Updates on Django's Async Story

0:00

01:04:56

Links Episode Deep Dive Transcript

For years, "Django and async" came with an asterisk. The docs themselves warned you off it. Scary performance notes, a story that felt half-finished. Well, that story just got rewritten, literally, and the person who rewrote it is here to tell you why the old framing was wrong.

Carlton Gibson is a former Django Fellow, sat on the security team for eight years, and he's on the steering council. On this episode we get into the async topic doc rewrite, what actually remains versus what was just fear, the new Tasks framework in 6.0, DB-level cascades and fetch modes landing in 6.1, and why free-threading is the bet that's about to pay off big for Django. If you've been told Django's async story isn't ready, this is the episode that puts that myth to bed.

Play on YouTube

Watch the live stream version

Episode Deep Dive

Guest Introduction and Background

Carlton Gibson is one of the most recognizable voices in the Django world, and this episode finds him at the center of a story he personally helped rewrite. Carlton is a former Django Fellow (he held that role for five years and stepped down in 2023), a current member of the Django steering council, and he served on the Django security team for eight years. On top of that, he maintains a large collection of packages across the Django ecosystem and builds real web applications for a living, so he speaks from the perspective of someone who both shapes the framework and has to ship with it.

English by birth and now living in Spain, Carlton has been on a busy conference run: a JetBrains PyTV livestream event in Amsterdam in March, DjangoCon Europe in Athens, and PyCon Italia in Bologna. He co-organizes Django on the Med with Paolo Melchiorre, a multi-day core-developer sprint in Italy. He co-hosts the Django Chat podcast, and lately he has been building applications with HTMX, Alpine, and Django's template partials, while working on two headline efforts we dig into here: the rewrite of Django's async documentation and a new serialization project called Django Mantle. In short, if you want an informed, honest read on where Django's async story really stands, Carlton is exactly the person to ask.

github.com/carltongibson: Carlton's open-source work and packages.
djangochat.com: The Django Chat podcast Carlton co-hosts.
djangoproject.com: The Django web framework.

What to Know If You're New to Python

This is an intermediate-to-advanced episode about web performance, concurrency, and how a mature framework evolves. You will get the most out of it if you understand a few mental models around how Python servers handle many requests at once and how Django talks to a database.

Synchronous vs. asynchronous code (async and await): Synchronous code runs one line at a time and waits for slow operations (like a database call) to finish before moving on. Asynchronous code uses async and await so a single worker can start a slow operation, do other useful work while it waits, and come back later. The entire episode is about when this trade-off is worth it in Django.
WSGI and ASGI: These are the two standard interfaces between a Python web app and the server that runs it. WSGI is the classic synchronous model (one request per worker at a time), and ASGI is the newer async-capable model that can hold many connections open on a single event loop. Carlton explains exactly when you should move from one to the other.
The GIL (Global Interpreter Lock): Python has historically allowed only one thread to execute Python code at a time, even on a multi-core machine. This is why CPU-heavy work does not automatically get faster with more threads, and it is the backdrop for the "free threading" excitement in this episode.
The ORM and the N+1 query problem: Django's ORM lets you work with database rows as Python objects. A common performance trap is the N+1 (or M+1) problem, where looping over a list of objects and touching a related object triggers one extra query per item. Django 6.1's fetch modes are a direct response to this.
Background task queues: Some work (like sending an email) should not make the user wait for the HTTP response. A task queue accepts the job, returns immediately, and lets a separate worker process finish it. Django 6.0's new Tasks framework standardizes this pattern.

Key Points and Takeaways

Django's async story is ready, and the documentation finally says so The central news of this episode is that Carlton rewrote Django's asynchronous support topic guide, and the reframing matters. That guide was first written when async support landed in Django (back in the Django 3.x and Python 3.6 era), and it was strikingly negative: it warned that the work was unfinished, not ready, and carried terrible performance implications unless you did everything perfectly. Carlton's point is that this framing is now wrong. The async API surface is effectively complete, with async support across the ORM, cache framework, authentication, sessions, and signals. Old performance warnings were downgraded to notes, because the costs they described have largely evaporated. The story was, quite literally, rewritten to stop scaring people away from something that works.

docs.djangoproject.com/en/stable/topics/async/: Django's asynchronous support topic guide.
djangoproject.com: The Django web framework.

Fetch modes take on the N+1 query problem in Django 6.1 When Carlton asked people what they were most excited about in Django 6.1, most pointed to fetch modes as their headline feature. The problem they solve is the classic N+1 (or M+1) query: you fetch a user, iterate over that user's bookmarks, and each access lazily fires another query. That laziness is fine at small scale but becomes a performance nightmare as you grow. The new "fetch peers" mode lets you say that if you touch the first element in a set of related objects, Django should go and fetch them all at once, effectively a deferred prefetch_related. This is easier to maintain than hand-adding prefetch_related everywhere, which tends to rot as code changes and you end up prefetching data you no longer use. Carlton notes a DjangoCon Europe talk from new Fellow Jacob Tyler Wells that walks through how fetch modes work and where they are not a magic bullet.

docs.djangoproject.com/en/dev/releases/6.1/: The in-development Django 6.1 release notes.

Database-level cascades land in Django 6.1 This is Carlton's personal favorite feature, and it fixes a real scaling landmine. When you delete a record that other rows depend on (say, an organization with team members and projects), Django's on_delete cascade has historically done that work in memory: it walks the object tree, pulls the objects into Python, then deletes leaves, then stems, then branches. At scale, deleting one organization that references tables with millions of rows can exhaust memory and take the whole application down. The database is dramatically better at this work than Python is, so Django 6.1 adds the option to push the cascade down to the database level. The ticket had been open for years because it was genuinely hard, and Carlton credits an in-person group at Django on the Med, including Mariusz Felisiak, Simon Charette, Lily Foote, and the new Django Fellow, for finally getting it over the line.

docs.djangoproject.com/en/dev/releases/6.1/: The in-development Django 6.1 release notes.

The built-in Tasks framework makes background work pluggable (Django 6.0) Carlton is excited about Django's new Tasks framework, and the motivating example is the everyday password-reset email. Historically you might send the email during the request and make the user wait for delivery before returning the HTTP response, which is a poor experience. The better pattern is to accept the request, write a task, return immediately, and let a separate worker send the email. People have done this for years with Celery, RQ, Django Q, Django Q2, or Huey, but neither Django itself nor third-party packages could safely depend on any single one of those. Django Tasks provides a backend-agnostic API for enqueuing tasks and checking results, so package authors can offer background work while the user picks the backend in settings. Backends already exist for Redis and for the ORM (storing tasks in your database), with a Celery adapter tracked as an open issue.

docs.djangoproject.com/en/dev/releases/6.1/: Django release notes covering the Tasks framework rollout.
github.com/celery/celery: Celery distributed task queue.
python-rq.org: RQ, a simple Redis-backed task queue.
github.com/coleifer/huey: Huey, a lightweight task queue.

Free threading is the bet that's about to pay off Carlton frames free-threaded Python as the big, exciting shift, and he argues it vindicates a Django design decision that once looked clumsy. Django's approach of pushing synchronous work into a thread seemed awkward while the GIL meant threads could not truly run in parallel. As free threading matures and works its way through the ecosystem, that bet ages very well, because Django always really wanted to do CPU work in a thread and now that thread can deliver actual parallelism. The practical payoff is running everything in one process instead of spinning up many processes just to get parallelism, which cuts memory overhead substantially. Trying it is now trivial: with uv you can run a command like uv python install 3.14t and have a free-threaded Python to play with in seconds. Carlton stresses that the remaining work is mostly ecosystem-level thread safety (module-level caches, library internals, running test suites), not framework architecture, which is already in place.

docs.astral.sh/uv: uv, which can install free-threaded Python builds.
peps.python.org/pep-0703: PEP 703, making the GIL optional in CPython.

The hard parts of async are Python's, not Django's One of Carlton's clearest messages is that the genuine pain points in async are structural Python issues shared by every framework, not Django-specific flaws. CPU-bound work either blocks the event loop or gets thrown to a thread where it hits GIL contention, and pushing it out to a separate process is heavy because you have to bootstrap the process and marshal the data. Database parallelism is the real ceiling: each connection is an ordered pipe that takes a query and returns a response, so whether you use threads (as Django does) or greenlets (as some other ORMs do), you are limited by your connection pool. Django's connections are also tied to a thread local, so you cannot await inside an atomic transaction block, though you can still run a transaction by wrapping it in a single function called through sync_to_async. Carlton points out you would meet the very same GIL, concurrency, and database-scaling problems in FastAPI or Flask. That is actually reassuring: these are general problems the whole community shares.

github.com/django/asgiref: The asgiref library that provides sync_to_async and async_to_sync.
peps.python.org/pep-0703: PEP 703 on free-threaded CPython.

WSGI vs. ASGI: when you actually need to switch Carlton gives a crisp rule for when to move from WSGI to ASGI, and the answer is not "as soon as you write an async view." You can run async views under WSGI just fine. The real trigger is long-lived requests: server-sent events, WebSockets, or streaming responses where you hold a connection open, like a stock ticker pushing updates every few seconds. On the WSGI model, each held-open connection consumes a worker, and pre-forked workers each live in their own process with all the memory overhead of loading Django, priming caches, and setting up database connections. An ASGI worker can hold many connections open on a single event loop instead. For larger apps, Carlton likes a hybrid deployment: keep the core app on well-understood WSGI scaling and run a small ASGI sidecar, routing async endpoints by path in Nginx (for example, everything under an /a prefix).

gunicorn.org: Gunicorn, a common WSGI server with pre-forked workers.
nginx.org: Nginx, used here to route by path to a WSGI or ASGI backend.

The function coloring problem is not a dead end Michael and Carlton tackle the common complaint that async is "viral," the idea that one async function forces every caller above it to become async too. Carlton names this the function coloring problem: once you have a red (or blue) function, callers have to match its color, much like a GPL license spreading through a codebase. But you are not trapped. You can cap the propagation by dropping back to synchronous code with async_to_sync, or by spinning up your own asyncio event loop to run just the async part. The cleanest illustration is also one of the easiest ways into async with Django: a single synchronous view that needs to call five external APIs can fire all five concurrently in an event loop and aggregate them into one response, finishing in a fraction of the time it would take to call them one after another. It is a clear win with the complexity contained to a single view.

journal.stuffwithstuff.com/2015/02/01/what-color-is-your-function: The essay that popularized the function coloring metaphor.

A real-world win: fewer, more capable workers save memory Michael shares a concrete case that grounds the theory. While trying to shrink his servers' memory footprint (roughly nine gigabytes of RAM spread across about 27 containers), he found that moving more request handling onto an ASGI server with genuinely async calls down to the database let him run fewer workers. Because each async worker was more capable and did not get bound up waiting on slow I/O, he was able to cut around 500 megabytes off a single web app just by reducing worker count. Carlton adds that the memory savings compound because you avoid duplicating shared initialization, cache priming, and connection pooling across many processes. The takeaway is that you do not need Instagram-level traffic to benefit; even at modest scale, doing more per box saves on DevOps, complexity, and money. One of his async deployments used the Granian server for this exact reason.

github.com/emmett-framework/granian: Granian, the Rust-based Python server Michael used.

Django Mantle: a modern, type-safe serialization story Django Mantle is Carlton's take on what serialization in Django should look like today. His argument is that Django REST Framework, while wonderful and battle-tested, has last-generation serializers that loop over model fields and pull values one at a time, which is slow by modern standards. Newer libraries like attrs, cattrs, msgspec, and Pydantic generate code that runs almost as fast as hand-written extraction. The catch is that Pydantic is not ORM-aware, so Mantle acts as a type-safe wrapper around Django's queryset core: you define attrs "shape" classes, instantiate a query, and get back fully typed objects populated in a single efficient fetch, with only the needed fields selected and prefetching handled to avoid N+1. Carlton describes these shape classes as "static islands" sitting on top of Django's powerful dynamic ORM, giving you clean, typed, separated domain logic without ripping out the dynamic patterns underneath. It grew out of the reality that a Django model naturally accretes display, aggregation, and validation logic until you need more structure, and Mantle offers a path to grow into that structure rather than over-engineering from day one.

django-rest-framework.org: Django REST Framework, the incumbent serialization toolkit.
attrs.org: attrs, the class-building library Mantle's shape classes use.
github.com/jcrist/msgspec: msgspec, a fast serialization and validation library.
docs.pydantic.dev: Pydantic, a widely used data-validation library.

Before you reach for async, fix your database indexes Michael puts a public-service announcement into the episode: if your requests feel slow and your instinct is to add parallelism, first spend an hour on your database indexes and query plans. Database indexes are, in his words, magic, and making sure every hot path uses one often solves the problem you were about to throw async at. Carlton reinforces the point from the internals: when Django makes a database query there is a little CPU time to compile the query, then the I/O of talking to the database, then CPU time to build the result rows into ORM objects. Crucially, psycopg already releases the GIL during that I/O wait, so you are getting concurrency there whether you are on async or not. More often than not, it was the database all along that was taking the time, and no amount of extra workers or async will fix a missing index.

psycopg.org: psycopg, the PostgreSQL adapter that releases the GIL during I/O.

A customizable multipart parser, and why release notes matter Carlton highlights a smaller Django 6.1 feature that shows how targeted optimizations reach the people who need them: you can now customize the multipart parser class on the request object. For most developers this is irrelevant, but if you handle large multipart requests with many parts, swapping in a different parser can save around 30 percent of the CPU spent parsing each request, which for some workloads is a decisive win. What excites Carlton more is the direction it points: opening up the request implementation so people can swap components, benchmark alternatives, and turn performance knobs that used to be locked away. It is also a reminder of the broader habit he recommends, because Django ships a long release note every eight months and it is worth reading through to pick out the features that matter to you.

docs.djangoproject.com/en/dev/releases/6.1/: The Django 6.1 release notes to skim for your own favorites.

Typing, "fit your brain," and the AI wrinkle Michael and Carlton have a thoughtful exchange about Python typing that avoids the usual tribalism. Carlton's core value is that Python should "fit your brain," and his grumble is that typing can push code past that point. He tells a story from PyCon Italia of five typing experts, genuine gurus, sitting around a table trying to work out the single correct type hint, which is hardly the picture of something that fits your brain. He notes that the original proposal that added type hints promised they would never be compulsory, even by convention, yet there is now real cultural pressure to adopt typing whether or not it helps you. AI adds a twist: models frequently write typed Python by default, and both agree that the type safety you get from a checker genuinely helps you (and the AI) verify that generated output is correct, which is an increasingly important benefit regardless of your personal taste.

docs.python.org/3/library/typing.html: Python's typing module documentation.

Dark Matter Django: an invisible community of millions In one of the episode's more memorable riffs, Michael coins the term "dark matter Django" to describe how much of the community is effectively invisible. Carlton cites a best estimate from Jacob Kaplan-Moss (given at DjangoCon US in 2024) of roughly four million active Django projects in the world, while GitHub's public "used by" count shows around two million repositories, and that excludes private and internal apps entirely. The truly visible community, the people who subscribe to the newsletter, answer the survey, or show up at DjangoCon, is only about four or five thousand, roughly a thousandth of the user base. Both agree that download statistics from PyPI have become an unreliable proxy in the age of Docker builds, CI pipelines, and layered caching, where one real install can register as many or as few. An old idea to add telemetry to Django went nowhere, for the obvious reasons, leaving the project to estimate the size of a community it mostly cannot see.

jacobian.org: Jacob Kaplan-Moss, whose estimate anchors the discussion.
github.com/django/django: Django on GitHub, where the "used by" count appears.

In-person sprints, the beta, and the developer survey Carlton makes a strong case that in-person work still moves Django forward in ways remote work cannot. Django on the Med, a three-day sprint he co-organizes with Paolo Melchiorre (this year in Pescara, Italy), is where the long-stalled database-level cascades feature finally got over the line, and where disagreements that had rolled on the features board for months got resolved in five-minute conversations. He talks candidly about how COVID knocked the community back by roughly half a decade, and how it is only now recovering with fresh contributors and obvious new steering-council candidates. The event is open to everyone, not just core developers, and is designed to pair experienced and new contributors. Carlton closes with two direct calls to action: install the Django 6.1 beta and run your test suite against it so regressions get caught before release, and take a few minutes to fill out the Django developer survey, which genuinely helps steer the project.

djangocon.eu: DjangoCon Europe, the conference whose sprints inspired the event.
djangoproject.com: Home of the Django developer survey and beta downloads.

Interesting Quotes and Stories

"It looks as if Django's bet is going to pay off. That bet will have aged very well, because we always really wanted to do CPU work in the thread." -- Carlton Gibson

"This idea that the topic guide presented, that somehow the story wasn't finished, well, that wasn't quite right. So we wanted to update that." -- Carlton Gibson

"The problems that I see come up time and time and time again are problems with Python and Python's async story that we all share, not Django-specific problems." -- Carlton Gibson

"It is an order of magnitude more complicated to correctly handle async code than to just code synchronously. These are really complex patterns; grow into them rather than jumping headfirst." -- Carlton Gibson

"Celery is truly wonderful software, but it's literally overkill for 99% of projects, and it's operationally difficult to keep going. You end up running three or four different things." -- Carlton Gibson

"If slow is a problem, please just look at the query plans, and make sure every hot path is using a database index. Database indexes are magic." -- Michael Kennedy

"It was your database all along that was the thing that was taking the time." -- Carlton Gibson

"As your application grows, the complexity grows with it, and you need more structure to manage that complexity. That's the joy of crafting the application over time. That's why we program." -- Carlton Gibson

"There's something about the bandwidth of face-to-face communication which just isn't replicable, which you can't replicate." -- Carlton Gibson

The "dark matter Django" moment is worth calling out as its own small story. Riffing on how Django's real user base (estimated around four million active projects) dwarfs the few thousand people who are actually visible through newsletters, surveys, and conferences, Michael offered Carlton a new term for the enormous, unseen majority of Django users. Carlton's immediate reaction captured the delight of naming something everyone had felt but never labeled:

"I have a new term for you, Carlton: dark matter Django." "Brilliant, I'm having that. Let me write that down: dark matter Django." -- Michael Kennedy and Carlton Gibson

There is also a nice bookend on how the framework evolves. Carlton points out that the database-level cascade feature, an old and famously hard ticket, only shipped because a handful of core contributors sat in the same room at Django on the Med. As he put it, it was "literally something which would never have happened except for the time that was available at Django on the Med," a concrete argument for why in-person sprints still matter.

Key Definitions and Terms

sync_to_async and async_to_sync: Helper functions from the asgiref library that bridge synchronous and asynchronous worlds. sync_to_async wraps a normal function so it can be awaited (running it in a thread), and async_to_sync lets synchronous code call an async function, which is how you can cap the "viral" spread of async.
Coroutine: A special function defined with async def that can pause at await points and resume later. Coroutines are the building blocks that let a single thread juggle many in-flight operations.
Event loop: The scheduler at the heart of asyncio that runs coroutines, pausing one whenever it awaits something slow and running another in the meantime. By default it is single-threaded, which is why CPU-heavy work can block it.
Server-sent events (SSE): A technique for pushing a stream of updates from server to client over one long-lived HTTP connection, used for things like live stock tickers. It is a prime example of when moving to ASGI and async is worthwhile.
select_related and prefetch_related: Django ORM tools for loading related objects efficiently in as few queries as possible, so you avoid the N+1 problem. Django 6.1's fetch modes build on the prefetch idea by triggering it automatically.
Active Record pattern: A design where a single object is both a database row and the home for behavior (a Django model is the classic example). It is convenient early on but tends to accumulate too many responsibilities as an app grows, which is the problem Django Mantle addresses.
Greenlets: Lightweight cooperative "threads" managed in user space rather than by the operating system, used by some ORMs to handle concurrency. Carlton mentions them to note that they still hit the same database-connection ceiling as Django's thread-based approach.
Deferred field: A Django ORM object that stands in for a model attribute and returns the converted value when accessed via the descriptor protocol. It is one reason static typing over the ORM is awkward, since the declared type and the runtime value differ.
Template partials: A Django templating feature that lets you define and render named fragments of a template, which pairs especially well with HTMX for swapping small pieces of a page.

Learning Resources

If this conversation made you want to go deeper on async, Django, or building interactive web apps the way Carlton describes, here are a few Talk Python Training courses that map directly onto the episode's themes. They range from Python's full concurrency toolkit to getting productive with Django itself.

training.talkpython.fm/courses/python-concurrency-deep-dive: Async Techniques and Examples in Python covers async and await, asyncio, threads, multiprocessing, and the GIL trade-offs discussed throughout this episode.
training.talkpython.fm/courses/getting-started-with-django: Django: Getting Started teaches the framework fundamentals, including the ORM, views, and templates, so the 6.1 and async topics here make sense in context.
training.talkpython.fm/courses/htmx-django-modern-python-web-apps-hold-the-javascript: HTMX + Django matches Carlton's enthusiasm for HTMX, Alpine, and template partials, and the server-driven, real-time patterns that async unlocks.

Overall Takeaway

The heart of this episode is a myth-busting: for years Django's own documentation told people its async story was unfinished and slow, and Carlton Gibson's rewrite corrects the record. The async API is complete across the ORM, cache, auth, sessions, and signals; the old millisecond overheads are now measured in microseconds; and the genuinely hard parts, GIL contention, CPU-bound work, and database connection limits, are Python-wide realities that every framework shares, not Django failings. Layered on top of that are real reasons for optimism: Django 6.1's fetch modes and database-level cascades attack long-standing ORM pain points, the 6.0 Tasks framework finally makes background work pluggable without marrying you to Celery, and free threading looks set to vindicate Django's long-held bet on doing work in threads.

The inspiring throughline is one of pragmatic craftsmanship. You do not have to adopt async, typing, or heavy infrastructure to be a "real" Python developer; you grow into complexity only when your application earns it. Check your database indexes before you reach for parallelism. Start with a single ASGI worker or a small sidecar before you rewrite everything. Reach for Django Tasks and the humble ORM queue before you stand up a Celery cluster at 2 a.m. Django is ready for ambitious, high-performance, real-time applications, and the smartest way to get there is gently, one well-understood step at a time.

Links from the show

DjangoCon Europe: djangocon.eu
PyCon Italia: pycon.it
Django on the Med: djangomed.eu
Django Mantle: noumenal.es
PyPI: pypi.org
release notes: docs.djangoproject.com
on_delete: docs.djangoproject.com
Fetch modes: docs.djangoproject.com
HttpRequest.multipart_parser_class: docs.djangoproject.com
async topic doc: docs.djangoproject.com
docs: docs.djangoproject.com
DEP 14: github.com
django-tasks: github.com
django-tasks-local: github.com
Celery: docs.celeryq.dev
PEP 703: peps.python.org
free-threading HOWTO: docs.python.org
PEP 779: peps.python.org
ASGI: docs.djangoproject.com
PGBouncer: www.pgbouncer.org
Channels: channels.readthedocs.io
sync_to_async / async_to_sync: docs.djangoproject.com
noumenal.es: noumenal.es
Django Chat: djangochat.com
@carlton@fosstodon.org: fosstodon.org
Article: Cutting Python Web App Memory Over 31%: mkennedy.codes

Watch this episode on YouTube: youtube.com
Episode #556 deep-dive: talkpython.fm/556
Episode transcripts: talkpython.fm

Theme Song: Developer Rap
🥁 Served in a Flask 🎸: talkpython.fm/flasksong

---== Don't be a stranger ==---
YouTube: youtube.com/@talkpython

Bluesky: @talkpython.fm
Mastodon: @talkpython@fosstodon.org
X.com: @talkpython

Michael on Bluesky: @mkennedy.codes
Michael on Mastodon: @mkennedy@fosstodon.org
Michael on X.com: @mkennedy

Episode Transcript

Collapse transcript

WebVTT format On GitHub

00:00 For years, Django and Async came with an asterisk.

00:04 The docs themselves warned you off it.

00:06 Scary performance notes.

00:08 A story that felt half-finished.

00:10 Well, that story just got rewritten, literally.

00:12 And the person who rewrote it is here to tell you why that old framing was wrong.

00:17 Carlton Gibson is a former Django fellow, sat on the security team for eight years,

00:22 and he's on the steering council.

00:24 On this episode, we get into the Async topic doc rewrite.

00:27 what actually remains versus what was just fear and the new tasks framework in 6.0,

00:33 DB-level cascades and fetch models landing in 6.1, and why free threading is the bet it's about to pay off big for Django.

00:41 If you've been told that Django's async story isn't ready,

00:44 this episode puts that myth to bed.

00:46 This is Talk Python To Me, episode 556, recorded July 2nd, 2026.

01:09 Welcome to Talk Python To Me, the number one Python podcast for developers and data scientists.

01:14 This is your host, Michael Kennedy. I'm a PSF fellow who's been coding for over 25 years.

01:20 Let's connect on social media.

01:22 You'll find me and Talk Python on Mastodon, Bluesky, and X.

01:25 The social links are all in your show notes.

01:27 You can find over 10 years of past episodes at talkpython.fm.

01:31 And if you want to be part of the show, you can join our recording live streams.

01:35 That's right.

01:35 We live stream the raw uncut version of each episode on YouTube.

01:39 Just visit talkpython.fm/youtube to see the schedule of upcoming events.

01:44 Be sure to subscribe there and press the bell so you'll get notified anytime we're recording.

01:48 This episode is brought to you by Sentry.

01:50 Don't let those errors go unnoticed. Use Sentry like we do here at Talk Python. Sign up at

01:55 Talk Python.fm slash Sentry. Carlton, welcome back to Talk Python To Me. Amazing to have you here.

02:01 Thank you, Michael. Thank you for having me on again.

02:03 Always fun to talk with you. Fun to have you back. We're going to talk about one of my favorite

02:08 topics, async, both for the reasons it's amazing and the reasons it's like, why? Why is it this way?

02:15 Why do we do this to ourselves is the thing, right?

02:18 Exactly.

02:20 Wait, I want to say, oh, but I didn't want it like that.

02:22 No, just kidding.

02:24 Specifically with regard to Django, you did some really interesting updates with Django

02:29 that we're going to talk about.

02:30 But first, your teams, they're still in the World Cup.

02:33 How are we doing?

02:34 Oh, yeah.

02:34 Well, so I'm British by birth, so English by birth.

02:37 So I was supposed to support England, but I live in Spain.

02:40 So depending on who's winning, who's doing better, I get to flip my allegiance.

02:44 you know i think that's really that's really amazing because you if one of your team's not

02:50 doing well you can just like i'm gonna just cheer for the other one it's it's great you get like two

02:54 shots at the thing yeah i mean in the last uh european championships we had the england spain

02:59 final so i couldn't lose really yeah that's amazing yeah so the u.s team um advanced yesterday as well

03:06 they played really well so i'm very very excited for that yeah it's lovely to see new new nation

03:11 emerging nations do well in the World Cup.

03:12 You know, a long time ago, that would have been like some kind of joke.

03:15 People are like, ah, surely you're joking.

03:17 Surely you're joking about this.

03:19 But honestly, soccer or football, non-American football, is popular in America.

03:25 So super cool.

03:26 Glad to hear it.

03:27 And you're back.

03:29 So tell people just what have you been up to?

03:32 You know, not everyone listens every episode.

03:34 So a quick introduction for yourself, maybe.

03:36 Okay.

03:36 So my name is Carlton.

03:38 I've long time worked with Django.

03:40 I'm a former Django fellow, did that for five years.

03:42 I'm on the Django steering council.

03:44 I was on the security team for eight years.

03:46 I maintain a whole load of packages in the ecosystem.

03:48 So, you know, I'm quite vested there.

03:51 And, yes, you know, basically I use Django and have done for, I don't know, however many years to build web applications to pay the bills.

04:01 Had a busy spring.

04:02 I was everywhere I went to.

04:04 There was a Pi TV event organized by JetBrains in Amsterdam in March that I went to.

04:09 That was all on YouTube.

04:10 There was, I don't know, 11 hours of live streaming or something.

04:13 It was amazing.

04:14 Then I was in DjangoCon Europe in Athens in, I think, May.

04:19 No, April.

04:20 And then in May, I was at Paikon Italia, which is in Bologna.

04:23 Paikon Italia really is a lovely conference.

04:26 You should make your way there.

04:27 And now I'm just home and I'm exhausted.

04:30 I had a quick holiday with the family, but now I'm hiding at home until September,

04:34 where we've got Django on the Med, which is a three-day sprint event,

04:37 which is going to be in Pescara, Italy, which we're looking forward to.

04:40 That sounds really, really nice.

04:43 I love traveling, but the jet lag kind of blows me out eventually.

04:47 I'm like, ah, this is 11 hours jet lag.

04:50 I don't really want to do this anymore.

04:52 It sounds to me like you're kind of able to keep it in the broader Euro neighborhood, which is nice.

04:57 Yes, I was.

04:58 I was, which it does help because there isn't that massive time change.

05:01 But it's still hard work, right?

05:03 Even though you have fun at these events, even though you're seeing your friends,

05:06 even though it's, you know, doing what you're vested in.

05:09 It's their long days and you have to be full on, you know, in action all day, each day.

05:15 And so by the time you get home, you are, you know, kind of exhausted.

05:17 And then obviously you've been away and you get home and, you know, you need to be present for your family too.

05:22 So there's a limit to how many of these I can do.

05:24 So I won't do any other events this year because it's just too much.

05:28 Yeah, that's good.

05:28 Just too much.

05:29 Yeah.

05:29 I remember traveling when we had little kids and that was a whole nother level because you'd come back.

05:33 It's like, you've been gone for a week, so I need some help.

05:37 But I'm exhausted.

05:38 You don't understand.

05:39 But at the same time, it's a fair request, you know?

05:41 Yeah, no, absolutely.

05:43 Absolutely.

05:43 I mean, our kids are older now.

05:45 The eldest is 18, or will be 18 next week, just about to go to university, and then 16,

05:50 and 13, 14.

05:51 So they are a lot easier than they used to be.

05:53 But still, it's a big ask on my partner to go away and they're alone for the whole week.

05:58 Absolutely.

05:59 In that point position, we might call it.

06:03 Yes, exactly.

06:04 so uh tell people you mentioned django on the med yes i talked to paulo about this a couple episodes

06:09 ago but yeah tell me okay you're going to be part of this too yeah yeah yeah so paulo and i

06:14 this was paulo's idea several years ago to have core developer sprints for django um because at the

06:19 the end of jank on europe or the end of jank on yes we always have a couple of days of sprints

06:23 and they're some of the best parts of the conference um but also it's around the time

06:28 people are leaving and your people are already kind of exhausted you know having the three days

06:32 of talks plus then of the sprints so it's like we wanted more and um we decided to to do this event

06:39 and we were like yeah this is a good idea but it didn't happen and then i think dublin a year a

06:44 year and a half ago now uh we were like no come on let's like it so we said we're going to have

06:47 it in palo vijay which is my hometown um we did that last september and all last october sorry

06:52 beginning of october and we're doing it in paulo's hometown pescara this september and it's as i say

06:57 It's three days to get together and work on Django.

06:59 And we had some really big features landed.

07:02 Like, for instance, Django 6.1 is in pre-release now.

07:06 And it's adding an ability to have DB-level cascade options for the on-delete.

07:10 So on-delete is, well, you've got a foreign key, so I've got a user, and that user has a bookmark.

07:16 Well, what happens when I delete the user?

07:18 What happens to the bookmark?

07:19 It's got a parent relationship.

07:21 And so normally you would cascade that.

07:23 Now, Django's always offered this option, but historically, it's done that in memory.

07:28 In Python, it's pulled, it's walked the tree, and it's got the objects, and then it's deleted the leaves, and then the stems, and only finally down to the branches.

07:37 And now, that can be very costly at scale.

07:41 I'm going to delete this one record that references several tables with a million entries.

07:46 That could be a lot of money.

07:47 Exactly.

07:48 You've got an organization that's got team members and projects, and you delete the organization.

07:53 it just brings down your application basically you get out of memory memory issues so the database

07:59 still has to do that work but the database is much much much better at doing that work than python is

08:05 um and so that's so anyway we've got this new option this ticket had been open for i don't know

08:10 so many years because it's it was really hard and we had marish feliciak who's a former django fellow

08:15 and one of the core um orm contributors simon charette is one of the core open source contributors

08:21 Lily Foote, Jacob Tyler-Wells, who's the new Django fellow. We had the four of them just there working

08:27 together and they were able to get this over the line. And it's literally something which would

08:32 never have happened except for the time that was available at Django on the Met. So we've got

08:38 exciting plans for this time. We're hoping to get people again, people from around the community to

08:42 come. And I know, I'm really excited about it. The first one was a proof of concept. It went really

08:47 well. The second one's lining up to be amazing too. And once we've done two, well, it's kind of

08:50 established and you know we're yeah well it's tradition you gotta do it yeah yeah exactly you

08:56 gotta go hang out with the mediterranean so i've done i've done a lot of remote work in my life

09:01 and i find you know when we do in-person meetups get the people who are all distributed together

09:06 it just changes the energy you know there's something about the bandwidth of face-to-face

09:12 communication which just isn't rec replicable record you which you can't replicate replica

09:18 online yeah thank you mike thank you my but yeah it's it's i don't know i mean look it works online

09:25 but as well we had another it was a good session as well so with um django we've got this um we've

09:30 had recently this new features board which was an attempt to take people suggesting features away

09:35 from django's track instance which is kind of old and scary and is a place for for the work to be

09:40 done and put it into a GitHub project and where we could have a project board and we could have

09:45 upvotes and we could have a Kanban board to see what were being advanced to try and give a bit

09:50 more visibility into the process. And sometimes there's disagreements and those disagreements

09:54 online, they can go on forever and ever, but we were able to have a session and there were

09:58 different voices from different opinions. And we were able to come to agreements in five minutes

10:03 on topics which had been rolling on the boards for months. And that kind of, look, I can see

10:10 why you're saying this. It's really hard to get that online. You know, there's money emojis

10:15 we use as much as we try and communicate openly it's just as much slack as you want to throw out

10:20 there and you know i mean since 2020 it's it's kind of been just oh yeah obviously remote is better

10:27 and so on and there are mega advantages don't get me wrong but it's not always better and in lots of

10:31 ways so yeah so i guess for this good gone you you mentioned 2020 and it's covid it's one thing that

10:37 um i was um django fellow over over that period and um you know it was a really hard time for

10:43 because i think that the pandemic really knocked the back out of the community not we had we

10:48 had older people stepping away we had young roots that were starting to come and then the put the

10:52 the pandemic came and it was kind of like this really really really tough period and it's now

10:58 five six years later that we're looking at it we're thinking actually now the community you know the

11:02 activity is going on with things like jang and on space and the new community the new members that

11:06 are coming through and the people who who are active in the community think well next year when the

11:11 steering council is re-elected. There's obvious candidates who weren't there two years ago.

11:18 And it's taken us, I think, half a decade to recover from the real knock that was COVID.

11:24 Wow. I believe it. I believe it. So is this for just for core devs or is this open to people who

11:30 just want to contribute? No, it's open to everybody. Obviously, if you're an established

11:36 contributor, it's going to be easy to get going. But it's also about bringing on new

11:41 and teaming people up.

11:43 And, you know, if you come and you've got a particular interest in Django,

11:48 if you're a total Django novice, it would be strange to come to choose Django on the Met

11:52 as your first event.

11:54 But if you're into Django and you know it and you want to get contributing,

11:57 it's, yeah, that's part of the plan too, is that we have both experienced and new contributors

12:03 and we can team people up and get people working together.

12:05 And, you know, part of the contributing to Django problem

12:09 is the learning curve.

12:10 So it feels like it's a big, scary thing.

12:12 Whereas if you sit down with somebody in a room and you work on what they're working on, you're like, actually, I could work on this side ticket.

12:17 And all of a sudden people are off.

12:19 Well, it suffers from what many of the successful projects out there do.

12:23 It's super polished.

12:25 It's used by millions and millions of projects.

12:27 It's hard to just look at it and go, oh, I'll just change this without regard to all that knock on effect, right?

12:33 Yes. And that's where part of the new features repo was about giving a space where people could put their ideas forward without having them immediately closed as won't fix.

12:45 Because if you went and, you know, oh, can we integrate React? No, we're not going to do that.

12:52 You can use React and here's how we use React, but we're not going to integrate it into JankRusate as an example.

12:57 And that would be open, it'd be no, no, close, won't fix.

13:00 And people felt that that was a little bit too...

13:03 A little too stack over flowy.

13:04 Well, yeah, it wasn't ideal.

13:06 So the new features repo is to allow space to develop for a conversation.

13:11 And then those ideas which have a bit of traction, we can say, no, do you know what?

13:14 A lot of people have said they support this one or we can advance it.

13:18 Hey, one second.

13:19 Normally, this would be an ad break from Sentry, but not this time.

13:23 Let's just thank them for supporting the show and get right back to the conversation.

13:27 Also, visit talkpython.fm/sentry after the show.

13:31 Thanks, Sentry.

13:32 You have another project coming along, The Mantle.

13:35 Tell us what this is.

13:37 Django Mantle.

13:38 So it's my take on a modern serialization story for Django.

13:43 So the big topic we've had is about REST Framework, and REST Framework is aging.

13:46 And REST Framework is wonderful, and it has been wonderful.

13:49 But the serializers in particular, they're kind of last generation.

13:52 And there are much more modern approaches available now, involved in Pydantic, in msgspec, in Attrs and Catters, which is what I use,

14:01 are much, much faster than they used to be.

14:04 They generate code which is almost as quick as you would see if you hand-wrote it.

14:09 If you wrote the code to literally pull this attribute,

14:12 this attribute, this attribute, the code that's generated for you by, say, caters

14:16 is as quick as that code, which is amazing.

14:19 Whereas REST Framework serializes, they go through and get the field object from the model.

14:24 They get the value from it.

14:26 It's like, hang on, all of those loopings are very slow by comparison.

14:31 So then the question is, well, why don't we just use Pydantic?

14:34 Well, Pydantic's fantastic, but it's not ORM aware.

14:37 It doesn't have any insight into how Django works or anything.

14:42 So Django Mantle is my take on that.

14:44 It's the type-save wrapper around Django's liquid core.

14:49 So you write these atters classes, and then you just instantiate a query object,

14:54 and you go fetch, and it fully instantiates your object to the shape you define,

15:00 and it fetches all the data all in one go, and it can do that for nested objects,

15:04 and you can do writes with validation, all these things.

15:07 And it's ORM aware in that it only selects the fields that it needs,

15:11 and it does the pre-fetch relateds for you so that you don't get this N plus one query problem, right?

15:17 And it's very good for performant fetches of just the exact data you want.

15:23 You end up with your objects being fully Python typed.

15:27 They're at its classes, so they're typed.

15:28 Your type checker knows the way around them.

15:30 mypy or PyRefly is now adding support for Atis classes too, I see this week.

15:36 Oh, cool.

15:36 So anyway, it's an attempt to show what a serialization story using these modern approaches

15:43 can look like.

15:44 And then from there, it's like, well, okay, what does an API framework in 2026 look?

15:48 I love it.

15:49 I love this idea.

15:49 I think there's a lot of value in taking some of these very focused, lightweight, dedicated

15:56 like data structures right like c attrs or pydantic or just straight data classes and somehow

16:03 integrating them in here without just this heavyweight stuff like you talked about the

16:07 eager loading that's and some of the other things like the typing can be weird as well like sql

16:12 alchemy you kind of have to lie to it yes somehow do you either lie about it being a database thing

16:18 or do you lie about it being an integer when it actually gets the value right like yeah yeah yeah

16:24 Yeah, because the actual type is a Django deferred field, right?

16:28 For a Django modifier, it's a deferred field.

16:30 What on earth is a deferred field?

16:32 Where it's an object that has the descriptive protocol

16:35 so that when you access it, it gives you the converted value.

16:38 I don't want to type that.

16:40 Exactly.

16:41 Even if you do, then you try to add it to an integer.

16:43 It's like, you can't add that to an integer.

16:44 That's not the same thing.

16:45 You're like, but it's going to be.

16:46 Yeah, it's going to be.

16:47 So this was the topic of my talk that I gave in Amsterdam

16:52 and Django in Europe and PyCon in Thailand.

16:53 It's slightly different versions because I had more time at DjangoCon Europe.

16:57 So I went more into the way we grow from, say, an active record pattern,

17:02 which is like a Django model, right?

17:03 It's an active record.

17:04 It's your database row, but it's also an object that's got logical behaviors

17:09 to a more separated approach where you're like, hang on,

17:12 I don't want my display logic and my aggregation logic

17:16 and my user validation logic all living on this one database class.

17:20 So how do we separate those off?

17:22 That's kind of where mantle came from originally.

17:24 But the idea was that, and my take is that Django's ORM is built on these very powerful dynamic patterns that Python allows.

17:33 And retrofitting static typing onto that is going to be a real problem.

17:36 So instead of that, that's also a dynamic C.

17:38 On top of that, we have these static islands where we can have all the nice type safety features of modern Python,

17:44 but we're not trying to eliminate the very powerful dynamic features that we have underneath.

17:50 And so the mantles, the shape classes that you define, they're your static islands.

17:55 And there you can add all your domain logic that you want to, and you can have it fully

17:58 separated from the database.

18:00 And they are just pure Python classes.

18:01 There's no data mutability in there.

18:03 There's no like, say, oh, I modified a property and saved, and now I've updated something.

18:09 No, there's none of that active record stuff going on.

18:11 So it's a kind of way of getting some of the benefits of this sort of separation layers

18:17 type approach whilst still keeping everything you love from the aura uh spoken like someone who

18:23 appreciates some nice architecture and layers are very cool well the issue is as your application

18:28 grows the complexity grows with it and you need more structure to manage that complexity like you

18:35 simply do like it the issue isn't that you started with a jango model it's that your jango model grew

18:40 and then grew and it grew and all of a sudden it's got it's got methods on it which are several

18:45 distinct responsibilities. And then you think, oh, I'm getting a lot of code here. So you start

18:50 doing dry, you start abstracting methods. But that abstracted method wasn't for a single

18:55 responsibility. So it's not clean so that you've got, ah. And everybody who's grown a Django app

19:01 has seen this. And it's perfectly natural and it's perfectly normal. It's just how applications grow.

19:05 And Mantle is kind of my take on, well, how do we deal with that? How do we grow from the active

19:10 record pattern that we began to, to something that's more robust. And it's not that you should

19:15 start with the more robust patterns because you're over-engineering them. There's a nice road we have

19:20 to walk between these two problems. And that's the joy of crafting the application over time,

19:25 right? That's why we're programmed. I love it. I love it. All right. Let's talk about Django 6.1.

19:35 Yes.

19:36 I think it's coming out.

19:38 It's not quite finished, but there's a lot of features we're going to talk about

19:42 that are pretty exciting and related here, right?

19:43 A lot of new things coming out of the database, for example.

19:45 Yes.

19:46 So I think it's in beta now.

19:47 So it's going to be a release candidate in a couple of weeks,

19:49 and then we're about a month away from the final release.

19:52 Oh, by the way, I see you've got the page up there.

19:54 The Django Developer Survey is on right now.

19:56 Do take it, folks.

19:57 We want as many replies as we can.

19:58 It really does help drive forward the development of Django.

20:01 So do give that five minutes there of your time to answer the questions.

20:05 But what's exciting?

20:07 Well, we already talked about database-level cascades, right?

20:10 That's my big feature because that's something that you just couldn't work around.

20:13 And there's no in Python solution to that.

20:16 We wanted that.

20:17 You'd have to, well, in the past, I have created a custom database backend and changed the way it creates.

20:23 No, that's not a sustainable way.

20:25 So database-level cascades are my favorite one.

20:28 The one that I think more people are excited about is called fetch modes, which are on query sets.

20:33 So we talked about the prefetch related with the way Mantle handles it.

20:37 But the idea is you've got an author and so you fetch a user and they've got a list of bookmarks.

20:43 And so you get the author, you get the user, and then you iterate for, bookmark in users.bookmark, do display the bookmark.

20:51 And so what happens is it goes through and you get this N plus one query where it goes and fetches the lazy related object.

20:57 And that's great at small scales and when you're getting going, but it's a performance nightmare, basically.

21:05 And this N plus one problem that comes up is something that people hit all and all again.

21:10 So fetch modes let you say, look, fetch peers, which is the exciting one.

21:13 It lets you say, if I access the first element in a list of related objects, go and get them all at once.

21:20 And so it's like a deferred pre-fetch related.

21:22 And that's going to be a lot easier to manage in a lot of cases than having to manually add the pre-fetch related in each case, because those often become difficult to do by hand.

21:33 They're all right to add the first time, but they become difficult to keep updated.

21:36 And then you're suddenly pre-fetch relating something which you're no longer using.

21:40 And so fetch mode.

21:41 Yeah, your data access layer, maybe you're right, says get me all the books.

21:45 But then you might have to add like some kind of flag that says and the authors.

21:50 Because sometimes you want to call that function and you want the authors, but sometimes you

21:53 want to just list the books, you know, and you don't need that.

21:55 And it's like, yeah, so this way kind of just lets you says, if you're ever going to do

22:00 this lazy loading, then basically upgrade it to an eager load.

22:04 Yes.

22:04 And there's, I don't think the videos are quite out yet, but Jacob Tyler was, you found

22:09 a great talk at DjangoCon Europe on the, on how fetch modes work.

22:12 They'll, they'll be out very soon.

22:13 I think they'll be on YouTube, but do look out for them in the interest, but how, how,

22:17 how fetchmates work and what the limitations were and you know where it's not you know it's not a

22:22 magic solution it's not just oh let's enable fetch peers in all cases you know without thinking about

22:27 it no but it it's a good change and um when i've spoken to people about what they're excited about

22:32 in 6.1 most people have pointed to this as their headline feature nice nice so that's good and then

22:39 i guess um a little bit yeah go ahead well i was going to say about the other thing that i'm

22:43 excited about that you know there's an old list of features the thing with django releases every

22:47 there's this big long release pack go and read it go and pick your favorite features but there's

22:50 always lots in there for everything but you we can now customize the multi-part parser class for

22:57 on the request object now whoever wants to do that well it turns out but that you can when you're

23:03 getting multi-part requests if you're getting big ones with lots of multi-parts in swapping out a

23:08 different multi-parts class could save you maybe 30 on your passing time which is 30 cpu on pass it

23:14 on request for a passing time, that may be a non-negagable game.

23:18 So I'm excited about that.

23:19 But I'm excited because it starts to open the pathway to more customizations on the request.

23:25 And so, you know, Django's got this implementation, but can we swap out that implementation and compare it

23:31 and see if we can get some performance gains by being able to turn some of these nods under the hood.

23:36 That's really nice.

23:37 And having it be customizable is really great.

23:39 I'm working on this.

23:40 I'm working with this project that I just want to use.

23:43 I don't want to be an author of it, but it has no real extensibility.

23:48 So anything I want to do, I have to write little custom bits and change the source code and run it.

23:52 And I'm like, as soon as there's a new version, how am I going to pay for this?

23:56 You know what I mean?

23:56 And so this is really nice.

23:58 Yeah.

24:00 Again, this is customizing the multi-part pass class.

24:04 I don't think that's for the majority of people.

24:06 But when you need it, it's going to be there.

24:08 And it's like, okay, actually, there's a big win.

24:10 So those are my favorites.

24:12 Yeah.

24:12 Okay.

24:13 What's the timeline?

24:14 It's in beta, you say?

24:15 It's in beta now.

24:16 So the pre-release is three months.

24:19 So the alpha was a month ago.

24:21 The beta is just now.

24:23 And then the release candidate will be in a week or so.

24:25 And then a couple of weeks after that will be the final.

24:27 So I guess August will be.

24:29 So it's coming fast.

24:31 Oh, yes, yes, yes.

24:32 It's very nearly there.

24:34 Excellent.

24:35 So just a call to action.

24:37 If you're using Django and you could just install it against your application and run the beta,

24:41 run your test suite against the beta.

24:43 That would be amazing because if it passes, then you're all good.

24:46 You're good to update.

24:46 If it doesn't pass, then we want to know that.

24:48 We want to know, well, what changed?

24:51 It's got to be such a challenge to work on a project that's used,

24:55 not just consumed by people, but as a programmer, code integrated with this project, right?

25:01 Yeah, yeah, yeah.

25:02 And there's the whole ecosystem, right?

25:03 So, I mean, why do people use Django?

25:06 One of the big reasons they use Django is for the third-party packages

25:09 that are available, CMSs or image processing packages or whatever it is that you want to do.

25:14 Did we break any integrations with them?

25:16 Ah, well, we hope not.

25:17 But unless people test and tell us, we don't find out.

25:19 And when I was a fellow for five years and in every release, we would have on release

25:24 day bug reports saying, oh, there's some regression here when we could have had those a week before

25:30 and fix them before the final release.

25:31 Yeah.

25:32 Have you looked lately what the...

25:36 I'll just do it like this.

25:37 Have you looked at what it says used by?

25:39 On GitHub here?

25:41 You know, it says, does it say how many projects?

25:43 I don't know.

25:44 Oh yeah, here we go.

25:45 This is just the public projects on GitHub, but two million?

25:48 Couple of million.

25:49 Yeah, right.

25:49 Okay, good.

25:51 There's so many that are private.

25:53 There's so many that have nothing to do with GitHub.

25:54 It's like a very small sample.

25:56 That's still crazy.

25:58 Jay, a couple of years ago at DjangoCon US, I think maybe 2024,

26:02 Jacob Kaplan Moss gave a sort of best estimate about the user base, about 4 million,

26:06 something like that as a kind of you know four million in action django projects in in the wild

26:12 so okay you know if you count all the internal apps given that two million public repos on gmail

26:19 if you count all the internal private things that people just threw up for like a little internal

26:23 company app i bet it's way more than four million now i obviously couldn't prove it but yeah i mean

26:28 i'd have to ask jacob how he came up with the figures but it was in terms of downloads and

26:31 looking at this and looking yeah yeah exactly yeah probably the big query data pipe yeah or

26:36 Yeah. Honestly, I feel like that stuff's kind of broken these days. Like 10 years ago,

26:40 that was pretty representative. But right now, I think it's kind of a little weird on both ends.

26:46 I think it's weird because there's so many Docker builds, right? This project can say,

26:51 well, it got a hundred downloads. Well, that's because we just built something in CI a bunch

26:56 of times today. We didn't really intend to download it, that many projects. But also at the same time,

27:01 there's a bunch of caching layers. Like you might grab a Docker image that has it or build a Docker

27:05 image and never reload that or use uv caching like it's just so opaque i think i mean a lot of people

27:11 run little local pi pi pi cache or pip caches sorry so when you do pip install you're not you

27:17 fetched it once you but you might have installed it a hundred times from that so yeah exactly what's

27:22 that meant to be counted as one or i i think it's why it's so hard i don't have no idea so anyway

27:28 um incredible and this obviously is right i was gonna say once upon a time there was a an idea

27:33 to add some kind of telemetry into Django.

27:35 And obviously that didn't go anywhere because no one wants to go anywhere near that.

27:38 But we still sometimes would like some idea about how we estimate

27:44 because the sort of visible Django community is like a thousandth of a percent.

27:48 So say Jacob's 4 million users isn't too far off.

27:51 The sort of visible community, the people who subscribe to the newsletter

27:54 or do the survey or come to Django cons or all these things,

27:58 they're about a thousandth of that, like 4,000 or 5,000 people

28:02 that we can sort of point to and go, I know who you are.

28:04 I know who you are.

28:05 Yes, I've heard of you.

28:06 Yes, I know you're on LinkedIn.

28:08 But that's a fraction of the total user base.

28:10 It would be lovely if we had some way of just reaching more.

28:14 I know.

28:15 I have a new term for you, Carlton.

28:16 Dark matter Django.

28:18 Yeah, brilliant.

28:18 I'm having that.

28:19 Let me write that down.

28:21 Dark matter Django.

28:22 Well, we're talking about stuff that is not dark matter.

28:27 So you recently did a bunch of work on the documentation and positioning of async in general for django yeah yes yeah so the topic

28:39 guide which is here that you've got up on the page the asynchronous support topic guide it was first

28:43 written when we very first added um async support to django back in like that jango 3. something or

28:51 i can't remember exactly but i think it was um python 3.6 it was was when it is and um async's

28:58 come an awful long way since then and it was very negative it was like oh we still haven't finished

29:04 this it's still in development it's still not ready and the performance implications are terrible

29:10 unless you do it 100 perfectly but you and there was regular back in um go back in um python three

29:17 five three six days it async and await and this async concept really had kind of a two to three

29:24 issue like i would even if i wanted to do it i got to call this library which is talking to the

29:30 internet or into the database but it's not async so what's the point yes yes and the costs of so

29:36 the the big thing we'll come and talk about a lot is threats so in um django uh we use a lot of what

29:42 of a helper um from the asgirf library called sync to async where we um you take async function and

29:48 you you wrap it into a call routine so that you can wait into a coroutine so you can await that

29:53 and it gets run in a thread and async io has the two thread thing which is the same um but essentially

29:58 the same option um sync to async has some ability to track which thread it was called from so for

30:05 instance database connections they're tied to a thread um and the transaction state is all tied

30:11 to that thread so it's important that database um queries are run on the same thread as the connection

30:17 came from. Otherwise, you're going to have problems. Sync2AdsAsync adds a few niceties

30:24 around that, but it's essentially the same as AsyncIO2 thread. Go and run this in an executor

30:29 on a separate thread. That's great, but that used to cost orders of milliseconds to bootstrap and

30:36 set up. With latest pythons, it's tens of nanoseconds, or tens of microseconds, sorry.

30:42 Tens of microseconds when you're doing milliseconds requests, they're not a significant thing,

30:46 But the documentation was like, no, this is going to slow down your requests.

30:49 And it's like, hang on, it's really not.

30:53 And the API is complete.

30:55 The ORM, the cache framework, authentication, session, signals, it's all got async API.

31:01 So from a user perspective, it really is all there.

31:03 It doesn't mean there's no work to be done.

31:06 It's an ongoing project.

31:07 We keep developing it.

31:09 We keep fixing new things.

31:10 We keep adding new things.

31:11 But this idea that the topic guy presented that somehow the story wasn't finished, well, that's not quite, that wasn't quite right.

31:19 So we wanted to update that.

31:20 And we wanted to take some of the warnings about performance and just downgrade them to notes.

31:24 You know, they're a bit overplayed in this latter day.

31:27 I agree.

31:28 I think the sync to async and back and forth is pretty interesting.

31:32 And more broadly, you know, one of the things people use as a, I don't really want to use async.

31:38 It's kind of bad.

31:39 they'll call it like a virus or something and a lot of times what they mean is like well you've

31:43 got some low level function it's async so in order to call that the thing above it has to be async

31:47 which means like the rest of your app like it just cascades you know it's sort of viral in the gpl

31:51 sense right um but yes that's a you don't have to live with that you can yeah go ahead sorry well

31:56 they call it function coloring problem right is that once you've got a blue function or red

32:01 function all the functions are called it have to be the same color yes but but you there's nothing

32:07 to stop you from going in a synchronous function just async to sync boom and like kind of just stop

32:13 the propagation there or create your own async io event loop and just run that one bit down below and

32:20 so you could have a synchronous function but it maybe calls five external endpoints and you could

32:24 do all that with async in parallel and just like run that but cap it right there i think there's a

32:29 lot more creativity that we can bring just just just say like oh it's virus either it's amazing or

32:34 we don't want it at all, you know?

32:35 Yes.

32:35 No, that example, that example you gave

32:38 of calling five, you know, five external things concurrently,

32:42 that's one of the easiest ways into AsyncVagenda.

32:45 So you've got your WSGI app, it's deployed,

32:47 it's got its, you've run Gunicorn, it's got, you know,

32:51 four workers running, saying handling requests,

32:53 each one in a single thread, nice and separated,

32:55 so they're not getting in each other's ways,

32:58 sorry, in a separate process, each kind of single threaded,

33:01 doing its thing.

33:02 And then you're like, I want to make some,

33:04 API requests.

33:06 So you can define a single async view and Django's handlers will run that

33:12 in an event loop for you and you can run, say, your five HTTP requests

33:17 and you can aggregate them together and you can return a single response.

33:21 And that can just be done in a fraction of the time of calling one and then the other.

33:26 Yeah, yeah, yeah.

33:27 And it's just so clearly a win, right?

33:31 Do you have to switch to an ASGI, an ASGI server mechanism if you're going to use async views in Django?

33:37 No, you can run async views under WSGI.

33:41 But what you don't get, this is a good question.

33:46 So say I'm deployed, say I'm a Django developer, when should I switch to ASCII?

33:51 When should I switch to async?

33:53 And the real question is if you want long-lived requests.

33:56 If you want to do, say, something where you hold open a connection and you, say, do service center events where every five seconds you get an update.

34:06 Like a stock ticker is a good example, right?

34:08 You get what was the latest stock price.

34:11 Okay, it keeps coming down the wire at you.

34:13 Well, you don't want to do that on the WSGI model because each connection is a worker on WSGI.

34:19 And so you say you've got four workers.

34:21 Well, that's great.

34:22 But then, well, hang on.

34:23 I've got three of those held open by clients.

34:25 well, now I've only got one serving requests.

34:28 Well, okay, double your workers.

34:30 But then you're going to double your users.

34:32 And if you're using pre-forked workers where each worker lives in a separate process,

34:37 that's a lot of times that your Django application is loaded into RAM and say the cache is primed

34:43 or the database connections are set up.

34:47 There's a lot of overhead to running multiple processes.

34:50 Whereas if you can do it with a NASGI worker, you can hold open that connection

34:54 and it can just be served on the event loop.

34:56 You can have many connections just for one worker.

34:58 It's basically tied to an event loop and then that can share that thread or that worker

35:04 as it interlaces across that one event loop, right?

35:06 Yeah, exactly.

35:07 And that's where the exciting things come.

35:09 If you've got long-lived connections, even if you're just doing polling,

35:13 you can do polling quite quickly, once a second, and you're not really going to overload your whiskey workers.

35:20 But if you're holding a connection open, Ah, at that point, Async and ASCII is really called for.

35:26 Yeah, that's cool.

35:27 So you can kind of have this progressive scale.

35:30 Yeah, yeah.

35:31 I mean, standardly, the people that, I mean, we have an awful lot of people doing very exciting things, you know, with Async.

35:39 They're either running small projects where a single ASCII worker is enough and it can handle enough connections and you don't actually need multiple processes.

35:49 Or they tend to be running at the moment a WSGI base for their app,

35:53 so a standard WSGI base, which is handling most of the work.

35:56 And then the async endpoints are just running in a single ASCII process

36:00 as a sort of sidecar.

36:01 And they do this because something we'll get into is you get what's called

36:06 you get kind of contention for CPU-bound work on the event.

36:11 And we can talk about that in a bit.

36:12 But I think that's the way people are deploying most successfully

36:17 at the slightly larger applications.

36:19 Yeah, that makes sense. And this contention, it is very interesting. It certainly is. I want it actually, I searched for the wrong thing. Hold on. That is, I wanted to actually show. I recently went through this whole process of like, how could I make my server use less memory?

36:36 brought and i had some of my web apps were running purely synchronous as whiskey and some of them i

36:44 had upgraded to like true async running on asgi on granian right and i'm just i was looking the

36:51 server i'm like it's using i don't know nine gigs of ram for all these problems like 27 containers

36:56 running but still it's a lot i'm like is there a way that i could just maybe reduce the workers

37:01 a little bit and make it chill.

37:03 And so I wrote this article about like going through all these different things.

37:07 But one of the things that I found really helpful was

37:11 if I could switch to more of the processes, more of the requests being processed by an ASGI server

37:19 with like truly async calls and down to the database layer and so on,

37:23 maybe I can use fewer workers.

37:25 And so I like, just off of like one of my web apps, I cut like 500 megs off because I could just reduce the number of workers because each one was more capable and didn't get bound up.

37:36 There's still this computational problem you have, but at least you're not doing like I'm waiting a second for this API endpoint, you know?

37:42 Yes, yes, yes.

37:44 I thought that was pretty interesting.

37:45 No, it is.

37:46 And it's like people, it's one reason why people run.

37:49 So I always run, not always, but I default to running, say, Gun & Corn with multiple workers, multiple processes as workers because it's separate and you don't get the guild contention issues.

38:00 But there is a great approach, depending on how much of this CPU work that you're busy doing and how the guild contention comes into play.

38:08 But if you run with the threaded worker, you've just got one process.

38:13 And you've got much less memory overhead from running one process than you have from running four.

38:18 Shared initialization and all that.

38:20 Connection pooling, everything.

38:22 Yeah, all of it.

38:23 But it's never as simple as, oh, this is the way to deploy.

38:27 How do you deploy your application?

38:29 how long you got and how many developers are we asking because we'll all give different answers

38:33 for all these really interesting reasons that trade off against each other yeah something i

38:38 would like to get your thoughts on is i feel like sometimes people say like why are we making python

38:43 so complicated like the magic of python was that it was simple and yet there's also another side of

38:49 that story where people's like well i have to switch to go because i can't i can't do this and so you

38:55 got to walk this tight balance of there has to be a simple story but you don't want there to not be

39:00 a nice upper bound or high upper bound because you're going to lose maybe your your biggest

39:05 proponents or your your biggest users right yeah what do you think about that um no i think look

39:11 python so you know i this fits my brain idea i think it's something i'm very into and one of my

39:17 sort of grumbles about the type about typing is that it becomes not pipe it fits your brain i mean

39:23 And, you know, I was told a story at PyCon Italia, I showed five typing experts sat around a table trying to work out the correct type in.

39:32 And these are the absolute gurus, right?

39:34 And that's not fit your brain, is it?

39:38 So part of me is like, ah, on that front.

39:42 But we have to be performant.

39:44 You don't want to have to change language just because you need to increase your throughput,

39:49 just because you're doing a more serious application.

39:51 That's not good for, that wouldn't be good for long-term sustainability of the project.

39:56 No, it definitely wouldn't.

39:58 And the fact that you don't have to add typing, you don't have to write async views.

40:03 I think that's part of where it's really good because you can grow into that.

40:07 There is space to grow into.

40:08 There is, though, a kind of pressure to the original DEP, which added type hints to Python.

40:16 said, we don't intend to make type hints compulsory, even by convention. But I think there is very much

40:23 people coming into the Python landscape. There is a cultural pressure to adopt typing, even if perhaps

40:29 it's not relevant to you. It's like, oh, no, but this is how you do it. And there is a you should,

40:35 which perhaps doesn't respect that original promise in the dead.

40:40 Fair. And I think AI throws a whole spanner into it as well. And let's not go too deeply

40:45 down this because we can spend the next three hours on but you know if you ask an ai to write

40:49 python a lot of times it'll do typed python now sure without asking yes and um the big issue

40:55 with um ai output is can you verify it and so you know type safety any type safety that you get from

41:02 the type checker is going to help the ai um verify the output or help you verify the output 100 i

41:09 think it's a big bonus to ai as well whether yes people care about that you know that's a different

41:13 deal. Indeed. So what's left? Where else do people need to focus? Okay, so performance realities is,

41:22 you know, DB parallelism is your limit, right? So each DB connection is a ordered, like,

41:29 it takes, you know, it takes a query and it gets a response. And so whether you're coming,

41:34 whether you're doing that in threads, the way Django does it, or if you're doing it with greenlets,

41:39 the way some other ORMs do it, you're still limited by the size of your connection pool.

41:45 Again, and that's whether you're doing it with WSGI or ASCII, the limit always has

41:49 been the number of database connections you could hold open at a time, basically.

41:54 So that doesn't really change anything.

41:55 If you want to scale up, you need to increase the size of your connection pool or handle

41:59 that more efficiently.

42:01 That's one thing.

42:02 Async transactions is something that, because of the way Django's connections are tied to a thread local,

42:11 we're never going to have async transactions in the sense that you can call await inside an atomic block.

42:19 Why not? Because that await call may not be on the same thread as the connection, right?

42:24 So the transaction state is held on the connection, which is on a thread,

42:29 and if that await call handles in a different thread, you're going to be in big trouble.

42:32 So it's always going to be the case with Django's choice that you need to take if you want.

42:36 It doesn't mean you can't have transactions in async.

42:38 You just wrap your transaction in a single function, which you then call within a single sync to async.

42:47 So that's always going to be the case.

42:49 Then there's CPU band work.

42:51 Look, if you do work on CPU, you either block the event loop or you throw it to a thread.

42:58 And then, you know, if there's a lot of CPU work going on, you've got GIL contention that we'll talk about momentarily.

43:03 Or you have to throw it out to a separate process.

43:05 But throwing work out to a separate process is just crazily heavy.

43:08 You've got to bootstrap that process.

43:10 You've got to marshal your data.

43:11 You've got to work out returns.

43:13 So, you know, async frameworks have tools to throw out to a separate process.

43:17 But it's not something you want to address lightly.

43:19 Really, really, threading was the way you wanted to go.

43:22 But you are going to hit the gill, essentially.

43:25 Yeah, yeah.

43:26 For people who don't know, if by default, when you use async and await on a standard event loop, that's single threaded.

43:35 Even though stuff is happening, it's literally single threaded.

43:38 And the way that it time slices it or preemptively shares it is anytime you hit an await block, like on a database call or whatever, it's like, okay, that's waiting.

43:48 So now the same thread can go process something else.

43:51 But soon as you're back in regular Python, just jamming on something, well, there's no way to time slice that, right?

43:58 So everything else is waiting on that computationally.

44:00 And you end up with really horrible things.

44:03 And I'm going to say it's horrible.

44:04 I think it really is horrible, right?

44:05 It's where you've got a loop of CPU-heavy work.

44:08 I know you're rendering it and you add in an asyncio sleep zero just to release work.

44:14 And it's like, why are we doing that?

44:15 We're doing that because we're running CPU work on the event loop.

44:19 The event loop wasn't for that.

44:21 It was for our handling I.O., by async I.O.

44:25 And we really want to throw CPU bound work to a thread.

44:30 We just want to.

44:31 But then we've got Python's guild.

44:32 So this is where I'm super excited.

44:34 It effectively says, well, you have as many threads as you want, but you still can run

44:37 one at a time.

44:39 Yeah, exactly.

44:40 Either way.

44:41 So either you do the CPU work on your one thread, which is running the event loop, or you throw

44:46 it out to different threads and they get GIL contention and so you end up with essentially

44:51 one cpu or one threads worth of clock time no matter which way you cut it which is python's

44:57 real problem and it has you know it it's not actually about switching threads it's not actually

45:02 about creating threads it's not actually about spinning up event loops those are relatively quick

45:06 things it's about the inability to genuinely run parallel work in in multiple threads in python up

45:13 until very recently. Yeah, it's another axis of that. You probably don't need it, but it's a low

45:19 upper bound that shouldn't be there for people who need to grow into it. Yes, entirely. And you

45:25 shouldn't need it. It can come in at quite moderate scales. Okay, my Hello World blog application,

45:33 I'm not going to hit gil contention with my 10 requests a minute. It's not a problem if I'm lucky.

45:39 Yeah, that's when it gets popular, yeah, in the article.

45:42 Yeah, yeah, yeah.

45:43 Oh, I hit Hacker News.

45:44 I've got 10 requests in a minute.

45:45 You know, you're not going to hit GitGil contention there.

45:49 But if you're, again, serious about using Python, a bigger application,

45:54 yes, you are going to have to do CPU work at some point.

45:56 Otherwise, the computer's not doing anything, right?

45:58 The program, if it's going to do something, it's going to have to use the CPU at some point.

46:02 And so if you're doing that at scale, the Git has been an impediment to that.

46:07 And people are, oh, I'm going to switch to Rast.

46:08 I'm going to switch to Go.

46:09 So do we have to switch language again?

46:12 Can we not do that?

46:13 Yeah, let's make it so we don't have to.

46:15 And you don't have to have an insane amount of requests

46:18 to benefit from this.

46:19 Kind of like my little example I just showed with the memory thing.

46:22 If you have to run enough workers that you needed two servers,

46:26 if this free threading thing and the async stuff let you shrink that back down to one more manageable server,

46:34 it saves you on DevOps.

46:36 It saves you on complexity.

46:37 It saves your money.

46:38 It saves your money.

46:38 So if you want money, all these things.

46:41 So you don't have to be Instagram level of Django requests

46:46 to benefit from some of these ideas, right?

46:48 No, absolutely not.

46:49 I mean, it turns out, and I've always said for years, with a medium-sized box, you can run a very high-traffic server

47:00 in realistic terms.

47:02 But the more you can do on the smaller box, absolutely the better.

47:06 Absolutely.

47:07 So three threading is the big win.

47:10 It's the big exciting thing because Django kind of looked clumsy almost when it said,

47:15 no, let's just put all the sync work into a thread.

47:18 But actually, as three threading becomes more established and it works through the ecosystem,

47:26 it looks as if Django's bet's going to pay off.

47:29 That bet will have aged very well.

47:31 Because we always really wanted to do CPU work in the thread.

47:36 I think it's great.

47:37 Yeah, I mean, you can have a thread pool.

47:40 Your threads could each have sort of a hanging around database connection for any time you need it, ready to go.

47:46 And yeah, you just delegate it out and you actually get parallelism.

47:50 Yeah, no, and that's the thing.

47:52 Actual parallelism within Python is really exciting.

47:56 And that's for everybody.

47:57 I mean, you know, to go back, why did we update the async topic docs?

48:01 It's because there are problems that remain with async, but they're not Django-specific problems.

48:06 The same GIL contention problems you have, you type FastAPI concurrency in, or scaling problems, and you read the same sort of problems.

48:15 And so it's not Django specific.

48:17 It's not a Django problem there.

48:20 It's a Python structural problem.

48:22 Same database scaling.

48:23 How do we do it?

48:24 What do database pools?

48:25 Well, same using Flask.

48:27 You're going to scale it that way, et cetera.

48:29 It's nice that these are general problems that we all share, I think.

48:33 Yes, absolutely.

48:34 yeah i mean and let's see i think let's open it no there we go i think it's just so easy these days

48:42 to try out free threaded python too it used to be oh you got to get this weird thing and build it or

48:48 whatever but now with uv it's just literally uv python install like what do you want do you want

48:54 three fourteen six t yeah that's all you're saying in two seconds you've got free threaded

49:00 Python to play with and try out.

49:02 Yes.

49:03 The thing here, there's still work to be done here.

49:06 Most of it is about the ecosystem, not about the architecture.

49:10 The architecture is already in Plex.

49:12 So if you're at an app level, because people have been running threaded Django for a long

49:19 time.

49:19 We're talking about people deploying Whiskey with threaded workers.

49:23 They've been doing that for a long time.

49:25 But standardly, at the app level, you don't touch things which are multi-thread.

49:30 You might have, I don't know, some module level caching.

49:33 You might have a property at the module level.

49:35 Well, hang on.

49:36 Is that really thread safe?

49:37 Do you need to wrap that in a lock?

49:38 But it's extensions, it's libraries.

49:42 Are they doing these kind of patterns?

49:43 And we just need to work through them and run all our test suites against them

49:46 and make sure that we're compatible with re-threading.

49:49 And then I think we're actually going to unlock an awful lot of performance for everybody.

49:55 So instead of having to run multiple processes to get parallelism,

49:58 you'll be able to run it all within one process.

50:01 You know, it's a really exciting thing.

50:03 Yeah, I have some really interesting experiments I want to put together with free threading,

50:06 but the more important work is still out there.

50:09 Yeah.

50:10 With AI, I'm not ready to, like, go and work on these.

50:12 Like, I don't, I can't do another open source project yet.

50:15 Yeah, no, and you must avoid it because the trouble is that,

50:19 I maintain a lot of packages, right?

50:21 And it's a burden.

50:23 It's a thing that once you've taken it on, you can't just put that.

50:26 And you might think, oh, it's just a casual thing.

50:28 must be very conscious and careful about what you package up and put out there in the world because

50:32 you're making a promise to people like yeah carlton i'm thinking this is going to be a really good gist

50:38 not a repo you know what i mean yes right exactly put up a gist do a blog post put up a gist say

50:43 here it is but don't put it up as a as a package and unless you mean it if you mean it then it's a

50:49 package on pypi and you know all the rest people can install pip installs directly from github

50:53 You don't have often version control.

50:55 You don't have to put it on PyPI for people to be able to play with it.

50:59 Yeah, absolutely.

51:00 So one thing I want to talk to you about, I know you're excited about, is the built-in

51:04 task framework.

51:05 Because all of this stuff about threading and all of that, you know, parallelism and, you

51:11 know, event loops, it's fine.

51:13 But if you could just say, let's just run that somewhere else.

51:17 Don't hassle me.

51:19 Tell us about this.

51:19 Okay.

51:20 So this is a great question, because when we first added async to Django, people were like, oh, this is background tasks.

51:26 And it's like, no, no, no, it's not.

51:27 It's all about I.O. and multiple connections and long-lived requests and things like that.

51:32 It's nothing to do with having a queue where you want to do some asynchronous task.

51:37 So asynchronous is the word has appeared, but it means other than making people wait.

51:43 So the standard web, you've got a request and you send a response, and you want that to be as quick as you can.

51:49 But let's say somebody has a password reset and you want to send them an email so that they can click on the password reset link and they can reset their password.

51:57 Well, historically, you would have made the request, sent the email, wait for the email to send, which can take almost any amount of time.

52:06 And then only then do they get the HTTP response.

52:09 Well, that's a long time to wait.

52:11 That's a bad user experience, right?

52:12 The much better is to make the request, say, yes, we've accepted it, put it, we'll talk about where we put it in a moment, and just send the response quickly saying, look, it's on its way.

52:20 Give it a few seconds to arrive.

52:21 It's already email.

52:23 It's already a single.

52:23 There's nothing you can do to wait for delivery.

52:26 Yeah, yeah, yeah.

52:26 So all you can do, though, is send.

52:28 So what you do now is you quickly write a task to, say, the database with the Django task framework.

52:34 And you would send that, there'd be a worker process running totally separately, which could then send out the email, say, for instance.

52:42 Now, people have historically done this with either Celery or RQ or Django Q or then Django Q2.

52:49 And there are Huey, there's a billion of these task queues out there.

52:53 And problem there is that, say, Django itself couldn't rely on them.

52:58 And neither could any third party package rely on them.

53:01 Well, I could, but then I'd have to commit to, we're using Django Q2 for this.

53:05 And all the people who were using Celery or RQ or Huey, they'd be excluded.

53:09 And so what Django Tasks does is to provide a kind of back-end API that says, look, any package can use this API to enqueue a task and to check results and do the core activities that you need to do around asynchronous tasks.

53:27 And then just by the user, just by configuring in the settings what their back end is, can either go for Celery or RQ or it depends.

53:35 As long as those packages have implemented a Django tasks adapter, you can use those for the back end.

53:42 And so I think, for instance, Celery hasn't yet done it, but there's an open issue and it will just it will become.

53:47 But there are a couple of back ends that are available.

53:49 There's a Redis one, there's one that uses the ORM to just store in your database and you can get going with those now.

53:55 And for me as a third-party package maintainer, that's fantastic because now I can add tasks to my third-party package and users can benefit from those without me having to know anything about which backend they're using.

54:06 Yeah.

54:07 Your pip install or uv install instructions for your package aren't like bracket celery, bracket, whatever.

54:15 And if none of those are there, it's going to throw an acceptance.

54:18 Well, you didn't install any of the queuing options.

54:20 So now we're busted, you know, like, yeah, you can just delegate.

54:23 Yeah.

54:24 And Cellerie's wonderful.

54:26 It really is a truly wonderful piece of software.

54:29 But it's literally overkill for 99% of projects.

54:33 And it's operationally difficult to keep going.

54:36 You end up running three or four different things.

54:37 You need a monitor, a beat server, the actual worker.

54:43 It's designed for a level of sophistication that most applications really don't need.

54:48 I love that you call out the operational complexity as well.

54:53 Because like you say, it would be nice to just let people who don't have something complicated use this feature without going, yeah, look, I know you used to, look, that was fun when you could just put it on a platform as a service thing and let it run.

55:05 But now what we're doing is we're setting up a private network and now we've got poison messages and we've got downtime because the celery version had to change.

55:14 Oh my gosh.

55:16 It's a step jump, right?

55:17 And just to be able to avoid that is great.

55:20 No, it's a serious, serious.

55:22 If you are taking on celery, then there is a serious question to ask.

55:28 You know, if you're an established team, but if you're growing and suddenly it's like, shall we add celery?

55:34 It's like, well, hang on, look at your ops capacity and look at your ability to, you know, do you have someone on call for when the workers break down at two o'clock in the morning and page notifications and all those things?

55:44 Because it's a world up from self-hosting, you know, your bog standard Django app.

55:51 As someone who does a lot of DevOps without a lot of support, I hear that.

55:56 So, you know, Carl, I think this could have been like a three-hour show easily, I'm starting to realize.

56:03 But let's sort of get some advice to kind of wrap things up.

56:07 We're on a glide path here for people.

56:09 Should they write async views today?

56:11 What do you think about this?

56:14 Yes, right.

56:14 So if I'm writing a small application, like just my local blog application, I want to add a little bit of async sprinkle to it.

56:21 I might just deploy a single ASGI worker by itself.

56:24 It's probably big enough.

56:26 You're not going to hit scaling problems, you know, with low traffic.

56:28 So, you know, just a single ASGI application.

56:31 I think the, as I say, the people who are doing this at a bigger level than that, I think they're running their core application as WSGI and the standard ways because the scaling patterns there are so established and known.

56:42 and it's almost trivial by comparison.

56:45 But then they're running a little sidecar to handle their app.

56:49 So just have, you know, engine XA can route by the path

56:52 or it can route by other things.

56:53 But, you know, if it's got connection header,

56:55 keep route, but that's a little advanced.

56:56 Just route by path, put everything under an A prefix.

56:59 And then, right, those are your async ones for,

57:02 I don't know, your long-lib service center events requests.

57:05 And then send everything else back by your whiskey.

57:06 I mean, just do that.

57:07 Get the hang of it.

57:08 You'll find the scaling, that scales very well.

57:11 And then you can say, okay, well, now if I move this to async, oh, okay, I start to see some of the complications of async appear.

57:17 And I think that's the danger I see is when people dive feet first into async without really understanding

57:24 that it is an order of magnitude more complicated to correctly handle async code than to just code synchronously.

57:32 For all these reasons about blocking the event loop and you get deadlocks.

57:37 These are really complex patterns that grow into them rather than jumping headphones.

57:41 yeah yeah but django is ready yes no django's async story is there the problems that i see come up

57:47 time and time and time again are problems with python and python's async story that we all share

57:53 not django specific problems yeah or even that asynchronous programming in general yes asynchronous

57:58 yeah they are yeah they are async or guild problems not um django specific django specific

58:06 problems. Absolutely. I want to just put this out in the universe. If you're looking at your web app

58:12 and you're like, ah, these are a little bit slow, these requests. So we're going to need to set up

58:17 some parallelism so that we can handle multiple slow requests at a time. Like just take an hour

58:22 and look at your database indexes, please. Database indexes are magic. And I'm not saying it's not a

58:29 benefit to go async or whatever, but if slow is a problem, please just do, you know, look at the

58:35 query plans, look and make sure every hot path is using a database index and so on.

58:40 Yes, yes, exactly.

58:42 I mean, and if it, so when you make a database, we always talk about the database, right?

58:46 But if Django makes a database request, there's a little bit of CPU time to compile the request.

58:52 And then there's the IO bit where it's talking to the database.

58:55 And, you know, PsychoPG already releases the gil in that context.

58:59 So you're already getting, you know, what concurrency or parallelism you would have at that point.

59:04 And then there's CPU time to sort of marshal the return into,

59:09 or structure the returned data into ORM objects.

59:13 So those bits are CPU bound.

59:15 But that core bit is already releasing the gil.

59:18 You're already going to get parallelism there.

59:20 So the chances are you're not going to get much more by jumping to Async or adding more workers.

59:26 It was your database all along that was the thing that was taking the time.

59:30 These things that we've been talking about, about CPU contention and all the rest.

59:34 It's, ah, they come up in only specialized cases.

59:38 They're kind of advanced topics, if that makes sense.

59:42 They're often not the problem.

59:44 Amazing when you need them, but it's not a panacea, yeah.

59:47 Yeah, and we, I mean, how would I phrase it?

59:50 We spend so much time thinking about the use cases for the advanced cases that we sometimes forget

59:56 that the simpler ones are much more simple stories.

59:58 And yeah, should I be writing async views?

01:00:00 well, do it gently.

01:00:03 Don't jump into it thinking it's some cure-all.

01:00:06 Yeah, but Django's ready.

01:00:07 That's cool.

01:00:08 Yeah, yeah, yeah.

01:00:08 No, absolutely.

01:00:09 And that's why we made the effort to update the topic,

01:00:14 because we realized we were giving users bad advice.

01:00:17 Yeah, you're scaring them off.

01:00:19 Yeah, no, we really were.

01:00:20 We were like, oh, it's still incomplete.

01:00:22 It's not ready.

01:00:22 No, it's there.

01:00:24 And yes, you will run into problems as you scale up, but that's not Django.

01:00:28 That's async.

01:00:29 Yeah, indeed.

01:00:30 Final call to action.

01:00:31 People consider Django.

01:00:33 Maybe we're considering async for their existing Django projects

01:00:37 or other things like the Django tasks, background tasks and so on.

01:00:40 What do you tell them?

01:00:41 Yeah, I mean, I would get going with Django tasks.

01:00:44 You install the database backend for that, which is all the Redis ones work perfectly as well.

01:00:49 If you've already got Redis in your stack and you want a little bit more,

01:00:53 but most people don't need that.

01:00:55 Just use the ORM database queue.

01:00:57 It's a great storage.

01:00:58 That's where you want to throw most of your work.

01:01:01 You don't really want to be...

01:01:03 With Async.io, you can, yes, throw something off to an executor.

01:01:08 No, but don't do that because you've got no queue controls.

01:01:11 You've got no result.

01:01:12 You can't say, did it fail and then have logic to retry,

01:01:15 any of those sorts of things.

01:01:17 It's just running on the event loop.

01:01:19 And if it crashes, well, what do you do about that?

01:01:21 How do you recover from that?

01:01:22 There's no elegance to that.

01:01:23 So, yes, Django tasks is probably where I would point people first.

01:01:27 And then do I need long-lived requests?

01:01:29 Do I want to do things like service and events or WebSockets?

01:01:31 Yeah, that's where.

01:01:32 Or streaming, streaming responses.

01:01:35 Async, great, great example.

01:01:37 Yeah.

01:01:37 You want to adopt Datastar or one of these frameworks that really leverages service and events.

01:01:42 Yeah.

01:01:43 Or HTMX version 4, which is in beta whatever now, which I think is the risk.

01:01:48 Oh, is that coming with that as well?

01:01:50 Yeah, yeah, yeah.

01:01:51 So they basically were like, right, it's okay.

01:01:53 So I think Carson was like, we're never going to do an HTMX version 3.

01:01:57 and he's kept his promise because he's jumped straight from two to four um but the it

01:02:02 was um it was inspired by some of the things from um data star and the surrounding projects it's

01:02:07 like hang on we can have a go at that as well i think said goss you have to get him on and he can

01:02:11 talk you through it yeah i i love carson i think what he's doing is great and i've had him on before

01:02:15 i'm a fan of hdmx so that's cool maybe we'll do another no other round yeah so when i stepped

01:02:20 down as a fellow in 23 i started a new application been building with hdmx and alpine since day one

01:02:26 and just absolutely adore it.

01:02:29 It's combined with template partials, which we added into Django,

01:02:31 it's really changed the way I write web applications.

01:02:34 It's amazing.

01:02:34 All for the better as well.

01:02:35 It's like JavaScript magic fairy dust.

01:02:38 Yeah, it's how it should be.

01:02:40 Yeah, exactly.

01:02:41 Sprinkle it on there.

01:02:42 It doesn't have to be a spa.

01:02:43 Everyone calm down.

01:02:44 We could still do some stuff on templates and so on.

01:02:47 But when I learned JavaScript, that was how it was.

01:02:51 You detach a few event listeners and just do your little thing.

01:02:54 And that was enough, right?

01:02:56 hashtag dollar document ready.

01:02:58 Let's go.

01:03:00 A little jQuery back in the day, right?

01:03:02 Yeah.

01:03:03 That's how it was.

01:03:04 Yeah, yeah.

01:03:04 Cool.

01:03:04 Select something, do something.

01:03:06 Exactly.

01:03:07 Carlton, thanks for first all the stuff on Django for everybody

01:03:10 and second, coming on the show to share.

01:03:12 No, thanks.

01:03:12 Thanks for the chat.

01:03:14 It's been really good.

01:03:14 It's really lovely because you pick on the topics and I'm like, yeah, yeah, this is right.

01:03:19 Thanks.

01:03:20 See you later.

01:03:20 Thank you, Michael.

01:03:21 This has been another episode of Talk Python To Me.

01:03:24 Thank you to our sponsors.

01:03:25 Be sure to check out what they're offering.

01:03:26 It really helps support the show.

01:03:28 Take some stress out of your life.

01:03:30 Get notified immediately about errors and performance issues

01:03:34 in your web or mobile applications with Sentry.

01:03:36 Just visit talkpython.fm/sentry and get started for free.

01:03:41 Be sure to use our code, talkpython26.

01:03:44 That's Talk Python, the numbers two, six, all one word.

01:03:48 If you or your team needs to learn Python, we have over 270 hours of beginner and advanced courses

01:03:54 on topics ranging from complete beginners to async code, Flask, Django, HTMX, and even LLMs.

01:04:01 Best of all, there's no subscription in sight.

01:04:04 Browse the catalog at talkpython.fm.

01:04:06 And if you're not already subscribed to the show on your favorite podcast player,

01:04:10 what are you waiting for?

01:04:12 Just search for Python in your podcast player.

01:04:13 We should be right at the top.

01:04:15 If you enjoy that geeky rap song, you can download the full track.

01:04:18 The link is actually in your podcast blur show notes.

01:04:20 This is your host, Michael Kennedy.

01:04:22 Thank you so much for listening.

01:04:23 I really appreciate it.

01:04:25 I'll see you next time.

01:04:46 We tapped into that modern vibe over Kingy Storm.

01:04:50 Talk Python To Me.

01:04:52 I-Sync is the norm.

01:05:22 .