TonIO and Multi-core Async Python: A Visual Guide to Talk Python Episode 561

If your async Python code is slow, the usual advice is to swap in a faster event loop. uvloop is the famous one. Joe wrote rloop, a Rust-based loop, and Marcelo Trilesinski, the maintainer of Uvicorn and Starlette, recently published zuvloop, which builds on libuv and is written in Zig. Doesn't that settle it?

Joe's answer starts with a yes: install one of them. It's performance you're otherwise leaving on the table, and these projects also shine a light on some of asyncio's inefficiencies. But a faster loop only speeds up the hot parts of the loop's cycle, and the shape of how things work stays the same. By his estimate, the best you get is something like 20%, and that's raw throughput in a benchmark doing TCP and nothing else. Real programs do a lot more than move bytes.

Michael put numbers on the other side of it. He'd just bought a Mac Studio with 18 cores. One thread is about 5% of that machine. "So what does it matter if it's 4.9% or 5.1% depending on which loop?" What he wants is 95%.

He also pointed out when a faster loop matters at all: when you're doing very fine-grained work, lots of it, with lots of switching. If you've got 10 tasks that each run for two seconds, a faster loop does nothing for you. You have to construct a benchmark that does almost nothing but wait efficiently before the loops start to pull apart.

Joe hit the same wall from the inside. Working on rloop last year, he reached the point where there was nothing left to optimize. Most of the CPU time went to asyncio's own primitives, and all of it ran on a single thread. No loop could fix that, since the limit was the single core underneath it, and that realization is where TonIO began.

“all of these event loops, yes, they can optimize some of the hot part of the cycle. But basically, that's it. Nothing really changed the shape of how things work.”

Giovanni Barillari

Michael recently wrote a post on his personal blog about cutting Python web app memory by over 31%. One of the apps in it is Talk Python Training, which is hardly a toy: it has 178,000 lines of Python. With lazy imports, a switch to async and a few other changes, he took it from about 1.3 GB down to 450 MB.

Why does a Python web app use that much memory in the first place? Part of the answer is that the server copies it. Granian, like every Python server, has to replicate your app to use more than one core. It either runs several single-threaded processes, or several threads that each get their own interpreter. Either way you end up with copies of your code, and you need them because of the GIL. Michael's server has eight cores, and a single Python process can normally only use one of them.

Why does that hurt more now? As Michael put it, the expensive thing on a server today is memory, not compute. If your process takes 500 MB and you need four of them, that's 2 GB just to run your code.

Joe sees the same thing at a much bigger scale. Sentry is an 18-year-old Django monolith. It's deployed several different ways, with only some parts switched on in each, and it still needs four processes and four to six gigabytes of memory per pod to use a single CPU of a cloud VM.

Memory is only the first thing that multiplies. As Michael added, any in-memory cache, like an LRU cache, gets built four times instead of once. Observability gets worse too. Point the Prometheus SDK at that app and you get four separate instances, so tracking one set of metrics takes some weird tricks with the multiprocessing library. "It's messy, man," as Joe put it.

That's why he doesn't buy the idea that free-threading arrived just because of AI and its appetite for parallel work. For him, it solves a whole set of very ordinary web development problems first.

“I know your process takes 500 megs of RAM, but we need four of them. So now you need two gigs just to run your code.”

Michael Kennedy

Remember Python 2 to 3, when strings stopped being bytes and everything became Unicode? Joe says that's nothing compared to the GIL going away. For him, free-threaded Python is the biggest thing to happen to Python in its 30 years, and he's honestly puzzled that more people aren't as excited about it as he is.

It's been a long road. Just about every room full of Python people has argued about the GIL at some point. Michael remembered Larry Hastings' Gilectomy, which seemed to be perpetually in the works and never really went anywhere. It took something like 15 years of talk before free-threading finally landed. In Python 3.13 it shipped as an experiment, with an asterisk on top. Since Python 3.14, Joe says, it's considered stable, on the same footing as the normal interpreter with the GIL.

Why does he care so much? Because Python is the language he loves most, mostly for its community, and now he can use it with actual threads. No more reaching for a C library just to use the CPU he paid for. And CPUs keep growing sideways: even a $200 CPU, he pointed out, probably has 12 cores or so.

So why isn't everyone on it? The main blocker is native code. Every compiled C extension has to be made thread safe, and for some of the older ones that's neither simple nor easy.

Getting the interpreter, on the other hand, is trivial. As Michael noted, it's one command: uv python install 3.14t. The one sore spot for Joe is Docker. There's still no official free-threaded 3.14 Docker image, which Michael pointed out has been the case for a year and a half.

And this free-threaded build is the only one TonIO will run on.

“finally I have the language I love the most. not because of the structure, not because of types or how it's designed, but because of this awesome community that only Python language has, at least in my experience. And so finally, I can use this language with actual threads.”

Giovanni Barillari

Once Joe saw that rloop had nothing left to squeeze out of a single thread, he asked a different question. Forget making asyncio faster. If asyncio had never existed, what would async Python look like? Would it be hard to build?

Apparently not that hard. Three or four weeks after starting from a blank whiteboard, he had a complete alternative async runtime running Python on multiple threads. That's TonIO. As an Italian, Joe says "Tonio", and he's happy for you to pronounce it however you like. Michael pointed out that the IO on the end tips its hat to asyncio, and Joe added that it couldn't be Trio, since Trio already exists.

The biggest influence isn't Python at all. It's Tokio, the number one async runtime in Rust, a language Joe spends a lot of his time in. By default it starts one standard worker thread per CPU core, so on Michael's 18-core Mac Studio you'd get 18 threads running your code. Would you want all of them? Probably, Joe says. And if your program needs to be a good citizen and leave some cores free, as Michael asked, you can set the size of the pool when you start the runtime.

There's one hard requirement: TonIO only runs on free-threaded Python. Real threads are the whole design, so if the GIL is present, it raises a runtime error and tells you to change your interpreter. There's no fallback mode; it just won't start.

How finished is it? At the time of recording TonIO was at version 0.9.14 and officially alpha. Joe thinks the API is close to stable and that few modules or primitives are left to build. The big missing feature was Windows support (more on that at the end of this guide). The alpha label, he says, is mostly because he's the only one working on it.

“let's just assume for a second, like asyncio never existed. And I want to do like async stuff in Python. What that look like?”

Giovanni Barillari

So far, the story covers three ways a Python program can use a modern machine:

• Many processes. Every core gets used, but only by copying the whole app, so memory, caches and metrics all multiply. See the real cost of the GIL.
• One event loop. asyncio juggles thousands of waiting operations cheaply, and uvloop or rloop make it faster, but it still runs on a single core. See why a faster event loop isn't enough.
• Real threads. Free-threaded Python 3.14t plus TonIO: one process, a worker thread per core, shared memory. See free-threaded Python and what TonIO is.

The table above rates TonIO's complexity as low. That's fair for the API, which has far fewer moving parts than asyncio. It's less true for your program. Shared memory across real threads brings back problems asyncio quietly hid, which is why Joe boils it down to two rules for locks and shared state.

Working on rloop taught Joe two things. One was the single-thread wall. The other was that asyncio is, in his words, overcomplicated sometimes.

Here's the example he gave. At PyCon Italy 2025, giving a talk about rloop, he put up a slide asking the audience a simple question: what's the difference between a protocol and a transport in asyncio? If you look up that chapter in the official Python docs, you'll find they try to explain the difference three different ways. For Joe, that's a red flag. The Zen of Python does say simple is better than complex, after all.

And protocols and transports are only one corner of it. There are events and futures. Anything you spawn onto the event loop gets wrapped in a task. "What is a task? Nobody knows." Then, when it actually runs inside the loop, it becomes a handle. None of this shows up when you write everyday async code. But the moment something goes wrong and you need to debug it, you need to understand all of it.

Michael's frustration comes from the other end: juggling loops. You want to make a request, but there's an event loop, and it's not that one, it's the one your async web framework created. Did you initialize the async database connection before the web app started, on the right loop? Nope, wrong one. He thinks this jankiness exists because Python never had true threads, and he'd much rather the runtime own the loop, so he can just use it without caring who started it. TonIO leans that way too.

TonIO's answer to all of this is to ship far fewer primitives. The core of it is two ways to start work, spawn and spawn_blocking, covered in the next two sections. That simplicity has a price, though. Code that leans on asyncio tasks or futures won't port directly, because those concepts don't exist in TonIO's runtime.

“if you need three different ways to explain the same thing, something is deeply wrong.”

Giovanni Barillari

If you know asyncio, TonIO's async code will look familiar. You write coroutines with async def and await things. The main difference is that you import sleep, timers and the rest from TonIO instead of asyncio, and there are far fewer of them. The one big new idea is spawn.

In asyncio, most operations tie launching to waiting. Await a coroutine and it starts and you wait for it, in one step. To start something now and collect it later, you create a task, and only then does it begin running right away. Or you reach for gather, task groups or Trio-style scopes. Michael described the dance: you build up a list of things that need to become tasks, turn them into tasks, then await each one, because if you just await them in turn, it all falls back to running one after another.

In TonIO, spawn starts the coroutine immediately. Then you choose: forget about it, await it right away, or park the result in a variable and await it later. And since TonIO runs a pool of real threads, spawned work runs in parallel across them.

To make it concrete, here's the pattern from TonIO's documentation that Joe walked through, using the async/await syntax:

import tonio.colored as tonio

async def double(v):
    await tonio.sleep(0.5)
    return v * 2

@tonio.main
async def main():
    parallel = tonio.spawn(double(2), double(3))
    v3 = await double(4)
    v1, v2 = await parallel
    print([v1, v2, v3])

main()

Save it as spawn_demo.py and run it with uv run --python 3.14t --with tonio spawn_demo.py. It prints [4, 6, 8] after about half a second, not a second and a half: the two spawned calls are already running while the third is awaited. (Checked against TonIO 0.10.4.)

Notice @tonio.main. In asyncio, even a simple script needs asyncio.run, and Joe finds that odd, since in JavaScript you don't decide anything. Michael's take is that Python has no foundation layer for async, so building one is your job. TonIO keeps the entry point but makes it small: configure everything through tonio.run, or put the tonio.main decorator on your function and be done.

The tonio.colored import is the other clue. TonIO supports two syntaxes. async/await is built on generators underneath, so supporting a plain yield syntax took Joe very little extra work. If you dislike sprinkling async and await everywhere, the base tonio module lets you yield instead. It's still colored, though: anything that calls a generator function has to be a generator too.

“So instead of await, you just yield from the next coroutine you want to suspend for or wait for. And that's it. You don't need to write async def everywhere.”

Giovanni Barillari

TonIO has two thread pools: the standard pool that runs your coroutines, and a separate blocking pool. When do you need the second one? Less often than you might think.

Joe's rule depends on what your program does. If it's purely CPU bound, just use the standard pool. There's nothing to block, and the default configuration already gives you one thread per core, which is exactly what CPU-bound work wants. If it's purely I/O bound, the standard pool is fine too. The blocking pool only matters when the two mix: network, disk or terminal work on one side, heavy computation on the other. Then the only way to keep the I/O side responsive is to move the CPU work to the blocking pool. In other words, the standard pool handles CPU work just fine on its own. It only needs protecting from long computations when I/O shares it.

How do you use it? Every await, yield or spawn runs on the standard pool. For the blocking pool, you call tonio.spawn_blocking. That's it. You don't create the pool or size it. Blocking threads are spawned on demand, up to a maximum you can configure, and if you want to limit how many run at once, you use barriers or semaphores.

Michael suggested a twist. What if a function could declare that it's computational with a decorator, so any caller, even code in another library built on TonIO, got it scheduled on the blocking pool without having to know? He used tonio.cpu as a placeholder name. Joe liked it and said he thought it would land in TonIO 0.10.

It did, under the name mark. TonIO 0.10 has tonio.mark.blocking to send a function through spawn_blocking, tonio.mark.max_concurrency(n) to cap how many run at once, and tonio.mark.cpu_bound(n) for both. Here's spawn_blocking next to the marker:

import tonio.colored as tonio

def crunch(n):
    return sum(i * i for i in range(n))

@tonio.mark.cpu_bound(4)
def crunch_marked(n):
    return sum(i * i for i in range(n))

@tonio.main
async def main():
    total = await tonio.spawn_blocking(crunch, 1_000_000)
    marked = await crunch_marked(1_000_000)
    print(total == marked)

main()

Run it with uv run --python 3.14t --with tonio plus the file name and it prints True. (Checked against TonIO 0.10.4.)

“The blocking thread pool is useful only when you have mixed workloads and you need to balance them”

Giovanni Barillari

A runtime is more than a scheduler. You need ways to wait, to coordinate and to talk to the outside world. TonIO groups these into a handful of modules:

• time has the time primitives: timeouts, timers and so on.
• sync has the synchronization primitives: locks, semaphores and barriers.
• net is the network module, with sockets and streams.
• fs is the filesystem module, with async open and a Path class.

The network module is where TonIO differs most from asyncio. Here's the odd part: asyncio has no async socket library of its own. You create a regular socket with the standard library's socket module, hand it to the event loop, and work with it through the loop. TonIO's net module contains a socket module with exactly the same interface as the standard library's, except that the methods that would block are async. On top of that sits a streams module, a higher-level, Trio-like interface that makes networking easier to manage.

The filesystem module follows the same idea. It has async versions of open, and it mirrors pathlib, so you can import Path from TonIO's fs module and get async versions of the methods that need to be async. There's also a pytest plugin, so your tests can run under TonIO. Fun fact: it's the one part of TonIO that Claude wrote.

How much is left to build? At version 0.9.14, Joe didn't think there was much missing in modules, primitives or features, apart from Windows support. He considers himself close to stabilizing the API.

“So right now, again, this is all alpha. That's mostly because there's only me working on this.”

Giovanni Barillari

Did the GIL ever really protect you from threading bugs? Michael's view is that people felt it saved them from locks and semaphores, but it didn't. It only made it less likely that another thread would interrupt you.

Joe sharpened that for async code. asyncio has a hidden feature: only one coroutine runs at a time, on one thread. That makes it really hard to deadlock yourself with a threading lock. As Michael put it, a threading lock in asyncio is effectively a no-op, other than costing some CPU. TonIO can run as many coroutines at once as it has threads, and that's exactly where people make mistakes. It has its own async locks, and you also still have thread locks. They're two different things, and they don't talk to each other.

That leads to Joe's two rules.

Rule one: never await inside a threading lock. If you enter a threading lock and then await a coroutine inside it, you've deadlocked TonIO. The good news is that it's easy to spot, since Python locks are taken with a with statement. If you need to hold a lock across an await, use an async lock, TonIO's own.

Rule two: watch out for shared state. In free-threaded Python, dictionaries and lists are thread safe. That doesn't make your program correct. If you write a value and expect to read it back, another thread can write to the same place in between. Whenever a sequence of operations on shared memory has to happen as one unit, you need a lock.

Thread locals and context variables are a related trap. Thread locals let you keep "global" state per thread, such as the current request, without passing it through every function. Flask's request object is the familiar example, and Joe copied the idea for his own framework, Emmett. Context variables do the same job for async code: when a coroutine suspends and resumes, it gets its own state back instead of a mix from other coroutines.

With async code spread across many threads, context variables get more complicated. So in TonIO they're a Boolean you pass when starting the runtime, and for now it defaults to false. Joe wants developers testing the alpha to say explicitly that they'll use context variables, as a sign that they've thought through the threading model. Once the API stabilizes, the default may well flip to true.

“my advice is just think about those two rules and then multi-threading is not that hard. Again, rule number one, never await inside a threading lock. That's a simple rule.”

Giovanni Barillari

One of asyncio's real strengths is that everyone uses it. If you write async with around an httpx client and await its get, it runs on asyncio. Does that work on TonIO? No. So what do you do?

Today's answer is tonio-monkey. As the name suggests, it monkey patches popular asyncio-native libraries so they run on TonIO. At the time of recording it covered psycopg, httpx and Redis, and it also has patches for FastAPI. Django is next: Joe met Carlton Gibson at PyCon Italy this year, they talked it over, and he hoped to have a Django patch in a month or so. (Michael had just had Carlton on the show to talk about async in Django, and Joe recognized a few of the ideas.)

Keep in mind that it patches specific libraries rather than making any asyncio code run as is. Joe wants to hear about the gaps. If you try TonIO on an existing codebase and need a package patched, open an issue on the tonio-monkey repo or start a TonIO discussion. Right now he needs as many real use cases as he can get.

He's also building a small set of libraries that target TonIO directly, mostly for the web. httpunk is a low-level HTTP library that works with both asyncio and TonIO. On top of it sits punkreq, a higher-level HTTP client that you can use with either runtime, because the adapter lives inside the package. And there's a very beta port of Uvicorn that runs on TonIO, which Joe built as a test server for the runtime.

The longer-term idea is less patching. Once the API is stable, Joe would like to talk to the AnyIO maintainers about a TonIO backend for AnyIO, which would be a much easier way to reach a lot of libraries at once. He isn't sure how feasible it is, though. TonIO doesn't support asyncio's primitives, and anything that relies heavily on them is a problem.

“If anyone from the listeners want to try Tonio to an existing code base and you have a package you would like to see monkey patched, just open an issue in the TonioMonkey repo or even in Tonio discussions, and I will look at it.”

Giovanni Barillari

How much of TonIO did AI write? Almost none. Joe puts it at maybe 1%. The only piece Claude wrote was the pytest plugin. Everything else he wrote the old way, in a text editor, without even code suggestions.

Where AI did help was in finding bugs. A new runtime has a chicken-and-egg problem: it needs real programs to shake out its bugs, and real programs don't exist until people trust it. So Joe had AI build the programs. His example was Pi, Mario Zechner's coding agent harness, which is written in TypeScript despite the name. He had Claude rewrite the whole thing in Python, mirroring every behavior and using TonIO for everything. That one project, by his estimate, caught probably 95% of the bugs in TonIO.

He's since started doing the same for the libraries around TonIO. As a solo open source developer who wants to focus on TonIO itself, he doesn't have time to write all the other packages himself, so this approach has been super useful to him.

Michael summed up the pattern: you need users before you have users. So you ask for the use cases you need covered. A web app that uses an async database. A terminal app. And so on.

One thing surprised Joe. When he pointed frontier models at a project and asked them to implement it with TonIO, they got the API right most of the time, even though there's no training data about TonIO at all.

“It's just like, I need users before I have users. I need these three use cases covered. So give me a web app that uses an async database. Give me this terminal app and so on.”

Michael Kennedy

A TonIO web stack is where things are heading, not where they are today. To see why, start with Granian.

In Python, for historical reasons, the web application and the HTTP server are separate packages. The server imports your app and talks to it through a protocol: WSGI, the original from PEP 3333, or ASGI, its async successor. Granian is one of those servers, an alternative to uWSGI, Uvicorn, Hypercorn and the rest. What makes it different is that it's written in Rust, on top of the Hyper library, so the whole HTTP and network layer runs outside the Python interpreter. When your app gets busy, a pure-Python server adds its own load to an interpreter that's already struggling. Granian doesn't.

That's why Michael picked it. He didn't choose it for being a ton faster. He chose it for the p95: how badly things slow down under weird circumstances. It now runs about 15 of the 33 containers on the Talk Python server. Swapping it in was easy, too, because WSGI and ASGI let you move a Flask or Django app to a new server without changing code. (Mostly. Joe had to add an environment key just for Werkzeug.) Granian recently passed Hypercorn in downloads, and it's used at Microsoft, Google and Sentry, which runs all three of its protocols, including Granian's own RSGI.

On free-threaded Python, Granian already uses threads. Since version 2.0, its workers become threads rather than processes, which skips the socket-sharing dance between processes that causes most of its GitHub issues. The catch is that each thread has its own event loop. A request that lands on one worker stays there, and connections can pile up on one thread while another sits idle.

Joe's plan for Granian 3 goes after that. He's designing revision two of RSGI, an async-independent, callback-based protocol, so Granian can support many runtimes: asyncio, Trio, TonIO, gevent, eventlet and more. No promises on timing. Michael asked whether that means the async endpoints in a FastAPI or Quart app would run on TonIO. That's the plan.

“the only thing I'm sure about Granian is that like the HTTP stack, that's super safe. Like that's like, I could bet like everything I have on that. So if something is wrong in Granian, like that's on me, not on Hyper”

Giovanni Barillari

What about Windows? Joe was frank: supporting Windows is a pain, and he's not joking. One example: a bug dating back to Windows NT 4 makes a shared socket become blocking. He also no longer owns a Windows machine. In 2026 he finally moved every one of his machines, gaming included, to macOS and Linux. And mio, the Rust library TonIO uses to drive its event loop, supports Windows only in a weird way.

A few months earlier his answer would have been never. Since then, working it through with an AI model, he thinks he's found a way to get mio to treat certain file descriptors as TCP sockets. Partial Windows support might arrive by the end of the year, but he made no promise. His reason isn't Windows developers. As Michael pointed out, they can use the Windows Subsystem for Linux. It's deployment. At the time of recording, TonIO only ran on POSIX systems, Linux and macOS, and Joe wants a program written with TonIO to run wherever it ends up.

Want to try it? You need the free-threaded interpreter and the package. With uv, that's two commands:

uv venv --python 3.14t
uv pip install tonio

Joe's pitch is that this is the perfect moment to try new things. Free-threaded Python is here, AI agents are here, and if asyncio has given you any of the pain points in this guide, he says now is the time to try something new. If it doesn't fit your use case, or the API could be better, he wants to hear about it: open an issue, start a discussion, or ping him. He's gi0baro on GitHub and on X, blogs at blog.baro.dev, and recently started a podcast on YouTube with Marcelo Trilesinski, @thewwwpod.

If you want the foundations first, Talk Python Training's Async Techniques and Examples in Python covers asyncio, threads, multiprocessing and thread safety. Granian, rloop and TonIO are all Rust underneath, and Up and Running with Rust goes from zero Rust to a PyO3 extension. And for the memory side of this story, there's Python Memory Management and Tips.

“it's not about developers. It's about like where the software runs at the end of the day.”

Giovanni Barillari