You've heard me go on and on about how Python 3.5's async and await features change the game
for asynchronous programming in Python. But what exactly does that mean? How does that look in the
APIs? How does it work internally? Today, I'm here with David Beasley, who has been deeply exploring
the space with his project Curio. And that's what this episode of Talk Python to Me is all about.
It's episode 107, recorded April 14, 2017.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the
ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter,
where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm
and follow the show on Twitter via at Talk Python. This episode is brought to you by Rollbar and
Hired. Thank them both for supporting the show. Check them out at Rollbar and at Hired underscore
HQ on Twitter and tell them thank you. David, welcome to Talk Python.
Hi, how are you doing? I'm doing great. It's great to have you back. It's been going on two years since
you were on one of my first episodes, episode number 12, talking about packaging and modules and
diving deep into understanding those. And I think we're going to be diving deep into another topic,
another area of Python today, but this time in concurrency.
Yeah, it should be fun. It's something I've talked about in the past.
Yeah, yeah. You've definitely been talking a lot about it lately and amazing presentations,
which we'll get to. I know many people know you. It's been two years since I last asked you this
question. So maybe just briefly, you could tell us how you got into Python programming,
that sort of thing.
All right. Well, not to get into too much detail, I guess I first found Python in 1996.
And I was doing some scientific computing, a lot of parallel computing kinds of things,
and just found it for doing, you know, like basically found it for scripting scientific
calculations. And it kind of grew from there. In the more modern era, you know, I'm known for
writing a couple of Python books. So that's where people sort of know me.
Yeah, absolutely. What books are the most famous ones you've written, most well known?
Yeah, the Python Essential Reference, that's been around for a while. And then they did the third
edition of the Python Cookbook with O'Reilly.
Okay, excellent. Yeah, those are both great. And these days, what are you doing in terms of
work and programming with Python and other things?
Most of my work is training, actually. So I do a lot of teaching of Python classes. That's what's
mainly paying the bills. And then also funding, you know, sort of hacking on various open source
projects and the other time when I'm not doing that.
Oh, that's great. Yeah, I've, you know, done a lot of training, a lot of in-person training
previously. And I thought it was just, I think it's a really great career. I think it's a perfect
balance or a close, a great balance, let's say, where you get to teach things to people,
see their reactions, see how they take it, see, you know, they kind of test your understanding of
it. That's part of your job. And then the part is just to research and learn and stay on top of
whatever it is you're teaching. It's really nice, I think.
Yeah, I'm always trying stuff out with teaching. I mean, I find it, you know, it kind of informs like
talks. Think of it, you know, at conferences and also informs books and things. You know, a lot of,
you know, a lot of what people would see at a conference is probably something that I've tested
out in the context of teaching, or I've tried to do it different ways and just kind of seen people's
reactions and confused looks. This isn't working. We'll get to try this. Of course, the iteration is
so much faster, right? You could teach two or three classes in a month. How many conference talks do you
give in a month, right? Or how many books do you write in a month? Not nearly as many.
Right, right. I mean, I'm supposed to be working on a book update right now. And it's, it's going slowly.
But, you know, I'm thinking a lot about these topics, you know, like how to present material, how to think
about it.
Yeah, that's great. Yeah. So let's go ahead and start talking about our main topic here, which is
concurrency. And maybe we could start by just kind of talking about the concurrent options in Python
in general a little bit. And how do you feel that we're doing in 2017 with Python 3.5, 3.6,
compared to, say, five years ago?
Oh, okay. That's an interesting question. Well, I mean, the gist of the problem with concurrency is
it's doing more than one thing at a time. I mean, that's the basic problem. And it comes up a lot in
network programming, especially. So that's where a lot of people are very interested in it. Python has
certainly been involved with concurrency for a long time. I mean, threads have been part of Python
since 1992, I think. So, you know, that goes way back. And there's certainly been the option of,
you know, launching separate Python interpreters. I mean, you could have like multiple interpreters
might be a way of doing that. So these, these have been kind of classic approaches that have been
around for a while. Kind of, kind of a side, alongside that, you have a lot of people messing around
with things like callback functions, event loops, packages based on that. So things like the
twisted framework sort of emerges out of that. So a lot of this has been going on for quite some time.
I mean, maybe over just throughout, throughout Python's history. This question about where it
goes in Python 3. I mean, it's kind of a, I don't know, I'm trying to think like how to, how to
chew on that. It's a big question, huh? No, it is a big question because you've got the,
well, I mean, obviously the big development is the asyncio library. I mean, you get that added to
Python and that is tying together a lot of, a lot of ideas from different places. I mean, a lot of,
a lot of concepts about, you know, event loops and generator functions and coroutines and all these
things are kind of coming together in, in that library. There's a lot of excitement around that
library, but it's also a really difficult concept. Like that's a very difficult library to wrap your
brain around. Yeah. It takes a bunch of things that are individually pretty conceptually hard and then
puts them all together. Yeah. I actually realized that, you know, in hindsight, I never really quite
understood that library the first time I heard about it. I mean, I, I watched Guido give a keynote talk
about it at PyCon and I'm trying to think which one that was. It might've been 2012, maybe 2013. And my
takeaway from the, from the talk is that, Oh, this is going to be cool. We're going to do, we're going
to do things like coroutines for, for async. And we'll probably talk about that later, but I went and rewatched
rewatched that talk recently, maybe three months ago. Cause I was, I was like, what did, what did Guido
like say in that talk exactly? And it was, it was not at all what I remembered, you know, you rewatched
the talk. He's talking more about, you know, trying to have like a common event loop in Python to have
some kind of interoperability with some of these libraries, like, you know, twisted and tornado and,
you know, maybe G event or something. And it, you know, the, the focus on coroutines was more
incidental. So that's really interesting. Yeah. So if for example, you're using twisted, it has some
kind of event loop that's doing the callbacks for you as, as your things complete or whatever. And if
you're using something else that also has an event loop, those things might not know anything about each
other. Right. Right. Right. Right. Yeah. That, that can definitely be a challenge. So that takes us up to,
what, three, three, three, four, when, three, three, three, right. Yeah. And then in three, five,
we got async and await, which was, you know, really take these ideas and make them more, more accessible
to people, I think. Right. I think, yeah. Trying to put like a better face on it. I think, you know,
it's like a putting like a different, I don't know, I don't know how to, almost like a different API on top of
that machinery, I think to present it in a more coherent way. It's certainly not a, that, that
approach is not a Python invention. I mean, I, I, I don't know that they directly cite it, but it had
been done in C# before. Yeah. And it, you know, what's interesting, like the, the history in C#
kind of follows the same way. They didn't come up with that initially either. They came up with just
this idea of like a task framework and it was all callback driven and whatnot. And then somebody said,
oh, look, this callback way of writing, this is super not nice, right? It, it works, but it's not
the same as writing serial code. If we put this async and await on it, it will be. And I, you know,
it has exactly the same benefit for Python is that it takes code that would have to look special and it
kind of makes it look serial, right? Yeah. It makes it like, I mean, that really, that is kind of
the whole focus of it, you know, writing code with callbacks. I mean, you see that like in every
talk about async, you know, there's usually a slide. It's like, oh, callback hell or something,
right? And everybody kind of moans. Yeah, exactly. I've seen, seen callback hell. And then, and then
there's different, different approaches for how to, you know, kind of untangle yourself from that.
And, you know, it's, you know, it's interesting, you know, where, where it tends to push everyone is
more into just like serial code. You know, people want something that looks a lot like maybe thread
programming or just kind of, you know, just kind of straight forward code. But then, you know,
there's this question, how do you get there? So how does it fooling around with generators and,
you know, tasklets and green threads and all the, all these things that you see in these libraries
are all kind of focused on that general, you know, that general problem.
Yeah. So that's really, that's a really interesting thing to understand kind of from the ground up,
I think. And so one of the subjects I wanted to cover while we were talking today is some of the
ideas that you brought up in a talk that you gave at PyCon 2015 called Python concurrency from the ground
up alive. And before we get into that, I just want to say that was such a masterful talk.
You did a fine job on that talk. I'm, for those of you who haven't seen it, David basically,
David basically pulls up the editor and says, we're going to write a web server and we're going to
explore all our TCP server. And we're going to explore all the ways that you might approach
concurrency from it and the ways we might invent something similar to the, the asyncio that's built
into Python and it was really well done. Okay. Thank you.
Yeah. Yeah. So people, I'll link to that in the show notes and people should definitely check it out,
but maybe we could just kind of talk through some of the ideas of like, if we start with a serial
server and a serial client, like how do we, what are the ways in which we can build up to that? Like
we can use threading, we could use coroutines. There's lots of things, right?
Right. Right. So obviously you start with serial code. It's fine long as you don't have too many
people requesting from the server. Right. But as soon as you have some long request, everything is
blocked. Right. Right. The whole big picture of that talk, I'm going to try to distill it down from
like high, high level view here. It's all about scheduling, basically task scheduling. And if you
have normal serial code, you're executing statements, you're going line by line through the code. If you
hit an operation, like receive, like I want to receive data on a socket or something, that code is going to
block. If it's like, if there's nothing available, it's going to block and you're going to have to wait.
And that really is the gist of the whole problem, which is what happens that, like what happens when
you block? And if you do nothing, then your whole program just freezes. I mean, just everything stops
and then nothing can happen. Some ways around that, one approach to do it is to use threads in the
operating system. Essentially with threads, you're running multiple sort of serial tasks at once.
And if one of them decides to block, needs to receive, well then, you know, the others are still
allowed to run. So you're essentially allowing the operating system to deal with it. It would take
care of scheduling the, scheduling the threads and making sure that things work. The other approach,
and this is something that the talk gets into at the end, is to do it yourself. Don't have the
operating system do it. Take care of that blocking, blocking on your own. And one of the tricks that's
used for that is to use Python generator function. And that is used there is actually just the
behavior of that yield statement. So if you haven't written a generator function before, I think most
people kind of know them in the context of iteration and the for loop. You can write this function where
you use the yield statement to emit values out of a for loop. And the thing that's really cool about
that yield statement is that it causes the function to just suspend itself right there at the yield. It's
like it emits a value and then it suspends. And that's exactly the kind of thing you need to do
this concurrency thing. You can say, well, if there's no data to receive, I can suspend myself.
And essentially, you can take over the role that like an operating system would normally do at that point.
Yeah. And I think that's a really, really interesting insight that you can say, I'm going to take this
thing that really doesn't generate anything and make it a generator anyway. And so you had some
interesting examples of like, we're going to basically simulate parallelism with generators.
And one of the reasons you might care about that is you can switch things over to threads,
but that actually slows things down quite a bit. And especially if there's computational stuff,
you might have to push that out to like do multi processing. And then it really slows down like 10
times what it might normally be. And so you had this great example that maybe we could talk about
just a little bit where you say like, let's just come up with a generators that can count down from
like, you give it a number like 10, it'll count down 10, nine, eight down to one. And then it's done.
And you generate a bunch of the well, you generate multiple of these with different numbers and whatnot,
and put them all into like a task list of things that have to be run. And then you one at a time,
sort of round robin work with those generators. And I think that really highlights, here's how this event
loop can work, right? We can actually process these in a semi fair way across all these different
generators, right?
Right. But you can like cycle, you know, you can kind of cycle between them.
Yeah, which is really doesn't make a lot of sense when you just have a countdown little thing,
right? But then you say, Okay, well, now let's apply the same idea to functions like a while true loop,
that is going while true, come over here and wait to receive from a socket, then process the response
while true way, you know, and if you put yield statements throughout, right before all the blocking
places, you can kind of accomplish the same thing with the same technique, right?
Yeah, yeah. Okay, I kind of describe it. And I don't know, this might be might be sort of silly,
but the whole approach in that talk, I sort of view it as analogous to maybe like the game of hockey or
something like you've got a, you've got a task, and it's out on the ice, you know, and it's doing its
thing. But if for some reason, it's got to read, it's got to receive and like, it can't proceed.
It gets thrown into the penalty box, right? One of the rules is you can't block and anytime it fails
that like, yeah, you take a blocking penalty, man. So it's like, it's like, okay, you blocked,
you're going to the penalty box, and you're going to sit in the penalty box for as long as it takes
until like some kind of data comes in. And then once some data has arrived, and it's like, okay,
you get to go back out on the ice. So it's very much that model, you know, it's like tasks get to
run as long as there's things to do. But you know, once there's nothing to do, it's now you go, you go
sit in the penalty box. And yeah, I think that's a really interesting analogy. And it's definitely makes
a it's a good way to think about it. The challenge, though, and what it was, I think this is pretty
obvious, right? I'm going through and if I could pull out any task, and I could ask it, hey, do you
have work to do? Then I'm going to let you do it. Otherwise, you go back to waiting, or you go to the
penalty box. And when you decided you have work to do, you can come back out. It's interesting.
But then how do you know when it actually has work to do, right? Like on the socket, how do I know,
or if I'm doing something computational, how do I know that that task is ready to run?
Oh, yeah. Well, to get that, you need the help of the operating system. So there's usually there are
some system calls related to like polling of sockets, like the select call is one there's
things with like, you know, like the poll function, you know, there's there's like low
level kind of event API's in the operating system where you can present it with like a whole collection
of sockets. And you can say, Okay, I have these 1000 sockets. Why don't you watch these? And then
if anything happens, tell me about it. Yeah. And that works really well for sockets. But what if I
gave it like a piece of just a function, a Python function that was computationally expensive?
Yeah, if it's computationally expensive, it's just going to run. It's actually a problem with these,
like this event loop thing. If you have something that runs like, I don't know, it's going to go
mine a Bitcoin or something. It's just going to run. And there's no way to, there's no way to get it
back until it finishes. So yeah, it's true. It's I mean, they're all really the event loop really is
on the same thread, right? I guess there are some things you can do, like you can say, Well,
this part is computational. So we're going to kick that off in some multi processing way,
or something like that, that's possible. You had some kind of socket trick where you're even when
you weren't using sockets, you were using it to signal with your commentary, like if you're going
to do like CPU work somewhere else, in some sense, you turn that back into an IO problem. I mean,
you might have some work that gets that gets carried out somewhere else. And then when it done,
when it's done, it gets signaled on a socket saying, Hey, you know, that thing was done.
Yeah. So okay. Yeah. So yeah, very, it was a very interesting technique. And in the end,
down in the internals, you kind of had to deal with some of the callbacks, but the way it got consumed,
right, it was a pretty, pretty straightforward. So that was your talk. And like I said, people should
absolutely go watch it. It's really quite amazing. And then the other thing that I kind of see as
the frameworkification of this, these ideas, I'm not sure what the origin is, I'll ask you,
which, you know, which one came first, but is this project you have called Curio? Do you want to tell
people what Curio is? Yeah. Okay. So Curio is a library for doing concurrency in Python. I mean,
it exploits a lot of this async and awaits the, I'd say ultimately sits in kind of the same spot as
asyncio, although it has a very different flavor to it.
One of the things that I think is really interesting with Curio is like, there's been,
as we talked at the beginning, there's been these ideas for a long time of doing some sort of
asynchrony through callbacks and things like that. Like with Twisted, we've got asyncio built in the
earlier versions of Python. But in Python 3.5, we have async and await and onward, of course. And you
kind of took all of those ideas and said, let's refresh, let's rethink them. Like, how would this
look? How would this API look if we actually had this version of Python concurrency, not what we had
before, right? Like that's, that was how I was reading it when I was going through it.
Yeah, that's part of it. I mean, okay, so there's a little bit of a complicated, complicated background
on this. So let me back up. In a past life, going back a ways, I was a professor in computer science.
And the main course that I taught was operating systems. And in that course, this is a typical
course where you'd make students cry on some huge project thing. And I was, I was just a bloodbath,
bloodbath of a course. So we'd make people write an operating system kernel and see.
And, you know, that curl had to do all of this stuff, had to do IO and had to do like multitasking
and task switching and all these things. And it turns out that all the stuff in that project
was exactly the same kind of thing that people have to do in these asyncio libraries.
And that's what they're doing. And it's like, they're doing IO and they're coordinating tasks and,
you know, switching stuff. So it's, it's, the problem is essentially the same.
It's just in a different environment instead of down at the low level of C and like interrupt
handlers and device drivers, you're up in Python and it's much higher, higher level, but it's,
it's a similar topic. And having done that, I mean, I've always kind of, kind of had a,
an interest in systems topics. So gave a, you know, a well-known PyCon presentation on the gill
that would have been, I don't know, maybe 2010 or something, something like that. And I also had
done some tutorials at PyCon about coroutines, sort of exploring, you know, this idea of using generators
and, and coroutines for concurrency. So it's been a, been a topic that personally been kind of
exploring for, for a long time. But one of the things that has kind of, I don't know,
bothered me over those years is that all of my presentations on that have been completely out of
line with what is actually going on in Python. I mean, like, if you look at that concurrency talk,
that is not at all how like asyncio approaches concurrency, like approaches that problem. If you look
at a lot of the, you know, a lot of the presentations and things that what the, what they talk about is
they're like, Oh, we have an event loop. And then we put callbacks on top of the event loop. And then
usually there's like a transition into, into a discussion of like futures or promises and like
a whole approach based on like futures and promises and tasks. And it starts getting like a lot of kind of
moving, moving gears. And frankly, I just have not been able to wrap my head around that stuff.
I look at that approach and it is completely different than anything that would have been
taught in an operating systems class. But I've never seen an operating system kernel built on top of
futures, for instance. I actually went and got like my old operating system books about, you know,
not too long ago. And I was like, God, did any of those books talk about futures or promises? And
they're nowhere in any of the, like the operating system texts. Do you, do you see that? So I've
been struggling with this kind of mismatch in a way where it's, where it's like, Hmm, there's this whole
approach with like futures. And then there's this thing that I did in the talk, which is a completely
different, different thing. And I've been kind of fascinated with that kind of mismatch,
you know, like, why is that? Or like, what is, what is, what is going on there? And in some sense,
the curio project is maybe a, maybe it's kind of like a green field project, just trying to do asyncio,
but more in the operating system kind of model where I'm thinking more in terms of like task scheduling and
the structure of how I would do it in like a kernel project,
not in the framework of like the, you know, the futures and promises and callbacks and,
you know, all of that stuff. So that's a big part of that project is like just kind of a re-envisioning
of how this might work.
Right. Okay.
Yeah.
Do you feel like it provides a cleaner model by not making you think about all the callbacks and
futures and stuff like that?
I think it does. I mean, it is really wild at first glance because it pretty much kills off
everything that you're used to. Like, in fact, there are no callbacks in curio at all. It's like,
there's, so there's no callbacks, there's no futures. There's like almost none of that machinery that you see in,
in kind of the asyncio library. You know,
one way I've described it, it's almost like I took like the async and await feature that I got added in 3.5 and it's used it as a starting point for some completely different approach.
Yeah, absolutely. And so it's really,
it's really involving a lot of async and await and coroutines basically. Right.
So a lot of the starting points, a lot of things you want to do, you're like, provide this class with some kind of async method, right?
An async coroutine. And then, then it runs from there. Right. Right. Right.
Yeah. So there, you know, it might, I don't know what people have looked at it or not, but there are a lot of building blocks,
a lot of really nice parts to this library.
When I first thought about it and first checked it out, I kind of thought, okay, this will be,
maybe it's got like a core event loop and it's doing a few things differently, but there's a lot of building blocks here.
You can build some really interesting things with it.
Yeah. There's some, some really odd stuff going on. I don't know.
It's like how we want to get into that. But one thing about actually getting back to the operating system model on that,
these async libraries, I mean, let's see if this makes sense. A lot,
a lot of these async libraries are kind of like an all in proposition where like you're either coding in the asynchronous world or you're not.
And it tends to be kind of a separation between those two worlds. Like it, even,
even if you're working with callbacks, it's kind of like you have to program in kind of the callback style or you're,
you're kind of out of luck. And I think Curio kind of embraces that as well.
I mean, one of the, one thing from operating systems is there's usually like a really strict separation between what is the operating system kernel.
And then what is like a user space program,
even at the level of like kind of like a protection kind of thing, like, you know,
like a user program isn't even allowed to like see the kernel in any meaningful way.
I mean, it's, it's, it's like very strict separation.
That is also something that's going on in the, in this Curio project.
I mean, there's kind of the world of async,
you have all these async functions and await and all of that.
And then there's kind of the kernel and like those two worlds are like really separated from each other in this,
in the Curio project.
So that's all, that's another kind of unusual thing about it.
Yeah, that is really interesting.
And I, I, now I can see the, the operating system analogies and there is a kernel,
a thing you actually call the kernel in Curio, right?
Yeah.
This portion of Talk Python to me has been brought to you by Rollbar.
One of the frustrating things about being a developer is dealing with errors,
relying on users to report errors, digging through log files, trying to debug issues,
or a million alerts just flooding your inbox and ruining your day.
With Rollbar's full stack error monitoring, you'll get the context insights and control that you need to find and fix bugs faster.
It's easy to install.
You can start tracking production errors and deployments in eight minutes or even less.
Rollbar works with all the major languages and frameworks, including the Python ones,
such as Django, Flask, Pyramid, as well as Ruby, JavaScript, Node, iOS, and Android.
You can integrate Rollbar into your existing workflow, send error alerts to Slack or HipChat,
or even automatically create issues in Jira, Pivotal Tracker, and a whole bunch more.
Rollbar has put together a special offer for Talk Python to me listeners.
Visit rollbar.com slash Talk Python to me, sign up, and get the bootstrap plan free for 90 days.
That's 300,000 errors tracked all for free.
But hey, just between you and me, I really hope you don't encounter that many errors.
Loved by developers at awesome companies like Heroku, Twilio, Kayak, Instacart, Zendesk, Twitch, and more.
Give Rollbar a try today.
Go to rollbar.com slash Talk Python to me.
Basically, you use the constructs and you pass these async coroutines to it, and that's that.
Okay.
So if you're doing, like you said on the all-in part, if you write some sort of coroutine,
and it comes down to a point where you're either doing something computational or blocking,
and you block there, you kind of take everyone out, right?
What do you mean take everyone out?
Well, you can clog up the event loop.
Oh, yeah.
Yeah.
Definitely.
Yeah.
I mean, you can throw a wrench into the hole.
Hey, we're just going to keep going and letting it.
You do your work, and when you're done, come back, and then we'll pick up where you left off.
Yeah.
So how do you deal with that?
Like, if I've got some async coroutine I want to run in Curio, and it's got something computational,
how do I make that work?
Sometimes you just have to do something that's going to take a while, right?
Yeah.
I mean, if it's concerned, you have to punt it out either to a thread or another process.
Yeah.
Okay.
Like with multiprocessing or something?
Yeah.
It's kind of the standard technique for all these async things.
It's like you've got something computational, and it's going to block.
Yeah.
You've got to punt it out somewhere.
And then do you have a way, like a construct, to await a thread or await some kind of multiprocessing call?
Yeah.
There's a function in there.
You can ask to run something in a thread or you can run something in a process.
Okay.
It will take care of it and wait for the result to come back, but it won't block the sort of internal loop, basically.
Okay.
Yeah.
Very nice.
So if we have some time, we could talk about some of the individual building blocks, but what can you build with Curio?
Curio, do you think?
I mean, it looks like it's a little bit below like a web framework, but it's close to a framework for building asynchronous programs on its own.
Like where do you think this fits in?
Okay.
Yeah.
It is definitely not a web framework.
In fact, I don't even, I think there's any HTTP support in it right now.
So I think it is more of a framework for concurrency, you know, actually setting up tasks, communicating between tasks, coordinating things.
It's the kind of thing that you might start building libraries on top of, you know, maybe libraries to, you know, interact with Redis or interact with databases or even to do HTTP.
But it's definitely like a lower level, a lower level thing.
So it's, you know, a lot of like coordinating tasks and things of that name.
I see.
So if I wanted to create some framework that was backed with Redis, I could use Curio to make a really nice async and await framework and somehow do the network IO internally.
And people might not even know that it's Curio, right?
They might just know my framework and it talks to Redis and internal part of that could be Curio.
Yeah, maybe.
Yeah.
I mean, I sort of see it personally as something I might implement a lot of sort of microservice code with.
Yeah.
Like a lot of little like web services and stuff, things like that.
Done a little bit of playing around trying to implement a game server with it.
Sure.
Like a socket-based game server.
Yeah.
Socket-based game server.
I think there's a lot of uses with like testing that kind of, I think there's a lot of people,
who do network programming that's not necessarily web programming.
And so I think it kind of fits into that.
Yeah.
And it has really good support for TCP and UDP type stuff, right?
Right.
Okay.
If I wanted to take a framework that maybe I'm already using, let's say Django, Flask, Pyramid, something like that, that doesn't have any support for this idea of concurrency or async await,
could I somehow use Curio in my own code?
If I'm willing to sort of do some kind of callback mechanism or like notification mechanism in my web app, right, for like asynchronous stuff?
Or would those things just not make any sense together?
I think it would be tough.
If you've got code that was written originally for kind of the synchronous world, getting that into any async framework, I mean, even asyncio or anything like that can be, it can be kind of a tough proposition just because there's just the programming model can be very different.
And you have to instrument a lot of code with like these async and await calls.
Right.
It's unclear.
Yeah, yeah.
I'm thinking like if you have like WebSockets or some kind of call and then pull JavaScript thing, like maybe those parts of it could somehow use Curio?
Maybe.
Yeah, maybe.
I don't know.
Maybe.
I mean, you actually kind of this opens up kind of a, I don't know, an avenue of discussion, which is, you know, actually where does this thing, you know, where does it fit in kind of the grand scheme of things?
And I think, you know, one thing with these async frameworks is just stepping back for a moment and thinking like, okay, where, like, what is the use case for these things?
Or like, what are they really good at?
And one of the things that they're really good at is handling a gigantic number of connections, like a high degree of concurrency.
Right.
Where you might have like, like, let's say I had 100,000 clients connected to some server and I've got to maintain, you know, some pool of, you know, 10,000, 100,000 socket connections.
That is where these async things tend to shine because you can't just spin up like 10,000 threads.
Yeah, you can't.
Just the memory required for the stack space would be problematic, right?
Right, right.
So they're really good at that.
But so you have like a high degree of concurrency.
But at the same, you know, even though you have like a high, like a lot of clients, it doesn't mean that those clients are all doing things at the exact same time either.
I mean, you might have like a server that has like, look, it has like 100,000 connections open, but maybe it's doing sort of push notifications or kind of low traffic stuff.
I mean, it's not like you're going to have like 100,000 connections kind of open, just completely hammering your machine all the time.
Right.
Something like Slack or something, right, where everybody's got it open, but the amount of traffic is actually quite low.
But you want it to be basically instant, right?
Right.
So the kind of stuff I'm thinking about is like, okay, so maybe let's say you have like 100,000 connections open.
Could you still use something like Flask or Django or something?
Like, could you still use that in some capacity?
Now, I mean, you can't, you're not going to be able to spin up like 100,000 threads running Django or whatever, but could you have some coordination between, you know, these like tasks and something like Curio or AsyncIO and coordinate that with maybe a smaller number of threads or processes or whatever that are running like a more traditional framework?
Yeah, exactly.
That was what I was thinking.
I'm not sure if it's possible, though.
I don't know.
I mean, so one thing in Curio that I think is actually one of the more interesting parts of the project is I'm trying to do a lot of coordination between async and traditional thread programming.
As an example of that, one thing that Curio has is it has this universal Q object.
This is probably one of the most insane things in the whole in the whole library, but a standard way to communicate between threads is just to use a queue.
Right, because shared shared data is problematic, right?
You've got a lock on it and all sorts of stuff.
Right.
So you have a queue and, you know, you like share between threads.
So there's this thing in Curio that lets, that basically allows queuing to work between async tasks in Curio and thread programs in a really like seamless way.
Like essentially the thread, the thread part of it just thinks it's working with a normal queue and everything works normal.
And like the Curio side works with like a, like an async queue and it thinks that everything is kind of normal.
And you get this like queuing going back and forth between kind of the two worlds.
And it's kind of, it's, it's sort of seamless in a really disturbing way.
It's a, it's like maybe you could have like, you know, a hundred thousand tasks kind of managing sockets, but then talking to some pool of threads through queuing and it, and it all kind of works.
And it's, this is sort of an area that is not, at least as far as I know, not being explored more traditionally in asyncio, for instance.
They have a queue there, but it's not compatible with thread.
Right.
So this is a really interesting idea.
This universal queue, it's kind of like a dual facade, right?
The, the different worlds can see it as part of theirs, right?
Yeah.
Somebody contributed a feature to Curio to allow it to submit work to asyncio.
Okay.
So like you could have Curio and you could have threads and you can have asyncio.
This queuing object in Curio actually works in all three of those worlds.
Done some tests on that.
So like you could have a queue where one end of the queue is like an asyncio task and the other end is a thread.
And then, or, you know, you could have like a thread and a Curio task putting things on a queue that's being read by asyncio and other, other things.
That's a really kind of wild, crazy thing to be playing with.
I mean, it's, yeah, that's really interesting.
I, you know, I didn't know about universal queue, but the library is full of these, these really amazing little data structures and, and functions and stuff.
So it's, it's quite neat.
I mean, the other thing that's kind of wild and well, they're kind of, kind of interesting in Curio too, is just the, the way that the task model works and has a lot of support for things like cancellation of tasks.
Right.
And that turns out to be a really tricky problem.
It's like, okay, you set up a whole bunch of work and then you want to cancel it.
Can you do that?
And that's something that you can do in Curio.
And I, it's, it's very interesting because it's something that you can't do with threads traditionally.
Right.
If you just kill a thread, maybe it's holding onto some kernel level thing and you forced it to just, you know, leak it or whatever.
Right.
It's going to be bad.
You have a way to, to even kill a thread.
Um, I mean, there's no API for killing a thread and then I think some people have done it going through C types, but that, that just makes my skin crawl.
It's like, like killing threads by going through C types seems like a really good way to just not have your program work.
So.
Yeah, exactly.
Like if it's holding onto something important, like let's say, imagine it's holding onto the GIL and you kill it.
What happens then?
That might not be great.
Yeah.
Okay.
So this cancellation thing, you're right that that is not simple.
How does it work?
Is it like basically every time one of these async coroutines yields or if it's not in a running state, you can just say, okay, this is getting canceled?
Pretty much.
That's it.
Yeah.
Since every operation requires the, you know, kind of the support of this kernel, if somebody wants to cancel something, I mean, as soon as it, you know, if somebody wants to cancel something, it's either blocked in there already.
Or you can just wait for it.
Or you can just wait for it to finish.
Um, yeah, you can, you can essentially just, just blow it away.
I mean, it's, you raise an exception at the yield statement saying, okay, you're done.
Yeah.
Okay.
That's what I was, that's always my next question is like, you can't just take it out of the running task list and throw it away because it might, might've been in the middle of something that needs to be unwound, like created a file and it needs to close the handle or something.
Right.
So you, you basically just raise like a task cancellation exception or something.
Uh huh.
Okay.
Yeah.
So you get, it gets a cancel there or, and then it, it can choose to clean up if it wants, but it's sort of a graceful, you know, graceful shutdown from that.
Yeah.
That's a nice, that's a really nice feature.
So let me ask you about integrating with some other things.
Like I'm, I have databases on my mind right now for, for some reason.
So there's a bunch of nice ORMs or ODMs.
If you're doing a SQL and Python, you know, SQLAlchemy, Mongo engine, Peewee pony, and so on.
The one ORM that I've seen that seems to integrate really nicely with async and await is Peewee ORM.
You can basically await on the queries that you're getting back from it, which is super cool.
Would Curio integrate pretty seamlessly with a framework like that?
I don't know.
Do you know how they, how they're doing that under the cover?
I looked at Peewee before we met here because you mentioned it.
I didn't see that feature off the top of my head, but.
To basically the extent that I know is I've seen reference to it where it basically supports async and await on, on the queries, right?
The things that it's going to talk to the database, but I don't know what's happening internally there.
Okay.
Yeah.
I don't know.
I would have to take a, I would have to take a look at it.
My initial guess is it probably would not work just because Curio is so out in left field right now.
It's, you know, like if they've written, if they've written that specifically to work on top of asyncio.
I see.
Yeah.
So it might not, right?
Hit or miss on that.
Yeah.
I mean, the API is you just have an async method and you just await either object.create or objects.query and so on.
But I don't know what's internal.
It's probably asyncio, I would guess.
Yeah.
This portion of Talk Python to Me is brought to you by Hired.
Hired is the platform for top Python developer jobs.
Create your profile and instantly get access to thousands of companies who will compete to work with you.
Take it from one of Hired's users who recently got a job and said,
I had my first offer within four days and I ended up getting eight offers in total.
I've worked with recruiters in the past, but they were pretty hit and miss.
I tried LinkedIn, but I found Hired to be the best.
I really like knowing the salary up front and privacy was also a huge seller for me.
Well, that sounds pretty awesome, doesn't it?
But wait until you hear about the signing bonus.
Everyone who accepts a job from Hired gets a $300 signing bonus.
And as Talk Python listeners, it gets even sweeter.
Use the link talkpython.fm/Hired and Hired will double the signing bonus to $600.
Opportunity is knocking.
Visit talkpython.fm/Hired and answer the door.
I've been thinking about this.
I mean, the same problem.
I'm not familiar with the PeeWee ORM, but I am familiar with SQLAlchemy.
Yeah, that was my next question.
What about things like SQLAlchemy or Mongo Engine that have no concept of this at all?
Could we somehow shoehorn them into working with Curio or these types of things?
Yeah, maybe.
Keep in mind, it's highly experimental.
It's highly experimental.
And what I'm about to talk, I mean, may not work.
But one thing that I've been playing with in Curio is this, there's a concept in there known as an async thread.
I mean, at first glance, it's like, oh, God, this is insane.
I thought that was wonderful.
That's really cool.
Oh, no.
Async threads are nuts.
Let me see if I can explain.
Okay, so in threads, with thread programming, you have threads, and then you have all these asynchronous primitives.
Or you have all these synchronization primitives.
You have locks and cues and semaphores and all this stuff.
So in thread programming, you have all this stuff that you normally use to write programs.
It turns out that almost all of that functionality is replicated in these async libraries.
I mean, like if you look at async.io, it has like events and semaphores and locks and cues and stuff.
And Curio has events and locks and cues and all that stuff.
But the limitation of that, of the async libraries, is that those things don't work with threads.
Like if you read the docs, it has like this huge warning on it.
It's like this is not thread safe.
You know, if you use a thread with this, you're going to die.
You'll be sorry.
Yeah.
So you have all these things that you would normally use with threads in these async libraries, but you can't use them in thread code.
So I've got this idea where I wonder if you could like flip the whole programming model around where you could create like an actual real live thread.
But then have the thread like sitting behind the thread.
You could have like a little task that interacts with the event loop.
Like it interacts with the async world.
Right.
So instead of being having a bunch of processes and the event loop is running on one of them or not a bunch of threads.
And you're like the event loop controls all the threads in a sense.
Right.
So what you have is you have like one event loop.
But then you have like a real thread.
Keep in mind this would be like a POSIX thread, a real life, like fully realized thread.
But like sitting like right next to that thread out of view, like out of sight would be like a little tiny, like asynchronous, like a little task on the event loop that is watching for the thread to make like certain kinds of requests.
And I was thinking, it's like, what if you took like in the thread, you took all these requests for all these synchronization primitives and you just kind of handed it over to this like little helper on the side.
And then you let it interact with the with the event loop.
And the thing that's really kind of wild about that.
So so Curio supports this concept.
It turns out you get all of these features with like tasks and stuff in Curio showing up in threads like you can cancel threads.
You can do all the synchronizations with threads and all these all these other things.
I've been I've been thinking about that in the context of some of this database, you know, this database stuff.
Like, let's say I did want to interact with something like SQLAlchemy.
Maybe I could have like a pool of threads or something that that would take care of the SQLAlchemy side of it, but then kind of coordinate it with with sort of tasks on the event loop through this kind of acing thread mechanism.
That's very interesting, like some kind of adapter that kind of looks like SQLAlchemy, but really routes over to another thread where maybe it creates the session and it does the all the filtering order by query stuff and then brings it back over when it's when it returns or something like that.
Yeah, maybe I'm not even sure it would be an adapter, but I'm kind of thinking of like this.
The model in my mind is that, OK, if you're using async, you know, let's say you did have a server and you've got like 10,000 connections sitting there.
It's extremely unlikely that I'm going to have 10,000 or that I would want to make 10,000 concurrent requests on the database.
I mean, most of these connections are probably sitting idle most of the time or doing other things.
So I'm thinking like, well, maybe I could have like, you know, maybe 100 threads or, you know, maybe that's too many.
But you could have like a pool of threads that are sort of responsible for doing the database side of it.
And then you could coordinate that with this, you know, 10,000 asynchronous tasks in some way.
So it's going to be kind of a hybrid model where like some of the work takes place in threads and other work takes place in these async tasks.
But it's kind of done in a more kind of seamless way.
Well, it sounds really cool.
I'd love to see it and try it out, but I don't know if it'll work either.
Yeah, I don't know.
Yeah, they're finding the time sometimes to explore that.
It's a challenge for sure.
Yeah.
Yeah, but if you could take these really popular, really nice frameworks like SQLAlchemy and somehow click them into this world without actually rewriting them from the ground up, that would be really cool.
Yeah, I agree.
I think that'd be I think it'd be fun.
Yeah.
It's something I want to try.
I mean, I've got this sort of web service project that I did a couple years ago.
And right now it's sitting on Python 2.7 with SQLAlchemy and a bunch of stuff.
And I look at that and it's like, man, I really want to rewrite this thing.
You know, in Python 3.6 with Curio and try this thing.
It's just, you know, there's only finite resources in a day.
So it's still on the to-do list to sort of get to that.
Sure.
But some sort of framework that bridge that divide, I think, would be generally applicable in a lot of places.
And a lot of people would be excited about it.
Very cool.
So you talked about the AsyncIO module a lot.
Curio is not built on it, right?
It's kind of something different?
It is a completely different universe.
Okay.
It doesn't use AsyncIO.
It's doing its own thing.
That was mostly because you said you want this to be like a greenfield.
Like, how would the ideal world of the space look like?
Not like, let's pick up the old model and see what we can do with it?
Yeah, partly that.
Yeah.
And partly, you know, I'm trying to, you know, actually, Curio is kind of a project where I'm trying to learn about this stuff myself.
You know, just trying to learn, like, you know, what is Async and Await all about in Python?
What can you do with it?
Or how can you abuse it?
You know, sort of, you know, what kind of horrible, insane things are possible with it?
In some sense, you know, Curio is a project exploring a lot of that.
You know, exploring a lot of ideas about APIs.
And really kind of even the programming environment itself.
So it's not built on AsyncIO.
And it's, I don't even think it's really meant to clone AsyncIO.
I mean, it's kind of its own thing right now.
Right, sure.
Okay.
So where are you going with this in the future?
What are your plans?
Well, part of the plan is just figuring out how to write about this in books.
I mentioned I'm supposed to be updating my Python book.
So a big part of it is I'm thinking about just how to approach Async and Await in the context of writing and teaching.
And so there's that element of it.
Actually, do you want a little rant on that, by the way?
Yeah, go for it.
I have convinced myself that the approach of teaching Async needs to be flipped in some way.
And let me describe what I mean by that.
If you see a typical tutorial on a lot of this Async programming, it ends up being this very kind of bottom-up approach.
Where it's like, okay, you have sockets, and then you have the event loop, and then you start building all this stuff on the event loop.
You have callbacks, and then it's like, oh, we have callbacks, and we have futures.
And then you start layering and layering and layering and layering.
And then at some point, you reach this like, oh, and we have Async and Await.
Yeah.
And it's like Async and Await is, yeah, finally, oh, it's awesome.
You have Async and Await.
The problem with this approach is that I just have never been able to teach it.
I mean, I've tried this in classes, kind of doing the bottom-up approach to this Async stuff.
And every single time, it seems like you get about halfway through it, and then you're just looking at a room with deer in the headlights.
Yeah, you've gone through this strainer, and you've stripped everybody's interest out by the time you get to the interesting part.
Oh, it's horrible.
So you're just looking at all this deer in the headlight look, and it's like, oh, oh, God.
You think about it, it's like, okay, wait a minute.
Okay, so let's say I had to describe file I.O. to somebody.
Like, you open a file on your computer.
Like, you open a file in Python, and you read from it.
Is my description of that going to start with, like, well, okay, you have CPU registers.
And what you do is you load the CPU registers with, like, a system call number, and then, like, a buffer address.
And then you execute a trap.
And then the trap goes into the operating system.
And it's going to do something.
I don't know.
It'll find, like, a file I know.
And it'll probably check the buffer cache.
And then, you know, it'll go do some stuff with the disk scheduler and bring some stuff in.
And then there's, like, copying.
And then is that the description of, like, how I'm going to describe file I.O. to somebody?
All right.
People are like, I just want to read JSON.
Exactly.
I'm like, I'm thinking of, you know, so this is my thinking on async, too.
It's like, does anybody actually care how this stuff works?
Like, seriously, like, do you care that there's an event loop or a future or a task or whatever it is in there?
And I'm not sure that you do.
I'm almost wondering whether, like, the approach to teaching this async stuff is to do, like, this total, like, top-down thing.
You just, like, you basically say, hey, you have async functions and you have a wait.
And you just start using it.
Yeah.
You don't even say, like, don't even mention, like, generators or coroutines or the yield statement.
Yes, it's built on that.
But do you care?
Yeah, I think you're totally right.
I mean, you probably care in, like, six months once you've been using it a while.
You might want to look inside, right?
But when you don't even know what async and await is, you're right.
You absolutely don't care.
I mean, it seems like let's write a program, show that it's blocked, show that we unblock it with async and await.
Awesome, right?
That could be the way to get started.
Yeah.
So I've been thinking about that a lot, you know, in the context of, like, the book writing.
It's like, hmm, how am I going to – how do I bring async and await into this book project?
It's like, am I going to do it – you know, am I going to go the top-down approach where it's going to require kind of a leap of faith where it's like, yeah, okay, you just do it.
It's like a file.
It's like you don't care.
Like, in my whole programming life, have I ever cared how, like, a system call works?
No.
The answer is no.
Like, I mean, other than teaching the operating system class, I have never once cared how a system call works.
And I kind of feel maybe the same way about async and await.
It's like if you approach it in the right way, you know.
Yeah, it doesn't have to be so daunting, right?
Yeah, it doesn't have to be so daunting.
And actually, in that context, I'm almost wondering whether something like the asyncio module is like an – it's like assembly code phrasing.
Yeah, a little bit.
Yeah, yeah, yeah, a little bit.
Yeah.
Very overwhelming, right?
I get in there and it's like, oh, coroutines wrapped by futures and tasks and blah, blah, blah, blah.
And you're like, ah, like head is exploding.
It's like, eh.
It doesn't have to explode, right?
Yeah, maybe you don't even need to know that stuff.
Yeah, that's awesome.
So what book is this that these ideas are going to land in?
Ultimately, it will be the Python Essential reference book.
Nice.
Awesome.
I'm trying to figure out how to put async and await into the first chapter, which is like tutorial introduction.
Yeah.
I got this idea where it's like, yeah, I'm just going to drop it in the tutorial like right away and just see what I can get away with.
Yeah, that'd be cool.
Just set the tone like, no, this is kind of normal now.
We're doing this now.
Yeah, it's just this normal, yeah.
Like what else would you use?
Yeah, sure.
No, I don't know whether I can get away with that or not.
Well, if you want a vote of one for your flipping this presentation or the presentation style for how you present it to people, I think that's the right way to do it.
So it's unanimous.
Yeah, okay.
In some sense, a curio project is kind of experimenting with that too.
You know, it's, I don't know, focusing more on the async and await side of the equation as opposed to the low-level mechanics.
Yeah, it's a really cool project.
And I like where you're going with it.
It's definitely worth checking out to understand this whole async world better.
So, David, I think we're getting pretty much out of time.
Don't want to use up your whole morning.
So let me ask you a couple of questions real quick, as I always do.
Yeah.
I think I can guess this from your presentations.
But if you're going to write some Python code, what editor do you open up?
It's got to be Emacs.
Emacs, right on.
And favorite PyPI package in addition to curio, of course.
Is curio on PyPI or is it just installable from GitHub?
It's on there.
It should probably be a GitHub version if you're going to do anything interesting with it, though.
It's moving pretty fast.
It's moving along.
Yeah.
I don't always update it.
Think of it more like this.
Like, if people maybe don't know about a package that you recently found, you're like, this is really cool.
You guys should try this out more than just the popularity contest.
One of the goals of curio is not so much curio itself, but to basically change a lot of the thinking around async and await.
You know, it really is kind of an exploratory project where it's like, let's see what we can do with async and await that's maybe outside the context of asyncio.
And this trio project is something that has been kind of inspired by curio, if you will, is taking things in a slightly different direction.
So I would recommend people look at that.
All right.
I mean, if you're interested in, like, concurrency and some of this async stuff, it will give you yet a third spin on the whole universe.
So that's also an experimental project, but maybe I would advise that.
All right.
Yeah.
Very, very cool.
And speaking of packages, I think the PeeWee async thing that I was talking about might be a separate package.
I'm not sure if it's built in.
So just be aware of that.
All right.
Final call to action.
Are you looking for people to contribute to this project?
What can people do now that they're, you know, know about curio?
Oh, it's definitely something that I'm looking for contributors.
I think the place where a lot of contributions could be made on a package are more in supporting some of these other networking protocols.
So getting it to hook up with things like Postgres, MySQL, Redis, ZeroMQ, things like that.
There's a whole, you know, there's kind of a whole space of things that could be done there.
Really big project would be interesting would be support for HTTP.
Yeah.
Like some sort of a WSGI integration.
Yeah.
And that might be a whole separate podcast because there is like a whole kind of interest in HTTP and HTTP2 right now where, you know, people are implementing the protocols independently of the actual IO layer.
This would be like Corey Benfield's work.
And I think Nathaniel Smith is also working on this with like HTTP2 where, you know, he's implemented the protocol as its own library.
But then the protocol can be used from threads or used from async or used from twisted or used from different places.
And that's actually a really, that's a really interesting avenue of work.
Yeah.
That's, yeah.
Thanks for recommending that.
That's cool.
So maybe people could use that to build on top of or something for the HTTP layer.
Right, right.
For their framework.
Yeah.
There's been some work with that in Curio already.
People, they've, it's actually been shown that you, I mean, you can use those libraries from, from Curio,
but it has not been sort of packaged at a level of into, into like what I would call like a nice framework.
Sure.
Kind of at a, we're kind of operating at a lower level right now.
And it's, you know, turning it into more of like a framework.
It's a whole different question really.
So.
Yeah.
Well, it's definitely a really great start.
And if it turns into one of those frameworks, I would, I would love to play even more with it.
So very nice work on Curio.
David, thank you for coming on the show to share all this async stuff with us.
All right.
Thank you very much.
This has been another episode of Talk Python to Me.
Today's guest has been David Beasley.
And this episode has been sponsored by Rollbar and Hired.
Rollbar takes the pain out of errors.
They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until your users complain, of course.
As Talk Python to Me listeners, track a ridiculous number of errors for free at rollbar.com slash Talk Python to Me.
Hired wants to help you find your next big thing.
Visit talkpython.fm/Hired to get five or more offers with salary and equity presented right up front and a special listener signing bonus of $600.
Are you or your colleagues trying to learn Python?
Well, be sure to visit training.talkpython.fm.
We now have year-long course bundles and a couple of new classes released just this week.
Have a look around.
I'm sure you'll find a class you'll enjoy.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm.
Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.
Corey just recently started selling his tracks on iTunes, so I recommend you check it out at talkpython.fm/music.
You can browse his tracks he has for sale on iTunes and listen to the full-length version of the theme song.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Smix, let's get out of here.
Stating with my voice, there's no norm that I can feel within.
Haven't been sleeping, I've been using lots of rest.
I'll pass the mic back to who rocked his best.
First, develop, first, develop, first, develop, first.
You're welcome.
Developers, developers, developers, developers.
.