WEBVTT 00:00:00.001 --> 00:00:06.040 You've heard me go on and on about how Python 3.5's async and await features change the game 00:00:06.040 --> 00:00:10.820 for asynchronous programming in Python. But what exactly does that mean? How does that look in the 00:00:10.820 --> 00:00:17.400 APIs? How does it work internally? Today, I'm here with David Beasley, who has been deeply exploring 00:00:17.400 --> 00:00:22.780 the space with his project Curio. And that's what this episode of Talk Python To Me is all about. 00:00:22.780 --> 00:00:27.320 It's episode 107, recorded April 14, 2017. 00:00:27.320 --> 00:00:56.920 Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the 00:00:56.920 --> 00:01:01.540 ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter, 00:01:01.540 --> 00:01:06.460 where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm 00:01:06.460 --> 00:01:12.880 and follow the show on Twitter via at Talk Python. This episode is brought to you by Rollbar and 00:01:12.880 --> 00:01:20.180 Hired. Thank them both for supporting the show. Check them out at Rollbar and at Hired underscore 00:01:20.180 --> 00:01:25.100 HQ on Twitter and tell them thank you. David, welcome to Talk Python. 00:01:25.420 --> 00:01:32.180 Hi, how are you doing? I'm doing great. It's great to have you back. It's been going on two years since 00:01:32.180 --> 00:01:38.000 you were on one of my first episodes, episode number 12, talking about packaging and modules and 00:01:38.000 --> 00:01:43.660 diving deep into understanding those. And I think we're going to be diving deep into another topic, 00:01:43.660 --> 00:01:47.340 another area of Python today, but this time in concurrency. 00:01:47.340 --> 00:01:50.720 Yeah, it should be fun. It's something I've talked about in the past. 00:01:50.900 --> 00:01:55.540 Yeah, yeah. You've definitely been talking a lot about it lately and amazing presentations, 00:01:55.540 --> 00:02:01.060 which we'll get to. I know many people know you. It's been two years since I last asked you this 00:02:01.060 --> 00:02:06.060 question. So maybe just briefly, you could tell us how you got into Python programming, 00:02:06.060 --> 00:02:07.000 that sort of thing. 00:02:07.000 --> 00:02:12.700 All right. Well, not to get into too much detail, I guess I first found Python in 1996. 00:02:13.540 --> 00:02:17.600 And I was doing some scientific computing, a lot of parallel computing kinds of things, 00:02:17.600 --> 00:02:21.720 and just found it for doing, you know, like basically found it for scripting scientific 00:02:21.720 --> 00:02:27.040 calculations. And it kind of grew from there. In the more modern era, you know, I'm known for 00:02:27.040 --> 00:02:30.240 writing a couple of Python books. So that's where people sort of know me. 00:02:30.240 --> 00:02:35.200 Yeah, absolutely. What books are the most famous ones you've written, most well known? 00:02:35.460 --> 00:02:40.360 Yeah, the Python Essential Reference, that's been around for a while. And then they did the third 00:02:40.360 --> 00:02:42.300 edition of the Python Cookbook with O'Reilly. 00:02:42.300 --> 00:02:48.200 Okay, excellent. Yeah, those are both great. And these days, what are you doing in terms of 00:02:48.200 --> 00:02:50.580 work and programming with Python and other things? 00:02:51.020 --> 00:02:56.760 Most of my work is training, actually. So I do a lot of teaching of Python classes. That's what's 00:02:56.760 --> 00:03:02.960 mainly paying the bills. And then also funding, you know, sort of hacking on various open source 00:03:02.960 --> 00:03:05.820 projects and the other time when I'm not doing that. 00:03:05.820 --> 00:03:09.700 Oh, that's great. Yeah, I've, you know, done a lot of training, a lot of in-person training 00:03:09.700 --> 00:03:15.680 previously. And I thought it was just, I think it's a really great career. I think it's a perfect 00:03:15.680 --> 00:03:22.560 balance or a close, a great balance, let's say, where you get to teach things to people, 00:03:22.560 --> 00:03:29.000 see their reactions, see how they take it, see, you know, they kind of test your understanding of 00:03:29.000 --> 00:03:33.240 it. That's part of your job. And then the part is just to research and learn and stay on top of 00:03:33.240 --> 00:03:35.640 whatever it is you're teaching. It's really nice, I think. 00:03:35.640 --> 00:03:40.100 Yeah, I'm always trying stuff out with teaching. I mean, I find it, you know, it kind of informs like 00:03:40.100 --> 00:03:45.120 talks. Think of it, you know, at conferences and also informs books and things. You know, a lot of, 00:03:45.180 --> 00:03:48.720 you know, a lot of what people would see at a conference is probably something that I've tested 00:03:48.720 --> 00:03:54.840 out in the context of teaching, or I've tried to do it different ways and just kind of seen people's 00:03:54.840 --> 00:04:01.800 reactions and confused looks. This isn't working. We'll get to try this. Of course, the iteration is 00:04:01.800 --> 00:04:06.380 so much faster, right? You could teach two or three classes in a month. How many conference talks do you 00:04:06.380 --> 00:04:09.660 give in a month, right? Or how many books do you write in a month? Not nearly as many. 00:04:09.820 --> 00:04:15.740 Right, right. I mean, I'm supposed to be working on a book update right now. And it's, it's going slowly. 00:04:15.740 --> 00:04:19.620 But, you know, I'm thinking a lot about these topics, you know, like how to present material, how to think 00:04:19.620 --> 00:04:20.080 about it. 00:04:20.080 --> 00:04:24.820 Yeah, that's great. Yeah. So let's go ahead and start talking about our main topic here, which is 00:04:24.820 --> 00:04:32.000 concurrency. And maybe we could start by just kind of talking about the concurrent options in Python 00:04:32.000 --> 00:04:39.120 in general a little bit. And how do you feel that we're doing in 2017 with Python 3.5, 3.6, 00:04:39.120 --> 00:04:41.620 compared to, say, five years ago? 00:04:41.620 --> 00:04:46.000 Oh, okay. That's an interesting question. Well, I mean, the gist of the problem with concurrency is 00:04:46.000 --> 00:04:51.940 it's doing more than one thing at a time. I mean, that's the basic problem. And it comes up a lot in 00:04:51.940 --> 00:04:58.060 network programming, especially. So that's where a lot of people are very interested in it. Python has 00:04:58.060 --> 00:05:03.060 certainly been involved with concurrency for a long time. I mean, threads have been part of Python 00:05:03.060 --> 00:05:10.800 since 1992, I think. So, you know, that goes way back. And there's certainly been the option of, 00:05:10.800 --> 00:05:15.360 you know, launching separate Python interpreters. I mean, you could have like multiple interpreters 00:05:15.360 --> 00:05:18.960 might be a way of doing that. So these, these have been kind of classic approaches that have been 00:05:18.960 --> 00:05:25.040 around for a while. Kind of, kind of a side, alongside that, you have a lot of people messing around 00:05:25.040 --> 00:05:30.080 with things like callback functions, event loops, packages based on that. So things like the 00:05:30.080 --> 00:05:36.200 twisted framework sort of emerges out of that. So a lot of this has been going on for quite some time. 00:05:36.200 --> 00:05:41.660 I mean, maybe over just throughout, throughout Python's history. This question about where it 00:05:41.660 --> 00:05:46.040 goes in Python 3. I mean, it's kind of a, I don't know, I'm trying to think like how to, how to 00:05:46.040 --> 00:05:51.060 chew on that. It's a big question, huh? No, it is a big question because you've got the, 00:05:51.260 --> 00:05:57.960 well, I mean, obviously the big development is the asyncio library. I mean, you get that added to 00:05:57.960 --> 00:06:05.400 Python and that is tying together a lot of, a lot of ideas from different places. I mean, a lot of, 00:06:05.400 --> 00:06:12.720 a lot of concepts about, you know, event loops and generator functions and coroutines and all these 00:06:12.720 --> 00:06:18.920 things are kind of coming together in, in that library. There's a lot of excitement around that 00:06:18.920 --> 00:06:24.180 library, but it's also a really difficult concept. Like that's a very difficult library to wrap your 00:06:24.180 --> 00:06:30.240 brain around. Yeah. It takes a bunch of things that are individually pretty conceptually hard and then 00:06:30.240 --> 00:06:35.460 puts them all together. Yeah. I actually realized that, you know, in hindsight, I never really quite 00:06:35.460 --> 00:06:41.900 understood that library the first time I heard about it. I mean, I watched Guido give a keynote talk 00:06:41.900 --> 00:06:50.640 about it at PyCon and I'm trying to think which one that was. It might've been 2012, maybe 2013. And my 00:06:50.640 --> 00:06:55.660 takeaway from the, from the talk is that, Oh, this is going to be cool. We're going to do, we're going 00:06:55.660 --> 00:07:02.160 to do things like coroutines for, for async. And we'll probably talk about that later, but I went and rewatched 00:07:02.160 --> 00:07:07.220 rewatched that talk recently, maybe three months ago. Cause I was, I was like, what did, what did Guido 00:07:07.220 --> 00:07:13.960 like say in that talk exactly? And it was, it was not at all what I remembered, you know, you rewatched 00:07:13.960 --> 00:07:21.340 the talk. He's talking more about, you know, trying to have like a common event loop in Python to have 00:07:21.340 --> 00:07:26.680 some kind of interoperability with some of these libraries, like, you know, twisted and tornado and, 00:07:26.900 --> 00:07:31.940 you know, maybe G event or something. And it, you know, the focus on coroutines was more 00:07:31.940 --> 00:07:38.960 incidental. So that's really interesting. Yeah. So if for example, you're using twisted, it has some 00:07:38.960 --> 00:07:45.600 kind of event loop that's doing the callbacks for you as, as your things complete or whatever. And if 00:07:45.600 --> 00:07:49.900 you're using something else that also has an event loop, those things might not know anything about each 00:07:49.900 --> 00:07:56.760 other. Right. Right. Right. Right. Yeah. That, that can definitely be a challenge. So that takes us up to, 00:07:56.760 --> 00:08:02.700 what, three, three, three, four, when, three, three, three, right. Yeah. And then in three, five, 00:08:02.700 --> 00:08:11.360 we got async and await, which was, you know, really take these ideas and make them more, more accessible 00:08:11.360 --> 00:08:16.760 to people, I think. Right. I think, yeah. Trying to put like a better face on it. I think, you know, 00:08:16.760 --> 00:08:22.420 it's like a putting like a different, I don't know, I don't know how to, almost like a different API on top of 00:08:22.420 --> 00:08:29.260 that machinery, I think to present it in a more coherent way. It's certainly not a, that, that 00:08:29.260 --> 00:08:34.160 approach is not a Python invention. I mean, I don't know that they directly cite it, but it had 00:08:34.160 --> 00:08:39.780 been done in C# before. Yeah. And it, you know, what's interesting, like the history in C# 00:08:39.780 --> 00:08:44.880 kind of follows the same way. They didn't come up with that initially either. They came up with just 00:08:44.880 --> 00:08:50.400 this idea of like a task framework and it was all callback driven and whatnot. And then somebody said, 00:08:50.400 --> 00:08:56.740 oh, look, this callback way of writing, this is super not nice, right? It, it works, but it's not 00:08:56.740 --> 00:09:01.020 the same as writing serial code. If we put this async and await on it, it will be. And I, you know, 00:09:01.020 --> 00:09:07.440 it has exactly the same benefit for Python is that it takes code that would have to look special and it 00:09:07.440 --> 00:09:12.000 kind of makes it look serial, right? Yeah. It makes it like, I mean, that really, that is kind of 00:09:12.000 --> 00:09:18.280 the whole focus of it, you know, writing code with callbacks. I mean, you see that like in every 00:09:18.280 --> 00:09:22.840 talk about async, you know, there's usually a slide. It's like, oh, callback hell or something, 00:09:22.840 --> 00:09:28.420 right? And everybody kind of moans. Yeah, exactly. I've seen, seen callback hell. And then, and then 00:09:28.420 --> 00:09:34.500 there's different, different approaches for how to, you know, kind of untangle yourself from that. 00:09:34.500 --> 00:09:40.340 And, you know, it's, you know, it's interesting, you know, where, where it tends to push everyone is 00:09:40.340 --> 00:09:45.980 more into just like serial code. You know, people want something that looks a lot like maybe thread 00:09:45.980 --> 00:09:50.720 programming or just kind of, you know, just kind of straight forward code. But then, you know, 00:09:50.720 --> 00:09:55.340 there's this question, how do you get there? So how does it fooling around with generators and, 00:09:55.340 --> 00:10:01.580 you know, tasklets and green threads and all the, all these things that you see in these libraries 00:10:01.580 --> 00:10:05.760 are all kind of focused on that general, you know, that general problem. 00:10:05.760 --> 00:10:11.460 Yeah. So that's really, that's a really interesting thing to understand kind of from the ground up, 00:10:11.460 --> 00:10:18.660 I think. And so one of the subjects I wanted to cover while we were talking today is some of the 00:10:18.660 --> 00:10:24.500 ideas that you brought up in a talk that you gave at PyCon 2015 called Python concurrency from the ground 00:10:24.500 --> 00:10:30.100 up alive. And before we get into that, I just want to say that was such a masterful talk. 00:10:30.100 --> 00:10:35.620 You did a fine job on that talk. I'm, for those of you who haven't seen it, David basically, 00:10:35.620 --> 00:10:41.820 David basically pulls up the editor and says, we're going to write a web server and we're going to 00:10:41.820 --> 00:10:47.660 explore all our TCP server. And we're going to explore all the ways that you might approach 00:10:47.660 --> 00:10:55.800 concurrency from it and the ways we might invent something similar to the asyncio that's built 00:10:55.800 --> 00:10:59.320 into Python and it was really well done. Okay. Thank you. 00:10:59.320 --> 00:11:04.340 Yeah. Yeah. So people, I'll link to that in the show notes and people should definitely check it out, 00:11:04.340 --> 00:11:10.380 but maybe we could just kind of talk through some of the ideas of like, if we start with a serial 00:11:10.380 --> 00:11:16.180 server and a serial client, like how do we, what are the ways in which we can build up to that? Like 00:11:16.180 --> 00:11:20.000 we can use threading, we could use coroutines. There's lots of things, right? 00:11:20.440 --> 00:11:24.580 Right. Right. So obviously you start with serial code. It's fine long as you don't have too many 00:11:24.580 --> 00:11:28.720 people requesting from the server. Right. But as soon as you have some long request, everything is 00:11:28.720 --> 00:11:34.020 blocked. Right. Right. The whole big picture of that talk, I'm going to try to distill it down from 00:11:34.020 --> 00:11:42.160 like high, high level view here. It's all about scheduling, basically task scheduling. And if you 00:11:42.160 --> 00:11:49.020 have normal serial code, you're executing statements, you're going line by line through the code. If you 00:11:49.020 --> 00:11:54.940 hit an operation, like receive, like I want to receive data on a socket or something, that code is going to 00:11:54.940 --> 00:12:00.240 block. If it's like, if there's nothing available, it's going to block and you're going to have to wait. 00:12:00.240 --> 00:12:07.040 And that really is the gist of the whole problem, which is what happens that, like what happens when 00:12:07.040 --> 00:12:15.200 you block? And if you do nothing, then your whole program just freezes. I mean, just everything stops 00:12:15.200 --> 00:12:22.240 and then nothing can happen. Some ways around that, one approach to do it is to use threads in the 00:12:22.240 --> 00:12:29.560 operating system. Essentially with threads, you're running multiple sort of serial tasks at once. 00:12:30.220 --> 00:12:36.040 And if one of them decides to block, needs to receive, well then, you know, the others are still 00:12:36.040 --> 00:12:41.220 allowed to run. So you're essentially allowing the operating system to deal with it. It would take 00:12:41.220 --> 00:12:48.220 care of scheduling the, scheduling the threads and making sure that things work. The other approach, 00:12:48.220 --> 00:12:53.540 and this is something that the talk gets into at the end, is to do it yourself. Don't have the 00:12:53.540 --> 00:13:00.200 operating system do it. Take care of that blocking, blocking on your own. And one of the tricks that's 00:13:00.200 --> 00:13:07.860 used for that is to use Python generator function. And that is used there is actually just the 00:13:07.860 --> 00:13:15.140 behavior of that yield statement. So if you haven't written a generator function before, I think most 00:13:15.140 --> 00:13:20.680 people kind of know them in the context of iteration and the for loop. You can write this function where 00:13:20.680 --> 00:13:27.200 you use the yield statement to emit values out of a for loop. And the thing that's really cool about 00:13:27.200 --> 00:13:33.740 that yield statement is that it causes the function to just suspend itself right there at the yield. It's 00:13:33.740 --> 00:13:39.400 like it emits a value and then it suspends. And that's exactly the kind of thing you need to do 00:13:39.400 --> 00:13:46.200 this concurrency thing. You can say, well, if there's no data to receive, I can suspend myself. 00:13:46.200 --> 00:13:51.940 And essentially, you can take over the role that like an operating system would normally do at that point. 00:13:51.940 --> 00:13:57.660 Yeah. And I think that's a really, really interesting insight that you can say, I'm going to take this 00:13:57.660 --> 00:14:04.320 thing that really doesn't generate anything and make it a generator anyway. And so you had some 00:14:04.320 --> 00:14:09.520 interesting examples of like, we're going to basically simulate parallelism with generators. 00:14:09.520 --> 00:14:13.700 And one of the reasons you might care about that is you can switch things over to threads, 00:14:13.700 --> 00:14:19.660 but that actually slows things down quite a bit. And especially if there's computational stuff, 00:14:19.660 --> 00:14:25.040 you might have to push that out to like do multi processing. And then it really slows down like 10 00:14:25.040 --> 00:14:29.560 times what it might normally be. And so you had this great example that maybe we could talk about 00:14:29.560 --> 00:14:34.340 just a little bit where you say like, let's just come up with a generators that can count down from 00:14:34.340 --> 00:14:38.940 like, you give it a number like 10, it'll count down 10, nine, eight down to one. And then it's done. 00:14:39.140 --> 00:14:44.300 And you generate a bunch of the well, you generate multiple of these with different numbers and whatnot, 00:14:44.300 --> 00:14:51.100 and put them all into like a task list of things that have to be run. And then you one at a time, 00:14:51.100 --> 00:14:58.600 sort of round robin work with those generators. And I think that really highlights, here's how this event 00:14:58.600 --> 00:15:04.620 loop can work, right? We can actually process these in a semi fair way across all these different 00:15:04.620 --> 00:15:05.500 generators, right? 00:15:05.500 --> 00:15:08.580 Right. But you can like cycle, you know, you can kind of cycle between them. 00:15:08.580 --> 00:15:13.620 Yeah, which is really doesn't make a lot of sense when you just have a countdown little thing, 00:15:13.620 --> 00:15:21.340 right? But then you say, Okay, well, now let's apply the same idea to functions like a while true loop, 00:15:21.340 --> 00:15:29.140 that is going while true, come over here and wait to receive from a socket, then process the response 00:15:29.140 --> 00:15:36.260 while true way, you know, and if you put yield statements throughout, right before all the blocking 00:15:36.260 --> 00:15:40.640 places, you can kind of accomplish the same thing with the same technique, right? 00:15:40.780 --> 00:15:45.240 Yeah, yeah. Okay, I kind of describe it. And I don't know, this might be might be sort of silly, 00:15:45.240 --> 00:15:50.360 but the whole approach in that talk, I sort of view it as analogous to maybe like the game of hockey or 00:15:50.360 --> 00:15:55.160 something like you've got a, you've got a task, and it's out on the ice, you know, and it's doing its 00:15:55.160 --> 00:16:00.100 thing. But if for some reason, it's got to read, it's got to receive and like, it can't proceed. 00:16:00.860 --> 00:16:06.240 It gets thrown into the penalty box, right? One of the rules is you can't block and anytime it fails 00:16:06.240 --> 00:16:11.200 that like, yeah, you take a blocking penalty, man. So it's like, it's like, okay, you blocked, 00:16:11.200 --> 00:16:16.880 you're going to the penalty box, and you're going to sit in the penalty box for as long as it takes 00:16:16.880 --> 00:16:21.380 until like some kind of data comes in. And then once some data has arrived, and it's like, okay, 00:16:21.380 --> 00:16:26.380 you get to go back out on the ice. So it's very much that model, you know, it's like tasks get to 00:16:26.380 --> 00:16:31.680 run as long as there's things to do. But you know, once there's nothing to do, it's now you go, you go 00:16:31.680 --> 00:16:36.380 sit in the penalty box. And yeah, I think that's a really interesting analogy. And it's definitely makes 00:16:36.380 --> 00:16:41.920 a it's a good way to think about it. The challenge, though, and what it was, I think this is pretty 00:16:41.920 --> 00:16:47.140 obvious, right? I'm going through and if I could pull out any task, and I could ask it, hey, do you 00:16:47.140 --> 00:16:52.100 have work to do? Then I'm going to let you do it. Otherwise, you go back to waiting, or you go to the 00:16:52.100 --> 00:16:55.960 penalty box. And when you decided you have work to do, you can come back out. It's interesting. 00:16:56.380 --> 00:17:02.500 But then how do you know when it actually has work to do, right? Like on the socket, how do I know, 00:17:02.500 --> 00:17:05.820 or if I'm doing something computational, how do I know that that task is ready to run? 00:17:05.820 --> 00:17:12.180 Oh, yeah. Well, to get that, you need the help of the operating system. So there's usually there are 00:17:12.180 --> 00:17:17.440 some system calls related to like polling of sockets, like the select call is one there's 00:17:17.440 --> 00:17:23.700 things with like, you know, like the poll function, you know, there's there's like low 00:17:23.700 --> 00:17:29.960 level kind of event API's in the operating system where you can present it with like a whole collection 00:17:29.960 --> 00:17:35.540 of sockets. And you can say, Okay, I have these 1000 sockets. Why don't you watch these? And then 00:17:35.540 --> 00:17:40.700 if anything happens, tell me about it. Yeah. And that works really well for sockets. But what if I 00:17:40.700 --> 00:17:45.240 gave it like a piece of just a function, a Python function that was computationally expensive? 00:17:45.240 --> 00:17:49.640 Yeah, if it's computationally expensive, it's just going to run. It's actually a problem with these, 00:17:49.640 --> 00:17:53.400 like this event loop thing. If you have something that runs like, I don't know, it's going to go 00:17:53.400 --> 00:17:59.280 mine a Bitcoin or something. It's just going to run. And there's no way to, there's no way to get it 00:17:59.280 --> 00:18:04.760 back until it finishes. So yeah, it's true. It's I mean, they're all really the event loop really is 00:18:05.120 --> 00:18:08.760 on the same thread, right? I guess there are some things you can do, like you can say, Well, 00:18:08.760 --> 00:18:13.180 this part is computational. So we're going to kick that off in some multi processing way, 00:18:13.180 --> 00:18:18.520 or something like that, that's possible. You had some kind of socket trick where you're even when 00:18:18.520 --> 00:18:21.960 you weren't using sockets, you were using it to signal with your commentary, like if you're going 00:18:21.960 --> 00:18:28.100 to do like CPU work somewhere else, in some sense, you turn that back into an IO problem. I mean, 00:18:28.100 --> 00:18:32.940 you might have some work that gets that gets carried out somewhere else. And then when it done, 00:18:33.080 --> 00:18:38.040 when it's done, it gets signaled on a socket saying, Hey, you know, that thing was done. 00:18:38.040 --> 00:18:43.240 Yeah. So okay. Yeah. So yeah, very, it was a very interesting technique. And in the end, 00:18:43.240 --> 00:18:48.940 down in the internals, you kind of had to deal with some of the callbacks, but the way it got consumed, 00:18:48.940 --> 00:18:55.020 right, it was a pretty, pretty straightforward. So that was your talk. And like I said, people should 00:18:55.020 --> 00:19:02.140 absolutely go watch it. It's really quite amazing. And then the other thing that I kind of see as 00:19:02.140 --> 00:19:08.300 the frameworkification of this, these ideas, I'm not sure what the origin is, I'll ask you, 00:19:08.300 --> 00:19:13.420 which, you know, which one came first, but is this project you have called Curio? Do you want to tell 00:19:13.420 --> 00:19:19.840 people what Curio is? Yeah. Okay. So Curio is a library for doing concurrency in Python. I mean, 00:19:19.840 --> 00:19:26.740 it exploits a lot of this async and awaits the, I'd say ultimately sits in kind of the same spot as 00:19:26.740 --> 00:19:30.880 asyncio, although it has a very different flavor to it. 00:19:30.880 --> 00:19:34.440 One of the things that I think is really interesting with Curio is like, there's been, 00:19:34.440 --> 00:19:40.200 as we talked at the beginning, there's been these ideas for a long time of doing some sort of 00:19:40.200 --> 00:19:47.100 asynchrony through callbacks and things like that. Like with Twisted, we've got asyncio built in the 00:19:47.100 --> 00:19:55.380 earlier versions of Python. But in Python 3.5, we have async and await and onward, of course. And you 00:19:55.380 --> 00:20:00.600 kind of took all of those ideas and said, let's refresh, let's rethink them. Like, how would this 00:20:00.600 --> 00:20:05.640 look? How would this API look if we actually had this version of Python concurrency, not what we had 00:20:05.640 --> 00:20:09.280 before, right? Like that's, that was how I was reading it when I was going through it. 00:20:09.380 --> 00:20:14.420 Yeah, that's part of it. I mean, okay, so there's a little bit of a complicated, complicated background 00:20:14.420 --> 00:20:22.320 on this. So let me back up. In a past life, going back a ways, I was a professor in computer science. 00:20:22.320 --> 00:20:28.840 And the main course that I taught was operating systems. And in that course, this is a typical 00:20:28.840 --> 00:20:35.580 course where you'd make students cry on some huge project thing. And I was, I was just a bloodbath, 00:20:35.700 --> 00:20:40.840 bloodbath of a course. So we'd make people write an operating system kernel and see. 00:20:40.840 --> 00:20:47.340 And, you know, that curl had to do all of this stuff, had to do IO and had to do like multitasking 00:20:47.340 --> 00:20:52.660 and task switching and all these things. And it turns out that all the stuff in that project 00:20:52.660 --> 00:20:57.400 was exactly the same kind of thing that people have to do in these asyncio libraries. 00:20:57.400 --> 00:21:02.000 And that's what they're doing. And it's like, they're doing IO and they're coordinating tasks and, 00:21:02.000 --> 00:21:06.700 you know, switching stuff. So it's, it's, the problem is essentially the same. 00:21:06.700 --> 00:21:12.040 It's just in a different environment instead of down at the low level of C and like interrupt 00:21:12.040 --> 00:21:17.660 handlers and device drivers, you're up in Python and it's much higher, higher level, but it's, 00:21:17.660 --> 00:21:23.400 it's a similar topic. And having done that, I mean, I've always kind of, kind of had a, 00:21:23.400 --> 00:21:30.020 an interest in systems topics. So gave a, you know, a well-known PyCon presentation on the gill 00:21:30.020 --> 00:21:35.780 that would have been, I don't know, maybe 2010 or something, something like that. And I also had 00:21:35.780 --> 00:21:42.340 done some tutorials at PyCon about coroutines, sort of exploring, you know, this idea of using generators 00:21:42.340 --> 00:21:49.180 and, and coroutines for concurrency. So it's been a, been a topic that personally been kind of 00:21:49.180 --> 00:21:55.060 exploring for, for a long time. But one of the things that has kind of, I don't know, 00:21:55.060 --> 00:22:01.740 bothered me over those years is that all of my presentations on that have been completely out of 00:22:01.740 --> 00:22:08.340 line with what is actually going on in Python. I mean, like, if you look at that concurrency talk, 00:22:08.340 --> 00:22:16.200 that is not at all how like asyncio approaches concurrency, like approaches that problem. If you look 00:22:16.200 --> 00:22:21.540 at a lot of the, you know, a lot of the presentations and things that what the, what they talk about is 00:22:21.540 --> 00:22:26.520 they're like, Oh, we have an event loop. And then we put callbacks on top of the event loop. And then 00:22:26.520 --> 00:22:32.540 usually there's like a transition into, into a discussion of like futures or promises and like 00:22:32.540 --> 00:22:40.100 a whole approach based on like futures and promises and tasks. And it starts getting like a lot of kind of 00:22:40.100 --> 00:22:47.020 moving, moving gears. And frankly, I just have not been able to wrap my head around that stuff. 00:22:47.020 --> 00:22:52.500 I look at that approach and it is completely different than anything that would have been 00:22:52.500 --> 00:22:58.320 taught in an operating systems class. But I've never seen an operating system kernel built on top of 00:22:58.320 --> 00:23:05.600 futures, for instance. I actually went and got like my old operating system books about, you know, 00:23:05.620 --> 00:23:09.820 not too long ago. And I was like, God, did any of those books talk about futures or promises? And 00:23:09.820 --> 00:23:15.620 they're nowhere in any of the, like the operating system texts. Do you, do you see that? So I've 00:23:15.620 --> 00:23:20.780 been struggling with this kind of mismatch in a way where it's, where it's like, Hmm, there's this whole 00:23:20.780 --> 00:23:26.460 approach with like futures. And then there's this thing that I did in the talk, which is a completely 00:23:26.460 --> 00:23:32.220 different, different thing. And I've been kind of fascinated with that kind of mismatch, 00:23:32.220 --> 00:23:38.740 you know, like, why is that? Or like, what is, what is, what is going on there? And in some sense, 00:23:38.740 --> 00:23:47.100 the curio project is maybe a, maybe it's kind of like a green field project, just trying to do asyncio, 00:23:47.100 --> 00:23:53.660 but more in the operating system kind of model where I'm thinking more in terms of like task scheduling and 00:23:53.660 --> 00:23:57.940 the structure of how I would do it in like a kernel project, 00:23:58.220 --> 00:24:04.800 not in the framework of like the, you know, the futures and promises and callbacks and, 00:24:04.800 --> 00:24:10.800 you know, all of that stuff. So that's a big part of that project is like just kind of a re-envisioning 00:24:10.800 --> 00:24:12.640 of how this might work. 00:24:12.640 --> 00:24:13.340 Right. Okay. 00:24:13.340 --> 00:24:14.040 Yeah. 00:24:14.040 --> 00:24:20.860 Do you feel like it provides a cleaner model by not making you think about all the callbacks and 00:24:20.860 --> 00:24:22.200 futures and stuff like that? 00:24:22.300 --> 00:24:28.300 I think it does. I mean, it is really wild at first glance because it pretty much kills off 00:24:28.300 --> 00:24:34.720 everything that you're used to. Like, in fact, there are no callbacks in curio at all. It's like, 00:24:34.720 --> 00:24:40.960 there's, so there's no callbacks, there's no futures. There's like almost none of that machinery that you see in, 00:24:40.960 --> 00:24:44.600 in kind of the asyncio library. You know, 00:24:44.600 --> 00:24:54.100 one way I've described it, it's almost like I took like the async and await feature that I got added in 3.5 and it's used it as a starting point for some completely different approach. 00:24:54.100 --> 00:24:56.320 Yeah, absolutely. And so it's really, 00:24:56.320 --> 00:25:03.420 it's really involving a lot of async and await and coroutines basically. Right. 00:25:03.460 --> 00:25:11.940 So a lot of the starting points, a lot of things you want to do, you're like, provide this class with some kind of async method, right? 00:25:11.940 --> 00:25:15.820 An async coroutine. And then, then it runs from there. Right. Right. Right. 00:25:15.820 --> 00:25:22.880 Yeah. So there, you know, it might, I don't know what people have looked at it or not, but there are a lot of building blocks, 00:25:22.880 --> 00:25:26.600 a lot of really nice parts to this library. 00:25:27.280 --> 00:25:30.980 When I first thought about it and first checked it out, I kind of thought, okay, this will be, 00:25:30.980 --> 00:25:36.320 maybe it's got like a core event loop and it's doing a few things differently, but there's a lot of building blocks here. 00:25:36.320 --> 00:25:38.280 You can build some really interesting things with it. 00:25:38.280 --> 00:25:42.440 Yeah. There's some, some really odd stuff going on. I don't know. 00:25:42.440 --> 00:25:49.980 It's like how we want to get into that. But one thing about actually getting back to the operating system model on that, 00:25:49.980 --> 00:25:53.260 these async libraries, I mean, let's see if this makes sense. A lot, 00:25:53.260 --> 00:26:03.720 a lot of these async libraries are kind of like an all in proposition where like you're either coding in the asynchronous world or you're not. 00:26:03.720 --> 00:26:07.800 And it tends to be kind of a separation between those two worlds. Like it, even, 00:26:07.800 --> 00:26:13.880 even if you're working with callbacks, it's kind of like you have to program in kind of the callback style or you're, 00:26:13.880 --> 00:26:19.740 you're kind of out of luck. And I think Curio kind of embraces that as well. 00:26:19.740 --> 00:26:27.280 I mean, one of the, one thing from operating systems is there's usually like a really strict separation between what is the operating system kernel. 00:26:27.280 --> 00:26:30.920 And then what is like a user space program, 00:26:30.920 --> 00:26:35.200 even at the level of like kind of like a protection kind of thing, like, you know, 00:26:35.200 --> 00:26:41.100 like a user program isn't even allowed to like see the kernel in any meaningful way. 00:26:41.100 --> 00:26:44.540 I mean, it's, it's, it's like very strict separation. 00:26:44.540 --> 00:26:49.320 That is also something that's going on in the, in this Curio project. 00:26:49.320 --> 00:26:51.100 I mean, there's kind of the world of async, 00:26:51.100 --> 00:26:54.660 you have all these async functions and await and all of that. 00:26:54.660 --> 00:27:00.600 And then there's kind of the kernel and like those two worlds are like really separated from each other in this, 00:27:00.600 --> 00:27:01.780 in the Curio project. 00:27:01.780 --> 00:27:04.560 So that's all, that's another kind of unusual thing about it. 00:27:04.560 --> 00:27:05.600 Yeah, that is really interesting. 00:27:05.600 --> 00:27:10.560 And I, I, now I can see the operating system analogies and there is a kernel, 00:27:10.560 --> 00:27:13.900 a thing you actually call the kernel in Curio, right? 00:27:13.900 --> 00:27:14.180 Yeah. 00:27:14.180 --> 00:27:14.540 Yeah. 00:27:28.380 --> 00:27:31.760 This portion of Talk Python To Me has been brought to you by Rollbar. 00:27:31.760 --> 00:27:35.280 One of the frustrating things about being a developer is dealing with errors, 00:27:35.280 --> 00:27:40.280 relying on users to report errors, digging through log files, trying to debug issues, 00:27:40.280 --> 00:27:43.480 or a million alerts just flooding your inbox and ruining your day. 00:27:43.480 --> 00:27:50.680 With Rollbar's full stack error monitoring, you'll get the context insights and control that you need to find and fix bugs faster. 00:27:50.680 --> 00:27:52.400 It's easy to install. 00:27:52.400 --> 00:27:56.660 You can start tracking production errors and deployments in eight minutes or even less. 00:27:57.420 --> 00:28:01.000 Rollbar works with all the major languages and frameworks, including the Python ones, 00:28:01.000 --> 00:28:06.000 such as Django, Flask, Pyramid, as well as Ruby, JavaScript, Node, iOS, and Android. 00:28:06.000 --> 00:28:10.860 You can integrate Rollbar into your existing workflow, send error alerts to Slack or HipChat, 00:28:10.860 --> 00:28:15.220 or even automatically create issues in Jira, Pivotal Tracker, and a whole bunch more. 00:28:15.220 --> 00:28:18.580 Rollbar has put together a special offer for Talk Python To Me listeners. 00:28:18.580 --> 00:28:24.280 Visit rollbar.com slash Talk Python To Me, sign up, and get the bootstrap plan free for 90 days. 00:28:24.280 --> 00:28:27.320 That's 300,000 errors tracked all for free. 00:28:27.720 --> 00:28:31.040 But hey, just between you and me, I really hope you don't encounter that many errors. 00:28:31.040 --> 00:28:37.040 Loved by developers at awesome companies like Heroku, Twilio, Kayak, Instacart, Zendesk, Twitch, and more. 00:28:37.040 --> 00:28:38.660 Give Rollbar a try today. 00:28:38.660 --> 00:28:41.260 Go to rollbar.com slash Talk Python To Me. 00:28:49.180 --> 00:28:54.480 Basically, you use the constructs and you pass these async coroutines to it, and that's that. 00:28:54.480 --> 00:28:55.520 Okay. 00:28:55.520 --> 00:29:01.700 So if you're doing, like you said on the all-in part, if you write some sort of coroutine, 00:29:01.700 --> 00:29:05.560 and it comes down to a point where you're either doing something computational or blocking, 00:29:05.560 --> 00:29:09.860 and you block there, you kind of take everyone out, right? 00:29:09.980 --> 00:29:11.360 What do you mean take everyone out? 00:29:11.360 --> 00:29:14.900 Well, you can clog up the event loop. 00:29:14.900 --> 00:29:15.620 Oh, yeah. 00:29:15.620 --> 00:29:15.900 Yeah. 00:29:15.900 --> 00:29:16.280 Definitely. 00:29:16.280 --> 00:29:16.700 Yeah. 00:29:16.700 --> 00:29:18.840 I mean, you can throw a wrench into the hole. 00:29:18.840 --> 00:29:21.020 Hey, we're just going to keep going and letting it. 00:29:21.020 --> 00:29:26.320 You do your work, and when you're done, come back, and then we'll pick up where you left off. 00:29:26.620 --> 00:29:27.780 Yeah. 00:29:27.780 --> 00:29:29.480 So how do you deal with that? 00:29:29.480 --> 00:29:35.680 Like, if I've got some async coroutine I want to run in Curio, and it's got something computational, 00:29:35.680 --> 00:29:37.560 how do I make that work? 00:29:37.560 --> 00:29:40.620 Sometimes you just have to do something that's going to take a while, right? 00:29:40.620 --> 00:29:41.120 Yeah. 00:29:41.120 --> 00:29:44.860 I mean, if it's concerned, you have to punt it out either to a thread or another process. 00:29:44.860 --> 00:29:45.400 Yeah. 00:29:45.400 --> 00:29:45.820 Okay. 00:29:45.820 --> 00:29:47.500 Like with multiprocessing or something? 00:29:47.500 --> 00:29:47.740 Yeah. 00:29:47.740 --> 00:29:51.220 It's kind of the standard technique for all these async things. 00:29:51.220 --> 00:29:53.620 It's like you've got something computational, and it's going to block. 00:29:53.620 --> 00:29:54.440 Yeah. 00:29:54.440 --> 00:29:55.960 You've got to punt it out somewhere. 00:29:56.360 --> 00:30:02.640 And then do you have a way, like a construct, to await a thread or await some kind of multiprocessing call? 00:30:02.640 --> 00:30:03.060 Yeah. 00:30:03.060 --> 00:30:03.960 There's a function in there. 00:30:03.960 --> 00:30:08.420 You can ask to run something in a thread or you can run something in a process. 00:30:08.420 --> 00:30:08.880 Okay. 00:30:08.880 --> 00:30:16.440 It will take care of it and wait for the result to come back, but it won't block the sort of internal loop, basically. 00:30:16.440 --> 00:30:17.000 Okay. 00:30:17.000 --> 00:30:17.380 Yeah. 00:30:17.380 --> 00:30:17.820 Very nice. 00:30:17.820 --> 00:30:25.820 So if we have some time, we could talk about some of the individual building blocks, but what can you build with Curio? 00:30:26.100 --> 00:30:27.900 Curio, do you think? 00:30:27.900 --> 00:30:36.540 I mean, it looks like it's a little bit below like a web framework, but it's close to a framework for building asynchronous programs on its own. 00:30:36.540 --> 00:30:38.200 Like where do you think this fits in? 00:30:38.500 --> 00:30:38.660 Okay. 00:30:38.660 --> 00:30:38.840 Yeah. 00:30:38.840 --> 00:30:41.320 It is definitely not a web framework. 00:30:41.320 --> 00:30:45.800 In fact, I don't even, I think there's any HTTP support in it right now. 00:30:46.040 --> 00:30:54.700 So I think it is more of a framework for concurrency, you know, actually setting up tasks, communicating between tasks, coordinating things. 00:30:54.700 --> 00:31:06.480 It's the kind of thing that you might start building libraries on top of, you know, maybe libraries to, you know, interact with Redis or interact with databases or even to do HTTP. 00:31:07.280 --> 00:31:10.860 But it's definitely like a lower level, a lower level thing. 00:31:10.860 --> 00:31:15.160 So it's, you know, a lot of like coordinating tasks and things of that name. 00:31:15.160 --> 00:31:15.540 I see. 00:31:15.720 --> 00:31:26.480 So if I wanted to create some framework that was backed with Redis, I could use Curio to make a really nice async and await framework and somehow do the network IO internally. 00:31:27.020 --> 00:31:29.420 And people might not even know that it's Curio, right? 00:31:29.420 --> 00:31:34.740 They might just know my framework and it talks to Redis and internal part of that could be Curio. 00:31:34.740 --> 00:31:35.400 Yeah, maybe. 00:31:35.400 --> 00:31:35.840 Yeah. 00:31:35.840 --> 00:31:36.300 Yeah. 00:31:36.300 --> 00:31:41.640 I mean, I sort of see it personally as something I might implement a lot of sort of microservice code with. 00:31:41.640 --> 00:31:41.920 Yeah. 00:31:41.920 --> 00:31:45.180 Like a lot of little like web services and stuff, things like that. 00:31:45.180 --> 00:31:48.840 Done a little bit of playing around trying to implement a game server with it. 00:31:48.840 --> 00:31:49.200 Sure. 00:31:49.200 --> 00:31:50.480 Like a socket-based game server. 00:31:50.480 --> 00:31:50.760 Yeah. 00:31:50.760 --> 00:31:52.020 Socket-based game server. 00:31:52.020 --> 00:31:57.000 I think there's a lot of uses with like testing that kind of, I think there's a lot of people, 00:31:57.000 --> 00:32:00.880 who do network programming that's not necessarily web programming. 00:32:00.880 --> 00:32:03.700 And so I think it kind of fits into that. 00:32:03.700 --> 00:32:03.960 Yeah. 00:32:03.960 --> 00:32:09.280 And it has really good support for TCP and UDP type stuff, right? 00:32:09.280 --> 00:32:09.740 Right. 00:32:09.740 --> 00:32:10.140 Right. 00:32:10.140 --> 00:32:10.660 Okay. 00:32:10.660 --> 00:32:24.860 If I wanted to take a framework that maybe I'm already using, let's say Django, Flask, Pyramid, something like that, that doesn't have any support for this idea of concurrency or async await, 00:32:24.860 --> 00:32:29.600 could I somehow use Curio in my own code? 00:32:29.600 --> 00:32:38.520 If I'm willing to sort of do some kind of callback mechanism or like notification mechanism in my web app, right, for like asynchronous stuff? 00:32:38.520 --> 00:32:41.060 Or would those things just not make any sense together? 00:32:41.060 --> 00:32:42.840 I think it would be tough. 00:32:42.840 --> 00:32:59.740 If you've got code that was written originally for kind of the synchronous world, getting that into any async framework, I mean, even asyncio or anything like that can be, it can be kind of a tough proposition just because there's just the programming model can be very different. 00:32:59.740 --> 00:33:04.100 And you have to instrument a lot of code with like these async and await calls. 00:33:04.100 --> 00:33:04.860 Right. 00:33:04.860 --> 00:33:06.000 It's unclear. 00:33:06.000 --> 00:33:06.800 Yeah, yeah. 00:33:07.080 --> 00:33:18.580 I'm thinking like if you have like WebSockets or some kind of call and then pull JavaScript thing, like maybe those parts of it could somehow use Curio? 00:33:18.580 --> 00:33:19.280 Maybe. 00:33:19.280 --> 00:33:19.940 Yeah, maybe. 00:33:19.940 --> 00:33:20.360 I don't know. 00:33:20.360 --> 00:33:21.160 Maybe. 00:33:21.160 --> 00:33:33.480 I mean, you actually kind of this opens up kind of a, I don't know, an avenue of discussion, which is, you know, actually where does this thing, you know, where does it fit in kind of the grand scheme of things? 00:33:33.480 --> 00:33:43.060 And I think, you know, one thing with these async frameworks is just stepping back for a moment and thinking like, okay, where, like, what is the use case for these things? 00:33:43.060 --> 00:33:45.420 Or like, what are they really good at? 00:33:46.300 --> 00:33:54.560 And one of the things that they're really good at is handling a gigantic number of connections, like a high degree of concurrency. 00:33:54.560 --> 00:33:54.960 Right. 00:33:54.960 --> 00:34:06.320 Where you might have like, like, let's say I had 100,000 clients connected to some server and I've got to maintain, you know, some pool of, you know, 10,000, 100,000 socket connections. 00:34:06.320 --> 00:34:12.720 That is where these async things tend to shine because you can't just spin up like 10,000 threads. 00:34:12.720 --> 00:34:14.080 Yeah, you can't. 00:34:14.080 --> 00:34:18.260 Just the memory required for the stack space would be problematic, right? 00:34:18.260 --> 00:34:18.860 Right, right. 00:34:18.860 --> 00:34:21.000 So they're really good at that. 00:34:21.000 --> 00:34:24.760 But so you have like a high degree of concurrency. 00:34:24.760 --> 00:34:34.060 But at the same, you know, even though you have like a high, like a lot of clients, it doesn't mean that those clients are all doing things at the exact same time either. 00:34:34.060 --> 00:34:44.300 I mean, you might have like a server that has like, look, it has like 100,000 connections open, but maybe it's doing sort of push notifications or kind of low traffic stuff. 00:34:44.400 --> 00:34:50.920 I mean, it's not like you're going to have like 100,000 connections kind of open, just completely hammering your machine all the time. 00:34:50.920 --> 00:34:51.260 Right. 00:34:51.260 --> 00:34:56.520 Something like Slack or something, right, where everybody's got it open, but the amount of traffic is actually quite low. 00:34:56.520 --> 00:34:58.420 But you want it to be basically instant, right? 00:34:58.420 --> 00:34:58.860 Right. 00:34:59.240 --> 00:35:04.400 So the kind of stuff I'm thinking about is like, okay, so maybe let's say you have like 100,000 connections open. 00:35:04.400 --> 00:35:10.100 Could you still use something like Flask or Django or something? 00:35:10.100 --> 00:35:14.300 Like, could you still use that in some capacity? 00:35:14.300 --> 00:35:35.380 Now, I mean, you can't, you're not going to be able to spin up like 100,000 threads running Django or whatever, but could you have some coordination between, you know, these like tasks and something like Curio or AsyncIO and coordinate that with maybe a smaller number of threads or processes or whatever that are running like a more traditional framework? 00:35:35.380 --> 00:35:36.260 Yeah, exactly. 00:35:36.260 --> 00:35:37.440 That was what I was thinking. 00:35:37.440 --> 00:35:38.640 I'm not sure if it's possible, though. 00:35:38.640 --> 00:35:39.360 I don't know. 00:35:39.360 --> 00:35:53.560 I mean, so one thing in Curio that I think is actually one of the more interesting parts of the project is I'm trying to do a lot of coordination between async and traditional thread programming. 00:35:53.560 --> 00:35:59.580 As an example of that, one thing that Curio has is it has this universal Q object. 00:36:00.140 --> 00:36:09.080 This is probably one of the most insane things in the whole in the whole library, but a standard way to communicate between threads is just to use a queue. 00:36:09.080 --> 00:36:12.240 Right, because shared shared data is problematic, right? 00:36:12.240 --> 00:36:14.300 You've got a lock on it and all sorts of stuff. 00:36:14.300 --> 00:36:14.680 Right. 00:36:14.680 --> 00:36:17.900 So you have a queue and, you know, you like share between threads. 00:36:18.060 --> 00:36:31.680 So there's this thing in Curio that lets, that basically allows queuing to work between async tasks in Curio and thread programs in a really like seamless way. 00:36:31.680 --> 00:36:37.420 Like essentially the thread, the thread part of it just thinks it's working with a normal queue and everything works normal. 00:36:37.420 --> 00:36:42.760 And like the Curio side works with like a, like an async queue and it thinks that everything is kind of normal. 00:36:43.000 --> 00:36:47.460 And you get this like queuing going back and forth between kind of the two worlds. 00:36:47.460 --> 00:36:51.920 And it's kind of, it's, it's sort of seamless in a really disturbing way. 00:36:51.920 --> 00:37:02.880 It's a, it's like maybe you could have like, you know, a hundred thousand tasks kind of managing sockets, but then talking to some pool of threads through queuing and it, and it all kind of works. 00:37:02.880 --> 00:37:10.460 And it's, this is sort of an area that is not, at least as far as I know, not being explored more traditionally in asyncio, for instance. 00:37:10.860 --> 00:37:13.540 They have a queue there, but it's not compatible with thread. 00:37:13.540 --> 00:37:14.000 Right. 00:37:14.000 --> 00:37:15.880 So this is a really interesting idea. 00:37:15.880 --> 00:37:19.600 This universal queue, it's kind of like a dual facade, right? 00:37:19.600 --> 00:37:23.880 The, the different worlds can see it as part of theirs, right? 00:37:23.880 --> 00:37:24.400 Yeah. 00:37:24.400 --> 00:37:24.720 Yeah. 00:37:24.720 --> 00:37:30.180 Somebody contributed a feature to Curio to allow it to submit work to asyncio. 00:37:30.180 --> 00:37:30.840 Okay. 00:37:30.840 --> 00:37:34.740 So like you could have Curio and you could have threads and you can have asyncio. 00:37:35.460 --> 00:37:39.780 This queuing object in Curio actually works in all three of those worlds. 00:37:39.780 --> 00:37:41.680 Done some tests on that. 00:37:41.680 --> 00:37:47.220 So like you could have a queue where one end of the queue is like an asyncio task and the other end is a thread. 00:37:47.220 --> 00:37:57.020 And then, or, you know, you could have like a thread and a Curio task putting things on a queue that's being read by asyncio and other, other things. 00:37:57.020 --> 00:38:00.580 That's a really kind of wild, crazy thing to be playing with. 00:38:00.740 --> 00:38:02.760 I mean, it's, yeah, that's really interesting. 00:38:02.760 --> 00:38:11.440 I, you know, I didn't know about universal queue, but the library is full of these, these really amazing little data structures and, and functions and stuff. 00:38:11.440 --> 00:38:12.780 So it's, it's quite neat. 00:38:12.780 --> 00:38:24.240 I mean, the other thing that's kind of wild and well, they're kind of, kind of interesting in Curio too, is just the way that the task model works and has a lot of support for things like cancellation of tasks. 00:38:24.240 --> 00:38:24.700 Right. 00:38:24.820 --> 00:38:27.180 And that turns out to be a really tricky problem. 00:38:27.180 --> 00:38:30.260 It's like, okay, you set up a whole bunch of work and then you want to cancel it. 00:38:30.260 --> 00:38:31.820 Can you do that? 00:38:31.820 --> 00:38:34.120 And that's something that you can do in Curio. 00:38:34.120 --> 00:38:39.000 And I, it's, it's very interesting because it's something that you can't do with threads traditionally. 00:38:39.000 --> 00:38:39.520 Right. 00:38:39.520 --> 00:38:46.100 If you just kill a thread, maybe it's holding onto some kernel level thing and you forced it to just, you know, leak it or whatever. 00:38:46.100 --> 00:38:46.400 Right. 00:38:46.400 --> 00:38:47.700 It's going to be bad. 00:38:47.700 --> 00:38:50.220 You have a way to, to even kill a thread. 00:38:50.400 --> 00:38:59.860 I mean, there's no API for killing a thread and then I think some people have done it going through C types, but that, that just makes my skin crawl. 00:38:59.860 --> 00:39:07.480 It's like, like killing threads by going through C types seems like a really good way to just not have your program work. 00:39:07.480 --> 00:39:08.000 So. 00:39:08.000 --> 00:39:08.780 Yeah, exactly. 00:39:08.780 --> 00:39:14.020 Like if it's holding onto something important, like let's say, imagine it's holding onto the GIL and you kill it. 00:39:14.020 --> 00:39:16.380 What happens then? 00:39:16.380 --> 00:39:18.580 That might not be great. 00:39:18.580 --> 00:39:19.060 Yeah. 00:39:19.340 --> 00:39:19.700 Okay. 00:39:19.700 --> 00:39:22.680 So this cancellation thing, you're right that that is not simple. 00:39:22.680 --> 00:39:23.620 How does it work? 00:39:23.620 --> 00:39:31.860 Is it like basically every time one of these async coroutines yields or if it's not in a running state, you can just say, okay, this is getting canceled? 00:39:31.860 --> 00:39:32.500 Pretty much. 00:39:32.500 --> 00:39:33.000 That's it. 00:39:33.000 --> 00:39:33.220 Yeah. 00:39:33.220 --> 00:39:46.900 Since every operation requires the, you know, kind of the support of this kernel, if somebody wants to cancel something, I mean, as soon as it, you know, if somebody wants to cancel something, it's either blocked in there already. 00:39:47.520 --> 00:39:48.520 Or you can just wait for it. 00:39:48.520 --> 00:39:49.140 Or you can just wait for it to finish. 00:39:49.140 --> 00:39:51.860 yeah, you can, you can essentially just, just blow it away. 00:39:51.940 --> 00:39:55.760 I mean, it's, you raise an exception at the yield statement saying, okay, you're done. 00:39:55.760 --> 00:39:56.040 Yeah. 00:39:56.040 --> 00:39:56.260 Okay. 00:39:56.260 --> 00:40:08.460 That's what I was, that's always my next question is like, you can't just take it out of the running task list and throw it away because it might, might've been in the middle of something that needs to be unwound, like created a file and it needs to close the handle or something. 00:40:08.460 --> 00:40:08.660 Right. 00:40:08.660 --> 00:40:12.820 So you, you basically just raise like a task cancellation exception or something. 00:40:12.820 --> 00:40:13.160 Uh huh. 00:40:13.160 --> 00:40:13.620 Okay. 00:40:13.620 --> 00:40:13.900 Yeah. 00:40:13.900 --> 00:40:22.260 So you get, it gets a cancel there or, and then it, it can choose to clean up if it wants, but it's sort of a graceful, you know, graceful shutdown from that. 00:40:22.360 --> 00:40:22.440 Yeah. 00:40:22.440 --> 00:40:23.940 That's a nice, that's a really nice feature. 00:40:23.940 --> 00:40:27.460 So let me ask you about integrating with some other things. 00:40:27.460 --> 00:40:31.120 Like I'm, I have databases on my mind right now for, for some reason. 00:40:31.120 --> 00:40:35.120 So there's a bunch of nice ORMs or ODMs. 00:40:35.120 --> 00:40:42.320 If you're doing a SQL and Python, you know, SQLAlchemy, Mongo engine, Peewee pony, and so on. 00:40:42.320 --> 00:40:49.220 The one ORM that I've seen that seems to integrate really nicely with async and await is Peewee ORM. 00:40:49.420 --> 00:40:55.480 You can basically await on the queries that you're getting back from it, which is super cool. 00:40:55.480 --> 00:40:59.400 Would Curio integrate pretty seamlessly with a framework like that? 00:40:59.400 --> 00:41:00.700 I don't know. 00:41:00.700 --> 00:41:03.720 Do you know how they, how they're doing that under the cover? 00:41:03.720 --> 00:41:07.160 I looked at Peewee before we met here because you mentioned it. 00:41:07.160 --> 00:41:10.900 I didn't see that feature off the top of my head, but. 00:41:10.900 --> 00:41:20.880 To basically the extent that I know is I've seen reference to it where it basically supports async and await on, on the queries, right? 00:41:20.880 --> 00:41:24.220 The things that it's going to talk to the database, but I don't know what's happening internally there. 00:41:24.220 --> 00:41:24.620 Okay. 00:41:24.620 --> 00:41:24.840 Yeah. 00:41:24.840 --> 00:41:25.420 I don't know. 00:41:25.420 --> 00:41:27.820 I would have to take a, I would have to take a look at it. 00:41:28.380 --> 00:41:35.260 My initial guess is it probably would not work just because Curio is so out in left field right now. 00:41:35.260 --> 00:41:40.220 It's, you know, like if they've written, if they've written that specifically to work on top of asyncio. 00:41:40.220 --> 00:41:40.820 I see. 00:41:40.820 --> 00:41:41.100 Yeah. 00:41:41.100 --> 00:41:42.460 So it might not, right? 00:41:42.460 --> 00:41:43.560 Hit or miss on that. 00:41:43.560 --> 00:41:43.820 Yeah. 00:41:43.820 --> 00:41:44.140 Yeah. 00:41:44.140 --> 00:41:44.380 Yeah. 00:41:44.380 --> 00:41:51.300 I mean, the API is you just have an async method and you just await either object.create or objects.query and so on. 00:41:51.340 --> 00:41:52.500 But I don't know what's internal. 00:41:52.500 --> 00:41:55.500 It's probably asyncio, I would guess. 00:41:55.500 --> 00:41:56.080 Yeah. 00:41:56.080 --> 00:41:59.860 This portion of Talk Python To Me is brought to you by Hired. 00:41:59.860 --> 00:42:03.260 Hired is the platform for top Python developer jobs. 00:42:03.260 --> 00:42:08.720 Create your profile and instantly get access to thousands of companies who will compete to work with you. 00:42:08.720 --> 00:42:11.960 Take it from one of Hired's users who recently got a job and said, 00:42:11.960 --> 00:42:15.900 I had my first offer within four days and I ended up getting eight offers in total. 00:42:15.900 --> 00:42:19.100 I've worked with recruiters in the past, but they were pretty hit and miss. 00:42:19.540 --> 00:42:21.940 I tried LinkedIn, but I found Hired to be the best. 00:42:21.940 --> 00:42:26.140 I really like knowing the salary up front and privacy was also a huge seller for me. 00:42:26.140 --> 00:42:28.160 Well, that sounds pretty awesome, doesn't it? 00:42:28.160 --> 00:42:30.020 But wait until you hear about the signing bonus. 00:42:30.020 --> 00:42:33.580 Everyone who accepts a job from Hired gets a $300 signing bonus. 00:42:33.580 --> 00:42:36.600 And as Talk Python listeners, it gets even sweeter. 00:42:36.600 --> 00:42:41.960 Use the link talkpython.fm/Hired and Hired will double the signing bonus to $600. 00:42:41.960 --> 00:42:44.040 Opportunity is knocking. 00:42:44.040 --> 00:42:47.740 Visit talkpython.fm/Hired and answer the door. 00:42:48.240 --> 00:42:49.840 I've been thinking about this. 00:42:49.840 --> 00:42:51.060 I mean, the same problem. 00:42:51.060 --> 00:42:55.460 I'm not familiar with the PeeWee ORM, but I am familiar with SQLAlchemy. 00:42:55.460 --> 00:42:56.580 Yeah, that was my next question. 00:42:56.580 --> 00:43:00.940 What about things like SQLAlchemy or MongoEngine that have no concept of this at all? 00:43:00.940 --> 00:43:05.240 Could we somehow shoehorn them into working with Curio or these types of things? 00:43:05.240 --> 00:43:06.100 Yeah, maybe. 00:43:06.100 --> 00:43:07.520 Keep in mind, it's highly experimental. 00:43:07.520 --> 00:43:08.300 It's highly experimental. 00:43:08.300 --> 00:43:11.860 And what I'm about to talk, I mean, may not work. 00:43:11.860 --> 00:43:19.480 But one thing that I've been playing with in Curio is this, there's a concept in there known as an async thread. 00:43:19.480 --> 00:43:23.400 I mean, at first glance, it's like, oh, God, this is insane. 00:43:24.120 --> 00:43:25.420 I thought that was wonderful. 00:43:25.420 --> 00:43:26.200 That's really cool. 00:43:26.200 --> 00:43:26.780 Oh, no. 00:43:26.780 --> 00:43:28.880 Async threads are nuts. 00:43:28.880 --> 00:43:31.060 Let me see if I can explain. 00:43:31.060 --> 00:43:37.840 Okay, so in threads, with thread programming, you have threads, and then you have all these asynchronous primitives. 00:43:37.840 --> 00:43:39.480 Or you have all these synchronization primitives. 00:43:39.480 --> 00:43:43.460 You have locks and cues and semaphores and all this stuff. 00:43:43.520 --> 00:43:47.540 So in thread programming, you have all this stuff that you normally use to write programs. 00:43:47.540 --> 00:43:53.680 It turns out that almost all of that functionality is replicated in these async libraries. 00:43:53.680 --> 00:44:01.300 I mean, like if you look at async.io, it has like events and semaphores and locks and cues and stuff. 00:44:01.300 --> 00:44:04.700 And Curio has events and locks and cues and all that stuff. 00:44:04.700 --> 00:44:11.220 But the limitation of that, of the async libraries, is that those things don't work with threads. 00:44:11.760 --> 00:44:14.640 Like if you read the docs, it has like this huge warning on it. 00:44:14.640 --> 00:44:16.500 It's like this is not thread safe. 00:44:16.500 --> 00:44:19.380 You know, if you use a thread with this, you're going to die. 00:44:19.380 --> 00:44:20.580 You'll be sorry. 00:44:20.580 --> 00:44:21.120 Yeah. 00:44:21.120 --> 00:44:21.560 Yeah. 00:44:21.560 --> 00:44:29.580 So you have all these things that you would normally use with threads in these async libraries, but you can't use them in thread code. 00:44:29.580 --> 00:44:41.160 So I've got this idea where I wonder if you could like flip the whole programming model around where you could create like an actual real live thread. 00:44:41.700 --> 00:44:46.080 But then have the thread like sitting behind the thread. 00:44:46.080 --> 00:44:50.280 You could have like a little task that interacts with the event loop. 00:44:50.280 --> 00:44:53.160 Like it interacts with the async world. 00:44:53.160 --> 00:44:53.540 Right. 00:44:53.540 --> 00:44:58.860 So instead of being having a bunch of processes and the event loop is running on one of them or not a bunch of threads. 00:44:58.860 --> 00:45:02.500 And you're like the event loop controls all the threads in a sense. 00:45:02.500 --> 00:45:02.760 Right. 00:45:02.840 --> 00:45:05.300 So what you have is you have like one event loop. 00:45:05.300 --> 00:45:07.340 But then you have like a real thread. 00:45:07.340 --> 00:45:11.220 Keep in mind this would be like a POSIX thread, a real life, like fully realized thread. 00:45:11.220 --> 00:45:26.760 But like sitting like right next to that thread out of view, like out of sight would be like a little tiny, like asynchronous, like a little task on the event loop that is watching for the thread to make like certain kinds of requests. 00:45:27.180 --> 00:45:37.060 And I was thinking, it's like, what if you took like in the thread, you took all these requests for all these synchronization primitives and you just kind of handed it over to this like little helper on the side. 00:45:37.480 --> 00:45:40.680 And then you let it interact with the with the event loop. 00:45:40.680 --> 00:45:43.340 And the thing that's really kind of wild about that. 00:45:43.340 --> 00:45:45.780 So so Curio supports this concept. 00:45:45.780 --> 00:45:53.720 It turns out you get all of these features with like tasks and stuff in Curio showing up in threads like you can cancel threads. 00:45:53.720 --> 00:45:59.420 You can do all the synchronizations with threads and all these all these other things. 00:45:59.420 --> 00:46:04.940 I've been I've been thinking about that in the context of some of this database, you know, this database stuff. 00:46:05.120 --> 00:46:08.220 Like, let's say I did want to interact with something like SQLAlchemy. 00:46:08.220 --> 00:46:22.000 Maybe I could have like a pool of threads or something that that would take care of the SQLAlchemy side of it, but then kind of coordinate it with with sort of tasks on the event loop through this kind of acing thread mechanism. 00:46:22.000 --> 00:46:40.500 That's very interesting, like some kind of adapter that kind of looks like SQLAlchemy, but really routes over to another thread where maybe it creates the session and it does the all the filtering order by query stuff and then brings it back over when it's when it returns or something like that. 00:46:40.600 --> 00:46:45.020 Yeah, maybe I'm not even sure it would be an adapter, but I'm kind of thinking of like this. 00:46:45.020 --> 00:46:52.440 The model in my mind is that, OK, if you're using async, you know, let's say you did have a server and you've got like 10,000 connections sitting there. 00:46:52.860 --> 00:47:00.460 It's extremely unlikely that I'm going to have 10,000 or that I would want to make 10,000 concurrent requests on the database. 00:47:00.460 --> 00:47:05.980 I mean, most of these connections are probably sitting idle most of the time or doing other things. 00:47:05.980 --> 00:47:13.940 So I'm thinking like, well, maybe I could have like, you know, maybe 100 threads or, you know, maybe that's too many. 00:47:13.940 --> 00:47:19.060 But you could have like a pool of threads that are sort of responsible for doing the database side of it. 00:47:19.060 --> 00:47:25.140 And then you could coordinate that with this, you know, 10,000 asynchronous tasks in some way. 00:47:25.140 --> 00:47:32.460 So it's going to be kind of a hybrid model where like some of the work takes place in threads and other work takes place in these async tasks. 00:47:32.580 --> 00:47:35.900 But it's kind of done in a more kind of seamless way. 00:47:35.900 --> 00:47:37.260 Well, it sounds really cool. 00:47:37.260 --> 00:47:41.000 I'd love to see it and try it out, but I don't know if it'll work either. 00:47:41.000 --> 00:47:42.080 Yeah, I don't know. 00:47:42.080 --> 00:47:45.320 Yeah, they're finding the time sometimes to explore that. 00:47:45.320 --> 00:47:47.100 It's a challenge for sure. 00:47:47.100 --> 00:47:48.120 Yeah. 00:47:48.120 --> 00:48:02.180 Yeah, but if you could take these really popular, really nice frameworks like SQLAlchemy and somehow click them into this world without actually rewriting them from the ground up, that would be really cool. 00:48:02.400 --> 00:48:02.880 Yeah, I agree. 00:48:02.880 --> 00:48:04.180 I think that'd be I think it'd be fun. 00:48:04.180 --> 00:48:04.440 Yeah. 00:48:04.440 --> 00:48:06.140 It's something I want to try. 00:48:06.140 --> 00:48:11.880 I mean, I've got this sort of web service project that I did a couple years ago. 00:48:11.880 --> 00:48:21.080 And right now it's sitting on Python 2.7 with SQLAlchemy and a bunch of stuff. 00:48:21.080 --> 00:48:24.840 And I look at that and it's like, man, I really want to rewrite this thing. 00:48:25.660 --> 00:48:30.980 You know, in Python 3.6 with Curio and try this thing. 00:48:30.980 --> 00:48:33.680 It's just, you know, there's only finite resources in a day. 00:48:33.680 --> 00:48:37.240 So it's still on the to-do list to sort of get to that. 00:48:37.240 --> 00:48:37.740 Sure. 00:48:37.840 --> 00:48:43.940 But some sort of framework that bridge that divide, I think, would be generally applicable in a lot of places. 00:48:43.940 --> 00:48:45.360 And a lot of people would be excited about it. 00:48:45.360 --> 00:48:46.480 Very cool. 00:48:46.480 --> 00:48:49.260 So you talked about the AsyncIO module a lot. 00:48:49.260 --> 00:48:50.800 Curio is not built on it, right? 00:48:50.800 --> 00:48:52.380 It's kind of something different? 00:48:52.620 --> 00:48:55.800 It is a completely different universe. 00:48:55.800 --> 00:48:56.340 Okay. 00:48:56.340 --> 00:48:58.260 It doesn't use AsyncIO. 00:48:58.260 --> 00:48:59.540 It's doing its own thing. 00:48:59.540 --> 00:49:02.880 That was mostly because you said you want this to be like a greenfield. 00:49:02.880 --> 00:49:05.860 Like, how would the ideal world of the space look like? 00:49:05.860 --> 00:49:08.860 Not like, let's pick up the old model and see what we can do with it? 00:49:08.860 --> 00:49:10.100 Yeah, partly that. 00:49:10.100 --> 00:49:10.440 Yeah. 00:49:10.440 --> 00:49:16.620 And partly, you know, I'm trying to, you know, actually, Curio is kind of a project where I'm trying to learn about this stuff myself. 00:49:16.620 --> 00:49:21.100 You know, just trying to learn, like, you know, what is Async and Await all about in Python? 00:49:21.820 --> 00:49:23.120 What can you do with it? 00:49:23.120 --> 00:49:24.520 Or how can you abuse it? 00:49:24.520 --> 00:49:29.400 You know, sort of, you know, what kind of horrible, insane things are possible with it? 00:49:29.400 --> 00:49:33.360 In some sense, you know, Curio is a project exploring a lot of that. 00:49:33.360 --> 00:49:35.900 You know, exploring a lot of ideas about APIs. 00:49:35.900 --> 00:49:39.420 And really kind of even the programming environment itself. 00:49:39.420 --> 00:49:42.040 So it's not built on AsyncIO. 00:49:42.040 --> 00:49:45.400 And it's, I don't even think it's really meant to clone AsyncIO. 00:49:45.400 --> 00:49:47.300 I mean, it's kind of its own thing right now. 00:49:47.300 --> 00:49:47.880 Right, sure. 00:49:47.880 --> 00:49:48.420 Okay. 00:49:48.420 --> 00:49:51.060 So where are you going with this in the future? 00:49:51.060 --> 00:49:52.400 What are your plans? 00:49:52.400 --> 00:49:57.020 Well, part of the plan is just figuring out how to write about this in books. 00:49:57.020 --> 00:49:59.740 I mentioned I'm supposed to be updating my Python book. 00:49:59.740 --> 00:50:07.580 So a big part of it is I'm thinking about just how to approach Async and Await in the context of writing and teaching. 00:50:07.580 --> 00:50:10.080 And so there's that element of it. 00:50:10.080 --> 00:50:12.200 Actually, do you want a little rant on that, by the way? 00:50:12.200 --> 00:50:12.840 Yeah, go for it. 00:50:12.840 --> 00:50:20.120 I have convinced myself that the approach of teaching Async needs to be flipped in some way. 00:50:20.120 --> 00:50:22.020 And let me describe what I mean by that. 00:50:22.020 --> 00:50:29.760 If you see a typical tutorial on a lot of this Async programming, it ends up being this very kind of bottom-up approach. 00:50:30.280 --> 00:50:36.300 Where it's like, okay, you have sockets, and then you have the event loop, and then you start building all this stuff on the event loop. 00:50:36.300 --> 00:50:39.780 You have callbacks, and then it's like, oh, we have callbacks, and we have futures. 00:50:39.780 --> 00:50:42.840 And then you start layering and layering and layering and layering. 00:50:43.020 --> 00:50:47.040 And then at some point, you reach this like, oh, and we have Async and Await. 00:50:47.040 --> 00:50:47.700 Yeah. 00:50:47.700 --> 00:50:50.600 And it's like Async and Await is, yeah, finally, oh, it's awesome. 00:50:50.600 --> 00:50:52.180 You have Async and Await. 00:50:52.180 --> 00:50:57.980 The problem with this approach is that I just have never been able to teach it. 00:50:57.980 --> 00:51:03.820 I mean, I've tried this in classes, kind of doing the bottom-up approach to this Async stuff. 00:51:03.820 --> 00:51:11.900 And every single time, it seems like you get about halfway through it, and then you're just looking at a room with deer in the headlights. 00:51:12.060 --> 00:51:17.920 Yeah, you've gone through this strainer, and you've stripped everybody's interest out by the time you get to the interesting part. 00:51:17.920 --> 00:51:19.960 Oh, it's horrible. 00:51:19.960 --> 00:51:25.980 So you're just looking at all this deer in the headlight look, and it's like, oh, oh, God. 00:51:25.980 --> 00:51:28.600 You think about it, it's like, okay, wait a minute. 00:51:28.600 --> 00:51:33.240 Okay, so let's say I had to describe file I.O. to somebody. 00:51:33.240 --> 00:51:36.500 Like, you open a file on your computer. 00:51:36.500 --> 00:51:39.460 Like, you open a file in Python, and you read from it. 00:51:40.380 --> 00:51:45.580 Is my description of that going to start with, like, well, okay, you have CPU registers. 00:51:46.520 --> 00:51:52.080 And what you do is you load the CPU registers with, like, a system call number, and then, like, a buffer address. 00:51:52.080 --> 00:51:53.940 And then you execute a trap. 00:51:53.940 --> 00:51:56.180 And then the trap goes into the operating system. 00:51:56.180 --> 00:51:57.860 And it's going to do something. 00:51:57.860 --> 00:51:58.260 I don't know. 00:51:58.260 --> 00:51:59.760 It'll find, like, a file I know. 00:51:59.760 --> 00:52:01.800 And it'll probably check the buffer cache. 00:52:01.800 --> 00:52:05.960 And then, you know, it'll go do some stuff with the disk scheduler and bring some stuff in. 00:52:05.960 --> 00:52:07.020 And then there's, like, copying. 00:52:07.320 --> 00:52:11.520 And then is that the description of, like, how I'm going to describe file I.O. to somebody? 00:52:11.520 --> 00:52:11.660 All right. 00:52:11.660 --> 00:52:13.500 People are like, I just want to read JSON. 00:52:13.500 --> 00:52:14.740 Exactly. 00:52:14.740 --> 00:52:18.900 I'm like, I'm thinking of, you know, so this is my thinking on async, too. 00:52:18.900 --> 00:52:24.540 It's like, does anybody actually care how this stuff works? 00:52:25.020 --> 00:52:34.620 Like, seriously, like, do you care that there's an event loop or a future or a task or whatever it is in there? 00:52:34.620 --> 00:52:36.800 And I'm not sure that you do. 00:52:36.800 --> 00:52:44.700 I'm almost wondering whether, like, the approach to teaching this async stuff is to do, like, this total, like, top-down thing. 00:52:44.700 --> 00:52:48.400 You just, like, you basically say, hey, you have async functions and you have a wait. 00:52:48.400 --> 00:52:49.940 And you just start using it. 00:52:49.940 --> 00:52:50.120 Yeah. 00:52:50.440 --> 00:52:56.780 You don't even say, like, don't even mention, like, generators or coroutines or the yield statement. 00:52:56.780 --> 00:52:58.520 Yes, it's built on that. 00:52:58.520 --> 00:53:00.980 But do you care? 00:53:00.980 --> 00:53:02.460 Yeah, I think you're totally right. 00:53:02.460 --> 00:53:06.200 I mean, you probably care in, like, six months once you've been using it a while. 00:53:06.200 --> 00:53:07.780 You might want to look inside, right? 00:53:07.780 --> 00:53:11.000 But when you don't even know what async and await is, you're right. 00:53:11.000 --> 00:53:12.060 You absolutely don't care. 00:53:12.060 --> 00:53:18.060 I mean, it seems like let's write a program, show that it's blocked, show that we unblock it with async and await. 00:53:18.060 --> 00:53:18.900 Awesome, right? 00:53:19.000 --> 00:53:20.880 That could be the way to get started. 00:53:20.880 --> 00:53:21.280 Yeah. 00:53:21.280 --> 00:53:25.120 So I've been thinking about that a lot, you know, in the context of, like, the book writing. 00:53:25.120 --> 00:53:30.360 It's like, hmm, how am I going to – how do I bring async and await into this book project? 00:53:30.360 --> 00:53:40.020 It's like, am I going to do it – you know, am I going to go the top-down approach where it's going to require kind of a leap of faith where it's like, yeah, okay, you just do it. 00:53:40.020 --> 00:53:41.440 It's like a file. 00:53:41.440 --> 00:53:42.340 It's like you don't care. 00:53:42.820 --> 00:53:47.420 Like, in my whole programming life, have I ever cared how, like, a system call works? 00:53:47.420 --> 00:53:48.400 No. 00:53:48.400 --> 00:53:49.220 The answer is no. 00:53:49.680 --> 00:53:56.240 Like, I mean, other than teaching the operating system class, I have never once cared how a system call works. 00:53:56.840 --> 00:53:59.720 And I kind of feel maybe the same way about async and await. 00:53:59.720 --> 00:54:03.440 It's like if you approach it in the right way, you know. 00:54:03.440 --> 00:54:05.040 Yeah, it doesn't have to be so daunting, right? 00:54:05.040 --> 00:54:06.580 Yeah, it doesn't have to be so daunting. 00:54:06.580 --> 00:54:15.440 And actually, in that context, I'm almost wondering whether something like the asyncio module is like an – it's like assembly code phrasing. 00:54:15.720 --> 00:54:16.320 Yeah, a little bit. 00:54:16.320 --> 00:54:17.420 Yeah, yeah, yeah, a little bit. 00:54:17.420 --> 00:54:17.700 Yeah. 00:54:17.700 --> 00:54:18.980 Very overwhelming, right? 00:54:18.980 --> 00:54:24.980 I get in there and it's like, oh, coroutines wrapped by futures and tasks and blah, blah, blah, blah. 00:54:24.980 --> 00:54:27.420 And you're like, ah, like head is exploding. 00:54:27.420 --> 00:54:28.420 It's like, eh. 00:54:28.420 --> 00:54:29.780 It doesn't have to explode, right? 00:54:29.780 --> 00:54:32.420 Yeah, maybe you don't even need to know that stuff. 00:54:32.420 --> 00:54:33.100 Yeah, that's awesome. 00:54:33.100 --> 00:54:36.540 So what book is this that these ideas are going to land in? 00:54:36.540 --> 00:54:39.260 Ultimately, it will be the Python Essential reference book. 00:54:39.260 --> 00:54:39.700 Nice. 00:54:39.700 --> 00:54:40.620 Awesome. 00:54:40.620 --> 00:54:46.260 I'm trying to figure out how to put async and await into the first chapter, which is like tutorial introduction. 00:54:46.260 --> 00:54:46.780 Yeah. 00:54:46.780 --> 00:54:54.360 I got this idea where it's like, yeah, I'm just going to drop it in the tutorial like right away and just see what I can get away with. 00:54:54.360 --> 00:54:55.220 Yeah, that'd be cool. 00:54:55.220 --> 00:54:57.540 Just set the tone like, no, this is kind of normal now. 00:54:57.540 --> 00:54:58.200 We're doing this now. 00:54:58.200 --> 00:54:59.680 Yeah, it's just this normal, yeah. 00:54:59.680 --> 00:55:02.420 Like what else would you use? 00:55:02.420 --> 00:55:02.960 Yeah, sure. 00:55:02.960 --> 00:55:06.480 No, I don't know whether I can get away with that or not. 00:55:06.480 --> 00:55:15.840 Well, if you want a vote of one for your flipping this presentation or the presentation style for how you present it to people, I think that's the right way to do it. 00:55:15.840 --> 00:55:17.320 So it's unanimous. 00:55:17.320 --> 00:55:18.520 Yeah, okay. 00:55:18.520 --> 00:55:22.560 In some sense, a curio project is kind of experimenting with that too. 00:55:22.560 --> 00:55:30.420 You know, it's, I don't know, focusing more on the async and await side of the equation as opposed to the low-level mechanics. 00:55:30.420 --> 00:55:32.600 Yeah, it's a really cool project. 00:55:32.600 --> 00:55:34.700 And I like where you're going with it. 00:55:34.700 --> 00:55:38.920 It's definitely worth checking out to understand this whole async world better. 00:55:38.920 --> 00:55:41.720 So, David, I think we're getting pretty much out of time. 00:55:41.720 --> 00:55:43.600 Don't want to use up your whole morning. 00:55:43.600 --> 00:55:46.900 So let me ask you a couple of questions real quick, as I always do. 00:55:46.900 --> 00:55:47.500 Yeah. 00:55:47.500 --> 00:55:50.840 I think I can guess this from your presentations. 00:55:50.840 --> 00:55:54.140 But if you're going to write some Python code, what editor do you open up? 00:55:54.140 --> 00:55:55.380 It's got to be Emacs. 00:55:55.380 --> 00:55:56.680 Emacs, right on. 00:55:57.240 --> 00:56:01.880 And favorite PyPI package in addition to curio, of course. 00:56:01.880 --> 00:56:05.080 Is curio on PyPI or is it just installable from GitHub? 00:56:05.080 --> 00:56:05.800 It's on there. 00:56:05.800 --> 00:56:09.940 It should probably be a GitHub version if you're going to do anything interesting with it, though. 00:56:09.940 --> 00:56:11.280 It's moving pretty fast. 00:56:11.280 --> 00:56:12.360 It's moving along. 00:56:12.360 --> 00:56:12.660 Yeah. 00:56:12.800 --> 00:56:14.140 I don't always update it. 00:56:14.140 --> 00:56:14.960 Think of it more like this. 00:56:14.960 --> 00:56:19.160 Like, if people maybe don't know about a package that you recently found, you're like, this is really cool. 00:56:19.160 --> 00:56:21.980 You guys should try this out more than just the popularity contest. 00:56:22.220 --> 00:56:31.220 One of the goals of curio is not so much curio itself, but to basically change a lot of the thinking around async and await. 00:56:31.220 --> 00:56:40.420 You know, it really is kind of an exploratory project where it's like, let's see what we can do with async and await that's maybe outside the context of asyncio. 00:56:40.420 --> 00:56:51.180 And this trio project is something that has been kind of inspired by curio, if you will, is taking things in a slightly different direction. 00:56:52.040 --> 00:56:53.460 So I would recommend people look at that. 00:56:53.460 --> 00:56:53.960 All right. 00:56:53.960 --> 00:57:01.980 I mean, if you're interested in, like, concurrency and some of this async stuff, it will give you yet a third spin on the whole universe. 00:57:01.980 --> 00:57:06.100 So that's also an experimental project, but maybe I would advise that. 00:57:06.100 --> 00:57:06.560 All right. 00:57:06.560 --> 00:57:06.740 Yeah. 00:57:06.740 --> 00:57:07.320 Very, very cool. 00:57:07.320 --> 00:57:13.360 And speaking of packages, I think the PeeWee async thing that I was talking about might be a separate package. 00:57:13.360 --> 00:57:14.380 I'm not sure if it's built in. 00:57:14.380 --> 00:57:16.260 So just be aware of that. 00:57:16.260 --> 00:57:17.300 All right. 00:57:17.300 --> 00:57:17.640 All right. 00:57:17.640 --> 00:57:18.640 Final call to action. 00:57:18.640 --> 00:57:21.320 Are you looking for people to contribute to this project? 00:57:21.860 --> 00:57:24.780 What can people do now that they're, you know, know about curio? 00:57:24.780 --> 00:57:27.120 Oh, it's definitely something that I'm looking for contributors. 00:57:27.120 --> 00:57:35.040 I think the place where a lot of contributions could be made on a package are more in supporting some of these other networking protocols. 00:57:35.200 --> 00:57:43.480 So getting it to hook up with things like Postgres, MySQL, Redis, ZeroMQ, things like that. 00:57:43.480 --> 00:57:47.460 There's a whole, you know, there's kind of a whole space of things that could be done there. 00:57:47.460 --> 00:57:51.540 Really big project would be interesting would be support for HTTP. 00:57:52.040 --> 00:57:52.180 Yeah. 00:57:52.180 --> 00:57:56.260 Like some sort of a WSGI integration. 00:57:56.260 --> 00:57:56.880 Yeah. 00:57:56.880 --> 00:58:13.960 And that might be a whole separate podcast because there is like a whole kind of interest in HTTP and HTTP2 right now where, you know, people are implementing the protocols independently of the actual IO layer. 00:58:14.380 --> 00:58:16.920 This would be like Corey Benfield's work. 00:58:16.920 --> 00:58:26.000 And I think Nathaniel Smith is also working on this with like HTTP2 where, you know, he's implemented the protocol as its own library. 00:58:26.260 --> 00:58:33.560 But then the protocol can be used from threads or used from async or used from twisted or used from different places. 00:58:33.560 --> 00:58:37.160 And that's actually a really, that's a really interesting avenue of work. 00:58:37.160 --> 00:58:37.480 Yeah. 00:58:37.480 --> 00:58:38.180 That's, yeah. 00:58:38.180 --> 00:58:39.020 Thanks for recommending that. 00:58:39.020 --> 00:58:39.400 That's cool. 00:58:39.400 --> 00:58:44.440 So maybe people could use that to build on top of or something for the HTTP layer. 00:58:44.440 --> 00:58:45.260 Right, right. 00:58:45.260 --> 00:58:45.840 For their framework. 00:58:45.840 --> 00:58:46.060 Yeah. 00:58:46.060 --> 00:58:48.420 There's been some work with that in Curio already. 00:58:48.420 --> 00:58:54.060 People, they've, it's actually been shown that you, I mean, you can use those libraries from, from Curio, 00:58:54.060 --> 00:59:00.540 but it has not been sort of packaged at a level of into, into like what I would call like a nice framework. 00:59:00.540 --> 00:59:00.980 Sure. 00:59:00.980 --> 00:59:04.180 Kind of at a, we're kind of operating at a lower level right now. 00:59:04.180 --> 00:59:07.640 And it's, you know, turning it into more of like a framework. 00:59:07.640 --> 00:59:10.180 It's a whole different question really. 00:59:10.180 --> 00:59:10.500 So. 00:59:10.500 --> 00:59:10.740 Yeah. 00:59:10.740 --> 00:59:12.340 Well, it's definitely a really great start. 00:59:12.340 --> 00:59:17.400 And if it turns into one of those frameworks, I would, I would love to play even more with it. 00:59:17.400 --> 00:59:19.160 So very nice work on Curio. 00:59:19.160 --> 00:59:23.700 David, thank you for coming on the show to share all this async stuff with us. 00:59:23.700 --> 00:59:24.120 All right. 00:59:24.120 --> 00:59:24.900 Thank you very much. 00:59:24.900 --> 00:59:28.580 This has been another episode of Talk Python To Me. 00:59:28.580 --> 00:59:31.060 Today's guest has been David Beasley. 00:59:31.060 --> 00:59:33.860 And this episode has been sponsored by Rollbar and Hired. 00:59:33.860 --> 00:59:36.840 Rollbar takes the pain out of errors. 00:59:36.840 --> 00:59:44.540 They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until your users complain, of course. 00:59:45.240 --> 00:59:51.700 As Talk Python To Me listeners, track a ridiculous number of errors for free at rollbar.com slash Talk Python To Me. 00:59:51.700 --> 00:59:54.600 Hired wants to help you find your next big thing. 00:59:54.600 --> 01:00:03.160 Visit talkpython.fm/Hired to get five or more offers with salary and equity presented right up front and a special listener signing bonus of $600. 01:00:04.120 --> 01:00:06.340 Are you or your colleagues trying to learn Python? 01:00:06.340 --> 01:00:09.380 Well, be sure to visit training.talkpython.fm. 01:00:09.380 --> 01:00:15.200 We now have year-long course bundles and a couple of new classes released just this week. 01:00:15.200 --> 01:00:16.180 Have a look around. 01:00:16.180 --> 01:00:17.640 I'm sure you'll find a class you'll enjoy. 01:00:18.380 --> 01:00:20.060 Be sure to subscribe to the show. 01:00:20.060 --> 01:00:22.260 Open your favorite podcatcher and search for Python. 01:00:22.260 --> 01:00:23.500 We should be right at the top. 01:00:23.500 --> 01:00:32.800 You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm. 01:00:32.800 --> 01:00:37.900 Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix. 01:00:37.900 --> 01:00:44.600 Corey just recently started selling his tracks on iTunes, so I recommend you check it out at talkpython.fm/music. 01:00:45.100 --> 01:00:49.960 You can browse his tracks he has for sale on iTunes and listen to the full-length version of the theme song. 01:00:49.960 --> 01:00:52.020 This is your host, Michael Kennedy. 01:00:52.020 --> 01:00:53.300 Thanks so much for listening. 01:00:53.300 --> 01:00:54.480 I really appreciate it. 01:00:54.480 --> 01:00:56.640 Smix, let's get out of here. 01:00:56.640 --> 01:01:00.900 Stating with my voice, there's no norm that I can feel within. 01:01:00.900 --> 01:01:03.740 Haven't been sleeping, I've been using lots of rest. 01:01:03.740 --> 01:01:06.600 I'll pass the mic back to who rocked his best. 01:01:07.280 --> 01:01:09.940 First, develop, first, develop, first, develop, first. 01:01:09.940 --> 01:01:16.020 You're welcome. 01:01:16.020 --> 01:01:18.680 Developers, developers, developers, developers. 01:01:18.680 --> 01:01:20.680 .