#98: Adding concurrency to Django with Django Channels Transcript

00:00 Michael Kennedy: One of the major areas of innovation in Python, especially Python three, is advances in async and concurrent programming. Yet, when working with any of the major web frameworks, Django, Flask, or Pyramid, there is basically no concurrency option. That's why Andrew Godwin decided to tackle the issue on the Django side with his project, Django Channels. This is Talk Python to Me, episode 98, recorded January 17th, 2017. Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via @talkpython. This episode is brought to you by Metis and Hired. Thank both of them for supporting the show by checking out what they have to offer during their segments. Andrew, welcome to Talk Python.

01:20 Andrew Godwin: Ah, thank you. It's very nice to be here.

01:22 Michael Kennedy: Yeah. I'm excited to have another Django topic on the show. I don't cover Django that often and people really love it and I'm excited to sort of mix together the worlds of asynchronous programming and Django in this cool project that you're working on called Django Channels.

01:36 Andrew Godwin: Yeah, I'm excited to talk about it too. Like, it's combining one of the hot new things in async with one of the boring old things in Django, one might say. But yeah, it's gonna be a good thing to talk about. Hopefully people can learn some stuff from this.

01:48 Michael Kennedy: Yeah, I'm sure they will, I'm sure they will. And before we dig into it, though, let's hear your story. How did you get into programming?

01:54 Andrew Godwin: So I have this sort of this almost cliche stories. Like I got into programming about the age of 14 or 13 or so. Just like sort of the young sort of programmer model. I started off, weirdly, on a, I think it was a Palm Pilot IIIx, just writing, like, sort of simple BASIC program, running small games. And that quickly went into like doing JavaScript in websites and then doing PHP and finally about, about 10 years ago now, going over into doing Python. And I've been doing Python ever since, pretty much.

02:25 Michael Kennedy: That's really cool. What language did you program the Palm Pilot thing in?

02:29 Andrew Godwin: It was like a weird variant of BASIC. Like I need to go back and, like, find the actual program, like for posterity. But it was like a very simple BASIC. Had like BASIC graphical stuff and audio stuff. It's like, you know, like a simple Breakout clone, a sort of touch screen keyboard, a few things like games and interactivity has always been a big part of my attraction to programming so like that's what I started out doing, definitely. And that's kind of almost still where I do, that's what all my hobby stuff outside of web stuff.

02:56 Michael Kennedy: Oh yeah, that's cool. Games are definitely fun although they're both easier and harder these days, right? The programming languages are better, the tooling's better but if you're gonna make a serious game, like that's a team of people for a year, right? Like it's not just you can throw out something really simple and it'll sort of compete.

03:14 Andrew Godwin: Or like starting a game is super easy. Finishing and polishing one is the hardest thing in the world. Like I've released one of them. It took me like almost two years for a simple puzzle game. So I'm very aware of how hard it is. I have a lot of respect for those teams that make actual proper games.

03:30 Michael Kennedy: I do as well, absolutely. I worked on some 3D simulators and stuff for awhile and after that I'm like okay, this is really interesting and it's probably not something that I wanna do anymore. So--

03:41 Andrew Godwin: I hear you.

03:43 Michael Kennedy: So you said you got into Python about 10 years ago. Like what was the, how'd you get introduced to it?

03:48 Andrew Godwin: It's all a bit hazy now. So I know in particular that, actually no, it was more than 10 years ago. It was about 12 years ago now. So the first thing I ran into, I was running PHP apps at the time, for, you know, for about three or four years before this and then I ran into TurboGears and TurboGears, back then, was just sort of a couple other packages glued together with extra stuff, and that was my introduction to web programming in Python, in particular like I was looking for an avenue away from PHP and, you know, the number of security vulnerabilities even in my first open source product back then was astonishing. Including the ability to read the ECC password file. So that was fun.

04:24 Michael Kennedy: Oh boy.

04:25 Andrew Godwin: So I was like looking at, I was looking for sort of alternate language and I think Python cropped up as like a common suggestion and TurboGears was one of the big things at that point. Django was around but very young and new as well. And so I sort of started out doing TurboGears for a couple of years and then I had the opportunity to go and work at an agency with Simon Willison who's one of the co-creators of Django. And it is those four or five short weeks with Simon, who is, if you ever meet him, one of the most enthusiastic people in the world that converted me from TurboGear to Django by almost just force of personality. And I've been doing Django pretty much ever since.

05:03 Michael Kennedy: Oh that's really great. It's lovely to be around people who are so excited about technology. It just, it draws you in, doesn't it?

05:10 Andrew Godwin: Oh yeah, I learned so much. Like, I've actually pair programmed with him for like, you know, what, three weeks continuously at most. And I just learned so much both about like, not only like how Django works and how Python works in general but also like a lot of the philosophies of like, you know, things like the Zen of Python that you don't necessarily learn going into the language, per se, but you sort of pick up along the way. So that sort of stuff is really valuable.

05:30 Michael Kennedy: Oh yeah, absolutely. That's really, really great. Let's talk about this project that you've created, Django Channels, can you just kind of tell us, briefly, you know, what is Django Channels?

05:41 Andrew Godwin: It's a very hard thing to describe but I'll have a go. So Django Channels is actually, it's kind of two things. Like it's important to, as we talk about it more in the podcast. So first of all, so the headline feature as you were is Django Channels is a way of having WebSockets supported in Django. And we can go in later on as to why that's difficult and hard but needless to say that until Channels existed, WebSockets was very hard to do in Django just due to some of the technical limitations therein. But the second part of Channels is sort of this underlying layer of like to solve the problem in WebSockets, there's a lot of interesting technical challenges that go with that. And so part of it is like well how do we solve the problem of writing semi-asynchronous code, not fully asynchronous code but semi-asynchronous code, with Django and having things like the Django RM and the middleware and stuff and things that people are familiar with still work. Like how do we keep that stuff familiar yet also have extra power in there? Like I said, it's almost this two-faced story of the underlying solution that powers everything and on top of it, the nice layer of and here's WebSocket supports to go on top of it.

06:48 Michael Kennedy: Okay, and just making that really easy and giving people a great API for it, huh?

06:51 Andrew Godwin: Yeah, so like, you know, the Channels part, sort of the top layer is very much the let's have a nice design thing with a good developer experience that says hey you can write WebSockets and more importantly, it's very hard to shoot yourself in the foot or deadlock or livelock or any of the things that asynchronous programming gives you as like sort of loaded ammunition very easily.

07:12 Michael Kennedy: Absolutely. So, let's, before we dig into your project too much, let's talk about asynchronous programming in Django without your project works today.

07:23 Andrew Godwin: Today, Django doesn't really do nearly anything asynchronously. So if you've talked with me with Python's asynchronous solutions, Python really wasn't designed to have asynchronous programming in it, which is fair enough, it's a very old language. It's incredibly old at this point in some ways. And so especially in Django which itself is over a decade old now and was built back when Python 2.4 was fresh and young. A lot of the ways Django was built are not built around the ideas of async. And in Python, you have to build with async specifically in mind and build around things like AsyncIO or Twisted to have support for that stuff. And so before today, you just couldn't do it in Django. Like, there's no way you could, like, yield and let something else run. Like, if you wanted to do, say, go and fetch a couple of web pages, you might be able to fetch those in parallel and one worker process, but that's about as much as you could do. You couldn't share the load. And in particular, with things like WebSockets, you'd have to tie up a single Python process for every terminated socket, which, as you can imagine, quickly wastes all of your worker processes. Like for HTTP, it's great. Like, the process spins up, does a request. You know, if you've got a good site like maybe two, 300 milliseconds, finish the request, moves onto the next request. With a socket, it holds open forever and so you're quickly going to run out of those non-asynchronous worker threads. And that's kind of the problem that you butt up against very quickly.

08:52 Michael Kennedy: Yeah, you're gonna run out of space really quick.

08:54 Andrew Godwin: Oh real quick.

08:54 Michael Kennedy: If that's the case. Sorry, our website is busy. We have seven people on it. Could you come back later? That, this is not acceptable.

09:01 Andrew Godwin: Yeah, you're in a queue. You're number five in the queue. Exactly.

09:04 Michael Kennedy: You could give them a cool little count down. That would be nice.

09:07 Andrew Godwin: Oh yeah, no, you could make it a really nice experience around this really awful internal programming you have.

09:14 Michael Kennedy: Nice, so what was the motivation or sort of what pushed you to go and create Channels? Like, it's one thing to say Django doesn't support this async model super well. It's another to go and I'm gonna fix it.

09:24 Andrew Godwin: Yeah, I mean, you think I would have learned as well. So like for those listeners who aren't familiar, my path into Django core team in particular, was based on writing South which was Django's sort of, well one of the initial migration frameworks for database migrations for Django. And over time, eight years it took, I think. It became the de facto solution and then the only solution, pretty much, and then finally it was moved into the core. And so I had basically just finished, like, doing all the porting into Django core. I finally, like after eight years, had a bit of a break and of course I went what other hard problem needs solving that I can go and look at? Like this is the awful part of my sort of like idle thinking. And in particular, I had been interested in WebSockets for quite a while, so like since they came on sort of the browser scene, well four or five years ago now. I'm not, I can't remember the timeline perfectly. I've been looking at them sort of very intently and like in particular for game-related reasons like WebSockets are this unique thing in the web where you have very low latency, bi-directional communication with no overhead. Like you could, Ajax is one way. It's got quite a lot of overhead, it's quite slow. And it's hard to even send things from the server to the client without prompting. So I was like okay, I'm interested in this and then back then I sort of played around a bit. I used event alerts and sort of kept up with the ever-changing WebSocket specifications sort of fixed event alerts as the spec changed a little bit and tried to keep up with it and then dropped off for awhile. And about that time, Aymeric Augustin sort of took on the challenge of WebSockets in Django himself and he made his own attempt at this where he had a different arrangement of it. And I sort of played around with that like, okay, this is pretty interesting, I like that. And there were lots of other attempts and a lot of the con patent was that you would, you would have Django running normally with WSGI and then you would also have a separate process and that separate process terminates WebSockets. And then somehow, often with either mystery or some kind of magic, the two processes communicate with each other and you can hand off things from Django to WebSockets and vice versa, but it was always sort of bolted on. It wasn't, it wasn't really a good sort of like clean solution. It often didn't feel like Django as well. You know, it was, all it was was this extra server you ran just WebSockets. And I was like at some point, you have to have that separate server idea as we'll come to later anyway, but I wanted to sort of like, how I can make the solution that is more encompassing, that supports both WebSockets and also other things that need the same kind of problems like long polling HTTP and sort of other protocols like Internet of Things protocols are often like this, too. It's like how do we encompass all those problems into a common thing that also feels like Django, integrates really well with Django?

12:04 Michael Kennedy: Right, that sounds like a great, great goal. And really definitely something to start working for to bring all that in there. I think one of the challenges is you wanna support WebSockets, but you also talked about, like, HTTP long polling and there's a few other styles as well. Some kind of like frame, like long, slow frame or something like that.

12:28 Andrew Godwin: Yeah, yeah, yeah.

12:28 Michael Kennedy: What's that called?

12:29 Andrew Godwin: There's lots of different names for it like... I forget. The idea in particular is that you open a, you have a response and you send it in the chunked encoding. You send like one chunk and then wait and browsers can actually, the JavaScript can actually look at chunks as they come in so you can sort of like fake a one-way socket that way.

12:47 Michael Kennedy: Exactly, so the challenge is you've got WebSockets which, when you started this project, what was, what year was that?

12:53 Andrew Godwin: The project was conceived in late 2014 I would say.

12:58 Michael Kennedy: Okay.

12:59 Andrew Godwin: But there was a long period of me playing around with prototypes and ideas and a lot of failed attempts and failed API versions and trying to make a good API before it sort of emerged in 2015 pretty much.

13:10 Michael Kennedy: Okay, so you didn't have too much of the problem of browsers not supporting it, but still, a few support WebSockets, long polling, some of the other variations. You've gotta have like graceful fallback and negotiation and all sorts of interesting things that not just implementing pure WebSockets solves, right? You've got kind of a better solution than that or a broader solution.

13:34 Andrew Godwin: Well actually so no, Channels doesn't do fallback and it's one of the things that's interesting. And like one of the things we plan to do but like, so in terms of scoping, one thing I have learned from South is trying to keep scope small. And in particular, I wanted to solve the problem of, the core problem of how do we have basic async stuff in Django initially without having the fallback stuff. And we are working on fallback but like in Django, in Channels today, you can do long polling, you can do the sort of frame-by-frame push, you can do WebSockets. The fallback stuff isn't there mostly because I didn't want to reinvent the wheel in that regard. I wanted to reuse one of the existing solutions like Sock.js Or Socket.io or one of the other ones that already exists in the JavaScript world. And so the initial version was okay let's build initial great support for just WebSockets, just long polling, just pushing stuff and then later on, come and then join them together into a more sort of cohesive whole, if you will want that stuff.

14:32 Michael Kennedy: I see, so, even though you support the different protocols, it's basically you're saying, like, look, if you wanna use this in your web app, put some bit of JavaScript that will select it for you and we'll just stick with that one rather than having some really complicated negotiation.

14:47 Andrew Godwin: It's partially that and partially also what I tell all the people is that in some ways, if you're just using one of the wrappers around sort of that hides all the different things it's transporting over, you don't have as many guarantees, right? Like if you have a wrapper, you don't know the underlying connection is actually good at bi-directional communication. It might just be a long poll Ajax call and those are really bad at, say, pushing data from the server to the client. So in many ways, I'm encouraging this for if you really need WebSockets, like 100%, like, you know, you're doing real time-ish kind of stuff or programming or games or whatever or chat stuff, then this is the right stuff to do, like you got the right level of abstraction. If you want that sort of wrap-around stuff, sure, you can do it, but you might be better off just building everything long poll anyway. Because then you can build to a set of guarantees that you can sort of understand.

15:36 Michael Kennedy: Right, that is the problem. If you let the system auto-select basically different transport protocols is you don't really know what you're gonna get, right?

15:44 Andrew Godwin: Yeah, and of course WebSockets are more efficient. Like often you'd want to pick them over long polling but in terms of scoping and trying to sort of pick what to work on, I push that stuff back in favor of just getting it working first.

15:56 Michael Kennedy: Yeah, that makes a lot of sense. So I haven't checked caniuse.com for WebSockets in awhile. What's the likelihood that people coming to my site would be able to use the best version of WebSockets?

16:07 Andrew Godwin: So you have a deadly combination of it's mostly supported by all the new browsers, so, you know, like your new Chromes and your Firefoxes. I think even Edge has support for it these days. But you got problems like mobile Safari I think still doesn't support it properly. The old Android Chromes probably won't do it properly either. And then on top of that, you have the problem where WebSockets isn't normal HTTP and so a lot of proxies and corporate firewalls just block it entirely and even then, even if that works properly, your server and your hosting has to support HTTP 1.1 to correctly get this protocol. It's like what happens is it starts off HTTP and then sort of upgrades itself midstream into a binary protocol then just binary happens in the middle of it. And so if you don't understand 1.1, then your HTTP server's gonna go like what's happening and just like basically return 200 okay and break everything. So it's really sort of, you need to trio things to work properly and sometimes, that can be pretty tricky to have.

17:02 Michael Kennedy: Right and you don't always have great visibility especially with the proxy servers or the firewalls.

17:06 Andrew Godwin: Yeah, the firewalls are the worst thing 'cause like browsers, you can test JavaScript pretty easily and the server site stuff you have control over but sometimes those proxies or firewalls can really sort of throw a wrench in the works.

17:18 Michael Kennedy: Oh yeah, absolutely. So we talked about WebSockets and long polling. Some of the, what are some of the other features that you support? There's things like custom events and tasks and a number of other things as well, right?

17:29 Andrew Godwin: This is what I mentioned in the beginning, like, it's not just the support for WebSockets on top. Like, there's this whole underlying layer that's basically an event-driven layer inside Django. And it happens that on top of that, there's WebSockets and long polling but also you can use the same code for whatever you like. So if you want, you can have a custom set of events or custom channels that you send things to. You can have custom protocols, interfaces, so right now, the people working on things like a Slack interface that sort of there are channels for when you get a Slack message sent to your server so you can write bots very easily.

18:01 Michael Kennedy: Nice, so you can do things like if there's like 10 people on the site, one person could send a message and you could, like, broadcast that out to the other people who are around?

18:10 Andrew Godwin: Exactly, yeah. And so like that in particular, the thing called Groups. And Groups is the sort of built-in primitive for almost like broadcasting to multiple receiving channels. So it's like basically Channels is a couple of primitives like there's a channel where you can send and receive stuff. There's a lot of guarantees around the way it's designed. There's Groups for broadcast and then sort of this interface or API for sort of low-level talking and communication between processes and on top of those primitives is built, the WebSocket spec is built the HTTP spec. And if you want to, as we're doing at Eventbrite, in fact, you can then take that low-level spec and build, say, a service or (mumbles) on top of it as well. And that's all we're doing at Eventbrite.

18:50 Michael Kennedy: Oh very cool. So you guys are using Django Channels at Eventbrite?

18:52 Andrew Godwin: Well so in particular, well the underlying layer. So the underlying layer is rather unimaginatively called ASGI or as-gee, and what it is is definitely not a WSGI replacement, 100% not. And what the idea is, it's like ASGI is a specification for there is a thing that can pass messages around on names channels. And you can send name channels and you can receive from them and it gives you sort of some, a basic, a few basic guarantees, the methods to call on objects, a bit like DB-API 2 is in Python, DB-API 2. So things like, you know, there's a send method, there is a receive method. You can do groups in certain ways. And if you just have that, you can take things like the Redis transport we've built, that Django Channels runs on, and use it for whatever you like and do message passing inside just pure Python. Like you don't even need Django. Like, that, that, part is a pure Python library.

19:45 Michael Kennedy: That's really cool. So basically, if you've got something that can act as a transport like Redis, like a Redis queue or something, then you're good?

19:52 Andrew Godwin: Well yeah, so it's more than that. So like what happens is, like, the ASGI spec gives you certain guarantees. For example, it guarantees messages delivered at most once, so either you get it or you don't. The opposite choice being at least once meaning you get it or you get two of them. And that's a choice that you have to make and it has a lot of these choices like so you have a message expiry, so you get like 60 seconds, you have backpressure so you know if a channel is full. Sort of all these sort of guarantees that if you're building any sort of distributed system, you want to know what these guarantees are and sort of program against them so you understand your trade offs you're making. And so it comes with a pre-set, a pre-setted set of trade offs and then code built against those ready, so there's one that uses Redis for transports using lists. There's one that uses a local or shared memory segment. There's one that just works inside a process for like sort of unit testing, so you can write a nice lightweight one to write unit tests against. And also we have somebody working on a RabbitMQ one right now as well, to use RabbitMQ if you want to use that stuff.

20:46 Michael Kennedy: Oh yeah, that sounds really cool. Lotta options there. Let me take just a moment and tell you about Metis, a sponsor of this episode. A data scientist's responsibilities can range far and wide. How can you be sure you have the necessary skills and training to enter the field or keep up with emerging technologies? Metis, a data science training company, based in New York City, San Francisco, Chicago, and Seattle provides full-time immersive bootcamps, evening, part time professional devlopment courses, online resources, and corporate programs all with the goal of training you to become a data scientist and help you stay on top of new and necessary skills. Metis has a long list of professional development courses starting soon in any of these four cities, taught by industry leaders. Interested in honing your skills? Then these courses are for you. How do they work? Courses on topics like data visualization, deep learning with TensorFlow, machine learning, and statistical foundations run two nights a week for six weeks during the evening hours that fit into your busy schedule. These courses are laser-focused on relevant topics and skills that are sure to enhance your career. They have a special offer for you at thisismetis.com/talkpython be sure to check out what they're offering. It helps support the show. You talked about ASGI and people are familiar with WSGI, web service gateway interface. What's ASGI, what's the A for, asynchronous?

22:08 Andrew Godwin: Asynchronous, exactly. As I said, it's very unimaginative. Essentially I had to find a name and I stupidly chose to go with changing one letter of WSGI. And it's not even that similar to WSGI. Like, there is, it does also have a way to transport HTTP over it and that's kind of where it's semi-related. But also it's basically more of a specification for message passing at some point as well, so, not the best name but it's now stuck in everywhere so I can't really change it.

22:34 Michael Kennedy: Right, absolutely. Well, and if you look at the WSGI interface, the API. It's really, really simple and there's really very few hooks for asynchronous programming in there, right? Especially bi-directional stuff.

22:46 Andrew Godwin: Exactly. And in fact, like, one of the things I love about WSGI so much is how simple it is. Like, the interface literally, you must be a callable, you must take two arguments. That's the entire interface at WSGI, pretty much. Like there's a specification like what the environment looks like and where you get your stuff from but it's very, very simplistic and so certainly, like, that was an inspiration to me like, well, I want something that's similarly very simplistic that you can write against as well and so that kind of came along. But the thing and the place where WSGI falls down is that it has no support for asynchronous programming at all. Like I said, it's a callable. It's not a yielding thing, it's not a Twisted deferred returner, it's just a blocking callable that returns when the request is done.

23:28 Michael Kennedy: Right, that's like the exact opposite of asynchronous APIs.

23:31 Andrew Godwin: Pretty much, yeah.

23:33 Michael Kennedy: Interesting. So if I write some code here against ASGI, how interchangeable is that? Like what web servers can it run around? Like, micro WSGI, Gunicorn, like, does it plug into any of those or what's the story there?

23:50 Andrew Godwin: There's two separate things. There's ASGI which is like sort of the say it like here's a thing you can send messages to and then there's the HTTP over ASGI specification. That's kind of the WSGI equivalent, right? That's like the here's how we encode requests. Here's where the URL variables come. Here's how you send a chunk response. Like, that stuff is specified as like the format of messages you send over the channels, basically. And because of the way it has to be, it supports more than WSGI, it is of course not directly compatible. And so in particular, right now, there is one reference server, which is called Daphne. Daphne's basically bits of Twisted and bits of Autobahn, which is the really good Python WebSockets library sort of hot glued together in a way that just sort of does this specification. And people are also working on, there's been a micro WSGI plugin being worked on. Somebody has very bravely been working on a Microsoft IIS plugin, which is really impressive.

24:46 Michael Kennedy: Oh yeah.

24:46 Andrew Godwin: Like, that's--

24:47 Michael Kennedy: That's interesting.

24:48 Andrew Godwin: Serious work going into that one. And so the, but in particular, there's also a, the idea that is a superset, and so an adapter from one to the other is not hard to write. And in fact, there is a half-written adaptor already in the code base where you can plug in an ASGI back end to a WSGI server. It sort of just translates between them. The key problem is, of course, that doesn't let you run WebSockets because WSGI doesn't support WebSockets. And so that lets you sort of proxy things to servers that look like old traditional websites, but doesn't give you all the new features that you might want. And for those new features, things will have to be, like, natively supported, basically.

25:27 Michael Kennedy: Yeah, so that's why you've got things like Daphne.

25:28 Andrew Godwin: Exactly. So Daphne is like sort of the reference server and I would love to get, like, a proper micro WSGI support in there as well. Like a second option, maybe Gunicorn. Like, I have to sit down with these servers and like seriously look at them and see if I can, like, think of what, like, can I patch them more? Or like can I change the design so it works better? You know, just solve that story so I'm not just like here, run my special web server. It'll be fine. It's written by a, it's written by just Andrew on his spare time. It's probably okay. And so it's like there is a ecosystem of things you can pick from is the idea.

26:01 Michael Kennedy: Yeah, that's really fantastic. And cheers to the guy doing IIS because when you're hosting on Windows or you wanna host it in somewhere like Azure or something, that's, like, your only choice, right? It's not like Linux where there's a variety of things you put together. It's just like people on Windows, they just use IIS and that's that. So that's really cool that it could unlock that. Let's talk a little bit about the sort of Django patterns. Like how is this similar or different to the stuff that people already know?

26:28 Andrew Godwin: Like the idea is to sort of keep it familiar. So obviously we can't make it the same as Django. 'Cause like some of the things that sort of hobble WSGI also hobble Django. Like Django views are the same kind of, they take a request, they return a response. You can't really do much inside them. And so my goal was to make things that look and feel familiar and crucially have the same safety guarantees as normal Django. So you can code them the same way, ut also support this extra abilities. And so like the main thing here is thinking well of consumers. And consumers map pretty much to Django views. And the idea is whereas the Django view takes a request and returns a response, a Channels consumer gets a message and then can send zero or more other messages. So for example, if you wanted to listen to incoming WebSocket like chat messages, the consumer would be tied to the requests channel and every time you sent a thing in on the WebSocket, it would be like, okay, I'm gonna message, it would launch the consumer, the consumer can run, handle the message, and then maybe send some messages to other clients or sort of broadcast it, store it to a database, and then it exits straightaway. And the key thing here is the consumer, it takes one argument, which is a message object, which is kind of like the request object. You can have session and auth support, the same way you can in Django 2 with Decorators, if you're like message.channel_session or message.user, that kind of stuff. And also, as long as you set the thing of not doing blocking options, blocking things in the consumer, it actually makes it almost difficult, almost impossible to deadlock the process. Because it's not proper async because you can't block or listen to other channels and because you sort of have to take the message, run through the logic and then exit pretty much immediately. It means we can actually churn through messages in the same way that like Django might churn through HTTP requests with a very similar kind of set of guarantees and safety concerns.

28:22 Michael Kennedy: Well that's really interesting. I hadn't really thought of the views themselves actually having to be so cognizant of this async world, but it totally makes sense.

28:33 Andrew Godwin: Yeah, and in particular, the nice thing is that views end up being just a subset of consumers. Because a view takes a request, which is just a message in Channels land, and the returns respond, which means sending a message. So it actually means that views end up being a subset of consumers and it all sort of neatly wraps around.

28:52 Michael Kennedy: Sure. You talked a couple times about the ability to basically deadlock yourself.

28:57 Andrew Godwin: Yes.

28:58 Michael Kennedy: I've seen that before in some of this WebSocket sort of callback-style programming where you're waiting and you can't receive the message 'cause you're waiting but you're not gonna get a response. And can you maybe describe some of the ways in which you could, like, wrap yourself around in that trouble or just sort of give somebody an idea of how, what to look out for?

29:17 Andrew Godwin: Yeah, so the classic thing is where you have a process that sort of it gets the message. And say it then waited for a second message. Like, okay, I'm gonna send a thing and say I want X and wait to get X back. The problem is if one process says I want X and the other process says okay, I've got X, I want Y, to the other process, they're both then blocked in the thing of waiting for their response. But they both can't serve the response to the other process and that's a deadlock, basically. And that's very easy to get into if you don't have either an excellent knowledge of your author system, architecture, and of asynchronous programming, or if you have a framework that sort of builds in so what Channels is is it, Channels deliberately doesn't give you full async so that it's much harder to shoot yourself in the foot. Like, we don't let you block on things because that makes it very hard to, like, well say, yeah.

30:08 Michael Kennedy: You can't be blocked, you can't deadlock because you're waiting on something if you can't wait on something, right?

30:14 Andrew Godwin: Exactly. And if you want to, if you want to have that sort of low-level control, you can just drop down into sort of normal Python and just write your own async stuff against the ASGI interface. But the Channels levels of the higher-level one, that we keep sort of safer and easier to stomach. So like the idea's like, you know, if you're coming in for this as a web developer or maybe even as a brand new developer, the number of scenarios where you can have not only errors but errors that are impossible to debug is greatly reduced. 'Cause deadlock's not only nasty, but knowing they've happened and debugging that that's what's happening is really difficult.

30:47 Michael Kennedy: It's super difficult, yeah. And I heard them described as Heisenbugs.

30:52 Andrew Godwin: Yes.

30:53 Michael Kennedy: Because, you know, even when, you know, like, know that they're there by observing them or interacting, like putting logging or other stuff, sometimes can change the timing. Which actually changes them, right?

31:04 Andrew Godwin: Yeah.

31:05 Michael Kennedy: So super hard.

31:05 Andrew Godwin: One of the, one of the worst things is they often happen only under high loads. So like imagine you have like 10 worker threads. If you just have like five requests a second, it will happen perfectly fine. And as soon as you hit, like, over 10 requests a second, say, and you use all 10 of the threads, that could then trigger the deadlock because all the threads, all the workers suddenly consumed up as they spare workers to handle these extra requests. And so like it might be fine in development and you deploy it to production, it's fine in production, and then one Sunday evening when your website's under high loads, suddenly it locks up and there's no trace back of like what's happening, I don't understand. It's like, it's a really nasty sign of error.

31:41 Michael Kennedy: Yeah, it's super bad. And I think it's awesome that your framework, like, makes that harder. That, usually, that's the kind of stuff that happens right when you need the website to work most. We're running a new ad or we got featured on Hacker News or whatever. Boom, it's dead.

31:56 Andrew Godwin: Yeah, and it's not perfect, obviously. Like, we can't stop everything. But like the Django philosophy is to try to make these things hard by default, and so that's kind of tried to have been perfected here.

32:04 Michael Kennedy: Yeah, that's absolutely a good philosophy. So some other design patterns around Django that are maybe related, like routing in messaging?

32:12 Andrew Godwin: Yeah, so the other things that's familiar. So routing, so the idea is when you say I have a consumer for this channel, we have to sort of say like a view, you have to tie that consumer to the channel. So like well this consumer handles incoming messages on the WebSocket.recieve channel and it handles the ones where the path is /chat. And so there's a thing that looks very like Django's URL routing but is channel routing instead. It's like, it's not quite the same but very familiar. It's like a list of regular expressions, basically, and it has, again, very similar guarantees and a lot of power in there. And then as I said before, messages are very much like requests and like they have information on them, you can store stuff on them, there's session support in there. It's like, it all feels a bit like you're writing Django views in URLs but it's a bit more powerful with a bit extra, with a few extra options.

33:05 Michael Kennedy: Okay, yeah, very cool. And what's the programming model look like? Like, how is it the same as Django, or how is it different?

33:12 Andrew Godwin: So to the end developer, it looks largely the same as normal Django. In particular, Channels, in its default state, runs all of your code, the writers, consumers and stuff, synchronously. Like you don't ever run into an async option. And typically we run the whole of Django synchronously too, 'cause like we can't make Django async, so we just run Django synchronously with these sort of, the way sign helps just run it in sort of the worker thread and all the async is handled away from Django, basically. So if you're coming in from that end, it looks pretty much like normal Django. However, if you want to do sort of more advanced stuff or handle different protocols or even do your own async programming, it sort of drops down and becomes more of a pure Python thing where you can use whatever framework you'd like, you can use AsyncIO you can use Twisted, you can use other things, and then you just call the ASG, ASGI API, send and receive stuff as you need to. So for example, Daphne is a purely asynchronous Twisted program that happens to call ASGI in the right way inside of it. So like you sort of get a choice of do you want the familiar, happy world of Django without much power, or do you want the raw, unbridled power of async but you sort of need to be prepared to go into that particular world?

34:24 Michael Kennedy: Yeah, going down the rabbit hole there. Interesting.

34:27 Andrew Godwin: Exactly.

34:27 Michael Kennedy: So is this, if I'm doing async variants, is this still in the same process as Django or is this like a totally different thing?

34:35 Andrew Godwin: So it's a different process. In particular, like, one of the things when I sat down, I'm not, part of that initial period when I was sort of like trying to design Channels and go through the different options, I had some things that were threading, some things was using the multiprocess module, some things were using green-threads or like yielding, like generators. And eventually, I realized that if you're building a large system, as, you know, as I do at work at a daily basis, you're always gonna have more than one server. Which means you're always gonna have to have more than one process. And so given that, if I made a model that was just natively multiprocess from the get go, that would make a lot of sense in like, not just multiprocess, multi-server, like, let's make the model that is, by default, it expects to run everything on like a different CPU entirely with no shared memory or no shared state and work backwards from there. And so everything is basically done in a separate process and message passing happens between them. And so that's how things are coordinated. So it's a bit like, say, like if you've ever used Go, for example, Go, and a lot of Go primitives rely heavily on passing things between Go threads in a thing they call Channels as well, which is similar but different to Channels.

35:42 Michael Kennedy: Yeah, I was gonna ask you, ask you that about the relationship. The name is the same, the idea is kind of similar in some ways, like, was there any inspiration or similarity? Is it just random similarity, or is there more to it with Go and their Channels?

35:56 Andrew Godwin: Oh no, that is, the direct similarity that they both share, a common ancestor, which is the language called CSP. So when I went to university in Oxford, they invented CSP, they definitely teach it and they're very proud of it. It's language for basically proving asynchronous programs. And so it's a sort of a very abstract grammar for like, oh, we have a process that emits event X under channel Y and then a process that consumes channel Y and then gets over X and there's XY. And at its core, it's kind of this way sort of like you can write a set of assumptions and prove that programs don't deadlock, which is an incredibly nice way of doing things. But it also has these fundamental ideas of processes and channels and splits and so that's where my ideas came from and then sort of via Go, I think maybe the name camp channels came from Go as well, but like a different take on that same core concept of these are communicating sequential processes which are what CSP stands for.

36:46 Michael Kennedy: Yeah, this whole actor model is quite powerful for avoiding deadlocks and race conditions and, you know, writing to the same piece of memory at the same time. And a lot of the threading problems go away, right?

36:59 Andrew Godwin: And not just that but also like understanding as a developer. Like reasoning about systems is much easier when they are very separate components. You can reason about individually. Like, if it's all integrated into one big lump then trying to work out what happening is really difficult. If they're separated out, it's like oh, here is Daphne. Daphne just translates HTTP requests to messages and messages to HTTP responses. Writing tests with that is a lot easier because we just write tests where we put one in, one thing in one end, check the thing comes out the other end and vice versa.

37:26 Michael Kennedy: Yeah, you have very clear, explicit boundaries. These are the--

37:28 Andrew Godwin: Exactly.

37:29 Michael Kennedy: The message, where's, here's where the messages come in, here's where they go out and basically this is what they consist of, right?

37:34 Andrew Godwin: Yeah, and then that scales up too, it's like if you're doing engineering at scale as in with a large team, those boundaries then become team boundaries. That really helps you sort of distribute not only the testing but also like the cognitive work like understanding what the system is doing at any one time that you can draw the same boundaries and have teams work on shared understanding in a much better way as well.

37:54 Michael Kennedy: Yeah, that's a really good point to think about, how it lets people scale, not just software.

37:58 Andrew Godwin: Oh yeah, that, scaling. I'm very big proponent these days. Scaling software development is about half code scaling and about half people scaling. Like it's very important to get your team all understanding each other and not being at each other's throat all the time and a big part of that is good specs and good APIs.

38:14 Michael Kennedy: Yeah, that's awesome. This portion of Talk Python to Me is brought to you by Hired. Hired is the platform for top Python developer jobs. Create your profile and instantly get access to 3500 companies who will work to compete with you. Take it from one of Hired's users who recently got a job and said, "I had my first offer on Thursday after going live on Monday, and I ended up getting eight offers in total. I've worked with recruiters in the past but they've always been pretty hit or miss. I tried LinkedIn but I found Hired to be the best." "I really like knowing the salary up front privacy was also a huge seller for me." Sounds awesome, doesn't it? Well wait until you hear about the signing bonus. Everyone who accepts a job from Hired gets $1000 signing bonus and as Talk Python listeners, it gets way sweeter! Use the link hired.com/talkpythontome and Hired will double the signing bonus to $2000. Opportunity's knocking. Visit hired.com/talkpython to me and answer the door. So if I'm gonna run Django app using Channels, like what's, how's this work? Can I run it the same? Probably Daphne is involved somewhere, right?

39:31 Andrew Godwin: Basic level, you run two processes. You run Daphne and what Daphne does is Daphne terminates HTTP and WebSockets for you. So like that's what you expose to the world on port 80 or whatever. Or violet, put it behind NGINX, do static file serving or something and that sort of fills the role, let's say, like Gunicorn might fill. Like here's a process that terminates HTTP. And then separately, you run one or more Django worker processes and those processes look at the channel queues that are in this sort of channel routing that we mentioned earlier and sort of just run the consumers on those route channels as messages come through. And so at a basic level, you run one Daphne and you run one worker and then it just sort of works as normal Django does. And in particular--

40:13 Michael Kennedy: I see, so the request comes in and if it's like a channel request for say WebSockets, it just stops at Daphne and gets processed there but if it's a, something that matches for Django view or something, it will like pass through to the Django worker process?

40:28 Andrew Godwin: No, so in fact, Daphne, Daphne does both of them. So like both HTTP and WebSockets both get turned into messages on Channels. So that's kind of the difference to a lot of those previous solutions I talked about at the start of the show is that Channels has a sort of slightly ambitious goal, maybe, of putting everything over Channels. This includes HTTP as well. And so what happens is if you have a HTTP request that comes into Daphne, Daphne takes it, it decodes it into the ASGI HTTP format, pass that to Django, and then Django has a native ASGI handler next to the WSGI handler that takes that, turns it directly into a request object and then runs the views on that stuff, too.

41:07 Michael Kennedy: Okay, yeah, that's very cool.

41:08 Andrew Godwin: So it is the same system as the Channels on Django basically.

41:12 Michael Kennedy: Okay, excellent. And you still have things like runserver and so on?

41:16 Andrew Godwin: Right so run server, like, as development, just, you type run server, like, if you install Channels and type runserver, it just run Daphne worker for you in the same processes and they're sort of fitting around there. It's just easy to get going. And it means you, basically, you can pip install Channels, put it in your installed apps, and then WebSockets just work on your development server straightaway, which is really nice to have for sort of hacking around. And then sort of the other thing you can have and that I sort of recommend, especially the people who are wary or like people who are like doubt in the dyed in the wool (mumbles) people, should we say, is you can run both WSGI and Channels at the same time and just route WebSockets to Daphne and then route normal requests to, say, micro WSGI or Gunicorn. And so that way, you can use sort of more proven software for your main site but still use Channels for your other stuff if you want that sort of that trade off.

42:03 Michael Kennedy: Oh that's pretty interesting. And you're thinking put something like NGINX in front of it and switch on like protocol or something?

42:10 Andrew Godwin: Yeah, so you can switch, the easiest to switch on is URL paths or like other things on us like /WSwebsockets. But you can with a couple of tricks switch on the upgrade header as well, which is how you upgrade WebSockets. So like there are ways and means. It's tricky to switch based on what's coming in like you can do it, it's very hard to, so generally I recommend like sort of a path-based solution. And like there's examples in the Channels docs and mailing lists about like how that might run as well and sort of examples of how to set that stuff up.

42:38 Michael Kennedy: All right so what's the extensibility story here?

42:39 Andrew Godwin: So Channels is in fact, is very well built around extensibility, so like as I said before, it's not that much in the base package. You get sort of, WebSockets supports there, that kind of stuff, but like it's designed to be built upon in sort of a community way like Django itself. So things, for example, like the ability to have proper heart beating of people connected. So you know they're definitely still connected, like that is a thing that you could put on top of Channels, and so what I'm trying to do is like foster a community of third-party apps that are built against Channels and the common specifications. In many ways, that's why I took it upon myself to build a sort of more official is not the right word, but a singular solution that so that we can sort of tout as Django's solution for this stuff so that there is a common thing that people can build against. 'Cause like a lot of the part of Django's success is having a common stand that people write against. Like, it's a common known factor. And so really is designed around that idea of like, yeah, everyone's gonna have the same routing, everyone's gonna have the same idea of consumers. Like things integrate the singular way and like trying to encourage that kind of ecosystem as it were.

43:43 Michael Kennedy: Yeah, absolutely. I, you know, that's definitely a hallmark of Django is all the pieces you can bring together to make it more than what it is out of the box.

43:51 Andrew Godwin: Exactly.

43:52 Michael Kennedy: Nice. Are there like some notable third-party packages or projects that you wanna talk about?

43:57 Andrew Godwin: Not, none that I can talk about off the top of my head. Like, I don't know them off by heart. We have a page on the docs that links to a couple of ones that really going good.

44:04 Michael Kennedy: Okay awesome.

44:05 Andrew Godwin: There's also a few more that aren't on there yet. I encourage the developers to add links to the doc.

44:10 Michael Kennedy: Okay. So let's see. We talked about Daphne a little bit. Tell me some of the packages that you're using to put this together.

44:16 Andrew Godwin: Yeah, so like one of the things was like, try and like, Django for a long time was criticized as being, oh, Django's one giant package, it's not split in separate bits, so I went the entire opposite way. There are five different packages as part of Channels. So you have the base Channels package, which is the basically the Django plugin side of things is Django third-party app, well second-party app I guess, or first-party app, but it's a Django pluggable app that you load into Django. It does all the routing and runserver overloading. We talked about that before. And gives sort of the nice user experience, but it doesn't give you the serving and so Daphne's the other half of that. Daphne's the service, sorry, Daphne's the server that terminates HTTP and WebSockets and sort of translates them through into Channels for you. If there was something else there, you could just use Channels in a different thing that wasn't Daphne, but for now, we've bundled them together pretty much. And then underneath that, there is the ASGI packages, so the things that build this sort of low-level protocol we build upon. So there's ASGIref, which is sort of the base ASGI library. It has the conformance test suite in it so you can like, if you're writing a new ASGI backend, you just load up the conformance tests and it just tests against the spec which is really nice to have for new implementations. And it has a memory backend, too, for testing, like in memory for unit tests. And on top of that, you have two different backends, the Redis backend, called ASGI_Redis and that is a standard Redis backend, that we recommend for production use. It uses Redis lists and blocking pops and lua scripts and a few other things in there and it has built-in sharding if you want to use more than one server, it does scale up. And then at sort of a midway point, there's a ASGI_IPC backend. That only works on one server and it's designed for if you don't want to run a separate Redis server as well, you just want to run things on one machine, keep it simple. It doesn't perform nearly as well but it is zero configuration. You basically just load it up as the option backend, just set the same prefix and things just talk to each other on the same machine for a shared memory segment.

46:10 Michael Kennedy: Oh yeah, that's really nice. Really nice. So what's the story with Python 2 versus Python 3 on this project?

46:17 Andrew Godwin: So everything is built Python 3 first. Matter of fact, Django itself's going Python 3 only very soon, as of the next major release, I believe. That's in Django 2. And so it's built with Python 3 first but also Python 2 support is still there. So Channels supports the previous LTS of Django and up, so that's 1.8 and up. And 1.8 supports Python 2.7, so it supports Python 2.7 and Python 3. All my testing and all my development's done on Python 3 and then we have Travis just running Python 2 tests occasionally. And I'll just occasionally pop down to Python 2 and make sure things's work expected but I've seen like one bug that was Python 2 specific in the entire time I've been developing so far. It's not really a concern at all in that regard.

46:59 Michael Kennedy: Oh, that's really fantastic. And I'd love to see you doing this in Python 3 first. You know, I talked to the guys at the BeeWare Project, and they're doing Python 3 only. I'm surprised to hear about Django going Python 3, although excited to hear it. I think Python 3 is really, it's crossed some kind of threshold or critical mass recently, I feel.

47:21 Andrew Godwin: Yeah, like I feel the same way, too. Like, in the last year or year and a half or so, we definitely crossed that threshold of like things are coming out in Python 3 only a lot of cases. We're finding libraries that work now that were like oh, this great accounting library is only on Python 3 and then something like we need to move code to Python 3. It's become almost this trend now. And like it's difficult for a big company like Eventbrite to move to Python 3 of course, but definitely this is, a sea change is happening these days and everything I write from scratch is done in Python 3.

47:52 Michael Kennedy: Yeah, that's fantastic. I feel like I'm noticing it in that people are apologizing for it being in Python 2 in the cases where it is. Or they feel guilty or bad about it or whatever whereas a few years ago, it was kind of like yeah, it's Python 2 of course. Ha-ha, Python 3, whatever, right? So this is, it's really cool. Tell me a little bit about Django going Python 3 if you know any of the backstory. What's going on there?

48:15 Andrew Godwin: So we decided quite awhile ago to draw a line in the sand, as it were. So like, you know, at some point, we need to get Python 3 openly. In many ways because Django's job is, in some ways, as a leader in this area. Like, if Django's Python 3 only, that's a big signal to a lot of other Python packages that yes, you can do this too.

48:32 Michael Kennedy: It's a huge signal.

48:33 Andrew Godwin: It's a huge signal! And so the was in sight and like we debated for a long while I'd be like, well, like what version of Django should we get on, like, when should we do it? And so different core developers were arguing at different times for like oh, well could we do it now? So like, I think like two or three years ago, people were saying like, oh, let's just do it now. Let's just switch straightaway. And then eventually we decided that Django 2.0 is the first release to only support Python 3. And in particular, Python 3.5 and above because the reasoning being that Python 3 is small enough that people are pretty much on the newest version of Python 3, there's not many differences to stop you upgrading to Python 3.5. And so what this means for Django in particular is we can remove swathes of compatibility code. Like there's so much in there. Some of the core devs are literally chomping at the bit to just delete huge reams of code that are just Python 2-specific. In particular, like, we can use things like all the new features, we can get some of the class inheritance stuff can get done properly. Like all this sort of stuff that we've had to sort of keep going for Python 2, keep it there, is all being swept away. And the nice thing, too, is that because it's Django 2.0, we can say, okay, this is a nice, big change. And, in fact, like the Django number system is changing entirely, so for reference, what's happening is we're making so that Django's releases are, there's, every major LTS will be the last number in its major number, so 1.11 will be the last one, then we'll have 2.0, 2.1, 2.2, then 2.3's the next LTS. Then 3.0, 3.1, 3.2, and 3.3 are the LTS and that kind of strategy. And it just happens that it's kind of timed nicely that Python 3.5 support happens in Django 2. So it's a good reason to call it Django 2, I think. We were worried for awhile in Django Core that there wouldn't be a good thing in Django 2, be like, oh, and this is just another maintenance release of Django with some bug fixes but it's nice to have a bit of a sort of big change in there as well.

50:22 Michael Kennedy: Yeah, that's, that's really cool. And I think changing the major version number sends a strong signal, like this thing is different.

50:28 Andrew Godwin: Exactly. And it's easy to port. Like I have been doing for the last three or so years now, porting code to Python 3, really not that hard. Like once you get your head around what's going on and if your encoding's already good, like Python 3, and I like this about Python 3, really punishes you for having good encodings. Like knowing where your bytes come from. Which, for developing Channels, which is all about like handling bytes over the wire versus like encoded messages or URLs, is incredibly useful. It's like oh, well you've tried to pass this byte string to a unicode string function, just blows up. It's like having, that's all in order and you know where your strings are coming from, it's pretty painless support, in my experience.

51:09 Michael Kennedy: Yeah, that's really excellent. And it is at sort of that network layer that you see some of the differences between Python 2 and Python 3 more when you're down at the string level and taking bytes and converting them. But yeah, it sounds really good.

51:22 Andrew Godwin: Yeah, it's, it's really good. Like my main complaint is that I keep typing print with a space rather than print with a parenthesis. Like muscle memory's a really hard thing to undo.

51:29 Michael Kennedy: Yeah, absolutely. So when we talked about WSGI and we talked about AWSGI, the asynchronous thing, what came to mind for me was this concept of WSGI 2, which is being sort of brought into existence by the pressure that HTTP2 is putting on the WSGI specification, right?

51:52 Andrew Godwin: Yeah, other pressures, too. WebSockets is part of those pressures as well.

51:56 Michael Kennedy: Yeah, absolutely, well I was, you're right, WebSockets predates HTTP2, but all this sort of we wanna run more than one thing over this channel and we wanna sometimes do it in binary just doesn't work at all. So can you talk a little bit about like WSGI 2 and maybe how that like would help or is parallel to what you're doing?

52:16 Andrew Godwin: Yeah, so like I was actually involved in discussions for awhile, so there's a group in Python called Web-SIG, and Web-SIG is sort of the place where things about WSGI get discussed. And there's been on and off, like a few years now at least, I came in pretty late to the discussion, this idea of WSGI 2. And one of the main problems is that if you wanna call it WSGI, it has to look a lot like WSGI and in theory, they're backwards compatible as well. And that really hinders the approach. And, like, as of the time of recording, there's been very little progress on WSGI 2 for like about two years, if not longer. Also different ideas about what should be done or different approaches, like how to even make things async and so my conclusion to that was that like Django needs something Django, we needed to do something that existed and worked. And I've read through all the threads and tried to educate myself on the arguments and the reasons behind different people's visions and wants for WSGI 2. And took some of those with me to the design of ASGI and in particular the HTTP layer on top of it. But at the same time, I don't want to stride in and declare a replacement for WSGI with ASGI, rather it is a different thing. I also think that a different thing is the right solution. I think that Python, Python web systems are more than just handling HTTP responses these days and so my ultimate goal is probably to try and sit down, get Django and get more implementations of this going through, and then when we have a good set of reference implementation, like two references on either end, it's proven in production. Then take that proven thing back to Web-SIG and say hey, can we decide on this thing that's already in place, that has proof behind it? And then we can see what's happening with it, rather than trying to argue in the abstract, basically.

54:06 Michael Kennedy: Yeah, I think arguing in the abstract is, it's really hard to do.

54:10 Andrew Godwin: It's very difficult.

54:11 Michael Kennedy: Yeah, it's, you can just go around and around and around and I, you know, I don't know exactly what's happening internally at Web-SIG but if it's been kind of idle for two years and HTTP 2's coming. The web is not standing still, so I feel like you know, maybe there's some of that going on.

54:27 Andrew Godwin: I am perfectly willing, if someone comes up to me tomorrow with an amazing WSGI 2 spec that everyone adopts, I will immediately drop in support of that stuff. But like given the lack of movement, I felt that someone has to do something, and so I decided to be that someone and do something.

54:42 Michael Kennedy: Yeah, this is really cool. I mean, if you get this to be successful and proven, like you said, it's really easy to take it back and go look, guys, this is more or less working. Let's make a spec, let's extract a spec out of this general idea. Rather than debate ad infinitum.

54:59 Andrew Godwin: Exactly, I've been trying to keep conversations going with like various authors or luminaries of Python web stuff. Like not enough, but like getting opinions and feedback on like the way things are done. Like, you know, silly things like cleaning up the, it's like I said encoding, cleaning up the encoding of how strings are passed around in ASGI, like ASGI has an exact specification of which bits of the request come through as unicode and which bits come through as bytes, which WSGI does not have and can kind of never have. So like I've been trying to keep a lot of those things in mind. I don't know if it'd be successful in the end but I'm hoping like pragmatism is worth a lot here. And even if the message passing sort of asynchronous bit doesn't survive, the basic format of here's how you take a request and put it into a message dict will still be useful.

55:45 Michael Kennedy: Yeah, I think it's really positive that you're working on this and hopefully, hopefully that can, you know, thaw this blockade or whatever's going on there.

55:55 Andrew Godwin: At some point, like doing open source collaboration is really difficult. Like it takes a lot of effort. So like, it takes not just me but other people so hopefully we can get something out.

56:04 Michael Kennedy: Yeah, hopefully. I mean, WSGI itself has been so important and so successful that it's just really scary and hard to change. So I think that's, it's almost a victim of its own success in that regard.

56:16 Andrew Godwin: Yeah, WSGI is like right next to CGI in like how pervasive and successful it's been. Like every like Python web program runs on it. It's incredibly, it's incredibly simple, it's very powerful and the plugability of different servers is like a huge boon in its deployment, right? Like deployment in Python is never as easy as PHP but it's pretty damn close, considering, like, we have all these different options. So that's really something to aspire to, I think. Like, WSGI has an amazing specification that I am always in awe of, like every time. I'm very pleased that the original authors like decided to settle on it like so long ago as well.

56:56 Michael Kennedy: Yeah, absolutely. I mean, it's been going on 15 years or something. It's still going strong. It's just starting to show its age. That world was much less about serving multiple things and bi-directional stuff over the same channel and all sorts of things like that.

57:11 Andrew Godwin: Right, survived 15 years in software is an incredibly long time. Like when Django hit 10, I was feeling, I was already feeling like a bit old.

57:18 Michael Kennedy: Right?

57:19 Andrew Godwin: And so yeah, it's really impressive.

57:21 Michael Kennedy: Yeah, absolutely. Okay, so that's really great. So maybe, maybe one final question is how's all this useful beyond Django? Like it's cool that we can plug it into Django. What else can we do with it?

57:34 Andrew Godwin: So like really my goal, my ambitious goal, at the end of all of this, I think, is to make writing distributed systems in Python easier. Maybe not even in Python. Maybe in general. But, you know, I am, as like my career goes on, I find myself running into stuff where distributed systems are the one challenge I keep both enjoying and that challenges me time and time again. And like I would love to have this idea of message passing, this idea of channels and passing messages, these primitives be the way that Python programs in general, like not just HTTP but like chat bots and like Internet of Things systems and like stats systems, all these things, have this common base construct to build upon and talk to each other and message pass. Like I think that's the ultimate goal beyond Django. There's certainly a long way to get there. The middle goal is to have other Python web stuff work with it as well, but I have, like, you know, and I think it's nice to have somewhat grand ambitions for improving Python in general. I like Python the language a lot. Like I'm not just a Django person, right? Like Python is my real home in many ways. So I want to make sure that everyone can benefit from this stuff and to really push the state of the art forward in that respect.

58:46 Michael Kennedy: Yeah, that's really cool that it's got that sort of broader goal. You know, you're talking a lot about moving to Python 3 and this is Python 3 first. Do you see a world where you could use this idea of channels and this as the infrastructure for those channels mixed in with like async and await to be, to make Python much more Go like?

59:07 Andrew Godwin: That was kind of one of the ideas, yeah. It's like Channels in the spec, or ASGI in the spec, rather, has provisions for an AsyncIO-compatible receive command so you can do, you can await or receive and have it block correctly and do all that kind of stuff. So the idea is really that you can, and moreover, it's not just that you can write AsyncIO code it's that you can also write synchronous code and have that intermix. Like I'm a strong proponent of if the code you can write can be synchronous or at least can be written in a synchronous fashion, that's fine, as long as it inter-operates properly. Like synchronous code is easier to reason about, it's easier to test in many cases. It's like if we can have a system that supports both AsynchIO natively and synchronous code natively, and they interact properly and you can plug and replace different parts of the system correctly and like oh well we've written this part of the system, like, you know, this waiting room system here is in AsyncIO, whereas like this emailing system here is in Twisted and this part that's over here is in synchronous code. I think that's also a good goal to aim for. Like, flexibility's really important.

01:00:09 Michael Kennedy: Yeah, absolutely. All right, that sounds like a cool future. I'm looking forward to seeing it.

01:00:13 Andrew Godwin: Let's hope we get there.

01:00:14 Michael Kennedy: For sure. All right so I think maybe we'll just leave it here. This is a really cool project that you put together and I'm excited seeing it gaining traction.

01:00:23 Andrew Godwin: Thank you.

01:00:24 Michael Kennedy: Yeah, you bet. So before I let you outta here, let me ask you two questions I always ask at the end of the show. First of all, if you could write some code, Python code, other code, what editor do you open up?

01:00:34 Andrew Godwin: I'm a Sublime Text user. I have been for like four or five years now. It's just, it's got Python in it, too, so you can't argue with that, really, can you?

01:00:40 Michael Kennedy: No, that's definitely the home for Python there. Have you checked out the Anaconda plugin for it?

01:00:45 Andrew Godwin: I haven't, no.

01:00:46 Michael Kennedy: Okay.

01:00:47 Andrew Godwin: I have my own sort of set of plugins and custom plugins I've clutched over the years. It sort of, they sort of work together mostly. It's pretty decent.

01:00:55 Michael Kennedy: Nice. All right and there's 96,000 packages on PiPIe, there's a ton that you probably come across that people haven't heard of necessarily or that you'd like to recommend, what's on your mind?

01:01:06 Andrew Godwin: The one I've used most recently I was really impressed with is Cryptography which is they're actually quite new, given the name is quite generic. It's a really well-done crypto-package that is designed in a way where it's very hard to shoot yourself in the foot. As you know I'm a big believer in this. And like it comes with these amazing, fully-featured primitives for oh, here is a way to do symmetrical mesh encryption. Here is a way to do asymmetrical public private key encryption. And it's just like, there's safe defaults, it's easy to use, and as long as you follow the docs, you end up with a really nice solution that is compatible properly and like is pretty quick. And like that's what we use inside ASGI Redis if you want at rest encryptions. So like Cryptography is my current favorite PiPI package.

01:01:45 Michael Kennedy: Oh, that's really cool. I definitely think that Cryptography and hashing and password management, that stuff is better put into a library or package and just really baked in with the best practices and you just follow the steps.

01:01:58 Andrew Godwin: Yeah, I do not want to have to think about it. I want somebody else who's more intelligent than me to solve the problem so I can use their code. It does that really well.

01:02:05 Michael Kennedy: Yeah, that's great. I'm gonna have to check that out. All right so before we get outta here, final call to action, how do people get started on Channels, like what kinda support or help are you looking for?

01:02:15 Andrew Godwin: So yeah, so like I'd encourage anyone who's interested to go over to channels.readthedocs.io it's sort of the main Channels documentation. You can see installation instructions, a short tutorial, stuff there. And there's also a big list of stuff we want to work on so our GitHub repository, which is github.com/django/channels there's a whole list of issues there, You can email Django developers or Django users with questions about Channels. It's all part of the Django Project. And then if you want to do work on a big project, we even have some funding available for bigger projects we can dish out for sort of things that are, you know, at least a couple of weeks. Sort of bigger tasks that wouldn't be tackled otherwise in spare time. So if you're interested in any of those things like get in touch with the mailing list or get in touch with me directly I'm @AndrewGodwin on Twitter. Just all that stuff, I'm happy to talk about Channels.

01:03:02 Michael Kennedy: Ah, that's really great. So lots of ways people can get involved.

01:03:05 Andrew Godwin: So many, yeah.

01:03:06 Michael Kennedy: Cool. Andrew, thanks so much for taking the time to talk with me and sharing your project.

01:03:11 Andrew Godwin: Thank you so much for having me on. It's been a pleasure.