Adding concurrency to Django with Django Channels

00:00 One of the major areas of innovation in Python, especially Python 3, is advances in async and

00:06 concurrent programming. Yet when working with any of the major web frameworks, Django, Flask,

00:11 or Pyramid, there is basically no concurrency option. That's why Andrew Godin decided to tackle

00:17 the issue on the Django side with his project Django Channels. This is Talk Python To Me,

00:22 episode 98, recorded January 17, 2017.

00:26 Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the

00:55 ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter

01:00 where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm

01:05 and follow the show on Twitter via at Talk Python. This episode is brought to you by Metis and

01:12 Hired. Thank both of them for supporting the show by checking out what they have to offer

01:16 during their segments. Andrew, welcome to Talk Python.

01:19 Thank you. It's very nice to be here.

01:21 Yeah, I'm excited to have another Django topic on the show. I don't cover Django that

01:26 often and people really love it. And I'm excited to sort of mix together the worlds of asynchronous

01:31 programming and Django and this cool project that you're working on called Django Channels.

01:36 Yeah, I'm excited to talk about it too. Like it's combining one of the hot new things in

01:40 Async with one of the boring old things in Django, one might say. But yeah, it's going to be a good

01:45 thing to talk about. Hopefully people can learn some stuff from this.

01:48 Yeah, I'm sure they will. I'm sure they will. And before we dig into that though, let's hear

01:52 your story. How do you get into programming?

01:53 So I have sort of those almost cliche stories. Like I got into programming about the age of 14

01:59 or 13 or so. Just like sort of the young sort of programmer model. I started off weirdly on a,

02:07 I think it was a Palm Pilot 3 X. Just writing in like sort of simple basic program running small

02:14 games. And that quickly went into like doing JavaScript and websites and then doing PHP.

02:18 And then finally about, about 10 years ago now going over into doing Python. And I've been doing

02:24 Python ever since pretty much.

02:25 That's really cool. What language did you program the Palm Pilot thing in?

02:28 It was like a weird variant of basic. Like I need to go back and like find the actual program,

02:33 like posterity, but it was like a very simple basic, had like basic graphical stuff and audio

02:38 stuff. Like a simple breakout clone, a sort of touchscreen keyboard, a few things like games

02:46 and interactivity has always been a big part of my attraction to programming. So like that's,

02:49 that's why I started out doing definitely. And that's kind of almost still where I do a lot

02:54 of my hobby stuff outside of web stuff.

02:56 Oh yeah, that's cool. Games, games are definitely fun. Although they're both easier and harder

03:01 these days, right? The programming languages are better. The tooling is better. But if you're

03:05 going to make a serious game, like that's a, that's a team of people for a year, right? Like

03:10 it, it's not just, you can throw out something really simple and it'll sort of compete.

03:14 Oh, like starting a game is super easy. Finishing and polishing one is the hardest thing in the world.

03:20 Like I've, I've released one of them. It took me like almost two years for a simple puzzle game.

03:25 So I'm very aware of how hard it is. I have a lot of respect for those teams that make actual

03:29 proper games.

03:30 I do as well. Absolutely. I worked on some 3d simulators and stuff for a while and

03:35 after that, I'm like, okay, this is really interesting and it's probably not something

03:39 I want to do anymore. So, so you said you got into Python about 10 years ago. Like what was the,

03:46 how do you get introduced to it?

03:47 Oh, it's all a bit hazy now. So I know in particular that, actually it was more than 10 years ago.

03:53 It's about 12 years ago now. So that the first thing I ran into, like I was running PHP apps at the

03:58 time for, you know, for about three or four years before this. And then I ran into turbo gears and turbo gears

04:03 back then was just sort of a couple of other packages glued together with extra stuff.

04:07 And that was my introduction to web programming in Python in particular. Like I was looking for an

04:12 avenue away from PHP and you know, the number of security vulnerabilities, even in my first open

04:17 source product back then was astonishing, including the ability to read the ETC password file. So that

04:23 was fun.

04:24 So I was like looking, I was looking for a sort of alternate language and I think Python

04:29 cropped up as like a common suggestion and turbo gears was one of the big things at that point.

04:32 Django was around, but very young and new as well. And so I sort of started out doing turbo gears for,

04:38 for a couple of years. And then I had the opportunity to go and work at an agency with Simon Willison,

04:44 who's one of the co-creators of Django. And it is those four or five short weeks with Simon,

04:51 who is, if you ever meet him, one of the most enthusiastic people in the world that converted

04:57 me from turbo gears to Django by almost just force of personality. And I've been doing Django pretty

05:02 much ever since.

05:03 Oh, that's really great. It's, it's lovely to be around people that are so excited about technology.

05:07 It just, it draws you in, doesn't it?

05:10 Oh yeah. I learned so much. Like I basically pair programmed with him for like,

05:13 you know, three weeks continuously almost. And I just learned so much both about like,

05:18 not only like how Django works and how Python works in general, but also like a lot of the

05:22 philosophies of like, you know, things like the Zen of Python that you don't necessarily learn going

05:26 into the language per se, but you sort of pick up along the way. So that's the kind of stuff that's

05:29 really valuable.

05:30 Oh yeah, absolutely. That's really, really great.

05:33 Let's talk about this project that you've created, Django Channels. Can you just kind

05:38 of tell us briefly, you know, what is Django Channels?

05:40 It's a very hard thing to describe, but I'll have a go. So Django Channels is actually,

05:44 it's kind of two things. It's important to, as we talk about it more in the podcast.

05:48 So first of all, so the headline feature as you were is Django Channels is a way of having

05:55 WebSockets supported in Django. And we can, we can go in later on as to why that's difficult and hard,

06:01 but needless to say that until Channels existed, WebSockets was very hard to do in Django just

06:06 due to some of the technical limitations therein. But the second part of Channels is sort of this

06:12 underlying layer of like to solve the problem in WebSockets, there's a lot of interesting

06:16 technical challenges that go with that. And so part of it is like, well, how do we solve the problem of

06:22 writing semi-asynchronous code, not fully asynchronous code, but semi-asynchronous code with Django and having

06:28 things like the Django ORM and the middleware and stuff and things that people are familiar with

06:33 still work. Like how do we keep all that stuff familiar yet also have an extra power in there?

06:38 And so it's sort of this two-faced story of the underlying solution that powers everything.

06:43 And then on top of it, the nice layer of, and here's WebSockets support to go on top of it.

06:47 Okay. And just making that really easy and giving people a great API for it, huh?

06:52 Yeah. So like, you know, the channels part, sort of the top layer is very much the let's have a nice

06:56 design thing with a good developer experience that says, hey, you can WebSockets. And more importantly,

07:02 it's very hard to shoot yourself in the foot or deadlock or live lock or any of the things that

07:08 asynchronous programming gives you as like sort of loaded ammunition very easily.

07:13 Absolutely. So before we dig into your project too much, like let's talk about how asynchronous

07:18 programming in Django without your project works today.

07:22 Today, Django doesn't really do nearly anything asynchronously. So like if you are at all familiar

07:28 with Python's asynchronous solutions, Python really wasn't designed to have asynchronous programming in

07:35 it, which is fair enough. It's a very old language. It's incredibly old at this point in some ways.

07:40 And so especially in Django, which itself is over a decade old now and was built back when Python

07:46 2.4 was fresh and young, a lot of the ways Django is built are not built around the ideas of async.

07:52 And in Python, you have to build with async specifically in mind and build around things

07:56 like async, hyalur, twisted to have support for that stuff. And so before today, you just couldn't

08:03 do it in Django. Like there's no way you could like yield and let something else run. Like if you wanted

08:08 to do say, go and fetch a couple of web pages, you might be able to fetch those in parallel in

08:15 one worker process, but that's as much as you could do. You couldn't share the load. And in particular,

08:19 things like WebSockets, you would have to tie up a single Python process for every terminated socket,

08:26 which as you can imagine, quickly wastes all of your worker processes. Like for HTTP, it's great. Like

08:32 the process spins up, does a request. You know, if you've got a good site, like maybe two,

08:39 300 milliseconds, finishes the request, moves on to the next request. With a socket, it holds open

08:44 forever. And so you're quickly going to run out of those non-asynchronous worker threads. And like,

08:49 that's kind of the problem that you butt up against very quickly.

08:52 Yeah. You're going to run out of space really quick. If that's the case. Sorry, our website is busy. We

08:57 have seven people on it. Could you come back later? Yes. You're in a queue. You're number five in the

09:03 queue. Exactly. You could give them a cool little countdown. That would be nice. Oh yeah, no. You can

09:08 make it really not a really nice experience around this really whole full internal programming you have.

09:12 Nice. So what was the motivation or what pushed you to go and create channels? Like it's one thing to say

09:19 Django doesn't support this ASIC model super well. It's another to go and I'm going to fix it.

09:24 Yeah. And you think I would have learned as well. So like for this list is not familiar. My path into

09:30 Django core team in particular was based on writing South, which was Django's sort of what one of the initial

09:37 migration frameworks for database migrations for Django. And over the time, about eight years it took, I think,

09:42 it became the de facto solution and then the only solution really much. And then finally it was moved into core.

09:49 And so I had basically just finished like doing all the porting into Django core. I finally,

09:55 like after eight years had a bit of a break. And of course I went, what other hard problem

10:00 needs solving that I can go and look at? Like this is the awful part of my sort of like idle thinking.

10:06 And in particular, I've been interested in WebSockets for quite a while. So like since they came on sort of the browser scene,

10:12 well, four or five years ago now, I'm not, I can't remember the timeline perfectly. I've been looking at them

10:18 sort of very intently. And like in particular for game related reasons, right? Like WebSockets are this

10:24 unique thing in the web where you have very low latency, bi-directional communication with no overhead.

10:29 like Ajax is like one way. It's got quite a lot of overhead. It's quite slow.

10:33 And it's hard to even send things from the server to the client without prompting. So I was like, okay,

10:38 I'm interested in this. And then back then I sort of played around a bit. I used eventlets and sort of

10:43 kept up with the ever-changing WebSocket specification, sort of fixed eventlets as the

10:49 spec changed a little bit and tried to keep up for it. And then, and then dropped off for a while.

10:53 And about that time, Aymeric Augustin sort of took on the challenge of WebSockets and Django himself.

10:59 And he made his own attempt to this where he had a different arrangement to it. And I sort of played around

11:04 with that, like, okay, this is, this is pretty interesting. I like that.

11:06 And there were lots of other attempts. And a lot of the common pattern was that you would,

11:10 you would have Django running normally with WSGI. And then you would also have a separate process.

11:16 And that separate process terminates WebSockets. And then somehow, often with either mystery or some

11:22 kind of magic, the two processes communicate with each other. And you can hand off things from Django

11:27 to WebSockets and vice versa. But it was always sort of bolted on. It wasn't, it wasn't really a good

11:31 sort of like clean solution. It often didn't feel like Django as well. You know, it was,

11:36 this extra server you ran just WebSockets. And I was like, at some point, you have to have that

11:40 separate server idea, as we'll come to later anyway. But I wanted to sort of like, how can we make the

11:45 solution that is more encompassing, that supports both WebSockets, and also other things that need

11:50 the same kind of problems, like long pulling HTTP, and sort of other protocols, like incident of things

11:56 protocols are often like this too. So how do we encompass all those problems into a common thing

12:00 that also feels like Django integrates really well with Django?

12:03 Right. That sounds like a great, great goal. And really definitely something to start working for

12:10 to bring all that in there. I think one of the challenges is you want to support WebSockets,

12:16 but you also talked about like HTTP long polling. And there's a few other styles as well.

12:23 Some kind of like frame, like long, slow frame or something like that.

12:27 Yeah.

12:28 What's that called?

12:29 There's lots of different names for it. Like, I forget. The idea in particular is that you open a,

12:34 you have a response and you send it in the chunked encoding. You send like one chunk and then wait.

12:39 And browsers can actually, the JavaScript doesn't actually look at chunks that come in. So you sort

12:44 of like fake a one-way socket that way.

12:46 Exactly. So the challenge is you've got WebSockets, which when you started this project, what year was that?

12:53 The project was conceived in late 2014, I would say.

12:58 But there was a long period of me playing around with prototypes and ideas and a lot of failed attempts

13:04 and failed API versions at trying to make a good API before it sort of emerged in 2015, pretty much.

13:10 Okay. So you didn't have too much of the problem of browsers not supporting it, but still,

13:16 if you support WebSockets, long polling, some of the other variations, you've got to have like graceful

13:21 fallback and negotiation and all sorts of interesting things that not just implementing pure WebSockets

13:29 solves, right? You've got kind of a better solution than that or a broader solution.

13:33 Well, actually, so no, so channels doesn't do fallback. And it's one of the things that's

13:37 interesting and like one of the things we plan to do, but like, so in terms of scoping, one thing I have

13:41 learned from South is trying to keep scope small. And in particular, I wanted to solve the problem of the core problem of how do we have basic async stuff in Django

13:51 initially without having the fallback stuff. And we are working on fallback, but like in Django,

13:57 in channels today, you can do long polling. You can do the sort of frame by frame push. You can do

14:04 the fallback stuff. And so the fallback stuff isn't there mostly because I didn't want to reinvent the

14:09 wheel in that regard. I wanted to reuse one of the existing solutions like Sock.js or Socket.io,

14:14 one of the other ones that already exists in the JavaScript world. And so the initial version was,

14:18 okay, let's build initial great support for just WebSockets, just long polling, just pushing stuff.

14:25 And then later on, come and then join them together into a more sort of cohesive whole,

14:30 if you will want that stuff.

14:31 I see. So even though you support the different protocols, it's basically you're saying like,

14:36 look, if you want to use this in your web app, put some bit of JavaScript that will select it for you.

14:42 And we'll just stick with that one rather than having some really complicated negotiation.

14:47 It's partially that. And partially also what I tell a lot of people is that in some ways,

14:52 if you're just using one of the wrappers around sort of that hides all the different things it's

14:56 transporting over, you don't have as many guarantees, right? Like if you have a wrapper,

15:01 you don't know your underlying connection is actually good at bi-directional communication.

15:04 It might just be a long poll Ajax call. And those are really bad at say,

15:08 pushing data from the server to the client. So in many ways, I'm encouraging this for,

15:14 if you really need WebSockets, like a hundred percent, like, you know, you're doing

15:18 real time-ish kind of stuff or programming or games or whatever, or chat stuff,

15:22 then this is the right stuff to do. Like you've got the right level of abstraction.

15:25 If you want that sort of wrapper around stuff, sure, you can do it, but you might be better off

15:30 just building anything long pole anyway, because then you can build to a set of guarantees that you

15:35 can sort of understand. Right. That is the problem. If you let the system auto select

15:39 basically different transport protocols is you don't really know what you're going to get, right?

15:44 Yeah. And of course, WebSockets are more efficient. Like often you want to pick them

15:48 over long polling, but in terms of scoping and trying to sort of pick what to work on,

15:53 I pushed that stuff back in favor of just getting it working first.

15:56 Yeah. That makes a lot of sense. So I haven't checked can I use.com for WebSockets in a while.

16:01 What's the likelihood that people coming to my site would be able to use the best version,

16:06 the WebSockets?

16:07 So you have a deadly combination of it's mostly supported by all the new browsers.

16:11 So, you know, like your new Chrome's new Firefox's I think even Edge has support for it these days,

16:16 but you've got problems like mobile Safari, I think still doesn't support it properly.

16:19 The old Android Chrome's probably won't do it properly either.

16:22 And then on top of that, you have the problem where WebSockets isn't normal HTTP.

16:27 And so a lot of proxies and corporate firewalls just block it entirely.

16:31 And even then, if that works properly, your server and your hosting has to support HTTP 1.1

16:38 to correctly get this protocol.

16:40 So what happens is it starts off HTTP and then sort of upgrades itself midstream into a binding protocol

16:45 and just binary happens in the middle of it.

16:47 And so if you don't understand 1.1, then your HTTP server is going to go like,

16:52 what's happening?

16:53 And just like basically return 200 OK and break everything.

16:55 So it's really sort of you need to trio things to work properly.

16:59 And sometimes that can be pretty tricky to have.

17:02 Right.

17:02 And you don't always have great visibility, especially with the proxy servers or the firewalls.

17:06 Yeah, the firewalls are the worst thing because like browsers, you can test in JavaScript pretty easily.

17:10 And the server side stuff you have control over.

17:13 But sometimes those proxies or firewalls can really sort of throw a wrench in the works.

17:17 Oh, yeah, absolutely.

17:18 So we talked about WebSockets and log polling.

17:21 What are some of the other features that you support?

17:24 There's things like custom events and tasks and a number of other things as well, right?

17:28 This is what I mentioned in the beginning.

17:29 Like it's not just the support for WebSockets on top.

17:32 Like there's this whole underlying layer that's basically an event driven layer inside Django.

17:37 And it happens that on top of that, there is WebSockets and log polling.

17:41 But also you can use the same code for whatever you like.

17:44 So if you want, you can have a custom set of events or custom channels that you send things to.

17:50 You can have custom protocols, interfaces.

17:53 So right now that people working on things like a Slack interface that sort of there are channels for when you get a Slack message into your server.

18:00 So you can write bots very easily.

18:01 So you can do things like if there's like 10 people on the site, one person could send a message and you could like broadcast that out to the other people who are around.

18:10 Exactly.

18:10 Yeah.

18:11 And so like that in particular, the thing called groups and groups is a sort of built in primitive for almost like broadcasting to multiple and receiving channels.

18:20 So it's like basically channels is a couple of primitives.

18:22 Like there's a channel where you can send and receive stuff.

18:24 There's a lot of guarantees around the way it's designed.

18:27 There's groups for broadcast and then sort of this interface or API for sort of low level talking and communication between processes.

18:34 And on top of those primitives is built.

18:37 The WebSocket spec is built the HTTP spec.

18:39 And if you want to, as we're doing it at Eventbrite, in fact, you can then take that low level spec and build, say, a service oriented architecture system on top of it as well.

18:48 And that's what we're doing at Eventbrite.

18:49 Oh, very cool.

18:50 So you guys are using Django channels at Eventbrite?

18:52 Well, so in particular, the underlying layer.

18:54 So the underlying layer is rather unimaginatively called ASGI or ASCII.

18:59 And what it is, it's definitely not a whiskey replacement.

19:02 100% not.

19:03 And what the idea is, it's like ASGI is a specification for there is a thing that can pass messages around on named channels.

19:11 And you can send to named channels and you can receive from them.

19:15 And it gives you a few basic guarantees, the methods to call on objects, a bit like dbapi2 is in Python, dbapi2.

19:22 So things like, you know, there is a send method, there is a receive method.

19:27 You can do groups in certain ways.

19:30 And if you just have that, you can take things like the Redis transport we've built that Django channels runs on and use it for whatever you like and do message passing inside just pure Python.

19:41 Like you don't even need Django.

19:42 Like that part is a pure Python library.

19:45 That's really cool.

19:45 So basically, if you've got something that can act as a transport like Redis, like a Redis Q or something, then you're good?

19:51 Well, yeah.

19:52 So it's more than that.

19:53 So like what happens is like the ASGI spec gives you certain guarantees.

19:57 For example, it guarantees messages delivered at most once.

20:00 So either you get it or you don't.

20:02 The opposite choice being at least once, meaning you get it or you get two of them.

20:05 And that's a choice you have to make.

20:07 And it has a lot of those choices like, so you have a message expiry, so you get like 60 seconds.

20:12 You have back pressure, so you know if a channel is full.

20:14 Sort of all these sort of guarantees that if you're building any sort of distributed system, you want to know what these guarantees are and sort of program against them so you understand your trade-offs you're making.

20:24 And so it comes with a preset set of trade-offs and then code built against those ready.

20:30 So like there's one that uses Redis for transports using lists.

20:32 There's one that uses a local shared memory segment.

20:35 There's one that just works inside a process for like sort of unit testing.

20:39 So you have a nice lightweight one to write unit tests against.

20:41 And also we have somebody working on a RabbitMQ one right now as well to use RabbitMQ if you want to use that stuff.

20:46 Oh, yeah.

20:46 That sounds really cool.

20:47 A lot of options there.

20:49 Let me take just a moment and tell you about Metis, a sponsor of this episode.

20:53 A data scientist's responsibilities can range far and wide.

20:57 How can you be sure you have the necessary skills and training to enter the field or keep up with emerging technologies?

21:03 Metis, a data science training company based in New York City, San Francisco, Chicago, and Seattle, provides full-time immersive boot camps,

21:11 evening part-time professional development courses, online resources, and corporate programs,

21:16 all with a goal of training you to become a data scientist and help you stay on top of new and necessary skills.

21:22 Metis has a long list of professional development courses starting soon in any of these four cities taught by industry leaders.

21:30 Interested in honing your skills?

21:31 Then these courses are for you.

21:33 How do they work?

21:34 Courses on topics like data visualization, deep learning with TensorFlow, machine learning,

21:39 and statistical foundations run two nights a week for six weeks during the evening hours that fit into your busy schedule.

21:45 These courses are laser-focused on relevant topics and skills that are sure to enhance your career.

21:50 They have a special offer for you at thisismetis.com slash talkpython.

21:55 Be sure to check out what they're offering.

21:56 It helps support the show.

21:58 You talked about ASGI, and people are familiar with WSGI, Web Service Gateway Interface.

22:05 What's ASGI?

22:06 What's the A for?

22:07 Asynchronous?

22:08 Asynchronous, exactly.

22:09 As I said, it's very unimaginative.

22:10 Essentially, I had to find a name, and I stupidly chose to go with changing one letter of WSGI.

22:17 It's not even that similar to WSGI.

22:19 It does also have a way to transport HTTP over it, and that's kind of where it's semi-related.

22:24 But also, it's basically more of a specification for message passing at some point as well.

22:29 So, not the best name, but it's now stuck in everywhere, so I can't really change it.

22:34 Right, absolutely.

22:35 Well, and if you look at the WSGI interface, the API, it's really, really simple.

22:41 And there's really very few hooks for asynchronous programming in there, right?

22:45 Especially bi-directional stuff.

22:46 Exactly.

22:47 In fact, one of the things I love about WSGI so much is how simple it is.

22:51 The interface, literally, you must be a callable.

22:54 You must take two arguments.

22:56 That's the entire interface of WSGI, pretty much.

22:58 There's a specification of what the environment looks like and where you get your stuff from,

23:02 but it's very, very simplistic.

23:03 And so, certainly, that was an inspiration to me of like, well, I want something that's

23:08 similarly very simplistic that you can write against as well.

23:10 And so, that kind of came along.

23:12 But the thing, and the place where WSGI falls down, is that it has no support for asynchronous

23:18 programming at all.

23:19 Like I said, it's a callable.

23:20 It's not a yielding thing.

23:22 It's not a twisted deferred returner.

23:24 It's just a blocking callable that returns when the request is done.

23:27 All right.

23:28 That's like the exact opposite of asynchronous APIs.

23:30 Pretty much, yeah.

23:32 Interesting.

23:33 So, if I write some code here against ASGI, how interchangeable is that?

23:41 Like, what web servers kind of run around?

23:44 Like, a micro WSGI, G-Unicorn.

23:47 Like, does it plug into any of those?

23:48 Or what's the story there?

23:49 There's two separate things.

23:51 There's ASGI, which is like sort of the same.

23:53 Like, here's the thing you can send messages to.

23:54 And then there's the HTTP over ASGI specification.

23:58 That's kind of the WSGI equivalent, right?

24:00 That's like the, here's how we encode requests.

24:02 Here's where the URL variables come.

24:05 Here's how you send a chunk response.

24:06 Like, that stuff is specified as like the format of messages you send over the channels,

24:10 basically.

24:10 And because of the way it supports more than WSGI, it is, of course, not directly compatible.

24:18 And so, in particular, right now, there is one reference server, which is called Daphne.

24:23 Daphne is basically bits of Twisted and bits of Autobahn, which is the really good Python

24:29 WebTockets library, sort of hot glued together in a way that just sort of does the specification.

24:35 And people are also working on, there's been a micro whiskey plugin being worked on.

24:40 Somebody has very bravely been working on a Microsoft IIS plugin, which is really impressive.

24:46 Oh, yeah.

24:46 That's interesting.

24:47 Serious work going into that one.

24:49 But in particular, there's also the idea that it's a superset.

24:54 And so, an adapter from one to the other is not hard to write.

24:57 And in fact, there is a half written adapter already in the code base, where you can plug

25:01 in an ASGI backend to a WSGI server.

25:06 It sort of just translates between them.

25:07 But the key problem is, of course, that doesn't let you run WebSockets because WSGI doesn't

25:12 support WebSockets.

25:13 And so, that lets you sort of proxy things to servers that look like old traditional

25:20 websites, but doesn't give you all the new features that you might want.

25:22 And for those new features, things will have to be like natively supported, basically.

25:26 Yeah.

25:26 So, that's why you've got things like Daphne.

25:28 Exactly.

25:28 So, Daphne is like sort of the reference server.

25:31 And I would love to get like a proper micro whiskey support in there as well.

25:35 Like a second option, maybe Junicle.

25:37 I just sit down with these servers and like seriously look at them and see if I can like

25:42 think of like, can I patch them or like, can I change the design set?

25:46 Like just solve that story.

25:48 So, I'm not just like here, run my special web server.

25:51 It'll be fine.

25:51 It's written by just Andrew on his spare time.

25:54 It's probably okay.

25:55 And change that story.

25:57 Like there is a ecosystem of things you can pick from is the idea.

26:01 Yeah.

26:01 That's really fantastic.

26:02 And cheers to the guy doing IIS because when you're hosting on Windows or you want to host

26:07 it in somewhere like Azure or something like, that's the only choice, right?

26:10 It's not like Linux where there's a variety of things you put together.

26:13 It's just like people on Windows, they just use IIS and that's that.

26:17 So, that's really cool that it could unlock that.

26:19 Let's talk a little bit about the sort of Django patterns.

26:23 Like how is this similar or different to the stuff that people already know?

26:28 Like the idea is to sort of keep it familiar.

26:30 So, obviously, we can't make it the same as Django because like some of the things that

26:35 sort of hobble Whiskey also hobble Django.

26:38 Like Django views are the same kind of they take a request, they return a response.

26:42 You can't really do much inside them.

26:44 And so, my goal was to take things that look and feel familiar and crucially have the same

26:49 safety guarantees as normal Django.

26:50 So, you can code with them the same way but also support these extra abilities.

26:55 And so, like the main thing here is and the sort of top of the thing is things called

26:59 consumers and consumers map pretty much to Django views.

27:02 And the idea is whereas a Django view takes a request and returns a response, a channel's

27:08 consumer gets a message and then can send zero or more other messages.

27:12 So, for example, if you wanted to listen to incoming WebSocket like chat messages, the

27:19 consumer will be tied to the request channel.

27:21 And every time you send a thing in on WebSocket, it would be like, okay, I've got a message.

27:25 It would launch the consumer.

27:26 The consumer can run, handle the message, and then maybe send some messages to other clients

27:32 or sort of broadcast it, store it to a database, and then it exits straight away.

27:36 And the key thing here is that the consumer, it takes one argument, which is a message object,

27:41 which is kind of like the request object.

27:42 You can have session and auth support the same way you can in Django too with decorators.

27:46 It's like message.channel session or message.user, that kind of stuff.

27:50 And also, as long as you stick to the thing of not doing blocking options, blocking things

27:56 in the consumer, it actually makes it almost difficult, almost impossible to deadlock the

28:00 process because it's not proper async because you can't block or listen to other channels.

28:05 And because you sort of have to take the message, run through the logic, and then exit pretty

28:10 much immediately, it means that we can actually churn through messages in the same way that

28:15 Django might churn through HTTP requests with a very similar kind of set of guarantees and

28:21 safety concerns.

28:21 Well, that's really interesting.

28:22 I hadn't really thought of the views themselves actually having to be so cognizant of this async

28:31 world, but it totally makes sense.

28:32 Yeah.

28:33 And in particular, the nice thing is that views end up being just a subset of consumers

28:38 because a view takes a request, which is just a message in channels land, and it returns

28:44 a response, which means sending a message.

28:46 So it actually means that views end up being a subset of consumers and it all sort of neatly

28:50 wraps around.

28:51 Sure.

28:52 You talked a couple of times about the ability to basically deadlock yourself.

28:56 Yes.

28:58 I've seen that before in some of this WebSocket sort of callback style programming where you're

29:03 waiting and you can't receive the message because you're waiting, but you're not going

29:08 to get a response.

29:08 Can you maybe describe some of the ways in which you could like wrap yourself around in that

29:13 trouble or just sort of give somebody an idea what to look out for?

29:17 Yeah.

29:17 So the classic thing is where you have a process that sort of it gets a message and say it then

29:22 waited for a second message.

29:23 It's like, okay, I'm going to send a thing and say, I want X and then wait to get X back.

29:28 The problem is if one process says, I want X, the other process says, okay, I've got X,

29:33 I want Y to the other process.

29:35 They're both then blocked in the thing of waiting for their response, but they both can't serve

29:40 the response to the other process.

29:42 And that's a deadlock basically.

29:43 And that's very easy to get into if you don't have either an excellent knowledge of your

29:49 system architecture and asynchronous programming.

29:52 Or if you have a framework that sort of builds in or so what channels it isn't, but channels

29:57 deliberately doesn't give you full async so that it's much harder to toot yourself in

30:01 the foot.

30:02 Like we don't let you block on things because that makes it very hard to like, we said,

30:07 yeah, you can't, you can't be blocked.

30:10 You can't deadlock because you're waiting on something.

30:12 If you can't wait on something, right?

30:13 Exactly.

30:14 And if you want to, if you want to have that sort of low level control, you can just drop

30:18 down into sort of normal Python and just write your own async stuff against the ASGI interface.

30:23 But the channels level, sort of higher level one that we keep sort of safer and easier to

30:29 stomach.

30:29 So the idea is like, if you're coming into this as a web developer or maybe using as a

30:33 brand new developer, the number of scenarios where you can have not only errors, but errors

30:39 that are impossible to debug is greatly reduced.

30:41 because deadlocks not only nasty, but knowing they've happened and debugging that that's

30:45 what's happening is really difficult.

30:47 It's super difficult.

30:48 Yeah.

30:48 And I heard them described as Heisen bugs.

30:52 Yes.

30:53 Because, you know, even when you like know that they're there by observing them or interacting,

30:58 like putting logging or other stuff sometimes can change the timing, which actually changes

31:02 them.

31:03 Right.

31:03 So super hard.

31:05 One of the, one of the worst things is they often happen only under high load.

31:09 So like, imagine you have like 10 worker threads.

31:11 If you just have like, like five requests a second, it will happen perfectly fine.

31:15 As soon as you hit like over 10 requests a second, say, and you use all 10 of the threads,

31:20 that could then trigger the deadlock because all the thread, all the work is suddenly consumed

31:24 up.

31:25 There's no spare workers to handle these extra requests.

31:27 And so like, it might be fine in development and you need to point to production.

31:30 It's fine in production.

31:31 And then one Sunday evening when your website's under high loads, suddenly it locks up and there's

31:36 no traceback.

31:37 Like what's happening?

31:37 I don't understand.

31:38 It's like, it's a really nasty sign of error.

31:40 Yeah.

31:41 It's super bad.

31:42 And I think it's awesome that your framework like makes that harder.

31:44 That usually that's the kind of stuff that happens right when you need the website to

31:50 work most.

31:50 We're running a new ad or we got featured on Hacker News or whatever.

31:53 Boom.

31:54 It's dead.

31:55 Yeah.

31:56 And it's, it's not perfect, obviously.

31:57 Like we can't stop everything, but like the Jenga philosophy is try and make these things

32:01 hard, but a fault.

32:02 And so that's kind of tried to be reflected here.

32:04 Yeah.

32:04 That's absolutely a good philosophy.

32:06 So some other design patterns around Jenga that are maybe related like routing and messaging.

32:11 Yeah.

32:12 So in the other things that's familiar.

32:14 So routing.

32:15 So the idea is when you say, I have a consumer of this channel, you have to sort of say like

32:20 a view.

32:20 You have to tie that consumer to the channel.

32:23 So like, well, this consumer handles incoming messages on the WebSocket.receive channel.

32:29 And it handles the ones where the path is slash chat.

32:34 And so there's a thing that looks very like Jenga's URL routing, but is channel routing instead.

32:40 It's like, it's not quite the same, but very familiar.

32:43 It's like a list of regular expressions basically.

32:45 And it has, again, very similar guarantees and a lot of power in there.

32:49 And then, as I said before, messages are very much like requests and that like they have

32:53 information on them.

32:54 You can store stuff on them.

32:56 There's session support in there.

32:57 It's like, it all feels a bit like you're writing Jenga views and URLs, but it's a bit more powerful

33:03 with a bit extra, with a few extra options.

33:05 Okay.

33:05 Yeah.

33:06 Very cool.

33:06 And what's the programming model look like?

33:09 Like how is it the same as Jenga or how is it different?

33:12 So to the end developer, it looks largely the same as normal Jenga.

33:16 In particular, channels in its default state runs all of your code, the writer's consumers

33:22 and stuff, synchronously.

33:23 Like you don't ever run into an async option.

33:27 And particularly we run the whole of Jenga synchronously too, because like we can't make

33:30 Jenga async.

33:31 So we just run Jenga synchronously with the sort of, the way design helps us run it in sort

33:35 of a worker thread.

33:36 And all the async is handled away from Jenga, basically.

33:40 So if you're coming in from that end, it looks pretty much like normal Jenga.

33:42 However, if you want to do sort of more advanced stuff or handle different protocols or even

33:47 do your own async programming, it sort of drops down and becomes more of a pure Python thing

33:53 where you can use whatever framework you like that you can use async.io, you can use twisted,

33:57 you can use other things.

33:59 And then you just call the ASGI API, send and receive stuff as you need to.

34:05 So for example, Daphne is a purely asynchronous twisted program that happens to call ASGI in

34:11 the right way inside of it.

34:12 So like you sort of get the choice of, do you want the familiar happy world of Jenga without

34:16 much power?

34:17 Or do you want the raw unbridled power of async?

34:20 But you sort of need to be prepared to go into that particular world.

34:24 Yeah, you're going down the rabbit hole there.

34:26 Interesting.

34:26 Exactly.

34:27 So is this, if I'm doing the async variant, is this still in the same process as Jenga

34:32 or is this like a totally different thing?

34:35 So it's a different process.

34:36 In particular, like one of the things when I sat down, part of that initial period when

34:40 I was sort of like trying to design channels and go through the different options, I had

34:43 some things that were threading, some things were using the multi-process module, some things

34:48 were using green threads or like yielding like generators.

34:52 And eventually I realized that if you're building a large system, as you know, as I do at work

34:59 on a daily basis, you're always going to have more than one server, which means you're always

35:03 going to have to have more than one process.

35:04 And so given that, if I made a model that was just natively multi-processed from the get-go,

35:10 that would make a lot of sense in like, not just multi-process, multi-server.

35:13 Like let's make the model that is by default, it expects to run everything on like a different

35:17 CPU entirely with no shared memory or no shared state and work backwards from there.

35:22 And so everything is basically done in a separate process and message passing happens

35:27 between them.

35:27 And so that's how things are coordinated.

35:29 So it's a bit like, say like if you've ever used Go, for example, Go, a lot of Go primitives

35:35 rely heavily on passing things between Go threads in a thing they call channels as well, which

35:40 is similar but different to channels.

35:41 Yeah, I was going to ask you that about the relationship.

35:44 The name is the same.

35:45 The idea is kind of similar in some ways.

35:47 Like, was there any inspiration or similarity?

35:51 Is it just random similarity or is there more to it with Go and their channels?

35:56 Oh, no.

35:56 So the direct similarity that they both share a common ancestor, which is the language called

36:00 CSP.

36:01 So when I went to university in Oxford, they invented CSPs.

36:05 They definitely teach it and they're very proud of it.

36:06 It's a language for basically proving asynchronous programs.

36:10 And so it's sort of a very abstract grammar of like, oh, we have a process that emits event

36:15 X onto channel Y and then a process that consumes channel Y and then gets event X and then does

36:20 Y.

36:20 And in its core, it's kind of this way of sort of like you can write a set of assumptions

36:25 and prove that programs don't deadlock, which is an incredibly nice way of doing things.

36:28 But it also has these fundamental ideas of processes and channels and splits.

36:32 And so that's where my ideas came from.

36:35 And then sort of via Go, I think the name channels came from Go as well, but like a different take

36:40 on that same core concept of these are communicating sequential processes, which is what CSP stands

36:46 for.

36:47 Yeah, this whole actor model is quite powerful for avoiding deadlocks and race conditions

36:53 and, you know, writing to the same piece of memory at the same time.

36:56 And a lot of the threading problems go away, right?

36:58 And not just that, but also like understanding as a developer, like reasoning about systems

37:02 is much easier when they are very separate components you can read about individually.

37:06 Like if it's all integrated into one big lump, then trying to work out what happening is pretty

37:11 difficult.

37:11 If they're separated out, it's like, oh, here is Daphne.

37:14 Daphne just translates HTTP requests to messages and messages to HTTP responses.

37:19 Writing tests for that is a lot easier because we just write tests where we put one thing

37:24 in one end, the other thing comes up the other end and vice versa.

37:26 Yeah, you have very clear, explicit boundaries.

37:28 These are the messages.

37:29 Here's where the messages come in.

37:31 Here's where they go out.

37:31 And basically, this is what they consist of, right?

37:34 Yeah.

37:34 And then that scales up too.

37:35 Like if you're doing engineering at scale, as in with a large team, those boundaries then

37:39 become team boundaries.

37:40 That really helps you sort of distribute not only the testing, but also like the cognitive

37:46 workload of understanding what the system is doing at any one time that you can draw the

37:49 same boundaries and have teams work on shared understanding in a much better way as well.

37:53 Yeah.

37:54 That's a really good point to think about how it lets people scale, not just software.

37:57 Oh, yeah.

37:58 Scaling.

37:59 I'm very proud these days that scaling software development is about half code scaling, about

38:04 half people scaling.

38:06 Like it's very important to get your team all understanding each other and not being at each

38:10 other's throats all the time.

38:11 And a big part of that is good specs and good APIs.

38:13 Yeah, that's awesome.

38:25 This portion of Talk Python To Me is brought to you by Hired.

38:28 Hired is the platform for top Python developer jobs.

38:31 Create your profile and instantly get access to 3,500 companies who will work to compete

38:35 with you.

38:36 Take it from one of Hired's users who recently got a job and said, I had my first offer on

38:40 Thursday after going live on Monday and I ended up getting eight offers in total.

38:44 I've worked with recruiters in the past, but they've always been pretty hit and miss.

38:47 I tried LinkedIn, but I found Hired to be the best.

38:50 I really liked knowing the salary up front.

38:52 Privacy was also a huge seller for me.

38:55 Sounds awesome, doesn't it?

38:56 Well, wait until you hear about the sign-in bonus.

38:58 Everyone who accepts a job from Hired gets a thousand dollars signing bonus.

39:02 And as Talk Python listeners, it gets way sweeter.

39:04 Use the link Hired.com slash Talk Python To Me and Hired will double the signing bonus to

39:09 $2,000.

39:10 Opportunity's knocking.

39:11 Visit Hired.com slash Talk Python To Me and answer the door.

39:22 So if I'm going to run Django app using channels, how's this work?

39:27 Can I run it the same?

39:28 Probably Daphne is involved somewhere, right?

39:31 Basic level, you run two processes.

39:33 You run Daphne.

39:34 And what Daphne does is Daphne terminates HTTP and WebSockets for you.

39:38 So that's what you expose to the world on port 80 or whatever, or probably behind Nginx doing

39:43 static file serving or something.

39:45 And that sort of fills the role that, say, like G Unicorn might fill.

39:47 Like, here's a process that terminates HTTP.

39:49 And then separately, you run one or more Django worker processes.

39:54 And those processes look at the channel queues that are in this sort of channel routing we

39:59 mentioned earlier and sort of just run the consumers on those channels as messages come

40:04 through.

40:04 And so at a base level, you run one Daphne and you run one worker.

40:08 And then it just sort of works as normal Django does.

40:11 And in particular, I see.

40:12 So the request comes in and if it's like a channel request for, say, WebSockets, it just

40:18 stops at Daphne and gets processed there.

40:21 But if it's something that matches for Django view or something, it will like pass through

40:26 to the Django worker process?

40:28 No.

40:28 So in fact, Daphne does both of them.

40:30 So like both HTTP and WebSockets both get turned into messages on channels.

40:35 So that's kind of the difference to a lot of those previous solutions I talked about near

40:39 the start of the show is that channels has a sort of slightly ambitious goal maybe of putting

40:45 everything over channels.

40:46 This includes HTTP as well.

40:48 So what happens is if you have a HTTP request that comes into Daphne, Daphne takes it, it decodes

40:53 it into the ASGI HTTP format, pass that to Django.

40:58 And then Django has a native ASGI handler next to the WHGI handler that takes that, turns it

41:04 directly into a request object and then runs the views on that stuff too.

41:06 Okay.

41:07 Yeah, that's very cool.

41:08 So it is the same system as the channels running over basically.

41:11 Okay.

41:12 Excellent.

41:12 And you still have things like run server and so on?

41:16 Right.

41:16 So run server, like as for development, just you type run, like if you install channels

41:20 or type run server, it just runs Daphne and worker for you in the same process.

41:24 It's no sort of fitting around there.

41:25 It's just easy to get going.

41:27 And it means you basically, you can pip install channels, put it in your installed apps, and

41:31 then WebSockets just work on your development server straight away, which is really nice

41:34 to have for sort of hacking around.

41:36 And then sort of the other thing you can have, and I sort of recommend, especially to people

41:40 who are wary or like people who are like died in the wall ops people, should we say, is

41:45 you can run both WSGI and channels at the same time and just route WebSockets to Daphne

41:51 and then route normal requests to say MicroWSGI or G-Unicorn.

41:55 And so that way you can use sort of more proven software for your main site, but still use

42:00 channels for your other stuff if you want that sort of that trade-off.

42:03 Oh, that's pretty interesting.

42:04 And you're thinking put something like Nginx in front of it and switch on like protocol

42:08 or something.

42:09 Yeah.

42:09 So you can switch these things, switch on as URL paths, like things on us like slash WS

42:14 WebSockets, but you can, with a couple of tricks, switch on the upgrade header as well,

42:18 which is how you upgrade WebSockets.

42:20 So like there are ways and means.

42:21 It's tricky to switch dynamically based on what's coming in.

42:26 Like you can do it.

42:27 It's very hard to.

42:28 So generally I recommend like sort of a path-based solution.

42:31 And like there's examples in the channels docs and mailing lists about like how that might

42:35 run as well and sort of examples how to set that stuff up.

42:37 All right.

42:38 So what's the extensibility story here?

42:39 So channels is in fact, it's very much built around extensibility.

42:42 Like as I said before, like it's not that much in the base package.

42:46 You sort of WebSocket supports there and that kind of stuff.

42:48 But like it's designed to be built upon in sort of a community way like Django's itself.

42:53 So things, for example, like the ability to have proper heart beating of people connected.

42:59 So, you know, they're definitely still connected.

43:00 Like that is a thing that you could put on top of channels.

43:03 And so what I'm trying to do is like foster a community of third-party apps that are built

43:07 against channels and the common specifications.

43:09 In many ways, that's why I took it upon myself to build a sort of more official, it's not the

43:16 right, but to build a singular solution that we can sort of almost tout as Django solution

43:19 for this stuff.

43:20 So that there is a common thing that people can build against.

43:23 Because like a lot of the part of Django success is having a common standard people write against.

43:29 Like it's a common known factor.

43:30 And so really it's designed around that idea of like, yeah, everyone's going to have the

43:34 same routing.

43:35 Everyone's going to have the same idea of consumers.

43:37 Like things integrate the same kind of way and like trying to encourage that kind of ecosystem

43:42 as it were.

43:43 Yeah, absolutely.

43:44 I, you know, that's definitely a hallmark of Django is all the pieces you can bring

43:48 together to make it more than what it is out of the box.

43:51 Exactly.

43:51 Nice.

43:52 Are there like some notable third-party packages or projects that you want to talk about?

43:57 Not, not, I can talk to my head.

43:58 Like I don't know them off by heart.

44:00 We have, we have a page on the docs that links to a couple of ones that are already going.

44:03 Okay.

44:04 Awesome.

44:04 There's also a few more that aren't on there yet.

44:07 I encourage the developers to add links to the doc.

44:10 Okay.

44:11 So let's see.

44:11 We talked about Daphne a little bit.

44:13 Tell me some of the packages that you're using to put this together.

44:16 Yeah.

44:17 So like one of the things was like trying like Django for a long time is criticized as being,

44:21 oh, Django is one giant package.

44:23 It's not split in separate bits.

44:24 So I went the entire opposite way.

44:26 There are five different packages as part of channels.

44:28 So you have the base channels package, which is the, basically the Django plugin side of

44:34 things as Django third-party app or second-party app, I guess, or first-party app.

44:38 But it's a Django pluggable app that you load into Django.

44:41 It does all the routing and run server overloading we talked about before and gives you sort of

44:44 the nice user experience.

44:46 But it doesn't give you the serving.

44:47 And so Daphne is the other half of that.

44:49 Daphne is the service.

44:51 So Daphne is the server that terminates HTTP and WebSockets and sort of translates them through

44:57 into channels for you.

44:57 If there was something else there, you could just use channels and a different thing that

45:01 wasn't Daphne.

45:02 But for now, we bundle them together pretty much.

45:04 And then underneath that, there is the ASGI packages.

45:07 So the things that build this sort of low-level protocol we build upon.

45:10 So there's ASGI ref, which is sort of the base ASGI library.

45:15 It has the conformance test suite in it.

45:16 So you can, like, if you're writing a new ASGI backend, you just load up the conformance

45:20 tests and it just tests your spec, which is really nice to have for new implementations.

45:24 And it has a memory backend too for testing, like, in-memory for unit tests.

45:27 And on top of that, you have two different backends.

45:30 The Redis backend called ASGI underscore Redis.

45:32 And that is a standard Redis backend, the one we recommend for production use.

45:36 It uses Redis lists and blocking pops and Lua scripts and a few other things in there.

45:42 And it has built-in sharding if you want to use more than one server to scale up.

45:45 And then as sort of a midway point, there's a ASGI underscore IPC backend that only works

45:51 on one server.

45:52 And it's designed for, if you don't want to run a separate Redis server as well, you

45:56 just want to run things on one machine, keep it simple.

45:58 It doesn't perform nearly as well, but it is zero configuration.

46:02 You just basically just load it up as the option backend, just set the same prefix and things

46:07 just talk to each other on the same machine via a shared memory segment.

46:10 Oh yeah, that's really nice.

46:11 Really nice.

46:11 So what's the story with Python 2 versus Python 3 on this project?

46:16 So everything is built Python 3 first.

46:19 And in fact, Django itself is going Python 3 only very soon.

46:22 as of the next major release, I believe.

46:24 That's in Django 2.

46:26 And so it's built with Python 3 first, but also Python 2 support is still there.

46:31 So channel supports the previous LTS of Django and up.

46:35 So that's 1.8 and up.

46:36 And 1.8 supports Python 2.7.

46:38 So it supports Python 2.7 and Python 3.

46:41 All my testing and all my developments done on Python 3.

46:43 And then we have Travis just running Python 2 tests occasionally.

46:47 And I'll just occasionally pop down to Python 2 and make sure things work expected.

46:50 But I've seen like one bug that was Python 2 specific in the entire time I've been developing

46:56 so far.

46:56 So it's not really a concern at all in that regard.

47:00 Oh, that's really fantastic.

47:01 And I'd love to see you doing this in Python 3 first.

47:04 You know, I talked to the guys at the Beware project, and they're doing Python 3 only.

47:09 I'm surprised to hear about Django going Python 3, although excited to hear it.

47:14 I think Python 3 is really, it's crossed some kind of threshold or critical mass recently,

47:20 I feel.

47:21 Yeah, like, I feel the same way too.

47:22 Like, in the last year or year and a half or so, we definitely crossed that threshold of,

47:26 like, things are coming out in Python 3 only, a lot of cases.

47:30 We're finding libraries at work now that were like, oh, this great accounting library is only

47:35 on Python 3.

47:36 And then suddenly, like, we need to move code to Python 3.

47:38 It's become almost this trend now.

47:40 And, like, it's difficult for a big company like Eventbrite to move to Python 3, of course.

47:44 But definitely, a sea change is definitely happening these days.

47:48 And everything I write from scratch is done on Python 3.

47:51 Yeah, that's fantastic.

47:52 I feel like I'm noticing it in that people are apologizing for it being Python 2 in the

47:59 cases where it is, or they feel guilty or bad about it or whatever.

48:03 Whereas a few years ago, it was kind of like, yeah, it's Python 2, of course.

48:06 Like, haha, Python 3, whatever, right?

48:08 So this is really cool.

48:09 Tell me a little bit about Django going Python 3, if you know any of the backstory.

48:14 What's going on there?

48:14 So we decided quite a while ago to draw a line in the sand, as it were.

48:18 So, like, you know, at some point, we need to do Python 3 only.

48:21 In many ways, because Django's job is, in some ways, as a leader in this area.

48:27 Like, if Django is Python 3 only, that's a big signal to a lot of other Python packages that,

48:30 yes, you can do this too.

48:32 It's a huge signal.

48:32 It's a huge signal.

48:33 And so the line was decided.

48:35 And, like, we debated for a long while.

48:37 I'd be like, well, like, what version of Django should we do it on?

48:41 Like, when should we do it?

48:42 And so different core developers were arguing at different times for, like, oh, well, can

48:47 we do it now?

48:48 So I think, like, two or three years ago, people would say, like, oh, let's just do it now.

48:51 Let's just switch straight away.

48:52 And then eventually we decided that Django 2.0 is the first release to only support Python

48:59 3.

48:59 And in particular, Python 3.5 and above.

49:01 Because the reasoning being that Python 3 is small enough that people are pretty much

49:05 on the newest version of Python 3.

49:06 There's not many differences that start upgrading to Python 3.5.

49:09 And so what this means to Django in particular is we can remove swathes of compatibility code.

49:15 Like, there's so much in there.

49:16 Some of the core devs are literally chomping at the bit to just delete huge reams of code

49:21 that are just Python 2.0 specific.

49:23 And in particular, like, we can use things like all the new features.

49:26 We can get some of the class inheritance stuff and get done properly.

49:28 Like, all the sort of stuff that we had to sort of keep going with Python 2 and keep it

49:32 there is all being swept away.

49:34 And the nice thing, too, is that because it's Django 2.0, we can say, okay, this is a nice

49:39 big change.

49:40 And in fact, like, the Django number system is changing entirely.

49:43 So for reference, what's happening is we're making it so that Django's releases are...

49:48 There's every major LTS will be the last number in its major number.

49:53 So like 1.11 will be the last one.

49:55 Then we'll have 2.0, 2.1, 2.2.

49:57 Then 2.3 is the next LTS.

49:59 And 3.0, 3.1, 3.2.

50:00 And 3.3, the LTS.

50:01 And that kind of strategy.

50:03 And it just happens that it's kind of timed nicely that Python 3.5 support happens in

50:08 Django 2.

50:09 So it's a good reason to call it Django 2, I think.

50:12 We were worried for a while in Django Core that there wouldn't be a good thing in Django

50:15 2.

50:15 We'd be like, oh, and this is just another maintenance release of Django with some bug

50:18 fixes.

50:19 But it's nice to have a bit of a sort of big change in there as well.

50:21 Yeah, that's really cool.

50:23 And I think changing the major version number sends a strong signal like this thing is different.

50:28 Exactly.

50:28 And it's easy to pull.

50:30 I've been doing for the last three or so years now, pausing code to Python 3, really not that

50:37 hard.

50:37 Once you get your head around what's going on, and if your encodings are already good.

50:42 It's like Python 3, and I like this about Python 3, really punishes you for not having good

50:47 encodings, like knowing where your bytes come from.

50:49 Which for developing channels, which is all about handling bytes over the wire versus encoded

50:54 messages or URLs, is incredibly useful.

50:57 Like, oh, well, you've tried to pass this byte string to the Unicode string function.

51:01 It just blows up.

51:01 It's like having, as long as that's all in order and you know where your strings are coming

51:05 from, it's pretty painless to port in my experience.

51:08 Yeah, that's really excellent.

51:09 And it is sort of that network layer that you see some of the differences between Python

51:15 2 and Python 3 more when you're down at the string level and taking bytes and converting

51:19 them.

51:20 But yeah, it sounds really good.

51:21 Yeah, it's really good.

51:23 Like, my main complaint is that I keep typing print with a space rather than print with a

51:26 parenthesis.

51:26 Like, muscle memory is a really hard thing to undo.

51:29 Yeah, absolutely.

51:30 So when we talked about WSGI and we talked about AWSGI, the asynchronous thing, what came

51:40 to mind for me was this concept of WSGI 2, which is being sort of brought into existence by

51:48 the pressure that HTTP 2 is putting on the WSGI specification, right?

51:52 Yeah.

51:53 Other pressures too.

51:54 WebSockets is part of those pressures as well.

51:55 Yeah, absolutely.

51:56 Well, I was, you're right.

51:58 WebSockets predates HTTP 2.

51:59 But all this sort of, we want to run more than one thing over this channel and we want

52:03 to sometimes do it in binary, like just doesn't work at all.

52:07 So can you talk a little bit about like WSGI 2 and maybe how that like would either help

52:13 or is parallel to what you're doing?

52:16 Yeah.

52:16 So like I actually involved in discussions for a while.

52:18 So there's a group in Python called WebSig and WebSig is sort of the place where things

52:23 about WSGI get discussed.

52:24 And there's been on and off for like a few years now, at least, I came in pretty late to

52:29 discussion, this idea of WSGI 2.

52:32 And one of the main problems is that if you want to call it WSGI, it has to look a lot

52:38 like WSGI and in theory, backwards compatible as well.

52:42 And that really hinders the approach.

52:44 And like, as of the time of recording, there's been very little progress on WSGI 2 for like

52:49 about two years, if not longer.

52:51 Lots of different ideas about what should be done or different approaches, like how do we

52:55 even make things async?

52:56 And so my conclusion to that was that like Django needs something, we needed something

53:04 that existed and worked.

53:06 And I read through all the threads and tried to educate myself on the arguments and the

53:10 reasons behind different people's visions and wants for WSGI 2 and took some of those

53:15 with me to the design of ASGI and in particular the HTTP layer on top of it.

53:20 But at the same time, I don't want to stride in and declare a replacement for the WSGI

53:26 with ASGI, rather it's a different thing.

53:28 I also think that a different thing is the right solution.

53:31 I think that Python web systems are more than just handling HTTP responses these days.

53:39 And so my ultimate goal is probably to try and sit down, get Django and get more implementations

53:46 of this going through.

53:47 And then when we have a good set of reference implementation, like two references on either

53:51 end, it's proven in production, then take that proven thing back to WebSIG and say, hey,

53:58 can we decide on this thing that's already in place that has proof behind it?

54:02 And then we can see what's happening with it rather than trying to argue in the abstract,

54:05 basically.

54:06 Yeah, I think arguing in the abstract is really hard to do.

54:10 It's very difficult.

54:11 Yeah, you can just go around and around and around.

54:14 And I, you know, I don't know exactly what's happening internally at WebSIG, but if it's

54:19 been kind of idle for two years and HTTP2 is coming, the web is not standing still.

54:23 So I feel like, you know, maybe there's some of that going on.

54:27 I am perfectly willing.

54:28 If someone comes up tomorrow with an amazing WSGI 2 spec that everyone adopts, I will immediately

54:34 drop and support that stuff.

54:35 But like, given the lack of movement, I felt that someone has to do something.

54:40 And so I decided to be that someone and do something.

54:42 Yeah, this is really cool.

54:43 I mean, if you get this to be successful and proven, like you said, it's really easy to

54:48 take it back and go, look, this is more or less working.

54:52 Let's make a spec.

54:53 Let's extract a spec out of this general idea rather than debate at an infinite item.

54:58 Exactly.

54:59 I've been trying to keep conversations going with like various authors or luminaries of

55:03 Python web stuff.

55:04 Like not enough, but like getting opinions and feedback on like the way things are done.

55:07 It's like silly things like cleaning up the, like I said, encoding, cleaning up the encoding

55:12 of how strings are passed around in ASGI.

55:14 Like ASGI has an exact specification of which bits of the request come through as Unicode

55:20 and which bits come through as bytes, which WSGI does not have and can kind of never have.

55:24 So like I've been trying to keep a lot of those things in mind.

55:27 And I don't know if it'd be successful in the end, but I'm hoping that like pragmatism is worth

55:33 a lot here.

55:33 And even if the message passing sort of asynchronous bit doesn't survive, the basic format of here's

55:41 how you take a request and put it into a message dict would still be useful.

55:44 Yeah.

55:45 I think it's really positive that you're working on this and hopefully, hopefully that can,

55:50 you know, fall this, this blockade or whatever, whatever is going on there.

55:55 Some point like doing open source collaboration is really difficult.

55:58 Like it takes a lot of effort.

55:59 So like it takes not just me, but other people.

56:03 So hopefully we can get something, something out.

56:04 Yeah.

56:05 Hopefully.

56:05 I mean, WSGI itself has been so important and so successful that it's just really scary

56:11 and hard to change.

56:12 So I think that's, it's almost a victim of its own success in that regard.

56:16 Yeah.

56:16 WSGI is like right next to CGI and like how pervasive and successful it's been.

56:22 Like every like Python web program runs on it.

56:26 It's incredibly, it's incredibly simple.

56:28 It's very powerful and the pluggability of different servers is like a huge boon in its

56:33 deployment, right?

56:34 Like deployment in Python is never as easy as PHP, but it's pretty damn close considering

56:39 like we have all these different options.

56:40 So that's really something to aspire to.

56:43 I think like WSGI is an amazing specification that I am always in awe of, like every time.

56:49 Like I'm very pleased that the original authors like decided to settle on it like so long ago

56:55 as well.

56:56 Yeah, absolutely.

56:56 It's been going like 15 years or something and it's still going strong.

57:00 It just, it's starting to show its age.

57:03 That world was much less about serving multiple things and bi-directional stuff over the same

57:09 channel and all sorts of things like that.

57:10 Right.

57:11 To survive 15 years in software is an incredibly long time.

57:14 Like when Django hit 10, I was feeling, I was already feeling like a bit old.

57:18 Right.

57:19 So yeah, like it's, it's really impressive.

57:21 Yeah, absolutely.

57:22 Okay.

57:23 So that's, that's really great.

57:24 So maybe, maybe one final question is how's all this useful beyond Django?

57:30 Like it's, it's cool that we can plug it into Django.

57:32 What else can we do with it?

57:33 So like really my goal, my ambitious goal at the end of all of this, I think is to make

57:39 writing distributed systems in Python easier.

57:42 And maybe not even in Python, maybe in general, but like, you know, I am, as like my career

57:47 goes on, I find myself writing software, distributed systems are the one challenge I keep both enjoying

57:53 and the challenges me time and time again.

57:54 And like, I would love to have this idea of message passing, this idea of channels and passing

58:00 messages and these, these primitives be the way that Python programs in general, like not

58:05 just HTTP, but like chatbots and like internet of things systems and like stats systems, all

58:12 these things have this common base construct to build upon and talk to each other and message

58:16 pass.

58:16 And like, I think that's the ultimate goal beyond Django.

58:19 There's certainly a long way to get there.

58:21 The middle goal is to have other Python web stuff work with it as well.

58:25 But I have like, you know, and I think it's nice to have somewhat grand ambitions for improving

58:31 Python in general.

58:33 Like the, like, I like Python, the language a lot.

58:36 Like I'm not just a Django person, right?

58:38 Like Python is my real home in many ways.

58:40 So I want to make sure that everyone can benefit from this stuff and to really push the state

58:44 of the art forward in that respect.

58:45 Yeah.

58:45 That's, that's really cool that it's, it's got that sort of broader goal.

58:49 You know, you're talking a lot about moving to Python three and this is Python three first.

58:54 Do you see a world where you could use this idea of channels and this as the infrastructure

58:59 for those channels mixed in with like async and await to be, to make Python much more go like?

59:06 That's kind of one of the ideas.

59:08 Yeah.

59:08 It's like, it's like channels in the spec, ASGI in the spec rather, has provisions for a

59:13 async code compatible receive command.

59:15 So you can do, you can await on receive and how it blocked correctly and do all that kind

59:19 of stuff.

59:19 So the idea is really that you can, and moreover, like it's not just that you can write async code,

59:25 it's that you can also write synchronous code and have that intermixed.

59:29 Like, cause I'm a strong proponent of if the code you can write can be synchronous or

59:34 at least can be written in a synchronous fashion, that's fine.

59:37 As long as it interoperates properly, like synchronous code is easier to reason about.

59:40 It's easier to test in many cases.

59:42 So if we can have a system that supports both async code natively and synchronous code natively

59:48 and they interact properly and you can plug and replace different parts of the system correctly

59:52 and like, oh, well, we've written this part of the system, like, you know, this waiting

59:56 room system here is in ASUNKO, whereas like this emailing system here is in twisted and

01:00:03 this part of the system over here is in synchronous code.

01:00:05 I think that's also a good goal to aim for, like flexibility is really important.

01:00:08 Yeah, absolutely.

01:00:09 All right.

01:00:10 That sounds, that sounds like a cool feature.

01:00:12 I'm looking forward to seeing it.

01:00:13 Let's hope we get there.

01:00:14 For sure.

01:00:15 All right.

01:00:16 So I think maybe we'll just leave it here.

01:00:18 This is a really cool project that you put together and I'm excited to see it gain

01:00:22 attraction.

01:00:22 Thank you.

01:00:23 Yeah, you bet.

01:00:24 So before I let you out of here, let me ask you two questions I always ask at the end of

01:00:28 the show.

01:00:29 First of all, if you can write some code, Python code, other code, what editor do you open

01:00:33 up?

01:00:33 I'm a Sublime Text user.

01:00:34 I have been for like four or five years now.

01:00:36 It's just, it's got Python in it too.

01:00:38 So you can't argue with that really, can you?

01:00:39 And now it's definitely the home for Python there.

01:00:42 Have you checked out the Anaconda plugin for it?

01:00:45 I haven't, no.

01:00:46 Okay.

01:00:46 I have my own sort of set of plugins and some custom plugins I've collected over the years

01:00:51 that sort of, they sort of work together mostly.

01:00:53 It's pretty decent.

01:00:54 Nice.

01:00:55 All right.

01:00:55 And there's 96,000 packages on PyPI.

01:00:59 There's a ton that you've probably come across that people haven't heard of necessarily or you'd

01:01:04 like to recommend.

01:01:04 What's on your mind?

01:01:05 And the one I've used most recently, I was really impressed with it is Cryptography, which

01:01:09 is actually quite new given the name is quite generic.

01:01:11 It's a really well done crypto package that is designed in a way where it's very hard

01:01:16 to shoot yourself in the foot.

01:01:17 As you know, I'm a big believer in this.

01:01:19 It comes with these amazing, fully featured primitives for, oh, here is a way to do symmetrical

01:01:25 mesh encryption.

01:01:25 Here is a way to do asymmetrical public private key encryption.

01:01:28 And it's just like, there's safe defaults.

01:01:30 It's easy to use.

01:01:31 And as long as you follow the docs, you end up with a really nice solution that is compatible

01:01:37 properly.

01:01:37 And like, it's pretty quick.

01:01:38 And like, that's what we use inside ASCII reddit if you want at rest encryption.

01:01:42 So like cryptography is my current favorite PyPI package.

01:01:45 Oh, that's really cool.

01:01:46 I definitely think cryptography and hashing and password management, that stuff is better

01:01:52 put into a library or package and just really baked in with the best practices and you just

01:01:57 follow the steps.

01:01:58 Yeah.

01:01:58 I do not want to have to think about it.

01:01:59 Like I want somebody else who's more intelligent than me to solve the problem so I can use their

01:02:03 code.

01:02:04 It does that really well.

01:02:05 Yeah, that's great.

01:02:05 I'm going to have to check that out.

01:02:06 All right.

01:02:07 So before we get out of here, final call to action, how do people get started with channels?

01:02:11 Like what kind of support or help are you looking for?

01:02:15 So yeah, so like I'd encourage anyone who's interested to go over to channels.readthedocs.io.

01:02:19 It's sort of the main channels documentation.

01:02:21 You can see installation instructions, a short tutorial stuff there.

01:02:25 And there's also a big list of stuff we want to work on.

01:02:27 So our GitHub repository, which is github.com/Django slash channels.

01:02:31 There's a whole list of issues there filtered by experience level.

01:02:34 You want to have that kind of stuff.

01:02:35 You can email Django developers or Django users with questions about channels.

01:02:39 It's all part of the Django project.

01:02:40 And then if you want to do work on a big project, we even have some funding available for bigger

01:02:44 projects.

01:02:45 We can dish out for sort of things that are, you know, at least a couple of weeks, sort

01:02:49 of bigger tasks that wouldn't be tackled otherwise in spare time.

01:02:52 So if you're interested in any of those things, like get in touch, mainly this or get in touch

01:02:57 with me directly.

01:02:57 I'm Andrew Goldbin on Twitter.

01:02:59 Just all that stuff.

01:03:00 I'm happy to talk about channels.

01:03:01 Oh, that's really great.

01:03:03 So lots of ways people can get involved.

01:03:05 So many.

01:03:05 Yeah.

01:03:06 Cool.

01:03:07 Andrew, thanks so much for taking the time to talk with me and sharing your project.

01:03:10 Thank you so much for having me on.

01:03:11 It's been a pleasure.

01:03:12 Yep.

01:03:12 Bye.

01:03:14 This has been another episode of Talk Python To Me.

01:03:17 Today's guest has been Andrew Godin, and this episode has been sponsored by Metis and Hired.

01:03:22 Thank them both for supporting the show.

01:03:24 Want to learn data science?

01:03:26 Well, don't forget to visit thisismetis.com slash talkpython to learn more about their upcoming

01:03:32 courses.

01:03:33 Get the skills that you need to succeed in the fast-paced world of data science.

01:03:37 Hired wants to help you find your next big thing.

01:03:40 Visit Hired.com slash Talk Python To Me to get five or more offers with salary and equity

01:03:45 presented right up front and a special listener signing bonus of $2,000.

01:03:49 Are you or a colleague trying to learn Python?

01:03:51 Have you tried books and videos that just left you bored by covering topics point by point?

01:03:56 Well, check out my online course, Python Jumpstart by building 10 apps at talkpython.fm slash

01:04:02 course to experience a more engaging way to learn Python.

01:04:05 And if you're looking for something a little more advanced, try my Write Pythonic Code course

01:04:09 at talkpython.fm/pythonic.

01:04:12 Be sure to subscribe to the show.

01:04:14 Open your favorite podcatcher and search for Python.

01:04:17 We should be right at the top.

01:04:18 You can also find the iTunes feed at /itunes, Google Play feed at /play, and

01:04:24 direct RSS feed at /rss on talkpython.fm.

01:04:27 Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.

01:04:32 Corey just recently started selling his tracks on iTunes, so I recommend you check it out at

01:04:37 talkpython.fm/music.

01:04:39 You can browse his tracks he has for sale on iTunes and listen to the full-length version

01:04:43 of the theme song.

01:04:44 This is your host, Michael Kennedy.

01:04:46 Thanks so much for listening.

01:04:48 I really appreciate it.

01:04:49 Smix, let's get out of here.

01:04:52 Staying with my voice.

01:04:53 There's no norm that I can fill within.

01:04:55 Haven't been sleeping.

01:04:56 I've been using lots of rest.

01:04:58 I'll pass the mic back to who rocked it best.

01:05:01 I'll pass the mic.

01:05:12 Bye.

01:05:13 .

01:05:13 you