#153: How Python Evolves Transcript

00:00 If you spent some time in the Python community, you've probably heard the term PEP, which stands

00:04 for Python Enhancement Proposal. In fact, the very first one was created in June 2000,

00:10 and its purpose was to define the PEP process itself. Our guest this week, Nick Coghlan,

00:16 was a co-author on that very PEP and many, many more. In this episode, we're going to discuss

00:23 PEPs and how Python officially evolves, but also there are many other forces that drive and influence

00:29 Python. So let's dig into all of these right now. This is Talk Python to Me, episode 153,

00:35 recorded February 6th, 2018. Welcome to Talk Python to Me, a weekly podcast on Python, the language,

00:55 the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on

01:01 Twitter where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm,

01:06 and follow the show on Twitter via at Talk Python. This episode is brought to you by Linode and Datadog.

01:12 Be sure to check out their offers during their segments. It really helps support the show.

01:16 Talk Python to Me is partially supported by our training courses. Have you heard about the 100

01:22 Days of Code Challenge? It's a challenge where you write code for an hour a day for 100 days. It's helped

01:28 many developers finally master programming, but it can be hard to know what to study or have resources

01:34 to focus on. That's why we wrote not one, but two 100 Days of Code courses. 100 Days of Code in Python,

01:41 which covers mostly pure Python, and 100 Days of Web in Python, which covers a whole spectrum of web

01:48 frameworks and concepts. If you've been thinking about taking the 100 Days of Code Challenge, be sure to visit

01:54 training.talkpython.fm and check out our courses. They are the 100 Days of Projects and Lessons with a tidy

02:00 bow on top just for you. Nick, welcome to Talk Python. Hey, very happy to be here. Oh, it's a real honor to have

02:07 you on the show. You've been at this Python thing for quite some time. So going on 15 years, I think. Yeah, that's

02:14 awesome. So our main focus today is going to be to talk about how Python evolves, all the different forces

02:20 that press on it, the PEP process, all these great things. But before we get into those, let's just get your

02:25 story. How did you get into programming in Python? So I'm actually one of those folks who got into it as a kid. My parents

02:31 got our first computer when I was like, nine or 10, I think, which was a Apple IIe that spent lots of time playing

02:40 Winter Olympics and stuff on that with my siblings. And anyway, I don't actually remember the name of

02:44 the book, but it was like this big green thing, which had instructions on how to do programming in

02:50 Apple Basic. So yeah, so the classic 10 print, hello world, go to 10 type thing. Yeah, maybe a little

02:58 indentation and it screams across the screen, zigzag. Yeah. Yeah, basically. And you couldn't save

03:04 anything because the computer didn't have a hard drive. It's so crazy how storage used to be. I remember

03:09 some of my friends, they had, it must've been an Apple. I can't remember what it was, but had like

03:15 a big cassette player and you would plug it in and like the cassette somehow provided data to the

03:22 computer. And I don't even remember how that works. I just missed the cassette era with the computers at

03:28 my place. So we had a, we had the big five and a quarter inch floppy, floppy disks, which actually

03:33 were floppy. They were actually floppy. That's where the name came from. Cause they weren't for a long time.

03:39 No, the three and a half were never floppy unless you broke them open. Once you broke the case open,

03:44 then the disc inside was very floppy. But yeah, a friend of mine did have a Commodore 64 though. And

03:51 that was the, you put the tape in, you play the game. So how did it work? Did the tape basically

03:56 load the data into memory and then it would just run from there? Or like, I don't know. Yeah.

04:01 Yeah. That's, that's technology used to be so crazy. Now it's a little more standardized.

04:05 It's still crazy, but less bizarre, I guess.

04:09 Yeah. We were just using a magnetic tape on reels instead of a, crystalline structure in a USB

04:14 key.

04:15 Yeah, that's right. I guess we'll, we'll look back on this time as quaint when we're talking

04:19 about quantum computers and you know, whatever comes in 10 years, right? And AI is in, who knows?

04:24 The one that amuses me is you look back at shows like Babylon five and stuff where they

04:28 have their data crystals and you go, we literally have data crystals. That's what a USB key is.

04:33 Yeah. We really live farther in the future than I think we really appreciate. All right. So you did

04:38 some of this like early, really programming, this funky era, which we're talking about,

04:43 but then moved on from there, I'm sure.

04:45 Yep. So then I discovered in high school that programming was something you could do as a career,

04:50 which was cool. And then I went into university as a computer systems engineer.

04:54 which is kind of you computer systems engineering basically lives at the boundary between electrical

05:02 and electronic engineering and software engineering.

05:06 Back then there were, there were not really software development degrees, right? You either,

05:11 I mean, I, in the, the sort of software engineering, not computer science side. Oh yeah. Nice.

05:18 So Australia is a bit different from the U S. So for us, you guys are ahead of us basically. Yeah.

05:24 I was in university from 95 through to 2000. And so when I started, you could do a straight up

05:35 information technology degree with a software, there was a software subspecialty in that,

05:39 or you could do computer system engineering under the engineering department, which is what I did.

05:46 I think by the time I graduated, there was a straight up software engineering program.

05:52 And I think that was through the engineering department.

05:56 That's cool. I know those were coming on right around then. I missed that whole time.

05:59 My college started in 91. So yeah, just, just missed that window as well. Like I started before the web.

06:06 So that was fun. I remember the web coming out. That was so fun, right?

06:10 That was the other thing was our high school it teacher was, he basically, he also ran the

06:17 afterschool computer club, funnily enough. And so he was introducing us to like the mosaic browser and

06:24 Netscape Navigator 1.0 and a bunch of us in 93, it would have been. So grade 10,

06:31 we're all getting into the, online multi-user dungeon type stuff.

06:35 Paulie.clarkson.sos.net, is a domain name that's embedded in my brain because that was the

06:43 mud we played on.

06:44 Those were such fun times. That's so awesome. I played a bunch of those as well. All right. So,

06:48 so how did you learn Python in college or did it come after that?

06:52 So I have a funny, university story about Python, which is our networking teacher in 99.

07:00 This was wanted to try and level the playing field amongst all the students for the networking project

07:06 he was giving us, which was like a protocol design type thing. And he basically, so he told us the

07:12 standard language for the assignment is going to be Python because none of you will know that,

07:16 which is, so that's funny now looking back at it 20 years later, it's like no lecturer would do that

07:22 today. But the other one was that I actually put my hand up in class and said, do we have to use

07:28 Python or we could, can we use a language we already know like Java? And so he said, sure, if you really

07:33 want to. But then I did the, did the assignment in Python. I'm going, okay, yeah, I don't want to anymore.

07:38 And so that was still Python 152 at that point.

07:41 Wow. Those are pretty early days, but yeah, the contrast with Java probably was pretty stark.

07:45 Yeah, absolutely. We were mostly interested in the wire protocol structure. So using the struct module to

07:52 construct messages to send them back and forth. So yeah, that was quite a cool assignment. But

07:57 interestingly, I didn't, didn't touch Python again for another like three years after that, I think.

08:02 Wow.

08:03 Yeah.

08:03 Nice. And then you probably graduated three years later or so. And then was it work that brought

08:08 it to you or what happened?

08:09 Yeah. So, so what happened was I'd actually started working for Boeing Australia on the

08:16 Australian defense forces, high frequency radio communications infrastructure as an undergrad.

08:20 And so that became a full-time job after I graduated. And the system I was working on was actually a

08:27 digital signal processing system for voice detection in HF. And essentially in HF, the problem is ordinary

08:34 scrunchies don't work because HF is just very crackly and poppy. And so if you want to try and detect voice,

08:39 you actually have to do signal analysis to say, where's the energy? Is this just random crackling

08:46 noise or does it actually look vaguely like a human voice? And so that was all going along fine.

08:51 C programming for the DSP, but we didn't really have a proper test harness for it. We just had a C

08:58 application that ran on the host system. And if it got to the end without crashing, awesome.

09:03 Basically it's good. Now we're going to ship it.

09:05 Funnily enough, we kept finding bugs in the DSP software in the next level of integration testing.

09:13 It's like, who knew?

09:15 Who knew? What a mystery.

09:17 And so anyway, so we needed to do a better job of testing this.

09:20 And so there was a, another part of the system to orchestrate starting and stopping

09:27 services and bringing sites online and offline. And they just recently got a

09:33 approval to use, introduce Python 2.2 into the system so they could use it as this control

09:40 and orchestration language, which was on the approved languages list, which meant, and it

09:45 had this wonderful thing called the Python unit test module.

09:48 Yeah. I remember back then C++ didn't have very many options and C even less, right?

09:53 We actually had a in-house thing that engineers in Boeing Australia had designed for writing

09:59 tests in C++. And this was around the time where a colleague had inspired me to learn Eiffel.

10:06 We were fully into the extreme programming, test-driven development, all that sort of stuff.

10:11 Yeah. All that stuff was going strong then. It was so amazing. Yeah.

10:14 Yeah. And so, so they'd written that. And so it was a choice between that and the, the Python

10:20 stuff. But the beautiful thing that the Python stuff had was, so our host system was running

10:26 Windows NT and Python had the wave module that would let us play wave files out of the host

10:36 system, which was very handy because we needed to play those to, to check for the reactions from

10:42 the signal processing software. And then, and this is why I sometimes say that it's Dave Beasley's

10:48 fault that I'm a Python programmer, the SWIG existed. And so what SWIG let us do is we had a C++ driver

10:56 that ran on the host that the main production application used to talk to the DSPs.

11:03 And so what SWIG let us do, or let me do was basically, I just wrote one .ifile for SWIG to wrap the production

11:12 driver. And then suddenly I had this unit test system where I could call the production driver to set things

11:21 up in the DSP, call the wave module to play audio files out of the test system, and then monitor over the

11:28 network to say, did the signal processing software send the messages we expected it to send?

11:33 Oh yeah, that's really cool. That's awesome. Yeah. So practicality wins the day, right? There's some

11:38 simple tools to solve the problem. Yeah.

11:40 That's kind of continued that I've always been a huge fan of Python for testing because it means that

11:46 like when you try to do testing in C and C++ directly, you spend a lot of time building test

11:53 scaffolding to let you say, is this thing the same as that thing? Or is this what I expected? And so the

11:59 beauty of being in Python is you can express a lot of things more simply in the Python code than you

12:05 can in the production code. Right. It lets you move faster. It lets your tests be easier to maintain

12:10 over time. Yeah, it's wonderful. And so you just get this really nice combination of the tests in Python

12:15 express your intent. And then it doesn't matter what your production code is in. You're just saying,

12:20 am I matching my intent, whether it's faster or through a different mechanism or whatever.

12:25 Yeah, really nice. So you're probably not still doing that same project today, right? What are

12:30 you working on now? I worked on that project in different roles for like 12 years. But yeah,

12:36 so technically right now I'm not doing anything. So I finished up at Red Hat at the end of last year. And

12:41 so right now I'm talking to a few different people about what I'm going to do next.

12:48 That's awesome. Yeah.

12:49 So take your time to kind of find the next big thing.

12:51 I've got the luxury of not having to make a quick decision about what I want to do next.

12:56 And so it means I'm kind of going, what do I really want to be doing? Because to be honest,

13:02 I've spent the six years since I left Boeing, I've been completely in the software infrastructure

13:07 space. And it's like, while I don't dislike that, I do really miss being able to point at a piece of

13:15 hardware and say, my software makes that work.

13:18 Yeah. So when are we going to be riding in your next self-driving airplane that you're creating or

13:22 your next Friday, right? It's coming a few years out, right?

13:25 So that's the thing of, it's very tempting to just say, you know what, I'm kind of going to get

13:30 out of the software infrastructure world and go back into the physical hardware that does stuff.

13:35 But at the same time, there's a lot of interesting stuff to be done in open source supply chain

13:39 management. So yeah, of course, there's so many interesting problems, right? There's definitely

13:44 no shortage of that. That's awesome. All right. So let's go ahead and talk about our main topic,

13:49 which is how Python evolves. And, you know, we were talking just before I hit record. And I feel like

13:55 the two major ways that kind of drive where Python goes is kind of from the official inside,

14:03 which is this PEP process and these other meetings and core developers. And then there's the outside

14:09 world where nobody asks for permission, they just go make stuff. And then you guys look at it and go,

14:13 wait, actually, they're doing really cool stuff over here. Does that really drive that control like a

14:17 satellite in space? How are you doing that? Right? You're dealing with Python. So there's these sort

14:23 of two forces, right? The internal peps and the external just what people are doing. So what do you

14:30 think about that?

14:30 I kind of mentally break it up into two problems. So it's the thing of applying Python to a problem

14:37 domain. And that's the case of like, even Python 152 was an entirely usable orchestration language.

14:45 Python 2.2 was exceptional. That's why we stuck with it for so long at Boeing. And so in that sense,

14:52 it becomes the case of take Python as it exists today, apply it to a problem you have today,

14:57 and write whatever libraries you need in order to model that domain successfully.

15:02 So like that, again, gets back to my original exposure to it as hardware testing and orchestration

15:10 for an existing system. Like, I don't recall any time when I was doing that saying, oh, I wish Python

15:17 whatever, because it was the case of it did everything I needed it to do. And anything it didn't already do,

15:23 we'd just add a library for it. And we were actually doing a lot of stuff with Corba at the time as our

15:28 distributed communication protocol. And there were actually a bunch of different request brokers for

15:35 Python out there. So that was, and so for that, Python dev doesn't need to be involved. And this

15:42 occasionally frustrates people because they want us to be the single authority for all things Python.

15:48 And they're like going, we, you need to do something about blah. And we're like going, yeah,

15:53 why doesn't need to be us. It's like, we're not the domain experts on that.

15:57 Yeah, that's really interesting. And that is definitely something that's different in,

16:02 in the Python world from other organizations that are much more sort of mothership oriented.

16:10 I'm thinking Apple and Microsoft, right? Like if something is going to happen in those ecosystems,

16:15 there's very much like a blessing from the people who make the decisions.

16:19 Yeah. And that just doesn't happen here, right?

16:21 Well, and this is one of the interesting things. So this might be a bit of a tangent,

16:25 but the hardware system that I worked on for Boeing, a lot of the time I spent working on that was

16:31 actually as a system architect rather than a software developer. And when you're doing system

16:34 architecture, you spend a lot of time doing make buy decisions of whether you're going to write

16:40 something or whether you're going to buy something that somebody else is selling, which means you end

16:44 up doing a lot of spending a lot of time thinking about incentives and trajectories and, and why is a

16:50 vendor doing a certain thing? And what does that mean for you using them as a vendor? And if you look at

16:56 the history of programming languages, like C came about because people just needed an abstraction

17:02 layer over assembling, like rewriting.

17:05 It was absolutely necessary. Yes.

17:07 Yeah. So, so it was absolutely necessary. And, and so that was kind of, that kind of evolved out of the,

17:15 the needs of hardware manufacturers because they didn't want to rewrite their own software for every

17:21 new version of their hardware. That was actually quite a user-driven evolution there.

17:26 Like at that point, that was vendors solving the problem that they had and not necessarily trying

17:32 to use it to sell, to lock in customers. But then you look at things like emergence of Java driven by

17:39 Sun to try and stay relevant in a world where Microsoft were eating their lunch. As it turns out, it just made

17:46 it easier for Linux to eat Sun's lunch. So, so I don't think Java actually worked out quite the way Sun might

17:54 have hoped. Then it ended up at Oracle and Oracle is still trying to use it as a, as a way for them to stay

18:00 relevant. C-Sharp came out of Microsoft after they got into a big argument with Sun over Visual J++. So,

18:08 very much driven by Microsoft's things of how do we, how do we do something about this Java thing and make sure we

18:15 stay relevant in the business software world. Go, driven by Google trying to write things.

18:20 Apple really, really want people to develop on Apple for Apple, exclusively for Apple and say,

18:26 we'll do your market segmentation for you by excluding anybody who can't afford our hardware.

18:31 But it means if you write for Apple platforms, you're writing for wealthy people who will probably be able

18:36 to pay you. And so, so yeah. And so it becomes that thing of most languages exist in this space of having

18:43 been designed for a company that's trying to sell you something. Sun were trying to sell you servers.

18:49 Microsoft wanted to sell you Windows. Apple want to sell you tablets and iPhones and Mac machines.

18:56 And so, so there's those commercial incentives behind there. And then you come into the Python world and it's

19:03 like, well, Guido wanted to program his computer more easily and he wanted to make it easier for people to

19:10 learn to program. And so, so the incentives behind it are very different. And then you also look at,

19:17 you look at how the Python Software Foundation was set up. It was set up as a public interest charity,

19:22 not as a trade association. And so, and then you add in the fact that like some of Python's biggest

19:30 vendors don't even recognize that they're in the Python business. They're just shipping this thing

19:35 that came with their Linux distro. They sure you might want to use this or we use this. You might want

19:39 to as well. And so the vendors are all really quite passive and hands-off as well. They're just going,

19:44 yeah, whatever, we'll just do whatever, we'll just ship it. And so, so yeah, so you end up with this

19:50 very different dynamic where the large corporations uses, but not really driving the direction of the

19:57 language.

19:58 It's really interesting. And I'm sure people who come from those spaces that you named,

20:01 they look around and they're like, what, there's really, there's really no one group that just tells

20:06 me the way, right? It's pretty interesting.

20:10 This portion of Talk Python to me is brought to you by Linode. Are you looking for bulletproof hosting

20:15 that's fast, simple, and incredibly affordable? Look past that bookstore and check out Linode at

20:20 talkpython.fm/Linode. That's L-I-N-O-D-E. Plans start at just $5 a month for a dedicated server

20:27 with a gig of RAM. They have 10 data centers across the globe. So no matter where you are,

20:32 there's a data center near you. Whether you want to run your Python web app, host a private get server,

20:38 or file server, you'll get native SSDs on all the machines, a newly upgraded 200 gigabit network,

20:44 24-7 friendly support, even on holidays, and a seven-day money-back guarantee. Do you need a little help

20:50 with your infrastructure? They even offer professional services to help you get started with architecture,

20:56 migrations, and more. Get a dedicated server for free for the next four months. Just visit

21:01 talkpython.fm/Linode. So I think maybe one of the places to start is the Python that you described

21:10 back when you started your project in Boeing was being used to solve different problems than it

21:18 necessarily is now. And there's basically, even if you just talk about Python, obviously the syntax is

21:24 there. But what that means and how people use that could be really different based on who they are and

21:29 how they're solving problems. Whenever I even read like data science code, I feel like I'm reading

21:34 almost like a different dialect of the language because the mannerisms and the libraries just make

21:40 it feel really different than say like a web app that's like really broken into little pieces and

21:44 stuff. So I think I've mainly used Python myself in the use cases of test automation, which I would

21:51 still say I've used it more for test automation than I have for anything else. Hardware simulators,

21:57 particularly when you're working on a big thing like HF radio communications, you're not going to

22:03 have a 10 kilowatt radio transmitter sitting next to your desk. You're not going to have giant antenna

22:09 switch matrix sitting next to your desk. And even if you do, you might have one or two, which isn't

22:14 going to scale to automate testing for the whole system. So by writing simulators for those things

22:19 in Python, you can just do that relatively easily and test things that you just cannot test in reality.

22:27 So that's, that's an interesting use case. And then yeah, the straight up web application development

22:32 dating from the CGI days, which is actually how we wrote the original version of the system

22:37 orchestration, distributed system orchestration, which you now mostly see in things like Ansible and

22:42 salt in the Python rapids around Docker and Kubernetes.

22:45 Yeah. You even have people doing things like building, like say microcontroller code where

22:50 your Python program is almost, almost is the operating system, right? Like you wire your Lambda

22:56 expressions directly to hardware interrupts. It's like, there's this whole, whole thing. And what I

23:02 think is really interesting and both a challenge, and I think it's a challenge that's been well met by the

23:08 language designers, the language team and Guido in particular, is Python can be used in all these

23:13 ways. And it, it doesn't feel like it's built specifically for one of them. You know what I

23:19 mean? Like, like, if I were to look at R, like R is made for a thing, right? It's for solving math

23:24 problems. And then you can kind of do other stuff with it. Whereas Python also solves math problems

23:28 really well, but it also builds web apps amazing or controls space telescopes or you name it or automates

23:33 hardware, right? And I think this gets back to the origins in the ABC research language,

23:39 where the goal of ABC was to be a language for teaching people how computers work or how to think

23:44 computationally. And then Python was basically Guido's version of that, that you could actually use to,

23:49 that he could use for automating things. And so it becomes that thing of that base layer of imperative

23:57 procedural programming of do thing A, do thing B, if condition, do thing C or do thing D,

24:05 do thing E five times, do thing F until we run out of stuff. That basic procedural stuff,

24:13 it's fascinating. Like if you go read standard operating procedures, work instructions and so forth

24:19 that are written for humans, those are your basic constructs of how we give directions to each other.

24:25 And so it's only, only when you go to build more complex systems, do you start needing all of the

24:34 other constructs of computer science. And, and it's the thing of you start with imperative programming

24:40 because that's the way humans think. And it's if humans are going to write a checklist or break down

24:45 a task step-by-step to give to each other, they're going to start with that procedural model.

24:50 But there's a reason computer science involved all these other things because that procedural

24:55 problem, procedural modeling has scalability issues.

24:58 Both in terms of complexity for sure. Yeah.

25:01 Yeah. And so, but it's, if you look at the way a lot of, a lot of languages are written, they're saying

25:07 they have a very strong opinion on this is how we're going to deal with complexity.

25:12 And that's great if you have a problem which fits well into that complexity management model.

25:18 And the beauty of Python is Python says, well, you're going to have procedural stuff in there

25:24 somewhere. So start with procedural. But we know procedural doesn't scale to large teams,

25:31 to large code bases. So we're going to offer you these, all these other options for refactoring your code

25:40 as you better understand the problem you have. And that basically means it's, you have this common

25:47 procedural subset, but then you have all of these different approaches to data modeling and computational

25:55 flow modeling and all that sort of stuff where once you figure out what kind of problem you have,

25:59 you can move in that direction.

26:01 Yeah. And I feel like you've kind of captured one of the really special things about the language.

26:06 And that is, you can start simple, but it's not a simplistic language. Whereas Java and C#,

26:13 they say you work in this big formal framework from Hello World onward, right? You take all the

26:19 complexity management from the beginning. And there's other languages like say, visual basic that are

26:24 like, no, the world is simple enough, we're going to build simple things, but you hit an upper bound,

26:29 you're like, okay, well, I can't use this anymore. Because you know, I don't know, threading or whatever,

26:33 right? There's some, there's just a bunch of things you can't build with this simpler language. And somehow

26:37 Python's managed to cross that gap. And that's pretty, pretty awesome.

26:41 So I think one of the ways we actually do that is if you look under the hood of a lot of those higher level

26:47 constructs, ultimately, they just boil down to procedural internals. Like at runtime, we are literally running

26:56 through and doing stuff procedurally. And we, I think of it as the kind of a no privileged access concept, which is not

27:06 actually true. We do have privileged access and the interpreter can do things that ordinary Python code can't. But as far as we

27:13 can, we make it so that all of the syntactic sugar that we provide for things like classes and so on and so forth, we do our

27:22 best to make sure that there is an actual procedural equivalent, that any syntactic construct, like beyond the basic

27:29 arithmetic, you can write out a pure Python equivalent may not run as fast, and it you may not get the error handling

27:37 right. But conceptually, there is a Python equivalent of that's just procedural code, which is quite fun.

27:45 That's a really cool philosophy. So maybe let's focus on some specific domain. So we touched on them a little bit, but we

27:52 have, you know, web frameworks, GUI libraries, scientific stacks, and these have all evolved over time. One of the

27:58 examples of space from the sort of infrastructure world you're talking about is Ansible.

28:03 One of the things I like about Ansible is if you look at, like Python's basically been part of Linux

28:10 distributions almost from the beginning. Like I think it was there, it was there in 1.5. So it was there in

28:18 Red Hat Linux, it ended up becoming part of Red Hat Enterprise Linux, they had to deal with the Python 2

28:24 transition. And now they're dealing with the Python 2 to Python 3 transition. But anyway, by being there by

28:29 default, and it was always there on all the Red Hat systems, because it was part of the package manager,

28:34 it was usually there on other distros, because some utility or other would be on the system that needed

28:41 it. And anyway, so it meant that sysadmins started doing lots of admin scripting in Python. They'd write

28:48 these tools in Python, and everybody would write their own. And, and then they kind of like going, well, hang on,

28:53 why is every admin having to write their own custom code for all the same tasks. And so in a lot of

29:00 ways, you can look at Ansible as a way of going, well, how about we turn the actual automation bits

29:09 into modules that you put up on a, put up on, say, Ansible Galaxy, which you then, which you can then use

29:18 from a declarative thing to say, I want all of this stuff to happen in this order. And that then gives you

29:23 that more structured notion of what are the concepts that admins need to care about, rather than the

29:29 precise syntax of a full loop or whatever. And so you get this kind of, you go from this procedural

29:34 model, model down at the Python layer, up to a declarative model further up the stack.

29:38 And you get so much flexibility when you say, this is the state I want it to be in. I don't care how that

29:44 happens. You just make that happen. If you want to change how you do that, I don't care. I just want the

29:48 end state to have Nginx installed with this SSL certificate or whatever, right?

29:53 One of the phrases I use for this is the idea of executable knowledge, which is the case of you can

29:58 have a domain expert write a piece of software and they know all the nitty gritty, horrible details

30:04 of how messy reality is because humans, humans in history. But as a user, you don't need to care.

30:15 You're just like going, I'm just going to trust you to know more about that than I do and say, make it so.

30:21 And then one of the things I love about the scientific Python stack is like people talk about, oh, programmers

30:27 need to be good at math. And you're like going, no, programmers are terrible at math.

30:31 I have this conversation a lot with a lot of people and they say, well, what kind of math did I study?

30:36 He's like, you know, I barely use algebra for my job.

30:39 And it's the thing of most of us stopped learning math in high school. And I did learn a bunch of

30:44 engineering mathematics and actually working a signal processing job. I finally understood what

30:49 a fast Fourier transform was really for, which had never clicked for me in university. But in my day to

30:54 day job now, I don't, I don't use any of that. I haven't used that since I was doing the signal

30:59 processing stuff. And even when I was using it, the actual fast Fourier transform, I just called TI's

31:05 library to do it. And so it's the same with scientific stack today. Like modern research

31:10 mathematics is spectacularly sophisticated, whether you're looking at the numeric algorithms and

31:18 array oriented programming or calculus or whatever. But in Python, you can just go, well, I'm not

31:24 going to understand all of that myself. I'm just going to import pandas and have magic happen.

31:30 Right. Or I'm going to set up a random forest or some deep learning thing by five lines of

31:35 configuration or something. Yeah.

31:36 And you're like, I'm going, and it's that thing of you should be simpler than implementation. Like

31:40 you should be able to use things without understanding how they're implemented. And I think that's the

31:46 beauty of how the domain specific tools evolve is people are able to write them, expose these common

31:53 APIs as import magic from here.

31:56 All right. All right. QXKCD.

31:57 And then you have access as a user to all of this amazing domain knowledge that you then don't have

32:07 to, you can learn it if you want to, or you may learn it by necessity when it breaks. And you're like,

32:12 it's like when it breaks, you get to keep all the pieces.

32:15 Yeah. There's also this difference of being like, I'm learning it enough to implement it or

32:20 completely work in it, or I'm learning the concept of it really well. Like, like I,

32:25 I understand the relational model of a database and indexes and all these things, but I don't know

32:31 that I could go write a database from scratch without a lot of research, but that's okay. I

32:35 don't mind using a database. It works fine, you know?

32:37 Yeah. Well, and SQLAlchemy is another really good example of executable knowledge. Like I highly

32:43 recommend some of the talks about the design of that and the work that goes into keeping things

32:49 responsive at the application level and avoiding round trips to the database while at the same time,

32:55 making sure you don't get yourself into trouble with case-to-coherency and all that sort of stuff.

33:00 It's an amazing piece of work, but at a user level, it's import.

33:04 Yeah. It's amazingly simple, right?

33:06 Yeah.

33:06 So another really interesting area, especially in the evolution part is because, you know,

33:11 Python is, it came out in 1991. So it's seen a lot of different UI stories along the ways,

33:20 right? And it's, it's spread so far and wide. And to be honest, the UI story today is so unlike it was,

33:27 you know, in the early days. So maybe touch a little bit on this evolution through the,

33:33 like the Windows macOS Linux UI story.

33:37 And one of the ways this comes up is people are going, why is TCLTK in the standard library?

33:41 And so TCLTK has been there longer than I've been using Python. And one of the interesting things

33:47 is, so when I started working professionally in 98, Linux just wasn't there for the design of big

33:54 systems. And this was actually a source of frustration on the system I was working on,

33:58 because there was a lot of servers where we're running Windows NT on them. And we're just like,

34:02 going, why? And there were a bunch of other servers, we were running Solaris on them. And we're like,

34:07 going, why? And then you looked at when those operating systems were picked and the state Linux

34:11 was in at that time and just like going, yeah, okay. I understand this decision completely.

34:16 Fair enough. Yeah.

34:17 And switching now would...

34:19 Committed. Yeah.

34:19 We did end up switching a bunch of those things out for Linux eventually, but,

34:23 but it took some time and it took some work. And so anyway, so you look in the late nineties,

34:27 Linux isn't really a presence yet. It's starting to become a thing, but, but it's

34:32 ad hoc and experimental and...

34:34 Yeah. And it's desktop story is even more ad hoc and more experimental, right?

34:37 And so lots of things we take for granted today just don't exist.

34:41 Like GNOME. GNOME was founded in 98. And so anyway, and so TKinter was just how you did cross-platform

34:48 Unix GUIs. If you wanted a GUI that worked on AIX and worked on Solaris and worked on various other

34:55 things, you use TKinter. And hey, it works on Windows too. Doesn't look anything like a native

35:01 Windows app, but it works on Windows.

35:03 It looks like transplated from another world, but yeah, you could put stuff in the text boxes.

35:09 Yeah. And so TCLTK was the way you did cross-platform in the late nineties. And so at that point that was

35:16 before the pip process, before SourceForge development. So before any of that. And so it was

35:21 like, yeah, idle ended up in the standard library. So there was a default editor and it was all cool.

35:25 But then as Windows became increasingly popular and as Linux became increasingly popular, you basically had

35:34 people writing native Win32 apps. You had QT as a C++ wrapper that tried to tie, tried to use native widgets

35:43 when it could, emulated when it couldn't. And then you had GTK emerging as this is the way Linux distros are going

35:50 to do their desktop applications. Right. And just to put it in this perspective in time, like when Windows 95

35:56 hit, that was a major, major change in the desktop environment, right? Like that really was a big splash, sort of

36:04 like the iPhone landing in the phone world. Yeah, absolutely. So it's not surprising that a lot of people, you know,

36:10 went to this Win32 space with MFC or whatever they were using. And it was also around about then that you basically

36:17 could make a good living either selling prepackaged Windows software or software for download. Like the

36:25 internet was just becoming a thing. Yeah, there was that two cows and all those places coming along. The

36:30 internet was coming along. Yeah, it was great. And so it was the idea of pay to download was a thing. You

36:37 basically clicked through and gave somebody money to download stuff from their website. Yeah. Another one

36:42 that came up was WX Python. This was the thing of so kind of along a similar line to QT, there was WX

36:49 widgets. And WX widgets was that thing of how do I write cross platform macOS Windows and Unixy

36:58 applications without having to rewrite my GUI for everyone. And then so that WX Python wrapped WX widgets.

37:05 We had PyQT from Riverbank to wrap QT. And so none of these ever got added to the standard library because

37:12 just the rate of evolution and the size of complexity of the libraries they were binding to

37:17 TCLPK is tiny by comparison. And so, yeah, and so it became the case of that kind of started the

37:24 to this day unresolved fight between QT and GTA on Linux. And personally, I use KDE just because it's,

37:31 I grew up with Windows, use Windows and DOS since DOS 5, I think. And so, yeah, and so I'm just used

37:40 to the Windows paradigms of how things work. And so as much as people who like, who are fond of macOS X

37:47 might go, oh, but this way is better. I'm like going, it might be better for you. It's not for me.

37:50 It just got so complicated that it was just like, there is no standard. So there's nothing we can pull

37:57 into the standard library and say, this is the way things are going to be from now on.

38:00 Right. And the licensing is a little weird on QT. And so there's just like a bunch of stuff that

38:06 just is not a good fit, right? And like GTK's efforts to support, well, so the problem with GTK

38:12 is that its efforts to support anything cross-platform isn't really a thing. Like it's the,

38:18 essentially the Linux and Unix-y equivalent of the Win32 APIs. Like it itself is not really

38:26 cross-platform because it doesn't delegate to platform native stuff when you're running it

38:32 somewhere other than Linux.

38:32 Right. And that's really important because that's what it takes to make your app look like

38:36 it belongs in that environment.

38:38 Yeah. You have to use the native widgets rather than skinning your own emulation.

38:42 Yeah. So right after this whole sort of thing, the world kind of said, okay, well, Java and C-sharp

38:48 and Windows forums and WPF and Swing and just all of this stuff kind of scrambled it further.

38:54 That was the C, C++ world, which is kind of CPython's native stomping grounds. That was complex

39:00 enough. And then we had the whole thing of, oh, writing C and C++ directly is really hard.

39:07 And it's really hard to secure. So managed languages are going to be the answer.

39:12 And then Sun and Microsoft get into a fight. And so you end up with Java and C-sharp, right? And the

39:19 JVM and the CLR rather than one platform. So that's like, okay, cool. That makes life even more

39:27 complicated for everyone. Then Apple came along.

39:30 Somehow they became relevant again, right? Because they were on the verge of being barely relevant.

39:35 Never gone away completely because they always had that graphic designer, creative artist market.

39:40 And so there was always that, there was always that strong core of like, they were always second.

39:47 Like Windows completely took over the desktop, but Apple always had that like five to 10% of diehard

39:55 macOS fans. And I think it was, I think it was Autodesk stuff like tended to run really well

40:02 on the Macs. And so anyway, but then the iPhone happened and people are like, oh, we need touch

40:12 interfaces. We can't assume there's a mouse and keyboard all the time. And so it's like, okay,

40:17 what are your cocoa bindings like? What are your carbon bindings like? And I lost track at carbon.

40:21 I haven't paid any attention since then. I don't know if it's still carbon or if they've moved on to

40:25 something else by now. But yeah. And so, and so that was like, oh, okay. Everybody needs to

40:31 care about macOS X again. And then Google came along and said, well, the native base platform for Android

40:37 isn't going to be the underlying Linux core. It's going to be the Dalvik layer. And so if you want

40:45 everything to run snappily and use native widgets and be nice, you really need to be running in the

40:52 platform Java runtime. And then you're just like going, and if you don't do that, then you're either

40:58 going to be bridging into Java, which is slow, or you're going to be using non-native widgets and

41:02 emulating the native ones. And you're just like, oh, really? This was already, this was already terrible.

41:09 And you just made it worse.

41:11 Yeah. And you've got things like Electron JS and Cordova and Ionic and all these other frameworks

41:16 are trying to make the web sort of fit into this world. And they're all pretty neat. And every one

41:20 of these kind of had their space, but it just makes the story harder for what is the path for Python

41:26 in some sort of UI story. There are some options though, right?

41:30 WX Widgets and WX Python, when they were designed, basically said, okay, there's three big things we need

41:35 to care about. We need to care about Windows. We need to care about macOS X.

41:38 We need to care about GTK. And so they basically said, we're going to provide an abstraction layer

41:43 over those three things. You write to our abstraction layer, and then we'll translate it as appropriate

41:48 for any given platform.

41:49 Have you been tracking the Phoenix project from them?

41:51 I was for a while because that was the gating thing for their Python 3 support. I haven't looked

41:57 at it recently though.

41:58 Yeah. So they just like this week, maybe released WX Python 4.0, which is the whole Phoenix release.

42:04 I don't know how much that was before, but it's, it seems like this might be coming back.

42:08 It's a solid option. I actually wrote an app in this morning and it was really nice.

42:12 That's really cool to hear because there's a, Robin Dunn wrote a book, WX Python

42:16 in action, like 15 years ago, 10 years ago.

42:19 Time for a second edition.

42:20 Yeah. Yeah. But I was one of the tech reviewers for that.

42:23 Oh, nice. I think the name says a lot, right? The code name is Phoenix. Like it's a back sort

42:29 of from the ashes.

42:30 They were really, because they were tightly tied to the CC++ libraries. They were kind

42:34 of that whole Java and C# and the web thing made life really difficult for them because,

42:40 because all these shifting paradigms and expectations were all happening.

42:46 This portion of talk Python to me is sponsored by Datadog with infrastructure, monitoring,

42:51 distributed tracing, and now logging Datadog provides end to end visibility into the health

42:56 and performance of your Python applications. With out of the box support for frameworks like

43:00 Django bottle and flask. You can start monitoring your application performance within minutes.

43:05 Start monitoring your Python applications with a free trial. And as an added bonus, Datadog will

43:11 send you a free t-shirt. Just visit talkpython.fm/Datadog for more details.

43:16 Yeah. Another one I have high hopes for is the project from Beware.

43:22 Yes. And Beware's Toga is the one I think is really interesting. So it's still relatively

43:28 young, still lots of missing pieces. But yeah, it's based on that core notion of to really give

43:36 a compelling user experience for somebody that is expecting a platform native experience,

43:41 you need to actually use the native widgets. That if you're emulating the widgets, then

43:46 users can tell because application can be cross-platform. Users, generally speaking, are not.

43:52 They're using one platform. They're expecting your application to behave like all the other things

43:59 on that platform.

44:00 Right. You just want to reach all the users, but they're probably in one place individually for the

44:05 most part. Yeah. Yeah. And generally speaking, users will be like, they have the native apps for

44:09 whatever client device they're using, plus they have a web experience. And then maybe they have a

44:15 native desktop OS as well. That's increasingly becoming a thing of, no, lots of people do everything

44:21 on their tablet or on their phone. They have no desktop. And so, yeah. And so Toga basically take the

44:28 point of view of, okay, CSS is good as a way of styling things and doing layout and all that sort

44:34 of stuff. So let's lift that in and use that as the basis of styling your application.

44:41 That's a brilliant design choice, I think.

44:43 Yeah. And say, okay, let's not reinvent that. Let's just use that. But at the same time,

44:47 no, I don't think they use the DOM. I don't think they pulled the whole DOM in, but they made sure that

44:55 whatever they did was easy to translate into HTML5 DOM. And so essentially what they let you do is

45:01 rather than having to choose between, am I writing a web app or am I writing a native app?

45:06 They treat HTML5 as another target environment. And so you basically, one of the things you can get out

45:15 of your project is a Django 5 progressive web app that say, okay, so we target the browser as a

45:21 platform. There's your web app. And here's your native iPhone app. Here's your native Android app.

45:27 And it's really impressive the amount of underlying infrastructure they've developed for this.

45:34 Yeah. There's a lot of building pieces, a lot of cross compilers, and a lot of stuff that makes

45:38 this possible.

45:39 And one of the big things they realized was that they didn't have to write a Python compiler for every

45:43 target environment. What they could do is they could use CPython as the core compiler and emit CPython

45:50 AST, I think, and then go from the AST layer into JavaScript by the Bartavia project and into the JVM

46:00 by the VOC project. And then anywhere else that has native C APIs, then they can just use CPython.

46:08 But it was that big thing of that in the web, they really needed the native JavaScript implementation

46:13 just because currently the cost of getting from another language runtime into JavaScript to access

46:23 the DOM is just too slow for really snappy responsiveness on applications.

46:30 So WebAssembly might change that once they add some DOM interfaces.

46:35 But for now, I'm really interested in to see how WebAssembly is going to affect this because

46:40 there's a lot of JavaScript, Python to JavaScript options like Brython, Sculpt, Transcript, Batavia.

46:45 If that becomes like a native thing that is fast, like that actually might be really interesting.

46:51 One of the ways I put it is JavaScript's in an interesting position where it's ubiquitous because

47:00 for historical reasons that...

47:02 Because Netscape.

47:03 We actually...

47:04 Because Netscape.

47:04 But also because we managed to avoid the vendor fight.

47:07 Yeah, that's true.

47:07 Microsoft actually eventually came to the party and said, okay, we'll just implement JavaScript

47:11 as written.

47:13 And so that Java versus C# style split never happened, which is kind of astonishing.

47:20 Those were bitter fights over the whole browser internet time.

47:23 So surprising that that language made it through as a shared item.

47:27 And that's just...

47:28 It's astonishing that it survived the vendor turf wars.

47:32 And full credit to them for dealing with that.

47:34 Yeah, but maybe it wouldn't have if JavaScript had been more important at the time.

47:38 But at the time, it was kind of like jQuery was the pinnacle.

47:42 You could animate in a little thing.

47:43 You could put a little calendar.

47:45 But it wasn't like Angular and all these other major front end things.

47:49 Whereas the major part of your app is now that thing.

47:52 And I think that is the thing of...

47:53 You kind of do need that freedom of the vendors are there.

47:57 So you have the credibility and the resources.

48:00 But they're not really paying attention.

48:03 And they're not trying to use you as a competitive weapon.

48:06 Because yeah, that's when things get really messy.

48:09 Yeah, that's really interesting.

48:10 So let's focus on another area for a moment.

48:12 Let's talk about sort of the meta tooling, like the packaging and software distribution story around Python.

48:17 Like we have things like Briefcase from the Bware guys.

48:21 You just the same area that we just talked about.

48:23 We've got CX Freeze.

48:24 I use PyInstaller in that project.

48:26 I talked about WX Python.

48:27 There's a bunch of options and more that we're not even listening.

48:30 There's PIP, obviously.

48:31 But maybe talk a little bit about the evolution of that kind of stuff.

48:35 This is a fascinating thing.

48:37 Because back when I worked for Boeing, we did a lot of stuff with Windows.

48:41 And so we were running Windows installers.

48:43 And writing software that actually properly respects platform conventions is genuinely difficult.

48:50 It's super difficult, yeah.

48:52 Back when I worked for Red Hat, one of my biggest complaints was that there basically isn't any developer documentation for here is how to write a well-behaved Linux application.

49:04 Like they just don't exist.

49:07 And you've got bits scattered over in the file system hierarchy stand and all that sort of stuff.

49:11 And so you're just like going, the reason people don't write well-behaved applications is they have no idea what the rules are.

49:19 Because none of the Linux vendors have actually written the equivalent of Apple's user interface guidelines or Microsoft's user interface guidelines.

49:28 And you need those to say, what should I be doing?

49:33 Because otherwise I'll just make something.

49:35 Where do I even put the program on the computer when I install it?

49:38 What's it supposed to do there, right?

49:40 And it's like, and like Linux devs will say, oh, but file system hierarchy standard.

49:45 And I'm like going, have you tried to read that documentation?

49:48 It's all written for system administrators.

49:50 It's like none of it's written for software developers of here is what your application should do.

49:56 And so you're just like going, okay.

49:58 So how about we just define a simpler format that lets developers not worry about the whole mess that's out there and say, like, you just write your application as a monolithic thing that drops into a directory and just tag bits of it with what they are.

50:14 And then we can write tooling that will distribute things out into the right places.

50:19 And the beauty of working, and this is something we've been working really hard to enable on the disutil sig level, which is that, so we focus primarily on the developer use case of I'm installing this to hack on it, not to just run it as an application.

50:39 And so you want to be able to run test suites, you want to install dependencies in order to run a modified version that you've made local changes to and all that sort of stuff.

50:48 We're really primarily interested in the developer experience.

50:53 But at the same time, we're trying to get to a point where we make our description formats actually specified standards so that applications other than Python itself can read them.

51:11 And then once you do that, and once you start describing things in static metadata formats, then suddenly you enable stuff like PIX, which is the Python executable format, and basically takes that and says, okay, well, here's a thing where you just have to add a Python runtime and say, run that thing.

51:31 And also getting to the point of actually having true single binary executables.

51:37 Yeah, for people who are listening, PIX is basically like a bizarro hack on a zip file that contains a bunch of Python code, right?

51:44 But it lets you package up everything that you need into like one distributable binary thing.

51:51 Slight tangent, that's actually a case where we made a change through a tracker issue to enable execution of zip files.

51:58 And we forgot to mention it in the Python 2.6 What's New?

52:04 Which they meant, we're like going, why does nobody know that Python can do this?

52:08 We're like going, oh, because the only mention is like buried in the news file for Python 2.6.

52:16 We never actually put it in the What's New.

52:19 How funny.

52:20 And so then I think it was around Python 3.4 or 3.5, we actually added zip app to the standard library.

52:28 Not to let you execute this thing, but to provide some tools for creating them.

52:32 And so then I think after we did that, then more people were aware that the capability existed and had existed since 2.6.

52:38 Wait, why does this exist?

52:39 Can you actually turn making that part of this?

52:41 So what do you think about, to me, one of the challenges of Python outside the server space, right?

52:48 The server space, everything seems basically handled to me.

52:50 But if I want to write an application, let's say I go and use WX Python or PyQT or something like that.

52:57 And I get this nice app.

52:58 I want to give this, you know, we talked about 2CALs and the download stuff.

53:02 And here's my zip.

53:03 And I'm going to, like, the story for that in Python is, I think it's coming along, but it's still pretty early days.

53:11 Things like PyInstaller and stuff, you can make these sort of binaries that don't even require Python on the machine or the dependencies, right?

53:18 You don't have to describe, hey, dear user, you begin by, you know, creating a virtual environment and pip installing these things.

53:24 Like, that's, you know, that's not the average, like, non-developer workflow.

53:28 Yeah, exactly.

53:29 So here's my .app or my .exe.

53:31 There's one thing.

53:32 I double-click it.

53:33 That's it.

53:34 How do you see the current space where we're going?

53:37 Are you optimistic around that?

53:38 Do you think it matters?

53:40 I think it does matter.

53:41 Interestingly, because it's one of those things, like, desktop, I think, is going to go the way of kind of like an artist's drawing table.

53:49 Like, most people don't have a drawing table in their house.

53:54 But if you're a professional artist, you probably will.

53:57 And so it becomes that thing of you get to the point where vast majority of stuff can be done with just a phone or a tablet.

54:05 But there will be things where professional things or hobbyist things.

54:11 We just like going, no, no, no.

54:12 I want a desktop.

54:13 Right.

54:13 You know, one example.

54:14 People say this stuff a lot.

54:16 And I do agree with you.

54:17 One example might be, like, if you're writing software, like, for your company of, like, 10,000 stock traders.

54:25 And they all look at the market.

54:26 They have, you know, those insane, like, six screen, nine screen configurations.

54:29 And they want their little dashboard to absolutely update insanely fast, just almost client server style.

54:36 Right.

54:37 Like, like a mobile app or a high latency web app or something is probably not going to be a replacement for those kinds of environments.

54:45 Yeah, exactly.

54:46 You're going to have those specialist applications.

54:48 And I think the direction we're going to go is, again, talking about the Beware stuff, I really like the direction they've taken with Briefcase, where the idea is, instead of trying to convince developers to care about all these potential deployment targets, you instead get them to describe the information that they have that the computer cannot figure out automatically in a way that's useful to them for their own development environments.

55:14 So, in this case, Python-level dependency descriptions, that sort of stuff.

55:18 And then you just try and automate everything from that point forward.

55:21 And so, Russell Keith McGee, the founder of RebootBware, has a wonderful demo of, what is it, six platforms in 20 minutes.

55:30 He does have his machine already set up to publish to all the different platforms.

55:35 So, like, he has Xcode stuff pre-installed.

55:38 But, yeah, so in 20 minutes, he goes from a blank page to a built Android application, a built iOS application, a built Django web application, a Windows installer.

55:49 I don't think he has Flatpak in there as well yet, but I think he was looking at it.

55:54 So, yeah, and so it becomes that thing of, as a developer, you just go, I write my app, and then I can do cross-platform stuff without too much pain.

56:02 Sounds a little bit like what Ansible does for server orchestration.

56:05 I think so. It's about encapsulating that executable knowledge of what does a native app look like on these platforms.

56:11 Right, and so are you optimistic on, like, Python as a language for building mobile apps?

56:16 Yeah, I think so.

56:17 Somewhere in the future.

56:18 And I don't know yet whether that will come about through the Beware VOC model or whether it will be a result of getting decent WebAssembly DOM access and doing things that way or some combination thereof.

56:33 Because one of the interesting things about WebAssembly DOM models is it then raises the prospect of potentially exposing native widgets through those interfaces, which all becomes quite interesting.

56:47 Yeah, that's for sure.

56:48 One thing that might be an interesting path that is not any of the ones you mentioned, those all could be good.

56:54 And we only need one, right?

56:55 One could be PyPy, actually, to some degree, or the Pigeon project, where the way that they got Csharpen.net onto iOS and Android was they basically used the JIT compiler and then just ahead of time compiled it.

57:10 So if you could get some form of, like, JIT compilation, say, via PyPy, and you just don't JIT it, you just fully compile it for the machine and then drop it there, maybe that would actually be a good path.

57:22 I don't know what the state of PyPy pre-compilation caching is, because certainly, like, PyPy's current challenge, I think, is with startup time and warming up the JIT.

57:35 And there's a lot of ways to mitigate that and manage that.

57:38 And in particular, mobile environments where people tend not to close applications down completely, they just suspend them in the background.

57:45 Yeah, that does have the potential to work quite well.

57:48 Yeah.

57:49 I don't see anybody pursuing it, but it's still yet one more potential path in the future.

57:53 It's actually one of the interesting things is that I still think there's not a lot of understanding of how compelling and revolutionary PyPy is as a technology.

58:04 And I think it kind of gets back to the case of so many of us are using Python for orchestration use cases where we were kind of like, well, if our Python code becomes the performance bottleneck, something's gone horribly, horribly wrong at a system architectural level.

58:19 Like, it's not supposed to be in the critical path.

58:22 We really frustrate the PyPy guys, by the way.

58:25 It's like, sorry, the PyPy crew trying to avoid guys for that kind of thing.

58:30 But Intel wrote a instruction set emulator in PyPy.

58:37 So I think that was the Pidgin project that you mentioned.

58:39 And yeah, and they're just like going, hey, without tuning the JIT, we're kind of rivaling QMU for performance.

58:47 And you're just like going, seriously?

58:49 That's astonishing.

58:51 Yeah, the Intel guys are doing a lot of stuff at the like chip level to optimize Python and PyPy.

58:57 And they're doing contributions back.

58:58 It's really pretty cool.

58:59 And yeah, and that's been really cool.

59:01 I've spoken to some of those folks.

59:03 But the interesting thing is, we haven't yet seen anybody try to write a Java bytecode interpreter or a LLVM IR interpreter or a JavaScript interpreter in the PyPy technology.

59:19 And the interesting thing is, those are the ones which actually have the really, really heated cross-interpreted performance debates.

59:26 Where it's like, oh, a new release came out.

59:29 Well, we will have multiple articles.

59:31 How fast is it on the competitive benchmarks between the different implementations?

59:36 And I mean, that's one of the things you always see for V8 or new browser versions or that sort of stuff.

59:42 It's like, oh, how does it do on the benchmarks?

59:44 Yeah, exactly.

59:45 We run this JavaScript benchmark.

59:46 It's slightly faster than V8 now.

59:49 So take that.

59:50 Yeah, things like that.

59:52 We just don't have that kind of competitive performance mindset in the Python world.

59:57 I think that we're heading down that path a little bit.

01:00:00 I mean, you look at the work that Victor Stinner has been doing to optimize things around the Python 3.5, 3.6 timeframe was like, look, performance is a feature.

01:00:10 And we're now going to get serious about making this faster.

01:00:12 I think there is a lot of work being done there.

01:00:15 But you're right.

01:00:15 There's still a long ways to go.

01:00:16 Yeah.

01:00:17 But it's just one of those things, like having been in the vendor space as well.

01:00:20 It's like you've never had customers coming to you and say, we want you to make our Python code faster.

01:00:25 And so when customers aren't actually banging on vendors' doors saying, we want you to make this faster, then it doesn't happen.

01:00:33 Whereas in the JavaScript world, we had the driver of browser vendors wanted people using their browser so that they could get their browsing data.

01:00:43 And it's like running JavaScript faster meant more responsive websites, which meant people enjoyed using the browser more, which then meant they would use your browser, which meant you could get their data.

01:00:53 Yeah.

01:00:54 That you're at the end of that line, that funnel, by the way.

01:00:56 But I think it's really interesting that, say, Google, for example, who really doesn't care how you use the web.

01:01:02 They just want you on the web more, doing more web stuff to a large degree, is like, we're going to build V8 and make this super fast so that we can make these more engaging apps.

01:01:11 So you're just on the web and you might do more searches.

01:01:13 So we get your ad money.

01:01:15 We can retarget you.

01:01:16 We can show you more ads more quickly.

01:01:18 Exactly.

01:01:19 Yeah, that's a pretty interesting thing.

01:01:22 So we're kind of getting short on time.

01:01:23 So let's maybe do just a little bit of a conversation about how the standard library and the language evolved.

01:01:29 Because when people think of evolution of Python, they might think, well, that means a new library or there's like new syntax for some other comprehension I haven't imagined.

01:01:38 This is what people like to think of when they go, how Python evolves.

01:01:41 And so the biggest thing is essentially CPython's issue tracker.

01:01:45 So one of the interesting things about Python is we haven't gone the direction of actually defining a formal international standard of what it means for something to be Python.

01:01:56 Instead, what we have is we have a informal language reference as part of the CPython documentation.

01:02:03 And then CPython itself is the reference interpreter.

01:02:08 And so I should actually probably say what I mean by CPython.

01:02:13 Because for a lot of people, they go, what's CPython?

01:02:15 We just use Python here.

01:02:17 Yeah, I type Python.

01:02:18 It goes.

01:02:19 I don't know what the CPython is.

01:02:20 And so this is one of the interesting things is.

01:02:22 So there's actually two different things.

01:02:24 There's Python the language, which is the thing you actually type into a text file, which then needs to be interpreted by something or compiled by something.

01:02:32 And then the thing that actually executes when you type Python at the command line, we call imaginatively CPython because it's Python implemented in C.

01:02:41 In truth, the code base is actually more Python than it is C.

01:02:45 But the core engine, like the compiler and the eval loop and the built-in types are written in C.

01:02:51 The core heart, the byte coder is handed off to C eval.c.

01:02:55 There's a for loop and a switch statement in C.

01:02:57 And that's kind of where the execution happens.

01:02:59 But yeah.

01:03:00 That's where the magic happens.

01:03:01 Yeah.

01:03:02 And so we started using CPython more deliberately and more consciously and more of our documentation actually after the 2010 language and VM summits, which was when we got a bunch of the implementers of different VMs together.

01:03:20 So CPython, Jython, IronPython, PyPy, and basically talked about how can we essentially help alternative implementations to grow and reduce the barriers to use.

01:03:32 Turns out the biggest barrier to use is C extension API compatibility.

01:03:36 Yeah, that's definitely true.

01:03:38 Yeah.

01:03:38 And that's...

01:03:40 That's both the cursing and the blessing of it, right?

01:03:43 Because without that, the scientific computation stuff just wouldn't happen here.

01:03:47 Yeah.

01:03:48 And that's right.

01:03:48 And so that's been an ongoing challenge for a long time.

01:03:51 We're working on that.

01:03:52 Which gets us back to the how the interpreter and language itself evolve.

01:03:57 And so the core of this is the issue tracker at bugs.python.org because that's actually not just bugs.

01:04:05 It's also enhancement requests.

01:04:07 So one of the categories there is enhancement.

01:04:09 And so that becomes a case of, oh, we think this API should be added.

01:04:13 We think this API could use another flag on it.

01:04:16 And so that basically becomes a case of, hey, can we make people's lives a bit easier once they can change their baseline version of Python more easily?

01:04:25 One of the other things that's making this more desirable and have it happen more often is when we're getting more and more organizations now that are...

01:04:34 They have automated testing regimes that are sufficiently robust that updating to a new version of Python is no longer scary.

01:04:42 Right.

01:04:42 Things like talks and whatnot that can do multi-version targeting and stuff, right?

01:04:47 Yeah.

01:04:47 But also I think when Python 3.6 came out, I think Facebook had upgraded their production infrastructure in less than a week.

01:04:55 Yeah.

01:04:56 Yeah.

01:04:56 They were like, I remember the tweet.

01:04:58 I think it was from Jason Fried.

01:04:59 It was something to the effect of Python 3.6 came out on Tuesday.

01:05:02 We're now running on Python 3.6 in production.

01:05:05 And it was like Thursday that he sent that message out.

01:05:07 It was incredible.

01:05:08 Yeah.

01:05:08 Might have been Lukash maybe.

01:05:10 Oh, yeah, yeah.

01:05:10 It could have been actually.

01:05:11 Yeah, yeah.

01:05:11 It could have been.

01:05:12 And then someone replied going, I just thought about the sheer scale of that.

01:05:18 And that's nothing short of astonishing.

01:05:20 But the thing is with CI auto-deployment pipelines, it becomes that thing of you submit the PR that changes the base version in your Docker image.

01:05:31 And if CI passes, well, you have just as much confidence in that change as you do in any other change to your code base.

01:05:39 And so you're no longer talking about, oh, rebase means weeks or months of manual QA anymore.

01:05:45 A rebase just runs through the same CI as everything else.

01:05:49 Well, and you also have these sophisticated deployment techniques, I'm sure, places like Facebook use, where they don't necessarily switch everything.

01:05:56 Maybe 1% of the traffic hits the new one, and then 20, and then 100%.

01:06:00 You know, they can, like, gradually throw that out to the world before they hit millions of people and see what happens.

01:06:05 It's that thing of, because automated testing has gotten so much better, rebasing to new Python versions is easier.

01:06:13 Which means that they can more quickly get to the point of, we got the enhancement into the core.

01:06:19 That means we can get rid of our workaround for that capability not being there.

01:06:24 And the payoff for that is, like, within the couple of years that you can plan around.

01:06:30 You can say, okay, we'll do the hacky way now.

01:06:33 We'll get the proper fix into the core.

01:06:35 Then we can throw away the hack in several months' time.

01:06:38 That gets a lot more attractive.

01:06:40 And so, whereas in more traditional environments, you may be supporting old Python versions for 5, 8, 10 years.

01:06:48 Which kind of kills your incentive to say, oh, I need this in the standard library.

01:06:53 And you're going, well, no, I really don't.

01:06:55 I need it in a library that I can upgrade.

01:06:57 Yeah, you've got to be able to pip install that, because you're not changing the base runtime.

01:07:02 Yeah, very interesting.

01:07:03 So, getting back to the enhancement requests.

01:07:05 Filing the enhancement request is kind of the first step.

01:07:08 It's like, hey, that can be, so you can either do that directly, or you can post to the Python ideas mailing list and say,

01:07:14 hey, should I file an enhancement request for this?

01:07:16 How is it likely to be received?

01:07:18 And so, enhancement requests, it's easiest to get a good review for modules that actually have an active maintainer.

01:07:24 So, in the Python developer guide, we have a page called the experts index.

01:07:28 And essentially, the expert index, as core developers, we can add ourselves to that in particular areas.

01:07:36 And essentially, when we add ourselves, we're saying, I feel both confident and entitled to make design decisions about this module or about this part of the interpreter.

01:07:46 And quite often, the biggest problem that folks with enhancement requests face is that they're not being explicitly told no.

01:07:55 It's that nobody's telling them, yes, we're going to do that.

01:07:59 And quite often, the reason nobody's telling them yes is that nobody feels both confident and entitled to say, yes, that's a good idea.

01:08:05 Right, exactly.

01:08:06 Because it's sort of out of their little specialty.

01:08:09 And so, you know, it's just floating out there.

01:08:11 And so, they're just like going, oh, it might be a good idea.

01:08:13 Like, I think for a long time, there was a proposed improvement to MIME type handling in the standard library.

01:08:20 And it just kind of sat there forever because we didn't actually have any MIME experts on the current team.

01:08:28 And it was a huge change that combined.

01:08:32 It didn't clearly separate a massive refactoring of the MIME handling module from here is the bits that are actually wrong and need to be fixed.

01:08:41 And at that point, you just kind of end up in an impasse because you're just like going, well, we're not confident enough in the change to say yes to it.

01:08:48 But at the same time, we're pretty sure the status quo isn't right.

01:08:51 Right.

01:08:52 That's tough.

01:08:53 But yeah, I see it.

01:08:54 All right.

01:08:55 So, how about the language?

01:08:55 What if we want a new feature in the language?

01:08:57 Like, I want a triple slash that means something.

01:08:59 I want to put that in there.

01:09:01 Okay.

01:09:01 Go to the triple slash.

01:09:02 Okay.

01:09:03 So, that actually starts the same way, either on the Python ideas mailing list and that's like, basically, it's very hard to go wrong starting with the Python ideas mailing list.

01:09:14 You're just going, the main way people go wrong is they present their idea with the implication that it's an obviously good idea and we're clearly idiots for not having implemented it already.

01:09:26 That tends not to go over so well.

01:09:28 But so long as-

01:09:30 So, how you win friends and influence people, right?

01:09:31 So long as people come in with the genuine questioning mindset of, would this be a good idea or might it contain the seeds of a good idea, that tends to be a much better foundation for discussion.

01:09:45 Because either it will not be a good idea.

01:09:49 There's a comment along the lines of, like, the number of ways we can make Python worse is unbounded.

01:09:56 The number of ways we can make it better is a relatively limited subset.

01:10:00 Or is always better in a language.

01:10:02 I feel like there are some language designers, you know, thinking like Swift and the C# team, they seem to just be like, well, we have a job as a language designer, so we're going to continue to design the language.

01:10:13 And so, what features are we adding?

01:10:15 The shipping period is like, well, there don't necessarily need to be more features.

01:10:19 You're just making it worse now, right?

01:10:21 It's getting complicated.

01:10:22 So I really like that the Python is cautious about that.

01:10:25 Cognitive burden is something we think about a lot.

01:10:28 Because every time you add to an API or add more keywords or anything, you are necessarily making things harder to learn.

01:10:35 Especially if there's multiple ways to do the same thing.

01:10:37 But there's like a slightly better, newer way.

01:10:39 It just makes it confusing.

01:10:42 When the new way is just obviously better in every way, those are the ones you're really happy about.

01:10:47 Right.

01:10:47 Like f-strings, perhaps, maybe.

01:10:49 F-strings are exactly the example I was going to bring up.

01:10:51 Because they're not only the most concise way of formatting strings, but when you can use them, they're also the fastest.

01:10:59 And this is something not a lot of people realize, is that with both percent formatting and the .format method, Python has to interpret the formatting string at runtime.

01:11:08 So it means it has to scan through it, find all the fields, calculate the field substitutions, and it's all just kind of horrible and slow.

01:11:16 The beauty of f-strings is because they're syntax, you can get the compiler to help.

01:11:21 And so with f-strings, we do all of that thing of breaking it up into segments at compile time.

01:11:29 So in the bytecode or the PYC file, it's like already sort of partitioned it.

01:11:34 In the compiled form, all you have is an alternating sequence of text segment, format operation, text segment, format operation, text segment, format operation.

01:11:46 And so it can basically, all of that string passing code happens at compile time.

01:11:53 And so it's not only the nicest to write and the easiest to read, it's also the fastest.

01:11:59 That sounds like a win in future.

01:12:01 Yeah.

01:12:01 So it was like, cool.

01:12:02 And then what it means is that then all the other runtime formatting options, they're all still there and you still have all your different trade-offs between them for when you might want to use them.

01:12:10 But for straight up text formatting, you don't need them anymore.

01:12:12 And those kinds of ideas that kind of win on all the boxes and it's like, those are the ones most likely to get approved.

01:12:19 That basic enhancement proposal process of proposing things on Python ideas or on bungas.python.org.

01:12:25 Essentially, we have an escalation mechanism for those.

01:12:27 So if it fits in somebody's personal area of expertise, we may just approve it on the issue tracker and say, yep, that's a great idea.

01:12:33 It's an area I'm confident making decisions about.

01:12:36 I'll just approve it on my own authority and merge it.

01:12:40 So those are wonderful from a contributor experience point of view.

01:12:43 It's like, hey, I made a suggestion.

01:12:44 It was accepted.

01:12:45 Cool.

01:12:45 I made a proposed change to Python.

01:12:48 Those are kind of the exception rather than the most common case.

01:12:53 And what happens when that doesn't work is we may ask that a suggestion get taken to Python ideas if it's relatively vague or to Python dev if the suggestion is clear and the bit that's unclear is should we do it or not?

01:13:09 Right.

01:13:09 It needs debate.

01:13:10 So Python dev is mainly for yes or no type questions.

01:13:13 Python ideas is for I don't even know what the question is.

01:13:17 What should the question be is kind of a useful way of thinking of those two mailing lists.

01:13:23 But sometimes that's still not enough to resolve the question.

01:13:30 So it's an enhancement proposal process.

01:13:36 And that's also a useful thing.

01:13:43 And that's also a useful thing.

01:13:50 And so the idea of the process is to provide a document that summarizes the discussion, summarizes the tradeoffs that have been discussed, and nominates a particular person to make the final decision, which is basically yes or no, or let's put it off and reconsider it later.

01:14:12 And so by default, that's always Guido as the lead language designer.

01:14:17 But we have a delegation mechanism in place where any core developer can say, I'm happy to make the final decision on this and basically be responsible for the fallout.

01:14:31 Yeah.

01:14:31 And that can be wide-ranging.

01:14:33 Yeah.

01:14:34 Yeah.

01:14:34 Yeah.

01:14:34 Yeah.

01:14:35 That's interesting.

01:14:36 In a lot of cases, we also use the PEP process when we're like going, we're happy with the idea in principle, but there's a bunch of lower-level details to work out before it becomes actually implementable.

01:14:48 And in those cases, there's usually arbitrary design decisions that need to be made along the way.

01:14:53 And it's like, it doesn't really matter which way you go.

01:14:56 You just have to pick one and stick with it.

01:14:58 And so the BDFL and delegation process is useful for that as well because that's when you just say, look, if there's an arbitrary decision to be made, this is the person who makes it.

01:15:10 And the beauty of this is that it gives us a way to say yes to complex suggestions.

01:15:16 I think the record for getting a complex proposal through from proposal to implementation was something like eight days to add the matrix multiplication operator from the point where Nathaniel submitted the PEP to when it was actually implemented.

01:15:36 And that was an interesting one because that actually came out of Nathaniel researching the history of operator proposals to Python.

01:15:45 And it turned out nobody had ever actually suggested, let's just add a matrix multiplication operator.

01:15:52 They'd always just gone, they'd all be in a case of let's add the ability for users to define their own infix operators.

01:16:00 And they'd always failed on the basis of not adequately demonstrating the use case.

01:16:06 And then what Nathaniel did was he went and looked and talked to all the scientific community and said, what are we actually feeling pain for about?

01:16:17 And he went, well, it turns out we're really only feeling the pain for matrix multiplication because you need a way to spell element wise multiplication and you need a way to spell matrix multiplication.

01:16:29 And Python just simply didn't offer enough infix operators to let you do that.

01:16:35 Like you needed, so you had to choose and it was all very complicated.

01:16:40 Nathaniel really did his research, understood the problem, wrote the PEP and said, this is the problem.

01:16:48 This is what we want to do about it.

01:16:49 This is why we chose the simple we did.

01:16:51 And Guido went, sure.

01:16:53 Yeah, that's awesome.

01:16:55 That's a real blueprint for other people who want to do stuff like this.

01:16:58 It's still my favorite enhancement proposal this day because it was just a beautiful example of researching the problem, understanding the need and explaining it clearly to people that weren't in that environment and weren't regularly writing scientific software.

01:17:13 And one of the cute things was that it would have been even faster, except that there was a question in there about left associativity versus right associativity, where the description in the PEP was, we didn't ask for this because we assumed you'd say no.

01:17:31 And Guido went, well, no, no, no.

01:17:33 I might say yes if you have a valid use case for it.

01:17:37 And so it was like, oh, okay, we need to go discuss that on the scientific mailing list to figure out what we actually want then.

01:17:43 We'll get back to you.

01:17:44 Yeah.

01:17:45 And so there was an extra couple of days in there while I went and figured out, do we actually want that?

01:17:49 And then they came back and said, well, we don't actually want it.

01:17:52 That's cool.

01:17:53 That's a really nice, really nice example.

01:17:55 All right.

01:17:56 We're totally out of time, but I want to still get you with the last two questions, as I always do.

01:18:00 So if you're going to write some Python code, what editor do you use?

01:18:02 I use VS Code these days, and I used KDE's Kate editor for a very long time.

01:18:08 And it was only the VS Code has better Python plugins.

01:18:12 I think VS Code with the Python plugin is a really nice option.

01:18:15 So I definitely like it as well.

01:18:17 And notable PyPI package.

01:18:19 We've seen many.

01:18:20 Notable PyPI package.

01:18:21 Oh.

01:18:23 Maybe something that somebody doesn't know about.

01:18:25 Not the most popular, but like, oh, they should know about this thing that they probably don't.

01:18:29 Hang on.

01:18:29 That's a different question.

01:18:30 One that I'm really, really pleased that it exists is Nipi, N-I-P-I.

01:18:35 And that is neuroimaging data manipulation.

01:18:39 Like, I personally have no use case for it.

01:18:41 I am just really, really happy that it exists because...

01:18:44 Yeah, that's awesome.

01:18:45 That's just really cool.

01:18:47 So one that I really, really love using for writing command line applications is Click.

01:18:53 So Click for writing command line application.

01:18:55 I think a lot of people have heard of that one.

01:18:57 But it's still a really, really nice option.

01:18:58 Yeah, it's just as a way of writing command line applications that are easy to test, easy to compose, easy to write in the first place.

01:19:08 And the fact that it puts your command line passing right next to the function that implements a given subcommand is just a really nice way of doing things.

01:19:17 Very, very good.

01:19:18 I definitely like those recommendations.

01:19:19 All right.

01:19:20 So people are excited.

01:19:21 They have ideas for improving Python at many levels.

01:19:24 They want to get in touch with you.

01:19:25 What's the final call to action?

01:19:26 Not you in particular, but you as a group of people who make these decisions.

01:19:29 Python developer guide is a really worthwhile document to read.

01:19:34 We get a lot of common questions about how do I get started hacking on Python?

01:19:38 How do I make suggestions for changing the language?

01:19:40 And essentially, we try and make sure the developer guide answers those.

01:19:44 It does make it rather large, but it's a complicated project.

01:19:49 The other one I would point out is that posting suggestions to Python ideas is almost never the wrong thing to do.

01:19:56 Just make sure they're phrased as suggestions, not as demands.

01:19:59 Yeah.

01:20:01 You're asking a favor.

01:20:02 So approach the situation that way, right?

01:20:04 Yes, exactly.

01:20:05 Yeah.

01:20:06 Awesome.

01:20:06 All right.

01:20:06 Well, Nick, it's been really great to share your perspective on all these changes in Python.

01:20:11 It's been great.

01:20:12 Okay.

01:20:12 Awesome.

01:20:12 I've had a great time.

01:20:14 Sorry if I ramble a bit, but I really enjoyed talking about this stuff.

01:20:18 Yeah.

01:20:19 It's really interesting.

01:20:20 So thanks for taking the time.

01:20:21 Talk to you later.

01:20:22 Okay.

01:20:22 Thank you very much.

01:20:23 This has been another episode of Talk Python to Me.

01:20:27 Today's guest has been Nick Coghlan, and this episode has been brought to you by Linode and Datadog.

01:20:33 Linode is bulletproof hosting for whatever you're building with Python.

01:20:37 Get four months free at talkpython.fm/Linode.

01:20:42 That's L-I-N-O-D-E.

01:20:43 Datadog gives you visibility into the whole system running your code.

01:20:48 Visit talkpython.fm/Datadog and see what you've been missing.

01:20:52 Don't even throw in a free t-shirt for doing the tutorial.

01:20:54 Are you or a colleague trying to learn Python?

01:20:57 Have you tried books and videos that just left you bored by covering topics point by point?

01:21:02 Well, check out my online course, Python Jumpstart, by building 10 apps at talkpython.fm/course

01:21:08 to experience a more engaging way to learn Python.

01:21:11 And if you're looking for something a little more advanced, try my Write Pythonic code course at talkpython.fm/pythonic.

01:21:19 Be sure to subscribe to the show.

01:21:20 Open your favorite podcatcher and search for Python.

01:21:22 We should be right at the top.

01:21:24 You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm.

01:21:33 This is your host, Michael Kennedy.

01:21:35 Thanks so much for listening.

01:21:36 I really appreciate it.

01:21:37 Now get out there and write some Python code.

01:21:39 I'll see you next time.

01:21:59 Thank you.