#431: Visualizing CPython Release Process Transcript
00:00 Every year, Python has a new major release. This year, it's Python 3.12, and it'll come out on
00:04 October 2nd, 2023. That's four days from when this episode was published. There is quite a process
00:11 involved to test build and ship Python across the many platforms and channels. We have Seth
00:16 Michael Larson here to give us a detailed rundown on exactly what is involved in releasing CPython.
00:21 This is Talk Python to Me, episode 431, recorded September 28th, 2023.
00:27 I agree.
00:27 Welcome to Talk Python to Me, a weekly podcast on Python. This is your host, Michael Kennedy.
00:44 Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython,
00:49 both on fosstodon.org. Be careful with impersonating accounts on other instances. There are many.
00:55 Keep up with the show and listen to over seven years of past episodes at talkpython.fm.
01:00 We've started streaming most of our episodes live on YouTube. Subscribe to our YouTube channel over
01:06 at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.
01:11 This episode is brought to you by JetBrains, who encourage you to get work done with PyCharm.
01:19 Download your free trial of PyCharm Professional at talkpython.fm/done dash with dash PyCharm.
01:25 Hey there. Before we jump into talking about building CPython, I have a new course to tell you about.
01:31 HTMX plus Django. Full web apps hold the JavaScript.
01:35 This is an excellent new Django course written by Christopher Trudeau.
01:39 In just a few hours, you'll learn how to make your app much more interactive,
01:43 and the code even a little cleaner than pure Django.
01:46 You may be familiar with our HTMX plus Flask course I released a while ago,
01:52 and this one is its Django sister course.
01:54 And it is out now at talkpython.fm/HTMX dash Django.
01:59 The link is in your show notes.
02:01 If you Django and have been wanting to adopt HTMX, you should definitely give this course a look.
02:06 And a heartfelt thank you to everyone who has purchased one of our courses or library bundles.
02:11 It truly makes this show possible.
02:13 Now, on to that interview.
02:15 Hey, all. Hey, Seth.
02:17 Hello.
02:17 Awesome to have you here on the show.
02:20 Yeah, thanks for having me.
02:22 Absolutely. I'm looking forward to talking about the details, the steps of releasing CPython.
02:28 And for those of you who have not been watching and paying attention, by the time this comes out,
02:34 it'll probably have already happened.
02:35 I'm sure it's already happened.
02:37 By the time this video is up, though, it's not quite happening.
02:40 So as of the recording, next Monday, this whole process that we're going to talk about goes into action, right?
02:46 Releasing Python 3.11.
02:48 Yeah.
02:48 This, not 3.12.
02:50 This will have happened.
02:51 3.12.
02:52 I got to get used to it.
02:53 It's like, you know, in January, you always write the year on.
02:58 So 3.12, the new version, the still faster, faster CPython 3.12 with lots of cool features is going to be out.
03:06 And how does that happen, right?
03:08 There's a lot going on behind the scenes.
03:10 And you wrote a really cool blog post called Visualizing the CPython Release Process or something like that.
03:16 I'm getting off memory.
03:17 And it has a great diagram with a whole bunch of boxes.
03:20 And so we're going to go through and kind of just talk about how that happens and so on.
03:24 But before we get into it, maybe a quick intro for folks who don't know you yet.
03:27 Yeah.
03:28 So I'm Seth Larson.
03:30 Online, everywhere, I am Seth M. Larson.
03:33 That's my handle.
03:34 And I am the security developer in residence at the Python Software Foundation, the inaugural security developer in residence.
03:41 Hopefully for more, obviously.
03:42 But yeah, this whole role is funded through the OpenSSF Alpha Omega project specifically.
03:50 And it's kind of a project that's focusing on Alpha being really, really important projects that everyone agrees, yep, that's a really important project to secure.
03:59 So among them is Python and PyPI.
04:01 And then Omega is kind of like the long tail, right?
04:04 Like all the other projects.
04:05 Because there's open source is huge.
04:07 There's millions of different projects.
04:08 How can we secure those?
04:10 And so, yeah, this role exists because of them.
04:13 So I'm very thankful.
04:14 And it's mostly.
04:15 As are we all.
04:16 Right.
04:16 Speaking for the community.
04:17 This has been something.
04:18 Yeah.
04:19 There's been a lot of places where it's like, well, why doesn't someone take care of this?
04:22 Like, well, there's nobody whose sole job it is.
04:25 But like Python has been slowly rolling in these new roles, which is fantastic.
04:28 Like Lukash Lenga at the developer in residence.
04:31 You're the security developer in residence.
04:33 We have Mike Fiedler, who I spoke to about on PyPI security.
04:36 Like there's people.
04:38 There's developers working on stuff.
04:40 It's awesome.
04:40 There's something to be said.
04:42 Like having someone working on stuff full time, like you can get a lot done.
04:47 It's quite nice because we're so used to, you know, how the speed of open source.
04:52 Right.
04:52 Which is not.
04:53 I guess when someone says the speed of open source, they don't immediately think, oh, yeah,
04:56 that's going to be that's going to be fast and snappy because people have lives.
04:59 Sometimes it is.
05:00 Yeah.
05:00 Well, sometimes it is.
05:01 When things are brand new, they can have a lot of momentum.
05:05 But, you know, my personal view is like on all these software projects, there's these
05:09 little gnarly edges that is just as fun for no one.
05:12 Right.
05:13 And like, I want to add this new feature or I want to add async to this thing so people can
05:17 do this and like you can get so inspired, even if it's not your job and it's just your
05:21 project and really go on it.
05:22 But like, you know, really polishing out that little bit of documentation or or updating that
05:28 detour.
05:28 Like it's it's all the the fine polish, I guess, that I think you all are bringing to
05:33 Python, which is awesome.
05:34 Yeah.
05:35 There's tons of work.
05:36 That's like it really does need that amount of time investment and like a sustained time
05:42 investment.
05:42 So like when you're working with external groups or like, you know, developing processes and
05:47 stuff like that, like you just need you just need to be there a lot more than, you know,
05:51 weekends.
05:52 So it's good to have that full time presence.
05:55 Yeah.
05:56 Yeah.
05:56 I think it is, too.
05:57 Although it's absolutely mind blowing how much stuff gets done in traditional pure open
06:02 source.
06:02 It really is.
06:03 Honestly, it gives a whole new perspective to it.
06:06 Right.
06:06 It's like, oh, yeah.
06:07 You know, all of this is happening.
06:09 But, you know, almost all of the people that I interact with on a day to day basis are volunteers.
06:16 And so like I when I kind of wrote about this a little bit in my opening when I announced
06:20 myself as a security developer in residence, but it's like you need to you can't have expectations
06:26 and like assign people work.
06:28 You know what I mean?
06:28 Like this is not a traditional work environment where you can just tell people what to do.
06:32 And, you know, it it's a certain way of working.
06:36 And I'm used to it because I am an open source maintainer myself.
06:39 So it makes a lot of sense to me and I feel very comfortable with it.
06:43 But it's very different than a regular security developer job would be.
06:46 Would you say fewer TPS reports?
06:48 Yeah, a few fewer.
06:51 Yeah, yeah.
06:51 Just a couple fewer.
06:53 Hey, you since you brought it up, maybe we'll come back to this if we have time at the end.
06:57 But, you know, you've got some pretty notable projects that you're working on here.
07:01 Like give a quick shout out to some of your open source work that predates.
07:04 Yeah.
07:04 So I am the lead maintainer of your lib3, which is one of the most downloaded Python packages
07:10 on PyPI.
07:10 I also help maintain requests.
07:13 It's a user of your lib3.
07:15 And then in addition to that, I have a library, which I quite love, which is called TrustStore,
07:19 which is being adopted by a lot of package managers right now, like pip and Conda and PDM,
07:25 which allows those those packages to use system certificates as opposed to something like Certify
07:33 so that you can take advantage of all of the benefits that you get for using a system trust store for HTTPS as opposed to like a static bundle of certificates.
07:42 Because operating systems are actually constantly checking, updating all of these things.
07:47 It's just a little bit better.
07:48 Some certificate store or some certificate authority turns out to be shady and they get revoked and.
07:54 Or just not have not have the best practices.
07:58 And so, yeah, just like, OK, we're going to pull that one.
08:01 But we're even pulling us.
08:02 Yeah, they even get stolen.
08:04 Right.
08:04 Sometimes these certificates get stolen.
08:06 They get stolen.
08:07 You know, there's all sorts of things that can go wrong.
08:10 Thankfully, it doesn't go wrong very often.
08:12 Otherwise, it would be a lot bigger of a problem.
08:14 But yeah, it does happen every once in a while.
08:17 Yeah, that's cool.
08:17 Yeah.
08:18 And you were on Python Bytes, my other podcast I do with Brian to talk about trust store, which is cool.
08:23 All right.
08:23 Back to this.
08:24 So tell us a bit about this developer, security developer in residence role.
08:29 Like people were like, oh, this maybe they've heard of it or is maybe even new to them.
08:33 But like, what is your what is your job here in the PSF space?
08:37 Yeah, it's it's it's kind of funny talking about my role, too, because people see security through so many different lenses.
08:44 And this is even like outside of talking to people online that are in the open source space.
08:48 So whenever I talk about this job, people will say, oh, so you do IT security for the PSF.
08:54 I'm like, well, no, I don't do that.
08:55 And then they'll say, oh, so you secure.
08:57 Yeah, right.
08:59 I'm in.
08:59 No, no, no.
09:00 So I don't do that.
09:02 I don't do IT security for the PSF.
09:04 And then the next question they usually ask is like, oh, so you make it so that Python is more secure, like the code of Python is more secure.
09:13 And I'm like, not really.
09:16 Honestly, I don't do that either.
09:17 And what I really do is try my best to make it so that bad stuff doesn't happen to Python users.
09:24 And so, you know, obviously those things are important, like securing Python, the actual code itself.
09:30 But on like the scale factor, maybe those are a little bit less important than some things like making sure that when you download something from Python.org, it is the correct thing.
09:42 Right.
09:42 Making sure that when a release is happening for Python, nothing can subvert that release and get injected in and then distributed to everyone.
09:51 And then it goes beyond Python when you start thinking about like the packaging space.
09:55 So like making sure that HIP release process is good, making sure that the all of these like tools and stuff that are using all of these binary libraries.
10:06 How can you be sure that those libraries that are bundled along with them don't have vulnerabilities?
10:11 And then a lot of things around like process.
10:13 So vulnerability management and making sure that, you know, vulnerabilities that are discovered in Python and reported to us, they actually get to the end of the process where they're fixed and released.
10:24 There's just a whole bunch of things like that where it's more of like a scale and safety and like making sure that my effort goes towards things that are going to keep on churning as opposed to things that are like spot fixes.
10:36 Right.
10:36 Because sure.
10:37 Spot fixes, they're important, but they don't have the scale and they don't have that like keep on giving you the value aspect of them.
10:45 Yeah.
10:46 And the consequence of a lot of what you described sounds like supply chain type issues.
10:51 The consequence of that is so bad.
10:54 It doesn't matter how patched your computer is or how good your firewall is.
10:58 If CPython itself ships with a virus.
11:01 Right.
11:02 Right.
11:02 Yeah.
11:03 Or somebody snuck it in right at the build.
11:05 Yeah.
11:05 Yeah.
11:05 Right.
11:06 Right.
11:06 Right.
11:06 And so it's good that you focus on those.
11:09 Nobody wants that.
11:10 Well, almost nobody wants that.
11:11 Very few people want this.
11:13 So those are not good people.
11:14 We don't want them to want this.
11:15 We don't.
11:16 We don't take them into consideration for their use cases.
11:19 We sure don't.
11:21 Awesome.
11:21 And let's see.
11:23 I believe you're on.
11:24 Well, when I read the article, you're on week nine.
11:26 So you're on week 10.
11:27 Is that where we are?
11:28 I think I'm on week 12 at this point.
11:30 Oh, man.
11:30 Time flies.
11:31 Yeah.
11:31 I know.
11:32 I'm actually drafting up like a quarterly review.
11:35 And it's a big thing.
11:36 So, yeah.
11:37 Okay.
11:38 See, a quick question from the audience here on the live stream.
11:40 You know, Lewis asks, is there a visibility or will there be a public visibility for CVEs in packages?
11:47 That is an excellent question.
11:49 It is a good question.
11:50 And I bring this up because didn't the PSF just get certified as like a CVE authority or whatever that is?
11:56 Like ability to create CVEs?
11:57 I don't know.
11:58 Yeah, I can I can talk so much about this.
12:00 Right.
12:00 So there there's a whole bunch of stuff happening in that space.
12:03 So I'll start off with stuff that I did not personally do, but I'm helping maintain now, which is there is an advisory database called if you go on GitHub, it's GitHub.com slash PyPA slash advisory database with a dash in between advisory and database.
12:21 And that database is essentially trying to categorize all of the CVEs and what versions they affect for every single package on the Python package index, which is an impossible task.
12:35 But so we're trying our best there.
12:37 But yeah, so one of the packages, no problem.
12:40 No problem.
12:41 Right.
12:41 Like that's a that's a manual task right there.
12:43 Yeah.
12:43 No.
12:44 So what this actually does is for the bulk of it, what it will do is it'll go out and it'll pull the CVE feed from NBD and MITRE.
12:51 And it will parse it and try to find references to Python packages and versions inside of CVEs.
13:00 So obviously that's sometimes there's false positives, but a lot of the times there's just it just works out OK.
13:04 And there's a little bit of triage involved.
13:07 There's a little bit of manual submission of this, too.
13:09 But yeah, so tools like pip Audit, which is a tool that I love so much, uses this database in addition to other scanning tools like any scanning tool can use this database.
13:21 But yeah, this is the canonical location for vulnerabilities affecting Python packages.
13:27 So for Python itself, the Python Software Foundation just recently, like a few weeks ago, announced that we are now a CVE numbering authority, which means that we are we kind of have like a scope of just Python and pip right now.
13:44 So those are the only two projects that we're emitting CVEs for.
13:46 And what that means we can do is it means that people have to submit reports to us, which is good because then they can't submit them to other places and get CVEs that way.
13:55 It can be kind of frustrating because in the past, sometimes that would happen.
13:58 And then we wouldn't know about the vulnerability until it got published as a CVE.
14:02 And we'd be like, what just happened?
14:03 Like one of the big deals is you're supposed to be told in an ideal world, you're supposed to be given like a come knock in the back door like that WebP code.
14:13 We're going to need to talk about the WebP code before.
14:16 Yeah.
14:16 Oh, gosh.
14:17 Don't get me started on WebP.
14:18 Don't get me started on WebP.
14:19 This portion of Talk Python to Me is brought to you by JetBrains and PyCharm.
14:25 Are you a data scientist or a web developer looking to take your projects to the next level?
14:30 Well, I have the perfect tool for you, PyCharm.
14:33 PyCharm is a powerful integrated development environment that empowers developers and data scientists like us to write clean and efficient code with ease.
14:43 Whether you're analyzing complex data sets or building dynamic web applications, PyCharm has got you covered.
14:48 With its intuitive interface and robust features, you can boost your productivity and bring your ideas to life faster than ever before.
14:56 For data scientists, PyCharm offers seamless integration with popular libraries like NumPy, Pandas, and Matplotlib.
15:02 You can explore, visualize, and manipulate data effortlessly, unlocking valuable insights with just a few lines of code.
15:10 And for us web developers, PyCharm provides a rich set of tools to streamline your workflow.
15:14 From intelligent code completion to advanced debugging capabilities, PyCharm helps you write clean, scalable code that powers stunning web applications.
15:23 Plus, PyCharm's support for popular frameworks like Django, FastAPI, and React make it a breeze to build and deploy your web projects.
15:31 It's time to say goodbye to tedious configuration and hello to rapid development.
15:36 But wait, there's more.
15:37 With PyCharm, you get even more advanced features like remote development, database integration, and version control, ensuring your projects stay organized and secure.
15:46 So whether you're diving into data science or shaping the future of the web, PyCharm is your go-to tool.
15:51 Join me and try PyCharm today.
15:54 Just visit talkpython.fm/done-with-pycharm, links in your show notes, and experience the power of PyCharm firsthand for three months free.
16:04 PyCharm.
16:05 It's how I get work done.
16:10 The real, like, the ideal process is someone would Google, like, Python and security, or, like, Python and reported vulnerability, and then they would find these, like, really lovely instructions that someone else wrote up to say, just send an email to security at python.org with whatever information you have, and then we triage that.
16:28 And if we decided it's a vulnerability, we would ask for a CVE on your behalf and, like, credit you and all this stuff, and then fix it.
16:35 And then the release and publication of the CVE would happen at the same time.
16:40 So that's what we call coordinated vulnerability disclosure.
16:43 And so instead of, like, oh, there's a vulnerability, but there's no fix available for that vulnerability, like, that's what you want to avoid.
16:49 Because it causes panic, and in theory, it could cause exposure, right, where there's some dwell time in between when people, or it doesn't minimize the dwell time between when people are able to patch and when the vulnerability is known to potential attackers.
17:05 Right.
17:06 There's already a latency boom from when it's announced to somebody actually fixing it.
17:10 But if when you hear it, you can't actually fix it for two more weeks because it's not released.
17:14 Well, that just gives people two weeks to try to dissect, like, what exactly do they mean by there's this image problem?
17:19 Exactly.
17:20 Exactly.
17:21 Oh, yeah.
17:21 I'm going to look at my image, yeah.
17:22 We're going to start talking about LibWebP, actually.
17:25 Yeah, the...
17:26 No, I'm just using it.
17:27 Like, it's the most recent example, because last night there was a...
17:30 And it's really relevant to CVEs because there was some drama that Apple patched some of their code.
17:36 When Google announced it, it looked like it was only applied to Chrome, but it actually applied to anything that used WebP, which, looking at the updates, is like, everything.
17:43 Everything on my computer needs to be updated because of it.
17:46 You know, like, that mismatch was, like, caused a lot of drama.
17:50 I definitely agree.
17:51 Yeah, there was, like, a week of time in between when that CVE just mentioned Chrome and Apple, and then when it actually ended up mentioning LibWebP.
18:02 And then right when it mentioned LibWebP, I'm like, oh, I know a lot of Python packages use that.
18:06 And so I then went out and go and talk to those people.
18:09 Oh, interesting.
18:09 And they fixed those things now.
18:10 So.
18:11 Yeah, that's great.
18:12 Another good reason to stay on top of dependency management and stuff, right?
18:16 Like tools.
18:17 One of the things I think we're coming to sort of understand and manage better in the Python space is not your dependencies, but the transitive closure of your dependencies.
18:28 Right?
18:28 So I might use, let me just say, I guess, what if I use Pillow to work with images, but Pillow itself uses some other library that it pip installs that itself might have some kind of image code.
18:39 If I say I use Pillow, oh, I need to update it.
18:41 pip install --upgrade.
18:44 Pillow doesn't upgrade its dependencies and those dependencies of the, right?
18:49 So like, but that could be where a lot of silence issues live.
18:53 And so things like poetry, flit, pip-tools, there's a lot of tools coming on that I think are gaining popularity that treat not just what you directly wanted to pip install, but its friends and its dependencies.
19:04 Yeah, definitely.
19:05 What do you think about that from a security perspective?
19:07 Yes.
19:07 So having the most important thing from like a kind of dependency management side of things is having those lock files and having them have hashes.
19:17 Like it's, it's less of a problem now.
19:21 Like I, it likely won't come to fruition that those hashes end up being important, but luckily lots of lock file formats do generate hashes anyways, which is great.
19:29 So that subverts attacks against PyPI partially.
19:32 We're in the middle DNS madness.
19:35 Right.
19:36 Yeah.
19:36 Like if somehow you've been tricked into using a bad certificate or something, right?
19:42 Like it, it saves you from a lot of stuff like that.
19:44 So having hashes, they're basically free.
19:46 They're great.
19:47 Just use those.
19:48 The other side of it is running a vulnerability checker against those dependencies.
19:53 Cause then you'll, or either that or having a tool that's doing that for you.
19:57 So like depend a bot is a tool that does that for you.
20:00 It will check vulnerability databases and then give you like a security notification or a PR to say,
20:05 Hey, this is affected by a medium severity vulnerability, or you can run them your own via pip audit with.
20:12 It can either check an already installed environment or it can check like a requirements file.
20:17 I don't know the full extent of all the different file types that I can check.
20:21 I know that it works on requirements.txt, but yeah, it, it'll check those files for known vulnerabilities and all of the transitive dependencies, just everything that's available.
20:30 And then from there, it'll tell you, Hey, these are the things that you need to fix.
20:34 Here are the fixed versions.
20:36 Like try upgrading to at least this, doesn't mean that it's easy.
20:39 advisory database you talked about.
20:41 It does go back to that advisory database.
20:43 Yeah.
20:44 And it also can hook into OSV, dot dev, which is kind of like this global, aggregation of tons of different vulnerability databases.
20:53 So for example, we have one for Python that I pointed out, but then like rust has one.
20:59 Debian has one.
21:00 what is it?
21:01 Rocky Linux, I think has one.
21:03 Like there's just a whole bunch of them and you can kind of interface with this database using the same API and it's, it's kind of nice.
21:09 So, yeah.
21:10 So making this a security show, we just talked to Mike Fiedler a little while ago about that as well.
21:17 Diving into it.
21:18 I mean, we'll come back to the release process, but I do want to ask you about this is one of the challenges I see is like I got to depend about a warning.
21:24 Luckily it was for basically the requirements dash dev side, not the true requirement.
21:31 You know, like I'm going to use a bunch more tools to do like Jupyter stuff on my data, but I'm really just running a website and there's like a much smaller, a smaller set of things on there.
21:40 But there was a vulnerability report for something, some package that was in my requirements, for the dev verge, the dev setup.
21:49 However, when I tried to go update it, one other package that is the reason that was there said, no, it has to be less than something.
21:57 And that something was less than the fix.
22:00 And I'm like, how do I square these two things?
22:03 Like I can pip install it, but then it keeps complaining that you're, you're running the wrong version.
22:07 Like, no, I'm not.
22:08 The other one has a remote CVE and, you know, a remote code execution in it.
22:12 And I'm like, I really don't want that.
22:14 I mean, not a server.
22:15 So it's probably fine, but I really don't want that.
22:17 No, I'm, I don't care if it's slightly more unstable.
22:20 I want the new one.
22:21 Give it to me.
22:22 Right.
22:23 And that's, that's a bit of a challenge.
22:24 I think, when you say like, I've got 50 dependencies and I've got to somehow square, I need to get a brand new one right away.
22:32 What are your thoughts on that?
22:32 Yeah, this, this is a super tough problem.
22:35 And I think it's one that open source will be coming to reckon with even more now, right?
22:40 Because we're kind of at a, in a place now where there's a few really important libraries, which are kind of moving something that is very extensively and widely used to be not maintained anymore.
22:51 For example, open SSL, open SSL one or 1.1.1, which is extremely widely used, is no longer maintained.
23:01 There's going to be no more security patches for that.
23:03 And it's, it's going to be tough because my guess is that there's also just a lot of software that's not built for open SSL three being the, the, the open SSL of choice.
23:13 Right.
23:13 Luckily, Python is not in that situation, but it's, it's certainly like a thing that other pieces of software might have to deal with.
23:20 in terms of Python packaging, this is kind of why the guidance on don't specify an upper bound, unless you know that that upper bound like makes sense.
23:32 And like, so like backwards incompatibility, like, yeah, maybe you want to say, okay, if this is version 2.1. whatever.
23:39 And it's some ver, you can say, okay, anything less than three is probably fine.
23:43 but even that, right.
23:45 If you, if that package goes unmaintained.
23:47 Yeah, this one from two to three and its main version, right.
23:49 It's, it's major version.
23:50 So.
23:51 Right.
23:51 Yeah.
23:52 So this is, this is a difficulty.
23:54 Yeah.
23:55 There's a lot of packages that set hard upper bounds.
23:57 And I just, I, I don't know why, because I, maybe it, it's going to break it, but I feel like it's speculative.
24:04 Like, ah, we should probably not let a major version run, but you know, that, that leads into trouble.
24:09 Right.
24:09 Like, even if you just want to use some other library that says greater than or equal to three, and this one says less than three.
24:14 You're like, great.
24:15 Now, what do I do?
24:16 Yeah.
24:16 I think that the toughest thing is that you can't go back and then like amend it.
24:19 yeah, that's the hardest part is because.
24:22 So I, I will speak very, very highly of all of these, anything that can make pip automatically get the right thing is just magical is so good.
24:33 And so people like that.
24:35 And I totally get why people are doing these upper bound, sorts of things, but yeah, it does have the issues that you, that you talked about.
24:43 But yeah, there's something to say about how great the, those bound checks are.
24:48 I wish there was a way that you could add something like some sort of guidance to pip to say after the fact, like, okay, I said this was open, but now I'm going to go ahead and tell you that like, this definitely doesn't work with this version.
25:00 Maybe that's something that can get recommended, but yeah, it's just so nice when it does work out.
25:05 Yeah.
25:05 Yeah.
25:05 I, I can relate to Frank here who writes with 1.3,000 or 1,300 dependencies in our project because of this dependencies are at all times.
25:15 Yeah.
25:16 I have on Talk Python Training, the courses website.
25:19 I think there's 248 packages.
25:21 If you look at what's installed and then the dependencies, and then you try to throw in the data sciencey stuff.
25:26 And it's just like, there's a 50, 50 chance that I can actually install everything.
25:30 Cause like half the time, something has got a less than, something's got a greater than.
25:34 And I'm like, Oh my God, as you upgrade, right?
25:36 Like at any point in time, it's deterministic, but you're over at over the evolution of it.
25:41 Right.
25:41 Yeah.
25:41 No, it's, it's tough.
25:42 It's tough when you start getting to that number.
25:44 Like there's just so many intertangled things.
25:47 There are.
25:47 And like, the reason I bring this up now is it's, it's a hassle when like, I want to use this library and that library and somehow I've got to like juggle things.
25:54 So they work, but it's concerning when you get a depend about alert that says there's a remote code execution.
26:00 You're going to need to deal with it and your dependencies won't let you upgrade.
26:04 You know what I mean?
26:05 Like that's, that's why I bring this up.
26:06 It's, it's definitely concerning.
26:08 Yeah.
26:08 Another thing that you can do, and this is, I feel like they don't talk about this as much with depend about is that there are ways.
26:14 to mitigate vulnerabilities that doesn't necessarily have to come from upgrading.
26:19 Like I'm always going to say upgrading your dependency is the best way to go forward because then you know that there's no chance of at least that vulnerability in particular being exploited.
26:28 But if it's something like remote code execution, but it's this one component, right?
26:33 As long as you're not using that component, then you're fine.
26:35 Or if it's remote code execution, when you get some sort of input, whatever, right.
26:40 That's, that's malformed or whatever.
26:42 And you're like, okay, well that component's not exposed to the network.
26:46 It's, I only use it internally.
26:47 So there is something to say.
26:50 It's not even a server.
26:51 It doesn't open a port, right?
26:52 It's just a thing I run on my computer.
26:54 It's like, it probably doesn't matter.
26:56 Exactly.
26:56 Yeah.
26:57 And that's, that's kind of the hard part.
26:58 I'm the one typing into it.
26:59 And it's like, but if it's in your website, then all of a sudden it gets a little trickier, right?
27:03 Then it gets dicier.
27:04 Yeah.
27:04 Definitely.
27:05 Yeah.
27:05 The stakes are higher when you attach the internet to anything.
27:07 But when it's like a local script, it's probably going to be okay.
27:10 Just look at your log file.
27:10 Just tell your log file.
27:12 And it's request, request, hack, hack, request, attempted hack, request.
27:16 It's just nonstop.
27:16 WP login.
27:17 Yes.
27:17 Yeah.
27:18 WP admin.
27:19 All them things.
27:20 Yeah.
27:20 It's usually, luckily it's WP something or something most of the time.
27:24 So you're good.
27:24 You're safe against 90%.
27:26 We're all good.
27:27 As long as you don't return a 200 to any of those, like your traffic will remain low.
27:31 Yeah.
27:32 Absolutely.
27:32 All right.
27:34 So that's what you're doing.
27:36 You're focusing on these kinds of things.
27:37 That's really, really awesome.
27:38 And thanks for giving us an even more detailed look than I expected into it.
27:42 Yeah.
27:44 Let's talk about your, let's talk about, you know, the timely thing.
27:48 I mean, security is always timely, but the CPython, let me see if I get this right.
27:52 312 release coming up here on Monday.
27:54 Dang.
27:55 He's got it.
27:56 Amazing.
27:56 I got it, man.
27:57 I'm ready.
27:58 I'm ready for this new, new world.
27:59 So there is a PEP and let's see who, who put this together.
28:03 This is Barry Warsaw and Guido.
28:05 And it is PEP 101.
28:07 I love it.
28:08 PEP 101.
28:09 The one you need, right?
28:10 Yeah.
28:11 What a perfect number.
28:12 But it's, it's the, the title is doing Python releases 101.
28:16 That's kind of a circular.
28:18 I love it.
28:19 And it just talks about all the steps and it's kind of a manual of now you do this and now
28:25 you do that.
28:25 But it's just a really long doc.
28:28 I love the stop, stop, stop, stop, stop.
28:31 There's like, there's like all caps all across.
28:35 Stop, stop, stop, stop, stop.
28:35 Did you do this right?
28:37 Right?
28:37 Yep.
28:38 That you can tell, you can always tell when there were problems in the past in a checklist,
28:43 right?
28:43 Like if there ever was something that happened incorrectly, it's, it's very obvious.
28:47 And so, yeah, the stop, stop, stop.
28:49 I feel that that, I don't personally know why that's there, but I do laugh every time
28:55 I see it.
28:55 So.
28:56 Yeah.
28:56 Let's see.
28:58 Please note how PEP 101 replaces PEP 102.
29:00 Time traveling was invented there.
29:02 What is the.
29:03 Yeah.
29:04 At the very top.
29:04 It's, I wanted to comment on that too.
29:06 It said replaces PEP 102.
29:08 And I, I wonder if it was more like a, oh, we should just use one-on-one sort of situation.
29:14 Well, I think this is doing micro releases.
29:16 Right.
29:17 Is this one maybe more like full, like the full deal?
29:20 I don't know.
29:21 Yeah.
29:21 I think security releases are, they're slightly different.
29:24 We don't actually build binary installers for security releases.
29:28 We only build source builds, like source tarballs for micro releases.
29:33 And so maybe that was the distinction in 102.
29:35 Sure.
29:36 Okay.
29:37 So this has existed since 2001, but you got to really dig into it.
29:43 And you know, there's some interesting stuff there, but picture a thousand words, all these
29:47 things, you put together a really nice graphic here and somehow you got this in like HTML.
29:52 I'm impressed.
29:53 Yeah.
29:54 That's embedded SVG.
29:55 I, I always use draw.io for my, my diagrams.
30:00 I take pride in my diagrams.
30:01 Yeah.
30:02 It looks good.
30:03 All right.
30:04 So there are 10 steps.
30:06 Some of the steps have ABCs in them.
30:08 So, and you know, just for people who pull up this picture and obviously being a podcast,
30:15 it's hard to see the picture.
30:16 If you're not watching the live stream, I will put a link to the article, which has the picture
30:22 in there.
30:22 And if I remember right, I'll even put it as the chapter art.
30:25 We'll see about that.
30:26 but there's a legend in here.
30:28 So you've got like different types of things.
30:30 You've got human actors, which are golden release artifact, which is blue, a source of risk, which
30:36 is where you're paying extra attention and the start and end state.
30:40 And so, the colors mean stuff here and the legends at the bottom, but yeah, let's, you
30:45 know, start with number one.
30:46 What happens here?
30:47 Yeah.
30:47 So starting with number one, and I'm actually going to increase the size of my screen as
30:52 well, because I'm going to pull it up separately myself.
30:54 Yeah.
30:55 There's a lot going on here.
30:57 There's a lot going on here.
30:59 You need some, you need some manager control plus on it.
31:01 Yep.
31:02 Okay.
31:03 All right.
31:03 So in the very beginning, what happens is essentially the release manager decides it's
31:09 time to make a release.
31:10 It'll be common source right now.
31:12 So we're, we're deciding, okay, we're going to make a release.
31:15 And so something that happens even before this is we kind of talk amongst all of the other,
31:21 it says release managers in there, like windows release manager, macOS release manager.
31:24 They kind of flip between that language and experts.
31:27 So like windows expert, macOS expert.
31:30 Someone who understands this weird windows things that can make a MSI installer.
31:34 The people.
31:34 Exactly.
31:34 What do you do on Mac?
31:36 I don't even know how to build that.
31:37 Right.
31:37 That person.
31:38 Yeah.
31:38 Yeah.
31:39 Talk to, talk to them, get, get their, get their buy-in that indeed they're going to be
31:43 around when you decide to do this, because this is kind of like a big coordinated thing.
31:47 Everyone has to be around to do their part.
31:50 And then if that is the case, then the release branch gets frozen.
31:54 So in this case, that release branch would be 312.
31:57 And so that's done by essentially putting a block on all future like pushes to the branch,
32:03 like in GitHub, you use like branch protection.
32:06 And that's basically to make it so that no one can merge pull requests anymore so that you
32:11 have something that's stable that you can kind of work off of.
32:14 And then after that, you'd move on to step two, which is the actual release manager will
32:19 update their fork of the repo with whatever is on that branch.
32:23 So like the 312 branch will get updated and then you'll pull that branch locally now on
32:29 step three.
32:30 So now you're on the actual release manager's machine.
32:33 And so like I kind of have like gray boxes drawn kind of throughout.
32:37 So those are kind of like security boundaries a little bit.
32:40 The top most one is the GitHub like organization of Python.
32:45 And then the middle, the tiny one that has the fork repo, that's the GitHub user for the
32:51 release manager in particular.
32:52 And then the furthest left, the gigantic one is the release manager's actual computer.
32:56 It's not documented anywhere.
32:59 It's just how I was thinking about it.
33:01 But yeah, so then there's this thing called release tool.
33:03 And so release tool is basically just a Python script.
33:06 It's on GitHub.
33:07 If you go to github.com/Python slash release dash tool, it's basically just a script that
33:13 runs and then does all of the stuff that's needed to build the source distribution and documentation
33:20 of a new Python release.
33:22 And so you run that.
33:23 And as a result, you kind of get like a huge amount of code that needs to be committed.
33:29 And so there's a whole bunch of different little tools that kind of get run in addition to
33:33 that.
33:33 So things like blurb, Sphinx gets run as a part of that.
33:37 Yeah, you need a whole lot of dependencies installed.
33:41 So you like you need like LaTeX, you need a whole bunch of stuff installed for this to actually
33:45 work.
33:45 Do you make sure you've got like the right, right compiler?
33:50 Is there a lot of concern about that?
33:52 Like, is it clang or LLVM or GCC or, you know, I believe it's GCC.
33:58 I believe it's GCC.
33:59 You know, I didn't dig into that exact point, but I'm pretty sure it is GCC.
34:02 But I feel like that's another part that you like that choice matters, right?
34:06 Like you might get different outputs.
34:07 You might discover bugs that weren't there.
34:09 So you got to kind of get that gray box all set up as well, right?
34:13 Yeah, definitely.
34:14 And so the fact that it's on someone's machine, I would say reproducibility is quite tough in
34:20 those sorts of situations, right?
34:22 Like if someone were to build a release of Python and then a day goes by, like I guarantee you
34:28 that if you try to do it again with the exact same starting input, it would be potentially
34:34 different, right?
34:35 And so that's actually a problem from a security perspective for reproducibility, because it means
34:40 that someone else can't verify that what you did is correct, right?
34:46 Like the hash might change or something like that, right?
34:48 Hashes might change or if there's like somehow the time is getting embedded into a binary
34:53 somewhere like that.
34:55 There's just a whole bunch of different ways that reproducibility could be different between
34:59 two different runs.
35:00 Yeah.
35:01 And so that after the release tool happens, you basically get handed a couple of source
35:07 tarballs that just have everything in them.
35:09 And then you also get a bunch of get like changes to the repository.
35:13 So these are things like the change log has been generated from all of the blurb notes.
35:18 And so what you do is then you take all of those changes and you create a big commit that
35:23 says this is the release of CPython 3.2, 3.12.0.
35:28 And then you tag it and you push that to the fork.
35:32 You don't push that to the main repository yet.
35:34 And so the reason you don't push that to the main repository, and this is actually something
35:37 that this phenomena is, is an interesting one, especially because a lot of security build
35:44 tooling doesn't take this work workflow into account, which is you don't push the tag to
35:52 the origin until the very end.
35:55 A lot of security tooling like salsa and all of these things will kind of like they won't
36:01 assume because you can you can do it without tags, but like they'll have features that are
36:06 Oh, we'll capture the get tag in the output of like this, like document that says that your
36:12 builders came from this repository, this set of tooling, whatever.
36:15 Right.
36:16 But a lot of projects actually wait on the tag until the very end because they want things
36:21 to be repeatable without causing confusion.
36:24 So if something went wrong throughout this whole release process, you wouldn't want there to
36:29 be a tag on the main repo that is now confusing people.
36:32 Right.
36:32 Because those tags cause notifications.
36:34 I'm going to get the code.
36:35 You're like, actually, no, the Mac build wouldn't build.
36:37 We're going to fix that or whatever.
36:39 Right.
36:39 Right.
36:40 Exactly.
36:41 And so this whole thing is like very it's a common thing.
36:45 So pip actually has this exact same workflow where they do everything in the release until
36:50 the very end and then they push the tag to the repo because they know that if they push
36:54 that tag and it needs to change, it's going to cause confusion.
36:57 So, yeah, that kind of is like step five in there.
37:01 It pushes that tag.
37:02 And so that tag is highlighted red as a source of risk because that is a fork of CPython and
37:09 tags tags are not in themselves verifiable because anyone can push a tag.
37:16 If you if you're just looking at just the name of the tag, anyone can write a tag if you have
37:22 right access to a get repository or a GitHub repository.
37:24 And so if you push a tag and say someone has access to your account, someone could move that
37:31 tag to a different commit, completely different commit.
37:33 And if it were pulled in that time and no one decided to like check something else.
37:38 So like the commit hash, for example, you would be like it would be able to inject code into the
37:44 process.
37:44 Right.
37:44 Right.
37:44 That's as far as we know, that has not happened.
37:46 Yeah, that is bad.
37:47 It is bad to inject arbitrary.
37:49 Someone else's arbitrary code into Python directly.
37:52 And it's also it's actually doubly bad because there's there's different degrees of bad in
37:57 term like in supply chain security.
37:59 One degree of bad is like if you're able to do bad stuff, but then people notice.
38:04 Right.
38:05 Or like people can see what happened.
38:07 And so this is kind of like degrees of bad of you can inject code and then you can clean
38:11 up after yourself afterwards.
38:13 Right.
38:13 Because you can tag back.
38:14 You know, put the tag back.
38:16 You have right access.
38:17 Right.
38:17 And so that that sort of like put the tag back or being able to circumvent it in that
38:22 way is not good.
38:24 Yeah.
38:25 Don't give people ideas.
38:26 These are not new ideas.
38:29 So I feel less bad talking about these.
38:31 Yeah.
38:32 Another thing in this article is actually like tying every step of the release process back
38:38 to a known attack that has succeeded against another project.
38:42 So like SolarWinds was about build like release artifact poisoning, essentially.
38:48 Right.
38:48 You're like taking advantage of the fact that these release artifacts are being distributed
38:53 and, you know, they're signed.
38:55 Everything looks good, but you've gotten code injected into the actual artifact itself and
39:01 it didn't get noticed.
39:01 Right.
39:02 And so like that sort of attack could happen against Python if if there's not mitigations
39:07 against it.
39:07 Right.
39:08 And so, yeah, and then the next step is six.
39:10 And so at this point, the release manager has sourced tarballs from the result of just
39:15 running the release tool.
39:16 And so they're kind of just waiting.
39:18 They're sitting around waiting for the other experts to to do their thing now.
39:22 And this is where Windows and macOS installers are actually built.
39:28 And so these aren't built for security releases, but for 312.0, these will be built.
39:33 And they're two completely different sets of build processes.
39:37 One of them happens in Azure pipelines.
39:40 So if you go to that release tool, there's actually like a Windows folder that just has
39:46 100 YAML files in it for all of the different Azure pipeline configurations.
39:50 Yeah.
39:51 Yeah.
39:51 Azure pipelines a little bit like GitHub actions.
39:53 People who are familiar with that, but not Azure.
39:56 Right.
39:56 Something roughly.
39:58 Yeah.
39:58 Yeah.
39:58 Yeah.
39:58 And I think there was like a time where Azure pipelines and GitHub actions were like basically
40:02 the same thing.
40:02 Yeah.
40:03 Yeah.
40:03 Like in the very beginning.
40:07 Yeah.
40:07 So Azure pipelines, there's a whole bunch of stuff that happens there because there's
40:11 Windows embeddable packages is something that gets uploaded to python.org.
40:15 Windows installers also get uploaded to python.org.
40:18 But that also there's like NuGet gets updated.
40:21 The Windows store gets updated there.
40:23 So like there's a lot happening there.
40:25 And there's actually a lot of things that happen that require the actual release manager,
40:30 the Windows release manager to like approve them.
40:33 So like they're gated.
40:34 These are things like signing keys that are, you know, as a part of the process, you have
40:39 to like sign off to say like, yep, we want to give this job access to the signing key.
40:43 So that's great.
40:43 Do you know if Winget is kind of like the newer package manager for Windows?
40:49 Do you know if Python goes there as well these days?
40:52 Let's see.
40:52 I haven't used Windows in a really long time.
40:55 So I don't know.
40:56 I do, but I only use it for play games and simple Windows tests.
41:00 I don't know.
41:01 All right.
41:01 But yeah, so NuGet is like one of these kinds of package managers for Windows and maybe
41:05 maybe Winget.
41:06 People in the audience, if anyone knows, they can let us know.
41:08 Yeah, definitely.
41:09 I've not even heard of Winget.
41:11 So I feel really out of the loop right now.
41:13 It's only a couple of years old, I believe.
41:15 Gotcha.
41:16 Gotcha.
41:17 And following on with that, those probably won't help you if you don't do a lot of Windows.
41:21 Lewis asked, does Python maintain Chocolaty, which is similar to Winget, but more independent
41:27 than Winget?
41:29 They tried Winget and it was buggy and it wasn't great.
41:32 Chocolaty is maybe a little older.
41:33 I don't know.
41:34 Same story probably, right?
41:35 Yeah.
41:35 I also don't know.
41:36 Unfortunately, Windows is not my strong suit.
41:39 Yeah.
41:40 Yeah, it's all good.
41:42 Definitely a tastier name.
41:43 That's how I choose all my tooling is how tasty is the name.
41:47 Yeah.
41:47 And so then the counterpart to that is the macOS binary installers.
41:54 And so there's basically just another set of scripts that get run on the macOS release
41:59 manager, like their machine.
42:02 And it builds just everything having to do with macOS.
42:06 It does notarization.
42:07 It does, you know, making, I know not as much about macOS either, but I know that notarization
42:13 happens.
42:13 And actually I've talked to a few folks about this one too, because this one, if you notice,
42:19 it's just one square and it's just red.
42:21 And I was like, yeah, there's a lot more to dig into there.
42:25 Yeah.
42:26 But yeah, the macOS notarization stuff is a serious pain as well.
42:29 Like it's a little bit like submitting something to the iPhone app store where it, you know,
42:34 like gets reviewed and yeah.
42:36 Yeah.
42:36 I think once you have it already notified, you can publish updates.
42:39 Cause I was thinking as you were talking about that, like how do you design or deal with the
42:44 latency, you know, of like I've submitted it and how long till someone picks it up and
42:48 reviews it?
42:49 I don't know.
42:49 But I guess once it's kind of approved for notarization, then it, it just goes through
42:53 pretty quickly.
42:54 You just set a really long timeout on your CI job.
42:58 Like exactly.
42:59 Timeout seven weeks.
43:00 Give it a week at least.
43:02 Yeah.
43:03 No.
43:04 And so then after all of those things kind of happen, there's this stage in, you know,
43:10 there's a stop, stop, stop line in the PEP 101 that just basically says for the release
43:14 manager to wait for everyone to be done with their thing.
43:17 And then once everyone's done with their thing and has uploaded everything to python.org, there's
43:22 basically like this phase of testing everything.
43:24 So you, you know, download everything, you make sure everything works on all the operating
43:27 systems, the way that's expected to, you don't want to like blow the horn and then realize
43:32 the very last mile that there's something wrong.
43:35 And so like lots of testing happens.
43:36 And then after the testing happens at that point, then everything that will get uploaded
43:41 to python.org or that has been uploaded to python.org gets signed by the release manager in particular
43:46 with sigstore and GPG.
43:49 And then from there, all of those signatures get published.
43:52 And that is the end of python.org.
43:54 All of the artifacts are published.
43:56 And only then does the release manager, this is step 10, do a git push to the actual upstream
44:02 CPython repo.
44:03 And at that point, then the branch can be unblocked.
44:07 All of these things, the release is done.
44:08 Everyone's happy.
44:09 We can celebrate.
44:11 Yeah, that's kind of the very high level view of what happens.
44:15 Amazing.
44:16 A couple of thoughts here.
44:18 First of all, the Windows square is like tangibly different than the macOS square.
44:25 The Windows square says, send off some source code to Azure pipelines, a external repeatable
44:32 CI build system.
44:33 Whereas the Mac one is like, build it.
44:35 You know what I mean?
44:36 That's actually pretty interesting.
44:39 And I don't know that there's an Azure pipeline for macOS, right?
44:42 Like that's just the sort of the story of Mac and servers.
44:45 But just highlighting those differences, right?
44:48 Like it's a different thing to say, I build on my machine with whatever I got versus I push
44:52 to something like Azure pipeline CI, CD and get some results back.
44:56 Yeah.
44:56 Both the source tarball builds and the macOS builds are all local, right?
45:02 And they're not done with a repeatable like CI provider.
45:07 I was actually able to get like the actual source tarballs to work on GitHub actions.
45:12 And so that's something that I've worked on and provided that to release managers.
45:18 And we're talking about it right now.
45:19 And macOS is actually, to my knowledge, is actually going to be revamped a little bit
45:25 because that tooling has been around for so long.
45:28 I think that there's going to be some work done on it.
45:30 And so I haven't dug into it as deeply.
45:32 I've kind of more just provided a bunch of context to the release managers about things
45:36 that I want to see.
45:37 Like if I were to do this myself, this is the other things I'm interested in, right?
45:42 Provided that you actually had access to a macOS system.
45:46 I'm sure you could do it all from the CLI because, for example, for the Talk Python courses apps,
45:50 we use Flutter and you just say Flutter run.
45:52 And it compiles that iOS or macOS version somehow through the command line.
45:59 And then like out comes a thing that is already deployed onto a simulator or something, right?
46:04 So it could happen.
46:06 I mean, there is some hosting, but it's kind of like, here's your Mac mini in the cloud.
46:12 Yeah, I mean, there is a GitHub Action supports macOS.
46:16 Does it?
46:17 Okay.
46:17 So possibly, yeah.
46:19 Maybe it could happen.
46:20 I'm not saying it has to, I'm just, it was kind of a striking difference that one is like
46:24 a CI, CD process and one is local and manual.
46:26 Yeah.
46:27 Actually, Windows is the most repeatable, most like actually isolated from any, from the,
46:34 just the machine itself, right?
46:36 The release manager themselves.
46:37 So, yeah.
46:38 Yeah, that's pretty interesting.
46:39 And then we have binaries and they get published and everybody is happy.
46:43 We all go and install it.
46:45 And it's been in testing for quite a while, right?
46:47 Like we've had a bunch of betas.
46:48 We had alphas.
46:49 We have had three release candidates and on Monday we'll have a dot zero.
46:54 It's exciting.
46:55 It's really exciting.
46:56 It is exciting.
46:57 The other question I would ask you is like, what is the time from like step one to step
47:02 10?
47:03 How much clock time has passed?
47:05 Wall time.
47:05 I would say that it is on, it's on the scale of hours.
47:11 It's not like a day, which is kind of incredible, right?
47:15 The fact that you get three volunteers together to do a bunch of all of this stuff.
47:21 And it only takes a couple hours.
47:23 It obviously depends on how many problems you run into on the way, because if you like get
47:27 to the very end and then there's a problem that you find in testing, like you have to kind
47:31 of start over, but yeah, if everything goes to plan, which has happened, I've coordinated security.
47:38 Like there was a couple of security fixes that were in 3.115 that I coordinated.
47:42 And yeah, I got to kind of witness the, okay, we've decided we're making a security release to the actual.
47:49 It's out there.
47:49 We can now talk about the vulnerabilities.
47:51 Yeah.
47:52 Was that the int parsing thing?
47:54 It was the, what was it?
47:56 There was an int parsing, or was it int parsing?
48:00 No, it wasn't int parsing.
48:01 Really long ints.
48:02 I don't know.
48:02 No, that's a different one.
48:03 No, it was, there was a TLS, and this one sounds really scary.
48:09 It's not as scary.
48:10 It's a TLS bypass on the server side.
48:16 So like if the client is supposed to be authenticated, for example, if you're using MTLS, you can,
48:23 if you like send a handshake and then immediately close your socket, you can get lucky during like a,
48:30 a brief period of time where the socket on the other side, like the server socket will say like,
48:35 oh, it's closed.
48:37 So it's fine.
48:38 Right.
48:38 And the data that whatever you sent over the wire won't, will still be readable, but then the
48:45 handshake hasn't actually like completed.
48:46 And so that sounds like scary, right?
48:49 But it's, it's, it's actually not as bad because so many protocols, well, for one, so many protocols
48:56 don't even use client authentication at all.
48:58 So that's one huge point.
49:00 The other side of it is that all protocols that people do end up using like mutual TLS with
49:07 are protocols that like, you have to like send something back to, so like HTTP, for example,
49:14 you have to send something back.
49:15 And at that point the socket is closed.
49:17 And so it can't send the data back.
49:20 So there's like no data exfiltration.
49:22 It's a pretty narrow scope of like what's actually vulnerable.
49:26 But yes, the, the, the fix for that vulnerability is in 3.11.5.
49:31 And there's another one too, but it's, it's a little bit more minor.
49:34 Well, we'll start fresh with 3.12 and hopefully you will not have to witness one of these releases.
49:39 Honestly, they're given how large CPython is and how wide ranging it, its standard library
49:44 attempts to be.
49:45 It's, it does not have many vulnerabilities.
49:48 I mean, you go look at like a web browser upgrade.
49:50 It's like, here's the 27, like remote code execution vulnerability.
49:55 Maybe not quite that many, but here's the 27 security fixes this month.
49:59 You know?
49:59 Yeah.
50:00 We've maybe, maybe it's a little more front of mind right now.
50:03 Now that like Chrome is on their fifth zero day of the week, but yeah, it's, it's pretty
50:07 tough week for Chrome right now.
50:09 Yeah.
50:09 And Firefox and the rest of us.
50:11 Yeah.
50:11 Yeah, for sure.
50:12 let's see another, another comment from Frank says six door and GPG seem a bit at the
50:19 end of the process.
50:20 Considering these are billed locally in some cases, how does one guarantee of the sign sources
50:25 are still right?
50:26 Yeah.
50:27 So this is kind of getting to the, like, I agree with you.
50:30 How, how does one, how does one track that they're still right?
50:33 so they, they get uploaded to python.org and at that point they're kind of like in a
50:39 holding pattern for testing.
50:41 And so those binaries are there, they're on python.org.
50:44 They've got, you know, the hashes have already been taken.
50:46 then the testing is happens.
50:48 So like anything that happens between that and the six or starting, like, yeah, at that point
50:53 it's kind of protected, but yeah, there is this really big dwell time.
50:56 Where you just have artifacts sitting on someone's machine and they're going to get signed and
51:02 sent out the door.
51:03 And that's kind of like the due diligence of release managers.
51:06 They need to make sure that those are the right artifacts that they've been tested properly,
51:10 that the ones that you tested locally are the ones that are going to get ended up being
51:14 signed.
51:14 yeah.
51:16 And so this is something, this is like a, a piece of the whole puzzle that gets mitigated
51:21 by having those builds happen on external services, as opposed to on someone's local machine.
51:26 Because at that point you're just giving, yeah.
51:28 Yeah.
51:28 You're just giving this like set of scripts, like very, very narrow set of input.
51:33 Like I want this exact get commit and this version number to get built.
51:38 And then out pops like a tar ball that's already been signed and verified and everything.
51:42 And then from there, you just put that on python.org.
51:45 Right.
51:45 Like that, that is a lot less.
51:47 there's a lot less of that risk where it's just on someone's machine.
51:51 And how, how do, how does one know that that is the exact thing that they built?
51:55 It's, it's not provable.
51:56 Yeah.
51:57 What about virtual machines for these gray boxes?
51:59 I mean, obviously the Azure one, that's, that's its own thing.
52:02 But have you considered like a, a Docker or an official just RLLs VM and go like, here,
52:09 take this, run that.
52:11 You know what I mean?
52:11 yeah.
52:13 To make it a little less dependent on the person whose role it is that year.
52:17 It definitely would help with the reproducibility side of things because as we know,
52:21 Docker is just the whole machine shipped in a box.
52:24 Right.
52:24 Yeah.
52:25 so yeah, in, in theory, if you build the same thing again, in that exact same image,
52:29 you would end up getting a pretty similar result.
52:31 So it helps on that front, which, which is good.
52:33 I think the, the tough thing is, is that you don't, you still don't have control of the
52:38 inputs and the output.
52:39 You also still don't have like this, like it comes out of the Docker container.
52:44 You get a tar ball.
52:45 There's still that time in between when the tar ball is out of the container and when it's
52:49 actually on python.org where it's on someone's machine that is not in a container.
52:54 and containers also aren't the best in terms of like, you, you did mention virtual
53:00 machines as well, but even, even that like on a machine that is a high value target,
53:07 you know, maybe it's worth it.
53:08 Who knows?
53:09 Yeah.
53:09 Yeah.
53:10 Who knows?
53:10 All right.
53:11 Well, it's pretty interesting and, it's, it's happening Monday.
53:15 So very exciting.
53:16 And I'm, I'm looking forward to all the new work.
53:19 I mean, are there new features in three 12 that are you particularly excited about?
53:24 You know, I, since being in this role, I actually haven't been paying attention to Python releases
53:30 as much Python features as much.
53:32 I'm, I'm mostly excited about it getting faster.
53:34 I mean, the generic, it's just straight up faster.
53:37 Like there's no one on the planet that's not happy about that.
53:40 what are the major features this, this release?
53:43 So there's like a, some more broad F string thing.
53:46 So you used to be able to have a subset of the language in the F string.
53:49 And now you can kind of like program in the F string.
53:51 I believe, type statement is better generic types.
53:55 Yeah.
53:56 There's like a, a simpler way to express generics and the type system.
54:00 And then I'm with you, you know, honestly, like f-strings being nicer.
54:05 That's awesome.
54:06 Thanks for that.
54:07 But faster, faster, faster is good.
54:09 this per interpreter GIL is kind of part of that faster CPython thing, right?
54:14 Like it's hard to leverage, but if you could just say in your threads, new interpreter for
54:18 this bit, right?
54:19 All of a sudden you escape the GIL for computational stuff.
54:23 That'd be pretty interesting.
54:24 but you know, also this, this is another interesting thing.
54:27 If you pull up this list, like the size of the scroll bar, I don't know.
54:31 We were talking about scary scroll bars earlier.
54:33 Like this is next level.
54:35 Let me see.
54:36 I'm going to, I'll throw this into omnivore.
54:38 Are you a fan of omnivore?
54:39 This app?
54:40 Omnivore.
54:40 Never heard of it.
54:41 What is it?
54:42 Omnivore.app.
54:44 Kind of a Instapaper pocket replacement.
54:46 Oh, I'm already logged into it.
54:47 Look at that.
54:47 Oh, super, super cool.
54:49 So I go here and you say add a link.
54:52 Actually, it just, I think it just went in there.
54:54 Let's see.
54:54 40 minutes just to read that what's new.
54:56 That's what I was trying to pull.
54:58 Oh, it just, yeah.
55:00 It calculates that for you.
55:02 It's like, it's going to take a while.
55:03 So just a sense of like, what is new in Python 3.12?
55:07 That's a ton, right?
55:08 Yeah.
55:09 A ton of stuff, like tons of fixes, tons of fixes and improvements.
55:13 Mike, people are out in the audience says, Python dash M SQLite 3 is nifty.
55:17 Mike, you're going to have to tell me more about this.
55:19 Does that, does that open up a database now?
55:22 That would be lovely.
55:23 Yeah.
55:24 That would be cool.
55:25 Kind of like the HTTP server.
55:26 Yes.
55:27 Yeah.
55:28 With the JSON tool.
55:29 The JSON tool is the one I use the most.
55:31 What's that one do?
55:31 So you can like pipe JSON into it and it'll just make it pretty.
55:35 It's pretty, it's questionable how useful it is, but it sure does make pretty JSON.
55:39 Yeah.
55:40 Which I, I really appreciate.
55:42 Like two, two space indented JSON.
55:44 That is, that's perfect.
55:46 Indeed.
55:47 All right.
55:47 One more comment from the audience and then we'll maybe wrap it up.
55:50 Karen says, let me, she's clarifying above like VM Docker for less, less changes and better reproducibility.
56:01 VM Docker for less heterogeneity reproducibility than the current situation, which is affected by a given user's environment.
56:10 Yeah.
56:10 Like what web browser did you install today or something completely unrelated or, you know,
56:15 or like you upgrade, if it's your personal machine and you're using tools, like if you upgrade those tools, it could pull some other system dependency.
56:23 Right.
56:23 That ends up being relevant to the build of CPython and yeah, it's just, there's just so much that could be different over the course of even a few days.
56:31 Yeah.
56:32 Yeah.
56:32 Which is why it's interesting that the windows one goes to Azure pipelines.
56:36 Cause it kind of pulls, puts that to the side, right?
56:39 Yeah.
56:39 All right, Seth.
56:40 Well, this has been super, super interesting and thanks for giving us a look into this world.
56:46 So yeah, very, very excellent.
56:48 And I guess, you know, final two questions.
56:51 You can write some Python code.
56:52 What editor using these days?
56:54 Oh yeah.
56:55 I'm using PyCharm.
56:56 I, I love PyCharm specifically.
56:58 You know, the, I, I have the, what is it?
57:01 I had the professional edition for a good long while and then I let that subscription lapse and I need to just redo it.
57:07 But so I'm using the community edition right now, but yeah, I love PyCharm so much.
57:11 So really good editor.
57:12 Awesome.
57:13 I'm there with you.
57:14 And then notable pipe I package, something that I already talked about these.
57:20 I, so pip audit, just use pip audit.
57:22 There's so much work happening somewhere.
57:24 With pip audit that it, it just makes sense for you to use it because there's just a ton of work happening.
57:30 And if you just add it to your workflow, you'll just know that things are vulnerable or like what's vulnerable and like what versions you should update to.
57:36 Yeah.
57:37 I already have some tooling that like some aliases that do three or four steps to actually update my dependencies and install them.
57:44 Like I might as well just throw this as like another and, and pip audit.
57:49 Yeah.
57:49 And you can actually, so you can, I think there's a way that you can tell pip audit to install packages, but then it will like say something.
57:58 If there is a vulnerability in whatever you're installing.
58:00 Right.
58:01 So like you can even replace like use pip audit.
58:05 I think that that is a feature.
58:06 All right.
58:07 I think so.
58:07 It has a features thing.
58:09 Let me look at it.
58:09 Oh, let's see it.
58:10 Oh, it's not going to work because it's not the read me.
58:12 It's true.
58:14 I got to, got to go to the homepage, which will take some here.
58:17 So I got to go to the source.
58:20 No, where's the GitHub?
58:21 Let me go to the stars.
58:22 All right.
58:22 You know what?
58:23 Maybe I don't want to click this thing.
58:24 Features.
58:25 There we go.
58:25 Multiple emitting.
58:27 We haven't talked about S bombs, but seamlessly reusing.
58:31 Yeah.
58:33 If it reuses the pip caches, maybe, maybe, maybe.
58:36 All right.
58:37 I'll play with it.
58:38 We'll see.
58:38 People can check it out.
58:39 Citation needed.
58:40 Citation needed.
58:42 Exactly.
58:42 Citation needed.
58:43 That's right.
58:44 All right.
58:44 Well, yeah.
58:45 People check out pip audit.
58:47 That looks excellent.
58:47 And yeah.
58:49 Final call to action.
58:50 People are interested in this.
58:51 I mean, give a thought to it on Monday, theoretically, at least in the past, if you've listened too
58:57 far in the future, but you know, the next release.
58:59 Yeah.
58:59 No, I, I think the biggest, so in terms of like, what can you personally do?
59:04 Because like, there's just, I just talked about so much stuff that is just kind of happening
59:08 in the background, but there's also stuff that like individual people can do.
59:11 And the most impactful URL that you can visit for security, if you're an open source consumer
59:17 or like a maintainer, like it doesn't matter.
59:20 This is just going to be impactful.
59:21 If you work in software is best.openssf.org.
59:26 So that is basically just like a webpage and it just has a few URLs and you click into
59:31 any of those URLs and it's just gives you like a checklist of here's things to think
59:36 about.
59:36 Here's, and then if you click into those checklist items, it gives you, it's kind of like this
59:40 recursive, nice, like guiding force of like, here's the things you could be doing.
59:46 Right.
59:46 And if you want to dig in more concise guide for developing more secure software, in other
59:50 words, NPM best practices and so on.
59:54 Yeah.
59:54 Excellent.
59:54 Yeah.
59:55 I'm hoping to put a like Python best practices guide there by the end of my, by the end of
01:00:00 the, the year that I'm here.
01:00:02 So don't you worry.
01:00:03 Look forward to that.
01:00:04 Excellent.
01:00:05 I'll be right there.
01:00:05 We'll get it above on the, on the list.
01:00:08 Yeah.
01:00:08 It'll, it'll somehow not sort alphabetically and it'll be okay.
01:00:14 Tip the scales a little bit.
01:00:15 Cool.
01:00:17 Well, thank you for being here, Seth.
01:00:18 Thank you.
01:00:19 And thank you for all this hard work that you're doing.
01:00:20 It's, it's good to know that you're out there.
01:00:22 keep an eye on all these things.
01:00:24 Thank you.
01:00:24 Thank you so much for having me on the show.
01:00:26 This has been lovely.
01:00:27 Yeah, you bet.
01:00:28 Catch you next time.
01:00:29 Bye.
01:00:29 This has been another episode of talk Python to me.
01:00:33 Thank you to our sponsors.
01:00:35 Be sure to check out what they're offering.
01:00:37 It really helps support the show.
01:00:38 The folks over at JetBrains encourage you to get work done with PyCharm.
01:00:44 PyCharm professional understands complex projects across multiple languages and technologies.
01:00:49 So you can stay productive while you're writing Python code and other code like HTML or SQL.
01:00:54 Download your free trial at talkpython.fm/done with PyCharm.
01:00:59 Want to level up your Python?
01:01:02 We have one of the largest catalogs of Python video courses over at Talk Python.
01:01:06 Our content ranges from true beginners to deeply advanced topics like memory and async.
01:01:11 And best of all, there's not a subscription in sight.
01:01:13 Check it out for yourself at training.talkpython.fm.
01:01:16 Be sure to subscribe to the show.
01:01:18 Open your favorite podcast app and search for Python.
01:01:21 We should be right at the top.
01:01:22 You can also find the iTunes feed at /itunes, the Google Play feed at /play,
01:01:28 and the direct RSS feed at /rss on talkpython.fm.
01:01:33 We're live streaming most of our recordings these days.
01:01:35 If you want to be part of the show and have your comments featured on the air,
01:01:39 be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
01:01:43 This is your host, Michael Kennedy.
01:01:45 Thanks so much for listening.
01:01:46 I really appreciate it.
01:01:48 Now get out there and write some Python code.
01:01:49 I'll see you next time.
01:02:10 Thank you.