WEBVTT

00:00:00.001 --> 00:00:05.380
PyPI has been in the news for a bunch of reasons lately, many of them great, but also some with a

00:00:05.380 --> 00:00:11.040
bit of drama or mixed reactions. On this episode, we have Dustin Ingram, one of the PyPI maintainers

00:00:11.040 --> 00:00:16.020
and one of the directors of the PSF, here to discuss the whole two-of-a-face story,

00:00:16.020 --> 00:00:21.020
securing the supply chain, and plenty more related topics. This is another important episode that

00:00:21.020 --> 00:00:25.480
people deeply committed to the Python space will want to hear. This is Talk Python To Me,

00:00:25.480 --> 00:00:29.620
episode 377, recorded August 11th, 2022.

00:00:29.620 --> 00:00:48.940
Welcome to Talk Python To Me, a weekly podcast on Python. This is your host, Michael Kennedy. Follow

00:00:48.940 --> 00:00:53.240
me on Twitter where I'm @mkennedy and keep up with the show and listen to past episodes at

00:00:53.240 --> 00:00:59.620
talkpython.fm and follow the show on Twitter via at Talk Python. We've started streaming most of our

00:00:59.620 --> 00:01:05.480
episodes live on YouTube. Subscribe to our YouTube channel over at talkpython.fm/youtube to get

00:01:05.480 --> 00:01:11.800
notified about upcoming shows and be part of that episode. This episode of Talk Python To Me is brought

00:01:11.800 --> 00:01:17.360
to you by Compiler from Red Hat. Listen to an episode of their podcast as they demystify the tech industry

00:01:17.360 --> 00:01:23.820
over at talkpython.fm/compiler. It's also brought to you by the IRL podcast, an original

00:01:23.820 --> 00:01:30.160
podcast from Mozilla. This season, they're looking at AI in real life. Listen to an episode at talkpython.fm

00:01:30.160 --> 00:01:35.880
slash IRL. Transcripts for this and all of our episodes are brought to you by Assembly AI. Do you need a great

00:01:35.880 --> 00:01:41.440
automatic speech-to-text API? Get human-level accuracy in just a few lines of code. Visit talkpython.fm

00:01:41.440 --> 00:01:49.820
Dustin, welcome back to Talk Python To Me. Yeah, it's great to be back. Good to see you. Yeah, good to

00:01:49.820 --> 00:01:56.400
see you as well. It's lovely to have you back. You know, we talked about the Python Packaging Authority.

00:01:56.400 --> 00:02:02.860
We talked about PyPI and all these things previously, and we're back to talk about them some more with a

00:02:02.860 --> 00:02:09.800
particular focus on security. Yeah, which is like kind of my new focus on my day-to-day, my like job

00:02:09.800 --> 00:02:16.140
hat, my PSF hat, all that stuff. It's fantastic when the job that you're paid to do like lines up with

00:02:16.140 --> 00:02:23.240
these other things, right? You can kind of learn on the job and then it really applies quickly. So, you know,

00:02:23.240 --> 00:02:29.280
maybe let's just start there. You're at Google working on security there. Maybe tell us about what

00:02:29.280 --> 00:02:33.260
you're up to and how it ties together. Last time we talked, I was working for Google Cloud as a dev

00:02:33.260 --> 00:02:37.720
advocate. And so that was a lot of like, you know, I think people mostly know me from that, a lot of

00:02:37.720 --> 00:02:42.220
conference talks and things like that. But since then, I've switched to like a brand new team at Google

00:02:42.220 --> 00:02:46.480
that I'm really excited about and I think is just really exciting in general. We're an open source

00:02:46.480 --> 00:02:52.000
security team. So, and we don't work just like on Google's open source, you know, libraries or

00:02:52.000 --> 00:02:56.800
whatever, but we just generally broadly work on open source security across the entire open source

00:02:56.800 --> 00:03:01.700
ecosystem and not just the Python ecosystem, but like every open source ecosystem. So we have our

00:03:01.700 --> 00:03:05.800
hand in like a lot of pots and there's like, you know, I think you're probably aware there's like a,

00:03:05.800 --> 00:03:10.740
this incredible wave of like focus on software security, but also open source software security.

00:03:10.740 --> 00:03:15.020
And so we're kind of riding that wave a little bit, but yeah, it's like, it's a dream team.

00:03:15.020 --> 00:03:19.500
Everyone I work with is like super talented and we're working on some really interesting new security

00:03:19.500 --> 00:03:21.040
stuff and yeah, really love it.

00:03:21.040 --> 00:03:24.280
I bet it's very exciting and it, you also have a chance to make a big impact, right?

00:03:24.280 --> 00:03:27.960
I've been working like kind of tangentially on software security ever since I started working

00:03:27.960 --> 00:03:32.460
on PyPI and I like cared about it for a long time, but it's like really, I think it's validating to

00:03:32.460 --> 00:03:36.300
sort of see that like, oh, now everyone kind of gets it. Like everyone's like, oh, this is,

00:03:36.300 --> 00:03:41.180
this is the thing we need to focus and make better. So it's cool to be there and be like ready to do it,

00:03:41.180 --> 00:03:43.340
you know, and, and have the tools to like make it happen.

00:03:43.500 --> 00:03:49.120
Yeah, absolutely. Have a lot of resources behind you through Google and the team and so on.

00:03:49.120 --> 00:03:52.500
Absolutely. Like an incredible amount of resources. Yeah. So nice.

00:03:52.500 --> 00:03:54.500
Most people probably don't fully appreciate it, right?

00:03:54.500 --> 00:03:54.800
Yeah.

00:03:54.800 --> 00:04:01.280
Yeah. So that is fantastic. The other thing that you're doing is working as the director of the PSF,

00:04:01.280 --> 00:04:01.520
right?

00:04:01.520 --> 00:04:07.220
Well, not the director. So the PSF has a board of directors. So I would, I call myself one of the

00:04:07.220 --> 00:04:08.020
directors of the PSF.

00:04:08.020 --> 00:04:09.540
So yeah. Tell us about your role at PSF.

00:04:09.540 --> 00:04:13.080
Yeah. Yeah. So I joined the board, I think it was about two years ago, because we just had an election.

00:04:13.080 --> 00:04:17.480
And so we sit for a three year term. So I've got another year left before I have to run again. But

00:04:17.480 --> 00:04:21.860
you know, it's been really nice to sort of like work on the PSF from the inside and do some community

00:04:21.860 --> 00:04:25.520
stuff. It's been a really weird time to join the board as well. Like there was like, right,

00:04:25.520 --> 00:04:31.000
the start of the pandemic and like the PSF derives most of its income from events like PyCon,

00:04:31.000 --> 00:04:35.260
like a lot of its income. And that was always sort of like identified as like kind of an existential

00:04:35.260 --> 00:04:40.720
threat to the PSF, but it like very much became a reality very quickly. And so there was a lot of

00:04:40.720 --> 00:04:45.620
work done before I joined and like sort of like after I joined as well to adapt to that. And I think

00:04:45.620 --> 00:04:49.840
PSF did an amazing job. Like we actually did really well, partly in thanks to like all of our

00:04:49.840 --> 00:04:54.460
sponsors and donors that like still continue to step it up, even though we weren't doing an in-person

00:04:54.460 --> 00:04:58.720
PyCon. We did a bunch of virtual PyCons. They went pretty well. You know, not as quite,

00:04:58.720 --> 00:05:03.520
quite as fun for me. You know, I like to see folks in person, but yeah, we, I think made it through the

00:05:03.520 --> 00:05:08.500
other side amazingly. It's been great. We got a really great board now. We just brought on a couple

00:05:08.500 --> 00:05:13.200
new folks as well. And I'm really excited to see what we're going to do for the next, you know,

00:05:13.200 --> 00:05:18.700
couple of years. I don't know that people fully appreciate how important PyCon is to the existence and

00:05:18.700 --> 00:05:22.960
financial wellbeing of the PSF. Maybe elaborate a bit on that.

00:05:23.120 --> 00:05:31.320
Yeah. I think the statistic, maybe at its peak, PyCon's revenue was about 85% of the operating

00:05:31.320 --> 00:05:36.640
budget of the PSF. So like almost all of the money that the PSF needs to like run and operate, which

00:05:36.640 --> 00:05:41.460
means pay staff, pay for infrastructure, all that kind of stuff came from ticket sales for PyCon.

00:05:41.460 --> 00:05:43.280
And sponsorship money and things like that.

00:05:43.380 --> 00:05:48.580
Yeah. Yeah. It's a little gray because like there are sponsors and they, they like both sponsor PyCon

00:05:48.580 --> 00:05:53.320
and sponsor the PSF. And it's like that money sort of just gets used by the PSF. But yeah, I mean,

00:05:53.320 --> 00:05:57.580
a lot of that sponsorship was really tied to the in-person event. So one thing we've done recently

00:05:57.580 --> 00:06:04.100
is like, if you're not a PSF sponsor, you should go psf.org slash sponsor. And there's, you know,

00:06:04.100 --> 00:06:08.060
kind of like a new menu for sponsorship. And we sort of adapted it in a way that's like,

00:06:08.060 --> 00:06:12.940
not exactly focused. Like you don't have to show up to PyCon to be a PSF sponsor and still you'll

00:06:12.940 --> 00:06:17.420
get a lot of benefit from it, including supporting things like PyPI and other infrastructure projects.

00:06:17.420 --> 00:06:23.260
There's a bit of a, it seems like a bit of a wave of large companies coming in and properly sponsoring

00:06:23.260 --> 00:06:29.780
the PSF. And I don't know if this is in reaction to what happened with PyCon and COVID, or it just

00:06:29.780 --> 00:06:36.260
happens to be the timing and the growth and especially the growth of Python in a more business corporate

00:06:36.260 --> 00:06:36.760
sense.

00:06:36.760 --> 00:06:41.120
I think it's a couple of things. One is that like PSF very much needs the support,

00:06:41.120 --> 00:06:45.980
right? And I think that's made obvious to the organizations that use Python and our infrastructure

00:06:45.980 --> 00:06:50.520
like PyPI. The other thing is, I think a lot of organizations are taking open source as like a

00:06:50.520 --> 00:06:55.320
dependency a lot more seriously. So making sure that they're like in some way contributing or,

00:06:55.320 --> 00:07:00.640
you know, providing support for the infrastructure tools, software that they use. The other thing that

00:07:00.640 --> 00:07:05.180
I want to call out here is like the PSF staff is incredible. They've done an amazing job about

00:07:05.180 --> 00:07:10.540
making it really an attractive thing to be a sponsor of the PSF and also like following through on our,

00:07:10.540 --> 00:07:15.680
you know, commitments and to our organization's commitments to us, our commitments to them,

00:07:15.680 --> 00:07:19.860
that kind of thing. And like finding new and interesting ways to like get funding as well.

00:07:19.860 --> 00:07:23.860
Right. So like we started doing an interesting thing a couple of years ago where we started

00:07:23.860 --> 00:07:29.460
applying for grants for like work on PyPI. And I think that's actually our first podcast was about

00:07:29.460 --> 00:07:33.980
like some funded work that I got hired to do as a contractor. And then we kind of like repeated that.

00:07:34.040 --> 00:07:38.880
And we like brought in a ton, a ton, a ton of money to fund like really big stuff, big stuff that like

00:07:38.880 --> 00:07:43.580
a volunteer would never get, you know, get done in a year of weekends. Right. Like it's just never

00:07:43.580 --> 00:07:46.920
going to happen that a volunteer is going to sit down and have the time to do this. So it's been

00:07:46.920 --> 00:07:52.020
really successful in terms of like shipping stuff that users need. That's like big, like large scale

00:07:52.020 --> 00:07:58.100
stuff. It seems a little bit like in the past, but when you go to PyPI.org, this is still kind of shiny

00:07:58.100 --> 00:08:04.260
new. Right. It got rewritten a couple of years ago and polished up and made a lot more modern. Right.

00:08:04.260 --> 00:08:10.040
Yeah. Yeah. I think 2018 we launched this and so it hasn't changed really, you know, visually much since

00:08:10.040 --> 00:08:14.940
then. A lot of new features and development and like the whole point behind the rewrite was to make it a lot easier

00:08:14.940 --> 00:08:16.060
to build on top of. Yeah.

00:08:16.060 --> 00:08:22.880
PyPI was like ancient, essentially. It predated everything on PyPI. So like it was, it was kind of wacky, but yeah,

00:08:22.880 --> 00:08:27.320
it's super modern. Yeah. That's fantastic. It's like more sustainable now as well. Right. Like we have, we have

00:08:27.320 --> 00:08:32.840
better like commitments from, you know, our in-kind donors for infrastructure. Shout out to like Fastly that pays our

00:08:32.840 --> 00:08:38.920
entire infrastructure bill and is like an amazing sponsor of PyPI. But also just like we have, we had just hired an

00:08:38.920 --> 00:08:44.360
infrastructure engineer to work on PyPI, which is super exciting and PyPI and other PSF infrastructure

00:08:44.360 --> 00:08:49.420
as well. And yeah, it's just like, it was a little more sustainable than it used to be. Like we have

00:08:49.420 --> 00:08:51.800
better core volunteers, moderators, all that.

00:08:51.800 --> 00:08:57.540
We talked back in 2018, I think maybe you and Donald stuffed. I don't remember if you all were on together.

00:08:57.540 --> 00:09:02.860
Those were two separate shows, but you were both involved. And one of the challenges was PyPI,

00:09:02.860 --> 00:09:11.140
the web app was so bespoken, sort of its own tangled mess that people would want to contribute.

00:09:11.140 --> 00:09:15.780
And they'd be like, you know what, now that I see this, maybe not so much. It sounds like it's in a

00:09:15.780 --> 00:09:19.820
better place. We have some peps that we're going to talk about, about extending some of its functionality

00:09:19.820 --> 00:09:25.260
and those sorts of things, which is probably a spinoff of just making it easier to work with.

00:09:25.260 --> 00:09:29.760
We did a full stack rewrite for our reasons, because it's easier to maintain for us. It's easier to

00:09:29.760 --> 00:09:35.880
contribute to, for other users, easier to propose new changes. And I think maybe the undertone for

00:09:35.880 --> 00:09:40.620
this entire interview is there needs to be progress, right? We can't just get to a point where it's

00:09:40.620 --> 00:09:46.140
just, that's it, that's good. There's just going to be, it's a constantly shifting landscape. So if we,

00:09:46.140 --> 00:09:51.480
the PSF, we, PyPI, want to continue to be successful and popular, and Python is doing amazing right now,

00:09:51.480 --> 00:09:53.260
we have to adapt to that to some extent.

00:09:53.260 --> 00:10:00.420
Yeah. It's not the same world it was built for when it first came out. And also this 393,000 packages,

00:10:00.420 --> 00:10:04.860
it's probably not something that was expected when this whole idea was put together.

00:10:04.860 --> 00:10:10.660
It's scaled like impressively well, I think over the years, like almost 4 million individual releases,

00:10:10.660 --> 00:10:15.500
like 6 million individual artifacts. Like that's a lot. That's a lot of stuff.

00:10:15.500 --> 00:10:21.140
Let's talk just a bit about the whole infrastructure side, not the tech or anything. We've covered that

00:10:21.140 --> 00:10:26.020
before and it was really interesting, but just how much data and expense there is to run this thing.

00:10:26.020 --> 00:10:32.340
I wrote a blog post sometime last year, and it was going to be essentially a five-year update from a

00:10:32.340 --> 00:10:37.920
previous post that Donald, who's one of the other PyPI maintainers, had written about just like what it

00:10:37.920 --> 00:10:42.300
takes to power PyPI. And it has some statistics in it that are at this point out of date. But like,

00:10:42.300 --> 00:10:49.100
yeah, we serve like almost, I think at this point, over 2 billion requests a day. We transfer like more than

00:10:49.100 --> 00:10:55.040
60 terabytes from PyPI.org. And that doesn't include files. So when we serve the actual files,

00:10:55.040 --> 00:11:00.460
actual distributions, that's like almost a thousand terabytes a day, like per day. That's a lot.

00:11:00.460 --> 00:11:01.900
A thousand terabytes?

00:11:01.900 --> 00:11:08.220
Yeah. If we had to pay like retail costs for our bandwidth from our CDN. So like almost like 99% of PyPI

00:11:08.220 --> 00:11:14.660
is served from CDN. It would be in the millions of dollars. Like it's a substantial infrastructure cost

00:11:14.660 --> 00:11:21.080
just to like serve the files, serve the requests. So it's not going down. It's not plateauing either.

00:11:21.180 --> 00:11:26.360
It's definitely going up, which is good in the sense that like, yeah, we want it to be popular,

00:11:26.360 --> 00:11:30.260
but like there's sustainability questions that come with that as well as we grow, just sort of like

00:11:30.260 --> 00:11:32.480
unfettered, you know? Yeah. Figuring that out.

00:11:32.480 --> 00:11:37.600
Honestly, that kind of blows my mind. I'm just wondering, what would you possibly do if you didn't

00:11:37.600 --> 00:11:40.440
have companies like Fastly really supporting?

00:11:40.440 --> 00:11:45.440
Honestly, it would be very hard to keep PyPI running if we didn't have the support of all our sponsors.

00:11:45.440 --> 00:11:50.020
And like, I think it's really important to make this distinction between PyPI and other indices,

00:11:50.020 --> 00:11:55.280
like, like NPM, for example, which is owned by a massive corporation and like run, has a whole

00:11:55.280 --> 00:12:00.300
support staff, has a whole like engineering staff. PyPI is like a couple folks and like a bunch of

00:12:00.300 --> 00:12:05.500
donated stuff, you know? And it's like on the same scale, you know, it's like as useful as something.

00:12:05.500 --> 00:12:11.940
When I think about how PyPI and npm and RubyGems, and you know, this is not to focus on,

00:12:12.080 --> 00:12:19.260
like to call out Python, but just all of these. It reminds me of the early internet back when we're,

00:12:19.260 --> 00:12:22.780
not maybe when we didn't have passwords, but when it was kind of like, oh, we'll just,

00:12:22.780 --> 00:12:27.920
we don't really need encryption here. And it was just, it was from a time when things were simpler

00:12:27.920 --> 00:12:32.280
and it feels like it's getting a little more complicated security wise and so on.

00:12:32.280 --> 00:12:37.000
Oh yeah, definitely. I mean, there was a point when me as a PyPI maintainer administrator,

00:12:37.000 --> 00:12:42.500
like we never had to respond to takedowns for like malicious stuff. Like it just never happened.

00:12:42.500 --> 00:12:47.120
And now it's like my inbox is on fire because I get multiple reports a day. Yeah. Yeah. I mean,

00:12:47.120 --> 00:12:51.960
it's, it's like, I think part of it is like people are trying to hit security bounties and do like

00:12:51.960 --> 00:12:57.080
research with PyPI, which is not the intended use case for PyPI, but it's a lot. Like it's definitely,

00:12:57.080 --> 00:12:57.880
there's an uptick.

00:12:57.880 --> 00:13:01.960
Yeah. There's been a lot of talk on the internet about things that might fix it,

00:13:01.960 --> 00:13:08.200
like signing packages and whatnot, but we'll, we'll talk about whether that actually has anything to

00:13:08.200 --> 00:13:13.100
offer there. Yeah. Yeah. One thing I did want to give a quick shout out to is there's from the open

00:13:13.100 --> 00:13:23.060
SF at open SSF. They just gave some big donation to make PyPI a little bit better. Right. So they,

00:13:23.060 --> 00:13:28.640
committed $400,000 to, in order to create a new role. Tell us a bit about this. What is this?

00:13:28.640 --> 00:13:33.260
Yeah. They're, they're very excited to announce that they're planning to support us with a new role.

00:13:33.260 --> 00:13:37.860
So it hasn't been, it hasn't been finalized. Okay. Contract hasn't been signed yet. No, no,

00:13:37.860 --> 00:13:43.420
like open SSF, a fairly new organization, bunch of member organizations, including Google, Microsoft,

00:13:43.420 --> 00:13:49.360
whatever to essentially support software security. Right. And so they're just kind of getting started

00:13:49.360 --> 00:13:55.520
pretty recently. And I think their marketing team kind of outpaced the like legal team here. So we

00:13:55.520 --> 00:13:59.880
haven't signed the contract yet, but it's like, you know, I feel confident saying that like, this is

00:13:59.880 --> 00:14:05.820
probably almost definitely going to happen. So yeah, they committed, I think $400,000 to doing a

00:14:05.820 --> 00:14:09.560
developer in residence that's security focused. And so this is sort of like piggybacking on something

00:14:09.560 --> 00:14:15.020
that I helped start two years ago at this point, which is create the CPython developer in residence.

00:14:15.020 --> 00:14:20.580
So that was started with funding from Google and, Lucas Schlange became the CPython developer

00:14:20.580 --> 00:14:26.020
in residence. And like, I love to see this because like, I'm very happy to say Google is not the

00:14:26.020 --> 00:14:30.300
sponsor of CPython developer in residence this year. It's Facebook. And then like, that's great because

00:14:30.300 --> 00:14:35.220
like, I think this is something that can be shared by all the PSF sponsors that's, you know, funding it

00:14:35.220 --> 00:14:39.540
each year, that kind of thing. So in a similar way, we're going to ideally hire someone that will focus

00:14:39.540 --> 00:14:45.240
on just security for Python. That might be security for CPython. It might be security for PyPI. They

00:14:45.240 --> 00:14:49.900
also want, I think, to fund a security audit of some like critical tooling for, for Python ecosystem.

00:14:49.900 --> 00:14:55.560
That might be PyPI. So, but yeah, this is super cool. And, they've also like announced funding

00:14:55.560 --> 00:14:57.780
for some other organizations like Eclipse foundation.

00:14:58.180 --> 00:15:02.740
This is fantastic news and it's too bad that it's not signed yet, but I'm, it sounds like it's

00:15:02.740 --> 00:15:06.400
definitely going to happen when it becomes official. I'll give it a, another shout out just to say

00:15:06.400 --> 00:15:10.520
things because this is going to probably make a big difference. That's a big chunk of money to

00:15:10.520 --> 00:15:11.200
contribute to it.

00:15:11.200 --> 00:15:15.740
Similar to the CPython developer residence role. Like we're going to do interviews and audit,

00:15:15.740 --> 00:15:20.360
uh, and hire someone for that role. So there'll be a job posting if this happens. And then,

00:15:20.360 --> 00:15:25.000
I'll be definitely tweeting, sharing that, trying to get people to get interested and apply because,

00:15:25.000 --> 00:15:26.580
uh, this is a super cool role.

00:15:26.580 --> 00:15:27.180
It sure is.

00:15:27.180 --> 00:15:34.240
This portion of talk Python enemy is brought to you by the compiler podcast from red hat.

00:15:34.240 --> 00:15:40.800
Just like you, I'm a big fan of podcasts and I'm happy to share a new one from a highly respected

00:15:40.800 --> 00:15:47.040
and open source company compiler and original podcast from red hat with more and more of us

00:15:47.040 --> 00:15:51.580
working from home. It's important to keep our human connection with technology with compiler.

00:15:51.580 --> 00:15:56.940
You'll do just that the compiler podcast unravels industry topics, trends, and things you've

00:15:56.940 --> 00:15:57.840
always wanted to know about.

00:15:57.840 --> 00:15:58.840
And I'm going to be talking to people who are going to be talking about.

00:15:58.840 --> 00:15:59.840
I'm going to be talking to people who are going to be talking about tech through interviews with people

00:15:59.840 --> 00:16:04.920
who know it best. These conversations include answering big questions like what is technical debt?

00:16:04.920 --> 00:16:10.560
What are hiring managers actually looking for? And do you have to know how to code to get started in open source?

00:16:10.560 --> 00:16:18.660
I was a guest on red hat's previous podcast, command line heroes in compiler follows along in that excellent and polished style.

00:16:18.660 --> 00:16:24.200
We came to expect from that show. I just listened to episode 12 of compiler. How should we handle failure?

00:16:24.380 --> 00:16:34.200
I really valued their conversation about making space for developers to fail so that they can learn and grow without fear of making mistakes or taking down the production website.

00:16:34.280 --> 00:16:42.180
It's a conversation we can all relate to. I'm sure. Listen to an episode of compiler by visiting talkpython.fm/compiler.

00:16:42.180 --> 00:16:50.180
The link is in your podcast player show notes. You can listen to compiler on Apple podcast, overcast, Spotify, pocketcast, or anywhere you listen to your podcasts.

00:16:50.180 --> 00:17:00.920
And yes, of course, you could subscribe by just searching for it in your podcast player, but do so by following talkpython.fm/compiler so that they know that you came from talkpython to me.

00:17:00.920 --> 00:17:06.180
My thanks to the compiler podcast for keeping this podcast going strong.

00:17:06.180 --> 00:17:16.120
Let's talk about 2FA. That's been a bit of a flashpoint in it. I don't feel like it should have, but it has been.

00:17:16.120 --> 00:17:20.520
What's the story of 2FA and critical packages and PyPI?

00:17:20.520 --> 00:17:34.340
Yeah. So yeah, flashpoint, like kind of almost unexpected for me, you know, like I think I'm just so close to security space and PyPI and all that stuff that like, I think the reaction was a little stronger than I think everyone expected.

00:17:34.340 --> 00:17:41.440
It feels to me like the reaction was if you had set up a rule that said, hey, you can't have the letter A as your password.

00:17:41.720 --> 00:17:51.040
And everyone who has the letter A, you have to change it. It's like, it's almost like that level of requirement change to me, it feels like. And yet it just, it just blew up. Right.

00:17:51.040 --> 00:17:55.280
Yeah. Let me give some background and then like we can talk about like realistically what it means.

00:17:55.680 --> 00:18:12.680
So yeah, we, we made an announcement and basically that we were going to designate some projects on PyPI as critical. And essentially we determined this based on download count because that's kind of like, it's not a great metric, but it's kind of the best metric we have for determining like if this project was compromised.

00:18:13.080 --> 00:18:24.100
And I'll talk about like how that might happen, you know, how many people would be affected. And it's like, if we measure the amount of times that this is getting downloaded a day, that's a pretty good proxy for like impact in terms of something being compromised.

00:18:24.480 --> 00:18:24.600
Right.

00:18:24.700 --> 00:18:31.460
So yeah, we, we made this designation and we sort of announced that like at some point in the future, did not announce a date, did not like enforce a requirement at this point.

00:18:31.460 --> 00:18:36.000
We're going to ask those maintainers to require that two FAs enabled for their account.

00:18:36.000 --> 00:18:39.680
And so we did that and then we sort of paired this with an incentive.

00:18:39.920 --> 00:18:44.960
We, my team at Google actually funded the purchase of a bunch of Titan security keys.

00:18:44.960 --> 00:18:57.740
These are like hardware keys for two factor authentication that Google manufactures, but we just, you know, essentially we give away discount codes to these maintainers of projects that have been designated as critical and they can get not one, but two for free.

00:18:57.740 --> 00:18:59.900
So if they're one of these maintainers.

00:18:59.900 --> 00:19:03.440
So yeah, we did that and the designation was 1%.

00:19:03.440 --> 00:19:07.820
We decided the top 1% of projects would be this point designated as, as critical.

00:19:07.820 --> 00:19:12.980
Right. I feel like there was a bit of a confusion when people saw this announcement.

00:19:12.980 --> 00:19:19.100
They saw, wait a minute, you're making me adopt hardware based 2FA because I have a PyPI package.

00:19:19.100 --> 00:19:23.940
The requirement is not that you have to use the hardware keys if you have a critical package, is it?

00:19:23.940 --> 00:19:37.660
And like, I would love if everyone used hardware keys because I think they're generally considered to be a little bit more secure, but no, the idea is that everyone should turn on 2FA and that's, PyPI supports, you know, TOTP, which is like what you're used to.

00:19:37.660 --> 00:19:43.020
Yeah. Like the free applications on the phone or other device and security keys.

00:19:43.020 --> 00:19:53.400
So like, and security keys is pretty broad now that, that doesn't just include the like USB devices, but also like you can do web auth and via like phones and like other physical hardware.

00:19:53.400 --> 00:19:56.740
Like it's pretty, you know, the integration with browsers is pretty good now.

00:19:56.740 --> 00:19:58.080
So there's a lot of support.

00:19:58.080 --> 00:20:03.280
Yeah. And just like the audience out there, Michael is asking, do they need to be hardware keys or just regular auth?

00:20:03.280 --> 00:20:04.360
It's just regular auth, right?

00:20:04.440 --> 00:20:07.040
And, and that's why I said, I don't feel like it's that big of a deal.

00:20:07.040 --> 00:20:10.320
It's like, well, you have to have a secure password or you have to have 2FA or whatever.

00:20:10.320 --> 00:20:16.100
Like kind of immediate reaction from some folks with like really big megaphones essentially was that this is a slippery slope.

00:20:16.100 --> 00:20:18.840
Like PyPI is asking something of its users.

00:20:18.840 --> 00:20:20.200
We don't do that very often.

00:20:20.200 --> 00:20:22.820
Like we sort of like let users do whatever they want.

00:20:22.820 --> 00:20:29.040
And we have some sort of like baseline requirements for like how to use PyPI, but we don't like often ask people to do extra stuff.

00:20:29.040 --> 00:20:33.120
There's a good reason why we're interested in asking people to do 2FA.

00:20:33.120 --> 00:20:38.420
And it's not because like Google has secretly conspired to like do it so that it's own open source security.

00:20:38.420 --> 00:20:40.860
You know, like it, there's, there's like a very good reason.

00:20:40.860 --> 00:20:49.900
There's a whole undercurrent, a whole thread of, well, it's these big corporate companies that are adopting Python that are making us do different security to support them.

00:20:50.020 --> 00:20:52.320
And that wasn't it at all, was it?

00:20:52.320 --> 00:20:53.840
Here's the main reason, right?

00:20:53.840 --> 00:20:55.320
And like that is valid, right?

00:20:55.320 --> 00:20:57.940
Like there are big corporations that consume stuff from PyPI.

00:20:57.940 --> 00:21:02.460
They would love to have more assurances about like that their projects haven't been compromised.

00:21:02.460 --> 00:21:05.140
I don't think 2FA is like exactly the right way to do that.

00:21:05.140 --> 00:21:12.280
At the end of the day, 2FA, like it protects against like two kind of critical attacks that could happen on a Python package.

00:21:12.280 --> 00:21:14.120
One is just like phishing, right?

00:21:14.120 --> 00:21:17.500
Like 2FA is essentially it completely eliminates the potential to get phished.

00:21:17.500 --> 00:21:19.400
I've never seen someone get phished on PyPI.

00:21:19.460 --> 00:21:20.860
I've never heard about phishing attack.

00:21:20.860 --> 00:21:25.120
But like PyPI is as susceptible as like a bank or anything else for phishing.

00:21:25.120 --> 00:21:26.200
Like it could happen to anybody.

00:21:26.200 --> 00:21:27.500
So that's one thing.

00:21:27.500 --> 00:21:33.300
The other thing is maybe like more specific to PyPI itself, which is what we call like domain resurrection attacks.

00:21:33.440 --> 00:21:38.400
So developers like really love their vanity domains, their personal domains, their personal email addresses.

00:21:38.400 --> 00:21:44.940
And so unlike maybe your bank, like you're the users on PyPI are more likely to have these like one off domains.

00:21:44.940 --> 00:21:46.480
And those domains like expire.

00:21:46.480 --> 00:21:48.420
People forget about they lose access to them.

00:21:48.420 --> 00:21:50.560
They get, you know, registered by someone else.

00:21:50.560 --> 00:22:03.660
And when that email address has the ability to reset a password on a PyPI account, an attacker can like keep an eye on your domain, watch when it expires, go and register it, do a password reset, and then take over your account and publish whatever they want.

00:22:03.660 --> 00:22:07.680
And so 2FA in similar way to phishing protects against that attack as well.

00:22:07.760 --> 00:22:08.900
I had never really thought about that.

00:22:08.900 --> 00:22:10.800
That's almost like the SIM card equivalent.

00:22:10.800 --> 00:22:11.420
Yeah, a little bit.

00:22:11.420 --> 00:22:13.340
But for email.

00:22:13.340 --> 00:22:22.520
So, you know, the SIM card problem is I could call up, you know, my, I could call up, I could call up someone else's phone provider and say, I lost my SIM card.

00:22:22.520 --> 00:22:24.000
Please issue me a new one.

00:22:24.000 --> 00:22:27.920
And then you start getting their SMS for like SMS authentication and stuff.

00:22:27.920 --> 00:22:31.800
This is, you've taken over their domain, not maliciously.

00:22:31.800 --> 00:22:35.240
They just decided to, you know, a credit card expired or something.

00:22:35.540 --> 00:22:38.680
And then you snatch it up, set up some MX records and off you go.

00:22:38.680 --> 00:22:39.200
Okay.

00:22:39.200 --> 00:22:50.060
Our ultimate goal, like PyPI's administrators, I'd love to protect all users from attacks that could be prevented from 2FA, but it's a little bit more like, like it's actually for our own benefit, right?

00:22:50.060 --> 00:22:51.000
Those kind of attacks.

00:22:51.000 --> 00:22:56.660
So one has happened recently, the CTX package had a domain takeover and a malicious release published.

00:22:56.660 --> 00:22:58.540
And we wrote a very long incident about it.

00:22:58.540 --> 00:22:59.680
It took a lot of our time.

00:22:59.680 --> 00:23:02.540
And we essentially like, it's not sustainable for these to happen.

00:23:02.540 --> 00:23:04.400
We can't, we don't have a support team.

00:23:04.480 --> 00:23:10.980
We can't do, you know, manually remove these packages and monitor things for, like, we just can't, we just can't handle it.

00:23:10.980 --> 00:23:15.940
So 2FA is like the folks that maintain PyPI asking users like, hey, help us out a little bit.

00:23:15.940 --> 00:23:25.080
Just do this thing for us to like kind of cut down on the potential for this and make it easier for us to do things that like we actually want to do to PyPI and not just like respond to security incidents.

00:23:25.080 --> 00:23:26.900
Right. Because there's only a couple of you.

00:23:26.900 --> 00:23:33.120
And if you're spending all your time putting out these fires, you're not adding JSON endpoints and other beneficial things.

00:23:33.120 --> 00:23:34.080
Yeah. All sorts of stuff.

00:23:34.080 --> 00:23:39.300
Like, yeah, the more time we spend putting out fires, the less we can do like useful and interesting things to PyPI.

00:23:39.500 --> 00:23:46.520
Yeah. So why 1% of the top packages? Why is that critical? And also what's the designation over time?

00:23:46.580 --> 00:23:57.220
The designation is if at any point it was in the top 1%. And I think we recompute this every day. So, you know, projects have since we announced this, they've moved into the 1% because it's constantly shifting.

00:23:57.220 --> 00:24:15.620
But yeah, why 1%? So that's a question that like was coming up a lot in the discussion after we made this announcement. And the like secret to the 1% is that in reality, if you were to go and figure out like, okay, how much traffic, how many downloads does this 1% of packages actually represent for PyPI?

00:24:15.820 --> 00:24:32.420
It's like over 95%. It's close to 99%. It's like most of what people are using from PyPI is in this 1%. So by saying 1%, we also essentially said like for the long tail of PyPI that people aren't using, we care a little bit less about that.

00:24:32.420 --> 00:24:45.340
We're going to cover like the majority of these, like I said, the potential for impact if something was compromised, we sort of maximize that. And we also kind of had to minimize that 1% too, because I think the thing, another thing that folks didn't really realize about

00:24:45.340 --> 00:25:14.700
what it takes to support 2FA is that there's an incredible maintenance burden for 2FA. Like we have to handle account recovery requests because people like they lose their phones, they lose their hardware keys, people are humans, right? And so this happens all the time. And it's expensive for us to handle this, right? And we can't just say, all right, great, you like you lost 2FA, I turned it off for your account, go wild, because that's essentially like a perfect way to circumvent 2FA. Instead, we have to do this like very manual process where we like verify other identities,

00:25:14.860 --> 00:25:43.380
emails, like if you have a GitHub associated, we ask you to do something on GitHub, just like prove that you own that account. And even then it's like, it's really not perfect. Like there is potential for someone to be compromised who did have 2FA enabled by someone who, you know, could take over this account or that account and like pretend like they need an account recovery. But yeah, this is a huge maintenance burden. So like, we actually like can barely handle account recovery requests right now. And I'm a little wary of how many we're going to get now that folks have started really turning on 2FA, but we think it's worthwhile.

00:25:43.380 --> 00:25:46.160
And maybe that's probably why 1% and not 100%, right?

00:25:46.160 --> 00:26:02.760
Oh, yeah. Like there's just zero chance we could handle 100% of like everyone on PyPI with 2FA enabled. Like we just couldn't handle it. I would love that. Like that would be great. But yeah, unfortunately, like the amount of people losing their stuff and having to come to us for resets, it's just, it's the burden is really high.

00:26:02.960 --> 00:26:09.180
Sure. For me, I used Authy for my 2FA, which syncs across devices. So at least if I lose one, I get it back.

00:26:09.180 --> 00:26:30.120
Yeah. And like Google Authenticator works really well for TOTP as well. And I think you can like download the codes or store them externally as well. So if you lose your phone, you can regain access to those TOTP codes as well. And there's a bunch, like, there's also like emulated TOTP stuff where you can like run it on your laptop. And it's not like maybe not technically true factor, but like a lot of people use that because it's more convenient.

00:26:30.240 --> 00:26:31.380
It's way better than nothing, right?

00:26:31.380 --> 00:26:32.640
All better than nothing. Exactly.

00:26:32.640 --> 00:26:40.800
Let's talk about James Bennett and opinions. You called out this article and I also read this. I think this is really good. What are some of your takeaways here?

00:26:40.800 --> 00:27:00.040
Yeah. James absolutely nailed the response here. And actually, like, you know, when we got a lot of feedback, I'm not going to say that it was bad feedback. You know, it was maybe somewhat uninformed feedback or it was somewhat sensational feedback, but we got a lot of feedback after this. And some of it was totally valid. Like, you know, at the end of the day, we are asking users to take a little

00:27:00.040 --> 00:27:14.420
more effort and some people, you know, they don't want to do that. And, you know, I like none of the PyPI administrators actually like explicitly responded to a lot of this. I think we were all like a little bit depressed about how upset some people were about the 2FA requirement that didn't even exist yet.

00:27:14.420 --> 00:27:23.460
But yeah, James, like really shout out to James, because I read this and I was like, I could really could not have written it better than he did. He really called out.

00:27:23.460 --> 00:27:27.140
There's a lot there. And it's I think it's very well thought. Yeah, I thought out. Yeah.

00:27:27.140 --> 00:27:42.080
Yeah. Shout out to James. You know, there was kind of like two arguments that he was making, which is that, like, you know, a lot of people were concerned this would be a slippery slope. And I think I don't really foresee PyPI making too many more mandates about stuff like this. Not because of the feedback, but because, you know, like we're never going to.

00:27:42.420 --> 00:27:56.440
I don't think we're ever going to mandate signing, for example, like that's always going to be the option of the maintainer. But, you know, things like 2FA for certain high profile stuff like, yeah, it really helps out helps PyPI continue to exist. Right. Like that's actually the motivation here.

00:27:56.440 --> 00:28:05.360
I definitely want to echo the message that you said about the overhead. Yeah. Like you have to deal the people who would otherwise be constructively working on this have to deal with these problems.

00:28:05.360 --> 00:28:11.820
Yeah. Every day. I mean, it's like I don't get paid to do it. I do it out of love, but it becomes larger and larger every day. And yeah.

00:28:12.120 --> 00:28:16.340
We're keeping our head above water right now. But yeah, there's plans also to make that better. But yeah.

00:28:16.340 --> 00:28:27.060
How much do you think the reaction, the I'll put it out of the way, I think the overreaction was of how much do you think that was perceived as it's got to be a hardware key versus it's just straight 2FA?

00:28:27.060 --> 00:28:34.080
Do you think people really rejected it being 2FA or did it seem like a bigger burden than just adding it to your Google Authenticator?

00:28:34.200 --> 00:28:40.160
If I were to say that whether we made some sort of failure here when we announced it, I would say like we didn't message this super well. Right.

00:28:40.160 --> 00:28:45.660
And that's because I'm a software engineer. I'm not a marketer or, you know, I'm an OK communicator.

00:28:45.660 --> 00:28:49.440
And the same is true for the rest of us. We don't have copywriters, anything like that.

00:28:49.440 --> 00:28:53.260
We don't have a PR team. So, you know, there was some stuff that people kind of missed.

00:28:53.340 --> 00:28:56.320
And I think one of the things was missed was like the mandate doesn't exist right now.

00:28:56.320 --> 00:28:58.820
We're just talking about enforcing it in the future.

00:28:58.820 --> 00:29:02.000
The other was like, what is actually being required of you today?

00:29:02.000 --> 00:29:04.500
Which for most folks, it was it was nothing.

00:29:04.500 --> 00:29:09.240
It was like, if you want to get a pair of free security keys, you have to do this today.

00:29:09.240 --> 00:29:11.260
And by the way, those are still available.

00:29:11.260 --> 00:29:16.700
I'm sure you all saw this as a positive, like, hey, we got this cool thing for people that they can get if they want or they just do 2FA.

00:29:16.700 --> 00:29:18.320
But like people are like, what is this?

00:29:18.320 --> 00:29:21.500
Yeah. And you're saying there's still some available for folks who want to get it. Right.

00:29:21.660 --> 00:29:26.440
Yeah. So through October 1st. So, yeah, if you this might be my call at the end as well.

00:29:26.440 --> 00:29:35.620
But yeah, if you go to pypi.org slash security dash key dash giveaway, you can check if you're a critical maintainer and you can get a key, get a pair of keys, actually.

00:29:35.620 --> 00:29:39.420
Yeah. So the pair of keys thing also, people weren't really sure why we were doing that.

00:29:39.420 --> 00:29:43.820
But the main reason is to help you not lose both of them, like lose all access.

00:29:43.820 --> 00:29:47.020
So if you have two keys and you've used both of them, you have some redundancy.

00:29:47.020 --> 00:29:50.880
You can stick someone, stick it in the garage or stick it somewhere else.

00:29:51.020 --> 00:29:52.020
You know, hand it to a friend.

00:29:52.020 --> 00:29:53.620
You put your brain in your backyard.

00:29:53.620 --> 00:29:55.480
Yeah, exactly. Exactly.

00:29:58.800 --> 00:30:04.920
This episode of Talk Python To Me is brought to you by the IRL podcast, an original podcast from Mozilla.

00:30:04.920 --> 00:30:10.240
If you're like me, you care about the ideas behind technology, not just the tech itself.

00:30:10.240 --> 00:30:14.140
We know that tech has an enormous influence on society.

00:30:14.140 --> 00:30:16.420
Many of these effects are hugely beneficial.

00:30:16.420 --> 00:30:21.600
Just think about how much information we carry with us every day through our cell phones.

00:30:22.320 --> 00:30:24.780
Other tech influences can be more negative.

00:30:24.780 --> 00:30:32.420
I really appreciate that Mozilla is always on the lookout for and working to mitigate negative influences of tech for all of us.

00:30:33.100 --> 00:30:37.600
If those kinds of ideas resonate with you, you should definitely check out the IRL podcast.

00:30:37.600 --> 00:30:43.000
It's hosted by Bridget Todd, and this season of IRL looks at AI in real life.

00:30:43.000 --> 00:30:44.300
Who can AI help?

00:30:44.300 --> 00:30:45.320
Who can it harm?

00:30:45.700 --> 00:30:51.420
The show features fascinating conversations with people who are working to build a more trustworthy AI.

00:30:51.420 --> 00:30:55.760
For example, there's an episode on how the world is mapped with AI.

00:30:56.160 --> 00:31:01.840
But it's the data that's missing from those maps that tells as much of the story as the data that's there.

00:31:01.840 --> 00:31:05.960
Another episode is about gig workers who depend on apps for their livelihood.

00:31:05.960 --> 00:31:10.880
It looks at how they're pushing back against algorithms that control how much they get paid,

00:31:10.880 --> 00:31:16.660
and how they're seeking new ways to gain power over data and create better working conditions for all of them.

00:31:16.660 --> 00:31:24.240
And for you political junkies, there's even an episode about the role that AI plays when it comes to the spread of disinformation around elections.

00:31:24.580 --> 00:31:27.640
Obviously, a huge concern for democracies around the world.

00:31:27.640 --> 00:31:35.060
I just listened to The Tech That We Won't Build, which explores when developers and data scientists should consider saying no to projects

00:31:35.060 --> 00:31:39.140
that can be harmful to society, even though we do have the tech to build them.

00:31:39.140 --> 00:31:41.260
Does this sound like an interesting show?

00:31:41.260 --> 00:31:46.440
Please use the link talkpython.fm/IRL to subscribe.

00:31:46.440 --> 00:31:52.260
Yes, you could search for it in your podcast player, but use the link talkpython.fm/IRL

00:31:52.260 --> 00:31:54.720
to let them know that you came from us.

00:31:54.720 --> 00:31:56.920
The link is in your podcast player show notes.

00:31:56.920 --> 00:32:01.420
Thank you to IRL and Mozilla for supporting Talk Python To Me.

00:32:04.660 --> 00:32:12.000
I guess part of the reason this is so much in the public awareness is because of this project called Atomic Rights.

00:32:12.000 --> 00:32:13.240
Yes.

00:32:13.240 --> 00:32:19.780
Want to give us all the rundown of why we're talking about this package?

00:32:19.780 --> 00:32:21.960
Let me just give people a really quick background.

00:32:21.960 --> 00:32:27.820
Atomic Rights is a package that lets you, within a with block, like you would do open file,

00:32:27.940 --> 00:32:33.480
but instead you say atomic write and it will write to a temporary file and only commit those

00:32:33.480 --> 00:32:36.520
changes to the real file, like at the very end, all in one shot.

00:32:36.520 --> 00:32:37.480
Pretty useful.

00:32:37.480 --> 00:32:43.840
Not super hard to do your own version of with a couple of built-in things in Python, like the temp files and what, but still.

00:32:43.840 --> 00:32:46.820
Kind of no longer necessary for modern Python is my understanding.

00:32:46.820 --> 00:32:49.000
Like this is a couple of lines of like modern Python.

00:32:49.000 --> 00:32:50.140
You don't have to worry about it.

00:32:50.140 --> 00:32:52.140
But it used to be, you know, something that you would use.

00:32:52.140 --> 00:32:52.440
Right.

00:32:52.500 --> 00:32:52.820
Exactly.

00:32:52.820 --> 00:32:54.420
How does this relate to 2FA?

00:32:54.420 --> 00:32:56.340
That has nothing to do with 2FA, does it?

00:32:56.340 --> 00:32:58.940
There's this thing that happens all the time, right?

00:32:58.940 --> 00:33:02.640
Like so IPI has this policy that everything on PyPI is essentially immutable.

00:33:02.640 --> 00:33:09.420
And that means that like individual files, file names, which can include a project name, a version and like a distribution type.

00:33:09.420 --> 00:33:10.240
Those are immutable.

00:33:10.240 --> 00:33:15.660
So if you upload something to PyPI that is like source distribution for some version or whatever, you publish that.

00:33:15.660 --> 00:33:16.380
It's there.

00:33:16.380 --> 00:33:17.580
You can't overwrite it.

00:33:17.580 --> 00:33:17.800
Right.

00:33:17.800 --> 00:33:20.800
So you can't surreptitiously like change what that points to.

00:33:20.920 --> 00:33:24.060
So like anyone installing it is always going to get the same thing, same SHA, everything.

00:33:24.060 --> 00:33:28.340
But that also means like if you want to delete something, you delete it and it's gone forever.

00:33:28.340 --> 00:33:30.380
You can't come back and overwrite it with something else.

00:33:30.380 --> 00:33:36.400
And so, you know, when and I don't encourage people to delete stuff from PyPI generally because, you know, you're almost definitely going to break somebody.

00:33:36.400 --> 00:33:41.720
There's better methods for, you know, kind of marking something as not useful and telling pip to not install it.

00:33:41.720 --> 00:33:43.980
That's our yanking, which is a whole weapon to itself.

00:33:43.980 --> 00:33:45.800
But yeah, so this thing happens all the time, though.

00:33:45.800 --> 00:33:50.620
Like we have a huge warning banner, big red button, like everything telling you if you're going to delete this thing.

00:33:50.620 --> 00:33:51.980
You're not going to be able to get it back.

00:33:51.980 --> 00:33:56.180
And so what happened here is like this maintainer didn't want to comply with 2FA.

00:33:56.180 --> 00:34:00.880
Their project was marked as critical because a lot of people were using it like a lot of people were using it still.

00:34:00.880 --> 00:34:03.220
And they thought that it would be a cool.

00:34:03.220 --> 00:34:09.760
They thought they'd discovered a cool hack where if they deleted it and then recreated it later that the mandate would no longer apply.

00:34:09.760 --> 00:34:14.820
And like that was kind of true because, like I said, our computation for critical projects runs once a day.

00:34:14.820 --> 00:34:20.180
So when they brought it back, like it didn't have that flag within 24 hours, that flag was added back to the project, essentially.

00:34:20.180 --> 00:34:22.900
But for a brief period of time, yeah, it was not marked as critical.

00:34:23.320 --> 00:34:26.220
But what happened was, you know, all these versions went away.

00:34:26.220 --> 00:34:31.340
And like a lot of people, I think, were depending on them, you know, like actual users of this project.

00:34:31.340 --> 00:34:37.280
And so, yeah, like there's a long discussion happening now about whether like it should even be possible to delete stuff from PyPI.

00:34:37.280 --> 00:34:40.220
And there's good arguments on both sides of the coin, right?

00:34:40.300 --> 00:34:44.140
Yeah, well, that was one of my first thoughts is like, wait, you can delete the releases?

00:34:44.140 --> 00:34:45.340
I knew they were immutable.

00:34:45.340 --> 00:34:47.540
You can't update them, but deleting.

00:34:47.540 --> 00:34:50.260
So what's the tradeoff there?

00:34:50.260 --> 00:34:52.020
Why can you delete them now?

00:34:52.020 --> 00:34:53.400
And maybe why wouldn't you in the future?

00:34:53.400 --> 00:34:55.880
This is like NPM's left pad incident, essentially.

00:34:55.880 --> 00:34:59.700
Like we right now there's potential for a high profile enough.

00:34:59.700 --> 00:35:02.880
And this package wasn't super high profile, but like it was in the critical list.

00:35:02.880 --> 00:35:04.620
It was in the top 1%.

00:35:04.620 --> 00:35:09.600
There's potential for some maintainer to decide, you know, you know, and it's their prerogative right now, right?

00:35:09.600 --> 00:35:12.940
Like there's no guarantees that these things continue to exist on PyPI.

00:35:12.940 --> 00:35:14.700
No one's necessarily paying for this.

00:35:14.700 --> 00:35:21.060
So like, yeah, maintainer absolutely has the ability now to just wipe something super popular and necessary off the face of PyPI.

00:35:21.060 --> 00:35:23.100
And that's the current status quo.

00:35:23.100 --> 00:35:25.100
It's not the same in a lot of other ecosystems.

00:35:25.100 --> 00:35:27.520
Some of them don't have that policy.

00:35:27.520 --> 00:35:28.400
Some of them do.

00:35:28.400 --> 00:35:35.900
But yeah, so there's a bit of debate about whether that should be necessary, especially when we have stuff like yanking, which actually is a more meaningful way to remove something.

00:35:36.240 --> 00:35:41.740
So let's suppose somebody, a pallets org or whatever, erases flask tomorrow.

00:35:41.740 --> 00:35:42.720
David, don't do it.

00:35:42.720 --> 00:35:44.720
David, please keep going, man.

00:35:44.720 --> 00:35:53.900
Is there a way to get a hold of the actual wheels and stuff as a community and put it back up under potentially a different name?

00:35:53.900 --> 00:35:55.120
Or is it just gone?

00:35:55.480 --> 00:35:57.460
How seriously gone is it when it's gone?

00:35:57.460 --> 00:35:59.620
Yeah, nothing published to PyPI is actually gone.

00:35:59.620 --> 00:36:06.980
So we don't actually, unless we're like legally required to, we don't delete any actual files off of our data store.

00:36:06.980 --> 00:36:11.400
So like the bucket that everything goes into, everything that's ever been published to PyPI is still there.

00:36:11.400 --> 00:36:12.920
So this actually played out.

00:36:12.920 --> 00:36:18.620
It's good that we have this in a couple instances because this is what exactly we used in this case of atomic writes.

00:36:18.620 --> 00:36:21.120
Because the maintainer was like, oh, I made a mistake.

00:36:21.120 --> 00:36:22.900
And they were like kind of humble.

00:36:22.900 --> 00:36:24.420
They were like, yeah, okay, this is a mistake.

00:36:24.420 --> 00:36:25.180
I shouldn't have done this.

00:36:25.180 --> 00:36:29.340
And then asked us to essentially restore the project from scratch.

00:36:29.460 --> 00:36:31.460
And like, we don't really have mechanisms to do that, right?

00:36:31.460 --> 00:36:34.000
That's not, that's not something that we do often.

00:36:34.000 --> 00:36:36.840
I think I can only remember maybe once when we've done that before.

00:36:36.840 --> 00:36:37.740
Maybe not even once.

00:36:37.740 --> 00:36:39.040
We generally just don't do this.

00:36:39.040 --> 00:36:41.100
Like if you delete something, we say it's gone.

00:36:41.100 --> 00:36:42.320
Like you need to publish a new version.

00:36:42.320 --> 00:36:45.380
But in this case, like we did decide to take the time.

00:36:45.380 --> 00:36:49.280
I think it took Donald like almost an hour to do this because it's a super manual process.

00:36:49.280 --> 00:36:50.600
But yeah, the files are still there.

00:36:50.600 --> 00:36:52.080
Something you don't do very often, right?

00:36:52.080 --> 00:36:53.620
No, no, like almost never.

00:36:53.620 --> 00:36:55.160
How can I even do this?

00:36:55.160 --> 00:36:58.120
The files are still there and they're still like externally addressable too.

00:36:58.240 --> 00:37:00.300
So they're, you know, they're always going to be available.

00:37:00.300 --> 00:37:04.320
And like, if, if something like that happened, David, don't do it.

00:37:04.320 --> 00:37:07.820
But like, if something like that happened, I think folks would probably be okay.

00:37:07.820 --> 00:37:08.860
We'd find ways around it.

00:37:08.860 --> 00:37:12.320
But yeah, I mean, it's a strong argument for not allowing it to happen.

00:37:12.320 --> 00:37:16.980
And, you know, when, when people publish stuff to PyPI, like RTOS is essentially,

00:37:16.980 --> 00:37:19.940
you give us the right to distribute this as we see fit forever.

00:37:19.940 --> 00:37:22.060
So, you know, PyPI is within its right.

00:37:22.060 --> 00:37:26.360
But there's, there's arguments for, you know, giving maintainers the ability to do it for various reasons.

00:37:26.780 --> 00:37:31.880
Has there been any thoughts to putting like levels of what pip will install?

00:37:31.880 --> 00:37:35.380
For example, I'm thinking like, I want to set up my pip though.

00:37:35.380 --> 00:37:42.800
It will only accept things that have 2FA set up, or it will only accept things with a certain number of downloads.

00:37:42.800 --> 00:37:45.900
Like I can only pip install something with 10,000 or more downloads.

00:37:45.900 --> 00:37:50.420
So, because maybe I'm trying to avoid typosquatting for very edge case things.

00:37:50.420 --> 00:37:52.820
In that general realm, have you all thought about this?

00:37:52.820 --> 00:37:53.820
Probably you have.

00:37:53.820 --> 00:37:54.180
Yeah, definitely.

00:37:54.180 --> 00:37:55.480
What are some of the thoughts?

00:37:55.480 --> 00:37:57.200
I think we have it on our list to talk about later.

00:37:57.200 --> 00:37:59.440
But yeah, I mean, there's definitely potential, right?

00:37:59.440 --> 00:38:03.000
There's all sorts of signals that you could potentially take into account here.

00:38:03.000 --> 00:38:07.540
TBD, like how meaningful some of them actually will be or how much that will actually protect you.

00:38:07.540 --> 00:38:14.480
But yeah, people have talked about like essentially defining a policy for what they'll consume and either having that be part of pip or something external.

00:38:14.480 --> 00:38:16.240
So yeah, it's definitely been discussed.

00:38:16.240 --> 00:38:16.580
Okay.

00:38:16.580 --> 00:38:22.700
You know, for example, like the web browsers, you can have no blocking.

00:38:22.700 --> 00:38:24.440
You can have blocked third-party cookies.

00:38:24.440 --> 00:38:26.620
You can block third-party cookies and trackers.

00:38:26.620 --> 00:38:31.940
And you can decide like how broken do I want my web to be versus how safe do I want my web to be?

00:38:31.940 --> 00:38:35.560
I feel like there might be something like that in the pip world.

00:38:35.560 --> 00:38:40.880
I think the reality, at least right now, is that any kind of policy like that would not be enforceable.

00:38:41.060 --> 00:38:45.120
Because there's going to be some edge case, some dependency that's super old or whatever.

00:38:45.120 --> 00:38:51.980
Like, you know, Python is not nearly as bad as an ecosystem like npm in terms of like breadth of dependencies for a given thing.

00:38:51.980 --> 00:38:52.260
Right.

00:38:52.260 --> 00:38:52.480
Yeah.

00:38:52.480 --> 00:38:54.560
Usually the dependencies are like thicker.

00:38:54.560 --> 00:38:55.580
Right.

00:38:55.580 --> 00:38:58.780
You don't have like three lines of code you're depending on.

00:38:58.780 --> 00:38:59.780
You just put that in your code.

00:38:59.920 --> 00:39:01.940
It does exist, but yeah, generally no.

00:39:01.940 --> 00:39:11.300
But so that's sort of like, but even still, like, I think you'd have a hard time saying like, I'm going to only consume packages that have 2FA enabled because there's so few of them right now.

00:39:11.300 --> 00:39:14.260
If all that's, you know, the tooling and stuff for that existed.

00:39:14.260 --> 00:39:14.660
Yeah.

00:39:14.660 --> 00:39:14.920
Sure.

00:39:14.920 --> 00:39:15.640
Okay.

00:39:15.640 --> 00:39:16.220
Interesting.

00:39:16.220 --> 00:39:22.440
So this Atomics Right story is everything was put back, but it just, it shows unintended consequences.

00:39:22.720 --> 00:39:26.740
And kind of ironic too, actually, because like PyPI is an open source project as well.

00:39:26.740 --> 00:39:28.700
And like, you know, people were upset.

00:39:28.700 --> 00:39:31.820
We were making demands of users to do a certain thing.

00:39:31.820 --> 00:39:36.960
But like at the end of the day, someone's making demands of us to use our time in ways that we don't necessarily want to.

00:39:36.960 --> 00:39:39.840
Again, see James Bennett's article, right?

00:39:39.840 --> 00:39:42.660
A lot of those ideas were really well spelled out there.

00:39:42.660 --> 00:39:43.100
Absolutely.

00:39:43.280 --> 00:39:48.460
People forget that it's a volunteer open source project and not like run by some corporation.

00:39:48.460 --> 00:39:52.040
I thought it was this conglomerate of corporate overlords.

00:39:52.040 --> 00:39:54.140
So this is what I got from Reddit.

00:39:54.140 --> 00:39:55.120
Shadowy Cabal.

00:39:55.120 --> 00:39:55.480
Yeah.

00:39:55.480 --> 00:40:00.140
What's this PyPI 2FA dashboard here?

00:40:00.140 --> 00:40:00.960
This looks pretty cool.

00:40:00.960 --> 00:40:02.160
Tell us about this project.

00:40:02.160 --> 00:40:03.620
I'll link to it in the show notes, of course.

00:40:03.620 --> 00:40:07.440
Switch to like in the top right, switch to like past three months or something like that.

00:40:07.440 --> 00:40:07.900
Yeah.

00:40:07.900 --> 00:40:08.160
Okay.

00:40:08.160 --> 00:40:09.160
You can really see the bump there.

00:40:09.160 --> 00:40:16.900
So yeah, this is the dashboard we put together essentially for us to monitor how the rollout for 2FA and security key giveaway was going.

00:40:16.900 --> 00:40:18.080
But we made it public.

00:40:18.080 --> 00:40:19.500
So like anyone can check this out.

00:40:19.500 --> 00:40:21.340
And I think we'll put link in the show notes.

00:40:21.340 --> 00:40:22.880
But yeah, the numbers are great.

00:40:22.880 --> 00:40:26.220
The one thing this isn't actually showing is how many security keys we've given away.

00:40:26.220 --> 00:40:31.080
So we've at this point, like I just checked earlier, we've given away more than 500 keys, which is awesome.

00:40:31.080 --> 00:40:33.860
And it's only a fraction of what we have to give away.

00:40:33.860 --> 00:40:38.760
So I would really like anyone listening wants a key and has a critical project like go and get the keys.

00:40:38.760 --> 00:40:41.680
And then, you know, this will find something to do with those keys.

00:40:41.680 --> 00:40:44.520
We don't give them all away by the time they expire in October.

00:40:44.520 --> 00:40:45.320
But yeah.

00:40:45.320 --> 00:40:46.920
So a bunch of keys given away.

00:40:46.920 --> 00:40:54.840
We also I didn't mention this, but like as part of this, we also turned on a feature that allowed any project to manually require 2FA for all their maintainers.

00:40:54.840 --> 00:40:58.740
So like anyone that just wasn't critical and wanted to opt into this, they could do that, too.

00:40:58.920 --> 00:41:01.700
So I think 300, almost 300 projects have done that.

00:41:01.700 --> 00:41:08.840
And then we're almost like we're so close to hitting 30,000 users on PyPI with two-factor enabled, which is huge.

00:41:08.840 --> 00:41:13.140
And that's up from like 27,000 before we did the giveaway.

00:41:13.140 --> 00:41:13.480
Cool.

00:41:13.480 --> 00:41:21.900
Well, I was one of the 27,000 before because my packages on PyPI are not super significant, but they are there.

00:41:22.480 --> 00:41:27.040
And so I definitely put 2FA on there and just have it running through my phone, basically.

00:41:27.040 --> 00:41:27.620
Appreciate that.

00:41:27.620 --> 00:41:27.940
Yeah.

00:41:27.940 --> 00:41:32.500
So people can go and see the progress here of how it's coming along.

00:41:32.500 --> 00:41:32.760
Yeah.

00:41:32.760 --> 00:41:35.520
And, you know, how many projects we've classified is critical, right?

00:41:35.520 --> 00:41:36.720
Like how many is 1%?

00:41:36.720 --> 00:41:42.640
Well, right now it's like almost 4,000 projects, which, you know, it's not a ton, but there's a lot of maintainers of those projects.

00:41:43.200 --> 00:41:44.400
Well, two things that are interesting.

00:41:44.400 --> 00:41:55.320
One is I think I can go to pypi.org and see 393,000 and say, well, 3,930 are probably critical, right?

00:41:55.320 --> 00:41:56.580
As that designation.

00:41:56.580 --> 00:41:59.920
But as you said, it's computed over time.

00:41:59.920 --> 00:42:04.340
So maybe there's something that was critical, but is no longer or something becomes critical, right?

00:42:04.340 --> 00:42:09.580
So this number could sort of outpace the actual number of just 1% of the total projects.

00:42:09.580 --> 00:42:09.780
Yeah.

00:42:09.900 --> 00:42:10.100
Yeah.

00:42:10.100 --> 00:42:10.340
Yeah.

00:42:10.340 --> 00:42:17.020
It will grow above 1% over time because, yeah, if something's been designated as critical, it just retains that designation indefinitely.

00:42:17.020 --> 00:42:23.260
And the other thing that we kind of snuck in here is that like anything that's a dependency of PyPI itself is also critical.

00:42:23.260 --> 00:42:26.420
So we just figured that would be a good idea for us.

00:42:26.420 --> 00:42:32.360
So, yeah, there's a couple of projects that maybe wouldn't normally be included, but we include them because like we personally care.

00:42:32.360 --> 00:42:34.980
Is that like maybe pyramid or stuff like that?

00:42:34.980 --> 00:42:35.200
Yeah.

00:42:35.200 --> 00:42:39.080
I don't know what the difference between those two sets is necessarily, but it would be interesting to figure out.

00:42:39.220 --> 00:42:42.120
If there's anything that potentially wasn't, then it is because of that.

00:42:42.120 --> 00:42:42.360
Okay.

00:42:42.360 --> 00:42:42.640
Yeah.

00:42:42.640 --> 00:42:49.720
The other thing that's interesting is there's 8,400 users identified as critical, even though there's 3,900 packages.

00:42:49.720 --> 00:42:54.040
So I guess because multiple people can be designated as a maintainer, huh?

00:42:54.040 --> 00:42:54.640
Yeah, exactly.

00:42:54.640 --> 00:42:59.560
So like it looks like the average is about two maintainers per critical project, which is a little scary.

00:42:59.560 --> 00:43:03.780
I think in that reality, there's a lot that just have one and a lot that have a lot more.

00:43:03.780 --> 00:43:04.080
Yeah.

00:43:04.080 --> 00:43:04.120
Yeah.

00:43:04.120 --> 00:43:06.680
It's a bimodal sort of distribution, right?

00:43:06.680 --> 00:43:10.340
There's a whole bunch by the one maintainer than these groups of people.

00:43:10.340 --> 00:43:10.780
Yeah.

00:43:10.780 --> 00:43:11.160
Yes.

00:43:11.160 --> 00:43:11.640
Interesting.

00:43:11.640 --> 00:43:12.080
Okay.

00:43:12.080 --> 00:43:12.460
This is cool.

00:43:12.460 --> 00:43:14.380
So people can check this out and see how it's going.

00:43:14.940 --> 00:43:21.220
This whole idea of critical and requiring 2FA, this npm is also doing something like this,

00:43:21.220 --> 00:43:21.360
right?

00:43:21.360 --> 00:43:22.960
So it's not completely out of the blue.

00:43:22.960 --> 00:43:23.280
Yeah.

00:43:23.280 --> 00:43:25.100
I mean, a lot of organizations are doing it.

00:43:25.100 --> 00:43:29.360
I think RubyGems said that they were working on a mandate or had proposed one as well.

00:43:29.600 --> 00:43:32.580
And yeah, NPM, they started with a pretty small cohort.

00:43:32.580 --> 00:43:36.300
It's only the top 100 projects, but they're going to expand that.

00:43:36.300 --> 00:43:40.760
And then the big one is, I think a lot of people aren't aware of this, but like GitHub

00:43:40.760 --> 00:43:45.460
announced that they are going to require 2FA for anyone that contributes code on GitHub,

00:43:45.460 --> 00:43:47.360
which is like, I guess everyone that uses GitHub.

00:43:47.360 --> 00:43:47.720
I don't know.

00:43:47.720 --> 00:43:52.200
I don't know what the user is that doesn't contribute code on GitHub, but like, yeah, everyone's

00:43:52.200 --> 00:43:54.800
going to have to have 2FA enabled, which is huge by the end of 2023.

00:43:55.040 --> 00:43:58.860
So they have some time, but like, I can't comprehend the size of the support team they

00:43:58.860 --> 00:44:01.040
must be hiring right now to satisfy that.

00:44:01.040 --> 00:44:02.320
Because that's like, it's crazy.

00:44:02.320 --> 00:44:05.760
Well, probably a lot of support team and a lot of automation that they're trying to get

00:44:05.760 --> 00:44:08.300
in place to, okay, here's what you got to do.

00:44:08.300 --> 00:44:08.780
But yeah.

00:44:08.780 --> 00:44:09.040
Yeah.

00:44:09.040 --> 00:44:13.720
The kind of, the secret here is that like, we have a, so via the OpenSF, it has a bunch

00:44:13.720 --> 00:44:17.660
of work groups and we have a new one that's pretty fun for me because it's about securing

00:44:17.660 --> 00:44:18.520
software repositories.

00:44:18.520 --> 00:44:22.720
But essentially, you know, like everyone that maintains a software repository, including folks from

00:44:22.720 --> 00:44:26.480
NPM, crates, RubyGems, like Maven Central, all that stuff.

00:44:26.480 --> 00:44:30.920
They all, we all come and talk twice a week or once every two weeks and, you know, talk

00:44:30.920 --> 00:44:31.540
about this kind of stuff.

00:44:31.540 --> 00:44:36.040
So like, we've been all talking about like 2FA mandates for a while now and like kind of

00:44:36.040 --> 00:44:38.560
working on our plans together and sharing notes and that kind of thing.

00:44:38.560 --> 00:44:38.820
Yeah.

00:44:38.820 --> 00:44:39.280
Fantastic.

00:44:39.280 --> 00:44:46.860
I only think that there's probably some users who just clone repos and post issues, but

00:44:46.860 --> 00:44:48.660
don't really, don't do any check-ins.

00:44:48.660 --> 00:44:49.400
They're just there to.

00:44:49.400 --> 00:44:49.780
Yeah.

00:44:49.780 --> 00:44:50.980
Just to save a copy.

00:44:50.980 --> 00:44:51.220
Yeah.

00:44:51.560 --> 00:44:51.960
Yeah.

00:44:51.960 --> 00:44:56.680
That's actually probably not the main use, right?

00:44:56.680 --> 00:45:01.880
Probably the main use is people contributing to their private repos or to maybe even less

00:45:01.880 --> 00:45:04.900
sort of public repos, probably mostly private repos, I would guess.

00:45:04.900 --> 00:45:05.620
Okay.

00:45:05.620 --> 00:45:07.400
But yeah, that's going to be a big, big deal.

00:45:07.400 --> 00:45:14.720
And it also, I guess it leads us into the whole supply integrity side of things, right?

00:45:14.720 --> 00:45:21.880
Because it's one thing to say, your account on PyPI has to be secured with 2FA and better

00:45:21.880 --> 00:45:22.360
security.

00:45:22.360 --> 00:45:29.460
But if somebody can just put in bad code through a very complex PR that happens to sneak, you

00:45:29.460 --> 00:45:33.900
know, a less than where there used to be a greater than, you know, some weird, weird little

00:45:33.900 --> 00:45:39.600
edge case into a PR or take over somebody's GitHub account, change the code and they don't

00:45:39.600 --> 00:45:40.640
realize it, right?

00:45:40.640 --> 00:45:44.680
These, that's probably more likely because there's no notification that you did your own

00:45:44.680 --> 00:45:46.980
commit in a significant way, right?

00:45:46.980 --> 00:45:52.320
So locking down GitHub will have really important knock-on effects for Python and all the other

00:45:52.320 --> 00:45:54.960
open source package locations, right?

00:45:55.060 --> 00:45:59.820
And there's a lot of work being done here just in terms of like making your source depository

00:45:59.820 --> 00:46:03.280
more secure and making your builds, you know, if you're building artifacts on GitHub, making

00:46:03.280 --> 00:46:07.780
those more secure, you know, making your publishing method more secure, like all of it is getting

00:46:07.780 --> 00:46:09.020
a ton of improvement right now.

00:46:09.020 --> 00:46:09.300
Yeah.

00:46:09.300 --> 00:46:16.060
I mean, you don't want something sneaking into mixing some code in a CI step or any of those

00:46:16.060 --> 00:46:16.540
types of things.

00:46:16.540 --> 00:46:16.700
Yeah.

00:46:16.700 --> 00:46:19.060
So there's so many insertion points, right?

00:46:19.060 --> 00:46:20.220
Yes, exactly.

00:46:20.220 --> 00:46:20.880
There's a lot of spots.

00:46:20.880 --> 00:46:21.260
Exactly.

00:46:21.720 --> 00:46:24.460
So I guess, what are some of the thoughts on what can be done?

00:46:24.460 --> 00:46:27.500
You know, like this is kind of what I was going back to, like, oh, there should be this world

00:46:27.500 --> 00:46:30.120
with no encryption and like lacks or no passwords.

00:46:30.120 --> 00:46:31.860
And now all of a sudden we need all this junk.

00:46:31.860 --> 00:46:35.820
And I feel like there's a little bit of that with people going, well, let's, let's just try

00:46:35.820 --> 00:46:36.820
to abuse PyPI.

00:46:36.820 --> 00:46:40.320
Let's try to abuse npm and sneak in what we can.

00:46:40.320 --> 00:46:45.720
And you maybe talk about some of the steps you all have taken to mitigate that and, you know,

00:46:45.720 --> 00:46:46.920
what you think can be done.

00:46:46.920 --> 00:46:50.000
There's a lot of talk about package signing, which I'm not so sure.

00:46:50.560 --> 00:46:52.180
Straight up signing is how useful that is.

00:46:52.180 --> 00:46:54.400
But yeah, maybe let's start at like whatever you thought about.

00:46:54.400 --> 00:46:57.620
The thing I want to start with is just like, none of these is a pancia, right?

00:46:57.620 --> 00:47:01.940
Like, so I think one of the arguments that was raised with the two factor stuff was just

00:47:01.940 --> 00:47:06.620
like, well, this isn't going to protect like us from a vulnerability or like a maintainer

00:47:06.620 --> 00:47:07.180
going rogue.

00:47:07.180 --> 00:47:09.000
It's like, yeah, no, obviously not.

00:47:09.000 --> 00:47:11.140
Like there's not one thing that's going to protect you from that.

00:47:11.140 --> 00:47:14.380
It's like we, there's a combination of features that are going to protect you from a combination

00:47:14.380 --> 00:47:16.160
of threat actors or vectors.

00:47:16.160 --> 00:47:19.820
And we have to use all of them if you want to like feel fully protected.

00:47:20.000 --> 00:47:24.300
So yeah, I think like we just spent a lot of time talking about 2FA, but like, like I

00:47:24.300 --> 00:47:26.220
said, it eliminates entire classes of attacks.

00:47:26.220 --> 00:47:27.720
So like, please turn on 2FA.

00:47:27.720 --> 00:47:28.180
Yeah.

00:47:28.180 --> 00:47:32.460
Another one, like since we're talking about GitHub, an interesting one is the security hardening

00:47:32.460 --> 00:47:33.300
with OIDC.

00:47:33.300 --> 00:47:35.380
If you've heard of OIDC, it's OpenID Connect.

00:47:35.380 --> 00:47:37.300
It's kind of built on OAuth protocol.

00:47:37.480 --> 00:47:40.640
But essentially like it allows you to give things identity.

00:47:40.640 --> 00:47:45.620
So like each individual GitHub Action workflow, like each run of the workflow actually gets

00:47:45.620 --> 00:47:46.380
its own identity.

00:47:46.380 --> 00:47:49.480
And that identity is like cryptographically verifiable.

00:47:49.480 --> 00:47:55.560
So IPIs, we're working on implementing support right now and it will exist very soon for what's

00:47:55.560 --> 00:47:59.220
essentially going to be credentialist publication from a GitHub Actions workflow.

00:47:59.220 --> 00:47:59.560
Okay.

00:47:59.820 --> 00:48:02.880
So that means like no password, no API token, nothing.

00:48:02.880 --> 00:48:08.080
You essentially say, okay, I trust this workflow and it has the ability to publish directly

00:48:08.080 --> 00:48:09.200
in a secure way.

00:48:09.200 --> 00:48:10.020
It's super cool.

00:48:10.020 --> 00:48:14.640
And then like a lot of other CI providers will hopefully support for OIDC as well.

00:48:14.640 --> 00:48:17.400
So yeah, we'll probably see this in a bunch of places.

00:48:17.400 --> 00:48:17.860
All right.

00:48:17.860 --> 00:48:22.220
So short-lived tokens directly from your cloud provider.

00:48:22.220 --> 00:48:22.840
Yeah, essentially.

00:48:22.840 --> 00:48:26.340
It works for like Google Cloud right now as well and a couple other things.

00:48:26.340 --> 00:48:31.220
But yeah, it's essentially like a way to verify the identity of like a really tightly

00:48:31.220 --> 00:48:33.920
scoped thing, like an actions run.

00:48:33.920 --> 00:48:38.780
And then, you know, verify it and authenticate it and give it the permission to like do something

00:48:38.780 --> 00:48:39.840
like publish to PyPI.

00:48:39.840 --> 00:48:40.680
This looks useful.

00:48:40.680 --> 00:48:47.760
I mean, handing out API keys or embedding username passwords into CI, CD doesn't sound like a

00:48:47.760 --> 00:48:48.220
great idea.

00:48:48.220 --> 00:48:48.840
Yeah, no.

00:48:48.840 --> 00:48:52.940
Especially with stuff like CodeCoveAttack or Travis CI had a similar attack where like all

00:48:52.940 --> 00:48:54.480
environment variables were exposed.

00:48:54.480 --> 00:48:57.300
Like everyone's got to just go and roll everything at that point.

00:48:57.300 --> 00:48:58.160
It's like such a mess.

00:48:58.160 --> 00:48:58.920
So yeah.

00:48:58.920 --> 00:49:01.820
Well, and it's one thing to say, okay, we've got to go reset your password.

00:49:01.820 --> 00:49:02.580
Fine.

00:49:02.580 --> 00:49:04.040
There is a race.

00:49:04.040 --> 00:49:08.080
I mean, as soon as that happens, there's a race from people looking to use those credentials

00:49:08.080 --> 00:49:11.640
and you looking to not have them used by other than you.

00:49:11.640 --> 00:49:14.680
The real problem, I think, is not the people who are paying attention.

00:49:14.680 --> 00:49:19.240
It's probably the maintainer who set up some project and hasn't touched it in a year and

00:49:19.240 --> 00:49:20.220
hasn't checked their email.

00:49:20.380 --> 00:49:22.480
They're just kind of not super engaged.

00:49:22.480 --> 00:49:25.860
That could stay open for a super long time in a bad way.

00:49:25.860 --> 00:49:26.200
Exactly.

00:49:26.200 --> 00:49:26.760
All right.

00:49:26.760 --> 00:49:27.540
Package signing.

00:49:27.540 --> 00:49:32.280
Donald Stuff talked about package signing and said, why package signing is not the holy

00:49:32.280 --> 00:49:32.560
grail.

00:49:32.560 --> 00:49:36.300
People just say it's a little bit like the 2FA stuff.

00:49:36.300 --> 00:49:40.440
Like if you just sign your packages to prove they come from you, everything is going to be

00:49:40.440 --> 00:49:43.820
fine unless the person just goes rogue.

00:49:43.940 --> 00:49:45.700
Like you can't protect from crazy.

00:49:45.700 --> 00:49:49.520
There was one of the packages that got messed up.

00:49:49.520 --> 00:49:50.480
I think it was on PyPI.

00:49:50.480 --> 00:49:56.080
That same person was arrested for like bomb making materials in New York.

00:49:56.080 --> 00:49:56.820
I don't know about this.

00:49:56.820 --> 00:49:57.900
Let's see if I can find the article.

00:49:57.900 --> 00:49:59.060
Let's see if I can find the article.

00:49:59.060 --> 00:50:04.620
It's like, clearly that's not a well-functioning sort of person.

00:50:04.620 --> 00:50:08.720
And it doesn't matter if they sign the package, if they just go bonkers, you know?

00:50:08.720 --> 00:50:09.040
Yeah.

00:50:09.080 --> 00:50:11.840
It's not the attack that we're trying to protect against with signing.

00:50:11.840 --> 00:50:12.260
But yeah.

00:50:12.260 --> 00:50:14.520
What can signing help with and what is it not going to help us with?

00:50:14.520 --> 00:50:14.820
Yeah.

00:50:14.820 --> 00:50:20.120
Donald wrote this post that's kind of like canon at this point for why package signing is not

00:50:20.120 --> 00:50:20.440
the holy grail.

00:50:20.440 --> 00:50:21.800
But he's really talking about GPG.

00:50:21.800 --> 00:50:24.500
Well, why GPG signing is not the holy grail.

00:50:24.500 --> 00:50:25.560
There's good points in here, right?

00:50:25.560 --> 00:50:29.840
Like so, like you said, package signing doesn't protect against an actual account compromise.

00:50:29.840 --> 00:50:33.320
If someone compromised your GPG key, like they can sign whatever they want.

00:50:33.320 --> 00:50:34.900
So there's no protections there.

00:50:34.900 --> 00:50:37.300
And then there's other problems with GPG as well.

00:50:37.300 --> 00:50:39.720
Like there's UX and usability issues.

00:50:39.720 --> 00:50:43.420
There's issues with web of trust, like actually establishing, okay, great.

00:50:43.420 --> 00:50:48.020
Like you sign this thing, but how do I establish that the person that signed it is actually a

00:50:48.020 --> 00:50:52.160
person that I trust and not someone's provided me with a malicious public key, right?

00:50:52.160 --> 00:50:53.360
Like how do you actually do that?

00:50:53.360 --> 00:50:53.660
Sorry.

00:50:53.660 --> 00:50:57.460
It's a little equivalent to people saying, well, just check that it's HTTPS for the URL.

00:50:57.460 --> 00:50:59.180
Like, well, that's not identity.

00:50:59.180 --> 00:51:00.380
That's just like encryption.

00:51:00.380 --> 00:51:00.880
Yeah.

00:51:00.880 --> 00:51:03.220
You can still serve crap over HTTPS.

00:51:03.220 --> 00:51:03.440
Yeah.

00:51:03.440 --> 00:51:08.640
I think a lot of people also, they don't realize that like PyPI has supported uploading

00:51:08.640 --> 00:51:12.860
GPG signatures for like a very long time, still does, hasn't gone away.

00:51:12.860 --> 00:51:14.400
And nobody does it.

00:51:14.400 --> 00:51:15.920
It's just like, it's just not used.

00:51:15.920 --> 00:51:16.560
Nobody does it.

00:51:16.560 --> 00:51:17.420
It's too hard.

00:51:17.420 --> 00:51:18.340
Tooling doesn't work right.

00:51:18.340 --> 00:51:20.300
Or it's just like not worth doing.

00:51:20.300 --> 00:51:23.880
So I think Donald's post is right in the context of a world.

00:51:23.880 --> 00:51:24.880
And this is written in 2013.

00:51:24.960 --> 00:51:30.660
So a world where like only GPG is the only signing feature or signing tooling that you

00:51:30.660 --> 00:51:31.200
have available.

00:51:31.200 --> 00:51:32.280
But that's not true anymore.

00:51:32.280 --> 00:51:37.480
And I'm really excited about this new tech called SigStore, partly because I like, I work

00:51:37.480 --> 00:51:37.700
with it.

00:51:37.700 --> 00:51:39.080
People on my team work on it.

00:51:39.080 --> 00:51:40.740
It's like really interesting technology.

00:51:40.740 --> 00:51:44.960
But SigStore is essentially like a new way to sign things.

00:51:44.960 --> 00:51:49.300
And it's not necessarily based on long-lived maintained keys.

00:51:49.300 --> 00:51:50.920
It actually uses ephemeral keys.

00:51:50.920 --> 00:51:54.620
Like when you sign something with SigStore, you generate a public private key pair, you

00:51:54.620 --> 00:51:56.520
sign it very quickly, and then you throw those away.

00:51:56.520 --> 00:51:57.840
Like you don't actually maintain them.

00:51:57.840 --> 00:51:58.600
They can't get leaked.

00:51:58.600 --> 00:51:59.920
They don't ever get rid of the disk.

00:51:59.920 --> 00:52:00.780
Like nothing.

00:52:00.780 --> 00:52:01.400
They don't exist.

00:52:01.400 --> 00:52:01.980
That's interesting.

00:52:01.980 --> 00:52:03.680
This is also based on OIDC.

00:52:03.680 --> 00:52:09.760
The chain of ownership of those keys, the keys like provided by some other trusted key

00:52:09.760 --> 00:52:12.300
or something that really is tied to you, something like that?

00:52:12.300 --> 00:52:12.840
No, no, no.

00:52:12.840 --> 00:52:16.800
It's just literally just a key that you generate out of thin air, you sign it, and then you throw

00:52:16.800 --> 00:52:17.060
it away.

00:52:17.060 --> 00:52:21.440
So the way that SigStore works, which is it's also built on top of OIDC.

00:52:21.440 --> 00:52:23.360
So you have these identities, right?

00:52:23.360 --> 00:52:27.540
You have like your email, your Gmail account, your GitHub account, that kind of thing.

00:52:27.540 --> 00:52:30.600
And they all offer like essentially an online identity.

00:52:30.600 --> 00:52:32.700
You essentially sign with these identities instead.

00:52:32.700 --> 00:52:37.600
So like you sign into something like Gmail, sign in something like GitHub.

00:52:38.020 --> 00:52:42.740
You share this identity with a certificate authority that SigStore runs.

00:52:42.740 --> 00:52:48.540
And this binds the identity to this like one time ephemeral private public key pair that

00:52:48.540 --> 00:52:49.120
you generated.

00:52:49.120 --> 00:52:51.940
Then that certificate is published on a transparency log.

00:52:51.940 --> 00:52:52.740
So it's there forever.

00:52:52.740 --> 00:52:54.520
There's a record of everything that gets signed.

00:52:54.520 --> 00:53:01.040
But then the thing that you have to trust is not like some however many digit long alphanumeric

00:53:01.040 --> 00:53:02.080
like public key ID.

00:53:02.600 --> 00:53:03.780
It's like an email address.

00:53:03.780 --> 00:53:05.640
It's like di at python dot org.

00:53:05.640 --> 00:53:09.920
You can like be pretty sure that someone hasn't, you know, and someone can still lose access

00:53:09.920 --> 00:53:13.060
to that identity via like compromise of that account.

00:53:13.060 --> 00:53:16.940
But it's like a little, it's much, much easier to use, sign, and maintain.

00:53:16.940 --> 00:53:21.180
And it's a little bit less likely that that actual identity is going to get compromised.

00:53:21.180 --> 00:53:21.700
Okay.

00:53:21.700 --> 00:53:23.440
Here's the one that I was thinking of.

00:53:23.440 --> 00:53:23.580
Sorry.

00:53:23.580 --> 00:53:24.900
It was NPM, not PyPI.

00:53:24.900 --> 00:53:25.300
Oh.

00:53:25.300 --> 00:53:30.960
NPM libraries, colors and fakers, sabotage, sabotage and protest by maintainer, a person

00:53:30.960 --> 00:53:32.140
named Mark Squires.

00:53:32.140 --> 00:53:36.140
And follow it up quickly from that.

00:53:36.140 --> 00:53:42.800
We have the resident of Queens home suspected in bomb making materials, arrested Mark Squires

00:53:42.800 --> 00:53:43.440
and so on.

00:53:43.520 --> 00:53:48.260
So like, this is what I was talking about when I'm saying like the no amount of signing

00:53:48.260 --> 00:53:52.920
or, you know, 2FA is going to help against this.

00:53:52.920 --> 00:53:56.980
And it's, it's just something, I mean, it's just, I think it's part of the deal.

00:53:56.980 --> 00:54:00.220
If you accept code from people, you got to vet that, you know?

00:54:00.220 --> 00:54:03.160
I've essentially like always said, you know, PyPI makes no guarantees.

00:54:03.160 --> 00:54:04.600
You can't trust anything that's on it.

00:54:04.600 --> 00:54:09.360
You need to like take your own steps to learn and build trust and things that are on there.

00:54:09.360 --> 00:54:11.200
We can give you some tools to help you do that.

00:54:11.260 --> 00:54:15.460
But like, yeah, essentially you're giving someone commit access to your project, your

00:54:15.460 --> 00:54:16.220
application, whatever.

00:54:16.220 --> 00:54:22.200
Like you're allowing them to introduce code into your project and run alongside your application

00:54:22.200 --> 00:54:22.580
or.

00:54:22.580 --> 00:54:29.920
What are your thoughts on private hosted package systems like PyPI dev server or whatever

00:54:29.920 --> 00:54:33.680
those, those, those where you can create the local ones and maybe even mirror stuff from

00:54:33.680 --> 00:54:33.980
PyPI.

00:54:33.980 --> 00:54:34.280
Yeah.

00:54:34.280 --> 00:54:38.420
I mean, if you're going to do like a really robust, like auditing pipeline where you pull

00:54:38.420 --> 00:54:41.000
stuff from public PyPI, you spend some time looking at it.

00:54:41.140 --> 00:54:44.480
You maybe run it through some tests and then you introduce it to your private server.

00:54:44.480 --> 00:54:48.140
That's a really good way to like insulate yourself from a couple of different types of attacks,

00:54:48.140 --> 00:54:50.800
like dependency confusion attacks, typosquatting, that kind of thing.

00:54:50.800 --> 00:54:55.280
Like if you're pointing your install at private server that just doesn't have any of that stuff

00:54:55.280 --> 00:54:59.920
on it because you have manually curated it, then yeah, I mean, that's, that's a pretty good

00:54:59.920 --> 00:55:00.200
practice.

00:55:00.200 --> 00:55:04.480
And so like a lot of people like Artifactory, Google Cloud, Artifact Registry, all those things,

00:55:04.480 --> 00:55:06.280
like they're all sort of similar in that regard.

00:55:06.460 --> 00:55:11.780
I guess related to that is if you automatically install the latest continuously, that maybe

00:55:11.780 --> 00:55:16.720
puts you at a higher level of risk than if you choose to upgrade at some point to a package,

00:55:16.720 --> 00:55:18.340
like pinning versus not pinning.

00:55:18.340 --> 00:55:19.000
Yeah, exactly.

00:55:19.000 --> 00:55:19.380
Yeah.

00:55:19.380 --> 00:55:21.440
We're getting pretty short on time here, Dustin.

00:55:21.440 --> 00:55:22.040
Yeah.

00:55:22.220 --> 00:55:23.040
What else can we cover?

00:55:23.040 --> 00:55:28.980
I know maybe pip on it or scorecard or what do you want to focus on for a couple of minutes

00:55:28.980 --> 00:55:29.140
here?

00:55:29.140 --> 00:55:33.640
Well, yeah, let me just quickly say that like 6door, we built a Python client, so you can

00:55:33.640 --> 00:55:37.580
pip install 6door and then sign, verify, do whatever from there.

00:55:37.580 --> 00:55:42.660
And I'm super excited to say that like the upcoming Python 3.11 release is, the releases

00:55:42.660 --> 00:55:46.080
are usually signed with GPG, but we're going to start signing it with 6door as well.

00:55:46.080 --> 00:55:46.980
The release maintainer.

00:55:46.980 --> 00:55:47.400
Oh, nice.

00:55:47.400 --> 00:55:48.880
Pablo is going to sign it.

00:55:48.880 --> 00:55:50.320
Can it be signed with two things at once?

00:55:50.320 --> 00:55:51.420
I guess it can, yeah.

00:55:51.420 --> 00:55:52.760
Yeah, multiple people can sign it.

00:55:52.760 --> 00:55:54.720
So we're just going to have their release manager sign it.

00:55:54.720 --> 00:55:56.120
But yeah, that's super exciting.

00:55:56.120 --> 00:55:56.740
Yeah.

00:55:56.740 --> 00:56:01.080
So another like area that we've been working on a lot lately, I've been working on is vulnerability

00:56:01.080 --> 00:56:02.300
auditing or remediation.

00:56:02.300 --> 00:56:04.100
So we have a new tool called pip audit.

00:56:04.100 --> 00:56:05.440
It's not part of pip right now.

00:56:05.440 --> 00:56:07.820
And there's some discussion about whether it should be or not.

00:56:07.820 --> 00:56:12.260
But essentially, this is a tool that allows you to audit your local environment, your Docker

00:56:12.260 --> 00:56:16.900
container requirements file, whatever for known vulnerabilities, not like unknown vulnerabilities,

00:56:16.900 --> 00:56:21.000
but stuff that's been known, reported and either fixed or upgraded.

00:56:21.340 --> 00:56:24.840
So it'll tell you like essentially if it finds a CD or something like that.

00:56:24.840 --> 00:56:30.480
But this also uses like a Python specific advisory database that we built that pairs with the open

00:56:30.480 --> 00:56:33.540
source vulnerability service and works pretty well.

00:56:33.540 --> 00:56:34.600
I'm pretty pleased with it.

00:56:34.600 --> 00:56:38.400
I would like encourage everyone to just like run it on their machine and see what vulnerabilities

00:56:38.400 --> 00:56:40.140
you have like lurking about right now.

00:56:40.140 --> 00:56:44.680
But also like integrate it into your CI pipeline, like run an audit, just make sure that, you

00:56:44.680 --> 00:56:47.640
know, your application is not going to like have a vulnerability introduced.

00:56:47.640 --> 00:56:50.660
It pairs really nicely with like depend about things as well.

00:56:50.660 --> 00:56:53.260
Yeah, there's a bunch of other stuff like working on Salsa.

00:56:53.260 --> 00:56:58.400
If you're not familiar with Salsa, it's essentially like a framework for thinking about how secure

00:56:58.400 --> 00:57:01.060
your build pipeline is when you're producing publishing artifacts.

00:57:01.060 --> 00:57:04.640
So if you're a maintainer, you might think about whether, you know, tampering is possible,

00:57:04.740 --> 00:57:05.300
that kind of thing.

00:57:05.300 --> 00:57:09.580
It's a sort of a way just to sort of think about how good of a job the build pipeline is

00:57:09.580 --> 00:57:10.340
doing in that regard.

00:57:10.340 --> 00:57:10.640
Yeah.

00:57:10.640 --> 00:57:12.140
Build pipelines are a little scary.

00:57:12.140 --> 00:57:16.800
I mean, they have a huge value, but they're also, you could sneak stuff in without even

00:57:16.800 --> 00:57:20.240
actually changing the code of the original repo and all sorts of stuff.

00:57:20.240 --> 00:57:20.440
Yeah.

00:57:20.520 --> 00:57:26.020
I saw this is managed by, or like in collaboration with Trail of Bits and also you're a maintainer.

00:57:26.020 --> 00:57:28.660
What's the, what's the origin story of pip Audit?

00:57:28.660 --> 00:57:28.940
Yeah.

00:57:28.940 --> 00:57:29.200
Yeah.

00:57:29.200 --> 00:57:31.660
So Trail of Bits is a security consultancy.

00:57:31.660 --> 00:57:35.720
They've done a lot of work in the Python space and software security space for a long

00:57:35.720 --> 00:57:36.220
time now.

00:57:36.220 --> 00:57:41.100
Folks like William Woodruff were actually involved in like way back when implementing two

00:57:41.100 --> 00:57:42.700
factor and some other stuff on PyPI.

00:57:43.200 --> 00:57:47.880
So my team, open source security team at Google, we've hired them as contractors to

00:57:47.880 --> 00:57:51.100
do some of this work, do some maintenance, build these open source projects, that kind

00:57:51.100 --> 00:57:51.380
of thing.

00:57:51.380 --> 00:57:56.280
So you'll see William and Alex and some other folks all over these projects because they've

00:57:56.280 --> 00:58:00.620
been working really hard to make them really useful and work really well and be really secure.

00:58:00.620 --> 00:58:00.920
Yeah.

00:58:00.920 --> 00:58:01.380
Fantastic.

00:58:01.380 --> 00:58:06.240
Let's maybe just round things out with the stuff on the PEP, the various PEPs.

00:58:06.240 --> 00:58:06.680
Oh yeah.

00:58:06.680 --> 00:58:07.540
Maybe the API.

00:58:07.540 --> 00:58:08.000
Yeah.

00:58:08.000 --> 00:58:09.120
Let's go through the PEPs real quick.

00:58:09.120 --> 00:58:10.660
Then we'll probably have covered enough.

00:58:10.660 --> 00:58:10.860
Yeah.

00:58:10.860 --> 00:58:12.620
I stuck my name on a couple of PEPs recently.

00:58:12.760 --> 00:58:13.620
Or as much we have time for.

00:58:13.620 --> 00:58:13.940
Yeah.

00:58:13.940 --> 00:58:18.440
I think most of these are, you know, I've like provided some like minimal input into

00:58:18.440 --> 00:58:18.620
them.

00:58:18.620 --> 00:58:22.260
I don't, can't say I claim that I authored them myself, but what I'm super excited about

00:58:22.260 --> 00:58:23.360
is PEP 621.

00:58:23.360 --> 00:58:27.860
So this is a way to do essentially static metadata for Python packages.

00:58:27.860 --> 00:58:31.880
So this includes source distributions, which means that like you don't have to use setup.py

00:58:31.880 --> 00:58:32.180
anymore.

00:58:32.180 --> 00:58:34.880
You don't know why you don't need to use setup.py anymore.

00:58:34.880 --> 00:58:39.900
It's essentially arbitrary code execution at install time, which is super scary and should not

00:58:39.900 --> 00:58:40.180
happen.

00:58:40.180 --> 00:58:41.420
So yeah.

00:58:41.420 --> 00:58:42.000
621.

00:58:42.320 --> 00:58:46.260
I compare that thought with this resident of Queens for bomb making.

00:58:46.260 --> 00:58:50.140
Like, do you want to have that person running arbitrary code on your machine?

00:58:50.140 --> 00:58:51.420
Probably no, is the answer.

00:58:51.420 --> 00:58:52.620
At install time too.

00:58:52.620 --> 00:58:52.820
In production.

00:58:52.820 --> 00:58:53.700
Not at run time.

00:58:53.700 --> 00:58:54.180
Yeah, yeah.

00:58:54.180 --> 00:58:55.100
Yeah, exactly.

00:58:55.100 --> 00:58:58.840
So I actually just saw, like right before I joined this, I saw a tweet from Prajan that

00:58:58.840 --> 00:59:00.700
like setup tools has full support for this.

00:59:00.700 --> 00:59:03.000
There's a bunch of folks been working really hard on it.

00:59:03.000 --> 00:59:07.780
So yeah, it's essentially like you don't need to use setup.py anymore, which is really nice.

00:59:07.780 --> 00:59:12.580
And like a lot of tools have sort of converged on highproject.toml as the sort of best standard

00:59:12.580 --> 00:59:14.060
for metadata and configuration.

00:59:14.060 --> 00:59:16.340
So like, it's nice to see that conversions.

00:59:16.340 --> 00:59:17.380
Yeah, it's great.

00:59:17.380 --> 00:59:18.600
Shout out to like Brett.

00:59:18.600 --> 00:59:20.520
I think mostly led this PEP.

00:59:20.520 --> 00:59:21.820
He did an amazing job.

00:59:21.820 --> 00:59:22.220
Yeah.

00:59:22.220 --> 00:59:22.480
Yeah.

00:59:22.480 --> 00:59:22.840
Very cool.

00:59:22.840 --> 00:59:25.400
And then PEP 691 is exciting as well.

00:59:25.400 --> 00:59:27.420
So my API has a couple of different APIs.

00:59:27.420 --> 00:59:29.300
Most of them are not standardized.

00:59:29.300 --> 00:59:34.180
One of them that is standardized is a simple API, which is essentially just an HTML page

00:59:34.180 --> 00:59:35.520
and tools like pip.

00:59:35.520 --> 00:59:36.600
It's kind of insane.

00:59:36.600 --> 00:59:38.000
But if you go to a simple.

00:59:38.000 --> 00:59:39.120
Are you really going to do it?

00:59:39.120 --> 00:59:41.480
Yeah, that's going to blow up your browser for sure.

00:59:41.480 --> 00:59:43.180
Because like essentially, yeah.

00:59:43.180 --> 00:59:45.020
So tools don't actually use this page.

00:59:45.020 --> 00:59:46.960
They use the individual pages for these projects.

00:59:46.960 --> 00:59:50.760
But tools like pip essentially have to parse HTML to like interact with PyPI.

00:59:50.760 --> 00:59:51.820
And that's not great.

00:59:51.820 --> 00:59:52.620
Like it used to work.

00:59:52.620 --> 00:59:53.980
Okay, it doesn't scale well now.

00:59:53.980 --> 00:59:57.820
So we're in the process of like standardizing a lot of our JSON APIs.

00:59:57.820 --> 01:00:02.260
And one of them that we sort of did, and Donald led this path with some input from Pragen

01:00:02.260 --> 01:00:06.620
and Cooper and myself, essentially like the same data, the same API, same files, everything

01:00:06.620 --> 01:00:07.520
that pip needs.

01:00:07.520 --> 01:00:08.660
That's not HTML.

01:00:08.660 --> 01:00:09.360
It's just JSON.

01:00:09.360 --> 01:00:14.900
So they can use standard library JSON parser to request and get this and do the stuff that

01:00:14.900 --> 01:00:15.960
pip needs to do to be pip.

01:00:15.960 --> 01:00:20.440
Yeah, probably make it a little more efficient, easier for other people to consume, right?

01:00:20.440 --> 01:00:20.740
Yeah.

01:00:20.740 --> 01:00:26.860
Is that something that you encourage other applications to go mess with PyPI APIs?

01:00:26.860 --> 01:00:30.620
Or is it, it's public, but you know, we'd rather you don't mess with it.

01:00:30.620 --> 01:00:31.480
What are your thoughts there?

01:00:31.480 --> 01:00:34.700
Like the stuff that we have standardized, definitely you can depend on it, right?

01:00:34.700 --> 01:00:36.380
Like it will continue to exist.

01:00:36.380 --> 01:00:39.880
And by standardizing it, we've said like, this is what you should expect.

01:00:39.880 --> 01:00:43.140
And this, you know, unless we change the standard, it's what it's going to continue to do.

01:00:43.140 --> 01:00:46.320
We have other APIs that existed before we were standardizing stuff.

01:00:46.320 --> 01:00:47.920
Like there's a legacy JSON API.

01:00:48.080 --> 01:00:51.460
There's this XML RPC API that's like such a nightmare to maintain.

01:00:51.460 --> 01:00:54.040
We've kept them running just because a lot of people use them.

01:00:54.040 --> 01:00:58.420
Like for example, Poetry uses this like kind of our unofficial JSON API.

01:00:58.420 --> 01:01:03.300
And yeah, I mean, there's times when we need to do stuff to that because it like isn't scaling,

01:01:03.300 --> 01:01:03.640
right?

01:01:03.640 --> 01:01:05.580
Or there's trade-offs that we need to make.

01:01:05.580 --> 01:01:09.560
And it's like, well, like we're probably going to have to break someone in order to like

01:01:09.560 --> 01:01:10.700
keep this afloat.

01:01:10.820 --> 01:01:14.280
But like with things like these standard APIs, like we've spent a lot of time designing

01:01:14.280 --> 01:01:15.780
them, planning them, standardizing them.

01:01:15.780 --> 01:01:18.460
Like those are definitely 100% cool to integrate against.

01:01:18.460 --> 01:01:18.740
Okay.

01:01:18.740 --> 01:01:19.340
Fantastic.

01:01:19.340 --> 01:01:24.720
Traditionally on these peps, you'll see it's accepted and planned for this version of, this

01:01:24.720 --> 01:01:25.580
one doesn't have that, right?

01:01:25.580 --> 01:01:28.120
Because it just goes against the web app, right?

01:01:28.120 --> 01:01:29.240
This is a packaging PEP.

01:01:29.320 --> 01:01:33.300
So we use the same process as CPython for their Python enhancements.

01:01:33.300 --> 01:01:35.100
But yeah, this is about packaging.

01:01:35.100 --> 01:01:38.400
So it doesn't necessarily tie to an individual Python release.

01:01:38.400 --> 01:01:41.120
You don't ship it to a binary that people get, right?

01:01:41.120 --> 01:01:42.140
Well, in a way we do.

01:01:42.140 --> 01:01:44.280
Like pip, you know, ships support for it in a binary.

01:01:44.280 --> 01:01:45.680
So, and it's implemented.

01:01:45.680 --> 01:01:47.240
Like this exists on PyPI now.

01:01:47.240 --> 01:01:48.680
Pip uses this now.

01:01:48.680 --> 01:01:53.820
So if I have the latest pip on my machine and I pip install something or do pip actions,

01:01:53.820 --> 01:01:56.940
which does it hit the old symbol or does it hit this new one now?

01:01:56.940 --> 01:01:59.520
If I'm remembering correctly, the support's been added to pip.

01:01:59.520 --> 01:02:00.120
It might not.

01:02:00.120 --> 01:02:01.360
I think it's been released.

01:02:01.360 --> 01:02:01.740
Yeah.

01:02:01.740 --> 01:02:02.320
I'm pretty sure.

01:02:02.320 --> 01:02:02.920
I could be wrong.

01:02:02.920 --> 01:02:04.460
I'm not a pip maintainer, but yeah.

01:02:04.460 --> 01:02:05.320
Yeah, sure.

01:02:05.320 --> 01:02:05.960
Okay, cool.

01:02:05.960 --> 01:02:08.560
And we have one more, one more to touch on.

01:02:08.560 --> 01:02:09.120
Oh yeah.

01:02:09.120 --> 01:02:12.460
I mean, this is, you only really care about this if you're like really working with PyPI,

01:02:12.460 --> 01:02:15.460
but we're going to have a new, like I said, we're standardizing all our APIs.

01:02:15.460 --> 01:02:17.600
So there's going to be a new upload API.

01:02:17.600 --> 01:02:20.340
The existing upload API has a lot of problems.

01:02:20.340 --> 01:02:21.060
It's fairly old.

01:02:21.060 --> 01:02:24.440
It's essentially just a big post request with like a metadata in it.

01:02:24.440 --> 01:02:26.180
So this should be a little bit better.

01:02:26.440 --> 01:02:30.600
And also enable things like draft releases where you can like publish something to PyPI

01:02:30.600 --> 01:02:32.340
that's in draft state.

01:02:32.340 --> 01:02:35.820
You can review it, practice installing it before you actually like publish it.

01:02:35.820 --> 01:02:39.320
And when it's in draft, it'll allow you to overwrite it, you know, like do things that

01:02:39.320 --> 01:02:39.920
we don't.

01:02:39.920 --> 01:02:40.700
To fix the problem.

01:02:40.700 --> 01:02:40.960
Yeah.

01:02:40.960 --> 01:02:41.660
Yeah.

01:02:41.660 --> 01:02:42.100
Okay.

01:02:42.100 --> 01:02:42.420
Exactly.

01:02:42.540 --> 01:02:47.800
Is this an alternative replacement or just another safety net compared to say like the

01:02:47.800 --> 01:02:50.420
test PyPI versus production PyPI?

01:02:50.420 --> 01:02:52.200
You mean like the draft stuff?

01:02:52.200 --> 01:02:52.520
Yeah.

01:02:52.520 --> 01:02:53.640
I think it would be the preferred.

01:02:53.700 --> 01:02:58.080
Like would stuff automatically upload and draft and you got to flip it so that you don't

01:02:58.080 --> 01:03:02.180
accidentally publish something that's not ready or first because you forgot to use the test

01:03:02.180 --> 01:03:02.640
PyPI.

01:03:02.640 --> 01:03:03.060
Yeah.

01:03:03.060 --> 01:03:07.860
Test PyPI is kind of weird because like it actually existed to be our test environment for

01:03:07.860 --> 01:03:08.180
PyPI.

01:03:08.180 --> 01:03:11.140
Like because we didn't have a great test suite with the old PyPI.

01:03:11.480 --> 01:03:14.000
And now it sort of hangs around as like a playground sandbox.

01:03:14.000 --> 01:03:14.600
Yeah.

01:03:14.600 --> 01:03:15.520
We don't care.

01:03:15.520 --> 01:03:19.400
But yeah, some people do like have it in their production, in their like release flow, like

01:03:19.400 --> 01:03:22.400
upload here first, make sure that it works and then like go to.

01:03:22.400 --> 01:03:27.860
So I think this will be a better use case and sort of consolidate folks to use PyPI for

01:03:27.860 --> 01:03:28.140
everything.

01:03:28.140 --> 01:03:32.320
And we can eventually shut down test PyPI because it's not as useful as it could be.

01:03:32.320 --> 01:03:32.780
All right.

01:03:32.780 --> 01:03:36.460
Well, I think that's probably all the time we got to talk about this.

01:03:36.460 --> 01:03:37.620
We could go on and on.

01:03:37.620 --> 01:03:41.320
We only touched on some of the stuff that we're thinking about talking about.

01:03:41.880 --> 01:03:44.520
But before we're done, maybe answer the final two questions.

01:03:44.520 --> 01:03:46.780
If you're going to write some code, yeah, what editor do you use?

01:03:46.780 --> 01:03:51.100
I do almost everything in BI and have since I sort of started doing any kind of development

01:03:51.100 --> 01:03:51.440
work.

01:03:51.440 --> 01:03:56.260
That said, I like I kind of try to do as much as I can in the browser in the like GitHub

01:03:56.260 --> 01:03:56.680
UI.

01:03:56.680 --> 01:03:57.400
Okay.

01:03:57.400 --> 01:03:58.680
I hit, you know, speed bumps there.

01:03:58.680 --> 01:04:02.800
Sometimes it can't be quite as fast, but like for little stuff I like, yeah, I kind of

01:04:02.800 --> 01:04:04.440
like, I kind of like seeing what I can do there.

01:04:04.440 --> 01:04:04.740
Yeah.

01:04:04.740 --> 01:04:05.220
Right on.

01:04:05.220 --> 01:04:07.660
Do you ever press the dot in GitHub?

01:04:07.660 --> 01:04:08.200
Yeah.

01:04:08.200 --> 01:04:08.700
All the time.

01:04:08.700 --> 01:04:09.360
Love the dot.

01:04:09.360 --> 01:04:09.600
Yeah.

01:04:09.700 --> 01:04:12.580
The dot converts it to like hosted VS Code basically.

01:04:12.580 --> 01:04:13.120
Yeah.

01:04:13.120 --> 01:04:15.140
And then a package you want to give a shout out to?

01:04:15.140 --> 01:04:21.420
So my biased answer to this is check out the SIG store package on PyPI and the pip dash

01:04:21.420 --> 01:04:22.300
audit package.

01:04:22.300 --> 01:04:24.500
Those are both stuff that I've been working on.

01:04:24.500 --> 01:04:25.700
My team has been working on.

01:04:25.700 --> 01:04:27.520
I'm really proud of the way that they work.

01:04:27.580 --> 01:04:31.700
And we're going to be working on integrating those into like more use cases, more patterns.

01:04:31.700 --> 01:04:32.580
Check them out.

01:04:32.580 --> 01:04:33.300
Like try them out.

01:04:33.300 --> 01:04:34.740
I'd love to get feedback on them.

01:04:34.740 --> 01:04:35.040
Yeah.

01:04:35.040 --> 01:04:35.440
Fantastic.

01:04:35.440 --> 01:04:40.000
They seem to fill like a super big, a super important hole that sort of backfill some

01:04:40.000 --> 01:04:42.760
of the security and supply chain stability and so on.

01:04:42.820 --> 01:04:46.940
And then I think my unbiased answer, I did want to give a shout out to the pip-tools, pip dash

01:04:46.940 --> 01:04:48.120
tools project on PyPI.

01:04:48.280 --> 01:04:50.060
That's maintained by the Jazzband team.

01:04:50.060 --> 01:04:53.160
So that's just like a roving revolving door of maintainers.

01:04:53.160 --> 01:04:55.080
But they do a good job keeping it up and running.

01:04:55.080 --> 01:04:59.800
It satisfies this really, I think, important use case that a lot of people don't do with

01:04:59.800 --> 01:05:04.600
their Python dependencies, which is like essentially allows you to compile your dependencies into

01:05:04.600 --> 01:05:08.740
a requirements file that has all the versions pinned, all the sub dependencies there.

01:05:09.200 --> 01:05:14.300
Hashes, which is like, please like use tips, hash checker, hash all your dependencies.

01:05:14.300 --> 01:05:17.320
It definitely protects you against another whole class of attacks.

01:05:17.320 --> 01:05:19.220
But yeah, pip-tools is great for that.

01:05:19.220 --> 01:05:20.740
I would love a future where that was part of pip.

01:05:20.740 --> 01:05:23.720
I don't know if that's going to exist or not up to the maintainer.

01:05:23.720 --> 01:05:24.920
But yeah, it's super cool.

01:05:24.920 --> 01:05:28.980
I have switched to using pip-tools for all of my packages and I love it.

01:05:28.980 --> 01:05:29.540
It's fantastic.

01:05:29.540 --> 01:05:30.160
Yeah, me too.

01:05:30.160 --> 01:05:30.980
We use it for PyPI.

01:05:30.980 --> 01:05:32.180
We use it for a bunch of other stuff.

01:05:32.180 --> 01:05:32.460
Yeah.

01:05:32.460 --> 01:05:32.780
Indeed.

01:05:32.780 --> 01:05:33.100
All right.

01:05:33.100 --> 01:05:33.800
Final call to action.

01:05:33.800 --> 01:05:37.500
People want to get more involved, maybe make their things more secure.

01:05:37.500 --> 01:05:42.420
I'd say call to action is go to pipi.org slash security dash key dash giveaway.

01:05:42.420 --> 01:05:45.760
See if you're eligible as a critical maintainer for security keys.

01:05:45.760 --> 01:05:48.600
If not, please just turn on 2FA anyway while you're there.

01:05:48.600 --> 01:05:50.460
As a maintainer, I'd really appreciate that.

01:05:50.460 --> 01:05:52.200
Keep your eye on the security space.

01:05:52.200 --> 01:05:56.160
I think like there's a lot of interesting stuff happening, a lot of focus, a lot of resources

01:05:56.160 --> 01:05:57.180
going into it right now.

01:05:57.180 --> 01:06:01.500
It's a good time to like try and adopt some additional security, both to protect yourself,

01:06:01.500 --> 01:06:02.320
your users, everything.

01:06:02.320 --> 01:06:03.020
So yeah.

01:06:03.020 --> 01:06:04.280
Yeah, I definitely second that.

01:06:04.280 --> 01:06:07.040
Dustin, thanks so much for coming and sharing all this.

01:06:07.160 --> 01:06:11.380
This has been great to talk about, get an insight and some of the ideas and thoughts

01:06:11.380 --> 01:06:12.340
behind all these decisions.

01:06:12.340 --> 01:06:13.040
It's great.

01:06:13.040 --> 01:06:13.420
For sure.

01:06:13.420 --> 01:06:15.700
Michael, it's always really great to talk to you.

01:06:15.700 --> 01:06:16.500
So glad to be here.

01:06:16.500 --> 01:06:16.780
You too.

01:06:16.780 --> 01:06:17.300
Thanks again.

01:06:17.300 --> 01:06:17.760
See ya.

01:06:17.760 --> 01:06:18.020
See ya.

01:06:19.420 --> 01:06:21.980
This has been another episode of Talk Python To Me.

01:06:22.640 --> 01:06:23.640
Thank you to our sponsors.

01:06:23.640 --> 01:06:23.820
Thank you to our sponsors.

01:06:23.820 --> 01:06:25.420
Be sure to check out what they're offering.

01:06:25.420 --> 01:06:26.820
It really helps support the show.

01:06:26.860 --> 01:06:31.440
Listen to an episode of Compiler, an original podcast from Red Hat.

01:06:31.440 --> 01:06:36.200
Compiler unravels industry topics, trends, and things you've always wanted to know about

01:06:36.200 --> 01:06:38.880
tech through interviews with the people who know it best.

01:06:38.880 --> 01:06:43.000
Subscribe today by following talkpython.fm/compiler.

01:06:43.240 --> 01:06:46.860
You care about the ideas behind technology, not just the tech itself.

01:06:46.860 --> 01:06:50.000
And you know that tech has an enormous influence on society.

01:06:50.000 --> 01:06:52.000
So check out the IRL podcast.

01:06:52.000 --> 01:06:53.720
It's hosted by Bridget Todd.

01:06:53.720 --> 01:06:56.540
And this season of IRL looks at AI in real life.

01:06:56.540 --> 01:07:00.360
Listen to an episode at talkpython.fm/IRL.

01:07:00.840 --> 01:07:02.040
Want to level up your Python?

01:07:02.040 --> 01:07:06.100
We have one of the largest catalogs of Python video courses over at Talk Python.

01:07:06.100 --> 01:07:11.280
Our content ranges from true beginners to deeply advanced topics like memory and async.

01:07:11.280 --> 01:07:13.940
And best of all, there's not a subscription in sight.

01:07:13.940 --> 01:07:16.860
Check it out for yourself at training.talkpython.fm.

01:07:16.860 --> 01:07:21.540
Be sure to subscribe to the show, open your favorite podcast app, and search for Python.

01:07:21.540 --> 01:07:22.840
We should be right at the top.

01:07:22.840 --> 01:07:28.000
You can also find the iTunes feed at /itunes, the Google Play feed at /play,

01:07:28.200 --> 01:07:32.200
and the direct RSS feed at /rss on talkpython.fm.

01:07:32.200 --> 01:07:35.640
We're live streaming most of our recordings these days.

01:07:35.640 --> 01:07:39.040
If you want to be part of the show and have your comments featured on the air,

01:07:39.040 --> 01:07:43.420
be sure to subscribe to our YouTube channel at talkpython.fm/youtube.

01:07:43.420 --> 01:07:45.320
This is your host, Michael Kennedy.

01:07:45.320 --> 01:07:46.600
Thanks so much for listening.

01:07:46.600 --> 01:07:47.780
I really appreciate it.

01:07:47.780 --> 01:07:49.660
Now get out there and write some Python code.

01:07:49.660 --> 01:08:10.440
I'll see you next time.