Michael Kennedy: Security has always been the vegetables of software.
Michael Kennedy: Everyone agrees it matters, and somehow it never quite makes it onto the plate.
Michael Kennedy: At PyCon US this year, that changed.
Michael Kennedy: For the first time ever, security got its own dedicated day-long track,
Michael Kennedy: one of just two at the whole conference sitting right next to AI.
Michael Kennedy: And the room was packed to the back wall.
Michael Kennedy: On this episode, I'm joined by three people at the center of it.
Michael Kennedy: Seth Larson, security developer in residence at the Python Software Foundation,
Michael Kennedy: and very recently a CPython core developer,
Michael Kennedy: Juanita Gomez, a PhD researcher at UC Santa Cruz
Michael Kennedy: in open source security who co-chaired the track
Michael Kennedy: and Mike Fiedler, PyPI's safety and security engineer,
Michael Kennedy: one of the very few people paid full-time
Michael Kennedy: to keep the packages you install safe.
Michael Kennedy: We use the arc of the track's talks
Michael Kennedy: to take the temperature of Python security right now.
Michael Kennedy: Supply chain attacks, dependency cool-down,
Michael Kennedy: zero trust, S-bombs,
Michael Kennedy: and the push to bring Rust into CPython.
Michael Kennedy: and why not one of us think security is anywhere close to solved.
Michael Kennedy: And it turns out that's why the room was full.
Michael Kennedy: This is Talk Python To Me, episode 556, recorded July 6th, 2026.
Michael Kennedy: Talk Python To Me, yeah, we ready to roll.
Michael Kennedy: Upgrading the code, no fear of getting old.
Michael Kennedy: Async in the air, new frameworks in sight.
Michael Kennedy: Geeky rap on deck.
Michael Kennedy: Quarth crew, it's time to unite.
Michael Kennedy: We started in Pyramid, cruising old school lanes.
Michael Kennedy: Had that stable base, yes sir.
Michael Kennedy: Welcome to Talk Python To Me,
Michael Kennedy: the number one Python podcast for developers and data scientists.
Michael Kennedy: This is your host, Michael Kennedy.
Michael Kennedy: I'm a PSF fellow who's been coding for over 25 years.
Michael Kennedy: Let's connect on social media.
Michael Kennedy: You'll find me and Talk Python on Mastodon, Bluesky, and X.
Michael Kennedy: The social links are all in your show notes.
Michael Kennedy: You can find over 10 years of past episodes at talkpython.fm.
Michael Kennedy: And if you want to be part of the show, you can join our recording live streams.
Michael Kennedy: That's right.
Michael Kennedy: We live stream the raw uncut version of each episode on YouTube.
Michael Kennedy: Just visit talkpython.fm/youtube to see the schedule of upcoming events.
Michael Kennedy: Be sure to subscribe there and press the bell so you'll get notified anytime we're recording.
Michael Kennedy: This episode is brought to you by Sentry.
Michael Kennedy: Don't let those errors go unnoticed.
Michael Kennedy: Use Sentry like we do here at Talk Python.
Michael Kennedy: Sign up at talkpython.fm/sentry.
Michael Kennedy: Talk Python and Python Bytes both now have MCP servers.
Michael Kennedy: Point your AI at 10 plus years of Python episodes, transcripts, and show notes.
Michael Kennedy: Free.
Michael Kennedy: Click MCP in the nav at talkpython.fm and at pythonbytes.fm.
Michael Kennedy: Juanita, Mike, Seth, welcome all of you to Talk Python.
Seth Larson: Hi.
Seth Larson: Hello.
Michael Kennedy: Awesome to have some of you back, some of you here for the first time.
Michael Kennedy: Really, really great.
Michael Kennedy: So we're going to talk about Python security.
Michael Kennedy: And I believe, Seth, I saw that you wrote up a post article just thinking back on this kind of unique Python security track and this extra focus at PyCon US this year.
Michael Kennedy: I thought, hey, that'd be great to just get everyone involved together, at least everyone involved in the track, not all eight, seven or eight speakers, whatever that number adds up to be many, many speakers.
Michael Kennedy: And just kind of talk broadly about Python security and use this arc of the topics and talks covered at PyCon and the security track as maybe a backdrop.
Michael Kennedy: So welcome, welcome.
Michael Kennedy: Now, before we get into that exactly, let's just do quick introductions about who you are and how you're involved in stuff.
Michael Kennedy: Juanita, you want to kick us off?
Juanita Gomez: Sure.
Juanita Gomez: My name is Juanita.
Juanita Gomez: I am a PhD researcher right now at UC Santa Cruz.
Juanita Gomez: I work with the hospital at my university doing research in open source security and sustainability.
Juanita Gomez: And I've been involved in like the Python and the Sign2Pay Python community for a while.
Juanita Gomez: So, yeah, I got invited to coaching on the track and that was very exciting.
Michael Kennedy: Very cool. Where are you in your PhD? Are you past qualifying exams?
Juanita Gomez: I'm graduating in August.
Michael Kennedy: What? Oh, my God. That's awesome. Congratulations. You're far past it.
Michael Kennedy: So what's the plans after?
Juanita Gomez: I just got a job. I can't talk about it yet, but I just got a job. But I'm going to industry. I'm going to keep working on security and sustainability in open source mostly. But yeah, an industry.
Michael Kennedy: Cool. Congratulations on both finishing the PhD and getting a job straight away. That's super cool.
Michael Kennedy: Thank you.
Michael Kennedy: Yeah. Mike, welcome back.
Mike Fiedler: Hey, glad to be here. Mike Fiedler, PyPI Safety and Security Engineer, working for the Python
Mike Fiedler: Software Foundation based out of New York City. I'm coming up on my third year here at the PSF,
Michael Kennedy: which is a full milestone that I will be sharing with my compatriot here, Seth.
Michael Kennedy: Very cool. And I know thanks for all the work. It's been, I think, pretty impactful in three
Mike Fiedler: years already. Yeah, it's been kind of a wild ride of kind of incorporating this because I was a
Mike Fiedler: volunteer PyPI admin and contributor long before this job opened up. And having the ability to sit
Mike Fiedler: here and think about and work on security features and stuff for PyPI full-time has been a big eye
Mike Fiedler: opener as to kind of where the gaps were and kind of where they remain, but also kind of what are
Mike Fiedler: the very important things that we can do in the interim immediately while building up the kind of
Mike Fiedler: the piece by piece bits that we need to get to these longer term ideas, but being able to kind
Mike Fiedler: of still make progress while we're getting there. Again, it's that notion of like rebuilding the
Mike Fiedler: plane while you're flying it. Like, okay, well, we had the plane, but now we're like adding better
Mike Fiedler: sensors better kind of steering better throttles better better toggles while we're still flying
Michael Kennedy: and that flight is accelerating yeah 100 and seat belts i think seat belts belong in the security
Mike Fiedler: airplane analogy uh probably but again it's arguable sometimes seat belts are a problem
Mike Fiedler: so like yeah it becomes a very good discussion of trade-offs and balances and kind of what what
Mike Fiedler: actually will move a needle versus this is a nice to have uh you know ashtray yeah ashtrays oh my
Michael Kennedy: gosh i remember i remember when there are ashtrays on airplanes yeah that was crazy i'm like why is
Michael Kennedy: there still an ashtray on this airplane like i know they're not gonna let anyone use it fun fact
Mike Fiedler: they still have them in airplane bathrooms because while it is against the law they still want you to
Mike Fiedler: have a safe place to throw an ash your cigarette but if you're gonna violate the law so that way
Mike Fiedler: you don't throw it into the trash can and set something on fire.
Michael Kennedy: Seems reasonable. Seems reasonable. So as a way of a segue, Mike, I'll have you take this first,
Michael Kennedy: and then Seth, you can pick this up. You pointed out being full-time on this project versus a bunch
Michael Kennedy: of kind of contributors who get excited about part of it, but then there's this whole herding
Michael Kennedy: cats across thousands, hundreds of thousands of projects and users and contributors. And
Michael Kennedy: role to this versus just like a core dev type of situation, you know? Yeah, I think there's kind of
Mike Fiedler: two facets here, right? And there is very much a trade-off, right? When it's full-time volunteers,
Mike Fiedler: you get what you pay for, which is nothing, right? You haven't paid for anything. Volunteers are
Mike Fiedler: going to do what volunteers want to do, and they're going to do their best effort. When you hire
Mike Fiedler: somebody, then you set expectations and you try to make sure that you meet them, right? The double-edged
Mike Fiedler: sword there is that by hiring full-time employees, volunteers often will take a step off because it's
Mike Fiedler: like, okay, I don't need to volunteer as much as I used to because now someone's getting paid for
Mike Fiedler: to do it. So it's both a good thing because it highlights and focuses effort on areas that need
Mike Fiedler: kind of sanitization and focused effort to make progress, but it also gives volunteers a much
Mike Fiedler: needed break, which then in turn puts more pressure on the full-time employees to execute,
Mike Fiedler: which again, not enough capacity, not enough time. So we're doing our best here, but it's shown that
Mike Fiedler: if you do pay for a focused security effort, you do get good security outcome. And I think that's
Michael Kennedy: the takeaway for me and for anybody is if you pay for it, they will cut. Fair. Seth, welcome back.
Michael Kennedy: Hey, Michael.
Michael Kennedy: So quick intro for people who haven't listened to your episodes and then maybe your thoughts on this full-time thing.
Seth Larson: Yeah, so I, everyone, Seth Larson, a security developer in residence at the Python Software Foundation.
Seth Larson: I work primarily on kind of like everything that is not IPI backend.
Seth Larson: So that's the Python language, the runtime, the security response team for Python and pip,
Seth Larson: packaging tools and standards, and then kind of like how Curie World interacts with Python and vice versa.
Seth Larson: Right. So like CBE and all that.
Seth Larson: And recently a core developer, very recently.
Seth Larson: So to answer the how does this role and all that interacting with volunteers,
Seth Larson: My the way that I've tried to like because my space is like so large, my area of where I'm trying to focus.
Seth Larson: What I end up doing is I try to find projects that are obviously like high impact, all of that.
Seth Larson: Right. Like you don't want to be doing things. You don't want to be nibbling or what is the there's a snacking.
Seth Larson: That's the phrase for it. You don't want to be snacking, but you also don't want to be doing things that volunteers could technically do.
Seth Larson: Right. So if it's a job that is or a task or a project that a volunteer could do, maybe I will like pause and wait on that because I'll leave space for someone to potentially contribute there.
Seth Larson: And then the things that are less likely to get volunteer contributions, things like spinning up a CBE numbering authority for the Python Software Foundation or being an admin on the PSRT where or the Python security response team where you're doing things like, OK, we're going to like be monitoring for spam being sent to our email thread.
Seth Larson: Like volunteers maybe don't want to do that, but maybe they would want to be on the security response team contributing to coordinating vulnerability reports and then getting like acknowledged for their work.
Seth Larson: Right. Like in the in the NCD record and advisory.
Seth Larson: So that's kind of like how I think about it, because the area is so wide.
Seth Larson: I try to make sure that I leave space for other people to contribute just in case that people do get around to it before I end up getting to it.
Seth Larson: So we want to build the community volunteer muscle also, not just have it be on work.
Michael Kennedy: Yeah, that's really cool. How much do you focus on popular third party projects like Django? I mean, Django already has full time people above Flask or something like that versus just pure Python?
Seth Larson: Yeah, so I would say my time doesn't necessarily always like land directly on the project, but I do spend time responding to maintainers if they have questions.
Seth Larson: So like, for example, David from Flask, him and I have talked many times about how to do like a security response program or a threat model or, you know, hey, I'm seeing this.
Seth Larson: Like, can you help? Like, what what should I do in this situation?
Seth Larson: Like, we've talked about that and I've talked about it with many other maintainers, not just Flask, just to try to give them like advice.
Seth Larson: But then also whenever, like one of the nice things about this role being at like a nonprofit is that you can be super, super open.
Seth Larson: You don't have to like keep the secrets to yourself about like what you've done and like how you've seen success.
Seth Larson: So like we publish everything that we do and like where we see success and people in theory can just like take that copy paste it into their projects or their processes and have a better time than they were before.
Seth Larson: So people can learn from each other.
Seth Larson: I really do appreciate that part of the work.
Michael Kennedy: That is a bit of a unique aspect of doing an open source security thing as opposed to,
Michael Kennedy: I can't talk about it.
Michael Kennedy: This is, no, we don't talk about anything.
Michael Kennedy: So very interesting.
Michael Kennedy: Let's talk about this track.
Michael Kennedy: So there's this trailblazing Python security that talks about, for the first time ever,
Michael Kennedy: a day-long track devoted to the latest in Python security.
Michael Kennedy: And if you pull up the schedule of talks, I believe it was Saturday was the track.
Michael Kennedy: And that was, there was just a whole room.
Michael Kennedy: And what I also liked, it was a room, wasn't it?
Michael Kennedy: That was like a fixed place, so you didn't have to keep moving.
Michael Kennedy: Yeah.
Michael Kennedy: So Juanita, you want to kick us off and talking about why this, why do you guys focus on this
Michael Kennedy: this year?
Michael Kennedy: And, you know, why did you decide it's going to take a specific slot in?
Michael Kennedy: There's only two dedicated slots, security and AI.
Michael Kennedy: And I think that's very fitting.
Michael Kennedy: But tell us about this track.
Juanita Gomez: Okay.
Juanita Gomez: Well, I will let Seth talk about how did the idea come to place because I was not part
Juanita Gomez: of that.
Juanita Gomez: but I'm going to talk about why I think it's important
Juanita Gomez: and why it was important in the conference.
Juanita Gomez: I think security is something that's parallel to everything in Python.
Juanita Gomez: I think that if you're a maintainer for a package,
Juanita Gomez: I think having security somewhere on your mind should be a thing.
Juanita Gomez: You should be thinking about security when you're developing a thing.
Juanita Gomez: And I think there's a big gap in maybe the time that maintainers spend
Juanita Gomez: thinking about security or even the expertise.
Juanita Gomez: And so I think this track was like a big opportunity for people that work in PyVon as maintainers or as users to just learn about some important security things when thinking about like their development processes.
Juanita Gomez: I thought it was great that it was like a dedicated track because I think it highlights the importance of security.
Juanita Gomez: Like it's not a bunch of talks that are kind of like in the schedule that people just kind of like figure out they want to go to.
Juanita Gomez: But it's like we like made it a thing by making it like a full track.
Juanita Gomez: And I think that elevates the importance of security within Python, which I think it's not necessarily always the case.
Juanita Gomez: I think every time I talk to Mike, I'm like, why are you the only getting paid for doing Python security?
Juanita Gomez: I think increasing the awareness of the importance of security is very important.
Juanita Gomez: And I think that the track, I think, was meant to do that.
Michael Kennedy: How much do you think what were in those seven talks is representative of the broader security
Michael Kennedy: space, you know, getting your PhD, studying the stuff like you get maybe a little more
Michael Kennedy: than just Python view?
Michael Kennedy: Like, what did it capture?
Michael Kennedy: And what did it miss?
Michael Kennedy: I guess.
Juanita Gomez: I think, honestly, we made it a track that was diverse enough that we could capture like
Juanita Gomez: very different things within the security brother, like image.
Juanita Gomez: And I also think that a lot of the things that we talked about are not even necessarily Python specific.
Juanita Gomez: Like you should care about credentials.
Juanita Gomez: You should care about phishing.
Juanita Gomez: Like all of these things are things that you should care about, whether you do Python or C or any other like language development.
Juanita Gomez: I think that something that maybe I was missing a little bit after I like saw maybe after the conference,
Juanita Gomez: I was thinking about, I would have liked maybe like an intro version of security, something.
Juanita Gomez: I think something that I work with in my research is actually like security practices,
Juanita Gomez: mostly for like starting projects.
Juanita Gomez: I work with like literally university projects.
Juanita Gomez: And like the goal of my OSP is to improve the practices of the university projects.
Juanita Gomez: And some of them are maybe not as advanced to take on some of the,
Juanita Gomez: like I would say more advanced security practices.
Juanita Gomez: And so maybe something like that
Juanita Gomez: is something that maybe we did not capture as much,
Juanita Gomez: like basic things,
Juanita Gomez: like literally just going to GitHub
Juanita Gomez: and like switching some switches
Juanita Gomez: on the configurations of like your repository.
Juanita Gomez: Some of those things maybe were missing.
Juanita Gomez: And I think there's value on that for people
Juanita Gomez: that are like literally nowhere in the security like path.
Juanita Gomez: But apart from that,
Juanita Gomez: I think the diversity was really good on the track.
Michael Kennedy: I love it.
Michael Kennedy: I think that'd be a great talk to kick off the track and like security 101.
Michael Kennedy: Let's not get hacked the first week or something like that, you know?
Michael Kennedy: Maybe I'll do it next year.
Michael Kennedy: Yes, let's do it.
Michael Kennedy: I would love to see that.
Michael Kennedy: Like, for example, maybe just add a little bit of rate limiting and some logging.
Michael Kennedy: So if something does happen, you could actually tell.
Michael Kennedy: And, you know, these kinds of things, right?
Michael Kennedy: It would be, you know, there's a whole list of that stuff, right?
Seth Larson: Another idea that like came up in when we were thinking about how the track went and how the themes.
Seth Larson: this is this is funny to his idea so i you're gonna have to correct me but uh the idea of like
Seth Larson: having a little bit more interactivity uh to kind of shake things up as opposed to it being because
Seth Larson: i mean with security with these talks maybe you're expecting it to i mean it's obviously like more
Seth Larson: experts based but that ends up if you're in a talk format it's mostly talking to people but having
Seth Larson: something like a panel where you start off with like a few questions to kind of set the theme
Seth Larson: and then you kind of open it up to kind of get some vibes from the audience uh about like what
Seth Larson: questions they're having, make it more interactive in that way. That seemed really interesting to me
Seth Larson: as well. So we're going to see if we can experiment with that next year.
Michael Kennedy: So am I hearing you say there's going to be a one next year?
Juanita Gomez: We're hoping for it, for sure.
Seth Larson: We think that we should have another one next year. I mean,
Seth Larson: there was no indication that we shouldn't do this again, is what we saw.
Michael Kennedy: Honestly, I mean, security might be solved next year.
Michael Kennedy: Oh, yeah.
Michael Kennedy: That'd be great, wouldn't it?
Michael Kennedy: For sure.
Michael Kennedy: AI will solve all of our security problems.
Michael Kennedy: It's what I've been told, what I've read.
Michael Kennedy: Mike, you're trying to take us down a path that is a very different path.
Mike Fiedler: Well, I mean, it's been a few minutes and we haven't said AI yet.
Mike Fiedler: Well, no, you actually did because you introduced the other tracks.
Michael Kennedy: Yeah.
Michael Kennedy: Well, we only mentioned it in passing, but not with regard to this topic.
Michael Kennedy: I do think actually maybe if we get some time, I do want to talk to you about it.
Michael Kennedy: There's a lot to cover.
Michael Kennedy: There is so much.
Michael Kennedy: Maybe this could just be its own show that we could go into because there is a lot to cover.
Michael Kennedy: I think there's like positives and negatives,
Michael Kennedy: and that kind of sums up AI right there, doesn't it?
Michael Kennedy: It sums up everything, honestly.
Michael Kennedy: It sums up a lot of life, yes.
Michael Kennedy: Hey, one second.
Michael Kennedy: Normally, this would be an ad break from Sentry,
Michael Kennedy: but not this time.
Michael Kennedy: Let's just thank them for supporting the show
Michael Kennedy: and get right back to the conversation.
Michael Kennedy: Also, visit talkpython.fm/sentry after the show.
Michael Kennedy: Thanks, Sentry.
Michael Kennedy: It's a lot of gray.
Michael Kennedy: Yeah, not so much black and white, a lot of gray.
Michael Kennedy: So it was like standing room only?
Michael Kennedy: I mean, that sounds like that bodes well for our next year.
Seth Larson: Yeah, next year, hopefully.
Seth Larson: I mean, it'd be really great to even have a bigger room.
Seth Larson: Or, yeah, we were quite happy with the turnout.
Seth Larson: We had a few talks that basically it was standing room
Seth Larson: and there was nowhere else to come into the room.
Seth Larson: So that's always great to see as an organizer for a first-time event
Seth Larson: is when the room is too small.
Seth Larson: It means you're doing something right.
Michael Kennedy: Exactly.
Michael Kennedy: You've overachieved compared to what you were trying to work with or whatever.
Michael Kennedy: Yeah, I guess just one in the test,
Michael Kennedy: what was your experience just co-sharing a track and so on?
Michael Kennedy: What was it like being on that side of the conference?
Juanita Gomez: I think it was, I thought it was great because I think,
Juanita Gomez: so I've actually been organizing SyPyre also for a couple of years.
Juanita Gomez: And for the longest time,
Juanita Gomez: I've actually tried to bring security as a conversation at SyPyre.
Juanita Gomez: So I've learned a couple of like reds of a feather session,
Juanita Gomez: but having a security track seems a little impossible.
Juanita Gomez: I don't know, like it's so I don't know, like making making this happen at PyCon was it made me really, really happy because it gets you the opportunity to bring people to talk about the things that you really care about.
Juanita Gomez: Like security in Python is really like personal to me.
Juanita Gomez: I care a lot about it.
Juanita Gomez: And I think part of the problem is the lack of awareness of some of the things that we discussed at the shark.
Juanita Gomez: So I thought it was a great opportunity to just kind of bring that awareness into the table and making it happen.
Juanita Gomez: Just like being part of that conversation.
Juanita Gomez: I felt really proud of doing that.
Juanita Gomez: And I think also, I think like Seth and I and other couple of people we had, I mean, I had a lot of fun reviewing some of the talks that people submitted.
Juanita Gomez: We were unfortunately like limited to like eight slots.
Juanita Gomez: But there was a lot of things that I learned by just reading some of the talk proposals.
Juanita Gomez: So it was like, I think it was also like a learning experience for me to be part of the truck.
Seth Larson: Yeah, I was going to say, compared to Juanita, my experience with organizing sessions and talks and conferences, this is like the very first time that I've like really dipped my toe into organizing at this level for a conference.
Seth Larson: So it was a lot of learning for me.
Seth Larson: And I thought it was a great experience.
Seth Larson: I think other people could definitely like would enjoy doing this too.
Seth Larson: So now I'm like just understanding and learning all of these things that are about how a conference track or a talk schedule is put together.
Seth Larson: And for me, like the day of stuff was like so, so much.
Seth Larson: It's you're running around and it was like very on the fly.
Seth Larson: And, you know, they have all the recommendations that once the ball is rolling, it's like it's rolling.
Seth Larson: And it was really, really fun.
Seth Larson: I don't know. We had a lot of fun up there.
Seth Larson: I think that it was a great session.
Seth Larson: The energy was so positive.
Seth Larson: People are always coming up to you afterwards and saying, oh, that was amazing.
Seth Larson: Thank you for doing this, all of that.
Seth Larson: And it was a really great experience.
Michael Kennedy: Cool.
Michael Kennedy: Yeah, PyCon was really fun this year, too.
Michael Kennedy: I thought Long Beach was a pretty nice venue.
Michael Kennedy: There's some nice food carts and other things to do there.
Michael Kennedy: So that was really looking forward to back then.
Michael Kennedy: Yeah, now that we know the cool little bar open, you know, rooftop bars or outside bars,
Michael Kennedy: we can just go meet up and hang out with friends or the cool restaurant you want to go back to or whatever.
Michael Kennedy: Yeah.
Michael Kennedy: My first event there was to rent one of those like Lime or Bird scooters and just spend an hour riding around the town and the beach just to like get used to it and then going to check in.
Michael Kennedy: So very fun.
Michael Kennedy: Let's bring it over to the conference with the first of seven, I think we have seven, six or seven tracks that I put up for us to kind of riff on here.
Michael Kennedy: And that would be by none other than Mike Fiedler, an anatomy of a fishing campaign.
Michael Kennedy: What is this?
Mike Fiedler: Oh, boy.
Mike Fiedler: What is this?
Mike Fiedler: So this was the first track of the day, wasn't it?
Mike Fiedler: Yeah.
Michael Kennedy: I'll tell you what this is.
Michael Kennedy: This is why we can't have nice things, Mike.
Michael Kennedy: This is why we can't have nice things.
Mike Fiedler: It really is true, right?
Mike Fiedler: Like there's, for me, it was a good opportunity to kind of do a little storytelling education
Mike Fiedler: because these are cases that happen, right?
Mike Fiedler: And this was a novel enough one that I was like, this deserves some disambiguation because
Mike Fiedler: people hear phishing attack is like, oh yeah, yeah, yeah, yeah. This was different, right?
Mike Fiedler: And I was very kind of excited to speak and be selected for the spot. And you mentioned standing
Mike Fiedler: room only. Standing up there on the stage and looking out into a large room and seeing people
Mike Fiedler: standing at the back wall and not seeing little dotted seats available throughout the room was a
Mike Fiedler: little intimidating because it's like, you know, sometimes people don't sit in all the seats and
Mike Fiedler: That's fine. Or like the first two rows are empty. But no, Juanita and Seth had basically packed this room with people, not intentionally, but purely purely by curating the timeline and like setting the tone of like, yeah, this is this is going to be a story.
Mike Fiedler: And I was able to relate my personal firsthand experience of dealing with a phishing campaign that some of our users were tricked into visiting a website that was basically a proxy for PyPI.
Mike Fiedler: In previous phishing campaign attempts, a lot of folks talk about building a byte for byte or pixel perfect version of a different website and steering traffic to that.
Mike Fiedler: Here, they didn't even need to do that. They just threw up another proxy in front of it. And it was, you know, using email addresses that are like publicly available that maintainers put on their GitHub profiles and put in their, you know, Python package metadata. And they farmed out and emailed some unknowable number of maintainers.
Michael Kennedy: What an interesting security issue, right?
Michael Kennedy: Like you've got to be pretty sure that the login people use for their GitHub is probably the same email address that they've registered at PyPI.
Michael Kennedy: And the packages that you want to take over from PyPI linked to their GitHub repo, right?
Michael Kennedy: It's a really, it really does condense what you've got to do.
Mike Fiedler: A little bit, right?
Mike Fiedler: Like so in a lot of phishing campaigns, it's kind of a spray and pray, right?
Mike Fiedler: They're casting a very wide net and they're going to email 2 million people.
Mike Fiedler: And if they get two, that's a success for them, right?
Mike Fiedler: Because we have to defend 100% of the time.
Mike Fiedler: They only have to be right once or twice, right?
Mike Fiedler: And they were in this case.
Mike Fiedler: Again, PyPI is open source, right?
Mike Fiedler: You can read on the entire code base.
Mike Fiedler: You can see what our email templates look like.
Mike Fiedler: And you can see that, yeah, we do ask maintainers now and then to do an action.
Mike Fiedler: And they emailed maintainers and asked them to do an action.
Mike Fiedler: And these maintainers did an action.
Mike Fiedler: They did what they were asked to do.
Michael Kennedy: For example, you emailed people a couple of years ago and said, hey, you need to turn on 2FA.
Michael Kennedy: Yeah.
Mike Fiedler: And that's totally legitimate.
Mike Fiedler: And sometimes we will email people and say, hey, there's a problem with your metadata.
Mike Fiedler: Go in and fix it.
Mike Fiedler: We will occasionally tell people there's a problem.
Mike Fiedler: This one in particular would tell them it wasn't using our email infrastructure.
Mike Fiedler: You know, if you had anti-spam things, then like maybe it would have caught it.
Mike Fiedler: But it didn't because the mail was coming from the domain advertised for this hack.
Mike Fiedler: And it was basically showing instead of PyPI.org, PyPJ.org.
Mike Fiedler: And I would challenge you at, you know, looking at an email on your phone to hover on a link and see that difference between an I and a J in the lowercase Arial font on your phone.
Mike Fiedler: Like, you're not going to notice it, right?
Mike Fiedler: Humans, I have no qualms for the humans that fell for this, right?
Mike Fiedler: Like, they did what they were supposed to do.
Mike Fiedler: What was annoying about this attack is we've been talking about 2FA for a very long time now,
Mike Fiedler: and this is still phishing.
Mike Fiedler: You know, you can still get phished with certain forms of two-factor authentication,
Mike Fiedler: notably the time-based one-time password that were TOTP or what folks might think of your, like, QR code.
Mike Fiedler: or Google Authenticator app,
Mike Fiedler: all of those things are the same thing.
Mike Fiedler: Whereas if you used WebAuthn
Mike Fiedler: or the underlying technology for pass keys
Mike Fiedler: or like a YubiKey,
Mike Fiedler: that's requiring a different sort of 2FA
Mike Fiedler: that is more phishing resistant.
Mike Fiedler: So anybody who had the phishing resistant part
Mike Fiedler: was not subject to this attack.
Mike Fiedler: And then
Michael Kennedy: I think it might be worth
Michael Kennedy: maybe just a quick conversation on that
Michael Kennedy: because I totally agree
Michael Kennedy: because you could just proxy, they send over the thing,
Michael Kennedy: and you've got 30, 15, whatever seconds to then, you know,
Michael Kennedy: curry those credentials over,
Michael Kennedy: and you probably are fine to hack through with a proxy, right?
Michael Kennedy: I mean, I think it's, I think the TOTP ones are more,
Michael Kennedy: like if somebody gets a hold of your credentials and they're leaked, right,
Michael Kennedy: that's not enough to get in anymore, right?
Michael Kennedy: Or if they're trying to brute force something,
Michael Kennedy: that can be a big challenge to limit that.
Michael Kennedy: But for phishing, it really doesn't add a lot of safety, does it?
Mike Fiedler: So it doesn't because the session token that you get once you have completed your username password, then you do your TOTP.
Mike Fiedler: Now you have a session cookie, right?
Mike Fiedler: This is what your browser uses instead of asking you for your password and to FA every single page view, right?
Mike Fiedler: You have a session cookie that is good for, I don't know, eight hours or something like that.
Mike Fiedler: So you don't have to do that all the time.
Mike Fiedler: And this session cookie is something that they can use to then, they don't even need the TOTP within those 30 seconds.
Mike Fiedler: They now have a session cookie and use that to log in as you.
Mike Fiedler: And then the next part that they did was create an API token, publish.
Mike Fiedler: API tokens today do not have any expiry.
Mike Fiedler: So they were like, all right, I'm going to create a new API token.
Mike Fiedler: And now I don't need your username, your password, your TOTP, the session token.
Mike Fiedler: I don't need any of that.
Mike Fiedler: I have an API token.
Mike Fiedler: I can go ahead and do the things that I want to do on this website with this token.
Mike Fiedler: So I think there's layers of security.
Mike Fiedler: Something we added in the aftermath of this was if you now are using a session token and it was obtained via TOTP and it's coming from a different IP address, we're not going to accept that.
Mike Fiedler: We're going to ask you to recertify that you are who you said you were because something is different.
Mike Fiedler: But if you use WebOFN, then you're fine.
Mike Fiedler: The people who will get impacted by this
Mike Fiedler: are folks who are like bouncing between VPNs
Mike Fiedler: or if you're riding a train
Mike Fiedler: and you keep changing IP addresses
Mike Fiedler: as you kind of go to different cell towers,
Mike Fiedler: which I don't think happens very often, but it can.
Mike Fiedler: Those are the people who will get caught in that net.
Mike Fiedler: But it's like, yeah, you know,
Mike Fiedler: we can't tell that you are who you said you were
Mike Fiedler: based on those credentials.
Mike Fiedler: Yeah.
Mike Fiedler: Juanita, Seth, thoughts?
Juanita Gomez: I want to say that this was one of my favorite talks of the track because it was very relatable.
Juanita Gomez: Like a lot of the people that are sitting down in the room are maintainers who published their packages in YPI.
Juanita Gomez: And just having Mike's perspective on like, oh, this can happen to you.
Juanita Gomez: Like, you need to be aware of this.
Juanita Gomez: I think it makes it much more impactful because it's like you are directly affected by some of these things.
Juanita Gomez: And I think also Mike had a really, he not only told us a story about how all of these bad things happened, but also at the end, a call for action.
Juanita Gomez: This is the things that you need to do in order to avoid these things from happening to you.
Juanita Gomez: And so I thought that was really impactful for the people that were sitting down in that room.
Michael Kennedy: Nice.
Michael Kennedy: So one of the things I think is, I guess, I'm not sure if you mentioned it, but the result was there were some packages that got malicious code in them and then published a PyPI, right?
Mike Fiedler: Just one that we know of, right?
Mike Fiedler: We audited kind of a lot of the attacker signatures and kind of their behaviors.
Mike Fiedler: We're only able to turn up one.
Mike Fiedler: So only four of the user accounts of the, again, we have over a million user accounts on PyPI right now, but only four people fell for this phishing and like click through and actually logged in.
Mike Fiedler: And only one package was kind of harvested and republished.
Mike Fiedler: And again, that doesn't mean that others weren't created, you know, as a backup, but they went after one that is a dependency of a dependency of a dependency, right? They're working their way through the supply chain. They didn't go after anything that, you know, these other four, sorry, the other three maintainers had.
Mike Fiedler: They went after the one maintainer who kind of fell for the fish.
Mike Fiedler: And theirs was a dependency of Hugging Faces, Transformers library.
Mike Fiedler: And if you're doing anything at all in AI land, which a lot of people are doing a lot in AI land, Transformers is kind of key.
Mike Fiedler: It gets, I don't know, 30, 40 million downloads a day.
Mike Fiedler: And Numb to Words is a sub-dependency of Transformers.
Mike Fiedler: So by publishing a new version of Numb to Words, any Transformers user is going to get it.
Michael Kennedy: Scary.
Michael Kennedy: And it's, yeah, you know, and you maybe think I'm going to take on this dependency, so you'll review it.
Michael Kennedy: But how far down the dependency chain do you follow, right?
Michael Kennedy: I mean, the answer is like, you know, what was that TV show?
Mike Fiedler: You are the weakest link.
Mike Fiedler: Goodbye, right?
Mike Fiedler: You're only as strong as the weakest link in your chain.
Mike Fiedler: And we all depend on something and somebody else and somewhere, right?
Mike Fiedler: None of us are writing 100% everything by ourselves based on like a C standards library.
Mike Fiedler: Like you didn't write your compiler on your own.
Mike Fiedler: You used somebody else's.
Mike Fiedler: You didn't write the Python language on your own.
Mike Fiedler: You used somebody else's.
Mike Fiedler: And even if you are authoring Python, you're using tools that somebody else wrote, right?
Mike Fiedler: You didn't write your editor on your own.
Mike Fiedler: You didn't write your operating system on your own.
Mike Fiedler: And again, if you are the one person who has done all of this, I commend you.
Mike Fiedler: I also think you have put in a lot of effort that is commendable, but is unnecessary in
Mike Fiedler: today's world.
Mike Fiedler: but you are likely more secure as a result.
Michael Kennedy: Yes.
Michael Kennedy: This portion of Talk Python, I mean,
Michael Kennedy: is brought to you by our AI tools.
Michael Kennedy: You know that thing where you ask an AI
Michael Kennedy: something about Python
Michael Kennedy: and it confidently tells you about the library version
Michael Kennedy: from 18 months ago?
Michael Kennedy: Well, we fixed that, at least for our shows.
Michael Kennedy: Talk Python and Python Bytes both have MCP servers now.
Michael Kennedy: Connect Talk Python and your AI can search
Michael Kennedy: over 550 episodes, full transcripts,
Michael Kennedy: every guest in the entire course catalog of Talk Python courses,
Michael Kennedy: ConnectPython Bytes,
Michael Kennedy: and he gets almost 500 episodes of Python news going back to 2016,
Michael Kennedy: including every link we've ever put in the show notes.
Michael Kennedy: This means you can say things like,
Michael Kennedy: ask Talk Python what astral joining OpenAI means for uv,
Michael Kennedy: or what has Python Bytes said about Locust,
Michael Kennedy: and get a real answer with real links, not a hallucination.
Michael Kennedy: Name one of our shows in your prompt,
Michael Kennedy: and your AI knows exactly where to look.
Michael Kennedy: And if you live in the terminal,
Michael Kennedy: Talk Python now has a CLI too.
Michael Kennedy: One line, uvtoolinstalltalk-python-cli.
Michael Kennedy: Then search the episodes, transcripts, guests, and courses
Michael Kennedy: without ever even opening a browser.
Michael Kennedy: It's open source and it outputs text, JSON, or Markdown,
Michael Kennedy: so it also feeds AI tools that don't speak MCP.
Michael Kennedy: And here's the real reason I built it.
Michael Kennedy: Both shows cover around 10 years of Python history.
Michael Kennedy: the people, the decisions, the packages that took over, and the ones that quietly didn't.
Michael Kennedy: This enhanced access to all of our information is free. No account, no API keys, nothing to buy.
Michael Kennedy: That history contained in these shows should be there for all of us. So visit talkpython.fm
Michael Kennedy: and pythonbytes.fm and click the MCP link in the nav bar. Connect them right now to your agents
Michael Kennedy: so that they will be accessible anytime they're needed in the future. Hope you all enjoy the access.
Michael Kennedy: One thing that came to mind when I was looking at your talk is this whole concept of dependency
Michael Kennedy: cooldowns, which is making some waves.
Michael Kennedy: I think I noticed it pretty early in, put this up for everyone, pretty early, I guess
Michael Kennedy: with uv, and I think pip is getting it as well, but also some of the other package managers.
Michael Kennedy: But I've seen it sort of picking up steam in other areas unrelated to Python, other package
Michael Kennedy: managers.
Michael Kennedy: So I really like this idea.
Michael Kennedy: I really like it.
Mike Fiedler: I think cooldowns are a good idea for your average consumer. So in a different attack that Seth and I talk about later, we talk about how the people without using version pinning or deciding I want to use version 1.0.0 only, if you use unbounded versions, you're going to get the latest one that the registry has to offer.
Mike Fiedler: And when attackers are looking to exploit you, that's kind of one vector they're looking to exploit to say these people are not pinning their versions.
Mike Fiedler: Dumb to words was not pinned in Transformers.
Mike Fiedler: So they're going to get whatever the latest one is.
Mike Fiedler: Now, sub-dependency pinning and version bounds is a whole other set of conversations.
Mike Fiedler: But as an end consumer, cooldowns is the easiest way for you to say, I don't want what the latest the registry has.
Mike Fiedler: I want to wait three days or seven days because in the scope of those three to seven days,
Mike Fiedler: it is very likely that some security scanners will have chewed through this and noticed
Mike Fiedler: if there's a problem and reported it.
Mike Fiedler: And if they have, there is a, again, a very high likelihood that that version is no longer
Mike Fiedler: available for download in that time.
Michael Kennedy: Yeah.
Michael Kennedy: So like you look at cool, the thing I'm linking to is cooldowns.dev.
Michael Kennedy: And in here, it pointed out, said, does it actually work?
Michael Kennedy: And they gave some stats, you know, they talk about LiteLLM, some of the other issues that have come along.
Michael Kennedy: They said, I think within a week, I think 90% of the vulnerable ones were taken down and most of them were taken down within a day or something like that, right?
Mike Fiedler: Yeah. I mean, a lot of that on PyPI is thanks to our group of volunteer reporters.
Mike Fiedler: And we don't have any kind of commercial relationship.
Mike Fiedler: There's just people out there in the community who like scanning stuff.
Mike Fiedler: There are businesses who do it, and some of them will report back to PyPI and say, hey, we found something fishy, something smelly.
Mike Fiedler: We recommend you take a look.
Mike Fiedler: Some of these folks have gotten such a very good or almost zero false positive rate that we kind of allow them on PyPI to put that into an automated quarantine status that requires a human like myself to review it and either clear it or tank it completely.
Mike Fiedler: But in that interim state, it's no longer downloadable.
Mike Fiedler: So those like our most notable one is Kamil Mankowski, who works out in Austria, and he
Mike Fiedler: has gotten it down to like his reports are often within, you know, 30 minutes of publish.
Mike Fiedler: His tooling has scanned it, analyzed it and reported it back with zero false positives.
Mike Fiedler: And, you know, that only comes with curation and time.
Mike Fiedler: And he's doing this as a volunteer effort.
Mike Fiedler: So kudos out to Kamil.
Mike Fiedler: It's a lot of data to be processing.
Seth Larson: Yeah.
Seth Larson: I wanted to comment on the cooldowns aspect too, like maybe about like the, who is it
Seth Larson: for?
Seth Larson: Cause I think this, this is always a challenge whenever we're building something at the scale
Seth Larson: of Python, right?
Seth Larson: Where it serves so many different communities and people and right.
Seth Larson: There's people that have, you're on a developer team in a really large organization and you
Seth Larson: have your own like security team who is deciding what your security policies are.
Seth Larson: They're running a mirror of PyPI, right?
Seth Larson: Like all this stuff is happening.
Seth Larson: But then Python also needs to serve like students or people who this is their first programming
Seth Larson: language.
Seth Larson: And my thinking is, is that this sort of config where you set it once and then you don't have
Seth Larson: to think about it and you benefit from this large apparatus that already exists out there
Seth Larson: in PyPI, people that are scanning, people that are manually reviewing things, taking
Seth Larson: them down, all of Mike's work, right?
Seth Larson: Like you get to benefit from all of that work without having to have this massive, you know,
Seth Larson: separate team or policy or having to necessarily think about like reviewing every single dependency
Seth Larson: right like this is something that I don't think that a lot of developers are doing especially not
Seth Larson: students right they want to run pip install with the thing that they want to install either for
Seth Larson: their course or because they're learning they're not necessarily thinking about oh this is
Seth Larson: potentially a way that malware is going to be installed on my system so set like a really
Seth Larson: aggressive like seven days or something like that on your cooldown one time and then you're pretty
Seth Larson: okay. Like, I think a lot of people should probably be using cooldowns, especially for like, developer
Seth Larson: machines or a personal machine that you're running Python on where you're not, you don't have this
Juanita Gomez: giant security apparatus working for you. Yeah, I want to second that and say that a lot of the
Juanita Gomez: people that use Python packages have no idea of what is PyPI, where they're coming from, what does
Juanita Gomez: pip install mean? And so having like, and like, like, seriously, but when I started doing all of
Juanita Gomez: they say, I had no idea.
Juanita Gomez: I was just running commands
Juanita Gomez: that I was reading somewhere in Reddit
Juanita Gomez: or somewhere else.
Juanita Gomez: And I think just having some default ways
Juanita Gomez: of keeping security
Juanita Gomez: in my own developer environment
Juanita Gomez: is, I think it's great.
Juanita Gomez: For the most unexperienced developers,
Juanita Gomez: this is definitely more usable.
Michael Kennedy: Yeah, I think it's a great idea.
Michael Kennedy: I really like the idea.
Michael Kennedy: So I adopted this to some degree
Michael Kennedy: and I thought it was amazing.
Michael Kennedy: And then I ran into some problems.
Michael Kennedy: And I'd love to hear your thoughts on this.
Michael Kennedy: So I don't have unpinned dependencies.
Michael Kennedy: But I use uvpipcompile, which will say, with an update,
Michael Kennedy: and say, OK, well, I've decided now I'm
Michael Kennedy: going to release a new version.
Michael Kennedy: Let's go ahead and do a refresh of the update of the dependencies
Michael Kennedy: and recompile.
Michael Kennedy: And it will repin them to the newest.
Michael Kennedy: And without this cooldown, it'll pin them to the very latest.
Michael Kennedy: And so if, for some reason, something
Michael Kennedy: were to git pushed into PyPI an hour ago, and I ran that command, well, I might as well have
Michael Kennedy: unpinned dependencies, right? So I'm going to put uv pip compile dash dash, like wait, whatever the
Michael Kennedy: command is, exclude newer one week. And I even put it into the Docker build for all of my server
Michael Kennedy: stuff so that the Docker build would fail if using a combination of pip audit and this. So if there's
Michael Kennedy: a CVE, it will fail because there's a problem with one of the packages. And then I would use this
Michael Kennedy: older ones but here's what i ran into is there would be a cve in something and then it would
Michael Kennedy: it would window into the seven days ago and the fix would be like one day later but it wouldn't
Michael Kennedy: allow it to install the one without the cve because that was excluding newer and so then i end up with
Michael Kennedy: all of these like x ignore this one because this one you know what i mean like it's these two things
Michael Kennedy: clashing it's kind of like one package says version has to be greater than two on a dependency
Michael Kennedy: and one says version has to be less than two you're like well now what do i do yeah michael you're
Seth Larson: You're touching on the exact reason why I would have a hard time,
Seth Larson: at least right this moment,
Seth Larson: pushing for dependency cooldowns to be a default.
Seth Larson: It confuses users,
Seth Larson: especially when there are other systems
Seth Larson: that are telling them that they need to resolve a problem.
Seth Larson: And when they run the commands necessary that they know work
Seth Larson: because they've used them before
Seth Larson: to upgrade a dependency to a specific version or whatever,
Seth Larson: they don't work.
Seth Larson: Or they'll say something like,
Seth Larson: oh, that version doesn't exist
Seth Larson: because pip is like literally dropping
Seth Larson: these versions from the candidate pool
Seth Larson: or something like this, right?
Seth Larson: It'll say that version doesn't exist
Seth Larson: and you get these really confusing messages.
Seth Larson: I mean, one day I do,
Seth Larson: I would really like to have something,
Seth Larson: it doesn't necessarily have to be cooldowns,
Seth Larson: but some sort of system that the default is,
Seth Larson: if you are not,
Seth Larson: the way I like to think about it is like,
Seth Larson: if you are not showing through your tool usage,
Seth Larson: whether or not you have like operational maturity
Seth Larson: is kind of how I think about it, right?
Seth Larson: Like you're not using a mirror,
Seth Larson: you're installing direct from latest.
Seth Larson: If you're not showing that sort of operational maturity,
Seth Larson: like asking for a lock or an exact version or whatever,
Seth Larson: then there is some sort of secure default.
Seth Larson: But that secure default policy
Seth Larson: needs to take into account a lot of things.
Seth Larson: And one of those things is
Seth Larson: whether a security vulnerability is being fixed
Seth Larson: by something that is within this cooldown window
Seth Larson: or reviewing window.
Seth Larson: And maybe this can be easier
Seth Larson: if there's more of this happening
Seth Larson: the index side. Like right now, IPI is mostly you upload it, it's immediately available, right? And
Seth Larson: the scan comes after the fact, there's no intermediate step where it's like being reviewed
Seth Larson: because we just can't, there's too much volume. And so like, if we get to a world where that is
Seth Larson: happening, maybe tools can implement this sort of policy a little bit more easily, because it'll be
Seth Larson: for the most part on the index side. But from a tool perspective, it's really difficult to get
Seth Larson: a secure default that takes and respects all of these use cases perfectly right as a default.
Seth Larson: It's very challenging.
Juanita Gomez: So what you're saying is we need to hire a lot more people to scan packages before they
Juanita Gomez: get published by BI.
Juanita Gomez: That's what you say, right?
Seth Larson: I'm open to emails.
Seth Larson: Email Seth at Python.org for any interested parties solving this issue.
Michael Kennedy: Indeed.
Michael Kennedy: So earlier, Mike, you talked about API keys and how they're kind of outside the bounds of two-factor and so on.
Michael Kennedy: So the next talk was Zero Trust in 200 Milliseconds, Identity Per Transaction with Tristan and McKinnon.
Michael Kennedy: And that comes from FediRAMP, which is a U.S. government thing, the Federal Risk and Authorization Management Program that provides approach to security and risk and so on.
Michael Kennedy: If I recall correctly, this is kind of a zero trust story where it's every single time you need to use an API key, you have to get a new version of an API key.
Michael Kennedy: Like, I want to talk to AWS.
Michael Kennedy: Great.
Michael Kennedy: Here's a key you can use for two seconds or something, right?
Michael Kennedy: Use it now.
Michael Kennedy: Yeah.
Michael Kennedy: What are your thoughts on this?
Michael Kennedy: What are the takeaways from this one?
Michael Kennedy: Anyone?
Michael Kennedy: How do you feel about it?
Seth Larson: So I was going to say, we're all pointing at each other.
Seth Larson: I really love this talk.
Seth Larson: because it talked about a side of Python and like development that a lot of people don't
Seth Larson: necessarily experience, which is where you work in an industry that's like regulated or needs to
Seth Larson: like implement FedRAMP, right? It is a tone of extra work to do this. And a lot of times it's
Seth Larson: not really talked about a lot online because like you said, that's not necessarily like open source
Seth Larson: communities that are doing this sort of work. It's mostly, you know, industry that is doing these
Seth Larson: things. And so it's like really, really great to see this sort of talk where you're talking about
Seth Larson: something like FedRAMP compliance and zero trust in Python with a lot of different like tools that
Seth Larson: people already use. So like using the AWS SDK and how to implement with this with AWS and then
Seth Larson: extrapolating that out to other different like backends. Right. So it's back end agnostic. I really
Seth Larson: enjoyed this talk because of those reasons. Like this is just stuff that you don't necessarily see
Seth Larson: at PyCon or at open source conferences,
Seth Larson: people talking about federal policy requirements
Seth Larson: and how to implement secure controls like this.
Seth Larson: So really, really interesting talk.
Michael Kennedy: Yeah, it is a bit of a look in an area
Michael Kennedy: that you don't normally see.
Michael Kennedy: And yeah, it's quite cool.
Michael Kennedy: This next one, the next one is Rust.
Michael Kennedy: And this by Emma Smith.
Michael Kennedy: And this one certainly, it's interesting
Michael Kennedy: because you can debate it technically,
Michael Kennedy: but then you also have to debate it
Michael Kennedy: from an organization perspective
Michael Kennedy: and so on. And the idea is, what if we could have more of CPython written in Rust, right?
Michael Kennedy: But a lot of the people who work on CPython, hey, guess what? What is the name there? CPython,
Michael Kennedy: as in the language, our C developers, and they don't necessarily want to rewrite stuff in Rust,
Michael Kennedy: and they don't necessarily have an expertise in it, and a whole bunch of things like that. So one,
Michael Kennedy: thoughts on the talk. Two, thoughts on Rust and Python.
Juanita Gomez: I want to give my thoughts on the talk. I think I love this talk for two reasons. The first one is it was a talk about Rust in a Python conference, which shows the intersection between these two communities. And I think, I mean, I think Emma did a great job of explaining like the technical implications of the migration and like of why we need to do this.
Juanita Gomez: But at the same time, I think she was very clear about what this means from like a community perspective. And I think part of the talk was actually talking about how can we actually make this happen? Not from a technical perspective, but like from organizing the people and like actually getting the people to like agree with doing this and like how do we make this happen?
Juanita Gomez: Which I think is very on point with open source and how open source works.
Juanita Gomez: You don't just make a decision and take a different path on something that you're doing.
Juanita Gomez: There's a whole lot of people that are involved in making decisions for a specific community.
Juanita Gomez: And I think Emma did a great job of illustrating how that specific aspect of open source works with this particular talk.
Juanita Gomez: So I loved it.
Juanita Gomez: And one of my highlights was that Wido was there and he ended up asking questions to Emma about this particular talk.
Juanita Gomez: that was like, isn't it cool that you have the creator of Python asking you questions about your
Seth Larson: talk? That was that was one of my highlights. Yeah, from like a security. So I am very
Seth Larson: intrigued about this idea. I, it makes sense. But it's also like you said, like in a social sense,
Seth Larson: extremely challenging. And also even like a technical sense, it's extremely challenging,
Seth Larson: right? This is this is not an easy project. I think that there's maybe when people think about
Seth Larson: rewriting and rust and all of this stuff, right? It's almost kind of like a joke, right? That you
Seth Larson: can all just rewrite it in rust and oh, rust Python already exists. Why? Why would this be hard? Right?
Seth Larson: Like this already exists out there. And it is extremely difficult. I am really excited to talk
Seth Larson: about this at the language summit at EuroPython is a talk there also. And I think that Emma and
Seth Larson: the whole team that's working on rust for Python have their work cut out for them. It's going to be
Seth Larson: long journey. Talking about like just the technical aspect of this, I think that they
Seth Larson: have made a really great choice in which library they're decided to potentially like attack first
Seth Larson: as like a first go, which is Zlib, which is a compression library that's pretty understood,
Seth Larson: processes a lot of like untrusted data, right? So there's the memory management, memory of security
Seth Larson: vulnerability aspect. Not necessarily that Zlib has had a ton of issues in that area,
Seth Larson: but it's a really good one to attack as a first module. Yeah, I'm going to be
Michael Kennedy: Give us an idea of exactly what they're proposing here. Is this like,
Michael Kennedy: there are portions of the standard library that right now are pure Python or C, and we propose
Seth Larson: rewriting some of them in Rust? So the proposal right now is to rewrite a single standard library
Seth Larson: module in Rust with the long term goal being maybe more and even potentially the like parts
Seth Larson: of the language being rewritten in Rust.
Seth Larson: So like the I believe it was like legit.
Seth Larson: The just in time compiler was another candidate for like a larger project or maybe like a here
Seth Larson: you can cut your teeth on this.
Seth Larson: We won't merge it, but we'll it'll it'll prove out whether or not the tooling that's around
Seth Larson: this like compiling stuff to Rust and having it actually work for Python would work just
Seth Larson: as an option.
Seth Larson: this got discussed a while back too.
Seth Larson: And so this has kind of been ongoing for quite a long time.
Seth Larson: And so I'm like really excited that this talk,
Seth Larson: this is kind of like the first time that Emma and the team have like poked
Seth Larson: their head out and be like, talk to the wider community about this is happening.
Seth Larson: We're discussing this and what it could mean.
Seth Larson: But yeah, it's going to be a long road ahead.
Seth Larson: I have a feeling.
Michael Kennedy: So yeah.
Michael Kennedy: All right.
Michael Kennedy: Next up, Asleep at the Wheel, S-Bombs for Python Builds.
Michael Kennedy: Sanchit Seye and Abhishek Reddy Piley.
Michael Kennedy: Sorry about that.
Michael Kennedy: Yeah, so SBOMs.
Michael Kennedy: What are the thoughts on SBOMs, folks?
Michael Kennedy: We need more of them.
Juanita Gomez: Yes, and better.
Juanita Gomez: They need to be accurate.
Juanita Gomez: Exactly.
Juanita Gomez: I think, honestly, I was just going to comment really quick that I was very proud of this talk because Abby and Sancheid are both students.
Juanita Gomez: I think Sancheid is a master's student.
Juanita Gomez: Abby is a PhD student.
Juanita Gomez: And as a PhD student, I love to see the research that people are doing in conferences are not academic conferences.
Juanita Gomez: And I think they have done a really good work on developing their research, but also interacting with the outside community to figure out how to make their tools available and usable.
Juanita Gomez: I love that they have made their project part of the OpenSSF, like, bigger ecosystem, because that kind of shows how they want to make sure that anything that they're doing is actually usable.
Juanita Gomez: And yeah, like it's not just a paper about something that's very abstract and not really relevant, but like it's actually something that the community can use and adopt.
Juanita Gomez: So I'm really proud of this talk.
Seth Larson: Yeah, I was going to add that this talk is the talk that showed me what SBOMIT actually was, which is really, really cool.
Seth Larson: I'd heard of it.
Seth Larson: I kind of knew what it meant, right?
Seth Larson: It's one of these things where there's just so many security tools and approaches and so much area to cover that you kind of like end up knowing about something.
Seth Larson: And, you know, like when you'd maybe reach over there to learn more, but you don't know like how deep the iceberg goes.
Seth Larson: right and how much work had to go into making this basically work like magic to record all of the
Seth Larson: stuff that happened during your build all the network calls all the command invocations and
Seth Larson: how it gets then exported into an s bomb but then also this verifiable record using i believe it's
Seth Larson: called in toto that's what the it is an s vomit um and like how it all works end to end and so it
Seth Larson: just gets my mind thinking about like if we were to release this like either like a build farm for
Seth Larson: PyPI if we ever to work on that project this is the sort of thing that we would like reach for and
Seth Larson: say like okay maybe we could integrate this technology into that or if we were going to
Seth Larson: create a like a recommended builder like a docker image that did building like this would potentially
Seth Larson: be a part of that story where you're recording network calls and then you can get this extremely
Seth Larson: accurate software bill of materials document at the end that shows how a package was built.
Seth Larson: And then you can have that be published with the package or you can use it as a user, however
Seth Larson: you want to use it.
Seth Larson: Like that sort of information is really important.
Seth Larson: And it's really difficult to get after the fact, as opposed to like when it's in flight
Seth Larson: and when the actual programs are doing the installations and the builds and all of that.
Seth Larson: And I think you mentioned...
Seth Larson: Oh, sorry, Godwin.
Juanita Gomez: I was just going to say that this talk actually did a very good job of making people aware
Juanita Gomez: that the current tools that generate S-BOMs are not as accurate.
Juanita Gomez: And there's still a lot of work needed to make S-BOMs actually usable
Juanita Gomez: within a security perspective.
Juanita Gomez: And they did a great job of just making that awareness.
Juanita Gomez: And they also were brave enough to do a demo,
Juanita Gomez: which is like people now don't do demos because they all break,
Juanita Gomez: but they did a demo and it worked.
Juanita Gomez: That was really cool.
Michael Kennedy: Very cool.
Michael Kennedy: Seth, you theorized there might be some magical process
Michael Kennedy: involving cooldowns and pin dependencies and review.
Michael Kennedy: And maybe SBOMs could be an input to that to say,
Michael Kennedy: well, this ain't got a problem,
Michael Kennedy: but it has some source down in the supply chain
Michael Kennedy: that actually we've discovered is problematic.
Michael Kennedy: So everything downstream from that,
Michael Kennedy: we need to check or flag or whatever.
Seth Larson: Yeah, SBOM, the way I think about SBOM
Seth Larson: is mostly as just this third party.
Seth Larson: It's not a part of an ecosystem,
Seth Larson: so it works really well for Python.
Seth Larson: because we have so many Python packages out there
Seth Larson: are not just Python.
Seth Larson: It's like C, Fortran, Rust, JavaScript,
Seth Larson: all of these different ecosystems, right?
Seth Larson: GPU shaders.
Seth Larson: Yeah, GPU shaders.
Seth Larson: Throw them in there.
Seth Larson: Why not?
Seth Larson: I mean, Mike knows this even more than I do.
Seth Larson: Like there's video.
Seth Larson: It's just all sorts of great, great content on PyPI.
Seth Larson: Yeah, not that we encourage that use case.
Seth Larson: But there's a lot of stuff in Python packages
Seth Larson: and SBOMs are a great way to represent
Seth Larson: like the provenance of all that software without it being Python specific.
Michael Kennedy: For sure. All right. Picking up speed because we got a couple more to go through.
Michael Kennedy: We've got maybe the follow on or the after case here is post incident runtime S-bomb generation
Michael Kennedy: from Python memory by Hala, Ali and Andrew Case. And the general idea is there's what you've
Michael Kennedy: declared to be your supply chain inputs and then there's what's actually there. So maybe look at
Michael Kennedy: what is loaded into memory?
Michael Kennedy: And you're like, oh, why is this library also loaded?
Michael Kennedy: Or some other hacker type of thing got put on the machine
Michael Kennedy: and then loaded in.
Michael Kennedy: And now we can detect that through this generation
Michael Kennedy: of looking at loaded modules and so on, right?
Seth Larson: Yeah, this one, I was really intrigued about this
Seth Larson: because it went so deep into Python memory management
Seth Larson: and how modules are loaded and how functions are called
Seth Larson: like extremely deep dive into this topic.
Seth Larson: And so I was like super I was like in awe watching this talk.
Seth Larson: I was like they just kept on being more cool stuff happening.
Seth Larson: And so I was like looking at like runtime analysis.
Seth Larson: You can see what functions are being called.
Seth Larson: You can like to see whether or not like a CD is affecting your program.
Seth Larson: So you can like introspect your program live while it's running and see whether
Seth Larson: interesting like something is even being used.
Seth Larson: You don't even have to like look at the source code to check or whatever.
Seth Larson: You just look at your program and see if it's using that code.
Michael Kennedy: Yeah, yeah, because if you bother, and I don't, maybe you all know better than I do, but if I were to speculate, I'd say less than 1% of people actually run pip audit or uv pip audit sort of thing, uv audit, whatever it is, on their code.
Michael Kennedy: But even if you do, or you look at the CVE and you see one, or maybe get the PendaBot tells you, oh, there's a CVE in one of your dependencies, you better go fix it.
Michael Kennedy: And then you got to go look and say, does it really matter?
Michael Kennedy: I remember one of the packages I was using, it said, oh, on Windows, there's this issue that if you take untrusted input and it's a PDF and it's this way, it's going to do this terrible thing.
Michael Kennedy: I'm like, yeah, but my deployment target is Linux on Docker and my dev machine is Mac.
Michael Kennedy: And I don't think I've taken untrusted input whatsoever for this.
Michael Kennedy: And so do I care?
Michael Kennedy: I don't really care.
Michael Kennedy: You know what?
Michael Kennedy: I'm just going to not worry about it.
Michael Kennedy: But PitBot will fill my build and then we're back to the cooldowns and all that.
Michael Kennedy: But nonetheless, it's like, I think that's part of the problem.
Michael Kennedy: And the fact, if you could say somehow combine the CVE and the CVE, actually, I can verify
Michael Kennedy: it doesn't touch that bit of code.
Michael Kennedy: That's super cool.
Mike Fiedler: Yeah, on that front, I think there's an ongoing effort through the open source vulnerability
Mike Fiedler: database to surface more rich details as to like code paths that are affected.
Mike Fiedler: So that way you could automatically audit, do I use those code paths?
Mike Fiedler: and then thus either fail or not fail your build.
Mike Fiedler: But again, in languages like Python,
Mike Fiedler: where dynamicism is used,
Mike Fiedler: getting that an accurate representation
Mike Fiedler: of am I using that code path
Mike Fiedler: is often a harder question to answer
Mike Fiedler: because you don't necessarily even have to import the library
Mike Fiedler: via an import statement.
Mike Fiedler: You can use import lib.
Mike Fiedler: You can do all sorts of other different things.
Mike Fiedler: So the challenge is how do we surface enough information
Mike Fiedler: make a confident decision without misleading you down the wrong path.
Michael Kennedy: Indeed.
Michael Kennedy: Let's keep moving on because time is a ticking.
Michael Kennedy: The next one is GitHub Action Security and Python Packages by Andrew Nesbitt.
Michael Kennedy: So, yeah.
Michael Kennedy: Terrifying.
Michael Kennedy: There's some really weird stuff.
Michael Kennedy: That was very eye-ocaching.
Michael Kennedy: Weird caching.
Mike Fiedler: So Andrew's a nerd, right?
Mike Fiedler: He's the best kind of nerd, right?
Mike Fiedler: He loves researching things.
Mike Fiedler: He loves accumulating data.
Mike Fiedler: He's the author behind Ecosystems.
Mike Fiedler: That's ecosystem.sys.ms.
Mike Fiedler: And he analyzes lots of projects
Mike Fiedler: and was able to analyze a lot of GitHub project
Mike Fiedler: IPI-linked repositories with Zismore,
Mike Fiedler: which he did not write.
Mike Fiedler: But it is a very cool tool
Mike Fiedler: and great to audit your GitHub actions
Mike Fiedler: for security vulnerability.
Mike Fiedler: It's scary what its surface is.
Mike Fiedler: And when we're talking about
Mike Fiedler: folks getting started, this should be on that checklist of did you run Zizmor on your repository?
Mike Fiedler: Better yet, did you add it as a GitHub action to run always on your repository? But, you know,
Mike Fiedler: it surfaces a lot of easy to fix findings, and then sometimes surfaces the scarier ones of you've
Mike Fiedler: left yourself open to an injection that leads to a exploit. And he talked about a couple of those.
Michael Kennedy: Very interesting. Yeah, the GitHub actions have a lot of nuance and weird, there was some weird
Michael Kennedy: cache poisoning that people could submit PRs that would trigger a GitHub Action build.
Michael Kennedy: And even if they had no rights to the target system, their PR could point at a malicious
Michael Kennedy: package somehow.
Michael Kennedy: And then that would get cached into the actual build because the build ran in the context
Michael Kennedy: of the target repo.
Michael Kennedy: And hey, wouldn't it be nice if we could just cache this so we don't save it?
Michael Kennedy: Maybe we should cache other people's inputs.
Michael Kennedy: You said all the keywords there, Michael.
Mike Fiedler: You said pull request and target.
Mike Fiedler: That is the trigger for a lot of these.
Mike Fiedler: GitHub Actions has a lot of different trigger keywords.
Mike Fiedler: Push, pull request.
Mike Fiedler: There is one called pull request target.
Mike Fiedler: That one runs on forks.
Mike Fiedler: It runs with elevated privileges.
Mike Fiedler: It is dangerous.
Mike Fiedler: Zismore calls that out as like,
Mike Fiedler: you are probably using this wrong
Mike Fiedler: because it is a very sharp, sharp knife.
Mike Fiedler: If you pick it up the wrong way,
Mike Fiedler: You have now cut your hand and it's easy to pick up the wrong one.
Mike Fiedler: So Runs is more.
Michael Kennedy: Please start trying to juggle three of those.
Michael Kennedy: Yeah.
Mike Fiedler: Don't juggle three.
Mike Fiedler: Runs is more.
Mike Fiedler: Follow its recommendations.
Michael Kennedy: Move on with your life.
Michael Kennedy: All right.
Michael Kennedy: One more here.
Michael Kennedy: Breaking Bad packages.
Michael Kennedy: Why traditional vulnerability tracking fails supply chain attacks.
Michael Kennedy: Shelby Cunningham and Madison Figuerely.
Seth Larson: So I really like this one because it talks about the CVE system,
Seth Larson: which is something that I've spent way too much time kind of digging around in.
Seth Larson: And it's this challenge between intentional malicious software, like malware that we see on PyPI, and vulnerabilities.
Seth Larson: And I think that there's this, sometimes we use CVEs for malware and sometimes we don't.
Seth Larson: And because if we did, there'd be, you know, millions.
Seth Larson: And it's like, do we, do we not?
Seth Larson: Like, what are the trade-offs?
Seth Larson: Does it overwhelm the system?
Seth Larson: Is it even useful?
Seth Larson: And it talked about all of these different watering hole attacks that ended up publishing malware to different open source ecosystems, including PIPI and how the this vulnerability record like CD or OSV responded in turn and like how people got the word out.
Seth Larson: How did people respond to it?
Seth Larson: And yeah, it's it's really challenging because it feels it feels so close to being the right system to use, but not quite.
Seth Larson: And maybe that means that we need to build a new system.
Seth Larson: Maybe that means that who knows what it means.
Seth Larson: We need to do some more thinking in this area.
Michael Kennedy: Yeah, I totally see it because malware is not a problem in valid software, right?
Michael Kennedy: It's just a bad piece of software.
Michael Kennedy: Whereas if I open up this PDF in Microsoft Word and it turns me into a botnet, that's a CVE.
Michael Kennedy: Because Microsoft Word isn't intended to be malware.
Michael Kennedy: All right, folks.
Michael Kennedy: I think that might be it for the time that we've got.
Michael Kennedy: So I guess final call to action here.
Michael Kennedy: We have all these talks on PyCon US, and everyone links to them.
Michael Kennedy: So every part of the show notes links up to them.
Michael Kennedy: So I'll put the links in the show page for people listening and their podcast player and so on.
Michael Kennedy: So go out and check out the interesting talks on YouTube.
Michael Kennedy: I can put that out there.
Michael Kennedy: But let's leave it with us.
Michael Kennedy: Juanita, I'll give you the first word, and you can each answer this one.
Michael Kennedy: So if everyone out there listening did one thing this week because of this episode,
Michael Kennedy: because of what they heard us talking about.
Michael Kennedy: What should they do?
Michael Kennedy: What would you tell them?
Juanita Gomez: Like, here's the one thing really I recommend most.
Juanita Gomez: If it's only one, I'll just pick you, Sizmore.
Mike Fiedler: Okay.
Mike Fiedler: Mike?
Mike Fiedler: Look at trusted publishing.
Mike Fiedler: Convert at least one of your projects to use it.
Mike Fiedler: And don't forget to delete the old token.
Mike Fiedler: Got it.
Mike Fiedler: Okay.
Michael Kennedy: Spoken like a person who works on PyPI security.
Michael Kennedy: Seth?
Seth Larson: Yeah.
Seth Larson: So my recommendation is if after hearing all of these security problems,
Seth Larson: solutions like what's happening in security and how it's not this like solve thing to I'm assuming
Seth Larson: that because you're watching this show that you have an interest in Python and you maybe you even
Seth Larson: work with Python. I would recommend if you depend on Python or third party packages to look upstream
Seth Larson: and to see that all of this work is being done and that there is so much more to do and support
Seth Larson: those open source ecosystems that you depend on to enable them to do security. So whether that's
Seth Larson: donating engineering time, so like working on a security team, making some of these security
Seth Larson: focused contributions to open source projects that you care about, or whether that comes as
Seth Larson: like funding, right? So like donating to the Python Software Foundation, for example,
Seth Larson: nonprofit that is the home for Mike and I's job. Security is not free. It costs a lot of time to
Seth Larson: implement all of these changes that everyone ends up benefiting from anyways. And it really would
Seth Larson: not happen without internal advocates at companies and organizations working with their management and
Seth Larson: all of that to convince them to support open source that they depend on. So that's my recommendation.
Michael Kennedy: Perfect. And one final thing, maybe Juanita, people want to keep in touch with you.
Michael Kennedy: Where's, you know, can they follow you on social? Do you got a blog?
Juanita Gomez: I have a website where I guess I sometimes put things that I have done or I'm doing currently.
Juanita Gomez: But honestly, the best way to reach me out is probably LinkedIn.
Juanita Gomez: I try to post where I'm going to be if I'm giving talks, where to find me.
Juanita Gomez: But if you just want to chat, just message me on LinkedIn.
Juanita Gomez: I'm very happy to connect with people that are interested in any of the things that I'm working on.
Mike Fiedler: Right on. Mike?
Mike Fiedler: Yeah, I have a landing page at MikeTheMan.dev, and that will link you out to all of my other different locations, socials, blogs, etc.
Mike Fiedler: And that is, yeah, I'm also on LinkedIn, but I also try to post on all the different socials of like, yep, I'm going to be at EuroPython.
Mike Fiedler: Here's where I'm going to be.
Mike Fiedler: So that way people can accost me on the hallway track and demand some new security feature, which is wonderful.
Michael Kennedy: Love talking about it.
Michael Kennedy: The most well-attended track of the conference indeed.
Michael Kennedy: Seth.
Seth Larson: Yeah, so you can find me online, sethamlarson.dev.
Seth Larson: Links out to everywhere.
Seth Larson: I prefer Mastodon, but I also respond to LinkedIn.
Seth Larson: And there's even Signal there.
Seth Larson: So if you want to just like privately ask me a security question or something like that,
Seth Larson: or you just want to chat for a second, I respond to Signal.
Seth Larson: So any of those locations work for me.
Seth Larson: I publish a blog there as well.
Seth Larson: So if you want to read anything about security or retro video games, those are your options.
Seth Larson: Solid.
Michael Kennedy: Great.
Michael Kennedy: Thank you all for being here.
Michael Kennedy: See you later.
Michael Kennedy: Thank you.
Michael Kennedy: This has been another episode of Talk Python To Me.
Michael Kennedy: Thank you to our sponsors.
Michael Kennedy: Be sure to check out what they're offering.
Michael Kennedy: It really helps support the show.
Michael Kennedy: Take some stress out of your life.
Michael Kennedy: Get notified immediately about errors and performance issues
Michael Kennedy: in your web or mobile applications with Sentry.
Michael Kennedy: Just visit talkpython.fm/sentry
Michael Kennedy: and get started for free.
Michael Kennedy: Be sure to use our code talkpython26.
Michael Kennedy: That's Talk Python, the numbers two, six, all one word.
Michael Kennedy: Talk Python and Python Bytes both now have MCP servers.
Michael Kennedy: Point your AI at 10 plus years of Python episodes, transcripts, and show notes.
Michael Kennedy: Free.
Michael Kennedy: Click MCP in the nav at talkpython.fm and at pythonbytes.fm.
Michael Kennedy: If you or your team needs to learn Python,
Michael Kennedy: we have over 270 hours of beginner and advanced courses on topics
Michael Kennedy: ranging from complete beginners to async code, Flask, Django, HTML, and even LLMs.
Michael Kennedy: Best of all, there's no subscription in sight.
Michael Kennedy: browse the catalog at talkpython.fm.
Michael Kennedy: And if you're not already subscribed to the show
Michael Kennedy: on your favorite podcast player,
Michael Kennedy: what are you waiting for?
Michael Kennedy: Just search for Python in your podcast player.
Michael Kennedy: We should be right at the top.
Michael Kennedy: If you enjoy that geeky rap song,
Michael Kennedy: you can download the full track.
Michael Kennedy: The link is actually in your podcast blur share notes.
Michael Kennedy: This is your host, Michael Kennedy.
Michael Kennedy: Thank you so much for listening.
Michael Kennedy: I really appreciate it.
Michael Kennedy: I'll see you next time.
Michael Kennedy: I'm out.