WEBVTT 00:00:00.001 --> 00:00:03.780 Python is often described as a batteries-included language and ecosystem. 00:00:03.780 --> 00:00:08.660 In fact, that's been taken so far, there's even a delightful Easter egg in the Python REPL. 00:00:08.660 --> 00:00:11.720 Just type import anti-gravity to see what I mean. 00:00:11.720 --> 00:00:13.820 Where do these powerful packages come from? 00:00:13.820 --> 00:00:17.380 Well, the Python Package Index, or PyPI. 00:00:17.380 --> 00:00:22.900 On this episode, you will meet Nicole Harris, Ernest Durbin III, and Dustin Ingram. 00:00:22.900 --> 00:00:29.380 They were part of the team that has just launched the new version of PyPI over at pypi.org. 00:00:29.620 --> 00:00:33.260 Not only have they given us a great new website around packaging and Python, 00:00:33.260 --> 00:00:37.520 they have laid the foundation for innovation in the space for years to come. 00:00:37.520 --> 00:00:42.800 This is Talk Python To Me, episode 159, recorded April 18, 2018. 00:00:56.540 --> 00:01:03.460 Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. 00:01:03.460 --> 00:01:05.580 This is your host, Michael Kennedy. 00:01:05.580 --> 00:01:07.580 Follow me on Twitter, where I'm @mkennedy. 00:01:07.580 --> 00:01:11.480 Keep up with the show and listen to past episodes at talkpython.fm. 00:01:11.480 --> 00:01:14.060 And follow the show on Twitter via at Talk Python. 00:01:15.180 --> 00:01:18.440 This episode is brought to you by ActiveState and Codicy. 00:01:18.440 --> 00:01:20.700 Please check out what they're offering during their segments. 00:01:20.700 --> 00:01:22.140 It really helps support the show. 00:01:22.140 --> 00:01:23.640 Hey, everyone. 00:01:23.640 --> 00:01:27.020 Before we get to the exciting news about the new PyPI launch, 00:01:27.020 --> 00:01:30.360 I want to tell you about a brand new course we just launched. 00:01:30.360 --> 00:01:33.140 It's called Python 3, an Illustrated Tour. 00:01:33.140 --> 00:01:38.720 And it's a five-hour visual and code-based tour of all the features in Python 3. 00:01:39.140 --> 00:01:44.680 It's written by Matt Harrison, who has authored 15 technical books and is a best-selling Python author. 00:01:44.680 --> 00:01:48.780 Check it out over at talkpython.fm/illustrated. 00:01:48.780 --> 00:01:53.720 And if you get the course this week, we'll throw in Matt's newest Python book for free, 00:01:53.720 --> 00:01:55.640 which is a perfect complement for the course. 00:01:55.640 --> 00:02:00.260 And if you have the Everything Bundle already, then you should definitely check out the course 00:02:00.260 --> 00:02:03.320 because it's included in your bundle, and you can just go take it. 00:02:03.560 --> 00:02:04.880 I hope you love this new course. 00:02:04.880 --> 00:02:09.660 We have many more coming down the pipe, and I'm looking forward to sharing those with you as well. 00:02:09.660 --> 00:02:11.320 Now, let's hear about the new PyPI. 00:02:11.320 --> 00:02:14.980 Nicole, Dustin, Ernest, welcome to Talk Python. 00:02:14.980 --> 00:02:15.840 Hey, thanks. 00:02:15.840 --> 00:02:16.900 It's great to be here. 00:02:16.900 --> 00:02:17.540 Thanks for having us. 00:02:17.540 --> 00:02:20.140 Yeah, you all have done something amazing. 00:02:20.140 --> 00:02:24.680 It's almost like you've caught a unicorn in the mythical sense of, like, 00:02:25.240 --> 00:02:33.740 there's been this talk of a new PyPI website and infrastructure for so long, and then, like, here it is. 00:02:33.740 --> 00:02:36.240 And you all are, you know, really central to doing this. 00:02:36.240 --> 00:02:42.480 So I'm super excited to talk about this, the rollout, the technology behind it, new features we're going to get. 00:02:42.480 --> 00:02:44.280 We have already gotten things like that. 00:02:44.280 --> 00:02:49.300 But before we get to that, let's start with your story just briefly since there are three of you. 00:02:49.300 --> 00:02:50.860 How do you get into programming Python? 00:02:50.860 --> 00:02:51.960 Nicole, go first. 00:02:51.960 --> 00:02:55.520 I started off with programming generally about 10 years ago. 00:02:55.520 --> 00:03:02.720 My degree is actually in film and photography, and I wanted to make a website to put up my animation works. 00:03:02.720 --> 00:03:07.640 And that kind of led me to HTML and CSS, which is still my specialization. 00:03:07.640 --> 00:03:16.280 And from that, kind of, I became what was back then a sort of generic web designer before we had lots of different specializations. 00:03:16.280 --> 00:03:19.720 And then my husband is actually a Python programmer. 00:03:19.720 --> 00:03:22.340 So that's how I got involved in the Python community. 00:03:22.340 --> 00:03:28.720 And I don't program in Python very much these days, but I do sort of dabble in it every now and again. 00:03:28.720 --> 00:03:29.760 Yeah, yeah, very nice. 00:03:29.760 --> 00:03:30.960 Ernest, how about yourself? 00:03:30.960 --> 00:03:39.440 I graduated from school with a degree in physics and math in the sort of peak of the recession back in 2007-8 era. 00:03:40.000 --> 00:03:43.900 And eventually conned my way into a job as a business analyst. 00:03:43.900 --> 00:03:49.020 And at that point, I started programming in order to stop using Excel. 00:03:49.020 --> 00:03:52.060 And then years later, I've come to this point. 00:03:52.060 --> 00:03:52.580 Very cool. 00:03:52.580 --> 00:03:57.140 I love how you sort of took your career and just kind of laddered it up or leveled it up. 00:03:57.140 --> 00:03:57.820 Right? 00:03:57.880 --> 00:03:58.680 Like, I'm math and physics. 00:03:58.680 --> 00:04:00.380 I'm not going to work at CERN. 00:04:00.380 --> 00:04:01.040 So now what? 00:04:01.040 --> 00:04:03.080 And then you just, you know, work your way up that ladder. 00:04:03.080 --> 00:04:05.920 Like, I also, I've said this several times on the show, of course. 00:04:05.920 --> 00:04:12.840 But I also was working my PhD in math and then kind of abandoned for my self-taught developer path many years ago. 00:04:12.840 --> 00:04:14.240 Dustin, how about you? 00:04:14.340 --> 00:04:14.540 Yeah. 00:04:14.540 --> 00:04:17.020 So I went to school for computer science. 00:04:17.020 --> 00:04:20.760 And I'm not really sure when I first was introduced to Python. 00:04:20.760 --> 00:04:29.880 But I do remember at some point, you know, after having done a lot of C and C++ in my studies, coming across this Python thing and being like, oh, this looks so much nicer. 00:04:29.880 --> 00:04:33.840 So I slowly sort of worked that in as much as I could. 00:04:33.840 --> 00:04:37.680 And, yeah, now I could probably call myself a Python developer. 00:04:37.680 --> 00:04:38.540 That's pretty awesome. 00:04:38.540 --> 00:04:40.260 So you're like, this can't work. 00:04:40.260 --> 00:04:41.520 There's only five lines, right? 00:04:41.520 --> 00:04:45.080 Like, in C++, I'd definitely have to, like, write a whole app around this. 00:04:45.080 --> 00:04:46.460 So, but it works. 00:04:46.460 --> 00:04:47.500 It's the beauty of Python, right? 00:04:47.500 --> 00:04:48.860 Nice. 00:04:48.860 --> 00:04:49.200 Okay. 00:04:49.200 --> 00:04:57.760 So first of all, I want to start with a big piece of news, which we've been hinting at, or I've been hinting at, but has a particular date. 00:04:57.760 --> 00:05:03.720 So the new PyPI.org, which, by the way, for a while is PyPI.io. 00:05:03.720 --> 00:05:04.700 I want to ask you about that. 00:05:04.700 --> 00:05:12.260 But PyPI.org has launched and Legacy PyPI is shutting down April 30th, right? 00:05:12.260 --> 00:05:14.240 That's on the blog recently announced. 00:05:14.240 --> 00:05:15.460 Congratulations. 00:05:15.460 --> 00:05:16.560 How do you all feel about that? 00:05:16.560 --> 00:05:17.040 Thanks. 00:05:17.040 --> 00:05:18.580 I think we're super excited. 00:05:18.580 --> 00:05:20.940 Yeah, I don't think there's anything negative to say about it. 00:05:21.020 --> 00:05:26.660 I mean, it's just to see the culmination of the effort come to, like, a moment has been incredible. 00:05:26.660 --> 00:05:35.260 And there'll be another sort of celebratory secondary on the 30th when we sort of say goodbye to something that's been around for so long. 00:05:35.260 --> 00:05:35.620 Yeah. 00:05:35.620 --> 00:05:40.300 We're going to have to get used to less gray, more red, or more blue, right? 00:05:40.300 --> 00:05:41.160 It's blue, isn't it? 00:05:41.160 --> 00:05:42.700 Is that your work, Nicole? 00:05:42.840 --> 00:05:48.720 It sounds like you might have done a fair amount of the redesign HTML bootstrap type of thing. 00:05:48.720 --> 00:05:57.880 I joined the project back in 2015 because Donald, who's our lead developer, I think you've already met and interviewed. 00:05:57.880 --> 00:05:59.140 Yeah, he's been on the show twice. 00:05:59.140 --> 00:05:59.700 He's great. 00:05:59.700 --> 00:06:00.120 Yeah. 00:06:00.120 --> 00:06:06.660 So he put a call out basically to say, I'm rebuilding this thing, but I'm terrible at design. 00:06:06.660 --> 00:06:09.240 So is there anybody out there who can help? 00:06:09.780 --> 00:06:22.960 And I got in touch, and so that's kind of how I ended up in charge of both the user interface, the user experience, and I also took charge of the HTML and the SCSS code base as well. 00:06:22.960 --> 00:06:25.560 So kind of front end minus JavaScript. 00:06:25.560 --> 00:06:26.640 Yeah, that's really cool. 00:06:26.640 --> 00:06:31.220 Anything that looks good is Nicole's doing and not any of the rest of us. 00:06:31.220 --> 00:06:39.760 I got to say congratulations because I do feel like it looks really modern, not overly designed, but it definitely feels like, you know, 2018. 00:06:40.260 --> 00:06:41.240 Somewhere you want to be. 00:06:41.240 --> 00:06:46.560 It doesn't look old, neglected, gray, and just like default, like browser font style, right? 00:06:46.560 --> 00:06:47.860 Like it's really, really good. 00:06:47.860 --> 00:06:51.300 And I think on one hand design, it's how much does it matter, right? 00:06:51.300 --> 00:06:52.640 It's like a package warehouse. 00:06:52.640 --> 00:06:57.040 But on the other, I think it sends a message to the community like this place is special. 00:06:57.040 --> 00:06:57.780 We care about it. 00:06:57.780 --> 00:07:02.540 We put in effort to style it and make it really look good and be usable, right? 00:07:02.540 --> 00:07:16.240 Yeah, and I think a lot of the design focus for me was thinking about how much Python is a teaching language and how for how many programmers it might be their first experience dealing with a package index. 00:07:16.240 --> 00:07:23.200 So it was really important to me that it looked friendly and it reflected the values of the Python community. 00:07:23.200 --> 00:07:31.180 So both in terms of the design, but also in terms of the accessibility features that we've built into the front end code base. 00:07:31.180 --> 00:07:35.580 We're trying to make sure that it's serving as many people as well as possible. 00:07:35.580 --> 00:07:36.100 That's cool. 00:07:36.100 --> 00:07:40.460 And do you mean things like ARIA, like screen reader indicators and stuff like that? 00:07:40.580 --> 00:07:47.400 We've done a reasonable amount of work on that so far and we've actually got an accessibility audit happening this week as well. 00:07:47.400 --> 00:07:50.280 So there'll be more improvements on that side. 00:07:50.280 --> 00:08:00.940 But given there's so many users of the site currently, it's just from a percentage perspective, you know, that there is going to be a portion of those users who are going to be using assistive technology. 00:08:00.940 --> 00:08:03.240 So we need to be looking after them. 00:08:03.240 --> 00:08:08.500 And I think that reflects the Python community and the way that we go about things offline as well. 00:08:08.500 --> 00:08:09.060 Very nice. 00:08:09.480 --> 00:08:12.240 Let's touch on the contributions the other two of you have made. 00:08:12.240 --> 00:08:15.920 So Ernest, what was your major part in this whole project here? 00:08:15.920 --> 00:08:16.320 Sure. 00:08:16.320 --> 00:08:23.980 So since about 2013, 12 or 13-ish, I've been contributing to the Python Software Foundation's infrastructure. 00:08:23.980 --> 00:08:31.940 And so this is the servers and services behind python.org, www.python.org, mail, wiki, etc. 00:08:31.940 --> 00:08:38.420 And so PyPI is one of the largest and most used of the services provided by the PSF. 00:08:38.780 --> 00:08:43.320 And I got involved primarily just keeping things turned on. 00:08:43.320 --> 00:08:51.580 In 2013, there was a large contribution that I did to modernize the infrastructure that hosted the old PyPI. 00:08:51.980 --> 00:08:59.100 And over the past few years, I've continued that work in adding to the reliability and telemetry of PyPI. 00:08:59.100 --> 00:09:11.260 And so with the warehouse project, Donald, Stuffed, and myself both sort of took a step back and said, if we were going to do it all over again, how can we make sure we have excellent infrastructure for warehouse? 00:09:11.260 --> 00:09:27.660 So my main contribution in the most recent work has been a mixture primarily of the infrastructure behind PyPI.org and also some code changes that features as well as just stuff to make it more compatible and easier to operate and do so reliably. 00:09:27.660 --> 00:09:28.500 Yeah, very cool. 00:09:28.500 --> 00:09:30.380 Dustin, how about yourself? 00:09:30.380 --> 00:09:34.540 Yeah, I joined the project just as a volunteer contributor about two years ago. 00:09:34.540 --> 00:09:43.200 I think I just had happened to come across it looking at Donald's GitHub and I was like, wow, this is a really usable PyPI, but it's not finished. 00:09:43.200 --> 00:09:50.780 And, you know, as a new contributor, I was pretty just attracted to it because I could actually contribute to it. 00:09:50.780 --> 00:09:55.420 Legacy is Behomoth and has very few tests. 00:09:55.420 --> 00:09:59.180 And even to run it locally, you have to actually go in and comment out a bunch of code. 00:09:59.180 --> 00:10:03.040 So it's really abrasive for new contributors. 00:10:03.480 --> 00:10:05.040 And warehouse is not like that at all. 00:10:05.040 --> 00:10:13.960 So I sort of started making some contributions, doing some like Elasticsearch tuning and that kind of thing, and just adding elements to the UI that weren't there before. 00:10:13.960 --> 00:10:23.220 And so I think I learned this to work necessary and also making a lot of contributions to the just tooling ecosystem. 00:10:23.220 --> 00:10:28.260 So that's other projects like Twine and pip and things like that, just to work with the new PyPI. 00:10:28.260 --> 00:10:29.120 Yeah, very cool. 00:10:29.260 --> 00:10:36.320 Now, the three of you are here, but you all have mentioned Donald Stufft, who's been spearheading this and deserves a lot of credit as well. 00:10:36.320 --> 00:10:37.960 So congratulations to him. 00:10:37.960 --> 00:10:39.100 Who else? 00:10:39.100 --> 00:10:43.400 Is there anyone else who we should sort of give a shout out to while we're talking to you all? 00:10:43.400 --> 00:10:44.180 Yeah, absolutely. 00:10:44.540 --> 00:10:46.260 I want to point out Sumina. 00:10:46.260 --> 00:10:50.940 Sumina, I actually have never tried to pronounce Sumina's last name out loud. 00:10:50.940 --> 00:11:00.000 So Sumina H took an incredible role in the project management and leadership over the past few months and bringing this together. 00:11:00.680 --> 00:11:01.120 Absolutely. 00:11:01.120 --> 00:11:10.580 It was a huge driver in a lot of the work that we did to encourage and welcome and have sticky contributors to the project. 00:11:10.580 --> 00:11:25.300 So I sort of said this a few, I don't remember when exactly, but there was a point where whenever Donald or I would tweet the PyPI team, what we meant was whichever one of us happened to have done something that week or that month with PyPI. 00:11:25.300 --> 00:11:38.420 And I attribute personally a lot of the reason why when I say the PyPI team now, it's a collection of more than like five, I mean, it's probably close to like seven or eight people who are regularly contributing and there's a team. 00:11:38.420 --> 00:11:42.260 And when I say that now, I say it earnestly, pardon the pun. 00:11:42.260 --> 00:11:48.440 But yeah, so Sumina must be, in my opinion, must be sort of encouraged and called out here as well. 00:11:48.520 --> 00:11:58.180 Yeah, I just want to say, I don't think the project would have been as big of a success as it was if Sumina hadn't been sort of organizing and hurting us along the way. 00:11:58.180 --> 00:12:00.120 She did an exceptional job. 00:12:00.120 --> 00:12:00.740 So glad to hear. 00:12:00.740 --> 00:12:02.600 So congratulations to you all. 00:12:02.600 --> 00:12:08.660 So I have two sort of burning questions around the new PyPI.org. 00:12:08.660 --> 00:12:09.160 Three. 00:12:09.160 --> 00:12:11.060 Let's start with a simple one. 00:12:11.600 --> 00:12:19.560 We have PyPI.python.org slash PyPI, which is a crazy location on the internet because why the duplication? 00:12:19.560 --> 00:12:21.820 But anyway, we have that. 00:12:21.820 --> 00:12:25.360 And then for a little while, you had PyPI.io. 00:12:25.360 --> 00:12:33.840 And then you switched to PyPI.org for like where the actual new warehouse, the new Python package index lives. 00:12:33.840 --> 00:12:36.020 Why did you change it halfway along the way there? 00:12:36.020 --> 00:12:36.640 Sure. 00:12:36.640 --> 00:12:37.820 This is a good story. 00:12:37.820 --> 00:12:42.940 So it started at Python.org slash PyPI is where it initially lived. 00:12:42.940 --> 00:12:51.680 And then it moved to PyPI.python.org slash PyPI because it was easy to change the domain and not so hard, not so easy to change the URLs. 00:12:51.680 --> 00:12:52.020 Right. 00:12:52.020 --> 00:12:57.340 It was more easy to separate it like the infrastructure to another server rather than behind the load balancer or something like that. 00:12:57.340 --> 00:12:57.560 Right. 00:12:57.560 --> 00:12:58.000 Mm hmm. 00:12:58.000 --> 00:12:58.340 Okay. 00:12:58.420 --> 00:13:07.200 And then eventually PyPI.io, I don't remember when we got it, but we've been using that for sort of the internal domain for PyPI for a long time. 00:13:07.200 --> 00:13:09.580 So for the actual servers behind the scenes. 00:13:09.580 --> 00:13:14.580 And when warehouse started to get to the point where Donald was like, oh, man, this is real. 00:13:14.580 --> 00:13:16.640 We can start deploying this somewhere. 00:13:17.160 --> 00:13:20.400 We went PyPI.io because we had it. 00:13:20.400 --> 00:13:28.740 The frustrating part is that PyPI.org was not owned by the PSF or Python community member for a long time. 00:13:28.740 --> 00:13:40.400 So basically, the reason why it switched midstream was that PyPI.org was successfully obtained by the PSF and by the PyPI maintainers. 00:13:41.280 --> 00:13:49.540 It was the sort of the gold standard of the domain that we desired, but it wasn't ours until I don't remember when it was when that happened. 00:13:49.540 --> 00:13:51.060 But when it became ours. 00:13:51.060 --> 00:13:51.820 Yeah. 00:13:51.820 --> 00:13:54.220 When it became ours, we immediately switched. 00:13:54.220 --> 00:13:54.640 I see. 00:13:54.640 --> 00:13:56.040 So that was what you wanted all along. 00:13:56.040 --> 00:13:59.540 But there was just this like squatter type of situation thing going on. 00:13:59.540 --> 00:14:01.880 It is the Internet, isn't it? 00:14:01.880 --> 00:14:04.220 All right. 00:14:04.220 --> 00:14:06.360 So whoever wants to take this one, feel free to jump in. 00:14:06.620 --> 00:14:17.720 One thing that I'm wondering is what features or benefits do we get other than the underlying system is more polished, easier to contribute to and so on. 00:14:17.720 --> 00:14:20.520 But as just a user, like suppose I don't care about that. 00:14:20.520 --> 00:14:22.640 It could be written in PHP for all I care. 00:14:22.640 --> 00:14:27.200 But when I go to it, what do I get to do that's better or different? 00:14:27.200 --> 00:14:28.880 Honestly, there's not much different. 00:14:28.880 --> 00:14:35.440 Most of the goal of this project was to move to a system that would allow us to more easily add new and exciting features. 00:14:35.440 --> 00:14:45.340 So we have a lot of ideas like new APIs and ability to deprecate packages and things like that that are now going to be, you know, not trivial, but much, much, much easier to implement. 00:14:45.340 --> 00:14:47.560 Much easier than they would have been on legacy. 00:14:47.560 --> 00:14:58.440 So a lot of this is just modernization efforts and taking what was originally just a proof of concept that became PyPI into something that's actually been thought through and designed and robust. 00:14:58.780 --> 00:15:17.460 Yeah, I think you mentioned earlier, and Donald himself had said this previously, that the original PyPI, the gray one, not the blue one, was really like based on almost like custom web programming, not even like Pyramid or Flask or something. 00:15:17.460 --> 00:15:19.660 It was really hard to get to. 00:15:19.660 --> 00:15:21.720 People would come and say, hey, I want to contribute the new feature. 00:15:21.800 --> 00:15:24.420 They would look and go, actually, not that much. 00:15:24.420 --> 00:15:25.900 And then they would go away, right? 00:15:25.900 --> 00:15:30.880 And so now, maybe this is a good place to switch into it. 00:15:30.880 --> 00:15:36.540 You know, we could talk a little bit about what the underlying technology for that is, right? 00:15:36.680 --> 00:15:40.280 So maybe Dustin Ernest, talk about the back end. 00:15:40.280 --> 00:15:44.380 And Nicole, we could touch on the front end as well, because that also got super modernized, I'm sure. 00:15:44.380 --> 00:15:44.860 Yeah. 00:15:44.860 --> 00:15:51.800 So the thing about legacy is that it was written at a time that predates a lot of the frameworks and tools that we know exist today. 00:15:51.800 --> 00:15:54.360 So, you know, it was doing the best with what it had, I think. 00:15:54.360 --> 00:16:02.820 It's not a real direct criticism of it, but it just it came into existence like really early before much of the other stuff, right? 00:16:02.820 --> 00:16:06.000 Like you pip install Flask, but where are you going to do that from if you don't have it? 00:16:06.100 --> 00:16:06.420 It's old. 00:16:06.420 --> 00:16:11.020 Yeah, the modern PyPI, the framework we chose to use is Pyramid. 00:16:11.020 --> 00:16:19.460 And that was after a little bit of experimentation that sort of Pyramid just allows us to have a little more control over various things that we need to do to be PyPI. 00:16:19.460 --> 00:16:24.160 And I think a big part of this project was the infrastructure work that Ernest did. 00:16:24.160 --> 00:16:26.360 And I think he should talk about that more. 00:16:26.360 --> 00:16:27.080 Yeah, go for it. 00:16:27.080 --> 00:16:32.600 We're now deployed on top of a nice, buzzy framework, piece of infrastructure called Kubernetes. 00:16:33.420 --> 00:16:40.560 And so we sort of looked at that as getting to the point where it as a technology, Kubernetes has come so far. 00:16:40.880 --> 00:16:48.020 And by the time warehouse is going to be really real, Donald and I are both comfortable with sort of targeting that. 00:16:48.020 --> 00:17:01.580 And the biggest drawback that that as a platform has is right now sort of the industry standard of the de facto for deploying to Kubernetes is you write a bunch of YAML or you use something to generate a template for YAML. 00:17:02.280 --> 00:17:17.140 So the goal was basically to have a lot of similar features to other platform as a service and do so without really having to have warehouse maintainers or PyPI maintainers worry too much about what's actually happening. 00:17:17.840 --> 00:17:31.500 And so a project came out of this work called Cavitage, which is a platform within Kubernetes and a web app and worker on top of it that just basically manage continuous deployment. 00:17:31.500 --> 00:17:39.680 So you can set and configure your environment variables and such and then deploy your service and it pops up in a known URL. 00:17:39.980 --> 00:17:40.640 That's really sweet. 00:17:40.640 --> 00:17:44.880 So you basically, as a contributor, I do a check-in to a Git branch, maybe a PR. 00:17:44.880 --> 00:17:53.180 And when that is accepted, that will trigger sort of Kubernetes to pull down a new version and just kick off, you know, sort of reroute the request? 00:17:53.180 --> 00:17:54.480 What happens there? 00:17:54.480 --> 00:17:55.440 Not yet. 00:17:55.440 --> 00:17:56.320 That's the dream? 00:17:56.320 --> 00:18:01.820 That is something that is another long-term benefit that we can sort of foresee out of this. 00:18:01.820 --> 00:18:14.180 Right now, the biggest benefit that we get from this is we have incredible flexibility in the way that we deploy warehouse and how we change how many resources it has effectively. 00:18:14.180 --> 00:18:21.200 So all of the primitives of the platform of Kubernetes effectively are really excellent. 00:18:21.200 --> 00:18:25.320 It's just that you have to bring them together and that's the part that's sort of difficult. 00:18:25.320 --> 00:18:29.300 So one of the biggest benefits we get is, you know, the zero downtime deployment. 00:18:29.300 --> 00:18:37.680 So since PyPI went live on Monday, we've already deployed like 30 times and nobody noticed, which is great. 00:18:37.680 --> 00:18:40.420 And then also just being able to be really flexible. 00:18:40.420 --> 00:18:47.220 We have, I think it's like five different types of things happening behind pypi.org. 00:18:47.400 --> 00:18:53.440 And we're running certain workloads under G Unicorn because they perform very well under G Unicorn. 00:18:53.440 --> 00:19:01.400 And the primary site is deployed using Twisted for that purpose. 00:19:01.400 --> 00:19:06.120 So overall, just, you know, having a little more flexibility and scalability was the main driver. 00:19:06.260 --> 00:19:12.560 And down the line, we're really excited to see about doing things like you mentioned, being able to do branch based deploys, et cetera. 00:19:12.560 --> 00:19:13.480 Yeah, that's really cool. 00:19:13.480 --> 00:19:14.120 Go, Dustin. 00:19:14.120 --> 00:19:21.720 I just wanted to mention, I totally forgot there is one feature that I'm super proud of that pypi.org does that Legacy did not. 00:19:21.980 --> 00:19:25.740 And I can't believe I forgot about this because this is my baby for a long time. 00:19:25.740 --> 00:19:30.000 But you can now write markdown descriptions on PyPI. 00:19:30.000 --> 00:19:31.320 Yeah, that's awesome. 00:19:31.320 --> 00:19:33.540 Which is a feature that people have wanted for a really long time. 00:19:33.540 --> 00:19:39.140 And that's really the one big thing that I'm super excited to say that the new PyPI does. 00:19:39.140 --> 00:19:39.580 That's cool. 00:19:39.580 --> 00:19:42.360 And that's part of that modernization that you're talking about, right? 00:19:42.360 --> 00:19:47.200 Like markdown, I don't know what people would have thought that meant back when it was created. 00:19:47.200 --> 00:19:56.280 But now, obviously, it's like the de facto way of formatted, structured input that doesn't break the site because it's missing a div or something, right? 00:19:56.280 --> 00:19:57.980 So it's really cool. 00:19:57.980 --> 00:20:01.220 Markdown didn't even exist when PyPI was first created. 00:20:01.220 --> 00:20:02.720 Yeah, I'm sure it didn't. 00:20:02.720 --> 00:20:06.360 Nicole, how did the sort of redesign look? 00:20:06.360 --> 00:20:09.900 Did you try to take what was there and like patch it up? 00:20:09.900 --> 00:20:15.840 Or are you like, I'm just going to recreate this from scratch and style it up from scratch? 00:20:15.840 --> 00:20:17.120 What was that process like? 00:20:17.120 --> 00:20:22.800 Before I answer that question, I actually have something to add on the infrastructure question that you asked. 00:20:22.800 --> 00:20:31.200 One of the things that I really appreciate about the project is how easy it is to set up as a contributing developer. 00:20:31.200 --> 00:20:40.160 So I am not the most technical contributor, but I found the project really, really easy to set up with Docker and Docker Compose. 00:20:40.660 --> 00:20:48.380 So the infrastructure that the team has set up in terms of being able to hack on this project is really, really amazing. 00:20:48.380 --> 00:20:52.800 And it really lowers the barrier to entry for a lot of people. 00:20:52.800 --> 00:20:59.040 We've seen people who've made their first open source pull request on this project. 00:20:59.280 --> 00:21:00.000 That's really great. 00:21:00.000 --> 00:21:00.320 Yeah. 00:21:00.320 --> 00:21:04.320 It's really accessible for people to actually come and contribute to the project. 00:21:04.320 --> 00:21:06.840 So I don't want to undersell that aspect. 00:21:06.840 --> 00:21:08.600 I think that's really important. 00:21:08.600 --> 00:21:09.900 I agree that it is. 00:21:09.900 --> 00:21:14.640 And I think that's one of the real powers of this whole Docker thing is, right? 00:21:14.700 --> 00:21:16.420 Like, it kind of comes all together. 00:21:16.420 --> 00:21:23.900 But Docker on its own brings almost equally many difficulties or challenges at the same time. 00:21:23.900 --> 00:21:29.240 And this, like, bringing in Kubernetes kind of, like, to make all the pieces fit, I think, is really, really clever. 00:21:29.240 --> 00:21:30.120 So quite nice. 00:21:30.120 --> 00:21:35.040 This portion of Talk Python To Me is brought to you by ActiveState. 00:21:35.040 --> 00:21:41.020 ActiveState gives you a faster way to build and secure open source runtimes from your first line of code through to production. 00:21:41.420 --> 00:21:47.760 Every second you spend building your Python distro or trying to secure your Python programs is less time spent doing the work you love. 00:21:47.760 --> 00:21:54.000 You've got better things to do than trying to resolve dependencies or making sure that you tick off all security boxes when you ship to production. 00:21:54.000 --> 00:22:00.360 Standardize on your Python builds so you can have less friction in the development cycle and you can deliver apps faster. 00:22:00.360 --> 00:22:04.840 You can also get a unique server-side way to verify your Python applications at runtime. 00:22:04.840 --> 00:22:08.280 Bake security right into your code without impacting performance. 00:22:08.880 --> 00:22:13.660 Go faster, spend more time doing the work you love, and comply with your enterprise security needs. 00:22:13.660 --> 00:22:19.300 Try ActiveState and see why it was chosen by IBM, Microsoft, NSA, Siemens, PepsiCo, and more. 00:22:19.300 --> 00:22:23.380 Join millions of developers who trust ActiveState to build their open source language distros. 00:22:23.380 --> 00:22:27.540 Visit talkpython.fm/ActiveState for a special offer. 00:22:27.540 --> 00:22:29.960 That's talkpython.fm/ActiveState. 00:22:31.440 --> 00:22:32.660 On your other question. 00:22:32.660 --> 00:22:49.800 So in terms of the redesign, basically Donald just gave me free reign to do whatever I needed to do because I hadn't – like to give you an impression of the old code base, I wasn't even – you know, Donald basically said don't even go and touch that. 00:22:49.800 --> 00:22:51.620 Like don't look at anything over there. 00:22:51.820 --> 00:22:52.820 Don't set it up. 00:22:52.820 --> 00:22:53.120 Don't set it up. 00:22:53.120 --> 00:22:59.580 Just avoid at all costs because he knew that that would be a world of pain for me. 00:22:59.580 --> 00:23:04.100 So I didn't really take any of the HTML or the CSS or the design from that. 00:23:04.100 --> 00:23:06.920 It was just like, okay, so we've got this fresh new thing. 00:23:06.920 --> 00:23:09.120 We want to show that it's a fresh new thing. 00:23:09.340 --> 00:23:14.220 And we want to bring it to the standards, modern design standards that people expect. 00:23:14.220 --> 00:23:20.600 We want it to be responsive so it works across all devices and we want it to be accessible. 00:23:20.600 --> 00:23:24.460 So I basically started from a completely clean slate. 00:23:24.460 --> 00:23:25.660 That's not true. 00:23:25.660 --> 00:23:30.080 Donald had put together some templates, but he was basically like throw that in the bin and start again. 00:23:30.080 --> 00:23:31.420 So that's what I did. 00:23:31.420 --> 00:23:32.200 That's really cool. 00:23:32.200 --> 00:23:35.180 So what are some of the technologies in the new one? 00:23:35.180 --> 00:23:39.100 It looks to me like it's probably bootstrap based, which I'm a fan of, so that's cool. 00:23:39.240 --> 00:23:39.900 And what else? 00:23:39.900 --> 00:23:41.260 No, it's not bootstrapped. 00:23:41.260 --> 00:23:42.280 No, it's not bootstrapped? 00:23:42.280 --> 00:23:42.600 No. 00:23:42.600 --> 00:23:43.140 Okay. 00:23:43.140 --> 00:23:44.200 What's involved there? 00:23:44.200 --> 00:23:44.580 Okay. 00:23:44.580 --> 00:23:49.500 So we're going to go into a bit of CSS and HTML naming methodology. 00:23:49.500 --> 00:23:58.000 So it's the HTML users BIM, which is a naming methodology for controlling the specificity of your CSS. 00:23:58.000 --> 00:24:06.880 And then basically each of the areas of the front end is a separate block or component within our SCSS code base. 00:24:07.060 --> 00:24:12.360 So basically the idea is we've built up a custom reusable CSS code base. 00:24:12.360 --> 00:24:13.600 Yeah, that's really nice. 00:24:13.600 --> 00:24:22.320 And you're using SAS, you said, or SASS, which is like programmable CSS that then compiles or transpiles to CSS, which is really nice. 00:24:22.320 --> 00:24:28.220 So it sounds like if people want to contribute to the UI side of things, it's pretty modern and fresh if they want to drop in. 00:24:28.340 --> 00:24:30.460 It is, and it's documented as well. 00:24:30.460 --> 00:24:32.980 So it's fairly clear how that system works. 00:24:32.980 --> 00:24:41.140 If you want to change variables, if you want to change what are called mixins, which are kind of like functions, reusable functions in SCSS. 00:24:41.540 --> 00:24:54.180 And if you want to modify a certain part of the code base, it's really obvious when you look, when you inspect the HTML, it's really obvious where the corresponding CSS is for that within the code base. 00:24:54.260 --> 00:24:58.420 So it's quite logical in terms of the way that the file structure is being set up. 00:24:58.420 --> 00:25:00.180 And I don't take credit for that. 00:25:00.180 --> 00:25:08.360 So it uses a system from a CSS guru called Nicholas Gallagher, who's, I mean, if anyone's into CSS, that's someone you should be following. 00:25:08.600 --> 00:25:11.620 So it uses the IT CSS system from here. 00:25:11.620 --> 00:25:12.120 That's cool. 00:25:12.120 --> 00:25:22.800 I feel like CSS and a lot of the web design stuff kind of gets the short end of the stick, but it could either be a serious drag to work on or it can be really beautiful depending on how you do it, right? 00:25:22.800 --> 00:25:27.360 The challenge with CSS is kind of achieving something at scale. 00:25:27.360 --> 00:25:33.940 Like I think most people can, you know, write a decent CSS code base for small projects. 00:25:33.940 --> 00:25:42.860 But when you start to scale projects, that's when you kind of have all this complexity with the cascade, things starting to break where you don't expect them to break. 00:25:42.860 --> 00:25:50.620 So that's why from the beginning, I introduced these kinds of systems that I knew would allow us to scale the code base as we add new features. 00:25:50.620 --> 00:25:57.680 I feel, I can't remember who on my show said it before, but somebody said they feel like CSS and large projects becomes a write only. 00:25:57.680 --> 00:25:59.460 Like you don't actually change anything. 00:25:59.460 --> 00:26:03.360 You only go to the bottom and maybe overwrite it or add another file that replaces it. 00:26:03.560 --> 00:26:04.900 You know, like adds to it. 00:26:04.900 --> 00:26:05.680 Pretty interesting. 00:26:05.680 --> 00:26:13.080 So let's talk about the actual rollout because actually before we talk about the rollout, let's talk about the traffic. 00:26:13.080 --> 00:26:21.700 I don't know, maybe Ernest, this is most clear on your mind, but this site and this underlying infrastructure, it handles a little bit of data, right? 00:26:21.700 --> 00:26:23.640 In total, PyPI does. 00:26:23.640 --> 00:26:26.760 The numbers are not directly in front of me. 00:26:26.760 --> 00:26:27.620 Why did I do that? 00:26:27.620 --> 00:26:30.740 But I have a slide deck somewhere that has this information. 00:26:30.740 --> 00:26:36.500 But it's many, it's like 30 or 50 terabytes a month, like something to that size, I think. 00:26:36.500 --> 00:26:38.460 It's a tremendous amount. 00:26:38.460 --> 00:26:43.160 I think it's like 10 billion requests per month is our running average right now. 00:26:43.160 --> 00:26:45.020 Let's go look at numbers. 00:26:45.020 --> 00:26:58.440 So if we go and look at the old service to get it in the last month, so that excludes two days, we did a total of 6.5 billion requests at the edge. 00:26:58.800 --> 00:27:06.020 6.8 billion requests per month and 1.5 petabytes of data at the edge. 00:27:06.020 --> 00:27:06.400 Petabytes. 00:27:06.400 --> 00:27:08.280 Holy moly. 00:27:08.280 --> 00:27:18.120 And so we're also doing that at around 150 milliseconds of latency and with not that many errors, all things considered. 00:27:18.400 --> 00:27:27.260 It's always important when we talk about these huge numbers to take one step back and go, yes, that is what the service is a total and total does. 00:27:27.260 --> 00:27:31.240 But it's all thanks to Fastly, which is the CDN provider. 00:27:31.240 --> 00:27:31.640 Right. 00:27:31.640 --> 00:27:32.820 Because of the CDN. 00:27:32.820 --> 00:27:33.100 Yeah. 00:27:33.100 --> 00:27:38.360 Which is the CDN provider that offered to front PyPI many years ago. 00:27:38.360 --> 00:27:45.700 And so just that one change was the most significant thing that happened to PyPI until, in my opinion, Monday. 00:27:46.340 --> 00:27:54.500 But at the back end, we still do something like 25 to 30 requests per second across a myriad of different routes. 00:27:54.500 --> 00:27:55.640 Yeah, that's really cool. 00:27:55.640 --> 00:27:57.360 And Pyramid is working out pretty well for you. 00:27:57.360 --> 00:28:03.620 Like my entire site, my course site, my podcast site, and various other pieces of infrastructure are almost all Pyramid. 00:28:03.620 --> 00:28:04.420 There's a little flask in there. 00:28:04.420 --> 00:28:06.380 And I think it's just been rock solid. 00:28:06.380 --> 00:28:07.960 So I've enjoyed it a lot. 00:28:07.960 --> 00:28:10.000 But how is it working for you? 00:28:10.000 --> 00:28:11.140 Yeah, I've had no complaints. 00:28:11.140 --> 00:28:14.540 I mean, I didn't really use Pyramid before I started contributing to the project. 00:28:14.720 --> 00:28:21.360 And now it's definitely my preferred framework for more intensive web applications in Python. 00:28:21.360 --> 00:28:22.840 So I like it a lot. 00:28:22.840 --> 00:28:24.120 Yeah, it broke my brain. 00:28:24.120 --> 00:28:28.440 I mean, like I got to the point where now I'm like, oh, of course, like this is how this works. 00:28:28.440 --> 00:28:32.540 And I go back and I work on some of the flask. 00:28:32.540 --> 00:28:36.300 I was like, oh, I can't do that here. 00:28:36.500 --> 00:28:49.280 And so overall, I think I agree with what Dustin sort of alluded to earlier around the control and precision that you can get from Pyramid that other frameworks sort of make you run around to do. 00:28:49.520 --> 00:28:49.860 Yeah, nice. 00:28:49.860 --> 00:28:51.600 So the rollout. 00:28:51.600 --> 00:28:54.840 So I set the stage with how much data you guys do, how much traffic you do. 00:28:54.840 --> 00:28:58.260 When you flip the switch on that, that's got to be a... 00:28:58.260 --> 00:29:00.600 So did you just go, it all goes here? 00:29:00.600 --> 00:29:06.920 Or did you like do some sort of like, let's take 1% of 1% of the traffic and like slowly roll it over? 00:29:06.920 --> 00:29:07.880 Like, what was it like? 00:29:07.880 --> 00:29:12.860 The main traffic sources for PyPI are pip installs, XML, RPC. 00:29:12.860 --> 00:29:19.280 So we have an XML RPC API and that gets a lot of traffic because it's mostly post requests and it's hard to cache that. 00:29:19.280 --> 00:29:22.360 And then, you know, a very small fraction of that is actual web traffic. 00:29:22.360 --> 00:29:32.520 So, you know, pypi.org existed for a long time before the launch and you could go and do everything on, you know, via the web interface that you could do on regular legacy PyPI. 00:29:32.520 --> 00:29:35.920 So that was, you know, didn't require a lot of traffic and worked fine. 00:29:36.100 --> 00:29:47.100 And so what we did was sort of some incremental load testing where we would switch certain either some pip traffic or XML RPC traffic over to pypi.org and see how it stood up. 00:29:47.100 --> 00:29:47.740 Yeah. 00:29:47.740 --> 00:29:51.840 So once again, Fastly was sort of predominant in that effort. 00:29:51.840 --> 00:29:57.600 So because we were doing those redirects at the edge, we were able to set rules there. 00:29:57.600 --> 00:30:03.460 And so like right now, actually, there's still quite a bit of traffic going to the old pipe or to the legacy PyPI backend. 00:30:03.960 --> 00:30:08.480 And we can do that because we're not redirecting the traffic over to PyPI. 00:30:08.480 --> 00:30:18.260 So we were able to like tune it at like 5, 10, 15, 20% for the heavy hitting stuff and test ahead of time. 00:30:18.260 --> 00:30:23.840 And so when we switched, basically all we did is we started issuing redirects from the old service. 00:30:23.840 --> 00:30:33.120 And so it was a one time click, but there were like weeks and weeks of like incremental quick load tests for where we would throw a bunch of traffic at it. 00:30:33.440 --> 00:30:35.880 There was some replaying we did ahead of time as well. 00:30:35.880 --> 00:30:36.220 Yeah. 00:30:36.220 --> 00:30:37.140 Oh, replaying. 00:30:37.140 --> 00:30:37.760 That's pretty cool. 00:30:37.760 --> 00:30:44.320 That's a basically capture the exact web traffic and you replay it against the domain and just see how it behaves. 00:30:44.320 --> 00:30:44.600 Right. 00:30:44.600 --> 00:30:45.980 It wasn't the exact traffic. 00:30:45.980 --> 00:30:53.280 We were taking measured, basically percentage stuff and then redispatching a request that looks like it. 00:30:53.360 --> 00:31:01.220 And because the problem is we can't just do every request blindly or people would like dual submit up, you know, dual submit an action or something. 00:31:01.220 --> 00:31:01.840 That's true. 00:31:01.840 --> 00:31:02.180 Right. 00:31:02.180 --> 00:31:06.060 You got to have non-modifying type of stuff or test data or something, I guess. 00:31:06.060 --> 00:31:06.260 Yeah. 00:31:07.340 --> 00:31:07.740 Yeah. 00:31:07.740 --> 00:31:08.180 Pretty cool. 00:31:08.180 --> 00:31:09.120 So how did it go? 00:31:09.120 --> 00:31:09.620 Perfectly. 00:31:09.620 --> 00:31:10.500 Not a thing went wrong. 00:31:10.500 --> 00:31:13.860 It was good for the first 15 minutes. 00:31:13.860 --> 00:31:15.440 I think we were all really excited. 00:31:15.440 --> 00:31:16.120 It's working. 00:31:16.120 --> 00:31:17.120 Wait a minute. 00:31:17.120 --> 00:31:17.960 It's working. 00:31:17.960 --> 00:31:20.940 Then the issues started rolling in. 00:31:20.940 --> 00:31:22.000 What do you guys run into? 00:31:22.200 --> 00:31:22.400 Sure. 00:31:22.400 --> 00:31:29.100 So previously, all files uploaded by users to PyPI. 00:31:29.100 --> 00:31:33.140 So these maintainers uploading their packages were hosted under the same domain. 00:31:33.140 --> 00:31:38.800 So packages were hosted at pypi.python.org slash packages, some stuff. 00:31:38.800 --> 00:31:45.500 During this switch, we decided to make a separate service, a separate domain for hosting user content. 00:31:45.500 --> 00:31:52.080 If you've ever seen the documentation that used to be hosted at or is still hosted, I'm sorry, pythonhosted.org. 00:31:52.080 --> 00:32:01.280 The main reason for that is that serving user-generated content from the same domain that you're actually operating a service from can be dangerous from some security perspectives. 00:32:01.280 --> 00:32:16.360 So the thing that went wrong is that when we switched over, there were redirect loops and all sorts of craziness happening for people trying to download files from files.pythonhosted.org, our new host. 00:32:16.860 --> 00:32:22.680 Ultimately, it was a bewildering and sort of bizarre thing because we had a number of factors at play. 00:32:22.680 --> 00:32:25.160 We had files that were cached were fine. 00:32:25.160 --> 00:32:28.540 Files that weren't cached were going to end up in this redirect loop. 00:32:28.540 --> 00:32:31.260 We had some host names involved. 00:32:32.080 --> 00:32:37.100 And overall, it was just, and it would happen, we realized that it's sort of the worst possible time. 00:32:37.100 --> 00:32:46.920 So if you go to status.python.org and scroll down a little bit, you can read an incident report that sort of describes in more detail what went wrong. 00:32:47.440 --> 00:32:52.300 But effectively, we were making this change as part of the rollout. 00:32:52.980 --> 00:33:04.340 And an esoteric thing that occurs, I guess, occasionally when you try to move a host name from one CDN configuration to another CDN configuration. 00:33:04.340 --> 00:33:06.080 We mishandled that. 00:33:06.080 --> 00:33:07.640 And so it was a one line. 00:33:07.640 --> 00:33:12.540 The fix was one line, and it was like 13 characters. 00:33:12.540 --> 00:33:14.600 But it resolved it. 00:33:14.600 --> 00:33:17.680 And so, yeah, not everything can go perfect. 00:33:17.680 --> 00:33:21.860 Well, sometimes the best, most memorable lessons are taught in production. 00:33:21.860 --> 00:33:35.500 What we talked about before we started the official recording that everyone was listening to is your overall, as a group, your overall thought was this was a big success, even if there was like a blip here or there. 00:33:35.500 --> 00:33:36.380 Yes, absolutely. 00:33:36.380 --> 00:33:43.320 Aside from like that, you know, files outage, which is kind of the core use of VPI. 00:33:43.320 --> 00:33:44.560 So that's kind of a big deal. 00:33:44.560 --> 00:33:48.540 But aside from that, you know, everything else worked great and continues to work great. 00:33:48.540 --> 00:33:50.500 So we're generally pleased. 00:33:50.500 --> 00:33:53.360 Like 99% of things worked perfectly. 00:33:53.360 --> 00:33:54.680 Yeah, that's really great. 00:33:54.680 --> 00:34:01.040 So I think this is one of those things, like I'm sure people were concerned about switching, like what might go wrong. 00:34:01.200 --> 00:34:17.780 Like, will we break like Netflix deployment because they can't get a pip install to work on some like Docker container in like a continuous build because something like, you know, these types of I may be affecting this, but you sort of had to go through that to be on the better side of the world. 00:34:17.780 --> 00:34:18.040 Right. 00:34:18.100 --> 00:34:22.580 So now you Nicole's designs up the pyramid app that you all built is up. 00:34:22.580 --> 00:34:25.420 And now it's it's just there to be polished and built upon. 00:34:25.420 --> 00:34:25.680 Right. 00:34:25.680 --> 00:34:26.280 Yeah. 00:34:26.280 --> 00:34:28.440 I think that we're our hands are. 00:34:29.020 --> 00:34:39.620 Well, once legacy is shut down, our hands are untied and we can make, you know, we can make progress in places that we would that we sort of wouldn't wouldn't have been able to in the future. 00:34:40.040 --> 00:34:50.380 Something that I like to point out about PyPI, the historical PyPI is that there was a point where it was pretty much the only non static web host that Python.org had. 00:34:50.380 --> 00:34:58.860 And so it would end up getting a bunch of features thrown into it that weren't necessarily critical to its operation. 00:34:59.100 --> 00:35:14.240 And so as we split into warehouse features were removed from PyPI legacy and sort of while they were both simultaneously in existence, we had to be very strategic about what things we added to warehouse or PyPI.org. 00:35:14.240 --> 00:35:26.020 And so once legacy is shut down, we can start to make much more progress and do so much more quickly and much more safely than we ever have been been able to before. 00:35:26.020 --> 00:35:36.500 And so that alone is probably the biggest long term benefit of this is being able to do the things that people need, whether it's design or functionality. 00:35:36.500 --> 00:35:49.740 I think if if you have to remain like with on parity with this older system that totally you're not designing one thing, you're designing almost like two things or you're constrained really painfully. 00:35:49.740 --> 00:35:51.220 So you'll be free. 00:35:51.220 --> 00:35:52.460 They share a database. 00:35:52.460 --> 00:35:55.320 So that also is a huge complicating factor. 00:35:55.320 --> 00:35:56.980 Very interesting. 00:35:56.980 --> 00:35:59.580 I guess a couple questions just really quick on that. 00:35:59.580 --> 00:36:01.880 And then I want to kind of talk about where things are going. 00:36:01.880 --> 00:36:03.280 You said they share a database. 00:36:03.280 --> 00:36:04.700 Like what database is that? 00:36:04.700 --> 00:36:10.340 Where is the actual web apps running your Kubernetes containers running these days? 00:36:10.340 --> 00:36:17.300 We use Postgres for a database and we have a very generous donation for in-kind service, basically. 00:36:17.520 --> 00:36:20.620 So AWS said, yeah, you can run PyPI here. 00:36:20.620 --> 00:36:25.020 And so right now we run we run the entire stack in Ohio. 00:36:25.020 --> 00:36:26.300 I picked where it deployed. 00:36:26.300 --> 00:36:27.160 So I picked Ohio. 00:36:27.160 --> 00:36:34.000 But in the Ohio region for AWS, we've got like I think it's like nine medium-ish sized servers running Kubernetes. 00:36:34.000 --> 00:36:39.640 And we're using RDS and Elastic Cache for Postgres and Redis and such. 00:36:39.780 --> 00:36:40.240 That's cool. 00:36:40.240 --> 00:36:45.200 And Dustin, I heard you talk about Elasticsearch, right? 00:36:45.200 --> 00:36:46.420 Is that that's involved as well? 00:36:46.420 --> 00:36:47.980 They're another sponsor in kind. 00:36:47.980 --> 00:36:55.680 And that's for the search on PyPI.org is far, far better than it was on Legacy, which is basically a super naive search. 00:36:55.780 --> 00:37:04.720 So now we can do full text search across descriptions and summaries and package names and even author maintainer names. 00:37:04.720 --> 00:37:10.240 And it's a little more performant than the previous search and a little more reliable and better results. 00:37:10.240 --> 00:37:11.440 Yeah, perfect, perfect. 00:37:11.440 --> 00:37:12.540 All right. 00:37:12.540 --> 00:37:16.620 So let's talk about where things are going, I guess. 00:37:16.620 --> 00:37:22.880 So you have a roadmap laid out at wiki.python.org slash PSF slash warehouse roadmap. 00:37:22.880 --> 00:37:24.400 I'll put that in the show notes, of course. 00:37:25.400 --> 00:37:28.960 So the very first thing, you have a bunch of stuff, which is pretty awesome. 00:37:28.960 --> 00:37:30.660 It's like, here's a milestone, closed. 00:37:30.660 --> 00:37:32.520 Here's a milestone, closed, completed, right? 00:37:32.520 --> 00:37:33.640 These are great. 00:37:33.640 --> 00:37:38.980 And then the current one that's like coming in progress is shut down Legacy PyPI. 00:37:38.980 --> 00:37:40.480 You all want to talk about that? 00:37:40.480 --> 00:37:42.000 That's coming on the 30th, right? 00:37:42.000 --> 00:37:44.360 Like that is, we're recording right now on April 18th. 00:37:44.360 --> 00:37:45.800 So 22 days. 00:37:45.800 --> 00:37:46.940 Yeah, go Dustin. 00:37:46.940 --> 00:37:55.000 We sort of kept Legacy up for now just because there are a few big users of PyPI that weren't able to sort of make the migration in time. 00:37:55.020 --> 00:37:57.880 So the idea is to keep it up for just a little bit longer. 00:37:57.880 --> 00:38:01.120 And then fully, the domain will continue to exist. 00:38:01.120 --> 00:38:04.620 So pypi.python.org will redirect to pypi.org. 00:38:04.620 --> 00:38:07.620 But the legacy service will cease to exist. 00:38:07.620 --> 00:38:14.460 That's the big change you were talking about, Ernest, where like you'll kind of be free to build this thing as its own creation, right? 00:38:14.460 --> 00:38:15.820 Not mirroring that. 00:38:15.980 --> 00:38:16.640 Yeah, it's interesting. 00:38:16.640 --> 00:38:23.500 I think the first thing that Warehouse ever did that was production impacting was take control of the database schema. 00:38:23.500 --> 00:38:26.460 And that was years ago. 00:38:26.460 --> 00:38:28.880 And so we started tracking database changes there. 00:38:29.100 --> 00:38:31.120 And then uploads came. 00:38:31.120 --> 00:38:35.880 And then the actual web app came up and was usable and such. 00:38:35.880 --> 00:38:38.460 And we added features there to get to parity. 00:38:38.460 --> 00:38:44.360 And so everything that the project was sort of undertaken up to this point, except for, you know, markdown descriptions. 00:38:45.020 --> 00:38:46.300 I think that's it. 00:38:46.300 --> 00:38:47.960 And the design, yeah. 00:38:47.960 --> 00:38:57.480 And, of course, the refresh design has been just to make sure that we're doing everything we can to keep from breaking too many people. 00:38:57.480 --> 00:38:59.700 It's impossible for us not to. 00:39:00.180 --> 00:39:07.080 I mean, it's impossible for any service to make progress without, at some point, deprecating older APIs and such. 00:39:07.080 --> 00:39:10.920 And so we're really getting to the point where we've pared down a lot of things. 00:39:10.920 --> 00:39:18.720 And we can start looking forward to, you know, value-add features, if you will, where it's like security features, audit features. 00:39:18.720 --> 00:39:23.020 Accessibility is a big thing that, you know, we're looking forward to as well. 00:39:23.020 --> 00:39:23.860 Yeah, very cool. 00:39:23.860 --> 00:39:25.720 So that comes on the 30th. 00:39:25.720 --> 00:39:30.600 And it'll be officially, the chains will be broken and warehouse will be its own thing. 00:39:30.600 --> 00:39:31.240 And that'll be great. 00:39:31.240 --> 00:39:35.940 This portion of Talk Python is brought to you by Codacy. 00:39:35.940 --> 00:39:41.000 If you want to improve code quality, prevent bugs and security issues from making it into production, 00:39:41.000 --> 00:39:46.760 and at the same time speed up your code review process by 20%, then you need to try Codacy. 00:39:46.760 --> 00:39:49.240 That's C-O-D-A-C-Y. 00:39:49.580 --> 00:39:56.900 Codacy makes it easy to track code quality and identify and fix issues by automatically analyzing your commits and pull requests 00:39:56.900 --> 00:40:00.020 with all the most widely used static analysis tools. 00:40:00.020 --> 00:40:03.420 Codacy helps great teams build great software. 00:40:03.420 --> 00:40:07.320 Join companies like DeliverHero, PayPal, Samsung, and more. 00:40:07.320 --> 00:40:13.980 Try your first code review by visiting talkpython.fm/Codacy and linking your GitHub or Bitbucket account. 00:40:13.980 --> 00:40:16.680 You can also just click on the Codacy link in the show notes. 00:40:18.720 --> 00:40:24.600 So then you have, under your roadmap, you have another section called Post-Lugacy shutdown. 00:40:24.600 --> 00:40:29.900 And then kind of beyond that, you have Cool but Not Urgent, which is a nice way to categorize it. 00:40:29.900 --> 00:40:31.900 So maybe we could kind of touch on those. 00:40:31.900 --> 00:40:35.500 And whoever feels most like it's in their space, just grab it. 00:40:35.500 --> 00:40:40.380 So like Dustin, there's something called incremental searching, search indexing rather, coming. 00:40:40.380 --> 00:40:41.940 Tell us about that. 00:40:43.240 --> 00:40:52.700 Yeah, so right now, the way the search index works, you upload a package, our index runs, I think every, now it's every three hours, roughly, when it actually runs. 00:40:52.700 --> 00:40:54.860 So there's a lot of packages index. 00:40:55.140 --> 00:40:59.820 And we don't have, at the moment, a way to sort of incrementally update the index. 00:40:59.820 --> 00:41:03.320 So as soon as you publish a package, you know, it shows up in search results. 00:41:03.320 --> 00:41:03.680 I see. 00:41:03.680 --> 00:41:07.220 So you could say, like, this part is super stale because I know it just got updated. 00:41:07.220 --> 00:41:10.660 So rerun the search, but only on this package, for example. 00:41:10.660 --> 00:41:13.380 The goal was, you know, we got search up and running on PyPI. 00:41:13.380 --> 00:41:15.840 And it was still a lot better than Legacy. 00:41:15.840 --> 00:41:17.240 So it was good enough for launch. 00:41:17.240 --> 00:41:18.860 But, you know, it can be better. 00:41:18.860 --> 00:41:20.780 So that's one of the things we're focused on adding. 00:41:21.020 --> 00:41:21.100 Sure. 00:41:21.100 --> 00:41:26.420 And while you're on it, like, there's the autocomplete for search, which will be pretty nice. 00:41:26.420 --> 00:41:28.560 There's also a search API. 00:41:28.560 --> 00:41:30.540 That's pretty cool. 00:41:30.540 --> 00:41:31.740 Like, does that exist? 00:41:31.740 --> 00:41:33.920 Is there a way to search now and in the future? 00:41:33.920 --> 00:41:35.180 Like, this is going to be a thing? 00:41:35.180 --> 00:41:35.760 What's the story? 00:41:35.760 --> 00:41:39.940 Technically, we have the XML RPC API that is technically deprecated. 00:41:39.940 --> 00:41:43.840 You probably shouldn't be depending on it or using it or adding new things that depend on it. 00:41:43.840 --> 00:41:46.780 It does have the word XML RPC in it, right? 00:41:46.780 --> 00:41:49.240 That should be an indicator that it's deprecated, but no. 00:41:49.600 --> 00:41:51.480 But you can technically search from this API. 00:41:51.480 --> 00:41:56.080 And this is how, like, if you type pip search, whatever, that's how you get results through there. 00:41:56.080 --> 00:42:00.780 But XML RPC, like I think I said before, is really hard for us to cache. 00:42:00.780 --> 00:42:04.100 It's a big consumer of our bandwidth and backend resources. 00:42:04.100 --> 00:42:08.140 So the idea is to sort of move to something that is a little more cacheable. 00:42:08.140 --> 00:42:12.880 So this would be, we have a lot of ideas about future APIs for PyPI.org. 00:42:12.880 --> 00:42:17.020 And, you know, something that might be included in that is a search API. 00:42:17.460 --> 00:42:23.100 The other one that's interesting to me is the Psycho PG 2 warning. 00:42:23.100 --> 00:42:26.260 So I guess that's just like you guys are using Postgres basically. 00:42:26.260 --> 00:42:30.520 Are you using the asynchronous stuff or just synchronously? 00:42:30.520 --> 00:42:33.420 No, warehouse is all synchronous right now. 00:42:33.420 --> 00:42:38.580 Are you thinking of any way to get something async in there or does it not matter? 00:42:38.840 --> 00:42:45.580 So a number of the services that are behind the entire sort of service, it's like the service umbrella, if you will, of what PyPI is. 00:42:45.580 --> 00:42:49.100 So PyPI, it has been broken up into hunks. 00:42:49.100 --> 00:42:54.960 And so for some things, it truly does matter the way these, you know, the way these requests are handled. 00:42:54.960 --> 00:43:06.020 A lot of the really incredible work that was done initially on warehouse by Donald was just how aggressively cached everything is. 00:43:06.020 --> 00:43:11.400 You know, the goal is basically to make as few requests require a transit to the backend as possible. 00:43:11.600 --> 00:43:16.120 So we don't have a ton of concurrency concerns around that. 00:43:16.120 --> 00:43:24.820 But for some services that do see lots of traffic, like we have a service that just translates old URLs to new URLs. 00:43:25.000 --> 00:43:28.160 And that is effectively proxying information. 00:43:28.160 --> 00:43:32.020 And so that's a knockout use case for async stuff. 00:43:32.020 --> 00:43:32.800 Yeah, pretty interesting. 00:43:32.800 --> 00:43:35.900 Let's see, what else is in your post-legacy shutdown here? 00:43:35.900 --> 00:43:37.620 We have stop having a staging environment. 00:43:37.620 --> 00:43:41.540 Is that because of the Kubernetes stuff that makes it not required? 00:43:41.540 --> 00:43:46.900 So that's talking about test PyPI, which would be at test.pypi.org now. 00:43:46.900 --> 00:43:54.720 That existed so that people can, or it currently exists so that people can do stuff and not worry about it being on the real PyPI. 00:43:54.840 --> 00:43:58.740 So you can practice uploading a package, see how it looks on PyPI. 00:43:58.740 --> 00:44:04.340 And I think there's a lot of reasons for it to exist, sort of just as an experimental and educational tool. 00:44:04.340 --> 00:44:10.240 But the main reason I think people use it is to see if their restructured text descriptions are going to break or not. 00:44:10.240 --> 00:44:14.880 Because historically, PyPI would just, it's sort of all or nothing. 00:44:14.880 --> 00:44:18.100 You either get a perfect description or it just looks like plain text. 00:44:18.100 --> 00:44:22.900 There's some ideas about doing some new things that might obviate the need for test PyPI, 00:44:23.100 --> 00:44:25.580 like the ability to stage your releases. 00:44:25.580 --> 00:44:27.860 So you're going to make a new release of your package. 00:44:27.860 --> 00:44:31.140 You can upload them all to PyPI, but they're not actually published yet. 00:44:31.140 --> 00:44:33.560 You can go and look at them, but no one can see them. 00:44:33.560 --> 00:44:35.800 And then you hit a button and they're released. 00:44:35.800 --> 00:44:36.720 Yeah, that's very cool. 00:44:36.860 --> 00:44:41.780 And a big reason why that's important is we have immutable releases on PyPI. 00:44:41.780 --> 00:44:48.040 And so right now, there's a lot of frustration that comes from users around they upload a package. 00:44:48.040 --> 00:44:49.720 They don't like what they saw. 00:44:49.720 --> 00:44:51.360 They try to delete it. 00:44:51.600 --> 00:44:55.220 And they get a warning that says, when you delete this, you won't be able to reupload it. 00:44:55.220 --> 00:44:58.220 And then they go to reupload it and they're frustrated. 00:44:58.220 --> 00:45:00.280 So then they try to delete the project. 00:45:00.280 --> 00:45:02.920 And then they go and they reupload it again. 00:45:02.920 --> 00:45:05.160 And it says, no, you still can't reupload that file. 00:45:05.340 --> 00:45:12.720 And so this is around primarily a caching and immutability thing to basically say that files can't be replaced. 00:45:12.720 --> 00:45:18.080 So if you've been installing a file from PyPI for however long, it will still be there. 00:45:18.080 --> 00:45:27.360 And so giving people a way to trial things without committing, basically, if you go to the permanence of the thing, is a big reason for that as well. 00:45:27.360 --> 00:45:34.700 And when you get billions of requests, one pip freeze can make it part of the history of the software, right? 00:45:34.740 --> 00:45:35.060 For sure. 00:45:35.060 --> 00:45:36.380 All right. 00:45:36.380 --> 00:45:37.740 So just really quick, some other things. 00:45:37.740 --> 00:45:42.440 You have GitHub sign-on coming along, renaming projects, a few other cool things. 00:45:42.440 --> 00:45:46.060 In the cool but not urgent, the one that stood out most to me was a mobile app. 00:45:46.060 --> 00:45:47.900 Like, what's the story with the mobile app there? 00:45:47.900 --> 00:45:51.700 Nicole, are you going to be designing a new mobile app for PyPI? 00:45:51.700 --> 00:45:52.940 I don't know whether or not. 00:45:52.940 --> 00:45:56.360 I mean, this has been a suggestion from the community. 00:45:56.360 --> 00:46:04.580 And I think we're still working out whether or not that is something that's justifiable in terms of our time and the resources that we have on the project. 00:46:04.640 --> 00:46:08.420 What exactly, like, do you guys know, like, what the goal of the mobile app? 00:46:08.420 --> 00:46:11.600 I mean, you're definitely not going to pip install, like, onto your mobile phone. 00:46:11.600 --> 00:46:12.860 Like, that wouldn't mean anything, right? 00:46:12.860 --> 00:46:14.960 Is it more about management and, like, seeing stats? 00:46:14.960 --> 00:46:22.400 I think it was more just about, like, can we offer this user interface as a mobile app as opposed to a responsive website? 00:46:22.400 --> 00:46:25.900 And for me, I'm not sure how much value that would bring. 00:46:25.900 --> 00:46:34.240 We probably have, like, I mean, I don't have the statistics in front of me, but it's less than 10% of our users are using a mobile or tablet device. 00:46:34.520 --> 00:46:38.820 So, and the site works on mobile now better than the old ones. 00:46:38.820 --> 00:46:41.860 So, I'm not sure whether or not we'll go down that road. 00:46:41.860 --> 00:46:55.480 What I think is most interesting about the mobile apps issue that's being tracked there is a prerequisite for that is effectively the next generation of an API for interacting with PyPI. 00:46:55.920 --> 00:47:01.340 That's one of the biggest things that is intended to be undertaken at the PyCon sprints this year. 00:47:01.340 --> 00:47:22.000 And so, now, putting my PyCon hat on and my warehouse hat on at the same time, I think it'd be an excellent idea for people who are interested in helping to contribute to the discussion and design of the next generation of APIs for PyPI to consider attending the sprints after PyCon this year. 00:47:22.140 --> 00:47:24.880 The sprints are Monday, Tuesday, Wednesday, Thursday after. 00:47:24.880 --> 00:47:29.320 And there will be a number of contributors to the project around. 00:47:29.320 --> 00:47:32.520 And that's one of the main things we plan on discussing. 00:47:32.520 --> 00:47:33.860 It sounds really good. 00:47:33.860 --> 00:47:34.180 Yeah. 00:47:34.180 --> 00:47:35.040 Very, very nice. 00:47:35.040 --> 00:47:37.020 A couple others. 00:47:37.020 --> 00:47:38.020 Let's see. 00:47:38.020 --> 00:47:41.220 The university, a simple one, package update feed. 00:47:41.220 --> 00:47:46.180 So, that's like I can subscribe to real-time changes to the back-end data. 00:47:46.180 --> 00:47:53.580 So, I know if I'm like, if I've pulled that down or something, I could refresh, say, like my local PyPI caching server type thing. 00:47:53.580 --> 00:47:53.980 Yeah. 00:47:53.980 --> 00:48:03.280 So, there's a lot of third-party services that depend on PyPI that kind of want real-time notifications about new package uploads or removals and that kind of thing. 00:48:03.280 --> 00:48:10.660 And so, this is just going to be a new API for, you know, like a tool like PyUp, which lets you like automatically upgrade your dependencies when they're released. 00:48:10.660 --> 00:48:10.980 Yeah. 00:48:11.040 --> 00:48:12.140 I use PyUp on my stuff. 00:48:12.140 --> 00:48:12.520 I love it. 00:48:12.520 --> 00:48:12.720 Yeah. 00:48:12.720 --> 00:48:13.080 It's great. 00:48:13.080 --> 00:48:17.320 So, we want to be able to support them, make it easier for them to do their job and use PyPI. 00:48:17.320 --> 00:48:19.460 So, that's one of the things we're thinking about. 00:48:19.460 --> 00:48:19.740 Yeah. 00:48:19.740 --> 00:48:22.640 You don't want to have to suck all that data down just to get a new batch. 00:48:22.640 --> 00:48:24.060 Kind of like your incremental search. 00:48:24.060 --> 00:48:25.900 This is like the external version of it, sort of. 00:48:25.900 --> 00:48:26.180 Yeah. 00:48:26.180 --> 00:48:26.540 Exactly. 00:48:26.540 --> 00:48:36.460 So, another one that's really closely related to it, like including related to PyUp.io, like you just mentioned, is security notification system for Python packages. 00:48:36.460 --> 00:48:38.560 That sounds really useful. 00:48:38.720 --> 00:48:48.080 We just had this year or is it in the last year, like some sort of test malicious stuff uploaded to PyPI, right? 00:48:48.080 --> 00:48:51.960 A couple of packages that were sort of hitting on typo squatting. 00:48:51.960 --> 00:48:54.780 Didn't really seem to do anything, but still kind of scary. 00:48:54.780 --> 00:49:03.460 So, knowing about security notifications, I guess not necessarily just people uploading malware, but like, hey, we actually forgot to check the password in this login field. 00:49:03.460 --> 00:49:06.860 You probably want to get the newer version that checks the password type of thing, right? 00:49:07.200 --> 00:49:14.820 On legacy, you could do this thing called hiding releases, which just made them not show up, but they basically still existed and it's not going to prevent you from using them. 00:49:14.820 --> 00:49:22.220 One of the things that we're thinking about doing with the new PyPI is either adding the ability to deprecate a release saying like, you should not use this anymore. 00:49:22.220 --> 00:49:26.100 It doesn't work or being able to market as insecure in some way. 00:49:26.240 --> 00:49:29.920 So, there's like a known vulnerability in it and you should upgrade to the new version. 00:49:29.920 --> 00:49:35.120 And, you know, this is something that's going to have to change in a lot of different parts of the packaging ecosystem. 00:49:35.120 --> 00:49:43.400 So, like, pip needs to be able to say, hey, you told me to install this version and PyPI says it's insecure and tell the user, give them a warning or whatever. 00:49:43.400 --> 00:49:44.560 But, yeah. 00:49:44.560 --> 00:49:44.820 Yeah. 00:49:44.820 --> 00:49:51.320 I mean, just related to that, I would love to be able to type pip security checkup on like an environment or something. 00:49:51.320 --> 00:49:53.440 And go, these two things have security warnings. 00:49:53.440 --> 00:49:57.080 These have updates, but they're feature only or something to that effect, right? 00:49:57.080 --> 00:49:57.640 That would be cool. 00:49:57.640 --> 00:49:57.960 Yeah. 00:49:57.960 --> 00:50:02.580 And I mean, to be clear, it doesn't happen very often that there are security vulnerabilities in Python packages. 00:50:02.580 --> 00:50:05.700 But it's something that could happen, might happen. 00:50:05.700 --> 00:50:06.900 We want to be able to support it. 00:50:06.900 --> 00:50:07.120 Yeah. 00:50:07.120 --> 00:50:10.980 For example, Django had one or two minor security issues patched, right? 00:50:10.980 --> 00:50:18.060 And you'd want to know if you were built upon Django, like, hey, you should probably install a new version before people start doing anything with that, right? 00:50:18.060 --> 00:50:18.480 Yeah. 00:50:18.480 --> 00:50:19.000 Very cool. 00:50:19.000 --> 00:50:20.520 So, just super quick. 00:50:20.520 --> 00:50:23.100 I'm about out of time, but just touch on one more thing. 00:50:23.100 --> 00:50:26.940 Like, this week, I think, pip 10 was released, right? 00:50:26.940 --> 00:50:27.520 That's correct. 00:50:27.520 --> 00:50:31.060 I don't know how much any of you all had to do with that, but still pretty good news, right? 00:50:31.060 --> 00:50:31.640 Yeah, it's great. 00:50:31.640 --> 00:50:32.500 It had been a long time. 00:50:32.740 --> 00:50:34.580 Since we had a pip release, so. 00:50:34.580 --> 00:50:36.480 Yeah, it's really exciting. 00:50:36.480 --> 00:50:43.120 I mean, the biggest thing is that it's a pretty foundational refactoring of a lot of the internal stuff. 00:50:43.120 --> 00:50:56.640 And it puts, in my opinion, anyway, one of the things I'm most excited about it is it puts a lot of the internal tooling of pip and makes it more available for more interesting things built around and on top of pip, not necessarily at a CLI basis. 00:50:56.640 --> 00:51:04.360 Because right now you've got to, like, if you want to use pip's stuff, you have to, you just have to jump into, like, super private APIs to do it, which isn't so great. 00:51:04.360 --> 00:51:05.100 That's really cool. 00:51:05.100 --> 00:51:09.140 Probably will make pairing it with this work that you're doing on the server side easier as well. 00:51:09.900 --> 00:51:10.200 All right. 00:51:10.240 --> 00:51:16.240 So I think I have other things I would love to talk to you about, but I think we're sort of running low on time. 00:51:16.240 --> 00:51:20.580 So let's get to just a couple of things here at the end. 00:51:20.980 --> 00:51:23.400 Final two questions, just quick, since they're straight of you. 00:51:23.400 --> 00:51:24.640 Nicole, start with you. 00:51:24.640 --> 00:51:29.480 If you're going to do some work on this project, what editor would you use? 00:51:29.480 --> 00:51:30.760 Like, what typical editor do you use? 00:51:30.760 --> 00:51:31.540 I use Atom. 00:51:31.540 --> 00:51:32.180 Okay. 00:51:32.180 --> 00:51:33.340 Very nice. 00:51:33.340 --> 00:51:34.960 I don't think that I know about that one. 00:51:34.960 --> 00:51:36.060 Tell me a little bit about it. 00:51:36.060 --> 00:51:37.300 You're talking about text editor? 00:51:37.300 --> 00:51:37.900 Yeah. 00:51:37.900 --> 00:51:38.440 Yeah. 00:51:38.440 --> 00:51:40.680 So it's Atom, which is developed by GitHub. 00:51:40.680 --> 00:51:41.420 Oh, Atom. 00:51:41.420 --> 00:51:41.800 Oh, yeah. 00:51:41.800 --> 00:51:42.240 Sorry, sorry. 00:51:42.240 --> 00:51:43.360 Yeah, I must have misheard you. 00:51:43.360 --> 00:51:44.380 Atom, of course, I know Atom. 00:51:44.380 --> 00:51:44.620 Yeah. 00:51:44.620 --> 00:51:45.860 Sorry, that's my accent. 00:51:45.860 --> 00:51:46.640 No, no. 00:51:46.640 --> 00:51:46.860 Yeah. 00:51:46.860 --> 00:51:48.580 Cool, Dustin. 00:51:48.580 --> 00:51:49.460 I'm a Vim user. 00:51:49.460 --> 00:51:50.380 Vim, right on. 00:51:51.100 --> 00:51:51.380 Ernest? 00:51:51.380 --> 00:51:52.580 I also am a Vim user. 00:51:52.580 --> 00:51:53.040 Nice. 00:51:53.040 --> 00:51:53.400 All right. 00:51:53.400 --> 00:51:57.940 Now, this particular question I ask of everybody, but it's kind of interesting because you're 00:51:57.940 --> 00:51:59.580 both on the inside and the outside. 00:51:59.580 --> 00:52:02.320 So, notable PyPI package. 00:52:02.320 --> 00:52:03.400 Ernest, how about you go first? 00:52:03.400 --> 00:52:06.040 Notable in what way? 00:52:06.040 --> 00:52:11.000 Notable in that, like, it's probably not necessarily the most popular thing. 00:52:11.000 --> 00:52:12.800 Like, people always say requests, which is fine. 00:52:12.800 --> 00:52:15.460 But, like, I learned about this thing. 00:52:15.460 --> 00:52:17.860 You should totally check it out, sort of notable. 00:52:18.240 --> 00:52:21.320 Like, it's not necessarily totally known, but it's actually amazing. 00:52:21.320 --> 00:52:23.240 And it's just a pip install away. 00:52:23.240 --> 00:52:28.400 Recently, with the type of squatting thing we sort of talked about, I was on the hunt for 00:52:28.400 --> 00:52:31.820 something that would just tell me all of the standard limb module names. 00:52:31.820 --> 00:52:33.260 And that exists. 00:52:33.260 --> 00:52:38.280 And go figure, it is called, I think it's called standard limb module names. 00:52:38.280 --> 00:52:41.120 Descriptive names are good. 00:52:41.740 --> 00:52:42.060 Yeah. 00:52:42.060 --> 00:52:48.780 And so, we were able to add that to PyPI and very quickly be able to have a good block 00:52:48.780 --> 00:52:50.120 of that first line of defense. 00:52:50.120 --> 00:52:54.180 Somebody didn't try to pip install regex or something. 00:52:54.180 --> 00:52:55.120 Right, right, right. 00:52:55.120 --> 00:52:55.360 Yeah. 00:52:55.360 --> 00:52:56.080 Pip install re. 00:52:56.080 --> 00:52:57.340 No, not doing that. 00:52:57.340 --> 00:52:58.540 Dustin? 00:52:58.540 --> 00:53:02.480 In the course of this project, I had this sort of favorite Python package I'd learned about, 00:53:02.580 --> 00:53:08.240 which is pretend, which we use pretty heavily on Warehouse for sort of mocking things out 00:53:08.240 --> 00:53:08.640 in tests. 00:53:08.640 --> 00:53:12.760 So, the new PyPI has like 100% test coverage. 00:53:12.760 --> 00:53:14.660 So, there's a lot of mocking going on. 00:53:14.660 --> 00:53:18.300 And so, that's, I think, Alex Gaynor's tool. 00:53:18.300 --> 00:53:19.300 And it's been really helpful. 00:53:19.300 --> 00:53:25.080 I think my, as of lately, my favorite package is not actually on PyPI, but I just discovered 00:53:25.080 --> 00:53:25.720 it the other day. 00:53:25.860 --> 00:53:28.980 I'm kind of a sucker for like funny little hacks or jokes. 00:53:28.980 --> 00:53:34.400 And so, this guy, Dominic Medzinski, he made this project called Import PyPI. 00:53:34.400 --> 00:53:36.000 And it's really interesting. 00:53:36.000 --> 00:53:38.740 What it does is it sort of wraps the import command. 00:53:38.740 --> 00:53:44.860 And if you don't have a given package on your system, it will go out to PyPI, get it, and 00:53:44.860 --> 00:53:46.160 install it, and it will just work. 00:53:46.160 --> 00:53:48.760 So, you never actually have to pip install anything again. 00:53:48.760 --> 00:53:50.660 I ran across that as well, and that's pretty interesting. 00:53:50.660 --> 00:53:52.320 It's quite ironic it's not on PyPI. 00:53:52.320 --> 00:53:54.100 But, yeah. 00:53:54.100 --> 00:53:55.640 Does it do that on the fly? 00:53:56.380 --> 00:53:57.240 Yeah, it does. 00:53:57.240 --> 00:53:57.540 I think it does. 00:53:57.540 --> 00:53:58.100 If it doesn't find it. 00:53:58.100 --> 00:54:03.020 I don't think it's really recommended for production grade usage, but it's a fun little hack. 00:54:03.020 --> 00:54:05.040 It is quite interesting for what it's worth. 00:54:05.040 --> 00:54:08.460 Hope it puts a --user on it, at least. 00:54:08.460 --> 00:54:10.280 All right, Nicole, do you have one? 00:54:10.280 --> 00:54:11.440 Oh, yeah, I do. 00:54:11.440 --> 00:54:13.520 As I said, I only dabble in Python. 00:54:13.520 --> 00:54:20.920 But when I was dabbling, I got really into testing, and I really liked Factory Boy, which 00:54:20.920 --> 00:54:23.520 creates factories. 00:54:24.020 --> 00:54:29.800 So, I use that a lot for running Selenium tests running over my Django code base when 00:54:29.800 --> 00:54:31.020 I was developing with Python. 00:54:31.020 --> 00:54:32.040 It was a really cool project. 00:54:32.040 --> 00:54:36.040 I think it's actually based off a Ruby project originally. 00:54:36.040 --> 00:54:36.720 Yeah. 00:54:36.720 --> 00:54:38.440 A ThoughtBots factory bot. 00:54:38.440 --> 00:54:39.240 So, yeah. 00:54:39.240 --> 00:54:40.660 It's a really great project to work with. 00:54:40.660 --> 00:54:41.020 Awesome. 00:54:41.020 --> 00:54:42.000 That sounds like a great one. 00:54:42.000 --> 00:54:43.220 All right. 00:54:43.220 --> 00:54:45.440 Well, thank you all for being on the show. 00:54:45.440 --> 00:54:48.060 I want to give you one final chance for a call to action. 00:54:48.280 --> 00:54:50.420 There's people who have packages they maintain. 00:54:50.420 --> 00:54:52.860 They should probably play with your stuff, right? 00:54:52.860 --> 00:54:53.900 Try the new thing. 00:54:54.060 --> 00:54:57.280 We have people who maybe want to contribute to open source. 00:54:57.280 --> 00:54:59.020 Ernest, you spoke about the sprints. 00:54:59.020 --> 00:55:00.020 What should people do? 00:55:00.020 --> 00:55:01.220 They should come to the... 00:55:01.220 --> 00:55:03.400 If they're going to be a PyCon, they should come to the packaging sprints. 00:55:03.400 --> 00:55:04.320 So, I'll be there. 00:55:04.320 --> 00:55:08.700 Ernest, some part of Ernest will be there after running PyCon. 00:55:08.700 --> 00:55:09.660 We'll see what's left of him. 00:55:10.220 --> 00:55:16.060 And we're going to just sprint on the packaging ecosystem, including PyPI, and see what we can build. 00:55:16.060 --> 00:55:16.400 Awesome. 00:55:16.400 --> 00:55:19.560 You should go verify your primary e-mail. 00:55:19.560 --> 00:55:22.920 Janitorial aspects of that. 00:55:22.920 --> 00:55:24.600 So, go verify your e-mail address. 00:55:24.600 --> 00:55:25.860 That's super helpful for us. 00:55:25.860 --> 00:55:26.540 Yeah, that's awesome. 00:55:26.540 --> 00:55:27.180 Yeah, Nicole. 00:55:27.180 --> 00:55:32.220 The other thing is I'm planning on running a sprint also at Europython this year in Edinburgh. 00:55:32.220 --> 00:55:38.760 So, for people who are based in Europe who want to contribute to the project, we'll be running a sprint there as well. 00:55:39.220 --> 00:55:54.860 And the other thing is people should consider donating to the Python Packaging Working Group because we actually were lucky enough to receive an award from Mozilla to be able to fund working on our warehouse for the last few months. 00:55:54.860 --> 00:55:57.220 But that money is about to run out. 00:55:57.220 --> 00:56:03.820 We have used that money to get to our goal, which is to launch the new PyPI and shut down Legacy. 00:56:03.820 --> 00:56:19.160 But in terms of the future development of the project, you know, any funding that we can secure is obviously going to mean that we can move faster and more reliably and be less reliant on our volunteers for our sort of core infrastructure. 00:56:19.680 --> 00:56:23.800 So, yeah, I know the PSF is currently running a fundraising campaign. 00:56:23.800 --> 00:56:26.720 So, certainly consider donating to the working group. 00:56:26.720 --> 00:56:31.840 And there's a handy link, actually, at the top of the new site if you do want to donate. 00:56:31.840 --> 00:56:34.720 So, yeah, any contributions would be most welcome. 00:56:34.720 --> 00:56:36.420 That is a great suggestion. 00:56:36.420 --> 00:56:39.040 And, yeah, I think people definitely should do that. 00:56:39.260 --> 00:56:43.580 I forgot to call out the Mozilla Open Source Foundation and say thank you for that. 00:56:43.580 --> 00:56:51.900 But, like, the reason we're here having this conversation and it got this major boost is largely, like, that was a major factor in it, right? 00:56:51.900 --> 00:56:53.380 Dustin, you wanted to add something. 00:56:53.380 --> 00:56:57.460 Yeah, the Mozilla Award is definitely the reason why this was all possible. 00:56:57.460 --> 00:56:58.040 So, yeah. 00:56:58.040 --> 00:56:59.980 I wanted to have a call to action. 00:56:59.980 --> 00:57:04.700 Anyone that wants to contribute to the project or just contribute to Open Source, come and find us on GitHub. 00:57:04.940 --> 00:57:11.020 We are a pretty friendly group and we have a bunch of issues tagged a good first issue that you could take a crack at. 00:57:11.020 --> 00:57:13.500 And we'd like to see more contributors every day. 00:57:13.500 --> 00:57:13.880 Absolutely. 00:57:13.880 --> 00:57:18.000 And it's much easier, as you all have laid out, for various reasons why that's the case. 00:57:18.000 --> 00:57:18.560 Ernest? 00:57:18.560 --> 00:57:21.820 Yeah, I definitely wanted to just, like, I'm shaking here. 00:57:21.820 --> 00:57:27.060 How did we not talk a little bit more about Mozilla Open Source Support Grant Program? 00:57:27.240 --> 00:57:38.740 Indeed, it is the sole reason why PyPI.org launched on Monday and not in, like, another year or 18 months. 00:57:38.740 --> 00:57:50.780 Because just the amount of work that went into making this all possible, I think, in retrospect and without being super optimistic looking forward, wasn't incredible. 00:57:50.780 --> 00:58:04.980 And just based on looking back, it probably would have been an indefinite period of time before this occurred without being able to have people committed and thinking and soliciting the community to help as well. 00:58:04.980 --> 00:58:12.080 So Mozilla was instrumental and forever indebted to them for how much they made this happen. 00:58:12.080 --> 00:58:12.960 Yeah, that's really awesome. 00:58:12.960 --> 00:58:14.020 And thank you to them. 00:58:14.020 --> 00:58:14.440 That's great. 00:58:14.440 --> 00:58:16.040 I want to add one final thing. 00:58:16.040 --> 00:58:19.500 People should donate to the Python Packaging Working Group. 00:58:19.500 --> 00:58:28.720 But they should also, if they have a company that massively depends upon Python, they should say, dear company, you're running a $5 billion business on this. 00:58:28.720 --> 00:58:32.640 Could we set up a $1,000 recurring donation monthly to this? 00:58:32.640 --> 00:58:37.600 Because without this, your business goes away, or at least a good chunk of it. 00:58:37.600 --> 00:58:43.020 Yeah, the number of organizations and companies that depend on PyPI are most of them, it seems like. 00:58:43.180 --> 00:58:45.900 So, yeah, it's now possible to make recurring donations. 00:58:45.900 --> 00:58:48.480 So we definitely appreciate the support. 00:58:48.480 --> 00:58:48.880 Right. 00:58:48.880 --> 00:58:49.280 Awesome. 00:58:49.280 --> 00:58:49.980 All right. 00:58:49.980 --> 00:58:51.520 Well, let's leave it there. 00:58:51.520 --> 00:58:52.620 Thank you all for being on the show. 00:58:52.620 --> 00:58:53.660 It's been a great conversation. 00:58:53.660 --> 00:58:55.340 And congratulations on the launch. 00:58:55.340 --> 00:58:56.540 I'm super excited to see it. 00:58:56.540 --> 00:58:57.080 Thanks, Michael. 00:58:57.080 --> 00:58:57.960 Thanks, Michael. 00:58:57.960 --> 00:58:58.600 Thank you. 00:58:58.600 --> 00:59:03.440 This has been another episode of Talk Python To Me. 00:59:03.440 --> 00:59:08.760 Our guests have been Nicole Harris, Ernest Durbin III, and Dustin Ingram. 00:59:08.760 --> 00:59:12.200 And this episode has been brought to you by ActiveState and Codicy. 00:59:13.120 --> 00:59:17.860 ActiveState gives you a faster way to build and secure open source runtimes. 00:59:17.860 --> 00:59:24.860 From your first line of code through to production, check it out at talkpython.fm/activestate. 00:59:24.860 --> 00:59:28.540 Review less, merge faster with Codicy. 00:59:28.540 --> 00:59:36.340 Check code style, security, duplication, complexity, and coverage on every change while tracking code quality throughout your sprints. 00:59:36.860 --> 00:59:40.020 Try them at talkpython.fm/codicy. 00:59:40.020 --> 00:59:41.900 C-O-D-A-C-Y. 00:59:42.600 --> 00:59:43.840 Want to level up your Python? 00:59:43.840 --> 00:59:50.860 If you're just getting started, try my Python jumpstart by building 10 apps or our brand new 100 days of code in Python. 00:59:50.860 --> 00:59:54.660 And if you're interested in more than one course, be sure to check out the Everything Bundle. 00:59:54.660 --> 00:59:56.920 It's like a subscription that never expires. 00:59:57.340 --> 00:59:59.120 Be sure to subscribe to the show. 00:59:59.120 --> 01:00:01.300 Open your favorite podcatcher and search for Python. 01:00:01.300 --> 01:00:02.560 We should be right at the top. 01:00:02.560 --> 01:00:11.840 You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm. 01:00:12.200 --> 01:00:13.740 This is your host, Michael Kennedy. 01:00:13.740 --> 01:00:15.100 Thanks so much for listening. 01:00:15.100 --> 01:00:16.180 I really appreciate it. 01:00:16.180 --> 01:00:18.120 Now get out there and write some Python code. 01:00:18.120 --> 01:00:38.620 I'll see you next time.