WEBVTT

00:00:00.001 --> 00:00:02.300
What is the most powerful part of the Python ecosystem?

00:00:02.300 --> 00:00:07.080
Well, the ability to say pip install magic library has to be right near the top.

00:00:07.080 --> 00:00:11.460
But what powers the Python package index and who are the people behind it?

00:00:11.460 --> 00:00:16.620
Did you know that they ship over 300 terabytes of traffic each month these days?

00:00:16.620 --> 00:00:21.480
Join me as we chat with Donald Stuffed to look inside Python's packaging infrastructure.

00:00:21.480 --> 00:00:26.860
This is Talk Python To Me, episode 64, recorded Wednesday, June 22, 2016.

00:00:26.860 --> 00:00:56.580
Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the

00:00:56.580 --> 00:00:58.100
ecosystem, and the personalities.

00:00:58.100 --> 00:01:00.240
This is your host, Michael Kennedy.

00:01:00.240 --> 00:01:02.240
Follow me on Twitter where I'm @mkennedy.

00:01:02.240 --> 00:01:06.120
Keep up with the show and listen to past episodes at talkpython.fm.

00:01:06.120 --> 00:01:08.640
And follow the show on Twitter via at Talk Python.

00:01:08.640 --> 00:01:12.880
This episode has been brought to you by SnapCI and Rollbar.

00:01:12.880 --> 00:01:14.440
Hey, everyone.

00:01:14.440 --> 00:01:18.500
I have a really fabulous look inside the Python package index on deck for you.

00:01:18.500 --> 00:01:22.420
Donald and I just scratched the surface, but there are a ton of fascinating topics that

00:01:22.420 --> 00:01:22.720
we cover.

00:01:22.720 --> 00:01:25.920
Before we get to our talk with Donald, I have one announcement for you.

00:01:26.300 --> 00:01:30.320
I've released my second online course called Write Pythonic Code Like a Seasoned Developer.

00:01:30.320 --> 00:01:35.460
It's over four hours and 50 concrete examples of how you can write more Pythonic code.

00:01:35.460 --> 00:01:39.620
It's jam-packed with tips that you can incorporate into your projects immediately.

00:01:39.620 --> 00:01:44.800
Topics covered include the expansive use of dictionaries, hacking Python's memory usage

00:01:44.800 --> 00:01:50.240
via slots, using generators to comprehend and generate expressions, creating subsets of collections

00:01:50.240 --> 00:01:53.060
via slices all the way to the database, and much more.

00:01:53.660 --> 00:01:58.460
Several of these topics are Python 3-only features, so you'll have even more reason to adopt Python

00:01:58.460 --> 00:01:59.640
3 for your next project.

00:01:59.640 --> 00:02:03.040
The response to the first two days has been super positive.

00:02:03.040 --> 00:02:07.620
I hope you'll take a moment to see what the course is all about at talkpython.fm/pythonic.

00:02:07.620 --> 00:02:09.300
Now, let's get to Donald.

00:02:09.300 --> 00:02:11.340
Donald, welcome to the show.

00:02:11.340 --> 00:02:11.880
Yes, hi.

00:02:12.280 --> 00:02:13.780
Hey, thanks for coming today.

00:02:13.780 --> 00:02:18.860
It's going to be really fun to talk about packaging and PyPI and all these sorts of things.

00:02:18.860 --> 00:02:23.520
But, of course, like always, before we get into the details, let's hear your story.

00:02:23.520 --> 00:02:25.020
How did you get started in programming in Python?

00:02:25.480 --> 00:02:27.800
Yeah, so programming in general.

00:02:27.800 --> 00:02:34.940
I was playing a video game back in high school on EverQuest, and I started to get into hacking

00:02:34.940 --> 00:02:35.900
that video game.

00:02:35.900 --> 00:02:40.740
And there was a tool called MacReQuest that I started writing add-ons for and such, and it

00:02:40.740 --> 00:02:44.060
sort of got me involved in programming at all.

00:02:44.280 --> 00:02:48.160
And then I started to get jobs in programming.

00:02:48.160 --> 00:02:51.560
My first job was with PHP.

00:02:51.560 --> 00:02:58.380
And when I was working with Drupal at the time, I kept hearing about this cool framework called

00:02:58.380 --> 00:02:59.920
Django, but it was written in Python.

00:02:59.920 --> 00:03:08.760
So I picked up a Python book and sort of taught myself Python over a week or so, so that I could

00:03:08.760 --> 00:03:12.600
then use Django to do websites instead of Drupal.

00:03:13.080 --> 00:03:17.340
So I was feeling constrained by the sort of CMS aspects of Drupal.

00:03:17.340 --> 00:03:18.840
Yeah, okay, that's cool.

00:03:18.840 --> 00:03:20.460
Drupal's PHP, right?

00:03:20.460 --> 00:03:28.020
Yes, Drupal's a PHP CMS that's sort of got some framework-y aspects to it, or at least it

00:03:28.020 --> 00:03:29.600
did back then.

00:03:29.600 --> 00:03:31.260
That was 2007 or so.

00:03:31.260 --> 00:03:32.360
I haven't really looked at it.

00:03:32.360 --> 00:03:33.420
I did recently.

00:03:33.420 --> 00:03:35.440
Yeah, it's definitely still going.

00:03:35.440 --> 00:03:39.440
So was it refreshing to get into Python from PHP?

00:03:39.440 --> 00:03:43.060
Yes, I found Python to be incredibly useful.

00:03:43.060 --> 00:03:44.780
concise and great to work with.

00:03:44.780 --> 00:03:52.540
I like the enforced white space coming from, you know, somewhere where PHP attends.

00:03:52.540 --> 00:03:57.940
And particularly at the time, a lot of the examples you would find online didn't always have the

00:03:57.940 --> 00:03:58.320
greatest formatting.

00:03:58.320 --> 00:04:05.880
So that sort of enforced formatting helped a lot with me reading other people's code to

00:04:05.880 --> 00:04:08.120
figure out what they were doing and how it worked.

00:04:08.120 --> 00:04:15.020
And sort of just a general grogking of the code bases to help me learn over time.

00:04:15.020 --> 00:04:15.800
Yeah, yeah.

00:04:15.800 --> 00:04:16.400
That's really cool.

00:04:16.400 --> 00:04:17.400
I'm sure it was.

00:04:17.400 --> 00:04:17.940
It was nice.

00:04:18.440 --> 00:04:24.920
So I think maybe a good place to start this conversation would be to talk about what you

00:04:24.920 --> 00:04:26.360
do for your day job.

00:04:26.360 --> 00:04:33.140
Yeah, so my day job is I'm employed by Hucule Packard Enterprise, full-time employee.

00:04:33.140 --> 00:04:38.020
And my sort of mandate is basically make Python packaging better.

00:04:38.620 --> 00:04:40.660
So from there, you know, I work on PyPI.

00:04:40.660 --> 00:04:43.100
I work on pip.

00:04:43.100 --> 00:04:48.120
I work on a project called Warehouse, which is essentially PyPI 2.0.

00:04:48.120 --> 00:04:53.500
The Warehouse theme is not really going to be exposed to end users other than if they start

00:04:53.500 --> 00:04:54.520
to work on the backend.

00:04:54.720 --> 00:04:57.320
But we call it Warehouse just to distinguish it.

00:04:57.320 --> 00:05:03.820
Then there's other little small tools like Twine and, you know, sort of a lot of the background

00:05:03.820 --> 00:05:10.620
efforts, you know, writing peps and coordinating things and whatnot sort of all falls under my

00:05:10.620 --> 00:05:13.720
banner of what I do for my day job.

00:05:13.720 --> 00:05:20.760
Yeah, so that's really awesome that HP is making that investment to more or less fund a developer

00:05:20.760 --> 00:05:23.020
to continuously work on Python packaging.

00:05:24.020 --> 00:05:26.540
Yeah, you know, you know, I think it's great.

00:05:26.540 --> 00:05:30.320
You know, they are a big contributor to the OpenStack community.

00:05:30.320 --> 00:05:34.520
And OpenStack is, I think it's not entirely written in Python now, but it's largely written

00:05:34.520 --> 00:05:34.980
in Python.

00:05:34.980 --> 00:05:40.420
So they're heavy users of, you know, Python and PyPI and pip and whatnot.

00:05:40.420 --> 00:05:46.660
You know, so HP felt that, you know, they're depending on these things, you know, made sense

00:05:46.660 --> 00:05:52.380
to invest in these things to make sure that they continue to be running and work and whatnot.

00:05:52.380 --> 00:05:53.540
Yeah, that's cool.

00:05:53.540 --> 00:06:00.460
So is this a position that you somehow managed to create or was there like a job announcement

00:06:00.460 --> 00:06:03.820
hiring packaging support person?

00:06:05.160 --> 00:06:05.600
Yeah.

00:06:05.600 --> 00:06:05.600
Yeah.

00:06:05.600 --> 00:06:07.700
So it was created for me.

00:06:07.700 --> 00:06:15.260
At the time there was somebody named Monty Taylor inside the HPE who sort of led the effort

00:06:15.260 --> 00:06:19.080
to convince the higher ups that this was something that they should do.

00:06:19.080 --> 00:06:22.880
And, you know, he's the one who reached out to me and offered this to me.

00:06:22.880 --> 00:06:26.760
You know, prior to this, I was working at Rackspace where they gave me half time.

00:06:26.760 --> 00:06:32.180
You know, 50% time to work on packaging, 50% time to work on their own projects.

00:06:32.180 --> 00:06:37.860
You know, so he sort of took that idea and just amped it up to the next level.

00:06:38.300 --> 00:06:44.360
And, you know, we're really pushed for getting that done in HPE and convinced me to come over.

00:06:45.220 --> 00:06:46.240
Yeah, that's awesome.

00:06:46.240 --> 00:06:46.500
Yeah.

00:06:46.500 --> 00:06:46.500
Yeah.

00:06:46.500 --> 00:06:46.500
Yeah.

00:06:46.500 --> 00:06:47.500
Yeah.

00:06:47.500 --> 00:06:54.240
I think we'll dig in a little bit more later on how companies are supporting PyPI and so on.

00:06:54.240 --> 00:06:56.720
But let's keep it high level for a little bit.

00:06:56.720 --> 00:07:02.700
So I've had a lot of interesting conversations with people basically around the pronunciation

00:07:02.700 --> 00:07:04.780
of P-Y-P-I.

00:07:04.780 --> 00:07:09.180
And I've heard a lot of people say PyPI.

00:07:09.180 --> 00:07:13.420
And I've heard a fair number of people say PyPI.

00:07:13.420 --> 00:07:15.960
What's the official way to say it?

00:07:15.960 --> 00:07:18.140
The official way to say it is PyPI.

00:07:18.140 --> 00:07:20.820
It's Python package index.

00:07:20.820 --> 00:07:23.820
So, you know, you put the emphasis on the PyPI.

00:07:23.820 --> 00:07:26.420
A lot of people do call it PyPI.

00:07:26.420 --> 00:07:31.700
I've heard Pippi and PeePee and all sorts of pronunciations.

00:07:31.700 --> 00:07:33.760
I really don't like the name.

00:07:33.760 --> 00:07:39.720
I think it's a confusing name that has this big sort of problem with pronunciation.

00:07:39.720 --> 00:07:41.780
Everyone pronounces it a little bit differently.

00:07:41.780 --> 00:07:45.680
And one of the most common pronunciations, PyPI,

00:07:45.680 --> 00:07:49.540
clashes with PyPI as P-Y-P-Y.

00:07:49.540 --> 00:07:54.200
The alternative Python implementation, however,

00:07:54.200 --> 00:08:01.740
I've sort of thus far been unsuccessful at convincing people that it's worth it to change the name

00:08:01.740 --> 00:08:09.080
since P-Y-P-I is so ingrained and everywhere across the ecosystem.

00:08:09.880 --> 00:08:13.580
Yeah, it's deeply ingrained, but it's also an insider term, right?

00:08:13.580 --> 00:08:21.780
Like, you can't walk up to somebody who's very barely familiar with Python and say PyPI and have them,

00:08:21.780 --> 00:08:22.940
oh, yeah, of course, right?

00:08:23.380 --> 00:08:27.540
But, you know, that's true, I think, with lots of the packaging systems.

00:08:27.540 --> 00:08:31.080
Like, if you said npm to somebody, they wouldn't know.

00:08:31.080 --> 00:08:34.240
If you said Gems, they wouldn't know.

00:08:34.240 --> 00:08:35.100
You know, NuGet.

00:08:35.100 --> 00:08:37.180
Like, there's all these packaging systems.

00:08:37.180 --> 00:08:41.280
They all seem to have poor names, but PyPI, okay, that's great.

00:08:41.660 --> 00:08:47.140
So I'm glad to hear that I've been saying it right, but I've been following Guido's lead,

00:08:47.140 --> 00:08:49.100
and I figured that's a pretty safe lead to follow.

00:08:49.100 --> 00:08:50.720
Yeah, yep.

00:08:50.720 --> 00:08:56.560
And, you know, PyPI is what it is, although I do slip up every once in a while and pronounce it wrong.

00:08:56.560 --> 00:08:57.760
Yeah.

00:08:57.760 --> 00:08:58.880
Okay, cool.

00:08:58.880 --> 00:09:03.940
So PyPI, one thing I always wondered about, maybe you can explain this for me.

00:09:03.940 --> 00:09:09.760
If I type pypi.python.org and hit enter, I get pypi.python.org slash pypi.

00:09:09.760 --> 00:09:11.760
Like, why does it appear on both ends?

00:09:11.760 --> 00:09:14.140
Is it just, like, really beloved?

00:09:14.140 --> 00:09:19.880
There's a long history there, and I'm, you know, I don't fully know it

00:09:19.880 --> 00:09:22.260
because a lot of it comes from before my time.

00:09:22.260 --> 00:09:29.620
But I believe PyPI was originally deployed under just python.org slash pypi,

00:09:29.620 --> 00:09:33.680
and then they eventually moved it to its own domain name.

00:09:33.680 --> 00:09:42.700
It's pypi.python.org, and they just kept the slash pypi because that prefix was baked into scripts,

00:09:42.700 --> 00:09:47.760
and it was easier to just change the domain name rather than changing the whole path.

00:09:47.760 --> 00:09:48.600
Oh, right.

00:09:48.600 --> 00:09:50.300
So it's just a compatibility thing.

00:09:50.300 --> 00:09:50.820
Yeah.

00:09:50.820 --> 00:09:53.180
And it was actually just recently we,

00:09:53.180 --> 00:09:57.260
and I knew that because we just recently, within the last year,

00:09:58.020 --> 00:10:03.580
broke compatibility for people who were still using python.org slash pypi,

00:10:03.580 --> 00:10:09.020
and apparently there were still people who had scripts and whatnot pointing to that

00:10:09.020 --> 00:10:13.180
because when we broke that, we got people yelling at us for it.

00:10:13.180 --> 00:10:14.380
You heard about it, huh?

00:10:14.380 --> 00:10:15.000
Yes.

00:10:15.000 --> 00:10:20.320
Well, that's the quickest, easiest way to tell if a service or a database

00:10:20.320 --> 00:10:22.680
or something like that is required is like,

00:10:22.680 --> 00:10:25.020
turn it off and see if anyone screams, right?

00:10:25.020 --> 00:10:26.560
Yes.

00:10:26.560 --> 00:10:33.960
And with PyPI, that's one of our main ways we figure out what's needed or not

00:10:33.960 --> 00:10:38.460
because it's never been documented well what you can depend on in PyPI,

00:10:38.820 --> 00:10:44.360
and sort of the entire history of PyPI has been someone depended on something weird in it,

00:10:44.360 --> 00:10:47.900
some sort of implementation detail, and suddenly that became the API.

00:10:47.900 --> 00:10:53.740
So a lot of things in PyPI are we have no idea what people are depending on,

00:10:53.740 --> 00:10:56.280
so we just change things and pray.

00:10:56.280 --> 00:10:56.800
Yeah.

00:10:56.800 --> 00:11:00.620
So you step very lightly making changes, I suspect.

00:11:00.620 --> 00:11:01.320
Yeah.

00:11:01.320 --> 00:11:06.420
Particularly on what we call legacy PyPI, which is what's running now

00:11:06.640 --> 00:11:09.880
because it's a very old code base.

00:11:09.880 --> 00:11:13.560
It's like 15 years old or so, and it's got no tests.

00:11:13.560 --> 00:11:16.360
It doesn't run locally very easily.

00:11:16.360 --> 00:11:20.380
Like, to actually run it locally, you have to modify the code

00:11:20.380 --> 00:11:23.040
and comment things out to get it to start up.

00:11:23.040 --> 00:11:26.020
So it's fun.

00:11:26.020 --> 00:11:26.580
Yeah.

00:11:26.580 --> 00:11:33.380
I'm sure this is really an interesting experience and long-lived Python code.

00:11:33.380 --> 00:11:36.360
So maybe it's a good place to start talking about the history.

00:11:36.580 --> 00:11:40.320
Like, PyPI is not as old as Python, right?

00:11:40.320 --> 00:11:43.100
And so there was a while where Python was just a thing.

00:11:43.100 --> 00:11:44.640
There was no packaging, right?

00:11:44.640 --> 00:11:46.240
And then PyPI came along.

00:11:46.240 --> 00:11:47.240
What's the story there?

00:11:47.240 --> 00:11:49.280
So, again, this predates me.

00:11:49.280 --> 00:11:53.920
So this is what my understanding is from what people have told me.

00:11:53.920 --> 00:11:58.480
So Python did originally have no packaging story.

00:11:59.040 --> 00:12:02.060
So people started sort of making their own story with make files.

00:12:02.060 --> 00:12:05.880
Make files obviously are not super repeatable.

00:12:05.880 --> 00:12:09.080
They had to write a whole new make file every time you switched to a different project.

00:12:09.080 --> 00:12:12.940
They don't run by default on Windows, you know, et cetera, et cetera.

00:12:13.620 --> 00:12:26.720
So someone whose name escapes me right now came up with distruteels to replace all these make files with a Python script that will run everywhere and it will sort of do all that stuff for you.

00:12:26.720 --> 00:12:29.420
So, which at the time, great.

00:12:29.420 --> 00:12:34.380
It was, you know, a really big step forward for Python and packaging in general.

00:12:35.420 --> 00:12:49.040
And I think roughly around that same time, maybe a little bit after, there were several sort of efforts to create sort of a CPAN, but for Python, one name that I know of was Vaults of Parnassus.

00:12:49.040 --> 00:12:53.860
I don't fully know all the names or what their specific implementations were.

00:12:54.060 --> 00:13:04.240
But then Richard Jones, I believe, came up with the idea of PyPI and sort of implemented a proof of concept of what PyPI could be.

00:13:04.240 --> 00:13:06.580
And they deployed it to Python.org.

00:13:06.580 --> 00:13:09.680
And we're still running that today.

00:13:09.680 --> 00:13:14.620
That proof of concept that was originally designed to be replaced quickly with the real thing.

00:13:14.780 --> 00:13:24.420
I'm always a little suspicious or leery of proof of concepts, showing them to people who are like, oh, this is amazing.

00:13:24.420 --> 00:13:25.200
Let's use this.

00:13:25.200 --> 00:13:25.860
We don't even have to.

00:13:25.860 --> 00:13:26.640
We're almost done.

00:13:26.640 --> 00:13:27.280
Let's go.

00:13:27.280 --> 00:13:27.700
Right.

00:13:27.700 --> 00:13:28.560
Yes.

00:13:28.560 --> 00:13:32.740
It can be a shifty, soft foundation, I suppose.

00:13:32.740 --> 00:13:33.920
Yes.

00:13:33.920 --> 00:13:39.640
And an important thing to realize about PyPI's code quality, it's not great.

00:13:40.260 --> 00:13:45.720
It was written 15, mostly 15 years ago with people hacking on it since then.

00:13:45.720 --> 00:13:51.000
And it predates much of what consists of the modern web stack.

00:13:51.000 --> 00:13:52.920
Django didn't exist.

00:13:52.920 --> 00:13:54.400
Pyramid didn't exist.

00:13:54.400 --> 00:13:55.600
Flask didn't exist.

00:13:55.600 --> 00:14:00.860
I think WSGI might have existed barely, or they were like contemporaries.

00:14:00.860 --> 00:14:03.880
So the original PyPI didn't use WSGI.

00:14:03.880 --> 00:14:07.600
It just sort of wrote its own Python handler.

00:14:07.600 --> 00:14:09.560
I think it used CGI.

00:14:10.260 --> 00:14:12.580
So a lot of that code is still there.

00:14:12.580 --> 00:14:22.600
And a lot of what we've done to try and semi-modernize it has been how do we hack in WSGI support into this thing that expects CGI?

00:14:22.600 --> 00:14:24.620
How do we deploy this?

00:14:24.620 --> 00:14:29.900
We're using a modern web server instead of this little script that just happens to sit inside the thing.

00:14:29.900 --> 00:14:30.380
Yeah.

00:14:30.380 --> 00:14:35.320
So it's got a lot of soap in it because it was big 15 years ago.

00:14:35.320 --> 00:14:35.960
Right.

00:14:35.960 --> 00:14:36.360
Of course.

00:14:36.740 --> 00:14:43.400
A lot of its problems really just stemmed from the fact it was written before we knew how to do good websites.

00:14:43.400 --> 00:14:44.260
Yeah.

00:14:44.260 --> 00:14:47.080
What year was that when the prototype was created?

00:14:47.080 --> 00:14:48.520
I believe it was 2003.

00:14:48.520 --> 00:14:49.300
Okay.

00:14:50.400 --> 00:14:56.740
So there were some examples of serious web activity, serious websites, right?

00:14:56.740 --> 00:14:59.480
That's .com, just post .com days.

00:14:59.480 --> 00:15:05.000
But still, Python was not nearly as polished around the web story there, was it?

00:15:05.000 --> 00:15:05.240
Yeah.

00:15:05.240 --> 00:15:05.600
Correct.

00:15:05.600 --> 00:15:06.000
Cool.

00:15:06.000 --> 00:15:06.440
Yeah.

00:15:06.460 --> 00:15:06.620
Okay.

00:15:06.620 --> 00:15:07.500
So that's really interesting.

00:15:07.500 --> 00:15:11.840
So what's the relationship with easy install these days?

00:15:11.840 --> 00:15:13.700
Easy install lets you install packages.

00:15:13.700 --> 00:15:15.300
PIP lets us install packages.

00:15:15.300 --> 00:15:17.380
It's all pip these days, right?

00:15:17.380 --> 00:15:19.500
Is there still a reason to use easy install?

00:15:19.500 --> 00:15:21.540
So easy install still exists.

00:15:21.540 --> 00:15:25.260
It does a few things that pip doesn't do.

00:15:25.860 --> 00:15:28.300
And pip largely doesn't do them on purpose.

00:15:28.300 --> 00:15:41.480
Like easy install supports multi-version installs where you have a single Python environment and you can install multiple versions of, say, requests into that single Python environment.

00:15:41.480 --> 00:15:51.620
And then at runtime, it will generate a sys.path that has the correct version for whatever thing you're running on that sys.path.

00:15:51.620 --> 00:15:57.040
PIP doesn't do that, which I believe is a good thing pip prefers to use.

00:15:57.040 --> 00:16:04.360
Is that because pip really assumes the presence of virtual environments and it's like, if that's what you want, just create a virtual environment, it's a different version?

00:16:04.360 --> 00:16:05.220
Yes.

00:16:05.220 --> 00:16:05.500
Correct.

00:16:05.800 --> 00:16:13.860
So pip says use virtual environments, create explicit environments, and install things into there.

00:16:13.860 --> 00:16:19.220
Easy install while it works with virtual environments because there's a way virtual environments work.

00:16:19.220 --> 00:16:29.300
Virtual environments, it sort of says you declare in your script or in your script wrappers, et cetera, what you depend on.

00:16:29.300 --> 00:16:42.000
And we will sort of create a dynamic virtual environment in memory by munging the sys.path, whereas virtual environments are you create an explicit named environment, a name by your file system path.

00:16:42.000 --> 00:16:48.240
And then, you know, pip will install things flat, you know, just a single version into that environment.

00:16:48.240 --> 00:16:55.780
That's a cool feature that you could hack it together and have multiple versions, but it seems like it's almost more trouble than it's worth.

00:16:55.780 --> 00:16:58.880
Yeah, so it's certainly an interesting feature.

00:16:58.880 --> 00:17:01.600
I don't think one is necessarily better than the other.

00:17:01.600 --> 00:17:03.840
They each have their trade-offs.

00:17:04.840 --> 00:17:17.920
We've sort of settled around using the virtual environment, a single flat install system, and I don't think there's enough of a reason to go back to the easy install sort of multi-version install system.

00:17:17.920 --> 00:17:23.180
Just because, you know, I think everyone's sort of figured out how to use the virtual environments.

00:17:23.980 --> 00:17:29.680
And, you know, I think trying to change gears now would just be a big disruption for not a whole lot of benefit.

00:17:29.680 --> 00:17:34.200
But if we had never done that, I think we would be in a perfectly fine place now.

00:17:34.200 --> 00:17:37.580
We just have a different mechanism for doing things.

00:17:37.580 --> 00:17:38.800
Okay, cool.

00:17:39.440 --> 00:17:46.600
So what's the relationship between the Python Packaging Authority and the Python Software Foundation?

00:17:46.600 --> 00:17:52.960
Yeah, so the PyPA is sort of not a real thing, sort of is a real thing.

00:17:52.960 --> 00:17:55.320
You know, it's not an official organization.

00:17:55.320 --> 00:17:59.220
We don't have a 5013C or whatever they're called.

00:17:59.220 --> 00:18:04.940
So Python Packaging Authority is like a shadow organization pulling the strings of Python packaging.

00:18:04.940 --> 00:18:05.700
Yeah, sort of.

00:18:05.700 --> 00:18:07.440
Yeah, yeah.

00:18:07.440 --> 00:18:09.440
You can think of it that way.

00:18:09.440 --> 00:18:18.020
You know, it sort of came around because pip was putting itself on GitHub and they needed an organization name.

00:18:18.020 --> 00:18:21.640
So they just kind of jokingly called it PyPI or PyPA.

00:18:21.640 --> 00:18:23.600
And I have too many things in my head.

00:18:23.600 --> 00:18:23.900
It's good.

00:18:23.900 --> 00:18:25.200
P-Y-P something.

00:18:25.200 --> 00:18:26.820
I confuse them.

00:18:27.400 --> 00:18:29.240
Yeah, so they sort of jokingly called it PyPA.

00:18:29.240 --> 00:18:44.740
And then as we sort of, this sort of push in the past couple of years to really standardize around these things and fix things and get moved forward, we just sort of started to take that name and say, okay, we're going to really take this name that started out as a sort of joke.

00:18:44.740 --> 00:18:46.640
And we're going to use this for a real thing.

00:18:46.640 --> 00:18:52.420
And it's sort of just an umbrella organization where it's like, okay, you're working on something that's packaging in Python.

00:18:52.420 --> 00:18:55.280
You can bring your project into PyPA.

00:18:55.280 --> 00:18:57.000
We don't have very many rules.

00:18:57.120 --> 00:19:02.660
I think the only rule we really have is that you have to, your project has to be governed by our code of conduct.

00:19:02.660 --> 00:19:08.280
And beyond that, you know, you can run your project however you want, use whatever VCS you want, et cetera, et cetera.

00:19:08.280 --> 00:19:13.280
But it sort of provides a central location for someone who's, okay, I want to work on packaging stuff.

00:19:13.280 --> 00:19:16.700
You know, go to the PyPA, you know, pages.

00:19:16.700 --> 00:19:19.160
You know, you can see the list of projects.

00:19:19.160 --> 00:19:26.700
And then since there's a lot of cross-pollination between people who work on these projects, kind of makes managing permissions and stuff a bit easier.

00:19:26.700 --> 00:19:32.500
But, like, funding, we're starting to try and get funding and stuff available for that.

00:19:32.500 --> 00:19:34.440
So that's largely going through the PSF.

00:19:34.440 --> 00:19:39.920
You know, so the PSF is sort of the legal entity that we hang our stuff off of.

00:19:39.920 --> 00:19:43.760
But the PyPA is sort of, you know, just our little unofficial organization.

00:19:43.980 --> 00:19:48.660
You can think of it similar to Python Dev, how Python Dev manages CPython.

00:19:48.660 --> 00:19:54.420
But, you know, the legal trademarks and stuff sort of hang off of the PSF.

00:19:54.420 --> 00:19:55.120
Right.

00:19:55.120 --> 00:19:55.400
Okay.

00:19:55.400 --> 00:20:05.500
So since the Packaging Authority is just, like, a loose group of people who work together on packaging, there's no legal component there.

00:20:05.500 --> 00:20:12.820
Like, for example, I made a donation to PyPI when you guys announced, like, hey, we need some supporters and whatnot.

00:20:13.060 --> 00:20:17.620
Small one, but still, to do that, I actually went to, through the PSF website.

00:20:17.620 --> 00:20:19.920
And I sort of donated it to them.

00:20:19.920 --> 00:20:21.260
And then they forwarded it on, right?

00:20:21.260 --> 00:20:26.920
So for things like that, where you need an entity, PSF is kind of there to support you.

00:20:26.920 --> 00:20:27.980
Yeah, exactly.

00:20:27.980 --> 00:20:35.500
It frees us up from having to deal with, you know, getting the board of directors and dealing with the mundane legalities.

00:20:35.500 --> 00:20:37.100
PSF already does that.

00:20:37.100 --> 00:20:41.880
And I believe they call it fiscal sponsorship or something along that lines.

00:20:41.880 --> 00:20:47.960
But, you know, it's basically, you know, they manage being the legal entity behind this stuff.

00:20:47.960 --> 00:20:58.280
And we manage actually doing the, you know, the code work and, you know, the roadmaps about how this is going to go forward and, you know, making those sort of decisions.

00:21:11.000 --> 00:21:23.080
SnapCI is a continuous delivery tool from ThoughtWorks that lets you reliably test and deploy your code through multi-stage pipelines in the cloud without the hassle of managing hardware.

00:21:23.080 --> 00:21:29.720
Automate and visualize your deployments with ease and make pushing to production an effortless item on your to-do list.

00:21:29.720 --> 00:21:35.620
Snap also supports Docker and M browser debugging, and they integrate with AWS and Heroku.

00:21:36.120 --> 00:21:44.300
Thanks, SnapCI for sponsoring this episode by trying them with no obligation for 30 days by going to snap.ci slash talkpython.

00:21:54.440 --> 00:21:58.000
So I kind of want to focus on three themes around PyPI.

00:21:58.000 --> 00:22:03.300
One is looking inside the infrastructure and the traffic and all that.

00:22:03.300 --> 00:22:06.620
One is about this new version you called warehouse.

00:22:06.620 --> 00:22:09.560
And then also, like, the funding and support and that kind of thing.

00:22:09.560 --> 00:22:11.820
So let's start with the traffic and infrastructure.

00:22:11.820 --> 00:22:16.800
You wrote a cool blog post called Powering the Python Package Index.

00:22:17.260 --> 00:22:20.680
In there, you really laid out a lot of the underlying technology.

00:22:20.680 --> 00:22:23.380
You talked about some of the bandwidth requirements.

00:22:23.380 --> 00:22:29.620
Like, I felt like I had bandwidth requirements until I saw what you guys are doing.

00:22:29.620 --> 00:22:33.040
Like, you had a comment where you said,

00:22:33.400 --> 00:22:41.460
We had 293 terabytes of traffic serving 3 billion requests in April, for example.

00:22:41.460 --> 00:22:43.100
Yes.

00:22:43.100 --> 00:22:44.160
Can you want to talk a little bit about that?

00:22:44.160 --> 00:22:45.580
I mean, that's pretty darn impressive.

00:22:45.580 --> 00:22:55.440
Like, you know, you've got to go talking to the video places, the Vimeos, the YouTubes, and the Netflixes to get more traffic than that or more bandwidth than that in some sense, right?

00:22:56.040 --> 00:23:07.020
Yeah, you know, and I mean, the vast bulk of that bandwidth is taken up by the package files themselves, although a not insignificant amount of that is taken up by API requests and stuff.

00:23:07.020 --> 00:23:08.440
I was actually curious.

00:23:08.440 --> 00:23:20.100
I looked at May's numbers, and May's numbers are 343 terabytes and, you know, 3 billion and change HTTP requests again.

00:23:20.280 --> 00:23:21.060
That's really amazing.

00:23:21.060 --> 00:23:31.120
And so it sounds like May had, I don't know, what is that, like 10% more traffic in terms of bandwidth than it did in April.

00:23:31.120 --> 00:23:34.780
So does that mean, how do you interpret it?

00:23:34.780 --> 00:23:40.900
Does that mean that we have more popularity of Python, like the popularity and usage of Python is growing?

00:23:40.900 --> 00:23:49.040
Or does that mean people are spinning up more little tiny VMs and pip install requirements.txt more often?

00:23:49.040 --> 00:23:55.800
Like, is this a use case variation or is this a adoption variation, you think, as these numbers are going up?

00:23:55.800 --> 00:23:59.960
Yeah, so I actually don't have a whole lot of insight into exactly what it is.

00:23:59.960 --> 00:24:04.940
It's something that I've tried to get some insight into, but it's kind of hard.

00:24:04.940 --> 00:24:13.980
I will say one thing is that as of pip 6, which was released in, I want to say, the end of 2014, I think.

00:24:13.980 --> 00:24:15.180
Don't quote me on that, though.

00:24:15.180 --> 00:24:19.040
Pip sort of aggressively caches locally.

00:24:19.680 --> 00:24:27.220
So you type pip install requests and you get request version 3.0 or 2.0 or whatever.

00:24:27.220 --> 00:24:30.900
It only downloads that file once per computer, basically.

00:24:30.900 --> 00:24:32.540
It uses HTTP caching.

00:24:32.540 --> 00:24:36.280
We have 10-year-long cache headers on those.

00:24:36.520 --> 00:24:39.640
So it's basically once per 10 years per file.

00:24:39.640 --> 00:24:43.100
It's basically once per machine, right?

00:24:43.100 --> 00:24:45.960
Like, probably the machine goes away before the cache does.

00:24:45.960 --> 00:24:50.580
Yeah, unless you blow away the cache or something along that lines.

00:24:51.080 --> 00:25:01.100
So we're definitely, I believe, not seeing increase in people doing pip install on their own machine multiple times.

00:25:01.100 --> 00:25:03.220
Or rather, I shouldn't say we're not seeing increase.

00:25:03.220 --> 00:25:09.380
We don't know if that's increasing or not because they download once and then we never see that download again from them.

00:25:09.380 --> 00:25:10.720
No matter how many times.

00:25:10.720 --> 00:25:10.820
Right.

00:25:10.820 --> 00:25:12.880
It's probably not reflected in those numbers.

00:25:12.880 --> 00:25:13.300
Yeah.

00:25:13.300 --> 00:25:13.700
Yeah.

00:25:13.700 --> 00:25:19.640
So this is going to be either new users or new machines, cloud machines.

00:25:19.860 --> 00:25:25.880
You know, I think probably more people switching to cloud-based workflows, you know, have an impact on this.

00:25:25.880 --> 00:25:30.780
You know, because each time you bring up a new cloud, you know, Docker containers, yes.

00:25:30.780 --> 00:25:37.040
Each time you bring up a new Docker container, a new cloud, you know, you start with a fresh cache that downloads again.

00:25:37.040 --> 00:25:41.760
I think people are doing CI more and particularly things like Travis and such.

00:25:41.760 --> 00:25:48.680
Unless you go out of your way to cache things between runs, Travis is going to give you a brand new cache each time.

00:25:49.480 --> 00:25:51.240
You know, so I think a lot of it is things like that.

00:25:51.240 --> 00:25:57.720
Yeah, I use SynapseEye and every time I check in something, it definitely pip installs a bunch of stuff.

00:25:57.720 --> 00:26:02.260
I'm not sure if it has the cache populated before that or not.

00:26:02.260 --> 00:26:03.840
But yeah, you're right.

00:26:03.840 --> 00:26:07.060
Every check-in in some sense triggers that kind of behavior.

00:26:07.060 --> 00:26:07.620
Yeah.

00:26:07.800 --> 00:26:19.680
You know, and plus I do think we are – I don't know if Python itself is growing in usage, but I think pip is particularly since there's been a lot more push from the Python doc side and things.

00:26:19.680 --> 00:26:24.000
Say, hey, you know, here's a thing that you can use to download other packages.

00:26:24.660 --> 00:26:29.360
I think particularly for Python 2.6 and 2.7, they're getting kind of long-handed too.

00:26:29.360 --> 00:26:33.720
And people are wanting some of the new features from Python 3 without actually switching to Python 3.

00:26:33.900 --> 00:26:43.700
So they're installing backports and stuff or they're reaching out for things that aren't included in the standard library as much more than they used to be.

00:26:43.700 --> 00:26:50.600
You know, I also think there is a – previously, PyPI wasn't very reliable.

00:26:51.100 --> 00:26:55.440
You know, we went down fairly regularly, sometimes even in the middle of file downloads.

00:26:55.440 --> 00:27:07.540
And sort of in the past three to five years, I believe it's gotten to a point where you can pip install something and be pretty confident you're going to download something and install it.

00:27:07.540 --> 00:27:15.000
So I think just usability overall has made people more willing to use PyPI than they were in the past.

00:27:15.000 --> 00:27:16.940
Yeah, and kudos to you guys for that, right?

00:27:17.220 --> 00:27:22.080
Yeah, although, I mean, I would admit a vast amount of that has been Fastly.

00:27:22.080 --> 00:27:38.180
I say Fastly is our secret scaling sauce because, you know, we really would not be able to do near the amount of traffic with the skeleton crew we have without offloading a lot of that to the CDN.

00:27:38.180 --> 00:27:38.920
Right.

00:27:38.920 --> 00:27:45.400
You guys don't have, like, a massive data center in San Antonio or something that's, like, in a bunker that you manage.

00:27:45.640 --> 00:27:48.260
You're pushing this to the cloud like all the modern companies, right?

00:27:48.260 --> 00:27:49.100
Yep, yep.

00:27:49.100 --> 00:27:53.100
And we actually have PyPI Legacy runs.

00:27:53.100 --> 00:27:56.460
We have three web nodes for all of that.

00:27:56.460 --> 00:27:59.100
And we have a Heroku database server.

00:27:59.100 --> 00:28:03.100
And, you know, we store our files in S3.

00:28:03.100 --> 00:28:08.100
And that's really about it for the infrastructure for PyPI itself.

00:28:08.100 --> 00:28:09.300
Okay, excellent.

00:28:09.440 --> 00:28:13.680
And can you give us a sense of what it costs to run PyPI?

00:28:13.680 --> 00:28:17.940
Yeah, so for us, the cost is zero.

00:28:17.940 --> 00:28:21.960
But besides the people's time, but, like, right, of course.

00:28:21.960 --> 00:28:27.920
Yeah, because we are lucky to have all these companies donate services.

00:28:28.720 --> 00:28:32.560
I think I clocked it at where somewhere around $35,000 a month.

00:28:32.560 --> 00:28:39.760
You know, some of those numbers are a little fuzzy because, like, Fastly, the billing numbers we have are for all of our use of Fastly and Python.

00:28:39.760 --> 00:28:40.280
Right.

00:28:40.280 --> 00:28:45.240
But you probably represent the vast majority of traffic.

00:28:45.500 --> 00:28:51.340
Yeah, well, like, in May, we said we have 340-some terabytes for PyPI.

00:28:51.340 --> 00:28:55.860
For Fastly in general, we had 399 terabytes.

00:28:55.860 --> 00:29:01.040
You know, and our Fastly bill in May was $33,000.

00:29:01.040 --> 00:29:06.840
You know, we have $6,000-some dollars in Rackspace for all of Python.

00:29:06.840 --> 00:29:09.640
And then we have, you know, DNS and such there.

00:29:10.140 --> 00:29:17.520
So not counting people time, you know, we're roughly $35,000, maybe a little bit more a month for just PyPI.

00:29:17.520 --> 00:29:18.540
Okay.

00:29:18.540 --> 00:29:23.160
This is not something that you could just run if you wanted to, right?

00:29:23.160 --> 00:29:29.480
This takes the support of the community and companies like HP that are really backing it and Fastly.

00:29:29.480 --> 00:29:30.120
Oh, absolutely.

00:29:30.120 --> 00:29:38.040
Because there's no person that's going to go, you know, I really believe in this project, so here's $35,000 a month to pay for bandwidth.

00:29:38.040 --> 00:29:39.660
Yeah, you know, I mean.

00:29:39.660 --> 00:29:41.000
I guess there could be people.

00:29:41.000 --> 00:29:44.700
Maybe you can find, you know, Bill Gates or someone.

00:29:44.700 --> 00:29:46.000
You could have been to them.

00:29:46.000 --> 00:29:49.180
That's almost $400,000 a year.

00:29:49.180 --> 00:29:51.780
That's, you know, that's more than most people make.

00:29:51.780 --> 00:29:53.140
But outside that, yeah.

00:29:53.140 --> 00:29:56.380
Let's talk about the people involved.

00:29:56.380 --> 00:30:06.420
So we know that HP is basically supporting you to do what needs to be done for Python packaging, mostly around PyPI, but in the general sense.

00:30:06.420 --> 00:30:09.100
How many people work on these projects?

00:30:09.180 --> 00:30:16.580
And how many people would you say are responsible for keeping pip install a thing a possibility?

00:30:16.580 --> 00:30:20.740
Ernest Durbin helps a lot with PyPI itself.

00:30:21.320 --> 00:30:22.660
He is not paid for that.

00:30:22.660 --> 00:30:24.660
He's completely volunteer.

00:30:24.660 --> 00:30:27.660
He's sort of the ops side of things.

00:30:27.660 --> 00:30:29.660
I do a little bit of ops.

00:30:29.660 --> 00:30:30.880
I'm not great at it.

00:30:30.880 --> 00:30:35.880
So he does a lot of the operation stuff and is super helpful with that.

00:30:35.880 --> 00:30:39.280
You know, Richard Jones does things on a volunteer basis also.

00:30:39.280 --> 00:30:41.440
He's largely stepped back lately.

00:30:41.440 --> 00:30:44.840
Just because his time has been taken up by other things.

00:30:44.840 --> 00:30:50.580
But he still comes around and helps with support requests and stuff, which I don't have the time to do.

00:30:50.900 --> 00:30:57.740
You know, then we start getting like setup tools, which you need for pip install to do source disks.

00:30:58.300 --> 00:31:03.700
You know, that's largely Jason Combs, Jericho.

00:31:03.700 --> 00:31:05.700
That's largely his baby now.

00:31:05.700 --> 00:31:06.800
You know, then you start talking about pip.

00:31:06.800 --> 00:31:08.700
You know, I'm there as well.

00:31:08.700 --> 00:31:26.660
You know, there's like a Paul Moore and Marcus is there and a few other people, you know, but you know, their time is all limited based on because they're doing it in their spare time, which I do in my spare time as well on top of the HP time.

00:31:26.660 --> 00:31:39.140
But most people outside of myself have a lot more limitations to the time they can spend on it just because they're completely volunteer based.

00:31:39.140 --> 00:31:39.720
Yeah.

00:31:39.720 --> 00:31:43.700
So just a handful of people full time.

00:31:43.700 --> 00:31:44.120
Oh, yeah.

00:31:44.120 --> 00:31:47.880
And everyone else is just a little here and a little there as they can.

00:31:47.880 --> 00:31:48.600
Yeah.

00:31:48.600 --> 00:31:50.840
Do you feel like PIPI is underfunded?

00:31:50.840 --> 00:31:51.860
Yeah.

00:31:51.860 --> 00:31:55.120
You know, I do believe PIPI is underfunded.

00:31:55.120 --> 00:32:00.240
You know, it's sort of the tragedy of the commons.

00:32:00.240 --> 00:32:06.460
You know, it's used by a lot of people, you know, as evidenced by our traffic numbers.

00:32:06.460 --> 00:32:09.400
That or someone's using it a lot.

00:32:09.400 --> 00:32:20.100
While we do get support from, you know, a number of companies, you know, realistically, I forget the exact number off the top of my head, but we're less than 10 companies who support us.

00:32:20.100 --> 00:32:23.760
And I'm pretty sure more than they use PIPI.

00:32:23.760 --> 00:32:26.280
They could be just installing a lot of stuff.

00:32:26.280 --> 00:32:29.600
Of course, I mean, everybody uses it.

00:32:29.600 --> 00:32:33.320
It's absolutely one of the foundational things, right?

00:32:33.320 --> 00:32:40.240
It is the thing that facilitates batteries included in the broad sense of Python, right?

00:32:40.240 --> 00:32:40.820
Yeah.

00:32:40.880 --> 00:32:41.000
Yeah.

00:32:41.000 --> 00:32:43.540
You know, I absolutely believe so.

00:32:43.540 --> 00:32:53.220
You know, I think, you know, it's one of the most important things, if not the most important things provided by the PSF infrastructure.

00:32:53.720 --> 00:32:57.980
It's debatable with, you know, you know, HG or HG instances and stuff.

00:32:57.980 --> 00:33:03.200
I think it's one of the most important things that, you know, the PSF infrastructure does provide.

00:33:03.880 --> 00:33:18.840
You know, and certainly when it goes down, people are very quick to notice that, you know, they, I get notifications through Twitter and email and IRC before our monitoring even notices it's down.

00:33:19.280 --> 00:33:20.520
People are telling me it's down.

00:33:20.520 --> 00:33:22.660
A lot of people depend on it.

00:33:22.660 --> 00:33:25.160
Is it stressful to you to be responsible for it?

00:33:26.200 --> 00:33:27.780
So there's definitely stress.

00:33:27.780 --> 00:33:30.720
Over the years, I've gotten better at dealing with it.

00:33:30.720 --> 00:33:36.540
You know, a lot of people, it's one of those things where, you know, when it's working, people don't think about it.

00:33:36.540 --> 00:33:42.160
And when it's not working, they're out in droves to tell you it's not working.

00:33:42.160 --> 00:33:48.200
You know, and I've never actually been technically on call for PyPI.

00:33:48.200 --> 00:33:53.200
Ernest has been, Nova Kanchowitz has been, been technically on call.

00:33:53.540 --> 00:34:03.120
But I say, you know, it's basically impossible for me to not be on call unless I completely unplug myself from the internet.

00:34:03.120 --> 00:34:07.780
You have to go into the forest and not look for sky riding airplanes.

00:34:07.780 --> 00:34:20.020
Yeah, because, you know, I'm so publicly known and associated with PyPI that, you know, people are reaching out to me as soon as it's down.

00:34:20.240 --> 00:34:24.300
And to their credit, they're mostly trying to be helpful to let me know it's down.

00:34:24.300 --> 00:34:31.580
But, you know, so it becomes a flood of communication anytime it suffers serious downtime.

00:34:31.580 --> 00:34:38.780
Now we've sort of engineered it so that pip install very, very rarely goes down.

00:34:38.780 --> 00:34:43.300
But uploads and stuff kind of go down more often than that.

00:34:43.300 --> 00:34:46.020
But that affects a much smaller percentage of people.

00:34:46.020 --> 00:34:48.200
So that doesn't get as bad.

00:34:48.200 --> 00:34:49.340
Absolutely.

00:34:49.340 --> 00:35:07.620
I'm really excited to tell you about a new sponsor of the show, Rollbar.

00:35:07.620 --> 00:35:10.800
One of the frustrating things about being a developer is dealing with errors.

00:35:11.660 --> 00:35:18.560
Relying on users to report errors, digging through log files, trying to debug issues, or a million alerts just flooding your inbox and ruining your day.

00:35:18.560 --> 00:35:25.400
With Rollbar's full stack error monitoring, you get the context, insights, and control you need to find and fix bugs faster.

00:35:25.400 --> 00:35:26.980
It's easy to install.

00:35:26.980 --> 00:35:30.600
Start tracking production errors and deployments in eight minutes or less.

00:35:30.600 --> 00:35:34.940
Rollbar works with all the major languages and framework, including the Python ones.

00:35:35.140 --> 00:35:40.520
Django, Flask, Pyramid, as well as Ruby, JavaScript, Node, iOS, and Android.

00:35:40.520 --> 00:35:43.420
You can integrate Rollbar into your existing workflow.

00:35:43.420 --> 00:35:50.300
Send error alerts to Slack or HipChat, or automatically create new issues in Jira, Pivotal Tracker, and lots more.

00:35:50.300 --> 00:35:53.320
We have a special offer for Talk Python listeners.

00:35:53.700 --> 00:35:58.920
Visit Rollbar.com slash Talk Python To Me and get the bootstrap plan for free for 90 days.

00:35:58.920 --> 00:36:01.900
That's 300,000 errors tracked for free.

00:36:01.900 --> 00:36:05.400
But hey, just between you and me, I really hope you don't need that many errors.

00:36:05.400 --> 00:36:12.020
Rollbar is loved by developers at awesome companies like Heroku, Twilio, Kayak, Instacart, Zendesk, Twitch, and more.

00:36:12.020 --> 00:36:13.740
Give them a try today.

00:36:13.740 --> 00:36:16.440
Go to Rollbar.com slash Talk Python To Me.

00:36:16.440 --> 00:36:31.880
If we had more people working on it and more people whose time was more seriously dedicated to it like yours is,

00:36:31.880 --> 00:36:35.300
we could probably do some amazing stuff, right?

00:36:35.300 --> 00:36:38.560
So there's a way to donate to this.

00:36:39.100 --> 00:36:47.240
You could actually have a donate.pypi.io as like the landing domain, I guess you would call it, because it redirects.

00:36:47.240 --> 00:36:50.480
But that allows people to donate.

00:36:50.480 --> 00:36:58.700
Like how if people are working for a company and they're like, yeah, our whole company does $100 million of revenue, depends upon this thing.

00:36:58.700 --> 00:37:01.740
And we've actually not even helped support it.

00:37:01.740 --> 00:37:02.440
Maybe we should.

00:37:02.440 --> 00:37:04.040
Like how would they get involved?

00:37:04.040 --> 00:37:08.140
Maybe those would be the biggest bang for the bucks if we could get some big companies to step up.

00:37:08.140 --> 00:37:10.180
Yeah, so obviously they can donate.

00:37:10.180 --> 00:37:13.280
If that's the way that they want to get involved, donations are easy.

00:37:13.280 --> 00:37:16.040
And there are tax write-off in the U.S.

00:37:16.040 --> 00:37:25.560
But if they want to donate developer time, one of the easiest things to do would be to contribute to Warehouse,

00:37:25.560 --> 00:37:30.180
which is on GitHub at github.com/PyPA slash Warehouse.

00:37:30.180 --> 00:37:33.580
That is deployed now to PyPI.io.

00:37:33.800 --> 00:37:36.000
That's where you're going to live in the future.

00:37:36.580 --> 00:37:43.600
And, you know, my big push right now is trying to get Warehouse to the point where we can say, okay, this is ready enough.

00:37:43.600 --> 00:37:46.320
Let's start directing people here by default.

00:37:46.860 --> 00:37:49.520
Because PyPI is sort of an old code base.

00:37:49.520 --> 00:37:52.240
It's slowly falling apart.

00:37:52.240 --> 00:37:57.220
Uploads, we get a sort of base 10% error rate on uploads right now.

00:37:57.220 --> 00:37:59.940
You know, it's just falling apart.

00:37:59.940 --> 00:38:07.300
It's sort of a full-time job to keep it from not falling apart, which doesn't give me any time to work on other things.

00:38:07.720 --> 00:38:11.980
All your fingers plugging holes in the dam rather than, like, building the dam.

00:38:11.980 --> 00:38:12.540
Yeah?

00:38:12.540 --> 00:38:13.300
Yeah.

00:38:13.300 --> 00:38:20.860
So I've sort of ignored some of the holes willfully to try and free up time to get Warehouse ready to launch.

00:38:20.860 --> 00:38:28.360
Because, you know, the idea is that hopefully Warehouse will get slotted into place and a lot of things will be better.

00:38:28.360 --> 00:38:30.720
I'm sure it's going to break things for a lot of people.

00:38:30.720 --> 00:38:37.440
But it will become more joyous to work on, and maybe you'll get more contributors using stuff that people understand, right?

00:38:37.440 --> 00:38:38.080
Yeah.

00:38:38.080 --> 00:38:38.380
Yeah.

00:38:38.380 --> 00:38:39.120
Oh, absolutely.

00:38:39.120 --> 00:38:46.700
Like, PyPI Legacy is largely two files, both over 4,000 lines long.

00:38:46.700 --> 00:38:50.260
And I think I'm the only person left that actually understands it.

00:38:50.260 --> 00:38:57.120
Richard hasn't touched it recently enough, and things have changed that he would have to get ramped back up.

00:38:57.120 --> 00:39:06.640
And so far, I've personally had 100% failure rate on getting new people involved in that just because they take one look at the code and then it just kind of didn't disappear.

00:39:06.640 --> 00:39:09.760
It's not a fun code base to work on.

00:39:09.760 --> 00:39:12.960
You know, it's particularly because you can't really run it locally.

00:39:13.500 --> 00:39:26.340
So a lot of times, changes are make a change, push it to production, maybe to test PyPI, which is sort of a staging slash sandbox instance we have, depending on what the exact problem is.

00:39:26.340 --> 00:39:29.740
And then just sort of pray and who the century is and start yelling at us.

00:39:29.740 --> 00:39:30.580
Yep.

00:39:30.580 --> 00:39:31.940
All right.

00:39:31.940 --> 00:39:35.540
So Warehouse is a huge topic that people are interested in.

00:39:35.540 --> 00:39:36.340
I want to get to that.

00:39:36.340 --> 00:39:41.240
But before we move off this topic, you also heard another blog post called A Year of PyPI Downloads.

00:39:41.240 --> 00:39:45.800
And there was a lot of insight that could be gleamed from those numbers.

00:39:45.800 --> 00:39:47.860
Can you talk about that really quick?

00:39:47.860 --> 00:39:48.460
Yeah.

00:39:48.460 --> 00:39:57.720
So sort of in general, in January of 2014, we started saving and archiving all of our download logs.

00:39:59.000 --> 00:40:15.360
In 2014 and again in 2015, I believe it was, I sort of pulled those all down and crunched some numbers on those and pulled out some what I thought were interesting numbers largely around, you know, what versions of Python are being used, you know, what versions of packaging tools are being used, things like that.

00:40:15.360 --> 00:40:19.500
And those blog posts were sort of very widely received.

00:40:20.240 --> 00:40:35.440
So what I've done in the mean since then is I've sort of improved our metrics stream to the point where now every single download generates a row in a BigQuery database.

00:40:35.440 --> 00:40:45.040
And, you know, which is got all sorts of information like, you know, what version of Python downloaded it, you know, what tool downloaded it, what country it came from, stuff like that.

00:40:45.180 --> 00:40:52.440
And as of, let's say, a month ago, that data is now completely public for anyone to go and query as long as they have a Google account.

00:40:52.440 --> 00:40:52.980
Nice.

00:40:52.980 --> 00:40:53.840
Yeah.

00:40:53.840 --> 00:40:56.620
Give me the link to where that is and I'll put it in the show notes.

00:40:56.620 --> 00:40:56.920
Yeah.

00:40:56.920 --> 00:41:03.460
So I'm not really that great at data visualization or pulling information from data.

00:41:03.460 --> 00:41:08.360
You know, I tried to do my best with my year of PyPI downloads posts.

00:41:09.620 --> 00:41:17.580
But I'm hoping that by making that completely public, A, people can do, can make data-driven decisions about what versions of Python they support.

00:41:17.580 --> 00:41:26.560
Because you can very easily, using an SQL-esque language, say what versions of Python are downloading X package over some period of time.

00:41:26.560 --> 00:41:35.880
You know, I'm hoping that people who are actually good at data visualization and pulling meaning from data can take a look at it and come up with some interesting information.

00:41:36.360 --> 00:41:45.940
Particularly maybe as it relates to, you know, correlating that with, you know, GitHub data or Bitbucket data or, you know, some other kind of data.

00:41:45.940 --> 00:41:47.500
Yeah, that sounds really cool.

00:41:47.500 --> 00:41:51.340
And if you give me a link to the details, I'll be sure to put it in the show notes.

00:41:51.340 --> 00:41:52.060
Yep, yep.

00:41:52.060 --> 00:41:54.220
It's not greatly documented right now.

00:41:54.220 --> 00:41:59.160
It's basically a post to Disutil SIG telling people about it.

00:41:59.520 --> 00:42:01.660
The historical data is not in there yet.

00:42:01.660 --> 00:42:07.520
You know, so that data starts, I want to say, March-ish, sometime late March.

00:42:07.520 --> 00:42:13.240
That data starts, I'm backfilling from January of 2016 right now.

00:42:13.240 --> 00:42:17.040
And then I have to backfill back beyond that.

00:42:17.040 --> 00:42:21.260
But that takes a bit more effort because the logs aren't in the correct format.

00:42:21.260 --> 00:42:27.020
So I have to come up with something to munch those logs into the new correct format.

00:42:27.380 --> 00:42:27.760
Yeah, okay.

00:42:27.760 --> 00:42:28.960
Well, that sounds really promising.

00:42:28.960 --> 00:42:34.220
And, you know, there's a lot of data scientists and data visualization professionals that listen to my show.

00:42:34.220 --> 00:42:40.360
And when I put data out, they seem to do amazing stuff with it in a surprisingly short amount of time.

00:42:40.360 --> 00:42:43.100
So maybe someone will come up with something cool for that.

00:42:43.100 --> 00:42:47.880
Let's talk about Warehouse because that's a really interesting project.

00:42:47.880 --> 00:42:50.500
And I sent some messages out on Twitter.

00:42:50.500 --> 00:42:53.380
I said, hey, what should I ask Donald while we're talking?

00:42:53.380 --> 00:42:58.060
And everybody came back with some variation of talking about Warehouse.

00:42:58.060 --> 00:43:01.060
So Warehouse can be found at PyPI.io.

00:43:01.060 --> 00:43:02.680
What is it?

00:43:02.680 --> 00:43:06.900
Yeah, so PyPI.io is the production-ish.

00:43:07.580 --> 00:43:11.460
I say ish because we're not monitoring it or anything.

00:43:11.460 --> 00:43:15.400
Deployment of Warehouse, which is PyPI 2.0.

00:43:15.400 --> 00:43:21.760
It is backed by the same PostgreSQL database, the same S3 instance, et cetera.

00:43:21.760 --> 00:43:27.960
So anything that changes on PyPI changes on Warehouse, and the reverse is true.

00:43:27.960 --> 00:43:28.380
Right.

00:43:28.380 --> 00:43:30.280
So they're watching the same data stores.

00:43:30.640 --> 00:43:34.060
It's more of the front-end and API implementation, right?

00:43:34.060 --> 00:43:34.600
Correct.

00:43:34.600 --> 00:43:43.020
So right now, the read-only portions of it, which is 90-some percent of our traffic, are pretty much done.

00:43:43.020 --> 00:43:43.880
There are a few.

00:43:43.880 --> 00:43:52.100
If you go through the UI, you'll see a few to-do stuff that we have to either finish or comment out before we make it official.

00:43:52.100 --> 00:43:56.460
A lot of the author UI stuff is not done yet.

00:43:56.460 --> 00:43:56.820
Okay.

00:43:56.820 --> 00:44:01.260
The client side, the read-only UI bit is really nice.

00:44:01.260 --> 00:44:07.880
It looks, you know, I feel like I'm going from SourceForge to GitHub equivalent of experience.

00:44:07.880 --> 00:44:11.580
It's much better than current story.

00:44:11.580 --> 00:44:13.960
So I think that's going to be delightful.

00:44:13.960 --> 00:44:14.380
That's cool.

00:44:14.380 --> 00:44:15.220
Yeah, yeah.

00:44:15.220 --> 00:44:24.520
And, you know, I think that was Nicole Harris who's done the design for that, which I think she's done a phenomenal job on that so far.

00:44:24.740 --> 00:44:37.340
You know, particularly going from what we had to what this is, you know, it really does feel like you're jumping forward an era or two, you know, as far as what modern design looks like.

00:44:37.340 --> 00:44:38.340
Oh, yeah, absolutely.

00:44:38.340 --> 00:45:01.420
No, I was just saying, you know, she's put a big focus on, you know, trying to get the usability of PyPI to be a whole lot nicer and better to use, you know, and surfacing the information people care about and, you know, hiding information that people really don't care about or maybe admitting it completely if it's, you know, just confusing information or information that maybe really needs to tend to know.

00:45:01.580 --> 00:45:02.100
Yeah, nice.

00:45:02.100 --> 00:45:07.460
So can you talk a little bit about what frameworks and internals use to create it?

00:45:07.460 --> 00:45:10.800
Yeah, so Warehouse is written in Pyramid.

00:45:10.800 --> 00:45:12.180
Now it is.

00:45:12.180 --> 00:45:23.220
Warehouse has got sort of a sordid history where I've gone from, I started out using just WordZig and making my own framework to then it went to Flask, then it went to Django.

00:45:24.020 --> 00:45:26.100
Now I've finally settled on Pyramid.

00:45:26.100 --> 00:45:28.720
Why do you make those changes?

00:45:28.720 --> 00:45:31.080
Like, what kept moving you along?

00:45:31.080 --> 00:45:40.140
Yeah, so one of the things I wanted to do with Warehouse was to 100% test coverage across the board.

00:45:40.140 --> 00:45:44.200
You know, coming from with PyPI where, you know, we had zero test coverage.

00:45:44.200 --> 00:45:55.060
One of the things that was very painful to me to make any change was, you know, figuring out what the impact was going to be, where I'm going to break code that I thought was unrelated but was really related.

00:45:55.060 --> 00:45:58.160
So I really wanted to make sure we had great test coverage.

00:45:58.160 --> 00:46:05.300
Unfortunately, a lot of the web frameworks out there tend to use a fair amount of globals.

00:46:05.300 --> 00:46:10.980
There's some argument about whether it's threat globals or technically globals or not, but I call them globals too.

00:46:10.980 --> 00:46:11.880
I think they're globals.

00:46:11.880 --> 00:46:16.780
You know, and, you know, that makes it more difficult to test things.

00:46:16.780 --> 00:46:31.680
So I really created my own framework on top of Workazig that was heavily influenced by Gary Bernhardt's boundaries talk where there was very few of these sort of bag of items.

00:46:32.260 --> 00:46:40.320
There wasn't a user class that had, well, that was a data model and it had, you know, all these sorts of things you could do to a user hanging off of it.

00:46:40.320 --> 00:46:44.460
So that makes it hard to test because you have to pass through a huge interface.

00:46:44.460 --> 00:46:46.200
It's sort of like a miniature global.

00:46:46.200 --> 00:46:49.620
They kind of have to pass through this huge interface and actually test things.

00:46:49.620 --> 00:46:55.700
You have to provide an object, implement all those things or else, you know, your tests are not actually really testing things.

00:46:56.080 --> 00:47:03.360
And so we didn't have a ORM, you know, it was using SQL Alchemist expression layer to some degree.

00:47:03.360 --> 00:47:06.640
It was also using just raw SQL in some places.

00:47:07.280 --> 00:47:17.440
And one of the things that we, that I discovered doing that was, A, I was reinventing a lot of things, which, while fun, wasn't necessarily the best use of my time.

00:47:17.440 --> 00:47:30.100
You know, two, it brought us back to the same problem we had with PyPI where other people found it hard to contribute to it because it was using something that, well, it fit my headspace quite well.

00:47:30.100 --> 00:47:32.560
It didn't necessarily fit other people's headspaces.

00:47:33.380 --> 00:47:34.160
And, you know, three.

00:47:34.160 --> 00:47:34.800
Right.

00:47:34.800 --> 00:47:36.320
Nobody had experience with it, right?

00:47:36.320 --> 00:47:37.080
Yeah.

00:47:37.080 --> 00:47:47.260
And, you know, and three, it, a lot of decisions were bottlenecked on me because it was, well, how, how do we do X thing in this completely custom framework?

00:47:47.260 --> 00:47:50.360
Well, nobody knows except Donald because Donald has to invent it.

00:47:50.360 --> 00:47:52.920
And so a lot of things were bottlenecked on me.

00:47:52.920 --> 00:47:58.160
So, you know, me and Richard sort of talked about it and we decided, you know, we need to move to something more standard.

00:47:58.160 --> 00:48:02.340
And I can't recall exactly if it was Flask or Django that came first.

00:48:02.920 --> 00:48:13.280
I know it was ported to Flask and I wasn't really too fond of that because one of Flask's big API things is thread locals as part of the API.

00:48:13.280 --> 00:48:22.420
And my experience with that is it becomes hard to then do anything in Flask without adding increasing amounts of thread locals.

00:48:23.440 --> 00:48:26.540
And so I kind of mixed that because I just didn't like the third locals.

00:48:26.540 --> 00:48:30.980
I went to Django and Django is a great code base.

00:48:30.980 --> 00:48:33.040
You know, I've used Django a lot.

00:48:33.040 --> 00:48:34.240
I just started with Django.

00:48:34.880 --> 00:48:48.280
I started porting it to that and I discovered that the Django ORM is not really powerful enough to handle all of the things that PyPay does in its database that it's sort of accumulated over time.

00:48:48.440 --> 00:48:51.440
We have, you know, we have tables with composite keys.

00:48:51.440 --> 00:48:53.120
We have composite foreign keys.

00:48:53.120 --> 00:48:55.820
We have tables without any primary keys.

00:48:55.820 --> 00:48:58.320
You know, we have, you know, a number of things.

00:48:58.320 --> 00:49:01.980
And, you know, I did get the user things ported over to Django.

00:49:02.380 --> 00:49:10.800
Then I just sort of gave up and said, you know, this is too much work to actually get this to slot it into the shape that Django wants it to be.

00:49:10.800 --> 00:49:16.340
And I said, I'm going to just have to use SQLAlchemy for this, which is another great ORM.

00:49:16.340 --> 00:49:22.060
But then once you sort of throw away the Django ORM, you'll lose a lot of the power of Django.

00:49:22.060 --> 00:49:32.160
And I also wasn't using the Django template language because prior to all of this happening, I worked on a sort of another alternative front end to PyPI called crate.io.

00:49:32.160 --> 00:49:35.760
Back before I was a PyPI administrator, which was written in Django.

00:49:35.760 --> 00:49:38.100
I used the Django template language in that.

00:49:38.100 --> 00:49:44.840
And that became a bottleneck because of how big some of our HTML pages were to list all the packages.

00:49:44.840 --> 00:49:46.600
That became a serious bottleneck.

00:49:46.600 --> 00:49:48.200
So then we're sitting there.

00:49:48.200 --> 00:49:50.680
So we use Jinja2.

00:49:50.680 --> 00:49:54.520
So then I'm sitting there looking, okay, we have Django.

00:49:54.520 --> 00:49:58.400
It's kind of hard to fit our database into Django's ORM.

00:49:58.400 --> 00:49:59.260
So we can't use that.

00:49:59.260 --> 00:50:02.020
The DTL was too slow at the time.

00:50:02.020 --> 00:50:04.360
I don't know if it is now to really use that.

00:50:04.360 --> 00:50:09.680
So we've sort of paired Django down to a glorified request router.

00:50:09.680 --> 00:50:12.900
And we've thrown away a lot of the power of Django.

00:50:12.900 --> 00:50:14.340
There's no third-party apps.

00:50:14.340 --> 00:50:15.040
There's no admin.

00:50:15.040 --> 00:50:17.120
So then I took a look at Pyramid.

00:50:17.120 --> 00:50:20.040
And Pyramid had a lot of the things I liked about Django.

00:50:20.040 --> 00:50:22.960
There was really no thread locals.

00:50:22.960 --> 00:50:27.020
There is a thread local, but it's optional whether you use it or not.

00:50:27.740 --> 00:50:33.920
And, you know, it had enough flexibility in it to sort of use whatever tools we wanted to do.

00:50:33.920 --> 00:50:35.960
So I could bring in SQLAlchemy and use that.

00:50:35.960 --> 00:50:43.320
I could bring in, you know, sort of change things around to sort of suit my purposes better than I could with Django.

00:50:43.700 --> 00:50:47.600
Now, the flip side of that is it doesn't do as much out of the box.

00:50:47.600 --> 00:50:51.080
You have to kind of configure it and make it do what you want to do.

00:50:51.080 --> 00:50:56.960
But, you know, given the long history of PyPI and all the sorts of little weird things that's grown over time,

00:50:57.040 --> 00:51:02.520
that fell a lot nicer to what we needed to do than sort of Django does.

00:51:02.520 --> 00:51:08.720
Because while Django is great, once you sort of get out of the Django workflow,

00:51:08.720 --> 00:51:12.320
you start fighting in the framework a lot more than you do with Pyramid.

00:51:12.320 --> 00:51:13.400
Yeah, okay.

00:51:13.400 --> 00:51:15.520
That kind of makes sense to me.

00:51:15.520 --> 00:51:18.640
The testing part with Flask, that I didn't know about.

00:51:18.640 --> 00:51:20.320
But I can imagine the stuff with Django.

00:51:20.320 --> 00:51:24.180
You know, I think you and I settled on, for very different projects,

00:51:24.180 --> 00:51:27.340
the same technology stack, more or less, like for all my web properties,

00:51:27.340 --> 00:51:29.760
I'm using SQLAlchemy and Pyramid and whatnot.

00:51:29.760 --> 00:51:32.040
So, yeah, very interesting.

00:51:32.040 --> 00:51:34.700
Let's see, we have just a few minutes left,

00:51:34.700 --> 00:51:37.880
and I have a couple of questions from the listeners,

00:51:37.880 --> 00:51:41.200
and then maybe one more thing I wanted to make sure we touch on.

00:51:41.200 --> 00:51:44.560
Mahmoud Hashemi, who's also been a guest on the show,

00:51:44.560 --> 00:51:47.920
asks if real-time download counters are coming back.

00:51:47.920 --> 00:51:49.280
Yeah, they are.

00:51:49.280 --> 00:51:54.060
So, the old metric stack sort of fell apart and died.

00:51:54.060 --> 00:51:56.440
because it was sort of hacked onto the side of PyPI,

00:51:56.440 --> 00:51:58.280
like a lot of what PyPI does has been.

00:51:58.280 --> 00:52:02.580
The new metric stack, which is based around BigQuery and such,

00:52:02.580 --> 00:52:05.820
is sort of designed to allow us to bring that back,

00:52:05.820 --> 00:52:11.260
as well as give us this sort of archival ability to query all sorts of things.

00:52:11.260 --> 00:52:14.660
I disabled them just because they were zero all the time,

00:52:14.660 --> 00:52:18.200
because the thing had broken, and I didn't have time to fix it.

00:52:18.700 --> 00:52:21.300
But we are planning on bringing those back,

00:52:21.300 --> 00:52:26.260
and hopefully they'll actually work this time, and it'd be nice.

00:52:26.260 --> 00:52:28.780
Yeah, would they reappear in the warehouse time frame,

00:52:28.780 --> 00:52:30.760
or would you bring them back to the legacy version?

00:52:30.760 --> 00:52:33.760
It'll probably be in the warehouse time frame.

00:52:33.760 --> 00:52:36.620
Might not be until after launch of warehouse,

00:52:36.620 --> 00:52:41.840
unless someone feels like coming around and figuring out how to do all that beforehand.

00:52:42.340 --> 00:52:44.060
I'm new to BigQuery.

00:52:44.060 --> 00:52:48.420
So far, I really enjoy using it, and it seems to work really great.

00:52:48.420 --> 00:52:56.420
But one of its constraints is that queries can take a couple seconds to a minute or two to run,

00:52:56.420 --> 00:53:00.400
which is fine if you're just looking at it for data archival.

00:53:00.400 --> 00:53:04.380
Not great to do in the middle of a request-response cycle.

00:53:04.380 --> 00:53:05.640
Yeah, that's for sure.

00:53:05.640 --> 00:53:10.820
So we need to do some work around figuring out, okay, how do we take this data that's in BigQuery

00:53:10.820 --> 00:53:18.140
and put it in a format that we can look at in warehouse in the request-response cycle,

00:53:18.140 --> 00:53:22.300
and then just take someone to sit down and figure that out.

00:53:22.300 --> 00:53:23.780
I just haven't had the time to do that.

00:53:23.780 --> 00:53:24.180
Sure.

00:53:24.180 --> 00:53:27.080
So final thing on warehouse.

00:53:27.080 --> 00:53:31.320
Nicola Kentar asks, what's the time frame for shipping that?

00:53:31.320 --> 00:53:33.740
So I've given a few dates before.

00:53:33.740 --> 00:53:35.240
We've passed them all so far.

00:53:35.240 --> 00:53:40.060
We're not historically been great at estimating how long until it's ready to go.

00:53:40.060 --> 00:53:41.920
I think we're pretty close, though.

00:53:41.920 --> 00:53:49.360
I've started – I just recently told people on District Till Sig to start switching their uploads to using warehouse,

00:53:49.360 --> 00:53:55.480
largely because we have a 10% standard failure rate on uploads to legacy.

00:53:55.480 --> 00:53:58.940
So far, everyone who's done that has said it's worked great.

00:53:58.940 --> 00:54:04.780
We just recently committed to Twine, which is a tool to replace setup.py upload.

00:54:04.780 --> 00:54:09.940
We switched in the master branch to default to using warehouse that's not been released yet.

00:54:10.560 --> 00:54:14.380
But I'm hoping that that will be released in the next couple weeks.

00:54:14.380 --> 00:54:21.980
And if that works out great, hopefully we will get Cpythron itself switched to the warehouse for uploads.

00:54:21.980 --> 00:54:28.360
And, you know, then that will hopefully propagate out and people will use that and that will solve a big problem.

00:54:28.360 --> 00:54:34.540
As far as switching the actual, you know, like redirecting the old domain, I think it's soon.

00:54:34.540 --> 00:54:38.240
I don't have a good target for exactly when.

00:54:38.980 --> 00:54:47.560
But hopefully – I'm really, really hoping it'll be in 2016 because I'm tired of the old code base.

00:54:47.560 --> 00:54:51.440
I want it to die a thousand deaths.

00:54:51.440 --> 00:54:53.320
Yeah, I can imagine.

00:54:53.320 --> 00:54:56.420
That sounds like a really cool new version.

00:54:56.420 --> 00:55:01.180
And it sounds like it's going to be great for everybody when it's working and it's the default.

00:55:01.360 --> 00:55:05.680
Certainly, my playing around with it, you know, it seems like a nice place to be.

00:55:05.680 --> 00:55:09.120
So I have a bunch of other questions I'd love to talk to you about.

00:55:09.120 --> 00:55:11.920
But we're just running out of time.

00:55:11.920 --> 00:55:14.820
So maybe we'll have to leave it there.

00:55:14.820 --> 00:55:20.160
Maybe when you guys do ship it, when it flips, maybe we can come back and do some kind of celebratory show.

00:55:20.160 --> 00:55:20.720
Sure.

00:55:20.960 --> 00:55:24.420
To celebrate the actual flipping of the DNS or the redirect.

00:55:24.420 --> 00:55:25.260
Cool.

00:55:25.260 --> 00:55:35.000
So a couple of questions I always ask people before they end at the end of the show is – and I think this question is particularly interesting to you given your relationship.

00:55:35.000 --> 00:55:41.060
But I ask this to all my guests is, you know, there are over 80,000 packages on PyPI these days.

00:55:41.060 --> 00:55:48.880
And like we talked about, there's so many amazing little packages people can grab and make their programs awesome.

00:55:49.060 --> 00:55:54.860
Like what one do you think is amazing you like to call attention to that's maybe not requests that everybody knows?

00:55:54.860 --> 00:55:55.660
You know, something like that.

00:55:55.660 --> 00:55:56.120
Yeah.

00:55:56.120 --> 00:55:59.600
I would have to say Bpython.

00:55:59.600 --> 00:56:04.380
It's sort of an alternative REPL for Python.

00:56:04.380 --> 00:56:06.480
It's got the, you know, syntax highlighting.

00:56:06.480 --> 00:56:11.480
You know, it's got autocomplete as you type things out.

00:56:11.480 --> 00:56:19.200
It really works well and I install it on all my virtual M's because it just works a lot nicer than the building one, I think.

00:56:19.200 --> 00:56:20.960
I use it a lot.

00:56:20.960 --> 00:56:21.380
Okay.

00:56:21.380 --> 00:56:21.960
That's awesome.

00:56:21.960 --> 00:56:25.180
And when you write Python code, what editor do you use?

00:56:25.180 --> 00:56:27.120
So lately I've been using Atom.

00:56:27.120 --> 00:56:30.000
I've been trying it out for the past two or three months.

00:56:30.000 --> 00:56:31.720
So far I've liked it.

00:56:31.780 --> 00:56:34.500
You know, previously I was using Sublime Text 3.

00:56:34.500 --> 00:56:36.420
Lately it's been Atom.

00:56:36.420 --> 00:56:37.160
Okay, cool.

00:56:37.160 --> 00:56:37.380
Yeah.

00:56:37.380 --> 00:56:41.520
It seems like Sublime and Atom are working at the same level.

00:56:41.520 --> 00:56:45.400
They have kind of a similar, they appeal to a similar group of people.

00:56:45.400 --> 00:56:46.140
That's cool.

00:56:46.140 --> 00:56:47.140
All right.

00:56:47.140 --> 00:56:49.840
Any final call to action while you've got the mic?

00:56:50.140 --> 00:57:03.140
Yeah, you know, I just, you know, I would love it if anyone who uses PyPI could come and, you know, contribute to that or to pip or, you know, talk to your companies, see about contributing to developer resources or even just some money.

00:57:03.140 --> 00:57:04.540
Anything helps.

00:57:04.540 --> 00:57:12.640
And, you know, hopefully we can keep moving things forward and everyone will be happy and stop yelling at me when things go down.

00:57:12.640 --> 00:57:14.480
That would be amazing.

00:57:14.480 --> 00:57:15.760
I just want to second that as well.

00:57:15.760 --> 00:57:22.620
Like, try to convince your companies, if they depend heavily on Python, to contribute just a little bit.

00:57:22.620 --> 00:57:26.940
Because imagine a world where pip install basically didn't work.

00:57:26.940 --> 00:57:28.400
You had to go piece that all back together.

00:57:28.400 --> 00:57:31.320
That's something we don't want to see happen.

00:57:31.320 --> 00:57:37.600
And it would be really great if we could make it a much more stable, supported, active thing instead of putting all the weight on you, Donald.

00:57:37.600 --> 00:57:40.360
And the few other guys we mentioned, right?

00:57:40.360 --> 00:57:40.680
Yep.

00:57:40.680 --> 00:57:45.700
And remember, donations are tax deductible in the U.S.

00:57:45.860 --> 00:57:46.140
Awesome.

00:57:46.140 --> 00:57:51.820
So there's just so many more things we could talk about, but we're going to have to leave it here just for the sake of time.

00:57:51.820 --> 00:57:53.140
So, Donald, thanks for being on the show.

00:57:53.140 --> 00:57:53.880
It was great to talk to you.

00:57:53.880 --> 00:57:54.140
Yep.

00:57:54.140 --> 00:57:54.980
Thanks for having me.

00:57:54.980 --> 00:57:55.220
Yeah.

00:57:55.220 --> 00:57:55.520
Bye-bye.

00:57:56.960 --> 00:57:59.760
This has been another episode of Talk Python To Me.

00:57:59.760 --> 00:58:04.560
Today's guest was Donald Stuffed, and this episode has been sponsored by SnapCI and Rollbar.

00:58:04.560 --> 00:58:06.320
Thank you both for supporting the show.

00:58:06.320 --> 00:58:09.700
SnapCI is modern, continuous integration and delivery.

00:58:09.700 --> 00:58:15.560
Build, test, and deploy your code directly from GitHub, all in your browser with debugging, Docker, and parallels included.

00:58:15.560 --> 00:58:18.640
Try them for free at snap.ci slash talkpython.

00:58:19.740 --> 00:58:21.620
Rollbar takes the pain out of errors.

00:58:21.620 --> 00:58:28.760
They give you the context and insight you need to quickly locate errors that might have otherwise gone unnoticed, until your users complain to you, of course.

00:58:28.760 --> 00:58:32.960
As Talk Python To Me listeners, you can track a ridiculous number of errors for free.

00:58:32.960 --> 00:58:36.340
Just go to rollbar.com slash talkpython to me to get started.

00:58:37.180 --> 00:58:39.240
Are you or a colleague trying to learn Python?

00:58:39.240 --> 00:58:43.980
Have you tried books and videos that left you bored by just covering topics point by point?

00:58:43.980 --> 00:58:51.980
Well, check out my online course, Python Jumpstart by Building 10 Apps, at talkpython.fm/course, to experience a more engaging way to learn Python.

00:58:51.980 --> 00:58:57.820
You can find the links from this episode at talkpython.fm/episodes slash show slash 64.

00:58:57.820 --> 00:59:00.220
Be sure to subscribe to the show.

00:59:00.220 --> 00:59:02.420
Open your favorite podcatcher and search for Python.

00:59:02.420 --> 00:59:03.660
We should be right at the top.

00:59:04.040 --> 00:59:12.940
You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm.

00:59:12.940 --> 00:59:17.660
Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.

00:59:17.660 --> 00:59:21.700
You can hear the entire song at talkpython.fm/music.

00:59:21.700 --> 00:59:23.400
This is your host, Michael Kennedy.

00:59:23.400 --> 00:59:24.680
Thanks so much for listening.

00:59:24.680 --> 00:59:25.860
I really appreciate it.

00:59:25.860 --> 00:59:28.000
Smix, let's get out of here.

00:59:28.000 --> 00:59:31.860
Stating with my voice, there's no norm that I can feel within.

00:59:31.860 --> 00:59:33.100
Haven't been sleeping.

00:59:33.100 --> 00:59:34.720
I've been using lots of rest.

00:59:34.720 --> 00:59:37.600
I'll pass the mic back to who rocked it best.

00:59:37.600 --> 00:59:41.160
I'll pass the mic back to who rocked it best.

00:59:41.160 --> 00:59:43.860
I'll pass the mic back to who rocked it best.

00:59:43.860 --> 00:59:45.180
I'll pass the mic back to who rocked it best.

00:59:45.180 --> 00:59:46.900
I'll pass the mic back to who rocked it best.

00:59:46.900 --> 00:59:49.080
I'll pass the mic back to who rocked it best.

00:59:49.080 --> 00:59:49.160
I'll pass the mic back to who rocked it best.

00:59:49.160 --> 00:59:49.860
I'll pass the mic back to who rocked it best.

00:59:49.860 --> 00:59:50.860
Thank you.