What if we distributed CPython, the runtime, in the same way we distribute Python packages,
as pre-built binary wheels that only need to be downloaded and unzipped to run? For starters,
that'd mean we could ship and deploy Python apps without worrying whether Python itself is
available or up-to-date on the platform. Nathaniel Smith has just proposed a PEP to do just that,
PEP 7.11, and we'll dive into it with him next. This is Talk Python to Me, episode 412,
recorded April 18th, 2023.
Welcome to Talk Python to Me, a weekly podcast on Python. This is your host, Michael Kennedy.
Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython,
both on fosstodon.org. Be careful with impersonating accounts on other instances. There are many.
Keep up with the show and listen to over seven years of past episodes at talkpython.fm.
We've started streaming most of our episodes live on YouTube. Subscribe to our YouTube channel over at
Talk Python.fm slash YouTube to get notified about upcoming shows and be part of that episode.
This episode is brought to you by Sentry and us over at Talk Python Training. Please check out what
we're both offering during our segments. It really helps support the show. Nathaniel, welcome back to
Talk Python to Me.
How's it going?
It's going real well. It's going real well. We're on the eve of the eve of PyCon. How about that?
Eve square? Okay. Yeah.
I don't know how many eve. Maybe it's the eve to the third, but we're very near PyCon. I'm pretty
excited.
Yeah. Anti-penultimate eve.
I don't know.
The penultimate eve, perhaps? Yeah, exactly.
I suspect a lot of people will be listening to this show on their way to PyCon. So if you are,
awesome. Come say hello to me. I'm going to be doing some live shows, some Ask Me Anything,
some various other things. Are you going to be at PyCon this year?
I'm not. I'm not, unfortunately.
So they're going to have to just shoot you a message on Twitter or
on Mastodon or something like that, right?
Yeah. I'm easy to find. GitHub, email, whatever. Yeah.
Cool. Well, everyone going to PyCon, hope you have a great time and do come say hi. And with that,
you know, we're going to be talking about this project, this new PEP about distributing Python
itself, kind of like you distribute Python packages, but a little bit more. Why not? I mean,
it seems pretty reasonable to me. And I'm super, super excited to see work in this area because
Python is so strong in so many areas. And there's just a couple of like really big gaps that other
technologies have nailed so well. Two of them that I see that are super significant is like,
hey, Michael, I want to build a mobile app. How do I do that in Python? Or I want to build a desktop
app. How do you do that in Python? Like, I'm not sure. I'm not sure you should even think about
doing like desktop, maybe mobile. I mean, Kivy is great, but it's not like a general purpose
UI toolkit. And so that's the one. The other is, hey, I have an application. I want to give it to
someone who is not a developer and have them run it. Right. And there are some tools that address
that. But one of them is just like, oh, how do they get Python at all? And your project, your pep,
and some of the ideas relate specifically to how do we make it easier to get a prebuilt
non-admin, not install for the whole system, Python for somebody so they can run an app or
even for developers, right? Yeah. I mean, in fact, I mean, I'm a developer, so that's kind of in some
ways the original use case, you know, scratch your initial app, right? Yeah. But yeah, I mean,
it's just, it's a very general capability, I think, once you have it. So yeah, I mean,
the motivation there is basically like, you know, there are lots and lots of ways to get Python,
right? You can get it from, you know, the Windows store, it comes with on your pre-installed on your
Mac, but not a very good one. Yeah, not quite, sort of. There's, you know, but there's also Homebrew
or Pyenv or your Linux distro has it. And you can, you know, get it through Conda. And if you download
Blender, oh, there's secretly a Python inside, you know, like, it's just, you know, there's just so
many different ways to do it. And that's great. You know, like, it's good to have all these options.
They also are different use cases, but it's sort of silly that, you know, it's obviously it's,
you know, it's flexible. There's lots of ways to do it, but there's no way to just be like, okay,
like, I just want a zip file that has Python in it. And it's like a standard way that's like supported
and, you know, kind of, we all use, so we can call sort of share benefits and improvements and all of
that. So that's kind of the key, you know, the PEP is not that innovative in terms of what it's
actually, you know, doing, right? It's a zip file with Python in it. But it's just sort of trying to
do kind of the logistics of like, okay, but like, let's all agree on how we're going to put it on PyPI.
Let's, you know, have tags and stuff. So tools can figure out what they're looking at and do stuff
automatically. And I think that unlocks a lot of use cases, just that one simple change.
I think it does as well. I mean, your abstract is one of the more concise ones, I would say.
Tell people about the abstract here.
Yeah, the abstract on the PEP is, it's, quote, like wheels, but instead of a prebuilt Python
package, it's a prebuilt Python interpreter. That is the full abstract. I figured that basically,
you know, tells people what they need to know.
Yeah. So the idea is kind of like you would say pip install request, you might say pip install
Python 3.11, except for that you can't use the word pip because pip is built on Python. And so
you need Python to run. I mean, it's a little bit circular there. So you kind of need something
outside of Python. But, you know, conceptually, it's, I have these things I need to run my app,
I need request SQLAlchemy, and beautiful soup. I also need Python 3.11. So those are my dependencies.
Give me that, right?
And I mean, you could even imagine potentially pip install Python 3.11 working, I suppose,
like, you would need a Python, some Python to run pip. But once you have that, then it's probably
still convenient to be able to say, okay, actually, shoot, so I got this bug report saying in 3.11.2
specifically, there's some issue, and I'm not sure if I can reproduce it, you know, like,
being able to just grab that in one command, pretty useful. But that said, you know, yeah, it's
not necessarily, pip isn't necessarily the target. I've been working on some stuff there as well.
I don't know if you want to get it together.
Yeah, we'll definitely get into it. We'll definitely get into it. I think,
you know, you need something a little bit on the outside. And I think ideally doesn't depend on
Python being on the system, because that it would be perfectly useful for it to depend on Python.
And this gives you a different version, this gives you a way to quickly toggle between these versions
and these different setups. But if you could omit that dependency on Python, then all of a sudden,
you give a way to give it to people who are not developers and use cases where it's not just I
already have Python and I need to zoom in. Maybe you're a developer, but you're not a Python developer.
Should you have to manage your own Python installation so that you can use, you know,
something that needs Python to run against your source code, right? That is not Python. So there's
a lot of scenarios where I think it gets unlocked if you use a different foundation.
Yeah, I mean, some of the one of the audiences definitely have in mind here is like, you know,
people with like taking their very first ever programming course on the first day of class.
Like right now, it's pretty awkward that you're like, okay, well, first, you have to go to Python.org
and then click through here and click there and download that. Oh, wait, no, not that version.
Oh, did you forget to check to put in your path? Oh, dear. Hold on.
Yeah. And oh, no, do you use the Pylauncher? Are you on Windows? Are you like, it was just,
it's this extra fiddliness. And you know, it's funny, we've spent all this effort
in the last few years kind of getting wheels to the point where they can just work, right? You can
just, you know, pip install NumPy and it works everywhere and, you know, stuff like that. But
Python itself isn't there yet. Another use case, I've sort of maybe my primary use case,
sort of audience I have in mind is, right, I develop Python packages and as open source and
distributed on PyPI, stuff like Trio. And so I have the problem of, you know, I want to be
welcoming to new contributors, right? I want to, you know, bring them in, get them started first,
quickly. They're volunteers. So like, I don't want them like faffing around and struggling and getting
stuck trying to just like run the tests or anything like that, right? Like both, that's just a waste of
their time. It's like kind of rude and inconsiderate. And it's also, you know, there's likely they'll just
give up, you know, if it's just a casual, like, you know, they aren't really invested yet. It's just
something they're doing for fun or out of interest. You know, I really want to, you know, make that easy
for them. And so part of the vision here is like, you know, be able to say, oh, yeah, so you check out
Trio, you know, here, you type this git clone command, and then you, you know, you have, you know,
this, some kind of Python management tool installed. And you type that tool, you know, run tests,
and it makes sure you get the right version of Python and set up the environment correctly. And then
it executes it for you. And it knows what tests are, how to run the tests in this project, because it
looks at, I don't know, pyproject.toml or whatever, right? And so just sort of capturing all, you know,
we have all those pieces, we don't really have anything that kind of brings them all together
into that, like, just type one thing. That's it. It's going, you know, and it just works.
We don't have many, we honestly don't have many tools that are outside looking in, in Python. So much of our
tooling and our infrastructure is you have Python. Now, now you use the tools, now you install it,
now you install black or, or rough or other. Yeah, there's just this kind of very old assumption,
which I mean, it made sense, like 10, 15, 20 years ago, where sort of everything was sort of
installed manually. And of course, you have, you're going to go through some work to like get set up
with Python. It's the foundation of your whole environment. And then, you know, kind of we add
stuff to make it easier on top of that, but kind of, I think it's time to kind of go back and re,
reevaluate that sort of foundational assumption. Yeah, absolutely. So you mentioned Trio, I know
before we dive too much further in, I'm going to give you a chance to kind of, let folks know what
you're up to. we'll talk about Trio at the end if we've got time, but you know, what have you
been up to since June 29th, 20, 2018, you know, five years ago, roughly last time I was on the podcast.
Yeah. wow. That was really early in Trio's life, I guess, actually. so, I mean, I've had
a lot of just, you know, real life, has happened. then I was, you know, sick for a while and trying
to kind of get back on my feet, did some consulting, just started a new job. So that's
kind of, you know, a lot of distractions, but, also, you know, yeah, I think, Trio is still,
uh, you know, I still like it a lot. it's definitely had more influence. Actually,
had I even published the structure concurrency blog post then? I don't remember.
I feel like it sounds familiar to me, even though it's been five years, it does sound familiar. So I
do think so. But what has happened since then certainly is Python has seen some of these ideas and
adopted them, right? Like, like 311s, concurrency stuff. Yeah. So there's been a, like sort of the,
the influence has gone a lot further than I ever expected. both actually in other languages. So
yeah, like Swift and, Kotlin and, you know, have all kinds of adopted ideas from here. Java
apparently is making some big changes, coming up soon with a whole new concurrency setup. And they're
like saying like, yeah, we're basing it on that Nathaniel Smith random blog post. Okay. Yeah.
Okay. Okay. Yeah. It's very flattering. but yeah. and yes, also, in the Python itself,
it's sort of complicated because we, it's sort of this awkward situation where there's a sink.io that's
in the standard library. And then there's my sort of competing thing trio, which, yeah, we guess we
should say trio is an async library for Python. that's, you know, portable, it's sort of, you know,
an alternative to a sink.io. There are some tools to kind of let you use both at once, but, it's not
a library for a sink.io it's its own thing. and so, you know, like, obviously we all wish there was just
one obvious choice. I kind of looked, you know, but async.io is also in this very difficult
position being in the standard library, and being sort of built up over time. And a lot of it
was designed before we even had stuff like a sink await. so there's just a lot of machinery in
there that's kind of already committed to other ways of doing things. And it's very hard to change.
And trio was sort of like, well, look, we have all these modern things and some new ideas coming in,
like structure. Concurrency is a better way to kind of, write your right concurrent programs.
Um, and it was able to set out a clean slate to like, really, you know, do that all from the start
and be much simpler. so that was important, you know, to have it be its own thing just so we
could, you know, work that stuff out. Then there's a question of, okay, now what do we all switch to trio?
Do we move it back into a sink.io? Do they both continue?
That's been a debate for some really popular things. I think that is interesting. A lot of
people say, well, why is library X, which everyone uses, why is that not built into Python? Why do I
need to pip install it? And a lot of times the answer is because making it part of Python will
harm its ability to innovate and change, right? It'll slow it way down.
Yeah. Like there was a whole debate some years ago, but like, you know, like we all know the
HTTP client in Python, you're a lib or whatever. It's just really bad. You should just never use
it. And it's broken a lot of ways. Like it's just, you just don't use it. but we still ship it
because it would be too disruptive to take it out. That's also why we can't change it. There's just
too much code out there depending on all the weird quirks. and we don't want to ship something else
because then it'll end up being your, like URL, you know, five years later. so we, you know,
it was questioning, like, should we put requests in the standard library or URL lib3 or what, you know,
one of these, and it's just, you know, then you can't ship security fixes. You can't improve
your API. You can't, you know, so as we've gotten better at packaging also, it's taken some of the,
um, pressure off the standard library to be all things to all people.
This portion of talk Python to me is brought to you by code cov from century. Have you heard
about code cov? They are the leading code coverage tool on the market, and they just joined century,
the error tracking and performance monitoring company that you know, and love. Code cov is
the all in one code coverage reporting solution for any test suite, giving developers actionable insights
to deploy reliable code with confidence. Code cov is easy to set up. If you are already both a code
cov and century user, GitHub integration is even enabled automatically for you. You'll get coverage
insights directly in your workflows. Code coverage pull request comments allow you to quickly analyze
your PR's coverage and risk without leaving your workflow. It'll reduce the guesswork. You set up
customizable quality gates and let your continuous integration do the rest. And code cov identifies where
tests can help you avoid errors in production through their century integration. If an error does occur,
you'll even see code coverage details directly in your stack traces. So you can see the untested,
partially or fully covered code that may be causing errors to help you fix your tests to avoid similar
errors happening in the future. Get started for free or take advantage of century's promo
pricing where with a century team or business plan, you can get your first five row code cov seats for
just $29 a month. That's a 40% savings. Visit talkpython.fm/century to get started. Remember to use the code
talkpython to let them know you came from us. It really does help support the show. That's talkpython.fm/s-e-n-t-r-y
century and the code talkpython. Thank you to century and code cov for supporting the show.
When the standard library first came into existence, there was no PyPI and there was no package. Like,
it had to come with it because how else you're going to hunt it down on Usenet and on, you know,
base 64 and code it and put it like, what are you going to do?
Yeah. Or I mean, maybe you find it, you know, you download, I don't know, Twisted or something like
from an FTP site. Yeah. Or an FTP site or something like that.
You have to manually unpack it and have to put it in your pipe. Like it was all totally,
yeah, you know, knocking rocks together.
Sharpen sticks.
You hope it's flint and it creates a spark.
Yeah.
So I think even though we're already far down the road and making changes is breaking and doesn't
make sense, it might make sense to ship less in the standard library, quite a bit less and just say,
oh, you're going to pip install some meta package that explodes out some section. I'm going to pip install
the collections area. Boom. And now I got a bunch more potentially. Who knows?
One thing I'd really like to see as a possible sort of future there is moving some of the standard
library into wheels that are installed by default.
Yes, exactly.
So you get, so, you know, that's sort of, it's sort of this halfway house, right? Where, you know,
it's still the case you download Python, install it, they're there. So we don't just like break
everyone in the world who just assumes they're there. But then it kind of gives us that both of,
you know, the long-term, if it's like, we want to get rid of it, it kind of gives us,
or push it out to PyPI or just remove it entirely. Then it gives us kind of a way to do that more
gradually. But also for, you know, libraries like asyncio that are big and complex and really,
you know, would benefit from being able to have their own release cadence and bug fixes and deprecation
cycles and all of that. Then it's like, yeah, it still ships with Python,
Python, but then you can pip upgrade it. You know, you're not stuck with that exact version.
That could only change when a whole new Python release comes out and you have to take all those
changes together at once.
Yeah, I, I've absolutely had this thought and I think it is, it's a really elegant solution because
on one hand, it lets the core developers focus more on the true essence of Python and it lets it be used
in more locations, right? Think PyScript for example, or MicroPython, right? It might be that you can
create a central core that is exactly the same on all of these. You don't have to consider like,
of course, this is what runs, it runs everywhere, but you still get that backwards compatibility and
you get the ability to say, actually, I want the newest version of asyncio because I want this
more high performance background worker or something.
Yeah. Or even just, I mean, for smaller, like, you know, I don't want the newest version of async
IO because like, I don't know if it works, but I want to install in a scratch environment, this like development
version so I can try it out and give them feedback before they, you know, really make the release and
set the API in stone. And again, like right now, like you'd have to go build your own Python and it's
like, it's just kind of a whole thing, right? You can't just do install --pre.
Right. Exactly. It's definitely a more of a barrier for people who are just casually wanting to test stuff
out. You got to be pretty committed to getting Python 312 alpha six or whatever we're at. Right. I don't
know. Yeah. Yeah. Yeah. Indeed. Okay. very cool. Well, maybe we'll come back and dive into tree a little
bit more, but yeah. And so what do you, what are you doing these days? You talked about doing a little
consulting and, well, yeah, well, so I just started at, a new job, just like last week, week before,
I guess. Yeah. It's been less than two weeks. Yeah. and working at Anthropic, which I don't know if
anyone's heard of, it's still somewhat stealthy, but, it's sort of, yeah, I mean, quickly
changing. I don't know the exact status currently. but yeah, so my understanding of the background
here is that, there's actually the team at open AI who trained GPT three, just sort of,
you know, sat down together and thought, you know, decided they really wanted to do more of like a pure
focus on like interpretability and safety. Like, how do you get these models? Like, how do you know what
these models are actually going to do and how do you get them to do what you want instead of kind of,
you know, stuff like making things up or just, you know, we've all kind of got the scene now how
these large language models can go just all over the place, do all kinds of strange things.
and so, and they decided to be in suit for slander, I believe. yes. Yeah.
Somebody in the UK, I think. Yeah. Well, yeah, there's definitely, there's one that just like,
if you ask the model, like, you know, can you give me some advantages of like, you know,
problems with sexual harassment in law schools and it just picks like five real law professors
and makes up stuff like it's really, really bad. Yeah. And it like sites sources that are all made
up. Like it's, you know, like they, you know, they're just, they're very powerful, but also
not well understood or how to like kind of make them useful and safe.
Just a little bit of devil's advocate though. They are incredibly powerful and they are incredibly
capable. And that's, I think part of the dangers you're like, oh my God, it knows this. Oh my gosh.
Yeah. It understood all of that. And I ask it. So the fifth thing it says, you're like, well,
at this point I'm convinced that it really is on. And then maybe that's the made up one. And I think
that's the dangers. Cause it's actually, it's almost an incanny valley. It's close enough to write that
you're like, okay, this thing's right. It knows. Yeah. So yeah. So personally, like I'm still kind
of up in the air on how you like, how impactful they'll be, where the impact will be. Like,
I think it's just a lot of open questions. You haven't bought a farm, like a goat farm in the
woods. Cause you just give them technologies. Okay. But I guess I do have stock options now,
apparently, or I will at some point if they vest. So I guess that's the other route. But no, but anyway,
so I was just saying, so yeah. So, Anthropix is just interesting company where, you know,
you actually get to play with some of those big models internally. They're kind of working on
on releasing products now. but it's also been kind of just a really interesting to kind of
get the sense internally of like, it's really kind of this like research culture, which is appealing
to me with, I'm sort of coming out of academia of a lot of like numerics background. and what's
also interesting is that part of the reason, we kind of, you know, connected is that apparently,
you know, turns out a ton of their internal infrastructure runs on trio. so, they're,
you know, partly hired me to support that, and are actually giving me time to work on open source,
um, paid time. so actually they are, funding this PEP 7.11, you know,
Python binary stuff, though. They don't know it yet. now they do. Yeah. Yeah. Yeah.
If they listen to the podcast, they'll learn on this. That's great. That's really, that's really cool.
It looks like a, an interesting area to be working. I agree that the research oriented places,
they are fun area to work, right? You're not. Yeah. And there's just a lot of flexibility,
kind of, you know, like it's clear this stuff is going to have effects, which effects and how big
and all that. I don't know, but you know, being at ground zeros, you know, it's exciting. It's really
exciting. Yeah. And a lot of chances maybe have some impact. So cool. All right. Let's dive into the PIP,
the PIP, the PEP, the PEP. That is not, it's not quite PIP, but it's kind of like PIP. Okay.
Yeah. Okay. So we talked a little bit about the motivation. we talked a little bit about what
it is. maybe, maybe tell us a bit about the spec. Like what is the PEP actually say?
what is it actually trying to deliver? And we can talk about like the use cases and
Yeah. the tools for it and so on later. Yeah. I mean, and so, like I said, you know,
that the abstract deals like wheels, but it's an interpreter instead of a package.
that's partly just sort of a tagline of like how you use it, but it's actually also a lot about how
the actual spec is written. it's just sort of like, well, you know, we, we've done a ton of
work over the last, you know, five, 10 years. A lot of people have put a lot of work into making wheels
work, right. In terms of like, figure out, okay, how do we, you know, have metadata that's usable to
like keep track of, you know, which packages are installed and their versions and which ones are compatible.
And if for a binary build, like which systems can you put this on and, you know, all the many
Linux work and, just all of that stuff. And it's just like, well, you know, so we have wheels,
we don't need to reinvent the wheel again. so I'm just sort of taking all of that.
so it's just like, okay, it was mostly, it's defined, it's just a Delta against the wheel spec. It's like,
okay, and the wheel spec, you have, you know, this directory for metadata, I have, you know,
that same directory, but I call this calling these Python binaries, pybys, just to have sort of a
short name, you can stick it in a prefix or sorry, in a file extension.
Yeah. P Y B I. Yes. Dot P Y B I. I like it. Yeah. The PYPY, the interpreter and PYPI, the,
package, repository work confusing enough. So I thought I'd add another near homonym to the,
uh, PYPY. It's PYBI. Yeah. Yeah. You know, anyway. but so, yeah, so like, but they look like,
you know, the file names look like wheels, like, you know, be something like, you know,
cPython dash version dash many Linux to 17.py bi. they context looks like wheels. They're
basically just zip files. There's some, you know, instead of a .dist info directory, you have a
pybyinfo directory and it has a metadata file that's in the same format as wheel metadata files with,
for the name and version and, you know, description, all that stuff.
there, there are a few tweaks, basically just what, you know, you need specifically for,
interpreters. so, okay. So like one thing that makes it a lot simpler is that, for an,
there's only one interpreter in a Python environment, right. Whereas wheels are kind of designed to be
flexible and be installed into different kinds of different kinds of Python environments with
different layouts. a PYBI, it's just like, it's just a raw set of files. You unzip it. That's it.
You're done. where wheels, there's like, well, okay, if you, you want to put this in site
packages, so you have to go find that. Whereas this goes in the bin directory. So you have to go
find that and do the special, you know, so that part's just, you know, not relevant. we leave
that out. there's some, slight, you know, we have to support sim links, which wheels don't,
uh, mostly just cause there's never been a big compelling reason. What's that?
Or the windows folks out there, maybe, and others who,
Oh yeah.
Yeah. What the heck are sim links?
Yeah. Okay. Well, so yeah. Sim link is a classic Unix concept, though. Windows does have them too.
Now, I guess where it's like a special magic file that, instead of having like its actual contents,
it just lists says, go look at this other location on the file system for my content.
Right. It's like, like an app shortcut, but yeah.
But like programs, not for UI.
Yeah. Well, like built into the file system.
Yes, exactly. So you try to open it, it goes to the other one.
Yeah. The, the, the operating system automatically opens that other file for you. but you could
also like look at, you know, if you, you can like say like, can you show me the sim link? And like,
it'll tell you about it if you ask, but if you don't, then it just, you know, magically works.
And it's mostly, it's just, it turns out that traditionally, Unix pythons tend to use these,
uh, both for things like, you'll, you know, in the, your bin directory, you'll have the Python
executable. And then you'll also have Python three as a sim link to Python and Python 3.11 as a sim
link to Python three. And so, you know, one to preserve that. and also it turns out on macOS,
um, they have this very specific kind of layout they want with like framework. Like,
I don't really understand it in detail, but like there's sort of like a, how a macOS app is
supposed to be structured and that it turns out to involve sim links. So we just, you know, we just
have to support, that said, I mean, the way we support them is like, it turns out there's a
standard way to put them in zips. So I say, let's do that. You know, like again, really trying to keep
this as boring as possible, you know? I did know. and then the last thing file, that's crazy.
Yeah. It's a, it's true. It's an extension from the info zip folks, but then it's become,
I don't know. Zip's a strange format. It's kind of like an oral
tradition as much as like an actual like specified format, but there's an entire documentary on zip.
And, I believe I came up with it. There's yeah, it's even controversial in its early days.
It's it's nuts, but yeah, it's even won the compression de facto standard these days for the
most. Yeah. It's definitely got trade-offs, but it's just in terms of, it's just really useful.
It's just a thing that everyone, everything could understand. It's just, it's just so compatible and it's
also convenient that you can do random access, unlike some of the alternatives. You can just,
you can pull out one file from the middle if you want to.
The fact that anyone can open it is so much better than it might save one more percent. Yeah, for sure.
Yeah. Cool. Okay. So we've got these, basically the PI by file is the zip file. Is that basically the
entire interpreter just kind of bundled into a zip file? Like what's the deal there?
Yeah. Okay. I mean, it's just literally like, you know, you install Python into a certain directory
and then you take that directory and you put it in a zip file. There's a little bit of tweaking to like
make sure it's self-contained and you can move it in a portable, portable, relocatable, I guess is a
better word. Yeah.
Yeah. So sometimes if you just install Python regularly, it's kind of, we'll have hard coded.
I know that I'm at this particular position on the file system. And so I need to make sure we don't do
that. And also to make it self-contained, it's like the same thing we do with wheels. Like you have to
vendor some libraries, right? If it wants to use read line as a library to, for like in the REPL to be able to
like edit your line as you're typing it, then, you know, we can't just assume it's on the system.
We have to include that inside the PyBuy. So, and, but again, like this is stuff we've all
already dealt with with wheels. There's tools for doing it. We understand how to do it. And I'm just
reusing those tools. So we're not, if I were to run a Python application delivered by one of these PyBuy's,
does it have to unzip the contents into a location and then run it there? Or can it just run it
straight out of memory or what does that work?
Well, so by itself, the format, I mean, it's just a zip file, right? So you can do with it,
what you can do with a zip file, which I mean, is not much on its own. You need some software to work
with it. Right. Now that said, I think, so, yeah. So like, if you just were starting with nothing and
you're like, I just, I have a URL to some PyBuy and I want to use it, then you'd have to download it,
run it on zip tool, and then you'd have a, you could go into that directory. It's a Python environment.
You know, you could run pip in there or whatever. That said, I think this is a really useful building
block for tools that want to go beyond that. So things like delivering a prebuilt application that
you can just run without unpacking. Like there are various tools to do that, like PyOxidizer,
PyToApp. I don't know. There's a ton of them actually. I'm probably forgetting like 10 more.
Yeah. The ones that come to mind for me are PyToApp, PyInstaller, and PyOxidizer for sure.
Yeah.
PyOxidizer being the newest of them.
Yes. Oxidizer because it involves Rust somehow.
Because all the things that involve Rust.
Yes. So, but yeah, so those tools that it's really useful to be able to say, okay,
like I'm going to do some clever thing to like set up, I don't know, a self-extracting executable or
whatever it is they do for their distribution mechanism. I'm going to create an installer
program. I'm going to, whatever it is, but you still need an actual Python to put into that.
Right. And so having a straightforward way where it's okay, that's not their problem anymore to
figure out how to find a Python and get it built and working for the target system.
So they can just say, okay, there's, I can just like grab, you know, okay, yeah, you want to target,
you know, many Linux. Cool. I'll just go grab the right Python. It's already there. I know it works.
And now I can take the files out of this PyBI and do whatever I want with them. I can pack them
into my installer or do clever things to make them usable out of memory or whatever. And they can focus
on that part instead of just the, like, how do you even get a Python? Yeah. Or how do you once
find yourself in the wrong Python, get the right Python. Yeah. Yeah. Yeah. That's a, I don't know if
that's trickier or less tricky, right? It's one thing to say, dear user, go get Python. You need
that. It's another thing to say, go upgrade your Python and hope you don't break something.
You know, I think. Yeah. Well, but also that's part of the point of these being self-contained is,
um, so, I mean, this is one of the more trivial use cases, right? But right now we all use virtual
ends and mostly that's fine, but also sometimes, you know, they can get, you know, janking stuff
could happen. Like, you know, you're on Linux, you do an app to upgrade and now your system
Python's changed and all the virtual ends that were based on it are broken now.
Yeah. Yeah.
Because it like had some kind of dependence on that exact binary. now I will say you
would always want to do this, but at least it's nice to have the option. You could say,
okay, instead of making virtual ends, I'm just going to make real ends. They're all just going to be,
I'm just going to drop a new copy of Python in each environment. And that way I just,
it's totally self-contained. I know exactly what I have. It upgrades when I decide to upgrade it.
Um, and it's just, you know, it's a nice, it's a nice option, right? Sometimes to have that.
Um, and, and also, you know, it gives you that total isolation, right? So you, you, then we were
just saying about that issue of like, oh, I wanted to use this. So I went and installed, upgraded my
Python. But now that other thing I was already using broke, cause they're using the same Python.
It's like very easy to say, nah, just give them different pythons. You know,
There's not that much that changes over time. That's a backwards breaking sort of thing. I mean,
two to three, but I think that's kind of, let's put that in the past. But that's a whole idea.
I did. Well, but I did recently, I was working with MongoDB using Beanie, which was using
motor, which was using the at async or at co-routine decorator, which was removed in 3.11 or 3.10,
one of those recent upgrades. And it had been deprecated forever. The people at MongoDB said,
we don't care. We're just going to leave it. Who wants to put the word async in front of my method?
That's tricky. I mean, they just probably won't pay attention. And my, my code wouldn't work. I'm
like, why doesn't this work? Oh, the thing I depend on, which the thing it depends on that,
that thing needed less than 3.10 or whatever, 3.10 or 3.11, whatever.
Yeah. Now we're back to it. I think I have struggles to adapt without breaking. Yeah.
Yeah. But yeah, stuff like this would, stuff, it does happen, you know,
and this kind of isolation gives you 100% confidence to say, I'm going to make this new app. We're going to
try running this app on, on this in production, and it's not going to hurt anything. And I don't
need Docker. And you can say, or you could use, say, I'm going to, you know, use this exact point
version on in development. And then I'm going to take that and build it, create my, use that to create
my Docker image. Like I don't need the like prebuilt Docker stuff. I could just grab Python
from PyPI. And I know it's the exact same version. Everyone else is using built by the python.org folks.
Hopefully, you know, we're not there yet, but like, that's kind of the, where we're trying to
a PEP and not something on GitHub, right? Yeah, sure. Well, it has been on GitHub for a while,
but you know, I have time. Yeah. Yeah. So maybe it's, it's worth jumping over that. But before we do
two questions, maybe two top level questions anyway. So this is about the concept of kind of like pip
install Python 3.11 or 10 beta two or whatever, whatever it is. Yeah. Yeah. Does that, is there a way to say,
say, and these three packages off of PyPI? Like, can I take and kind of bring a virtual environment
effectively along with me with what you're doing so far? So PyPI is again, by themselves, I mean,
it's just an archive format, right? In a package format. It doesn't do anything that said,
obviously, yeah, part of what we want is for these to be useful for things like, you know, building
environments that have other packages in them and stuff. so that's the one other thing I didn't,
uh, I forgot to mention about, defining the format, probably the most interesting part actually,
is that we do add some new, static metadata that we put into the package and kind of the motivation
there is that, we're trying, I try to figure out, okay, what do I need to know in order to install
wheels into this Python without running it? Right. Because right now, right. Like pip assumes that it's
running on the Python it's installing into. So anytime it wonders like, you know, okay,
like what ABIs does this Python support and what version is it? What platform am I on? It can just
ask the interpreter it's running on. Right. And it's like, okay, well, it would be really nice if you
didn't have to do that both for like efficiency. Like you want to be able to, you know, figure out which,
you know, have your like installer, your resolver, figure out which versions of everything it wants,
uh, without having to like download and run multiple versions of Python and stuff. Like
you'd really like to avoid that. and it's also things like, I want to build a cross, I want to
build, you know, release, distributions for, you know, macOS, but I'm on windows or vice versa.
Um, or I just want to, you know, I, I'm developing on, I'm developing my package like trio on
Linux personally, but I would like it that when I lock my, you know, version. So I know all my,
you know, collaborators are using the same versions that we figure out locks that also
work on windows and macOS. And I can't just trivial run that all those pythons, you know,
from one resolver, cause it's not running on all three at once. and we, you know, in Python
packaging does have the ability to like have different dependencies on different OSs and,
you know, it gets kind of, you know, can get very kind of complicated to figure out like,
which packages do I need where? and so I want to put a bunch of metadata into the PyBI,
like all the stuff you need to solve those problems. so yeah, so the PyBI itself,
I think normally they won't ship with any packages, maybe again, call back, maybe in the future,
we'll start moving some of the standard library into wheels that are pre-installed. You can do that.
Um, but, I'm guessing like, you know, for now it'll, they'll mostly just be, you know,
a plain vanilla Python install, but then you could take that, you could take some wheels,
bundle them all together into a new archive if you want, or yeah, again, whatever you want to do with
it, stick it in a Docker image, whatever. It's a step towards, but not necessarily trying to propose an
entire solution of here is the interpreter and all the dependencies and the code and just run it as if it
was, it had no, no dependency on your system, just treat it as like a dot exe or a dot app.
I can just double click. Yeah. I mean, it makes that a lot easier than it is. Right. Right. Like
right now, first step is just to figure out like, how do I even build a Python that'll work like that?
And that is like some arcane dark knowledge written on a tome in, you know, black ink on black paper and
a black tomb you have to go find or something like, you know, it's just, yeah, it's not easy. and so
just having the ability to say like, yeah, just grab this file, unzip it, drop some wheels in it,
zip it up again. Now that's a package you can drop, you can hand to someone and it'll work on their
system. You know, like that's, that makes it a lot more accessible. it's not the thing I'm most
personally, like I'm not immediately going to go build that one last extra tool, but I bet someone will.
Yeah. I can imagine someone will, for sure.
This portion of Talk Python To Me is brought to you by us over at Talk Python Training with our courses.
And I want to tell you about a brand new one that I'm super excited about. Python web apps that fly
with CDNs. If you have a Python web app, you want it to go super fast. Static resources turn out to be
a huge portion of that equation. Leveraging a CDN could save you up to 75% of your server load and make
your app way faster for users. And this course is a step-by-step guide on how to do it. And using a
CDN to make your Python apps faster is way easier than you think. So if you've got a Python web app
and you would like to have it scaled out globally, if you'd like to have your users have a much better
experience and maybe even save some money on server hosting and bandwidth, check out this course over at
talkpython.fm/courses. It'll be right up there at the top. And of course, the link will be in your
show notes. Thank you to everyone who's taken one of our courses. It really helps support the podcast.
Now back to the show.
My next question is not on our shared screen here, but is what impact do you think this would have
on PyPI? First of all, do you see PyPI the way the CDN that delivers packages like Trio and wheels
like Trio? Do you see that as the same channel through which CPython 3.11 is delivered?
Yeah. I mean, so I would like these to literally be like you go to PyPI/project/cpython and it says,
like, here's the latest release and you click on downloads and it shows you the, yeah, I'd like it
to just literally be stuff you upload to PyPI. Right. And when you pip and solve from there,
it figures out the platform to pick from and it downloads that wheel and off it goes. Right.
Do you think that that would add like a huge burden to the amount of traffic or do you think it would be okay?
No. I mean, we'll have to see and adapt, but Python itself, it's like, shoot, it's tens of megabytes.
Okay. So there's plenty of other packages. There are a lot of much bigger, you know,
like go look at TensorFlow or something. There are hundreds of megabyte packages on Python that are
very popular. Also, I mean, Python.org downloads go through the same CDN anyway. It's just sort of
different infrastructure on the backend, but it's still fastly serving it and donating the bandwidth.
So, you know, in that regard, I wouldn't expect much change. And also just, you know, like people
tend to install, you know, wheels more often than they install Python. Right. Again, it's hard to know
sort of the second order effects. Like maybe virtual lens will be less popular in favor of full lens
if this takes off. And then people would start installing Python more than they do now, but still,
nonetheless, I don't think it's, you know, a huge, I don't, I wouldn't, I wouldn't anticipate it being a
huge change. And if it, you know, it turns out to be a problem that, you know, we can kind of figure,
address it then, you know. Well, you could also do to a large degree, you could do things like
pip does already that caches, you could just cache the CPython wheel, the PI BI into the user profile.
And the second, third, fourth time you get it is really the CI systems and all the doctors and all
that stuff that don't understand what a cache is or any of those things. Yeah. But then, you know,
so like, if it comes to real problems, then you go to GitHub and you're like, hey, can we work
something out so that, you know, you stick a cache in front of PI PI and, you know, like stuff like
that, like, it's not, you know, trivial, but, you know, you could talk to people and solve problems.
Like, I don't, certainly, I don't think we should, you know, hold back the entire design of like how we
distribute Python and make it available because, oh, maybe it'll be too easy and people will use it too much.
You know, like, that's a good problem to have, right?
Yes, exactly. Look, they're using it. This is terrible.
Yeah. Like first you make it easy and then figure out how to solve any problems that causes don't.
Yeah. I think we've more than once solved the problem of, oh my gosh, they're using it like
Google, Netflix, you name it, you know. Think of the benefit that you'll be doing for all the
developers, especially those who have Python skills and are looking for a job. I mean, if the popularity of
Python by downloads is one thing, if you could like 4x that, we'd all be more demand. Like,
look, really, really downloaded now.
Right. Yeah. Just go out there and just download it five times in every CI job. Just, you know.
Exactly.
Just throw a forum away, but you know.
Just do it a couple of times. Just show.
Right. Yeah.
Awesome. The question that you put into the PEP here on the screen though, is why not just Conda? And I,
not being a particularly data focused person, I definitely prefer using pip over Conda because
especially it seems like a lot of the web packages are not as close to update, up to date. You know,
there's a latency before it hits Conda and it's like immediately on PIP. That said, there's a bunch
of people who are like, I kind of use Conda for this.
Um, yeah. And, and right. If you're just like, look, I don't really care about all this. Like,
I just, you know, I want to run my Jupyter notebooks and I, you know, just need a Python that can do that.
And maybe, you know, some NumPy or whatever. Conda solves that really well. And this thing could,
you know, I'm working on, could also potentially solve that really well. And so it feels duplicative,
um, to those people and to them, it is, you know, it doesn't really, you know,
they're both two solutions that work, but there isn't necessarily reason for them to choose one or the
other. but this could also be a foundation for the way that Conda provides Python to itself.
Maybe, I don't know. Like there's a whole other question about how like we could bring Conda and,
you know, PyPI pip, that kind of world closer together and interoperate better. But it's sort of,
that's a whole like, you know, can of worms, lots of complicated stuff. I don't think,
I don't think, you know, this PEP itself is going to, be the thing that makes a big difference
there. Okay. But it's not an anti-Conda type of thing. No, no. Yeah. Well, and so,
right. And so, I mean, you can also get to see a version of this in the past, but,
like basically the way I think about it is that the key reason why we just like,
why PyPI is a critical piece of infrastructure that, you know, cannot be replaced by anything else
is not because it's, of its use for end users. I mean, it's great that end users use it and find it
helpful and all that, but like, that isn't the people who absolutely need it and could not have
any replacement. the reason why we just absolutely need it is for package developers,
because the way, again, you're talking about all those different ways you can get Python,
right? And there's all these different ways Python packages get distributed, right?
You can brew install packages. There's versions, you know, NumPy, a patch version of NumPy used to be
part of the standard macOS install. Maybe it still is. I don't know. you know, like when you
install Blender, there are Python packages in there, install some game using, was it RenPy? it's
going to have Python packages in there or just, you know, there's just like, there's so many different
ways that Python code goes out of the world and gets used in all these different contexts.
and if you're developing some upstream library, like, you know, trio again, or, but, you know,
or requests or NumPy or anything, then what you absolutely don't want to do is have to maintain
a separate distribution for all of those different things. You don't want to have to upload your
package to conda forge and also to Debian and also to Fedora and also to Blender forge or like,
like you just like, that's not, that doesn't make any sense. Right. so we need this, you know,
and then having every different package maintainer do that, like that just would be terrible, right? We
just would be unworkable. so the critical role that PyPI serves that just nothing else can
is it's this intermediation point between package uploaders and package users and including package
redistributors. and so, and so, you know, like I, you know, make a release of my package,
I upload it to PyPI and then that's where conda forge gets it. That's where Debian gets it. That's
where, you know, end users get it. If they're pulling straight from PyPI, like sort of it's fans out from
there. And the key difference in terms of design between pip and conda is that pip's metadata formats
and you know, wheels and the metadata and sourcetists and all that are designed around this abstraction
of, you have some kind of Python environment, but it could be any of those, right? It could be on
different OSs. It could be different, you know, ways of building it, different layout, different
pieces could be missing, like whatever you could be laid out in all kinds of different ways. I just know
that there is some kind of Python environment and I have the metadata to like figure out how to adapt to
how this particular Python environment is put together. And conda on the other hand is one of
these sort of downstream systems. It's, it can, the reason people like love it and like data science,
right, is because it's a full fledged, like arbitrary application distribution thing, right? You can install
random, you know, C libraries and you know, you can install R and R packages. Like, like it's just,
it's got, you know, compilers that are all there in the one thing. But because of that,
it isn't have this abstraction of, oh, I can handle any Python environment. A conda package of a Python package
is set up to install in a conda Python that's laid out in the way a conda environment is laid out in the way
using the libraries a conda library has, right? And so it isn't have that flexibility. If you just release
something for conda, then it's great for conda, but it's not, you know, usable to Debian and Homebrew and all of those other folks.
And so like, sort of that's the key thing that PyPI does, right? It's like, it has that abstraction
that lets you, you kind of have the Python packaging ecosystem of all those packages and their dependencies
on each other. And then you kind of project it down into each of these more specific, specialized
packaging systems. Right. And then also, because, you know, as a, that's the, that's the other thing,
as a package maintainer, I don't just write my package and upload it. Like I'm also using all by other,
you know, all the other open source maintainers work as I do it, we're all working together. Right.
And I'm depending on their work and they're depending on mine. And so I need to be able to say like,
okay, you know, my package needs those three other packages and here are the versions that I need to be able to
download, you know, create an environment with those versions and test it before I upload my package to PyPI.
And so again, all that work has to happen at that higher abstraction level. Like you can't just say,
I'm going to take the latest version from Conda and test against it. Because that's not necessarily
the version that other packet, you know, other people will get where you take the versions from PyPI.
Those are like the original ones. I can get exactly, I can have access to anything anyone has ever uploaded
as soon as they upload it and I can test them all together. And then, you know, if Conda wants to take
some curated subset of those or whatever, that's great. That's a really valuable service, but you know,
they kind of need that underlying set of packages to curate. And that's what PyPI is for. Yeah.
PyPI is kind of the definitive source of truth as the package creator intended it to be.
Yeah. And then, so, right. And then of course, yeah, for a lot of end users, it turns out they're
just going straight to that without any intermediary works great for them. And that's really cool.
But also, you know, it's not like I don't have anything against people who prefer to go through
Debian or Conda or whatever. I think that's also great, you know, if that works better for you.
But for the folks who are, you know, developing, you know, packages to upload or who just, you know,
would rather just, you know, get stuff straight from the source, the PyBIs, I think, can solve a lot
of problems that Conda, you know, just, it just doesn't address those. It has a different focus.
Trying to make it a swap over to do that might kill a little bit of what it's, it's good for, you know?
Yeah. All right. Let's, let's see. So let's move on to your announcement here. I think.
Okay. Right. Yeah.
So over on discuss.python.org, when was this? This was January 21st.
So a few months ago, you announced PyBI and Posi.
Yes. And Posi is, we talked about this mythical pip that could pip install CPython 3.11.
Posi is that mythical pip, right? Yeah. In a sense.
Yeah. So yeah, the pip 7.11, the PyBI stuff is, is just the one, one brick in my master plan.
Oh. So, right.
Right. Because yeah, because sort of the, this vision I had in mind, I kind of alluded to earlier
talking about like, you know, okay, if somebody does, you know, get clone my project, I want them to,
you'll just run the tests and know that they have the right version of Python and the right version
of the dependencies. And it just kind of, you know, and they know how to run the, you know,
just, just do it. Right. And code all that information somewhere. And Posi is sort of my
experimental, it's not ready to use, but it does have a lot of, a lot of stuff working,
is my attempt to solve that part of the problem. So the vision is Posi is a, it is a full
re-implementation of pip with, you know, the like metadata parsers and result dependency resolvers and
archive installers and all of that. Except I rewrote it all in Rust as is the style. And, but
it also, it's all built around PyBIs. So it is, it doesn't, I mean, maybe at some point,
we'll also start supporting, you know, VNs or your user installed pythons, but, you know, for now,
sort of the, for the MVP, it's just says like, okay, yeah, you have a PyBuy, I will grab that. I will
grab packages that are compatible with it. I will arrange them all to run together based on, you know,
you just, you just say what you need. I turn that into like a lock file. I fetch those packages. I run,
you know, your test script or whatever. And yeah, I mean, that, that, that is the core idea.
Yeah. And one advantage of being in Rust is that it's, you know, just because just, so obviously,
you know, if you want to hack on it, then you need like a Rust compiler and stuff. But if you just want
to use it, then we can, you know, just take a, we can compile it down to a single binary that you just,
you know, upload to wherever, install it from wherever, and you just, you drop it on your system,
you run it and it's self-contained. It can handle everything from there. So the, again,
thinking of that target audience of like beginners, right? You say, okay, install this one program.
And now you type, you know, posy run and oh, look, you're in a REPL. And it's like, and now internally,
of course, I had to go find the latest version of Python and grab it and figure out if, which build is
right for your system and do that. But like, but you know, you don't have to think about that. You just hit
enter and it happens and there's your REPL. Or then you say, you know, posy add requests and then, or posy add
Jupyter and then posy run notebook, you know, like, and it kind of is handling the, the environments behind the
scenes. Yeah. Yeah. We did a panel discussion with a bunch of core developers around packaging recently. And a
lot of them were saying things like, I don't really want to put words in the mouth, but kind of getting the, the
sense that like, okay, so we have a bunch of tools that are really neat that live within Python, you
know, I'm thinking hatch poetry, those that, that category of tools, pip itself, even, and some of the
challenges or problems that they would like to solve, they could unlock a simpler API if it was turned
inside out, right? If the tool itself controlled Python and didn't depend on Python to get started,
they mentioned rust up as a way to get started, which is a way to kind of install
a version of rust and get started. Right. And it feels to me like this is pretty close to that.
Yeah. There's a lot of overlap for sure in terms of sort of the goals and approach and all of that.
Mm-hmm. One challenge I see is so like, for example, to run the application for the,
with the Python that's bundled up inside of one of these PyBIs is you would say posy run or posy,
some kind of file or something like that, right? Sure. Yeah. Whatever. Exactly. Yeah. Yeah. Yeah.
Yeah. Yeah. Yeah. Yeah. Whatever the CLI that's yet to be fully spec'd out comes out to be, but you
know, could you do things like, could you create, you know, speaking of sim links and other types of
stuff, could you create just in the same folder where that app lives, a Python that actually just calls
posey the Python inside instead of Python itself and, and pip that says, you know, posey run pip inside
this Python to kind of bridge, to unify the API from where people are coming from, to kind of
expose the, the same tools that are inside a little bit, you know what I mean?
Well, so yes. I mean, so the way I am currently, sort of in my current prototype, basically,
um, it doesn't work like that just because it felt sort of more complicated to then like try to expose
those things. So, sort of the sense is like, you know, let's see how far we can get with treating
sort of the UI paradigm of like posey is just your front end to Python. Like you don't, you just,
you start your command with posey and that's that you, that's the only command you need to know kind
of. and so, and that also allows some interesting things. So like the way posey
does environments right now is it doesn't, you can have multiple environments within a project.
Like if you need to test against multiple Python versions or you need different installs for,
I don't know, tests and for building your docs and whatever. but it doesn't actually materialize
all those as separate independent virtual environments. Instead, what it does is it,
for like each unique wheel or pyby that it needs, it unpacks that into its own directory. And then on
the fly, it assembles environments by like setting up environment variables so that it can launch a
Python in such a way that it's, you know, it picks the right Python and launches in such a way that it
sees the right packages it's supposed to see. But there's only one copy of those packages on
desk. If you have multiple environments, if you want to like try out, you know,
different versions or whatever, it can just sort of like do that, without having to go
rearrange everything on desk. and that's just, you know, it's convenient. It's just a really nice
way to work with sort of having declarative Python environments. So you never update an environment
in place. There is no environment in place. It's just on each command you run, it sort of knows
declaratively, okay, these are which packages are supposed to be there. I'll give you those packages.
So there isn't even a concept exactly of like pip upgrade or pip install.
Um, you can just say that next time I invoke environment, I'm going to give you a different
specification for which versions I want and it'll make sure that happens.
It feels a little like Docker, right? Like if you create a Docker image and you run a container,
you want to make changes to it. You don't log into the container typically and
in that sense. Right. Yeah. Yeah. But you would just say, okay, well, we changed the Docker fire.
We shut it down and we started back up with the new up better version of itself. Right.
That, that sounds like Docker. Yeah. The main difference, the big, a big difference would be
that your Docker, then when you build it, like the actual Docker file is this big old imperative,
go scribble here and then delete that and then put something else, you know, that kind of thing.
It's not just like, here's the list of things you need.
Um, well, it feels to me like maybe a better solution than what Docker is giving you. If what
you really just want to do is run a Python and isolated Python thing repeatably, because with
Docker, the idea is like, well, you want it, you want it isolated. So let's do this. Let me give you an
entire separate copy of Linux. Right. I know you're running it in like a, not a full VM way, and it's not
as heavy as a VM, but it's still, you're configuring a Linux computer inside of this container in the way that,
whereas this is just like, I just want Python configured, not everything.
Well, and even more like in Docker, if you want to make sure that you run Docker build twice and get
the same package versions, like you have to do that yourself. You have to come up, you have to use,
I don't know, pip compile or something like create a lock file and then install from that instead of
your original requirements. And, you know, it's a whole thing. There's lots of ways to get it wrong.
Whereas in POSY, the way I've written it currently, it's just like, there is, you know, there's one
operation, one internal function that takes a set of like, okay, these are the packages I want.
And it like renders that down into a lock file. Like, here's the exact set of packages you need,
including all the dependencies and all their versions. And then that's the thing that you hand
to the run me an environment. So like, you have to go through that step. It's just built in. And so we
can like, you know, so, and of course, you know, as we build up the, you know, CLI and stuff,
idea will be that that will be that, you know, written to disk, similar to a cargo.lock or poetry.lock
or whatever. And so you just automatically get that reproducibility, which, you know, you don't
get that automatically from Docker, right?
This is a thing that people could go get.
Yeah, on your GitHub profile, they could check this out. And they could they could try it, right?
So yeah, let's see. So yeah, so obviously, there's a lot of moving parts here. If folks want to help
with the PI BI part that PEP 711, there's lots of stuff could use help with. But it is also there's a
draft pap up and I have built lots of PI BI packages for lots of different versions of Python for Mac,
Windows Linux, they're up on a CDN. So you know, if you use the same API as PI PI, so you could pip install
from there if you had a pip that did it. So it's it is stuff you can try out right now, experiment with
at least. And then yeah, as for the posy part, again, like I said, it's still it's mostly the
back end stuff, but it is a pretty complete implementation of like all the packaging stuff.
It had it can actually do that demo. I was just saying of like, I need these three packages in this
version of Python, and it can do the dependency resolution for a named specific operating system,
which may not be the one you're running on, and then generate that environment and actually invoke
it. Well, you can only invoke it if it's for the operating system you're running on, of course.
But you know, it can do all that stuff. There isn't really a UI in front of it yet.
But so it's not like something I'm suggesting you go start, you know, rolling out to your company.
Everyone adopt it. Yeah.
But if this is like an exciting project for you, then you could check it out, see where it's at,
join in, whatever. There's definitely tons and tons of stuff to do. But it's, you know,
there's a good solid start. And I think it's at this point, I'm pretty confident like everything
could work, right? You know, kind of the proving it out part is pretty much there, done.
So question from Marwan in the audience. Hypothetically speaking, does that posy.lock
work as is on different platforms?
Right? Yeah. So cross platform support is a huge issue with locking. I don't know if anyone's ever
tried to do this with like pip compile, like it just doesn't work. If you have anything complicated,
like multiple Python versions, like it just doesn't work. You know, good. They tried hard. But yeah.
So what I'm doing right now in posy is I've tried to kind of keep it simple. I just say like,
tell me which platforms you care about, you know, like, you know, you know, recent dish Linux and
Windows 64 and macOS arm and Intel or something, you'll give me like a list. And then it will,
it can go through just like loop through that. And for each one, find the right PI BI, look up the
metadata to figure out which kinds of packages are appropriate to install there and generate a lock
file for each of those. And then, you know, you can somehow like merge the common parts and write them
to a file. So the individual things that like, you know, resolve this set of versions or set of package
requests into a exact set of versions that only runs at for one specific platform at a time. But you know,
you can run it multiple times. There are, it might be possible to do something smarter. So I know like
poetry has some algorithm that I don't really understand very well, where they try to sort of
simultaneously resolve all the platforms into their lock file. And then the way the lock file works is
then you actually you've, it's like only like mostly resolved. And then when you actually go to install,
it does like that last step to try to narrow it down to the exact platform you're on. And I just,
I don't quite, no one's been able to explain to me at how exactly that algorithm works or even like,
I'm not a hundred percent sure it's even like if it's fully correct or if it's heuristics based or
what. So I don't know, but like, you know, I, we can change the, there's lots of options, right? You
know, we can change the code if there's a better way to do it. Just that's where I'm at so far.
Yeah.
But so there's something that people can play with, but it's early days and you wouldn't mind having
help if people wanted to jump in.
No, for sure. If you want, if you've been looking for an excuse to learn Rust,
if you want to, yeah, play around with, you know, cool. I mean, there's interesting problems
in terms of things like how you efficiently resolve, you know, do package resolution.
It's like this whole messy, like logic programming problem. it's, you know, MP complete.
there's just like interesting system engineering problems of like, okay, if we're going to really
make this a really nice to use, like you want it, like, how do you unpack 20 wheels as fast as possible?
You get to use threads and concurrency and all kinds of stuff. yeah, there's lots of,
lots of cool technical bits too. and of course, just making something that's like a joy to use,
fits nicely in your hand. Lots of fun user interface problems. Yeah. I'm excited about it.
If you can probably tell, like, I just, I just love kinds of problems. So yeah, absolutely. I,
so this announcement was on discuss.python.org and I thought, okay, well, it says there's 72 responses.
Let me flip down and see how this landed with people. Right. Okay. Yeah. Okay.
It was, you know, a wide variety of responses. Yeah. Yeah. Well, but I mean, I would say that at
least the top batch, the first bunch of people, Paul Moore deeply involved with pip jumps in and says,
this is beyond awesome. I had realized you were working actively on this. I'll take a look.
I'd love to help out too. Talks about rest a little. Frederick says, really nice to see this.
This is a great direction. Giannis says, well, certainly blew my mind. Count me in on how we could
explore how this might work for conda and so on. And just, I thought it was really, really quite positive.
of how many, you know, next person. This checks many of the boxes of what I have in mind.
So I, it seems like it's landing well with the community. I hope that, I hope that it continues
to make good progress. Yeah. I think the biggest thing is like, there were definitely some folks
going like, okay, but why are you writing everything in rust? Like, especially it's like, you know,
we've spent a lot of effort, not just making standards for Python packaging, but also like implementing
those. So like you can pip install packaging and like right now, and that is a library that
could do things like unpack wheels and access the pipe. I, or I forget exactly which set,
but like a lot of the tricky stuff, you know, parse Python metadata, formats and, you know,
just all these different tricky things. And it's like, why are you re-implementing this? And also,
does it like send the wrong message that like, you know, when we wanted to do something complicated,
we thought pipe, I will, or sorry, Python wasn't good enough. We needed to switch to rust. And I get
where they're coming from. but well, I mean, there's a few things. So one is just that
I thought writing in rust would be fun. You know, I'm not telling you, you can't use Python for
anything. Sure. I mean, we'll, let's take a step back and say, yeah, how would, how would you propose
writing that in Python? Well, so there is no Python, right? That's no, no, it's possible. Right. So
like, so condos written Python, right. But, and the way, so it makes the distribution a little
complicated because like when you, you get your conda.sh or mini conda, you know, like the installer,
it has a Python package inside it, which it unpacks. So it uses it to run conda to install
like another, another Python or whatever it is that you want to install with conda.
Uh, but it sort of has one built in and you could do the same thing for something like posy.
That's something like pie installer to build an executable.
Exactly. Yeah. But then for my installer effectively, right. Like,
like, just, you know, kind of recursively do that. it's totally, you know, it's totally
something one could do. the, so yeah, but the main reasons, I'm not to go that way is one is,
like I said, like, it's just was more interesting to me. It's one thing. There's also like, you know,
every, language has trade-offs, right. And the exact set of things you want from a package
installer are kind of right in Rust sweet spot and not pythons. So there's sort of
four things that are really important for something, a tool like posy. So it's, there's the initial
install there's, the, how quickly it starts up because like, this is in between you and invoking
Python or whatever it is you actually want to do. there's how quickly you can resolve packages.
Um, and how quickly can it unpack packages? Like, those are the things that, you know, you care about.
Like those are the big load bearing pieces. And those are like kind of four of Python's weakest spots,
honestly. Right. Like, so like we just talked about the deployment part. you can make it work,
but you know, it's not as straightforward as some things. There'd be more possible things,
moving parts, things that could go wrong. It's not the strongest argument, but it is there.
for startup speed, just notoriously when a Python's weak spots, because if it has to like,
do all those imports from scratch every time, something, think tools like Mercurial have struggled
with a lot. you know, lots of Python applications doesn't matter, but you know,
for this particular one, that, that would be a challenge. and then, resolving is big,
heavy, like that MP complete, like just really gnarly burning as many CPUs as you can on complicated,
logical, operations. Again, not Python's strongest point. It's not something you could
like use NumPy for, and it's not like, you know, I'll down or anything. Yeah.
Like you can do something like Cython potentially, using no Gil. Yeah.
But then you're basically writing it in C at that point. Like you're, you're pretty far from core Python.
Yeah. Yeah. And then, and then finally they just, you know, unpacking files. This is totally IO bound.
And so simple that it's actually a big advantage to be able to just like, like, like IO is so fast
these days with like SSDs and everything and NVMe drives that like almost any overhead in the unpacking
path actually is pretty substantial as a relative proportion. So like you, you add like one Python
operation per, you know, you know, a hundred kilobytes written. And that might suddenly be like a 2x
slowdown just because everything else is so fast that even that small amount of Python overhead,
could be large. And, you know, in a tool like this, like people are really sensitive. Like
they really care if it takes 10 seconds versus one second to unpack those, the environment.
Like that's just a huge difference in usability. So I think it's just, it's kind of really an,
like, it just happens to be exact combination things that makes Rust pretty compelling, but you
could do it the other way too. You know, I'm not making a, it isn't meant as a political point.
Yeah. Okay. Got it. All right. Yeah. Python is written in C.
Yeah, sure.
So it's a PYPY, but I mean, the, the, the core, you know, the core bit of it is written in C,
right? No, but yeah. PYPY is written in Python, right? Hence the name.
Yeah, that's true. Yeah. Yeah. C PYPY written in C. It says it right there.
Yeah, exactly. Okay. Interesting. I think we're probably out of time to, to dive much further in
this, but sure. Yeah. I think we covered a lot. I think we covered a lot. I guess.
Anything else you want to give me your thoughts on the future. Like, what do you think? Does,
is the PEP gaining traction, right? This is in draft mode. I don't know how much I emphasized at the
beginning, but it is not an accepted. Ah, yes. That is important to be clear on that. Yeah.
Yeah. It's just, I wrote it. I posted it for feedback. That's as far as it's gone. It, you know,
there's no, there's no commitment on anyone's part to like that. This is what's actually going to
happen. That said, I'm pretty optimistic. Like, like I said, I got a little bit of pushback
on the posy part because of like the rust and whatever, but, I don't think I've gotten,
I can't think of really like any pushback on the PEP 7.11, the actual PYBI part, except people are
like, well, but why aren't you using conda or something? Which, you know, fair question, but you know,
there's an answer and I don't think anyone like, like it's not something that like the people that
you need use agreement, you need to get this accepted or like, you know, the PYPI maintainers
and Python packaging maintainers, and they are totally okay with like conda is not the solution
to everything, obviously. Right. Sure. Well, how complicated would it be to have
bit conda into this particular use case? Right. Yeah. I mean, there's definitely room to collaborate
better there. and I would love to see that in the future, but, but yeah, but my sense is that
there's just really hasn't been a lot of like, people just seem pretty much like, yeah, this is
cool. I guess actually the biggest thing is that there's been, some feedback from folks like the
PY oxidizer folks saying like, Hey, we would like a bit more metadata so we can fully, you know,
to solve some of our other things we want to do. Like we want to be able to cross compile for a given
Python and we need to know a bit more about the target Python in order to do that. so that's just a
very technical, it's like, yeah, okay. More stuff we should add and tweak.
It's not against the idea of evolving it. Yeah. I think the basic idea, generally people seem to be
on board. I'm not going to like, you know, make a commitment to what like Python.org and PYPI
and all they're actually going to do, but my, I don't, I'm a pretty hopeful. I think that, you know,
they're definitely some of the folks involved in like building the Python.org downloads right now
are like, Oh yeah, I'd build one of these if that was standard. Sure. so it is, it isn't all signed off
on and you know, full, but there seems to be a pretty reasonable consensus that like,
this is a good direction that we're interested in moving in. So, well, it sure caught my attention
when I saw it. So, I'm excited to see. Yeah. Well, thanks for having me on to, yeah. Talk about it.
Yeah. You bet. No, let me just ask you real quick, the final two questions since it has been five years
since I asked them of you. Python code. Do you work on this? what editor are you using these days?
Uh, I am using Emacs same as I've been since I was 13. So that's not a political position. That's just, I, you know,
I'm stuck. Like all your commands to come in chords, right? Okay. You know, like that's it. Yeah. Excellent.
And then notable PYPI package. just some random PYPI package. Yeah. Just something you like ran across.
Gosh, that's awesome. People should know about this. It could be very popular or not popular at all.
Oh man. Shoot. I did not prepare for this. Should've. I mean, I don't like, there's some obvious,
like obviously I like, you know, trio has been thinking about a lot, but, that's not an
interesting answer for this. you mentioned rough earlier. I think you mentioned rough,
but rough is pretty cool. Maybe I didn't mention rough. Yeah. Okay. rough is very awesome.
if anyone doesn't know rough is sort of, you know, flake eight, and such re-implemented in rust. So it's like
a hundred times faster. Like you just like instantaneously lends all your code, which is
very sweet. There's a selling point for rust integrated with Python, right? Like
another use case that looks pretty neat. Yeah. And I, you know, sort of, as I'm digging into it,
I'm really impressed at how they, those two, how well they fit together. People put a lot of work
into like making that really smooth and having them collaborate. Well, actually it's something I've just
been working on at work is, we've been having trouble with, so, in an async library like
trio, the, you have lots of tasks running concurrently, but they only like the scheduler
only gets to switch from one task to another when one task explicitly lets go, like says, okay, I can
stop here. We're using an await statement. so it's possible to write code where you accidentally
like don't do that for a long time. And that task will just like hog, all the runtime and block
other tasks from running. And it'd be nice. It's hard to kind of tell when that's happening. and
a similar thing could happen with the GIL. so if you have like an extension library, like, you know,
PyTorch or something, and they forget to drop the GIL before doing some big heavyweight operation,
then it could just block any other threads from running. that's really awkward. We've been
having trouble with that. And, but well, you know, like PySpy, there's another really cool package.
If anyone's seen it, is a rust profiler for Python that can just sit outside your process and,
uh, can, like give you, you know, tell you what it's doing. and, but also it being in rust
and it's up on, you know, crates.io, I could just write a little program that imports, PySpy
is use it as a library, but, and tweak it. So that instead of looking for like, where's code
spending time, it detects, okay, is something hogging, you know, the GIL or the run loop and,
and, and give me the trace back, show me which code is doing that. and it's again, really neat to be
able to, you know, get that really deep insight into this Python stuff that, you know, we're still,
you know, we were using it. It's still Python, but, the rust really is a great flavor that goes with it.
Cool. PySpy. All right. People can check that out. That's
Sampling Profiler for Python programs. Yes. Yeah. PySpy is really cool.
Indeed. All right. Well, okay, cool. Thanks for being here. if people are interested in the pep,
what should they do? I mean, I guess the, the post on discuss.uh, python.org,
is the best place for like feedback. It's also, where I posted about, Posey. So if you want to
like see the discussion or join in, that's a good place. If you want to like, you know, help then yeah.
GitHub.com/njsmith/posy is the repository, you know, jump in, send PRs, file issues, whatever.
Um, or just, you know, send me a, you know, I don't know. Yeah. What's the toot at me, I guess,
the Mastodon version. I'm not really on Twitter these days, but yeah. njs@mastodon.social.
Um, and I'll see it. Cool. All right. Well, Nathaniel, thanks for being here. Thanks for this
pep. It looks interesting. Yeah. Thanks. It's great. Great being here. Yeah, you bet.
That this has been another episode of talk Python to me. Thank you to our sponsors. Be sure to check
out what they're offering. It really helps support the show. Take some stress out of your life. Get
notified immediately about errors and performance issues in your web or mobile applications with
Sentry. Just visit talkpython.fm/sentry and get started for free. And be sure to use the promo code,
talk Python, all one word. Want to level up your Python? We have one of the largest catalogs of
Python video courses over at talk Python. Our content ranges from true beginners to deeply advanced topics
like memory and async. And best of all, there's not a subscription in sight. Check it out for yourself
at training.talkpython.fm. Be sure to subscribe to the show, open your favorite podcast app and search
for Python. We should be right at the top. You can also find the iTunes feed at /itunes, the Google Play
feed at /play and the direct RSS feed at /rss on talkpython.fm. We're live streaming most of our
recordings these days. If you want to be part of the show and have your comments featured on the air,
be sure to subscribe to our YouTube channel at talkpython.fm/youtube. This is your host,
Michael Kennedy. Thanks so much for listening. I really appreciate it. Now get out there and write
some Python code.
it's just a couple of tips on the show. I'll see you next time. I'll see you next time. I'll see you next time.