PEP 711 - Distributing Python Binaries
Episode Deep Dive
Guest introduction and background
Nathaniel Smith is a well-known Python core developer and open-source contributor. He helped create and maintain Trio, an async library built around structured concurrency principles. Over the years, Nathaniel has contributed heavily to the Python ecosystem's packaging discussions and is deeply involved in improving Python’s distribution story. Most recently, he proposed PEP 711, which aims to distribute prebuilt, ready-to-run Python interpreters as easily as regular Python packages. In addition, he is also working on “posy,” a Rust-based toolset that aims to simplify environment management and the handling of these new Python binaries.
What to Know If You're New to Python
Many of the ideas discussed here rely on installing and managing Python and its libraries. If you haven’t worked much with package managers or virtual environments, it’s helpful to know that Python typically installs packages via pip
into different “environments.” Understanding basics like pip install
or how Python searches for installed packages will help you follow along.
Key points and takeaways
- 1) Introducing PEP 711 for Distributing Python Binaries
PEP 711 proposes distributing Python itself as a prebuilt binary, similarly to how wheels deliver Python packages. This approach would let you “install” Python—complete with all needed libraries—by just downloading and unzipping a single file. It aims to make Python more portable, more self-contained, and less reliant on system-level constraints. This could eliminate issues like missing or outdated system interpreters, especially in container or CI environments. Because Python is crucial in so many fields, having a single standard to deliver it anywhere—just like a wheel—would be a massive win for both new and experienced developers.
- Links and Tools:
- 2) Why Self-Contained Python Installations Matter
Many Python developers have struggled with environment collisions and system-wide installs. Self-contained Python distributions let you avoid interfering with system Python or forcing operating-system-level upgrades. The conversation highlighted that having a portable interpreter means fewer conflicts in user environments, especially when you’re juggling multiple Python projects. It also simplifies onboarding, giving beginners or team members a direct, isolated Python environment without manual configuration. Ultimately, this leads to more reliable deployments, since everything is reproducible and consistent.
- Links and Tools:
- 3) posy: A Rust-based Tool for Python Environments
Nathaniel is also developing “posy,” a command-line tool that fetches and configures these new Python binaries. Rather than running on top of Python like
pip
, posy is written in Rust, which provides benefits for speed, concurrency, and distribution as a single binary. It can download the right interpreter for your project, unpack only the necessary components, and help you keep multiple versions side by side. posy also aims to integrate Python package resolution in a more declarative manner—generating a lock file for your entire environment’s Python version and dependencies. While still an early-stage project, posy could become the front-end piece that brings PEP 711’s vision to life.- Links and Tools:
- 4) Comparisons with Conda and Other Packaging Solutions
Listeners may wonder: Why not just use Conda? The episode explains that Conda is excellent at bundling a full data science environment, but it’s also deeply integrated with its own ecosystem of packages and layout assumptions. PEP 711 focuses on delivering a standard binary Python that could be reused by any downstream system—Conda included. Because PyBI (the proposed binary format) is simpler and more universal, it could ultimately act as a foundational piece for many different Python delivery methods. The takeaway is that Conda and PyBI-based workflows solve different problems and can coexist without replacing each other.
- Links and Tools:
- 5) Challenges in Packaging the Entire Standard Library
An interesting thread discussed whether the entire standard library could be shipped independently of the interpreter. Some even questioned if distributing smaller standard library wheels might help with updating or removing outdated modules. However, big structural changes like shipping the standard library separately would require major buy-in from the community. According to Nathaniel, the real step is to start small—just distributing the interpreter. Over time, advanced scenarios might emerge where standard library components also become more modular.
- Links and Tools:
- 6) Potential Impact on Beginners and Teaching
By letting people grab a single PyBI file to run Python code, PEP 711 could significantly streamline the “getting started” story for new developers. The episode highlights how often teachers spend valuable class time dealing with environment setup, installing the correct version, and path issues. Having a prebuilt binary means drastically fewer installation steps—just unzip and run on Mac, Windows, or Linux. Additionally, educators could hand out a fully configured environment to students, lessening the friction for coding assignments and projects. This simpler route benefits experienced teams as well, by reducing overhead across large organizations.
- Links and Tools:
- 7) Python Packaging Ecosystem Evolution
The podcast placed the PyBI concept in the broader context of packaging changes. Python packaging has come a long way, from the old
distutils
days to modern wheels, pip improvements, and advanced dependency resolvers. PEP 711 extends that same principle of consistent, universal binaries to the interpreter itself. This could unify how we share not only Python packages but the entire Python runtime. In effect, PyBI plus tools like posy represent a new generation of solutions aiming to give the developer more control and fewer headaches.- Links and Tools:
- 8) Performance and Efficiency Gains with Rust
Implementing an installer in Rust (like posy) comes with tangible benefits for Python packaging. Rust binaries start up quickly, handle parallel tasks well, and let you distribute a single executable that’s independent of a Python interpreter. This addresses issues around startup overhead for large Python programs that do significant amounts of importing. For example, by using Rust’s concurrency, posy can rapidly unpack multiple wheels or Python binaries in parallel. While Python itself remains an excellent language, the conversation underscores how Rust can complement Python in packaging or tooling scenarios.
- Links and Tools:
- Rust Async Book
- PyOxidizer for packaging Python apps with Rust
- Links and Tools:
- 9) posy’s Approach to Declarative Environments One of the novel aspects of posy is its declarative concept for locking and environment creation. Instead of repeatedly modifying a single environment in place, posy can build new environments from a “lock file” that has pinned versions for Python and each dependency. This approach feels more like Docker’s reproducible containers, but with far less overhead than bundling a whole OS layer. Reproducibility and clarity become the default, reducing confusion for both local development and automated CI pipelines. While still evolving, this direction holds a lot of promise for Python’s future developer workflows.
- 10) Real-World Use Cases and Future Outlook
Nathaniel described potential immediate benefits for open source projects, letting maintainers easily distribute “ready-to-run” development environments to newcomers. He also mentioned how organizations dealing with legacy or locked-down systems could instantly upgrade to the latest Python by just unzipping a PyBI. The conversation touched on expansions like cross-compiling, shipping stripped-down embedded Pythons, or even bridging further with specialized tools (e.g., PyInstaller, PyOxidizer). If the community embraces PEP 711, the entire Python ecosystem could see faster innovation around environment management, distribution, and user onboarding. Overall, the future looks bright if these ideas move forward and become widely adopted.
- Links and Tools:
Interesting quotes and stories
- “The entire idea is basically: Like wheels, but the wheel *is* Python itself.” – Nathaniel Smith, summarizing the abstract of PEP 711 in one sentence.
- “If you have a single file you can just unzip—your problems around system Python vanish.” – A highlight about how PyBIs could revolutionize day-one experience for new or non-Python developers.
- “Tools like posy can truly turn Python inside out. Instead of ‘pip install Python’, we’re controlling Python from the *outside*.” – On how a Rust-based approach might shift the packaging mindset.
Key definitions and terms
- PEP 711 (Python Enhancement Proposal 711): A draft proposal to distribute CPython as a portable binary (PyBI) that can be treated similarly to Python wheels.
- PyBI: A potential new file format that packages a prebuilt Python interpreter in a single zip archive.
- posy: A Rust-based command-line tool designed to fetch and manage PyBIs, while also providing dependency resolution and lock-file-based environment generation.
- Structured Concurrency: A programming model promoted by Nathaniel in the Trio library, ensuring tasks are systematically started and torn down, improving reliability and safety for async code.
- Wheels: The modern standard for distributing binary Python packages. They are
.whl
files that can be quickly installed without requiring local compilation.
Learning resources
- Modern Python Projects: Discover best practices in Python packaging, project structure, and how to handle multi-environment setups.
- Python 3.11: A Guided Tour Through Code: Stay up to date with the newest Python features and improvements—especially helpful if you’re curious about next-gen distribution.
Overall takeaway
Python’s packaging ecosystem continues to evolve to meet developers’ needs, and Nathaniel Smith’s PEP 711 shows how distributing the interpreter like a library could transform the user experience. Tools such as posy illustrate how Rust and Python can harmonize, delivering high-performance, low-overhead management layers. The goal is simpler onboarding for new users and more powerful distribution options for seasoned developers. By aligning Python’s own runtime distribution with modern packaging practices, the community can eliminate many of the installation, environment, and versioning hassles we’ve long faced. If these ideas move forward, they could usher in a new era of flexibility and freedom for anyone building or deploying Python applications.
Links from the show
[announce] Pybi and Posy: discuss.python.org
PEP 711: peps.python.org
Py2App: readthedocs.io
PyInstaller: pyinstaller.org
py-spy: github.com
Anthropic: anthropic.com
Trio: github.com
Trio on Talk Python: talkpython.fm
Zip Documentary: The Dark History of Zip Files: youtube.com
Watch this episode on YouTube: youtube.com
Episode transcripts: talkpython.fm
--- Stay in touch with us ---
Subscribe to Talk Python on YouTube: youtube.com
Talk Python on Bluesky: @talkpython.fm at bsky.app
Talk Python on Mastodon: talkpython
Michael on Bluesky: @mkennedy.codes at bsky.app
Michael on Mastodon: mkennedy
Episode Transcript
Collapse transcript
00:00 What if we distributed CPython, the runtime, in the same way we distribute Python packages,
00:05 as pre-built binary wheels that only need to be downloaded and unzipped to run? For starters,
00:11 that'd mean we could ship and deploy Python apps without worrying whether Python itself is
00:16 available or up-to-date on the platform. Nathaniel Smith has just proposed a PEP to do just that,
00:22 PEP 7.11, and we'll dive into it with him next. This is Talk Python to Me, episode 412,
00:29 recorded April 18th, 2023.
00:31 Welcome to Talk Python to Me, a weekly podcast on Python. This is your host, Michael Kennedy.
00:49 Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython,
00:54 both on fosstodon.org. Be careful with impersonating accounts on other instances. There are many.
01:00 Keep up with the show and listen to over seven years of past episodes at talkpython.fm.
01:06 We've started streaming most of our episodes live on YouTube. Subscribe to our YouTube channel over at
01:11 talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.
01:17 This episode is brought to you by Sentry and us over at Talk Python Training. Please check out what
01:23 we're both offering during our segments. It really helps support the show. Nathaniel, welcome back to
01:29 Talk Python to Me.
01:30 How's it going?
01:31 It's going real well. It's going real well. We're on the eve of the eve of PyCon. How about that?
01:37 Eve square? Okay. Yeah.
01:40 I don't know how many eve. Maybe it's the eve to the third, but we're very near PyCon. I'm pretty
01:44 excited.
01:45 Yeah. Anti-penultimate eve.
01:48 I don't know.
01:48 The penultimate eve, perhaps? Yeah, exactly.
01:50 I suspect a lot of people will be listening to this show on their way to PyCon. So if you are,
01:55 awesome. Come say hello to me. I'm going to be doing some live shows, some Ask Me Anything,
01:59 some various other things. Are you going to be at PyCon this year?
02:01 I'm not. I'm not, unfortunately.
02:03 So they're going to have to just shoot you a message on Twitter or on Mastodon or something like that, right?
02:09 Yeah. I'm easy to find. GitHub, email, whatever. Yeah.
02:13 Cool. Well, everyone going to PyCon, hope you have a great time and do come say hi. And with that,
02:20 you know, we're going to be talking about this project, this new PEP about distributing Python
02:27 itself, kind of like you distribute Python packages, but a little bit more. Why not? I mean,
02:33 it seems pretty reasonable to me. And I'm super, super excited to see work in this area because
02:39 Python is so strong in so many areas. And there's just a couple of like really big gaps that other
02:47 technologies have nailed so well. Two of them that I see that are super significant is like,
02:52 hey, Michael, I want to build a mobile app. How do I do that in Python? Or I want to build a desktop
02:56 app. How do you do that in Python? Like, I'm not sure. I'm not sure you should even think about
03:01 doing like desktop, maybe mobile. I mean, Kivy is great, but it's not like a general purpose
03:07 UI toolkit. And so that's the one. The other is, hey, I have an application. I want to give it to
03:13 someone who is not a developer and have them run it. Right. And there are some tools that address
03:19 that. But one of them is just like, oh, how do they get Python at all? And your project, your pep,
03:24 and some of the ideas relate specifically to how do we make it easier to get a prebuilt
03:30 non-admin, not install for the whole system, Python for somebody so they can run an app or
03:37 even for developers, right? Yeah. I mean, in fact, I mean, I'm a developer, so that's kind of in some
03:43 ways the original use case, you know, scratch your initial app, right? Yeah. But yeah, I mean,
03:48 it's just, it's a very general capability, I think, once you have it. So yeah, I mean,
03:54 the motivation there is basically like, you know, there are lots and lots of ways to get Python,
03:57 right? You can get it from, you know, the Windows store, it comes with on your pre-installed on your
04:03 Mac, but not a very good one. Yeah, not quite, sort of. There's, you know, but there's also Homebrew
04:09 or Pyenv or your Linux distro has it. And you can, you know, get it through Conda. And if you download
04:16 Blender, oh, there's secretly a Python inside, you know, like, it's just, you know, there's just so
04:20 many different ways to do it. And that's great. You know, like, it's good to have all these options.
04:26 They also are different use cases, but it's sort of silly that, you know, it's obviously it's,
04:31 you know, it's flexible. There's lots of ways to do it, but there's no way to just be like, okay,
04:36 like, I just want a zip file that has Python in it. And it's like a standard way that's like supported
04:41 and, you know, kind of, we all use, so we can call sort of share benefits and improvements and all of
04:46 that. So that's kind of the key, you know, the PEP is not that innovative in terms of what it's
04:51 actually, you know, doing, right? It's a zip file with Python in it. But it's just sort of trying to
04:57 do kind of the logistics of like, okay, but like, let's all agree on how we're going to put it on PyPI.
05:01 Let's, you know, have tags and stuff. So tools can figure out what they're looking at and do stuff
05:05 automatically. And I think that unlocks a lot of use cases, just that one simple change.
05:09 I think it does as well. I mean, your abstract is one of the more concise ones, I would say.
05:15 Tell people about the abstract here.
05:19 Yeah, the abstract on the PEP is, it's, quote, like wheels, but instead of a prebuilt Python
05:26 package, it's a prebuilt Python interpreter. That is the full abstract. I figured that basically,
05:30 you know, tells people what they need to know.
05:32 Yeah. So the idea is kind of like you would say pip install request, you might say pip install
05:41 Python 3.11, except for that you can't use the word pip because pip is built on Python. And so
05:46 you need Python to run. I mean, it's a little bit circular there. So you kind of need something
05:51 outside of Python. But, you know, conceptually, it's, I have these things I need to run my app,
05:57 I need request SQLAlchemy, and beautiful soup. I also need Python 3.11. So those are my dependencies.
06:03 Give me that, right?
06:04 And I mean, you could even imagine potentially pip install Python 3.11 working, I suppose,
06:09 like, you would need a Python, some Python to run pip. But once you have that, then it's probably
06:14 still convenient to be able to say, okay, actually, shoot, so I got this bug report saying in 3.11.2
06:19 specifically, there's some issue, and I'm not sure if I can reproduce it, you know, like,
06:22 being able to just grab that in one command, pretty useful. But that said, you know, yeah, it's
06:29 not necessarily, pip isn't necessarily the target. I've been working on some stuff there as well.
06:36 I don't know if you want to get it together.
06:38 Yeah, we'll definitely get into it. We'll definitely get into it. I think,
06:41 you know, you need something a little bit on the outside. And I think ideally doesn't depend on
06:46 Python being on the system, because that it would be perfectly useful for it to depend on Python.
06:51 And this gives you a different version, this gives you a way to quickly toggle between these versions
06:56 and these different setups. But if you could omit that dependency on Python, then all of a sudden,
07:00 you give a way to give it to people who are not developers and use cases where it's not just I
07:05 already have Python and I need to zoom in. Maybe you're a developer, but you're not a Python developer.
07:09 Should you have to manage your own Python installation so that you can use, you know,
07:14 something that needs Python to run against your source code, right? That is not Python. So there's
07:19 a lot of scenarios where I think it gets unlocked if you use a different foundation.
07:24 Yeah, I mean, some of the one of the audiences definitely have in mind here is like, you know,
07:29 people with like taking their very first ever programming course on the first day of class.
07:33 Like right now, it's pretty awkward that you're like, okay, well, first, you have to go to Python.org
07:37 and then click through here and click there and download that. Oh, wait, no, not that version.
07:41 Oh, did you forget to check to put in your path? Oh, dear. Hold on.
07:44 Yeah. And oh, no, do you use the Pylauncher? Are you on Windows? Are you like, it was just,
07:48 it's this extra fiddliness. And you know, it's funny, we've spent all this effort
07:53 in the last few years kind of getting wheels to the point where they can just work, right? You can
07:58 just, you know, pip install NumPy and it works everywhere and, you know, stuff like that. But
08:01 Python itself isn't there yet. Another use case, I've sort of maybe my primary use case,
08:08 sort of audience I have in mind is, right, I develop Python packages and as open source and
08:14 distributed on PyPI, stuff like Trio. And so I have the problem of, you know, I want to be
08:21 welcoming to new contributors, right? I want to, you know, bring them in, get them started first,
08:26 quickly. They're volunteers. So like, I don't want them like faffing around and struggling and getting
08:31 stuck trying to just like run the tests or anything like that, right? Like both, that's just a waste of
08:36 their time. It's like kind of rude and inconsiderate. And it's also, you know, there's likely they'll just
08:40 give up, you know, if it's just a casual, like, you know, they aren't really invested yet. It's just
08:45 something they're doing for fun or out of interest. You know, I really want to, you know, make that easy
08:49 for them. And so part of the vision here is like, you know, be able to say, oh, yeah, so you check out
08:56 Trio, you know, here, you type this git clone command, and then you, you know, you have, you know,
09:01 this, some kind of Python management tool installed. And you type that tool, you know, run tests,
09:07 and it makes sure you get the right version of Python and set up the environment correctly. And then
09:10 it executes it for you. And it knows what tests are, how to run the tests in this project, because it
09:15 looks at, I don't know, pyproject.toml or whatever, right? And so just sort of capturing all, you know,
09:21 we have all those pieces, we don't really have anything that kind of brings them all together
09:24 into that, like, just type one thing. That's it. It's going, you know, and it just works.
09:30 We don't have many, we honestly don't have many tools that are outside looking in, in Python. So much of our
09:36 tooling and our infrastructure is you have Python. Now, now you use the tools, now you install it,
09:41 now you install black or, or rough or other. Yeah, there's just this kind of very old assumption,
09:47 which I mean, it made sense, like 10, 15, 20 years ago, where sort of everything was sort of
09:53 installed manually. And of course, you have, you're going to go through some work to like get set up
09:56 with Python. It's the foundation of your whole environment. And then, you know, kind of we add
10:00 stuff to make it easier on top of that, but kind of, I think it's time to kind of go back and re,
10:04 reevaluate that sort of foundational assumption. Yeah, absolutely. So you mentioned Trio, I know
10:11 before we dive too much further in, I'm going to give you a chance to kind of, let folks know what
10:16 you're up to. we'll talk about Trio at the end if we've got time, but you know, what have you
10:21 been up to since June 29th, 20, 2018, you know, five years ago, roughly last time I was on the podcast.
10:28 Yeah. wow. That was really early in Trio's life, I guess, actually. so, I mean, I've had
10:37 a lot of just, you know, real life, has happened. then I was, you know, sick for a while and trying
10:44 to kind of get back on my feet, did some consulting, just started a new job. So that's
10:50 kind of, you know, a lot of distractions, but, also, you know, yeah, I think, Trio is still,
10:56 you know, I still like it a lot. it's definitely had more influence. Actually,
11:02 had I even published the structure concurrency blog post then? I don't remember.
11:06 I feel like it sounds familiar to me, even though it's been five years, it does sound familiar. So I
11:11 do think so. But what has happened since then certainly is Python has seen some of these ideas and
11:18 adopted them, right? Like, like 311s, concurrency stuff. Yeah. So there's been a, like sort of the,
11:25 the influence has gone a lot further than I ever expected. both actually in other languages. So
11:29 yeah, like Swift and, Kotlin and, you know, have all kinds of adopted ideas from here. Java
11:37 apparently is making some big changes, coming up soon with a whole new concurrency setup. And they're
11:42 like saying like, yeah, we're basing it on that Nathaniel Smith random blog post. Okay. Yeah.
11:48 Okay. Okay. Yeah. It's very flattering. but yeah. and yes, also, in the Python itself,
11:59 it's sort of complicated because we, it's sort of this awkward situation where there's a sink.io that's
12:03 in the standard library. And then there's my sort of competing thing trio, which, yeah, we guess we
12:08 should say trio is an async library for Python. that's, you know, portable, it's sort of, you know,
12:13 an alternative to a sink.io. There are some tools to kind of let you use both at once, but, it's not
12:18 a library for a sink.io it's its own thing. and so, you know, like, obviously we all wish there was just
12:26 one obvious choice. I kind of looked, you know, but async.io is also in this very difficult
12:31 position being in the standard library, and being sort of built up over time. And a lot of it
12:36 was designed before we even had stuff like a sink await. so there's just a lot of machinery in
12:42 there that's kind of already committed to other ways of doing things. And it's very hard to change.
12:46 And trio was sort of like, well, look, we have all these modern things and some new ideas coming in,
12:50 like structure. Concurrency is a better way to kind of, write your right concurrent programs.
12:55 and it was able to set out a clean slate to like, really, you know, do that all from the start
13:00 and be much simpler. so that was important, you know, to have it be its own thing just so we
13:06 could, you know, work that stuff out. Then there's a question of, okay, now what do we all switch to trio?
13:09 Do we move it back into a sink.io? Do they both continue?
13:12 That's been a debate for some really popular things. I think that is interesting. A lot of
13:17 people say, well, why is library X, which everyone uses, why is that not built into Python? Why do I
13:23 need to pip install it? And a lot of times the answer is because making it part of Python will
13:28 harm its ability to innovate and change, right? It'll slow it way down.
13:32 Yeah. Like there was a whole debate some years ago, but like, you know, like we all know the
13:37 HTTP client in Python, you're a lib or whatever. It's just really bad. You should just never use
13:42 it. And it's broken a lot of ways. Like it's just, you just don't use it. but we still ship it
13:47 because it would be too disruptive to take it out. That's also why we can't change it. There's just
13:52 too much code out there depending on all the weird quirks. and we don't want to ship something else
13:58 because then it'll end up being your, like URL, you know, five years later. so we, you know,
14:03 it was questioning, like, should we put requests in the standard library or URL lib3 or what, you know,
14:07 one of these, and it's just, you know, then you can't ship security fixes. You can't improve
14:12 your API. You can't, you know, so as we've gotten better at packaging also, it's taken some of the,
14:18 pressure off the standard library to be all things to all people.
14:21 This portion of talk Python to me is brought to you by code cov from century. Have you heard
14:28 about code cov? They are the leading code coverage tool on the market, and they just joined century,
14:34 the error tracking and performance monitoring company that you know, and love. Code cov is
14:38 the all in one code coverage reporting solution for any test suite, giving developers actionable insights
14:44 to deploy reliable code with confidence. Code cov is easy to set up. If you are already both a code
14:51 cov and century user, GitHub integration is even enabled automatically for you. You'll get coverage
14:57 insights directly in your workflows. Code coverage pull request comments allow you to quickly analyze
15:02 your PR's coverage and risk without leaving your workflow. It'll reduce the guesswork. You set up
15:08 customizable quality gates and let your continuous integration do the rest. And code cov identifies where
15:15 tests can help you avoid errors in production through their century integration. If an error does occur,
15:21 you'll even see code coverage details directly in your stack traces. So you can see the untested,
15:27 partially or fully covered code that may be causing errors to help you fix your tests to avoid similar
15:33 errors happening in the future. Get started for free or take advantage of century's promo
15:38 pricing where with a century team or business plan, you can get your first five row code cov seats for
15:45 just $29 a month. That's a 40% savings. Visit talkpython.fm/century to get started. Remember to use the code
15:53 talkpython to let them know you came from us. It really does help support the show. That's talkpython.fm/s-e-n-t-r-y
16:02 century and the code talkpython. Thank you to century and code cov for supporting the show.
16:08 When the standard library first came into existence, there was no PyPI and there was no package. Like,
16:17 it had to come with it because how else you're going to hunt it down on Usenet and on, you know,
16:23 base 64 and code it and put it like, what are you going to do?
16:26 Yeah. Or I mean, maybe you find it, you know, you download, I don't know, Twisted or something like
16:30 from an FTP site. Yeah. Or an FTP site or something like that.
16:32 You have to manually unpack it and have to put it in your pipe. Like it was all totally,
16:35 yeah, you know, knocking rocks together.
16:38 Sharpen sticks.
16:41 You hope it's flint and it creates a spark.
16:43 Yeah.
16:43 Yeah.
16:52 So I think even though we're already far down the road and making changes is breaking and doesn't
16:57 make sense, it might make sense to ship less in the standard library, quite a bit less and just say,
17:03 oh, you're going to pip install some meta package that explodes out some section. I'm going to pip install
17:08 the collections area. Boom. And now I got a bunch more potentially. Who knows?
17:12 One thing I'd really like to see as a possible sort of future there is moving some of the standard
17:18 library into wheels that are installed by default.
17:21 Yes, exactly.
17:22 So you get, so, you know, that's sort of, it's sort of this halfway house, right? Where, you know,
17:26 it's still the case you download Python, install it, they're there. So we don't just like break
17:30 everyone in the world who just assumes they're there. But then it kind of gives us that both of,
17:37 you know, the long-term, if it's like, we want to get rid of it, it kind of gives us,
17:40 or push it out to PyPI or just remove it entirely. Then it gives us kind of a way to do that more
17:46 gradually. But also for, you know, libraries like asyncio that are big and complex and really,
17:51 you know, would benefit from being able to have their own release cadence and bug fixes and deprecation
17:56 cycles and all of that. Then it's like, yeah, it still ships with Python,
18:00 Python, but then you can pip upgrade it. You know, you're not stuck with that exact version.
18:04 That could only change when a whole new Python release comes out and you have to take all those
18:09 changes together at once.
18:10 Yeah, I, I've absolutely had this thought and I think it is, it's a really elegant solution because
18:16 on one hand, it lets the core developers focus more on the true essence of Python and it lets it be used
18:23 in more locations, right? Think PyScript for example, or MicroPython, right? It might be that you can
18:29 create a central core that is exactly the same on all of these. You don't have to consider like,
18:35 of course, this is what runs, it runs everywhere, but you still get that backwards compatibility and
18:40 you get the ability to say, actually, I want the newest version of asyncio because I want this
18:45 more high performance background worker or something.
18:48 Yeah. Or even just, I mean, for smaller, like, you know, I don't want the newest version of async
18:53 IO because like, I don't know if it works, but I want to install in a scratch environment, this like development
18:59 version so I can try it out and give them feedback before they, you know, really make the release and
19:04 set the API in stone. And again, like right now, like you'd have to go build your own Python and it's
19:09 like, it's just kind of a whole thing, right? You can't just do install --pre.
19:14 Right. Exactly. It's definitely a more of a barrier for people who are just casually wanting to test stuff
19:19 out. You got to be pretty committed to getting Python 312 alpha six or whatever we're at. Right. I don't
19:25 know. Yeah. Yeah. Yeah. Indeed. Okay. very cool. Well, maybe we'll come back and dive into tree a little
19:31 bit more, but yeah. And so what do you, what are you doing these days? You talked about doing a little
19:34 consulting and, well, yeah, well, so I just started at, a new job, just like last week, week before,
19:41 I guess. Yeah. It's been less than two weeks. Yeah. and working at Anthropic, which I don't know if
19:48 anyone's heard of, it's still somewhat stealthy, but, it's sort of, yeah, I mean, quickly
19:55 changing. I don't know the exact status currently. but yeah, so my understanding of the background
20:00 here is that, there's actually the team at open AI who trained GPT three, just sort of,
20:07 you know, sat down together and thought, you know, decided they really wanted to do more of like a pure
20:13 focus on like interpretability and safety. Like, how do you get these models? Like, how do you know what
20:18 these models are actually going to do and how do you get them to do what you want instead of kind of,
20:22 you know, stuff like making things up or just, you know, we've all kind of got the scene now how
20:27 these large language models can go just all over the place, do all kinds of strange things.
20:33 and so, and they decided to be in suit for slander, I believe. yes. Yeah.
20:38 Somebody in the UK, I think. Yeah. Well, yeah, there's definitely, there's one that just like,
20:43 if you ask the model, like, you know, can you give me some advantages of like, you know,
20:48 problems with sexual harassment in law schools and it just picks like five real law professors
20:52 and makes up stuff like it's really, really bad. Yeah. And it like sites sources that are all made
20:59 up. Like it's, you know, like they, you know, they're just, they're very powerful, but also
21:04 not well understood or how to like kind of make them useful and safe.
21:07 Just a little bit of devil's advocate though. They are incredibly powerful and they are incredibly
21:13 capable. And that's, I think part of the dangers you're like, oh my God, it knows this. Oh my gosh.
21:17 Yeah. It understood all of that. And I ask it. So the fifth thing it says, you're like, well,
21:23 at this point I'm convinced that it really is on. And then maybe that's the made up one. And I think
21:28 that's the dangers. Cause it's actually, it's almost an incanny valley. It's close enough to write that
21:33 you're like, okay, this thing's right. It knows. Yeah. So yeah. So personally, like I'm still kind
21:37 of up in the air on how you like, how impactful they'll be, where the impact will be. Like,
21:42 I think it's just a lot of open questions. You haven't bought a farm, like a goat farm in the
21:47 woods. Cause you just give them technologies. Okay. But I guess I do have stock options now,
21:56 apparently, or I will at some point if they vest. So I guess that's the other route. But no, but anyway,
22:02 so I was just saying, so yeah. So, Anthropix is just interesting company where, you know,
22:08 you actually get to play with some of those big models internally. They're kind of working on
22:12 on releasing products now. but it's also been kind of just a really interesting to kind of
22:19 get the sense internally of like, it's really kind of this like research culture, which is appealing
22:24 to me with, I'm sort of coming out of academia of a lot of like numerics background. and what's
22:29 also interesting is that part of the reason, we kind of, you know, connected is that apparently,
22:35 you know, turns out a ton of their internal infrastructure runs on trio. so, they're,
22:40 you know, partly hired me to support that, and are actually giving me time to work on open source,
22:45 paid time. so actually they are, funding this PEP 7.11, you know, Python binary stuff, though. They don't know it yet. now they do. Yeah. Yeah. Yeah.
22:56 If they listen to the podcast, they'll learn on this. That's great. That's really, that's really cool.
22:59 It looks like a, an interesting area to be working. I agree that the research oriented places,
23:04 they are fun area to work, right? You're not. Yeah. And there's just a lot of flexibility,
23:08 kind of, you know, like it's clear this stuff is going to have effects, which effects and how big
23:14 and all that. I don't know, but you know, being at ground zeros, you know, it's exciting. It's really
23:19 exciting. Yeah. And a lot of chances maybe have some impact. So cool. All right. Let's dive into the PIP,
23:24 the PIP, the PEP, the PEP. That is not, it's not quite PIP, but it's kind of like PIP. Okay.
23:28 Yeah. Okay. So we talked a little bit about the motivation. we talked a little bit about what
23:35 it is. maybe, maybe tell us a bit about the spec. Like what is the PEP actually say?
23:42 what is it actually trying to deliver? And we can talk about like the use cases and
23:45 Yeah. the tools for it and so on later. Yeah. I mean, and so, like I said, you know,
23:51 that the abstract deals like wheels, but it's an interpreter instead of a package.
23:56 that's partly just sort of a tagline of like how you use it, but it's actually also a lot about how
24:01 the actual spec is written. it's just sort of like, well, you know, we, we've done a ton of
24:06 work over the last, you know, five, 10 years. A lot of people have put a lot of work into making wheels
24:11 work, right. In terms of like, figure out, okay, how do we, you know, have metadata that's usable to
24:16 like keep track of, you know, which packages are installed and their versions and which ones are compatible.
24:21 And if for a binary build, like which systems can you put this on and, you know, all the many
24:26 Linux work and, just all of that stuff. And it's just like, well, you know, so we have wheels,
24:32 we don't need to reinvent the wheel again. so I'm just sort of taking all of that.
24:39 so it's just like, okay, it was mostly, it's defined, it's just a Delta against the wheel spec. It's like,
24:44 okay, and the wheel spec, you have, you know, this directory for metadata, I have, you know,
24:49 that same directory, but I call this calling these Python binaries, pybys, just to have sort of a
24:54 short name, you can stick it in a prefix or sorry, in a file extension.
24:58 Yeah. P Y B I. Yes. Dot P Y B I. I like it. Yeah. The PYPY, the interpreter and PYPI, the,
25:06 package, repository work confusing enough. So I thought I'd add another near homonym to the,
25:11 PYPY. It's PYBI. Yeah. Yeah. You know, anyway. but so, yeah, so like, but they look like,
25:20 you know, the file names look like wheels, like, you know, be something like, you know,
25:24 cPython dash version dash many Linux to 17.py bi. they context looks like wheels. They're
25:31 basically just zip files. There's some, you know, instead of a .dist info directory, you have a
25:36 pybyinfo directory and it has a metadata file that's in the same format as wheel metadata files with,
25:41 for the name and version and, you know, description, all that stuff.
25:46 there, there are a few tweaks, basically just what, you know, you need specifically for,
25:52 interpreters. so, okay. So like one thing that makes it a lot simpler is that, for an,
25:59 there's only one interpreter in a Python environment, right. Whereas wheels are kind of designed to be
26:03 flexible and be installed into different kinds of different kinds of Python environments with
26:07 different layouts. a PYBI, it's just like, it's just a raw set of files. You unzip it. That's it.
26:14 You're done. where wheels, there's like, well, okay, if you, you want to put this in site
26:17 packages, so you have to go find that. Whereas this goes in the bin directory. So you have to go
26:21 find that and do the special, you know, so that part's just, you know, not relevant. we leave
26:25 that out. there's some, slight, you know, we have to support sim links, which wheels don't,
26:33 mostly just cause there's never been a big compelling reason. What's that?
26:37 Or the windows folks out there, maybe, and others who, Oh yeah.
26:40 Yeah. What the heck are sim links?
26:41 Yeah. Okay. Well, so yeah. Sim link is a classic Unix concept, though. Windows does have them too.
26:46 Now, I guess where it's like a special magic file that, instead of having like its actual contents,
26:53 it just lists says, go look at this other location on the file system for my content.
26:57 Right. It's like, like an app shortcut, but yeah.
27:00 But like programs, not for UI.
27:02 Yeah. Well, like built into the file system.
27:04 Yes, exactly. So you try to open it, it goes to the other one.
27:07 Yeah. The, the, the operating system automatically opens that other file for you. but you could
27:11 also like look at, you know, if you, you can like say like, can you show me the sim link? And like,
27:16 it'll tell you about it if you ask, but if you don't, then it just, you know, magically works.
27:21 And it's mostly, it's just, it turns out that traditionally, Unix pythons tend to use these,
27:27 both for things like, you'll, you know, in the, your bin directory, you'll have the Python
27:32 executable. And then you'll also have Python three as a sim link to Python and Python 3.11 as a sim
27:37 link to Python three. And so, you know, one to preserve that. and also it turns out on macOS,
27:43 they have this very specific kind of layout they want with like framework. Like,
27:47 I don't really understand it in detail, but like there's sort of like a, how a macOS app is
27:51 supposed to be structured and that it turns out to involve sim links. So we just, you know, we just
27:55 have to support, that said, I mean, the way we support them is like, it turns out there's a
27:59 standard way to put them in zips. So I say, let's do that. You know, like again, really trying to keep
28:04 this as boring as possible, you know? I did know. and then the last thing file, that's crazy.
28:09 Yeah. It's a, it's true. It's an extension from the info zip folks, but then it's become,
28:14 I don't know. Zip's a strange format. It's kind of like an oral tradition as much as like an actual like specified format, but there's an entire documentary on zip.
28:24 And, I believe I came up with it. There's yeah, it's even controversial in its early days.
28:30 It's it's nuts, but yeah, it's even won the compression de facto standard these days for the
28:35 most. Yeah. It's definitely got trade-offs, but it's just in terms of, it's just really useful.
28:39 It's just a thing that everyone, everything could understand. It's just, it's just so compatible and it's
28:45 also convenient that you can do random access, unlike some of the alternatives. You can just,
28:50 you can pull out one file from the middle if you want to.
28:52 The fact that anyone can open it is so much better than it might save one more percent. Yeah, for sure.
28:57 Yeah. Cool. Okay. So we've got these, basically the PI by file is the zip file. Is that basically the
29:08 entire interpreter just kind of bundled into a zip file? Like what's the deal there?
29:13 Yeah. Okay. I mean, it's just literally like, you know, you install Python into a certain directory
29:18 and then you take that directory and you put it in a zip file. There's a little bit of tweaking to like
29:22 make sure it's self-contained and you can move it in a portable, portable, relocatable, I guess is a
29:26 better word. Yeah.
29:27 Yeah. So sometimes if you just install Python regularly, it's kind of, we'll have hard coded.
29:32 I know that I'm at this particular position on the file system. And so I need to make sure we don't do
29:37 that. And also to make it self-contained, it's like the same thing we do with wheels. Like you have to
29:42 vendor some libraries, right? If it wants to use read line as a library to, for like in the REPL to be able to
29:49 like edit your line as you're typing it, then, you know, we can't just assume it's on the system.
29:53 We have to include that inside the PyBuy. So, and, but again, like this is stuff we've all
29:59 already dealt with with wheels. There's tools for doing it. We understand how to do it. And I'm just
30:03 reusing those tools. So we're not, if I were to run a Python application delivered by one of these PyBuy's,
30:12 does it have to unzip the contents into a location and then run it there? Or can it just run it
30:17 straight out of memory or what does that work?
30:20 Well, so by itself, the format, I mean, it's just a zip file, right? So you can do with it,
30:27 what you can do with a zip file, which I mean, is not much on its own. You need some software to work
30:30 with it. Right. Now that said, I think, so, yeah. So like, if you just were starting with nothing and
30:37 you're like, I just, I have a URL to some PyBuy and I want to use it, then you'd have to download it,
30:42 run it on zip tool, and then you'd have a, you could go into that directory. It's a Python environment.
30:46 You know, you could run pip in there or whatever. That said, I think this is a really useful building
30:52 block for tools that want to go beyond that. So things like delivering a prebuilt application that
30:56 you can just run without unpacking. Like there are various tools to do that, like PyOxidizer,
31:02 PyToApp. I don't know. There's a ton of them actually. I'm probably forgetting like 10 more.
31:06 Yeah. The ones that come to mind for me are PyToApp, PyInstaller, and PyOxidizer for sure.
31:12 Yeah.
31:12 PyOxidizer being the newest of them.
31:15 Yes. Oxidizer because it involves Rust somehow.
31:18 Because all the things that involve Rust.
31:21 Yes. So, but yeah, so those tools that it's really useful to be able to say, okay,
31:29 like I'm going to do some clever thing to like set up, I don't know, a self-extracting executable or
31:34 whatever it is they do for their distribution mechanism. I'm going to create an installer
31:38 program. I'm going to, whatever it is, but you still need an actual Python to put into that.
31:43 Right. And so having a straightforward way where it's okay, that's not their problem anymore to
31:48 figure out how to find a Python and get it built and working for the target system.
31:52 So they can just say, okay, there's, I can just like grab, you know, okay, yeah, you want to target,
31:56 you know, many Linux. Cool. I'll just go grab the right Python. It's already there. I know it works.
32:01 And now I can take the files out of this PyBI and do whatever I want with them. I can pack them
32:06 into my installer or do clever things to make them usable out of memory or whatever. And they can focus
32:13 on that part instead of just the, like, how do you even get a Python? Yeah. Or how do you once
32:20 find yourself in the wrong Python, get the right Python. Yeah. Yeah. Yeah. That's a, I don't know if
32:26 that's trickier or less tricky, right? It's one thing to say, dear user, go get Python. You need
32:31 that. It's another thing to say, go upgrade your Python and hope you don't break something.
32:35 You know, I think. Yeah. Well, but also that's part of the point of these being self-contained is,
32:40 so, I mean, this is one of the more trivial use cases, right? But right now we all use virtual
32:43 ends and mostly that's fine, but also sometimes, you know, they can get, you know, janking stuff
32:48 could happen. Like, you know, you're on Linux, you do an app to upgrade and now your system
32:52 Python's changed and all the virtual ends that were based on it are broken now.
32:55 Yeah. Yeah.
32:56 Because it like had some kind of dependence on that exact binary. now I will say you
33:00 would always want to do this, but at least it's nice to have the option. You could say,
33:03 okay, instead of making virtual ends, I'm just going to make real ends. They're all just going to be,
33:07 I'm just going to drop a new copy of Python in each environment. And that way I just,
33:10 it's totally self-contained. I know exactly what I have. It upgrades when I decide to upgrade it.
33:14 and it's just, you know, it's a nice, it's a nice option, right? Sometimes to have that.
33:18 and, and also, you know, it gives you that total isolation, right? So you, you, then we were
33:23 just saying about that issue of like, oh, I wanted to use this. So I went and installed, upgraded my
33:27 Python. But now that other thing I was already using broke, cause they're using the same Python.
33:30 It's like very easy to say, nah, just give them different pythons. You know,
33:33 There's not that much that changes over time. That's a backwards breaking sort of thing. I mean,
33:38 two to three, but I think that's kind of, let's put that in the past. But that's a whole idea.
33:42 I did. Well, but I did recently, I was working with MongoDB using Beanie, which was using
33:47 motor, which was using the at async or at co-routine decorator, which was removed in 3.11 or 3.10,
33:56 one of those recent upgrades. And it had been deprecated forever. The people at MongoDB said,
34:02 we don't care. We're just going to leave it. Who wants to put the word async in front of my method?
34:06 That's tricky. I mean, they just probably won't pay attention. And my, my code wouldn't work. I'm
34:10 like, why doesn't this work? Oh, the thing I depend on, which the thing it depends on that,
34:14 that thing needed less than 3.10 or whatever, 3.10 or 3.11, whatever.
34:19 Yeah. Now we're back to it. I think I have struggles to adapt without breaking. Yeah.
34:25 Yeah. But yeah, stuff like this would, stuff, it does happen, you know, and this kind of isolation gives you 100% confidence to say, I'm going to make this new app. We're going to
34:33 try running this app on, on this in production, and it's not going to hurt anything. And I don't
34:38 need Docker. And you can say, or you could use, say, I'm going to, you know, use this exact point
34:43 version on in development. And then I'm going to take that and build it, create my, use that to create
34:49 my Docker image. Like I don't need the like prebuilt Docker stuff. I could just grab Python
34:53 from PyPI. And I know it's the exact same version. Everyone else is using built by the python.org folks.
34:59 Hopefully, you know, we're not there yet, but like, that's kind of the, where we're trying to
35:02 a PEP and not something on GitHub, right? Yeah, sure. Well, it has been on GitHub for a while,
35:08 but you know, I have time. Yeah. Yeah. So maybe it's, it's worth jumping over that. But before we do
35:13 two questions, maybe two top level questions anyway. So this is about the concept of kind of like pip
35:21 install Python 3.11 or 10 beta two or whatever, whatever it is. Yeah. Yeah. Does that, is there a way to say,
35:28 say, and these three packages off of PyPI? Like, can I take and kind of bring a virtual environment
35:35 effectively along with me with what you're doing so far? So PyPI is again, by themselves, I mean,
35:40 it's just an archive format, right? In a package format. It doesn't do anything that said,
35:46 obviously, yeah, part of what we want is for these to be useful for things like, you know, building
35:51 environments that have other packages in them and stuff. so that's the one other thing I didn't,
35:55 I forgot to mention about, defining the format, probably the most interesting part actually,
36:01 is that we do add some new, static metadata that we put into the package and kind of the motivation
36:07 there is that, we're trying, I try to figure out, okay, what do I need to know in order to install
36:13 wheels into this Python without running it? Right. Because right now, right. Like pip assumes that it's
36:19 running on the Python it's installing into. So anytime it wonders like, you know, okay,
36:23 like what ABIs does this Python support and what version is it? What platform am I on? It can just
36:28 ask the interpreter it's running on. Right. And it's like, okay, well, it would be really nice if you
36:33 didn't have to do that both for like efficiency. Like you want to be able to, you know, figure out which,
36:39 you know, have your like installer, your resolver, figure out which versions of everything it wants,
36:44 without having to like download and run multiple versions of Python and stuff. Like
36:48 you'd really like to avoid that. and it's also things like, I want to build a cross, I want to
36:53 build, you know, release, distributions for, you know, macOS, but I'm on windows or vice versa.
37:00 or I just want to, you know, I, I'm developing on, I'm developing my package like trio on
37:05 Linux personally, but I would like it that when I lock my, you know, version. So I know all my,
37:10 you know, collaborators are using the same versions that we figure out locks that also
37:15 work on windows and macOS. And I can't just trivial run that all those pythons, you know,
37:20 from one resolver, cause it's not running on all three at once. and we, you know, in Python
37:26 packaging does have the ability to like have different dependencies on different OSs and,
37:31 you know, it gets kind of, you know, can get very kind of complicated to figure out like,
37:35 which packages do I need where? and so I want to put a bunch of metadata into the PyBI,
37:40 like all the stuff you need to solve those problems. so yeah, so the PyBI itself,
37:45 I think normally they won't ship with any packages, maybe again, call back, maybe in the future,
37:50 we'll start moving some of the standard library into wheels that are pre-installed. You can do that.
37:54 but, I'm guessing like, you know, for now it'll, they'll mostly just be, you know,
38:00 a plain vanilla Python install, but then you could take that, you could take some wheels,
38:05 bundle them all together into a new archive if you want, or yeah, again, whatever you want to do with
38:10 it, stick it in a Docker image, whatever. It's a step towards, but not necessarily trying to propose an
38:16 entire solution of here is the interpreter and all the dependencies and the code and just run it as if it
38:23 was, it had no, no dependency on your system, just treat it as like a dot exe or a dot app.
38:29 I can just double click. Yeah. I mean, it makes that a lot easier than it is. Right. Right. Like
38:34 right now, first step is just to figure out like, how do I even build a Python that'll work like that?
38:38 And that is like some arcane dark knowledge written on a tome in, you know, black ink on black paper and
38:45 a black tomb you have to go find or something like, you know, it's just, yeah, it's not easy. and so
38:53 just having the ability to say like, yeah, just grab this file, unzip it, drop some wheels in it,
38:58 zip it up again. Now that's a package you can drop, you can hand to someone and it'll work on their
39:02 system. You know, like that's, that makes it a lot more accessible. it's not the thing I'm most
39:07 personally, like I'm not immediately going to go build that one last extra tool, but I bet someone will.
39:11 Yeah. I can imagine someone will, for sure.
39:17 This portion of Talk Python To Me is brought to you by us over at Talk Python Training with our courses.
39:22 And I want to tell you about a brand new one that I'm super excited about. Python web apps that fly
39:28 with CDNs. If you have a Python web app, you want it to go super fast. Static resources turn out to be
39:35 a huge portion of that equation. Leveraging a CDN could save you up to 75% of your server load and make
39:42 your app way faster for users. And this course is a step-by-step guide on how to do it. And using a
39:49 CDN to make your Python apps faster is way easier than you think. So if you've got a Python web app
39:54 and you would like to have it scaled out globally, if you'd like to have your users have a much better
40:00 experience and maybe even save some money on server hosting and bandwidth, check out this course over at
40:06 talkpython.fm/courses. It'll be right up there at the top. And of course, the link will be in your
40:11 show notes. Thank you to everyone who's taken one of our courses. It really helps support the podcast.
40:16 Now back to the show.
40:17 My next question is not on our shared screen here, but is what impact do you think this would have
40:26 on PyPI? First of all, do you see PyPI the way the CDN that delivers packages like Trio and wheels
40:35 like Trio? Do you see that as the same channel through which CPython 3.11 is delivered?
40:40 Yeah. I mean, so I would like these to literally be like you go to PyPI/project/cpython and it says,
40:47 like, here's the latest release and you click on downloads and it shows you the, yeah, I'd like it
40:52 to just literally be stuff you upload to PyPI. Right. And when you pip and solve from there,
40:56 it figures out the platform to pick from and it downloads that wheel and off it goes. Right.
41:00 Do you think that that would add like a huge burden to the amount of traffic or do you think it would be okay?
41:05 No. I mean, we'll have to see and adapt, but Python itself, it's like, shoot, it's tens of megabytes.
41:16 Okay. So there's plenty of other packages. There are a lot of much bigger, you know,
41:20 like go look at TensorFlow or something. There are hundreds of megabyte packages on Python that are
41:25 very popular. Also, I mean, Python.org downloads go through the same CDN anyway. It's just sort of
41:31 different infrastructure on the backend, but it's still fastly serving it and donating the bandwidth.
41:35 So, you know, in that regard, I wouldn't expect much change. And also just, you know, like people
41:41 tend to install, you know, wheels more often than they install Python. Right. Again, it's hard to know
41:46 sort of the second order effects. Like maybe virtual lens will be less popular in favor of full lens
41:51 if this takes off. And then people would start installing Python more than they do now, but still,
41:57 nonetheless, I don't think it's, you know, a huge, I don't, I wouldn't, I wouldn't anticipate it being a
42:02 huge change. And if it, you know, it turns out to be a problem that, you know, we can kind of figure,
42:08 address it then, you know. Well, you could also do to a large degree, you could do things like
42:14 pip does already that caches, you could just cache the CPython wheel, the PI BI into the user profile.
42:21 And the second, third, fourth time you get it is really the CI systems and all the doctors and all
42:26 that stuff that don't understand what a cache is or any of those things. Yeah. But then, you know,
42:31 so like, if it comes to real problems, then you go to GitHub and you're like, hey, can we work
42:34 something out so that, you know, you stick a cache in front of PI PI and, you know, like stuff like
42:39 that, like, it's not, you know, trivial, but, you know, you could talk to people and solve problems.
42:44 Like, I don't, certainly, I don't think we should, you know, hold back the entire design of like how we
42:49 distribute Python and make it available because, oh, maybe it'll be too easy and people will use it too much.
42:54 You know, like, that's a good problem to have, right?
42:58 Yes, exactly. Look, they're using it. This is terrible.
43:00 Yeah. Like first you make it easy and then figure out how to solve any problems that causes don't.
43:05 Yeah. I think we've more than once solved the problem of, oh my gosh, they're using it like
43:09 Google, Netflix, you name it, you know. Think of the benefit that you'll be doing for all the
43:14 developers, especially those who have Python skills and are looking for a job. I mean, if the popularity of
43:20 Python by downloads is one thing, if you could like 4x that, we'd all be more demand. Like,
43:25 look, really, really downloaded now.
43:28 Right. Yeah. Just go out there and just download it five times in every CI job. Just, you know.
43:33 Exactly.
43:33 Just throw a forum away, but you know.
43:35 Just do it a couple of times. Just show.
43:38 Right. Yeah.
43:39 Awesome. The question that you put into the PEP here on the screen though, is why not just Conda? And I,
43:46 not being a particularly data focused person, I definitely prefer using pip over Conda because
43:52 especially it seems like a lot of the web packages are not as close to update, up to date. You know,
43:58 there's a latency before it hits Conda and it's like immediately on PIP. That said, there's a bunch
44:02 of people who are like, I kind of use Conda for this.
44:05 yeah. And, and right. If you're just like, look, I don't really care about all this. Like,
44:10 I just, you know, I want to run my Jupyter notebooks and I, you know, just need a Python that can do that.
44:16 And maybe, you know, some NumPy or whatever. Conda solves that really well. And this thing could,
44:22 you know, I'm working on, could also potentially solve that really well. And so it feels duplicative,
44:27 to those people and to them, it is, you know, it doesn't really, you know,
44:31 they're both two solutions that work, but there isn't necessarily reason for them to choose one or the
44:36 other. but this could also be a foundation for the way that Conda provides Python to itself.
44:44 Maybe, I don't know. Like there's a whole other question about how like we could bring Conda and,
44:50 you know, PyPI pip, that kind of world closer together and interoperate better. But it's sort of,
44:55 that's a whole like, you know, can of worms, lots of complicated stuff. I don't think,
45:00 I don't think, you know, this PEP itself is going to, be the thing that makes a big difference
45:05 there. Okay. But it's not an anti-Conda type of thing. No, no. Yeah. Well, and so,
45:10 right. And so, I mean, you can also get to see a version of this in the past, but,
45:13 like basically the way I think about it is that the key reason why we just like,
45:18 why PyPI is a critical piece of infrastructure that, you know, cannot be replaced by anything else
45:23 is not because it's, of its use for end users. I mean, it's great that end users use it and find it
45:32 helpful and all that, but like, that isn't the people who absolutely need it and could not have
45:36 any replacement. the reason why we just absolutely need it is for package developers,
45:41 because the way, again, you're talking about all those different ways you can get Python,
45:45 right? And there's all these different ways Python packages get distributed, right?
45:49 You can brew install packages. There's versions, you know, NumPy, a patch version of NumPy used to be
45:55 part of the standard macOS install. Maybe it still is. I don't know. you know, like when you
46:00 install Blender, there are Python packages in there, install some game using, was it RenPy? it's
46:08 going to have Python packages in there or just, you know, there's just like, there's so many different
46:12 ways that Python code goes out of the world and gets used in all these different contexts.
46:17 and if you're developing some upstream library, like, you know, trio again, or, but, you know,
46:22 or requests or NumPy or anything, then what you absolutely don't want to do is have to maintain
46:29 a separate distribution for all of those different things. You don't want to have to upload your
46:33 package to conda forge and also to Debian and also to Fedora and also to Blender forge or like,
46:41 like you just like, that's not, that doesn't make any sense. Right. so we need this, you know,
46:47 and then having every different package maintainer do that, like that just would be terrible, right? We
46:52 just would be unworkable. so the critical role that PyPI serves that just nothing else can
46:59 is it's this intermediation point between package uploaders and package users and including package
47:05 redistributors. and so, and so, you know, like I, you know, make a release of my package,
47:10 I upload it to PyPI and then that's where conda forge gets it. That's where Debian gets it. That's
47:14 where, you know, end users get it. If they're pulling straight from PyPI, like sort of it's fans out from
47:19 there. And the key difference in terms of design between pip and conda is that pip's metadata formats
47:26 and you know, wheels and the metadata and sourcetists and all that are designed around this abstraction
47:32 of, you have some kind of Python environment, but it could be any of those, right? It could be on
47:37 different OSs. It could be different, you know, ways of building it, different layout, different
47:41 pieces could be missing, like whatever you could be laid out in all kinds of different ways. I just know
47:46 that there is some kind of Python environment and I have the metadata to like figure out how to adapt to
47:52 how this particular Python environment is put together. And conda on the other hand is one of
47:58 these sort of downstream systems. It's, it can, the reason people like love it and like data science,
48:02 right, is because it's a full fledged, like arbitrary application distribution thing, right? You can install
48:08 random, you know, C libraries and you know, you can install R and R packages. Like, like it's just,
48:13 it's got, you know, compilers that are all there in the one thing. But because of that,
48:17 it isn't have this abstraction of, oh, I can handle any Python environment. A conda package of a Python package
48:25 is set up to install in a conda Python that's laid out in the way a conda environment is laid out in the way
48:30 using the libraries a conda library has, right? And so it isn't have that flexibility. If you just release
48:36 something for conda, then it's great for conda, but it's not, you know, usable to Debian and Homebrew and all of those other folks.
48:43 And so like, sort of that's the key thing that PyPI does, right? It's like, it has that abstraction
48:48 that lets you, you kind of have the Python packaging ecosystem of all those packages and their dependencies
48:54 on each other. And then you kind of project it down into each of these more specific, specialized
48:59 packaging systems. Right. And then also, because, you know, as a, that's the, that's the other thing,
49:05 as a package maintainer, I don't just write my package and upload it. Like I'm also using all by other,
49:11 you know, all the other open source maintainers work as I do it, we're all working together. Right.
49:15 And I'm depending on their work and they're depending on mine. And so I need to be able to say like,
49:19 okay, you know, my package needs those three other packages and here are the versions that I need to be able to
49:24 download, you know, create an environment with those versions and test it before I upload my package to PyPI.
49:29 And so again, all that work has to happen at that higher abstraction level. Like you can't just say,
49:35 I'm going to take the latest version from Conda and test against it. Because that's not necessarily
49:38 the version that other packet, you know, other people will get where you take the versions from PyPI.
49:43 Those are like the original ones. I can get exactly, I can have access to anything anyone has ever uploaded
49:48 as soon as they upload it and I can test them all together. And then, you know, if Conda wants to take
49:53 some curated subset of those or whatever, that's great. That's a really valuable service, but you know,
49:59 they kind of need that underlying set of packages to curate. And that's what PyPI is for. Yeah.
50:04 PyPI is kind of the definitive source of truth as the package creator intended it to be.
50:11 Yeah. And then, so, right. And then of course, yeah, for a lot of end users, it turns out they're
50:16 just going straight to that without any intermediary works great for them. And that's really cool.
50:21 But also, you know, it's not like I don't have anything against people who prefer to go through
50:26 Debian or Conda or whatever. I think that's also great, you know, if that works better for you.
50:31 But for the folks who are, you know, developing, you know, packages to upload or who just, you know,
50:37 would rather just, you know, get stuff straight from the source, the PyBIs, I think, can solve a lot
50:42 of problems that Conda, you know, just, it just doesn't address those. It has a different focus.
50:46 Trying to make it a swap over to do that might kill a little bit of what it's, it's good for, you know?
50:54 Yeah. All right. Let's, let's see. So let's move on to your announcement here. I think.
51:00 Okay. Right. Yeah.
51:01 So over on discuss.python.org, when was this? This was January 21st.
51:06 So a few months ago, you announced PyBI and Posi.
51:11 Yes. And Posi is, we talked about this mythical pip that could pip install CPython 3.11.
51:18 Posi is that mythical pip, right? Yeah. In a sense.
51:22 Yeah. So yeah, the pip 7.11, the PyBI stuff is, is just the one, one brick in my master plan.
51:27 Oh. So, right.
51:29 Right. Because yeah, because sort of the, this vision I had in mind, I kind of alluded to earlier
51:34 talking about like, you know, okay, if somebody does, you know, get clone my project, I want them to,
51:38 you'll just run the tests and know that they have the right version of Python and the right version
51:42 of the dependencies. And it just kind of, you know, and they know how to run the, you know,
51:45 just, just do it. Right. And code all that information somewhere. And Posi is sort of my
51:51 experimental, it's not ready to use, but it does have a lot of, a lot of stuff working,
51:56 is my attempt to solve that part of the problem. So the vision is Posi is a, it is a full
52:04 re-implementation of pip with, you know, the like metadata parsers and result dependency resolvers and
52:12 archive installers and all of that. Except I rewrote it all in Rust as is the style. And, but
52:21 it also, it's all built around PyBIs. So it is, it doesn't, I mean, maybe at some point,
52:26 we'll also start supporting, you know, VNs or your user installed pythons, but, you know, for now,
52:32 sort of the, for the MVP, it's just says like, okay, yeah, you have a PyBuy, I will grab that. I will
52:37 grab packages that are compatible with it. I will arrange them all to run together based on, you know,
52:42 you just, you just say what you need. I turn that into like a lock file. I fetch those packages. I run,
52:48 you know, your test script or whatever. And yeah, I mean, that, that, that is the core idea.
52:54 Yeah. And one advantage of being in Rust is that it's, you know, just because just, so obviously,
53:00 you know, if you want to hack on it, then you need like a Rust compiler and stuff. But if you just want
53:05 to use it, then we can, you know, just take a, we can compile it down to a single binary that you just,
53:10 you know, upload to wherever, install it from wherever, and you just, you drop it on your system,
53:15 you run it and it's self-contained. It can handle everything from there. So the, again,
53:21 thinking of that target audience of like beginners, right? You say, okay, install this one program.
53:27 And now you type, you know, posy run and oh, look, you're in a REPL. And it's like, and now internally,
53:32 of course, I had to go find the latest version of Python and grab it and figure out if, which build is
53:36 right for your system and do that. But like, but you know, you don't have to think about that. You just hit
53:40 enter and it happens and there's your REPL. Or then you say, you know, posy add requests and then, or posy add
53:46 Jupyter and then posy run notebook, you know, like, and it kind of is handling the, the environments behind the
53:54 scenes. Yeah. Yeah. We did a panel discussion with a bunch of core developers around packaging recently. And a
54:04 lot of them were saying things like, I don't really want to put words in the mouth, but kind of getting the, the
54:10 sense that like, okay, so we have a bunch of tools that are really neat that live within Python, you
54:15 know, I'm thinking hatch poetry, those that, that category of tools, pip itself, even, and some of the
54:22 challenges or problems that they would like to solve, they could unlock a simpler API if it was turned
54:29 inside out, right? If the tool itself controlled Python and didn't depend on Python to get started,
54:34 they mentioned rust up as a way to get started, which is a way to kind of install
54:40 a version of rust and get started. Right. And it feels to me like this is pretty close to that.
54:45 Yeah. There's a lot of overlap for sure in terms of sort of the goals and approach and all of that.
54:51 Mm-hmm. One challenge I see is so like, for example, to run the application for the,
54:59 with the Python that's bundled up inside of one of these PyBIs is you would say posy run or posy,
55:06 some kind of file or something like that, right? Sure. Yeah. Whatever. Exactly. Yeah. Yeah. Yeah.
55:10 Yeah. Yeah. Yeah. Yeah. Whatever the CLI that's yet to be fully spec'd out comes out to be, but you
55:15 know, could you do things like, could you create, you know, speaking of sim links and other types of
55:20 stuff, could you create just in the same folder where that app lives, a Python that actually just calls
55:27 posey the Python inside instead of Python itself and, and pip that says, you know, posey run pip inside
55:34 this Python to kind of bridge, to unify the API from where people are coming from, to kind of
55:40 expose the, the same tools that are inside a little bit, you know what I mean?
55:44 Well, so yes. I mean, so the way I am currently, sort of in my current prototype, basically,
55:51 it doesn't work like that just because it felt sort of more complicated to then like try to expose
55:59 those things. So, sort of the sense is like, you know, let's see how far we can get with treating
56:05 sort of the UI paradigm of like posey is just your front end to Python. Like you don't, you just,
56:11 you start your command with posey and that's that you, that's the only command you need to know kind
56:15 of. and so, and that also allows some interesting things. So like the way posey
56:20 does environments right now is it doesn't, you can have multiple environments within a project.
56:25 Like if you need to test against multiple Python versions or you need different installs for,
56:29 I don't know, tests and for building your docs and whatever. but it doesn't actually materialize
56:36 all those as separate independent virtual environments. Instead, what it does is it,
56:41 for like each unique wheel or pyby that it needs, it unpacks that into its own directory. And then on
56:47 the fly, it assembles environments by like setting up environment variables so that it can launch a
56:52 Python in such a way that it's, you know, it picks the right Python and launches in such a way that it
56:56 sees the right packages it's supposed to see. But there's only one copy of those packages on
57:00 desk. If you have multiple environments, if you want to like try out, you know,
57:05 different versions or whatever, it can just sort of like do that, without having to go
57:10 rearrange everything on desk. and that's just, you know, it's convenient. It's just a really nice
57:15 way to work with sort of having declarative Python environments. So you never update an environment
57:20 in place. There is no environment in place. It's just on each command you run, it sort of knows
57:24 declaratively, okay, these are which packages are supposed to be there. I'll give you those packages.
57:28 So there isn't even a concept exactly of like pip upgrade or pip install.
57:32 you can just say that next time I invoke environment, I'm going to give you a different
57:36 specification for which versions I want and it'll make sure that happens.
57:41 It feels a little like Docker, right? Like if you create a Docker image and you run a container,
57:46 you want to make changes to it. You don't log into the container typically and
57:49 in that sense. Right. Yeah. Yeah. But you would just say, okay, well, we changed the Docker fire.
57:54 We shut it down and we started back up with the new up better version of itself. Right.
57:58 That, that sounds like Docker. Yeah. The main difference, the big, a big difference would be
58:02 that your Docker, then when you build it, like the actual Docker file is this big old imperative,
58:07 go scribble here and then delete that and then put something else, you know, that kind of thing.
58:10 It's not just like, here's the list of things you need.
58:13 well, it feels to me like maybe a better solution than what Docker is giving you. If what
58:18 you really just want to do is run a Python and isolated Python thing repeatably, because with
58:24 Docker, the idea is like, well, you want it, you want it isolated. So let's do this. Let me give you an
58:28 entire separate copy of Linux. Right. I know you're running it in like a, not a full VM way, and it's not
58:35 as heavy as a VM, but it's still, you're configuring a Linux computer inside of this container in the way that,
58:41 whereas this is just like, I just want Python configured, not everything.
58:45 Well, and even more like in Docker, if you want to make sure that you run Docker build twice and get
58:51 the same package versions, like you have to do that yourself. You have to come up, you have to use,
58:55 I don't know, pip compile or something like create a lock file and then install from that instead of
58:59 your original requirements. And, you know, it's a whole thing. There's lots of ways to get it wrong.
59:04 Whereas in POSY, the way I've written it currently, it's just like, there is, you know, there's one
59:09 operation, one internal function that takes a set of like, okay, these are the packages I want.
59:13 And it like renders that down into a lock file. Like, here's the exact set of packages you need,
59:18 including all the dependencies and all their versions. And then that's the thing that you hand
59:22 to the run me an environment. So like, you have to go through that step. It's just built in. And so we
59:26 can like, you know, so, and of course, you know, as we build up the, you know, CLI and stuff,
59:31 idea will be that that will be that, you know, written to disk, similar to a cargo.lock or poetry.lock
59:35 or whatever. And so you just automatically get that reproducibility, which, you know, you don't
59:41 get that automatically from Docker, right?
59:44 This is a thing that people could go get.
59:46 Yeah, on your GitHub profile, they could check this out. And they could they could try it, right?
59:51 So yeah, let's see. So yeah, so obviously, there's a lot of moving parts here. If folks want to help
59:59 with the PI BI part that PEP 711, there's lots of stuff could use help with. But it is also there's a
01:00:06 draft pap up and I have built lots of PI BI packages for lots of different versions of Python for Mac,
01:00:12 Windows Linux, they're up on a CDN. So you know, if you use the same API as PI PI, so you could pip install
01:00:19 from there if you had a pip that did it. So it's it is stuff you can try out right now, experiment with
01:00:24 at least. And then yeah, as for the posy part, again, like I said, it's still it's mostly the
01:00:29 back end stuff, but it is a pretty complete implementation of like all the packaging stuff.
01:00:33 It had it can actually do that demo. I was just saying of like, I need these three packages in this
01:00:38 version of Python, and it can do the dependency resolution for a named specific operating system,
01:00:44 which may not be the one you're running on, and then generate that environment and actually invoke
01:00:48 it. Well, you can only invoke it if it's for the operating system you're running on, of course.
01:00:52 But you know, it can do all that stuff. There isn't really a UI in front of it yet.
01:00:56 But so it's not like something I'm suggesting you go start, you know, rolling out to your company.
01:01:01 Everyone adopt it. Yeah.
01:01:02 But if this is like an exciting project for you, then you could check it out, see where it's at,
01:01:09 join in, whatever. There's definitely tons and tons of stuff to do. But it's, you know,
01:01:16 there's a good solid start. And I think it's at this point, I'm pretty confident like everything
01:01:20 could work, right? You know, kind of the proving it out part is pretty much there, done.
01:01:26 So question from Marwan in the audience. Hypothetically speaking, does that posy.lock
01:01:31 work as is on different platforms?
01:01:33 Right? Yeah. So cross platform support is a huge issue with locking. I don't know if anyone's ever
01:01:41 tried to do this with like pip compile, like it just doesn't work. If you have anything complicated,
01:01:46 like multiple Python versions, like it just doesn't work. You know, good. They tried hard. But yeah.
01:01:51 So what I'm doing right now in posy is I've tried to kind of keep it simple. I just say like,
01:02:00 tell me which platforms you care about, you know, like, you know, you know, recent dish Linux and
01:02:07 Windows 64 and macOS arm and Intel or something, you'll give me like a list. And then it will,
01:02:15 it can go through just like loop through that. And for each one, find the right PI BI, look up the
01:02:19 metadata to figure out which kinds of packages are appropriate to install there and generate a lock
01:02:25 file for each of those. And then, you know, you can somehow like merge the common parts and write them
01:02:29 to a file. So the individual things that like, you know, resolve this set of versions or set of package
01:02:37 requests into a exact set of versions that only runs at for one specific platform at a time. But you know,
01:02:45 you can run it multiple times. There are, it might be possible to do something smarter. So I know like
01:02:52 poetry has some algorithm that I don't really understand very well, where they try to sort of
01:02:57 simultaneously resolve all the platforms into their lock file. And then the way the lock file works is
01:03:02 then you actually you've, it's like only like mostly resolved. And then when you actually go to install,
01:03:08 it does like that last step to try to narrow it down to the exact platform you're on. And I just,
01:03:14 I don't quite, no one's been able to explain to me at how exactly that algorithm works or even like,
01:03:19 I'm not a hundred percent sure it's even like if it's fully correct or if it's heuristics based or
01:03:24 what. So I don't know, but like, you know, I, we can change the, there's lots of options, right? You
01:03:29 know, we can change the code if there's a better way to do it. Just that's where I'm at so far.
01:03:34 Yeah.
01:03:34 But so there's something that people can play with, but it's early days and you wouldn't mind having
01:03:38 help if people wanted to jump in.
01:03:40 No, for sure. If you want, if you've been looking for an excuse to learn Rust,
01:03:43 if you want to, yeah, play around with, you know, cool. I mean, there's interesting problems
01:03:50 in terms of things like how you efficiently resolve, you know, do package resolution.
01:03:55 It's like this whole messy, like logic programming problem. it's, you know, MP complete.
01:04:03 there's just like interesting system engineering problems of like, okay, if we're going to really
01:04:06 make this a really nice to use, like you want it, like, how do you unpack 20 wheels as fast as possible?
01:04:12 You get to use threads and concurrency and all kinds of stuff. yeah, there's lots of,
01:04:16 lots of cool technical bits too. and of course, just making something that's like a joy to use,
01:04:21 fits nicely in your hand. Lots of fun user interface problems. Yeah. I'm excited about it.
01:04:26 If you can probably tell, like, I just, I just love kinds of problems. So yeah, absolutely. I,
01:04:31 so this announcement was on discuss.python.org and I thought, okay, well, it says there's 72 responses.
01:04:39 Let me flip down and see how this landed with people. Right. Okay. Yeah. Okay.
01:04:43 It was, you know, a wide variety of responses. Yeah. Yeah. Well, but I mean, I would say that at
01:04:48 least the top batch, the first bunch of people, Paul Moore deeply involved with pip jumps in and says,
01:04:53 this is beyond awesome. I had realized you were working actively on this. I'll take a look.
01:04:58 I'd love to help out too. Talks about rest a little. Frederick says, really nice to see this.
01:05:03 This is a great direction. Giannis says, well, certainly blew my mind. Count me in on how we could
01:05:09 explore how this might work for conda and so on. And just, I thought it was really, really quite positive.
01:05:16 of how many, you know, next person. This checks many of the boxes of what I have in mind.
01:05:19 So I, it seems like it's landing well with the community. I hope that, I hope that it continues
01:05:25 to make good progress. Yeah. I think the biggest thing is like, there were definitely some folks
01:05:29 going like, okay, but why are you writing everything in rust? Like, especially it's like, you know,
01:05:35 we've spent a lot of effort, not just making standards for Python packaging, but also like implementing
01:05:40 those. So like you can pip install packaging and like right now, and that is a library that
01:05:45 could do things like unpack wheels and access the pipe. I, or I forget exactly which set,
01:05:51 but like a lot of the tricky stuff, you know, parse Python metadata, formats and, you know,
01:05:56 just all these different tricky things. And it's like, why are you re-implementing this? And also,
01:06:01 does it like send the wrong message that like, you know, when we wanted to do something complicated,
01:06:06 we thought pipe, I will, or sorry, Python wasn't good enough. We needed to switch to rust. And I get
01:06:11 where they're coming from. but well, I mean, there's a few things. So one is just that
01:06:17 I thought writing in rust would be fun. You know, I'm not telling you, you can't use Python for
01:06:21 anything. Sure. I mean, we'll, let's take a step back and say, yeah, how would, how would you propose
01:06:25 writing that in Python? Well, so there is no Python, right? That's no, no, it's possible. Right. So
01:06:30 like, so condos written Python, right. But, and the way, so it makes the distribution a little
01:06:34 complicated because like when you, you get your conda.sh or mini conda, you know, like the installer,
01:06:40 it has a Python package inside it, which it unpacks. So it uses it to run conda to install
01:06:45 like another, another Python or whatever it is that you want to install with conda.
01:06:49 but it sort of has one built in and you could do the same thing for something like posy.
01:06:53 That's something like pie installer to build an executable.
01:06:57 Exactly. Yeah. But then for my installer effectively, right. Like, like, just, you know, kind of recursively do that. it's totally, you know, it's totally
01:07:07 something one could do. the, so yeah, but the main reasons, I'm not to go that way is one is,
01:07:14 like I said, like, it's just was more interesting to me. It's one thing. There's also like, you know,
01:07:18 every, language has trade-offs, right. And the exact set of things you want from a package
01:07:25 installer are kind of right in Rust sweet spot and not pythons. So there's sort of
01:07:30 four things that are really important for something, a tool like posy. So it's, there's the initial
01:07:35 install there's, the, how quickly it starts up because like, this is in between you and invoking
01:07:43 Python or whatever it is you actually want to do. there's how quickly you can resolve packages.
01:07:49 and how quickly can it unpack packages? Like, those are the things that, you know, you care about.
01:07:54 Like those are the big load bearing pieces. And those are like kind of four of Python's weakest spots,
01:07:59 honestly. Right. Like, so like we just talked about the deployment part. you can make it work,
01:08:03 but you know, it's not as straightforward as some things. There'd be more possible things,
01:08:07 moving parts, things that could go wrong. It's not the strongest argument, but it is there.
01:08:12 for startup speed, just notoriously when a Python's weak spots, because if it has to like,
01:08:16 do all those imports from scratch every time, something, think tools like Mercurial have struggled
01:08:20 with a lot. you know, lots of Python applications doesn't matter, but you know,
01:08:24 for this particular one, that, that would be a challenge. and then, resolving is big,
01:08:30 heavy, like that MP complete, like just really gnarly burning as many CPUs as you can on complicated,
01:08:36 logical, operations. Again, not Python's strongest point. It's not something you could
01:08:41 like use NumPy for, and it's not like, you know, I'll down or anything. Yeah.
01:08:45 Like you can do something like Cython potentially, using no Gil. Yeah.
01:08:50 But then you're basically writing it in C at that point. Like you're, you're pretty far from core Python.
01:08:56 Yeah. Yeah. And then, and then finally they just, you know, unpacking files. This is totally IO bound.
01:09:01 And so simple that it's actually a big advantage to be able to just like, like, like IO is so fast
01:09:07 these days with like SSDs and everything and NVMe drives that like almost any overhead in the unpacking
01:09:14 path actually is pretty substantial as a relative proportion. So like you, you add like one Python
01:09:20 operation per, you know, you know, a hundred kilobytes written. And that might suddenly be like a 2x
01:09:26 slowdown just because everything else is so fast that even that small amount of Python overhead,
01:09:31 could be large. And, you know, in a tool like this, like people are really sensitive. Like
01:09:34 they really care if it takes 10 seconds versus one second to unpack those, the environment.
01:09:40 Like that's just a huge difference in usability. So I think it's just, it's kind of really an,
01:09:45 like, it just happens to be exact combination things that makes Rust pretty compelling, but you
01:09:50 could do it the other way too. You know, I'm not making a, it isn't meant as a political point.
01:09:55 Yeah. Okay. Got it. All right. Yeah. Python is written in C.
01:09:59 Yeah, sure.
01:10:00 So it's a PYPY, but I mean, the, the, the core, you know, the core bit of it is written in C,
01:10:07 right? No, but yeah. PYPY is written in Python, right? Hence the name.
01:10:10 Yeah, that's true. Yeah. Yeah. C PYPY written in C. It says it right there.
01:10:15 Yeah, exactly. Okay. Interesting. I think we're probably out of time to, to dive much further in
01:10:24 this, but sure. Yeah. I think we covered a lot. I think we covered a lot. I guess.
01:10:28 Anything else you want to give me your thoughts on the future. Like, what do you think? Does,
01:10:32 is the PEP gaining traction, right? This is in draft mode. I don't know how much I emphasized at the
01:10:36 beginning, but it is not an accepted. Ah, yes. That is important to be clear on that. Yeah.
01:10:40 Yeah. It's just, I wrote it. I posted it for feedback. That's as far as it's gone. It, you know,
01:10:45 there's no, there's no commitment on anyone's part to like that. This is what's actually going to
01:10:50 happen. That said, I'm pretty optimistic. Like, like I said, I got a little bit of pushback
01:10:56 on the posy part because of like the rust and whatever, but, I don't think I've gotten,
01:11:02 I can't think of really like any pushback on the PEP 7.11, the actual PYBI part, except people are
01:11:08 like, well, but why aren't you using conda or something? Which, you know, fair question, but you know,
01:11:11 there's an answer and I don't think anyone like, like it's not something that like the people that
01:11:17 you need use agreement, you need to get this accepted or like, you know, the PYPI maintainers
01:11:21 and Python packaging maintainers, and they are totally okay with like conda is not the solution
01:11:25 to everything, obviously. Right. Sure. Well, how complicated would it be to have
01:11:29 bit conda into this particular use case? Right. Yeah. I mean, there's definitely room to collaborate
01:11:34 better there. and I would love to see that in the future, but, but yeah, but my sense is that
01:11:39 there's just really hasn't been a lot of like, people just seem pretty much like, yeah, this is
01:11:43 cool. I guess actually the biggest thing is that there's been, some feedback from folks like the
01:11:49 PY oxidizer folks saying like, Hey, we would like a bit more metadata so we can fully, you know,
01:11:54 to solve some of our other things we want to do. Like we want to be able to cross compile for a given
01:11:59 Python and we need to know a bit more about the target Python in order to do that. so that's just a
01:12:03 very technical, it's like, yeah, okay. More stuff we should add and tweak.
01:12:06 It's not against the idea of evolving it. Yeah. I think the basic idea, generally people seem to be
01:12:12 on board. I'm not going to like, you know, make a commitment to what like Python.org and PYPI
01:12:17 and all they're actually going to do, but my, I don't, I'm a pretty hopeful. I think that, you know,
01:12:23 they're definitely some of the folks involved in like building the Python.org downloads right now
01:12:27 are like, Oh yeah, I'd build one of these if that was standard. Sure. so it is, it isn't all signed off
01:12:32 on and you know, full, but there seems to be a pretty reasonable consensus that like,
01:12:36 this is a good direction that we're interested in moving in. So, well, it sure caught my attention
01:12:40 when I saw it. So, I'm excited to see. Yeah. Well, thanks for having me on to, yeah. Talk about it.
01:12:45 Yeah. You bet. No, let me just ask you real quick, the final two questions since it has been five years
01:12:50 since I asked them of you. Python code. Do you work on this? what editor are you using these days?
01:12:56 I am using Emacs same as I've been since I was 13. So that's not a political position. That's just, I, you know,
01:13:04 I'm stuck. Like all your commands to come in chords, right? Okay. You know, like that's it. Yeah. Excellent.
01:13:11 And then notable PYPI package. just some random PYPI package. Yeah. Just something you like ran across.
01:13:17 Gosh, that's awesome. People should know about this. It could be very popular or not popular at all.
01:13:21 Oh man. Shoot. I did not prepare for this. Should've. I mean, I don't like, there's some obvious,
01:13:27 like obviously I like, you know, trio has been thinking about a lot, but, that's not an
01:13:31 interesting answer for this. you mentioned rough earlier. I think you mentioned rough,
01:13:37 but rough is pretty cool. Maybe I didn't mention rough. Yeah. Okay. rough is very awesome.
01:13:41 if anyone doesn't know rough is sort of, you know, flake eight, and such re-implemented in rust. So it's like
01:13:47 a hundred times faster. Like you just like instantaneously lends all your code, which is
01:13:51 very sweet. There's a selling point for rust integrated with Python, right? Like
01:13:55 another use case that looks pretty neat. Yeah. And I, you know, sort of, as I'm digging into it,
01:14:00 I'm really impressed at how they, those two, how well they fit together. People put a lot of work
01:14:05 into like making that really smooth and having them collaborate. Well, actually it's something I've just
01:14:09 been working on at work is, we've been having trouble with, so, in an async library like
01:14:16 trio, the, you have lots of tasks running concurrently, but they only like the scheduler
01:14:22 only gets to switch from one task to another when one task explicitly lets go, like says, okay, I can
01:14:27 stop here. We're using an await statement. so it's possible to write code where you accidentally
01:14:32 like don't do that for a long time. And that task will just like hog, all the runtime and block
01:14:37 other tasks from running. And it'd be nice. It's hard to kind of tell when that's happening. and
01:14:41 a similar thing could happen with the GIL. so if you have like an extension library, like, you know,
01:14:46 PyTorch or something, and they forget to drop the GIL before doing some big heavyweight operation,
01:14:50 then it could just block any other threads from running. that's really awkward. We've been
01:14:54 having trouble with that. And, but well, you know, like PySpy, there's another really cool package.
01:14:59 If anyone's seen it, is a rust profiler for Python that can just sit outside your process and,
01:15:04 can, like give you, you know, tell you what it's doing. and, but also it being in rust
01:15:10 and it's up on, you know, crates.io, I could just write a little program that imports, PySpy
01:15:16 is use it as a library, but, and tweak it. So that instead of looking for like, where's code
01:15:21 spending time, it detects, okay, is something hogging, you know, the GIL or the run loop and,
01:15:27 and, and give me the trace back, show me which code is doing that. and it's again, really neat to be
01:15:33 able to, you know, get that really deep insight into this Python stuff that, you know, we're still,
01:15:37 you know, we were using it. It's still Python, but, the rust really is a great flavor that goes with it.
01:15:43 Cool. PySpy. All right. People can check that out. That's Sampling Profiler for Python programs. Yes. Yeah. PySpy is really cool.
01:15:51 Indeed. All right. Well, okay, cool. Thanks for being here. if people are interested in the pep,
01:15:56 what should they do? I mean, I guess the, the post on discuss.uh, python.org,
01:16:03 is the best place for like feedback. It's also, where I posted about, Posey. So if you want to
01:16:09 like see the discussion or join in, that's a good place. If you want to like, you know, help then yeah.
01:16:15 GitHub.com/njsmith/posy is the repository, you know, jump in, send PRs, file issues, whatever.
01:16:22 or just, you know, send me a, you know, I don't know. Yeah. What's the toot at me, I guess,
01:16:27 the Mastodon version. I'm not really on Twitter these days, but yeah. njs@mastodon.social.
01:16:34 and I'll see it. Cool. All right. Well, Nathaniel, thanks for being here. Thanks for this
01:16:38 pep. It looks interesting. Yeah. Thanks. It's great. Great being here. Yeah, you bet.
01:16:43 That this has been another episode of talk Python to me. Thank you to our sponsors. Be sure to check
01:16:50 out what they're offering. It really helps support the show. Take some stress out of your life. Get
01:16:55 notified immediately about errors and performance issues in your web or mobile applications with
01:17:00 Sentry. Just visit talkpython.fm/sentry and get started for free. And be sure to use the promo code,
01:17:07 talk Python, all one word. Want to level up your Python? We have one of the largest catalogs of
01:17:12 Python video courses over at talk Python. Our content ranges from true beginners to deeply advanced topics
01:17:18 like memory and async. And best of all, there's not a subscription in sight. Check it out for yourself
01:17:23 at training.talkpython.fm. Be sure to subscribe to the show, open your favorite podcast app and search
01:17:29 for Python. We should be right at the top. You can also find the iTunes feed at /itunes, the Google Play
01:17:35 feed at /play and the direct RSS feed at /rss on talkpython.fm. We're live streaming most of our
01:17:43 recordings these days. If you want to be part of the show and have your comments featured on the air,
01:17:47 be sure to subscribe to our YouTube channel at talkpython.fm/youtube. This is your host,
01:17:53 Michael Kennedy. Thanks so much for listening. I really appreciate it. Now get out there and write
01:17:57 some Python code.
01:17:59 it's just a couple of tips on the show. I'll see you next time. I'll see you next time. I'll see you next time.