Future of Pydantic and FastAPI
Episode Deep Dive
Guests Introduction and Background
Samuel Colvin is the creator of Pydantic, the popular data validation and settings management library for Python. His work on Pydantic has led to a brand-new company and a major update, Pydantic v2, which includes Rust-based performance optimizations.
Sebastián Ramírez is the creator of FastAPI, a high-performance Python web framework that uses type hints to deliver automatic validation, serialization, and documentation. Together, they share a passion for harnessing Python’s modern language features and exploring interesting performance optimizations.
What to Know If You're New to Python
If you’re just starting your Python journey, a basic understanding of data structures (like lists, dictionaries, and sets) and how Python’s type hints work will help you follow along. Many of the discussions revolve around validating data and leveraging Python’s runtime type information. If you’re unfamiliar with these topics, just know that Python's typing system helps you (and your tools) catch mistakes early. The topics in this episode will be clearer once you’ve used Python in data validation or basic web projects.
Key Points and Takeaways
- Massive Performance Gains with Pydantic 2
The new version of Pydantic brings a significant speedup—often cited as up to a 20x improvement in certain validation scenarios. This boost comes from major refactoring in both Python and Rust, drastically reducing overhead for large or complex data structures. Faster validation translates to lower latency in web services and less resource usage overall. Many real-world applications, including those using FastAPI or data pipelines handling large JSON structures, benefit from these performance improvements. Even if your project doesn’t need the absolute best speed, having more efficient operations can open up possibilities or allow you to process bigger workloads with fewer resources.
- Links / Tools:
- Pydantic GitHub
- Rust (general) at rust-lang.org
- Links / Tools:
- Refactoring into Pydantic Core and Pydantic (Python Layer) Pydantic v2 divides the library into a high-performance core (written in Rust) and a Python layer for developer-facing features. This refactoring simplifies code maintenance and allows each layer to focus on its strengths: Rust for speed and Python for readability. The separation also makes it easier to introduce new features without risking major regressions to core functionality. Because Pydantic Core is where the heavy lifting happens, bug fixes and optimizations apply broadly, while the Python layer can evolve around better developer ergonomics. This structure sets the stage for more ambitious updates down the road.
- Partial Rewrite in Rust for Efficiency
Rust was chosen for speed, memory safety, and the ability to integrate efficiently with Python. By shifting critical logic into Rust, Pydantic can handle massive data validation tasks more efficiently than in pure Python. The Rust code also uses Serde for JSON serialization, making it possible to parse data directly from bytes in a very controlled manner. This approach benefits not only CPU-bound operations but can also reduce overall resource usage, since Rust helps avoid performance pitfalls like the Python GIL in certain parts of the workload. While developers need not write Rust themselves, they benefit from the low-level optimizations in Pydantic.
- Links / Tools:
- Tight Integration with FastAPI
FastAPI is one of Pydantic’s largest “customers,” using it deeply for data validation and serialization. Pydantic 2 will allow FastAPI to drop a few internal hacks and fully rely on the Rust-based core for speed. Sebastian Ramírez notes that FastAPI users will often see a speed boost, especially for response serialization, without changing a single line of application code. Additionally, the improved control over how data is parsed and serialized will support more flexible workflows, like returning partial objects without exposing sensitive attributes. This synergy underlines why both projects share a close development relationship.
- Links / Tools:
- Alpha Release and Testing Pydantic v2
An alpha of Pydantic 2.0 has been made available, and early adopters can try it in non-critical environments. The Pydantic team welcomes feedback and bug reports to help finalize compatibility layers and fix any unforeseen edge cases. By experimenting with the alpha, projects can identify subtle validation changes and decide if they need migration paths or code tweaks. The earlier people test, the more likely the final 2.0 release will address real-world issues. This community-driven approach ensures the Pydantic ecosystem remains stable during a major transition.
- Links / Tools:
- New Company and VC Backing for Pydantic
Samuel Colvin announced that he secured venture capital funding from notable firms like Sequoia, which allowed him to hire a dedicated team. This is a rare scenario where a Python open-source project receives major VC support to evolve beyond a single maintainer. The company’s immediate goal is finishing the Pydantic v2 rollout, but over time, they plan to explore commercial services built around Pydantic. While these plans remain flexible, the ability to raise capital demonstrates both the library’s success and the broader trust in open-source Python solutions.
- Links / Tools:
- Implications for Other Libraries Relying on Pydantic Many popular libraries, such as Beanie (MongoDB ODM) and Django Ninja, use Pydantic under the hood for data validation. Improvements in Pydantic v2 mean these libraries can often remove specialized code previously needed for edge-case handling or performance workarounds. As a result, the entire ecosystem benefits—users of these libraries could see response times drop or gain access to new features for data serialization. Engaging with alpha releases helps library maintainers ensure a smooth migration. Ultimately, Pydantic’s enhancements cascade to a wide range of Python web and data solutions.
- Adoption of
Annotated
and Next-Generation Typing Features Both Samuel and Sebastián highlighted Python’s typing evolution, especially theAnnotated
type fromtyping
which simplifies how metadata can be stored with type hints. In Pydantic v2, annotated metadata is used to define constraints likegreater_than=0
or to specify more advanced validation rules. Meanwhile, FastAPI usesAnnotated
to differentiate default values from parameter metadata. This approach benefits other tools like Hypothesis for property-based testing, which can automatically generate valid inputs based on type and metadata constraints. The alignment of typed frameworks is ushering in a new era of clearer, more maintainable code.- Links / Tools:
- Upgrading from Pydantic v1 to v2
The majority of v1 code should “just work,” but some edge cases exist, like certain coercions that are now stricter. Projects that override
__init__
or use advanced features like root validators may need to adapt to the new validator system. Overall, the maintainers plan to offer clear migration documentation and possibly tooling to automate code modifications. Users are encouraged to test v2 in staging or sandbox environments to catch any changes in data parsing behavior. By the time v2 is fully stable, the combination of speed and clearer validators should outweigh migration overhead for most teams. - PyCon 2023 and Community Energy The live interview setting at PyCon 2023 brought about an energetic discussion of success stories and hallway-track conversations. Both guests described the excitement of meeting other open-source contributors face to face for the first time. PyCon fosters these real-world collaborations, leading to better synergy and faster iteration on new features. Many library maintainers gather at PyCon to share direct feedback, which feeds right back into the roadmap. Getting first-hand developer stories continues to play a vital role in guiding Python’s next wave of libraries and frameworks.
Interesting Quotes and Stories
“The best thing about it is I didn’t think about how long it was going to take, so I tried to do everything.” – Samuel Colvin on rewriting Pydantic in Rust.
“One of the coolest parts is that people won’t have to change their code to get that performance benefit—just bump the version.” – Sebastián Ramírez on FastAPI’s upcoming speedups via Pydantic v2.
“I gave a talk in the Typing Summit. Now, there’s an understanding that runtime use of type hints is a legitimate thing.” – Samuel Colvin acknowledging how far Python typing has come.
Key Definitions and Terms
- Pydantic: A Python library for data validation and settings management. It uses type hints to parse and validate input data, returning Python objects with minimal fuss.
- FastAPI: A high-performance Python web framework that uses Pydantic for data handling and validation, offering automatic documentation and easy asynchronous code support.
Annotated
(PEP 593): A typing construct that allows attaching metadata to Python types, helping frameworks like Pydantic and FastAPI do more advanced validation or documentation.- Serialization: The process of converting complex Python objects or models into simpler data formats (e.g. JSON) for storage or transmission.
Learning Resources
- Modern APIs with FastAPI and Python: Dive deeper into building APIs using FastAPI’s type-centric approach.
- Rock Solid Python with Python Typing: Develop a solid understanding of Python’s type hints and how projects like Pydantic leverage them.
- MongoDB with Async Python: See Pydantic in action with Beanie and FastAPI, handling async data persistence with MongoDB.
Overall Takeaway
Pydantic v2 and FastAPI’s next iteration represent a major leap in both speed and developer experience for Python web apps. By harnessing Rust under the hood, Pydantic delivers large-scale performance gains that percolate throughout the Python ecosystem—especially for data-intensive services. Thanks to close collaboration between Samuel Colvin and Sebastián Ramírez, FastAPI stands ready to provide a seamless upgrade path with fewer hacks and more elegant validation. Their combined efforts also highlight Python’s growing support for advanced typing features and the convergence of runtime and static type-checking benefits. Ultimately, these improvements underscore how vibrant and innovative the Python community can be when tackling real-world challenges together.
Links from the show
Samuel Colvin: @samuel_colvin
FastAPI: fastapi.tiangolo.com
Pydantic: pydantic.dev
Pydantic V2 Pre Release: pydantic.dev
Watch this episode on YouTube: youtube.com
Episode transcripts: talkpython.fm
--- Stay in touch with us ---
Subscribe to Talk Python on YouTube: youtube.com
Talk Python on Bluesky: @talkpython.fm at bsky.app
Talk Python on Mastodon: talkpython
Michael on Bluesky: @mkennedy.codes at bsky.app
Michael on Mastodon: mkennedy
Episode Transcript
Collapse transcript
00:00 The release of Pydantic 2.0, its partial rewrite in Rust, and its refactoring into Pydantic Core and top-level Pydantic in Python is big news.
00:09 In fact, the alpha of Pydantic 2 was just released.
00:12 Of course, these changes will have potentially wide-ranging and positive effects on the libraries that are built upon Pydantic, such as FastAPI, Beanie, and others.
00:22 That's why this chance I had to catch up with Samuel Colvin from Pydantic and Sebastian Ramirez from FastAPI together, live from PyCon 2023, was really timely.
00:32 It's a super fun and wide-ranging interview I'm sure you'll enjoy.
00:36 Plus, there's a bit of an Easter egg in the middle.
00:38 This is Talk Python to Me, episode 415, recorded on location at PyCon in Salt Lake City on April 23rd, 2023.
00:47 Welcome to Talk Python to Me, a weekly podcast on Python.
01:02 This is your host, Michael Kennedy.
01:04 Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython, both on fosstodon.org.
01:12 Be careful with impersonating accounts on other instances.
01:14 There are many.
01:15 Keep up with the show and listen to over seven years of past episodes at talkpython.fm.
01:20 We've started streaming most of our episodes live on YouTube.
01:24 Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.
01:31 This episode is sponsored by Sentry.
01:34 Don't let those errors go unnoticed.
01:36 Use Sentry.
01:36 Get started today at talkpython.fm/sentry.
01:39 And it's brought to you by InfluxDB.
01:42 InfluxDB is a database purpose-built for handling time series data at a massive scale for real-time analytics.
01:49 Try it for free at talkpython.fm/InfluxDB.
01:53 Samuel, Sebastian, super nice to see you here at PyCon.
01:56 Welcome to the show.
01:57 Thank you very much for having us.
01:59 It's strange and exciting to do this live and to see you.
02:03 Yes, I know.
02:04 Normally it's remote over screen share over half the world or something like that.
02:09 Yeah, I've been able to talk with you directly.
02:11 Also with Samuel here.
02:12 It's super cool, super cool to hear you.
02:14 Yeah, thank you very much.
02:15 Yes, it's great to be here.
02:16 It's really fun being at PyCon and then doing this is like, yeah, even more fun.
02:19 I've done my talk, so I'm much more relaxed than I would have been if it had been this time yesterday.
02:23 I was just thinking, talking to someone else, like one of the best parts about giving a talk is that when it's over, you can really relax.
02:30 You know what I mean? You're like, okay, now I can enjoy the conference.
02:33 Absolutely, yeah.
02:34 And the parties, because you can't go too big on the parties if you've got to talk.
02:36 I feel like all the best ones were last night.
02:38 I'm afraid. We were at a pretty good one last night, weren't we?
02:41 Yeah.
02:41 Yeah.
02:42 But that was excellent.
02:43 All right, well, really good to have you both at the show.
02:46 I guess you hardly need introductions.
02:49 You both are doing such cool work.
02:51 We've had you on the show several times each.
02:53 So maybe just let's start with a catch up.
02:55 Like you both have lots of big news.
02:57 Don't want to necessarily spoil too much, but you know.
03:00 What have you been up to?
03:01 Yeah, so I raised money earlier this year.
03:03 Well, it was all sorted last year.
03:05 Money came in in January this year.
03:06 Started a company around Pydantic.
03:08 So I've been busy hiring.
03:10 Got a team of seven now.
03:12 One more going to join in June.
03:13 And yeah, currently we're working full-time on Pydantic version 2, getting that released.
03:18 And then after that, we're going to move on to the commercial plans, which I'm not talking about too much, mostly because they're up in the air a bit.
03:25 Also because if you start talking about them, you have to finish talking about them.
03:28 And then that's like, I'll just like take over the whole podcast.
03:30 So I'll say that, yeah, working on Pydantic V2 for now and then moving on soon.
03:35 Well, first from the whole community, congratulations.
03:36 You must be really thrilled.
03:38 Yeah, it's amazing.
03:39 It's very surreal, right?
03:41 Because I was going to say, did you see this coming?
03:43 No, I didn't.
03:44 My plan had been to start a different company once Pydantic V2 was done.
03:49 And then in November, Bogomil from Sequoia, who Sebastian knew, Sebastian recommended,
03:55 he chatted to me.
03:56 We had a call.
03:57 We had another call two weeks later.
03:58 And then he said, let's have the final meeting with like a few more partners to decide whether
04:02 to invest in two weeks time.
04:03 So I thought, oh, I should probably go and speak to some other VCs.
04:07 So Sebastian very kindly got me lots of intros.
04:08 My girlfriend also got me some intros.
04:10 I had like five meetings lined up.
04:12 And then the like floodgates opened and I got another like 20 or so VCs emailing me being
04:17 like, please, can I call?
04:17 Starting to hear about, oh, why are we not part of this?
04:20 Right.
04:21 And then I got COVID.
04:23 So I spent a week like locked in the bedroom upstairs doing VC calls.
04:27 Most of them with the like camera off feeling absolutely horrific.
04:29 And yeah, and then came back full circle back and had the big call with Sequoia and took
04:35 their money.
04:35 They've been amazing.
04:36 So it was Sequoia that invested.
04:37 Yeah.
04:37 Wow.
04:38 Awesome.
04:38 That's a big name to have behind you.
04:40 So Sequoia and Partec who are the smaller VC who are like French American and then Irregular
04:45 Expression, which is this really cool CTO network based kind of again, like New York and Paris.
04:51 And then a bunch of angels.
04:52 Yeah.
04:53 Last time we spoke, it was about Pydantic V2 and then all of this broke.
04:56 Yeah.
04:57 Yeah.
04:57 So I feel like I'm just back up to, as in problem was, although I was doing it to speed
05:02 up, that was there were then two months of basically doing meetings and doing legals.
05:06 So I think I've now got a team sufficiently that I'm like caught up to where I would have
05:09 been if I had just sat there and written code all along.
05:11 Yeah.
05:12 That's how it goes, right?
05:14 Yeah.
05:15 You got to put a little more sand in the gears to grow, I guess.
05:19 And Sebastian, how about you?
05:21 What do you, what have you been up to?
05:22 How have you been?
05:22 Oh, I've been good.
05:24 I've been good.
05:24 Yeah.
05:25 Very excited about like what they are doing at Pydantic is like the team they are assembling
05:29 is like just amazing.
05:30 And like, yeah, just like recently working a bunch in FastAPI and like Typer and actually
05:36 like in also some of the low level things of FastAPI and also not just FastAPI, but like
05:43 the things that go underneath.
05:44 Right now, like one of the things that I am pushing for is having documentation of the
05:50 API reference of the reference of the, each one of the parameters for what it is for each
05:54 one of the methods, like all that stuff.
05:56 Yeah.
05:56 And I want to do it in a, in a better way that is more maintainable and that I can test
06:01 the actual documentation for those parameters and consistency between like, there's a bunch
06:06 of things that I'm trying to do.
06:08 And like, it also goes to the low levels of like typing and interacting with the people
06:12 that is handling typing and like all the stuff that is super cool, super exciting.
06:16 But like, I think it can work and it can, it can make these things have like, you know,
06:21 like the API reference for the tools is something that a lot of people have been requesting and
06:26 like being able to have that in a way that is easy to maintain and that can work well and
06:31 that I can handle.
06:32 Like, I think that's, that's super exciting on that side.
06:34 And on the other side, of course, like the integration with Pydantic V2 is super exciting
06:38 now that they have like the first alpha available.
06:41 It is.
06:42 I mean, here you are going along working on FastAPI.
06:45 Everyone I talked to was just universally impressed with it.
06:49 You know, honestly, like I've never heard a bad thing about FastAPI and people are really
06:54 enjoying it.
06:54 And then here comes Samuel just changing the foundation, changing up Pydantic.
07:00 No, no, no.
07:02 I'm just teasing.
07:03 So how much work is that actually going to be to kind of make the change?
07:07 Is it kind of nothing or is it some work?
07:09 No, like there will be some working FastAPI.
07:11 The thing is for final users, it will be like almost transparent.
07:15 Right.
07:15 They will probably like, if they are doing like weird stuff, complex things like that touches
07:21 the corner cases or things like that, like they will probably have to update some things.
07:25 But for most of the use cases, it will be pretty much transparent for people.
07:29 And they will just like get like the, like, I don't know, like 10, 20x performance from
07:33 Pydantic v2.
07:33 And also like the...
07:35 I was going to say, just on the performance, I'm sorry to interrupt you.
07:37 One of the big things that we will be able to, you'll be able to drop from FastAPI is the,
07:42 I'm going to call it a hack, but it's not your fault.
07:44 It's my fault.
07:44 Of like, don't ask the type problem of serialization.
07:48 So I think that the like speed up on serialization in FastAPI could exceed, like, you know,
07:54 could be even bigger than that.
07:54 I don't know that yet, but I'm really hopeful for some like massive improvements because of
07:59 fixes in Pydantic that make FastAPI simpler and more elegant.
08:03 Yeah.
08:03 And since they're turning up the lights, we'll see how long we last here.
08:07 We'll stay as long as we can.
08:10 If you hear any noise in the background, that's because they're trying to tear down PyCon,
08:13 but we're going to...
08:14 They're tearing down PyCon around us, it seems.
08:16 We will not let it be torn out.
08:17 It will live on.
08:19 It's not because of Pydantic V2.
08:20 So, yeah, we may have to pause and move, but we'll find out.
08:24 Anyway, from a user for Pydantic, Samuel, if you haven't gone to like deeply gone into like
08:32 root object validation and all that kind of stuff, it's probably you won't even know, right?
08:36 So I think the hardest thing...
08:38 Yeah, you're right.
08:38 The vast majority of your code will either continue to work or we'll have a...
08:44 If you get run over by a forklift, it's going to really slow down the development of Pydantic,
08:49 by the way.
08:49 We're going to have a mod tool to change the name of methods.
08:55 So with luck, the vast majority of the changes should be automated.
09:00 I suspect that, and I was saying this earlier in the open space, the hardest thing is probably
09:05 going to be where your API subtly changes in its restrictions because of effectively edge
09:11 cases that Pydantic has fixed.
09:13 So for example, in Pydantic v1, we would coerce a int to a string.
09:17 If you passed a string to an int to a string field, we would coerce it.
09:21 I think that that's wrong and we shouldn't have done it.
09:23 And so now we don't.
09:24 But I was saying in the example, if for some reason you stored your IDs as strings and therefore
09:29 your API had the ID field as a string, but your user was just like pumping them into your
09:34 API as integers because that seemed to make sense to them, that's going to break.
09:38 And you probably haven't gotten a unit test that test that because you know your ID field
09:41 is a string.
09:42 So I feel sorry for those people.
09:44 And my biggest request would be if you're a user, try Pydantic v2 as soon as possible.
09:50 I know if you use it via FastAPI, you can't yet, but like all the other libraries.
09:54 But the sooner you can try it, the sooner you can tell us and the more easily we can fix things.
09:59 And we are prepared to add compatibility shims.
10:02 Okay.
10:03 Well, there, I mean, in Python, we have sort of a from futures import.
10:06 Well, there'd be from a, from history import type of like reverse thing.
10:10 Just slow that down or is it going to be a deprecation or is it just...
10:14 So we're doing deprecation warnings everywhere we can or deprecation errors saying this has
10:19 gone away.
10:19 You probably want to replace it with this thing.
10:20 We're working really hard on that.
10:22 We haven't got a like from future import or a compact layer yet for actual like validation
10:28 logic.
10:29 But if we have to, we will.
10:30 Yeah.
10:31 Okay.
10:31 You'll see, right?
10:32 See how much screaming...
10:34 What we didn't want to do was try and guess at what the problems were and build a compatibility
10:38 layer that people didn't need.
10:39 Yeah, of course.
10:40 So that's why we're doing it this way.
10:41 Yeah.
10:42 That makes a lot of sense.
10:43 You want to go as minimal backwards trying to fill those gaps as possible, right?
10:48 And if I'm brutal about it, if insert name of big bank that use Pydantic locks and never
10:53 engage with the open source community, get stung by this, they never paid me a penny and they've
10:57 never engaged, then like, I'm sorry for them, but I'm not as sorry as I would be if they
11:02 had like come and reported an issue and tried to like help along the way.
11:05 Yeah.
11:06 Yeah.
11:06 Like, can we work with you to just like smooth this over?
11:08 And, you know, worst case, can it be, you know, equal, equal, Pydantic, equal, equal,
11:13 1.10?
11:14 I think we'll carry on supporting critical security fixes for a year.
11:18 Okay.
11:19 So there's something of an LTS type of thing you're thinking?
11:21 Yeah.
11:21 For a while.
11:22 We have to, right?
11:23 For a while.
11:24 And yeah, we'll see.
11:25 Look at the download numbers and play it by ear.
11:27 Yeah.
11:27 All right.
11:28 Cool.
11:28 While we're talking about compatibility, if people are like doing a lot of the overriding
11:33 functions and stuff in their Pydantic models, like what do they, what should they expect?
11:37 Too many changes are pretty similar.
11:38 One of the biggest changes is that the init method of a model is now no longer called when
11:45 you're, unless you literally call init.
11:47 So if you call model validate, or if your model is nested inside another model, init is no longer
11:53 called.
11:53 The solution for that is to use a wrap validator or a model validator, but that's going to
11:58 be one of the pain points for people.
12:00 But there's just, it turns out with the Rust API, it's literally impossible without a massive
12:04 performance hit to do that.
12:07 This portion of Talk Python to Me is brought to you by Sentry.
12:10 You know that Sentry captures the errors that would otherwise go unnoticed.
12:14 Of course, they have incredible support for basically any Python framework.
12:18 They have direct integrations with Flask, Django, FastAPI, and even things like AWS Lambda and
12:25 Celery.
12:26 But did you know they also have native integrations with mobile app frameworks?
12:30 Whether you're building an Android or iOS app, or both, you can gain complete visibility into
12:36 your application's correctness both on the mobile side and server side.
12:40 We just completely rewrote Talk Python's mobile apps for taking our courses, and we massively
12:46 benefited from having Sentry integration right from the start.
12:50 We used Flutter for our native mobile framework, and with Sentry, it was literally just two
12:55 lines of code to start capturing errors as soon as they happen.
12:58 Of course, we don't love errors, but we do love making our users happy.
13:02 Solving problems as soon as possible with Sentry on the mobile Flutter code and the Python server
13:08 side code together made understanding error reports a breeze.
13:12 So whether you're building Python server side apps or mobile apps or both, give Sentry a try
13:19 to get a complete view of your app's correctness.
13:22 Thank you to Sentry for sponsoring the show and helping us ship more reliable mobile apps
13:27 to all of you.
13:27 Do you already have a roadmap?
13:30 Have you already tried the alpha on FastAPI?
13:33 Like, what's the story for you guys?
13:35 So, like...
13:36 Pedantic-wise.
13:37 Yeah, yeah, yeah.
13:37 So, like, we have actually been interacting a lot, like, with what are the changes that are
13:42 needed?
13:42 Like, what is it going to be?
13:43 And like, as someone was saying, like, I have a lot of code that is quite hacky.
13:48 I was actually surprised it didn't break much.
13:51 It just, like, really worked.
13:52 And like, it's for this particular use case where you can have, like, they are so loud.
13:58 They really want to tear us down.
13:59 I know.
14:00 We might have to be weird to say it.
14:02 But let's go ahead.
14:02 Let's finish this up.
14:03 Yeah.
14:03 So, like, validation error.
14:08 Can it concentrate?
14:09 So, imagine this use case where you have user model.
14:13 And then this, you want to return this user model.
14:16 But then you have an authenticated user model, as you were showing in the talk, in the PyCon talk.
14:22 And then this authenticated user model has a field that is a password.
14:26 If you return that use, the authenticated user directly, FastAPI does a lot of tricks to make
14:33 sure that what you receive in the client side is the actual user without the password.
14:38 That is the thing that you declared that you were going to return.
14:41 But by default, if you don't do it through FastAPI, but you do it with just plain Pydantic,
14:46 it will just check, like, hey, is this an instance of the other?
14:48 And then it will include the field.
14:50 Because, like, you know, because in thinking about types, it makes sense.
14:54 Like, oh, this is a subclass of that.
14:56 So, it makes sense that it's valid.
14:58 But when you think about data in an API, it doesn't make sense that it will include more
15:02 data than what it should.
15:03 Right, right, right.
15:04 Because you don't want to either have a mass injection attack on the inbound data or an
15:10 over-exposure on the way out, right?
15:12 You know, give away the password from users.
15:14 Is that bad?
15:15 I think it's pretty bad.
15:16 Okay, yeah, all right.
15:18 Like, you know, for example, like, some months ago or years or something, like, I remember
15:23 that Kaggle, the Kaggle API was returning, like, some of the hashes of the experiments.
15:28 So, like, you know, it's like, it's a mistake and a bit, but it's sad that it will end up
15:33 just, like, filtering more data than what it should be returning.
15:36 And it's something that can happen very easily.
15:38 It can happen very easily to FastAPI applications if people don't specify what is the response
15:43 model, the thing that they want to return.
15:45 And they just return a bunch of data directly.
15:47 So FastAPI does a lot of things to make sure that when you declare a response model that
15:52 should filter this data, the data is filtered.
15:55 But that's a lot of code in FastAPI to make it compatible.
15:58 With the new Pydantic V2, that's going to be pretty much transparent.
16:02 So that's amazing.
16:03 That is amazing.
16:04 Yeah.
16:04 There's going to be, like, a bunch of things that require, like, some refactoring and
16:08 also making sure that the Pydantic V1 and V2 are compatible at the same time in some way
16:14 so that people can have, like, a migration path.
16:16 But yeah, like, we have been making sure that, like, all the things that need to be changed
16:21 or that need to be updated or, like, all the things that need to be exposed from the Pydantic
16:25 side are actually available.
16:26 Yeah, it's awesome that you guys are working so closely together on this.
16:29 I mean, it's going to make it.
16:30 Yeah, absolutely.
16:31 So in my mind, these two projects are pretty closely tied.
16:34 I know that they're not, but that's a big use case.
16:36 Yeah, I think that's true.
16:37 And we know that FastAPI is by far our biggest dependent.
16:41 But also Django Ninja, which is, I think, now second or third, maybe third after SQL model
16:46 by Starz, is, like, Vitaly, who maintains that, has been engaging a lot with us on V2.
16:51 So yeah, lots of other projects are interested in it.
16:54 And I think, yeah, lots of people will be able to remove messy code because of that problem.
16:57 Because, yeah, like, the invariance of the response interface problem.
17:01 That's fantastic.
17:02 Coming back to your previous question about...
17:04 Before we go to that, I think we should probably find it.
17:06 What do you think?
17:07 Yeah, I think you might be right.
17:08 Not even necessarily, I think the audio may be okay, but just for a concentration, it's
17:11 very loud with the trucks around us.
17:13 I feel like I'm on the deck of that aircraft carrier, so I throw things off the side.
17:16 Yeah, okay, let's pause this for a moment.
17:17 We'll be back.
17:18 Hold on.
17:18 So we have survived the disassembly.
17:22 We have returned to continue.
17:24 We were talking about the integration of FastAPI and Pydantic, and that was really cool.
17:29 I think something I'd like to kind of move to real quick is this big announcement, alpha
17:34 of version two.
17:35 Samuel, last time you were on the show, we spoke about the plan for version two, and
17:40 now you're released in an alpha stage.
17:41 Tell people where we are with this.
17:43 Yeah, so we're, yeah, we've had two alphas, maybe three alphas now out.
17:47 We're basically pretty close to a feature freeze, and the plan is to release the package, release
17:54 a beta, and effectively, we hope that we can then release the full release, say two weeks
17:58 after that.
17:59 But there'll be bugs, and we'll fix them, and we'll have more betas.
18:01 But effectively, once we get to beta, the plan is that active development is stopping
18:06 bar fixing bugs.
18:07 Now it's performance and bugs, right?
18:09 Yeah.
18:10 And obviously, one of the big things will be once that's out, there'll be a lot more work
18:13 on, say, FastAPI, Django Ninja, et cetera, et cetera.
18:16 And that might come back with, we really need this thing.
18:18 Either this is broken, or we really need that to make it, you know, to reduce the overhead
18:23 of upgrading.
18:24 One of the things I did for Pydantic 1.10, which was super valuable in beta, was to go
18:30 through packages that use Pydantic, initially sorting by star, but then looking at what they
18:35 actually do, and trying to upgrade them.
18:37 And that, like, found me a bunch of bugs in either libraries or in Pydantic.
18:41 So we're not promising we're going to go and upgrade the whole of GitHub to Pydantic V2,
18:46 but we'll do a bit of that mostly to try and find bugs.
18:50 One of the things that would be really useful is if anyone has an application that uses Pydantic,
18:54 preferably without FastAPI or another library, that they would like help upgrading, we would
18:59 love to come and help.
19:00 And it might be a really powerful way of us, again, seeing the real pain points and identifying
19:05 bugs before we get to V2.
19:07 Yeah.
19:07 And I guess another thing to mention that is a real headline, and I also want to get your
19:12 thoughts on this, Sebastian, is the performance numbers, right?
19:15 I mean, you put out some pretty impressive performance numbers, and Sebastian gets to
19:19 piggyback on that, right?
19:20 Yeah.
19:20 I mean, yeah, I'm really proud of it, right?
19:23 Yeah, you should be.
19:24 I think to go in a change of release, in a bump of release, to be in the ballpark of 22
19:30 times faster, not 22%, but 22 times faster, I don't know of another package that's made
19:35 an upgrade of that kind of order of magnitude.
19:37 What's crazy is it's not numerical computing, right?
19:41 It's general purpose.
19:42 If you look at the example I gave in my talk earlier, it's a completely standard Pynantic
19:46 model with four fields.
19:47 And that's where we're seeing that kind of like 22 times speed up.
19:50 So I think it's going to be massive.
19:51 I have my own cynicism about people who hype about performance as being the most important
19:57 thing.
19:57 I don't think most applications, it's actually the thing that matters most.
20:01 But I think it matters.
20:02 A, it matters to everyone, and everyone wants it to go in the same direction.
20:05 And two, it matters to the whole world and to the whole community that we effectively reduce
20:10 the energy that we consume, like doing compute, basically.
20:14 Right.
20:14 That's absolutely true.
20:15 And also, even if people don't actually need it, there's a group of people who are like,
20:20 well, I'm going to leave for Go because it's not fast enough.
20:22 I can't do enough concurrency or whatever.
20:24 And if they don't have that feeling, even if they didn't actually need that percentage
20:28 increase, that's still really good for the Python community.
20:30 Even me, I was saying I had a gigabyte of data from web analytics that I needed to load
20:36 into a Polar's data frame.
20:38 And for that, I needed to A, extract some attributes from nested fields and B, parse dates and things
20:43 like that.
20:44 And I used Pynantic V2.
20:45 And it was vastly faster with V2.
20:49 It went from like toodling, what's the word I'm looking for?
20:51 Twiddling your thumbs to like it happens virtually instantly, right?
20:56 And that's fantastic.
20:57 And that'll be even more true when you have an order of magnitude or two orders of magnitude
21:01 more data.
21:02 Yeah, for sure.
21:03 Sebastian, how about you?
21:04 What's the knock-on effect for FastAPI?
21:06 So I think one of the coolest things is that people won't have to change the code to get
21:11 that performance benefit.
21:12 It's just going to be like a bump suddenly.
21:13 And because of the new ways that Pynantic can handle the data, we're going to be able
21:19 to, there's something that needs to be done in FastAPI, but we're going to
21:21 be able to let the parsing of the data, let Pynantic handle that in the Rust site.
21:28 So Pynantic will be able to just read the JSON bytes instead of reading them in the Python
21:34 site and let Pynantic do that.
21:35 And then Pynantic give the models back to the rest of the code of FastAPI.
21:39 That alone will boost performance a lot.
21:41 But the fact that it's being done in Rust, in the Rust site, it's just going to be amazing.
21:45 Yeah.
21:46 Like one of the other things that I want to do that is on the plans is to let users
21:50 define how they want to serialize data and not have like, this is just like by default,
21:55 it's just like Pynantic models and like it converts automatically to JSON.
21:59 But I want to allow users to decide how to serialize the objects and the data so that
22:03 they can just...
22:04 Like data classes or something like that?
22:06 Yeah.
22:06 For example, they can say like, oh, I don't want to serialize with the standard JSON serializer.
22:10 I want to serialize with our JSON, which is like the...
22:13 I see.
22:14 Rust based implementation to serialize JSON.
22:17 Or they can say like, I want Pynantic to be the one that serializes this because I know
22:21 that this is just a model that can handle that.
22:22 They can also like...
22:23 And this is one of the things that I think is super cool.
22:25 They can also create...
22:27 They will be able to create a way to say, I want to serialize this response to XML or something
22:33 like that or to YAML and to let Pynantic handle the validation, but then like do the serialization
22:38 in a way that they can customize the whole thing without having to do it directly in the code.
22:43 Maybe even some of these crazy stream buffer type of protocols.
22:47 You know, like, yeah, like protocol buffers with GRPC or like even message pack or like,
22:51 you know, like a bunch of these things that there's no obvious way and there's no native
22:56 way to have support for that, for reading the data and for exporting the data.
23:00 And like, that's one of the things that I have like implants.
23:02 I like, I'm probably saying too much and then I'm going to account all the pieces of the code.
23:06 And now they're like, you know what?
23:07 You promised this.
23:08 You did promise it.
23:09 Well.
23:10 Can I just come back on serialization for a minute?
23:12 Yeah, please.
23:12 Yeah, please.
23:13 Yeah.
23:13 So what we've, I've worked from October, quite like putting to one side, the whole funding
23:19 round in the middle of it was working solidly on serialization.
23:21 So we have, there's almost as much code in Pynantic core now for serialization as there
23:26 is for validation.
23:26 Yeah.
23:27 Well, we can serialize directly to, to Jason using the same underlying library that, or Jason
23:32 uses using Serdi, but one, and also we can, you can customize how your serialization
23:38 goes on a per field basis rather than on a per type basis, which is like incredibly powerful.
23:42 But we also allow you to effectively serialize to Python.
23:46 So not just what used to be the dict method, but basically do JSONable Python.
23:51 So you effectively set the mode when serializing to Python, to Jason, and you will only get the
23:55 like, whatever it is, seven primitive Jason types in Python, which is super valuable if you
24:00 want your output to be XML, because then you know your XML encoder only needs to bother,
24:04 needs to take in dictionaries, list, ints, floats, none, bool.
24:08 Yeah.
24:08 Rather than whatever complex data you have.
24:11 So there's an, or I, yeah, I'm like super proud of lots of the advantages of serialization.
24:15 My 45 minute talk earlier, I was able to touch about half of the, the big new features, which
24:20 kind of talks about quite how much has changed.
24:23 Yeah.
24:23 That's really exciting.
24:24 I think we definitely, I was saying earlier, if I had known how long it was going to take,
24:28 I would never have set out on this journey.
24:30 So the best thing about it is I didn't, I didn't think about how long it was going to
24:33 take because we didn't try and do a bit more.
24:36 We tried to do everything or I tried to do everything.
24:38 And that's how it's disadvantaged.
24:40 It's taken longer than we had hoped, but.
24:41 But here you are.
24:42 You're pretty much here.
24:43 That's really good.
24:43 That's really good.
24:44 And so when you think about performance, right, obviously the 22 times faster is awesome.
24:49 The FastAPI speedup is awesome.
24:51 But if you do something like SQL model and FastAPI or Beanie and FastAPI, you're getting
24:58 on both ends, you're getting that Beanie or the Pydantic benefits with the Beanie integration
25:03 and the fast.
25:04 So you, you're kind of putting Pydantic in both those layers.
25:08 And so those speedups are like twice as good or something like that.
25:11 Yeah.
25:12 Right.
25:12 I think you, well, they're probably, yeah, they're like, they're the same relatively,
25:16 but more in absolute terms.
25:17 Yeah.
25:18 Yeah.
25:18 Yeah, absolutely.
25:19 So, yeah, I think it's, you know, the fact that it's a lot of people, so many things have been built upon Pydantic means you've just sped up a bunch of projects
25:26 without them doing too much.
25:28 Yeah.
25:29 We get the, like the win.
25:30 It's like CPython itself getting faster, helps everyone.
25:32 This is like the next layer down, but we, you know, as a dependency of lots of packages,
25:36 we get to speed up lots of the community with one package, like one person devoting a year
25:41 to it.
25:41 Does this surprise you to see all these projects coming out?
25:44 Like here's another project based on Pydantic plus, you know, name your other thing that
25:48 it's been crazy, particularly in the like machine learning space where, you know, Langchain, who are one of the like big names right now in these like big language
25:56 models, large language models, all based on Pydantic, right?
25:59 Yeah.
25:59 You were saying, I think on Twitter, that OpenAPI use a bunch of FastAPI, right?
26:04 OpenAI?
26:04 Sorry.
26:05 Yeah.
26:06 OpenAI, not OpenAPI.
26:08 Marvin from, I think from Prefect is built on Pydantic again.
26:11 So like the wave of machine learning stuff seems to have leveraged Pydantic a whole lot.
26:16 Docker array being another, another big example.
26:18 Some for Elastic and some other things as well.
26:24 This portion of Talk Python to me is brought to you by Influx Data, the makers of Influx
26:28 DB.
26:28 Influx DB is a database purpose built for handling time series data at a massive scale for real
26:35 time analytics.
26:36 Developers can ingest, store, and analyze all types of time series data, metrics, events,
26:42 and traces in a single platform.
26:44 So dear listener, let me ask you a question.
26:46 How would boundless cardinality and lightning fast SQL queries impact the way that you develop
26:51 real-time applications?
26:52 Influx DB processes large time series data sets and provides low latency SQL queries, making
26:58 it the go-to choice for developers building real-time applications and seeking crucial insights.
27:04 For developer efficiency, Influx DB helps you create IoT analytics and cloud applications
27:10 using timestamped data rapidly and at scale.
27:13 It's designed to ingest billions of data points in real-time with unlimited cardinality.
27:18 Influx DB streamlines building once and deploying across various products and environments from
27:24 the edge, on-premise, and to the cloud.
27:27 Try it for free at talkpython.fm/influxDB.
27:31 The link is in your podcast player show notes.
27:33 Thanks to Influx Data for supporting the show.
27:36 I'm sure one of the big things you're thinking going forward with FastAPI is like how do you
27:42 guys work together and make this a seamless change?
27:44 What else you got?
27:45 What else are you working on?
27:46 What else do you see in the future?
27:48 I have a bunch of things.
27:49 Are they secret or can you tell us?
27:51 No, no, no.
27:52 I can't tell.
27:52 Most of it I can't tell.
27:54 I just feel the accountability.
27:57 But I can't tell.
27:59 So like I have a bunch of projects and like the funny thing is that in some way it's kind
28:04 of a dependency graph of things that I should work on.
28:07 So for example, I have this project generator for FastAPI to generate a project with a SQL
28:12 database.
28:12 I haven't updated it in a long time and it uses SQLAlchemy.
28:16 I built SQL model for that project to use it there.
28:19 But I haven't updated it there because first I want to upgrade more things in SQL model.
28:23 I want to finish the documentation, finish the story about migration.
28:26 But then for the story about migration, I need typer for SQL model.
28:30 So I need to update things in typer.
28:32 And then for typer, I want to add support for annotated, which is actually one of the big
28:36 things.
28:36 One of the recent big things in FastAPI is that now there's support for annotated.
28:41 So annotated is this feature from Python.
28:44 It's like standard Python typings.
28:46 You import from typing, from typings import annotated, and then you can use that to meet
28:52 information to the types that you define for parameters.
28:54 Like what?
28:55 I haven't used this.
28:56 I love typing.
28:56 I use it all the time.
28:57 And here I'm learning more about typing.
28:59 The thing is it exists there in the standard library, but it doesn't have like a canonical
29:04 use in Python itself.
29:05 It's there mainly for FastAPI and Pydantic to use it.
29:09 You know, like it's just that like I haven't, I hadn't pushed for that before.
29:13 But the thing is you import from typings, import annotated, and then you create a function that
29:19 takes like a username.
29:20 And then this function will normally be of type string.
29:23 So it will be like the parameter of the function will be username colon str.
29:28 Now you can say username colon annotated, and then open square brackets as if it was like
29:33 a list or like a dict or something.
29:35 Open square brackets.
29:36 And then the first thing that you put there, that's the actual type.
29:38 So you will say annotated string.
29:40 Ah, okay.
29:41 And then you can pass additional metadata afterwards.
29:43 And the additional metadata is the thing that you will use to tell FastAPI this will be extracted
29:48 from the query parameters or from the cookies or from headers.
29:51 Before, and like up to recently in FastAPI, the only way to do that was using the default value of the parameter.
29:57 Right.
29:57 You would set the default to like a depends or...
29:59 Yeah, to depends or equals a cookie or equals header or something like that.
30:04 And then FastAPI can take the information from that to give you the data in your function.
30:08 But the thing is, if you call that function manually somewhere else, the editor and Python
30:13 won't complain that you are not passing some parameter that is required.
30:16 And then you're going to end up with some strange value internally that is just for FastAPI.
30:20 Right.
30:21 Or the type checker complains, you're not passing a depends.
30:24 Like, no, I'm passing a string.
30:25 That's what it's supposed to be.
30:26 But that's...
30:26 Yeah, exactly.
30:27 Some weird thing like that.
30:28 Exactly.
30:28 Yeah, okay.
30:29 So for those cases, having annotated, like all the type is exactly what it is.
30:32 And if it has a default value, it's the actual default value instead of like some strange
30:37 internal concept in FastAPI.
30:40 And having support for that allows having like much better support for typings, for editors,
30:45 autocompletion, inline errors, all these things.
30:47 It's reusing the same functions in other places.
30:49 And it will also, having support for that in typer, will allow users to have the same
30:54 function being used for FastAPI and typer, having the custom metadata necessary for each
31:01 one of the parameters for FastAPI and for typer and things like that.
31:04 So it's something that is super powerful and super interesting.
31:06 Yeah.
31:07 I'm going to come in on annotated because I'm excited about it too.
31:09 So Pydantic V2, we use annotated for all of our custom types.
31:13 So for example, positive int type is just annotated of int.
31:17 And then we use the annotated types package, which is some reusable metadata for annotated.
31:22 So we would use like positive int is annotated of int and then greater than zero.
31:28 And what's even cooler is that will be used by Pydantic, of course.
31:33 Hypothesis is going to get support for that really soon.
31:36 So it will only pass a positive value in if it sees greater than zero there.
31:40 And then typer, I guess, could like, even if it's still based on click, it can go and take
31:44 that greater than and infer it as, you know, must be greater than zero.
31:47 So I think it was one of the things that typing guys, when they first created annotated, hoped
31:52 was going to build a rich network of libraries that interchanged metadata.
31:56 It's taken a bit longer than they expected, but we're getting there.
31:59 Here the two of you are kind of doing that a little bit, right?
32:02 That's cool.
32:02 That's really cool.
32:03 One of the areas where I feel like typing is a little janky is on ORMs and ODMs.
32:08 When you define a class, you say, for example, it's like a SQLAlchemy column, or it's a beanie
32:15 column or something like that.
32:17 And the type is it's a string column, but really it's a string.
32:21 It's not a string column.
32:22 And so there's this weirdness of using types to kind of drive behavior.
32:28 That's a perfect case for using annotated.
32:31 That's what I was thinking.
32:31 Yeah.
32:31 What it doesn't do is the other case where there is a context where you'd want to get the
32:36 column object of some sort rather than the integer in a row.
32:40 So it does mean two different things, the kind of dot objects in the Django context.
32:44 But yeah, absolutely.
32:45 Yeah.
32:46 It's there precisely to solve this kind of problem.
32:48 And it's also because like currently, as far as I remember, there's no way.
32:52 So the thing is that this is all based on something called descriptors.
32:55 And it's that when you call like, I don't know, class user dot name, it's actually the
33:01 attribute in the class.
33:02 But when you create an instance of that user and then say user dot name, that is the attribute
33:07 on the actual instance.
33:08 And the way that these ODMs or ORMs or these things work is that they have a special way
33:14 to say like when someone calls the actual class, this is a different thing than when someone
33:18 calls an instance attribute.
33:20 Right.
33:20 And there's sort of two behaviors.
33:22 Yeah.
33:22 Yeah.
33:23 And like it's super powerful.
33:24 That's how SQLAlchemy works.
33:26 And it's super powerful because then like all the queries and all the stuff is actually
33:29 like consistent with how Python works.
33:32 And you can say like greater than or equals to using Python syntax, which is great.
33:36 But then currently, as far as I know, there's no way to define that with type annotations
33:41 in a standard way.
33:42 I think it's something that will probably be improvable.
33:45 But I think there's currently no way.
33:46 There will probably be a way at some point.
33:47 But to be able to say like, hey, this SQLAlchemy column is a column when it's accessed
33:53 at the class level.
33:54 But this is going to be a string when it's accessed at the instance level.
33:57 A scope level in the annotated, you know, the class is this and this.
34:01 And it's the instance is that and that.
34:03 Yeah.
34:03 That's really interesting.
34:04 While we're talking types, and I know you both are really big fans because it's such a
34:09 central part, both your libraries.
34:11 What do you want to see coming?
34:13 It feels like this is an area of Python changing fast.
34:15 But is there something like they just if they could just and I have another question on types
34:19 by the way after that.
34:20 If I remember.
34:20 So I gave a talk at the typing summit asking for certain things.
34:23 So now we're going to test my can I remember PEP numbers challenge, which I'm going to fail
34:28 in.
34:28 But pep472 is that keyword args to so one option would be to allow keyword arguments to get
34:36 item, which would make annotated even more powerful because then you could use keyword arguments
34:40 to describe the meaning of your metadata, rather than having to have these kind of identifier
34:44 types like greater than one of the big things that I hope we're going to persuade.
34:49 So I think one of the things that's happened recently is that everyone gets that runtime
34:52 use of type hints is a legitimate thing.
34:54 They might not want to do it themselves, but they get that like, it's a legitimate thing
34:58 to do.
34:58 How much pushback was there when you first came out by Tandic there?
35:01 I think we were like the black sheep of Python.
35:03 I was a black sheep of Python.
35:04 This is supposed to have no meaning.
35:05 What are you doing?
35:06 You're doing it wrong.
35:07 And I think nowadays that's changed and everyone gets it's a real thing.
35:10 But so for example, the hash of a union is independent of the order of the members of the union.
35:16 That makes sense in the context of static typing, where the union of int float is exactly the
35:21 same as a union of float int.
35:23 It turns out in static typing, particularly when you're doing coercion, there are some cases
35:27 where that is not the case.
35:28 And so it's really difficult right now that effectively when you unions are fine on their
35:34 own.
35:34 But if you have a union, say within a string, the capital string square brackets, it will
35:40 the order will be match the order that the first time you call that not what you actually call
35:45 unless you use lowercase string when it is the right order, except there's a PR open right
35:49 now to break it on string to string as well.
35:53 So anyway, we are on lowercase lists as well.
35:58 Anyway, so things like that, where I do think that we'll see what happens on that particular
36:02 case.
36:02 But I feel like the voice of people doing runtime use of types, we're not the only people, are
36:07 being heard better.
36:08 And yeah, I think things are going to continue to improve.
36:11 Yeah, there was a PEP that proposed an optimization for typing that kind of broke the runtime behaviors
36:17 of it a little bit for both of y'all.
36:18 Yeah, it did in some edge cases.
36:20 And that's going to be fixed soon by the successor pep.
36:24 Absolutely.
36:24 So that's really good.
36:26 Generic alias is another thing that kills us internally in Pydantic.
36:30 I won't go into all of the details of it.
36:31 But yeah, the high-level takeaway is that the typing community seemed happy with the idea
36:38 that they might make a change to typing to make it easier for us.
36:41 And I think that's also for the Pydantic team to engage better.
36:44 And instead of spending ages...
36:46 Problem is, right?
36:46 You have a problem.
36:47 You see a solution in typing.
36:49 You submit the PR.
36:50 Even if it gets accepted in a week, which it won't, you wait five years before we can
36:54 remove the code that deals with the other case.
36:56 So it's very tempting not to engage with typing, but just go and write the work around where
37:00 we should be better Python citizens and go and submit the PR to see Python to try and fix
37:04 it properly.
37:05 Yeah.
37:05 What's your wish list for typing?
37:07 So, well, the first thing is like this thing that I have been trying to work on and like
37:11 trying to do to have like better ways to do documentation of the APIs that's also related
37:15 to typing and to the annotations.
37:17 Like, let's see if I can pull it off.
37:19 The other thing is like, it's actually not that necessarily that related to the things
37:23 that we have been talking about, but it's quite relevant for the data science and machine
37:28 learning community.
37:28 That is that there are many, many APIs for many libraries that decorate in some way, some
37:35 function.
37:36 And then the function is expected to give some types of values to the body of the function
37:41 internally, but to be able to receive different types of values.
37:45 That sounds a bit abstract, but like that's the core idea that is like replicated across
37:51 several libraries.
37:52 And this will apply to, for example, to Ray, the distributed machine learning or computing
37:56 system, to Dask, to, I think, Dagster also uses something like that.
38:00 Monad, this system for deploying workloads and machine learning and things like that, also uses
38:06 these types of ideas.
38:07 So there are many of these libraries that like the way that they are designed is that you create
38:12 some function and then you are going to tell something to call this function.
38:16 And then in the function, you say, I want to expect this value.
38:19 And instead of you calling all the functions that will generate that value, you tell it like,
38:23 hey, distributed system, blah, blah, blah.
38:25 Give me this value.
38:26 Execute this for me.
38:27 But that means that you will have or no type annotations or invalid type annotations or red
38:34 squiggly lines in some places or no auto completion or auto completion for the wrong things, just
38:39 because there's currently no way to define this specific thing of saying, hey, this function after
38:45 being decorated is going to be able to receive a different type.
38:49 Oh, yeah.
38:50 Than what it's going to give to internal.
38:52 So I think that's something that, and it's probably quite challenging and like a big thing to tackle,
38:57 but it's something that is replicated across several libraries, in particular for these things
39:01 that do distributed processing of data.
39:04 I think that's something that will be great to improve.
39:07 Does paramspec fix some of that?
39:10 It's very close, but the paramspec only does it for being able to sort of copy the params
39:18 that come from one function to another.
39:20 And actually, I use all that for, for example, for async and for other things to be able to
39:24 get like auto completion for the decorated function or for the generated function or things like
39:30 that.
39:30 And it will probably like the change will probably be somewhere around paramspec to
39:35 be able to say like, not just the paramspec, but like this function will not only have the
39:40 paramspec, but will receive a modification like this of the parameters.
39:45 Almost making paramspec generic.
39:50 All right.
39:50 One more typing question.
39:52 Do you all think typing is going too far with like the generic stuff?
39:56 And is it going too much like C++ and C# and Java?
39:59 Or is it, is it still, still good?
40:02 I think it's, I think the way Python is growing is super interesting because like,
40:07 we all have to agree that Python 3.12 is not the same Python 2.7.
40:12 Yeah.
40:12 It's quite different.
40:13 And I think it's a different in a good way.
40:15 The users are different and the light, the, the focus of the runtime is different.
40:20 And like the things that we can do with types now, and like the fact that in Python, we can
40:26 access these types at runtime, which means like, I don't know, I was always confused with the
40:31 term runtime.
40:32 It's like, what does that mean?
40:33 And it's like, when you execute Python, the same Python code can inspect and like see what
40:38 are those types.
40:39 That's what FastAPI and Pydantic do.
40:40 It's just like seeing like, what are those types?
40:42 We can do that in Python.
40:43 You cannot do that in things like TypeScript, or you cannot do it in Java.
40:48 You cannot do it in many other languages.
40:49 You get access to this typing information to be able to do additional things with that, like
40:53 validation, data serialization, documentation, all that stuff.
40:56 So I think that's, to start, that's super powerful in Python.
41:00 The language in Python for typings is not as powerful as, for example, TypeScript.
41:06 That is just like so much stuff that you can do with that.
41:09 Nevertheless, I feel that in Python is just like, is growing and is growing organically.
41:14 And like, we have growing pains, you know?
41:16 Like, there are some things that is like, oh, this little thing here is slightly incorrectly
41:21 named.
41:22 Yeah.
41:23 But like, now there's a better way to do that in Python 3.10.
41:26 So we don't care much about that slightly incorrect name.
41:29 Things like that.
41:30 Yeah, I feel like there's some tensions of people who are in the mypy side, and they want
41:34 perfect validation of, I want to prove my code hangs together like a static compiler.
41:39 And folks like you all who are like, we want to leverage typing to make it behave in interesting
41:46 ways.
41:46 And maybe that behavior expression doesn't exactly match what it looks statically like, but
41:51 it is.
41:52 Everybody wants it, but it might trip up mypy.
41:55 And I feel like there's this tension between those two things.
41:57 That's kind of what I was thinking when I asked that question.
41:59 I guess there's a little bit of that, but at the same time, there's, is much less than
42:04 you could imagine.
42:05 There are so many people that are so close to, you know, core mypy and these things that are
42:11 actually very excited about the things that we are doing.
42:13 So they're like, you know, like the, it's actually quite friendly, all the communication.
42:17 It's just that there is some people that just don't really care about runtime types and
42:22 that's fine.
42:22 But like, I feel like it's much more, you know, much more closer together and like much more
42:28 stronger the relationship, I think.
42:30 Yeah.
42:30 Yeah.
42:30 That's great.
42:30 Yeah.
42:31 I think actually we've gone in the, we've, typing's got better for someone who's not
42:35 like, it's actually got less verbose, cleaner, easier to understand.
42:38 You don't have to import union.
42:40 You can do pipe operator.
42:41 You don't have to import list from typing.
42:44 You can use list, which makes complete sense.
42:46 Any is an unfortunate one, but I also understand why the any function might, it would not make
42:51 sense.
42:52 I give up.
42:52 I just, I can't deal with this part.
42:54 So yeah.
42:54 No, in general, I think it's got much better.
42:57 I do think that the interchange between runtime, so there's a PEP open now to add to data class
43:03 transforms, a converter function.
43:06 I forget exactly how it works, but I think that is awareness in the static typing space
43:11 that data gets converted when you construct something that looks like a data class.
43:15 So no, I think, I think it's really positive.
43:18 I think we're incredibly lucky that we're like, I could say TypeScript is the other, is in some
43:23 ways the best untyped language typing system.
43:25 But the fact that they're not available at runtime means we're, we're killing it.
43:29 I think I spoke to someone who maintains a library that does a type analysis at runtime in
43:35 TypeScript and all his types are strings and like they're valid TypeScript, but they're
43:40 strings.
43:40 And that's, you know, he was saying that doesn't matter and it's all fine.
43:44 I tend to feel like it probably does a bit.
43:46 We're really lucky to have them at runtime.
43:48 Then you go to the other end where I've been writing a lot of Rust.
43:51 I am like, Rust's great.
43:52 It has many advances, but if you want to just get something done and not have to,
43:55 think too hard about what the types are, it's really nice.
43:58 I write a lot of Python that's untyped when I'm just trying to get something to, something
44:01 to work.
44:01 I'm not a like, everything must have a type on it kind of person.
44:05 So no, I think we're in a really great place.
44:06 And I think most of the advantages are actually cleaning it up.
44:09 So the new six something, 649, the new generics syntax to me, 695.
44:18 There are people who know the names of their numbers of their peps and there's everyone
44:25 everyone's saying.
44:26 That for me, it cleans up generics, right?
44:30 Yes, it's a fundamental change to the language.
44:32 Yes, it makes the syntax of a function look a bit more like Rust or something.
44:36 But like, if you look at it independently of our experience, it's a heck of a lot more elegant
44:41 than importing type var.
44:42 All right.
44:43 Yeah.
44:43 Yeah.
44:44 Let me ask you one that's, this is purely theoretical because I don't think it'll get adopted,
44:48 but we have int, pipe, none.
44:50 We have optional of int.
44:51 A lot of languages have question mark for nullable types.
44:55 Like it'd be int.
44:56 You could even say it like int.
44:57 I'm not sure.
44:58 Is it an int?
44:59 It could be an int.
44:59 It might be nullable.
45:00 I don't know.
45:01 Right?
45:01 Or use int as an int.
45:03 You just know.
45:03 There's no question mark.
45:04 And those types, what are your thoughts about null, call us, annual care?
45:08 I'm really happy with the new situation and not having the optional that isn't optional.
45:12 That's been a problem for a long time.
45:14 So not needing to use optional is being able to use pipe none is great.
45:19 I actually think one of the things that's going to happen with, particularly with the advent
45:23 of the match syntax and with increased use of type dict, we're going to need a new union
45:29 type that operates much more like an enum in Rust.
45:32 So basically a union that keeps track of which member of the enum you have an instance of.
45:38 I keep meaning to build a package to demonstrate what I mean.
45:40 And I haven't got around to it, but like if you have a union.
45:44 Type dict, which is a legitimate thing to do.
45:46 It's effectively impossible without starting to do effectively validation to work out which
45:50 member you're on.
45:51 So I think we need, and it would be really neat if you could use a match expression to process
45:55 each branch of your union.
45:57 Sebastian?
45:57 You already said everything.
45:58 No, but like, you know, like I feel, I feel that way, I was saying that like, I feel
46:04 Python is just like growing and like the typing system is growing.
46:08 I feel it's growing in a very healthy way because it's not, you know, it's not like just some,
46:12 some, some academics hidden in some corners somewhere saying like, this is how it should
46:17 be done.
46:17 I did my thesis on this type system and here we are.
46:19 Yeah.
46:19 And then like everyone should just use it.
46:21 It's just like a lot of hearing everyone and just receiving the feedback from everyone
46:26 and just like growing in the ways that it should grow.
46:29 I think that's amazing.
46:30 I think like we are, you know, it's like a kind of renaissance of like typing in Python
46:35 and like how we can build all these things.
46:37 I think that's amazing.
46:38 I think it absolutely is.
46:39 All right.
46:39 I think we're pretty much out of time.
46:41 We've used up all the various places we've escaped to at a shutting down conference here.
46:44 Final question for you both.
46:46 Just, you know, what's your big takeaway?
46:47 What's the experience like here at PyCon?
46:49 Like how's it been 2023?
46:51 For me, it's been amazing.
46:52 It's my first PyCon in the US.
46:55 Oh, it is.
46:55 Yeah.
46:55 Like I have never been in a PyCon in the US.
46:57 I have been in PyCons in like many other places, but not in the US.
47:00 And like I got to see, I got to put faces to so many handles in Twitter and GitHub.
47:06 I got to meet you in person.
47:08 That's amazing.
47:09 Wow.
47:09 It's great.
47:09 Like a bunch of other people that I only knew, you know, just like on the internet,
47:14 a bunch of core developers and like, that's so cool.
47:16 They are so cool.
47:16 Like I knew they were super cool, but just like, you know, talking on Twitter and like
47:20 didn't see me in person.
47:21 That's amazing.
47:23 It's my favorite part of the whole conference.
47:25 It's just the people and the getting together.
47:27 Definitely.
47:28 I think I attended like two talks.
47:31 I was just on the hallways talking to everyone.
47:34 You feel the hallway track.
47:35 Yeah.
47:35 I was all the way on the hallway track.
47:38 Awesome.
47:38 Well, that's great.
47:38 Yeah.
47:39 I absolutely love it.
47:40 I remember Sebastian and I joined the Language Summit remotely two years ago, the year when
47:44 there was no PyCon.
47:44 And the most interesting bit of the like four hours Zoom call was the five minutes between
47:49 talks when people just chatted.
47:50 And I remember then thinking how cool PyCon must be to have that same group of people
47:54 like in a room rather than on a Zoom call.
47:57 So no, I love it.
47:58 I think it's, I've really enjoyed it.
48:00 Last year was my first year.
48:01 This year is even more fun.
48:03 Yeah.
48:03 I really enjoy it.
48:03 Awesome.
48:04 Yeah.
48:04 It's been great to meet you both in person.
48:06 Meeting you has been the best bit of all.
48:07 Thank you very much.
48:08 No, no.
48:08 It's been really great to spend some time with you all here.
48:10 Thanks for coming on the podcast.
48:12 Part two now here to wrap things up.
48:14 So it's, thanks for taking the time and congrats both on the success of your projects.
48:18 They're amazing.
48:19 Thanks so much.
48:20 Thank you very much.
48:21 Thanks for having us.
48:21 Thanks for seeing us.
48:23 Yeah.
48:23 Bye guys.
48:24 This has been another episode of Talk Python to Me.
48:26 Thank you to our sponsors.
48:27 Be sure to check out what they're offering.
48:29 It really helps support the show.
48:30 Take some stress out of your life.
48:32 Get notified immediately about errors and performance issues in your web or mobile applications with
48:38 Sentry.
48:38 Just visit talkpython.fm/sentry and get started for free.
48:43 And be sure to use the promo code talkpython, all one word.
48:46 InfluxData encourages you to try InfluxDB.
48:49 InfluxDB is a database purpose built for handling time series data at a massive scale for real-time analytics.
48:56 Try it for free at talkpython.fm/InfluxDB.
49:00 Want to level up your Python?
49:01 We have one of the largest catalogs of Python video courses over at Talk Python.
49:05 Our content ranges from true beginners to deeply advanced topics like memory and async.
49:10 And best of all, there's not a subscription in sight.
49:13 Check it out for yourself at training.talkpython.fm.
49:16 Be sure to subscribe to the show, open your favorite podcast app, and search for Python.
49:20 We should be right at the top.
49:22 You can also find the iTunes feed at /itunes, the Google Play feed at /play,
49:26 and the direct RSS feed at /rss on talkpython.fm.
49:31 We're live streaming most of our recordings these days.
49:33 If you want to be part of the show and have your comments featured on the air,
49:37 be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
49:41 This is your host, Michael Kennedy.
49:43 Thanks so much for listening.
49:44 I really appreciate it.
49:45 Now get out there and write some Python code.
49:47 I'll see you next time.
49:48 Bye.
49:48 Bye.
49:48 Bye.
49:48 Bye.
49:49 Bye.
49:50 Bye.
49:50 Bye.
49:50 Bye.
49:51 Bye.
49:52 Bye.
49:52 Bye.
49:52 Bye.
49:53 Bye.
49:54 Bye.
49:54 Bye.
49:54 Bye.
49:54 Bye.
49:54 Bye.
49:54 Bye.
49:54 Bye.
49:54 Bye.
49:55 Bye.
49:55 Bye.
49:55 Bye.
49:55 Bye.
49:56 Bye.
49:57 Bye.
49:58 Bye.
49:59 Bye.
50:00 Bye.
50:01 you you Thank you.