By now, surely you've heard how awesome Pydantic version 2 is.
The team led by Samuel Colvin spent almost a year refactoring and reworking the core
into a high-performance Rust version while keeping the public API in Python and largely unchanged.
The main benefit of this has been massive speedups for the frameworks and devs using Pydantic.
But just how much work is it to take a framework deeply built on Pydantic and make that migration?
And what are some of the pitfalls?
On this episode, we welcome back Roman Wright to talk about his experience converting Beanie,
the popular MongoDB async framework based on Pydantic, from Pydantic 1 to 2.
And we'll have some fun talking about MongoDB while we're at it.
This is Talk Python to Me, episode 432, recorded August 16th, 2023.
Welcome to Talk Python to Me, a weekly podcast on Python.
This is your host, Michael Kennedy.
Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython,
both on fosstodon.org.
Be careful with impersonating accounts on other instances.
There are many.
Keep up with the show and listen to over seven years of past episodes at talkpython.fm.
We've started streaming most of our episodes live on YouTube.
Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows
and be part of that episode.
This episode is brought to you by Studio 3T.
Studio 3T is the IDE that gives you full visual control of your MongoDB data.
With a drag and drop visual query builder and multi-language query code generator,
new users will find they're up to speed in no time.
Try Studio 3T for free at talkpython.fm/Studio 2023.
And it's brought to you by us over at Talk Python Training.
Did you know we have over 250 hours of Python courses?
And we have special offers for teams as well.
Check us out over at talkpython.fm/courses.
Roman, welcome back to Talk Python to Me.
Hi.
Hey, so good to have you back on the show.
See you again.
Yeah, you as well.
Thank you.
How have you been?
So it was a nice adventure for me.
Like, these two years of building Python projects, of moving to other countries.
So, yeah.
Are you up for sharing that with people?
What are you up to?
Sorry?
Are you up for sharing where you've moved to?
Last time we spoke, you were in Berlin, I believe.
I was in Germany.
And now I'm in America.
So, completely different country.
Completely different continent.
Yeah.
Are you enjoying your time there?
Yeah.
So, I like it so much.
Honestly, I like Germany too.
So, Germany is a great country.
But I like different things.
Yeah, but you're probably looking forward to not freezing cold winters.
I want to try.
Honestly, I don't know if I will like it or not.
But I want to try until just a few years, for example.
I moved to San Diego, California a long time ago.
I didn't really enjoy the fact that there was no winters and there was no fall.
It was just always nice.
Always.
Until one day I realized, you know what?
We just headed out mountain biking in the mountains.
And the weather was perfect.
And it was February.
And we didn't even check if it was going to rain or be nice.
Because it's always nice.
You know what?
That's a good trade-off.
You could do a lot of cool stuff when you live in places like that.
So, I'm glad to hear.
I'm glad to hear.
You'll have to let us know how it goes.
And Beanie has been going really well as well.
Right?
So, when we first spoke, Beanie was just kind of a new project.
And the thing that caught my eye about it was two really cool aspects.
Asynchronous and Pydantic.
I'm like, oh, those things together plus MongoDB sound pretty awesome.
And so, it's been really fun to watch it grow over the last two years.
And it's gaining quite a bit of popularity.
I don't know.
It's kind of popular, but not that much.
As Pydantic itself, as FastAPI.
But, yeah, still popular in context of MongoDB.
I think it's a little bit different than a web framework, potentially.
Right?
Like, it's hard to say, you know, is it as popular as FastAPI or Flask or something like that.
Right?
Because those things, you know, those are the contexts in which this is used.
But not everyone uses Mongo.
Not everybody cares about async Mongo.
All that, right?
There's a lot of filters down.
But it's, I think you've done a really great job shepherding this project.
I think you've been really responsive to people.
I know I've seen the issues coming back and forth.
I've seen lots of releases on it.
And I guess the biggest news is around, yeah, you're welcome.
I think the biggest news is around, you know, when Pydantic 2 came out.
That kind of changed so many things.
You know, I spoke to Samuel Colvin about his plan there.
I spoke to Sebastian Ramirez from FastAPI about, like, what he was thinking and where that was going.
And it sounded like it was not too much work for people using Pydantic, but quite a bit of work for people like you that was, like, deep inside of Pydantic.
Yeah?
Yeah.
So, honestly, I have a Discord channel for support Bini users.
And there I was talking with other guys, like, probably, I'm sorry,
but probably I will not support both versions at the same time.
Maybe I will have two different branches of Bini with V1 and V2 supporting.
But finally, I am ended, you know, in a single branch.
And this was very challenging, honestly, because...
Okay, interesting.
I didn't realize you were going backwards in that maintainability there for the people who didn't want to move to Pydantic 2.
Yeah, there are legacy stuff.
It must support new features for Pydantic 2.0 also, definitely, because it may be...
Maybe people want to move to Pydantic 2.0, but they could be stuck on other libraries that support only V1, for example.
And it can go for a while, like, months.
And even for me, it took, like, three weeks to make it work.
Just so people know, pockpython.fm and pythonbytes.fm, both of those are based on MongoDB and Bini.
And when you came out with the new one, I saw when the release for V2 came out, like, all right, how long until Bini supports this?
You know, and I saw that you were, like, right on top of it and working on that.
That was great.
And then when I went to upgrade it, it was really easy, right?
Just I use pip-tools and I use pip compile to just get all the latest versions of the dependencies and update the requirements.
So then I installed a new one and it wouldn't run because there's not because of anything that Bini did,
but just some changes to Pydantic 2.
For example, if I had a, let's say there's a database field that was URL and it was an optional string.
In Pydantic 1, you could say URL colon optional bracket str.
That's it.
But there's no default value explicitly set.
So in Pydantic 2, that's not accepted, right?
It says, no, no, no.
If you want it to be none by default, you have to set it to be none explicitly.
So I had to go through and, like, find all my database documents.
Basically, anytime there's an optional something, set it equal to none.
And then that was it.
That was the upgrade process.
And now the website runs faster.
Thank you.
Welcome.
Yeah, I invented a middleware, like a few classes and functions that just check if you use Pydantic
1 or Pydantic 2.
And based on this, you use a different kind of backends.
But interface is the same for both.
So I'm a unified interface inside of Mini.
So before we get too far down this conversation, give us two quick bits of background information
here.
First of all, why MongoDB?
There's a lot of excitement around things like MySQL, but especially Postgres, relational
databases, MongoDBs, document database.
Give us the elevator pitch.
Why do you like to work with Mongo?
Honestly, I like to work with all the databases.
I become databases fun and nerd.
So I like them all.
But yeah, MongoDB is a document database.
And it means the schema, the data scheme is much more flexible than in SQL databases.
Because in SQL, you use tables, plain tables, while in MongoDB, you use extra documents,
which could be nested.
And the level of this nestedness could be really, really high.
There are some trade-offs based on this.
The relation system for plain tables could be implemented much simpler than for documents
because this flexy structure is hard to make nice relations.
But I'd say it's much more useful for if you use nested data structures in your applications,
it's much simpler to keep this same data structure in your database.
And this makes all the processes of development much more easy and more simple to understand
them, I'd say.
You don't have the so-called object relational impedance mismatch, where it's like, well,
you break it all apart like this in the database, and you reassemble it into an object hierarchy
over here.
And then you do it again in the other way.
And all that stuff, it's kind of just mirrored the same, right?
True.
And I really like MongoDB to make small projects.
I mean, when I just want to play with something.
And I play with data structures a lot.
And using Postgres or MySQL, I have to do a lot of migrations.
Because when I change the type of field or just the number of fields, I have to do this
stuff.
And this is kind of annoying, because I just want to make fun and to play.
Yes, exactly.
Exactly.
For me, it's easy to make MongoDB fast.
And it's operationally almost trivial, right?
If I want to add a field to some collection, I just add it to the class and start using it.
And it just, it appears, you know, it just shows up.
And you want adding an asset object?
You just add it.
And it just, you don't have to keep running migrations and having server downtime and all
that.
It's just, it's glorious.
Okay, so that's the background where people maybe haven't done anything with Mongo.
What about Beanie?
What is Beanie?
Really quick for people.
We're talking about it a bit.
But give us the quick rundown on Beanie and why you built it.
Like there were other things that talked to MongoDB and Python before.
Yeah.
So there's a lot of tools.
There is Mongo Engine, which is nice and which is official.
Yeah, I like Mongo Engine too.
Yeah.
But one day I was playing again with new technologies and FastAPI was super new at
time.
It was like, it wasn't that famous at time, like three years ago.
And already was super nice.
And I wanted to play with it before I can use it in my production projects.
And I found that there is no nice, let's say, connector to MongoDB from FastAPI because
there is nothing that could support and Pydantic and asynchronous MongoDB driver motor.
And I decided like, but why?
I think I can implement it myself.
Why not?
And I made a very small, tiny ODM.
I even thought it would be tiny all the time.
It was like, you know, it could support only models of the documents.
It could insert them.
And all the operations of MongoDB, it wasn't hidden inside of Beanie.
You had to use MQL, Mongo query language there.
So I just released this.
And somehow in one month it got not that popular, but people just came to me and like, I like what
you did.
Could you please add this feature and that feature?
And this part works wrongly.
So please fix this.
And I was like, whoa, I didn't know, but I made an open source product.
Wow.
Yeah, that's cool.
Some weird podcaster guy goes, this is great, except for where the index is.
Yeah, this was like first or maybe second week after I published it.
And yeah, you came to my, to GitHub issues.
Could you add indexes?
And I was like, I forgot about indexes.
Yeah, I have to add them.
Yeah, indexes are like database magic.
They're just awesome.
So yeah, and this was kind of playground project.
And now this is nice.
So damn, I'm going to be.
It's been really, really reliable for all the work that we've been doing.
So good work on that.
Let's see.
I guess there's two angles to go here.
One, if we go over the releases, the big release is this 1.21.0, which says Pydantic V2 support.
So I want to spend a lot of time talking to you about like, what was your experience going from Pydantic 1 to 2?
Because as you said, there's the really famous ones like FastAPI and others.
But there's many, many projects out there that use Pydantic.
I wonder if we could get it to show it.
So, you know, GitHub has that feature where it shows used by 229,000 projects.
228,826 projects.
Yeah, they're all the projects with Pydantic now, right?
Exactly.
Just on GitHub, use Pydantic.
So, you know, many of them still haven't necessarily done this work to move to 2.
And so I want to make that the focus of our conversation.
However, since we had a nice episode on Beanie before, before we get into that aspect,
let's just do a catch up on like what's happened with Beanie in the last two years.
What are some of the cool new features and things that you want to highlight for folks?
I added a lot of features, honestly.
But there were a few really big.
I really like one.
I didn't know that it could be needed for anybody.
But I was continuously asked about, please add, add this.
And I didn't want to add.
But finally, I added.
And now I love this so much.
This is called inheritance.
You can inherit documents.
So you can make a big inherited structure like car, then vehicle, then
from here you can inherit bicycle, by car, and from car inherit something else.
And the thing is, everything will be stored in the same collection, in the same MongoDB collection.
And if you want to make statistics over all the types, you can do it.
And when you need to operate only with a type or subtype, you can do it as well.
You can choose what you want to do.
And I know this feature is used in productions now in many projects.
And this is nice.
Yeah, this is really cool.
So when I first heard about it, my first impression was, okay, so instead of deriving from beanie.document, you create some class that has some common features, maybe properties and validation and stuff.
And then other documents can derive from it.
So, like you said, bicycle versus car.
But in my mind, those would still go into different collections, right?
They would go in different collections.
And that would just be a simpler way to have the code that would have a significant bit of reuse.
But the fact that they all go into the same collection and the documents are kind of supersets of each other, I think that's pretty interesting.
I hadn't really thought about how I'd use that.
This portion of Talk Python to me is brought to you by Studio 3T.
Do you use MongoDB for your apps?
As you may know, I'm a big fan of Mongo and it powers all the Talk Python web apps and APIs.
I recently created a brand new course called MongoDB with Async Python.
This course is an end-to-end journey on getting fully up to speed with Mongo and Python.
When writing this course, I had to choose a GUI query and management tool to use and recommend.
I chose Studio 3T.
It strikes a great balance between being easy to use, very functional and remaining true to the native MongoDB shell CLI experience.
That's why I'm really happy that Studio 3T has joined the show as a sponsor.
Their IDE gives you full visual control of your data.
Plus, with a drag-and-drop visual query builder and a multi-language code generator, new users will find they're up to speed in no time.
For your team members who don't know MongoDB query syntax but are familiar with SQL,
they can even query MongoDB directly with Studio 3T using SQL and migrate tabular data from relational databases into MongoDB documents.
Recently, Studio 3T has made it even easier to collaborate, too.
Their brand-new team-sharing feature allows you to drag-and-drop queries, scripts, and connections into permission-based shared folders.
Save days of onboarding team members and tune queries faster than ever.
Try Studio 3T today by visiting talkpython.fm/studio2023.
The links in your podcast player show notes.
And download the 30-day trial for free.
Studio 3T.
It's the MongoDB management tool I use for talkpython.
You can do even two different...
If you want to count all the vehicles, you can do it without making requests to each of the collections,
because everything is in the single collection.
And you can do this with different fields there as well.
And you can make aggregations over all of them.
And even over cars separately.
This is nice.
And yeah, I like it.
Does the record have something, some kind of indicator of what...
Yeah, inside there is...
What class it is.
It's like, I'm a car class.
I'm a bike class.
Yeah, yeah.
You can specify...
Originally, it is called class name or something like this with underscore.
But you can specify which fields would work for this.
So you can specify the name of this field.
And in this field, it stores not only the name of the class, but the structure itself.
Like for bus, it will keep vehicle, car, bus.
Yeah.
So in this field.
Okay.
That's why it will be able to...
Even on the database level, it will understand the hierarchy of this object.
Right.
And so if you want to do data science-y things, you could use the aggregation framework to run a bunch of those types of queries on it.
Right.
Yeah.
And it's better to do all this stuff on the database layer because Python is not that fast with iterations.
While MongoDB is super fast.
Yeah.
And plus, you know, you don't need necessarily to pull all the data back just to read some field or whatever.
Right.
So, yeah, that's really cool.
This is not what I expected when I first heard about it, but this is quite cool.
First time I heard this about this feature, I was like, nobody wants this.
Why do you try to...
But then I found how flexible this is getting to be.
And so, yeah, this is nice.
The reason I guess it's a surprise to me is it leverages an aspect of MongoDB that's in document databases in general that are interesting, but that I don't find myself using very much is in that you don't have to have...
There's not a real structured schema.
And a lot of people say that and kind of get a sense for it.
For me, that's always meant like, well, the database doesn't control the schema, but my code does.
And that's probably going to be the same.
Right.
So there's kind of an implicit static schema at any given time that matches the code.
But you can do things like put different records into the same collection.
You wouldn't do it just like, well, here's a user and here's a blog post and just put them in the same collection.
That would be insane.
But there's, you know, if you have this commonality of this base class, I can see why you might do this.
It's interesting.
Yeah.
And in this context, blog post or video post could be different by structure, but could be stored in the single collection.
One other thing on the page here that we could maybe talk about is link.
I want to tell people about what link.
MongoDB is not a non-relational database, but you can force it to work with relations.
There are even, there is a data type in MongoDB called db-ref, db-reference, which is used to work with this link type in Bini.
So in Bini, with this generic type link, you can put inside of the link any document type.
It can make relations based on this link.
So you can fetch linked documents from another collections using just standard find operations in Bini.
This is, there is kind of magic under the hood.
I use, instead of using find operations, MongoDB find operations, I use aggregation framework of MongoDB, but it is hidden under the hood of Bini.
And so, yeah, and you can use relations then.
And the nice thing about new features, because link already was implemented, I think, two years ago.
But again, I don't come up with my own features, I think.
Every feature somebody asked me for.
And I was asked for another feature to make backtracking, back references for these links.
Like, if you have a link from one document to another, another document should be able to have to fetch this relation backwards.
So in this case, you've got an owner which has a list of vehicles.
But given a vehicle, you would like to ask who is its owner, right?
True.
Okay.
And I implemented this.
I named backlinks, backlink, and it can just fetch it in reverse direction.
That's cool.
And the nice thing about this is it only uses the magic of aggregations, and it doesn't store anything for backlinks fields in the collection itself.
Okay.
In the MongoDB document, you never will find this field for backlink, because everything you need is on the link.
And this is nice.
Yeah, that is really cool.
In the queries, in the find statement, you add fetch underscore link equals true.
And that's kind of like a join.
Is that how that works?
There are options.
Eager versus lazy loading type of thing.
When you find without this option, default it is false.
You will see in the field, in the link at field, you will see only link itself.
It will be linked with ID inside of the object.
But if you put fetch, and you can fetch it manually, like with method dot fetch, it will work.
But when you put fetch links true, it will fetch everything automatically on the database layer.
And yeah, it will return all the link documents.
That's really cool.
Other one?
Lazy parsing.
I mean, we all want to be lazy.
But what are we doing here?
What is this one?
Yeah, so this is, in some cases, Bini could be used for really high load projects.
And sometimes you need to fetch like thousands of documents in a moment.
And the nature of Pydantic is synchronous, not synchronous, because it uses CPU bound operations there.
And when you fetch hundreds of documents, even thousands of documents, you completely block your system.
Because a lot of loops to parse data, to validate data, and etc.
And if you use it in asynchronous framework, this is not behavior that you like to have.
And to fix this problem.
Maybe even in a synchronous framework, it might be the behavior you don't want to have as well.
Yeah.
Right?
Even then.
Yeah, yeah.
This is true.
But even in asynchronous, you don't accept.
Yeah, exactly.
But it's totally reasonable to think, well, I'm going to do a query against this document.
And it's got some nested stuff.
Maybe it's a big sort of complex one.
But you really just want three fields in this case.
Now, you can use projections, right?
Like that is the purpose of projections.
But it limits the flexibility.
Because you only have those fields that were projected.
And in different situations, maybe you don't really know what parts you're going to use, right?
You can have a complicated approach.
Yeah.
So this kind of lets the consumer of the query use only what they need, right?
Yeah.
And so when you use this lazy parsing, Pydantic doesn't parse anything on the initial call.
Like you receive everything and store everything in a raw format in dictionaries, in Python dictionaries there.
And when you call any field of the document, just parse it using Pydantic tools to parse as Pydantic do it internally.
So is this lazy parse primarily implemented by Pydantic?
Or is this something you've done on top of Pydantic?
I implemented my own library for this.
Like it's on top of Pydantic for sure.
But it uses, in Pydantic, there are tools and different in V1 and V2.
So the name of this tool is different, but you can parse something into type, not into base model, but just into type.
You can provide a type and the value to be parsed into this type and it can parse it.
So I use this.
And additionally, I had to handle with all the validator stuff because this is a very important part of Pydantic and you have to be able to validate things.
And with lazy parsing, if it sees that there are root validators, then it will validate it against any field.
Or if there is a field-specific validator, it will validate it against the field if this field was called.
So yeah, I had to do some magic with Pydantic there.
But it, especially with Pydantic V1, which was slower than V2, significantly slower, it was very helpful for people who have to fetch really big amounts of documents and to not block their pipeline in this step.
This is a nice feature also.
Yeah, I think this is a really nice feature.
What's the harm?
And does it make certain things slower if you're going to use every field or, you know, why not just turn this on all the time, right?
That's my question.
Yes, for sure.
There are trade-offs.
If you will use this all the time, it would be like around twice slower than just Pydantic validation if you will use all the fields.
If you will use just a few fields, it would be faster.
But I didn't turn it on by default because in general case, when people just want to fetch 10, maybe 20 documents and use all the fields of them, it would be slower.
That's kind of what I expected.
But if you've got a really complicated document and you only use a few fields here and there, then it seems like a real win.
But you're going to use everything anyway.
And especially for Pydantic V2, when all the validation happens on the Rust layer.
But here I cannot do this because I cannot, you know, I cannot put the logic into the Rust layer because there is no Rust layer for Beanie.
And if you will fetch all the fields of the documents using this lazy parsing, everything will happen on the Python layer instead of the Rust layer.
And it would be as slow as V1.
So we will not see benefits.
It's interesting, even in the V1 version of Pydantic, but now with Pydantic 2 being roughly 22 times faster, that all of a sudden you want to let Pydantic do its thing if it can.
Yeah, true.
Speaking of Pydantic getting some speed up from Rust, is any part of Beanie some other runtime compilation story than pure Python?
Is there like a Cython thing or a Numba or any of those?
The thing about the speed of Beanie is, Beanie is not about, so as Pydantic is very CPU bound, all the stuff happens on the CPU layer.
While Beanie uses mostly input-output operations because it interacts with the database.
And for this, just default, I think, await pattern of Python uses, works the best.
And all the time, if there are any delays, it's most likely about this interaction process between application and MongoDB.
It could be network, it could be just delay from the query and et cetera, but not Beanie because Beanie doesn't compute anything.
Beanie doesn't do very much, I guess, right?
It coordinates motor, the asynchronous engine from MongoDB, and it coordinates Pydantic.
It kind of clicks those together using async and await, a nice query API that you put together.
And so, right, it's more about letting motor be fast and letting Pydantic be fast and getting out of the way, I suppose.
This is true, so yeah.
Beanie is mostly about making some magic and convert Python syntax into MongoDB syntax.
And thank you for that.
That's really nice.
It's super nice the way that the syntax works, right?
The fact that you're able to use native operators, for example, right?
You do the queries.
I really like that.
Yeah, it was.
It is.
I don't like when somebody uses this in production applications.
I mean, when, because it's hard to find problems.
But when we are talking about libraries, this is really nice when it supports Python syntax.
So that's why I decided to implement it.
People shouldn't get too crazy with overloading their own operators.
But as an API, it's really good.
So for example, in this case, you have a sample document and it has a number.
And so the query is sample.find.
And then the argument is sample.number =10, right?
Which is exactly the way you would do it in an if statement.
You know, contrast that with other languages or other frameworks such as Mongo Engine, which I used previously and is nice.
But you would say sample.find and then just number_eq =10, right?
And you're like, I know what that means.
But it's not speaking to me the same way as if I was just doing a raw database query or writing pure Python, right?
Yeah, it sounds like you have to learn another one of English, right?
Exactly.
You've got to like, if you want to do a nested thing, it's the double underscore, you know, it'd be like number double underscore item underscore eq =10.
You're like, oh, my goodness.
So yeah, that's kind of tricky.
Definitely true.
Cool.
Okay, well, let's talk about this, this upgrading story for the 22, 229,000 other folks out there who maybe haven't done this.
So a while ago, back in 2022, almost exactly to the day, a year ago, I had Samuel Colvin on to talk about the plan to move to Pydantic v2, why he did it.
It was really interesting.
So that's worth listening to.
If people want to learn more, as well as I had Sebastian Ramirez and Samuel Colvin on to talk about it live at PyCon.
And that was fun too.
So if people want background on what is the story of Pydantic 2, they can check that out.
But the big announcement was on June 30th, a couple months ago, I guess that's a month and a half ago, Pydantic v2 is here after just one year of hard work.
That was a huge project for the Pydantic folks, which, you know, they've done a great job on it.
And I guess the big takeaway really is that Pydantic v2 is quite a bit faster.
Maybe you could speak to that.
And it's mostly but not exactly the same because, as you already pointed out, the core of it is rewritten in Rust for performance reasons.
I've made a lot of tests.
It is much faster when you talk about validation of the models itself, especially when you validate parts really nested and complicated models.
Then it's much, much faster than Python, then v1 implementation.
While with Bini, you still can see significant upgrade, performance upgrade, but not that much because with Bini works with MongoDB.
And there is this input output operation, which is slow and which could not be upgraded by just by decreasing processing time.
Sure.
When we are talking about simple documents, it's not that visible, like 10, sometimes 20% faster.
But when we talk about nested documents, when there are nested dictionaries or nested lists of dictionaries, then it's much, much faster.
In my Allo test, it is twice faster, v2 against v1.
And I was super impressed by this because I was expecting this, expecting this, that it would not be that, that faster than as a pedantic v2 itself because of this output operation.
But yeah, this is so this crazy.
Right.
Because it's not just the parsing that Bini does, right?
Bini sends the message over to Mongo.
The network does some stuff.
Mongo does its thing, sends it back, serialized as Bison.
And then it's got to deserilize into objects somehow.
And then the pedantic part kicks in.
Right.
And plus all the extra bits you've already talked about.
Right.
So it, it can only affect that part.
But I think it, your example here shows, you know, standard computer science answer.
It depends, right?
As a, as a faster, it depends.
I would guess the more complicated your document is, the bigger bonus you get, which you've already said.
And the more documents you return.
So if I return one record from the database, it's got five fields.
The amount of that processing that is pydantic is small.
But if I return a thousand records and there's all that serialization, like, you know, the database
has kind of done more or less the same amount of work.
It's streamed the stuff back.
But at this, when it gets to Python, it's like, whoa, I've got a lot of stuff to validate and parse.
And I suspect that also matters how many records are coming back.
Yeah, this is true. That's why this affected this case for lazy parsing that was implemented for
v1. And now, now it's not necessary for many cases, only for very extreme, extreme high load.
That's really cool.
What makes me smile from this is the more pydantic you use, the more awesome this upgrade to 2 becomes.
And it's like I said, it's almost no work.
Technically, I had to set all the optionals to be none.
That's not a beanie thing.
That's a pydantic thing, but it's not a big deal.
So upgrading basically means all the parts of your frame, all the frameworks you use that are
built on pydantic get faster.
So for me, when I upgraded the website, it went about, I don't know, 40% faster or something like
that, which is a huge speed up for very little work.
It's already really fast, right?
If you go to, you know, the podcast, you pull up an episode page.
It's like 30 milliseconds.
You go to the courses and you pull up a video to play.
It's got many queries.
It does, but it's probably 20 milliseconds.
So now it's 15 milliseconds or 14 milliseconds.
But still to get that much of a speed up and do basically no work on my part.
That's awesome.
And I'm not using a framework like FastAPI where the other side of that story is also pydantic.
So if you're using FastAPI and beanie, which I think is probably a common combination,
both the database side that gets you the pydantic things is faster.
And then the outbound and inbound processing that the API itself does is a lot faster because
of pydantic.
And so you get this kind of multiplicative doubling of the speed on both ends, right?
The numbers you just told about your website is like this time is only about this communication.
I mean, how bits are going from one computer to another.
There is no computation there probably.
It's super, super impressive.
You know, people say, well, Python is not fast enough.
That may be true for a few very rare, you know, extremely high load situations, but I would bet
it's fast enough for most people.
You know, if your website end to end processing is responding in 15 milliseconds, like, you know,
you've got other parts of your system like CDNs and an amount of JavaScript you send to worry about,
not your, not your, pretty awesome.
Because of indexes also not, don't forget your indexes.
Yeah.
Each time, each time when I have any requests about my things are slow and probably I have
to switch Python to anything else.
Usually the problem is about the data model, not about the processing because, you know, people could
store things a bit, a bit wrongly, a bit too nested or less nested than it should be.
And yeah, indexes.
Okay.
Yeah.
Indexes.
Or you've done a query where you return a hundred things of huge documents and you only want,
like say the title and last updated, but you don't do a project projection.
And so you're sending way too much data.
It's like select star instead of select title, comma, date, you know, something like that.
Sometimes, sometimes, sometimes also works to, to make protection on the database layer,
like to, to make some, like to find maximum elements or minimum elements and to do this
stuff on the database is better than in Python, because in Python, you have to iterate towards
the objects to, to find things which is not that efficient.
So we just have to,
you got to pull them all back, deserialize them and validate them and then iterate over
it rather than just let that happen on just the fields in the database.
That's, that's a good point.
And maybe the query is just bad as well.
Okay.
So what was your experience?
You know, they, they shipped the migration guide about the things that you've got to do.
And if you look at the scroll bar here, the migration guide is large.
I don't know how many pages.
Let's see if I press print, if it'll tell me how many pages.
No, sadly, it doesn't want to tell me how many pages it would print,
but there are many pages here, I guess, if I were to print this out,
it was this a daunting experience.
How did it go for you?
It was very.
So when I just, we should, we can install a potential tool.
Or you're like, let's see if it just works.
Come on.
And just run tests and everything was.
Firstly, it was like only one error.
Nothing can work.
I don't run any tests.
So one here.
Yeah.
I handled this and each test was read and everything.
It was really interesting and challenging.
Interesting because it's kind of really computer science problem sometimes there.
So Pydantic moved a lot of logic to the Rust layer and it got hidden for me as a user of Pydantic.
For example, there is in Python, there is a thing called forward ref.
What is it?
And even you have two classes in a single module, for example, and in one class you have field of the second class.
And in the second class, you have field of type of the first class.
You cannot just put these names without any magic.
You can make input from annotation or you can just write it as a string instead of the class itself.
I see.
But then Python will understand it as a forward ref, forward reference for another class.
And Pydantic can resolve it and make it an actual class finally in the base model.
And in V1, this mechanism was implemented in Python and I could use it and I could use the results of this.
While in Pydantic V2, everything is in the Rust layer.
And when it updates this reference, it updates inside of the Rust and I cannot see the actual class finally.
And I had to implement my own resolver there.
And there are a few stuff like this.
Another one big change was about...
That does not sound easy.
Yeah.
I was like, so will I?
Oh no.
Am I going to do this?
Yes, exactly.
Exactly.
And, you know, for people who maybe haven't played with this, it's really important.
Because Pydantic and hence Beanie depends on the types, right?
The parsing and the validation and all of those things, you have to know exactly what type a thing is, right?
And so if you lose access to the forward reference resolution, that's going to be bad.
Yeah.
For example, in links or in backward links, if you don't clearly understand which document is linked to this document,
you cannot build a query for this.
So yeah.
And I have made this resolver.
Honestly, I just was reading Pydantic with one code and was...
Hold on.
I didn't copy paste, but I was using nearly the same algorithm.
And another big feature was about how validation of the custom types happens.
Yeah.
And get Pydantic core schema.
So everything is in the schema inside of the Rust layer now.
And you can write instructions in the Python, which would be important to this Rust layer and would be called from there.
That's why the whole syntax of this completely changed.
Completely.
And that's why I was thinking like, but so how can I have two completely different syntaxes for the same thing inside of the same class?
And I was thinking like, maybe I have to switch branches now and have Beanie V2 after Pydantic V2 and Beanie V1 and support all the new features in both versions.
And it was, oh no, it will be a nightmare.
You're already busy with one project.
Do you need...
Because I already have Bonnet.
Yeah.
Do you want two projects?
Yeah.
And I already have Bonnet, which is a synchronous version of Beanie, which also would be split into two then.
And I will have four projects, which is too crazy.
Yeah.
No kidding.
I can split myself.
This is a problem.
And yeah, finally, finally, finally, I found...
I honestly, I just went to FastAPI code and I was reading how they deal with this.
And like, nice idea.
I will do the same.
Thank you guys.
And that's the power of open source.
Yeah, it is.
I feel like Sebastian and FastAPI are kind of examples for a lot of different projects and different people.
You know, people look to them for kind of an example.
Yeah, this is true.
And so, and finally, finally, I solved all the problems.
It took me to solve all the problems.
It took me like one, maybe one and a half weeks.
But then I published a better version and there were performance problems with better version.
There were some corner cases that I didn't catch and community found them.
And this is great also about open sourcing, because honestly, I would not be able to find
all of these problems myself for this short period of time.
Definitely.
I guess how much of this is at runtime and how much of it, I guess it's all at runtime,
but how much of this is startup versus kind of once the app is up and running.
So, you know, you've got your beanie init code where you pass the motor async client over and you pass the models.
And it's got to like verify that they all click together correctly.
You know, they all have the settings that say, you know, what database they go to and the indexes are set up correctly and whatever else is in there.
And then once I imagine once that gets processed, it kind of just knows it and runs.
So how much of this stuff that you're talking about was kind of the setup, get things working and how much of it is happening on every query, every insert?
Most of this was about runtime of about every query and every, not even this way.
Not even this way.
Like how do I set up the document itself?
How do I set up validators?
Because I also have validators in the documents to make things simpler and this index changed and et cetera.
So yeah, there are, for example, the config, there was config nested class called config before in Pedantic V1.
And now this is a field called model config, which is, you see, it's completely different interface again, because this is not a class.
This is just a field now.
And I had to, and I was using and still use this config stuff.
And I had to not only switch to the new syntax, but support config class if you use Pedantic V1.
And that's why inside of my classes, I have conditions like which field I want to define based on the version of Pedantic.
And yeah, most of the, most of the changes were about, about this, the tabs of the documents and the tabs of the types, but you use it not on the initialization layer, but on the runtime.
Yeah.
All these things.
You also talked about the performance story.
Do you do any profiling or have any tools like that?
Honestly, I didn't.
So there are P profile and, and other tools to do this stuff.
I was measuring using just time, start, time, end, but I was doing this for, for different parts of the code, just to see what's happening there and here.
And honestly, it was when I faced a performance problem.
And this was because there are some methods in Pedantic V2 that they keep, but marked as deprecated.
They keep from, from V1.
In some cases, I just didn't switch it to the, to the new versions.
And this was the performance problem.
And when I found all the places where I use deprecated methods, then everything, first of all, then.
You switch it to the new, more intended one.
And that's got the, you know, fully optimized version or something like that.
Yeah.
True.
Because, because Bini is, is working with a lot of internal things of Pedantic and it uses this very heavily.
And, sometimes I just don't know that in this, I mean, don't remember that in this, specific part, there is something internal for Pedantic and, I use this also.
And, so I had to check everything and Pedantic code basis is big already.
So it's hard to keep everything in mind.
Yeah.
I feel like that's the challenge for people like you and FastAPI and others.
You have your, your code way deeper in the internals of Pedantic than people who just consume FastAPI or consume Beanie like I do.
This is very interesting.
Like, I mean, this is true computer science problem.
Like when you have to swap interfaces and, and you don't even know where all these interfaces are used and you have to detect them.
So yeah, this is nice.
It's super interesting.
Yeah.
And again, on your, your side, you've got to do that to adapt to the new Pedantic, but you've also got to present some kind of consistent forward looking view to people consuming Beanie.
So they don't have to rewrite all their code too.
Right?
Yeah.
True.
So, and, and all the interfaces, all the,
I didn't change any Beanie interface when I was working with this, like all the interfaces,
still the same for Beanie and, nobody should change their code, which uses Beanie.
I implemented kind of middleware between Pedantic and Beanie and this middleware, have static interfaces.
And inside of this, there are these conditions like if Pedantic V2, then, and, and yeah, this kind of logic.
Nice.
It's pretty good question on the audience from Marwan here.
It says, other than getting the code to execute correctly, were there any gnarly parts you had to figure out to appease type checkers and linters and that kind of stuff?
I don't, remember any changes about this time, about these things like, my pie and, rough.
I use rough for instead of flake eight currently.
Mm-hmm.
I didn't fix this stuff.
Uh, everything was okay.
It was, it was new Pedantic.
I think, I think, guys made a really great job about this to, to make everything work because yeah, I have, I have other checks about my pipe and about rough and everything went smoothly about this.
So yeah, this is, I can't imagine being Samuel and team to have to rewrite Pedantic with such a major refactoring realizing quarter million other projects depend on this.
And then people depend on those projects, you know, and like, how are we going to do this without the world just completely breaking, you know?
Currently they have a discord channel also for Pedantic.
And I see, I see many people asking for, for help for, and, some questions about new, new syntax, but I see that it could be much more because so current, current syntax is, is very similar to, to the previous syntax, but the change it's completely rewritten.
Right.
Oh yeah.
This is, this is very impressive work.
Yeah.
Completely rewritten a good part of it in another language.
Yeah.
Broken into multiple modules and still it seems to go pretty well.
Yeah.
And the assumptions about the default values for, for optionals, that was the only thing that seemed to have caught me out.
You can put in, in, in the configs that, you, you can have options, like the same, use the same logic as for V1, but you just have to mention it in the, in the config now.
But I don't remember.
Honestly, there was a few other things that maybe I ran across, like, and honestly, I don't know why some of these changed.
So it used to be able to call dot JSON on an object.
And now it's model underscore dump underscore JSON or there was dict model dump, right?
Everything is, has prefix model currently in Pydantic.
And I believe this is to not make conflicts with the user logic, because when you, when you have your own, suffix or prefix, then yeah.
Then it's simpler for, for users to, to have their own methods.
And I, I like this solution.
This, this is really nice.
I guess this is the other thing that I ran into is I have a few places I was calling dot dict, I think.
For me as well.
And I have like conditions.
I see.
Anything else you want to talk about, about the migration, like what's, go okay for you or any other thing you want to highlight about how it went?
Honestly, I think I mentioned everything and yeah, it, it, it, it, it's when, it's when very, it's mostly thanks to, to, to Podentic team.
And, yeah, I really, I really like, Podentic V2.
I still really like Podentic V1 actually, like both, but this completely, this completely different libraries inside and very similar,
libraries, I regard the interfaces.
So I really like how, how it's.
Yeah.
It's not too often that you get that massive of a speed up of your code and you didn't have to do anything.
Right.
You just, I was using this library and it was really right in the core of all the processing and now it's a lot faster.
So, so it was my code.
When you read the code, the, the Rust part of the, of Podentic, I started to read it just to learn.
And, this is really nice how it can interact with Python parts.
I mean, I, I can, I can write Rust itself, just logic.
I even implemented database in the Rust, but how to do all this Python stuff inside of another language.
So this is super impressive.
It is super impressive.
We'll get a little short on time here.
Let's see what, I guess I'll give a shout out to this course.
I recently released MongoDB with async Python, which of course uses Beanie as well.
And I created this course on Beanie one dot, something prior to the one dot 21 that switches to Pydantic two and it's all Pydantic one.
And so I was always wondering like, well, how, you know, how tricky is it going to be to upgrade?
And there was really either zero or very little changes.
I think maybe the default values on optionals was also a thing I had to adjust on there.
But if people want to learn all the stuff that we're talking about, you know, just talk Python.fm, go to courses, check out the MongoDB async course.
It's all about Beanie and stuff like that.
I was reading this course also and I can highly recommend it.
They are.
Thank you so much.
One thing I do want to give a shout out to is I use Locus.
Are you familiar with the Locus?
Locus IO?
Yeah.
What a cool, what a cool project.
So this lets you do really nice modeling of how users interact with your site using Python.
And then what you get is, I don't know if there's any cool pictures that show up on here in terms of the graphs,
but you get really nice graphs that show you real time about how, you know, how many requests per second and different scenarios you get.
And on this one, I'm pretty sure when I upgraded it to Pydantic 2 and ran it again,
trying to think of all the variations that you knew, there could be something that has changed that I wasn't aware of.
Like maybe, maybe I recorded on my M1 Mac mini and then ran it again on my M2 Pro Mac mini.
So that could affect it by like a little bit as well, like 20%.
But I think just so using Beanie and FastAPI and upgrading all those, those paths to Pydantic 2 and the respective Beanie and
FastAPI versions.
I think it went 50% or double fast, two times as fast, 100% faster just by making that change.
So that was pretty awesome.
This is great.
Yeah.
I see, I see, I see this 50% also.
And yeah, all I did is I just reran pip-tools to get the new versions of everything and reran the load
tests and look how much faster they are.
And so that's a real cool example of kind of what I was talking about.
So yeah, if you want to see all this stuff in action, check out the the MongoDB with async Python course.
Thanks for coming on the show and updating us on Beanie and especially giving us this look into your
journey of migrating based on Pydantic 1 to 2.
I think that's really cool.
Yeah.
Thank you very much.
Thank you very much for having me here.
Of course.
So before you get out of here, I got a PIPI project library to recommend to people something
besides of course Beanie and Pydantic, which are pretty awesome and obvious.
I would recommend to use motor. This is this is kind of by Mongo, but asynchroneous.
Integrates with async and await perfectly, which is real, real nice.
If you want to do something more low levels and then Beanie, then you have to at least meet with
motor because this is really nice library. And even after that many years, it's still very actual.
And what else? Honestly, I don't have anything.
Yeah. Pydantic is good stuff. Awesome.
Okay. I'll throw Locus out there for people. They can check out Locus. That's pretty cool.
If you are going through the same process, you've got code built on Beanie or just Pydantic in general,
and you want to see, you know, how does my system respond before and after?
Locus is like ridiculously easy to set up, run it against your code, pip install, upgrade,
run it again and just see what happens. I think that'll be a really good recommendation too.
Yeah. Final call to action. People want to get started with Beanie. Maybe people out there already
using Beanie want to upgrade their code. What do you tell them?
Just pip install it. Everything would be, would work fine. So just try it. But at least you have to
upgrade everything. So please write tests.
Absolutely.
And if something will go wrong, go to my Discord channel and me or other people will answer your
questions. Sounds good. All right. Well, congrats on upgrading Beanie. You must be really happy to have
it done. Yeah, I think. Thank you very much.
Yeah, you bet. See you later. See you.
This has been another episode of Talk Python to me. Thank you to our sponsors. Be sure to check out
what they're offering. It really helps support the show. Studio 3T is the IDE that gives you full
visual control of your MongoDB data. With a drag and drop visual query builder and multi-language query
code generator, new users will find they're up to speed in no time. You can even query MongoDB directly with
SQL and migrate tabular data easily from relational DBs into MongoDB documents. Try Studio 3T for free
at talkpython.fm/studio2023. Want to level up your Python? We have one of the largest catalogs of
Python video courses over at Talk Python. Our content ranges from true beginners to deeply advanced topics
like memory and async. And best of all, there's not a subscription in sight. Check it out for yourself
at training.talkpython.fm. Be sure to subscribe to the show. Open your favorite podcast app and search
for Python. We should be right at the top. You can also find the iTunes feed at /itunes, the Google Play
feed at /play, and the direct RSS feed at /rss on talkpython.fm. We're live streaming most of our
recordings these days. If you want to be part of the show and have your comments featured on the air,
be sure to subscribe to our YouTube channel at talkpython.fm/youtube. This is your host,
Michael Kennedy. Thanks so much for listening. I really appreciate it. Now get out there and write
some Python code.
"The RSS feed on YouTube and check out the video. I'll see you next time. I'll see you next time. I'll see you next time."
"The RSS feed on YouTube and I'll see you next time."