Talk Python to me. Episode number two with guest Jesse Davis recorded Sunday, April 5th, 2015.
Hello and welcome to Talk Python to me, a weekly
podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your
host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy and keep up with the show and listen
to past episodes at talkpythontome.com. This episode, we'll be talking to Jesse Davis from
MongoDB about PyMongo and of course, MongoDB. Before we get to the interview, I have a quick
message to share. Since we launched a week ago, the response has been overwhelming. I've received many
feedback. I want to thank everyone who contacted the show. However, I could use your help to make
sure the show continues to grow and thrive. If you know someone who would be interested in listening to
the show, please send them a link to talkpythontome.com or share this on Twitter or Facebook. Do you know of
someone who would make a great guest or have a great show topic in mind? Send me a note and I'll set it up.
In other excellent news, we have a show sponsor. I want to thank Python Gear from pythongear.com for
sponsoring this episode and you'll hear more about them later. If you'd like to sponsor a future episode,
please contact us at talkpythontome.com slash sponsor. Now onto the show.
Let me introduce Jesse. Jesse Davis is a staff engineer at MongoDB in New York City. He works
on the MongoDB driver team and develops PyMongo and the MongoC driver. He's the author of the async
MongoDB driver called Motor and he contributes to Tornado and AsyncIO.
Jesse, welcome to the show.
Thanks, Michael.
It's really great to have you here on the show. And, you know, we've known each other
sort of as acquaintances for a couple of years. As you know, I'm a MongoDB master,
which is kind of like an MVP community expert program you guys run. So yearly, we'll come up there and we'll have some really interesting conversations.
And we've always enjoyed the sessions where you come down and talk to sort of the external experts about working with MongoDB from Python.
Yeah, we've been doing those about once a year and we've got the next one coming up in a month. And I really look forward to those too. We get some of our best ideas. It definitely creates a year's worth of ideas, if not more, to kind of mull over and implement after each one of those sessions.
Yeah, those are really fantastic meetings. I really, really enjoy them. So I've seen that you've done a ton of stuff with Python. You know, before we get into the details of MongoDB and PyMongo and all that, you know, how'd you get started?
So it's a funny story, really, as they say. I began when I graduated from Oberlin College 15 years ago, I was a C++ guy, to the extent that I knew any programming language, really, particularly well at the age of 22.
I thought that I was a C++ and graphics guy.
flight patterns.
Wow, that sounds like a really interesting thing to jump into.
It was a great gig and I did really, really poorly at it. So I spent about two years there and realized that I wasn't yet a grown-up. I was really screwing up my life and I was a bad software engineer. So I quit the job and went out into the world to try to get my head straight. And I spent a summer biking through France.
And then I spent a year at a Zen monastery in Southern California. And when I checked back with Austin Digital whether they wanted me to come back to work for them, they said no, because I had not proven myself there.
So I came to New York to continue my Zen study with a place called the Village Zendo here and to start being a layman and a software professional again. And there were no C++ jobs in New York at the time.
What kind of jobs were there?
Well, this was fall of 2004. So the whole market was kind of in bad shape. There had been the NASDAQ crash in July of 2001 and then there was September 11th. And New York hadn't recovered from that yet. So all of the C++ people from all of the banks were still unemployed and I couldn't compete with them.
So what I did find was an educational startup called Wireless Generation in Brooklyn. In recent years, it's become Amplify Education. And they were willing to take a shot at me, even though the job was in Python and Oracle. And I didn't know either of those.
That's a long ways from graphics and C++.
Yeah, it was a huge leap. And it was pretty tough, but I had good mentors and I started using Python there.
That's excellent. So then you carried on digging into Python and I saw that you have a ton of open source projects on GitHub that are successful or contribute to them.
And somehow you found your way over to MongoDB from there, huh?
Yeah. So after a few years working for Wireless Generation, I wanted to get a little more breadth. And I also wanted to make sure that I didn't become so senior that I couldn't continue to program, which was a pressure that I experienced there to move into management.
That's kind of the curse of success for some programmers is, you know, you're really good at this. Stop doing it now. Go manage people, right?
Yeah, exactly. And so to jump ahead a little bit, now at MongoDB, we've figured out how to do that by creating this whole separate track of staff engineers, which is the track I'm on now.
Oh, that's excellent.
Data storage layer for applications like that.
Even though MongoDB was brand new at the time, like I started using it at version like 0.8 or something.
Yeah, is this like 2009, 2010 timeframe or something like that?
Yeah, exactly.
And it was such a cool product.
And within the New York tech startup scene, it was such a rarity as a big infrastructure systems project in a New York City startup that when I finally got tired of freelancing and I wanted to settle down and make a substantial contribution to a single product, I called Elliot and I said, I'm ready to come in from the cold.
And he said, great.
That's excellent.
That's Elliot Horowitz, who's the CTO of MongoDB, right?
Yeah, exactly.
Yeah.
So I suspect most people who are listening to this show have heard of MongoDB, although maybe not everybody.
And they might maybe just know it as a buzzword.
Can you give us the quick elevator pitch of what MongoDB is?
Sure.
So it stores your data.
It's a database.
And it stores your data not in rows and columns, but in a non-relational document format.
And the format is called BSON, which is a binary JSON format.
So if you know JSON, MongoDB's data format is very familiar.
It consists of objects, which have a set of key value pairs.
And these documents can also contain arrays, strings, numbers, dates, and about a dozen primitive data types.
MongoDB lets you index and query this kind of object-oriented data in a very rich way.
So among the document databases that we compete with, we have a particular advantage when it comes to our ability to declare multiple indexes on a collection,
the sophistication of our query language and our statistical aggregation capabilities,
and our ability to let you do very complex update operations where you can add a member to a set within a document
or do math on numbers within those documents.
One of the things that I find people coming from a relational database world feel like,
a lot of times they're like, well, it's really cool you can have these kind of hierarchical structures
that more closely match the way your objects look in memory in your program.
But you probably can't query properly deep down with this stuff.
So if I've got, let's take a super simple example, like a bookstore,
and the bookstore has books and the books have reviews as nested, like a nested array.
Well, what if I just want to know all the books that have five-star reviews?
Could I query that?
Right, exactly.
And we do provide that, and that distinguishes us somewhat from the much simpler sort of key value store
or other simplified document database product.
Yeah, definitely.
And I think, you know, of all the NoSQL databases,
MongoDB is one of the few that would reasonably be something you could consider
as your standard general purpose database, not just some kind of high-scale special use case.
Yeah, that's exactly right.
And MongoDB is not the best answer to every single question, obviously.
There is data that is naturally relational,
and then there is data that should naturally be put in some other simpler,
more specialized NoSQL database.
But MongoDB is very much targeted to be the best answer to many questions
questions and a pretty good answer to an even broader set of questions.
So you can use it as your default database in the way that you might have in the past been used to,
MySQL or Postgres being a pretty good answer to many questions and the best answer to many others.
This episode is sponsored by Python Gear.
We know you're a huge fan of Python,
and Python Gear has an excellent way to put your enthusiasm for Python on display.
Visit pythongear.com and pick up Python or Django t-shirts, stickers, and more.
Hand screen printed on American apparel,
these are shirts that are made to last and are very comfortable.
What's more, a portion of all sales will benefit either the Python Software Foundation
or the Django Software Foundation.
Tell Python Gear thank you for sponsoring this podcast
by visiting their site at pythongear.com and ordering a t-shirt.
They're also helping us with a small contest.
We're giving away a free t-shirt to one lucky listener.
Visit talkpythontimi.com, click on Friends of the Show,
enter your email address, and we'll pick a winner before the next episode.
Now, back to the show.
So, I've got MongoDB, and by the way, in case people didn't know,
it's open source, you can go to GitHub and check it out or see the progress.
I've gotten it, but probably I downloaded it from mongodb.org,
and it's running.
Now I've got my Python app.
What do I do?
Right.
So, MongoDB's network protocol is called the MongoDB Wire Protocol,
and it's a basic TCP protocol.
So, you need something that knows how to talk that protocol
and knows how to convert between your Python data structures,
your dicts and lists and strings and numbers,
to BSON and back.
So, you need a driver.
And the standard driver for MongoDB is called PyMongo,
and you install it from PyPI via PIP, install PyMongo.
The current version is about to be 3.0,
which we'll release in just about a week,
which is very exciting.
Yeah, that's big news.
Like, you guys have been trying to have a sort of a major unification
of all the different drivers for the different languages.
Is this part of that effort?
Yeah, that's exactly right.
So, PyMongo 3 has big behavioral and API improvements
and standardizations,
and that those changes are matched by the MongoDB Ruby driver 2.0,
the C driver 1.2,
the nose driver 2.0,
and so on.
And much more than ever before,
we are all converging on the same set of behaviors
and the same set of APIs.
That's really cool.
One of the real benefits of Mongo, I think,
is it has great support for so many languages.
So, if you choose your database,
you're like, oh, wait, maybe this is better from, you know,
Java for some reason.
It still has a good data access story.
So, that's fantastic.
That's getting even better.
Yeah, right.
That's exactly right.
So, we have drivers in 10 programming languages,
and plus, even if you're using something weird like R or Haskell or Erlang,
there's something out there in the community for you
because writing a basic driver is actually fairly easy.
We're really focused on making sure that each of these drivers
feels right to experts in that language.
So, PyMongo is very Pythonic,
and it's written by Python experts,
and its style and its documentation and so on
are all very Python-y
while at the same time balancing that
with some degree of consistency
with the nine other programming languages that we support.
Yeah, there's got to be some interesting tension there.
Huge, huge tension.
It's the toughest problem that we face,
and we are just in the last year or two
really figuring out good bits to tackle that
and to make those decisions correctly.
Yeah, cool.
So, you play a pretty big part in PyMongo, right?
Yeah, I've been Bernie Hackett.
My boss in Palo Alto is the PyMongo developer and maintainer,
and I've been assisting him for the last three years
as his second-in-command.
And my main contributions to the driver are its concurrency design,
its implementation of distributed systems type problem solving,
and the connection pool.
And with the 3.0 release, that's actually kind of done for the moment.
And so I'm putting a lot of that work to rest now
and moving on to become the primary maintainer of the C driver for MongoDB
so that that part of the team can move into the kernel team
and make contributions there.
Oh, that's excellent.
And that's even a little bit back to your roots from Austin,
in some ways, I guess, right, with the C++ story.
Yeah, exactly.
Parts of my brain that has been idle for a decade are coming back online,
and it's a really fun feeling.
I know it is.
Yeah, yeah.
If audience members, if you've been programming Python for 10 years straight,
like I have, I really can't recommend enough
learning a very different programming language or reviving one.
It's incredibly satisfying.
Yeah, and gives you interesting problem solving skills
that you don't necessarily develop if you stay in just one language.
So that's great.
Yeah.
Now, there's a bunch of ways to talk to MongoDB,
even just from Python, right?
So there's PyMongo.
What else is there?
So PyMongo is the general purpose driver,
and it's the most featureful, the most standard,
the best maintained,
but it's not optimized for some specialized use cases.
And you can think of these as CPU-bound versus I.O.-bound use cases.
Right, okay.
So for the I.O.-bound,
cases where you've got a web application
that has a huge number of client connections,
but they're often kind of idle or sleepy connections,
like if you're implementing a chat server
or something with web sockets,
you want to use an async framework in Python,
like Tornado or Twisted
or in the Python 3.4 standard library,
we've got async.io now.
Yeah, that's a cool new feature.
Right, so these are awesome async frameworks,
and they solve that problem brilliantly,
but they've got a gigantic compatibility issue.
none of the existing libraries work with them.
None of the existing sort of outside MongoDB libraries don't work with them,
or like PyMongo itself doesn't work with them,
or how do you mean?
Well, I mean both of those.
So if you've got basically a driver for any database
that's not written specifically for one of these async frameworks,
then it won't work with that async framework.
So you need a specialized database driver for Tornado and MySQL.
You need a specialized database driver for Tornado and Postgres.
Right.
And you need a specialized driver for Tornado and MongoDB.
So I wrote that over the last few years,
and it's called Motor,
because it's taking the beginning of Mongo and Tornado.
Excellent.
Right.
Plus it's a cool name,
and somehow it was not yet taken on PyPI.
So Motor is the now standard official async driver for MongoDB and Tornado,
and over the next year I'm going to be expanding it out to cover async.io next,
and then eventually twist it as well,
so that it will just integrate with whatever you're using right now.
Nice.
And does that work with Python 3 and 2,
or is that sort of a 2 thing for now,
or what's the story there?
That will work with Python 2.6 plus,
so 2.6, 2.7, 3.3, 3.4, 3.5, when 3.5 is out.
So how does that work with different implementations,
like PyPI, for example?
Sure.
In the past, Motor and PyPI didn't work together very well,
but that was about a year ago that I last personally tested them.
It was correct, but it was slow,
due to some very specific details about PyPI.
In recent months,
somebody that I didn't know posted benchmarks
that showed that Tornado, Motor, and PyPI were actually blazingly fast,
but I haven't personally reproduced that,
so at the moment it's just kind of a hopeful sign
rather than something that I would officially endorse.
Sure.
That's really good news, though.
It looks like PyPI is moving on and has a lot of activity there,
so that's really cool.
I agree.
Yeah.
I've also heard of something called Mon Area.
What's that?
Right.
So we've got this other branch of specialization.
So the three categories that I think of are general purpose,
PyO bound, and that's for Motor,
and then there's CPU bound, and that's what Monary is for.
Monary is a NumPy driver for MongoDB.
Oh, that's interesting.
Wow.
Isn't it?
I found out about it a few years ago.
It was written by a quantitative analyst named David Beach.
He needed it for something, some specific financial application that he was doing.
And he noticed that if you stream BSON data through PyMongo and then into NumPy from there,
it's pretty slow.
And your data conversion is typically your bottleneck.
MongoDB is fast.
NumPy is fast.
But converting each number from one data format to the next is very expensive,
and it's a lot of wasted work.
So he wrote a little bit of C code, which queries MongoDB using the C driver rather than PyMongo,
converts the BSON data directly into NumPy arrays without passing through any Python data structure,
and then hands you giant buffers of numbers that it got from MongoDB.
And then you can use NumPy's incredibly fast statistical methods on that data.
That's really fantastic.
So if maybe you're storing a bunch of data in Mongo for big data on numerical type stuff,
this would be the thing for you from Python.
Exactly.
And that's a pretty common use case among financial institutions.
There's also a lot of universities doing big bioinformatics with MongoDB.
And there's generally a lot of use within the scientific community for storing numeric data in MongoDB.
Since NumPy has such a rich set of statistical routines that you can just take off the shelf,
being able to go between the two in Python incredibly fast is an awesome feature.
So Monary can do upwards of a million queries per second on commodity hardware.
And, or query upwards of a million documents per second.
That's amazing.
We've been adding features to it over the years.
We had a couple of interns last summer, Matt Cotter and Kyle Suarez.
And now a new hire who's working for me, Anna Hurley,
is adding more and more features to Monary every month.
So it's becoming better documented.
It's now read-write.
So you can insert some NumPy arrays into MongoDB.
And it's similarly optimized along that path.
And we're adding SSL and authentication, which financial institutions will probably want if they're analyzing financial data.
And that's kind of, that fills in the other portion of the environment where you're doing single-threaded CPU-bound calculations on numeric data within MongoDB.
Yeah, that really does open up the whole science story for MongoDB a little bit more from Python anyway.
That's really cool.
So what surprises you most about what you see people doing with Mongo from Python or even with PyMongo specifically?
What most often surprises me is there are certain mistakes that are incredibly common.
And I wish we could figure out how to stamp them out.
And the main mistake that I see people make is that they create a new Mongo client class instance for every HTTP request.
And so they pay the price of TCP setup, very often SSL and authentication setup, and then the TCP slow start algorithm.
All of this incredible overhead involved in opening a socket.
And then they do one query and shut it all down.
And when they say, oh, and they defeat connection pooling as well.
And there's this sort of strange resistance to just creating a single global variable, which is the Mongo client.
And I don't understand.
I think they're coming from Java, where it's frowned upon to have one module-level global variable.
But it means that among Python programmers who make this mistake, which is a huge number of people, their throughput is probably a third or a quarter or a fifth of what they should be seeing.
Wow, that's amazing.
And it's easy to fix.
I don't understand apps to persuade people.
It's incredibly easy to fix.
That's the thing, right?
Right.
Doing it correctly is easier than doing it wrong, and yet doing it wrong is very common.
Yeah.
Well, hopefully people out there listening will go make a global Mongo client.
I hope so.
And we're also adding a best practices document to the PyMongo documentation in the next release that I hope will further dissuade people from making this error.
Yeah, that'd be great.
Almost would be cool to have a list of, like, these are the top five worst things you could do that we see out in the wild, so don't do them.
You know, something like that would be nice to put up somewhere.
That's a good idea, although honestly, it's really only top one.
Like, if you don't do that, you're probably doing a good job.
Excellent.
So you guys just had a big release as well.
You've kind of changed the entire underlying file system with something called WireTiger, and you've released version 3.0.
What's the quick rundown of that?
That's pretty exciting.
Right.
So it's not the default at the moment.
It's opt-in.
But with MongoDB 3.0, we made the option available to swap our old storage engine with WireTiger.
And the performance characteristics are still a little complex, so I'm not going to make any inexpert pronouncements on that.
Sure.
But for the large number of use cases, WireTiger seems to be much, much higher performance, and it especially has much better concurrency.
So if you're doing a lot of simultaneous writes on certain kinds of hardware, you should see a lot better throughput with WireTiger.
So it's really exciting.
And we also acquired that company.
It was our first acquisition so that we could...
The WireTiger open source project continues, but it means that we have that expertise in-house to make sure that WireTiger and MongoDB work perfectly together.
That's right.
They were from Berkeley DB or something like that originally, right?
I think a lot of the WireTiger people were among the Berkeley DB developers, yes.
Yeah, excellent.
You have some opinions on editors, right?
Like Vim versus PyCharm.
What are your thoughts there?
Yeah.
So in recent years, I've spent a lot of time mentoring young people or just working together with a lot of junior developers.
And what I particularly noticed is that everybody uses Vim for everything.
And it's kind of sad to watch because I say, go find this method.
And then I sort of have to sit around for a little bit, drinking tea and watching them grepping their files and doing like an incremental search through each file with Vim until, you know, they're searching for like death space, create collection or something in order to find this method.
And it's just sad.
And when people watch me, it's just, you know, this isn't bragging.
It's just, if you use the right tool for the job, you can leap around really easily.
And it's not just navigation.
It's for huge search and replace jobs.
PyCharm has all of these modes and all of these wonderful ways of breaking down big code change tasks to make sure that you complete them correctly.
Using a visual debugger is also completely invaluable.
Like when I see somebody say, I have this bug.
And I say, well, what if you try it?
And they say, well, I added 100 print statements.
I know that something's wrong.
And invariably, even though they've spent hours on it, I say, well, let's get it into PyCharm.
And I use a couple of breakpoints and a few watchpoints.
And we find it because that's what a debugger is for.
That's right.
A picture is worth a thousand words and you can kind of say the same about a visual debugger, right?
Yeah, exactly.
You don't have to ask via a print statement, is this what I expect?
Oh, it is.
Okay.
Well, is this what I expect?
Oh, it is.
Right.
You just have everything displayed for you and you can immediately see what it is about the code that isn't matching your expectations.
Yeah, that's awesome.
I totally agree with you.
And I'm a big fan of PyCharm as well.
I use it for all my Python work.
And you can just hit like shift two times and type something and it'll just take you to a file or to a method or to a variable.
It's fantastic.
Yeah, I agree.
It also shows you your unused imports and it checks your style while you go.
So you start making that error as well.
Yeah, you're sort of PEP 8 compliant straight away.
Exactly.
Awesome.
All right.
Well, let me ask you one more question.
So out there on PyPI, there's tons and tons, you know, 65,000, 56,000, something like that packages.
There's some great ones out there.
Do you have any favorites?
PyMongo and Motor maybe, huh?
Yeah.
PyMongo is a great package.
It's one of the most popular.
And Motor is a fun little project written by some smart guy somewhere.
I wonder who that was.
I'm also a big fan.
And so another package I wrote that I'm very fond of is called Toro.
It's T-O-R-O.
And it's a set of things that are like locks, queues, conditions, event variables.
But it's not for multi-threading.
It's for Tornado coroutines.
Oh, okay.
So it kind of extends the analogy between coroutines and threads.
And it makes it possible for your asynchronous coroutines, which are optimized for I-O-bound applications using something like Tornado.
It lets them coordinate using the same kinds of patterns you're used to with threads.
So you can make consumer-producer coroutines using the exact same pattern as you're used to for producer and consumer threads.
I'm working with Tornado's author, Ben Darnell, to contribute Toro piece-by-piece into Tornado so that Tornado will have those features built in with Tornado 4.2.
It has nothing to do with MongoDB.
I think it's one of the things I like about it is that it's just this separate project that I kind of use recreationally.
And it's also just got a lot of really fun patterns that it's fun to think about.
Yeah, that's really cool.
You know, that kind of programming, trying to coordinate asynchronous coroutines, sounds like a really interesting sort of problem or puzzle to solve, you know?
Right.
And a lot of those questions have already been answered for threads.
So if you have the same primitives or if you have analogous primitives to use with coroutines, then you can use the same answers that people have developed for threads for the last few decades.
Fantastic.
So, yeah, people should check out Toro.
Very cool.
People have been listening to us talk and rave about MongoDB.
And in case you guys don't know, I'm a huge fan of MongoDB as well and use it for many projects.
If people are excited, they want to get started, what do they need to do?
So you can pip install PyMongo and then you can go to mongodb.org and download the MongoDB server.
It's among other awesome things about MongoDB.
You install it by untarring it and then starting it, period.
And if you want to learn more, the PyMongo documentation online at api.mongodb.org slash python has a complete tutorial for you.
And if you want to go a little deeper, there's an excellent online course called M101P.
So that's MongoDB 101 for Python.
And the next section of that class starts on May 26th, I think.
Excellent.
And is that education.mongodb.org?
I forget the URL.
Yeah, that's correct.
Yep, exactly.
Excellent.
Okay, well, is there anything else you want to sort of give a shout out to or get people's attention focused on?
I'm just really excited to be on this podcast.
And I think that this podcast is excellent.
And I'm looking forward to listening to the next episodes while I'm on the subway.
Excellent.
Thank you, Jesse.
It's been a really great conversation.
And I think people are going to be super interested in this stuff.
It's good.
MongoDB is such a fun project to work with.
When I was working with relational databases, I always felt like, oh, there's this sort of database and it's a necessary evil.
And it's kind of resisting what I'm trying to build in my app.
And you switched to Mongo and it's just sort of frictionless design and evolution in your app.
And so thanks for all your work on making that a possibility.
Thanks so much, Michael.
You bet.
Talk to you later.
Bye.
This has been another episode of Talk Python to Me.
I want to thank our sponsor, Python Gear, for making this show possible.
Please visit pythongear.com, get an awesome T-shirt or sticker, and let them know you heard about them on Talk Python to Me.
Smix, take us out of here.
Bye.
Thank you.