WEBVTT

00:00:00.001 --> 00:00:02.860
Database design and decisions used to be fairly straightforward.

00:00:02.860 --> 00:00:06.000
Pick your relational database engine, map out the general entities,

00:00:06.000 --> 00:00:09.000
apply the third normal form to them, and you're basically done.

00:00:09.000 --> 00:00:15.440
With the Cambrian explosion of database options and variations created from about 2009 until present,

00:00:15.440 --> 00:00:17.300
it's way harder to even choose the database,

00:00:17.300 --> 00:00:20.980
much less follow the well-worn path of third normal form for modeling.

00:00:20.980 --> 00:00:24.800
On this episode, you'll meet Rick Copeland, a fellow MongoDB master

00:00:24.800 --> 00:00:28.460
and author of the book, MongoDB Applied Design Patterns.

00:00:28.840 --> 00:00:33.680
We'll discuss modeling data using documents in a document database such as MongoDB

00:00:33.680 --> 00:00:38.860
and some techniques and situations that apply particularly to MongoDB's implementation.

00:00:38.860 --> 00:00:45.140
This is Talk Python To Me, episode 109, recorded April 26, 2017.

00:00:57.680 --> 00:01:15.160
Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the ecosystem,

00:01:15.160 --> 00:01:16.260
and the personalities.

00:01:16.260 --> 00:01:18.380
This is your host, Michael Kennedy.

00:01:18.380 --> 00:01:20.380
Follow me on Twitter where I'm @mkennedy.

00:01:20.700 --> 00:01:24.260
Keep up with the show and listen to past episodes at talkpython.fm

00:01:24.260 --> 00:01:26.800
and follow the show on Twitter via at Talk Python.

00:01:26.800 --> 00:01:31.340
This episode is brought to you by Advanced Digital and Hired.

00:01:31.340 --> 00:01:33.860
Please check out what they're offering during their segments.

00:01:33.860 --> 00:01:35.060
It helps support the show.

00:01:35.060 --> 00:01:37.380
Rick, welcome to Talk Python.

00:01:37.380 --> 00:01:38.680
Thanks for having me on, Michael.

00:01:38.680 --> 00:01:40.440
Oh, I'm very excited to have you on.

00:01:40.440 --> 00:01:43.660
Anytime I get to talk about MongoDB, it really makes me happy.

00:01:43.780 --> 00:01:45.460
So we're going to have a lot of fun doing that.

00:01:45.460 --> 00:01:50.420
And I think the work that you've done in MongoDB in a couple areas, in your ODM, as well as

00:01:50.420 --> 00:01:52.440
your book that we're going to talk about, it's super work.

00:01:52.440 --> 00:01:55.360
So I'm looking forward to sharing it with everyone and talking to you about it.

00:01:55.360 --> 00:01:55.580
Awesome.

00:01:55.580 --> 00:01:56.020
Yeah.

00:01:56.020 --> 00:01:59.960
Well, of course, before we get into that, we've got to hear your story.

00:01:59.960 --> 00:02:01.500
How did you get into programming in Python?

00:02:01.500 --> 00:02:06.620
Well, so when I was, I don't know, when I was a kid, my dad got an Apple II, and I guess

00:02:06.620 --> 00:02:07.780
that dates me a little bit.

00:02:07.780 --> 00:02:11.800
But I learned how to program on BASIC starting there and then ended up just getting into

00:02:11.800 --> 00:02:13.340
computer science in college.

00:02:13.340 --> 00:02:16.520
So I was pretty hardcore in C, C++.

00:02:16.520 --> 00:02:23.240
Out of college, I did some systems programming and various other types of things with visual,

00:02:23.240 --> 00:02:28.980
C++, and ended up having a little period when I was exploring new programming languages and

00:02:28.980 --> 00:02:33.180
I ran across an essay by Eric Raymond about why Python.

00:02:33.180 --> 00:02:35.900
And it was kind of like his new favorite programming language.

00:02:35.900 --> 00:02:40.260
And I'd seen a little bit of Python before, like I'd run across Gen 2 and I saw it's got

00:02:40.260 --> 00:02:42.920
this crazy indentation syntax and I kind of dismissed it.

00:02:42.920 --> 00:02:44.760
But then when someone credible...

00:02:44.760 --> 00:02:45.500
This is a weird language.

00:02:45.500 --> 00:02:46.520
It uses white space.

00:02:46.520 --> 00:02:47.340
Let's keep going.

00:02:47.340 --> 00:02:47.760
Yeah.

00:02:47.760 --> 00:02:48.120
Yeah.

00:02:48.120 --> 00:02:52.140
So when someone credible said, you know, this is really cool, I figured that I would get

00:02:52.140 --> 00:02:52.600
into it.

00:02:52.600 --> 00:02:58.420
And so I kind of taught myself Python, started to introduce it at that time into a enterprise

00:02:58.420 --> 00:03:00.520
programming environment, which was interesting.

00:03:00.520 --> 00:03:04.480
They were using mainly C# and Visual Basic.

00:03:04.800 --> 00:03:07.980
It was kind of one of those things where whenever there was a problem and they needed something

00:03:07.980 --> 00:03:10.680
solved really quick, I would say, oh, I'll use Python for that.

00:03:10.680 --> 00:03:12.340
And they said, don't tell me what you're doing.

00:03:12.340 --> 00:03:13.420
Just fix the problem.

00:03:13.420 --> 00:03:15.360
So kind of flying it under the radar.

00:03:15.360 --> 00:03:17.800
And then, you know, discovered that I really loved it.

00:03:17.800 --> 00:03:22.180
So the next job, you know, I sought out Python programming positions after that.

00:03:22.180 --> 00:03:23.440
I think I was in...

00:03:23.440 --> 00:03:26.060
Python was around version 2.3 then.

00:03:26.300 --> 00:03:30.580
So obviously it's grown a lot of new bells and whistles since then.

00:03:30.580 --> 00:03:33.440
And just it's a lot more fun to program in even than it was then.

00:03:33.440 --> 00:03:33.700
Yeah.

00:03:33.700 --> 00:03:39.560
It's just both in terms of the language and whatnot, but also the ecosystem and all the

00:03:39.560 --> 00:03:40.080
packages.

00:03:40.080 --> 00:03:40.660
Right.

00:03:40.660 --> 00:03:41.560
It just keeps getting cooler.

00:03:41.560 --> 00:03:44.100
I think it's a fun place to be for sure.

00:03:44.220 --> 00:03:44.340
Yeah.

00:03:44.340 --> 00:03:51.860
Well, even back in, you know, 2002 or 2003 when I started doing Python, it was so far...

00:03:51.860 --> 00:03:55.220
Like what you got when you just downloaded the standard library was so much more than you

00:03:55.220 --> 00:03:58.040
got in any of the other languages that I was familiar with.

00:03:58.040 --> 00:04:03.280
So, you know, if you wanted to do anything in C++, you had to go out and find something

00:04:03.280 --> 00:04:06.140
that you would configure, make and make install.

00:04:06.140 --> 00:04:09.020
And then, you know, you'd be able to get those development libraries.

00:04:09.020 --> 00:04:14.160
But with Python, you know, you can download URLs, you can create an FTP server or, you know,

00:04:14.160 --> 00:04:15.060
things like that.

00:04:15.060 --> 00:04:16.160
And it's just built in.

00:04:16.160 --> 00:04:20.060
So that was a nice aspect of the batteries included at that time.

00:04:20.060 --> 00:04:20.680
Yeah, absolutely.

00:04:20.680 --> 00:04:23.100
So what are you doing with Python these days?

00:04:23.100 --> 00:04:24.260
What's your day job?

00:04:24.480 --> 00:04:28.600
So these days I'm a consultant, which basically means I do kind of a little bit of everything.

00:04:28.600 --> 00:04:34.080
So sometimes I'm out training different companies in how to do Python.

00:04:34.080 --> 00:04:40.320
So I've been to D.C. and California in the last two weeks.

00:04:40.320 --> 00:04:43.200
I sometimes do custom development for folks.

00:04:43.200 --> 00:04:46.220
I've got, I think, three active projects going on for that.

00:04:46.220 --> 00:04:48.620
And I'm working on a startup on the side.

00:04:48.620 --> 00:04:51.840
So, you know, everybody's got to have their little side hustle that they want to eventually

00:04:51.840 --> 00:04:52.800
make into something.

00:04:52.920 --> 00:04:55.860
So I guess I've got more of a three side hustles going at this point.

00:04:55.860 --> 00:04:57.060
Yeah, that sounds fun.

00:04:57.060 --> 00:04:59.940
It's a challenge to do all these different things.

00:04:59.940 --> 00:05:00.800
I know.

00:05:00.800 --> 00:05:05.280
But it's also fun to have a wide variety and not just be doing the one thing, right?

00:05:05.280 --> 00:05:05.760
Yes.

00:05:05.760 --> 00:05:06.560
Yeah, it is.

00:05:06.560 --> 00:05:08.560
It's never boring.

00:05:08.560 --> 00:05:10.300
Sometimes it can be a little bit overwhelming.

00:05:10.300 --> 00:05:12.200
Like, what am I supposed to work on today?

00:05:12.200 --> 00:05:14.420
And the context, which can get a little bit much.

00:05:14.420 --> 00:05:16.340
But, you know, it's all good.

00:05:16.340 --> 00:05:18.100
Yeah, it's better than the alternative, I think.

00:05:18.100 --> 00:05:18.700
Although you're right.

00:05:18.700 --> 00:05:19.720
It's definitely overwhelming.

00:05:20.240 --> 00:05:23.120
And you're still using MongoDB for some of these projects?

00:05:23.120 --> 00:05:23.780
I am.

00:05:23.780 --> 00:05:28.780
One of the things that I discovered is just it's kind of my go-to at this point.

00:05:28.780 --> 00:05:34.020
I know most people, they learn relational databases because it's much more widespread use.

00:05:34.020 --> 00:05:36.720
But I just kind of got used to using MongoDB.

00:05:36.720 --> 00:05:40.500
In my last business, it was the main database of choice.

00:05:40.760 --> 00:05:44.720
So, you know, I just, I have all the tooling and I have the familiarity with it.

00:05:44.720 --> 00:05:47.980
So that's just the first thing I reach for when I'm implementing something for someone.

00:05:47.980 --> 00:05:49.360
Yeah, I'm with you.

00:05:49.360 --> 00:05:52.880
I feel like it just has so much more flexibility and whatnot.

00:05:52.880 --> 00:06:03.800
I feel like a lot of people fall into using relational databases because that's considered the safe choice or that's what they already know or what they were taught in college.

00:06:04.140 --> 00:06:08.460
Or like the people they're working with, they already know that, but it's not necessarily because it's the best choice.

00:06:08.460 --> 00:06:08.920
Right.

00:06:08.920 --> 00:06:09.780
Yes.

00:06:09.780 --> 00:06:11.280
Yeah, absolutely.

00:06:11.780 --> 00:06:20.740
So I thought maybe, you know, we're mostly going to focus on sort of advanced MongoDB design patterns and implementation concepts.

00:06:20.740 --> 00:06:26.500
But not everybody listening to this is totally familiar with Mongo or NoSQL or document databases.

00:06:26.500 --> 00:06:34.500
So maybe you could give us a quick view just into the summary of what is NoSQL?

00:06:34.500 --> 00:06:36.160
What is a document database?

00:06:36.380 --> 00:06:42.020
And I'm also interested to hear your thoughts on what is NoSQL because everybody seems to have a slightly different definition of that.

00:06:42.020 --> 00:06:42.760
Okay, sure.

00:06:42.760 --> 00:06:47.840
So I would say, you know, NoSQL is kind of anything besides SQL.

00:06:47.840 --> 00:06:54.660
So if you want to kind of drop some of the constraints that SQL puts on you if you're programming.

00:06:54.660 --> 00:07:00.240
So if you look at things like transactions, do you want them to be atomic, consistent, isolated, and durable?

00:07:00.240 --> 00:07:02.820
If you drop some of those, then maybe you're in the NoSQL land.

00:07:03.820 --> 00:07:09.680
So a lot of the NoSQL databases, well, I guess I should say they kind of run the gamut.

00:07:09.680 --> 00:07:13.640
So a NoSQL database could be just a key value store.

00:07:13.640 --> 00:07:15.480
So you're able to look up things very quickly.

00:07:15.480 --> 00:07:25.360
It could be something that's more complicated, has an exotic model like Cassandra with a column, you know, column data store or MongoDB, which is a document data store.

00:07:25.360 --> 00:07:30.800
So, and that's more like the storage model, the programming model, what are you actually putting into this database?

00:07:31.100 --> 00:07:42.880
So I guess the, how I describe MongoDB is, or its document model is, if you think of the relational databases, you've got tables, which are made up of rows and columns.

00:07:42.880 --> 00:07:46.260
In MongoDB, we don't call them tables, we call them collections.

00:07:47.000 --> 00:07:49.760
And what you put in those collections are JSON objects.

00:07:49.760 --> 00:07:53.320
So if you've done web programming, you've probably run into JSON.

00:07:53.320 --> 00:07:58.400
So, and if you haven't, then it's Python dictionaries and lists embedded in each other.

00:07:58.400 --> 00:08:00.640
Basically, that's the data model you're looking at.

00:08:00.820 --> 00:08:06.720
So you just have a collection of these documents and, and we call them those JSON objects documents.

00:08:06.720 --> 00:08:10.600
And you can kind of query into those collections.

00:08:10.600 --> 00:08:16.620
So you can say, give me, say, all of the restaurants that are bakeries in this collection.

00:08:16.620 --> 00:08:22.440
And so you have some field in each JSON document that says, you know, the type of cuisine and it's a, it's a bakery.

00:08:22.780 --> 00:08:32.900
So you could do that sort of a query on a MongoDB database, which makes it a little bit different from a key value store because you don't have to just query based on the key of that document.

00:08:32.900 --> 00:08:40.560
Right. Key value stores are like the most basic, fastest, most scalable, but also the most limiting, right?

00:08:40.560 --> 00:08:46.400
Because you've got the key, the primary key, like an ID, or maybe you could use an email address or whatever if it's a user.

00:08:46.400 --> 00:08:50.520
But then it makes it really hard to ask interesting questions.

00:08:50.700 --> 00:08:53.720
Like the rest of the data is fairly an opaque blob.

00:08:53.720 --> 00:08:58.360
I know there's ways to kind of like add extra stuff around some of the databases, but still.

00:08:58.360 --> 00:08:58.660
Yeah.

00:08:58.660 --> 00:09:05.560
Well, and a lot of the time, those things, you know, you have the key value store and then people will create their manual indexes around that.

00:09:05.560 --> 00:09:11.320
So, you know, you've got kind of the natural key of something, maybe it's some restaurant ID and you want to look up things by cuisine.

00:09:11.320 --> 00:09:18.260
So what you have is maybe a whole bunch of things in another collection that is like cuisine is the key.

00:09:18.540 --> 00:09:20.520
So then those point to the restaurant ID.

00:09:20.520 --> 00:09:22.960
So they build their own indexes out of these things.

00:09:22.960 --> 00:09:26.660
MongoDB takes care of that for you in a lot of cases.

00:09:26.660 --> 00:09:27.540
Right.

00:09:27.540 --> 00:09:33.700
You know, one of the big distinctions, I think, that it takes people who are new to this idea of document databases.

00:09:33.700 --> 00:09:37.240
And MongoDB is not the only one, just the most popular and probably the best.

00:09:37.380 --> 00:09:43.180
But there's, you know, things like Azure DocumentDB, there's CouchDB, there's a variety of them.

00:09:43.180 --> 00:09:43.400
Right.

00:09:43.400 --> 00:09:55.720
And I think it's easy to look at these databases and go, oh, it's kind of like a JSON field embedded in another database or like storing a blob of data or something like that.

00:09:55.720 --> 00:09:56.020
Right.

00:09:56.060 --> 00:10:02.780
You're like, oh, I could do this and say Microsoft SQL server and just make like a text field or a JSON field and stick a blob in there.

00:10:02.780 --> 00:10:06.820
But the big difference is you can index deep into these things.

00:10:06.820 --> 00:10:08.700
You can do rich queries into them.

00:10:08.700 --> 00:10:08.900
Right.

00:10:08.900 --> 00:10:12.460
They're like, they're not just hierarchical things you can store.

00:10:12.460 --> 00:10:18.840
But, you know, even look at some of the object databases, things like ZODB, things like that.

00:10:18.840 --> 00:10:19.020
Right.

00:10:19.020 --> 00:10:19.460
Yes.

00:10:19.460 --> 00:10:25.740
So that is the big difference between this and a key value store, I would say, is that you can index into these things.

00:10:25.880 --> 00:10:36.480
If you're using something like, like you could take MySQL and throw a, you know, a blob column and an ID column into a table and call it a NoSQL database if you want to.

00:10:36.480 --> 00:10:43.640
But you don't really get the ability to do these sorts of rapid lookups on something other than the key.

00:10:43.640 --> 00:10:45.400
And that's what MongoDB gives you.

00:10:45.400 --> 00:10:48.100
That plus it gives you some scaling advantages as well.

00:10:48.100 --> 00:10:59.420
But my background, I haven't taken advantage of that as much as I've just taken advantage of the fact that you can do reasonable things very quickly as opposed to doing unreasonable things reasonably quickly.

00:10:59.420 --> 00:11:00.420
Right.

00:11:00.420 --> 00:11:03.640
I think that's a really interesting comment you made about the performance.

00:11:03.640 --> 00:11:03.940
Right.

00:11:03.940 --> 00:11:11.500
Like one of the things you can do with a lot of these NoSQL databases, Mongo included, is you can do lots of replication.

00:11:11.500 --> 00:11:17.580
You can do specifically, I'm thinking of like sharding, like I could set up a 10 sharded cluster.

00:11:17.860 --> 00:11:24.960
And so then when we do inserts, they're like crazy fast and we can parallelize our queries and our aggregation and MapReduce stuff, all those kinds of things.

00:11:24.960 --> 00:11:25.380
Right.

00:11:25.380 --> 00:11:28.400
But that is like the thing that draws people to it.

00:11:28.400 --> 00:11:30.020
They're like, oh, look what we can do with performance.

00:11:30.020 --> 00:11:35.380
But at the same time, like very few people actually end up needing that much performance.

00:11:35.380 --> 00:11:41.880
I mean, I've seen a few places where it was really needed, but 99% or more of the use cases don't.

00:11:41.880 --> 00:11:47.360
But everybody has a, my relational schema is a pain to deal with.

00:11:47.620 --> 00:11:48.660
It's hard to add columns.

00:11:48.660 --> 00:11:50.320
It's hard to change the shape of it.

00:11:50.320 --> 00:11:51.580
It's like a pain.

00:11:51.580 --> 00:11:52.640
It's slowing me down.

00:11:52.640 --> 00:11:56.640
It's have a, everybody has a complexity problem with their software.

00:11:56.640 --> 00:12:02.460
And I feel like modeling in these documents solves that complexity problem for everyone, not just the 1%.

00:12:02.460 --> 00:12:02.880
Right.

00:12:02.880 --> 00:12:09.060
To me, the sharding and the ability to scale out horizontally has always been kind of a safety feature.

00:12:09.060 --> 00:12:18.820
Like if things go really super well, my system's not going to fall over and I'm not going to have to come back and do manual sharding or partitioning in my database and re-architect my whole application.

00:12:19.020 --> 00:12:22.440
I know that there is a path forward if something like that happens.

00:12:22.440 --> 00:12:28.900
But for now, I can get things done faster than I could with any of the relational approaches.

00:12:28.900 --> 00:12:29.420
Right.

00:12:29.420 --> 00:12:29.720
Yeah.

00:12:29.720 --> 00:12:30.300
I totally agree.

00:12:30.300 --> 00:12:30.840
Totally agree.

00:12:30.840 --> 00:12:37.300
So one of the, there's a bunch of different ways to access MongoDB from Python, right?

00:12:37.300 --> 00:12:42.620
You've got the official Python driver, PyMongo from the MongoDB folks.

00:12:42.940 --> 00:12:47.920
You've got this new thing, BSON, NumPy, NumPy, BSON.

00:12:47.920 --> 00:12:52.960
I don't remember the order of it that goes straight into the data science type structures in NumPy.

00:12:52.960 --> 00:13:03.380
And then on top of PyMongo, we've got things like MongoEngine, Ming, MongoKit, and all these ODMs, right?

00:13:03.380 --> 00:13:03.800
Right.

00:13:03.800 --> 00:13:08.200
And one of these, named Ming, actually, is one that you created.

00:13:08.200 --> 00:13:09.100
Yes.

00:13:09.100 --> 00:13:12.660
Maybe give us a quick overview of what are the trade-offs?

00:13:12.660 --> 00:13:15.120
When would you consider using one of these ODMs?

00:13:15.120 --> 00:13:16.620
Like, what the heck is an ODM anyway?

00:13:16.620 --> 00:13:19.760
So it really comes down to the idea.

00:13:19.760 --> 00:13:21.640
So I said we don't have tables.

00:13:21.640 --> 00:13:29.680
And one of the other things that you don't really have when you're dealing with MongoDB is you don't have a database-enforced schema.

00:13:29.680 --> 00:13:34.040
So you have, it's kind of like this big bag of things.

00:13:34.040 --> 00:13:37.120
And I said that, you know, they're kind of like Python dictionaries.

00:13:37.120 --> 00:13:40.020
So I'll have to say this very quickly or very carefully.

00:13:40.020 --> 00:13:45.380
But you don't want to end up with a big bag of dicts when you're working on this.

00:13:45.380 --> 00:13:51.640
So what you need to actually have is some sort of a schema that tells you the sorts of things that you're going to put in these collections.

00:13:51.640 --> 00:13:57.820
Because it turns out whatever you're putting into them, when you read them back out, your code's going to have to do something with that data.

00:13:58.240 --> 00:14:03.340
You can't just say, well, I'm just going to store everything in there and it's magically going to reappear.

00:14:03.340 --> 00:14:11.000
Your code is making certain assumptions about what fields are in those dictionaries, what keys, and, like, what is the structure of the data that you're storing?

00:14:11.720 --> 00:14:16.240
So that's really what these ODMs or object document managers do.

00:14:16.240 --> 00:14:20.480
That tells you, you know, in this collection, we're putting things in that look like this restaurant.

00:14:20.480 --> 00:14:28.440
So although MongoDB, until very recently, didn't have any form of enforcing schemas, this would be something in your code where you're documenting it.

00:14:28.440 --> 00:14:36.000
At the very least, you're documenting what sorts of dictionaries or BSON documents you want to be putting into these collections.

00:14:36.000 --> 00:14:48.940
Right, and so these, if you go through one of these ODMs, object data mappers, layers, you basically go through predefined classes and objects in Python, which themselves have a fixed structure.

00:14:48.940 --> 00:14:55.440
And so you're kind of, you filter through, like, a known layer of schema, and that works pretty well, right?

00:14:55.440 --> 00:14:56.200
Yeah, yeah.

00:14:56.200 --> 00:15:04.040
And that allows you to kind of, you can get a long way without having any kind of a documented schema if you're the only programmer on the project.

00:15:04.040 --> 00:15:14.400
But once you start having multiple people, you need to have kind of a common understanding of, well, I'm going to write things that look like this to the collection, and I'm going to read things and expect them to look like this other thing.

00:15:14.400 --> 00:15:17.380
So that's kind of the base level of why you need something like this.

00:15:17.380 --> 00:15:19.320
You need a library or a data access layer.

00:15:19.320 --> 00:15:20.900
Sometimes people will write their own.

00:15:20.900 --> 00:15:22.360
That's a pretty common thing.

00:15:22.360 --> 00:15:30.740
If you don't use an ODM, then people will typically write Python modules that have, you know, getters and setters for different types of data that they want to put into the database.

00:15:31.260 --> 00:15:34.540
That's the approach that MongoDB uses on their training materials, I think.

00:15:34.540 --> 00:15:38.740
They just build this Python module that does these things.

00:15:38.740 --> 00:15:45.480
So an object document mapper allows you to kind of abstract that out and write those more quickly.

00:15:45.700 --> 00:15:56.280
So rather than saying, you know, I want to get a restaurant or I want to write a function that calls, you know, this is called get restaurant, then I can have a restaurant class that has a get method.

00:15:56.280 --> 00:15:58.140
But the get method is not something I have to write.

00:15:58.140 --> 00:16:00.060
It's something that the ODM provides me.

00:16:00.060 --> 00:16:01.720
I hope that makes sense.

00:16:01.940 --> 00:16:02.820
Yeah, I think it does.

00:16:02.820 --> 00:16:07.800
And a lot of these, I can't speak to me, you'll have to fill us in on the details.

00:16:07.800 --> 00:16:08.800
I don't remember exactly.

00:16:08.800 --> 00:16:13.240
But the one that I'm using right now is called MongoEngine, which is also one of the more popular ones.

00:16:13.240 --> 00:16:16.760
And it has a lot of additional things that it helps you with.

00:16:16.760 --> 00:16:19.580
Like you define a class and it's much like SQLAlchemy.

00:16:19.580 --> 00:16:22.860
You say these are the fields that go into the database.

00:16:22.860 --> 00:16:25.540
And this one's a string and it has to be unique.

00:16:25.620 --> 00:16:29.180
Like this one's an integer and I want an index on it and things like that.

00:16:29.180 --> 00:16:31.460
And so it'll actually apply the uniqueness constraints.

00:16:31.460 --> 00:16:33.440
It'll apply the index.

00:16:33.440 --> 00:16:37.380
It'll create and enforce the indexes, all those sorts of things as well.

00:16:37.380 --> 00:16:38.820
That's part of what Ming does.

00:16:38.820 --> 00:16:39.240
Definitely.

00:16:39.240 --> 00:16:46.380
You can put these constraints in there, these indexes that you want it to define, and it'll go ahead and create those indexes for you.

00:16:46.380 --> 00:16:53.380
Another thing that it's helpful or that it helps you with is your schema evolves as you're building your application.

00:16:53.600 --> 00:16:59.320
So maybe you didn't need a zip code when you first started or you forgot that you were going to need a zip code.

00:16:59.320 --> 00:17:01.640
And maybe that's a bad example.

00:17:01.640 --> 00:17:07.100
But there's some fields that you want to add later on and you need a sensible default for the existing documents.

00:17:07.100 --> 00:17:13.780
Like let's say you have an account class and you eventually want to start verifying that they've verified their email address.

00:17:13.780 --> 00:17:15.400
And you didn't think of that at first.

00:17:15.400 --> 00:17:17.820
You don't have a is email verified or something like that, right?

00:17:17.820 --> 00:17:18.160
Yeah.

00:17:18.320 --> 00:17:22.340
So maybe you want all of the existing documents to have that be default false.

00:17:22.340 --> 00:17:23.660
They haven't verified their email.

00:17:24.300 --> 00:17:36.420
So you can write a validator or a type into your schema that says, you know, when this field is not found, then I want you to populate the Python object with false.

00:17:36.420 --> 00:17:41.040
It helps you to do these sorts of on-the-fly data migrations.

00:17:41.560 --> 00:17:41.680
Right.

00:17:41.680 --> 00:17:49.200
And it can be kind of like you describe it in your book as a lazy, lazy migrations or lazy schema migrations.

00:17:49.200 --> 00:17:53.080
Because in a relational database, that wouldn't fly very well, right?

00:17:53.080 --> 00:17:59.320
You'd have to say, well, we're going to have we're going to do a schema transformation and we're going to add a column and it's going to be a type bool and it's going to be default.

00:17:59.320 --> 00:18:01.520
There's some kind of script you got to run probably.

00:18:01.520 --> 00:18:01.940
Yeah.

00:18:01.940 --> 00:18:04.300
And those I guess that's the deal.

00:18:04.300 --> 00:18:10.160
You when you're going to be changing the schema in a in a SQL database, you do it up front.

00:18:10.160 --> 00:18:13.020
So you've got to make sure that all of the rows conform to the schema.

00:18:13.020 --> 00:18:15.480
And, you know, the database is going to enforce that.

00:18:15.480 --> 00:18:20.460
So you do this alter table statement and it makes sure that everything conforms to it.

00:18:20.460 --> 00:18:21.760
So that's one approach.

00:18:21.760 --> 00:18:23.420
And you can do that in MongoDB as well.

00:18:23.420 --> 00:18:27.220
You just go in and you and you overwrite all of the existing documents.

00:18:27.220 --> 00:18:27.520
Right.

00:18:27.520 --> 00:18:29.940
It's maybe as in a SQL script.

00:18:29.940 --> 00:18:32.800
It's just a JavaScript script or something like that.

00:18:32.800 --> 00:18:33.000
Right.

00:18:33.000 --> 00:18:33.580
And you run that.

00:18:33.580 --> 00:18:33.840
Yeah.

00:18:33.840 --> 00:18:41.020
Mongo gives you the option of kind of waiting until you actually load a particular document to make sure it conforms to your current schema.

00:18:41.460 --> 00:18:49.380
So that's something that we built into Ming as well so that you could actually like read a document and then check to see does that document actually conform to my current schema.

00:18:49.380 --> 00:18:54.440
And if it doesn't, there was actually the ability to fall back and run a migration function on that document.

00:18:54.440 --> 00:18:58.360
So you could actually bring things forward at the moment when they're loaded out of the database.

00:18:58.360 --> 00:18:58.920
Oh, yeah.

00:18:58.920 --> 00:18:59.600
That's really cool.

00:19:00.220 --> 00:19:07.360
So one of the early success stories from MongoDB, I think, comes from SourceForge, actually.

00:19:07.360 --> 00:19:08.940
And you were part of this.

00:19:08.940 --> 00:19:10.380
I remember SourceForge.

00:19:10.380 --> 00:19:13.020
This is before there's a GitHub or anything like this, right?

00:19:13.020 --> 00:19:14.680
And SourceForge was used quite frequently.

00:19:14.680 --> 00:19:18.040
And I remember it was getting painfully slow.

00:19:18.040 --> 00:19:20.500
And then one day it was fast again.

00:19:20.500 --> 00:19:23.740
And you were involved in that somewhat.

00:19:23.740 --> 00:19:26.440
And that's actually partly where Ming came from, right?

00:19:26.440 --> 00:19:27.620
Do you want to tell us about that?

00:19:27.760 --> 00:19:28.000
Yeah.

00:19:28.000 --> 00:19:41.500
So when I came to SourceForge, I remember in the interview when I was about to come to SourceForge, I had worked on building a SQLAlchemy-like library for, I guess you could call it a NoSQL database.

00:19:41.500 --> 00:19:44.560
It was a private thing that the company that I was working for had developed internally.

00:19:44.560 --> 00:19:47.240
And so they asked me about that.

00:19:47.240 --> 00:19:51.700
And they said, well, you know, we've got this thing called MongoDB that we're thinking about working on.

00:19:51.700 --> 00:19:53.520
And it kind of stores Python dictionaries.

00:19:53.520 --> 00:19:56.240
So what would your approach be to doing something like that?

00:19:56.840 --> 00:19:58.500
And so, you know, I kind of talked about it.

00:19:58.500 --> 00:20:00.020
And I guess my answer was good enough.

00:20:00.020 --> 00:20:00.720
And they hired me.

00:20:00.720 --> 00:20:03.180
And they said, you know, we did some performance evaluations.

00:20:03.180 --> 00:20:05.020
And at that time, it was like 2009.

00:20:05.020 --> 00:20:07.380
They looked at various different approaches.

00:20:07.380 --> 00:20:09.820
And they said, MongoDB is going to give us the performance that we need.

00:20:09.820 --> 00:20:12.520
And we're comfortable with the data model.

00:20:12.520 --> 00:20:16.720
So we like the idea of storing things that look like Python dictionaries into the database.

00:20:16.720 --> 00:20:22.640
But we would like to have something like some kind of a schema enforcement layer or an ODM.

00:20:22.640 --> 00:20:25.620
Although I don't know that that was really a big term at that point.

00:20:25.820 --> 00:20:27.320
They maybe called it an ORM even.

00:20:27.320 --> 00:20:28.100
Yeah.

00:20:28.100 --> 00:20:31.040
I call it an ORM for a non-relational database.

00:20:31.040 --> 00:20:31.500
Yeah.

00:20:31.500 --> 00:20:32.580
ORM minus the R.

00:20:32.580 --> 00:20:32.920
Yes.

00:20:32.920 --> 00:20:36.000
So, you know, we started working on that.

00:20:36.000 --> 00:20:40.060
And I was the main developer on Ming.

00:20:40.480 --> 00:20:47.640
And so Ming formed kind of the data layer of a complete rewrite of all of the SourceForge developer tools.

00:20:47.640 --> 00:20:52.560
So when you think about SourceForge, there's kind of two sections of it.

00:20:52.560 --> 00:20:54.200
I mean, if you think about SourceForge these days.

00:20:54.200 --> 00:20:58.440
But there's sort of the, this is the site for the developers to build their software.

00:20:58.860 --> 00:21:02.360
And this is the site for users to download software.

00:21:02.360 --> 00:21:06.560
This portion of Talk Python is brought to you by Advanced Digital.

00:21:06.560 --> 00:21:10.400
How would you like to build one of the most visited news sites in the U.S.?

00:21:10.400 --> 00:21:11.200
That sounds fun.

00:21:11.200 --> 00:21:13.620
The folks at Advanced Digital would love to talk to you.

00:21:13.620 --> 00:21:16.960
They're primarily a Python shop located in beautiful Jersey City.

00:21:16.960 --> 00:21:19.300
Just one subway stop from lower Manhattan.

00:21:19.980 --> 00:21:22.500
Spend your time building an amazing web app with Python.

00:21:22.500 --> 00:21:25.740
And do it with a small team of developers focused on agile development.

00:21:25.740 --> 00:21:29.740
Are you going to miss PyCon this year because your company wouldn't fund the travel and expense?

00:21:29.740 --> 00:21:33.100
If you join this team, they'll cover your conference and training initiatives.

00:21:33.100 --> 00:21:35.460
It's time to take your Python to the next level.

00:21:35.460 --> 00:21:37.120
Build an amazing web app.

00:21:37.120 --> 00:21:41.060
Get started by visiting python.advance.net right now.

00:21:41.060 --> 00:21:44.480
So we rewrote kind of all of the developer tools.

00:21:44.480 --> 00:21:48.580
And we rewrote a lot of the download side of things as well.

00:21:49.000 --> 00:21:52.740
And that was actually a migration from PHP to Python.

00:21:52.740 --> 00:21:58.100
And a migration from largely Postgres backed to mostly MongoDB.

00:21:58.100 --> 00:21:59.700
And we kind of did it in stages.

00:21:59.700 --> 00:22:01.580
But Ming was a big part of that.

00:22:01.580 --> 00:22:05.160
Being able to kind of come in and say, we've got a group of programmers working.

00:22:05.160 --> 00:22:09.940
What's our common understanding of the data that we're storing in this weird database that none of us has seen before?

00:22:09.940 --> 00:22:10.600
Right.

00:22:10.600 --> 00:22:12.560
And how did it go?

00:22:12.560 --> 00:22:18.740
I recall that there were some pretty major stats in how much better the site got.

00:22:18.960 --> 00:22:22.880
How much fewer, how many fewer database servers there were?

00:22:22.880 --> 00:22:23.500
Things like that.

00:22:23.500 --> 00:22:23.880
Do you recall?

00:22:23.880 --> 00:22:30.360
Well, I remember we went from handling, it was something like 13 servers that were running the PHP front end.

00:22:30.560 --> 00:22:36.900
Our first deployment, we went down to, I believe, four Python servers doing basically the same work.

00:22:37.420 --> 00:22:40.340
So that was a nice, nice thing for Python.

00:22:40.340 --> 00:22:44.560
And of course, the PHP was backed by Postgres and the Python was backed by Mongo.

00:22:44.560 --> 00:22:54.740
And one of the other things in the first version, this is what a lot of people did with Mongo at the time, and I guess still probably do, is when you're introducing this new technology, you kind of take baby steps.

00:22:54.860 --> 00:22:57.780
So Mongo was not our system of record initially.

00:22:57.780 --> 00:23:02.900
We would use it as kind of a cache for all of the Postgres data that was coming from the legacy system.

00:23:02.900 --> 00:23:05.080
So all of that went into Mongo.

00:23:05.740 --> 00:23:17.660
And then as long as you obey a few little rules like make sure your working set fits into RAM, Mongo behaved, its performance was closer to memcached than it was to a relational database.

00:23:17.660 --> 00:23:22.120
So, you know, super fast for a read mostly workload.

00:23:22.880 --> 00:23:25.460
And that's why we were able to do nice things.

00:23:25.460 --> 00:23:30.460
And then we, like a lot of people who first deploy MongoDB, we think, oh, this is great.

00:23:30.460 --> 00:23:33.200
It can probably do anything I want it to do.

00:23:33.200 --> 00:23:36.540
So we wrote a little rate limiter in MongoDB.

00:23:36.540 --> 00:23:41.580
And we did it in a really stupid way, it turns out, by basically just logging every request.

00:23:41.580 --> 00:23:49.660
And then every time a request comes in, we would query to see how many requests from that IP in the last X seconds or minutes or whatever a rate limit was.

00:23:50.580 --> 00:23:55.240
And that worked until it didn't, which was when the index got bigger than our RAM.

00:23:55.240 --> 00:23:59.080
And you got this nice cliff of performance.

00:23:59.080 --> 00:24:01.200
So we reworked that.

00:24:01.200 --> 00:24:04.120
But, you know, for the most part, it was a pretty good rollout.

00:24:04.120 --> 00:24:10.200
And, you know, a lot of success moving from PHP to Python.

00:24:10.200 --> 00:24:13.300
And there's still things that run on.

00:24:13.300 --> 00:24:17.660
Last I heard, there were still things that ran on Postgres at SourceForge.

00:24:17.660 --> 00:24:20.380
But it was primarily MongoDB later on.

00:24:20.660 --> 00:24:21.160
Yeah, okay.

00:24:21.160 --> 00:24:22.060
That makes a lot of sense.

00:24:22.060 --> 00:24:22.880
That's really cool.

00:24:22.880 --> 00:24:25.240
Is it still running in Mongo, do you think?

00:24:25.240 --> 00:24:25.520
Do you know?

00:24:25.520 --> 00:24:26.240
Well, it is.

00:24:26.240 --> 00:24:31.040
So the first version that we rolled out was only for the download side.

00:24:31.040 --> 00:24:34.440
And then we ended up rewriting all of the developer tools in Python and MongoDB.

00:24:34.720 --> 00:24:42.000
And then that ended up being outsourced, not outsourced, open sourced as the Apache Allura project.

00:24:42.000 --> 00:24:45.040
So it's now an official Apache Software Foundation project.

00:24:45.040 --> 00:24:50.600
And anybody can run the same tools that SourceForge is running for developing software.

00:24:51.180 --> 00:24:53.080
And there's a little bit of setup involved.

00:24:53.080 --> 00:24:55.240
But it's still out there.

00:24:55.240 --> 00:25:01.160
It's something that was kind of a goal early on that we wanted to make sure that we gave back to the community with what we were doing.

00:25:01.160 --> 00:25:03.480
And, of course, Ming was always open source from the beginning.

00:25:03.480 --> 00:25:10.560
SourceForge has had its moments of evil, but generally has been a good supporter of open source software.

00:25:12.020 --> 00:25:16.640
Yeah, I say historically, it's probably got a positive grade, all in all.

00:25:16.640 --> 00:25:16.940
Yeah.

00:25:16.940 --> 00:25:17.280
All right.

00:25:17.280 --> 00:25:24.200
So one of the things I really want to dig into while we're talking is your book called MongoDB Applied Design Patterns.

00:25:24.200 --> 00:25:31.980
But before we get to that, I just want to quickly write an idea by you and maybe make a plea to anyone who is either running or considering running Mongo.

00:25:31.980 --> 00:25:34.640
I think I'd love to hear your opinion.

00:25:34.640 --> 00:25:47.020
One of the things, I think Mongo is super great, but I think they've made a few fairly minor decisions that have come back to haunt them in certain ways that get amplified from the early days.

00:25:47.020 --> 00:25:52.700
And I think one of those is, by default, not running encrypted connections.

00:25:52.700 --> 00:25:55.460
And another is, by default, not running with authentication.

00:25:55.460 --> 00:25:56.200
Yes.

00:25:56.200 --> 00:25:58.880
So their defaults have always been interesting.

00:25:58.880 --> 00:26:00.440
Maybe I'll use that word.

00:26:00.440 --> 00:26:07.760
I think they've optimized too much for performance and scalability and not enough for durability and safety nets.

00:26:07.760 --> 00:26:10.800
I'm thinking of the initial write concern defaults.

00:26:10.800 --> 00:26:13.460
I'm thinking of the lack of journaling in the early days.

00:26:13.460 --> 00:26:15.000
You know, all these things.

00:26:15.000 --> 00:26:17.820
And each one of them maybe made sense in their original world.

00:26:17.820 --> 00:26:22.640
But I think people have taken these and not knowing they need to be aware of them got themselves in trouble.

00:26:22.640 --> 00:26:23.160
Absolutely.

00:26:23.160 --> 00:26:39.040
So when we started out, the default way that you wrote to MongoDB, if you didn't change any of the settings and you do an update or an insert or whatever, basically you got an acknowledgement from the server that, hey, I received your request to write this data to the database.

00:26:39.040 --> 00:26:43.340
What you didn't have was any assurance that it actually made it onto disk.

00:26:43.340 --> 00:26:44.920
We didn't even get an acknowledgement.

00:26:44.920 --> 00:26:45.340
Yeah.

00:26:45.340 --> 00:26:47.820
Not even into the data set and memory.

00:26:47.820 --> 00:26:50.340
Just the servers received your socket request, basically.

00:26:50.340 --> 00:26:52.880
I think we even didn't get that initially.

00:26:52.880 --> 00:26:53.720
Yeah, I think you might be right.

00:26:53.720 --> 00:26:54.840
Yeah, you could be right about that.

00:26:54.840 --> 00:27:01.440
So everybody learned, first of all, that you needed to have this magic argument when you connected called safe equals true.

00:27:01.440 --> 00:27:08.480
So by default, MongoDB was running an unsafe mode, which is kind of a silly thing to do when you think about it.

00:27:08.480 --> 00:27:08.820
It's cool.

00:27:08.820 --> 00:27:09.360
It's fast.

00:27:09.360 --> 00:27:10.700
Yeah, it was certainly fast.

00:27:10.700 --> 00:27:15.020
And somebody made a nice web video about web scaleness from that.

00:27:15.020 --> 00:27:16.940
Dev null is very fast, too.

00:27:16.940 --> 00:27:17.300
Yeah.

00:27:17.300 --> 00:27:20.360
I can write an infinite amount of data to it super quick.

00:27:21.040 --> 00:27:23.520
But so everybody, you know, we moved over to safe equals true.

00:27:23.520 --> 00:27:30.460
But even then, you just got an acknowledgement that server received your request and maybe it didn't violate any unique key constraints.

00:27:30.460 --> 00:27:31.540
So, okay, great.

00:27:31.540 --> 00:27:33.860
That's some progress, but it might not make it to disk.

00:27:33.860 --> 00:27:37.020
And so they told you, well, you need to really run in replication.

00:27:37.500 --> 00:27:45.000
So then you could get some, you could say, well, I want to only consider my right to be complete once it's been also written to another server.

00:27:45.000 --> 00:27:45.740
Okay, fine.

00:27:45.740 --> 00:27:46.720
Well, that's pretty good.

00:27:46.720 --> 00:27:53.420
That's if you're actually getting verification of replication, then you're probably running in a slightly safer mode than most people are writing to MySQL.

00:27:53.840 --> 00:27:56.300
So I would say that's a good place to be.

00:27:56.300 --> 00:28:07.080
But then they've also got this network issue that by default, you get MongoDB, you fire it up, and it's going to bind to all of the IP addresses on the machine with no authentication and no encryption.

00:28:07.620 --> 00:28:11.180
And anybody can connect to it, read, write any of the data that's on the database.

00:28:11.180 --> 00:28:16.380
So that is not really a good default state to be in.

00:28:16.380 --> 00:28:21.220
And it turns out a lot of people didn't read their docs when they moved to production.

00:28:21.220 --> 00:28:30.900
And there was a big exploit recently where there were thousands of production MongoDB databases that were compromised because they were running completely wide open to the intranet.

00:28:30.900 --> 00:28:32.060
So, yeah, be careful.

00:28:32.060 --> 00:28:32.900
Yeah, absolutely.

00:28:32.900 --> 00:28:35.200
So basically, I bring this up for two reasons.

00:28:35.820 --> 00:28:45.940
One is there's a lot of FUD about Mongo involving things about this write concern and the journaling, and all those are changed, right?

00:28:45.940 --> 00:28:48.260
The defaults are to do the right thing these days.

00:28:48.260 --> 00:28:50.340
So those are basically phased out.

00:28:50.340 --> 00:28:53.620
But this last thing about the security is not.

00:28:53.620 --> 00:28:56.580
If I were king of Mongo, I'm not a king of MongoDB.

00:28:56.580 --> 00:29:04.960
But if I were, I would make it a change that unless you set up authentication, it will only listen on localhost by default.

00:29:05.280 --> 00:29:05.680
Right.

00:29:05.680 --> 00:29:07.260
That would be my rule.

00:29:07.260 --> 00:29:08.240
And that's kind of safe.

00:29:08.240 --> 00:29:12.220
Like, if you're running the server next to your web app or for dev, it's fine.

00:29:12.220 --> 00:29:15.720
And if you want to do something production-wise, you've got to configure it a little better.

00:29:15.720 --> 00:29:17.500
But that's not how it works.

00:29:17.500 --> 00:29:22.640
So just if you guys are listening and you want to run Mongo, definitely, we both definitely recommend it.

00:29:22.640 --> 00:29:25.500
Just make sure you turn on security or you don't listen.

00:29:25.500 --> 00:29:28.000
Just unprotected on the internet, right?

00:29:28.040 --> 00:29:34.760
Just take a few steps to enable encryption if you're going to go across networks and security authentication, things like that.

00:29:34.760 --> 00:29:34.940
Yeah.

00:29:34.940 --> 00:29:42.660
And I think the latest versions of the RPMs and the Debian packages do bind only to the local host.

00:29:43.000 --> 00:29:44.680
So at least they're a little bit more secure.

00:29:44.680 --> 00:29:49.620
But still, if you're just running the MongoDB binary by default, it's going to listen to anything.

00:29:49.620 --> 00:29:50.820
So, yeah, be careful.

00:29:50.820 --> 00:29:51.000
Yep.

00:29:51.000 --> 00:29:51.220
Yep.

00:29:51.280 --> 00:29:54.620
And that also, it's not just a server production thing, right?

00:29:54.620 --> 00:29:57.360
Like, that could be a dev issue.

00:29:57.360 --> 00:30:04.160
Your dev machine could be on the network and you could be running a dev version with live data and it could have the same problem.

00:30:04.160 --> 00:30:05.560
So just be careful about this.

00:30:05.560 --> 00:30:05.820
Yes.

00:30:05.820 --> 00:30:06.380
All right.

00:30:06.380 --> 00:30:10.360
So let's talk about your book, MongoDB Applied Design Patterns.

00:30:10.360 --> 00:30:11.360
That's the title, right?

00:30:11.360 --> 00:30:11.660
Yes.

00:30:11.660 --> 00:30:12.020
Okay.

00:30:12.020 --> 00:30:12.540
I didn't copy.

00:30:12.540 --> 00:30:13.540
It's not a paraphrasing.

00:30:13.540 --> 00:30:13.960
Okay, good.

00:30:13.960 --> 00:30:19.220
So this is a book that looks at MongoDB from a Python developer's perspective.

00:30:19.800 --> 00:30:22.440
And really, I think it's a super book.

00:30:22.440 --> 00:30:28.160
The idea is to look at a bunch of different use cases and challenges and try to solve them, right?

00:30:28.160 --> 00:30:28.600
Right.

00:30:28.600 --> 00:30:35.500
The genesis of the book is MongoDB needed to, or they wanted to have something like a list of different use cases.

00:30:35.500 --> 00:30:38.440
Like, how do you use MongoDB in this situation?

00:30:38.440 --> 00:30:43.660
And so I wrote up a bunch of use cases for them and then they said, you know, this would be a really good book.

00:30:43.660 --> 00:30:51.760
So let's see if we can introduce you to some people at O'Reilly and see if we can kind of flesh these out into a full O'Reilly title.

00:30:51.760 --> 00:30:53.420
And so that's what we ended up doing.

00:30:53.420 --> 00:30:53.700
Yeah.

00:30:53.700 --> 00:30:56.520
And that book came out in 2013, right?

00:30:56.520 --> 00:30:57.040
Sounds right.

00:30:57.040 --> 00:30:57.260
Yeah.

00:30:57.260 --> 00:30:57.620
Yeah.

00:30:57.620 --> 00:30:57.980
Yeah.

00:30:57.980 --> 00:31:00.240
And MongoDB 2.

00:31:01.000 --> 00:31:04.240
Something 2.2, 2.4 sort of time zone.

00:31:04.240 --> 00:31:06.660
How much of it do you think is still current?

00:31:06.660 --> 00:31:12.020
And how much do you think is sort of slightly changed with a release like, say, MongoDB 3?

00:31:12.200 --> 00:31:19.340
There are definitely changes to some of the performance concerns that have to do with the way that the storage engine works since version 3.

00:31:19.340 --> 00:31:23.620
Because they switched to WiredTiger by default and not mem map files, yeah?

00:31:23.620 --> 00:31:23.920
Yeah.

00:31:24.020 --> 00:31:34.260
So the nice thing, nice maybe in quotes here, for programming against MongoDB in the olden days before WiredTiger is it was really easy to understand the memory model.

00:31:34.260 --> 00:31:38.460
Because what they did is they just took your whole database and they mapped it into RAM.

00:31:38.460 --> 00:31:44.780
And they used the Linux virtual memory system to decide what was in and what was out.

00:31:44.780 --> 00:31:49.400
So if you know how to modify memory, then you knew the most efficient way to modify MongoDB.

00:31:49.400 --> 00:31:51.620
With WiredTiger, that's changed.

00:31:51.740 --> 00:31:53.560
They have a real storage engine.

00:31:53.560 --> 00:31:56.940
You know, it has multi-version concurrency control.

00:31:56.940 --> 00:32:05.780
It has some interesting, interesting in a good way, performance characteristics of being able to, you know, have multiple writers going at the same time.

00:32:05.780 --> 00:32:13.860
So I would say some of the things that really optimized for in-place modification in my book don't really apply as much.

00:32:14.320 --> 00:32:29.140
Because there was a huge difference in performance in the old storage engine between writing to something in place on the disk and doing something that, say, changed the size of a document and required MongoDB to write a whole new copy of the document somewhere else on the disk.

00:32:29.140 --> 00:32:29.820
Right.

00:32:29.820 --> 00:32:32.100
And the way it works now, it's totally different.

00:32:32.100 --> 00:32:32.860
So, all right.

00:32:32.860 --> 00:32:37.300
But my look, when I went through it, I felt like this is really still quite current.

00:32:37.300 --> 00:32:41.860
I think you're right about probably the considerations around the memory map files and whatnot.

00:32:41.860 --> 00:32:44.300
But other than that, it looked really good.

00:32:44.300 --> 00:32:49.260
So let me read a really quick excerpt from the book just to kind of set the stage.

00:32:49.260 --> 00:32:59.160
So you say, traditionally, relational databases, while familiar, present significant challenges and complications when trying to scale up to big data needs.

00:32:59.160 --> 00:33:03.100
And into this world steps MongoDB to address the scaling.

00:33:03.100 --> 00:33:18.600
And around all of this height and excitement, a bunch of sites grabbed a NoSQL database, MongoDB database, and threw it out there and just started working with it without really understanding that it takes a different thinking about it.

00:33:18.600 --> 00:33:18.800
Right.

00:33:18.800 --> 00:33:19.600
It's paraphrasing.

00:33:19.600 --> 00:33:19.700
Right.

00:33:19.700 --> 00:33:27.700
But it's basically – and some of these things we just talked about around the durability and security were one of the things.

00:33:27.700 --> 00:33:46.280
But I think more – probably the biggest mind shift that you have to make in this world, and you start – you dedicate a significant part of your book to this right at the beginning, and I think you should, is schema design and document design relative to, say, first normal form and third normal form and all that.

00:33:46.280 --> 00:33:46.940
Yes.

00:33:47.480 --> 00:34:00.060
So I would say that the biggest mindset shift that you've got to get through to be effective at MongoDB schema design is to say how you – what happens when you get rid of joins and what happens when you get rid of transactions.

00:34:00.480 --> 00:34:02.040
So it's kind of the reads and writes.

00:34:02.040 --> 00:34:05.420
So MongoDB does not support the join operator.

00:34:05.420 --> 00:34:07.840
Well, there's a way to do it in the aggregation framework.

00:34:07.840 --> 00:34:19.880
But besides putting that aside, generally, when you do a query in MongoDB, you can get a collection of – or you can get a set of documents in your results set, but you're not going to be talking to two different collections when you do that.

00:34:19.980 --> 00:34:22.340
You're going to be making a query against a single collection.

00:34:22.340 --> 00:34:24.880
And you'll get documents from that single collection.

00:34:24.880 --> 00:34:29.640
And so the question is, how do you actually use that in an efficient way?

00:34:29.740 --> 00:34:37.780
So if I was building a blog in a relational database, then maybe if I need to render that blog post, I would maybe fetch something from the posts collection.

00:34:37.780 --> 00:34:40.360
I would fetch something from the authors collection.

00:34:40.360 --> 00:34:45.340
I would fetch something from – or not collection, but the post table, the authors table, comments table.

00:34:45.340 --> 00:34:54.460
And I'd do a join of all these things, and you'd end up with all of the data that you need to represent that blog post to a web user.

00:34:54.940 --> 00:34:56.900
Well, with MongoDB, you can do the same thing.

00:34:56.900 --> 00:35:03.440
You could have a posts collection and a comments collection and an authors collection, and you can do kind of the join-y work in memory.

00:35:03.440 --> 00:35:12.840
But you've gotten rid of a lot of the benefits of MongoDB because the nice thing about MongoDB is you can design your schema so that a single document can satisfy that web request.

00:35:12.840 --> 00:35:18.880
So you could have the post with the embedded author information with all of the comments all in a single document.

00:35:18.880 --> 00:35:21.820
So basically, you're doing a single fetch, a single round trip to the database.

00:35:22.380 --> 00:35:28.640
And even on the database, if you're using a disk or you're using an SSD, whatever the case is, you've got all the locality right there.

00:35:28.640 --> 00:35:32.580
So the whole document is right where Mongo is looking at that time.

00:35:32.580 --> 00:35:38.920
And so it's able to basically just do things much more efficiently if you design your schema right.

00:35:38.920 --> 00:35:44.380
Yeah, and I think it's very much a Shakespearean type of thing, like to embed or not to embed.

00:35:44.380 --> 00:35:45.380
That is the question, right?

00:35:45.380 --> 00:35:51.220
Like really, every time I sat down to design a new data model for MongoDB, it's like, what are all the pieces?

00:35:51.220 --> 00:35:55.320
What embeds where and what shouldn't be embedded for various reasons, right?

00:35:55.320 --> 00:36:04.980
So like, for example, you mentioned you could have your post and it could have the author embedded and it could have the comments embedded and so on.

00:36:04.980 --> 00:36:07.560
Maybe even there's categories, right?

00:36:07.560 --> 00:36:15.040
Like categories and things that you could theoretically embed the category data into the post, but then you have to replicate that across all the different posts, right?

00:36:15.040 --> 00:36:15.580
Sure.

00:36:15.580 --> 00:36:16.320
Yes.

00:36:16.320 --> 00:36:18.480
That may or may not be something you want.

00:36:18.480 --> 00:36:18.800
Yeah.

00:36:18.800 --> 00:36:21.020
So you still have relationships in your data.

00:36:21.020 --> 00:36:22.900
That's a logical concern, right?

00:36:22.940 --> 00:36:28.420
You can do an entity relationship diagram and you can still map that onto MongoDB.

00:36:28.420 --> 00:36:39.800
The difference is with Mongo, when you have one of these one-to-many relationships, all of a sudden you now have the option, if it makes sense performance-wise, that you could take both of the entities and put them into a single collection.

00:36:39.800 --> 00:36:43.480
But you can't do that in a relational database, right?

00:36:43.480 --> 00:36:47.940
Relational kind of first normal form says you don't have multiple entries in a column.

00:36:48.580 --> 00:36:50.740
But with MongoDB, that's sort of the norm.

00:36:50.740 --> 00:36:54.260
You're allowed to have these array types that are being stored there.

00:36:54.260 --> 00:36:56.900
So now you've got to decide, does it make sense to put it there?

00:36:56.900 --> 00:37:07.240
Or if you've got a many-to-many joint or a many-to-many relationship, the old way of doing it or the SQL way of doing it is you've got to have a join table that's got IDs from table one and IDs from table two.

00:37:07.240 --> 00:37:09.240
And it tells you which ones match up with which ones.

00:37:09.240 --> 00:37:13.000
MongoDB, if you're doing a blog, again, it's just an easy example.

00:37:13.000 --> 00:37:15.100
So you've got tags or categories.

00:37:15.660 --> 00:37:19.200
A lot of the time, that'll just be a list of strings that you put into the post.

00:37:19.200 --> 00:37:25.480
And there's no need to actually have that join collection or that join table that you would have in SQL.

00:37:25.480 --> 00:37:26.780
I think that's totally right.

00:37:26.780 --> 00:37:37.120
And even if your tag thing was more complicated, right, you can do these many, many relationships and maybe store a list of tag IDs in every post.

00:37:37.120 --> 00:37:37.540
Right.

00:37:37.660 --> 00:37:39.460
And then reach back into the other table.

00:37:39.460 --> 00:37:39.760
Yeah.

00:37:39.760 --> 00:37:40.080
Yeah.

00:37:40.080 --> 00:37:44.620
You'd almost never want to have something like a join table in MongoDB.

00:37:44.620 --> 00:37:46.520
I can't think of a good case.

00:37:46.520 --> 00:37:53.240
You'll almost always want to either have a list of IDs in collection A or a list of IDs in collection B or both.

00:37:53.240 --> 00:37:59.340
But you wouldn't want to have a separate collection where the existence of a document means that these other two documents are joined.

00:37:59.600 --> 00:37:59.740
Yeah.

00:37:59.740 --> 00:38:01.540
I find that to be almost never.

00:38:01.540 --> 00:38:05.140
I don't think I've ever seen that in a well-designed case either.

00:38:05.140 --> 00:38:08.260
I definitely have never made use of it in the apps that I built.

00:38:08.260 --> 00:38:12.580
That was one of the problems with people coming from the SQL world is they know how to model things there.

00:38:12.580 --> 00:38:19.380
And they just assume that if I take the same schema that I had in SQL, it's going to be like that but faster if I do it in MongoDB.

00:38:19.380 --> 00:38:20.040
Yeah.

00:38:20.040 --> 00:38:21.480
Because I heard Mongo is faster.

00:38:21.480 --> 00:38:23.900
So it'll be faster if I just put this over here.

00:38:23.900 --> 00:38:24.360
Exactly.

00:38:24.360 --> 00:38:24.940
Yeah.

00:38:24.940 --> 00:38:30.280
It probably is faster, but not because you copied over your schema design from a relational database.

00:38:30.280 --> 00:38:30.540
Yeah.

00:38:30.540 --> 00:38:39.020
Or in many cases, it would end up being slower because you're doing all of the logic of the join at that point, but you're doing it in whatever your programming language is.

00:38:39.020 --> 00:38:44.520
So I love Python, but it's not this super high-performance bare metal language.

00:38:44.520 --> 00:38:48.040
If you're building a join engine in Python, yeah, you can do that.

00:38:48.040 --> 00:38:52.680
But you are now talking about introducing network latency to talk to the database.

00:38:52.680 --> 00:38:54.720
You're talking about it's written in Python.

00:38:54.720 --> 00:39:00.880
It's not written in C++ like the MongoDB engine is or like C database engines might be in other cases.

00:39:00.880 --> 00:39:05.140
So you're kind of, if it's faster, then it's an unusual situation.

00:39:05.140 --> 00:39:07.440
You're usually going to kill yourself performance-wise.

00:39:07.440 --> 00:39:11.880
This portion of Talk Python To Me is brought to you by Hired.

00:39:11.880 --> 00:39:15.300
Hired is the platform for top Python developer jobs.

00:39:15.300 --> 00:39:20.760
Create your profile and instantly get access to thousands of companies who will compete to work with you.

00:39:20.860 --> 00:39:23.980
Take it from one of Hired's users who recently got a job and said,

00:39:23.980 --> 00:39:27.920
I had my first offer within four days and I ended up getting eight offers in total.

00:39:27.920 --> 00:39:31.140
I've worked with recruiters in the past, but they were pretty hit and miss.

00:39:31.140 --> 00:39:33.960
I tried LinkedIn, but I found Hired to be the best.

00:39:33.960 --> 00:39:38.160
I really like knowing the salary up front and privacy was also a huge seller for me.

00:39:38.160 --> 00:39:40.180
Well, that sounds pretty awesome, doesn't it?

00:39:40.180 --> 00:39:42.060
But wait until you hear about the signing bonus.

00:39:42.060 --> 00:39:45.620
Everyone who accepts a job from Hired gets a $300 signing bonus.

00:39:45.620 --> 00:39:48.620
And as Talk Python listeners, it gets even sweeter.

00:39:48.940 --> 00:39:53.980
Use the link talkpython.fm/Hired and Hired will double the signing bonus to $600.

00:39:53.980 --> 00:39:56.060
Opportunity is knocking.

00:39:56.060 --> 00:39:59.760
Visit talkpython.fm/Hired and answer the door.

00:40:00.920 --> 00:40:01.400
Yeah.

00:40:01.400 --> 00:40:09.000
So one of the things while we're on this document design stuff is in MongoDB, there's no concept of a foreign key constraint or relationship.

00:40:09.000 --> 00:40:09.560
Right.

00:40:09.560 --> 00:40:13.400
I can't have one document with a strict relationship to another document.

00:40:13.400 --> 00:40:14.000
Right.

00:40:14.000 --> 00:40:15.980
I'm not really sure how much value you get.

00:40:15.980 --> 00:40:17.660
There's no joins and things like that.

00:40:17.660 --> 00:40:22.660
Like, so oftentimes people think that means there's no relationships in MongoDB.

00:40:22.660 --> 00:40:23.000
Right.

00:40:23.000 --> 00:40:23.320
Yeah.

00:40:23.400 --> 00:40:24.540
But I don't think that that's true.

00:40:24.540 --> 00:40:27.180
I think you can put them into these models.

00:40:27.180 --> 00:40:28.740
They just don't span documents, right?

00:40:28.740 --> 00:40:29.140
Right.

00:40:29.140 --> 00:40:29.420
Yeah.

00:40:29.420 --> 00:40:35.740
You can have, you know, the relationships can exist within a document and you get atomic updates and things like that.

00:40:35.780 --> 00:40:38.680
So you get the database to enforce some consistency there.

00:40:38.680 --> 00:40:50.080
And you can also model the relationships with, I mean, it's not enforced by Mongo, but you can have a foreign key concept where you've got an ID of a different document in another collection and you're storing that ID.

00:40:50.080 --> 00:40:55.520
Differences that you always have to take into account the possibility that that document might not actually exist.

00:40:55.520 --> 00:40:56.040
That's right.

00:40:56.040 --> 00:40:56.280
Yeah.

00:40:56.280 --> 00:40:57.320
I think of them as two things.

00:40:57.320 --> 00:41:00.360
I have a slightly different name that I've used over the years for it.

00:41:00.360 --> 00:41:06.060
Like for the stuff that's within your document, you've got a post and it has a list inside of it of comments.

00:41:06.060 --> 00:41:08.080
Like that is a super strong relationship.

00:41:08.080 --> 00:41:10.180
You can't have a comment without the post.

00:41:10.180 --> 00:41:11.280
It is the same thing.

00:41:11.280 --> 00:41:18.620
But if you were like reaching back to an author table through just a foreign key constraint, that doesn't really exist, but it's logically there.

00:41:18.620 --> 00:41:23.000
I call those soft foreign keys or something like that.

00:41:23.000 --> 00:41:26.560
Like they're not enforced, but they technically, they fill the same role, right?

00:41:26.560 --> 00:41:26.860
Yeah.

00:41:26.860 --> 00:41:27.760
They fill the same role.

00:41:27.860 --> 00:41:30.660
And sometimes people call them references or document references.

00:41:30.660 --> 00:41:39.420
Way back when I started with MongoDB, one of the patterns that they kind of promoted was storing the collection name along with the ID.

00:41:39.420 --> 00:41:42.720
I never found that super valuable, but that's another thing that you can do.

00:41:42.720 --> 00:41:46.440
If you want to have a reference that could go to any collection, then you can just throw the collection name in there.

00:41:46.440 --> 00:41:46.900
Yeah.

00:41:46.900 --> 00:41:47.200
Yeah.

00:41:47.200 --> 00:41:47.920
Interesting.

00:41:47.920 --> 00:41:50.480
That works well at the low level, at like the PyMongo level.

00:41:50.480 --> 00:41:52.420
Less good at the ODM level.

00:41:52.420 --> 00:41:52.900
Right.

00:41:52.900 --> 00:41:55.300
So let's talk about some of the use cases.

00:41:55.420 --> 00:41:59.860
So we've kind of set up this, you talk a lot about like, this is what modeling in this world looks like.

00:41:59.860 --> 00:42:05.240
You also talk about mimicking transactional behavior with compensation models that work well in MongoDB.

00:42:05.240 --> 00:42:07.860
But let's just kind of leave that as there.

00:42:07.860 --> 00:42:10.080
So you kind of set the ground with some of these foundational things.

00:42:10.080 --> 00:42:18.280
And then you say, let's talk about six different use cases, all the performance considerations and how you model it and everything.

00:42:18.280 --> 00:42:18.600
Right.

00:42:18.840 --> 00:42:24.120
So do you want to touch on some of your favorite ones there and maybe like what was non-obvious or maybe something like that?

00:42:24.120 --> 00:42:24.400
Yeah.

00:42:24.400 --> 00:42:30.760
So the first one is, has some of the more interesting parts, I think, or some of the things that I found really interesting.

00:42:30.760 --> 00:42:34.280
And I guess that's why I put it first, but that's the operational intelligence chapter.

00:42:34.560 --> 00:42:40.200
And it's really focusing on analytics and dealing with kind of high volume data that's coming in quickly.

00:42:40.200 --> 00:42:44.500
There were two different use cases in there, or maybe there were three in there.

00:42:45.180 --> 00:42:49.680
But two in particular that I remember were, one of them was incremental aggregation.

00:42:49.680 --> 00:42:56.120
So this is, you've got something coming in, you've got these aggregate statistics that you want to report out immediately.

00:42:56.120 --> 00:43:05.960
So one approach that you could do for aggregation is you can run a big MapReduce job on a Hadoop cluster, and that'll come back in a few minutes.

00:43:05.960 --> 00:43:10.380
But if you actually want something that's up to the minute, then how do you do that in an efficient way?

00:43:10.560 --> 00:43:22.120
And so this relied a lot on the in-place updating, and it was based on MongoDB's own, it's not called Cloud Manager, but their monitoring service, which would actually monitor MongoDB performance for you.

00:43:22.120 --> 00:43:23.940
And they offered this as a free service.

00:43:23.940 --> 00:43:25.440
So it was like, how do we deal with this scale?

00:43:25.440 --> 00:43:33.480
So let me show you how you can build your schema to deal with that kind of scale and how you can keep the performance high, even with an in-map storage engine.

00:43:33.480 --> 00:43:44.760
No, I just, I think it was, what I found interesting about this was you start from, like, let's start with log file data, like something out of Apache web request or something like that.

00:43:44.760 --> 00:43:46.280
Let's put that in the database.

00:43:46.280 --> 00:43:50.780
And then let's start doing, like, processing and analysis of it.

00:43:50.780 --> 00:43:58.600
And you have some really interesting graphs and various things that say, like, let's look at, if we design it this way, what are the trade-offs?

00:43:58.860 --> 00:44:00.640
What is the benefits, what are the drawbacks?

00:44:00.640 --> 00:44:08.280
And there was a number of non-obvious ways in which things kind of slowed down or got out of control.

00:44:08.280 --> 00:44:24.520
And you ended up with quite an interesting aggregation report database, right, where you pre-computed and pre-allocated a whole bunch of pieces and then used some of the in-place update operations to sort of, like, increment the numbers at the right levels as these things came in, right?

00:44:24.520 --> 00:44:26.800
Yeah, that was the incremental aggregation one.

00:44:26.880 --> 00:44:33.420
So that was, the problem there is it was storing the aggregates in these large documents.

00:44:33.420 --> 00:44:36.980
And sometimes the documents would grow and that would cause performance problems.

00:44:36.980 --> 00:44:51.420
And then you get into a secondary issue, which is that even though you think of these things as Python dictionaries, which are super fast to access any item in them, physically they're stored as a list of key value pairs on the disk.

00:44:51.420 --> 00:44:57.020
And so it turns out it takes longer to access something towards the end than it does to take to access something towards the beginning.

00:44:57.020 --> 00:44:58.640
So how can we mitigate that issue?

00:44:58.640 --> 00:45:05.260
And those were just some sort of the sorts of things that you can only see when you've actually run some performance metrics against it.

00:45:05.260 --> 00:45:06.740
Again, just a shout out to Python.

00:45:06.740 --> 00:45:14.220
I did all this with, at that time, IPython notebook and printed out the graphs and, you know, just threw those into the book right there.

00:45:14.220 --> 00:45:17.940
So I think those are actually screenshots from IPython, now Jupyter notebook.

00:45:17.940 --> 00:45:21.400
Yeah, they looked like some Matplotlib graphs or something, which is cool.

00:45:21.400 --> 00:45:21.780
Yeah.

00:45:21.900 --> 00:45:22.140
All right.

00:45:22.140 --> 00:45:29.620
So another thing that people, at least in the early days, were like, oh, you can't use MongoDB for this was e-commerce, which I totally disagree with that statement.

00:45:29.620 --> 00:45:34.700
But you have a section where you talk about using MongoDB for like an e-commerce site, right?

00:45:34.700 --> 00:45:34.980
Yeah.

00:45:34.980 --> 00:45:42.060
So one of the big things or one of the difficulties with existing e-commerce, I guess the big one is Magento.

00:45:42.060 --> 00:45:46.220
So Magento uses an entity attribute value store.

00:45:46.220 --> 00:45:51.140
So they're still stuck on SQL, but they use SQL in a way that makes it non-relational.

00:45:51.660 --> 00:46:02.440
Basically, instead of keeping your products in a products table where each one of the attributes of that product is a column, they just say, I've got one big table that says for this entity, maybe it's a shirt.

00:46:02.440 --> 00:46:05.160
I have an attribute, which is a size, and it's an Excel.

00:46:05.160 --> 00:46:11.540
For this entity, which is a drill, it has, you know, some other attribute, and it's, you know, 120 volts or whatever.

00:46:11.540 --> 00:46:14.480
And so out of that, they're able to get this very flexible schema.

00:46:15.040 --> 00:46:20.360
So it's kind of like, well, that's not really a fantastic way to map to the relational model.

00:46:20.360 --> 00:46:29.360
But they kind of have to because you want to deploy to a store that might have all sorts of different items in it that have different attributes that you want to store.

00:46:29.360 --> 00:46:35.500
Nice thing about MongoDB is not all of your documents have to look like each other inside the collection.

00:46:35.500 --> 00:46:39.340
So Mongo lets you actually say, well, I want to store drills and shirts in this collection.

00:46:39.340 --> 00:46:40.600
Can I do that?

00:46:40.600 --> 00:46:41.900
And it turns out you can.

00:46:41.900 --> 00:46:44.520
Maybe there's certain attributes that they all have in common.

00:46:44.520 --> 00:46:46.400
They have an SKU number.

00:46:46.400 --> 00:46:47.980
They have a price.

00:46:47.980 --> 00:46:50.060
They have maybe a quantity available.

00:46:50.060 --> 00:46:54.020
But then they've all, you know, got their other things that are custom to each one.

00:46:54.020 --> 00:47:03.900
And so you can introduce this polymorphism with MongoDB in a much more natural way, I think, than using something like an entity attribute value schema in a relational database.

00:47:03.900 --> 00:47:06.900
Yeah, I think that's leveraging a pretty interesting aspect.

00:47:06.900 --> 00:47:12.700
And you're in some sense implementing inheritance for specialization.

00:47:12.700 --> 00:47:14.980
Not exactly, but something to that effect, right?

00:47:14.980 --> 00:47:24.300
And that because the schema is really enforced at the application layer, not in the database layer, that flexibility pretty much just flows through.

00:47:24.360 --> 00:47:25.620
And you end up with these sparse objects.

00:47:25.620 --> 00:47:29.280
Like maybe one document has a drill bit size or something.

00:47:29.280 --> 00:47:31.380
The other one has a shirt size, right?

00:47:31.380 --> 00:47:33.260
And those don't appear in both records.

00:47:33.260 --> 00:47:34.280
You don't waste the space.

00:47:34.280 --> 00:47:34.980
Yeah, exactly.

00:47:34.980 --> 00:47:40.160
And you can build your ODM to kind of take care of that.

00:47:40.160 --> 00:47:52.420
I don't think I haven't been doing a lot with Ming super recently, but I'm not sure if we had the ability to kind of discriminate based on the data that it loads out as to which physical type of object it's creating.

00:47:52.420 --> 00:47:55.420
But that's certainly something that you can do with an ODM.

00:47:55.420 --> 00:48:07.140
And I know it's something that SQLAlchemy does with relational databases, but it requires you to either do a super complex schema in SQL or it requires you to waste a lot of columns.

00:48:07.480 --> 00:48:11.320
And those are kind of your two options to do this sort of object oriented polymorphism.

00:48:11.320 --> 00:48:11.920
Nice.

00:48:11.920 --> 00:48:15.160
So what are some of the other ones that you cover that you really like?

00:48:15.160 --> 00:48:26.980
So I did have some fun with the online gaming chapter because that was just, I don't know, games are fun, but kind of like brainstorming out like what are some of the data structures that you might need when you're building this?

00:48:26.980 --> 00:48:31.040
How do you do these in a, say, it's a massively multiplayer online game?

00:48:31.040 --> 00:48:32.200
How would you actually store this?

00:48:32.200 --> 00:48:33.040
How would you scale it?

00:48:33.040 --> 00:48:34.380
How would you do the sharding?

00:48:35.080 --> 00:48:42.060
The online advertising networks was also interesting just because it's a very high frequency sort of application.

00:48:42.060 --> 00:48:45.380
And it's something that I had seen a little bit of at SourceForge.

00:48:45.380 --> 00:48:51.700
And, you know, one of the things that you mentioned earlier on was, you know, SourceForge got slower and slower and slower.

00:48:52.540 --> 00:48:58.580
So part of that we can blame on maybe PHP and Postgres, but part of it we just have to blame on the ad networks.

00:48:58.580 --> 00:49:04.360
Because SourceForge is an advertising supported site, a lot of these ad networks just took a long time to render the ad.

00:49:04.360 --> 00:49:09.540
And that's kind of slowing down your browsing experience and can cause various other problems.

00:49:09.680 --> 00:49:16.740
So what if we could speed those things up and deliver contextual advertising to people in a way that doesn't make them want to pull their hair out?

00:49:16.740 --> 00:49:18.520
So that was also an interesting one.

00:49:18.520 --> 00:49:19.900
Yeah, that's a fun one to work on.

00:49:20.020 --> 00:49:31.160
And I know a couple of people working in this ad network space and they're using Mongo and they have some pretty intense requirements around the traffic that they handle.

00:49:31.160 --> 00:49:37.980
Because if you run ads on a site that gets, you know, a million views a day and that's just one of the places, right?

00:49:37.980 --> 00:49:39.980
You all of a sudden are getting a million requests a day.

00:49:40.080 --> 00:49:47.540
You're getting a million requests a day and you're trying to target those ads now based on some content, you know, that's going on in the article.

00:49:47.540 --> 00:49:54.220
So presumably you've indexed that and you know something about the keywords, but then you probably have some real-time bidding going on for those too.

00:49:54.220 --> 00:49:58.160
So how do you actually choose the ad inside that request-response cycle?

00:49:58.160 --> 00:50:08.120
Because you know that your content people that are actually paying or that, you know, you're advertising on their site, they're not going to like it if you slow down the experience for their viewers.

00:50:08.120 --> 00:50:09.280
No, absolutely not.

00:50:09.280 --> 00:50:11.980
So, yeah, that's definitely a cool example.

00:50:11.980 --> 00:50:17.600
So there was a bunch of great examples and I learned a lot from looking at how you implemented them and the trade-offs.

00:50:17.600 --> 00:50:19.020
And it's a great book.

00:50:19.020 --> 00:50:25.240
I definitely recommend if people are, they know a little bit of Mongo and they're like, I think I should be using this, but I don't really know how to solve this problem.

00:50:25.240 --> 00:50:28.360
There's a lot of good stuff to study there around schema design and whatnot.

00:50:28.360 --> 00:50:28.800
Well, thanks.

00:50:28.800 --> 00:50:29.520
Yeah, you bet.

00:50:29.520 --> 00:50:34.520
So there's a couple of options on where you might run your MongoDB server.

00:50:34.520 --> 00:50:39.240
And I guess it depends on how complicated of a situation you have.

00:50:39.240 --> 00:50:42.880
on how much you want to think about this or need to think about this.

00:50:42.880 --> 00:50:47.740
If you're just running a single server and it's just like there on a machine, maybe you can run that on a VM.

00:50:47.740 --> 00:50:49.700
You still got to deal with backups and whatnot.

00:50:49.940 --> 00:50:52.700
But there's also like hosted Mongo.

00:50:52.700 --> 00:50:56.300
They have MongoDB Atlas.

00:50:56.580 --> 00:51:06.040
What are your thoughts on like if somebody comes to you and says, hey, I want to do the site and run maybe let's say a three node replicated cluster?

00:51:06.040 --> 00:51:07.680
Like what would you consider?

00:51:07.680 --> 00:51:12.980
I would, by default, I would hope that their budget would afford them to get Atlas.

00:51:13.220 --> 00:51:15.920
So Atlas is actually the cloud service by MongoDB.

00:51:15.920 --> 00:51:17.940
They'll host your Mongo for you.

00:51:17.940 --> 00:51:23.060
They'll host the latest copy or the latest version, handle your backups and everything.

00:51:23.060 --> 00:51:27.500
Now, if you're dealing with a large amount of data, the backups can start to get pretty pricey.

00:51:27.500 --> 00:51:29.380
So that might not be an option.

00:51:29.860 --> 00:51:37.820
But unless you have strong operations people on your team, I wouldn't immediately jump to saying, oh, I need to self-host.

00:51:37.820 --> 00:51:38.500
I need to build it.

00:51:38.500 --> 00:51:40.020
I need to run it on my own VMs.

00:51:40.020 --> 00:51:42.440
So there's other options that you can go to.

00:51:42.440 --> 00:51:46.420
You can go to ImLab is one that I've used in the past.

00:51:46.420 --> 00:51:47.900
I've really enjoyed working with them.

00:51:47.900 --> 00:51:52.940
They provide, you know, hosted MongoDB, Compose.io, Object Rocket.

00:51:52.940 --> 00:51:56.900
These are all hosted MongoDB options that you can go with.

00:51:56.900 --> 00:52:03.120
And then if you are going to decide to self-host, there's actually some MongoDB provided tools to do that.

00:52:03.120 --> 00:52:19.160
So if you actually go into the MongoDB Cloud Manager, provide them your EC2 account keys, for instance, and you say, I want to use these three servers or these three virtual machines that I've provisioned to make a three node replica set, then they can do that for you as well.

00:52:19.360 --> 00:52:29.760
So that would probably be, you know, the next step is get your own VM or get your own VMs and then install Cloud Manager and go ahead and have Cloud Manager install that.

00:52:29.760 --> 00:52:30.420
Okay, cool.

00:52:30.420 --> 00:52:32.360
And the Cloud Manager, that's from MongoDB themselves?

00:52:32.360 --> 00:52:33.840
Yeah, that's also from MongoDB.

00:52:33.840 --> 00:52:37.820
So all of these things kind of run in the same UI on MongoDB.

00:52:37.820 --> 00:52:39.380
I guess it's .com.

00:52:39.380 --> 00:52:41.600
I know they have .com and .org both.

00:52:41.600 --> 00:52:41.920
Yep.

00:52:42.160 --> 00:52:43.200
That used to be a big confusion.

00:52:43.200 --> 00:52:45.140
You couldn't find the download link on .com.

00:52:45.140 --> 00:52:45.900
Yeah.

00:52:45.900 --> 00:52:52.280
One thing I'd like to say is I have used MongoLab before, MLab.

00:52:52.280 --> 00:52:53.260
It used to be MongoLab.

00:52:53.260 --> 00:52:53.980
They renamed it.

00:52:54.440 --> 00:52:59.320
And I think they're one of the few options that has a free Mongo server.

00:52:59.320 --> 00:53:08.440
So if you want to just set up a little prototype and get started and play around, they have like a half a gig free server you can set up and use there.

00:53:08.440 --> 00:53:10.300
And so that's pretty sweet.

00:53:10.300 --> 00:53:10.960
They're great.

00:53:10.960 --> 00:53:12.080
I use them.

00:53:12.080 --> 00:53:13.440
I still use them today.

00:53:13.440 --> 00:53:16.360
I use Atlas a little bit, but I use MLab as well.

00:53:16.360 --> 00:53:19.820
One of the nice things about MLab is that there's an integration to Heroku as well.

00:53:20.120 --> 00:53:26.400
So if you're using Heroku, you can get the MLab plan for free, and then it's just kind of like I'm not running a server anywhere.

00:53:26.400 --> 00:53:32.620
Somebody else is doing it for me, and I can play around with things and have them work and with authentication enabled as well.

00:53:32.620 --> 00:53:32.860
Yeah.

00:53:32.860 --> 00:53:35.500
Yeah, those all come set up correctly, let's say.

00:53:35.500 --> 00:53:36.440
Yes.

00:53:36.440 --> 00:53:37.000
Perfect.

00:53:37.000 --> 00:53:38.160
All right.

00:53:38.160 --> 00:53:38.440
Awesome.

00:53:38.440 --> 00:53:45.280
So, yeah, just right now I'm running my own MongoDB server on my own VM, but I've been working with Mongo for six years.

00:53:45.280 --> 00:53:51.200
So I feel like that's probably a point at where I can go run my own VM and do my own backups daily, things like that.

00:53:51.200 --> 00:53:56.860
But, yeah, these are all good options, and I know that jumping on one of the hosted ones is pretty nice to get started.

00:53:56.860 --> 00:54:00.140
So let's talk about some other stuff that you've been up to.

00:54:00.140 --> 00:54:06.460
First of all, like all of this MongoDB work, you now just came out with a MongoDB course for Python developers, right?

00:54:06.460 --> 00:54:12.420
Yeah, so I'm working with Pact Publishing, and they wanted to put out some courses on MongoDB.

00:54:12.420 --> 00:54:25.080
And I just came out with a video course called Developing with MongoDB and kind of a three-hour course that gives you an intro, both of what is MongoDB, how does it work, how is it different from relational databases,

00:54:25.080 --> 00:54:30.460
takes you through using it with Python, takes you through some schema design.

00:54:30.780 --> 00:54:40.260
It doesn't get into some of the big data analytics, you know, using it with Hadoop or some of the other things, but it does give you a good foundation in MongoDB.

00:54:40.260 --> 00:54:45.740
And, you know, I was happy to say that that was just published yesterday, which would be the 25th of April.

00:54:45.740 --> 00:54:48.940
We're recording on the 26th, so happy to see that out there.

00:54:48.940 --> 00:54:49.300
Yeah.

00:54:49.300 --> 00:54:50.440
How's that for timing?

00:54:50.440 --> 00:54:51.040
Perfect, huh?

00:54:51.040 --> 00:54:51.760
Yeah.

00:54:51.760 --> 00:54:52.920
Nice.

00:54:52.920 --> 00:54:53.300
That's cool.

00:54:53.300 --> 00:54:54.520
That must have been fun to make.

00:54:55.000 --> 00:55:00.560
And you also, speaking of ODMs, you wrote a book with the R instead of a D in there as well, the ORM, right?

00:55:00.560 --> 00:55:01.140
I did.

00:55:01.140 --> 00:55:06.340
This is prior to my involvement with MongoDB, and the name of the book is Essential SQLAlchemy.

00:55:06.340 --> 00:55:07.940
It's also an O'Reilly title.

00:55:07.940 --> 00:55:17.700
So SQLAlchemy, if you are using Python and you are using an SQL database and you are not using SQLAlchemy, then you're missing out, I would say.

00:55:17.780 --> 00:55:25.440
And you're probably a Django developer because they have a really nice ORM themselves, and it has a lot of other features, if you're using Django, that are nice.

00:55:25.440 --> 00:55:30.160
But SQLAlchemy is one of the best libraries, object-relational mappers.

00:55:30.160 --> 00:55:33.080
I mean, it is the best I've ever seen.

00:55:33.080 --> 00:55:34.540
Yeah, it's really, really good.

00:55:34.540 --> 00:55:36.780
I've used it a lot, and it's been perfect.

00:55:36.780 --> 00:55:37.200
Yeah.

00:55:37.200 --> 00:55:46.020
A lot of the time when you get something like an object-relational mapper, then you give up a lot of the goodness of, like, a lot of the strengths of SQL.

00:55:46.160 --> 00:55:57.260
And I think that Mike Baer, who is the author of SQLAlchemy, really did a good job of giving you the abstractions of an ORM while still allowing you to get the performance of raw SQL.

00:55:57.260 --> 00:56:01.880
So I was really happy with that, and a second edition of that came out in the last year.

00:56:01.880 --> 00:56:06.400
I didn't have a lot to do with the second edition, but because I wrote the first edition, I get to have my name on the cover.

00:56:06.400 --> 00:56:07.720
Nice.

00:56:07.720 --> 00:56:09.480
Perfect.

00:56:09.480 --> 00:56:14.540
Yeah, and I actually had Mike Baer on one of the first episodes, episode five, so dug into that.

00:56:14.540 --> 00:56:15.940
Yeah, I like SQLAlchemy a lot.

00:56:16.040 --> 00:56:17.060
He is a smart dude.

00:56:17.060 --> 00:56:18.020
Indeed.

00:56:18.020 --> 00:56:20.460
All right, Rick.

00:56:20.460 --> 00:56:21.840
So we're about out of time.

00:56:21.840 --> 00:56:30.840
I don't want to take all your day up, but so let me ask you two quick questions before I let you out of here, and then one more thing after that.

00:56:30.840 --> 00:56:34.160
So if you're going to write some Python code, what editor do you open up?

00:56:34.160 --> 00:56:35.900
I open up Sublime Text 3.

00:56:35.900 --> 00:56:36.880
Sublime Text.

00:56:36.880 --> 00:56:37.260
All right.

00:56:37.260 --> 00:56:38.800
Definitely a solid one.

00:56:38.880 --> 00:56:43.780
Do you have extra plugins, or do you use the Anaconda IDE thing that plugs in there?

00:56:43.780 --> 00:56:45.520
Not the Continuum thing, but something else.

00:56:45.520 --> 00:56:49.220
No, I pretty much use almost the default install.

00:56:49.220 --> 00:56:51.120
I mean, package control is in there.

00:56:51.120 --> 00:56:57.880
Occasionally do some React programming to mention a different programming language, but get the JSX plugin and things like that.

00:56:58.000 --> 00:57:00.620
But it's Sublime, pretty vanilla for me.

00:57:00.620 --> 00:57:01.040
Nice.

00:57:01.520 --> 00:57:04.600
There's a ton, over 100,000 packages on PyPI.

00:57:04.600 --> 00:57:08.400
Is there one that's kind of notable you think maybe people haven't tried or heard of that you want to recommend?

00:57:08.400 --> 00:57:17.620
Well, other than things like PyMongo and SQLAlchemy that we've already mentioned, one of the ones that it just comes up over and over, and people may have already, a lot of people have heard of it, is Requests.

00:57:17.620 --> 00:57:22.260
It's the most un-Google-able package name.

00:57:22.840 --> 00:57:28.820
But if you're going to do any web programming in Python as a client, you need the Requests library.

00:57:28.820 --> 00:57:29.700
Yeah, absolutely.

00:57:29.700 --> 00:57:34.340
So I think it would be un-Google-able if it weren't so popular.

00:57:34.340 --> 00:57:35.920
Yeah, true.

00:57:35.920 --> 00:57:37.880
So Python Requests is your best bet.

00:57:37.880 --> 00:57:38.120
Yeah.

00:57:38.120 --> 00:57:39.040
Exactly, exactly.

00:57:39.040 --> 00:57:40.240
All right.

00:57:40.240 --> 00:57:43.200
Well, that's about all the time we have to talk about Mongo for today.

00:57:43.200 --> 00:57:44.600
Any final call to actions?

00:57:44.600 --> 00:57:46.280
People are excited about this stuff.

00:57:46.280 --> 00:57:48.080
How do they learn more, do more?

00:57:48.080 --> 00:57:51.580
So MongoDB.org can teach you a lot about MongoDB.

00:57:52.260 --> 00:57:58.660
You know, obviously the course, which will be in the show notes, but there's also MongoDB World coming up this summer in Chicago.

00:57:58.660 --> 00:58:02.020
So that might be a good place if you're really interested in this database.

00:58:02.020 --> 00:58:04.640
It's probably the cheapest education that you can get.

00:58:04.640 --> 00:58:08.420
And it's, you know, two days of talks and tutorials before that.

00:58:08.420 --> 00:58:11.020
So I guess those are my calls to action.

00:58:11.020 --> 00:58:11.420
Yeah, cool.

00:58:11.420 --> 00:58:14.480
MongoDB World, that's like the PyCon of MongoDB.

00:58:14.480 --> 00:58:15.460
Yes.

00:58:15.460 --> 00:58:17.020
That's the big one to go to.

00:58:17.020 --> 00:58:19.120
It's in Chicago, and that's cool.

00:58:19.120 --> 00:58:21.440
It used to be in New York City every time.

00:58:21.680 --> 00:58:24.720
Yeah, this is the first time that they've kind of ventured out of Manhattan.

00:58:24.720 --> 00:58:26.920
So it'll be interesting to see what goes on there.

00:58:26.920 --> 00:58:27.440
Yeah, indeed.

00:58:27.440 --> 00:58:28.020
All right.

00:58:28.020 --> 00:58:29.720
Well, Rick, thank you so much for being on the show.

00:58:29.720 --> 00:58:31.780
It's been great to chat about Mongo.

00:58:31.780 --> 00:58:32.220
All right.

00:58:32.220 --> 00:58:32.740
Well, thank you.

00:58:34.500 --> 00:58:37.360
This has been another episode of Talk Python To Me.

00:58:37.880 --> 00:58:40.140
Today's guest has been Rick Copeland.

00:58:40.140 --> 00:58:43.480
And this episode has been sponsored by Advanced Digital and Hired.

00:58:43.480 --> 00:58:51.480
Advanced Digital would love to work with you to build and extend one of the most visited websites in the U.S. in Python.

00:58:51.480 --> 00:58:55.920
Reach out to them at python.advance.net to see if there's a fit.

00:58:57.200 --> 00:58:59.180
Hired wants to help you find your next big thing.

00:58:59.180 --> 00:59:07.740
Visit talkpython.fm/hired to get five or more offers with salary and equity presented right up front and a special listener signing bonus of $600.

00:59:08.820 --> 00:59:10.920
Are you or your colleagues trying to learn Python?

00:59:10.920 --> 00:59:13.980
Well, be sure to visit training.talkpython.fm.

00:59:13.980 --> 00:59:19.780
We now have year-long course bundles and a couple of new classes released just this week.

00:59:19.780 --> 00:59:20.760
Have a look around.

00:59:20.760 --> 00:59:22.220
I'm sure you'll find a class you'll enjoy.

00:59:22.220 --> 00:59:24.640
Be sure to subscribe to the show.

00:59:24.640 --> 00:59:26.840
Open your favorite podcatcher and search for Python.

00:59:26.840 --> 00:59:28.080
We should be right at the top.

00:59:28.080 --> 00:59:37.400
You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm.

00:59:37.740 --> 00:59:42.480
Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.

00:59:42.480 --> 00:59:49.180
Corey just recently started selling his tracks on iTunes, so I recommend you check it out at talkpython.fm/music.

00:59:49.180 --> 00:59:54.520
You can browse his tracks he has for sale on iTunes and listen to the full-length version of the theme song.

00:59:54.520 --> 00:59:56.600
This is your host, Michael Kennedy.

00:59:56.600 --> 00:59:57.900
Thanks so much for listening.

00:59:57.900 --> 00:59:59.100
I really appreciate it.

00:59:59.100 --> 01:00:01.260
Smix, let's get out of here.

01:00:01.260 --> 01:00:05.480
Stating with my voice, there's no norm that I can feel within.

01:00:05.480 --> 01:00:06.720
Haven't been sleeping.

01:00:07.020 --> 01:00:08.320
I've been using lots of RSS.

01:00:08.320 --> 01:00:11.160
I'll pass the mic back to who rocked his best.

01:00:11.160 --> 01:00:14.500
First, developers, developers, developers, developers.

01:00:14.500 --> 01:00:17.500
First, developers, developers, developers.

01:00:17.500 --> 01:00:20.500
First, developers, developers.

01:00:20.500 --> 01:00:22.640
First, developers, developers.

01:00:22.640 --> 01:00:23.400
First, developers.

01:00:23.400 --> 01:00:25.460
you