WEBVTT

00:00:00.001 --> 00:00:05.140
In this episode, we'll be discussing two powerful tools for data reporting and exploration,

00:00:05.140 --> 00:00:06.800
Dataset and Dogsheep.

00:00:06.800 --> 00:00:12.260
Dataset helps people take data of any size or shape, analyze and explore it, and publish

00:00:12.260 --> 00:00:14.980
it as an interactive website and an accompanying API.

00:00:14.980 --> 00:00:20.500
Dogsheep is a collection of tools for personal analytics using SQLite and Dataset.

00:00:20.500 --> 00:00:25.320
Imagine a unified search engine for everything personal in your life, such as Twitter, photos,

00:00:25.320 --> 00:00:30.600
Google Docs, Todoist, Goodreads, and more, all in one place and outside of the cloud companies.

00:00:30.600 --> 00:00:34.740
On this episode, we talk with Simon Willison, who created both of these projects.

00:00:34.740 --> 00:00:39.160
He's also one of the co-creators of Django, and will discuss some of the early Django history.

00:00:39.160 --> 00:00:44.600
This is Talk By The Nome, episode 299, recorded November 18th, 2020.

00:00:44.600 --> 00:01:02.460
Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the

00:01:02.460 --> 00:01:03.880
ecosystem, and the personalities.

00:01:03.880 --> 00:01:05.680
This is your host, Michael Kennedy.

00:01:05.680 --> 00:01:09.820
Follow me on Twitter, where I'm @mkennedy, and keep up with the show and listen to past

00:01:09.820 --> 00:01:14.740
episodes at talkpython.fm, and follow the show on Twitter via at Talk Python.

00:01:14.740 --> 00:01:18.640
This episode is brought to you by Linode and Talk Python Training.

00:01:18.640 --> 00:01:20.940
Please check out the offers during their segments.

00:01:20.940 --> 00:01:22.240
It really helps support the show.

00:01:22.240 --> 00:01:24.600
Simon, welcome to Talk Python To Me.

00:01:24.600 --> 00:01:25.740
Hi, great to be here.

00:01:25.740 --> 00:01:26.960
Hey, it's great to have you here.

00:01:26.960 --> 00:01:31.880
We're going to talk about some really interesting projects that you've been working on, some

00:01:31.880 --> 00:01:36.320
that are extremely broad and far-reaching, and anybody who touches Python or maybe even

00:01:36.320 --> 00:01:40.480
knows about it, and then another, which I feel is such a personal project.

00:01:40.480 --> 00:01:44.840
It's for everybody, but it reveals so much data about you, you know?

00:01:44.840 --> 00:01:45.900
Absolutely, yeah.

00:01:45.900 --> 00:01:49.540
We're going to talk about Django a little bit, which you were part of creating.

00:01:49.540 --> 00:01:55.420
We're going to talk about data set, which is a way to interact, basically put a UI on

00:01:55.420 --> 00:01:59.600
top of any database in a friendly way that lets you explore it and treat it as an API.

00:01:59.600 --> 00:02:05.240
And then Dog Sheep, which has a funny name, nice story around the name, that allows you

00:02:05.240 --> 00:02:10.380
to basically take all of the data about you and turn it into your own Google.

00:02:10.380 --> 00:02:10.700
Right.

00:02:10.700 --> 00:02:13.640
It's your personal data warehouse, I've started calling it.

00:02:13.640 --> 00:02:15.580
I'm fascinated by where that could go.

00:02:15.580 --> 00:02:18.740
Now, before we get to all three of those cool things, let's just start with your story.

00:02:18.740 --> 00:02:20.260
How did you get into programming into Python?

00:02:20.260 --> 00:02:27.520
I learned to program with my dad on a Commodore 64 back in the 80s, and then kind of moved on

00:02:27.520 --> 00:02:31.600
to early Windows and DOS, where you didn't really get to program anything from QBasic.

00:02:31.600 --> 00:02:37.700
But I got really into programming when actually during the first dot-com boom, like even like

00:02:37.700 --> 00:02:45.160
Web 1.0, 1999, 2000, when I was working in London for a online gaming company, a company

00:02:45.160 --> 00:02:50.060
called Gameplay.com, which was doing like selling boxed games where you mail ordered them, but

00:02:50.060 --> 00:02:55.080
also running online gaming servers for like Team Fortress Classic and Half-Life and Quake and

00:02:55.080 --> 00:02:56.140
all of those kinds of things.

00:02:56.140 --> 00:02:56.760
Oh, yeah.

00:02:56.760 --> 00:02:58.780
Team Fortress and Counter-Strike.

00:02:58.780 --> 00:02:59.860
I love that stuff.

00:02:59.860 --> 00:03:00.580
That was so fun.

00:03:00.580 --> 00:03:01.340
Oh, absolutely.

00:03:01.340 --> 00:03:06.440
And so I was working for their online gaming division for a year and a half before the

00:03:06.440 --> 00:03:09.080
dot-com crash and everyone got laid off and it all collapsed.

00:03:09.080 --> 00:03:14.860
But during that time, I started the company as the downloads editor, so responsible for

00:03:14.860 --> 00:03:19.500
the section of the website where you download plugins and patches and mods and so forth.

00:03:19.500 --> 00:03:24.260
And I essentially taught myself web development as part of that and as part of some other gaming

00:03:24.260 --> 00:03:25.800
related side projects I had.

00:03:25.800 --> 00:03:31.080
And so, yeah, I'm a veteran of the first round of dot-coms back when no one had a slight

00:03:31.080 --> 00:03:32.180
idea what we were doing.

00:03:32.180 --> 00:03:32.840
Yeah, yeah.

00:03:32.840 --> 00:03:34.620
Everyone was just making it up on the spot, right?

00:03:34.620 --> 00:03:35.140
Oh, totally.

00:03:35.140 --> 00:03:39.660
So when you were working on that web development stuff, it probably wasn't Python, I'm guessing.

00:03:39.660 --> 00:03:42.120
Certainly wasn't Django because Django was what, 2005?

00:03:42.120 --> 00:03:45.240
2003, 2004 when we built it.

00:03:45.240 --> 00:03:47.360
I think it was open sourced in 2005.

00:03:47.360 --> 00:03:47.820
Yeah.

00:03:47.820 --> 00:03:48.460
Right, right.

00:03:48.660 --> 00:03:49.980
Okay, so what were you programming then?

00:03:49.980 --> 00:03:55.080
Oh, so Gameplay.com was running on a vastly expensive content management system called Media

00:03:55.080 --> 00:04:01.940
Surface, which was a combination of Perl for templates, Java and Oracle under the hood and

00:04:01.940 --> 00:04:06.360
insanely expensive and very, very tricky to get things done with.

00:04:06.360 --> 00:04:09.280
And then I had side projects which were classic PHP and MySQL.

00:04:09.280 --> 00:04:09.980
Right.

00:04:09.980 --> 00:04:13.560
So I was very much the sort of the classic PHP programmer.

00:04:13.560 --> 00:04:18.620
And actually, I mean, this is where Django came from is Adrian Holabati and I were working

00:04:18.620 --> 00:04:20.780
together at this local newspaper in Lawrence, Kansas.

00:04:20.780 --> 00:04:26.580
And we were both PHP developers and we saw the siren call of Python and wanted to figure out

00:04:26.580 --> 00:04:28.500
how we could build websites using this.

00:04:28.500 --> 00:04:31.120
We felt much more exciting programming language.

00:04:31.120 --> 00:04:31.940
Oh, that's awesome.

00:04:31.940 --> 00:04:33.400
It's definitely more exciting.

00:04:33.400 --> 00:04:36.620
But I do want to ask you, it's such a different world, right?

00:04:36.620 --> 00:04:42.400
We can go to the cloud providers and pay $5 a month and probably get better infrastructure

00:04:42.400 --> 00:04:43.480
than you guys had.

00:04:43.480 --> 00:04:44.840
I mean, who knows how much, right?

00:04:44.840 --> 00:04:45.720
Enormously.

00:04:45.720 --> 00:04:49.520
I mean, back then, a content management system, because you could spend a million dollars on

00:04:49.520 --> 00:04:50.660
your content management system.

00:04:50.660 --> 00:04:55.360
And they were terrible because nobody knew what a good content management system looked like.

00:04:55.360 --> 00:04:57.600
This was back before things like WordPress as well.

00:04:57.600 --> 00:05:03.140
Like your only option was to spend huge amounts of money on these like giant enterprise systems

00:05:03.140 --> 00:05:05.720
and then cross your fingers and hope that they could work for you.

00:05:05.720 --> 00:05:06.080
Yeah.

00:05:06.440 --> 00:05:11.120
So how did you feel when you worked, you go to work, you work in this kind of like ultra

00:05:11.120 --> 00:05:13.500
expensive, clunky environment and then go home.

00:05:13.500 --> 00:05:17.620
And even though it was PHP, still like do PHP and MySQL and kind of like, I paid nothing

00:05:17.620 --> 00:05:18.040
for this.

00:05:18.040 --> 00:05:21.880
It maybe felt as good as what this crazy online system you guys are working with.

00:05:21.880 --> 00:05:23.320
What was that back and forth like?

00:05:23.320 --> 00:05:26.480
I don't think it did feel nearly as good, to be honest, because there was no open source

00:05:26.480 --> 00:05:27.980
was still just starting up.

00:05:27.980 --> 00:05:31.620
So if you wanted to build something in MySQL, you wrote everything.

00:05:31.620 --> 00:05:32.240
Oh my goodness.

00:05:33.120 --> 00:05:36.600
You would start with the authentication system and work on it from there.

00:05:36.680 --> 00:05:42.400
So to be honest, I feel like today, like open source for me has solved the whole code

00:05:42.400 --> 00:05:43.240
reuse problem.

00:05:43.240 --> 00:05:46.820
How do we stop wasting our time to rebuilding the same things over and over again?

00:05:46.820 --> 00:05:51.720
But yet, you know, 20 years ago, you just built everything from scratch and everything

00:05:51.720 --> 00:05:52.740
took months to do.

00:05:52.900 --> 00:05:57.980
It's really interesting because I started out in C++ and I felt like a lot of the stuff

00:05:57.980 --> 00:06:02.920
I built was extremely low level, not just because it was C++, but because of this sort of difference

00:06:02.920 --> 00:06:03.800
that you're talking about.

00:06:03.800 --> 00:06:09.020
And throughout the years, I've seen it just get like what used to be really hard is now

00:06:09.020 --> 00:06:10.480
just grab this library and plug it in.

00:06:10.480 --> 00:06:12.700
That other thing used to be hard, grab this library and plug it in.

00:06:12.700 --> 00:06:17.400
And it just seems like the natural consequence would be, well, we need fewer developers because

00:06:17.400 --> 00:06:18.300
they used to have to build this.

00:06:18.300 --> 00:06:19.180
Now they plug it together.

00:06:19.180 --> 00:06:22.500
And yet all we've done is decided to solve more ambitious problems.

00:06:22.500 --> 00:06:23.140
Absolutely.

00:06:23.140 --> 00:06:24.720
In more interesting ways, right?

00:06:24.720 --> 00:06:30.220
It amazes me the quality of software that we build these days, you know, like 10, 15 years

00:06:30.220 --> 00:06:32.340
ago, we weren't writing tests for everything.

00:06:32.340 --> 00:06:34.900
You know, the quality of the stuff we wrote was just abysmal.

00:06:34.900 --> 00:06:38.000
And today we've got continuous integration and continuous deployment.

00:06:38.000 --> 00:06:42.800
And it's really quite, it's really easy to get out there and analyze like 15 different

00:06:42.800 --> 00:06:46.260
options and pick the one that's most, the one that is the highest quality.

00:06:46.260 --> 00:06:52.280
Not only do you have CI/CD, you've got it for free in the cloud on demand, you know, on actions

00:06:52.280 --> 00:06:53.120
that trigger it, right?

00:06:53.120 --> 00:06:55.920
It's just, it's such a cool place, a cool time to be doing this stuff.

00:06:55.920 --> 00:06:56.320
Definitely.

00:06:56.320 --> 00:06:56.640
Yeah.

00:06:56.640 --> 00:06:57.020
Yeah.

00:06:57.020 --> 00:06:59.940
So before we get to Django, how about now?

00:06:59.940 --> 00:07:00.960
What are you up to these days?

00:07:00.960 --> 00:07:05.940
So these days I spent the last year at Stanford University doing a fellowship.

00:07:05.940 --> 00:07:10.280
There's a fellowship program called JSK and it's journalism fellowships.

00:07:10.280 --> 00:07:14.740
The idea is to get sort of journalists from around the world together at Stanford for a year,

00:07:14.900 --> 00:07:17.040
thinking about problems facing journalism.

00:07:17.040 --> 00:07:22.520
And I managed to make my own to it as a sort of computer scientist with journalism leanings.

00:07:22.520 --> 00:07:27.840
And basically thinking about, okay, what are the open source tools that I can build that

00:07:27.840 --> 00:07:31.760
can help make data journalism more powerful, more widely accessible?

00:07:31.760 --> 00:07:36.900
And so a lot of the data set project was really accelerated by doing that.

00:07:36.900 --> 00:07:38.140
It finished a few months ago.

00:07:38.420 --> 00:07:43.680
The problem I'm having is that at Stanford, I was essentially paid to tinker on my own

00:07:43.680 --> 00:07:46.020
projects and go after whatever I thought was interesting.

00:07:46.020 --> 00:07:46.960
That sounds good.

00:07:46.960 --> 00:07:48.840
And it's great, but I'm having real trouble stopping.

00:07:48.840 --> 00:07:53.500
So now I'm not getting paid, but I'm tinkering on my own projects, going after things I find

00:07:53.500 --> 00:07:53.920
interesting.

00:07:53.920 --> 00:07:59.100
So I'm calling myself a consultant and I am available for consulting opportunities, especially

00:07:59.100 --> 00:08:01.680
around the sort of set of tools that I've been building.

00:08:02.040 --> 00:08:07.560
But mainly, yeah, I'm focusing on continuing to build out this ecosystem of tooling for

00:08:07.560 --> 00:08:09.480
data journalism and related projects.

00:08:09.480 --> 00:08:09.820
Yeah.

00:08:09.820 --> 00:08:11.080
Well, we're going to get to those.

00:08:11.080 --> 00:08:13.820
And they're definitely super interesting, the ones.

00:08:13.820 --> 00:08:17.880
And like I said, this personal aspect, I think that could help a lot of people, not just journalists

00:08:17.880 --> 00:08:18.340
for sure.

00:08:18.340 --> 00:08:20.000
But let's talk a little bit about Django.

00:08:20.000 --> 00:08:22.540
You mentioned that it came out of, what was it?

00:08:22.540 --> 00:08:23.700
Journal Lawrence?

00:08:23.700 --> 00:08:25.680
Lawrence Journal World or something like that?

00:08:25.680 --> 00:08:26.260
Journal World.

00:08:26.360 --> 00:08:31.060
Yeah, it's a tiny newspaper in Lawrence, Kansas, a town I'd never even heard of.

00:08:31.060 --> 00:08:34.060
And this was back in 2002, 2003.

00:08:34.060 --> 00:08:35.260
I was a blogger.

00:08:35.260 --> 00:08:40.260
I had a blog about web development and about 100 other people had blogs about web development.

00:08:40.260 --> 00:08:41.680
So we all read each other's blogs.

00:08:41.680 --> 00:08:48.300
And Adrian Holovaty, who was a journalism and programmer, posted a job ad on his blog saying,

00:08:48.300 --> 00:08:51.600
hey, I want somebody to come and join me in Lawrence, Kansas, building websites for local

00:08:51.600 --> 00:08:52.020
newspapers.

00:08:52.400 --> 00:08:57.680
And it coincided with my university course, giving me the option to spend a year in industry,

00:08:57.680 --> 00:09:00.020
which is something that UK degrees do quite often.

00:09:00.020 --> 00:09:06.180
So I could take a year out, get a student visa, which means I could track traveling, work in

00:09:06.180 --> 00:09:09.720
different countries, spend a year working, and then go back and finish my degree.

00:09:09.720 --> 00:09:12.680
And so the opportunities sort of aligned themselves.

00:09:12.680 --> 00:09:16.520
And I had huge respect from Adrian, just based on what I'd seen that he'd been doing.

00:09:16.520 --> 00:09:19.800
And it felt like a pretty interesting adventure to run off to Kansas.

00:09:19.800 --> 00:09:20.720
Yeah, yeah.

00:09:20.720 --> 00:09:21.640
That's a cool adventure.

00:09:21.900 --> 00:09:22.820
And yes, so I did that.

00:09:22.820 --> 00:09:26.320
So essentially, it was a year long, almost a paid internship.

00:09:26.320 --> 00:09:29.240
But it was in Lawrence, Kansas at this little newspaper.

00:09:29.240 --> 00:09:35.720
And it was a fascinating place because the family that owned the newspaper had laid fiber optic

00:09:35.720 --> 00:09:39.920
cable around a bunch of states like a bunch of years beforehand, when everyone thought they

00:09:39.920 --> 00:09:44.560
were crazy, and then sold the whole lot to maybe Comcast, one of these big companies.

00:09:44.560 --> 00:09:49.060
So financially, they were very secure, which meant they could invest huge resources in their

00:09:49.060 --> 00:09:50.740
local newspaper for this little town.

00:09:51.180 --> 00:09:55.120
And so this newspaper, despite serving a town with a population of like 100,000 people,

00:09:55.120 --> 00:09:59.720
it had way, way more resources than you would expect any local newspaper to have.

00:09:59.720 --> 00:10:03.500
They had their own software engineering team who were building out the websites for things.

00:10:03.820 --> 00:10:06.800
And because the family owned the local cable company for the town.

00:10:06.800 --> 00:10:12.500
So everyone in the town of Lawrence, Kansas had broadband internet in 2003.

00:10:12.500 --> 00:10:20.780
And that meant that the news websites could have like online videos and stuff, which no one else was doing, because what newspaper had an audience who could actually watch that kind of stuff.

00:10:20.780 --> 00:10:26.420
So it was a really exciting place to be inventing like things around online news.

00:10:26.420 --> 00:10:39.100
And we also had a very ambitious boss, this chap called Rob Curley, who basically wanted us to act like we were the New York Times, even though we were like six nerds in a basement somewhere.

00:10:39.100 --> 00:10:40.260
This little local newspaper.

00:10:40.260 --> 00:10:48.180
We had things like the local, there was a softball league where all of the local kids in softball teams competing against each other.

00:10:48.480 --> 00:10:56.420
And it turns out this is an amazing thing for a local newspaper to cover, because if you have good coverage of the softball league, everyone who knows a child in your town will buy your newspaper.

00:10:56.420 --> 00:11:02.040
So we went all in on kids softball and we ended up building a website for them that treated them.

00:11:02.040 --> 00:11:04.400
The idea was to treat them like the New York Yankees.

00:11:04.400 --> 00:11:08.240
So we had like player profiles and match reports and photo galleries.

00:11:08.240 --> 00:11:15.540
And then we sent two interns out to take 360 degree photographs of every softball pitch in town.

00:11:15.540 --> 00:11:16.940
And we had those on the website.

00:11:17.100 --> 00:11:19.660
And this was like quick VR or something.

00:11:19.660 --> 00:11:21.440
It was absolutely astonishing.

00:11:21.440 --> 00:11:23.340
That's so, so neat in the early days.

00:11:23.340 --> 00:11:25.900
Like, I'm sure the kids felt so special as well.

00:11:25.900 --> 00:11:29.120
I bet they still have like saved copies of their time.

00:11:29.120 --> 00:11:34.360
The best website we worked on was there was a local entertainment portal for the town of Lawrence, Kansas.

00:11:34.360 --> 00:11:37.300
And it was basically a website that had the events calendar.

00:11:37.300 --> 00:11:38.940
It had band profiles.

00:11:38.940 --> 00:11:40.160
It had restaurant reviews.

00:11:40.160 --> 00:11:46.660
It was like sort of a super hyper local version of Yelp crossed with music magazines,

00:11:46.720 --> 00:11:49.620
crossed with an events website just for this one little town.

00:11:49.620 --> 00:11:58.000
And we had features like there was a downloads page where you could download MP3s of bands who were playing in town that week.

00:11:58.000 --> 00:12:02.540
Because we had the MP3s for all of the local bands, again, in like 2003.

00:12:02.540 --> 00:12:05.580
And a little radio widget that you could click play on and stuff.

00:12:05.580 --> 00:12:06.420
It was astonishing.

00:12:06.740 --> 00:12:12.640
I have never seen an entertainment website since that was as good as this one that we were building back in Lawrence, Kansas.

00:12:15.380 --> 00:12:17.320
This portion of Talk Python To Me is sponsored by Linode.

00:12:17.320 --> 00:12:21.680
Simplify your infrastructure and cut your cloud bills in half with Linode's Linux virtual machines.

00:12:21.680 --> 00:12:25.740
Develop, deploy, and scale your modern applications faster and easier.

00:12:25.740 --> 00:12:29.120
Whether you're developing a personal project or managing large workloads,

00:12:29.120 --> 00:12:33.140
you deserve simple, affordable, and accessible cloud computing solutions.

00:12:33.280 --> 00:12:37.400
As listeners of Talk Python To Me, you'll get a $100 free credit.

00:12:37.400 --> 00:12:41.540
You can find all the details at talkpython.fm/Linode.

00:12:41.540 --> 00:12:47.260
Linode has data centers around the world with the same simple and consistent pricing regardless of location.

00:12:47.260 --> 00:12:50.100
Just choose the data center that's nearest to your users.

00:12:50.240 --> 00:12:56.800
You'll also receive 24-7-365 human support with no tiers or handoffs regardless of your plan size.

00:12:56.800 --> 00:13:00.120
You can choose shared and dedicated compute instances,

00:13:00.120 --> 00:13:04.220
or you can use your $100 in credit on S3-compatible object storage,

00:13:04.220 --> 00:13:07.020
managed Kubernetes clusters, and more.

00:13:07.020 --> 00:13:09.500
If it runs on Linux, it runs on Linode.

00:13:09.500 --> 00:13:14.180
Visit talkpython.fm/Linode or click the link in your show notes,

00:13:14.180 --> 00:13:16.880
then click that create free account button to get started.

00:13:18.780 --> 00:13:20.240
I have a history with Lawrence.

00:13:20.240 --> 00:13:24.140
I actually went to college there and got my math degree at University of Kansas.

00:13:24.140 --> 00:13:24.760
No way!

00:13:24.760 --> 00:13:25.860
I love that town.

00:13:25.860 --> 00:13:26.560
I love it.

00:13:26.560 --> 00:13:31.540
The Mass Street little downtown with Mass Street Brewery and all that was such a beautiful place.

00:13:31.540 --> 00:13:33.520
I really enjoyed my time there.

00:13:33.520 --> 00:13:34.500
Yeah, it's a great town.

00:13:34.500 --> 00:13:38.720
I think when I was there, I'd never been to anywhere else in America, basically.

00:13:38.720 --> 00:13:43.880
So it felt like a cool town, but I didn't really understand how cool it was until 20 years later.

00:13:43.880 --> 00:13:47.860
I've now lived in America, and I've been to lots of towns, and Lawrence is special.

00:13:47.860 --> 00:13:49.240
It's a very special.

00:13:49.240 --> 00:13:50.220
It definitely is.

00:13:50.220 --> 00:13:54.920
But what I never knew was how cool this newspaper was.

00:13:54.920 --> 00:13:56.380
I mean, I was just like a kid in college.

00:13:56.380 --> 00:13:58.460
I didn't read the newspaper a lot and whatnot.

00:13:58.460 --> 00:14:02.720
So this is a really interesting cradle from which Django sprung.

00:14:02.920 --> 00:14:06.160
So tell us about Django and how it fits into this world.

00:14:06.160 --> 00:14:06.600
Sure.

00:14:06.600 --> 00:14:11.620
So basically, Adrian and I, Adrian had built Lawrence.com, this amazing assignments website, in PHP.

00:14:11.620 --> 00:14:19.900
And both Adrian and I had hit that point in our PHP careers where it was straining under the size and complexity of the things that we wanted to do with it.

00:14:20.080 --> 00:14:22.500
This was like before PHP 4 even.

00:14:22.500 --> 00:14:29.000
So classes were very new, and the PHP language was pretty primitive compared to what you have today.

00:14:29.000 --> 00:14:31.780
And meanwhile, Python was exploding in popularity.

00:14:31.780 --> 00:14:37.220
We were both huge fans of Mark Pilgrim's Dive Into Python and Mark Pilgrim's blog where he talked about this.

00:14:37.620 --> 00:14:41.680
And so we decided that we really wanted to be working with Python for building these websites.

00:14:41.680 --> 00:14:46.980
But the web options back then were not particularly – the main thing was Zope.

00:14:46.980 --> 00:14:51.240
And Zope was pretty good, but it didn't match the way Adrian and I thought about the web.

00:14:51.240 --> 00:14:56.280
We cleared about things like designing our URLs carefully and separating our CSS from our HTML.

00:14:56.280 --> 00:15:00.800
The sort of modern MVC framework that people almost take for granted now, right?

00:15:00.800 --> 00:15:01.220
Exactly.

00:15:01.220 --> 00:15:04.560
But there weren't really any great options for that in Python.

00:15:04.780 --> 00:15:10.020
So we were looking at ModPython, the Apache module, as the way that we would put Python on the Internet.

00:15:10.020 --> 00:15:14.940
And we were a little bit worried about it because ModPython wasn't being used very widely.

00:15:14.940 --> 00:15:20.360
And we're like, okay, what happens if we bet the newspaper on ModPython and it turns out not to work, right?

00:15:20.360 --> 00:15:21.140
That would be –

00:15:21.140 --> 00:15:22.100
Yeah, yeah, yeah, yeah.

00:15:22.100 --> 00:15:29.480
So we said, okay, what we'll do, we'll have a very thin abstraction layer between us and ModPython so that if we have to swap ModPython for something else, we can do so.

00:15:29.480 --> 00:15:33.280
And that basically is what – well, that was the initial seed to Django.

00:15:33.280 --> 00:15:38.460
We wanted a request and response object, a basic way of doing templating, basic URL routing.

00:15:38.460 --> 00:15:39.820
And so we built that out.

00:15:39.820 --> 00:15:41.160
And we never thought it was a framework.

00:15:41.160 --> 00:15:43.160
We called it the CMS, right?

00:15:43.160 --> 00:15:45.160
It was the CMS of the newspaper.

00:15:45.160 --> 00:15:48.380
And it kept on evolving these additional little bits and pieces.

00:15:48.380 --> 00:15:53.800
Django admin was something which I went away to the South by Southwest Festival for like four days.

00:15:54.080 --> 00:16:00.740
And when I came back, Adrian had written a code generator for admin websites that was churning out all of this stuff.

00:16:00.740 --> 00:16:03.100
We just kept on building these extra bits out.

00:16:03.100 --> 00:16:04.620
And then I went back to England.

00:16:04.620 --> 00:16:06.000
My year in Kansas ended.

00:16:06.000 --> 00:16:09.300
And about six months later, they open-sourced Django.

00:16:09.300 --> 00:16:11.500
At the time I was working, it wasn't called Django.

00:16:11.500 --> 00:16:14.560
There were various ideas for names, which are truly terrible.

00:16:14.680 --> 00:16:18.120
But yeah, Jacob Kaplan Moss had joined the team at that point.

00:16:18.120 --> 00:16:21.700
And yeah, they made the case to the newspaper that they should open-source this thing.

00:16:21.700 --> 00:16:23.420
That's early days for that, right?

00:16:23.420 --> 00:16:25.380
Like it's not – like now it would be easy sell.

00:16:25.380 --> 00:16:26.900
But back then that was weird, right?

00:16:26.900 --> 00:16:27.480
Well, yeah.

00:16:27.480 --> 00:16:28.720
I asked them about this.

00:16:28.720 --> 00:16:33.980
And apparently one of the arguments they used is Ruby on Rails had just come out and was just exploding in popularity.

00:16:34.200 --> 00:16:39.720
And they could see that this company that released Rails was hiring people left, right, and center and was doing really well out of it.

00:16:39.720 --> 00:16:47.500
So they went to the newspaper and said, hey, look, if we open-source this, it's a great way for us to get talent and get free fixes and all of that sort of thing.

00:16:47.500 --> 00:16:48.160
And it worked.

00:16:48.160 --> 00:16:52.480
Like you and I are sitting here talking about this small newspaper in Lawrence right now, right?

00:16:52.480 --> 00:16:54.220
I mean, we wouldn't be doing this otherwise.

00:16:54.220 --> 00:16:54.960
That's true.

00:16:54.960 --> 00:17:00.060
But the argument that worked is they said to the newspaper owners, we've been building on open-source, right?

00:17:00.060 --> 00:17:05.460
The newspaper – we run Linux and we run Apache and Perl and Python and we've used all of this open-source stuff.

00:17:05.460 --> 00:17:07.000
This is a way of us giving back.

00:17:07.000 --> 00:17:08.980
And that's the argument that apparently resonated with them.

00:17:08.980 --> 00:17:10.280
They said, no, that completely makes sense.

00:17:10.280 --> 00:17:12.080
We can give back in that way.

00:17:12.080 --> 00:17:16.860
And yeah, and so Django open-sourced and it's been – that was 15 years ago, I think.

00:17:16.860 --> 00:17:18.760
And it's just been growing in it ever since.

00:17:18.760 --> 00:17:19.220
Yeah.

00:17:19.220 --> 00:17:22.540
Did you predict – like you look around now, it's just ubiquitous.

00:17:22.540 --> 00:17:24.400
Like does it blow you away what's happened?

00:17:24.400 --> 00:17:25.300
Completely blows me away.

00:17:25.300 --> 00:17:31.160
The thing that really amuses me is that I keep on seeing people talking about Django as the boring option.

00:17:31.160 --> 00:17:34.700
And like Django and Rails, yeah, those are the safe, boring options for things.

00:17:34.700 --> 00:17:39.280
And I actually saw someone on Twitter the other day say, well, nobody ever got fired for choosing Django.

00:17:39.280 --> 00:17:44.140
And I was like – I actually – I direct messaged Adrian and Jacob with that quote.

00:17:44.140 --> 00:17:45.760
I'm like, can you believe this?

00:17:45.760 --> 00:17:49.940
That we are now – no one ever got fired for choosing IBM option for web framework.

00:17:49.940 --> 00:17:50.140
Exactly.

00:17:50.140 --> 00:17:50.880
Exactly.

00:17:50.880 --> 00:17:51.580
Amazing.

00:17:51.580 --> 00:17:52.040
Yeah.

00:17:52.040 --> 00:17:54.080
I think there's definitely some truth to that.

00:17:54.080 --> 00:17:54.940
Quite interesting.

00:17:54.940 --> 00:17:57.240
It seems like it's got a ton of momentum.

00:17:57.240 --> 00:18:00.060
It's really starting to embrace the async and await world.

00:18:00.060 --> 00:18:01.540
It looks strong.

00:18:01.540 --> 00:18:02.000
Yeah.

00:18:02.000 --> 00:18:07.320
So a lot of my projects these days don't use Django, but they do use ASCII.

00:18:07.320 --> 00:18:12.380
I am really into the whole – the ASCII ecosystem that's growing up is so exciting.

00:18:12.380 --> 00:18:20.840
And Django is getting better at ASCII itself, so I'm going to be able to merge a bunch of my ASCII projects back into my Django projects pretty soon, which is super exciting.

00:18:20.840 --> 00:18:21.960
Yeah, that's exciting.

00:18:21.960 --> 00:18:26.000
I'm super excited about FastAPI, and it's one of those that fits really well in that world.

00:18:26.000 --> 00:18:26.360
Yeah.

00:18:26.360 --> 00:18:30.540
I mean, I haven't really done FastAPI, but I love Starlet, which is the framework.

00:18:30.540 --> 00:18:31.520
Yes, that's the foundation.

00:18:31.520 --> 00:18:31.840
Yeah.

00:18:31.840 --> 00:18:32.240
Yeah.

00:18:32.240 --> 00:18:33.180
Yeah, super cool.

00:18:33.180 --> 00:18:33.620
All right.

00:18:33.620 --> 00:18:37.220
Well, congratulations on Django to you and everyone who worked on it.

00:18:37.220 --> 00:18:38.580
I do think it's really interesting.

00:18:39.000 --> 00:18:48.640
If you look at the timing, Ruby on Rails came out of 37 Signals, now Basecamp, and was extracted from – they were using inside, right?

00:18:48.640 --> 00:18:48.780
Yeah.

00:18:48.780 --> 00:18:50.480
And they built it for them and extracted it.

00:18:50.480 --> 00:18:55.940
You guys built it at the newspaper and said, this thing we can pull out of and make it something else.

00:18:56.000 --> 00:19:01.600
I think it's really interesting that it was sort of polished and proven in this real place, right?

00:19:01.720 --> 00:19:05.060
I think the way I see it is Rails was extracted from Basecamp, right?

00:19:05.060 --> 00:19:06.360
They built Basecamp up.

00:19:06.360 --> 00:19:07.600
They pulled a framework out of it.

00:19:07.600 --> 00:19:10.120
Django, the goal was always Lawrence.com.

00:19:10.120 --> 00:19:16.160
We had this entertainment website and PHP and MySQL, and we knew that we wanted the thing we were building to power that.

00:19:16.160 --> 00:19:27.440
So Django was actually – there was an existing target, and we evolved the functionality of the framework until it could run a very high-quality newspaper, entertainment, listings, website.

00:19:27.440 --> 00:19:34.240
And so it's almost – it's like one was extracted and one was evolved in the direction of supporting this one site.

00:19:34.240 --> 00:19:34.700
Yeah.

00:19:34.700 --> 00:19:35.900
It's neat to see.

00:19:35.900 --> 00:19:39.120
I think they both came out quite successful from those experiences.

00:19:39.120 --> 00:19:39.760
All right.

00:19:39.760 --> 00:19:43.200
Let's start at the foundation of this recent work you've been doing.

00:19:43.200 --> 00:19:47.240
And in some sense, it's sort of a natural progression, right?

00:19:47.240 --> 00:19:50.700
It's in the journalism side of things is where the origin came from.

00:19:50.700 --> 00:19:52.620
So tell us about Dataset.

00:19:52.620 --> 00:19:59.020
So Dataset is – on its website, I call it an open-source multi-tool for exploring and publishing data.

00:19:59.020 --> 00:20:08.420
Basically, it's a web application which you can point at a SQLite relational database, and it gives you pages where you can browse the tables and run queries.

00:20:08.420 --> 00:20:12.780
It lets you type in custom SQL queries and run them against that database.

00:20:13.120 --> 00:20:15.480
It lets you custom templates and how things turn.

00:20:15.480 --> 00:20:20.060
It lets you get everything back out as JSON or CSV so you can use it for API integrations.

00:20:20.060 --> 00:20:23.280
And it lets you publish the whole thing on the internet really easily.

00:20:23.280 --> 00:20:24.560
So it's a lot.

00:20:24.560 --> 00:20:24.880
Yeah.

00:20:25.040 --> 00:20:33.480
And one of the biggest challenges I've had is how do I turn this into a bite-sized description that really helps people understand what the software does.

00:20:33.480 --> 00:20:40.840
I'm at a point now where if I can get somebody on a video chat, I can do a 15-minute demo, and at the end of it, they come out going, I totally get this.

00:20:40.840 --> 00:20:41.540
This is amazing.

00:20:41.940 --> 00:20:44.940
But that's not explaining software.

00:20:44.940 --> 00:20:46.800
It doesn't scale particularly well.

00:20:46.800 --> 00:20:47.200
Yeah.

00:20:47.200 --> 00:20:47.280
Yeah.

00:20:47.280 --> 00:21:00.960
Well, let me see if I can, with my limited exposure to it and knowing somewhat where we're going, you have this data source that's pretty ubiquitous or can become ubiquitous in terms of like some sort of ETL with SQLite, right?

00:21:00.960 --> 00:21:01.920
SQLite is everywhere.

00:21:02.440 --> 00:21:08.620
What's beautiful about it is there's no, please set up the server and make it not run as root and then put it on your network and so on.

00:21:08.620 --> 00:21:09.360
Yeah.

00:21:09.360 --> 00:21:09.520
Right?

00:21:09.520 --> 00:21:10.940
It's the magic of SQLite.

00:21:10.940 --> 00:21:17.240
SQLite, it boasts it's the most widely distributed database in the world, which it is because it runs on every phone.

00:21:17.240 --> 00:21:19.840
My watch has SQLite tracking my steps.

00:21:19.840 --> 00:21:24.240
Every iPhone app, every Android app, every laptop, they're all running these.

00:21:24.240 --> 00:21:24.660
Yeah.

00:21:24.660 --> 00:21:24.860
Yeah.

00:21:24.860 --> 00:21:25.260
Your phone.

00:21:25.260 --> 00:21:25.780
That's crazy.

00:21:25.780 --> 00:21:27.620
It's a file format, right?

00:21:27.620 --> 00:21:39.200
It's a SQLite database is a single .db binary file on disk, which like you said, makes it so convenient because I don't have to ask a sysadmin to set me up a Postgres schema or anything like that.

00:21:39.200 --> 00:21:42.520
I just create a file on my laptop and that's my new database.

00:21:42.520 --> 00:21:43.020
Yeah.

00:21:43.020 --> 00:21:44.580
And it's even built into Python, right?

00:21:44.580 --> 00:21:46.260
It just comes with Python.

00:21:46.260 --> 00:21:46.740
Yeah.

00:21:46.740 --> 00:21:48.020
It ships in the standard library.

00:21:48.020 --> 00:21:48.720
Yeah, exactly.

00:21:48.720 --> 00:21:50.560
So that's super cool.

00:21:50.720 --> 00:21:59.340
And it's great that we have this data format that if we either have data in there or you could do like an API call and then jam the data in there, right?

00:21:59.340 --> 00:22:02.300
Like something to get it into that format, which is great.

00:22:02.300 --> 00:22:09.680
But you can explore that with like Beekeeper Studio or some data visualization, SQL Management Studio.

00:22:09.680 --> 00:22:11.720
But that doesn't work for journalists.

00:22:11.720 --> 00:22:13.840
That doesn't work for getting it on the internet.

00:22:14.040 --> 00:22:16.900
That doesn't give you like the transformations.

00:22:16.900 --> 00:22:25.060
In some sense, I kind of see it almost like as a really advanced web-based like data IDE, but user-friendly.

00:22:25.060 --> 00:22:25.760
Kind of.

00:22:25.760 --> 00:22:26.100
Yeah.

00:22:26.100 --> 00:22:29.060
But the emphasis is absolutely on publishing.

00:22:29.060 --> 00:22:30.220
It's on getting it online.

00:22:30.220 --> 00:22:32.180
And then it's on being web native.

00:22:32.180 --> 00:22:35.600
Like everything in dataset can be got out as JSON as well as HTML.

00:22:35.600 --> 00:22:36.820
It can all get beat.

00:22:36.820 --> 00:22:38.720
It can return CSV to you.

00:22:38.720 --> 00:22:45.280
It uses you pass the SQL query in a get in a query string so you can bookmark queries, all of that kind of stuff.

00:22:45.280 --> 00:22:45.600
Yeah.

00:22:45.600 --> 00:22:47.920
I think the key, that's really the key idea.

00:22:47.920 --> 00:22:55.680
It's how do you take relational databases and make them as web native as possible and as cheap and inexpensive to host and to run as possible.

00:22:55.680 --> 00:23:06.340
So you can take any data that fits in a SQLite database, which is almost everything, and stick it online in a way that people can both explore it and start like integrating with it as well.

00:23:06.840 --> 00:23:10.520
And another key idea in dataset is dataset has a plugin system.

00:23:10.520 --> 00:23:19.300
I've actually written over 50 plugins for it now that add all sorts of different things, different output formats to get your database out as an Atom feed or an iCal feed.

00:23:19.300 --> 00:23:25.260
I've got plugins for visualizations that plot your data on a map or give you charts and line graphs and so on.

00:23:25.260 --> 00:23:31.980
I just this morning released a authentication plugin that supports the IndieAuth authentication mechanism.

00:23:31.980 --> 00:23:37.560
So you can use IndieAuth login to password protect your data, all of these different things.

00:23:37.560 --> 00:23:44.560
And honestly, having a plugin system is so much fun because I can come up with a terrible idea for a feature and I can build it to plugin.

00:23:44.960 --> 00:23:53.060
And it doesn't matter that that's just an awful idea that nobody should ever have implemented because I'm not causing any harm to that core project.

00:23:53.060 --> 00:23:54.340
It's a super interesting idea.

00:23:54.340 --> 00:24:01.320
I also think it might be a way to encourage others to contribute because they don't have to understand the whole system and be afraid of breaking it.

00:24:01.320 --> 00:24:05.280
They just have to understand, like, here's the three functions I implement to make this happen.

00:24:05.280 --> 00:24:11.740
Real dream there is when people contribute to open source, that's more work for me because I have to review their pull requests and figure it out and so on.

00:24:11.740 --> 00:24:17.780
If you write a plugin, you can release that plugin to the Python package index and I don't even have to know about it.

00:24:17.780 --> 00:24:24.400
Like, I can wake up one day and my software has new features because somebody built a plugin and shipped it, which I think is fantastic.

00:24:24.400 --> 00:24:25.460
It's fantastic.

00:24:25.460 --> 00:24:29.820
And they don't have to go through you as a gatekeeper, even if you might be super friendly and whatnot.

00:24:29.820 --> 00:24:32.760
They just don't have to have that interaction, right, which is pretty cool.

00:24:32.760 --> 00:24:33.000
Yeah.

00:24:33.180 --> 00:24:41.600
So one of the things that's interesting about Dataset is the way you get your stuff online is you basically just run Dataset against a SQLite database.

00:24:41.600 --> 00:24:44.800
And now you have this website that lets you explore it like you described.

00:24:44.800 --> 00:24:50.000
So you say Dataset space, path to a SQLite file, and now you have a web app running, right?

00:24:50.000 --> 00:24:55.860
So you type Dataset, name a file, hit enter, and it runs on your local laptop and you can start browsing it and exploring it.

00:24:55.860 --> 00:25:00.780
But then if you want to put it online, I've been building out integrations with a bunch of different hosting providers

00:25:00.780 --> 00:25:10.760
where you can, from the command line type, Dataset space, publish space, pick your provider, say, say Google Cloud Run, Dataset publish Cloud Run, name of database, enter.

00:25:10.760 --> 00:25:17.240
And it'll upload that database to the internet, wrap it in the application itself, and give it a URL and start serving.

00:25:17.240 --> 00:25:22.520
So it's like a one-liner for publishing data online with a URL that other people can start using.

00:25:22.520 --> 00:25:28.120
And that's basically, that's enabled by all of these fascinating serverless hosting providers.

00:25:28.120 --> 00:25:32.420
And that was actually one of the original inspirations for Dataset.

00:25:32.420 --> 00:25:37.340
So I was looking at server hosting and saying, okay, what can I build with this that I couldn't have built beforehand?

00:25:37.340 --> 00:25:41.100
And the limitation of serverless host is that they generally don't let you run databases.

00:25:41.100 --> 00:25:46.040
They're like, you can run like a sort of stateless application that does stuff, it's fine.

00:25:46.220 --> 00:25:50.720
But if you need to store data in the database, you're going to have to pay extra or get an RDS.

00:25:50.720 --> 00:25:51.320
Right.

00:25:51.320 --> 00:26:00.040
The only real option you have is please subscribe to our managed database service of some sort, which turns you into a DB admin and all sorts of stuff you don't want to be.

00:26:00.040 --> 00:26:04.600
The realization I had is, hold on a second, that makes sense if you have to write to your database.

00:26:04.600 --> 00:26:17.300
But if it's read-only, if you're publishing like interesting data sets throughout the world, which aren't going to change, why not just stick that data in SQLite and then put that .db file in as an asset as part of your source code?

00:26:17.300 --> 00:26:19.600
Like basically bundle it in with the application.

00:26:19.600 --> 00:26:20.060
Right.

00:26:20.060 --> 00:26:24.080
It gets loaded in AWS Lambda or whatever, just like it's right there.

00:26:24.080 --> 00:26:24.260
Yeah.

00:26:24.260 --> 00:26:30.740
It's basically a giant loophole that lets you abuse serverless host providers, which are really cheap to serve interesting relational data.

00:26:31.040 --> 00:26:36.180
As long as you don't need to accept rights, it just works, which I think it's kind of devious and it works incredibly well.

00:26:36.180 --> 00:26:36.560
Yeah.

00:26:36.560 --> 00:26:37.540
It's really interesting.

00:26:37.540 --> 00:26:46.460
One of the things that I heard you say in a talk that we'll get to in a little bit is, you know, coming from Django, Django has the beautiful Django ORM, right?

00:26:46.460 --> 00:26:49.460
Which is a really nice way to talk to data and manage it.

00:26:49.460 --> 00:26:54.560
It's not so good for completely dynamic, unknown schemas, but it's good if you know it.

00:26:54.660 --> 00:27:01.660
One of the cool things about ORMs that I really like is it makes it very hard to do to write bad code that has SQL injection.

00:27:01.660 --> 00:27:02.300
Absolutely.

00:27:02.300 --> 00:27:02.880
Yeah.

00:27:02.880 --> 00:27:03.140
Right.

00:27:03.140 --> 00:27:05.540
And yet what you said to me is, here's the thing on the internet.

00:27:05.540 --> 00:27:09.920
I can go to it and then I can type in, edit the SQL directly.

00:27:09.920 --> 00:27:10.280
Yeah.

00:27:10.280 --> 00:27:12.300
This seems a little bobby tables all over the place.

00:27:12.300 --> 00:27:12.720
Totally.

00:27:12.720 --> 00:27:15.620
It's SQL injection as a core feature of the application.

00:27:16.000 --> 00:27:19.240
And the reason it's okay to do that is that it's read only.

00:27:19.240 --> 00:27:20.840
Like the data is read only.

00:27:20.840 --> 00:27:24.480
I open the SQLite database file in read only mode and so forth.

00:27:24.480 --> 00:27:30.960
And so the only damage you can cause by putting in an evil SQL query is you could try and write one that uses up too many resources.

00:27:30.960 --> 00:27:32.560
But I've got a time limit on those.

00:27:32.560 --> 00:27:36.820
So if your SQL query takes a second to execute, I'll cut it off and I'll return an error.

00:27:36.820 --> 00:27:42.380
And yeah, it means that SQL is the API language itself, which is kind of fascinating.

00:27:42.800 --> 00:27:51.120
People have been very excited about GraphQL over the past few years because it's a language that you can use to define exactly what you get back from an API.

00:27:51.120 --> 00:27:53.860
SQL has been doing that since the 1970s.

00:27:53.860 --> 00:28:01.160
So Dataset essentially gives you a JSON API that you can feed SQL queries to and get back the results in JSON.

00:28:01.160 --> 00:28:01.760
Yeah.

00:28:01.760 --> 00:28:02.700
It's really interesting.

00:28:02.700 --> 00:28:08.140
The other problem that you can run into a SQL injection is exposing private data, right?

00:28:08.140 --> 00:28:16.380
So if I want to say show, do a query that shows just my orders, but I'm allowed to get a or true into the query, I get all the orders, right?

00:28:16.380 --> 00:28:22.120
But the whole point of Dataset is to say we're going to create just an explorer of all the data.

00:28:22.120 --> 00:28:24.300
So there's no private data and it's read only.

00:28:24.300 --> 00:28:25.480
So you're good to go.

00:28:25.480 --> 00:28:25.880
Exactly.

00:28:25.880 --> 00:28:28.840
And so suddenly SQL injection becomes this really powerful feature.

00:28:29.080 --> 00:28:40.180
I've done some work where I've written JavaScript applications where the JavaScript, it creates a SQL query in the JavaScript and then it sticks it in the query string and runs it against Dataset and pulls back the results.

00:28:40.180 --> 00:28:43.260
And the first time I did that, it was genuinely meant as a joke.

00:28:43.260 --> 00:28:50.040
I was trying to troll my coworkers by saying, ha ha ha, look, I've got JavaScript that's injecting SQL statements into this thing.

00:28:50.840 --> 00:28:56.200
It turned out it took me a couple of hours to build quite a sophisticated internal search engine for our documentation.

00:28:56.200 --> 00:29:01.460
And it feels like running SQL in your JavaScript is a terrible idea.

00:29:01.460 --> 00:29:04.740
But actually, for prototyping, it's really good.

00:29:04.740 --> 00:29:11.740
Like you end up with about 20 lines of JavaScript that do something very sophisticated and let you build an interactive UI on the top of it.

00:29:11.740 --> 00:29:19.660
Yeah, it's a pretty simple way to expose a searchable sort of REST API, read only REST API on top of just a data schema.

00:29:19.860 --> 00:29:20.640
Exactly, yeah.

00:29:20.640 --> 00:29:24.340
Yeah, but you have the ability to edit.

00:29:24.340 --> 00:29:26.760
You go in there and you write some more SQL and you work with it.

00:29:26.760 --> 00:29:33.300
And then you can say, give me the URL for this result, which I think is also a pretty neat thing, like deep linking into it.

00:29:33.300 --> 00:29:38.000
The way I think about that is I've written applications, which is a SQL query.

00:29:38.000 --> 00:29:40.560
Like it's a SQL query that is a great example.

00:29:40.560 --> 00:29:46.060
I've got a database of all of my GitHub issues and issue comments across all of my projects.

00:29:46.060 --> 00:29:54.000
And I wrote a SQL query which finds me GitHub issues which were created by someone who's not me and when none of the replies are from me.

00:29:54.000 --> 00:29:57.520
So it's issues in my repos where I have not replied to them yet.

00:29:57.520 --> 00:29:59.840
You basically somehow missed them potentially.

00:29:59.840 --> 00:30:00.380
Exactly.

00:30:00.580 --> 00:30:08.260
And so I've got a query for that and now I've bookmarked it and now I've got an application which is the here are the GitHub issues that you should go and look at application.

00:30:08.620 --> 00:30:12.280
So an entire application ends up being a URL that you can bookmark.

00:30:12.280 --> 00:30:14.200
And that's really interesting.

00:30:14.200 --> 00:30:17.940
That's a really, again, a very web native way of thinking about the problem domain.

00:30:17.940 --> 00:30:22.560
You know, you were talking about starting out working in the first dot com boom.

00:30:22.560 --> 00:30:26.360
And one of the things that was all the rage back then were mashups.

00:30:26.360 --> 00:30:29.280
Do you remember like Yahoo mashups and all that kind of stuff?

00:30:29.280 --> 00:30:30.120
Absolutely.

00:30:30.120 --> 00:30:30.680
Yep.

00:30:32.080 --> 00:30:35.000
Talk Python To Me is partially supported by our training courses.

00:30:35.000 --> 00:30:39.740
Python's async and parallel programming support is highly underrated.

00:30:39.740 --> 00:30:47.140
Have you shied away from the amazing new async and await keywords because you've heard it's way too complicated or that it's just not worth the effort?

00:30:47.140 --> 00:30:52.740
With the right workloads, a hundred times speed up is totally possible with minor changes to your code.

00:30:53.120 --> 00:30:55.180
But you do need to understand the internals.

00:30:55.180 --> 00:31:03.360
And that's why our course, Async Techniques and Examples in Python, show you how to write async code successfully as well as how it works.

00:31:03.360 --> 00:31:09.280
Get started with async and await today with our course at talkpython.fm/async.

00:31:09.280 --> 00:31:15.940
So it sounds to me like what you can almost build here is like a super interesting mashup.

00:31:15.940 --> 00:31:18.040
So you can like extract the data from GitHub.

00:31:18.040 --> 00:31:21.580
You can extract it from over here and you put it together in this new form.

00:31:21.680 --> 00:31:25.040
And then now you've got this API on top of it, right?

00:31:25.040 --> 00:31:29.520
A massive realization I've had working on this stuff is lots of websites have APIs.

00:31:29.520 --> 00:31:33.760
And the APIs sometimes have a lot of features like the GitHub API can do some pretty powerful stuff.

00:31:33.760 --> 00:31:38.240
But I can always think of something the API can't do that they didn't predict I'd won.

00:31:38.240 --> 00:31:43.740
If I can get all of my data out of that API and into a SQLite database, then there are no limits.

00:31:43.740 --> 00:31:48.640
And any question I can think to ask, I can apply against that thing.

00:31:49.120 --> 00:31:53.840
So I've basically sort of given the only thing I use APIs for now is to give get everything.

00:31:53.840 --> 00:31:54.860
Just give everything.

00:31:54.860 --> 00:31:55.340
It's here.

00:31:55.340 --> 00:31:56.600
Download it and sync.

00:31:56.600 --> 00:31:57.900
Get everything into a database.

00:31:57.900 --> 00:32:00.820
And now I can start introspecting my data and building this.

00:32:00.820 --> 00:32:00.980
Yeah.

00:32:00.980 --> 00:32:05.460
I think where it gets interesting, as we'll see as we get on to like the final dog sheep side of things,

00:32:05.460 --> 00:32:07.760
is it's great that GitHub has an API.

00:32:07.760 --> 00:32:09.080
It's great that Twitter has an API.

00:32:09.080 --> 00:32:11.540
It's great that Gmail has IMAP.

00:32:11.540 --> 00:32:15.140
And all these different things have rich, deep ways to talk to them.

00:32:15.140 --> 00:32:22.440
But if you want to talk to all of it at the same time and say, I want to know about what I've tweeted, emailed, or whatever else I've done about it.

00:32:22.440 --> 00:32:26.180
Like, you don't want to try to build that integration of all those APIs.

00:32:26.180 --> 00:32:27.560
Then it gets super gnarly.

00:32:27.640 --> 00:32:32.060
But if you get it into some kind of SQLite database, all of a sudden it becomes an option.

00:32:32.060 --> 00:32:32.300
Right.

00:32:32.300 --> 00:32:34.580
It's this personal data warehouse idea.

00:32:34.580 --> 00:32:36.240
And it's not just a personal data.

00:32:36.240 --> 00:32:42.540
Like, as a company, if you're a company with 50 different Git repositories, which lots of companies have,

00:32:42.540 --> 00:32:46.020
getting all of that metadata from all 50 of those repos into one place,

00:32:46.020 --> 00:32:49.120
and I've got tooling that will let you do exactly that, is crazy useful.

00:32:49.400 --> 00:32:52.880
It lets you build stuff across all of your issues and all of your comments.

00:32:52.880 --> 00:32:57.520
It lets you talk about, like, here is what our software team as a company has accomplished,

00:32:57.520 --> 00:32:59.580
or how many, like, that kind of stuff, right?

00:32:59.580 --> 00:33:02.720
Which is still super hard, even if you go to GitHub and do that.

00:33:02.720 --> 00:33:06.780
So I want to talk through a few examples that maybe we could mention really quickly.

00:33:06.780 --> 00:33:13.980
You gave a talk that covered both Dataset and Dogsheep at PyCon AU online this year, right?

00:33:13.980 --> 00:33:14.960
Was it this year or last year?

00:33:14.960 --> 00:33:15.560
Yes.

00:33:15.560 --> 00:33:16.200
Yes, I did.

00:33:16.200 --> 00:33:17.460
It was this year.

00:33:17.460 --> 00:33:17.700
Yeah.

00:33:18.000 --> 00:33:20.440
So in there, you talked about all sorts of interesting things.

00:33:20.440 --> 00:33:21.840
So I want to cover some of the examples there,

00:33:21.840 --> 00:33:24.000
because they kind of, like, really made this stuff connect for me.

00:33:24.000 --> 00:33:24.280
Cool.

00:33:24.280 --> 00:33:24.640
Okay.

00:33:24.640 --> 00:33:25.080
Yeah.

00:33:25.080 --> 00:33:27.740
So the first one was, you said, let me just say,

00:33:27.740 --> 00:33:32.440
we just search for random SQLite databases on my Mac.

00:33:32.440 --> 00:33:35.940
And you said, oh, look, here, we randomly found one in the photos library.

00:33:35.940 --> 00:33:37.100
Let's look at that, right?

00:33:37.100 --> 00:33:39.600
You just pointed Dataset at that, like, search for it.

00:33:39.600 --> 00:33:42.760
What do you do, like a .db or something like that in Spotlight?

00:33:42.760 --> 00:33:44.900
Yeah, there's a Spotlight command that you can run,

00:33:44.900 --> 00:33:47.440
which will show you every SQLite database on your Mac,

00:33:47.660 --> 00:33:48.540
and it's fascinating.

00:33:48.540 --> 00:33:49.520
Oh, my goodness.

00:33:49.520 --> 00:33:53.640
The number of weird little databases that you already have,

00:33:53.640 --> 00:33:56.260
your Firefox history, your Chrome history on there,

00:33:56.260 --> 00:33:57.920
Evernote uses SQLite.

00:33:57.920 --> 00:34:01.720
There was quite a few databases I found where I still don't quite know what they are,

00:34:01.720 --> 00:34:05.240
but they've got things like places that I've been over the past couple of years.

00:34:05.240 --> 00:34:09.400
I just sat there in a SQLite database somewhere, which is super interesting.

00:34:09.400 --> 00:34:10.440
Yeah, for sure.

00:34:10.900 --> 00:34:16.540
So in this demo, what you did is you used that command to find the SQLite database back in your photos library,

00:34:16.540 --> 00:34:20.120
and then said, well, let's just pull that up and poke around, right?

00:34:20.120 --> 00:34:20.740
Tell us about that.

00:34:20.740 --> 00:34:27.020
Yeah, so the photos, this was always one of my sort of white whale was, I want my photos data.

00:34:27.020 --> 00:34:28.600
I've taken 40,000 photos.

00:34:28.600 --> 00:34:31.600
They've got timestamps and latitudes and longitudes and all of this.

00:34:31.600 --> 00:34:37.500
How can I get that metadata into a SQLite database so I can run queries against my life in photos?

00:34:37.980 --> 00:34:41.160
And I've tried getting this to work with things like Google Photos in the past.

00:34:41.160 --> 00:34:45.740
Google Photos doesn't give you access to latitudes and longitudes, I think, for privacy reasons.

00:34:45.740 --> 00:34:51.780
But anyway, the big realization I had was that the Apple Photos on your phone and on your laptop uses SQLite.

00:34:51.780 --> 00:34:52.660
There's a little SQLite.

00:34:52.660 --> 00:34:54.480
They've already done it for you.

00:34:54.480 --> 00:34:55.360
You just got to go find it.

00:34:55.360 --> 00:34:56.200
It's probably huge.

00:34:56.200 --> 00:34:56.660
Yeah.

00:34:56.660 --> 00:35:00.020
800 megabytes of data in one SQLite database file.

00:35:00.020 --> 00:35:05.100
And it's not very easy to query because it's not designed for people to use it from outside.

00:35:05.300 --> 00:35:08.900
But if you jump through a bunch of hoops, you can get that data back out again.

00:35:08.900 --> 00:35:11.600
And then you start finding some really interesting things.

00:35:11.600 --> 00:35:16.820
So obviously, they've got a record for each of your photos with when it was taken in the latitude and longitude.

00:35:16.820 --> 00:35:18.800
They've reversed geocoded those locations.

00:35:18.800 --> 00:35:21.380
They can actually tell you in the database, this was in San Francisco.

00:35:21.380 --> 00:35:23.520
This was in the Mission District, those kinds of things.

00:35:23.520 --> 00:35:29.180
But then the coolest thing is Apple use machine learning to identify what your photos are of.

00:35:29.180 --> 00:35:31.420
So is it a dog or a cat or a pelican?

00:35:31.420 --> 00:35:33.660
Right, because you can go to your photos app and search for that.

00:35:33.660 --> 00:35:37.360
But like, show me cars and like some insane way cars come up.

00:35:37.360 --> 00:35:37.740
Exactly.

00:35:37.740 --> 00:35:41.340
But the beautiful thing about that is it turns out they run the models on your laptop.

00:35:41.340 --> 00:35:45.860
Like Google and Facebook will upload your photos to the internet and put them in a data center somewhere.

00:35:45.860 --> 00:35:52.200
Apple downloads these big binary machine learning model, like weights files onto your device.

00:35:52.200 --> 00:35:53.780
They actually run it on your phone overnight.

00:35:53.780 --> 00:35:56.460
And they use those to identify what's in your photos.

00:35:56.780 --> 00:36:03.220
So from a privacy point of view, this is perfect because you're not uploading your photos somewhere for some creepy machine learning model to run against.

00:36:03.220 --> 00:36:05.500
It's all happening on devices that you control.

00:36:05.500 --> 00:36:08.120
And the results of that go in a SQLite database.

00:36:08.120 --> 00:36:09.400
So they go into data set.

00:36:09.400 --> 00:36:11.620
And yes, so I can get them out and get them into data set.

00:36:11.620 --> 00:36:12.520
So yeah, I have a data.

00:36:12.620 --> 00:36:20.320
I have an example query that shows me photographs I've taken of pelicans based on Apple's machine learning labeling, labeling my photos of pelicans.

00:36:20.320 --> 00:36:24.180
And I can visualize those on a map because they've got lattes and longitudes with them and so on.

00:36:24.520 --> 00:36:31.620
And then the really fun thing is there's this there are various clues in the photos app that they're doing quality evaluations.

00:36:31.620 --> 00:36:38.020
Like if they show you all of your photos for a month, they'll pick a good photo to show us the sort of title of that album or whatever.

00:36:38.020 --> 00:36:39.800
That's machine learning as well.

00:36:39.800 --> 00:36:42.580
It's running on your device and it's based on these scores.

00:36:42.580 --> 00:36:51.940
And the scores are sat there in the database and they have names like Z overall aesthetic score and Z pleasant camera tilt score and Z harmonious color score.

00:36:52.300 --> 00:37:00.960
So you can say things like show me my pelican photograph with the most harmonious colors, with the most pleasant camera tilts and just get things back that way.

00:37:00.960 --> 00:37:10.360
You could even set up like a walking tour of show me where I've taken aesthetic photos of pelicans, starting with the best one and then the next and then the next.

00:37:10.360 --> 00:37:11.660
That is such a good idea.

00:37:11.660 --> 00:37:13.660
Yeah, that's just a SQL query.

00:37:13.660 --> 00:37:14.540
Yeah, yeah, yeah.

00:37:14.540 --> 00:37:16.760
Order by aesthetic descending or something.

00:37:16.760 --> 00:37:17.240
Yeah.

00:37:17.240 --> 00:37:21.800
And there's also facial recognition, which again is trained by you and runs on your device.

00:37:21.900 --> 00:37:23.660
So it's the least creepy version of it.

00:37:23.660 --> 00:37:33.880
So I go on a SQL query saying, show me photographs of my wife, Natalie, and my friend, Andrew, and show me the one with the most pleasant camera tilt that was taken outdoors.

00:37:33.880 --> 00:37:35.480
And this stuff all just works.

00:37:35.480 --> 00:37:37.660
It's baffling and really super fun.

00:37:38.020 --> 00:37:49.920
So one piece that we probably should connect for folks, if they tried to follow along, they find that SQLite database and then they throw a dataset at it, they're going to end up with like binary blobs where this data lives, right?

00:37:49.920 --> 00:37:50.600
Oh, totally.

00:37:50.600 --> 00:37:50.900
Yeah.

00:37:50.980 --> 00:37:55.460
I think Apple's SQLite format uses like binary plists in some of the columns.

00:37:55.460 --> 00:38:02.900
And also, it's actually quite hard to even open it because it'll crash and tell you that you don't have that custom something extension running.

00:38:03.300 --> 00:38:16.560
So the way I've addressed that, I wrote this, I found this software on GitHub called OSX Photos, which is someone's open source library for talking to the OSX SQLite database and working around some of these weird issues.

00:38:16.560 --> 00:38:27.420
And then I built my own tool on top of that tool called Dog Sheep Photos, which basically both pulls out the metadata for your photos into a nicer format, including getting the machine learning labels and stuff.

00:38:27.420 --> 00:38:31.680
But it's also got a tool for uploading the photo files themselves to an S3 bucket.

00:38:31.680 --> 00:38:38.640
Because if you want to really take control of your photos, you need them to have URLs so that you can embed them in pages and link to them and so on.

00:38:38.640 --> 00:38:49.080
So I've got a whole tool chain for uploading all of my photographs to S3 and extracting the metadata into a separate database and then publishing that database with data sets somewhere so that I can run queries against it.

00:38:49.080 --> 00:38:49.420
Yeah.

00:38:49.420 --> 00:38:50.360
Yeah, super cool.

00:38:50.360 --> 00:38:50.900
All right.

00:38:50.900 --> 00:38:54.600
Well, I think that probably is a good transition over to Data Sheep.

00:38:54.600 --> 00:38:57.060
So we have these different sources of data.

00:38:57.060 --> 00:38:59.960
Like it's nearly unbounded at this point, right?

00:38:59.960 --> 00:39:03.340
Of where data on the internet might live about that could be for you.

00:39:03.340 --> 00:39:08.840
There was this firebrand of a character that you mentioned around Wolfram Alpha.

00:39:08.840 --> 00:39:13.600
And he does some crazy, crazy, weird stuff.

00:39:13.600 --> 00:39:19.040
But he also had this idea that inspired you to sort of try to bring those together and build it on top of the data set.

00:39:19.040 --> 00:39:22.640
So like maybe start it with that story and we can tell people what Dog Sheep is.

00:39:22.640 --> 00:39:22.960
Okay.

00:39:22.960 --> 00:39:29.500
So there's this chap called Stephen Wolfram who is, he created Mathematica and Wolfram Alpha.

00:39:29.960 --> 00:39:38.220
And he's the CEO of a thousand person company, which it turns out he's a remote CEO, runs the entire company from home, which is kind of fascinating.

00:39:38.220 --> 00:39:38.700
Yeah.

00:39:38.700 --> 00:39:39.960
And he has been for a while, right?

00:39:39.960 --> 00:39:41.060
Like before it was cool.

00:39:41.060 --> 00:39:41.520
Yeah.

00:39:41.520 --> 00:39:42.180
No, absolutely.

00:39:42.180 --> 00:39:44.840
He's been doing the COVID thing for years and years and years.

00:39:44.840 --> 00:39:52.260
And so in February of last year, he published an essay called Seeking the Productive Life, Some Details of My Personal Infrastructure.

00:39:52.760 --> 00:39:57.720
And this thing, I would thoroughly recommend everyone take a look just to marvel at quite how long it is.

00:39:57.720 --> 00:40:03.140
He has spent 40 years optimizing every single inch of his personal and professional life.

00:40:03.140 --> 00:40:04.940
And he wrote about all of it in one place.

00:40:04.940 --> 00:40:09.740
Like he scanned every document he's worked on since he was 11 years old and got them OCR'd.

00:40:09.740 --> 00:40:13.100
And he's got a green screen in his basement for giving remote talks.

00:40:13.260 --> 00:40:18.640
And his heart rate monitor showed him that his heart rate is better when he's – so he had a standing desk.

00:40:18.640 --> 00:40:21.700
But his heart rate monitor showed him that walking outside is better for his heart.

00:40:21.700 --> 00:40:26.680
So he rigged up a sort of little tray mechanism so he could use his laptop while walking in the woods.

00:40:26.680 --> 00:40:28.280
And it's just astonishing.

00:40:28.800 --> 00:40:32.300
And I read through this essay thinking this is a lot.

00:40:32.300 --> 00:40:34.400
Like this is not –

00:40:34.400 --> 00:40:35.380
This is next level.

00:40:35.380 --> 00:40:35.820
Right.

00:40:35.820 --> 00:40:36.960
Totally next level.

00:40:36.960 --> 00:40:40.060
But there was this one little bit in it that caught my eye.

00:40:40.060 --> 00:40:44.600
He talked about how he has a personal search engine, something he calls his meta searcher.

00:40:44.600 --> 00:40:56.900
And so he's got his own private search engine that searches everything, every email he's sent, every paper he's written, everyone he knows who might know things about it, everything he's read, all of the files on his machine, all in one place.

00:40:57.220 --> 00:40:58.880
And I thought, well, that's something I'd like.

00:40:58.880 --> 00:41:05.280
Like I would love to have one place with as much of my personal data from different sources in one place where I can query it.

00:41:05.280 --> 00:41:09.140
Like I know I was talking to this person, but was it on iMessage?

00:41:09.140 --> 00:41:10.080
Was it an email?

00:41:10.080 --> 00:41:11.260
Was it over Slack?

00:41:11.260 --> 00:41:14.580
Like where the heck did I tell them this thing that I need to get back?

00:41:14.580 --> 00:41:15.300
Absolutely.

00:41:15.300 --> 00:41:20.440
And combine that with, you know, your bookmarks and your GitHub issues and, yeah, messages and all of these –

00:41:20.440 --> 00:41:20.800
Your tweets.

00:41:20.800 --> 00:41:22.660
Oh, yeah.

00:41:22.660 --> 00:41:24.340
And it felt like there was something interesting there.

00:41:24.700 --> 00:41:33.120
And so I then – I'll be honest, the best idea I've had in all of this is I thought, well, it's inspired by Stephen Wolfram, but it's not as good as what he's done.

00:41:33.120 --> 00:41:40.540
So if he's Wolfram, maybe I should be doing something called Dog Sheep, like as the less alpha versions of those animals.

00:41:40.540 --> 00:41:44.160
And then I thought, well, he's got a search engine called Wolfram Alpha.

00:41:44.520 --> 00:41:46.760
I could build a search engine called Dog Sheep Beta.

00:41:46.760 --> 00:41:49.760
And that joke stuck in my head.

00:41:49.760 --> 00:41:52.940
And I enjoyed it so much that I spent like 12 months on and off thinking with –

00:41:52.940 --> 00:41:54.260
Yeah, I have to build this.

00:41:54.260 --> 00:41:56.140
It must exist now that it's so good.

00:41:56.140 --> 00:41:57.080
I have to put it.

00:41:57.080 --> 00:41:57.800
Yeah.

00:41:58.200 --> 00:42:01.160
So this entire project is basically – it's pun-driven development.

00:42:01.160 --> 00:42:04.400
It's driven out of this pun that I came up with a year and a half ago.

00:42:04.400 --> 00:42:06.060
And, yeah, so that's what I've been building.

00:42:06.060 --> 00:42:13.360
And so the idea with Dog Sheep, it's basically an umbrella project for a whole bunch of tools around this idea of personal analytics.

00:42:13.360 --> 00:42:16.220
Like what data is there about me in the world?

00:42:16.440 --> 00:42:20.580
How can I get that data out of lots of different sources and get it into SQLite databases?

00:42:20.580 --> 00:42:23.760
Because once it's in SQLite, I can run data set on top of it.

00:42:23.760 --> 00:42:28.380
And now I've got this personal data warehouse of my data from all of these different sources.

00:42:28.380 --> 00:42:33.520
And then on top of that, I can build a search engine, which I've now built, which ties all of this stuff together again.

00:42:33.520 --> 00:42:40.060
And so, yeah, so I've been tinkering around with all sorts of tools in this category for, yeah, just over a year now.

00:42:40.060 --> 00:42:45.400
And I've got – so right now I've got data in my personal Dog Sheep from Twitter.

00:42:45.680 --> 00:42:48.780
I've got all of my tweets, but also all of the tweets that I've favorited.

00:42:48.780 --> 00:42:54.800
And I've favorited like 30,000 tweets, but I can search those and see who I've favorited the most from and so on.

00:42:54.800 --> 00:42:57.420
I've got all of my photos, as we discussed earlier.

00:42:57.420 --> 00:43:03.760
I've got my health kit data from my Apple Watch, which means I can tell you my heart rate going back the last three years or something.

00:43:03.760 --> 00:43:05.540
How do you get it off the Apple Watch?

00:43:05.540 --> 00:43:08.120
So, again, Apple are really good for this kind of stuff.

00:43:08.120 --> 00:43:10.940
They don't publish it to the – they don't upload it somewhere.

00:43:10.940 --> 00:43:13.100
They keep it on your phone and on your watch.

00:43:13.100 --> 00:43:14.440
But there's an export button.

00:43:14.440 --> 00:43:18.560
In the health app on the iPhone, there's a button that says export my data.

00:43:18.560 --> 00:43:25.560
And it actually creates a zip file full of XML on your phone and then gives you the option to like airdrop it to your laptop.

00:43:25.560 --> 00:43:29.220
So I do that and I get a 300 megabyte zip file full of XML.

00:43:29.220 --> 00:43:32.840
And then I write a script which reads that XML in terms of SQLite.

00:43:32.840 --> 00:43:34.300
And so, yeah, so I've got all of that data.

00:43:34.380 --> 00:43:43.020
The best thing about that is anytime you record an outdoor workout, like if you go for a run or even if you go for a walk, it records your latitude and longitude every few seconds.

00:43:43.020 --> 00:43:44.500
And that's available in the data.

00:43:44.500 --> 00:43:50.340
So I've got – so, like, within 10-meter maps of every walk I've taken for the past year, which is super fun.

00:43:50.620 --> 00:43:51.260
I mentioned GitHub.

00:43:51.260 --> 00:43:54.220
I've got all of the data from all of my GitHub projects.

00:43:54.220 --> 00:43:55.860
I've got over 400 repositories now.

00:43:55.860 --> 00:43:57.660
So that's actually quite a lot of stuff.

00:43:57.660 --> 00:44:00.660
I use Foursquare Swarm and check-in to places.

00:44:00.660 --> 00:44:02.740
So I've got 4,000 swarm check-ins.

00:44:03.080 --> 00:44:05.440
Oh, you have Google Takeout, which is insane.

00:44:05.440 --> 00:44:05.900
Right.

00:44:05.900 --> 00:44:08.120
I've only done a little bit of work with Google Takeout.

00:44:08.120 --> 00:44:09.580
That's one of the least developed tools.

00:44:09.580 --> 00:44:19.840
But, yeah, you can get Google's version of your location history, which is – like, for me, it's like 250,000 latitude, longitude points that they've – I don't even know where they got that stuff from.

00:44:19.840 --> 00:44:20.080
Yeah.

00:44:20.080 --> 00:44:23.500
I recently did a Google Takeout, and I think it was probably zipped.

00:44:23.500 --> 00:44:26.000
I think it was 61 gigabytes or something.

00:44:26.000 --> 00:44:27.240
I mean, it's a lot of data.

00:44:27.240 --> 00:44:28.080
It's a lot of data.

00:44:28.080 --> 00:44:31.580
And a lot of that is, like, photographs and document files and stuff.

00:44:31.840 --> 00:44:35.940
But there's just a ton of very detailed JSON data about you as well.

00:44:35.940 --> 00:44:47.320
With those exports, it's always fun to look for the ad targeting stuff because you'll find out that you have been assigned the role of a, like, middle-aged tech executive or something.

00:44:47.320 --> 00:44:49.900
And you can see what they're targeting ads are actually based on.

00:44:49.900 --> 00:44:51.220
I've got Evernote.

00:44:51.220 --> 00:44:53.400
I've got, like, 600 notes from Evernote.

00:44:53.400 --> 00:44:56.740
My Goodreads data on books that I've read, which is synced from my Kindle.

00:44:56.740 --> 00:45:00.000
Oh, and then the most fun one is I've got a copy of my genome.

00:45:00.600 --> 00:45:02.820
Because I did 23andMe a few years ago.

00:45:02.820 --> 00:45:04.680
And I found out they've got an export button.

00:45:04.680 --> 00:45:12.420
And you can get back a CSV file of 600,000, like, gene pairs from your genome, which I can now run SQL queries against.

00:45:12.420 --> 00:45:18.900
So I have a SQL query that tells me what color my eyes are based on interrogating my own copy of my genome, which delights me.

00:45:18.900 --> 00:45:19.900
That is pretty amazing.

00:45:19.900 --> 00:45:21.340
That's just insane.

00:45:21.760 --> 00:45:23.400
Yeah, so that's a ton of data, right?

00:45:23.400 --> 00:45:24.300
This is a lot of stuff.

00:45:24.300 --> 00:45:27.900
And I'm barely even scratching the surface of what I could be doing into this system.

00:45:27.900 --> 00:45:28.080
Right.

00:45:28.080 --> 00:45:30.240
Well, and these are all, like, plugins or other types.

00:45:30.240 --> 00:45:32.260
So it's wherever the data lives.

00:45:32.260 --> 00:45:34.680
If there's an API or web scraping, you can have it.

00:45:34.680 --> 00:45:34.920
Right.

00:45:35.040 --> 00:45:43.080
And it's some, the tools are all called things like Twitter to SQLite or GitHub to SQLite or I think I've got genome to SQLite somewhere.

00:45:43.080 --> 00:45:45.320
That's just the naming convention that I use.

00:45:45.560 --> 00:45:53.420
But, yeah, the core idea is you knock out a quick Python command line tool, which either takes a zip file you got from somewhere or hits an API with API credentials.

00:45:53.420 --> 00:45:58.260
And it slurps down as much data as it can, wallops it in a SQLite database, and that's all it does.

00:45:58.260 --> 00:46:04.160
And then it's up to you to run data set against it and start doing the, like, doing fun querying some things.

00:46:04.160 --> 00:46:04.700
That's cool.

00:46:04.800 --> 00:46:20.060
So I think maybe it would be good to connect this, maybe tell a story or an example, again, what you did at PyCon and you talk about your dog and figuring out using Twitter to graph the weight of your dog over time and Twitter to map where your dog likes to go on walks.

00:46:20.060 --> 00:46:20.680
Absolutely.

00:46:20.680 --> 00:46:21.820
So my dog is called Cleo.

00:46:21.820 --> 00:46:24.380
First of all, tell me how dog and Twitter go together.

00:46:24.380 --> 00:46:26.400
Not dog sheep, but just like your dog.

00:46:26.400 --> 00:46:32.960
So Cleo has a Twitter account because, I mean, to be honest, most dogs have Instagram these days, but Cleo is a bit more old fashioned.

00:46:32.960 --> 00:46:33.900
So Cleo is on Twitter.

00:46:34.160 --> 00:46:36.760
She's more on the tech side, less on the young influencer side.

00:46:36.760 --> 00:46:37.260
Yeah, exactly.

00:46:37.260 --> 00:46:37.780
Got it.

00:46:37.780 --> 00:46:41.540
Cleo, C-L-E-O-P-A-W-S.

00:46:41.540 --> 00:46:42.600
Cleo pause on Twitter.

00:46:42.600 --> 00:46:44.160
And she tweets about things.

00:46:44.160 --> 00:46:47.200
She tweets selfies and like things that she likes and so on.

00:46:47.200 --> 00:46:53.660
And every time we go to the vet, she tweets a selfie of herself at the vet and they weigh her and she tweets how much she weighs.

00:46:53.660 --> 00:46:56.280
She tweets, I weigh 49.3 pounds.

00:46:56.280 --> 00:46:57.260
I grew more dog.

00:46:57.260 --> 00:46:58.240
And there's a selfie.

00:46:58.240 --> 00:47:02.360
And one of the things I've done with dog sheep is I've imported all of her tweets.

00:47:02.820 --> 00:47:08.800
And so now I can run a SQL query that just pulls up the tweets where it contains LB for pounds and or weigh.

00:47:08.800 --> 00:47:12.060
So I can pull back just the tweets where she said how much she weighs.

00:47:12.060 --> 00:47:20.240
And then I've got a regular expression extension for data that adds a custom SQL function that can do regular expressions because that's a useful thing to have.

00:47:20.300 --> 00:47:23.880
So I can pull out her weight with a regular expression into a separate column.

00:47:23.880 --> 00:47:27.580
And then I've got a charting plugin that can chart something against something.

00:47:27.580 --> 00:47:38.520
So I can chart date against weight and see a chart of how much she weighs based on her self-reported weight in the selfies that she's posted on Twitter, which is clearly a...

00:47:38.520 --> 00:47:39.760
That's a killer act right there.

00:47:39.760 --> 00:47:41.660
That is a useful thing.

00:47:41.760 --> 00:47:43.880
It's absolutely frivolous.

00:47:43.880 --> 00:47:49.960
But what I think it shows is so interesting is what you can do if you can put these pieces of data together, right?

00:47:49.960 --> 00:48:03.980
If all of a sudden I can just do arbitrary queries and apply some special filtering, but to these other data sources you never expected to be able to, all of a sudden you have a graph that you never knew you were keeping about the weight of your dog or some other thing you're interested in.

00:48:03.980 --> 00:48:05.400
This is just a SQL query.

00:48:05.400 --> 00:48:07.420
The SQL query goes in a bookmark.

00:48:07.420 --> 00:48:12.560
So the entire application, the show me a chart of my dog's weight based on her selfies is a bookmark.

00:48:12.560 --> 00:48:14.800
It's just a bookmark that I've got.

00:48:14.800 --> 00:48:17.120
It's actually super useful.

00:48:17.120 --> 00:48:18.360
Super cool.

00:48:18.360 --> 00:48:21.460
Now, how do you get a map of where your dog is from Twitter?

00:48:21.460 --> 00:48:24.840
So that one, I mentioned I use Foursquare Swarm and I check in places.

00:48:24.840 --> 00:48:30.300
Every time the dog's with me, I use the wolf emoji in the check-in message, which looks a little bit like her.

00:48:30.300 --> 00:48:33.020
And it turns out SQL does emoji these days.

00:48:33.120 --> 00:48:38.200
So you can run a SQL query where you look for things like percentage wolf emoji percentage.

00:48:38.200 --> 00:48:38.720
Right.

00:48:38.720 --> 00:48:40.160
It's just a character that's unique.

00:48:40.160 --> 00:48:40.680
Exactly.

00:48:40.680 --> 00:48:44.240
So then you get back the check-ins where my dog was there.

00:48:44.240 --> 00:48:48.020
And then because I've got a latitude and longitude in that query, I get them on a map.

00:48:48.020 --> 00:48:54.220
So I've got a map of places my dog likes to go based on the wolf emoji in my swarm check-ins.

00:48:54.220 --> 00:48:55.960
And again, it's just a bookmark.

00:48:55.960 --> 00:48:58.360
It's these custom applications that's a bookmark.

00:48:58.360 --> 00:49:04.160
I think those two examples really bring home the unexpected power of what you kind of unleash when you get this stuff.

00:49:04.160 --> 00:49:04.580
Completely.

00:49:04.580 --> 00:49:07.520
There's a project I should mention that relates to this.

00:49:07.520 --> 00:49:10.400
So I've been writing a lot of these tools that create SQLite databases.

00:49:10.400 --> 00:49:14.360
All of these dog sheep tools pull something from somewhere and turn it into SQLite.

00:49:14.880 --> 00:49:20.100
And the way I do that is using a Python library that I've been building called SQLite hyphen utils.

00:49:20.100 --> 00:49:26.420
SQLite utils is a bunch of utility functions to make it really productive to create new SQLite databases.

00:49:26.760 --> 00:49:30.040
So the core idea is, say you've got array of JSON objects.

00:49:30.040 --> 00:49:33.800
You can say dot insert bracket array of JSON objects.

00:49:33.800 --> 00:49:38.720
And it will create a SQLite table with the schema that's needed to match those JSON objects.

00:49:38.720 --> 00:49:39.180
Wow.

00:49:39.180 --> 00:49:41.540
So it just looks and says, these are the top level keys.

00:49:41.540 --> 00:49:43.400
So we're going to make those columns, something like that.

00:49:43.400 --> 00:49:44.200
These ones are integers.

00:49:44.200 --> 00:49:44.820
These ones float.

00:49:44.820 --> 00:49:45.740
This is a text.

00:49:45.740 --> 00:49:47.840
And it creates the table automatically.

00:49:48.060 --> 00:49:53.800
Which means if you're working with an API that's well designed, like the GitHub API returns lists of JSON objects.

00:49:53.800 --> 00:49:58.560
So it's a Python one-liner to turn those into a SQLite table with the correct columns.

00:49:58.560 --> 00:50:03.880
And you can say, oh, and make the ID one the primary key and set these up as foreign keys and those kinds of things.

00:50:03.880 --> 00:50:12.440
And that's been crucial because it means that I didn't have to come up with a database schema for Swarm and Twitter and GitHub and Apple Photos and all of that.

00:50:12.440 --> 00:50:16.660
I just had to get the data into a list of objects and the schema was created for me.

00:50:16.840 --> 00:50:19.260
No, it's got a little bit of a NoSQL feel in SQL.

00:50:19.260 --> 00:50:20.020
Exactly.

00:50:20.020 --> 00:50:22.760
And SQLite, it turns out, can deal with JSON as well.

00:50:22.760 --> 00:50:25.900
So you can stick a JSON document in a SQLite column.

00:50:25.900 --> 00:50:30.660
And then there are SQLite functions for pulling out the first key of it and that kind of thing.

00:50:30.660 --> 00:50:32.760
But yeah, it means it's all super productive.

00:50:32.760 --> 00:50:36.300
SQLite utils also comes with a command line tool.

00:50:36.300 --> 00:50:39.240
So for simple things, you don't have to write any Python at all.

00:50:39.240 --> 00:50:45.880
You can W get a JSON blob, pipe it into the SQLite utils command line tool and tell it to insert it into a table.

00:50:46.200 --> 00:50:49.220
And it will create a database file on disk and populate the table.

00:50:49.220 --> 00:50:55.500
And then you can do stuff like configure full text search against it or set up extra foreign keys or whatever it is.

00:50:55.500 --> 00:50:56.360
Oh, that's super neat.

00:50:56.360 --> 00:51:06.080
So all of these different integrations that you built, it sounds to me like they could potentially just be useful for people listening and go, you know, I'd really love to get four square swarm data as a SQLite database.

00:51:06.080 --> 00:51:07.920
And they don't necessarily want to use dogsheet.

00:51:07.920 --> 00:51:11.440
Like it sounds like these plug in pieces might be cool building blocks.

00:51:11.440 --> 00:51:17.480
If you can get yourself a OAuth token for your swarm account, which I've got an online tool that will do that for you.

00:51:17.480 --> 00:51:25.160
You pip install swarm to SQLite and you type swarm hyphen to hyphen SQLite swarm dot DB --token equals that and hit enter.

00:51:25.160 --> 00:51:25.940
And that's it.

00:51:25.940 --> 00:51:27.900
That's it's like a one line on the terminal.

00:51:27.900 --> 00:51:31.480
And that will give you a SQLite database with all of your swarm check ins in it.

00:51:31.480 --> 00:51:31.840
Wow.

00:51:32.140 --> 00:51:37.320
I'm pretty fascinated by the idea of like I could go to one place and just search everything about me.

00:51:37.320 --> 00:51:37.820
Absolutely.

00:51:37.820 --> 00:51:42.500
So that feature, because I had this stuff in like hundreds, dozens of different databases and tables.

00:51:42.500 --> 00:51:47.140
I actually built the dog sheep beta search engine just a couple of months ago.

00:51:47.140 --> 00:51:52.320
And basically the way that works is you give it SQL queries to run against all of your other tables.

00:51:52.320 --> 00:51:57.360
So you say for the GitHub one, select title from issues and create a date and so forth.

00:51:57.360 --> 00:51:59.200
For the Twitter one, select this, select that.

00:51:59.560 --> 00:52:05.440
And you run a script and it will run those SQL queries against all of your like 20 different databases,

00:52:05.440 --> 00:52:09.660
load the results into a new database table and set up full text search on it.

00:52:09.660 --> 00:52:16.120
So it's kind of like using something like elastic search where you have to query your data from lots of different sources into one index.

00:52:16.120 --> 00:52:18.700
And in this case, the index is just another SQLite table.

00:52:18.700 --> 00:52:24.960
And then that gives you a fasted search index on the top that lets you search across all of the different things that you've ingested into it.

00:52:24.960 --> 00:52:25.200
Right.

00:52:25.200 --> 00:52:26.920
If you do an index, it'll be nice and fast.

00:52:26.960 --> 00:52:30.400
And then you just say, well, you've got to go back to these five tables and get these various things.

00:52:30.400 --> 00:52:30.620
Right.

00:52:30.620 --> 00:52:30.840
Yeah.

00:52:30.840 --> 00:52:42.980
And that's actually part of the tool is you can then set up a SQL query for each type of content that says, oh, and to display it, run this SQL query to grab these details, stick them in this Jinja template and stick that on the page.

00:52:43.480 --> 00:52:47.100
So when you display the results, it can use all of the rich data that's come back.

00:52:47.100 --> 00:52:51.500
But the actual index underneath it is basically title content and the date and that's it.

00:52:51.500 --> 00:52:51.780
Yeah.

00:52:51.780 --> 00:52:52.280
Wow.

00:52:52.280 --> 00:52:52.920
Okay.

00:52:52.920 --> 00:52:53.620
Pretty interesting.

00:52:53.620 --> 00:52:54.540
What about email?

00:52:54.540 --> 00:52:58.480
I don't see in this list like connect to email, although there's Google takeout.

00:52:58.480 --> 00:53:00.600
That's not exactly necessarily the same.

00:53:00.600 --> 00:53:10.520
I will admit I have not done email yet because I am terrible at email and I'm almost a little terrified what will happen if I start running SQL queries against 10 years of mostly unread emails.

00:53:10.520 --> 00:53:14.120
But I'm sort of transitioning into doing freelancing and consulting.

00:53:14.120 --> 00:53:17.700
One of the most important aspects for consultants is that they're on top of that email.

00:53:17.820 --> 00:53:25.080
So I think that's probably the next task that I'll have to take on is getting good at email and then ingesting that into the dog sheepers.

00:53:25.080 --> 00:53:25.400
Yeah.

00:53:25.400 --> 00:53:28.080
I mean, it's kind of like this open plugin type of architecture.

00:53:28.080 --> 00:53:31.880
So someone else could create it as well if they're listening and they just want it to exist, right?

00:53:31.880 --> 00:53:32.300
Absolutely.

00:53:32.300 --> 00:53:40.680
I mean, honestly, like, you know, the email standards are good enough now that writing a tool that turns your email archive into a SQLite database is pretty trivial.

00:53:40.680 --> 00:53:43.760
Apple mail.app uses SQLite anyway.

00:53:43.760 --> 00:53:47.040
So I've actually done a little bit of poking around just looking at that database.

00:53:47.120 --> 00:53:47.520
Okay.

00:53:47.520 --> 00:53:48.040
Interesting.

00:53:48.040 --> 00:53:48.360
Yeah.

00:53:48.360 --> 00:53:49.640
I just haven't got to it.

00:53:49.640 --> 00:53:55.580
You could probably point it at an Outlook PST file if, you know, the world has cursed you and like you have to work in Outlook.

00:53:55.580 --> 00:53:58.900
Or you just use IMAP or POP3 or something like that, right?

00:53:58.900 --> 00:54:03.320
There's a very solid Python library for reading Outlook mailboxes.

00:54:03.320 --> 00:54:04.720
So you could totally use that.

00:54:04.720 --> 00:54:05.140
Okay.

00:54:05.140 --> 00:54:11.220
And then it would be a very thin layer of code on top to turn that into a list of JSON objects and pipe them into the SQLite library.

00:54:11.220 --> 00:54:11.700
Yeah.

00:54:11.700 --> 00:54:12.280
All right.

00:54:12.280 --> 00:54:13.320
Super cool, Simon.

00:54:13.320 --> 00:54:16.420
This is like a bunch of levels building on top of each other.

00:54:16.420 --> 00:54:16.820
I like it.

00:54:16.820 --> 00:54:19.440
And also thank you for the history of Django.

00:54:19.440 --> 00:54:23.940
It was really cool to hear how you experienced as it came to existence.

00:54:23.940 --> 00:54:24.480
That was neat.

00:54:24.480 --> 00:54:24.720
Cool.

00:54:24.720 --> 00:54:25.140
Yeah.

00:54:25.140 --> 00:54:28.000
Could talk about a bunch more, but I don't want to take all of your time.

00:54:28.000 --> 00:54:30.880
So let me just ask you the final two questions here.

00:54:30.880 --> 00:54:31.700
I always ask.

00:54:31.700 --> 00:54:34.900
So if you're going to write some Python code, what editor are you using these days?

00:54:34.900 --> 00:54:36.740
And these days I'm all about VS Code.

00:54:36.740 --> 00:54:43.340
Especially the most recent version of their Python integration, which is controversial because it's the one bit of it that's not open source.

00:54:43.340 --> 00:54:45.620
But that thing is just miraculous.

00:54:45.620 --> 00:54:46.120
Nice.

00:54:46.120 --> 00:54:46.860
PyLance, right?

00:54:46.860 --> 00:54:47.440
I think so.

00:54:47.440 --> 00:54:47.640
Yeah.

00:54:47.640 --> 00:54:52.720
It's showing me, hey, this variable hasn't been used yet and this import wasn't working and all of that kind of stuff.

00:54:52.720 --> 00:54:54.420
See, I'm all into VS Code now.

00:54:54.600 --> 00:54:55.080
Yeah, okay.

00:54:55.080 --> 00:54:58.760
That's definitely one that seems to be coming along and catching a lot of traction.

00:54:58.760 --> 00:54:59.420
Cool, cool.

00:54:59.420 --> 00:55:01.000
Notable PyPI package.

00:55:01.000 --> 00:55:03.180
Something you've run across that you're like, oh, this thing is cool.

00:55:03.180 --> 00:55:04.280
You should really know about it.

00:55:04.280 --> 00:55:06.680
This is great because I get to answer this using dog sheep.

00:55:06.680 --> 00:55:11.580
Because I've got all of my starred GitHub repos are pulled into my database, into dog sheep.

00:55:11.580 --> 00:55:17.440
And I can actually run a dog sheep, a beta search and say, show me everything I've starred or sorted by most recent.

00:55:17.980 --> 00:55:22.720
So the most recent Python one I started, I just started Astor, A-S-T-O-R.

00:55:22.720 --> 00:55:23.340
I've heard of that.

00:55:23.340 --> 00:55:24.240
I forgot what it is, though.

00:55:24.240 --> 00:55:27.520
Yeah, it works with the Python abstract syntax tree.

00:55:27.520 --> 00:55:30.000
So it's for building software on top of Python.

00:55:30.000 --> 00:55:39.680
And the reason I found it is that I found this tool called Flint, which re-does all of your .format calls and turns them into Python 3.6 format strings.

00:55:39.680 --> 00:55:40.280
Yes.

00:55:40.280 --> 00:55:41.800
And I was like, oh, how does that work?

00:55:41.800 --> 00:55:43.500
And it turns out it uses Astor under the hood.

00:55:43.500 --> 00:55:45.900
I was actually going to give a shout out to Flint.

00:55:46.040 --> 00:55:53.260
The reason was on Dataset, I just saw the last commit at the time of the recording was use f-strings in place of format.

00:55:53.260 --> 00:55:59.120
And you can just point Flint at the entire, just the top level directory and it just fixes it.

00:55:59.120 --> 00:56:00.020
That's why I started.

00:56:00.020 --> 00:56:01.140
I was playing around with that.

00:56:01.140 --> 00:56:06.260
And then the other, I mean, the other one, it's not a recent favorite, but I'm going to promote HTTPX.

00:56:06.260 --> 00:56:06.860
Oh, yeah.

00:56:06.860 --> 00:56:08.160
I'm a big fan of that one as well.

00:56:08.160 --> 00:56:10.180
You talked about the ASCII side.

00:56:10.180 --> 00:56:13.940
This is like consuming services from inside an ASGI server.

00:56:14.060 --> 00:56:14.180
Yeah.

00:56:14.180 --> 00:56:16.520
One way to look at it, it's the new requests, right?

00:56:16.520 --> 00:56:25.080
It's essentially, it was almost, I think, called requests three at one point in its history, but it's basically the modern version of requests with full async support.

00:56:25.080 --> 00:56:27.980
So you can use it synchronously and you can use it asynchronously.

00:56:28.180 --> 00:56:39.040
But the killer feature from my point of view is you can instantiate a HTTPX client and you can point it at a Python ASCII or WSGI object and then start running requests through it.

00:56:39.040 --> 00:56:40.640
So it's an amazing test harness.

00:56:40.640 --> 00:56:41.260
Oh, yeah.

00:56:41.260 --> 00:56:47.000
All of the dataset tests and every dataset plugin that I've written, they're actually, they're using HTTPX.

00:56:47.000 --> 00:56:49.400
So it's like being able to do HTTP testing.

00:56:49.400 --> 00:56:51.900
You don't even have to spin up a local host server.

00:56:51.900 --> 00:56:53.480
It's all happening in memory.

00:56:53.480 --> 00:56:54.860
And that's just extraordinary.

00:56:54.860 --> 00:56:56.260
Oh, that sounds fantastic.

00:56:56.380 --> 00:56:59.620
Like that's such a good product of writing unit tests against things.

00:56:59.620 --> 00:57:00.560
Yeah, I've never tried it.

00:57:00.560 --> 00:57:05.540
But a lot of times you can create, you'd like install the web test and then wrap it in a test framework.

00:57:05.540 --> 00:57:10.140
Or, you know, maybe you've got to even run it, like fire it up and run it and then talk to it.

00:57:10.140 --> 00:57:14.440
But this is just like connect the two pieces in code and skip the network and go.

00:57:14.620 --> 00:57:17.780
And I've got a really nerdy thing that I've just started doing with it.

00:57:17.780 --> 00:57:22.340
So dataset has plugins and dataset plugins can do a bunch of different things.

00:57:22.340 --> 00:57:25.040
But I realized that dataset itself is an API.

00:57:25.040 --> 00:57:31.240
The whole point is that it gives you a JSON API that you can use to interrogate your tables and run queries and so on.

00:57:31.240 --> 00:57:34.080
And I wanted my plugins to be able to use that API.

00:57:34.080 --> 00:57:39.380
But I didn't really want them to be making outbound HTTP requests against themselves and so on.

00:57:39.380 --> 00:57:45.860
So I just a couple of weeks ago added a feature to dataset where it gives you an internal call.

00:57:45.860 --> 00:57:48.720
They can say client.get brackets and then feed it a URL.

00:57:48.720 --> 00:57:51.940
And that's actually using HTTPX and ASCII under the hood.

00:57:51.940 --> 00:57:59.400
So the idea here is that any feature of dataset that is an external JSON API is now also an internal API that plugins can use.

00:57:59.400 --> 00:58:01.620
And I've started building plugins against that.

00:58:01.620 --> 00:58:10.360
The Dogsheep beta plugin actually runs internal searches against the dataset search API for things like adding faceted search and so on.

00:58:10.360 --> 00:58:16.360
Dataset GraphQL is a plugin I'm writing that adds a GraphQL API on top of dataset.

00:58:16.360 --> 00:58:18.540
And that's going to be using this client as well.

00:58:18.540 --> 00:58:25.680
So you'll run a GraphQL query which gets turned internally into an internal JSON query and runs over this ASCII mechanism.

00:58:25.680 --> 00:58:26.700
It's cool.

00:58:26.700 --> 00:58:28.920
I hadn't really thought about that with HTTPX.

00:58:28.920 --> 00:58:31.780
I've always used it to just like I'm doing some async methods.

00:58:31.780 --> 00:58:33.220
So here's a good choice for a client.

00:58:33.220 --> 00:58:33.440
Yeah.

00:58:33.440 --> 00:58:36.100
The deep integration with ASCII I think is really exciting.

00:58:36.100 --> 00:58:36.520
Yeah.

00:58:36.520 --> 00:58:37.040
Super neat.

00:58:37.040 --> 00:58:37.440
All right.

00:58:37.440 --> 00:58:39.100
Well, those are all good recommendations.

00:58:39.100 --> 00:58:40.800
Now, final call to action.

00:58:40.800 --> 00:58:42.280
People are interested in dataset.

00:58:42.280 --> 00:58:43.160
Dogsheep.

00:58:43.160 --> 00:58:49.060
First, I just want to throw out you should really go watch the 25 minute or whatever it was talk that you did at PyCon AU.

00:58:49.060 --> 00:58:50.820
That'll connect a ton of things for people.

00:58:50.820 --> 00:58:52.180
I'll throw in another recommendation.

00:58:52.180 --> 00:58:52.900
Yeah, go for it.

00:58:52.900 --> 00:59:00.160
I gave a talk last week, the GitHub Octo Speaker Series, which I think is the best talk I've given about dataset and Dogsheep.

00:59:00.160 --> 00:59:01.800
It's got a lot of very recent demos in.

00:59:01.800 --> 00:59:03.040
And that's linked to on my blog.

00:59:03.040 --> 00:59:05.220
It's a talk about building personal data warehouses.

00:59:05.220 --> 00:59:08.100
But yeah, that's got dataset and Dogsheep demos.

00:59:08.100 --> 00:59:09.020
Yeah, very neat.

00:59:09.020 --> 00:59:17.240
And also, you actually, this is somewhat unusual for an open source project, but I think cool because promoting open source projects is always like, you know, why do they take off or don't?

00:59:17.240 --> 00:59:18.840
You have a dataset weekly newsletter.

00:59:18.840 --> 00:59:19.640
Yes, I do.

00:59:19.640 --> 00:59:20.700
It's not quite weekly.

00:59:20.700 --> 00:59:22.920
Maybe I should have picked a different name.

00:59:22.920 --> 00:59:28.800
But yeah, I've got a newsletter which goes out every week or so with the latest from the dataset ecosystem.

00:59:28.800 --> 00:59:31.680
So that's dataset.substack.com.

00:59:31.680 --> 00:59:35.980
My blog, Simon Willison.net, I update at least once a week with all sorts of bits and pieces.

00:59:35.980 --> 00:59:41.220
And then if you're interested in the Dogsheep stuff, I would love it if people started building these themselves.

00:59:41.220 --> 00:59:43.740
There is quite a bit of assembly required.

00:59:43.740 --> 00:59:50.720
All of the code that I've written is open source, but you have to track down your authentication tokens and run cron jobs and find somewhere to host it.

00:59:50.720 --> 00:59:52.780
And so it's not easy to get up and running.

00:59:52.780 --> 00:59:58.180
But if you do get it up and running, I would love to hear from you about what kind of things you managed to do with it.

00:59:58.380 --> 01:00:02.100
And if people want to build tools that are part of this ecosystem, I'd be absolutely thrilled.

01:00:02.100 --> 01:00:02.840
Yeah, that'd be awesome.

01:00:02.840 --> 01:00:07.060
If they want to build a star to SQLite, whatever star is, let you know, right?

01:00:07.060 --> 01:00:09.280
Well, congratulations on this project.

01:00:09.280 --> 01:00:10.680
I think it's super neat.

01:00:10.680 --> 01:00:12.100
And thanks for coming on to share with everyone.

01:00:12.100 --> 01:00:12.880
Awesome.

01:00:12.880 --> 01:00:13.920
Thanks a lot for having me.

01:00:13.920 --> 01:00:14.440
You bet.

01:00:14.440 --> 01:00:14.680
Bye.

01:00:16.160 --> 01:00:18.780
This has been another episode of Talk Python To Me.

01:00:18.780 --> 01:00:25.220
Our guest on this episode was Simon Willison, and it's been brought to you by Linode and us over at Talk Python Training.

01:00:25.220 --> 01:00:30.280
Simplify your infrastructure and cut your cloud bills in half with Linode's Linux virtual machines.

01:00:30.280 --> 01:00:33.640
Develop, deploy, and scale your modern applications faster and easier.

01:00:33.640 --> 01:00:38.600
Visit talkpython.fm/linode and click the create free account button to get started.

01:00:38.600 --> 01:00:40.560
Want to level up your Python?

01:00:40.560 --> 01:00:45.380
If you're just getting started, try my Python Jumpstart by Building 10 Apps course.

01:00:45.740 --> 01:00:53.520
Or if you're looking for something more advanced, check out our new async course that digs into all the different types of async programming you can do in Python.

01:00:53.520 --> 01:00:58.200
And of course, if you're interested in more than one of these, be sure to check out our Everything Bundle.

01:00:58.200 --> 01:01:00.080
It's like a subscription that never expires.

01:01:00.080 --> 01:01:02.220
Be sure to subscribe to the show.

01:01:02.220 --> 01:01:04.640
Open your favorite podcatcher and search for Python.

01:01:04.640 --> 01:01:05.860
We should be right at the top.

01:01:05.860 --> 01:01:14.860
You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm.

01:01:15.320 --> 01:01:16.940
This is your host, Michael Kennedy.

01:01:16.940 --> 01:01:18.440
Thanks so much for listening.

01:01:18.440 --> 01:01:19.480
I really appreciate it.

01:01:19.480 --> 01:01:21.240
Now get out there and write some Python code.

01:01:21.240 --> 01:01:21.860
I'll see you next time.

01:01:21.860 --> 01:01:22.100
Bye.

01:01:22.100 --> 01:01:22.160
Bye.

01:01:22.160 --> 01:01:22.360
Bye.

01:01:22.360 --> 01:01:22.420
Bye.

01:01:22.420 --> 01:01:22.440
Bye.

01:01:22.440 --> 01:01:23.440
Bye.

01:01:23.440 --> 01:01:24.440
Bye.

01:01:24.440 --> 01:01:25.440
Bye.

01:01:25.440 --> 01:01:26.440
Bye.

01:01:26.440 --> 01:01:27.440
Bye.

01:01:27.440 --> 01:01:28.440
Bye.

01:01:28.440 --> 01:01:29.440
Bye.

01:01:29.440 --> 01:01:30.440
Bye.

01:01:30.440 --> 01:01:31.440
Bye.

01:01:31.440 --> 01:01:32.440
Bye.

01:01:32.440 --> 01:01:33.440
Bye.

01:01:33.440 --> 01:01:34.440
Bye.

01:01:34.440 --> 01:01:35.440
Bye.

01:01:35.440 --> 01:01:36.440
Bye.

01:01:36.440 --> 01:01:37.440
Bye.

01:01:37.440 --> 01:01:38.440
Bye.

01:01:38.440 --> 01:01:38.940
you

01:01:38.940 --> 01:01:39.440
you

01:01:39.440 --> 01:02:09.420
Thank you.