On this episode, I'm joined by Dr. Jeff Boeing, an assistant professor at the University of Southern California,
whose research spans urban planning, spatial analysis, and data science.
We explore why OpenStreetMap is such a powerful source of global map data,
and how Jeff's Python library, OSMNX, makes that data easier to download, model, and visualize with Python.
Along the way, we talk about what shapes city streets around the world,
how urban design influences everything from daily commutes to disaster resilience,
and why turning open data into accessible tools can open up completely new ways of understanding
our cities. If you've ever wondered how to build or analyze your own digital maps in Python,
you won't want to miss this conversation. This is Talk Python to Me, episode 495,
recorded Thursday, January 16th, 2025.
Welcome to Talk Python to Me, a weekly podcast on Python. This is your host, Michael Kennedy.
Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython,
both accounts over at fosstodon.org. And keep up with the show and listen to over nine years
of episodes at talkpython.fm. If you want to be part of our live episodes, you can find
the live streams over on YouTube, subscribe to our YouTube channel over at talkpython.fm/youtube
and get notified about upcoming shows. This episode is sponsored by Posit Connect from
the makers of shiny, publish, share and deploy all of your data projects that you're creating
using Python streamlet dash shiny bokeh FastAPI flask quarto reports, dashboards and API's
Posit Connect supports all of them.
Try Posit Connect for free
by going to talkpython.fm/posit, P-O-S-I-T.
And it's also brought to you by us
over at Talk Python Training.
Did you know that we have over 250 hours of Python courses?
Yeah, that's right.
Check them out at talkpython.fm/courses.
Jeff, welcome to Talk Python To Me.
- Thank you for having me.
- It's great to have you here.
We're gonna talk maps and consuming map data
a lot of interesting research that you've done with that data.
So you have a very cool project, OSMNX, Open Street Map Network X,
that we're going to dive into.
And it really makes all of this street and map data
super accessible to users of Python, right?
- Yeah. - Yeah.
Excellent. I'm really looking forward to talking about that.
Before we dive into it though, just a quick introduction.
Who are you?
- My name is Jeff Bowen.
I'm an assistant professor
at the University of Southern California
in Los Angeles. I'm in its Department of Urban Planning and Spatial Analysis. And
most of my research and teaching are around that intersection of urban
planning and spatial analysis. I did my PhD in city planning at UC Berkeley and
my focus is on urban data science. So using Python tools, usually open data, to
try to understand cities and housing markets and transportation networks and
how people move through space,
how we can make our cities more resilient
to natural disasters, which is relevant in LA this week.
Also more equitable, nicer places to walk around,
and basically just better for urban living.
- Okay, well, that sounds like amazing, amazing research.
It's probably interesting to live in LA
as one of the bigger, crazier locations
in terms of traffic and congestion and population.
It's not New York in terms of density,
but it's probably its own special beast, I imagine.
- Yeah, it is.
LA is an ironic place to be an urban planning professor
because it can be so frustrating
for a lot of our best practices in urban planning,
but at the same time,
there is no better place in the US to be studying
and trying to intervene into some of these crises
that urban planning has wrought.
- Yeah, yeah, a lot of it was just,
I imagine in the early days, just a free for all, right?
There was people, especially out in the West, right?
the US West, people just move out there
and there's not a whole lot going on.
So when a city just neighborhoods go that way
and not quite enough density for a lot of public transit
makes it challenging.
And then they become huge cities
but they're stuck with those consequences.
Like LA and other cities. - Yeah, exactly.
And it's hard to retrofit a city once it's built out
because those are people's homes and private properties
that you can't just cut from whole cloth,
lay down the new city plan.
- Yeah, yeah.
Are there cities out there that are basically planned
from scratch?
- Yeah, there are places like Brasilia,
the capital of Brazil was a master planned city
in the modernist era, mid-century city planning
where it's designed to look almost like the shape
of an airplane for the jet age,
a modern city for the jet age.
- Amazing.
- But there's a wide range of how cities come to be.
Some of them are very organic,
like you think old medieval cities
have formed over centuries.
There are places like LA that built out very rapidly,
or like in Chicago, all these straight lines
following from these 19th century ideas of rationality
and orthogonal street networks.
And then there are these master plan cities
where strong central leadership lets you lay it down
however you think is best.
- I imagine that how the cities are built
has a lot to do with the people who helped construct the city
or there in the early days, how they perceived distance.
People perceived distance in LA by how far can you go
in a car in 20 minutes.
The people perceived distance in Stuttgart,
how far can you walk?
Maybe you can take a horse, potentially.
And that really probably changes things.
- Yeah, you know what, one way we talk about that idea
of the perception of distance
and how the urban form shapes up
is this idea of transportation technologies
And in general, you really see something of a death of geography.
That is, if you look at a 1990s suburb of Philadelphia or of Los Angeles, they look
very similar to each other.
You don't see that sort of local culture.
What you see is really building around the transportation technology and design aesthetic
of the era, rather than the local hands building the place up.
And it's, you know, when cars are how you get around,
you build a city, it's designed for cars.
- Yeah, and then the cars get to be huge SUVs
and they still don't fit.
All right.
Let's talk, you have this really cool visualization,
which I'm going to link to in the show notes,
called City Street Orientations Around the World.
And it's got some pretty wild graphics.
You wanna tell people about this
and kinda try to describe it for the folks listening?
Yeah, so what these show is the orientation
of your city streets.
By orientation, we mean on a compass.
Like, do they point north, south, east, west?
Are they off that north, south, east, west orientation?
Or do your streets point more chaotically
in all directions around the compass?
Like, there isn't a more consistent orientation.
So to visualize these, I used what's basically
just a histogram, but it's a polar histogram.
So where a normal histogram will count your observations
across some set of bins, for a polar histogram,
we've wrapped those bins around the circle of a compass
so that each bin represents 10 degrees around the compass.
And there's 36 bins total for 360 degrees around the circle.
And then the length of each of the bars in those bins
represents the frequency of streets
pointing in that direction in the city.
So in a place like Chicago or Beijing on screen here,
most of the streets point north, south, east, west.
And what that tells us is it's kind of grid-like.
And this is common in a lot of cities around the world.
Even a lot of pre-modern cities
would still orient themselves generally
north, south, east, west,
just based on patterns where the sun was rising and setting.
- Yeah, I'm super surprised to see, for example,
of Barcelona is pretty, not north-south,
but it's 90 degrees.
- Yeah, so in Barcelona, you see that 19th century
and early 20th century, Eixample development.
So around medieval old Barcelona,
El Defensor de la Dada laid out his grand city plan
that is fairly grid-like around it.
You can really see that here.
- Yeah, absolutely.
For people listening, some of them look like
airplane fore-blade propellers,
and some look maybe like boat propellers,
like really dense, but yeah, it's quite wild
to get a quick look.
Do we have LA in there?
We must, right?
- I have LA in the big one.
If you scroll down, we have American cities.
- Oh yeah, yeah, yeah.
Okay. - Yeah, there it is.
- That's a pretty stark contrast.
It's like all of them are propeller looking.
Other than Boston and interestingly, Charlotte.
Boston has a city. - Yeah, really interesting.
- Yeah, but Charlotte doesn't feel like a city
like that to me, but what do we have?
- Yeah, so Boston is kind of one of those pre-modern,
American cities. And its street orientation looks like that for a couple
of reasons. One, Boston is an agglomeration of multiple smaller towns
over several centuries and each of them had grids to some extent but they didn't
all face the same way so they all got merged together. And then it has a lot of
hills that made it harder to build especially in early days and it would
really predated America's obsession with grids. Charlotte is kind of the opposite
it's late 20th century, the sort of winding streets
and cul-de-sacs to make up those arterial roads
and then these sort of secluded in the center
super blocks of subdivisions.
- Nice, well I'm happy to see that Portland, where I live,
is like a perfect propeller blade shape.
You know, in our town, you have northeast, northwest,
southeast, and then street numbers,
like northeast 120th street, right?
And it just screams we're on a perpendicular
North Southeast West grid and it definitely shows up here.
And so this is some of the stuff that you came up with OSM NX.
- Yeah, so I used OSM NX to build all of these
street network models from OpenStreetMap data.
And then OSM NX has built into it a visualization module
that creates these kind of polar histograms
to understand how streets are oriented.
- Awesome, it's super interesting.
Let's, it's like it communicates to you just immediately
as you look at it, it's great.
Let's start by talking about OpenStreetMap.
This is the backing data, obviously,
the important backing data of your library.
So what's the origin of OpenStreetMap?
- Yeah, so OpenStreetMap started maybe 20 years ago or so.
For folks who haven't seen it before,
I often describe it as being akin
to Google Maps meets Wikipedia.
So it's a worldwide mapping project with global coverage.
Anyone can edit or contribute to it.
There is some editorial oversight as well to validate those edits.
But what's cool about OpenStreetMap is that its data license allows you to
pull data from it.
So unlike, say, Google Maps, which is fairly limited in what all it can draw,
you can really pull all of these features from OpenStreetMap.
And if you look, if you just zoom in on one of these cities here.
>> Yeah, let's go to Riverside, yeah.
- Yeah, it was surface streets, rail lines,
building footprints, parks, train stations,
often grocery stores, restaurants, public schools.
All this stuff we're interested in
from an urban planning perspective.
- Yeah, oh, it's really interesting.
And you can add in additional layers,
like cycling maps or transportation maps
or topology maps, topological maps,
all sorts of interesting things here.
- Yeah, exactly.
And it really lets you kind of hone in
on the specific characteristics of the city
that you're interested in.
And there's a humanitarian layer that you can see,
and this has been one of the big uses of OpenStreetMap
now for 15 years or so.
Often when there is a disaster in a less developed country,
humanitarian aid organizations will go in
and use OpenStreetMap and contribute to it
to create a kind of a digital layer
of the facts on the ground.
So the Haiti earthquake a few years back
was a primary example of that,
where it helped them understand where urbanization was,
where there were water pumps, toilets, and so forth,
the kind of things that people need,
but aren't well mapped, often in these informal settlements.
- Yeah, oh, it's amazing.
We have, you know, I think everyone was amazed
when Google Maps came out, right?
Before then we had MapQuest and all those things.
You'd have big little up arrow, up arrow, right arrow
to like pull in tiles of the world and you know,
Google Maps and the world view that they had.
What was that product called?
Remember?
Where you could kind of fly, it was like a 3D thing.
You could kind of fly around in the early days
but it was the same basic data thing.
Anyway, when that stuff came out, it was like,
wow, this stuff is, you can now know about the world.
You can say, where did that road go?
I never went down.
You could just follow it and see
or go to street view or whatever.
But a lot of that is locked behind.
however Google wants you to use it, right?
And it's not super accessible.
And so having something like this,
where people can contribute,
especially if they care about their neighborhood, right?
A lot of this people are contributing back
into OpenStreetMap data.
- Yeah, exactly.
Yeah, so the nice thing with this is that
it really is much more in that open source,
open data model of the world,
which means we can do open science with it,
citizen science with it.
People who just notice that stuff is missing
in their own community, you're able to fix it.
It's no longer that frustration of,
well, when will Google get this thing right?
Or, hey, we have this new highway
that was built outside my town.
When's it gonna show up?
I mean, it's up to you.
You can make it show up later today.
There is a drawback to that though.
The drawback is you get what we call sampling biases.
That is, you tend to see more coverage
in places where people tend to contribute more.
And those things tend to be correlated
with demographic characteristics.
Men contribute more than women.
higher income places more than lower income places
just from a resourcing perspective.
And so one of the challenges we face as researchers
using OpenStreetMap data for our analytics work
is keeping those in mind
because it's hard to do comparative analysis.
If you don't know it's because the infrastructure
is actually different in two places
or because it's been digitized to a different extent
in two places.
- Right, right, right.
Just maybe higher fidelity information
might skew some analysis one way or another, right?
- That's right, exactly.
And it tends to work pretty well comparatively
for stuff like roads, streets,
whereas things like pedestrian footpaths
or fresh food markets might be well digitized
in San Francisco and Portland,
but probably not, say, in a small town in India
or Indonesia, let alone like a suburb of Des Moines, Iowa.
- Yeah, yeah, absolutely.
One of the apps that I use a lot is OnX.
You know OnX?
- Yeah, OnX is cool.
You can, I do a lot of off-road motorcycle riding
and it'll show you all the little trails and stuff,
but someone's got to go and enter all that information.
And so some places are super well mapped
and others are not on the map at all.
But I imagine they probably get some of their data
from OpenStreetMap as well,
although I haven't been able to tell.
- Yeah, it's a good question.
A lot of people downstream do use OpenStreetMap data
because of its quality and accessibility.
- Yeah, Mark out in the audience is asking,
is there some kind of mobile app that kind of gives you
access to the OpenStreetMap data?
Kind of like Google Maps.
- Yeah, there are some mobile apps for it.
I don't use them.
It has some challenges with routing
because it is publicly editable.
Sometimes there'll be little breaks in the connections
where it's not digitized quite right.
But there are routing apps.
You could replace, say, Google Maps on your phone
with OpenStreetMap and you can route with it.
It works particularly well in big cities
and places like Europe and the US,
less so in small rural areas or in less developed countries.
- Yeah, yeah, the onX thing is,
it doesn't really route you,
but it shows you on a map
and then it shows you where you go on it,
and if you choose to stay on the path,
then it'll really help you,
but it doesn't really say,
you can't say, "Give me directions."
At least not when you're off-road.
If you're on-road, maybe it'll fall back
to something that works, but yeah, amazing.
- So what's the licensing story behind all this data?
How, the name probably gives a lot of it away,
OpenStreetMap, but how is this data usable?
What can people do with it?
- Yeah, so it's basically usable
for non-commercial purposes with attribution.
So for example, as researchers,
when we are using these data,
we just need to attribute where it came from,
OpenStreetMap and its contributors.
And I ended up including that license information
somewhere on the read me for OSMNX,
because I think a lot of time people will look over
these underlying data sources and tools
when they're using these kind of downstream,
downstream tools and apps.
So because of that, I just wanted to call it,
yeah, you can see it on screen here.
So I mentioned OpenStreetMaps open data license
requires that derivative works provide proper attribution,
which is really nice 'cause it's a very easy thing to do
for the cornucopia of data that they make available.
- Yeah, yeah, that's amazing.
This portion of Talk Python to Me
is brought to you by the folks at Posit.
Posit has made a huge investment
in the Python community lately.
Known originally for RStudio,
they've been building out a suite of tools
and services for Team Python.
Over the past few years,
we've all learned some pretty scary terms,
of squatting, supply chain attack, obfuscated code, and more. These all orbit around the idea
that when you install Python packages, you're effectively running arbitrary code off the
internet on your dev machine, and usually even on your servers. The thought alone makes me shudder,
and this doesn't even touch the reproducibility issues surrounding external packages.
But there are tools to help. Posit Package Manager can solve both problems for you.
Think of Posit Package Manager as your personal package concierge.
You use it to build your own package repositories within your firewall that keep your project safe.
You can upload your own internal packages to share or import packages directly from PyPI.
Your team members can install from these repos in normal ways using tools like pip, poetry, and uv.
Posit Package Manager can help you manage updates ensuring you're using the latest,
most secure versions of your packages. But it also takes point-in-time snapshots of your
repos, which you can use to rerun your code reproducibly in the future.
Posit Package Manager reports on packages with known CVEs and other vulnerabilities
so you can keep ahead of threats.
And if you need the highest level of security, you can even run Posit Package Manager in
air-gapped environments.
If you work on our data science team where security matters, you owe it to you and your
org to check out Posit Package Manager.
visit talkpython.fm/ppm today
and get a three month free trial to see if it's a good fit.
That's talkpython.fm/ppm.
The link is in your podcast player show notes.
Thank you to Posit for supporting the show.
If I wanted to build a commercial product,
I wanted to build on X squared or whatever,
where I could have people drive around and use that
and map it and make notes and maybe contribute back.
Do you know what the story,
I mean, obviously not a lawyer, right?
But what's the, would people be able to do that?
Or is there a paid API or how does it work?
- It's a good question.
So I know in practice people can
because I see commercial tools
that use OpenStreetMap data all the time.
I don't know the details though
because one of the nice things about being in academia
is you end up being so divorced from the commercial world
that I don't have to worry about that.
- Yeah, that's pretty good, right?
Just do your research, answer interesting questions
and follow the lead, right?
- Those are the benefits, yeah.
It comes with drawbacks, but those are the benefits.
- Yeah, I'm aware of some of the drawbacks as well,
for sure.
All right, well, so that is the foundation
of this project, OSMNX.
Tell us about your project.
- Yeah, so OSMNX is a Python package
that lets you interact with OpenShift's APIs
to make it really easy to download data
and build models or a data frame of those data.
So the two main things that it does is it can create a graph model.
So that is a model of a network like streets or rails or highways or anything else.
Or you can create a geo data frame.
That's kind of a spatialized version of a pandas data frame using GeoPandas of OpenStreetMap features.
So that's all the stuff in OpenStreetMap.
It could be points, lines, polygons.
It could be traffic signals or parks or schools or roads.
But you can download either of those
and then do whatever it is you wanna do with them.
And OSMNX has some other stuff built in.
So it has a simple routing engine built in,
some visualization tools.
You can work with elevation
so that you can figure out street grades.
Particularly relevant, for example, if you're a cyclist
and you're trying to choose a shortest path
that maybe minimizes those grades and so forth.
- Even for cars, Google started offering a,
this path saves, uses the least fuel,
or in my case, the least electricity,
whatever you put into it to make it go.
That's also, uses like that would be interesting.
- Yeah, exactly.
I mean, kind of superficially,
we always imagine that our shortest path to get somewhere
is the one that minimizes our travel time.
But in reality, there are a lot of other things
minimize including fuel consumption or for pedestrians.
Empirically, we know that people will take
longer routes if it's a nicer route.
We're not trying to shave off a minute to walk along
the arterial road with the aluminum.
>> You go through the fence
by that old abandoned factory back here,
and you go straight across, it'll be great.
You'll save a minute.
>> Exactly.
>> Amazing. Okay. Yeah, super fun.
As part of those visualizations,
Can you draw maps?
You can, yeah. So it has some mapping capabilities.
You can also plot routes on it.
It ties into a folium package,
which is a Python package to develop leaflet JavaScript web maps.
So you can do interactive web maps from it.
And then on top of OSMNX, there's a cool downstream package
called Pretty Maps that I'm not affiliated with,
but does really nice, aesthetically pleasing mapping.
It's really customizable and really cool.
>> Okay. Do you have a gallery or something?
>> I do. Yeah. So if you scroll up a little bit,
there's a link to the examples gallery.
It's on GitHub, OSMNX examples.
>> You can pull up different notebooks?
>> Yeah, exactly. So each notebook covers a different theme of analysis.
It would be commonly done with OSMNX.
>> Oh, nice. Yeah, that's really cool.
People just pull up the notebook and see how it works.
What one do you think is a good representative one
to maybe like get a sense of how to work with it?
- So we count from zero
'cause we're in the Python world.
So notebook zero zero is a good place to start.
That kind of steps through some of the basic functionality
that most users would be interested in starting with.
- Sure, okay.
Yeah, so people can download these notebooks
in the examples repository, which I'll link to.
- Yeah, exactly.
So there's a getting started guide in the documentation
after kind of reading through that,
that examples gallery is a really good way
to just kind of get a sense of code
that you can copy and paste to get started.
- Okay, excellent.
And 5,000 GitHub stars, almost a thousand forks.
You know, that's quite a bit of traction.
Congratulations.
- Yeah, thanks.
I mean, it's weird.
This is very much my side hustle,
but it's also really been a labor of love
for a long time now.
There is no reward in academia
for creating open source packages.
I think if you explain it very carefully
to academic power brokers, they say,
"That makes sense, that's great, that's a good contribution."
But prima facie, that doesn't mean anything.
They have no idea.
- It's really, it's tough to put it
on a 10-year application, right?
- Yeah, it makes it sound like you're wasting your time
and you don't understand what you're supposed to be doing
with your time.
(laughing)
- Like, but you understand, the research is so easy now.
- Yeah.
I mean, that's really how that started.
I thought I was gonna be able to do this research
without having to create my own tool,
but it turned out I was not able to.
- Interesting, what was your background
in programming at that point?
- Yeah, so my undergrad and master's degree,
I did computer systems and information systems.
I worked in the private sector for a few years
and decided to go back to school
for my PhD in urban planning.
But when I was coming into that,
I was pitching myself as this sort of like
data science analytics kind of person in that world,
which was starting to get kind of hot at that time.
So it ended up being a nice fit.
And when I started my dissertation,
I had this dissertation proposal that was, you know,
very, very nice, this kind of big grand proposal.
I ended up only doing the first chapter of it
for my entire dissertation
because of how these things expand to fill space.
And when I pitched my proposal to my dissertation committee,
there are a lot of assumptions about the tools
that were out there.
You know, I knew OpenStreetMap was out there.
I'd seen there were some tools to download
and work with data from it.
So I was pretty sure we'd be able to create
a nice mathematical graph model
of OpenStreetMap data automatically.
As I started on my first week of making that happen,
I realized the tool did not exist.
And in fact, it was profoundly difficult
to get data from OpenStreetMap.
If you wanted to filter it,
like I just want pedestrian paths, not everything.
I just want drivable streets, not everything else.
And so the first year of my dissertation work
was creating the first version of OSM and X accordingly.
- So you must have grants and stuff for your research
as academics do.
How much time are you allowed to sort of bring some of that
into adding features back to this project?
- Yeah, back to the benefits of academia,
the nice thing is as long as you get
what they want to see done,
as long as those outputs are there,
your time is really completely your own, right?
So you teach a couple classes,
you have to publish a few papers a year.
Other than that, you're free to do whatever.
So I do get grants, helps me fund my graduate students.
Ironically, it's very hard to get grants
to do this kind of open source software,
open commons development.
Everyone talks about it,
and especially grant funders know
how much we all use this stuff,
and they really like it,
but it just doesn't get funded.
So what you have to do is you have to kind of hack it in
or sideload it, I guess, into your grants
where the grant will answer this empirical question
that's near and dear to the funder's heart.
But to do that, we have this tool building phase
for six months, be able to unlock the analytics for it.
And so that's really how a lot of the features
and a lot of the modules have been added to OSMNx over time.
There's this new project I ought to do,
so we'll just build it into the package.
- Right, we need to be able to answer new questions,
so we gotta have new ways to get to the data, yeah?
- Yeah, exactly.
And it is poorly incentivized in academia,
but academia has really been home
to a lot of open source tool development,
especially niche tools, you know,
from stuff like Astropy,
for the astronomers and astrophysicists,
to things like PySAL,
the Spatial Analysis Library in Python,
but a lot of things that academics
have kind of been at the forefront of,
But for us, it's always a side hustle.
- Right, stuff with biology, psychology,
a lot of the sciences, I imagine.
- Yeah, yeah.
- Working on open source projects like this
feels to me very much like the concept
of funding basic science, right?
Not funding, we need to solve this problem
so we can create a bomb or new energy or go to space,
but we're just doing it
'cause we need to understand the science better
and it'll help us then apply it eventually,
maybe, at some point, right?
Same sort of thing here.
If these exist, it will supercharge the urban planning space
to a huge degree, but getting people to fund it
or to care about helping build that up,
I imagine it's a little bit frustrating.
- No, that's exactly, I mean, the way I talk about it
is it really is a sort of basic science work
that everyone takes for granted
as though these things just exist, right?
This tool just exists, we don't know where it comes from,
it has an API, that API just exists,
And it really kind of obscures all of these human decisions
and social structures and processes and institutions
that make it difficult or make other things easy
and really shape what we end up being able to work with.
- Yeah, absolutely.
Well, someday maybe you can put down,
yeah, I only wrote two papers,
but I have 10,000 get up stars, so.
- Yeah, well.
- Here's your tenure assignment, you know.
This is the future, I'm there for it.
- It's a race between whether ChatGPT overtakes
all of that stuff or you get there and people,
I don't know, that's a whole different deal.
We can talk about that later.
Let's talk about the Overpass API though.
That's the way that you get data from OpenStreetMap.
What is this?
- Yeah, so the Overpass API is another open source tool
OpenStreetMaps database and download stuff, basically,
any of OpenStreetMaps features.
So what's nice about Overpass is that there's some public
instances that you can just query as a web API.
You can also host it locally if you want to work around
throttling limits, rate limits.
Challenge with the Overpass API--
you can see a little bit of it here on screen-- is that the
OverpassQL, it's query language,
can be really hard to work with to do complex stuff.
So this is a fairly simple one.
On screen we can see there's a bounding box
with some lat long coordinates
to explain where we wanna download data from.
The output will be in JSON format.
The server timeout is 90 seconds.
Big queries can take a long time to run.
And then we're querying here for way elements.
So way would be a line feature in OpenStreetMap.
Any line, it could be street,
it could also be the building footprints boundary line.
- I see, the property lines are zoning or whatever, right?
- Yeah, exactly.
Anything represented as a line.
And so we get that back out, it'll be kicked out as a JSON.
And what OSMNx does is it works with this overpass API
to try to automate some of these queries.
So rather than having to write the query yourself,
download the data, and then convert that raw
OpenStreetMap data into a useful data format,
like drop it into Geopandas or into a NetworkX graph,
it assembles all of that under the hood.
It's open source, it's well-documented.
I have a few peer-reviewed articles about the exact process.
So we don't all have to reinvent the wheel
trying to work with this data
because none of this is automatic.
And in fact, 10 plus years ago,
People were very laboriously having to write
these long scripts.
- You know what I'm gonna do?
I'm gonna get a grad student
and it's gonna be their job for the summer
to convert this to Excel or MATLAB for me.
Something like that.
- Yeah, exactly, yeah.
Grad students are great.
But what's much greater, I think,
is having one or two lines of code
to replace your grad student for the summer.
- 100%, let them do meaningful work
that they also like, right?
- Yeah, that's the whole--
- Find it out.
Yeah, yeah, that's fantastic.
As I look at this and think about this,
It feels to me like OSM and X is a little bit like an ORM is to SQL and databases.
Right?
Like, let's you, you think more in Python language and concepts and don't
worry about the query syntax and the data and all that kind of stuff.
Is that a reasonable analogy?
>> Yeah, that's a good way to put it.
And, and the goal for doing something like that is to try to democratize
doing some of these analyses.
You know, open street map data are really amazing.
Overpass is an amazing resource,
but it's really hard to work with
unless you have some software development
or database experience.
And a lot of urban planners or urban geographers
or urban sociologists do not.
They have great questions, they have great theory,
they want a hypothesis test, right?
But we limit science if our tools are too difficult to use
or the right data are too hard to access.
- Yeah, back to that basic science ideas, right?
- Yeah.
- Yeah, so for something like this, what OSMNX does
is it puts that kind of overpatch query
into much more natural language.
Since Python is so readable in the first place,
you can just say something like graph from place,
give it a city name, say Pasadena, California.
My network type will be drive, and then just like that,
I have that drivable street network model
within the city limits of Pasadena.
- Right, yeah, that sounds entirely more usable.
- And one of the things you said is that it's,
it can be pretty expensive to query the Overpass API,
and so your library caches that stuff locally, right?
- That's right, yeah, so expense in the computational sense,
all these things are free to use.
- That's what I mean, computationally expensive.
- Yeah, exactly, and there's a lot of time complexity
to it as well.
So what we do is we cache everything,
which helps you avoid rate limiting.
let's OSMNX be a good neighbor
to these free public commons resources that are out there.
- Yeah, interesting.
It sounds almost like you could have a Docker containers
for different cities that are preloaded
with all the data that might be interesting
to just instantaneously answer a bunch of questions.
- Yeah, which is really my work.
So a lot of what I do will be,
I study every street network
for every urbanized area in the world, right?
So we'll have like eight to 9,000
of these large urban models.
And so I'll just have things cached locally
so that the model building and analytics
are really at the forefront
rather than the downloading and querying for a day.
- Right, it's not only lets you be a good citizen
and not hammer on OpenStreetMaps API,
but it just lets your stuff work faster.
So the incentives are aligned there, right?
- Yeah, and in fact, a lot of the time
when people first use OSMNX,
you know, you'll download the, say the model of Los Angeles, California,
and it takes a minute to run the query on their server, download whatever,
a hundred megabytes of data, and then build that graph,
theoretic model, do all the spatial stuff to it. but after that,
it's nice and fast the next time that you do it,
because you're able to skip all of that pre-processing from the cache.
It probably puts it into some super efficient data structure.
or some others, maybe dictionaries, who knows, right?
- Yeah, yeah, and we use spatial indexes
to accelerate all of the spatial queries locally too,
as well.
- Is that built into Geopandas?
- Yeah, built into Geopandas, there is an R tree,
which is probably the most common spatial index.
But in addition to that, we also use SciPy's KD tree.
They have like a C implementation of a K dimensional tree
that we can do nearest neighbor search.
So for example, Google Maps,
you say from my origin to my destination,
we can do the same kind of thing.
Say from this point to that point,
we'll snap the nearest neighbor on the network
and then solve a path between them.
- Okay, so just the k-nearest neighbor algorithm
sort of applied to, what is that,
the vectors of the intersections or something?
Or then you--
- Yeah, so normally when you do origins and destinations,
you would snap to a node in the network.
So the nodes would be like the intersections and dead ends.
And then the edges would be the links,
the street segments that connect everything together.
And when you solve a path on a graph,
it would be from a start node to an end node.
Then you'd minimize it by something,
either the fewest hops or the shortest distance
or the shortest travel time.
If you incorporate, say, speed limits, everything else,
you can bring in congestion information
to minimize traffic congested travel time
or minimize going up and down hills or anything else.
- Excellent.
If I got this data downloaded
and I wanted to put it into a proper database,
and by proper I mean one that is its own process,
different machines can talk to it in parallel and whatever,
you know, like Postgres or Mongo or something,
are there good databases that you would recommend
or that you've tried, like EdgeDB or Neo4j
or graph databases the right way to go,
or what do you think?
- It depends on what you're trying to do
In terms of OpenStreetMap features,
so if you're just downloading features
and you're not worried about,
you say those topological relationships between things,
like in a street network,
any tabular data structure will do.
Any kind of like SQL database will do just fine.
For a graph model,
yeah, you'd want some kind of graph database
because they can handle graph algorithms
to do stuff like routing
or other kinds of graph analytics
like understanding connectivity on the network.
- Yeah, I have this, given this node,
tell me all the things that are connected to it.
Like that, those kind of questions
could be answered. - Yeah, those kind
of questions, yeah. - Right, yeah.
Yeah, they're kind of pre-computed by the indices
in the structures.
What about bringing in other data?
You know, what if I have real estate data
that doesn't come from OpenStreetMap,
but I want to pull that in and overlay it
with, say, zoning information,
overlapped with census data that gives me revenue,
you know, how much that person in that area
makes per year on average or something like that.
Can you answer questions like that?
- Yeah, really common use case.
So OSMNX can kind of do that by itself,
but mostly as a tool in the toolkit to do it.
So, you know, a common way that we often will do this
is for accessibility analyses.
So for example, here at my house,
how many grocery stores could I access
within a 15 minute walk?
And it gives you a sense of, you know,
how car dependent or not a place is.
And also we can start looking at food deserts
communities where people don't have access to fresh food.
So for that, what you're doing is you have a model of the network, and then you also
have all of this point of interest data, right?
So where all these amenities are located, you would then snap them, load them onto
the graph, and then you can calculate accessibility by looking at where those
things are located.
And you can either look at cumulative opportunities within some time horizon or
distance to the nearest opportunity.
You can put different decays on it, like the utility of it being so far away, drops
the linear decay the further I have to go
or a non-linear decay if it really drops off fast.
But it's exactly that.
- Yeah, if a grocery store is three miles away,
it's not one third as useful as one that's one mile away.
It's a little bit more to carry your groceries that far.
- And similarly, there's marginal utility
for the 30th grocery store that you can reach.
Doesn't give you that much benefit
over the 29th that you can reach.
Whereas the first and second are very valuable.
- Yeah, if it goes from zero to one and one to two,
those are big jumps right there.
- Yep.
- Okay, very interesting.
How about some projects that people might be thinking about?
They look at this data and they're like,
"Wow, this looks really detailed and neat,
"and we have all these different aspects we can ask about,"
but maybe they just wanna play around.
- Yeah, I mean, hopefully,
if you look at the Getting Started Guide
or these usage examples,
you'll see it's really easy to play around,
two, three lines of code,
and you have a visualization of your local street network.
things that we've done in my lab at USC that may spark some interest. We look at
air pollution exposure and driving patterns in Los Angeles to try to
understand you know who's driving and where and then in turn who is exposed to
that tailpipe, those tailpipe emissions from those cars. So we use OSM and X to
model those street networks to simulate those trips and to look at what census
tracks that stuff goes through. We also use it for studies of urban form. So that is kind
of the shape, texture, density, the grain of a city. I have this series of square mile
visualizations of different cities around the world. And you can really quickly see
how different forms of urbanism look relative to each other. So a place like Irvine, California,
in Southern California has this really coarse-grained street network.
A place like Venice in Italy has incredibly fine-grained, like spiderweb street network.
Stuff like that really helps to kind of convey these different modes of urbanism and urbanization.
We also use these to look at disaster resilience.
That is, different street networks will be better if things start going wrong around
Things with more redundancy, more connectivity built in
are easier to route around if parts start failing
in a fire, in an earthquake.
And so we can capture some of these characteristics
and then make recommendations to planners and policy makers
about how to make your street network,
how to intervene in it to make it more resilient.
- Yeah, in my area, there's an East-West highway,
one of them, very few, that goes from the city west.
It has to go through this mountain tunnel
and it's two lanes each way in the mountain,
or three lanes in the mountain tunnel.
and there's even on a Saturday afternoon,
there's stopped cars trying to go through that tunnel.
And those are on good days.
Imagine if there was an emergency, it would just be done.
- Yeah, that's the 26, right?
- Exactly, it is the 26.
- Yeah, I used to live in Portland.
I lived in Goose Hollow, and my gym was in Beaverton,
so I would have to go through that tunnel to go work out.
It would always be like, you know, one of these days,
one of these days.
- Exactly, yeah, I live about a mile from Goose Hollow
up through that tunnel.
So yeah. - Oh, cool.
- Yeah, yeah, very cool.
Yeah, but there's just, and that one's hard to fix
because there's houses all along there.
It's a big hill, hard to go over it.
You know, it would take a lot of willpower
to make another tunnel.
- Yeah, interesting thing.
I did my PhD at Berkeley,
and there is another tunnel there
that goes from that kind of inner East Bay,
Berkeley, Oakland border,
out to the further East suburbs
like Orinda, Moraga, and so forth.
And that tunnel is really old.
I mean, it's, I don't know, 80 years old,
maybe something like that.
And they determined that when the big one
does eventually hit, it will fail, it will collapse.
And it's not, it's cost prohibitive
to actually prevent that from happening.
So the idea is it will collapse
and crush everyone in that tunnel,
and they will then build a new tunnel
at that point when they have to.
- Oh my goodness.
- We have a lot of that to look forward to.
(laughing)
- Yeah, just go quickly when you're in the tunnel.
Get out of there.
>> I always did.
>> Amazing.
All right.
Let's talk real quickly about integrating OpenOSMNX with other data science tooling,
right?
We have so many different things.
We have PyTorch, we have Pandas, we have Polluters, all the different things.
Like what are the different integration features?
You know, you talked about data frames.
That sounds like Pandas and GeoPandas, right?
That's one sort of path across those things.
- Yeah, so OSMNX is built primarily on top of three things.
It's NetworkX for the graph models,
Geopandas for the spatial data frames,
and Matplotlib for the visualizations.
In the spatial Python world,
most stuff happens in the pandas ecosystem
because Geopandas and PySAL and similar tools
are very pandas-oriented.
I know there's been some work toward geopolars,
but it's kind of in its infancy,
as far as I know, to this extent.
But that'll be great when that's available.
NetworkX is another funny one
because it's a pure Python implementation.
It has a lot of loops,
and so it can go really slow for calculating
some of these big time complexity graph algorithms,
things like betweenness centrality,
where you have this massive OD native,
basically all the nodes to all the nodes.
- Yeah, yeah, some combinatorial craziness.
- It ends up being very time consuming
to calculate those kinds of things
for a metro area sized network model.
So we'll integrate with other tools.
iGraph does that in C.
Also using tools like QGraph
where you can do it on the GPU instead
can be really fun. - Oh wow, okay, yeah.
- But it's limited because of the weighting scheme
that you say, I want to minimize travel time
or stuff like that.
- I see.
- Yeah, and then also integrated with Folium
to create interactive leaflet web maps out of the box.
- Yeah, I wonder how that the free-threaded Python stuff
will affect some of those computational things.
Like that everything compared with everything else
sounds like a so-called embarrassingly parallel
sort of problem, right?
And so, you have 10 cores, so let's just do 10 blocks
of them all in parallel and you know.
- Yeah, so that's what we do when we do it in pure Python.
So we'll use multi-processing generally with all your CPUs.
The only challenge is that your OD matrix,
your origin destination matrix is growing
at this exponential rate, but then there's a linear number
of cores, so going from one to 24,
you're dividing by a factor of 24.
But when you have say a million nodes
times a million nodes in your graph,
it's going to take a long time.
- It's still gonna be slow, but it's--
- You've got from months down to weeks.
- Yeah, I mean, you're still dividing it
by the number of cores,
but it doesn't necessarily mean it's super fast.
Although the GPU stuff seems like
it could absolutely fly, right?
- Yeah, that makes a big difference.
And I think that's really the area of growth
where the handling, I mean, basically getting out
of Python loops in the first place
is the biggest area of growth.
And then doing things on a,
since these are really just
very simple linear algebra problems,
being able to do them on a GPU
makes it much more efficient.
- Yeah, did you hear about this thing,
this new NVIDIA, what, I don't know what it's called,
Project Digits, where they announced
a $3,000 desktop AI computer?
I mean, they call it AI, but really it's just a GPU.
- That is, no, I hadn't heard about that.
- And this thing, it's the size of a Mac mini.
It's this, if you look at the pictures,
a little bit of a golden thing.
It's kind of clever they made it gold,
but that's a massive computer sitting next to you.
- Yeah.
- You could run that.
This is pretty interesting actually, I think.
- Yeah, yeah, that is.
I mean, it's shocking too,
because it looks so much smaller
than just an Nvidia card in your desktop.
- I think it basically is just the GPU.
- Right.
- Nothing to do with graphics.
just remote into it over, I don't know, a terminal or however you, or maybe you set
it as the Docker host and then you just do Docker locally, but it runs there, you know,
that type of thing.
I don't know how it works, but I kind of want one.
>> Yeah.
I mean, it looks sexy enough.
Of course you would want one, right?
>> Of course.
But if you had questions like that, if you say I have a million nodes and I could run
that on a GPU, getting something like this that's, you know, kind of the H100 style of
like the really large just meant for compute, not really a graphics card.
like that would really unlock some interesting options.
Yeah, totally.
I mean, yeah.
So I, I have a, a GeForce card in my desktop that I'll pass
them and that stuff locally onto.
But, another one of the academic benefits at USC, we have a high
performance computing cluster that, I can schedule tasks to run on that,
say over winter break and then come back in January and, and the month
of, of, 1000 CPUs is all done.
That's amazing.
And we had a mainframe, a silicon graphics mainframe
when I was studying math that we could all work on
in this group of maybe 10 of us.
And it wasn't as big as what you're saying,
but it was a quite expensive, big machine.
It had its own room and it was really allowed
to go into that room.
And I remember we came in to,
I came in to do whatever I was working on
as a grad student or whatever, I think I was a senior.
Anyway, I came in to work on my project
and nobody could access it.
and it was still in there making loud noises
like it was running, but it acted like it wasn't running.
And so I'm like, what is going on?
One of the other grad students had run a job
like you're talking about,
but they were having trouble getting into work
'cause they were math people, not programmers.
And they started logging everything.
And there were no limits on what people could do
to the computer, the people that set it up.
And so what had happened is it ran 100% CPU
on all the cores all night logging
until it ran out of space.
And then apparently the computer won't run anymore
if it has zero more bytes.
needed some temp file or something.
And so it just, it was just done.
I imagine you guys have more guardrails than that.
- Yeah, a lot of guardrails.
I would love to try to do something like that
just for the yuck sometime, but we got a lot of-
- The admins of that computer
would not please with that grad student.
- I can only imagine.
- It took a long time to get it going.
It's like a day of work to get it turned back on.
'Cause they didn't even know what was wrong with it at first
it just wouldn't turn on.
- Oh well, jumping back a moment to the,
no, that's not the one I wanted to copy,
to the integration, have you considered
working with narwhals?
- I looked into narwhals recently.
My understanding is one of the things that does well
is it kind of abstracts away from pandas versus polars,
so you're less tool dependent, right?
- Yeah, and it also lets people
who want to consume your library,
who want to do their work in polars,
talk polars to your project,
and people want to do pandas, talk pandas,
as well as QDF, modem, pyro, and other.
It kind of is an adapting layer,
so people can work in their native framework
and then talk to you.
- Yeah, I mean, yeah, on paper it sounds great.
I haven't played with it yet,
other than just being familiar with it,
but I mean, that sounds like the ideal, right?
We shouldn't be enforcing these certain APIs
or languages on people.
- Yeah, it does look interesting.
I believe its foundational internals are maybe just the API,
But there are pollers, not pandas,
so maybe that's not as good of a fit.
But it could be an interesting thing to look in the future to
make it native to
all the different DataFrame libraries that people want to talk in.
>> Yeah, definitely.
>> Are you going to need a new grant?
>> Yeah. Here's the basic science stuff.
If I'm interested in network models,
I don't know about getting into the nuts and bolts of
which DataFrame API we're going to have people use.
- Yeah, yeah, absolutely.
All right, well, let's close this whole conversation out
by talking about something that I know
is on a lot of people's minds, and that's LLMs.
And what do LLMs mean for this mapping data
and the types of questions you can answer
in OSMNx, all these types of things?
Like, how's that shaping your world?
- Yeah, so two ways that I like to think about it.
One is what LLMs have to offer us today.
And then one is what LLMs have to offer us,
say five to 10 years from now.
That is, how do we future-proof ourselves,
our careers, our livelihoods?
And I mean, or just optimistically,
how do we really get ahead of it to take advantage
of nice things that they'll have to offer?
Today, I am deeply unimpressed with most of the use cases
that I see for it.
I think that the outputs only look good
to people who aren't experts
and don't know how wrong it is.
Does some things well, you know,
like having a co-pilot while you're coding
and having it recommending.
That's nice.
I mean, it's like tab complete from years ago.
It's the same kind of thing, just iterating.
But you know, when someone tells it to,
you know, here's the prompt,
generate my algorithm for me.
Two things.
One, it very often doesn't do that good of a job
even at this point.
And number two, if you don't enjoy doing that kind of thing,
I'm not sure why you're in software development
in the first place.
- I know.
- For me, when I interface with it,
it's usually in the classroom.
And you'll have, so, you know,
people like to complain kids these days.
It's the same way it was when I was in school.
You know, a third of the class loves a subject.
They're just killing it.
They wanna be there.
They want a job in this.
A third of the class, they're like,
"Well, it's a required class.
"I'll try to do what I can."
And a third of them say, "I don't know why I'm here.
"Why is this a required course?
"I hate this.
"Is there any way I can cheat
"to just get a passing grade?"
- Where's my shortcut?
- Exactly.
I mean, and you know, the difference today
is that LLMs make it so much easier
and they really normalized cheating as well,
where we don't have that good of systems in place
to be able to detect it for code,
for writing, for anything.
And I think one of the problems here,
you know, it goes back to like licensing or attribution.
When we normalize this kind of cheating,
we're normalizing taking credit
for things that aren't our work.
And if we're doing that in school, we build those habits,
what's it gonna look like
when we're working in the workforce?
We either don't have the skills that people think we have,
or we think it's just normal, acceptable practice
to copy and paste and say,
here's my intellectual contribution.
- It's too much of a crutch,
and if maybe it solves 90% of the problems,
but that 10%, it doesn't,
you're fully incapable of approaching that problem, right?
- Exactly, yeah.
And I think, so that's the first part,
what LLM's offered us today.
I think the second part is say five to 10 years from now.
Part of it is scary and part of it is exciting.
I think that coding as it looks today
is going to largely go away over my lifetime
because coding is this mix of art and intellectual labor
and manual labor.
The manual labor parts will be going away
and that's where a lot of people use LLMs now
to reduce some of that grind of coding.
And I think it's important for us to figure out
where those intellectual contributions are,
where the art, the poetry of coding
remains a human contribution moving forward,
where we have that sort of human in the loop
to make ethical decisions that we don't pass off,
where it is humans who are doing some of the architecting
and not just handing it off saying,
"Here's a prompt, now redesign our financial system
from the ground up for us,"
but that it remains human guardrails and human ethics
and human notions of justice that we use AI to help us,
but doesn't replace that.
- Yeah, I feel like there's gonna be a lot of people
who are programmers now are gonna have to kind of be
a senior tech team lead type of thing,
where the team is asking the LLM to do stuff for you.
And that's gonna be awesome for those people, us,
but for the people who are brand new,
there's gonna be a wider gap to try to cross.
It's gonna be harder and harder to go from real beginner
to that level when, who's going to employ you
when it takes you a day and I can ask the LLM
to do it in five minutes and I got to review it either way.
That's a very concerning social consequence, I think.
- No, exactly.
I think those kind of entry-level jobs
are very much more labor-focused, right?
The senior dev isn't doing that work
because someone with less experience, less pay can do it.
And if LLMs replace that over time,
It's really hard to get your foot in the door in that industry.
- Yeah, it can be like COBOL programmers
did not a program without an LLM someday.
(laughing)
- Yeah.
(laughing)
- I don't know if that's good or bad,
but it's gonna be weird.
Anyway, I think it's probably pretty interesting.
And given, you know, access with your API
and with all this data,
I'm sure people will be able to ask
and answer interesting questions with it.
- That's the hope.
- Indeed, all right.
Two really quick things to close out our conversation.
Roadmap, anything on the roadmap coming real soon
that people have been asking about, thinking about?
- Yeah, so I just released V 2.0,
I don't know, a month or so ago.
And so, thank you.
And that was really just a year of rethinking the API.
I was gonna allow myself breaking changes,
really cleaning things up.
I added type annotations for the first time.
I really changed the linting and formatting,
CI, pre-commit, everything.
So it's a much more rigorous package at this point.
I'm taking a breather right now,
where I don't have that much big on the horizon.
I try to keep a really clean issue board as well,
where I quickly triage, this is in scope, this is not.
- Yeah, I noticed you have three open issues
and zero PRs, which is awesome.
- Yeah, I try to do it courteously,
where I'll tell people a lot of the time,
you know, this isn't in scope for the project,
but I appreciate the suggestion, but I'm busy,
so keeping a clean issue board makes my life simple.
So short answer, I'm taking a breather
instead of road mapping in January.
- But you just had a big release, which is cool.
- Yeah.
Contributions accepted?
- Absolutely.
And very often what I ask when someone opens an issue
and I say, "Yeah, that does sound useful,
would you like to open a PR?"
And as much as this is my side hustle,
most of the pull requests that I add to it
are very much aligned with a current
or recent research project that I'm working on.
So for others, if you see a gap or a cool feature
that could make people's lives easier,
better analytics down the road,
please propose it and I'd love a contribution.
- Excellent.
All right, final call to action.
People are psyched about mapping.
They wanna get started.
What do you tell 'em?
- I tell them to work through existing examples
from the tools that are out there.
If you haven't done any kind of mapping before,
there's a lot of cool packages in R and in Python.
Find something you're comfortable with.
work through some tutorials,
and then figure out something close to home.
It's so much easier if you have that kind of
local motivation, local knowledge.
Do some mapping, do an analysis in your community
for something that's bothering you,
you know, you think could be better.
- Excellent.
All right, well, thank you for being here
and congrats on that cool project.
- Thank you very much.
- This has been another episode of Talk Python to Me.
Thank you to our sponsors.
Be sure to check out what they're offering.
It really helps support the show.
This episode is sponsored by Posit Connect from the makers of Shiny.
Publish, share, and deploy all of your data projects that you're creating using Python.
Streamlit, Dash, Shiny, Bokeh, FastAPI, Flask, Quarto, Reports, Dashboards, and APIs.
Posit Connect supports all of them.
Try Posit Connect for free by going to talkpython.fm/posit.
P-O-S-I-T.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at Talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show, open your favorite podcast app, and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the Direct
RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air, be sure
to subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.