On this episode, Rob Emanuel and Tom Augsberger join us to talk about building and running
Microsoft's Planetary Computer Project. This project is dedicated to providing the data around
climate records and the compute necessary to process it with the mission of helping us all
understand climate change better. It combines multiple petabytes of data with a powerful
hosted JupyterLab notebook environment to process it. This is Talk Python to Me, episode 334,
recorded September 9th, 2021.
Welcome to Talk Python to Me, a weekly podcast on Python. This is your host, Michael Kennedy.
Follow me on Twitter where I'm @mkennedy and keep up with the show and listen to past episodes
at talkpython.fm and follow the show on Twitter via at talkpython.
We've started streaming most of our episodes live on YouTube. Subscribe to our YouTube channel over
at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.
This episode is brought to you by Shortcut, formerly known as clubhouse.io,
and us over at Talk Python Training. And the transcripts are brought to you by Assembly AI.
Rob, Tom, welcome to Talk Python to Me.
Thank you.
Good to have you both here. We get to combine a bunch of fun topics and important topics,
data science, Python, the cloud, big data, as in physically lots of data to deal with. And then
also climate change and being proactive about studying that, make predictions and do science
on huge amounts of data.
For sure. Looking forward to it.
Yeah, this will be fun.
Yeah, absolutely. Before we get into those, let's just start real quickly. How do you two get into
programming and Python? Rob, start with you?
Yeah, sure. So I've been a developer for, I don't know, let's say 14 years. I started at a shop that
was doing Sybase Power Builder.
That goes back a ways.
That's back a ways. And I actually, I come from a math background, so I didn't know a lot about
programming and started using Python just sort of like on the side to parse some bank statements and
do some personal stuff and started actually like integrating some of our source control at the
company with Python and had to write some C extensions. So got into the Python source code
and started reading, you know, that code and being like, oh, this is how programming should
work. Like this is really good code. And that year went to my first PyCon. It was just like all in.
I need to get, you know, a different job where I'm not doing Power Builder.
And yeah, really, I kind of credit Python and the code base and setting me on, you know,
a better development path for sure.
Oh, that's super cool. Python's a fun experience, isn't it?
Oh, yeah.
Yeah. It's like my geek holiday, but sadly, the geek holiday has been canceled the last two years.
Oh, no.
Yeah.
Yeah. Tom, how about you?
Kind of similar to a lot of your guests, I think. I was in grad school and had to pick up
programming for research and simulations. This is for economics.
They started us on MATLAB and Fortran. It goes back maybe further, almost as far as you can go.
And anyway, I didn't really care for MATLAB. So moved over to Python pretty quickly and then
just started enjoying the data analysis side more than the research side and got into like that whole
open source ecosystem around pandas and stats models and econometrics library. So I started
contributing to open source, dropped out, got a job in data science stuff, and then moved on to
Anaconda where I worked on open source libraries like Pandas and Dask for a few years.
Yeah. In a weird turn of coincidence, a weird coincidence, I was just the previous episode
with Stan Sievert, who you worked with over there, right?
Yeah.
Yeah. So he's the director of community innovation. And then, yeah, it was a great place to work at.
Really enjoyed it. And then came on to this team at Microsoft almost a year ago now working on the
planetary computer.
Yeah. Cool. Well, the planetary computer stuff sounds super neat. You get to play with all the
high-end computers and the big data and whatnot, right?
Yeah. It's a lot of fun. Although I did have a chance to play on, I think it was Summit, which is one of our
nation's supercomputers at my last show. So that was a lot of fun too.
Okay. Well, it's hard to beat that, right? That's one of the ones that's like, it takes up a whole room,
a huge room. That's pretty fantastic. Awesome. All right. Well, what are you two doing today?
You know, you're both on the planetary computer project. Are you working at Microsoft? What are you doing there?
Yeah. So we're on a pretty small team that's building out a planetary computer, which really
is sort of three components, which is a data catalog, you know, hosting a lot, you know,
petabytes, petabytes of data, openly licensed satellite imagery and other datasets on Azure's
blob storage. We're building APIs and running API services that ETL the data, encode metadata,
according to the stack specification, which we can get into later about that, those datasets,
putting them into a Postgres database and then building API services on top of that.
That's a lot of what I do is manage the ETL pipelines and the APIs and then expose that data
to users, environmental data scientists, and really anybody. It's just publicly accessible.
And yeah, that's sort of my side. And then there's a compute platform, which Tom can talk about.
Yeah. So all this is like in service of environmental sustainability. And so we have our primary users are like
people who know how to code mostly in Python, but they're not developers. And so, you know, we don't want them having to worry about things like Kubernetes or whatever to set up a distributed compute cluster.
So that's where we're kind of this hub comes as it's a place where users can go log in, get a nice convenient computing platform built on top of Jupyter hub and dash where they can scale out to these really large workflows to do whatever analysis they need, produce whatever derived data sets they need for them to pass along to their decision makers and environmental sustainability.
Yeah, that's super cool. The platform miller building people who might have some Python skills, some data science skills, but not necessarily high end cloud programming.
Right. Yeah. Handling lots of data, setting up clusters, all those kinds of things. You just push a button, end up in a notebook. The notebook is nearby petabytes of data. Right. Right. Exactly.
So we'll talk a lot about like cloud native computing data analysis. And so really what that means is just putting the compute as close as the data as possible. So in the same Azure region.
So you just need a big hard drive.
A really, really, really, really big hard drive.
That's what the cloud is. It's one third hard drive.
Exactly. It is. Yeah. So super neat. Before we get into it, though, let's just maybe talk real briefly about, you know, Microsoft and the environment. This obviously is an initiative you all are putting together to help client climate scientists study the climate and whatnot. But, you know, I was really excited to see last year that you all announced that Microsoft will be carbon negative by 2030.
Yeah, for sure. I mean, Microsoft and, you know, prior to me joining Microsoft, I didn't I didn't know any of this, but Microsoft's been on the forefront of, you know, corporate efforts and environmental sustainability for a long time.
And, you know, there's been an internal carbon tax that we place on business groups that were, you know, there's actual payments made based on how much carbon emission each business group creates.
And that's been used to fund the environmental sustainability team and all these efforts.
And that sort of culminated into these four focus areas and commitments that were announced in 2020.
So carbon is a big one, not just carbon negative by 2030, but by 2050, actually having removed more carbon than Microsoft has ever produced since its inception.
And that's over scope one, scope two and scope three, which means accounting for, you know, downstream and upstream providers.
And then there's a couple more focus areas around waste.
So by 2030, achieving zero waste and around water becoming water positive and ensuring accessibility to clean drinking and sanitation water for more than 1.5 million people.
There's an ecosystem element to by 2025, protecting more land than we use.
And then also creating a planetary computer, which is really using Azure's resources in the effort to model, monitor and ultimately manage Earth's natural systems.
That's awesome.
And that's the part you all come in, right?
Yeah, exactly.
The planetary computers in that ecosystem commitment.
And that's what we're working towards.
Yeah, very cool.
The removing all the historical carbon, I think is pretty fantastic.
And being carbon negative, right?
So much stuff runs on Azure and on these couple of large clouds that that actually is a statement about a large portion of the data center usage as well.
For sure.
How many data centers does Azure have?
Like three or four, right?
I think it's the most out of any of them.
It's a lot, right?
It's over 50 or something like that?
Large data centers?
I don't know.
But it's a big number.
They're building them all the time too.
Yeah.
Yeah, yeah.
It's like constant.
So that's a big deal.
Super cool.
All right.
Let's talk about this planetary computer.
You told us a little bit about the motivation there.
And it's made up of three parts, right?
All right.
Yeah.
So tell us about it.
So there's technically four parts.
We recognize that technology for technology's sake is just kind of spinning your wheels, right?
We have to be building all of this data, all this data access, the analytics platform towards
applying data and insights to actually making an impact on environmental sustainability concerns.
And that's done not by us, an engineering team, like kind of, you know, trying to figure out
the climate scientists, right?
We're engaging with organizations to build out applications specifically on these data and
services, you know, and partnering with organizations that have specific goals.
So there's an applications pillar to the planetary computer.
But from an engineering standpoint, we're mostly focused on the data catalog, the APIs,
and the hub that we had touched on briefly.
Yeah.
And then the applications is what the partners and other people building on top of it are really
doing, right?
Exactly.
And we participate in that and, you know, help bring different organizations together
to build out the applications and, you know, use the money that we have to actually fund
applications that are, you know, specifically aimed at different use cases.
Yeah.
Very cool.
There are some other things that are somewhat like this, right?
Like Google Earth Engine and AWS and probably could just grab this yourself.
You want to do a compare and contrast for us?
Sure.
Yeah.
So Google Earth Engine is sort of the bar that's set as far as using cloud compute resources
for earth science.
And it's an amazing platform that's been around for a long time and is really just like a
giant compute cluster that has interfaces into an API and sort of like a JavaScript interface
into it that you can run geospatial analytics.
And so it's a great, you know, like I said, a great tool.
Can't sing its praises enough.
One of the aspects of it that make it less useful in certain contexts is that it is a little
bit of a black box, right?
The operations, the geospatial operations that you can do on it, the way that you can
manipulate the data are sort of whatever Google Earth Engine provides.
If you wanted to run a PyTorch model against a large set of satellite imagery, that's like
a lot more difficult.
You can't really do that inside a Google Earth Engine.
You have to like ship data out and ship data in and getting data in and out of the system
is a little tough because it's like sort of a singular solution.
And they can optimize a lot based on that.
So the approach we're taking is more a modular approach, leaning heavily on the open source
ecosystems of tools, trying to, you know, make sure that the open source users are first class
users that we're thinking of first.
And that if people want to just use our data, we just have cloud optimized geotiff, you know,
these, these flat file formats on blob storage, go ahead and use it.
You don't have to use any of the other stuff that we're building.
But if you want to do space to temporal searches over it, we provide an API that's free access
that allows you to do searches and get metadata about the data so you don't have to actually
read in the bytes.
And then also providing the hub experience, which brings together that really rich open source
ecosystem of Python tooling, including our tooling.
And we're also building out, you know, other mechanisms to access this data.
But those current focuses is really on that Python data science.
But yeah, considering the open source ecosystem sort of as our user experience and trying to
treat that as like the first class use case.
Yeah, that's fantastic.
Tom, tell me if I have this right.
I feel like my limited experience working with this is you've got these incredible amounts
of data, but they're super huge.
You all built these APIs that let you ask questions and filter it down into like, like, I just want
the map data for this, you know, polygon or whatever.
And then you provide a Jupyter notebook and the compute to do stuff on that result.
Is that pretty good?
Yeah.
Yeah, that's pretty good.
If you just think like the API is so crucial to have and we'll get into what it's built on.
But just for like the Python analogy here is like, imagine that you only had lists for your
data structure.
You don't have dictionaries.
And now you have to like traverse this entire list of files to figure out where is this one
at like in space on Earth?
Where is it at?
Or what time period is it covering?
And, you know, the nice thing about the API is you're able to do very fast lookups over
space and time with that to get down to your subset that you care about.
And then bring it into memory on ideally on machines that are in the same Azure region,
bring those data sets into memory using tools like Xarray or Pandas and Dask, things like
that.
Yeah.
Very cool.
So, Rob, you mentioned the Postgres database.
Does that do you parse this data and generate the metadata and all that and then store some
of that information in the database so you get to it super quick and then you've got the
raw files as blob storage, something like that?
Yeah, for sure.
I mean, that's as much metadata that you can capture and to describe the data so that you
can kind of, you know, do what Tom said and like ignore the stuff that you don't care about
and just get to the area that you care about.
We try to extract that and we do that according to a spec that is this really interesting, like
community driven spec that one of the biggest complaints about dealing with satellite imagery
and this Earth's observation imagery is that it's kind of a mess.
There's a lot of different scientific variables and sensor variables and things.
So there's been a community effort over the past, say, three or four years to develop specifications
that make this type of information machine readable.
And so we've kind of bought fully into that and have processes to look at the data, extract
the stack metadata, which is just a JSON schema specification with extensions, and then write
that into Postgres.
And one of the things that we, you know, I've been trying to do for like transparency and
contribution to open source is a lot of that ETL code base, those Python, the Python code that
actually works over the files and extracts the metadata is open source in the stack utils,
GitHub organization.
So we're trying to contribute to that sort of body of work of how to generate stack metadata
for these different data types.
You want like the metadata for the exact same image that's coming from like the USGS public
sector data set.
You want the stack metadata to be identical for that, whether you're using our API or Google
Earth engines, who also provides a stack API.
And so like we're working together on these kind of like shared core infrastructure libraries.
This portion of Talk Python to Me is brought to you by Shortcut, formerly known as clubhouse.io.
Happy with your project management tool?
Most tools are either too simple for a growing engineering team to manage everything or way
too complex for anyone to want to use them without constant prodding.
Shortcut is different though, because it's worse.
No, wait.
No, I mean, it's better.
Shortcut is project management built specifically for software teams.
It's fast, intuitive, flexible, powerful, and many other nice positive adjectives.
Key features include team-based workflows.
Individual teams can use default workflows or customize them to match the way they work.
Org-wide goals and roadmaps.
The work in these workflows is automatically tied into larger company goals.
It takes one click to move from a roadmap to a team's work to individual updates and back.
Type version control integration.
Whether you use GitHub, GitLab, or Bitbucket, clubhouse ties directly into them so you can update
progress from the command line.
Keyboard-friendly interface.
The rest of Shortcut is just as friendly as their power bar, allowing you to do virtually
anything without touching your mouse.
Throw that thing in the trash.
Iteration planning.
Set weekly priorities and let Shortcut run the schedule for you with accompanying burndown
charts and other reporting.
Give it a try over at talkpython.fm/shortcut.
Again, that's talkpython.fm/shortcut.
Choose Shortcut because you shouldn't have to project manage your project management.
Well, let's dive into some of the data, actually, and talk a little bit about all these data sets.
So a lot of data, as we said over here.
Maybe highlight some of the important data sets that you all have on offer.
So Sentinel-2 is our largest and is incredibly important.
It's multispectral imagery, optical imagery that is 10 meter resolution.
So it's the highest resolution.
And when we talk about satellites, we often talk about what is the resolution that's captured.
Because, you know, something like Landsat, which we also have Landsat 8, is 30 meter resolution.
So once you get down to like street level, you can't really see.
Everything's blurry, right?
Right.
Each pixel represents 30 meters on the ground.
Right, right.
Okay.
So Sentinel is 10 meter.
You get a lot clearer picture.
You can, you know, track.
I mean, if you're doing, you know, sort of deforestation monitoring, for instance,
like you can really track the edge of the deforestation a lot better with 10 meter imagery.
Right.
Or glaciers and you want to understand the boundary of it or something.
Exactly.
And, you know, it's still pretty low resolution compared to commercially available imagery.
But as far as open data sets, it's high resolution.
It's passively collected.
I think the revisit rate is, I should have this offhand, I think it's eight days.
So you can really do like monitoring use cases with that.
It generates petabytes and petabytes of data.
So it's a lot to sort of work over.
I mean, generating the stack metadata for that, you know, it's like you got to fire up like
10,000 cores to kind of run through that.
And you end up actually reaching the limits of how fast you can read and write from different
services.
But my gosh.
Yeah.
But it's a really great, really great data set.
A lot of work is being done against Sentinel too.
So a lot of what I'm seeing and I'm reading through here is this annually or this from 2000
to 2006 or like the one we were just speaking about is since, you know, from 2016.
This data is getting refreshed.
And can I ask questions like, how did this polygon of map look two years ago versus last
year versus today?
Totally.
Yeah.
And you can do that with the sort of API to say, okay, here's my polygon of interest.
This is over my house or whatever, you know, fetch me all the images.
But a lot of satellite imagery, I mean, most of it is clouds.
It's just the earth is covered with clouds.
You're going to get a lot of clouds.
So there's also a metadata about the cloudiness.
So you can say, okay, well, give me these images over time, but I want the scenes to
be under 10% cloudy.
Right.
I'm willing for it to not be exactly 365 days apart, but maybe 350 because I get a clear
view if I do that.
Something like this.
Exactly.
And then you can make a little time lapse of how that area has changed over time.
And in fact, I think there was somebody who actually demoed a time lapse, a similar type
of time lapse, just grabbing the satellite imagery and turning it into a video over an area.
I forget.
Very neat.
That was.
Yeah.
That one, the Sentinel, the large one, the revisit time is every five days.
That's a lot of data.
That was a, yep.
Yeah.
It ends up a lot of data.
A lot of clouds in the cloud.
Yeah.
What about some of these other ones here?
The DayMet, which is gridded estimates of weather parameters in North America.
That's pretty interesting.
Yeah.
So DayMet's actually an example of a lot of our data is geospatial, like satellite imagery
or things that are derived from that, like elevation data sets where you're using the
imagery to figure out how, what's the elevation of the land or things like land cover data sets.
So if you scroll down just a tad, the land cover data set there, that's based off Sentinel, actually.
And so there's a saying, you know, for every pixel in Sentinel, they took like a mosaic over
a year.
What is the, that pixel being used for?
Is it water, trees, buildings, roads, things like that.
So those are examples based off of, of satellite imagery or aerial photography.
And then DayMet's an example of something that's like the output of a climate or a weather model.
So these are typically higher dimensional.
You're going to have things like, you know, temperature or maximum, minimum temperature,
water, pressure, vapor, all sorts of things that are stored in this really, you know, big
in dimensional cube at various coordinates.
So latitude, longitude, time, maybe height above.
So those are stored in typically in formats like Czar, which is this cloud native, very
friendly to object storage way of storing chunked in dimensional arrays.
Is it like, like streaming friendly?
You can stream part of it and seek into it, that kind of thing.
Exactly.
And all the metadata is consolidated.
So you can load in the whole data set and like, you know, less than a few hundred milliseconds,
but then access a specific subset very efficiently.
Sure.
Yeah.
Very neat.
Another one that's not directly based off of satellites is the high resolution electricity
access, I'm guessing.
And I guess you could sort of approximate it from lights, but is it, do you think it's light?
I think it is from Rod Dino.
Yeah.
So I think it's from.
Yeah.
It's V-I-I-R-S.
Satellite.
Okay.
So it is off of basically just steady and light.
Interesting.
And we have a few more that are coming online shortly, which are kind of more tabular.
So there's things like U.S.
Census gives you like the polygon.
So, you know, the state of Iowa has these counties or census blocks, which are this shape.
So giving you all those shapes and it has this population, things like that.
Things like GBIF has, which is, I think on there now, has occurrences of like, I think
they're like observations of somebody spotted this animal or plant at this latitude, longitude
at this time, things like that.
So lots of different types of data.
A mink was spotted running through the streets.
Okay.
Yeah.
Oh, you have one for agriculture.
That's pretty interesting.
If you're doing something with agriculture and farming and trying to do ML against that.
That's interesting because that's actually run by the National Agriculture.
That's actually aerial imagery, RGB, red, green, blue, and then also infrared aerial imagery
that's collected every about every three years.
So that's an example of high resolution imagery that's, you know, more than a, I think it's
one meter resolution.
Yeah.
You can see the little trees and stuff.
Exactly.
It's very, very accurate.
Yeah.
Great data set specific to the US.
So again, like Sentinel-2 is global in scope, but if you are doing things in the United States,
NAEP is a great data set to use.
Yeah.
You've got the USGS 3D elevation for topology.
That's cool.
And then you have some additional data sets.
What's the difference between the main ones and these additional ones?
Why are they separated?
We're catching up to where our stack API has all of the data sets we host, but the AI for
Earth program, which hosts all these data sets has been going on since 2017.
So there's plenty of data sets that they've been hosting that haven't yet made their way
into the API.
And that's, you know, just because we're getting there.
It's a bunch of work.
I see.
So for these additional ones, maybe I could directly access them out of blob storage, but
I can't ask API questions.
Exactly.
Okay.
And then another point, which is kind of interesting talking back to the tabular data is that some
of these data formats aren't quite, I mean, rasters and imagery is like fits really nicely
in stack.
And we know how to do spatial temporal queries over them.
But some of these data formats, you know, they're not as mature as maybe the raster data format, or it's not as clear how to host them in a cloud optimized format and then host them in a spatial temporal API.
So we're actually having to do work to say, okay, what are the standards?
Is it like geo parquet?
Or, you know, what are the formats that we're going to be using and hosting these data sets?
And then how do we actually index the metadata through the API?
So there's a lot of sort of data format and specification, metadata specification work before we can actually host all of these in the API.
Yeah, really nice.
A lot of good data here and quite large.
Let's talk about the ETL for just a minute because you threw out some crazy numbers there.
We're looking at the Sentinel-2 data and it gets refreshed every five days and it's the Earth.
Talk us through what has to happen there.
Yeah.
So for the Sentinel, it's actually, you know, daily.
So it's passive satellite collections.
So the satellites are just always monitoring, always grabbing new imagery.
And so that comes off to ground stations through the European Space Agency.
And then we have some partners who are taking that, converting it to the Cloud Optimus GeoTiff format, putting it on blob storage, at which point we run our ingest pipelines, look for new imagery, extract the stack metadata, insert that into the database.
And we just have that running in an Azure service called Azure Batch, which allows us to run parallel tasks on clusters that can auto scale.
So if we're doing an ingest of a dataset for the first time, there's going to be a lot of files to process and we can scale that up.
And it runs Docker containers.
So we just have a project that, you know, defines the Docker commands that can run.
And then we can submit tasks for chunks of the files that we are processing.
That creates the stack items.
And then another separate process actually takes the stack items and inserts it into the database.
That's cool.
So it's a little bit like data driven rather than a little bit like Azure Functions or AWS Lambda.
But processing, we just got to get all this data and just work through it kind of at scale.
Interesting.
Yeah, for sure.
Right now it's a little bit.
We're still building the plane as we're flying it.
But the next iteration is actually going to be a lot more reactive and based on another Azure service called Event Grid, where you can get notifications of new blobs going into storage and then put messages into queues that can then turn into these Azure Batch tasks that are running.
Right.
I see.
So you just get something that drops it in the blob storage and it kicks off everything from there and you don't have to worry about it.
Yep.
And then we publish those to, you know, our users saying, hey, this is ready now.
If they subscribe to that, that Event Grid topic.
Oh, that's cool.
There's a way to get notified of refreshes and things like that.
Not yet.
We're hoping to get that end of year.
But yeah, the idea is that we would have basically a live feed of new imagery.
I mean, what I would really like to see just for myself, my own interest is like,
to be able to have my areas of interest and then just go to a page that shows like almost an Instagram feed of Sentinel images over that area.
It's like, oh, this new one, it's not cloudy.
Look at that.
Look at that one.
You know, it's something I'm monitoring.
But yeah, generally we'll be publishing new stack items so that if you're running AI models off of the imagery as it comes in, you can do that processing based off of events.
Yeah, that'd be cool.
I'm only interested in Greenland.
I don't care if you've updated Arizona or not.
Just tell me if Greenland has changed.
Then I'm going to rerun my model on it or something.
Right.
Cool.
All right.
Let's see.
So that's the data part, data catalog.
And then we have the API and the hub, which I want to get to.
But I kind of want to just sort of put some perspective on what people have been doing with this, some of your partner stuff under the applications thing.
So, Tom, which ones do you think we should highlight from those that are interesting?
We kind of talked about the land cover data set.
So we worked with Impact Observatory to do that.
And so we had some tips about how to use Azure Batch because that's a very big Azure Batch job to generate that land cover map.
So pulling down the Sentinel data that we're hosting and then running their model over it.
So that was a fun data set to see come together and then use now.
The Carbon Plan, Carbon Monitoring Risk Assessment application.
That's like a really cool.
It's a cool like JavaScript application that you can view risks on.
So these companies are buying like carbon offsets that are forest trees that are planted to offset carbon.
But there's a problem that, you know, that we know about now is like the wildfires are burning down those some of those forests.
And so.
Right.
It doesn't help if you planted a bunch of trees to offset your carbon if they go up and smoke, right?
Right.
Yeah.
So Carbon Plan did a bunch of research, first of all, on essentially they did the research before that before our hub existed.
But, you know, we were working with these community members to and they have a very similar setup to what we have now to do the research, to, you know, train the models and all of that.
That goes into this visualization here of how likely, you know, what are the different risks for each plot of land in the U.S.?
Yeah.
So that was a great collaboration there.
One of the things I was wondering when I was looking at these is you all are hosting this large amounts of data and you're offering compute to study them.
How does something like Carbon Plan take that data and build this seemingly independent website?
Yeah.
Does that run directly on that data or do they like export some stuff and then run it on their side or what's the story?
They would have been doing all like the heavy duty compute ahead of time to train the models and everything to gather the statistics necessary to power this.
So then at that point, it's just a static JavaScript application just running in your browser now.
Oh, interesting.
And I think that's a good point because it's running against our data, but it's running in their own infrastructure.
Right.
So it's sort of on the planetary computer, but it's like really in this case, like using the planetary computer data sets in sort of a production setting that an infrastructure that they own, which is a use case we really want to support.
If, you know, if they need to use search in order to find the images that they need, they can use our stack APIs.
But really, it's like just an application running in Azure that, you know, in certain cases with our grants program, we'll end up supporting and sponsoring Azure subscriptions to run this type of infrastructure.
But at the end of the day, it's really just applications running in the cloud.
Right.
It's just better if that it's in Azure that it's nearby, but they could run it anywhere technically.
Right.
And just get signed blob storage access or whatever.
Yeah.
We'll throttle access at a certain point if you're trying to egress too much.
But yeah, I can imagine.
Yeah.
Very cool.
I can come over here and zoom in on Portland and it looks like we're in a decent bit of greenness still.
It does rain up here a lot.
Same for Seattle.
Yeah.
Quite cool.
You talked about this grant program.
What's the story that people out there listening, they're like, I want to get into working with this data and building things.
Grant might sound good to them.
What is that?
Awesome.
Yeah.
Look up AI for Earth grants.
We have rounds of supporting folks that are doing environmental sustainability work.
And there's a sort of a range of grant rewards.
The lowest level is like giving Azure credits.
You know, being able to sponsor an account or sponsor resources for applications that are being developed or research that's being done for environmental sustainability.
And we have folks running the grants program and go take the applications.
And there's different classes that we have and summits for each of the classes.
And then there's more involved grants and larger grants as usually as people sort of show progress, we can we actually can end up bringing additional resources or, you know, paid projects to accomplish specific goals.
But yeah, if anybody's out there and they're doing work in environmental sustainability that could benefit from the cloud, we'd love to work with you.
Well, just to clarify.
So there's, you know, the grants are great for like if you have like a complex deployment that's using a ton of Azure services and you want to like integrate this all together and use the planetary computer data, then the grants are a great approach.
If you're just like an individual researcher, a team of researchers or whoever who wants to use this data, the data is there.
It's publicly accessible.
And if you need a place to compute from that's in Azure, so close to the data and you don't already have an Azure subscription, then you can sign up for a planetary computer account.
And so like that's a way lower bar of barrier to entry.
There is you just sign up for an account, you get approved by us and then you're off to the races.
That's a great point.
If you think you need a grant to if you think you need a grant to use the cloud, try using the planetary computer first because you might not.
Yeah, very good.
Talk Python to me is partially supported by our training courses.
When you need to learn something new, whether it's foundational Python, advanced topics like async or web apps and web APIs, be sure to check out our over 200 hours of courses at Talk Python.
And if your company is considering how they'll get up to speed on Python, please recommend they give our content a look.
Thanks.
So what's the business model around this?
Is there going to be a fee for it?
Is there some free level?
Is it always free, but restricted how you can use it?
Because right now it's in like a private beta, right?
I can come down and request access to it.
Yeah, it's a preview.
We're still like getting access by requiring requests.
And, you know, there's a larger number of requests we're approving over time.
We're still coming up with the eventual final sort of target.
Most likely it will be some sort of limits around what you can do as far as compute, as far as data storage, once we have features around that.
And with clear offboarding of like if you're an enterprise organization that wants to utilize this technology, there should be paid services that allow you to just as easily do it as you're doing on the planetary computer.
But if you're doing low usage use cases or if your use cases is super environmental sustainability focused and you apply for a grant, we could end up, you're still using a paid service, but we're covering those costs through our grants program.
So we're still figuring that out as far as, you know, we're not, I don't see this as something that we're, you know, trying to turn into a paid service necessarily.
I think that there's a number of, you know, enterprise level services that could end up, you know, looking a lot like the planetary computer.
But really, we want to continue to support uses, particularly for environmental sustainability use cases through this avenue.
Yeah.
One of the nice things about our overall approach is since we're so invested in the open source side of things is if we're, you know, you might have requested an account a while ago and we're like very slowly going through them because there's just like so much to do.
But if we're too slow approving your account, then you can replicate the hub in your own Azure subscription.
If we're blocking you or if your needs are just like so vastly beyond what we can provide within this one subscription, then you can go ahead and do your own setup on Azure and get access to our data from your own subscription.
Right.
Because the blob storage is public, right?
Exactly.
Yep.
Okay.
Yeah.
Very nice.
Maybe the next two things to talk about are the API and the hub, but I think maybe those would be good to see together.
What do you think?
Yeah.
Yeah, definitely.
I think I'll let you talk us through some scenarios here, Tom.
Cool.
Yeah.
So I'm, in this case, I've, you know, logged into the hub here.
So I've, this is a, yeah.
Before you go further, there is a choice you get when you go there.
You've got an account and you click start my notebook up.
Yeah.
It's actually going to fire up a machine and it gives you four choices, right?
Python with four cores and 32 gigs of memory and a Pangeo notebook.
It gives you R with eight cores and R geospatial and GPU PyTorch, as well as UGIS, which I don't really know what that is.
Maybe tell us about getting started.
Yeah.
Got it.
So this is a Jupyter Hub deployment.
So Jupyter Hub's this really nice project.
I think it came out of UC Berkeley when they were kind of teaching classes, data science courses to like thousands of students at once.
And, you know, even with like Condor or whatever, you don't want to be trying to manage a thousand students, Condor installations or whatever.
So that's just a nightmare.
So they had this kind of cloud-based setup where you just log in with your credentials or whatever, and you get access to a compute environment to do your homework in that case or do your geospatial data analysis in this case.
And so this kind of, you mentioned Pangeo, this is this ecosystem of geobusinesses, geoscientists who are trying to do scalable geoscience on the cloud that Anaconda was involved with.
And so they kind of pioneered this concept of a Jupyter Hub deployment on Kubernetes that's tied to Dask.
So you can create, easily get a single node compute environment here, in this case, he's in the Python environment, or multiple nodes, a cluster of machines to do your analysis using Dask and Dask gateway.
Yeah, it's just a Kubernetes-based computing environment.
That's cool.
And I noticed right away the Dask integration, which is good for like this massive amount of data, right?
Because it allows you to scale across machines or, you know, more stream data where you don't have enough to store it, memory and things like that.
Yeah, exactly.
So this is a great thing that we get for Python.
So Dask is Python-specific.
We do have the other environments like R for, if you're doing geospatial in R, which there's a lot of really great libraries there, that's an option.
That is, unfortunately, single node.
There's not really a Dask equivalent there.
But there's some cool stuff that's being worked on, like multi-D plier and things like that.
Cool.
And if people haven't seen Dask running in Jupyter Notebook, there's the whole cluster visualization.
And the sort of progress computation stuff is super neat to see it go.
Yeah.
So it's when you're doing these distributed computations, it's really key to have an understanding of what your cluster's up to.
It's just crucial to be able to have that information there.
And then the example code that you've got there, the cloudless Mosaic Sentinel-2 notebook, it just has basic create me a cluster in Dask, get the client, create four to 24 workers, and then off it goes, right?
Yeah, exactly.
What is the limits and how does that work, right?
As part of getting an account on there, you get access to this cluster?
Yep.
So this is the first thing that we've talked about today that does require an account.
So the hub requires an account, but accessing the Stack API, which we'll see in a second, and even downloading the data does not require an account.
You can just do that anonymously.
Yeah, and in this case, I think the limit's like a thousand cores, something like that, some memory limit as well.
So that's the limit that you're into there.
So you can get quite a bit out of this.
That's real computing right there.
Yeah, definitely.
And in this case, we're using Dask's adaptive mode.
So we're saying, right now there's nothing to do.
It's just sitting around idly.
So I have three or four workers.
But once I start to actually do a computation that's using Dask, it'll automatically scale up in the background, which is a neat feature of Dask.
Yeah, and so the basic computation, the problem that we're trying to do here is we have some area of interest, which I think is over Redmond, Washington, Microsoft headquarters, which we're defining as this.
It's an exact square, yeah.
Yeah, maybe.
Heck.
Yeah, some sort of, I think it's a square polygon.
Anyway, we draw that out and then we say, okay, give me all of the Sentinel-2 items that cover that area.
So again, back to what we were talking about at the start is like, if you just had files in blob storage, that'd be extremely difficult to do.
But thanks to this nice stack API, which we can connect to here at planetarycomputer.micsop.com, we're able to quickly say, hey, give me all the images from 2016 to 2020 from Sentinel that cover, that intersect with our area of interest here.
And we're even throwing in a query here saying, hey, I only want scenes where the cloud cover is less than 25%, according to the metadata.
Very likely summer in Seattle because the winter, not so much.
Much fewer.
Yeah, much fewer.
So, you know, quickly within a second or two, we get back the 138 scenes items out of the, I don't know how many there are in total, but like hundreds of thousands, millions of individual stack items that comprise.
20 million?
20 million.
Okay.
That comprise Sentinel-2.
So we're quickly able to filter that down.
Next up, we have a bit of signing.
So this is that bit that we talked about where you can do all this anonymous, but in order to actually access the data, we have you sign the items, which basically appends this little token to the URLs.
And then at that point, they can be opened up by any geospatial program like QGIS or...
Right.
It converts a private block storage URL to a temporary public one.
Yep.
Exactly.
So you do that.
It's just like this kind of incidental happenstance that stack and dask actually pair extremely nicely.
If you think about dask, the way it operates is it's all about lazily operating, lazily constructing a task graph of computations.
And then at the end of your whatever you're doing, computing that all at once.
That just gives really nice rooms for optimizations and maximizing parallelization wherever possible.
The thing about geospatial is, again, if you didn't have stack, you'd have to open up these files to understand where on earth is it?
What latitude, longitude does it cover?
Right.
You have to open up all 20 million files and then look and see what its metadata says in it, right?
Yeah.
Okay.
And in this case, we have like 138 times three files.
Those 600, you know, whatever, 450, 600 items files here.
You know, each opening, each one of those takes a few, maybe 200, 400, 500 milliseconds.
So it's not awful, but it's like too slow to really do interactively on any scale of any large number of stack items.
So that's where it stacks great.
It has all the metadata.
So we know that this TIFF file, this cloud-optimized geotiff file that contains the actual data, we know exactly where it is on earth.
What latitude, longitude it covers, what time period it covers, what asset it actually represents wavelength.
So we're able to very quickly stack these together into this X-ray data array.
That's in this case, it's fairly small since we've chopped it down.
If we leave out the filtering, it'd be much, much larger because these are really large scenes.
But anyway, we're able to really quickly generate these data arrays.
And then using Dask, using our Dask cluster, we can actually load those, persist those in distributed memory on all the workers on our cluster.
So that's like very easy.
It's like a few lines of code, a single function call, but it represents years of effort to build up these stack specification and all the metadata and then the integration into Dask.
So it's just a fantastic, fantastic result that we have.
Yeah, and it's super cool.
Once you just call data.persist on the Dask array, you can just see in the dashboard of Dask, like all these clusters firing up and all this data getting processed.
Yeah, very neat.
Yeah, exactly.
So in this case, since we have that adaptive mode, we'll see additional workers come online here.
As we start to stress the cluster, it's saying, oh, I've got a bunch of unfinished tasks.
I should bring online some more workers.
And that'll take either a few seconds if there's empty space on our cluster or a bit longer.
Yeah, I feel like with this, if it just sat there and said, it's going to take two minutes and just spun with a little star, the Jupyter star, that would be boring.
It has this cool animated little dashboard like, oh, I'm going to just watch it go.
Look at it go.
No.
It's kind of like defragging your hard dive with the old days.
It's just you watch these little bars go across.
It's very bizarre.
Bizarrely satisfying.
Yeah, I will definitely just spend some time sitting here watching it.
Essentially, like monitoring.
There's like a lot of communication here.
There shouldn't be, but really, I'm just watching the lines move.
While the thing is working, let me take a question from the live stream.
Sam Paria asks, can users bring their own data to this sort of processing or, you know, because you've got the data sets that you have.
Is there a way to bring other research data over?
Yeah.
So the answer now is like, yes, but you kind of have to do a lot of effort to get it there.
Like, so your own data, you probably, maybe you do have like your own stack API and database setup and all of that.
But that's publicly accessible or you have a token for.
So most users don't already have that.
So you can't, this real divide between the data sets that we provide with our nice stack API and like your own custom data set.
That might be a pile of files and blob storage.
And you could access it that way, certainly.
But there's kind of a divide there.
So that is definitely something that we're interested in improving is making user data sets like that are private to you feel as nice to work with as our own public data sets.
Yeah.
Another thing that I saw when I was looking through, it said, under the data sets available, it says, or if you have your own data and you'd like to contribute, contact us.
And that's a slightly different question than they were just asking.
That's the one question was, well, I have my own data.
I want to use it.
This is like how I've, I work at a university or something.
I've got all this data.
I want to make it available to the world.
What's the story of that?
We have a backlog of data sets that we're onboarding onto Azure Blob Storage and then importing into the API.
We're still working through that backlog, but always on the lookout for good data sets that have real use cases in environmental sustainability.
If there's, you know, a group that's doing some research or doing building applications that have environmental sustainability impact and they need a data set, that certainly bumps it up on our list.
So, yeah, I would love to hear from anybody that has data sets that you're looking to expose publicly, hosts on the Planetary computer for anybody to use and need a place to host it.
Yeah, very cool.
All right, Tom, your graph stopped moving around.
It might be done.
Yeah, so we spent quite a while loading up the data and then that's like, yeah, just how it goes.
You spend a bunch of time loading up data and then once it's in memory, computations tend to be pretty quick.
So in this case, we're taking a median over time.
Is this the median of the image?
What is that a median of?
Yeah, so right now we have for like a list of numbers.
I'm not sure what it means for an image.
Yeah, so this is a median over time.
So our stack here, our data arrays, a four-dimensional array and the dimensions are time, first of all.
So we had like 138 time slices, wavelength.
So these, you know, red, green, blue, near-infrared, Sentinel captures like 10 or 12 wavelengths and then latitude and longitude.
So we took the median over time.
And the idea here is that like stuff like roads and mountains and forests tend not to move over time.
They're static relatively compared to something like clouds.
So again, clouds are always a problem.
And once you take the median over time, you kind of get like the average image over this entire time period, which turns out to be an image that doesn't have too many clouds in it.
Yeah, it might have no clouds because if you kind of averaged them out across all of them because you already filtered it down pretty low.
Yeah.
So now we can see a picture of the Seattle area where it's a cloud-free composite or a cloudless mosaic.
Yeah, beautiful.
Looks like you got maybe that's, what is that, Lake Washington and you got Rainier there and all sorts of good stuff.
Yeah.
I'm sure I actually do not know the geography of that well, but I have been looking at lots of pictures.
We tend to use this as our example area a lot.
Yeah.
Super cool.
Anyway, and one nice thing here is like, so we're, again, investing heavily in open source, investing in building off of open source.
So we have like all the power of Xarray to use.
Xarray is this like very general purpose, in-dimensional array computing library.
It kind of combines the best of NumPy and pandas.
In this case, you know, we can do something like group by.
So if you're familiar with pandas, you're familiar with group bys, we can group by a time dot month.
So I want to do like a monthly mosaic.
Maybe I don't want to combine images from January, which might have snow in them with images from July, which wouldn't have as much.
So I can do a.
So you'll get like 12 different images or something like that.
Here's what it kind of averaged out to be in February.
Exactly.
And so now we have a stack of images, 12 of them, and we can go ahead and representing a median.
So we have multiple years and we group all of the ones from January together and take the median of those.
And then we get a nice little group of cloud free mosaics here, one for each month.
Yeah.
And sure enough, there is a little less snow around Rainier in the summer than in the winter as you would, you know, the cascades.
Yep, definitely.
So that's like a fun little introductory example to what the hub gives you.
It gets you the single node environment, which that alone is quite a bit.
You know, you don't have to mess with, you know, fighting to get like the right set of libraries installed, which can be especially challenging when you're interfacing with like the C and C++ libraries like GDAL.
So that environment is all set up, mostly compatible, should all work for you on a single node.
And then if you do have these larger computations, you know, we saw it took a decent while to load the data, even with these fast interop between the storage machines and the compute machines in the same Azure region.
But you can scale that out on enough machines that your computations complete in a reasonable amount of time.
Yeah.
And because the animations, I don't need, you don't even mind.
Now it's, it's super cool.
So you use the API to really narrow it down from 20 million to like 150 or 138 images and then keep, keep running on.
So one thing that I was wondering when I was looking at this is, you know, what libraries come included that I can import and which ones, you know, if there's something that's not there, maybe I really want to use HTTPX and you only have requests or whatever.
Like, is there a way to get additional libraries and packages and stuff in there?
We do have a focus on geospatial.
So that's like, we'll have most of that there already.
So, you know, XRA, Dask, Rasterio and all those things.
But if there is something there, our container.
So these are all Docker images built from Conda environments.
That's all comes from this repository, Microsoft slash planetary computer containers.
So if you just, you know, you want HTTPX, you add it to the environment.yaml and we'll get a new image built and then available from the planetary computer.
And so these are public images.
They're just on the Microsoft container registry.
So if you want to, you know, we use our image, like you don't want to fight with getting a compatible version of, say, PyTorch and lib JPEG.
Not that I was doing that recently, but if you want to, if you want to avoid that pain, then you can just use our images locally, like from your laptop.
And you can even like connect to our Dask gateway using our images from your local laptop and do like some really fun setups there.
Yeah, I see.
Because most of the work would be happening in the clusters, the Dask clusters, not locally anyway.
Yeah.
So all the compute happens there and then you bring back this little image that's your plot, your result.
Okay.
Yeah, very cool.
So how do I get mine in here?
Like I see the containers.
I see you have the last commit here.
Yeah.
So there's one per, one per.
Right now, honestly, the easiest way is to send me, you know, open up an issue and I'll take care of it for you.
I see.
Just because I haven't got this continuous deployment quite working out.
There's an environment, you know, DIMO file there.
That gets.
Oh, yeah.
So you go see, yeah, there's quite a few packages in here already.
Yep.
And those are just the ones we explicitly asked for.
And then all their dependencies get pulled into a lock file and then built into a Docker images.
And so this is building off a project from Pangeo, that group of geo scientists that I mentioned earlier, who have been struggling with this problem for several years now.
So they have a really nice Docker eyes set up.
Right.
And we're just building off that base image.
Cool.
Yeah.
Based on the Pangeo container.
Very cool.
Sam Aparia asks, how long is the temporary URL active for the signed URL, the blob storage?
So that actually depends on whether or not you're authenticated.
We have some controls to say the planetary computer hub requires access, but also you get an API token, which gives you a little bit longer lasting tokens.
But forget what the actual current expiries are.
If you use the planetary computer Python library, you just pip install planetary underscore computer and use that dot sign method, it will actually request a token.
And then as the token is going to expire, it requests a new token.
So it reaches the token and caches it.
But it should be long enough for actually pulling down the data files that we have available.
Because we're working against smaller cloud optimized formats.
There aren't these 100 gig files that you should have to pull down and need a single SAS token to last for a really long time.
So you can re-request if you need a new one as it expires.
And like I said, that library actually takes care of the logic for you there.
That's cool.
Yeah, very nice.
All right, guys.
Really good work with this.
And it seems like it's early days.
It seems like it's getting started.
There's probably going to be a lot more going on with this.
Yeah, for sure.
I'm going to go out on a limb and make a big prediction that understanding the climate and climate change is going to be more important, not less important in the future.
So I suspect that's also going to grow some interest.
It might.
You know, the new report at PCC is making some heavy predictions.
And, you know, within the decade, you know, we might reach, you know, plus 1.5 Celsius.
And, you know, we're already in it.
We're already feeling the effects.
And, you know, this is the data about our Earth.
And it's going to become more and more important as we mitigate and adapt to these effects.
So, yeah, I agree.
I think that's a good question.
Yeah, if we, thanks.
If we are going to plan our way out of it and plan for the future and, you know, science our way out of it, we're going to need stuff like this.
So, well done.
All right.
I think we're about out of time.
So let me ask you both the final two questions here.
If you're going to write some Python code, what editor do you use?
Rob?
VS Code.
I suspect I could guess that, but yeah.
Yeah, actually, I was a big Emacs user.
And then when I got this job, switched over to VS Code.
It just integrated better with Windows.
And then really got into the PyLance and the typing system, you know, doing type annotations and basically having a compiler for the Python code.
Like really a change.
Instead of having all of the types in my head and having to like worry about all that, actually having the type hinting.
Yeah.
It was something I wasn't doing a year ago.
And now it's like drastically improved my development experience.
It's a huge difference.
Yeah.
And I'm all about that as well.
People talk about the types being super important for things like mypy and other stuff.
And, you know, in a lot of cases it can be.
But to me, the primary use case is when I hit dot after a thing, I wanted to tell me what I can do.
And if I have to go to the documentation, then it's kind of like something is failing.
I shouldn't need documentation.
I should be able to just, you know, auto complete my way through the world mostly.
Totally.
And I come from a, I was a Scala developer for, you know, six, about six years.
So I was used to very heavy, heavily typed system and kind of got away with it from Python.
I was like, you know what?
I like that there's not types, but I feel like the Python ecosystem is really hitting that sweet spot of like introducing enough typing, which is really great.
And then the inference flies along for the rest of the program.
Yeah.
Totally.
Yeah.
All right, Tom, how about you?
VS Code as well for most stuff.
And then Emacs for Magit, Magit, the Git client, and then a bit of Vim every now and then.
Right on.
Very cool.
Yeah.
All right.
And then the other question is for either of you, there's like a cool, notable IPI or Conda package that you're like, oh, I came across this.
It was amazing.
People should know about it.
Any ideas?
Have you got one?
Sure.
I'll go for Seaborn.
It's a plotting library from Michael Wascom built on top of Matplotlib.
It's just really great for exploratory data analysis.
Easily create these great visualizations for mostly tabular data sets, but not exclusively.
Oh, that's interesting.
I knew Seaborn.
I knew Matplotlib.
I didn't realize that Seaborn was like, let's make Matplotlib easier.
Yeah.
Essentially, for this very specific use case, Matplotlib is extremely flexible, but there's a lot of boilerplate and Seaborn just wraps that all up nicely.
Yeah.
Super cool.
All right.
Well, thank you so much for being here.
Final call to action.
People want to get started with Microsoft Planetary Computer.
Maybe they've got some climate research.
What do they do?
Planetarycomputer.microsoft.com.
That'll get you anywhere you need to go.
And then if you want an account, then it's slash account slash request, I believe.
Yeah.
There's a big request access right at the top.
So you can click that.
Awesome.
Yeah, exactly.
All right.
Rob, Tom, thank you for being here.
And thanks for all the good work.
Thanks for having us.
This is great.
Awesome.
Thanks so much.
Yeah.
Bye.
See ya.
This has been another episode of Talk Python to Me.
Our guests on this episode were Rob Emanuel and Tom Augsberger.
And it's been brought to you by Shortcut, formerly Clubhouse I.O., us over at Talk Python Training,
and the transcripts were brought to you by Assembly AI.
Choose Shortcut, formerly Clubhouse I.O., for tracking all of your project's work.
Because you shouldn't have to project manage your project management.
Visit talkpython.fm/shortcut.
Do you need a great automatic speech-to-text API?
Get human-level accuracy in just a few lines of code.
Visit talkpython.fm/assembly AI.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at Talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show, open your favorite podcast app, and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play,
and the direct RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air,
be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
Thank you.