What's the biggest challenge facing human civilization right now?
Fake news, poverty, hunger, oppression?
Yes, all of these are huge problems right now.
But if climate change kicks in, you can bet that it'll amplify these problems and many more.
That's why it's critical that we get answers and fundamental models to help understand where we are,
where we're going, and how we can improve things.
On this episode, you'll meet Dr. Damien Irving.
He's a climate science researcher using Python to understand what the climate models are telling us.
This is Talk Python to Me, episode 134, recorded October 16, 2017.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy.
Follow me on Twitter, where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm, and follow the show on Twitter via at Talk Python.
Hey, everyone.
I'm super excited to bring you this climate science, climate change episode.
But before we do, I want to just catch you up really quick on my free MongoDB course that I talked about last week.
If you're looking to learn MongoDB, especially with Python, check out freemogadbcourse.com.
Just last week, over 5,000 people signed up and really enjoyed it.
So drop by the website, sign up, and check it out.
Now let's talk with Damien about climate research and Python.
Damien, welcome to Talk Python.
Thanks for having me.
Very happy to be on.
Yeah, I'm really excited about our topic that we've got lined up today, Python and climate science.
And I think there's just so many aspects to talk about that.
Well, we're going to cover a bunch of them, right?
We're going to cover a bunch of different things, the programming, the problems you're trying to solve,
the education problems in terms of educating data scientists, and much more than that, right?
Yeah, absolutely.
Hopefully, we can have a wide-ranging chat.
Yeah, I think that's the way we should definitely do this.
But before we get to that, let's start with your story.
How did you get into programming in Python?
Well, I mean, I did my undergraduate degree in science, majoring in meteorology, way back in 2008 now.
But that actually kind of involved very little programming, which sounds shocking,
but is actually pretty common for people who study science.
So I think the only programming course I did was a short summer intensive learning FORTRAN in the engineering department.
And that actually put me ahead of the curve and ahead of most undergraduate scientists.
And so I picked up a summer job.
Yeah, that's amazing.
Yeah, it's scary to think back.
Yeah, and then so I was able to pick up a job off the back of that,
basically doing some work for an assistant professor in the meteorology department
after I finished my undergraduate degree.
And he kind of sat me down in front of a command prompt,
pointed me towards Ferris, which is a scripting language used a bit in oceanography,
and kind of left me to it.
And I bumbled my way through that.
Nice.
Was that actually in FORTRAN, or what language was that in?
No, so Ferris is kind of a standalone language, I guess,
just for doing basic data analysis in oceanography.
So it's a fairly limited language, but it's good if you're dealing with oceanographic data.
But yeah, I kind of bumbled my way through that,
and then bumbled my way through using FORTRAN and things like that in an honors degree,
which is kind of a year-long research project you do after you're undergraduate in Australia.
Yeah, and that was my kind of introduction to programming.
And it's kind of a pretty typical scientist experience, kind of self-taught,
no formal training or education in programming.
Right.
Here's the problem to solve.
We think programming can help.
Here's some tools.
Yeah.
I have at it, right?
Go.
Yeah.
But then, actually, after my honors year, I was lucky enough to get a job at CSIRO,
which is a national science organization in Australia.
And it was kind of a half research, half support scientist role.
And in my support science role, I got to spend a lot of time with IT people who basically introduced me to Python.
And that's kind of where my path kind of diverged from typical.
I got to spend a lot more time with IT nerds than your typical scientist would.
So I was kind of really lucky in that respect.
And they pushed me onto Python.
Thank goodness.
Yeah.
What is this language?
Nobody knows what you're doing.
Go learn Python.
Yeah.
Nice.
And so nowadays, what do you do?
Yeah.
So today, nowadays, I'm a postdoctoral research fellow at CSIRO.
So I'm back there after my PhD, which is nice.
That's cool.
What was your PhD in?
My PhD was in waves in the upper atmosphere and how they affect the weather down at the surface.
Okay.
Yeah.
I didn't even know there were waves in the upper atmosphere.
That's awesome.
Yeah.
It's kind of, I mean, the atmosphere is kind of a fluid.
You kind of, I guess you don't think of it like a fluid like you would an ocean, but it flows the same way as a fluid and stuff.
So there are waves, just like if you drop a stone into a pond and they ripple and things like that.
So looking at those waves in the atmosphere.
Okay.
Very cool.
Yeah.
But yeah, so the work involves, my work right now involves looking at climate model data.
There's probably about 40 or so modeling groups around the world that have computer climate models.
And periodically, every five or 10 years or so, they all perform sets of common experiments.
So they'll have worlds with humans in them and worlds without humans in them and worlds with different levels of greenhouse gases and all those things.
And they'll make all that data available.
And the entire data set is, you know, multiple petabytes in size.
But individual researchers like me typically only require, you know, a subset of the data set.
And so all that data in Australia is at a national computing facility.
And rather than have people download all that data to their own institution, which would obviously be impractical, they build a lot of, I guess, analysis infrastructure on top of the data.
So you log in remotely.
I see.
That's cool.
So there's so much data that it's in one place and you send the code and analysis to the data rather than the other way, huh?
Yeah.
So they have all the analysis infrastructure on top of the data so you don't have to move it anywhere.
So I live in Hobart in Australia, but I spend my days on a computer that sits on top of all the data in Canberra.
Yeah.
So that's kind of what I spend my days doing.
That's awesome.
That sounds really fun.
That's an incredible amount of data.
How much of what you work with, is it simulation?
And how much is it analyzing observational type data?
Yeah, for me, it's pretty much all simulation.
So it's all scenarios of the future world to look at climate change.
Or it's kind of reruns of the past 150 years with various elements taken out.
So humans might be removed or different aspects to kind of figure out what's caused what, if you like, in the observed climate.
I see.
What if we never ate beef or something like this, right?
Never raised cattle?
Yeah.
You can have models that have different kind of treatments of what's happening at the surface in terms of vegetation.
You know, yeah, livestock and stuff like that.
Yeah, farming practices.
Yeah.
So I deal almost exclusively in fictitious worlds, but there are plenty of people who deal with actual observations.
Yeah, and I'm sure that there's a really important place for both.
So maybe that's a good place to start.
Let's talk about the types of problems that you guys are trying to solve with Python.
Maybe some of the modeling and how that goes, things like this.
I guess most problems that climate scientists are trying to solve when they're using Python is that
someone's run one of these models, either a fictitious kind of world like I'm looking at,
or ones where a whole bunch of observations have been thrown in and the model is used to fill the gaps, basically.
So obviously, we don't observe temperature and rainfall and all those things everywhere.
So models can be used to kind of fill in the gaps between the observations using the observations as kind of a ground truth.
And so either way, whichever type of model you're using, it'll output large multidimensional arrays of data.
So they'll have a time axis, a latitude axis, a latitude axis, and a depth or a height altitude axis.
So large multidimensional arrays.
And then it's basically trying to draw insights from that data about how the climate system works, really.
So a lot of kind of time series analysis on those large arrays, other statistical analysis.
And just a lot of the time, it's just the fundamental task of actually visualizing what the model is simulating.
So actually visualizing the temperature data or the humidity data or whatever output it is, just what is this model doing?
And a lot of the challenge is just seeing it.
Right.
I can imagine, right?
Like once you've got so much data, just great ways to visualize it so you actually understand because that's a lot of axes.
Yeah.
It seems like a simple thing, but actually just seeing what a model is simulating is quite a challenge at times.
Yeah.
So do you feel like there's a lot of pandas, numpy, matplotlib, Jupyter Notebooks, those types of things?
Yeah, definitely numpy or libraries built on top of that for multidimensional arrays.
Pandas and stuff would be probably more common among, I guess, a lot of scientists alike me and have model data that comes out on these nice grids.
And then you have, I guess, the other scientists who is kind of, they're more GIS problems.
They've been out on a research vessel and they've dropped a temperature probe down at one point.
And then 100 nautical miles later, they drop another one and then another one.
So then they have this kind of unstructured spatial data.
I mean, it becomes much more of a GIS pandas kind of problem.
Right.
Because when you simulate the data, you don't have to clean it up as much.
No, it becomes fairly nice.
And the grid is, you know, all the latitudes and longitudes are perfectly spaced.
And it's this very nice regular grid.
Whereas if you're more of an observational scientist and you're just taking observations wherever you can, you have to clean it.
You have to get it on a grid that's a bit nicer.
And so, yeah, I kind of, I think my lucky stars some days that I'm not in that space, but they don't have the ridiculous amount of data that I've got to deal with.
So maybe, maybe there's some nice aspects to it.
Yeah, of course.
Of course.
That's pretty interesting.
And so for the simulations, I suspect that you have Python in action there, but it's not, is it all in Python or is it C++ or what's the mix there?
Most climate models themselves are written in Fortran.
So mostly you're using Python for the post-processing.
So analyzing the data once it's come out of the model.
And this, I guess, one reason most of them are in Fortran is kind of legacy issues that most climate models out there have been developed for decades.
So when they started, Fortran was the only thing available.
And the other one, I guess, is, well, I guess from a speed point of view, you know, with Siphon and things like that these days, you probably could do Python, write a climate model in Python.
And I think some very simple models written in Python do exist.
But in general, yeah, the models are written in Fortran and the post-processing data analysis is done with Python.
Right.
Okay.
Yeah, that's pretty interesting.
It scares me a little to think of that much Fortran.
Oh, yeah.
But that's okay.
Yeah.
Yeah, when I started in – I studied engineering for a while and they said the most important language you're going to learn is Fortran.
And I said, please let me take C++.
They said, nope, you have to learn Fortran.
Then you go do these elective courses if you want.
Like, oh, okay.
I see how that's going to be.
But, yeah, it sounds like it really might be the case in climate science.
That's cool.
So you talked about, like, tremendous amounts of data.
Probably parallelism is, like, an important thing there, right?
A lot of the problems are embarrassingly parallel in that you want to just do the same process on every single latitude point or something like that.
So there's no kind of talking between analysis points.
Or you want to do the same process across lots of different climate models.
So, yeah, so some people will write, I guess, parallel code explicitly, if you like, using the multiprocessing Python library or something like that.
Or, increasingly, I guess, it's kind of built into the package they're using.
So Dask is, I'm sure, one that's been talked about on previous episodes of your podcast.
But that's kind of built into XArray, which is kind of a build-on of pandas for multidimensional arrays.
And so it kind of chunks your problem and does all this parallel scheduling and stuff for you.
So some people are doing that.
But sometimes I feel like it's a little bit like everyone talks about big data, but most people aren't actually in that space with the analysis they do.
And it's a little bit the same.
I think most people are kind of still in the medium data space where even if they're dealing with a big data set, they're only interested in a smaller subset.
And you can kind of get around the need for writing parallel code by just using the numpy functions that allow you to vectorize your problem rather than looping.
And a lot of the packages in climate science kind of do lazy loading where you can kind of load all the metadata about the data first and have a look at what's in there.
And then you can just actually load the data that you want instead of the whole thing at once.
Okay.
Yeah.
That sounds pretty interesting.
Do you leverage like GPUs, you know, the running on GPUs for computational stuff?
Any or is it mostly just straight CPU?
I think I've spoken to people who are using GPUs as well as CPUs.
So I know it can be done.
I haven't personally.
But yeah, some people do.
Yeah.
Okay.
That's cool.
Yeah.
There's probably even some computational bits you can plug in that like what you're actually doing could harness the power of GPUs maybe.
So all this data, is this like in a giant database?
Is it in a bunch of CSVs?
Where do you get your data from?
Most of the output from client models these days goes into a file format called NetCDF.
I think Network Common Data Form must be what it stands for.
So it's a subset of HDF5.
So it's a self-describing format.
So it carries all its kind of metadata with it, which is really cool.
Right.
And then in kind of weather, climate, ocean, science, there's a whole set of CF conventions they're called.
So climate and forecast conventions around what metadata you use in the file.
So how exactly do you describe the time axis?
Do you say days since YYY slash MM slash DD?
Yeah.
So all these things so that then when people build libraries in weather and climate science, they can assume this basic level of metadata compliance in the files, which means that they can write functions that really speed up your analysis because they can assume certain things about the data.
So yeah, NetCDF in this kind of metadata format is kind of ubiquitous through weather, ocean, climate, science now.
All right.
Yeah, that's interesting.
So it sounds like maybe if you've got this common format, like a bunch of different libraries and packages can read and understand it.
So what are the major packages that you might use in Python to work with climate science?
Probably these days, I mean, first off, you would be installing your environment using Conda for sure.
So a big headache used to be just getting all the non-Python dependencies installed.
So obviously there's a lot of our libraries that are C underneath or there's NetCDF libraries that have to be installed and things like that.
And they used to be a complete nightmare.
And so Conda has absolutely been a game changer.
So yeah, people are installing their stuff with Conda.
Yeah, and Conda gives you basically a virtual environment, but it delivers the packages pre-compiled in a binary form for your OS rather than via source like pip or something like that, right?
But it used to be that if you wanted to use a particular library that had C dependencies or something, you had to figure out how to install those C dependencies yourself as well as pip installing it.
But now it's just, yeah, you go, Conda install, one line, it's all done.
It's amazing.
But yeah, so basically in terms of the main libraries that get used, so I mentioned XArray before.
So basically that library takes Pandas, which is obviously that labeled array concept for two-dimensional arrays and then expands it for multi-dimensional arrays.
And it was actually a climate scientist, Stephen Hoyer, who initially wrote it.
And now it's kind of been taken up by the broader PyData community.
But yeah, so that one is very popular.
And then the other one is the Met Office in the UK have written one called Iris, which is similar to XArray, except that XArray will let your files not be so kind of CF, net CDF metadata compliant.
So if you have files from a project that isn't very strict on their metadata and their net CDF files, it'll work just fine.
Whereas Iris really demands those things of you.
are very good about the metadata and stuff.
Because it's just like, you can do tasks slightly faster if you like, with less commands because it can make more assumptions of the input data.
Yeah, so I use Iris.
If you're going to draw results, you probably want to have some strictness on the data you're basing those results on, right?
Some people find it overly restrictive in that they want some files that don't have, you know, standard banner data type things.
So they'll go the XArray route.
But basically, if you're a climate scientist, you're kind of choosing between Iris or XArray for the bulk of your work.
So for input-output to net CDF files, for calculating basic statistical quantities and basic visualization and stuff like that,
it's all kind of, they're the core libraries you'll kind of leverage off.
This portion of Talk Python to me is brought to you by ParkMyCloud.
Every second your cloud servers are running is costing you money.
Cut your monthly cloud spend and stop paying for idle instances and VMs with ParkMyCloud,
a cloud cost management tool that turns off resources when you don't need them.
From their dashboard, automatically schedule your instances to be turned on or off,
saving you as much as 65% or more on your cloud spend.
Manage databases, auto-scaling groups, set up logical groups and servers
to turn off during nights and weekends when they're not in use.
Whether you're using AWS, Azure, or Google Cloud, it's easy to save money with ParkMyCloud.
Try ParkMyCloud and see why it's chosen by McDonald's, Capital One, Unilever, Fox, and more,
saving customers tens of thousands of dollars every month.
Visit talkpython.fm/park and cut the cost of your cloud today.
That's talkpython.fm/park.
It seems to me like the whole Python data science space in the last five, ten, definitely five years,
has gotten really polished and really blown up.
Probably it was a little harder to work with Python and stuff in the beginning, right?
When I started, yeah, obviously Condor didn't exist.
Iris and X-Array weren't there.
There was something a little bit like those that existed.
Even these days, they're taking it a step forward.
And I'm not sure if you've heard of the GeoViews or HoloViews library for visualization in Python?
Yeah, I don't think I've heard of that one.
Tell us about it.
So basically, I guess it's getting into that kind of declarative visualization space,
the idea being that for data exploration, you don't want to spend all your time kind of
tweaking the axes of a plot and deciding on exactly which type of plot you want to create
to look at this particular data.
You basically just want to tell the library the characteristics of your data and have it decide
what the best axes would be, what the best map projection would be, and all those things.
So there's kind of two major libraries getting into that space.
One's called HoloViews and the other one's called Altair.
So HoloViews is much more established, I guess.
And it can kind of use Matplotlib or Bokeh under the hood, depending on whether you want
like a static image or an interactive image.
Right.
That sounds great.
And you can maybe almost publish it to the web really easy with Bokeh, yeah?
Yeah.
So with HoloViews, you can definitely do that.
But anyway, so HoloViews doesn't have support for geographic plots of the type on a world map
of whatever kind of map projection you'd like, which is a common thing in climate science.
So the Met Office, again, have developed GeoViews on top of HoloViews.
And so basically, the idea being that you just throw your data at it, you give it a description
of the basic characteristics of your data, and it figures out the rest.
And you can have a static image, or you can have an interactive image that you can publish
to the web.
And this is really kind of taking down the barrier to kind of what I was talking about before,
of just that task of visualizing the data you've got quickly and easily.
Right.
Yeah.
That's awesome.
And the more the system can do it automatically, just look at it and go, well, it looks like
the axis should be this, and it should all do this.
Like, that's really great.
These type of things are just game-changing in terms of the amount of analysis you can get
done.
You know, you're not spending your days mucking around with axes and map projections.
It just happens.
And what used to take weeks now takes a couple of hours.
Oh, that's awesome.
So either now or going in the future, how do you see, like, machine learning and AI starting
to work its way into, like, analyzing climate data and making predictions and stuff?
There is some active stuff happening in that space.
I was at a conference last year at Lawrence Berkeley National Lab in San Francisco, and there
was a group there looking at applying machine learning to weather and climate type problems.
So I think it'll definitely happen.
I think at the moment they're in that kind of space where they're trying to figure out
which problems are kind of amenable to machine learning type solutions.
It's almost like it doesn't make sense to apply to everything, but there will be certain
questions within weather and climate science that could be answered much better by applying
that.
So it seemed like they have, if you like machine learning as a solution, and now they're trying
to find good questions to answer with it.
Yeah, there's a bit of that.
Yeah.
That was kind of interesting.
It was kind of, it was an interesting conference.
Oh, I'm sure.
I know that machine learning has been taught to do things like look at mammograms and predict
breast cancer as good or better than professionals, right?
At least under some circumstances.
So it feels like, you know, it's really good at looking at pictures and finding subtle, subtle
trends that people might miss.
And I'm just wondering, like, it seems like there's probably some good ways to use it to
understand things that are subtle, but then explode later in climate evolution.
Well, particularly like you could imagine it could look at a weather map and identify the
fronts or other weather features.
And it's just a question of, would it do that better than existing ways that we do it, where
you might look at a temperature field and you can look at the gradient, the temperature gradients
and identify it.
And it's just, the question is, would it actually do it better than we do it now?
Or yeah.
So that's kind of the, I think the question at the moment is figuring out what it would be
most useful for.
Sure.
I guess one of the real challenges is if you're trying, especially for prediction,
if you're trying to predict the future, you know, in the longterm, not, not like what's
it going to be like tomorrow in this town, but like, what's it going to be like in 20 years?
Machine learning is really good when you feed it lots of examples.
It was like this.
And the outcome was that it was like, is that, you know, like that, the breast cancer one
was given like a hundred thousand mammography scans.
And the answer, then it was asking more, right?
We only live from the past towards the future once, right?
Like we can't, we don't have a bunch of examples to feed it, right?
Yeah, you're right.
Which makes it difficult in a climate sense because there's only one realization of our
past climate.
Whereas the applications might be more in forecasting and stuff where a cold front has come across
Portland, Oregon hundreds and hundreds of times in the past.
So yeah.
So maybe there's more of a weather forecasting application.
Right.
That's interesting.
Not as cool as if we could answer major, major questions with machine learning, but who
knows?
Maybe someday someone will figure out a way to make it work, but speaking out of figuring
out how to make it work.
One of the challenges I suspect is all of this programming and these data tools and programming
tools are great, but you said your experience was you weren't giving tons of programming support
as a scientist, and that's probably true for many of them.
So what do we need to be doing to help like instill those right real programming skills?
I don't think the, the situation, I mean, I only graduated.
I graduated from my undergraduate degree back in 2008.
I don't think the situation has changed much since.
And so I think obviously software carpentry is a big one, a big organization that's kind
of helping with this.
And I think you've had Greg Wilson on a, on a previous episode talking a little bit about
software carpentry and everything they do.
Yeah.
I actually had Jonah Duckles on the podcast and that was back on episode 93.
And we talked about software carpentry, which is a group that basically works with scientists
around the world to help them become better programmers and better data scientists.
It's really cool.
So there's some aspect that with, with climate science stuff.
Is there a particular course for climate scientists or is it just the general stuff that software
carpentry teaches?
I'm actually working on some, I guess, climate specific stuff.
Basically the Australian Meteorological and Oceanographic Society, which is like the professional society
in Australia for weather ocean climate scientists have hosted a software carpentry workshop alongside
their annual conference for the last four years now.
And so I'm, I'm basically in the process of writing those material, those teaching materials
we've been using for that up into a, I guess, a more climate specific course that'll be hosted
with data carpentry, which is actually a sister sibling organization of software carpentry that
actually has discipline specific materials.
Yeah.
So if you're, if you're lucky enough to, I guess, to be a young scientist coming through who, who
has someone at your institution who's really into software carpentry or a professional society
like Amos who, who offer these types of things, then I guess your situation is, is a bit better
than, than mine was when I came through.
But, there's still, yeah, a long way to go.
And a lot of people who kind of slipped through the cracks and just kind of get lumped on a
research project with really not much assistance at all in terms of learning how to program,
which is, which is really sad on a lot of fronts, sad personally for them, but also in
terms of limiting the progress of their research, it's terrible.
Right.
I also suspect that there's a, a sharing component that's limited as well, right?
Like if, if you can write, go to MATLAB and write a script that will analyze something
and come up with an answer.
That's one thing.
If you could form it in a way and say in Python, in a way that is reusable and general and tested,
then you could put it out on GitHub and all sorts of people could use it and add to it.
And I think there's a pure scientific research sort of sharing the knowledge upside there as
well.
When you think about where, I guess the, the state of climate science is at the moment,
and I guess for lack of a better word that the computational literacy of the community,
you're basically at the point where you're just introducing people to, to say GitHub and
to having their own personal code under version control and things like that, to, to go the
next step and have, and have them writing code that the wider community can use.
And it's, you know, it's up on GitHub and people are submitting pull requests and, you know,
it's tested and there's continuous integration.
And all those things is just, it seems like there are exceptional, I guess, individuals
you like within, within the discipline who, who do those things, but they're very rare.
And so I think the discipline is another five or 10 years off kind of just having the, the
computational literacy, if you like to, to do those, those things that would, as you say,
make, make life so much easier for everyone.
Yeah, it would really be great, but that, that's definitely, those are some advanced ways of
working and if, especially if, if people don't get started early in that, right.
They, they don't build up those skills solely over time.
You know, they're busy solving real problems with science and computation and probably with
some thing like MATLAB or something.
Yeah, I can definitely see how that's, that's really, really quite challenging, but it's, I
think it's important for a lot of things.
So hopefully the software and data carpentry folks can keep that going.
That's great.
They're doing great things.
They're an amazing organization.
What else would you recommend to people out there who are scientists to level up,
their skills or to keep improving or whatever?
Really just encouraging people to, to kind of participate in the wider Python community.
Go to a PyCon conference or a SciPy conference or something like that, which, which seems obvious
to people who are developers in Python and stuff, but academics usually don't really think about,
you know, attending conferences outside of their research discipline.
Right.
They might go to a climate science conference and not PyCon, for example, right?
The PyCons that I've been to, there are a lot of kind of support staff there.
So the support staff at the institution that I work at or at the Bureau of Meteorology or
wherever it might be, they're all there, but the actual scientists who are doing a lot of
data science with Python aren't and it probably wouldn't occur to them.
I must say my, my first kind of PyCon really blew my mind in terms of compared to an academic
lecture, academic conference with like, you know, the recorded lectures that go up straight
away.
Even simple things like you could use your own laptop rather than having to give them a USB
and put it into their Microsoft Windows machine.
And people like doing things like live, like live coding on the screen and stuff, as opposed
to just a static PowerPoint presentation.
And it was kind of, it was all very mind blowing compared to an academic conference.
So I think, yeah, I'd really encourage people to kind of try and get actively involved in the
community in some way and really kind of broaden their horizons and keep their skills improving.
Yeah, I would totally second that.
I think especially the PyCon conference conferences, you know, pick your continent, maybe there's a lot of
science conversation and data science conversations going on there.
Like at PyCon US this year, the keynote speaker was Jake Vanderplass on the first day who opened it,
basically surveying all the different ways people use Python, how it's used in astrophysics,
how it's used in space telescopes and all sorts of things.
And it was very, I think it would actually be a really welcoming environment to people
who have, say, some programming skill and some programming ambition, but mostly are doing data analysis type stuff.
I think they should definitely check it out.
Yeah, for sure.
Even like in Australia, there isn't quite a critical mass to have a standalone SciPy conference.
But at PyCon Australia, there's always a data science track.
So you don't have to kind of sit there and, you know, listen to talks on web development and Django.
Yeah, exactly.
And stuff like that.
I don't care anymore about what Instagram did with Python.
I'm over it now, even if it was cool.
No, you can definitely have an almost a pure data science experience when you go to one.
Sure.
Oh, that's awesome.
Yeah, absolutely.
The hallway track of those conferences is great as well.
You know, just the people you meet doing similar stuff.
A lot of the times you don't have much in common with researchers once you get outside of your specific niche.
But you often have a lot in common with, in terms of the Python libraries you use and the types of basic data analysis you do.
So you end up, you find, oh, I have a lot in common with all these people who are from very different fields.
And that doesn't usually happen if you're just talking about your specific research discipline.
Yeah, I was really blown away at how many different people kind of solve similar problems with similar tools.
But, you know, the outcome is really different because of the questions they ask.
Cool.
So I definitely want to, yeah, encourage people to go to the local Python conferences.
They're really good.
Some of that software carpentry stuff we talked about, it sounds like it would be really beneficial to one of the major problems in science is the whole reproducibility thing.
Right.
The better you can get your code on GitHub, maybe create a Docker image that people can download and run exactly.
Like the more that you could share, distribute, and sort of save your computation.
It seems really valuable.
The reproducibility crisis, I guess, is a big one.
And it's, so I guess the central tenet is obviously if someone publishes some research that they describe their methods in such a way that someone else working in that field would be able to reproduce their results if they wanted to.
It turns out that most papers published these days aren't reproducible.
And it's for a variety of reasons.
Some of them are to do with experimental design, the availability of the data sets that were analyzed and things like that.
But a big one is computational reproducibility in that most papers don't make the code available that they wrote to do the analysis or the details of, you know, the software environment that that code was executed in.
So, yeah, the software carpentry skills and just the basic things about, you know, using version control and all those types of things are huge if we're ever going to actually get past the reproducibility crisis and get to a point where our research is truly reproducible.
Yeah, I definitely think that's something that's important.
I mean, the more that our research depends upon code and data, the more important it is that I think that that's accessible.
This portion of Talk Python to Me has been brought to you by Rollbar.
One of the frustrating things about being a developer is dealing with errors.
Relying on users to report errors, digging through log files, trying to debug issues, or getting millions of alerts just flooding your inbox and ruining your day.
With Rollbar's full-stack error monitoring, you get the context, insight, and control you need to find and fix bugs faster.
Adding Rollbar to your Python app is as easy as pip install Rollbar.
You can start tracking production errors and deployments in eight minutes or less.
Are you considering self-hosting tools for security or compliance reasons?
Then you should really check out Rollbar's compliant SaaS option.
Get advanced security features and meet compliance without the hassle of self-hosting, including HIPAA, ISO 27001, Privacy Shield, and more.
They'd love to give you a demo.
Give Rollbar a try today.
Go to talkpython.fm/Rollbar and check them out.
I know the guys at the Large Hadron Collider are doing some really interesting stuff with taking their code and putting it in an escrow thing.
It's not GitHub, but it's something kind of like GitHub where it's like we promise we won't change or delete this code because it's linked to by this paper.
Things like that.
It's been a while.
I forgot what the name was.
It was like two years ago when we talked about this.
But it sounds really important.
Some papers that I published recently, I mean, I give the link to GitHub of where the code is if people want to see, I guess, the latest version, if you like.
But then also, I guess, at the time of publication, you kind of take a snapshot, if you like, of your code repository at that point.
And there's websites.
There's one called FigShare and another one called Zenodo where you can put – they kind of call it the long tail of your research.
So things like code, supplementary figures, supplementary tables and stuff.
You put it all out there, it gets a DOI, a digital object identifier, and those –
Yeah, that's right.
And those websites guarantee that they're not going to disappear and it'll be around for, you know, all eternity for people to be able to get.
So, yeah.
So, in general, the best practice these days is definitely give people the link to GitHub so they can see the latest version, but also have a version up on persistent storage place like FigShare or Zenodo.
So if you ever change the name of your GitHub repo or something like that, it doesn't all just disappear.
Right.
Exactly.
I mean, it's great that it is – GitHub is there and you do get the latest version, but you could delete that if you just were in a bad mood or whatever, right?
Or your account gets suspended or hacked or whatever, right?
You want to – you definitely want to be careful.
So that sounds really cool.
And, yeah, that's what the Large Hadron Collider guys were talking about as well, the digital object identifiers for the papers.
That sounds great.
What about, like, Docker containers and these other things where you sort of ship, like, whole systems?
Do you guys – do you see much of that being used?
I hear people talking about it as a possible solution.
I think because I'm kind of more tuned in than most climate scientists to the kind of reproducibility scene.
But I think in general, it would probably be asking too much, I guess, of a regular climate scientist to be so up on these things to be able to do the whole Docker thing themselves.
So I think it's a possible solution in future, but it's kind of out of the reach of a regular climate scientist right now, if you know what I mean.
Yeah, yeah, I definitely know what you mean.
I think it's out of reach of a lot of developers as well.
Like, not that many people are actually doing, you know, complicated Docker things or containers in practice.
But still, it seems like it would be a great solution because you can – you capture the whole platform and its dependencies, not just the code plus the data.
I mean, I guess a lighter scale – very lighter scale version of that is Conda environments that – because you can post your environment on anaconda.org.
And then someone can just kind of go Conda ENV and then the URL to where you posted it on your profile on anaconda.org.
And then it'll install your software environment.
Oh, that is really cool.
I didn't know that that was a feature they had.
That's awesome.
Yeah, it's really cool.
And they actually – they go a step further now.
They've got a – I guess a bit of a beta thing called Conda Capsule.
Well, Capsule.
I think it's K-A-P-S-E-L.
Yeah.
In that one, instead of posting an environment to anaconda.org that someone can install, you basically write the specifications of the libraries you use in essentially your readme file.
And then Conda Capsule just takes your readme file and then installs it all.
Yeah, that sounds really cool.
They're doing all the stuff around Conda and kind of Conda environments is really exciting for kind of things that are doable for a regular scientist right now because it takes one line of code.
Yeah, which is totally possible.
It sounds like the Conda guys are doing tons of good stuff.
I know with the anaconda distribution and all that, but it sounds like even – maybe even more than I realize.
That's awesome.
We talked a little bit about reproducibility and the various things people should be doing.
What are some of the steps that you think are within reach to get people to sort of become better at participating in the wider community, do more stuff on GitHub, more libraries, things like that?
One of the major issues I think up until now has been – up until recently has been there isn't really an incentive for people to take the code that they've hacked together for themselves for the research problem that they were doing today for this particular paper and put in the time and effort to make it general enough that the wider community could use it and they could pip install it or Conda install it and those types of things.
There's a few journals out there now.
There's the Journal of Open Research Software, which I'm actually an associate editor on, or there's one called the Journal of Open Source Software as well.
Basically, the idea is because citations of academic papers is currency in academia.
That's how you get promoted.
That's what your career depends on.
That's the kind of incentive.
If people can write a paper documenting this research, scientific research software that they've released, and then if people start citing that every time they use it in their papers, then that's the kind of incentive that academics kind of need to go that extra step and actually take their personal code and make it community code.
That's starting to happen, and certainly we're getting a lot of submissions at the Journal of Open Research Software and stuff.
Now, I guess it's a matter of it becoming part of the culture for people to actually look for those papers and cite them in the methods section of their papers just to absolutely make sure that the authors of that software are getting the academic credit they deserve for it.
That's a really good point.
You're right that it definitely is the currency of academia.
I guess sort of making that a habit, right?
It's one thing to go to GitHub and grab a package and just go and do some analysis with it, but I guess it almost would be great to have in the package, and if you're going to cite this or use this in academic paper, please, this goes in the bibliography or whatever, right?
Just getting people to start the methods section of their paper with just one paragraph talking about the software that they use and citing the actual publications that relate to that software.
That would make a huge difference.
And so it's kind of – it's getting that to be a cultural thing where our method sections always have a bit about the statistical methods that we use.
Why don't they have a paragraph on the code that we used and where we got it from?
I think that's a great idea, and it's so low-hanging fruit, right?
It would be really easy to do that.
Yeah.
So hopefully as these journals get bigger and people start pushing people to cite them, that'll happen.
And in 10 years' time, it'll just be every method section has a paragraph on code.
I definitely see that as a possible future for us.
So speaking of possible futures, let's have a conversation a little bit about climate change.
You know, you've studied it more than almost – I'm sure you've studied it more than anyone else I've spoken to, more than most people, let's say.
So what do you think?
Climate change?
Is this a real thing?
Do people cause it?
Absolutely.
I guess the frustrating thing from a climate scientist's point of view is that that question, is climate change real and are humans causing it, hasn't been an active research question for more than 30 years.
It's been accepted in the scientific community for at least that long.
And so, you know –
Well, hold on.
If I turn on the news here in the U.S., there's always some other – there's like half of the people on the TV channel saying, oh, I have 400, 4,000 scientists who say this is not real.
Just give us a sense of like why it's so accepted and whatnot.
You know, maybe put a little pushback on that vision that gets projected by news.
Yeah, those ones are very – even their statistics where they go, you know, 97% of climate scientists agree.
It's actually 100%.
Like I've been going to climate science conferences for a decade now and I've never, ever, ever sat in a presentation or read a paper that suggests that climate change isn't happening.
It's a given and it's been a given for a very long time.
So the disconnect between kind of what's being discussed in the science community and what's being discussed in public is very frustrating.
I'm sure it is, yeah.
And it's not unique.
Obviously, like if I went to a health conference, you know, the difference between, you know, public discussions and policy on health would be very different from what experts in health think should be happening.
And it's not a unique thing, if you like, but it is particularly frustrating just because it – yeah, that's a question that we moved on from.
30 plus years ago.
Yeah, I think one of the things that like makes people feel that this is more up for debate than it is at least in the U.S.
I haven't – I don't know outside the U.S.
But there's this tendency of if you're going to present something, you're going to present both sides of it.
So if you're going to talk about climate change, you have somebody for and somebody against.
And that makes it feel like it's 50-50.
And it's not 50-50.
It's like 1,000 to 1 or something.
It doesn't seem like it really needs like this other side to say, well, here's the other side of the argument.
This guy, he only works for the – you know, this coal company.
But he's really studied it.
It'll be fine.
You should listen to him.
No, that is – it gets you tearing your hair out when you – the people that they put up on television.
Yeah, for the against case, like it's in a debate on physical reality.
Yeah.
Right.
So I'm with you.
I think this is an important – this might be the fundamental challenge of our generation, our generations, you know, probably multi-generational actually.
But, you know, what are some of the things you think we can do as just citizens of the world?
And what do you think we can do as people with the magical wand of software development where we can actually make things that analyze change and so on?
I definitely think getting, I guess, politically active is important.
So, you know, writing to congresspeople, attending protests, getting involved in, you know, community organizations, whether they're against a pipeline that's being built or they're encouraging people to divest their money from fossil fuel companies or whatever it may be.
I think it's got to just be that groundswell of kind of grassroots activism that gets things changed because, I mean, there are a lot of vested interests in keeping things the way they are.
I mean, the biggest companies in the world are fossil fuel companies like, you know, ExxonMobil and companies like that.
So, it's a formidable opponent and it really needs a massive grassroots effort.
And I guess the more I've got to know about the issue, the more I've kind of got involved in those types of things.
And I certainly, when I first started as a climate scientist, I wasn't politically aware or active at all.
But all the, I guess, organizations that I've been involved with as I've become more active are, you know, really crying out for help with IT stuff, whether it's their website, whether they want to analyze some voting data or whatever the case may be.
Like, if you rocked up at a grassroots organization doing, you know, climate activism and said, hey, I've got a bunch of IT skills, they'll fall over themselves with gratitude and with things that you can help with.
And you'll be up to your eyeballs in way too many things that you've got time to do.
But yeah, no, definitely, I think developers and people with IT skills have a particular role to play just because they're such important skills.
And these organizations are just crying out for help with that kind of stuff.
So yeah, if you want to get involved, there's absolutely ways to put your skills to good use.
Yeah, that sounds like a really good thing to do.
Certainly, if you gave a couple of hours a week or something to one of these organizations, and if they really are totally missing the software side of the story, right, they don't have a lot of people to help out there, you could probably make a pretty big difference there.
Oh, yeah, absolutely.
So how much of this do you think is like a political fight versus an economic fight?
I mean, like every time you take an action or you buy something or you don't buy something, you're kind of voting with your dollars or your pounds or whatever, right?
And do you think it's more important to act as consumers or to push on the political side?
Or where do you think the leverage points are?
I mean, I think it's both.
I mean, I think I used to think that it was just definitely talk to politicians and get them to change their minds.
But if you read, you know, the writing of major climate activists like Bill McKibben and stuff like that, he has a whole book about the fact that he started all his campaigning in Washington thinking he had to be there to, you know, tell the politicians.
And then in the end, he comes to the realization that it's corporations that rule the world.
And that's why 350.org, his organization, focused so much on divestment and of actually thinking about where you're spending your money.
Because at the end of the day, maybe where you spend your dollars is more important than what you write down at the ballot box in terms of the impact it has.
Right. And sure. And once the voting is kind of done, like things are set for a while.
Right. And so but you buy stuff every day, you consume things or don't every day.
That's pretty interesting.
Here in the U.S., I don't think there's a lot of positive policy that's going to be passed in the next two years on climate stuff.
But I feel like we still get many, many choices on what we buy, what we don't buy, where we get our energy from, things like that.
So there's still lots of things that people can do, but I totally agree with you on donating some time to activist groups.
So maybe what do you think is the most exciting or encouraging development in the last couple of years around sort of positive change to fight climate change?
And then maybe like what do you think is like a setback that we've had that is unfortunately.
The most exciting would definitely be just the growth in renewable energy.
So I'm sure it's similar in the U.S., but here in Australia in particular, the growth of people having solar panels on their own roofs and stuff has been huge.
And it's kind of in spite of government efforts to kind of slow it down, if you like, because obviously, you know, energy companies have a lot of lobbyists and things like that.
But in spite of that, renewables are just going from strength to strength.
So that's definitely the most exciting.
I would definitely agree with you that like, you know, a little bit with the politics versus dollars, like renewable energy is becoming the financially smart choice.
And once that happens, like it's just forget the politics, right?
It's going to solve itself at that point.
But, you know, we can slow or hasten it for sure through politics.
Probably your question of what was the most discouraging thing is probably the ability of kind of vested interest to slow things down.
It kind of, it feels like, it almost feels like sometimes with climate change is that we will eventually get there and we will eventually reduce our emissions significantly.
But like, it's a slow victory is kind of a loss because of all the heat that will be of accumulated in the climate system in the time it took to get there.
We have to win and we have to win fast.
And the fact that we're potentially winning, but winning incredibly slowly is a big problem.
It's both encouraging and really frustrating at the same time.
Yeah, so yeah, no, winning slowly is not really kind of an option, but the ability of vested interest to kind of slow things down makes it feel like it could be a very slow victory, which would be really not a victory at all in the end.
Right.
All right, well, maybe we'll leave it there with that for the climate science stuff.
And let me just ask you the final questions that I ask everyone who's on the show.
So if you're going to write some Python code, what editor do you open up?
I'm a simpleton.
It'll be like text wrangler or g edis or a simple graphical text editor like that.
I used to be kind of self-conscious about that.
And then I was teaching at a software carpentry workshop.
And one of the helpers was a core Python developer.
And he told me that he uses simple graphical editors like that too.
Ever since he told me that, I've felt really good about myself.
Hey, if the core developers can do it, you could definitely do it.
Yeah, exactly.
All right.
And a notable PyPI package.
I mean, you named a few that are involved in climate science.
Yeah, I thought I might actually give a shout out to a little one, unknown one.
I use Git Python.
I'm not sure if you've ever used that.
It's basically just a hook to Git.
But basically...
G-I-T Python?
Yeah.
Yep.
All one word, G-I-T Python.
And so basically I use it because a lot of...
With these net CDF files that we use that have the metadata in them, some of the tools that
have been built kind of in the global history attribute of these files, it keeps a record
of what was entered at the command line to produce this file.
And so I can do that.
Basically, I can have a script that at the end puts in the history attribute.
At the command line, it was Python, the name of the script, and then whatever input arguments
it was.
So I have a complete record of what was entered at the command line to produce this file.
And then with Git Python, I can also have...
You know, every commit in Python has a unique 40-character string associated with it.
I can basically put the first seven or so digits of that so I know which version of the code
was executed.
So yeah, I actually use Git Python.
Yeah, that's awesome.
Yeah, Git Python helps a lot just with reproducibility down to which version of that script did I use.
Yeah, and the arguments as well, which is really important.
Awesome.
All right.
So we've heard a lot about the tools, a lot about how we can contribute, maybe some of the
problems.
What's the final call to action?
People are interested, how do they get further involved or do something along those lines?
I mean, I can give a shameless plug for myself.
So I have a blog, drclimate.wordpress.com, where I talk a lot about Python.
I talk about research best practice in general, but a lot of the time that means I'm focusing
on Python, basically, in climate science.
So if people want to subscribe to that, that's a good way to kind of keep up to date with things.
You follow you on Twitter, right?
Yeah, at drclimate as well.
Yeah, at drclimate, which I'll put in the show notes, of course.
Just to kind of, it'd be good to have, I guess, more people in the climate science Python discussion,
if you like.
I feel like through some of my involvement in software carpentry with other disciplines, particularly in the biosciences, bioinformatics and stuff like that, the community around R in those languages is huge.
And they seem to be a lot further down the track in terms of dealing with the reproducibility crisis and releasing packages that other people use.
I feel like climate science is a smaller community, but we need that strong sense of community around Python to really help with some of those challenges.
Right.
Maybe some people who could actually convert something into a package that could be reused and help getting it on PyPI or on GitHub.
Maybe those type of contributions could be helpful as well.
Oh, yeah.
Absolutely.
Yeah.
There will be a lot of people that have very interesting code from a research perspective that would need a bit of assistance in actually releasing it.
Sure.
All right.
Well, that sounds great.
Thank you so much for all your thoughts and sharing what you guys are up to in the climate space.
Oh, no worries.
Thanks.
Thanks a lot for inviting me on the show.
You bet.
Talk to you later, Damon.
This has been another episode of Talk Python to Me.
Today's guest has been Dr. Damian Irving, and this episode has been brought to you by ParkMyCloud and Rollbar.
Do you hear that sucking noise?
That's your cloud provider making you pay for your idle instances.
Turn on ParkMyCloud, plug the leaks, and save money.
Visit talkpython.fm/park to get started.
Rollbar takes the pain out of errors.
They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until your users complain, of course.
As Talk Python to Me listeners, track a ridiculous number of errors for free at rollbar.com slash talkpythontome.
Are you or a colleague trying to learn Python?
Have you tried books and videos that just left you bored by covering topics point by point?
Well, check out my online course, Python Jumpstart by building 10 apps at talkpython.fm/course to experience a more engaging way to learn Python.
And if you're looking for something a little more advanced, try my write Python code course at talkpython.fm/pythonic.
Be sure to subscribe to me.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
Python code.
Thank you.