Learn how Python is being used in research to understand the inner workings of the Earth.
This week, you'll meet Lindsay Hagee, a PhD student in geophysics at the University of
British Columbia. She shares how she's using Python to solve these computational problems,
along with an amazing framework for viewing scientific writing itself
through the lens of Python and open source. This is Talk Python to Me,
episode 163, recorded May 3rd, 2018.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the
ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter,
where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm,
and follow the show on Twitter via at talkpython. This episode is brought to you by MongoDB and
Anvil. Please check out what they're offering during their segments. It really helps support
the show. Lindsay, welcome to Talk Python.
Thanks. Good to be here.
It's great to meet you. You're doing some really cool stuff in geophysics intersecting with Python,
so I'm super excited to explore that with you today. But before we do, let's get started with
your story. How did you get into programming in Python?
Most of that actually was undergrad, so I really hadn't touched much computing until I started at
university. I did my undergrad in Edmonton at University of Alberta. And it was in my first
year that I sort of got introduced a bit to programming ideas, but that was actually in Perl.
So did programming look like a good thing you wanted to do, or were you like, whoa, what is this?
It was more like, what is this?
I mean, it was interesting to just start getting exposed to the types of things you can automate to
make your life a bit easier. And so that was all that we had really done in the first year.
And then got a bit into C++ in my second year when I was starting in physics. And that was intimidating,
for sure. And then finally, actually, once I started getting much more into geophysics,
we were all in MATLAB. And that was much closer to what sort of the day-to-day scientific work
is readily supported by. And so, yeah, that was really my first foray into computing.
Yeah. And there's a pretty easy switch from MATLAB to Python. I mean, they're not the same.
Obviously, they have really different philosophies, right? Like MATLAB is like,
let's make it all super commercial. And every little function you want, you got to buy that
individually and so on. But, you know, the sort of feel of scripting it is kind of the same.
So that's pretty cool. So how did you get into Python? Actually, was that in grad school?
Yeah, it was grad school. And it was actually, I'll credit Rowan Cockett,
who I've done a lot of work with. And he just recently finished his PhD.
And so that was, I guess, my first or second year at grad school. He had really suggested that we
started working together on a number of projects as like a geophysics group. And so that's how
Simpeg started, which I'm sure we'll come back to. But then starting to build software together was
when, you know, we looked at Python because there is such a healthy community and so many tools for
making that easy. That, yeah, that's how we got into it.
Yeah, that's cool. And what year was that?
That was only a few years ago. So I guess like my second year of grad school.
Yeah. Earlier than that. Maybe 2014. Yeah.
I'm just trying. I'm just thinking because if you look at the history of Python, and especially how
it's appeared in data science, it looks like around 2012, I don't know what the actual trigger for this
was. But something happened that just really drove the adoption of Python, especially in the data science
space. I'm pretty sure Jupyter has a lot to do with it. But I'm sure there's other factors as well.
So it makes sense. Yeah, yeah.
Yeah. So what do you do today? Like, you're still in grad school? Is that right?
Yes, I am. My sort of day to day, I am trying to wrap up my PhD. So it's a lot of writing at the
moment. Like the thousand little details you didn't know you had to finish up or are attacking you?
Exactly. Yeah. So that's where I'm at at this point. But I'm fortunate to be a part of like a really
great group of people. And so there's a lot of collaboration that goes on back and forth.
So like most of my PhD and grad study journey has really been a lot of collaborative work,
which has been a lot of fun.
Nice. And so you're studying geophysics. And you're working at this place called the
Geophysical Inversion Facility. Is that correct? Is that right?
Yes, that's right. Yeah.
So first of all, what is geoscience or geophysics? Like, generally?
Geophysics would be a subset of geoscience. So geoscience is basically anything concerned with
trying to understand the earth. So that is very, very broad.
Right. It could be plate tectonics. It could be magnetic field. I guess it could even be climate
change, right?
Yeah. It could be climate change. It could be atmospheric studies. It could be like economic
geology, trying to map out where different rock units are. All of that.
Okay. So what's geophysics?
So geophysics, we're then understanding the earth through physics. And so what I do specifically in
geophysics is we're looking at geophysical imaging. So it's a lot like medical imaging in a lot of ways.
So, you know, when you go to the doctor, you're hoping that they don't have to drill into you to
get information about what's going on inside of you. So there are sensors, like if you go into an MRI or
something like that, there's sources and sensors on one side. And we can take the data that have been
collected in that survey of you and work with those to then get an image of what's going on inside your
head. And so we basically do the exact same thing, but then on the earth. And so it's a larger scale
survey, but it's the same general principles.
Okay. That sounds really interesting. I've seen some really amazing graphics and I'll link to a
couple of them, of course, in the show notes. What are some of the types of questions that you are
trying to answer or people generally in the field are trying to answer?
So it totally ranges. Our group, a lot of the history has been really connected with minerals.
So trying to map out and locate where mineral deposits are, characterize them, delineate the
different units and things like that. One of the big topics that's becoming more and more relevant
is characterizing groundwater. And so trying to figure out where do we have pockets of aquifers? You know,
how much is in that aquifer? Can we quantify that using geophysics? Because in a lot of places right
now, what's done is just wells are drilled and you can get water levels at, you know, single points,
but that obviously doesn't characterize the whole aquifer.
Right. Of course. And that's going to be an increasingly interesting question. You know,
we saw California go through some really serious droughts and they're finally out of it, but that
kind of stuff is going to be happening more and more, most likely. So those questions become more
critical, right?
Yeah, absolutely. So groundwater and seawater intrusion is another big thing in California. So it's not only
that you are losing, you know, aquifer water, the seawater is actually also coming in. So there's a whole
bunch of different things going on that we really need to get a handle on fairly soon.
All right. So what are some other questions or some other areas people are focused on?
A lot of the work that I've been doing, I've been looking at trying to monitor subsurface injections,
like if carbon dioxide or hydraulic fracturing is done, you know, we're injecting fluids into the
earth and trying to track out where those are growing. And so I've been doing a lot of work
looking at electromagnetics when we have steel cased wells there, because often, you know,
we've got this well that and we're going to inject the fluid through there. But what's
interesting about a well is that it is actually very, very conductive. So in a lot of ways,
it's like a big electrode. So it can help us get current to depth. But we're then trying to
understand, you know, the mechanisms and how does that actually happen? And how does it work?
Yeah.
So that's been a lot of, yeah, what I've done.
That sounds really, really interesting. I guess the first place we should probably start looking at
some of the programming side of things is, I guess, let's talk about Simpeg. So Simpeg is this thing
for simulation and parameter estimation. What does that mean? Put that in plain English for us.
Fair enough. What we need to do if we're trying to collect any sort of data,
so we can maybe go out and do a magnetic survey. And so in that case, the source is the Earth's
magnetic field. And then basically, whatever kind of rocks that you have that are susceptible,
meaning that they have, they act like little dipoles. And so they'll try and line up their
dipoles with the Earth's field. And then we can go over with a magnetometer and try and find these.
And so that would be a magnetic survey. And what we need to be able to do is we actually need to be
able to simulate the physics of that process. So in this case, we'd be simulating like the
magnetostatic equations.
Yeah, I was gonna say, so different types of minerals maybe react differently to the magnet.
So if you're looking for a cobalt or lithium or something else, like you can detect that using
magnets?
Yeah, some of them. So it depends if they're magnetic or not. So not all minerals are.
And then the strength can vary. And so mag is just one survey that you can do. But generally,
we'll try and do a few different surveys. So they might be magnetic, they might also be conductive.
So if you have something like iron, it's pretty easy to get current through that. And it's magnetic.
So if you go and do both of those surveys together, we can start to like pinpoint what these different
minerals are.
So iron is easy?
Iron's easier. Yeah.
Nice. Okay, so back to the simpeg. So you want to you want to analyze the magnetic response
of the stuff deep underground?
Yeah, so we first need to be able to like simulate the set of equations that governs that process.
So that's the simulation piece is we're going to solve some sort of partial differential equation
that's governing the physics. So we assume that we know what the earth is, like what the earth model
is, what the physical properties are, and we can predict what the data should be. So that's a
simulation piece. And then the parameter estimation piece is trying to go backwards,
is that we have data. And now from those data, we're going to try and estimate.
And so we do that through an optimization process, we can basically say, okay, I know what these data
are. And I know how to guess or I know how to simulate data. And so now we're going to try and
find a model that fits those data. And it's somewhat geologically reasonable.
Okay, that's pretty cool. And this is all done in Python. Is there maybe some like underlying
C or Fortran code that gets brought in there? Is it all straight Python?
It's basically all Python, we interface to lower level solvers, and things like that. And there's
a little bit of Cython for some of the meshing stuff. But for the most part, it's pure Python.
Okay. Yeah, that's, it's really cool. There's a picture, you mentioned Rowan Cockett, and I happen
to just have grabbed something off Twitter from him. So that's kind of a funny tie in. But if people are
kind of interested in seeing what the those look like, I'll put in the show notes, this cool,
sort of three dimensional graph of what is called the Richard equations. That's pretty cool. So
are you familiar with this?
Yes, I'm a co author on that paper, actually. So
Oh, awesome. Yeah. Well, tell us, tell us a little bit about like what,
what the main question was, and what you guys found using Simpeg on it.
What this was, this paper actually describes a lot of the fluid flow machinery that we have in Simpeg.
So Richard's equation describes fluid flow, like going through soils. So it's a two phase flow where
you have air and water in some sort of medium like a soil. And so depending on how hydraulically
conductive pieces of that soil are basically how easy water goes through that front propagates down
differently. And so in this case, what we developed in this paper is describing like, how do we solve
those equations? But then the important piece is when you want to actually go back and estimate things
like hydraulic conductivity, we need that inversion piece. And so we need to have gradients that tell
you basically, if I change this model parameter this much, it changes my data in this manner.
I see. Like how responsive is it to variation in parameters and estimation and stuff?
Absolutely. And so that's like such an essential piece to be able to solve that parameter estimation
problem. So in this paper, we go through and basically derive all the mathematics for that, and then show a couple
examples.
Yeah, the pictures are really, really compelling. And it's pretty interesting. So I think it's really nice what you guys have
put together here. So are a lot of people using Simpeg for other work? Or is it mostly within your group?
It started mostly within our group, but it is starting to branch out a bit. So we've got some collaborators at Colorado
School of Mines. There's some people with USGS who are starting to dabble in it. We recently gave a short course that
traveled around the world. And so that was a great opportunity to start introducing people to the Python ecosystem.
And for some researchers, you know, Simpeg was an appropriate tool. So they're starting to come in and explore that a bit.
Hey, everyone, Michael here. Did you know I'll be doing a three-part webcast series about MongoDB and Python from May to June?
We'll see why MongoDB is a great choice for Python web apps. In this series, we'll go through the entire process of
building a clone of PyPI, Python's packaging website over at pypi.org. Everything from building the front end to
deploying the web app and MongoDB to the cloud. You'll learn everything from document modeling basics to special
considerations for running MongoDB in production. The webinar is free. So just click on the link in the show notes or go to
mongadb.com slash webinar slash Python and sign up. See you in May.
It sounds like a lot of what you've learned from Simpeg has sort of fed back into some of the other work that you're doing as well around like more general education, research and science, right?
Yeah, absolutely. I think for our group and for me in particular, Simpeg was like really my first entry point to the whole open source ecosystem and like what it's like to operate and run an open source project and be involved in an open source project.
And what some of the strategies are things, you know, like peer review and issue tracking and all of that sort of stuff. That was all really learned when we first jumped into Simpeg.
I think it's really interesting. I mean, you come from a place like say working with MATLAB as a community in general, right? Like they publish packages you can add on and you kind of wait for them to release new versions and they give you what you, you know, you get what they give you, right?
And there's not a whole lot of give and take there. And that's just so, so much the contrast with things like Simpeg or other open source projects, right? Like even on the courses that I do, I have people will come and they'll say, oh, you did this little demo like this, but actually I've refactored it to really take account for this other thing.
And it's really, you know, just you get these out of the blue, people are helping you out.
Even if you don't ask for it, it's really quite an interesting experience.
It really is. And like just the ability to put code out there, share it and get other people's opinions is like such a part of the Python community and like what people follow.
Whereas if you look at something like MATLAB that just, there's not easy mechanisms to do that.
So even if you write something useful, like you email code then to your friends and that just is not sustainable.
Which one do you want me to run? The one that you sent me on May 23rd or the one that you sent in June?
Like I can't remember which one we're working from, right? That's not a great version control, is it?
No.
So maybe let's spend a little time talking about this presentation that you gave at PySci, CyPy?
2016.
And it was called Using Open Source Tools to Refactor Geoscience Education.
And I think you kind of put it around the scope of geoscience, but I would say scientific education more generally, right?
Yeah. I mean, geoscience is our domain.
And so that's where like we've thought through a lot of these things.
But, you know, as we've been developing along the way, really trying to take a bit of a big picture perspective and figure out,
are there things here that we can learn and perhaps translate to teaching and learning as well as scientific publication?
Yeah. So maybe give us the big idea on your talk and then we'll touch on some of the pieces.
Geosci.xyz is a collection of like open source.
They're basically living textbooks.
And so the idea is that we're trying to take a lot of what we've learned from developing open source software
and try and apply that to open source educational resources.
And so that there are opportunities for collaboration, for peer review, for iterating on things and, you know,
to grow and develop resources as a community.
One of the things that we've really noticed in geophysics in particular is it's a really small field.
And so there really aren't actually many good textbooks, especially for introductory level classes.
And so what happens is, is, you know, there are professors scattered around the world teaching this
and they're all developing all of their own course notes from scratch.
And each of them has, you know, different expertise and different background.
And so there's one aspect of that that's going to be really strong.
But then the rest of it, they're having to go and learn from A to Z, like a whole bunch of techniques that there may or may not be.
They maybe learned it in grad school, but they haven't used it for 15 years.
But now they got to write about it, right?
Because they got to be comprehensive.
Yeah.
And so we're hoping to eliminate some of that so that, you know, people who are experts in given topics can contribute that
and then can leverage what other people know.
So, yeah, I think that is really quite an interesting perspective.
You talked about some of the problems with the sort of teaching and learning and writing sort of largely focused around like textbooks and stuff, but more generally as well.
If there are bugs in the book, you don't know.
They're hard to be tracked.
I mean, maybe you can find it, right?
Like versioning is difficult.
Do you have the current version?
You don't know, right?
How do you diff a book like that is, you know, two textbooks.
You just kind of flip through it, right?
I don't know.
It's not super easy, is it?
And sometimes it's really obvious, like, oh, we added a chapter on this or a figure on that.
But other times you said, you know, it's much more subtle.
Like, oh, there was an error.
Should have been a minus sign in this equation right here.
And like that's, you probably wouldn't catch that, right?
That's really a challenging position to be in when you're first trying to understand a topic, right?
It's like you work through the whole question.
You create the plot based on your understanding.
And it doesn't match what theirs is.
And then what do you do?
Like they're different.
And they could be wrong.
You could be wrong.
And there's no way to sort that out.
And like there's no way to contact the author really in most cases.
Yeah, it's quite hard.
And a lot of times you're a student.
So you're like, well, I'm wrong because I'm new at this.
I must be wrong, right?
Until, you know, somehow you maybe decide, no, I think I really am right.
This is really broken.
But yeah, that can be frustrating.
So you had a really nice way of breaking down the sort of things involved in that type of creation, that educational content creation, and sort of framing it in terms of concepts from the Python space.
And you started with functions.
And you asked the question, like, what are functions in the context of science and writing?
So what are functions in that context?
There's a few different things to think through.
But one of the things that we first looked at, maybe I'll give you a little bit of the backstory before diving into this.
One of the big motivating factors for this project was that my supervisor had developed a website quite a number of years ago instead of a textbook.
So he was really forward-looking in the sense that he wanted to get content out there for students in a very tangible, like easy to interact with way.
And so they built this website.
It's a great site.
But then we found a few typos in it and wanted to try and go in and fix them and realized it was tangled up in this crazy HTML mess.
The first, I guess, in the refactor of this was really identifying, like, what actually is the data here?
What's the data and what is the packaging?
So in this case, really, the data is just the text and the equations and the images.
The HTML and CSS, all of that is just packaging.
And so in this case, perhaps a function is actually like Sphinx.
And so you take your data, which is text and images, and then you compile that into a website.
And so what's powerful about that is then as styles and things like that update and as there's better ways to interact or somebody builds a fantastic new search tool, you haven't, like, you've separated out what is data and what is the packaging.
And so you can immediately leverage all of those new developments.
Yeah, it sounds super obvious.
I mean, as people who write software, like, you think, of course, you're going to do these types of things.
But then in practice, you go look at, you know, for example, the website you're talking about.
And it's all crammed together.
A lot of times these are written by people who don't have sort of formal software training either.
So they maybe don't even have some of these ideas sort of in the back of their mind when they come to it.
And so, yeah, so you're like, all right, well, let's break this out into restructured text.
People can edit that super easy.
I mean, you probably know LaTeX or something that's like an easier version of that.
So that's good.
And then, obviously, the styles.
So that's one part.
Another part you talked about was capturing inputs.
So, you know, you probably have some picture, but that picture has, you know, like a view onto it.
It has parameters for the equation.
It has all kinds of stuff, right?
So that's another aspect of sort of the function analogy, right?
One of the great things in this Sphinx documentation is the Matplotlib plugin.
And so that's what we've been leveraging in order to capture the inputs to your figures.
And so because to create a figure, we're running some sort of code with, as you said, some inputs.
So maybe we're looking at the electrostatic response of a sphere.
And you want to change it from a resistive sphere to a conductive sphere.
That should be something that, like, the user of this resource should be able to, like, readily do.
And so you can't do that with a textbook, obviously.
But in this case, if you've preserved that source code, then that's actually an entry point for people to actually take that single picture and start to be able to explore that.
And, I mean, once you, yeah.
It sounds like that's touching on one of the whole, like, super significant things in science in general these days is the whole reproducibility thing, right?
And this just makes the whole paper or the book or whatever it is more reproducible if you can re-execute it to regenerate the output, right?
Well, Rowan and I gave a talk at JupyterCon last year and sort of touching on reproducibility.
And I think one of the things that's important to keep in mind, too, is, like, what is the point of reproducibility, I guess?
So, I mean, it is obviously good practice to have your content be able to be regenerated.
That is a good thing.
But the way that you actually build upon somebody's ideas in science is you take what they've done and then extend it.
And so, also, by being able to, you know, at least capture all of the instructions to get to point A, then somebody can immediately pick that up and start to play around with it and hopefully get to point B, which is then actually maybe some sort of new discovery that extends on that work.
Yeah, that's cool.
So, you can actually literally build on the sort of algorithm and steps.
So, there's probably a lot of data exploration in Jupyter as well in this.
Like, you could, you know, copy little bits into Jupyter, play around, put it back in restructured.
Or can you load restructured directly?
You know, I don't actually know about that.
I'm sure that somebody's written something.
It's got to be out there somewhere, right?
Yeah.
But we've sort of been developing Jupyter Notebooks in parallel.
We've been teaching a lot of courses, actually, where computation is, like, not taught at all.
But we want to be able to have people play with figures.
So, we've been leveraging Jupyter and IPy widgets to basically wrap functions that compute things and give you plots to make that interactive.
And so, that's been really exciting to see that, you know, people can actually get up and running.
And they're running code, but they don't necessarily even need to know anything about Python, what is Jupyter, any of these things.
Right, right, right.
Just inputs and pictures.
Another thing you said is once we have this concept of sort of a reusable function, you can test it, like in Travis CI or continuous integration, right?
This has been exciting to see.
There's a few aspects to the testing.
First off is, like, when we have code snippets and things like that in the textbook, we can test them.
Same thing with all of the figures.
We can test those.
And so, I mean, if there's API changes or things like that down the road, we'll catch that so that the code always continues to work.
Because I think we've all seen the case, you know, where somebody actually wrote a textbook and there's printed examples of code in there.
And there's inevitably a bug.
And there's nothing that can be done about that once it's been published.
And so, then you just end up with generations of frustrated students who can't even get the code to run at the first stop.
So, here, at least, that's something that we can test.
Another thing that's actually been kind of interesting, too, is you can go in and test links.
And so, make sure that all of the things that you are pointing to and all of the extra resources that you are connected to continue to be there.
And if not, then you can go in and find something else relevant to point people to.
Yeah, that's a really good point because you don't necessarily control all the external resources that exist, right?
And you don't keep checking them.
Yeah.
Yeah, yeah, really cool.
So, the functions, that's a pretty low-level concept in structuring code.
So, the next level up would be classes, maybe?
So, starting to get to a bit more organization.
So, I mean, a function, you've defined a piece that is reusable.
And now a class, we're going to try and define something that you can perhaps inherit and build upon.
One of the things that we pointed to in this analogy is just looking at, like, a given page structure.
So, when you're talking through a concept, I mean, there's a few obvious things that every page has.
Like, it has a title.
It has contributors to that page.
One of the things that we've been trying to promote is a purpose statement on each of the pages.
You know, to give just, like, a high-level overview of why should I care about what is on this page.
I think that's a really good idea because one of the major benefits of people writing, say, like, unit tests against real proper, you know, software code is you know when you're done.
Like, if you put out all the things it's supposed to do in the test and it does them, well, you can stop messing around.
Because people can, like, fiddle with the code and think about what it might need in the future forever, right?
And so, there's this really clear, this is what I wanted to do.
I've done it.
Now, what's next, right?
And you obviously have the same problem in writing.
I really love this idea of, like, we're going to give this a purpose and almost test it.
You also said that this leads really well to a collaboration, right?
Because people coming to it know what the purpose is.
They all agree upon the purpose and it sort of helps communicate that.
Yeah, absolutely.
And, I mean, because when you have multiple authors contributing content to one resource, everybody's got a bit of a different writing style and all of those sorts of things.
And that's totally fine.
But it can lead to sort of a hodgepodgey resource.
But at the very least, you know what is going to be achieved in each page.
It's so much easier to collaborate and give meaningful feedback as well.
So, if I am reviewing somebody else's page and I know what they're trying to accomplish and have maybe a couple ideas about some different examples that they could include to help achieve their goal, that's easy to then point them to.
But if the purpose isn't clear from the outset and it's not immediately transparent, it's very hard to then give productive feedback.
Yeah, for sure.
It also helps with peer review.
You know, a simple peer review question is, have you achieved, does this thing do what it says it does, right?
Rather than, is it good or is it accurate or is it good, you know, whatever.
Those are really hard to answer.
Another thing is you say it leads to templates, right?
So, you could say this page is for, you know, a case study of this type.
And then that means it has this structure, right, which can really help with writing.
Yeah, and it helps, you know, solicit input from other authors as well.
So, what we've done with case histories, a case history and how we've defined it is it's basically like an exploration or geophysics example.
So, we walk through, you start with some sort of question.
And then we're going to walk through what are the relevant physical properties?
So, what are the different rock types or things like that that we're going to look for?
How are we going to try and detect those?
What do the data look like?
Then we go through and, you know, how do we process those data, interpret the results?
And then did we actually answer our initial question?
So, that's sort of like seven steps that we've broken all of these case histories into.
And once you actually start having a few examples of that, we've been able to send these templates out to researchers around the world who have experience in different applications than we do.
And just said, hey, do you have a good example that you could put into this framework?
And once you lay out the pieces, people are a lot more willing to go in and put their content there.
Because you've just removed, like, all of that overhead of figuring out how should I structure this.
Right.
How long should it be?
What should I say?
What's important?
What's not?
Yeah, it's super, super interesting.
I had Jesse Davis on the show quite a while ago.
And he did something similar for blogging.
He talked about how to write a good developer blog.
And he had come up with five design patterns for blog posts.
Right.
What are your goals?
Then this pattern applies.
And it's just, like, once you know what you're trying to do and you have the pattern, you're like, okay, these three steps.
This is what I do.
And then all of a sudden, like, it's the writer block can be largely gone.
It's much, much quicker.
And so, yeah, I really like this idea.
This is where we've seen definitely the most contribution coming in is because it's an easy place to jump in once something's structured.
Okay.
So once you have functions in the classes, you might want to reuse them other places.
That might be, like, import.
The import saving.
Yeah.
So this is where things get a little fuzzy.
But we've played around with this analogy.
And one of the things that has been kind of exciting is we've developed an equation bank.
And so one of the resources we've been working on is about electromagnetics.
And so Maxwell's equations are going to show up all over the place in multiple places.
And so that's something that you don't want to repeat writing.
And especially when you have multiple people, we want to try and stick to the same notation conventions and all of that sort of stuff.
So ideally, we don't want people rewriting these.
So we've actually set up an equation bank.
And then that's something that you can just include in your page is just include Maxwell's equations.
That's cool.
And so if there's a mistake, you fix it in one place, right?
And it fixes it everywhere.
Exactly.
Yeah.
Nice.
And let's see.
You could also maybe think of links as external, as sort of an thing you import, like an external resource that you depend upon.
Yeah, in a sense, because in like what you're doing in that way is you've got some word that's linked.
So, you know, maybe we've linked the word to some sort of specific geophysical system.
And so that is containerized piece of knowledge that you can sort of bring in and expose to the user in a meaningful way that also is in context.
So that's one piece that I think fits into that analogy.
Yeah, yeah.
Another one is once you start importing things, you see structures and dependencies.
And then you could almost say like, well, we should refactor this into this other form that's better once you see the large overall structure.
That's pretty cool.
Because then you get to looking at ideas of like which concepts build upon each other and what concepts you need to understand, you know, this given method or another method, which is kind of cool to be able to actually like introspect the field.
Yeah.
Yeah, it's pretty amazing.
And I'm sure there'll be some good visualizations at some point.
And then at the very outer end, someone else wants to use the thing you've created.
So you have the pip and packaging analogy as well.
This is something that I would love to see this idea evolve a bit more.
But I think at the basic level, making it clear how people can use things.
So applying a license, showing which concepts build upon others.
So if you're importing this more advanced resource, what are the things that you should be familiar with before that?
And then as well, versioning and all of that.
Make sure that that's clear when you're changing things.
This portion of Talk Python to Me is brought to you by Anvil.
With Anvil, you can build full stack web apps with nothing but Python.
Building for the web is complex.
You typically have to write JavaScript, HTML, CSS, some front end framework like React.
And then you've only done the front end.
You still have the server to write.
And then you have to decide where and how to deploy it.
With Anvil, all you need to know is Python to build production-ready apps and deploy and scale them with a single click.
You have a visual designer for your page.
And you've got the entire Python ecosystem to integrate with.
It even comes with a built-in database as a service.
I've been using Anvil myself.
And I'm really excited how accessible it makes the web, even for people who are not excited about writing HTML.
And if you happen to take my 100 Days of Code course, you'll see near the end, we actually spend a lot of time building a really cool web app with Anvil.
I'll put that app in the show notes.
But you can find it at pypoint-100days.anvilapp.net.
Get started at talkpython.fm/Anvil.
And they'll throw in a 10% discount on an individual plan just for you Talk Python listeners.
If you've been afraid of the web, go have a look.
This is something special.
And they're doing really interesting things with Python.
So that's the whole overall conceptual way of thinking about the work that you guys are doing in education and writing in the software Python space.
But you actually took a lot of the tooling literally from the software space, right?
Things like Git and continuous integration.
What all did you use there?
Everything's hosted on GitHub.
So that's our peer review mechanism.
That's all the versioning for us.
Issue tracking, all of that.
We have used Sphinx to actually build the pages.
And then I did mention that the Matplotlib plugin has been one that we're using to generate these reproducible figures.
We started out hosting stuff on Read the Docs.
But then the site got way too big.
So we host it separately now.
Yeah, and then Travis CI for all of the testing pieces.
It's not just the concept of it.
You're actually applying a lot of these tools and techniques to it.
That's pretty cool.
Oh, yeah.
And then Jupyter throughout as well.
Yeah, I'm sure Jupyter is in there.
Are you guys moving to Jupyter Lab these days?
Are you sticking with Jupyter?
What's the thought there?
I've dabbled in Jupyter Lab.
I'm quite excited to start diving into it a bit more.
I've just been with writing.
I'm trying to wrap up the PhD.
I'm hesitating diving into new and exciting tools because it's easy to lose track of time there.
Yeah.
Yeah, I can imagine that the fence probably has a top priority.
Yeah, at this point.
Yeah, Jupyter Lab looks really cool.
I haven't done anything with it.
But I've kind of looked and said, oh, this looks a little nicer than Jupyter.
Maybe I should start learning this.
I've been excited to see some of the Markdown plugins and things like that that they've been working on and actually being able to execute and test code within Markdown.
So that because I think there's a lot of utility there for the writing that we've done to make that process a lot easier for contributors.
So that I'm excited to start playing with.
Yeah, I really like this idea of testing your work.
And one of the things that I've seen as something of a detraction from the whole notebook way of working, and I definitely see the exploration and flow benefits.
But one of the things I see is less good is it's harder to say run tests over your Jupyter notebook or do code coverage of the code in your Jupyter notebook as part of those tests and things like that.
And you actually tweeted out a really cool project that when I first saw it, I didn't realize it actually had to do.
Its origins were in geophysics.
But over at github.com/open geophysics slash test I pi in B is a thing that lets you unit test Jupyter notebooks, right?
We've just started this.
This basically got actually pulled out of the GeoSci ecosystem because as a part of a lot of these courses, we were distributing notebooks.
And there were a whole bunch of different people who were contributing to these notebook repositories.
And then we were deploying them either on Microsoft Azure or MyBinder and then stepping up in front of a course and using them to teach.
And so, you know, when you're in front of a classroom, you really don't want errors popping up, especially if you didn't write the original notebook.
Yes, exactly.
Yeah.
So this was really born out of need to make sure that at least if you are standing in front of the class that the notebook should run.
And so what we've been trying to do here is we've extracted a lot of the work that we've done using NB Convert that just runs the notebook and make sure that it completes with no errors.
So I think there's a lot that we can think about to, you know, increase the utility of this.
But as a first pass, like just making sure that the notebook goes from A to Z without erroring, that makes sure, too, that you've properly defined all your dependencies and all of those pieces that are so easy to forget, especially, you know, to new contributors.
It's not always clear that, you know, if it works on my machine, why doesn't it work over here?
Yeah, exactly. So it tests things like that your Python environment has the dependencies installed and stuff like that.
Yeah.
Which that can be challenging in the whole data science, scientific computing space, right?
Another one of the reasons I wanted to have something like this out there is that we're sharing notebooks that go along with our publications.
And so a lot of them are built on Simpeg.
We know that, you know, down the road, we're going to make changes that are not backwards compatible.
And if you catch that right away, it's very easy to fix.
But if you let the notebook lag by a year, it's really hard to then go in and maintain and upgrade that.
And so part of this, too, was just to be able to put like the Travis Cron jobs that run once a month on our research notebooks and make sure that the research still continues to run.
Yeah, that's a really good idea.
Just run it periodically and just go grab everything new and see if it works.
Yeah, that's cool.
So do you have any way to do more specific data?
Like result validation?
So, for example, like if you have a cell, it would be great if it could convert it like to a function that you could call with parameters and get the response out or, you know, things like that, right?
Like not only does it still run, but it actually gives me the same results.
Yeah, I think that that would be super cool.
I mean, the simplest way right now to do that is like include an assert statement that like downloads your archive data set and make sure that you can still reproduce that.
But having something like that a little more exposed on the outside, I think, could be quite neat.
Same thing with even sort of checking figures.
So make sure that your figure looks the same.
Right.
Just like visually or, you know, pixel by pixel.
Just compare that the picture is the same.
Of course, when, you know, matplotlib updates to have like slightly faded cool axes, you know, that's going to break.
But you could just say, oh, no, no, this new picture is still OK for us.
We'll just upgrade that.
Right.
Or update the baseline.
Yeah, that would be nice.
But it sounds like a lot of work, right?
Yeah, absolutely.
Yeah.
But I think the biggest thing is just knowing when stuff changes.
So even if, you know, you compare these two things and they are different, but you can visually tell that it was just a style update.
Like, that's fine.
Yeah.
A friend of mine, Luan Falco, has this project called Approval Tests.
And I don't know.
It's integration with Python.
I know he was doing some there.
And it's all based on that idea.
Like, you write a test and you either get a result as a picture or as, say, a JSON document or whatever.
And you just go, yeah, that looks good.
And then all subsequent tests just go, is it the same?
Like, you don't have to do all sorts of testing.
You just feed it two pictures and it goes, they're the same, they're different.
And you can either reapprove the new one or it's an error.
Right.
And so, like, that idea, I don't know.
Maybe somehow these can be put together.
It sounds cool.
Yeah, that'd be interesting.
Yeah.
Nice.
So, one thing that sort of comes to mind in this whole space is there's got to be a lot of data that you're collecting to get all these pictures and stuff, right?
Like, the Earth is big.
So, what are some of the challenges around, like, big data in geophysics?
So, in a lot of cases, we're not necessarily encountering sort of the same style of big data problems that you think about when you think of, like, social media.
We don't have data sets that are that big, at least in our group.
Not like CERN, for example?
No, not nearly like that.
But a lot of what we're working with is small, disparate data sets.
And so, we'll have collected, you know, a whole bunch of different types of geophysical surveys over one setting.
And now we want to try and integrate all those different data types and figure out, okay, like, what is this telling us in terms of the rocks?
Like, what does that mean in terms of the geology?
So, that's one aspect where we're starting to see machine learning coming in in a very powerful way is actually trying to either take data that have been interpreted independently and then try and sort of merge those interpretations to then give you something stronger in terms of, like, I think this is rock A and this is rock B.
Rather than this is magnetic and susceptible, this one is conductive.
I see.
More like trying to draw the proper conclusions from the raw data.
Yeah.
So, really trying to drive much more so at, like, the geologic interpretation.
Yeah.
So, another thing I was wondering is how is machine learning being used there?
Because with these pictures and a lot of this data, it seems like somebody could come along and make some really interesting uses of TensorFlow or PyTorch or something like that.
Yeah.
I mean, I haven't seen a ton of neural network work yet, but that's also just my one sample point.
Where I have seen a fair bit of work done is on the clustering side of things.
So, really trying to either cluster interpretations or one of the things that I think is really kind of cool is when we start to meld together deterministic inversions.
So, that's a lot of the work that we've done in the past where you're running some sort of optimization problem to fit your data.
But then connecting that with statistical or machine learning approaches where you say, okay, I've got some sort of geologic knowledge about these rocks and how do I now couple that to my physics?
So, that's one area where I think there's a lot of potential.
Yeah.
We're just at the beginning of all this machine learning stuff, right?
So, who knows what it'll look like in 20 years.
Yeah.
It'll be wild.
So, it sounds like you do a fair amount of programming around your research projects and stuff.
And I know what the world looks like as a full-on software developer, right?
You write a lot of code and things like that.
But, you know, how do you think about balancing like programming and working on some of the libraries versus, say, research versus writing?
Like, I know right now it's writing because of the time frame and all.
But I feel like that's the question of any academic is how do you balance all these things?
So, if anybody knows, I would love to know.
But I think in a lot of ways what's been powerful with the group that I'm in is that a lot of this is collaborative.
So, that there are pieces that you're working on together in each of these aspects.
Because, I mean, as an academic, you do need to write about the things that you've been doing.
So, you need to do things.
And that involves, you know, programming and research.
And then you need to share that with the world.
And so, that's writing.
Yeah.
And I think having this open source angle to it, the collaboration really helps, right?
Because other people can do some of the programming where they have the specialty, right?
Absolutely.
And there's less duplication of efforts, especially on a lot of those like more mundane tasks of just like parsing data files and things like that.
Because there's no need for everybody to be writing that.
But you all need it.
Yeah.
Somebody gave me this thing in IDL out of this program or generated by IDL.
And like, we'll just import this thing and we'll read it.
We'll be good, right?
Yeah.
So, another thing that is a major in that space is getting sort of citations and credits, right?
Those are sort of like your upvotes for your career.
Yeah.
Are you familiar with the Journal of Open Source Software?
Yes, I am.
I'm actually the geoscience editor.
You are?
I didn't realize that.
Yeah.
Yes.
Okay.
That's awesome.
Yeah.
So, what are your thoughts?
I mean, obviously, you must be a supporter if you're one of the editors, right?
But just, you know, maybe tell people really quickly what it is since they if they didn't listen to, you know, my show a while ago with our fun.
So, it's really a developer-friendly journal.
And so, you know, I think it's a lot of people that I'm going to do.
In that sense, the biggest thing is you're actually sort of getting peer review much more so on your code development and practices.
There's a short paper.
It's like one to two pages.
So, it's meant to be pretty lightweight.
Like, that's supposed to be, you know, a couple hours work.
What we're really evaluating is the software.
So, I'm obviously very supportive of this.
I think it's fantastic to be getting credit tied basically immediately to the software.
Because that, I think, is a big piece that is missing in academia.
But one of the things that I've been really sort of stunned by, I've been with Joss now for almost a year, is just like how, what a positive process it is.
You know, I send out requests for reviewers.
And people not only say yes, they're very enthusiastic to jump in and learn about projects.
I've seen people, like, do small pull requests to fix typos or fix small things.
They're writing very well thought out issues.
And then the authors are so grateful to have somebody with a fresh set of eyes coming in and looking at what they've done.
And so, it's just been such an amazingly positive process in, like, in peer review, which is not something I've really seen elsewhere.
Normally, it's like, oh, I don't want to do this, right?
No, it's, I think it's a really cool project.
And I definitely wanted to give a shout out to it.
Because I think it really ties in well with, you know, the way you're thinking about the writing and structuring it in terms of software and open source.
And I think this is just the perfect complement to it.
Yeah, and related to that, Jose is the Journal of Open Source Education is starting up very soon.
In fact, they might actually even be accepting submissions.
It's parallel.
And I think there's a lot of interesting things sort of from the geoside perspective where we're now thinking about developing open education resources.
And they'll be accepting submissions like that.
So, a few avenues.
Yeah, that's really, really cool.
And I definitely think the work that you are doing, even though it sort of found its roots in geophysics, I think it could equally apply to biology or chemistry or lots of things.
Hopefully.
Yeah, that's the goal.
Hopefully.
With a slight adaptation.
Yeah.
Another thing that I ran across doing some research for the show is EarthPy at earthpy.org.
So, over at EarthPy, it's a collection of IPython notebooks with examples of earth science and the related Python code.
So, if people are listening and they're into geoscience of some sort, like maybe there's some really interesting things to draw from there.
Oh, cool.
I had not seen that before.
So, I will also be checking that out.
Yeah, that's really quite neat.
What's the IoT story in all of your research?
I mean, I can imagine like a bunch of little sensors planted around, but I don't know if you're involved in any of that.
I'm not involved in any of that, but there's definitely things coming.
There's a lot of interest in looking at the use of drones for smaller scale surveys.
I think actually, too, looking at more of the precision farming and precision viticulture for vineyards.
I think there's a lot of potential on those smaller scales for bringing in sensors that are giving you much more real-time feedback.
And you can make decisions about which areas need to be irrigated or not based on what you're seeing.
And you could get that information like, at 2 p.m., it needs to be irrigated, not at 3, right?
Yeah, exactly.
Like, that kind of, like, super detailed.
With the advent of such cheap connected devices, it just seems like, and the ability to program them so easily with Python, right?
Like, things like MicroPython and, like, $5 microchips that you can set up.
I don't know how long you've run those on battery, but they've got to last a long time.
So, it just seems like you could do such great stuff with that.
Yeah, and this is something I'm going to be very curious to watch and see what happens over the next few years.
Because I think a lot of these ideas are just starting to be worked out.
And people are figuring out how to make geophysical sensors a lot smaller.
Because that's been a problem for a long time, as they're, in a lot of cases, they're big.
But people are making progress.
It totally depends on the survey.
So, there's some, if you're trying to go out and do a gravity survey, it's very big, slow.
Because you need to go out, you need to level it, you need to stay level.
If the wind picks up, that's a problem.
That's a challenging one.
But looking at magnetometers or things like that, people can now start to make them only a few centimeters.
So, that's then something that is readily adapted to an IoT application.
Yeah, that sounds quite cool.
All right, well, I think maybe we'll leave it here for our geophysics talk.
But, of course, I have two more questions for you before you go.
So, if you're going to write some Python, yeah, if you're going to write some Python code, what editor do you use?
Or editors, if you use Horde 1, I guess.
Well, it'll be Jupyter and Sublime is the combination I use.
Nice.
Yeah, I figured Jupyter must be in there.
Although, maybe JupyterLab someday.
I don't know.
I'll see.
Yes, absolutely.
Nice.
And then, notable PyPI package.
I mean, we definitely have Simpeg out there, which people can install Simpeg.
And that's really cool that it's that easy, right?
Yeah.
So, what else that's, like, really people maybe haven't heard of that would be great?
I guess there's two.
So, there's Diskretize, which Simpeg is built on.
And that does a lot of finite volume meshing and all of that.
We've got Octree and Tensor Meshes and Cylindrical Meshes.
So, that's an interesting package if you want to simulate PDEs.
But one that I'm very excited about is called Properties.
And it's, basically, it does, like, strong typing, validation, serialization, all of that in Python.
But it does a really good job of helping you design an API that's meant to be interactive.
So, in the sense of Jupyter.
Like, you want to design an explorable API.
Properties is a great package to help you build that.
That sounds really cool.
And I've never heard of that.
That's great.
That's why I asked this question.
That's awesome.
Okay.
So, final call to action.
Maybe there's educators or scientists out there who would like to adapt some of the work you've done.
What would you say to them?
Yeah.
I mean, I think that if they have ideas and want to contribute directly to any of these resources,
we are always keen to have more contributors.
I think, too, that, you know, trying to build in a way that promotes community and build in a way that you can invite others to go in and contribute is such a powerful thing to keep in mind.
It's because then you yourself don't have to do all of the work.
And the end product that comes out is going to be better than any one person could have done.
And sort of the parallels to that, then, too, is if you see something that you have ideas about how to contribute to, get involved and get in touch.
People are always looking for contributors, and everybody has something to add, even if you don't know how they are approaching it or where to even start.
Yeah.
It doesn't have to be a major rewrite of some system, right?
Like, first contributions can be much, much smaller, just a little thing you can fix, right?
Yeah.
Absolutely.
All right.
Well, Lindsay, thank you so much for being on the show and sharing what you're up to.
It's been really fun to chat.
Thanks, Michael.
It's been good.
Yeah.
And good luck on the dissertation defense.
Thank you very much.
Yeah.
Bye.
This has been another episode of Talk Python to Me.
Our guest for this episode has been Lindsay Hagee, and it's brought to you by MongoDB and Anvil.
Interested in seeing how web apps are built with Python and MongoDB?
Register for my webinar I'm doing with MongoDB over at mongodb.com slash webinar slash Python.
See you there.
Anvil lets you build your web apps quickly and easily.
With Anvil, you can get your web app in Python up and running in hours, not weeks.
Want to level up your Python?
If you're just getting started, try my Python jumpstart by building 10 apps or our brand new
100 days of code in Python.
And if you're interested in more than one course, be sure to check out the Everything Bundle.
It's like a subscription that never expires.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct
RSS feed at /rss on talkpython.fm.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
Thank you.