In this episode, we cover how Python is being used to understand the electrical markets and grid in
Australia. Our guest, Jack Simpson, has used Python to uncover a bunch of interesting developments
as the country has adopted more and more solar energy. We round out the episode looking at some
of the best practices for high-performance, large data processing in pandas and beyond.
In addition to that, we also spend some time on how Jack used Python and OpenCV,
computer vision, to automate the study of massive bee colonies and behaviors. Spoiler alert,
that involved gluing wingding fonts on the backs of bees. This is Talk Python to Me,
episode 320, recorded June 6th, 2021.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem,
and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy,
and keep up with the show and listen to past episodes at talkpython.fm,
and follow the show on Twitter via at Talk Python. This episode is brought to you by Square and Linode,
and the transcripts are provided by Assembly AI. Please check out what all three of them are offering.
It really helps support the show. Jack, welcome to Talk Python to Me.
Thank you, Michael. It's great to meet you after hearing your voice for so many years.
It's so great to have you on the show. It's always fun to have people who are listeners,
but have interesting stories to tell. Come on the show, and you definitely have some
interesting stories about the energy grid and doing data science around really important stuff like
keeping the lights on in Australia.
Absolutely.
Yeah, I'm definitely looking forward to diving into that stuff. It's going to be a lot of fun,
I think.
Absolutely.
Before we get to it, though, let's start with your story. How did you get into programming,
and what brought you to Python?
Yeah. Well, I guess I have a very strange background. I actually started off at university
enrolling in journalism and politics right at the start of the GFC. I had never programmed before,
and I didn't even realize I was interested in it. And my lecturers kept telling me how many
journalists were losing their jobs during the financial crisis. And so I actually dropped out,
and I was trying to consider what I wanted to do. And I'd always had a passion for biology and science.
And my hobby was I was actually a beekeeper. I had six of my own hives at home. I really love that.
Amazing. Are these like honeybee type of bees or what kind of bees?
Honeybees. Yes, absolutely. The ones that stink. I actually also had a couple of Australian native bees,
tetragongula carbonaria, which I'm not sure if you've heard of them before, but they're actually,
they almost look like tiny little flies. They're stingless bees. And they will actually make their
hives out of the resin of trees. And they will build their brood in this beautiful kind of spiral
pattern going up through the hive. And so I was just really interested in, I guess, all things bee and
insects related at the time. And so I actually started blogging about bees and beekeeping. And
that was actually my introduction to code because I had, I think I had a website on blogger. And one
day I suddenly thought, well, I'd love to actually be able to make my own website. How do I do that?
And so I started learning HTML and JavaScript. And it was literally just so I could talk about my bees.
I had no interest in programming. Amazing. Well, I think so many people get into programming that way
who don't necessarily feel like my goal is to go be a programmer, but they just really have something
they're into. And programming is almost in the way, right? It's just like something you've got to figure
out so that you can actually get to the thing that you actually like. But then a lot of people find out,
well, hey, this is actually kind of cool. And what else can I do now that I know this, right?
Absolutely. That was really what made me change my degree. So initially I was going to do
a pure biology degree. And so I decided I would do biology and web development. And as I kind of went
along with the degree, I suddenly started realizing that the programming skills I was picking up during
my degree. So I learned, you know, PHP, Perl and Python. Suddenly I realized that these skills could
actually help me with working with scientific data. We kind of hit this point where there's just so much
genomic data. Really, most people at the time I was doing undergrad, but one of the things I've
really noticed is most people these days that enroll in a biology PhD, you join the lab and it's
almost like, right, you're learning Python or you're learning R. There's no other way you're working
with this data. And so suddenly I kind of hit this point where I was like, wow, these kind of technical
skills were letting me do things and be useful in ways that I never thought. And it let me answer
research questions that I was really fascinated by. And that was my motivation to actually, I guess,
go into and do a PhD and then try and take those skills further.
What was your PhD in?
So it was in computational biology. It was trying to develop software to automate the analysis of
honeybee behavior in the hive. So the thing that was interesting was it was both a physical setup and
the code as well. So the physical side was actually, how do we set up a beehive in a building with a kind
of like a glass window in so that I can film them with an infrared camera in the dark? And how do I
put little tags with patterns on them on the backs of the bees that I can then use Python and machine
learning to identify and track over the course of several weeks? And that kind of process ended up being
much harder than I had anticipated. Because when I started out, I read a couple of papers
by some computer scientists who mentioned that they'd printed out some card tags. And they said
that they filmed the bees for a couple of hours, got the data and did an analysis. And I thought,
great, I'm going to do that. Problem solved. It was only until later that I realized the reason that
they only filmed them for a couple of hours was because that was how long it took the bees to chew the
cardboard off each other in the hive.
I know. Oh, did they help each other? Like, hey, I've got this thing on my back, get it off me?
Yes. Yes, they actually did. They actually, and that was the thing I came, I would actually find time
and time again, I would come up with a material and I would try and stick it on the back of the bees.
And you would see their friends effectively come over and start trying to pry it off them in the hive.
And so it was actually a process to find something that didn't, I guess, trigger them, so to speak.
And one of the things, a really immensely frustrating experience I had when I was doing these experiments
was I thought I had found the perfect fabric and the perfect glue to put them on the bees.
And I'd spent hours tagging hundreds of them. I put them into the hive. And then I came back,
I would come back a few hours later and all my tags had disappeared. And I couldn't understand why. And I kept doing it.
And then at one point I thought, you know what, I'm going to put a bucket outside the hive entrance just to see what happens.
And I'm going to watch in the dark. And what actually happened was the bees didn't like the smell of the glue.
So they were actually physically grabbing bees that I'd tagged, dragging them to the entrance and flinging them out of the hive.
And because the hive, because these bees were juvenile bees, they were too young to fly yet.
The ants were actually dragging them away. So I thought my tags were dropping off or being pulled off.
But actually, my poor bees were getting eaten by the ants because they couldn't fly away.
Oh my goodness.
And so I guess it's another example as well. You know, when you've got missing data, understand the process.
Sometimes the process that made that data missing is significant in a way.
I would have never guessed. That's really pretty insane, actually.
And so the solution for dealing with this was I would actually go through the process of tagging the bees.
Then I would put them in this heated incubator on a frame of honey for several hours until all of the smell had kind of faded away.
And then I could introduce them to the hive and they would be accepted.
Oh, I see. Okay. Wait, basically wait till they dried and it was really on them.
Yeah. Yeah. Well, wait until all the fumes were gone and then they would be accepted and then it would work.
Because I think this was the real challenge of my project was we weren't interested in tracking, showing that we could write software that could track the bees.
We had a specific application was to look at the social development over several weeks of these bees.
So we needed a kind of experimental setup and the code to support it that would let us look at these extended periods of behavior.
Did they have different markings based on like their age or their role in the colony or something?
Like were they all tagged the same and you just said, well, they kind of move around like this or were they're like, did you group them or something?
What I would often do is I had use a laser engraver to burn patterns in the fabric that I would put on them.
And so each bee had a unique pattern that I could use to identify it.
Like a QR code on the bee?
Kind of like that, but not.
In fact, I think if you scroll through the website to the bottom of the page.
Yeah, there's some right there.
There's some little patterns.
This was some initial prototypes for the bottom.
Just if you scroll up a little bit more, the last image.
Yep, that one.
I was literally using a wingdings pod to try out different patterns.
Wingdings.
Okay.
On the bees.
Because I just had the idea was to have a relatively inexpensive 4K camera that could pick up the different patterns.
Of course, if you had a really expensive high resolution camera, then you could do more with QR codes, for instance.
And what I would do is I would do these experiments where I would, half of the bees would be that I would introduce.
They would all be juvenile, except I would also mark the queen so I could know how they were interacting with the queen.
But half the juvenile bees I would introduce into the hive would receive a label that I could reference later on.
And half would receive a different label that I knew about.
And the reason I did this was so I could actually do these, have these control and treatment groups in my experiment.
Because I would do these experiments where I would treat the bees with caffeine to see how it would actually affect their social development in the hive.
I guess to give a little bit more context to that, diving a little bit into the way that bees develop.
You could almost think of a worker bee in the hive like the pictures I have on my site.
The jobs that a bee does over its lifetime are influenced by how old it is.
So these juvenile bees I was first introducing to the colony.
They really would just have quite menial colonies.
They would do little cleaning tasks around the hive.
They wouldn't do much.
Then when they're a little bit older, they would start nursing other juvenile bees.
And then the eldest bees are the ones that you actually see out and about flying and collecting nectar and pollen.
So those are actually the eldest of the bees in the colony, typically.
And so I wanted to see how this caffeine would affect that kind of behavioral process in the juvenile bees.
How interesting.
Short, briefly, what did you find that caffeine does to bees?
One of the things I found was it effectively meant that bees sped up how quickly they adjusted to the rhythms of the colony.
So for a bit of context, if you're a juvenile bee in the hive, you don't really care about circadian rhythms, day-night cycles, because you're in a hive, it's completely dark all the time.
And so what we found was that we hadn't seen before was these juvenile bees, even though they weren't exposed to the light on the outside, they would actually pick up these circadian rhythms by interacting with the older bees that were coming back.
It was effectively like a socially acquired circadian rhythm.
And so what we found was that bees that were treated with caffeine effectively picked up this rhythm more quickly than bees that weren't and kind of progressed in their roles in the colony more quickly.
So there was that.
Okay.
Yeah.
So there was that.
And I had a few other areas, but yeah, to be honest, a lot of the work was really just making the software and the bees all play nice together.
Yeah.
Absolutely.
It was probably one of the most immensely, I will say one of the things that is quite nice about the energy sector is I don't have to deal with, I guess I can deal with machines, which are a little bit less frustrating at times.
More reliable, more predictable and certain, that's for sure.
This portion of Talk Python to Me is brought to you by Square.
Payment acceptance can be one of the most painful parts of building a web app for a business.
When implementing checkout, you want it to be simple to build, secure, and slick to use.
Square's new web payment SDK raises the bar in the payment acceptance developer experience
and provides a best-in-class interface for merchants and buyers.
With it, you can build a customized, branded payment experience and never miss a sale.
Deliver a highly responsive payments flow across web and mobile that integrates with
credit cards and debit cards, digital wallets like Apple Pay and Google,
ACH bank payments, and even gift cards.
For more complex transactions, follow-up actions by the customer can include
completing a payment authentication step, filling in a credit line application form,
or doing background risk checks on the buyer's device.
And developers don't even need to know if the payment method requires validation.
Square hides the complexity from the seller and guides the buyer through the necessary steps.
Getting started with a new web payment SDK is easy.
Simply include the web payment SDK JavaScript,
flag an element on the page where you want the payment form to appear,
and then attach hooks for your custom behavior.
Learn more about integrating with Square's web payments SDK at talkpython.fm/square,
or just click the link in your podcast player's show notes.
Before we move on to the energy sectors, just give us a quick overview of the software that you use.
Was Python part of this role here?
Yes, absolutely.
So I used a mix of Python and OpenCV for a lot of the image processing.
And of course, TensorFlow and Keras for training my neural network to identify the different tags.
And that actually ended up being quite an interesting process,
building up that data set and improving it over time.
Because one of the things I found when I started trying to train that data set was I thought,
okay, I can take my patterns, film them, add a little bit of noise and rotation,
and then that's my kind of starter, you know, machine learning model.
The problem was that when you put the tag on the bee,
the way that they kind of walk around the hive,
you'll see different kind of angles of,
they kind of have this bit of wobble walk as they go around.
So it kind of introduces this level of distortion to the tag.
And then other, and so then also you could have other situations where
bees would walk over each other, the block occluded tags as well.
So one of the things I ended up having to do was I had to introduce a class to my,
a predictive class to my model that was literally just like the,
I don't know what this is class.
And effectively, the idea was, I'm going to see this bee,
I'm going to have multiple attempts to classify this bee as it's walking around.
So I want to only attempt a classification when I'm seeing enough of the tag
and I'm confident enough in that to attempt it.
And so that was one of the,
one of the techniques I found that helped improve the classification.
And really it ended up just becoming a process where I would,
I had a bit of a pipeline that would go through,
it would extract tags,
it would use the model at the current iteration to label them.
I would then go in and manually review it and then figure out where it had stuffed up,
where it was doing well,
and then use that corrected data set to retrain the model and then improve and see how well that
iteration did.
And it became kind of like a,
almost like a semi-supervised problem to an extent when I was building it out.
And at a certain point,
it became just as good as me at doing these classifications.
And then it was fully automated as well.
But I think I ended up labeling about seven or 800,000 images as part of doing this.
And my,
my wife was actually,
she was a PhD student in working in genetics at the time.
She was helping me in her spare time.
So she does not look favorably upon that,
on that project.
She probably doesn't love wingding fonts.
Maybe in a bit of a surprise,
she's like,
oh,
not you again.
Yeah,
absolutely.
But I'd say,
yeah,
so Python and OpenCV were big ones.
And then the other tool I was using a lot of was Python Cython library,
where I would,
for certain parts that I wanted to run really efficiently,
I wrote those in C++ and then used Cython to expose some of those methods to it.
And that worked amazingly well.
It was so impressive how you could call,
pass a list,
a Python list to my,
my C++ class,
and it would interpret that as a vector.
And then it would pass back the information as well.
I think this is the reason I'm such a fan of Python is just how well it lets me
do so many different things that I'm working on.
That's a really interesting point.
you know,
a lot of people talk about,
well,
Python is slow for this or it's slow for that.
And yet here's all these really intensive computational things that Python seems to be
the preferred language for.
And I think this is one of the hidden secrets that's not apparent as people come into the
ecosystem,
right?
Obviously people have been here for a long time and they,
they kind of know that story,
but you know,
as people come in,
because there's,
there's all sorts of people coming into the Python world drawn in a little bit like
you,
you talked about how you started out in biology,
not necessarily to be in software development specifically,
but then you kind of got sucked into it.
Right?
Absolutely.
Yeah.
I think all of the conversations around the performance of Python is super interesting.
It's like,
oh,
it's,
it's really slow,
except for in this time where it's like as fast as C++.
Wait a minute.
Is it,
is it slower?
Is it fast as C++?
Well,
it's both,
right?
it varies,
but you can bring in these extra like turbo boosts,
right?
Like Cython and,
or do your work in NumPy rather than in,
in straight lists and stuff like that.
Absolutely.
And like one of the,
initially when I started off my PhD,
I actually wrote an initial prototype version of it all in C++ using open CV,
open C++ library and a machine learning,
a deep learning library called cafe,
which was a bit of a thing back in the day.
And the pro that process for dealing with data and even just converting data
between like,
I think the best thing about Python is the fact that NumPy arrays is just
understood by all the scientific libraries,
whereas sometimes with other languages,
it can be painful moving data between different libraries and tools.
Oh,
interesting.
Yeah,
you're right about that.
Yeah.
And so like,
I,
I remembered at one point during my PhD with,
with that initial C++ version,
I had like a page of code to convert between an open C,
the matrix and a cafe,
I think blob.
And it was a page of code that I was terrified of breaking because I didn't understand how it
worked.
Whereas Python,
it was like everything I can move between,
you know,
scikit-learn,
pandas,
and all these other libraries.
And it's all kind of got that common foundation that makes me really efficient.
Yeah.
And that I understand really well.
That's a really interesting insight that there's this sort of common data structure across the
libraries,
because you're right.
I remember in C++ and other languages like C# and whatnot,
this one will take something like this and you've got to reorder the data and reformat it to pass it over.
And if you have to do that back and forth,
it would completely slow things down.
All sorts of stuff.
Yeah.
Very interesting.
In a way as well.
I really loved that the Python stack has let me do things in during my PhD and then post PhD as well.
In just the skills that I developed in analytics here,
I've gone on to be able to use that in so many different places.
For instance,
one of the pieces of analysis I did was look,
I used Python's network X library to look at the social interactions between the queen and worker bees.
And I would build out these network graphs that would explore the number of interactions and the length of time of those interactions between the queen and the worker bee.
And this actually recording independent interactions actually became important because sometimes the queen would literally fall asleep behind another worker.
And it would look like she loves that worker,
but she just was resting for like over an hour or two.
What I've actually found is that those skills for working with data and with network analysis,
when I was working consulting,
I would use network X to analyze the corporate structure of organizations that we're doing an org review for.
And then more recently I have done work in the energy sector,
looking at building out networks of power stations as well.
And so it's,
I think that's,
that's one of the things I love about this area is that you have this kind of transferable skill set that you're more limited by what you can think of,
but using it by rather than what you can actually do with it,
with it itself.
Yeah,
absolutely.
And I think for a lot of people,
if they're out there listening and they're doing,
you know,
academic type stuff or working in one area,
but maybe that's not the area they necessarily want to stay in.
A lot of these skills are super transferable.
One of the things that's blown my mind as I've spent more and more time in the software industry was,
I remember I was doing professional training and I spent one week at a stock brokerage in New York City teaching programming.
And then I spent two weeks later,
I was like at an air force base working with some of the engineers there.
The stuff that those two groups need to know,
it sounds like it's entirely different worlds,
right?
It's like 90% identically the same.
It's just a little bit of what do you do with that once you know it?
Like what's the secret sauce on top of it that puts it together?
But yeah.
And it sounds like you kind of got that skill in your research.
Absolutely.
And I think this is one of the things I've noticed is that some PhDs can struggle to transition into industry.
And often it's because there's people on the industry side that don't really understand how those skills can help them.
But at the same time,
I think it's actually a skill to be able to explain how you can link what you already know,
what you're capable of and solve their kind of business problem.
And in fact,
I think when I went into management consulting and I would do some work for some of the partners,
eventually it took me a little while to figure out that they weren't that interested in,
you know,
the code I was doing or even some of the raw data.
But if I could figure out a way to link that to the business problem they were trying to solve,
then they were interested.
Yeah, absolutely.
Being able to kind of communicate and act as like a bridge between those was something I didn't realize was a skill,
but it is hugely valuable in organizations.
I've really noticed.
Yeah, absolutely.
All right.
One final question about your research before we get into the energy sector.
That what year did you do that?
Oh,
so 2014 to 2017.
Yeah.
And that's not that long ago.
And yet the machine learning story has probably progressed really quite a bit with deep learning,
transfer learning,
all sorts of stuff going on,
the different use of GPUs and tensor compute units and whatnot.
What would it look like now if you're doing it versus then?
What would be different?
I think now one of the big differences was really that TensorFlow only came out towards the second half of my PhD.
So I think a lot of the,
so I think that was a difference.
Having more accessible machine learning libraries and tools really made a big difference.
The other one was,
I think when I started my project,
I actually spent a lot of time playing around with,
you know,
now if you started your PhD,
you would go image analysis,
it's going to be deep learning.
Whereas when I started,
I was actually pointed in the direction of,
oh,
go check out,
you know,
support vector machines,
try out a random forest,
try out a whole bunch of different feature engineering and machine learning techniques.
And so I spent a lot of time kind of moving around between those before I literally had a,
I got in touch with a researcher in the computer science department because I was in the biology department doing this work.
And he literally,
I had a chat with him and he literally looked at what I was doing and he said,
use deep learning.
And he said,
go check out,
check out these libraries,
but this is what you need to do it.
And I think,
yeah,
in a way like that type of,
the libraries and the understanding about how you would solve this problem now is,
is a lot further along and probably would have shortcut a lot of my initial frustration compared to previous.
Yeah,
probably.
But think of all the lessons you've learned with those late nights of it not working and,
and whatnot.
Right.
One other thing really quickly is I love to look at this graph here,
this,
the stack overflow trends,
and I'll link to this in the show notes.
There was,
back in 2017 article by stack overflow,
their data science team set called the incredible growth of Python.
And they predicted,
Oh,
Python's going to overtake some of these languages.
And you're not going to believe it.
It's going to be more popular than JavaScript,
more popular than Java.
And people are like,
no way.
This has got to be something wrong with the data.
And obviously,
here we are,
you know,
in 2021,
where I think they underestimated.
Honestly,
I don't want to have the exact picture in my mind,
but I'm pretty sure they underestimated the last couple of years,
which is pretty interesting.
But that's not what I want to talk about.
What I want to talk about is that 2012,
you know,
Python had been around for at that time,
25 years or something.
It was well known.
It was a fairly popular language,
but it was kind of just steady state.
And then it's like somebody just lit the afterburner on that language.
And it just,
you know,
it just started going up and up right around that time.
And this is the time that you got into Python as well,
more or less,
right?
Absolutely.
I feel like so many people came from these not traditional programming spaces.
I mean,
still interested in programming,
but not like a CS degree type of programming in.
And it just brought so much diversity in terms of the problems being solved.
And I think this graph is exactly what's happening here.
It sounds like you're part of that,
making that curve go up there.
Yes.
Yeah,
I guess so.
And I think for me as well,
when Pandas came out,
I think around in 2012 for working with,
you know,
data frames as objects,
I've used R.
I really liked that kind of data frame feature in R initially.
And it was a little bit frustrating before Pandas was a thing,
being able to,
having to deal with,
you know,
CSV files and having to treat them as lists and indexing.
So when Pandas became a thing,
that was almost one of the big reasons I pushed into using Python for so much.
And I still feel like I've been using Pandas for,
yeah,
I guess eight or nine years.
And I'm pretty sure the project I'm on currently,
I'm pretty sure I've learned a few extra things about the library just in the last couple of weeks.
Yeah.
It's crazy how that works,
right?
You're like,
I've been doing this forever.
How did I not know about this part of it?
Right?
Absolutely.
Amazing.
All right.
Well,
super cool project you had there.
Let's talk about energy.
So you work for the Australian Energy Market Commission?
Yes.
Yeah.
What's this?
What do you do there?
You can almost think of them as the rulemaker for the energy market.
We don't run the energy market.
That's the Australian Energy Market Operator.
But they effectively pass the legislation that determines how people have to act within the
energy market.
The reason I really joined the organization was because when I was working in consulting,
I started doing work in the energy sector.
And I do work for energy retailers, the people that you pay for your electricity.
I just work for some industrial companies.
And one of the things I found was when I bumped into the wholesale energy data, it was almost
like this, what was it?
The, you know, the city of gold in some way.
It was immense amounts of reasonably well-structured and cleansed data where the limitation wasn't,
you know, the data or cleaning it.
The limitation was understanding the domain well enough to do interesting things with it.
Right.
Okay.
And so that's really became my obsession was to learn as much as I could.
So I could actually do more and more interesting things with the data.
And so the reason I joined the AAMC was because it's one of the most amazing workplaces in terms
of the capability of everyone there is so passionate and incredible at what they do.
And so just being around these people and learning from them is just an experience in itself.
Yeah.
Fantastic.
That sounds super interesting.
And it sounds like things like your network experience, you know, there's probably a lot
of networks and energy and suppliers and whatnot.
There might go together.
Absolutely.
Yeah.
Basically there's, it's kind of a market that they set the price energy and then generators
like private companies that are, you know, have power plants and solar forms and whatnot.
They can decide whether or not they want to participate at that very moment in the grid or how does
it work?
Yeah, absolutely.
So yeah, this is one of the fascinating things about the wholesale energy market.
You can almost think that every five minutes, the market operator is effectively running
an auction where all power stations around on the East Coast of Australia are bidding in
bids for how much they were willing to sell different volumes of electricity at.
So for instance, a wind farm might say that they will sell this volume of power quite cheaply,
whereas a gas generator that has quite a high cost of fuel will set a higher price.
And the market operator will take all of these bids and it knows the locations of these generators.
It knows the capabilities of the transmission lines and the network.
And it will run this linear optimization to figure out, okay, what is the cheapest mix of
generators that I should dispatch to satisfy demand while still making sure the network is secure?
Okay.
So it's like trying to optimize certain goals.
Like we are going to need however much energy in the grid at this very moment.
And these people are willing to supply it at this, like, you know, who do we take however
much energy from until we get like both enough people that are willing to participate from a
financial perspective and then what people also need, huh?
Yes, absolutely.
And that's the thing that's so fascinating about this market is that at all times, supply
and demand have to be matched.
Very, very carefully because it'll break the grid if there's, well, too much is probably
worse than too little because you just get a brownout, right?
But too much could destroy things, right?
Yeah.
You don't want too much.
If you have too much, then you need generators to start to try and reduce the output.
And at the same time, if you have too little, then it can also create problems.
And in fact, the grid has to be kept at this such a precise level of balance that if it
actually you have too much or too little for too long, it will damage the machines that are
connected to it.
And in fact, to protect themselves, you will actually see them start to disconnect and it
can actually create these kind of cascading problems.
So unless you, we actually had a fascinating example recently in Queensland where a turbine,
a coal turbine blew up and it then tripped a whole bunch of other coal power stations that
then couldn't, that then stopped creating load.
And so you suddenly had this situation where you had all this demand for electricity and
suddenly they just lost all of this generation ability.
And what actually happened is the system just started disconnecting.
Well, it caused a blackout.
But there was this automated system in a fraction of a second that just started disconnecting
load or demand to try and balance it as quickly as possible to try and arrest the problem.
So, and one of the things I've actually been doing has been looking at this at like a, on a four second basis, the events that happened on this day and how different units responded to these events.
It's amazing.
Like there's, there's almost like the energy sector and the market.
It's almost like there's the physical infrastructure of make and making everything work and all that amazing engineering.
And then there's the financial market and the bids and everything like that, that's built on top of it.
And the market and the bids are fascinating, but at the end of the day, everything has to bow to the engineering.
It has to work.
Right.
It has to work.
Yeah.
Or it's all just going to come apart.
Interesting.
Yeah.
AR out in the live stream says, is AEMC doing anything with energy web?
I'm not sure if I've come across that before, but I'd be interested in looking into it.
Yeah.
And then also, it sounds like DERMS, what you're describing or D-E-R-E-M-S.
I'm not sure whether, how you pronounce it.
That might be something, an acronym from the, from the, the U.S. energy markets.
Everyone has their own kind of different acronyms.
Oh yeah.
That makes it easy, right?
To not even be consistent.
If you go to the, the market operators website, they have a glossary page where you can just
scroll for, for all those days on all the acronyms that are used in the sector.
Probably like an acronym thesaurus.
We call it this, what do they call it?
Exactly.
This portion of Talk Python to Me is sponsored by Linode.
Visit talkpython.fm/Linode to see why Linode has been voted the top infrastructure
as a service provider by both G2 and TrustRadius.
From their award-winning support, which is offered 24, 7, 365 to every level of user, to the ease
of use and setup, it's clear why developers have been trusting Linode for projects both
big and small since 2003.
Deploy your entire application stack with Linode's one-click app marketplace, or build
it all from scratch and manage everything yourself with supported centralized tools like Terraform.
Linode offers the best price-to-performance value for all compute instances, including GPUs,
as well as block storage, Kubernetes, and their upcoming bare metal release.
Linode makes cloud computing fast, simple, and affordable, allowing you to focus on your
projects, not your infrastructure.
Visit talkpython.fm/Linode and sign up with your Google account, your GitHub account,
or your email address, and you'll get $100 in credit.
That's talkpython.fm/Linode, or just click the link in your podcast player's show notes.
And thank them for supporting Talk Python.
So we have on the graph here, this picture on the screen, where energy went negative, actually.
And so this is where people are willing to pay to take energy that you've generated?
Like, that sounds completely insane.
Yeah, I know.
It sounds weird.
So yeah, to explain this figure, what's been happening this year in South Australia, the
wholesale price of electricity has been around, averaged around negative $20 during the middle
of the day, pretty much consistently.
And so the way this works is because the generators submit bids for how much they're willing to
sell their electricity for, because they'll effectively, when they run the optimization,
the price of the bid that satisfies demand is the price that everyone gets paid.
So what a lot of generators will do is they'll bid in quite cheaply at negative prices so that
they're sure that they will get dispatched.
But if everyone bids in a negative prices, then everyone gets the negative price.
And so what we've actually been seeing is because there's now so much generation in the middle
of the day, you're ending up with these really fascinating market events.
Like, yeah, these negative prices where literally you can get paid to consume electricity.
As a consumer, that sounds pretty good.
You know, get it nice and chilly and we'll all be fine.
True.
Yeah, that's true.
One of the drivers of this, it sounds to me like, is solar energy in Australia, right?
Yes.
We now have so much rooftop solar.
I can't remember the exact percentage, but a significant percentage of Australian households
now have solar panels because the cost has come down so much.
And so a lot of our work has involved looking at how that is impacting the grid.
Because if you imagine historically the energy market, it was a process where, you know, the
market operator could instruct generators to turn on or turn off.
And now we're in a world where there's so much of these kind of small scale solars that
solar that you can't tell what to do.
How do you factor that into balancing supply and demand in the grid?
Yeah.
Well, it definitely sounds like some interesting Python must be at play there.
So give us an overview of sort of the kind of tools you're using, the types of problems
you're solving.
Yeah, sure.
In the solar place, we've been using a Python, a software package called SAM, which is a system
advisor model, which is actually released by the National Renewable Energy Laboratory
in the States.
And so what it lets you do is if you provide solar irradiance data and data from weather
stations, you can use it to simulate the generation of rooftop in different areas around the country
on a granular, on a half hourly basis over the course of the year.
And so what this lets us do is I can use, they've got a Python library that lets me kind of call
and run this tool.
And I can simulate different PV system sizes and different locations and angles and all
for setups all around the country.
So I can effectively simulate hundreds and hundreds of different PV systems.
And if I combine that with how much the household is consuming and what the actual cost of electricity
was and those half hourly intervals, you can suddenly build up a picture for the economic
effect of different PV panels for different households around the country.
Yeah, how interesting.
Is this the right thing I pulled up here?
This Pi Sam?
Yes.
And I will say that for your US listeners, the laboratory release all of the data for
the US mainland in a format that's ready for you guys to go.
I had to, a big part of my project was actually trying to turn the Australian data into a format
that this program could understand.
And that in itself was an interesting exercise in data cleaning and manipulation because, for
instance, all of the data on the irradiance for the country came as tens of thousands of
these text files that were just these kind of grids, which pretty much they said, each value
represents a five by five kilometer grid on the Australian mainland.
It starts at this coordinate.
So I had to pretty much try and convert this text file into a map and then convert that into
format so I could know where the house fell in that as well.
Oh, wow.
How interesting.
Yeah, you're normally taking a bunch of text files and piecing those together in a map,
but I guess you do.
Eugene, who was on the show a little while ago about the life lessons from machine learning,
brought an interesting quote.
It was something to the effect of the data cleaning is not the grunt work.
It is the work of so much of this, right?
Like it's getting everything right, making sure it's correct, converting it, formatting it,
and then you feed it off to the magic library and get the answer, right?
Yep.
Absolutely.
And I think, yeah, the lessons that you learn from working with and cleaning the data often
help inform your analysis later on.
For instance, one of the things I was doing recently was I've been trying to correct for
errors in this really large data set measuring output from these power stations.
And so one of the pieces of advice I received was that if I see a data where the generation
value from the power station does not change, it effectively says this power station is generating
100 megawatts and that value doesn't change for more than a minute, for at least a minute,
that means there's an error in the data collection.
And so as I was cleaning up the data and I implemented that and I started looking for outliers,
and I actually discovered that you could see for some solar farms that it looks like if I
used this metric that I'd implemented to pick out the bad data, it was actually removing
cases where the power station, the solar farm was deliberately keeping their output perfectly
level to match this instruction from the market operator.
And so I think this is a case where they were actually foregoing additional generation to be
more predictable.
And I would have missed this whole interesting power station behavior if I just, you know,
if I wasn't thinking about what the implications were of these different, you know, cleaning
techniques that I was doing.
Okay.
Yeah.
Because maybe that advice comes from, I don't know, a gas power plant or a coal plant where
they have to fluctuate because, you know, whatever reason, right?
And this new world, the assumptions changed or the situation changed and the assumptions didn't,
right?
Absolutely.
And I think, yeah, like in a way, like part of the reason I think I've always gravitated towards
being passionate about combining the programming and the analytics with like deep domain expertise
is that I really love when I'm working with the data set, when I see something weird, I
love that I can go, that's wrong.
I can remove that or that looks weird.
I'm going to investigate this because I think that's interesting.
And one of the things I found in consulting was the projects where I didn't understand
the data or the industry as well were always a bit, and I was brought into the team to provide,
you know, the analytics capability, but I was effectively, you know, turning the understanding
of others into code.
I've always found them a little bit less satisfying from a personal perspective because I didn't
feel like I was the one who was really, you know, getting, I felt like I was a vehicle other
people turn their thoughts into code.
Whereas I really like that if I understand the domain, then suddenly I can investigate and
understand the area that I'm working in.
Yeah.
It becomes a puzzle, not just, I don't know, just more, get information from these people,
apply it to the data, see what comes out.
Yeah, for sure.
You know, someone asked me recently, they were looking to hire somebody.
It was, I don't know if it was exactly in the data science world, but it's close enough.
They were asking something to the effect of, should I go and try to find a computer science
type of background person who I can then teach the subject matter to and kind of get them
up to speed there because we need good programmers.
Or should I find some people who really understand what we're doing and then try to teach them
Python?
Yeah.
What would you say to that?
I had a thought on it, but I'd love to hear yours.
I think to a certain extent, the experience, you want to have, I guess, the passion for learning
about the domain.
And obviously, if they understand the domain, that's really valuable, but you probably want
them to be exposed at least a little bit to some programming concepts for them to know
that they like it.
In fact, I remember when I had a chat with my former boss who hired me into my current
role.
And he said that he's hiring philosophy is he looks for people with interesting backgrounds.
You know, my background, he saw computational biology.
And, you know, a lot of people would be like, oh, how does that apply to the energy sector?
Yeah.
That does.
That's not for us.
That's something totally different, right?
Exactly.
But he said, you know, for him, that's an interesting story.
And he could see how those skills can generalize to different areas.
And then it's more about, are you passionate about the thing you're working on as well?
So like, I think people can learn, you know, people with domain expertise.
I think learning Python can be like adding a bit of a superpower to your, you know, your
skills and domain skills as well.
But I also think that you wouldn't want to say, for instance, you wouldn't want to hire
someone who had good domain expertise into the team to be a programmer who'd never programmed
before and discovered they hated programming as well.
Yeah.
I think the assumption was that they had a little bit of programming experience or they were super
interested in it, but maybe not all the way there.
I think the subject matter expertise is really valuable.
It's, I think these days there's so many amazing libraries and Python so accessible that it's
really important to understand like deeply what's happening.
But you should probably also have one or two people who have like a true software engineer
experience like, hey, has anybody told anyone around here about Git?
We need to be using source control.
And what about continuous integration?
And have you heard of testing, right?
Like those kinds of things matter.
But I think also having this like deep understanding of it really matters.
Absolutely.
Cool.
All right.
So are there, I mean, you talked about this system advisor model.
Are there other things like in say the astronomy space, there's AstroPy, like all the astronomers
talk about, this is the library.
These are the things you talked about pandas and numpy and whatnot already, but is there something
like that or a couple of libraries like that in the energy space?
The closest I would probably say is the Pyomo optimization library.
So that I think Clark mentioned in a previous interview.
Yeah, yeah.
We had Clark come on to talk about that.
And he was doing really cool stuff.
Clark, yeah.
I'm trying to, I'm going to set up a chat with our team with Clark.
And that's the plan at a later date.
Because, yeah, it was very interesting that what he was able to do during his masters with
linear optimization.
And so I think, yeah, like really, there may be libraries out there that I haven't come
across yet at this point.
But I've really found that the whole, yeah, the Python stack of your pandas, Pyomo for
optimizations, and even things like, have you come across a library called Geopandas at all?
Yes.
Which adds spatial elements to data frames.
I use that for a lot of analysis.
Yeah.
Geopandas sounds cool.
I haven't done anything with it, but I would love an opportunity to do something fun with
Geopandas.
I did that when I was working in consulting.
I used that library once for looking at data from the Australian Bureau of Statistics.
And then suddenly, I was in demand for every proposal to be making these heat maps of the
country.
I suddenly was just making, I had heat maps coming out everywhere.
Yeah, they're like, if Jack knows how to make these graphs, give it to Jack.
He'll build a flag.
Exactly.
But Geopandas, if you know a bit about using, you know, pandas and data frames for working
with data sets, it's pretty much like using a pandas data frame, but it just adds a whole
bunch of capability for working with spatial data sets and creating beautiful figures as
well.
It's amazing.
Yeah.
Oh, it sounds super cool.
Super cool.
Yeah.
It works with Shapely.
It sounds like it would work really well with your 10,000 text files almost even.
Yeah.
Some of those things linking it up.
Yeah.
Alexander out in the live streams coming back.
Just one quick thought says, I wish people learned at least some programming.
Making customs software to cover simple cases is definitely tiring.
And most of the time, it's just a simple script.
Yeah.
I mean, kind of the automate the boring stuff could take a lot of people a long ways for sure.
Yeah.
At that level.
Absolutely.
In a way, a lot of the little things that I would go around and be useful for when I
was working in consulting, if people had a little bit of programming background, then
yeah, they almost wouldn't need my input because they understood the area better than me.
And if they had a little bit of Python and knew how to link up some data sets, then it would
be like they could just automate so much of some tedious things in their lives.
One thing I heard a lot in that sort of realm was, if you automate all these things, you're
going to take our jobs away.
What are we going to do?
Like this painful, tedious manual stuff that should be automated.
Like that's our job because that's what a lot of the people that I had worked with for
a while.
That's what they did.
And they were legitimately a little concerned that if we wrote software that would do those
things automatically, well, then what would they do?
And I saw year after year, we would write that software.
They would say, thank goodness, we don't have to do this again.
And they would just solve more problems, take on more data.
Like they would just do more and almost never did it result in, well, we don't need these
people anymore.
It just meant they got to do more interesting stuff in a bigger scale.
Absolutely.
Like this is what I find that when I work with new data sets or problems, and once you've
kind of, you know, solved the problem, you understand that data, you fix the issues with
it.
Suddenly having that kind of foundation and curated data set lets you actually build on it and do
more interesting things going forward.
It's not like, you know, you've done that.
You can, you know, you never need to do that again.
Yeah.
And that's what drew me to the energy sector because it was like, the more I worked with
these data sets, the more I understood and the more interesting questions I could answer,
which is really satisfying.
Yeah.
And the more things that are batch processes can become almost real time and really change
things.
So speaking of data, it sounds like you guys work with a ton of data over there.
Give us a sense for the scale.
Yeah.
So I guess the more standard data set is there's a database that has pretty much everything
going on in terms of dispatch on a five minute basis.
And so for most of your users, if you just want to see what the power station is doing,
what it's bidding, you can use that data.
It's large, like pulling out some of these data sets in, it's in, you know, a hundred or
200 million rows looking at certain parts of it.
But the thing I'm working on at the moment, it almost makes this kind of look small.
And this is kind of that same data from that same database, but it's on a four second basis.
So a single month of data is about 750 million rows.
And it gets all released as thousands of zip holders containing CSVs.
One CSV for every half hour.
Oh my goodness.
And so is there like a big process that just goes along, unzips it, grabs it, inserts it
into some database or something along those lines?
I think that was how it gets shared in some format.
So what I actually, so this is how I was given the data on my current project.
And so it's so big, I can't actually unzip it on the machine.
So I have to use Python to kind of spin up a number of separate processes that will kind
of work through the different zip folders.
It will then use, I think Python has a library called zip file.
So it will unzip the folder in memory, read in the CSVs, process them, and then it will
eventually concatenate it all back into a cleaned data frame that I can work with going forward.
And so, and so then I'm trying to use those sets of tools to try and then turn this into
a more compressed, cleaned data set that I can work with going forward.
So does that fit in memory or do you have to like only pull out slices sort of dynamically
with the zip file processing?
Yeah.
So I can fit about a month in memory on my machine.
We do have some large servers.
And so I will transition to processing this in parallel on the servers, which should get
an even better speed up.
But yeah, at the moment, I really just kind of look at things on a monthly basis.
And so what I can actually do is, and then there's a ton of processing I have to do with
this four second interval data, because what I can see is I will break the data up then into
five minute intervals.
And I, what I can do is I can see what the generator was doing on a four second basis.
And then I can see what its target was.
So when they run their optimization, they will say, we know you're sitting at this point here,
you have to ramp up to hit this target here at the end of this five minute period.
And so I can use this data to tell how well the generators are actually hitting their targets
and how well they're following instructions.
But the funny thing is, even though, you know, you think of four second data as being, you know,
very, very, such a short interval of time.
But if you look at some of the big batteries in the grid, that data is actually too slow for
some of the batteries, because batteries can actually turn on, inject power into the grid.
turn off again, and I can miss it in the four second data.
It's amazing.
Some of these big grid scale batteries are like the big Tesla battery in South Australia at
Hornsdale.
They're amazing feats of engineering that you really appreciate when you realize you're
missing things at a four second interval.
They break your sensors and things like that.
Yeah, Australia is really well known for having some of these big batteries in the energy sector.
I think for some reason, Tesla seemed to have partnered up with you guys to build these.
Yeah, the plan is to roll out a bunch more of these batteries around the grid as well.
And they're just really impressive.
You know how I mentioned the whole challenge of constantly balancing supply and demand?
And really, that's what batteries are so good at doing.
Right.
Their response time is almost instant.
Yeah.
So you could just, they could take it in.
Exactly.
They could eat the energy or they could initially fill, like immediately fill the gap, right?
For a while.
Exactly.
And sometimes with some of this data, you can actually see the battery will receive an
instruction.
It will quickly turn on.
It will discharge some power.
And then a couple of seconds later, it will actually then, because there's too much power
in the grid, the battery will actually then suck up some of that power and recharge and
do the opposite effect.
It's just amazing.
Yeah.
Fantastic.
I would love to dive into that, but let's say I'm super fascinated with batteries and
their potential.
Talk Python to me is partially supported by our training courses.
When you need to learn something new, whether it's foundational Python, advanced topics like
async or web apps and web APIs, be sure to check out our over 200 hours of courses at Talk Python.
And if your company's considering how they'll get up to speed on Python, please recommend they
give our content a look.
Thanks.
With all of this data, you said that you had basically learned some good advice, like certain
things you can just easily do in Pandas and NumPy on small data sets, maybe not so much on
large data sets like that.
Give us some of the things that you found to be useful and some of the tips and tricks.
Absolutely.
So there's a concept called vectorization.
I'm not sure if you've come across it, but it's effectively, how can you apply an operation
to a whole column?
So you're not writing a manual loop or using conditionals.
For instance, if I try to multiply a column with millions of rows by a number, it's really,
really fast because that's all kind of optimized to see under the hood.
And so with a lot of this, when I'm working with smaller data sets, you can get away with
doing some manual loops yourself or using Pandas to group by a column.
For instance, I would often say, this is the identifier for a power station.
I want you to group by this column identifier and then sum up.
And even those things start to become too slow once your data is kind of at this scale.
And so the real trick I find is, yeah, it's how do you find ways where you can apply some
operation, a calculation to the whole column?
And, but the tricky part with that starts to be what happens if you want to do conditional
calculations?
And so one of the things I find is sometimes I want to see how much the output of, on a four
second basis, how much is the generation of a power station changing?
And so you can imagine that Pandas has a calculation that lets you effectively calculate the difference
between the previous value that came before really, really efficiently.
But because, you know, I've got all these different generators and intervals, like kind of all
in the same, you know, data frame, I don't want to consider the first value in a five minute
interval, because that's affected by, you know, a different time interval as well.
So what you can do is NumPy has these great functionality called where or select, where you can pretty much
pass it a column that turns out to be true or false for the whole data set.
And it will then replace the value with something else really efficiently.
So what I can do is I can, I can run my calculation for the whole column.
And then I can use a NumPy where to replace the first value in each five minute interval with a
missing value.
And that pretty much does things in, you know, a few seconds that would have taken, I don't
even know how long with the other way hours, at least.
Wow.
It's amazing.
Yeah, I think that the whole computational space with pandas and with NumPy's, there's,
you know, like in Python, we speak about Pythonic code, right?
You would use a foreign loop instead of trying to index into things and so on.
And then there's a whole special flavor of that in the pandas world, right?
And a lot of it almost has the guidance of if you're doing a for loop, you're doing it
wrong, right?
Like there should be some sort of vector operation or something passed into pandas or something
along those lines, right?
Absolutely.
Like, yeah, it's almost like its own kind of type of problem solving in a way, because it's
like, how can I apply this calculation to everything in a column, but also in these cases, do something
else.
That's really the problem solving in a lot of ways.
Yeah.
It's a lot more set-based thinking, almost like databases.
Yeah.
What about things like threading or multiprocessing or stuff like that?
Like, have you tried to scale out some of the things that you're doing in that way?
So yeah, so we have a server that has about 60 cores and about 700 gig of RAM on it.
So that's the plan is I can shift my things over there once this project-
60 cores?
That's pretty awesome, actually.
Yeah.
We've got a couple of them, which is very useful for all of the energy modeling that we do.
And usually what I'm doing is kind of a mix of, yes, using Python's multiprocessing library
to try and, yeah, just split.
Usually what I'm doing is I'm just processing a whole heap of data frames in parallel and then
concatenating them back into a single data frame once they're kind of processed.
That workflow seems to work pretty well for a lot of the requirements that I have on projects.
Yeah.
Because you can get each subset data frame bit to do its own computation in parallel, right?
Yes.
Yeah, exactly.
And the other thing too that can also benefit is that usually as part of the cleaning process,
I'm kind of subsetting the data as well.
So while the data is starting off at an immense size as well, I'm figuring out which parts of
it I need and cleaning it.
And then so then the data frame I end up concatenating back together can be a more manageable size as
well.
Right, right, right.
Interesting.
Have you looked at Dask for any of this?
Yes.
We've had, I did some work on the server for a different project looking at it.
And I think Dask might be, once I've kind of built up a more curated version of this four
second interval data, I think Dask will probably be what I'll use on the server for working with
the whole data set in the future.
Yeah.
It sounds like it might really be, I mean, I haven't actually tried to apply to that much
data that you got there, but it's sort of its functionality.
It sounds like it really might be the thing to do because it'll take, it basically, your
description of breaking into the many data frames, having them run and then bringing it back
together.
That sounds to me like what Dask is built for, right?
Yeah.
And the brilliance as well, I think of the Dask project is how they were able to kind of
emulate the Python, sorry, the Pandas way of doing things as well, which is great because,
you know, it's nice not to have to relearn too many things and be efficient from the beginning.
Also, I should probably just flag as well that these data sets that I'm working with, the four
second interval, and then the actual database on the five minute basis, one of the things
that got me so into the energy sector, and that's, I think, unique in a way about the Australian
energy market, although I stand to be corrected, is that this is all public data.
If you want, you can go and download all of this data from the market operators website, which
is, you know, an amazing amount of kind of openness to be able to go in and look at what
these power stations are doing and what prices that we're getting on such a granular basis
as well.
That's been really what fascinated me.
Yeah, that's super cool.
You know, if people are out there trying to do research, working on a thesis or something
like that, they could just grab this data.
And it's like you said, it's real.
And a lot of people have both physical machine reasons and financial reasons.
Keep it accurate, right?
Yes, absolutely.
And really, I guess I come back to that original point where the limitation for this data isn't
how clean it is or the amount of data.
It's just understanding the processes well enough to know what you can do with it.
Right.
A little bit like that example you had about the solar farms versus coal generation, right?
Exactly.
Yeah.
Know that I mean different things.
Yeah.
Very cool.
All right.
Other advice or interesting things going on before we're getting short on time, but
anything else you want to throw out there?
Work with all this data?
One of the areas I'm quite interested in at the moment is Python's number library.
N-U-M-V-A.
Yeah.
I think it uses the able to kind of compile Python code under the hood to get really high
performance.
And it can be perfect for my use.
And the reason I'm rather interested in it is that if I can vectorize a calculation,
you know how I'm applying it to the whole column.
But for instance, if I'm trying to do some calculation where I want to loop through it
because my current calculation depends on the state of a previous calculation, then that can
be a limitation of this kind of vectorization approach.
Whereas...
Right, right.
You can't just apply it to the set and go, hey, every row here, just look back at yourself
a little bit in that place and then do the thing.
It's got some dependencies on what happened before, right?
And that, I have no idea how you would fix that.
Maybe it's possible, but it's very tricky, right?
And so number does that, okay?
So number is effectively a way to write Python loops that run as fast as C, but within for
numeric calculations.
So this is why I'm very interested in this for some of the areas that are a little bit
trickier for me to vectorize using data frames.
I'm very interested in the number library as a solution for some of those challenges.
Yeah.
So number makes Python code fast.
It says it's an open source JIT compiler that translates a subset of Python and NumPy
code into fast machine code.
So I haven't used it, but it sounds like it really knows about NumPy in addition, right?
Because you could use Cython, but Cython doesn't necessarily know anything about NumPy, for
example, right?
Yes.
Yeah, you're exactly right.
Like in a way, number has kind of replaced what I would have used Cython for in the past,
but for my application.
What I could do, say for instance, if I have a NumPy array and I want to loop through it
and there's some calculation I want to do and then my next calculation depends on that previous
calculation going forward.
This lets me do that.
Yeah, in a way, and I'm writing Python code.
I don't have to write C or C++.
You can see it.
You're literally adding decorators, just like how Flask has that lovely kind of decorator
syntax.
It's almost like that to an extent.
Yeah, exactly.
You say at number.jit and then do you want this to run in parallel or not?
Yeah, sure.
Why not?
Exactly.
And also it even lets you run your calculations in parallel as well.
You can see that little parallelism true.
And it's actually true parallelism without the GIL as well, which is amazing.
Yeah, this is super interesting.
I knew that it was a compiler along the lines of Cython, but I didn't realize that it had
this special integration with NumPy.
That's very neat.
Yeah, I really recommend, like, I think for most people, Pandas is probably what you need.
But if you're running into these kinds of, yeah, these types of problems, then I think
NumPy would be a solution before you would have to look to, necessarily have to look to
a different programming language.
Yeah, absolutely.
And like, this is how I started off our conversation together.
The performance side of Python is super interesting because it's like, oh, well, it's not really
fast enough.
Until you apply this decorator, then all of a sudden it's just as fast and it's amazing.
There's just all these little edge cases that are super neat.
Yeah.
Very, very cool.
I look back to some of the code I wrote during my PhD and I cringe at some of the bad performance
practices I probably had for working with them.
And I think this comes back to your point about having people in the team who are a bit more
experienced in this area.
Because if you can have people who understand this and point team members to these tools
for optimizing their code, then I think that can deal with a lot of the issues for people
who may not be as experienced with writing Python themselves.
Yeah, that's super advice.
And then if you've got data that maybe is like the other side of your story, it's like
too big to fit in RAM or you want more sort of automatic, rather, parallelism across, say,
a large data frame, then Dask is definitely a good thing to look at.
Absolutely.
All right, Jack.
This has been super interesting.
I think we're getting a little long on time.
I don't want to take up your entire day.
So I will have to call it a show on that bit.
No problem.
But before we get out of here, of course, you have to answer the two final questions,
as always.
So if you're going to write some code, work on some of these projects, what editor do you
use?
VS Code.
Especially since they added Jupyter Notebooks into the editor as well.
Yeah, it's really interesting to see both what VS Code and PyCharm are doing to try to bring
either just bring notebooks into the space or try to come up with a more native alternative,
right?
Like, well, here's a cell and it's separated by like this special comment, but you can still
run it in notebook style.
But it's like feels like a text file you're working with.
And yeah, it's an exciting time for that stuff.
Absolutely.
Yeah.
And then you've already given a shout out to a couple of different projects.
But any notable PyPI packages you want to tell people about?
Yeah, I mean, Numbers is the one I'm fascinated with in terms of learning more about at the
moment.
But I think for in terms of getting things done, just Pandas and Geopandas and the kind
of scientific stack, it's just, it's amazing what people, the code that people have done
that makes me so effective just by understanding, just being able to use their libraries.
It's phenomenal.
Yeah.
Are you a Jupyter or a JupyterLab person or something else?
I use mainly Jupyter, mainly because at least at the time when I tried out JupyterLab, I couldn't
collapse some of the outputs from the cells as well.
They may have fixed that, but I'll take a peek at JupyterLab every couple of months or so
and see what's new.
All right.
Fantastic.
All right.
Well, final call to action.
People are interested in, maybe they work in energy somewhere else in the world or they're
trying to do research with it.
What do you tell them?
If you're interested in getting in touch with me, we run a few meetups for energy modelers
in Sydney.
So if you're ever interested in getting in touch to chat about some of the data in Australia
or some of the work we're doing, feel free.
Is that Zoomable these days or is it in person?
The plan is for the meetup to be up as a Zoom one.
So that's the plan.
Yeah.
All right.
Fantastic.
Well, Jack, it's been great to have you here.
Thanks so much for sharing what you're up to.
Thanks so much, Michael.
It was great meeting you.
Thanks for having me.
Yeah.
Keep the lights on down under.
Thank you.
I will.
Bye.
Have a good night.
This has been another episode of Talk Python to Me.
Our guest on this episode was Jack Simpson, and it's been brought to you by Square, Linode,
and Assembly AI.
With Square, your web app can easily take payments, seamlessly accept debit and credit cards, as
well as digital wallet payments.
Get started building your own online payment form in three steps with Square's Python SDK
at talkpython.fm/square.
Simplify your infrastructure and cut your cloud bills in half with Linode's Linux virtual machines.
Develop, deploy, and scale your modern applications faster and easier.
Visit talkpython.fm/linode and click the Create Free Account button to get started.
Transcripts for this and all of our episodes are brought to you by Assembly AI.
Do you need a great automatic speech-to-text API?
Get human-level accuracy in just a few lines of code.
Visit talkpython.fm/assembly AI.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at Talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show, open your favorite podcast app, and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play,
and the direct RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air,
be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I'll see you next time.
Thank you.