We all know Python is becoming increasingly important in both science and machine learning.
This week, we journey to the very forefront of physics.
You'll meet Michaela Paganini, Michael Kagan, and Matthew Feikert.
They all work at the Large Hadron Collider and are using Python and machine learning
to help make the next major discovery in physics.
Join us this week on Talk Python to Me, episode 144, recorded December 14, 2017.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the
ecosystem, and the personalities.
This is your host, Michael Kennedy.
Follow me on Twitter, where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm.
And follow the show on Twitter via at Talk Python.
This episode is brought to you by Linode and Talk Python Training.
Be sure to check out what the offers are for both of these segments.
It really helps support the show.
Hey, everyone.
Before we get to the interview, I want to share a quick update about our Python courses with
you.
Do you work on a software team that needs training and could really use a chance to level up their
Python?
Maybe your entire company is looking to become more proficient.
We have special offers that make our courses here at Talk Python the best option for everyone
you work with.
Our courses don't require an ongoing subscription like so many corporate training options do.
And they're roughly priced about the same as a book.
We're here to help you succeed.
Send us a note at sales at talkpython.fm to start a conversation.
Now, let's get to the interview.
Michael, Michaela, Matthew, welcome to Talk Python.
Hi, everyone.
Thanks for having me.
Yeah, it's an honor to be on the show.
Thanks so much for inviting us.
Yeah, you bet.
I'm really, it's, you know, the honor is mine, to be honest.
You guys are doing amazing stuff.
You are pushing the boundaries of science.
And I'm just really excited that you're here to share what you're working on and how that
intersects with the Python space with my listeners.
So we're going to talk about large Hadron Collider, particle physics, and machine learning and how
those three amazing things go together.
But before we get into them, let's just start real quickly with how you guys got into programming
in Python.
Michaela, you want to go first?
Yeah.
So actually, when I started programming, it certainly wasn't Python for me at first.
It was IDL and MATLAB and some undergraduate physics and astronomy labs.
But then when I got to CERN and I started my career as a graduate student in Atlas, that's
when I first encountered Python and C++.
And so just like everybody else, I guess I learned on the job.
And I'm really thankful that I had great mentors that really taught me what good code and bad
code looked like, such that I could start writing good code instead.
Yeah, I feel like so much of what people do in programming really is learning on the job.
Even for people who have degrees in computer science, you study one thing, but then you go
and you actually build something different.
So I think, you know, it's probably not that different, actually, than what most people went
through.
Yeah, absolutely.
I think that's something that a lot of people could relate to.
Yeah, nice.
And IDL, that's the astrophysics type language.
Is that right?
That's correct.
Yeah, I started as an astronomer in undergraduate.
And so that was my first encounter, I would say, with real coding.
I see.
So you started out with really tremendously large things.
You're like, no, no, let's look at the really small stuff instead.
Yes, it was quite a transition.
All right.
Michael Kagan, how about you?
Yeah.
So I guess my story is pretty similar to Michaela's.
But so I took a few classes in undergraduate.
And actually, the first language I really used on the job was Fortran, because a lot of the
old physics simulation and code for doing calculations is built in Fortran.
So that was kind of my first real projects.
And then once I got to graduate school, everything on the modern high energy physics experiments
was C++.
And that's where I kind of learned on the job.
And then Python is kind of used there a bit.
And so I was beginning to learn Python through some of the glue scripting that's done there.
And then once I kind of got into more machine learning stuff and realized how many great
tools there were, then I dove a lot more into using Python.
Yeah, that's cool.
At CERN, a lot of the actual data processing is done in C++.
But then the consumption of that is done in Python, if I understand it right.
Yeah, that's right.
So basically, the enormous kind of database of code that kind of crunches on the data is
all in C++.
And then there's a wide range of different Python scripts that help guide the set of code
that helps crunch on data from one type of experimental apparatus and then from another
piece of experimental apparatus and then put that information together.
Nice.
Matthew, how about you?
Yeah.
So similarly, the first time I really did any programming was in a CS 101 course at university
where I learned a little bit of C and some MATLAB.
And nothing was really kind of clicking then.
But then I joined an undergraduate.
Well, as an undergraduate, I joined the research group at the University of Illinois.
And my advisor at the time was like, hey, do you want to do this stuff?
And it seemed really exciting.
He was telling me about his work at the Large Hadron Collider.
So I said, sure.
And he was like, great.
And he sent me down in front of a Linux terminal.
And that's when I first got introduced to Bash and C++ and things like that.
Then actually, I was starting to get to see how I could use programming to actually solve
problems in analysis.
That's kind of when I had this aha moment and it started to click.
But I really didn't start using Python much until I got to grad school.
And yeah, as Michael said, once I started to get a bit more introduced to how the physics
community is using machine learning, then Python.
And it's a really great ecosystem for machine learning tools.
That kind of became a really obvious choice for me to start to hone my Python skills.
But for me, pretty much all of my programming was just learned on the job.
That's really cool.
I think one of the things that really draws people in once they get started with Python
is all the extra libraries that you could use.
Oh, you can just pip install TensorFlow.
You can just do this, just do that.
And it's like, wait a minute.
It's just, it's all right here.
That's been fantastic, especially with these like very, very quickly growing set of libraries
and tools.
And even in the machine learning domain, there's a new package pretty frequently that you want
to try out.
And yeah, pip and Kanda and these kinds of these package managers that make this so quick
is really much faster development time than a lot of the stuff we're doing in C++, where
a new package comes out.
Given the code base is enormous, can take a relatively long amount of time just to compile
any changes you want to make.
Yeah, that's for sure.
So it's really, I think it's a really, I think of it as one of Python's superpowers.
It's really cool.
Michaela, one of the things I wanted to ask you about is it feels like this open source
aspect of Python fits really well with the sort of open science.
You know, this research is for public consumption.
A lot of stuff you can do here seems like it's much better done with an open source set of
software than say like MATLAB and proprietary paid add-ons or whatever.
Is that for Michael or Michaela?
Michaela.
Yeah, Michaela.
Oh, hi.
Sorry.
Yes.
No, I totally agree with you in the sense that it's a lot easier for us being a large
collaboration to be able to share tools and be able to collaborate across so many different
domains, even within physics, using these libraries that one can very simply, as you both
said, install in their environment, as opposed to perhaps using something closed source, which
then needs to be distributed correctly.
And so I totally agree that it fits very well with the design that at least I have in
mind for what our workflows should look like.
And I think a lot of the other people on this podcast would agree with me.
Yeah, very cool.
I was going to say, I would just add, I mean, and this is something like even the experiments
take quite seriously, even, you know, kind of within any language, even if there's proprietary
software that could be quite useful, we're very, very hesitant about, you know, kind of engaging
with it because of our inability to look inside the box and our inability to really know that
it's doing what we want to do.
So, which is why even within the experiment, we end up writing a lot of our own code, even
if there's a proprietary solution, just because it's not really adequate for us to be able
to do the research we need to do.
That makes a lot of sense.
Is that new?
You know, is that something like 15 years ago would have people said the same thing?
Yeah, absolutely.
Because a lot of these decisions that I'm even thinking about are about packages from
15 years ago at the beginning of the experiment where there's even a nice neural network package
that from 10 years ago that was decided not to really be used because it was proprietary
at the time.
Okay.
Yeah, very interesting.
I would say that those decisions were even more important back then than they are right now.
I think these days, like we see more adoption of standard tools, still open source tools, but
a lot of the standard tools from industry, whereas I would say that back in the days, maybe
like decades ago at CERN, we would tend to really customize every single piece of code
and write it ourselves.
I guess you're probably right.
That's an interesting point because today we're swimming in open source software and these ideas
and it kind of seems more accepted.
So yeah, it's even more important to have that sort of early when it wasn't so obvious.
Nice.
So let's talk about what you guys each do day today.
You all are doing such amazing stuff.
You're all involved in Atlas to some degree.
And we'll talk about the Large Hadron Collider a little bit, but maybe you could just touch
on what that is.
So maybe Matthew, let's start with you.
Yeah, sure.
So I'm a graduate student at Southern Methodist University in the United States, but I'm stationed
over at CERN.
And so like you said, I work on Atlas.
So as a graduate student, in some sense, you could say like one of my main responsibilities
is to make plots.
But I mean, what I mean by that is that I'm both like learning how to do analyses and then
I'm actually one of the people who's kind of going in and writing the code.
And so I do work on a specific analysis that's looking at trying to measure specific properties
of the Higgs boson given a specific decay channel that it might have or that it does have.
And then the way that I'm kind of doing this right now is actually by using Jupyter notebooks.
So I can actually go in and use some of the great Python tools like Keres to be able to
interact with the data and actually write some neural networks and actually try and do some
exploratory data analysis.
So in addition to that, I also do some operations work on Atlas.
So I work on something that's called our trigger system.
We can talk a bit more about that if there's time.
Yeah, the trigger system is really amazing and critical as well, right?
Yeah.
We'll definitely talk about that.
Yeah.
So there I'm basically just writing a combination of C++ and Python to try and do optimization studies
and provide support for the trigger system.
Okay, cool.
How about you, Michaela?
What do you do day to day?
What are you doing on all these experiments?
Yeah, okay.
So, well, these days I always joke about this, but what I spend most of my time on is preparing
talks and posters for conferences and workshops and interviews and all of that.
But I guess my average day as a PhD student in Atlas consists primarily of three things,
I would say.
First of all, training neural networks.
And then while those are training, I contribute to the experiments or my analysis code base.
I read lots of papers on the archive, but I would say certainly roughly 90% of my time
is spent on coding and documenting the code.
And I'm also involved in various reconstruction and analysis groups.
And my goal personally is to bring better algorithmic thinking to the table to help identify bottom
quarks or pairs of Higgs bosons in my case or any other particle that we might be interested
in at the Large Hadron Collider.
I can go into more detail if you want about some of the work that I'm doing with neural networks.
Specifically, I'm working on speeding up a part of our simulation that is very competentially
intensive.
And my idea is to use generative adversarial networks to have a higher accuracy, but at the
same time, a faster simulator that is powered by deep learning.
That is really awesome.
And I do want to ask you more about that.
But one question that came to mind while you're describing that is, you say you spend 90% of
your time writing code.
When you got into physics, did you think, and you thought about, what am I going to do as
a physicist once I'm done with the books?
Like, is that what you actually saw yourself doing?
Absolutely not.
I was not ready for that at first.
It came as a surprise.
It was a pleasant surprise, I have to say.
It turned out that I love coding a lot more, perhaps, than I like more traditional ideas
of physics.
But it was a transition, let's say, because not much of our coursework prepares us for what
the reality of the work of a graduate student in an experiment like Atlas is really like.
And as I said, the majority of it is coding.
Yeah, for sure.
I didn't take any graduate classes in physics, but I took a number of high-level ones.
And I don't remember doing hardly any coding for them.
It was all pen, paper, you know, prove this, prove that.
A lot of equations, not much actual software.
I would certainly advocate that some of the curriculums should be probably updated to reflect
what the real life of a graduate student in experiments in high-energy physics looks like.
Yeah, that makes a lot of sense.
All right, Michael, how about you?
I'm a research scientist at Slack without the K. So that's the Stanford Linear Accelerator Center.
Not the chat thing, but the really fast thing at Stanford, right?
Exactly. Yeah.
So it's a linear accelerator center.
It's a kind of one of the DOE, Department of Energy National Laboratories, that's run by
Stanford.
And actually, most of what they...
This used to be a high-energy physics lab, primarily.
And that's kind of been turned into basically a big X-ray laser, a free electronic laser.
But anyways, I don't really work on that stuff.
I'm part of the team that works on Atlas.
And so kind of as a research scientist, you know, we work in Atlas, which is this 3,000-person
collaboration.
And, you know, each of us kind of work for our own institutions.
And so one aspect of that is with 3,000 people trying to, you know, run, improve and run a
large piece of equipment and then do all this data analysis is we have to organize.
So I found myself now in kind of a phase of my career where, you know, I'm more and more
a part of that organization.
So I'm helping to run one of the groups which looks at a certain kind of particle, which we
might find in our detectors, which is called a bottom quark.
And we basically develop algorithms to find those particles in the detector.
And so once we find them, we can then give those algorithms out, supply them to any other
analyzer on the experiment who wants to, you know, look at some data and find out, you
know, in a given collision, how many bottom quarks were there in that collision.
So that's, I spend a lot of my time kind of running that group, which is kind of maybe 50
or 60 people kind of organized together, working together to get that moving forward and working.
And then, you know, with the free time that I have left, you know, working with kind of
postdocs and grad students on data analysis.
I mean, I've worked both with Michaela and Matthew on various data analysis projects.
And then also on exploring how we might take some new ideas in machine learning or even
develop some when needed to solve some of our specific tasks.
Yeah, that sounds really, really interesting.
And you're using some machine learning and those types of algorithms to create these techniques
for discovering these bottom quarks?
Yeah, absolutely.
So kind of machine learning, you know, machine learning is, you know, has been around for a
while.
And especially, you know, so we've had algorithms that that kind of work.
And I guess this is probably a pretty, you know, still a very common and powerful paradigm,
which is we look at our data.
And, you know, based on our domain knowledge, we can we can develop all sorts of features
or all sorts of use all sorts of algorithms to say, OK, well, this looks like the properties
of a bottom quark.
And we can compute all these features and then run them, train machine learning algorithms to,
you know, for instance, classify whether this set of data really was a bottom quark or not.
And so that's been around for a while.
And one of the things we're also working on then is saying, OK, well, let's if we can take
a step back and look at this data and see if, you know, if we can think about it in different
ways, sometimes that maps on to problems like problems in vision or even natural language
processing, where we can then start to use some of these super modern techniques, you know,
coming from things like deep learning so that, you know, we can even improve our classification
or in some cases we're doing regression or even generative type problems.
So there's a lot of work.
Yeah, you basically are working.
Yeah, you're working with a huge camera, basically.
Exactly.
With Atlas.
Yeah, it's it's it is a keep thinking of it in meters, but it's like a 75, 75 foot tall and
100 and 120 foot long detector that sits 100 meters underground.
And it's built of like many different kinds of cameras.
And so we have to kind of take that and these different kinds of cameras detect different
kinds of particles.
And we take all that information together to build a picture of what happened every time
we collide.
Yeah.
So maybe that's a great place to segue into just like a really quick summary of this particle
physics stuff.
So Matthew, maybe we'll start let you start this off.
You know, we were told I was told in, I don't know, fifth grade or something that atoms, that's
the smallest stuff that everything is made of.
Right.
But, you know, not so much.
Right.
Tell us about it.
Yeah.
So we I think everyone kind of has this idea going through like schooling that, you know,
you have the periodic table where you have like your atoms that are made of protons and
neutrons and electrons.
And that's it.
Right.
Well, yeah.
So it turns out that's not really the whole story.
And that while electrons, as it turns out, do seem to be fundamental particles, protons
and neutrons, these are these are actually composite particles that are made of even more
fundamental particles that we call quarks.
And so and there's also gluons in there, which are other subatomic particles.
So it turns out that there's kind of this whole particle zoo, if you will.
But kind of the amazing thing is that when we go and kind of explore the world and and
actually look, it turns out that there's 12 matter type particles in the world that there's
six of these quark particles that make up the protons and neutrons.
We call these things protons and neutrons more generally hadrons.
And then there's things like the electron that are called we call leptons.
And there's both electrically charged things, electrically charged leptons like the like the
electron.
And then there's also kind of their ghostly neutral cousins that hardly even interact with
matter at all.
So I think maybe Michaela and Michael can talk about the fundamental forces and other
things.
Yeah, sure.
Michaela, take it away.
Yeah, of course.
But on top of all of these matter particles that Matthew just described, we also have what
we call the force carriers or gauge bosons in a way.
And these are particles that can be thought of as being exchanged among other fundamental
particles to mediate some of the forces and attractions that connect these that are more
fundamental particles.
And so we can think of the photon, for example, as being the force carrier for electromagnetism.
And then I think to complete the puzzle, we have the most recently discovered particle
for the standard model of particle physics, which is the Higgs boson.
And by the way, the standard model of particle physics is just this great theory that we've
come up with over the years that kind of puts all of our fundamental particles in this periodic
table that Matthew and I have just tried to describe to you.
So there is a little bit of an analogy maybe to chemistry, but at even more fundamental
level than the atom itself.
Yeah.
And it's really amazing that this was created somewhat theoretically.
And then the machine to go find things like the Higgs boson was built.
And then it really was there.
That's true.
It's very fascinating.
I think at a certain point in history, experiment was ahead of theory.
And then theory surpassed experiment once again.
And so it's a very fascinating field to be in.
And I think at different historical moments, things were very different from what they look
like today.
Yeah.
How much do you think that comes from people just getting better at theory versus computational
techniques assisting theory?
I think it's probably part of what you said, but as well as the energy regimes that we are
trying to probe now.
So I think most particle physicists would agree that in terms of the energies that we were able
to probe right now, we think we have a good understanding of all of the particles that could exist there.
But what we're really searching for right now is something that is an even higher energy.
And that's the main issue.
At that point, the complication is not so much whether the theory is there or not, but it's
being able to produce machines.
So it's the hardware technology even to go search for these particles.
And I think certainly software will help us get the most out of the hardware that we currently
have and the next hardware that we will build in the future generations.
But it's both hardware and software, in my opinion.
Yeah.
Okay.
Really cool.
All right.
Maybe the last one on this physics intro stuff.
Michael, what was the big deal about finding the Higgs boson and what did people learn from
it?
Yeah, absolutely.
The Higgs boson is probably the hardest of the particles to describe in some ways.
So it sounds enormous, but the job of the Higgs boson is many things.
And it's probably the easiest way to explain is it gives mass to all the other particles.
And so the way you can think about that is as particles move around, they bump into Higgs
bosons.
And the Higgs bosons kind of slow them down and effectively give them mass.
Kind of like if you're trying to walk through water, it takes a lot more force or a lot more
effort to kind of move your body.
And that's the same way that's kind of happening with the particles.
This is an imperfect analogy.
So if any theoretical physicists are listening, I apologize.
That's roughly the idea.
But it turns out the Higgs boson really played a fundamental role in making this theory that
Michaela and Matthew explain, which is incredibly predictive, maybe the most predictive theory
ever, kind of makes sense.
Without the Higgs boson, the theory effectively predicts things that have probabilities larger
than one, which we knew didn't make any sense.
And that's kind of a little bit how you were saying that that knowledge of the theory breaking
down really helped drive what became a 40 or 50 year long experimental search.
Yeah, it's amazing how people get excited when they're wrong.
It's like, oh, the theory might be wrong.
It would be so exciting.
Yeah, absolutely.
I think, you know, that's every time something doesn't make sense, the theoretical physicists
get incredibly excited that they're going to have to come up with a completely new theory.
Yeah.
And I don't think we're done with that at all.
That's cool.
So if people are out there listening and they really want to get a sense for what's going
on LHC and particle physics, I definitely want to recommend the documentary Particle Fever,
which I think is available on Netflix.
But I'll link to at least the trailer.
And there's a book called Present at Creation, Discovering the Higgs boson, which is great.
And then there's another one, We Have No Idea, A Guide to the Unknown Universe.
Who recommended that one?
Yeah, so I recommended that.
That's actually co-written by our Atlas colleague, Daniel Whiteson, and then also the famous PhD
comics cartoonist, Jorge Sham.
I really like that book because I think it's both like a celebration of how much we still
don't know about the universe and how now is really a great time to get into science because
we're truly in an age of discovery.
But it also talks about just how much we do know as well.
So I think it's both, it's just a great celebration of science.
And it also, given that it's co-written by a physicist, it really does convey ideas really
well.
Yeah, excellent.
Yeah, so people can check all three of those things out.
They're really good background information.
This portion of Talk Python to me is brought to you by Linode.
Are you looking for bulletproof hosting that's fast, simple, and incredibly affordable?
Look past that bookstore and check out Linode at talkpython.fm/Linode, L-I-N-O-D-E.
Plans start at just $5 a month for a dedicated server with a gig of RAM.
They have 10 data centers across the globe.
So no matter where you are, there's a data center near you.
Whether you want to run your Python web app, host a private Git server, or a file server,
you'll get native SSDs on all machines, a newly upgraded 200 gigabit network, and 24-7 friendly
support, even on holidays, and a seven-day money-back guarantee.
Want a dedicated server for free for the next four months?
Use the coupon code python17 at talkpython.fm/Linode.
So let's catch up on the LHC just a little bit.
Michaela, maybe could you just give people a sense of the scale?
Like Michael said, there are 3,000 people working on Atlas, and Atlas is just part of one of the
experiments.
Give us a sense of what this place is like.
Yeah, absolutely.
I really love CERN.
It's a group of fantastic people, some of the brightest minds in the world.
And it brings together researchers and engineers and computer scientists from all around the
world, probably hundreds of different nationalities.
And one of my favorite thing about it is that I can absolutely say that every single time I
am there, I'm never hanging out with more than one or two people from the same nationality.
So to me, that's the best thing about it.
Again, there's probably tens of thousands of people across the various different experiments.
We've been talking a lot about the LHC, a large hadron collider that hosts four experiments.
So not only Atlas, but also LHC-B, ELISE, and CMS.
But then again, the large hadron collider is only one small part of the entirety of CERN.
There are a lot of other experiments going on.
For example, some antimatter experiments at the antiproton decelerator, as well as even
astronomy experiments, as far as I know.
So it's a large laboratory that spans across two different countries at the intersection,
at the border, basically between France and Switzerland.
So fantastic place to work at.
Yeah, it sounds really, really cool.
And people can go tour it, right?
They can set up a tour?
Absolutely, yes.
Anybody can just show up and do the quick tour of, for example, Point One, which is where
Atlas is located.
So you can visit our control room and learn more about our experiment.
And if you're lucky enough to be able to visit during a shutdown period, so oftentimes in the
winter, that's when we have quick shutdowns.
You might even be able to go underground and visit the actual experiment, which is absolutely
breathtaking.
Yeah, I'm sure that it is.
Just the scale from the pictures, it looks like, is just incredible.
Yeah.
I just want to jump in and just make an additional comment about what Michaela said, in the sense
that CERN really is an open laboratory.
And it's a big part of CERN's mission to make sure that the scientific discoveries that are
made there are made for all of humanity.
So CERN really welcomes the public getting involved and being curious and coming and asking
questions.
So yeah, if you're ever traveling through nearby Geneva, Switzerland, sign up for a tour.
Come visit.
Yeah, that sounds great.
Michael, the data that flows through these experiments out to the collectors and then into the trigger
that Matthew mentioned, and then on to the larger computing structures that are there is
pretty insane.
Do you want to give us like a kind of an overview of the scale of data?
Yeah, absolutely.
The way that kind of the LHC works is, well, I'll give you a little bit of the physics
background about why we have to design the systems this way.
So a lot of the things we're searching for are very rare.
And the physics that we deal with is probabilistic.
So we might be looking for something that's interesting that only happens in one out of a trillion
collisions or maybe even less frequently.
And so we have to collide protons as many times as possible.
So we collide protons 40 million times a second.
And those collisions fly out into the massive detectors that Michaela was describing or that
we've been discussing.
So the thing is, we can't record all that data.
So we can only record a fraction of that data because it would simply be too much.
So we have a set of systems called the trigger, which allows us to go from 40 million collisions,
of which many are not super interesting, down to about kilohertz, so about 1,000 a second.
And each of those collisions, the data that comes out of the detector, I think it's about a megabyte
per collision.
So that means we're taking about a gigabyte of data per second.
That's the data that made it through the trigger that was not discarded by the hardware, right?
Exactly.
That's just the data that made it through the detector.
And the actual processing there is a combination of custom-built hardware and FPGAs, which are
fast enough to deal with looking at the data really quickly at 40 million times a second and
helping us pipeline it down through various both hardware and software systems down to this
kind of kilohertz rate.
So we don't run the detector all the time.
We run it a lot.
There's shutdown periods.
There's times when you have to kind of refill the beam.
And I think we accumulate something like, what is it, like three or four petabytes of data
a year.
Yeah, that's just crazy.
And then it's not just there in Geneva.
It's also broadcast out, right?
Right.
So I think I was, yeah, I think there's something like 170 institutions around the world that make
up the kind of worldwide computing grid or the LHC worldwide computing grid.
And there's something like 300,000 or maybe more at this point computing cores, which then
can, well, we distribute the data around the world.
And then we often need to process and reprocess and analyze that data.
And that's done on this enormous computing grid.
And so that's kind of once it's stored and distributed, that's how we go and analyze it is by basically
sending jobs to this grid, which you can kind of think of as a precursor to what the cloud
is now.
Right.
And so there's so much data probably that I suspect putting it on your laptop doesn't
make a lot of sense, right?
You need to send your computation to the data rather than bringing the data to you, right?
Exactly.
I know.
I just said exactly.
That's exactly how it works.
You certainly wouldn't want to be downloading petabytes of data onto your computer.
So thankfully, we have this grid of tier zero, tier one, tier two, tier three locations spread
all around the world where you can send your scripts and they'll be run and then you'll
get the results back.
Yeah.
So give me a sense of what that's like.
You have a question you want to ask about the data.
You could write some C++.
You could write some Python.
You could write maybe even Fortran.
What is the mechanism from?
I have this thing, this Python, let's say Python here.
And I would like to make it run there and analyze the data.
What's the steps look like there?
From the user's perspective, which is the one that I get, it's very simple because of the
work of hundreds of people who've made it simple for us.
So we simply have an interface with our computing grid where we can specify specific locations
if we want to do so.
We can specify the length of the job, the number of cores that we're requiring, the number of
nodes, et cetera.
And then we can simply submit our script as long as it's in a format that is compliant
with what our systems are able to handle.
And then you specify what data to operate on, whether it's true data that has been collected
from the LHC or if it's simulated data, you can also operate on that.
And then you're able to monitor all of your jobs and eventually get the results back and
download the histograms or whatever format your results will come to you as.
That sounds really cool.
And Michael, you said 300,000 computing cores?
It's something I think it might even be larger at this point.
As of 2010, it was more than 200,000.
It's ever growing with the amount of data we have and the amount of computing we need.
Just as Michaela was saying, we kind of send our jobs out and it's kind of built on top of
a kind of virtual machine file system where we actually, all these sites are kind of working
in coherently with having the same distribution of the Atlas software located all these sites.
So you can send your job with a known even version of the software and it's already available
locally to you.
Wow.
Okay.
That sounds really, really fun.
I guess one thing that I was wondering looking at the LHC is we have Atlas, we have LHCB and
Alice and CMS.
What is the purpose of Atlas?
Like what is Atlas trying to do relative to the larger goal of LHC?
Maybe Matthew take that.
So Atlas and CMS, these are two of our general purpose detectors.
And so the idea there is these detectors were explicitly designed and then built to be sensitive
to a wide range of interesting physics.
Whereas for example, backtracking a little bit like Atlas and CMS, they kind of have, if
you will, like they're sometimes referred to as like cylindrical onions in the sense that
the architecture is you kind of have your beam pipe and then you have successive layers of
very detailed and detectors going out around them.
Is that so you can basically take a 3D picture?
Yeah, exactly.
Because when we have these collisions, when you have the really hard collisions, then in
some sense you have like just the result of the collision is that you have sprays of
particles kind of coming out in all directions.
And so you want to have as much coverage as possible.
And the idea is if you have both like calorimetry systems and tracking systems, then you're
able to get a much more detailed picture of what actually happened.
Because we're trying to reconstruct essentially point-like interactions that are happening at
the subatomic level.
But we're doing that by seeing what kind of is just coming splattering through our detector.
So it's kind of like trying to reconstruct what might have happened if you took like two,
if there is some sort of like car crash.
And the only way you could investigate what happened is if you were able to look at what
the walls of a tunnel or something nearby.
But so Atlas and CMS, they're general purpose detectors.
And then Elise and LHCB, their geometries are a little bit different.
And so they're more specialized detectors that are looking at specific types of physics.
All right.
Okay.
Yeah.
Very cool.
So one of the things I want to dig into with each one of you is what you're doing day to
day and sort of how Python and machine learning fit into that.
So Michaela, let's start with you.
You're doing some, you already mentioned your generative adversarial networks and some really
amazing stuff.
You said that you were able to speed up some of these simulations like 100,000 times.
That's correct.
Cool.
Techniques.
That's incredible.
Can you talk about how you're doing that?
Yeah.
So these are obviously preliminary results and there's a lot more R&D that is now just being
started within our collaboration.
But the point is that we built this great prototype we call the Calogan.
Calo stands for calorimeter.
That is one of the detector layers inside of this big onion-like structure that we just described
our detector to look like.
And the calorimeter measures the energy deposited by certain particles as they travel through,
as Matthew was just describing.
The issue is that because some of these physical processes that the particles undergo are so
complicated, simulating these traversals of the particles through the calorimeter is really,
really computationally intensive.
And that's actually taking more than half of the computing grid power that we were just
describing.
So it's billions of CPU hours per year.
So what I'm working on is this new technique to speed up that part of the simulation, which
currently occupies the majority of our computing resources worldwide.
And what I'm using is generative adversarial networks.
I think some of your audience maybe will recognize this word.
So we use GANs to provide a function approximator to our simulator while retaining, hopefully, the majority
of the accuracy that the slower physics-driven simulator possesses.
And again, as I said, multiple preliminary results have been put out so far.
And we are achieving speed up times of over 100,000 times.
But now the complicated part will really be to learn how to calibrate all of these, this machinery and
bring it, port it into the real simulation within the experiment.
But speaking of Python, I think the cool thing for everybody to know is that this is very
easily built using Keras and TensorFlow.
So very standard machine learning tools from Python, as well as other standard tools from the Python
ecosystem, such as NumPy, scikit-learn, H5Py, and Matplotlib all make it into my project.
Yeah, that's really cool.
I think people are probably familiar with most of those.
But H5Py, what is that?
Oh, it's just the interface for HDF5 in Python.
HDF5 being like a very standard data format that can be ported across various different languages,
C++, Python, and H5Py certainly saved my life in terms of being able to open these files.
Yes.
Of course.
That's really cool.
Michael, you want to talk a bit about how you're using machine learning for what you're up
to?
Yeah, absolutely.
So kind of in the past, I've been working a lot on, again, these kind of ideas of taking
detector measurements and turning them into classifying whether this data was from a given
kind of particle or not.
And so I've been working on connecting kind of the data that we have with ideas of machine
learning.
So we have, so when we were talking about quarks, and when you produce quarks, it turns out
they produce kind of collimated streams of particles that smash into these calorimeters and
kind of leave a bunch of energy with kind of distributed in space.
It turns out we can connect those distribution of energy in space with basically imaging type
approaches.
And then we can, we've been running a lot of computer vision type techniques to study those
jets, those quarks.
And so that's, you know, really jumped into connecting with things like convolutional neural
networks and modern computer vision.
And so, yeah, I've been working on that just like Michaela with tools like Keras and TensorFlow
and, you know, built on top of core packages like SciPy and NumPy.
Well, one of the things I was going to ask you is it seems like the things you're actually
looking for are quite rare.
Like there was only a few collisions that produced the Higgs boson, for example, back in 2013,
14.
Right.
And my sense, I haven't done a lot of machine learning, but my sense is you have to give a
lot of examples to machine learning.
And then the machine learning can find more of those, even if they're subsequently rare.
But how do you like bootstrap this?
How do you get it started and where there's enough?
How do you teach them?
I guess is what I'm asking.
And so that they can then go find these things, especially when those occurrences are rare.
In some sense, it is kind of a bootstrap, but it's based on this idea that those super
rare particles like Higgs bosons, we don't observe them directly.
They decay into other things that we know about, like electrons.
I see.
So they're easy to find.
Decay result is easy to find.
And you can teach it to find those.
And when you find them in certain configurations, you're like, oh, this may have originated
from what we're looking for.
Exactly.
So I work a lot on making sure we can find not electrons, but other types of particles that
are produced copiously.
And then we can say, okay, in this case, okay, I know how to find electrons.
I can train an algorithm using our simulation, which is very precise.
I can go look for those in our real data to make sure that the simulation makes sense in
a different configuration.
And then I can go hunting with my well calibrated and well tuned algorithm for finding electrons.
I can go hunt for configurations with four or five electrons.
And that might be a really rare thing you want to look for.
Okay.
Yeah.
Very interesting.
I see how that works.
Because I was thinking about this.
You know, how do you get started trying to find very rare things?
But I can see that now.
Okay.
I'll throw this question out to everyone.
Are any of you using special hardware?
Or are you just leveraging the many, many cores out there on the computing environment?
Like are you using the tensor CPU type things or GPUs or those sorts of things?
I think the majority of us are using GPUs to train some of our machine learning models these
days on top of obviously the worldwide grid that we just described.
But that's more reserved for simulating samples or doing, you know, more standard types of analysis.
But I think most of us rely on GPUs these days.
I would add, I think one of the interesting potential future directions on top of all the
work with GPUs and CPUs is if we ever want some of these algorithms to work in our trigger
in that super fast system that needs to operate at 40 megahertz, this may be a place where,
you know, there's some people already in the community beginning to look at, you know,
putting neural networks onto FPGAs so that we can actually run them at super high speed.
So that may be a future direction in the field move.
Okay.
Yeah.
That would be really amazing.
Maybe Matthew, this is a good time to talk about that trigger thing.
So you have to take 40 million observations across this thing and get it down to 1000.
And you've got to do that extremely, like every second.
Yeah.
So how do you do that?
Yeah.
So Michael's already given a very nice intro summary there, but the trigger is an immensely
complex system.
And so, I mean, I definitely don't understand how all of it works.
I actually just work on a subsystem of the trigger subsystem.
So if you want to actually do an analysis, then you need to have, let's say like you're looking for a Higgs decay that goes to two B quarks.
Well, if you want to go searching for that, then you want to have some confidence that there ever could have been a recording of an event in the Atlas detector that had two B quarks in it.
So you want to make sure that some of these interesting collisions aren't getting thrown out.
And so we have, that's one of the reasons we have the trigger system is that we kind of have what we call a trigger menu, which is a list of basically logical sequences in that we are looking for in the different subsystems of the detector to say that, oh, okay.
So I think that's a lot of the way.
So I think that's a lot of the way that we're looking for a lot of the things that we're looking for.
We're looking for a lot of the things that we're looking for, but we're looking for a lot of the things that we're looking for, but we're looking for a lot of the things that we're looking for.
But we want to make sure that as we go up to higher energies and as we go up to even more collisions, so basically what we call luminosity is the number of collisions that we have per crossing increases.
We want to make sure that our trigger system can still deal with this because in a single crossing of the beam, we don't just get one kind of collision point.
We might get right now somewhere between like 30 and 50, but as we go to higher energies and to higher luminosities, we're looking at getting something like 200 collisions that are happening every beam crossing.
So if you're trying to say, oh, I have like an electron or I have a B-jet that looks interesting over here, I wonder if it might also have a partner that can tell me if I had a collision, you have a really difficult problem because you're trying to now pick out what might these other energy deposits or tracks be from 200 other collisions that are happening at the same time.
Yeah, it's one of these combinatorial types of things.
How many different relationships between, you know, 30 or 200, right?
That turns out to be astronomically bigger, right?
Yeah, it gets pretty crazy.
I'm sure it does.
So that's mostly like embedded C++.
Is that right?
But what other stuff are you doing there?
Is there any Python at that layer?
So actually like in the trigger system, no.
But so as a student, what I'm doing a lot more is performance studies of how our trigger is doing.
And so there I do use kind of both a hybrid of Python and C++.
We have colleagues at the University of Chicago that have written a really nice analysis framework that a lot of people in the collaboration use.
And that's it's all implemented in C++.
But then there is also a way for us to interact with it from Python.
So that's really nice because then that allows us to, for example, I was able to write some tools, some command line interface tools using arg parse and things like that.
So I can give, if I want to do a quick performance study, I can, without really having to ever write a line of C++, I can kind of spin up a small analysis to say like, oh, okay.
How is the trigger performing under these scenarios and things like that?
And using things like Jupyter Notebooks and stuff to sort of do that exploration?
For the trigger, I don't use Jupyter Notebooks so much.
That's more regular.
I use Jupyter more for the exploratory data analysis that I'm using for, if you will, like my actual like PhD research.
But for the operations work that I do on Atlas, that's more just kind of going into your favorite editor and then writing some command line tools in Python.
And then also making sure that it interfaces well with these C++ frameworks that we've developed.
Okay.
Very interesting.
So, Michaela, maybe we could talk a little bit about where you guys are going in the future.
LHC has been a really long running project.
It started producing major results in 2013, 14, but it was started way before then, right?
So where's the future going now with what you're up to?
Of course.
So there are still so many open questions, I think, about the standard model.
We know, as Michael said, that the standard model has been validated over and over again.
And it's probably one of the most predictive theories in the history of all theories.
But there are some missing pieces.
So, for example, we don't have a complete quantum theory of gravity, for example.
So there is this hypothetical particle called the graviton that some are searching for.
So I think there are very many open questions that could potentially be answered through some of the theories that have already been proposed, whether it's supersymmetry or others.
And so I think right now the goal is to continue looking for those while we also continue making very high precision measurements of all of the particles that we know and love.
So I think some ideas for what else could be out there doesn't necessarily come from directly searching for these new particles, but it could come from measuring properties of particles that are already a part of the standard model.
So I think that's a good thing.
I think that's a good thing.
So I think that's a good thing.
It could look entirely right.
But there could be these subtle, subtle deviations.
Like just look at Newton and gravity and then Einstein and gravity, right?
Newton looked right.
Yes, of course.
I mean, to first degree, it was.
But obviously, there are then in certain regimes, some deviations from the more simplistic theory.
And so we think that perhaps there could be other corrections to what we're measuring that
could come from more complicated theories that we're still to validate.
Cool.
So, Michael, one thing I wanted to ask you is, how do you feel that machine learning and
doing that with Python and TensorFlow and all these things have changed physics and the
physics exploration?
Like how much would have it just been more work or stuff actually being discovered that
wouldn't have been discovered?
What do you think?
Yeah, I think a lot of algorithms may not have been discovered or maybe, or I should say,
maybe applied or implemented in a reasonable timescale.
I mean, so, you know, we're not, since we're not professional machine learning, you know,
machine learning engineers or machine learning researchers, a lot of the times what, you know,
the tools we're using, we either built from scratch or use what's available.
And so the kind of this availability of things like TensorFlow make it really easy for us to implement and just try things out on our data and then try to make these connections between the machine learning world.
So things like these generative models, you know, they might not, we may not have access to these kind of really, potentially really fast speed ups or new ways to look at the data, like, you know, with vision or natural language processing approaches.
You know, these algorithms may not even be implemented anywhere.
Nowadays, you can just kind of download the model if you want and use that as a place to start.
And I think that one of the things that's been really helpful is the core kind of C++ libraries that we use at CERN are built with this, with kind of internal dictionaries with the kind of have this idea of reflection so that they can really easy be bound to Python.
And we can just take our data that's in one format and really quickly switch into Python and just start pounding or, you know, running through all these different kind of algorithms and see what comes out.
That's really cool.
So it's a little bit like what NumPy did for numerical analysis.
You kind of have something similar where it's got this sort of native layer really close to Python.
Exactly.
And so, yeah.
And in fact, then we can also interface directly with NumPy and pull our data right into formats that are kind of standardized in the machine learning and data science community.
Yeah, very cool.
Michaela, maybe I could ask you the same question because your work has made computational so much faster.
And it seems like the machine learning played an important role there as well.
If you could really make these simulations 100,000 times faster, that's almost like going from regular computers to quantum computers.
It's such a jump.
Yes, of course.
I mean, obviously, we would like to empower more physicists to be able to do the analysis that they want to do at the precision levels that they require.
And right now, obviously, that simulation bottleneck means that some analysis cannot necessarily be performed with the statistical uncertainties that we would like to have.
Or some of them cannot be performed at all in terms of the required accuracy that they need.
And so hopefully this type of projects will enable more physics to be done and more analysis in the future of the LHC.
Yeah, very cool.
So, Matthew, do you think there's room for normal developers, people who want to just go contribute to open source, maybe some kind of open science they want to play around and just like, hey, they're really good at programming, but they're not physicists.
Is there a place for them to kind of jump in and be part of this?
Yeah, I think so.
I mean, so we have, as Michael mentioned earlier, it's really important that our code is open source.
And so if people actually want to go take a look at it, it's out there.
And then there's also, I think there's some efforts to get something like this started on Atlas, but then on one of our other, one of the, at LHCB, one of the other experiments, they have what's called the LHCB Starter Kit.
And that was started by two PhD students, Kevin Duns and Tim Head, who are now, have now left the field to go work at Google and be a data science consultant.
But they thought like, hey, you know, a lot of our students are coming in and we don't have, and we're all physicists, but we're not all necessarily software experts.
So they created this thing called the LHCB Starter Kit, and they've done some partnering with softwarecarpentry.org to actually go ahead and hold training seminars where people from software carpentry come in and help them learn how to actually get started to do data analysis and programming and physics.
So I think software carpentry and data carpentry, those are really great organizations.
So if people want to get involved there, they definitely can as well.
Yeah, that's really cool.
And I had the software carpentry guys on the show a couple of months back.
Yeah, that was a good episode.
Yeah, thanks.
Yeah, it was really nice.
So I think that's, I think it's great.
I think, you know, there's more and more of these tools are becoming accessible to everyone.
And I suspect uploading to the whole computing system and running stuff there is probably restricted to you guys.
But still, people can work on these algorithms.
And in a sense, they're contributing, right?
People who make TensorFlow or Keras better, they're in a sense making it better for you guys as well, right?
Yeah.
Absolutely.
Yeah, there's a big push right now.
I think Michaela is a great example of this to really kind of go out into the communities that are going and actually building these great tools and interact with them directly and contribute.
So, I mean, we want to try and use really powerful tools to do the best analysis we can.
So the people that are making the Python tools that we use better, they're really directly impacting science.
So we appreciate it.
Yeah, that's amazing.
All right.
Well, I have so many more questions to ask you all, but we're pretty much out of time.
So I want to be respectful of that as well.
Before we go, I'm going to ask you all the two questions.
I'll start with Michael, favorite editor.
Emacs.
Michaela.
Sublime.
Matthew.
Adam with Vim key bindings.
All right.
Right on.
And Michael, notable PyPI package?
PyTorch and Scikit Optimize.
I don't think I've talked about PyTorch on the show before.
Maybe just really briefly talk about what that is.
Yeah.
So PyTorch is a deep learning library that's built on a different way of building the graphical representation of your deep neural network for all the downstream computations.
But actually, the API is just really, it's very, I find it very smooth and easy to really quickly spin up a neural network and have it running.
Excellent.
Michaela, how about you?
I'm a huge fan of Keras.
I've been using it since it was first started as a project.
And it's been enabling me to go from idea to experimentation very quickly.
And then huge shout out, I think, to Matplotlib.
It allowed me to make all the plots that I've ever made during my PhD.
So those two for sure.
All right.
Excellent.
Matthew?
Definitely Keras.
I mean, like Michaela said, like, that's kind of a bread and butter thing for us.
And then also Scikit-HEP, which is a...
HEP is high energy physics?
Yeah.
So similar to Scikit-Learn, how that was supposed to be a toolbox for machine learning, Scikit-HEP is a collection of Python tools that have been developed inside the high energy physics community, but are meant to try and help us interface with things like with NumPy and Pandas data frames and make our lives a lot easier when we're actually trying to go from our root data file format to things like NumPy.
Yeah, very cool.
I'll give you guys a chance for a final call to action.
And people are excited about what they heard.
They want to get involved.
Michael, you can go first.
Sure.
Yeah.
I think there's a lot of ways in which you can get involved, especially through the CERN, you know, open data portal and CERN Open Science.
You can download some of our data.
You can play around with it and also just help spread knowledge about, you know, science and scientific reasoning and, you know, how that can benefit society, both from advancing science and advancing the way we think.
All right.
Excellent.
Michaela?
Well, certainly if you're interested in working at the intersection of science and machine learning or data science, CERN, I think, is quickly becoming one of the best places in the world to do that because the scale and the fascinating problems that we're working on are completely unparalleled.
And we're always looking for new skilled software engineers that are curious about mysteries of the universe.
And as I said before, the community of CERN is truly fantastic.
So we can always use more Python experts.
And you absolutely don't have to be a physicist to make a huge impact at CERN.
Oh, excellent.
Sounds really fun.
I'd love to work there.
All right, Matthew, you got the final word.
Yeah.
So physicists, we love to collaborate and we like to think at least that we have really cool and really hard problems.
So we were always looking for it.
And you consider those to be the same thing, right?
Cool equals hard.
Yeah, probably.
So and we've we've started in the last couple of years to have some really fruitful collaborations with our CS colleagues.
So, you know, if you if you know a friendly neighborhood particle physicist and you want to talk with them, please do.
We we're happy to talk and we really want to try and have our field grow with other fields and try and do the best science we can.
All right.
Thank you all so much for being on the show.
It's been really interesting and I've learned a lot.
Thanks for sharing what you're doing and keep up the good work.
Thank you.
Yeah.
Thanks for having us on, Michael.
Yeah, thanks.
And thanks for for hosting this great show.
You're welcome.
Bye, everyone.
Bye bye.
Bye.
This has been another episode of Talk Python to Me.
This week's guests have been Michaela Paganini, Michael Kagan and Matthew Feigert.
And this episode has been brought to you by Linode and Talk Python Training.
Linode is bulletproof hosting for whatever you're building with Python.
Get your four months free at talkpython.fm/Linode.
Just use the code Python 17.
Are you or a colleague trying to learn Python?
Have you tried books and videos that just left you?
bored by covering topics point by point?
Well, check out my online course Python Jumpstart by building 10 apps at talkpython.fm/course
to experience a more engaging way to learn Python.
And if you're looking for something a little more advanced, try my write pythonic code course at talkpython.fm/pythonic.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
Thank you.
I'm out.