Gene therapy holds the promise to permanently cure diseases that have been considered
lifelong challenges, but the complexity of rewriting DNA is truly huge and lives in its
own special kind of big data world. On this episode, you'll meet David Born, a computational
biologist who uses Python to help automate genetics research and helps move that work
to production. This is Talk Python to Me, episode 335, recorded September 15th, 2021.
Welcome to Talk Python to Me, a weekly podcast on Python. This is your host, Michael Kennedy.
Follow me on Twitter where I'm @mkennedy and keep up with the show and listen to past episodes
at talkpython.fm and follow the show on Twitter via at talkpython. We've started streaming most
of our episodes live on YouTube. Subscribe to our YouTube channel over at talkpython.fm
slash YouTube to get notified about upcoming shows and be part of that episode. This episode
is brought to you by Shortcut, formerly known as clubhouse.io, and us over at Talk Python
Training. And the transcripts are brought to you by Assembly AI. David, welcome to Talk Python
to Me.
Thanks, Michael. It's great to be here.
Yeah, it's great to have you here. One of the things I really love to explore is the
somewhat non-traditional use cases of Python that are not straight down the, I'm building
an API and something that talks to a database or a startup, right? Something like that, but
blending it with other technologies and science and whatnot and genetics plus Python, it's going
to be interesting.
It sure is.
When you got into this, did you start out on the biology side or the programming side of
the world?
I definitely started out on the biology side. Yeah. So I went all the way through grad school
with relatively minimal formal programming experience.
Yeah. Okay. So you studied biology and genetics and whatnot. And then how'd you end up here
on a Python podcast?
Yeah. So I always thought that programming would be cool, but I didn't really have much of an
opportunity through my undergraduate studies to really do much formal programming. I took our one
computer science class that my college had to offer. It was in C++. I think I wrote a Boggle program,
something with some recursion in there. It was pretty fun. Didn't really get to use Python until
graduate school. I was in a genetics course and we were basically tasked with doing some data analysis
on published data and then reproducing some plots in a figure, then extending it further.
My partner and I decided to learn Python, teach it to ourselves so that we could do this. We heard that
it was a good way to do data analysis in biology. And so we basically taught it to ourselves and we used
NumPy, some basic string searching and things to redo this analysis. And it was really amazing what we could
do with Python for that civil project.
That's awesome. How'd the learning project go? Like coming from not having a ton of programming,
what was your experience like?
It was relatively easy. I would say I think my brain sort of fits pretty well with how
programming languages work, but it was definitely a lot in a short amount of time to really dive into
how to make sure your while loops don't stay open. And then someone tells you maybe you shouldn't use
a while loop at all. I was learning a lot of things not to do right away.
Yeah, of course. But you had to get all the analysis done, right? So you got to solve all the problems
and power through.
Yeah, we did. We got our analysis done there and we got some, we reproduced some plots and got some
new analysis made. And I think we really just, I really came to appreciation of how much you can do
in a short amount of time with just a little bit of coding knowledge or essentially none.
Yeah. I think that's really an important takeaway that a lot of people, you know, maybe many people
listen to podcasts already kind of know, but I think looking in from the outside, it feels like,
oh, I've got to, you know, go get a degree in this to be productive or useful. And really what you need
is like a couple of weeks and a small problem and you're already already there.
Absolutely. Yeah. I've all, I've definitely found that just learning through, through doing has been
the way I've, I've worked entirely. I have essentially no formal programming training, no,
no coursework and I'm using Python every day.
Yeah. That's fantastic. Yeah. I didn't take that much computer science and college either. Just
enough to do the extra stuff from my math degree. Very cool. All right. Now how about today? You're
working at Beam Therapeutics doing genetic stuff. Tell us about what you do day to day.
Yeah. So I'm on the computational sciences team at Beam Therapeutics. We're a gene editing company. So
we develop these precision genetic medicines that are, we're trying to develop them to cure genetic
diseases that are caused by single genetic changes in the genome.
Like a point shift mutation or something like that.
Yep. Yep. And so if you have one of these genetic changes, you might have a disease that is lifelong
and there aren't any cures for most of these diseases. So we're trying to create these, we call
them hopefully lifelong cures for patients by changing the genetic code back to what it should be.
That's incredible. It seems really out of the future. I mean, it's, I think it's one thing to
understand genetics at play and it's even amazing to be able to read the gene sequences, but it's
entirely another thing I think to say, and let's rewrite that.
Yeah. We're definitely at the cutting edge of a lot of biotechnology and science that has really come
to have had in the last decade with the, with CRISPR and technologies that use CRISPR, which ours do be able to precisely target genetic sequences. It's a really fascinating place to work and it's a privilege.
I bet a lot of people go to work and they end up writing what you might classify as forms over data. It's like, well, I need a view into this bit of our database or I need to make, be able to run a query to just see who's got the most sales this week or, you know, something like that.
And that's important work and it's, it's useful and there's cool design patterns and whatnot you can focus on, but it's also, it's not like what people dream of necessarily building when they wake up.
But this kind of science, like maybe so, right? Like these are the really interesting problems that both have a positive outcome, right? You're helping cure disease, not just, you know, shave another one, one hundredth of a percent off of a transaction that you get to keep or, you know, something like that. Right. From like in finance.
Yeah.
You get to use really cool tech to do it too, like programming wise.
Yeah. I mean, one of our dreams that we joke about on the computational team is that like it's conceivable one day we could, you know, say like, Hey Alexa, how do we cure sickle cell disease? And it'll tell you what parts of our technology we should put together to cure that disease. That's sort of like the pipe dream of, of where we could go if we combine all our data in the right ways. And I think all of the stuff we'll talk about today is really just laying out the framework for that.
Yeah, absolutely. So you mentioned CRISPR. Maybe tell people a bit about that biotechnology.
Yeah. So CRISPR is molecular machine, which a lot, which targets a very specific place in a specific genetic sequence. And so usually people are using CRISPR to target a specific place in the genome, a specific sequence. And what CRISPR does naturally is to cut at that sequence.
So it'll cut in a very specific place in the genome. And as people were using CRISPR, we could actually decide where it's going to cut by giving it a different targeting sequence. This sort of like directed molecular machine is a basis of a whole new field of biotechnology using CRISPR and CRISPR derived technology.
So our technology is like kind of a CRISPR 2.0 where we don't use CRISPR to cut. We use the localization machinery and we add onto it another protein, which just changes a base instead of cutting the DNA itself.
So it's slight variation, but it's still using this same CRISPR technology.
Okay. So let me see if my not very knowledge filled background understand here is kind of a, has a decent analogy. So does it work basically like you give it like almost like find and replace, like you give it a sequence of pairs and it says, okay, if I find a TTCAT, like enough specificity and that it's like, that's the unique one. And then it does like cut at that point. Is that kind of how it works?
Yep. It's pretty much just like that. We give it the sequence you want to target. And then if it finds that sequence in the genome, it will cut the genetic material, the DNA at that position for normal CRISPR.
When you're coming up with these and you say, we're going to rewrite the DNA to solve this problem. How do you do it on a large enough scale? Right. Like how do you, I mean, how much of the body has to be changed for this to be kind of, to be permanent, right?
Right. It's definitely a tricky question. I think we are, we definitely leverage human biology, how the human body works for a lot of these problems. For example, our, some of our leading drug candidates are for sickle cell disease. And because of the way sickle cell disease manifests in red blood cells and red blood cells are created through a set of, through, there's a specific type of cell that creates red blood cells.
And we can, if you access that type of cell and you cure, cure a sickle cell in these progenitor cells, the stem cells, then you can create all red blood cells from a cured population.
So if you can target it to the regenerative cells, you can cure sickle cell throughout the body essentially, because the symptoms are from red blood cells. There's a lot of diseases that, which by curing a single organ, you can cure the symptoms of the disease because that's where it actually manifests.
Right. Like, like, like diabetes or sickle cell anemia or something like that?
Yeah. Like sickle cell diseases in the liver, like some blindnesses in the eye. By just targeting specifically where the symptoms occur, you can cure the disease.
Yeah. That's amazing. Is it a shot? Is it a blood transfusion or like, how do you deliver this stuff?
Yeah. Delivery is a huge place of research and it definitely depends on the type of targeting we're doing. So for something in the eye, it would probably be an injection.
For something in the blood, it's slightly more complicated and it's not an injection. For something in the liver, it would probably be more akin to a injection or dosing regimen.
Okay. It sounds really fascinating. Like I said, it feels like it's a little bit out of the future to be able to come up with these and just say, no, we're just going to rewrite this little bit of the genetics and I think it'll be good. But if you can make it happen, it's pretty clear how it's obviously a benefit, right? Like those progenitor cells, they eventually have to recreate new ones. And then the way they do that is they clone their, their current copy of the DNA, which is the fixed one, right? So if you can get enough of them going, it'll just sort of propagate from there.
Yep. And we're also the benefit that sometimes you only need to cure a small fraction to remove the symptom. So lots of things going for us.
That's awesome. So I'm sure there's a lot of people involved in this type of work. What exactly are you and your team working on?
Yeah. So our team, we call ourselves the computational sciences team. We really sit in the, in the middle of the research and development arm of the organization, processing all of our sequencing data and some other data as well. And as you can imagine, with our technology changing DNA, changing genomes, there's a lot of sequencing data because what we're trying to do is change a genetic sequence.
So we have to read out that genetic sequence and then figure out has it changed, how many copies are changed and things like that.
Yeah.
The field of the techniques of next generation sequencing NGS are pretty broad. And we deal with a lot of different types of these next generation sequencing assays that are being done at the end. Our team really processes, analyzes, and collaborates with the experimental scientists on performing and developing these experiments.
Cool. So the scientists will do some work and they'll attempt to use CRISPR-like technology to make changes. And then they measure the changes that they've made. And it's, you all sort of take that data and compare it and work with it.
Right. Yeah. The act of measuring the changes itself is relatively computationally intensive. So we run and support pipelines for all of these standard assays as well, which is, is part of our, our job.
How much data is in DNA? I mean, I know that human biology stores or just biology stores an insane amount of data, but also computers store an insane amount of data and processing. So I'm not sure like which, where the sort of trade-off is, but what sort of data are we talking about? How much?
Yeah. I mean, it definitely depends on the type of assay that we're doing, like the scale of the data. I would say usually we're not looking, sometimes we're looking at things that are looking at all the bases in the genome. More often we're looking at defined regions of the genome where we're trying to make the change.
Right. You're like, this one gene is the problem, right? And let's look at that.
Right. Yeah. We're trying to target here. So that's where we're looking. It definitely depends on the assay, but, but I guess in terms of data scale, in terms of file sizes, perhaps that would be accessible.
Yeah.
For standard things, it would be on the order of a few gigabytes per experimental run. For some of our larger assays, it's 10 to 100 times that.
Right.
Per experiment.
That's a lot of data, but not impossible to transmit sort of amounts of data, right? Or store.
Right. Every little piece of it is, is pretty manageable. When you start combining them together and looking at some doubt, like your downstream results of things, the data does get pretty, pretty large.
But I wouldn't say we're at the scale of like big data analytics at Google or anything like that.
Yeah. Yeah. Or the LHC. If you've ever looked at the data flow layers at LHC, it's like the stuff near the collectors is just unimaginable amounts of data. Right.
Yeah. I haven't looked, but I'm sure it's certainly.
Yeah. They've got a lot of stuff built in, like onboard the detectors to filter it down a bunch. And then it goes for processing. Then it gets filtered down some more. And then like eventually it gets to a point where like now we have enough space.
on hard drives to save it.
But before that, it was like,
it couldn't be saved on hard drives.
It was just too much.
Yeah, very interesting.
But it sounds to me like
some of the real big challenges for you all
is the computational bits, right?
Because if you take all these things
and you can end up in like
combinatorial comparison scenarios,
I'm sure that that can really blow up
the computing time.
Yeah, I think definitely
there's a lot of the challenges
sometimes in these combinatorics.
And there's also just a lot of,
there's steps that have to go on
to process a lot of this data.
There's a lot of biology specific
pieces of software that we use
for various things.
And we have to string them together
to create a complete
data processing pipeline.
And a lot of that art is part of our job,
how to get all these tools to behave together
and act in unison
to actually process the data effectively.
I can just imagine
some of these pipelines are tricky.
It's like, well, okay,
so it starts over here
where the robot gets it
and it reads this data
and we can access that off the hard drive.
And then it has to be sent
to this Windows app
that actually doesn't have an API.
So we got to somehow like
automate that thing
and then get data out of it.
And then is that what it's like?
That's some of it for sure.
Tell us about your life.
Okay.
Yeah.
I mean, there's a couple aspects
and I think we might touch on them,
but for sequencing experiments,
the pipelines are more defined
because we usually get the data
from a source that's already in the cloud,
which I'm always happy about.
If we can start in the cloud,
we'll stay in the cloud
and that's a nice place to be.
For data that's coming directly
from instruments on premises,
there is another layer of art
that has to do with software
on the instrument,
software that gets the data to the cloud
and moves it around
between our other database sources.
And that is a fun project in itself there.
Yeah.
I can imagine.
I've worked in some areas
where it's like collecting data
from all these different things
and then, you know,
to process it and move it along.
And yeah, it's not always just
send it from this Python function
to that Python function
to that Python function.
There's a lot of janky stuff
that wasn't really meant
to be part of pipelines possibly.
Absolutely.
Yeah.
This portion of Talk Python to Me
is brought to you by Shortcut,
formerly known as clubhouse.io.
Happy with your project management tool?
Most tools are either too simple
for a growing engineering team
to manage everything
or way too complex
for anyone to want to use them
without constant prodding.
Shortcut is different though,
because it's worse.
No, wait, no, I mean it's better.
Shortcut is project management
built specifically for software teams.
It's fast, intuitive, flexible, powerful,
and many other nice positive adjectives.
Key features include
team-based workflows.
Individual teams can use
default workflows
or customize them
to match the way they work.
Org-wide goals and roadmaps.
The work in these workflows
is automatically tied
into larger company goals.
It takes one click
to move from a roadmap
to a team's work
to individual updates and back.
Type version control integration.
Whether you use GitHub,
GitLab, or Bitbucket,
clubhouse ties directly into them
so you can update progress
from the command line.
Keyboard-friendly interface.
The rest of Shortcut
is just as friendly
as their power bar,
allowing you to do
virtually anything
without touching your mouse.
Throw that thing in the trash.
Iteration planning.
Set weekly priorities
and let Shortcut run the schedule
for you
with accompanying burndown charts
and other reporting.
Give it a try
over at
talkpython.fm
slash shortcut.
Again,
that's
talkpython.fm
slash shortcut.
Choose shortcut
because you shouldn't have
to project manage
your project management.
with robots,
do you have to actually
talk to the robots
and like any of the
type of automated things?
Yeah,
for lab automation
is what we
call our
team that has
the robots
as we like to
call them.
As you can imagine
with a lot of
these types of experiments,
they can be
made much more efficient
if we can have
robots doing
the actual
transfer of liquids
and incubation
and centrifugation,
these scientific
techniques that
sometimes you need
someone in the lab
to do,
but oftentimes
you can automate them.
So the
lab robotics
aspect is an
important part
of how we can
efficiently generate
data.
A lot of the
issues around that
come with
how to
pass
instructions
to the instrument
and how to get
back
data from the
instrument
when it's done.
and then
there's a
whole other
art of
making the
instruments
actually
orchestrate
together,
which is
held in a
different world
of software.
I don't actually
work on that
part myself.
Yeah,
that part of
programming
definitely seems
a little bit
magical,
getting factories
or automations
of different
machines to
work together.
It's very cool.
There's a whole
lot of
proprietary software
involved in
actually running
the instruments,
but in terms of
the problems
in getting the
data to them,
I think it's
one of those,
it could be a
relatively common
software problem
of getting
information from
somewhere in
the cloud.
We have
electronic lab
notebooks and
laboratory data
systems that
are in the
cloud and
users will be
submitting
information about
how they want
their samples
processed by
the robots.
There's the
problem of
getting that
to the
instruments
themselves.
And I think
it actually
sort of
reminded me of
the talks you
guys had in
episode,
I think it was
327, the
small automation
projects.
We end up
with quite a
few of those
here where we
have these
relatively small
tasks of
take data from
an API,
put it somewhere
where a robot
can access it.
Usually we use
AWS S3,
and these sort
of very small
data handling
tasks end up
being these nice
little projects
for Python to
come into play.
Awesome.
I can see that
that definitely
happens, right?
if a file
shows up
here, grab
it, upload
it to that
bucket, and
name it
whatever the
active experiment
is with the
data or
something like
that, right?
That's a very
small program
to write, but
then that becomes
a building block
in this flow,
putting these
pieces and
machines together.
Right, yeah.
Once it comes
into the cloud,
we do another
set of data
processing on it,
we upload it
to our
current databases,
and all of
that we
orchestrate on
AWS using,
I guess they
call it their
serverless design
patterns, so we
don't have to
handle anything on
our own computers.
Yeah, that's
really nice, and
the serverless
stuff probably
helps you avoid
running just tons
of VMs in the
cloud, right?
Like, it can all
be on demand,
like the Lambda
trigger is a file
appears in this
S3 bucket, so
then it starts
down the flow,
right?
Absolutely, yeah.
I really don't
like maintaining a
lot of infrastructure,
although we do
have,
a good amount
of it that we
do have to
maintain, I
find that these
small Python
functions are
the perfect
use case for
those event-driven
Lambda functions,
which are running
these very simple
pieces of code.
When an object
appears in S3,
they get a
small event about
when the object
was uploaded,
and then they
do their thing,
a little bit of
data conversion,
send it to an
API, and now
the data is in
our data store.
And those
things just
happen, and
they're super
consistent,
they don't
require anything
on my end to
maintain.
It's a pretty
beautiful pattern.
That's awesome.
They don't need
things like,
oh, there's a
new kernel for
Linux that
patches a
vulnerability, so
let's go and
patch our
functions, right?
Just it's all
magic.
It all happens
on its own.
Right.
It does feel
like magic a
lot of the
time, yeah.
Setting them up
can be a little
bit of a challenge,
but once they're
there, they are
very consistent.
Yeah, yeah,
cool.
One of the
things you
talked about
doing, you
seen, was
the AWS
CDK, or
Cloud
Development
Kit.
I think I've
heard of this
before, but
it's like, I've
definitely not
used this
personally.
Tell us, what
is this?
How does it
help you?
Right, so
when I started
dabbling with
these Lambda
functions, what
I found pretty
soon after I
had four or
five of these
functions, which
I had uploaded
on the
AWS Council,
and if
anyone's used
that, know
that it can
be pretty
tedious once
you have a
few things
running, getting
all your
permissions and
things set up.
And I was
looking for
better ways to
do this, and
at that point,
it was early
days for this
AWS Cloud
Development,
AWS CDK, and
what it is, is
a way to write
your infrastructure
as code in
essentially any
common language,
programming language.
write JavaScript,
TypeScript, I
think you can
do Java, Go,
and Python
perhaps.
And using it,
you can define
your cloud
infrastructure in
an object-oriented
way with all the
parameters they
need, and you
can deploy it to
different AWS
accounts, you
can take it
down, you can
reconfigure it,
you can look at
how it's changed
since your last
commit, and you
can store all
that configuration
in source
control.
And this has
allowed us to
scale up,
in terms of
number of
Lambda functions,
I think we
have maybe
60 or
something now,
which would
be unmaintainable
in the
console, but
they're
essentially just
completely maintained
inside of
source control
these days using
AWS CDK and
Python.
Yeah, it seems
super neat.
I have not used
it, but on
the page, which
I'll link to in
the show notes,
they have
Werner Vogels,
the CTO of
Amazon, AWS, and
talks about some
of the benefits and
kind of how it all
fits together.
And like you
said, you can
store your cloud
structure definition
in source control.
You can run unit
tests against your
infrastructure to
say, if I apply
all these commands
to AWS, so I
actually get what I
was hoping to get
out of it.
yeah, it seems
like a really neat
thing for this
infrastructure's code
bits.
Yeah, and I
think it definitely
really shines when
you're developing
larger pieces of
infrastructure, but
I would encourage
people to check it
out, even if they
have a small
automation type
project.
This is what I
was thinking of
when I was
listening to
episode 327 the
other day.
When you have
these things you
want to run on
your computer with
a cron job, you
can actually run
them for free on
AWS Lambda.
You get a bunch
of free time on
the free tier, and
you try it out.
You don't need to
make sure your
system D processor
for whatever is
running, and it's
a pretty cool way
to get familiar
with how to do
some of these
things on AWS.
I'm not sure if
this also exists for
other cloud
providers.
We use AWS in
particular, so that's
what I know, but
it might also exist
for things like
Azure and Google
Cloud.
Sure.
Yeah, maybe.
I don't know if it
does either, but it
definitely seems
useful for what
you all are doing
there.
One of the things
it could be useful
for, and one of the
challenges I suspect
that you run into
more than a lot of
places, certainly more
than like an e-commerce
site or something like
that, is
reproducibility.
If you're going to
say, we're going to
come up with a
treatment that
literally gets
injected into people,
you've got to go
through FDA, you've
got to have
possibly peer-reviewed
stuff.
There's a lot of
reproducibility and
data stewardship
going on, I
imagine, right?
Absolutely.
That's definitely the
case.
We are very
cognizant of the
fact that we have
to make our
software as
reproducible as
possible.
I think a lot of
that lends itself to
just using good
development practices,
containerizing everything
that you're doing for
data processing,
getting all your
versions, appropriate
source control, all
of these things.
And the
infrastructure is
definitely a piece
of that because if
we can't deploy the
pipeline in the same
way a year from
now, we won't be
able to get the same
results from the
data.
And that's a
problem.
That's a big
challenge.
It's tricky, right?
Because things like
containers and source
control, they'll
absolutely get you
very far.
But then you're also
depending on these
external things that
have been very
stable and very likely
will be, like
Lambda, for
example, but what
if Amazon went out
of business, right?
Which, you know, is
kind of laughable,
right?
It's only like
doubled its revenue
this year or
something.
But, you know,
theoretically, it's
possible that AWS
decided to shut down
or something like
that, right?
Yeah, it's
definitely true.
So you got to
think, these are
trade-offs, right?
But at the same
time, it's so
enabling for you to
just scale out all
this workload.
Yeah, I mean, in
terms of how we
create data pipelines,
I think a lot of
the ways we do it,
we do have to be
aware of creating
them in a way that
you can run them
outside of the cloud.
Sometimes we need to
allow a third party
to run our data
analysis in a
regulated way.
And that requires us
to have essentially
running it internally,
we run it in the
cloud, which is
efficient for scaling,
but we also need to
be able to take that
same piece of
software and run it
in a way that may
not be on AWS,
may not be in the
cloud at all.
And that creates
some interesting
software challenges.
Yeah, I'm sure,
because so many of
the APIs are, you
know, cloud native,
right?
You import Bodo,
Bodo do this thing,
or something like
that, right?
How do you deal
with that?
I think a lot of it
comes down to
structuring your
workflows and your
data pipelines in
ways that are,
they're not really
using the cloud as
much as they're,
they're using the
cloud as a, well,
the file system
and a way to
gain compute,
but they're not
using any piece of
the cloud to actually
affect the data
processing itself.
So ideally, if you
create the data
pipeline in a way
that is appropriate,
you can both run it
in the cloud and run
it locally and
achieve identical
results because it's
containerized.
Primarily, that's the
reason, I guess.
Yeah.
But doing that, I
think, is a challenge
in itself.
It's really like the
process of getting a
data pipeline built is
a big ask in our
field.
I think a lot of
companies spend a lot
of time thinking about
this.
Yeah.
I'm sure they do.
Getting it right is a
huge enabler, right?
So let's talk about the
process of maybe coming
up with a new data
pipeline.
What does a software
project look like for
you all?
Yeah.
So it usually begins
with a collaborative
meeting with some
experimental scientists
where we discuss
what's the experimental
design going to be
like?
What are we going to
be looking at in the
data?
And then the
experimentalists will
go and they'll
generate some form of
sequencing data.
At that point, we
generally take the
data, open up some
Jupyter notebooks or
some small Python,
sometimes even just
bash scripts to try
to use some of those
standard third-party
tools.
These are things like
making sure all the
sequences are aligned
to each other so that
you know when there's
differences and making
sure the quality is
correct, things like
that.
These are pretty
standard bioinformatics
things.
Right.
Then for sequencing
assays, there's usually
a couple spots where
there's some real
experimental logic going
into it where often
we'll have to write
custom code in Python
to say, if there's this
sequence here, it means
we should keep the
sequence.
And if there's this
sequence here, we
should divide the
sequence in half or
something like that.
And so that code gets
written in Python, maybe
it's in the Jupyter
notebook or another
script.
And we sort of do this
really slow testing
depending on the size of
the data.
It might be locally on
a laptop or in a small
cloud-based HPC type
cluster.
Right.
This is like where we're
doing...
You're not trying to
process all the results.
You just want to spot
check and see if it's
coming out right before
you turn it loose,
right?
Right.
Or we're very patient.
It's a little bit above.
Sometimes it's very
difficult to take only a
small fraction of the
data, but we try when we
can.
Once we settle on
something that we think
is pretty locked down,
we'll take it out of the
Jupyter notebooks.
We don't try to use
Paperville or anything
like that.
We try to get it out of
there as soon as
possible into some more
complex scripts.
They might be a shell
script that runs a number
of other scripts in order,
or we might start using
some sort of workflow
manager.
workflow managers in
bioinformatics are pretty
common because everyone
has the same problem of
running all these third
party tools together and
custom code.
Right.
It's a lot of shared
tools as well, probably.
Right.
They all use this library
or that app.
Absolutely.
Yeah.
There's definitely...
Whatever it is, right?
There's a whole bunch of
standard bioinformatics
tools that we run on
almost everything.
And so some of the
workflow managers are
designed to specifically
work very well with those
tools and others are
pretty agnostic of what
you're doing with them.
Yeah.
One of the things I find
interesting and listening
to you talk about this,
it just reminded me, is
so often we see these
problems that people are
solving, right?
Like, oh, here we're
using CRISPR to do all
this work.
And then you talk about
the tools you use.
It's like, oh, yeah,
we're using like NumPy
and Pandas and Jupyter
and these kinds of
things.
And the thing that I find
really interesting is for
software development,
there's so much of the
stuff that is, it's just
the same for everyone,
right?
They're doing the same
thing.
And then there's 10 or
20% that like, yeah,
this field does this part
different, but there's
like 80% of, yeah, we
kind of, we should use
source control.
We're using Python.
We're using notebooks.
We're using Pandas and
that kind of stuff.
and it's the
similarities are way
more common than I
think they appear from
the outside.
It's a great point.
And I think we'd, we'd
all be better off if we
reminded ourselves of
that more often that
we're not just because
we're doing biotechnology
and things with Python
that we're largely very
similar to other
software developers doing
data science on business
topics or finance or
just standard web
development.
You could be at a hedge
fund and you're like,
this is that different
than what I'm used to
actually.
Yeah.
And if you think like
that, we, you end up
with, I think, better
practices overall in
software.
Yeah.
Oh, I totally agree.
Also, I suspect that
this fact is what also
makes the data science
tools and all the
tooling around Python,
the libraries and
whatnot.
So good because it's
not just like, oh, the
biologists have made this
part of it really good.
It's like most of the
people are all really
refining like the, the
core parts.
Right.
Yeah.
I think that's, that's
definitely the case.
And some of the biology
specific tools are, are
a little, a little
walkie when you, when
you start using them.
I think, I mean, things
like pandas are, are
really amazing.
You can tell the amount
of attention or something
is different.
Cool.
what is moving to production
look like for you.
So you talked about
sometimes it's, you start
the exploration and stuff
in notebooks, which is
exactly what they're built
for.
And then moving to maybe a
little more composition of
scripts and whatnot.
And eventually somehow you
end up with, you know,
Lambda, cloud databases,
things like that.
What's that flow?
Yeah.
So the, the process of, I
guess we say like
productionizing a pipeline
is we've had pretty well
set now.
and generally how it
works as we say this
pipeline is about done
and we'll hand it off to
myself or one of my
colleagues to start the
process of getting it
fully cloud capable and
scalable.
And what that means for us
is to take the software
in whatever form we've
gotten it from our
colleagues and put it
into a workflow manager.
And we, I think every
company has their own
version of workflow
manager that they, they
choose.
we're using Luigi, which
is fully Python based.
It was originally
developed at Spotify to
do this sort of task.
It uses like a GNU make
type target file dag
creation.
I don't know all the
technical terms to
describe how the tasks
are built, but you
essentially, you have a
task at the end and you
say it requires the
output of this other
task.
And then that task
requires the more.
And as they build up,
you create a graph of
what tasks need to be
done to get to our
output.
Right.
A directed acyclic
graph.
And then the workflow
can decide like, oh,
these two things are
independent.
So let's scale them out
separately.
now this one has to wait for
these two to finish and
then get its results.
And that coordination can
be really tricky.
Exactly.
And there's a number of
common workflow managers
and bioinformatics.
I think the two most
common are snake make and
next flow.
Luigi is, has also been
really good for us.
We like it primarily
because it is fully
Python based and it uses
standard Python syntax,
which allows us to
really, if we need to,
get out of the hood and
add some customization,
extend it where we need
to, or, or fix things
that we don't like about
it.
And that was a really
important part of our
decision and choosing
Luigi over some of these
other workflow managers.
Yeah, for sure.
I had a nice conversation
with the Airflow, Apache
Airflow folks, and not
too long ago.
And one of the things
that really struck me
about this is the ability
for people to work on
little parts of the
processing, a little bit
like that, little Python
automation tools or a
little Python projects
that you described
earlier in episode 327
in that instead of
trying to figure out
all this orchestration,
you just have to figure
out, well, this little
task is going to do a
thing.
And that, like I said,
maybe it means see the
file and then copy it
over there.
And if your job was
see a file here, copy
it over there, like
that's a really simple
job.
You can totally nail
that.
You know what I mean?
Yeah.
Whereas if your job is
to orchestrate this,
this graph and make sure
that it runs on time
and here's the failure
case, all of a sudden
that becomes super,
super hard.
So these seem really
empowering, almost like
the promise of
microservices is like
you get to focus on
one little part and do
that.
Yeah, it definitely,
it definitely helps.
And it helps with that
idea of having these
small tasks.
It really helps with how
you can develop it and
reuse the components
for each task.
Like we might, as we
said earlier, that there
are these third party
tools that end up being
used in almost all of
our pipelines and using
something like Luigi or
any workflow manager.
You can reuse the tasks
in different contexts as
need be and you can have
your perfectly optimized
way of using that task
everywhere.
That reuse is really nice
and it's something I think
a lot of software
developers appreciate.
Yeah, for sure.
So if you look at some of
the folks that are using
Luigi, so Spotify, as you
said, created it, but also
Foursquare, Stripe, Asana,
SeatGeek, a lot of companies
that people probably heard
of like, oh, these places
are doing awesome stuff.
Let's be like them.
Yeah.
A lot of places use it for
like Hadoop and things like
that.
And one of the nice things is
you mentioned about like how
Airflow has the same model
where you can create these
contributions, which are
different connectors for
Luigi or for Airflow, where
you can connect them to
their different cloud
providers or different data
stores, things like that.
And that allows you to use
Luigi or any workflow manager
in numerous different
contexts, whether it's
locally on your own
computer running things in
Docker containers or whether
it's deploying out to AWS
and scaling massively
horizontal.
These workflow managers
really support that.
And that's why they're a
necessary component in how
we productionize our data
pipelines.
Yeah.
Again, they just seem so
empowering for allowing
people to focus on just
each step independently,
which is excellent.
Did you consider other ones
like did you consider
Airflow or DAXer or any
of these other ones or did
you find this fit and
you're like, we're going
with this?
We did look at some
other ones.
We were using Nextflow
for a little bit, which is
a bioinformatic
flavored workflow
manager.
It's very focused on
bioinformatics as its
primary use case,
although you could use it
for anything.
Syntax is similar to
Groovy and it's based
in Groovy.
And that was one of the
detractives for us is that
it was a little hard to get
under the hood and use
that because of it.
I did briefly look at DAXer
after hearing a few
episodes, I think.
No, maybe a different
podcast.
Yeah, I did have Tobias
Macy on to give us an
overview of the whole data
engineering landscape.
So possibly, I know he
spoke about it then, but I'm
not sure when you heard
about it.
Yeah, so I heard about it
on a podcast and probably
this one as well.
And I did look into it, but
it didn't have it at that
time.
It was pretty early and it
didn't have any connectors
to AWS in the ways that we
like to use.
The connectors, right?
That's such an important
thing because otherwise
you've got to learn the API
of every single thing you're
talking to.
Yeah.
These days, knowing how
Luigi works, it actually
wouldn't have been that big
of a task to look under
the hood.
But yeah, so we did choose
Luigi and particularly we
like how it handles
deployment to AWS and we
use it on the service
called AWS Batch, which
is, I guess it might be
similar to like
Kubernetes pod, although I
haven't done anything with
K8s or anything like that.
so I'm not speaking from
experience, but it
essentially scales up EC2
instances, these elastic
compute instances on the
cloud as you need them.
And it gives out jobs to
the virtual computers as
necessary.
So it spins them up,
allocates jobs in a
Docker container.
They run when there's no
more jobs on the instance,
it shuts off.
Okay.
So you come up with like an
AMI, an Amazon machine
image that's pre-configured,
set up, ready to run.
And then you say, I'm going
to give you a bunch of
data.
Each one of these data gets
passed to the machine and
it runs and then maybe even
shuts down when it's done.
Yeah.
There is an AMI.
We keep the AMI pretty
simple because it's the sort
of the base for all of
them.
And then the way batch
works is you have your top
level AMI that's called the
compute environment, I
believe.
And then inside of it, you
run the actual job and the
job runs inside of a
Docker container.
Oh, I see.
So the Docker container is
pre-configured with like all
the Python dependencies and
the settings that it needs and
whatnot.
Right.
So we have each task in
Luigi, each little piece of
work as its own Docker
container.
and then we push those
out into the cloud and they
get allocated out onto these
machines.
They run their task.
Data comes in from S3, goes
back out to S3.
Nothing is left on the hard
drives and then they
disappear.
They're these little ephemeral
compute instances.
And that's all managed by a
workflow manager such as
Luigi or Airflow or
Nexflow.
Wow.
That's pretty awesome.
One of the things that I
remember reading and
thinking that's a pretty
crazy use of the cloud was
this Ars Technica article
from look at that year,
2011.
Oh, wow.
So if you think back to
2011, like the cloud was
really new and the idea of
spending a ton of money on
it and getting a bunch of
compute out of it was still
somewhat foreign to people.
So there's this article I'll
link to called the 100 or
1,279 per dollar per hour,
30,000 core cluster built on
Amazon EC2 cloud, which is
this company, this
pharmaceutical company that
needed to do a lot of
computing.
And they said, instead of
buying a supercomputer,
basically, we're going to
come up and just fire off an
insane amount of cores.
And I think if I remember
reading this correctly, that
they weren't allowed to make
that many cores in a single
data center.
so they had to do this
across.
They also had to coordinate
in like multi data center
type of processing as well
because, yeah, it's just the
scale of everything.
This seems like the type of
work that you all might be
doing.
Yeah, it does look very
familiar.
Okay.
You got any war stories?
Can you tell us anything
about this?
Yeah, we do occasionally have
a certain type of molecular
modeling job that we can scale
very wide.
And I think these, this sort
of 30,000 number looks pretty
familiar.
I think our largest jobs
today have been about 10,000
CPUs wide and running for a
few days.
So maybe like four or five
days.
I think the number was like
four or five days on the 10,000
cores.
Yeah.
It's a, it's, it's a lot.
And I, so I think it was like
over a million CPU hours on AWS
batch.
And that was just something that
we could really heavily
parallelize and we needed the
data fast and it worked.
You pull it all back and
aggregate it all together at the
end that it was a really useful
data set.
And it's pretty amazing what you
can do on some of these cloud
providers by going really wide.
Yeah.
It's crazy.
You know, I mean, there's some
places where it makes sense to
like build true supercomputers
like Oak Ridge has this thing
called summit, which is like
this insane supercomputer that
they have.
But a lot of times there's the
latency of getting something like
this set up.
There's the overhead of, guess
what?
Now you're an admin for like an
insane, there's only three in
the world supercomputer type
system.
It's got to be empowering to be
able to just go hit go and then
let all this happen and not worry
about all those details.
Right.
Yeah.
And there's definitely still
some stress involved in, in
starting one of these jobs.
If you just, you know, it's not
cheap anyway, slice it.
We do, we do try to, you know,
do everything we can to make it
as, as efficient in cost as
possible.
But yeah, I'm sure there's,
there's two aspects to it that
really jump out to me.
One is the, Oh, was that the
wrong version of the code?
The one that still had the bug?
We got to throw away all the
data.
Whoops.
We just ran for five days and,
you know, thousands of dollars
got burned up and there's a bit
of a problem there.
So whoops.
And now it's five days later until
you get the answer as well.
Right.
That's one.
And then I'm losing track of my
thought on my other one, but
yeah, anyway, it's just got to be
stressful to be like, to set that
up and then press go.
Right.
it definitely is.
I think one of the benefits of a
lot of our computational team does
have a spare metal background and
doing experiments in the lab, these
sort of numbers, like tens of
thousands of dollars for an
experiment are not really that
uncommon.
And those kinds of things, you get
really careful and you check all
the boxes and double check
everything.
And so I think a lot of us have
had that, that experience.
And so even when we're dealing
with software, we'll be very
careful and we'll do the tests and
quality control before we really
let her rip.
Yeah.
I, you know, I worked on some
projects when I was in grad school
that were on a Silicon graphics, big
mainframe type thing.
And obviously much lower importance
than, you know, solving diseases and
stuff is just solving math problems.
But I remember coming in to work on
the project one day and none of our
workstations could log in to the
Silicon graphics machine.
And what is wrong with this thing?
And it was so loud.
It was in the other room.
You could hear it roaring away.
in there.
It clearly was loud.
And what happened was it wasn't
me.
It was someone else in the group
had written some code.
And these, these things would run
all night.
We'd come in in the morning and
we'd check them.
And what had happened was they had
a bug in their code, which they
knew they were trying to diagnose
it.
So they were printing out a bunch
of log stuff or something like
that.
Well, they did that in a tight
loop on a high-end computer for a
night.
And it filled up the hard drive
till it had zero bytes left.
And apparently the Silicon
graphics machine couldn't operate
anymore was literally zero bytes.
And so it just stopped working.
They couldn't get it eternal.
I was like, it took days to get it
back, I believe.
Oh, oops.
But it's like that kind of stuff,
right?
I mean, this, you're not going to
break EC2, but, you know, you don't
know until the next day that, oh, look,
you filled up the computer and it
doesn't work anymore, right?
When you're doing that much
computing, you could run out of
different resources.
You could run into all kinds of
problems.
Absolutely.
And we are without our worst stories
of doing this, but I think we've
definitely learned a lot of lessons
along the way of how to monitor your
jobs effectively and double check
things.
But sometimes, yeah, you run a big
job and it doesn't quite turn out
right.
But it's the cost of doing business,
I guess.
Yeah.
I mean, it's very computational to
explore stuff like this kind of stuff.
But that's also what enables it,
right?
Without this computing power, it would
just not be a thing.
Absolutely.
A lot of the data just takes a lot of
time to process and there's just not
really any way around it.
And even when you're iterating, you
have to go through all the hoops to
look at the data at the end.
Yeah.
So you talked about APIs.
You talked about data store.
What are you using for a database?
Is this like a hosted RDS AWS thing or
what is the story with that?
Yeah.
So we have a few different places to
store data.
our larger scale internal data, we
store in Django-based web app and we
use the Django ORM for SQL-based
database, so MySQL database on AWS.
And that has worked surprisingly
effectively, actually.
I've heard some people say that the
Django ORM is really slow when you
scale out and things.
But if you design it correctly, I
think it'll surprise you.
I think that's so true.
I hear so many things about ORMs are
slow or this thing is slow in this way.
And wow, if you have the queries
structured well, you do the joins ahead
of time.
If you have indexes and you've put the
work into finding all these things, it's
mind-blowing.
When I go to sites, I won't call any
out.
I don't know if they've been updated or whatever.
But you go to a site and you're like, this site is taking four or five seconds to load.
What could it possibly be doing?
I mean, I know it has some data, but it doesn't have unimaginable amounts of data.
Right?
Surely somebody could just put an index in here or a worst case, a cache in something.
And it would just transform it.
Right?
So yeah, I'm glad to hear you're having good experiences.
Yeah, we definitely fairly regularly run into slow queries.
They're usually not too bad to solve.
I'm sure at some point we'll get to something really wonky that will be challenging.
But I think for the most part, yeah, we've been able to solve it through better query design
and better indexing.
Yeah.
Do you ever do things where you opt out of the sort of class-based query syntax and go
straight to SQL queries here and there to make sure that that part works better?
We have tried it for some particular sequence-based searches that we do.
And I actually found that most of the time you can write it in the ORM.
It's just a little more complicated.
But I do expect that at some point we will be writing raw SQL queries because out of necessity.
Yeah, but it's not the majority.
Mainly you're using the ORM and then it's okay.
Yeah.
And I think a lot of benefits from the appropriate data model will help the queries along the way.
Yeah, absolutely.
The thing I found the slowest about ORMs and ODM, if you're doing document databases,
is the deserialization, actually.
It's not the query time, but it's like, I got to generate 100,000 Python objects
and that just happens to be not that fast relative to other things.
You're talking about my Monday morning pipeline.
And in those cases, I think that's the place where it makes sense to maybe do some kind of like a projection or something.
I don't know how to do it in Django ORM, but in say Mongo Engine, you can say,
I know what I'm going to get back is an iterable set of these objects that match to the data,
but only actually add, just fill out these two fields.
Most of the data, just throw it away.
Don't try to parse it and convert it.
Just like, I just want these two fields.
And that usually makes it dramatically faster.
Yeah, we've run into a number of bottlenecks at the serialization layer.
And we have been experimenting with a variety of different ways to solve those issues.
And sometimes it means putting fewer Python objects between you and the data.
And that often speeds it up, even if it makes it a little bit harder to interpret in your development environment.
Yeah, absolutely.
Or just say, you know what?
I just need dictionaries this time.
I know it's going to be less fun, but that's what it takes.
That was the fix on Monday morning.
We do try to extensively use data classes for a lot of our interoperability.
When data comes in and out of a pipeline, we like to have it in a data class, in a centrally stored repository.
And then our Django web app also has access to that.
So it knows what the structure of the data coming in is, and it knows what to serialize it to when it's coming out.
And the Python data classes has been a really useful tool for that.
But I think we were talking about that on another podcast a few weeks ago.
Maybe it was the Python bytes one.
The data classes can be slow.
And sometimes it's better to just have a dictionary, even if it is a very highly structured dictionary.
Yeah, and the problem is maintainability and whatnot.
But, you know, if it's five times slower, you know what?
This time it matters.
So we're going to just bite the bullet and have to deal with it.
Let's see.
We had time for just a little bit more digging into it.
So you talked about the Django ORM, Django REST framework, which is all great.
What's the server deployment story?
Like, how do you run that thing?
Is it with G-Unicorn?
Is it Micro Whiskey?
You know, what's your setup on that side of things?
Ours is a little custom.
I guess in some ways it's pretty standard.
I think we're blanking on exactly how it's set up now.
But there's an NGINX proxy I am blanking on.
It might be G-Unicorn.
I feel like G-Unicorn and Django go together frequently.
I'm not sure why they got paired up specifically.
But yeah, it's a good one.
And we end up deploying it out to AWS Elastic Beanstalk, which is a source of some conversation in our team.
Because there's some things we really like about it.
And there's some things that are really annoying in terms of the deployment is much more complicated than we would like it to be.
But we have everything wrapped up in a pretty gnarly CDK stack that does a lot of this work.
It was messy, but you've solved it with a CDK.
Now it's just you push the button and it's okay.
It's exactly like that.
We have a very automated deployment process.
I wouldn't like to refactor it, but it's there and it works.
So that works for us.
But I think, yeah, it's a pretty standard Django deployment on the cloud.
And it works well.
Yeah, cool.
All right.
Well, I think that's probably about it for time to talk about the specifics of what you all are doing.
But there's the last two questions, as always.
So let's start with the editor.
If you're going to write some Python code, what editor do you use?
The yes code.
A lot of the remote development environment on that.
Oh, yeah.
You use the remote aspect of it?
Yeah.
We're doing a lot of work on EC2 instances as our day-to-day work.
And the yes code, the way it works with instances in the cloud is really amazing.
So I would encourage anyone to check out that extension.
Yeah, you get access to the file system on the remote machine.
And basically, it's just your view into that server.
But it's more or less standard VS Code, right?
But when you hit run, it just runs up there.
It feels exactly like you're on your own computer.
Sometimes I actually get confused whether I'm on or not.
Oh, so does it work?
Because I'm in Virginia.
I see.
Yeah.
All right.
And then notable PyPI package?
I'll have to shout out some of the ones that we talked about.
I would encourage people to look at AWS CDK.
If they're on AWS, I think it has some really interesting things there.
And then also, Luigi as a workflow manager, if people are doing any of these types of data pipelines
that have tasks that get reused, these sort of workflow managers are really cool.
And Luigi is a pretty accessible one, probably one that's familiar with Python.
Yeah.
All right.
Fantastic.
Yeah.
I'm just learning to embrace these workflow managers, but they do seem really amazing.
All right.
So for all the biologists and scientists out there listening, thinking you've got this really
cool setup and all this cool computational infrastructure, what do you tell them?
How do they get started?
Maybe biology or wherever?
I think biology is a good place to start.
We're also happy to have people come from a software background that are really interested
in learning the biology.
And I guess as a final plug, we do have a few open positions.
So if you're interested, check out our careers page and give us an application.
Are you guys a remote place?
Remote friendly?
Or what's the story these days?
Yep.
We're remote friendly.
I'm actually living in Philadelphia and our company is based in Cambridge.
So we are.
I mean, it's a short trip to the cloud, no matter where you come from, basically.
Absolutely.
Right on.
David, thank you for being here and giving us this look inside all the gene editing Python
stuff you're doing.
Thank you, Michael.
It was a pleasure.
Yeah, you bet.
Bye.
This has been another episode of Talk Python to Me.
Our guest on this episode was David Bourne, and it's been brought to you by Shortcut, us
over at Talk Python Training, and the transcripts were brought to you by Assembly AI.
Choose Shortcut, formerly Clubhouse.io, for tracking all of your project's work, because you shouldn't
have to project manage your project management.
Visit talkpython.fm/shortcut.
Do you need a great automatic speech-to-text API?
Get human-level accuracy in just a few lines of code.
Visit talkpython.fm/assembly AI.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at Talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show.
Open your favorite podcast app and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct
RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I really appreciate it.