Do your applications call a lot of APIs?
Maybe you have a bunch of microservices driving your app.
You probably don't have the crazy combinatorial explosion
that Zapier does for connecting APIs.
They have millions of users automating things
with thousands of APIs.
It's pretty crazy.
And they're doing it all with Python.
Join me and Brian Helmig, the CTO and co-founder of Zapier,
as we discuss how they pull this off with Python.
This is Talk Python to Me, episode 199,
recorded January 21st, 2019.
Welcome to Talk Python to Me, a weekly podcast on Python,
the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy.
Follow me on Twitter, where I'm @mkennedy.
Keep up with the show and listen to past episodes
at talkpython.fm.
And follow the show on Twitter via at Talk Python.
This episode is brought to you by Linode and Ting.
Check out what they're offering during their segments.
It really helps support the show.
Brian, welcome to Talk Python.
Thank you, thank you.
I'm excited to be here.
I'm excited to have you here.
It's so cool to have a look inside a company like Zapier
and all the interesting ways you're using Python,
and especially some of the scalability
and sort of interoperability stuff you got going on
is like next level stuff you got there.
Yeah, yeah.
It's some really cool stuff.
It's really fun.
Yeah, awesome.
Now, before we get into all that, though,
let's start with your story.
How did you get into programming in Python?
I got into programming mostly, I want to say,
early university years, right?
So kind of getting into that,
hmm, I want to start some startups.
I want to kind of create some businesses online.
And I was big into music,
so I actually did like online guitar lessons for a while
and needed to build a site around that.
And it just kind of all expanded from that.
But I never trained it like formally.
I just literally like got addicted to doing it
like day in and day out.
And then all throughout college,
like that's basically like all you could find me doing
was like building things
and creating all these little side projects
and little businesses that didn't really work.
Where's Brian?
He's in the computer lab, man.
He's down there working on something.
That's cool.
I really like it.
It's like you learn programming
because you have these ideas.
And you're like, well, programming is,
you know, it's the hammer I need to build my idea, right?
The hammer, the saw, and whatever.
Yeah, exactly.
And then you, I mean, you learn all the more formal stuff,
like after having like practical,
like applications for it.
And for me, that always just worked better for,
I mean, same thing for music.
Like I learned like the licks and all the cool things.
And then I'd learn like the theory of why it works.
And that was always much more satisfying to me,
like as a person.
So I found it worked well.
Oh, that's great.
So did you start out with Python
or did they have you doing C++?
Or where did you start when you're in college?
With those sites like the WordPresses
and the Joomla's and the Drupal's and stuff,
it was a little bit of PHP.
But once I found Python and particularly Django
back in the very early days of Django,
I was like addicted to that.
And that was really, I mean,
I would say really Django was like the thing
that got me into the Python ecosystem
and got me stuck on it.
And it's funny, Zapier's built on Django
and I've spent a lot of time
in the Django ecosystem since then.
Yeah, that's really cool.
It's just like you found it
and it's still serving you well.
You know, I think that's one of the things
that's super interesting about Python in general.
And I think of Python as a full spectrum language, right?
There's languages that are really good
for getting you started to build something simple,
but then they kind of top out
before you can go like pro with them.
And then there's pro languages like Java.net C++,
where it's like you'll start there.
You start with all the complexity.
And Python's one of those few ones
that you start easy,
but as you need more complexity,
you can start to bring in those features of the language,
but they're not forced on you from the beginning.
Right.
And the industry support for Python's like outrageous.
Like a lot of these like more niche sort of things,
which are kind of fun and cool,
or like they just don't have as much like adoption
throughout the industry.
And when you look at like, you know,
employment opportunities and ease of starting
and what the ecosystem looks like community,
like Python's really hard to beat in both a practical
and I also think it's a really well-designed,
nicely done language as well.
So you kind of like, it's firing on all cylinders.
It's really quite incredible.
Yeah, absolutely.
I totally agree with you.
All right.
So let's talk about just really quickly your role day to day,
you know, just so folks know where you're coming from, right?
So right now you're the CTO of Zapier, right?
Correct.
Yeah, yeah.
I'm CTO.
I was one of three founders and we've grown,
man, it's Zapier is a little over seven years old now.
So we've grown from just us three,
Wade, Mike and myself to 200 people.
So obviously the things I'm doing day to day
are a lot different than they used to be.
It used to be a lot of hacking
and then it quickly turned into a lot of hiring.
And that's really been where a lot of my time has been.
You know, some of the cool things are like,
once you get to certain scales,
you can start to really hire amazing,
amazing people to take over different slices
who are way better than you.
And we recently did something similar.
We just hired a VP of engineering
who I'm hoping will be able to kind of take us
to that next level for organizational and hiring
and all this stuff that's so critically important
to like building a team.
And maybe I can spend a little more time
on the code side and the product and architecture side.
So I'm really excited in 2019
to spend a bit more time on that.
That's awesome.
A little bit back to your roots.
Yeah, yeah.
No, it'll be really,
it's going to be really fun.
I'm really excited.
Yeah, I think in the early days,
if it's the three of you starting a company,
you all have to do so many things that you just,
you are unprepared and unqualified to do, right?
Like, you're like, all right,
who's going to do accounting and taxes?
You're like, wait a minute.
Nobody knows about this.
Who's going to do, you know, deployment?
Like, well, we've never deployed it.
Like, you know, there's just so many things,
but now you can find somebody who's like,
say, super good at DevOps
and you don't have to worry about that.
And that's a big transition, right?
Yeah, it is.
And you kind of, you know,
whenever you're just starting out,
you can kind of get away with the mess, right?
Like if you mess up a deploy,
it's like, well, only 10 people were using your site.
So no one noticed, you know,
you have little things like that,
that really help.
And same thing.
I mean, you mentioned like accountant hiring
and like, you're not doing so much of it.
So if you're not world-class at it,
you know, you can kind of get away with it.
But at the heart of it,
like you got to build that product
that really connects to people.
And if you do that well in the beginning,
the rest kind of comes
and you can get amazing people
who know all the,
know how to build a great deployment system,
know how to run taxes in 50 different states,
know how to do all this other stuff.
So yeah, it changes.
Yeah, yeah, I'm sure.
All right, cool.
So we talked a little bit about it.
You work at Zapier and stuff,
but you know, I'm sure most folks listening
have more or less heard of it,
but maybe they don't really know.
They're like, oh, I think that's
some kind of online startup thing.
Tell us what is Zapier.
Yeah, so you can think of Zapier
as sort of the glue between
all the different SaaS services out there.
So if you use Gmail,
if you use, especially like Google Sheets,
if you use things like Dropbox or Slack,
like all these different services
that people use on a day-to-day basis,
they have data in them
and you want to trigger on that data
and do other things and other services.
Zapier is that like mechanism that does that,
which is very much, you know,
I think of it a little bit like,
you remember all the,
and I guess there still exists,
some of the languages like Scratch
who are kind of those visual programming
and they never really caught on,
but they kind of were like,
oh, we're going to give the power of automation
and the power of coding to the masses, right?
Yeah.
It was always, it's an awesome goal,
but it was always really difficult.
And I think we're kind of doing that
and kind of coming in through the back door.
So you can do these cool things
where when you, you know,
star a message in Slack,
you can add it to your to-do app
and you can choose one of, you know,
a dozen to-do apps that are supported on Zapier.
And then you can think,
oh, well, if I say like super important
in that message,
I had an extra filter
and I send a message to this room in Slack
or that room in Slack,
maybe, or I send, you know,
my colleague a DM,
like you can start to build these like workflows.
And that's really the power of Zapier
is that you can kind of start
to string these actions together
and do some really complex stuff
that normally couldn't do
unless you had a fleet of engineers
to like custom build this bespoke software.
This is interesting.
So, you know, you talked about Scratch
and some of these other like edgy block
type of application.
Right.
I'm not going to call it application,
but programming frameworks
that are these visual things.
Right.
And I think the idea of those is not terrible.
I think the problem is
they work at too low of level.
Right.
They're like,
you issue a print statement,
you do a for loop.
You, I mean,
like they literally work almost like line by line
as you would with text.
Whereas what you guys have is like
an email came into Gmail
and that's like an integration with an API
and OAuth and like tons of other stuff.
And then, you know,
you've condensed it down to the right size
of the draggy droppy thing.
You know what I mean?
So it's like,
there's a few blocks,
not 50,000 blocks,
which makes no sense.
Yeah.
And you get away from the idea of like
trying to teach people control structures.
Right.
And how data structures work
and like all this stuff is like
really heady stuff.
Like once you get it,
it's fine and it doesn't bother you.
But like until you get it,
it's another world, right?
It's completely foreign.
And we avoid that.
And we try to keep it just simple
to like,
oh, I get a new email
and I add a message to Slack.
Like, got it.
Like I can connect those two together.
And that's always like the beginning point.
And then I think
the really powerful stuff comes
where you have a team in there
and they're like building zaps together
or they're adding their own
like little flavors of,
oh, I want an extra,
you know, filter here.
Only do it if this matches
or add an extra step to this
and don't just send it to Slack,
you know,
send it to my ToDo app
or send it to this,
send it to my CRM.
Like we have lots of business folks
and marketers who are doing
all this like cool stuff as well.
So this is,
this is not for like,
only for engineers.
We do have some,
some of my favorite stuff to do
is like you can literally embed Python
into these workflows,
which of course like
is very topical for this podcast.
I love it.
But this is for like marketers as well.
So they can build these things.
This is for real estate agents
who have,
you know,
form software
and they want to like trigger off
of people submitting that
and get e-signatures
and like all of this stuff
is like in this ecosystem.
So it's not just engineers
or devs that use this.
It's kind of regular folks,
right?
People who are just trying
to get work done, right?
It's cool.
And you can bring,
bring the level down a little bit.
You don't have to be,
you know,
if you're kind of good with computers,
but you're not a developer,
right?
Like this could be something
you could maybe
really make what,
maybe like a developer would go,
I'm just going to write an app
that'll just call this API
than this and write.
But like,
there's so many people who,
even if they could write that,
they're not going to say,
deploy it as some kind
of background process
in like on Linux
and put it in the cloud.
Like they're just not going to do that.
Yeah.
So yeah,
this is great.
Yeah.
And you think about,
that's a great example.
It's like we often get engineers
or devs who are like,
I really don't want to build
that thing you just asked me to build.
I'm just going to use Zapier
because if you think of like,
oh, I got to pull this service
and now I have to come up
with this stateful store
of which things I've seen
and haven't seen,
I got to deploy it to like,
I don't know,
maybe we'll put it on a Heroku.
Maybe we'll like,
being a DevOps person,
we'll get it still,
you know,
like it just starts to spider
into this whole thing.
Whereas like with Zap,
you just like create a trigger,
create this,
and then you can hand it off
to the marketer.
You can hand it off
to your colleague
and they can just dig into it.
So that's,
that's a powerful pattern.
Exactly.
It's also like,
I think this is a good idea
to build this little integration,
but I don't want
another puppy
that I have to take
for a walk and maintain.
I didn't want to,
I don't want a thing
to care for.
So if I build it myself,
it's my problem.
If I build it here
and hand it off,
like you guys,
that could be your problem.
You'll keep it running
if the API changes.
Yeah, we do.
And there's a lot of things
that change in the API world
that we stay on top of.
So it just kind of,
it just takes that whole layer
of like what's going on away
because we have,
and I'm sure we'll get into,
we have tons of tooling
that like watches this.
Yeah, yeah, yeah.
It's really cool.
I definitely want to talk
about that because,
you know,
you have thousands
of integrations
and these integrations
integrate together,
which is even more
like, you know,
combinatorially insane.
So that's going to be fun.
And I want to talk about that,
but let's just talk briefly
about this at a high level
while we're talking
about Zapier.
You know,
a lot of folks,
they get up
and they get in the car
and they drive
through angry drivers
and traffic for 30 minutes.
Right.
And they all have to live
around their company
and they go to work, right?
But as developers
and tech companies
and things like that,
right?
Like a lot of times,
even if you're sitting
next to somebody,
you're working
in asynchronous modes,
you're making a GitHub issue,
you're doing a PR,
you're throwing a message
in Slack,
you're like,
there's no reason
you've got to sit
next to the person, right?
So that's kind of what
you guys decided as well.
And with your 200 people,
you're more or less
100% remote, right?
That's correct.
Yeah, 100% remote.
We don't have a single office.
We're not geographically bound.
We have people in,
I'm going to say,
I think it's like 18
or 19 different countries
right now that work
for Zapier.
And we've been that
since the very beginning.
So we started 100% distributed.
We've been distributed
throughout the entire time.
So every single person,
it's funny,
like basically everyone
we've hired,
we had not met in person, right?
There's like a few exceptions
to that rule,
but like everyone we've hired
has been just like a Zoom call
or a Skype call
to kind of do the interview
or interacting over email
or interacting over,
you know,
text communications in some way,
which really mimics the way
that most of us work anyways,
right?
That's been just kind of
a core part of Zapier
since day one.
I think that's really,
it's pretty exciting, right?
I mean,
there's certainly drawbacks
to working from home, right?
You know,
people can get a little bit lonely.
You have to be a little more
self-directed, I think,
because some people see
working from home
means I have no boss.
No boss means I can just do
whatever the heck I want.
Like if I want to just watch football
for the second half of the day,
I'm doing that.
Like those types of people
don't thrive,
but I feel like a lot of developers
and technical folks
are pretty driven anyway.
Yeah.
And if they have a place
for a proper outlet
that's really,
they feel like they can contribute to,
like it's pretty easy
to work from home,
honestly,
as a developer.
Yeah, I think so too.
And, you know,
everything you're,
everything,
all the artifacts you're creating
in a tech company,
like you're writing code,
you're fixing bugs,
you're working on features.
It's all,
like the output's like all there.
You can see it either shows up
in the repo or it doesn't.
It either lands on the site
or it doesn't.
And so you can,
it's not that complicated
to like track the work.
I often get folks who are,
tend to be a little bit more,
especially in the early days of Zap,
you're very skeptical
about the idea of like remote.
Like how do you know people
are doing their work?
And how do you,
like that has not played out
to be the biggest problem for us.
Many of the problems around,
and you mentioned like
these are trade-offs,
like a lot of the challenges
you get into
are just challenges
that you hit in a normal company.
You just hit them sooner, right?
Like that communication challenge
doesn't happen,
you know,
when you're in the hundreds
of like employees.
It happens when you're
in the dozens of employees, right?
Where everybody's off doing
all this like cool stuff
and it starts to get hard
to coordinate, right?
So we had to invest
in coordination
and communication
a lot earlier in the org
because you couldn't do that
implicitly by picking it up
around the water cooler,
so to say, right?
So that's always been
a big focus for us
in the early days
was like what's breaking now
that probably wouldn't break
until, you know,
we were double
or triple the size
in a normal company.
So what are some of the tools
and things you put in place?
You know, to me,
it feels like
distributed teams 2018
is 10 times easier
than distributed teams
2006, 2008,
but still like
there's a lot of tools
to pick from.
Like what's important part
of that whole structure there?
I mean, they're the same tools
I think a lot of companies
are using.
I mean, they're the GitHubs,
right?
They're the Slacks.
We use Zoom
for like calls.
We do so many video calls,
constantly hopping on calls
to like chat over stuff.
We've even built
a couple like internal tools
that are custom to us.
So we've built
an internal tool called Async
modeled around the idea
of asynchronous communication,
right?
Where, you know,
you look at something like Slack,
it's very synchronous, right?
Like you drop information in,
it scrolls away and it's gone.
It's not gone,
but it's like basically gone.
It feels ephemeral, right?
It does definitely feel ephemeral, right?
It's hard to keep permanent actions
that you need to do
like somewhere in your Slack history.
Yeah, that's a poor,
it's a poor idea.
You want to push that out.
So what we've done
is created this tool called Async
and you can kind of think of it
like WordPress meets Reddit
meets Twitter kind of a thing.
So it's basically a blog mechanism
that has some structured ways
to describe, for example,
we require each post
have like a TLDR, right?
Like where you can,
under a hundred words
or less, give or take,
kind of like summarize
what your decision is
that's enclosed in that Async post.
Man, I love it.
That's such a good idea
because then you can tell
do you need,
how much attention
you need to pay to it
and like just an email,
like that would be a great
inter-office communication way
because I mean,
I personally get so much email
that I'll periodically
have to take,
you know,
eight hour days
and just answer email
just so it doesn't get too bad.
And it's like,
those were those little short vets?
Like that would help sometimes.
That's awesome.
I like it.
Yeah, it really would.
And we've also added things
like depending on
how many people view it
or comment or like it
or interact with it,
it gets hotter.
Like it's a very like Reddit style
or Hacker News style thing.
Yeah, yeah.
Which helps like surface
just important information.
So earlier this year,
Wade posted a whole bunch
of like essays
on like what we're trying
to do in 2019
and everybody was interacting
and like diving in
and chatting about them
and they'd popped
right to the top.
So you don't even have to like
really work super hard
to get that information.
You just kind of hit
the homepage of Async
and you get a flavor
of what's going on
in the company.
You can read those TLDRs
and you get a good idea.
You can click in,
you can go deeper if you want,
you can interact.
So this is something,
as an example of something
we built internally for this,
but it's not crazy software, right?
It's not super complicated.
This is something that
in conjunction with other tools
like the classic,
like, yeah,
like we use lots of Google apps
and then the GitHub
and the Elysians
and the Jira's
and all that stuff.
So we do do some custom stuff.
Yeah, that sounds really awesome
and I think it's great.
You know, I've worked for 10 years
for a company that was remote
and we had people
all over the world
and there are definitely trade-offs,
but I really felt like
the people I was working with
were some of the very best people
because they were chosen
from the population of the world,
not from the population
of the town I happen to be in,
which maybe is a small town
or whatever, right?
Like, how has that either been
like a positive or negative for you?
I've seen like a lot of tension
between some companies
are like, oh,
we have to have somebody local
because it's super important
we have this like sort of control
or like view over them
and interaction with them,
but I don't know
that that's always the best.
No, I don't think so.
We believe,
and we've seen it in reality,
we believe that like talent
is like evenly distributed
throughout the world,
but opportunity isn't, right?
So in a large way,
we see this as an incredible opportunity
that even just like talk about America,
a small town in America, right?
And you have an amazing engineer
that lives there.
They live there because they have family
and it's a great cost of life
and they love where they're at.
The opportunities as an engineer
are likely pretty slim.
It's one of those opportunities
where those amazing folks,
those really talented folks
can actually contribute
at the level
that they can contribute at.
That's pretty special.
Like that's really, really hard to do
when everybody's geographically located
and you require someone
sit there in an office
that just wouldn't really like scale out.
So for us,
we think of it as
something that is good for business,
but also just really good
for people as well.
It's just a better way to work.
It's a better way to live.
You can live where you want.
You can have a better,
like just look at like the cost of living
for your family,
for everything.
It's just so much better
in every way.
And you just have to like work
through the little challenges
that pop up
in a remote environment
and it really helps
to be 100% distributed.
I think when you get that mix thing,
yeah, it gets a little trickier.
The mix is tricky
because you got the insiders
who are at the office
and they don't necessarily
fully trust the remote workers,
but maybe the remote workers
are really talented
so they have to work with them.
But there's always
these little undercurrents
of weirdness.
Like I've been in those situations
as well.
This portion of Talk Python to me
is brought to you by Linode.
Are you looking for hosting
that's fast, simple,
and incredibly affordable?
Well, look past that bookstore
and check out Linode
at talkpython.fm
slash Linode.
That's L-I-N-O-D-E.
Plans start at just $5 a month
for a dedicated server
with a gig of RAM.
They have 10 data centers
across the globe,
so no matter where you are
or where your users are,
there's a data center for you.
Whether you want to run
a Python web app,
host a private Git server,
or just a file server,
you'll get native SSDs
on all the machines,
a newly upgraded
200 gigabit network,
24-7 friendly support
even on holidays,
and a seven-day
money-back guarantee.
Need a little help
with your infrastructure?
They even offer
professional services
to help you with
architecture, migrations,
and more.
Do you want a dedicated server
for free for the next
four months?
Just visit
talkpython.fm
slash Linode.
You were just talking about
having a company retreat
in New Orleans,
and the company I worked for
before when I was
working remotely
before I started my own.
Certainly, we had the same
challenges of how do you connect,
and so having some kind
of yearly or bi-yearly
get-together physically,
and also you talked
about Zoom.
The video aspect
of the Zoom and Skype
actually makes a big difference,
I think.
It really does,
and the internet connections
are getting so good now
that that video connection
can feel very real,
can feel very,
like you can interact,
you can see someone's body language,
so you do get that.
Yeah, you and I are talking,
and it's almost as if we're
sitting across the table
from each other.
It basically is good.
Yeah, it works really well.
So we found it to be
exceptionally good.
Yes, I do think having
the visual interaction
is important,
like video calls,
but then periodically
these actual get-togethers,
right?
Yes, the retreats.
Yeah, like these retreats,
I always look forward to them
because it was like
these people I really liked
and I would have more of a work.
Like you wouldn't hang out
on Skype.
You'd be like,
hey, I'm just,
what are you up to, man?
Let's hang out on Zoom, right?
Like that's kind of weird,
but you would have these meetings.
So this is your chance
to be more friendly
with like your coworkers, right?
So I think it's a good combo.
I think it works really well.
Yeah, I do too.
And when it comes to the retreats,
we've done those
since the early days.
I remember our first retreat
was seven people
and it's grown up to 185
or 190 people.
I think we're at the last retreat,
which brings about a whole set of,
you know,
unique logistics challenge,
but we found them
to be really, really useful
to kind of help set the stage
for the stuff
that we want to talk about.
So you get that,
you know,
one of the cool things is,
at least for me,
is when you're in an office,
you kind of are pretty much
like flatline energy level.
Like it's just kind of
day in, day out.
It's what you do every day.
When you meet the team,
you get them together.
For us, it's like twice a year.
That energy level's
through the roof.
Everyone's really excited
to see each other.
You're diving into things.
You're hacking on stuff.
You're going,
like there's some really cool stuff
that comes out as a result.
So you get a little bit
of the best of both worlds
where you get to go back.
You get to be in your space.
You get to like zone in.
You get to like have
your kind of flow state,
but you also get to have
that really high energy
connecting with people,
spending time with them,
which works really well.
And we try to emulate,
some of that serendipity things.
Like we've used,
there's this product
called Donut for Slack
and it will like pair you up
with folks.
So we have these like two by,
two people like pairings
that happen every week,
three people pairings
that happen every week.
And they just let you get together
and just chat about whatever.
And you get matched up
with random folks on the team.
So you get a little bit
of that as well.
Oh, nice.
And throw in little jokes
or just stuff
just for more personal things.
Yeah, yeah,
you definitely do.
And you get to hear
a little bit about
what people are up to
and like, you know,
hear about their family
and hear about this
or hear about that.
And it's those sorts
of unstructured conversations
that kind of bring out
that humanness,
which, you know,
you do lose a little bit of
when you're remote.
So we often have like advice
for folks to try to plan
one day of your week
where you go to a coffee shop
or you go somewhere
just to get out
and move around.
And, you know,
some people don't need that.
Some people need more of that.
And just be aware
of that for yourself
because you're not often
going to get that directly
from working on Slack
or something like that all day.
I think that's great advice.
Like people can go
kind of stir crazy.
Like I work 100%
from my home office
or somewhere else, right?
Everyone I work with
is some kind of contractor
or author or something
who's also not
in any form of office with me.
I know all the coffee shops around,
especially in the summer,
you know,
and sometimes hit the libraries,
but just getting out
that makes a big difference.
So I guess one more question
that people out there
might find interesting
and maybe they will more so
after what we talk about
in just a moment,
but it seems like
it fits better here
is they're out there listening.
They're like,
you know,
I have this Java thing I'm doing
and I'm not really super happy.
I love a Python job.
You guys are pretty much
doing tons of Python
and they could work
from where they are.
Are you hiring?
Yeah, we're doing
a lot of hiring in 2019.
We've been really fortunate
that, you know,
we've talked about this publicly.
Zapier has been profitable
for years and years
and we've been growing
at a really respectable pace
and we have a lot
of ambitions for 2019.
A lot of cool stuff
we got to build.
So yeah,
we're 100% hiring.
We're looking for folks
who have Python background,
have JavaScript background.
We're going to be hiring
slightly different,
you know,
on the manager track,
we're going to be hiring folks
on that side as well,
which we kind of think of
as like a different skill set.
We're also hiring
like security folks.
We're hiring like
test automation,
you name it.
We're hiring all the things
in 2019.
Lots of cool stuff.
All right.
Awesome.
Well,
people can,
I'm sure there's a career page
at zapier.com,
right?
Slash jobs.
Yeah.
Zapier.com slash jobs
and we got them all on there.
All right.
Awesome.
Nice.
All right.
So let's dig into
some of the technology
because I think there's
two sides of things
that are super interesting here.
One,
you've got
your infrastructure
and just the amount
of people running jobs
and triggered actions
on your site
is really,
really interesting.
There's so much
happening there.
And then also
you're calling
thousands of APIs
and these APIs
are not totally stable
and all sorts of stuff.
So let's dig into that
a little bit.
Like maybe just tell us
real quick,
like what kind of tech
is involved
and things along that line.
We've tried to keep it
pretty simple
because we have plenty
of complexity
when it comes out
on the other side
with talking to APIs
and some of the
interesting combinations
users make.
So for us,
we're Python.
I mentioned earlier,
we're Django based.
So we're still using
a lot of the ORM,
a lot of the kind
of standard tooling
that comes with Django,
a lot of the caching bits,
a lot of the session bits,
all that stuff
we still use
really, really heavily.
What kind of database
back in are you guys
talking to with that ORM?
Yeah, yeah.
We're talking to MySQL
right now.
We're starting to kind
of shard off
and we've done
and not sharding
in the sense of like
sharding over
consistent sharding
over IDs
or anything like that,
but like functional sharding
where we try to extract
this service
from kind of the monolith
and say,
okay, well,
now this is an opportunity.
Maybe we'll use
a more specialized database.
Either maybe we'll
stay relational
and we'll use Postgres
or upgrade
the MySQL
to something
a little different
or maybe we'll use
some sort of NoSQL
or specialized
read and write patterns
that make that
use case even better.
So for that,
we're generally,
I mean,
we're looking at like
mostly like
vanilla Python
like on the back end,
the workflow engine side,
right?
which is probably,
you think of like
that big behemoth
that's running
in the background
doing all this
like crazy stuff,
talking to all these APIs,
kind of your little robots
running around the internet
and that's all like Python
and that's usually doing
a lot of,
I mean,
you think of like celery,
we use a ton of celery,
right?
We use a ton of that
and we use RabbitMQ
for a lot of the message
passing between that.
We're still using Django
on the HTTP front,
I mentioned.
we use a lot of
just pretty like vanilla
like ingest for like webhooks.
Like I know we do a lot of pulling,
which is a lot of back end stuff,
but we also pull in a lot of webhooks.
So this is great
when services send us those webhooks
and let us know about things changing.
Right.
Webhooks have to be basically
the best possible outcome,
right?
Because something like an email
or an action happens
and that service tells you
rather than just
you got anything for me,
you got anything for me.
Yeah, yeah, yeah.
That's,
and that's,
it's huge.
We did a big analysis
and we actually even shared
and try to get more partners
on board with like webhooks
and what we call rest hooks,
which is just a way
to subscribe to webhooks.
Because I think for a while,
especially in the early days
of webhooks,
it was like,
hey user,
please copy this URL
into this other service.
And then,
whereas like now that we have
rest APIs,
like we can just post
to a subscription
or a webhook endpoint,
it'll create a webhook
and send you those events
and then you can tear it down.
So we've done a lot there
and we've done,
and we did a bunch of analysis
in the early days
and it's something like
66 times more efficient,
like,
because like on average,
you're asking hundreds of times
for new data
and there's nothing there.
And then the one time
there is,
right?
Yeah.
Just super,
super wasteful.
So we try to push towards
like webhooks
and event-driven sorts of things
as much as we can.
Yes.
So you go to these partners,
you're like,
we would like to not call you
100,000 times a day.
Yes.
Could we make it
so we just call you
100 or 50 or 1,000
or whatever the right number is?
Yeah.
Yep,
exactly.
And some partners
are really like into that
and they have like some services
and some endpoints set up
to kind of create those
webhooks and subscriptions
and some don't.
And in those cases,
we do fall back
kind of that polling architecture.
But yeah,
we really like it
when partners do add those.
I'm sure.
So I feel like
this asynchronous,
cueing,
message passing,
message bust type of world
must be pretty important
for you guys.
Like,
you know,
a new trigger has come in
from like a new Gmail,
but then you want to talk
to Dropbox.
Like,
you don't necessarily
want to block that up.
You're like,
okay,
well,
kick off a message.
Like,
you need to take this thing
from Gmail
and move it over to Dropbox
when you have a moment.
All right.
And just sort of pass it
down the flow.
You want to talk a little bit
about when that happens
and like how that helps?
Yeah.
I mean,
if you go into Zapier
and you go into
build a Zap,
like a workflow,
you'll see multiple steps.
Maybe it's a,
I'll use a starring,
a message in Slack,
and then you add it to,
let's say,
Todoist,
right?
Those are two distinct steps
that we've broken apart
on the back end.
And on the back end,
they're stored in kind of
a self-relational,
sort of a tree.
You can kind of think of it
as a tree.
We try not to,
there's no real ways
to do cyclical,
right?
Sorts of like things
because you get into,
now you're into
control flow stuff
and that's,
you know,
a little bit crazy.
For most users.
Recursion with drag and drop,
probably not.
Yeah, yeah.
It gets tricky.
So we avoid that.
So you can kind of think of this
as like,
that's a two-step Zap,
Slack,
Todoist,
and we just pull that data
out of Slack,
we pass it on to the next one
and we have a whole like
templating system
that kind of like
passes in that data
into the inputs
of the next step
in a particular way
and users can define that
however they want,
which is kind of this
cool customizability.
So you can think of
that task would say,
you know,
the message
and maybe the time
and who like
actually made that
message in Slack
and you put that all
in the description
field for
Todoist action,
which would be maybe
create a task or something
or a to-do item.
So that would trigger on
and push down
to a totally separate step
that's going through
RabbitMQ in that case.
And then once that,
you call into Todoist
and you're complete with that,
you get a 200,
everything's hunky-dory.
You look for any other
children you have
and in that case,
you have nothing
and you insert
no child task
and you're done.
This app is finished running.
That's pretty much
kind of the workflow
and you can see
how that would expand
and you could have
dozens of steps
and it walks down
through a dozen steps
until it gets to the last one.
Nothing to do,
so it's finished.
Yeah, yeah.
How do you deal with
like the poison message
equivalent, right?
So like the third step
is I want to call
the Todoist API
and I'm a big fan
of Todoist by the way,
but you know,
maybe that day
the Todoist API
is down
for like two hours
and you're like,
well,
do you like
throw it back in the queue?
Do you put it
in some other place
like we're going to get to this
maybe when things are better?
Because I'm sure you're like,
there's probably an API
permanently
at any moment in the day
there's one of your APIs
that is like
not behaving that great,
right?
That you got to interact with.
There's so many.
Yeah, definitely.
And there's a couple ways
that we handle that.
So we do a ton,
since we do so many requests
and we log all these requests
and we track all these requests
just for understanding
the health of APIs.
When we see one
starting to fail,
we see if it's failing
for a lot of users
or if it's just one user.
If it's just one user,
that's an error.
Maybe, you know,
something about the
to-do item
they were adding
is actually invalid, right?
Right.
They deleted the project
they're trying to add it to
or whatever, right?
Yeah, exactly.
It could be anything like that
and that's a user-specific thing.
So that would pop up
as an error.
We'd email them,
hey, heads up,
something funny is going on
with your Zap.
Click here to kind of learn more.
But if it's something
that's happening
across a high cardinality
of users,
a bunch of different users
are hitting this,
now we know,
okay, something looks
a little fishy here.
And at that point,
we'll start to hold things back
and periodically
like kind of let them through
to see,
do we have things going
from, you know,
maybe a 403,
we're getting like
nonstop 403s,
someone shipped a bug
and da-da-da.
It's a mess.
And we'll retry those
until we start to see 200s
and then we'll let
the rest of them go.
We'll try to trickle them out
until we see them clear up.
And that's a really
common pattern for us.
And we have,
I mean,
we've invested a lot
in just trying to understand
when APIs go down
because I joke that we're like
an edge case catching machine
for the internet
and anything that will go wrong,
we will find out about it
in some way.
So we have to have all this
like really intelligent monitoring
to kind of stop that
from happening,
really keep it from impacting users
or at least like
give them the awareness
that,
hey,
heads up,
there's some internet stuff
going on.
you know,
maybe AWS is having
some sort of an outage
and look at all these services
that are 503.
And we can kind of tie that together.
Yeah.
Cross your fingers
and hope that a US East 1A
doesn't go down.
Right.
Yeah.
Like how much stuff runs
on the Virginia data center
of AWS.
So what?
Yeah.
A lot of ours does.
And we can often tell when,
like if we see any funniness
with AWS,
we can always just go look
at our like partner dash
partner status health dashboard
and see if anyone else
is having issues.
And if it's just us,
it's like,
okay,
so we must be doing
something funny.
But if everybody's
like having issues,
we're like,
okay,
so Amazon is having an issue
and they haven't quite updated
their status page yet.
Right.
Exactly.
You're like a little bit
of a canary on that.
Yeah, definitely.
Interesting.
This portion of Talk Python
is brought to you by Ting.
Let me tell you about Ting,
a new mobile service
available in the US
that's targeted developers
and other technically savvy folks.
First of all,
their average customer
only pays $23 a month,
but they're no discount provider.
Their service runs over T-Mobile's
and Sprint's fast nationwide network.
If you don't use that much data
because you're usually on Wi-Fi
like many of you are,
then Ting will save you
a ton of cash.
But don't worry,
you can still use
as much data as you like
for just $10 per gig.
One mobile feature
I use all the time
is tethering.
And with Ting,
you get unlimited tethering
at the same data rate
with your account.
$6 a month for a phone line,
$10 a gig,
$3 a month for text
if you usually chat
over iMessage or WhatsApp.
Think about it,
no contracts
and super clear
and fair billing.
Visit python.ting.com,
that's python.ting.com,
and check out their savings calculator.
Enter your usage
and see exactly what you'd pay.
Use that link
and you'll get a $25 credit
to try them as well.
That's python.ting.com
or just click the link
in the show notes.
I guess it sounds like
you host a lot of your stuff
at AWS
and you probably don't have
like a big in-house data center.
Like that made more sense
if you have a big office
and like part of it
could be the data center.
But if you have no office,
like who's going to offer up
their bedroom
to get fiber
and like cooling
and like,
all right,
that just doesn't make sense.
So are you running AWS?
Where's your services hosted?
We're predominantly AWS.
We have a lot US East,
of course,
but we also have a couple things
that fail over
to multiple different regions.
So yeah,
I mean,
we've just leaned into AWS
since I think the early,
I think maybe we did
like Linode or Linode
back in the early days
for a hot minute
and then that went over
to AWS since then.
We try to use
some of the managed services
where we can.
So you're looking at the RDSs
and the Elastic Caches
and of course,
like the ELBs and ALBs
that exist out there,
tons of S3.
So we try to lean on
some of the managed services
where we can
to just knock down
the amount of like
deep,
intricate knowledge
we have to have
about like Postgres
or MySQL
or whatever, right?
Yeah,
let them do the operational
bits of that.
Interesting.
So to me,
it feels like the cloud,
especially AWS
and Azure
and Google,
those three clouds
feel like that's
the new lock-in,
right?
And I don't necessarily
mean that in a bad way,
right?
Like when you
take full advantage
of a platform,
you get all of the features
of it, right?
Like you could treat
AWS as a fancy Linux
hosting environment,
right?
Or you could use
all the databases
and all the files.
all of the APIs
and really get
a lot more out of it,
but then you're
kind of permanently
stuck to it.
Like how do you guys
think about like
lock-in and cloud
and all that?
We kind of like
tread the line
a little bit in between.
So all of our workloads
that you think of
or a large majority
of our workloads
that you think of
just like running tasks
and performing
these actions
for users,
like they're running
pretty bare bones
like EC2 instances.
We found that
that generally has
the best performance
characteristics.
we have the most
kind of control over it.
So in that case,
like it's not really
like locked in a whole lot.
We're starting to lean
into things that are
like open source
orchestration frameworks,
the classic Kubernetes
like transition
that I think a lot
of companies
are like trying to do now.
Again, we're talking
about pretty open source
standard sort of a thing,
which we really like,
but we're also not afraid
that on an occasion
using some of these
specialized services,
if it makes our lives
a lot better,
we'll use it.
Like RDS is a great example
of kind of them
splitting the middle.
like it's standard MySQL.
So in the future,
if you decide
you want to like swap out,
it's not the end of the world,
but you don't have to worry
about it in the meantime,
right?
You get all the goodies
kind of a little bit,
certainly not for free,
but with less work
on your side
and I can focus
on the product.
So we've definitely
leaned into that a lot.
We haven't done a whole lot
of like hyper specialized stuff.
Maybe the only thing
I think that we do a lot of
is use a ton of AWS Lambda.
In fact,
like back in the early days,
we built our own version
of AWS Lambda.
It was,
I forget what we called it,
but where we could
more securely run
a bunch of
partner provided scripts
and user provided scripts.
That way they could do
this kind of Python
and his app thing,
or they could like
stand up their own
partner apps.
So this allows partners
to kind of deploy code
to us,
to talk to APIs
and integrate with
any of our,
you know,
1400 different apps
that was running
on our own little flavor.
And then as soon as AWS
came along,
it was like,
actually,
we got this
and it's way better.
We immediately
swapped it all over
to that as fast as we could.
So we do use things
like that,
but even that has been
commoditized by,
to a large degree.
I mean,
both Google
and Microsoft's
cloud offerings
have,
you know,
the serverless
sort of vibe
going on too.
So it's kind of interesting.
we've tried to
balance it
on the whole
to where we can
focus more time
on the product,
but not go like
too hardcore,
use really specialized stuff.
Right, right.
It's that kind of stuff
that lets you do
what you're doing
on a 200 person team
instead of a 500
or a thousand person team,
right?
There's not like a fleet
of database engineers
and other infrastructure people
and yeah, right.
It's like,
they're really big benefits
to this.
And I kind of do
the same thing.
Like there's certain things
where like,
okay,
that's really not
available elsewhere
or so much more work.
I'm going to just,
you know,
plug into that API
or this API,
but yeah,
definitely.
Yeah.
So you did talk
about calling
a few APIs
and if you're doing
that with Python,
like the most common
way to do that
probably is the request,
the module by Kenneth,
right?
Right.
Are you guys using
requests for all this?
How's this work?
Yeah.
We use requests,
but we use a bunch
of our own.
I mean,
the great thing about requests
is how like extensible
it is.
Like you can swap in
and out your own
HTTP adapters.
And then from there,
you can swap out
almost everything.
So we have custom
DNS resolvers.
We have custom
request adapters
or HTTP adapters.
We have custom like
response and request types
that all kind of encapsulate
some of the behaviors
and the logging
and the monitoring
that we want to have.
And we have like
different classes.
Give us some concrete
examples.
Like,
so you have a custom
DNS thing.
Is that like,
so you can do better
caching and not like,
like make the calls
faster?
Like what's the idea there?
The idea there is
if you,
for us,
like we want to be able
to have a lot of kind
of control over what
DNS services we have
with which like adapter
we're using.
For example,
we use requests for
internal communications,
which is like a different
DNS sort of provider.
And then when it comes
to like user outbound
user stuff,
we would want to use
either like the Google
8.8.8.8 or maybe
like open DNS or even
maybe some sort of a
custom thing for a user.
So in those cases,
we want to be able
to swap that out.
And that's a fairly
low level.
I mean,
it's kind of automagically
done for you a lot of times.
But if you want to
control that,
you have to kind of
start digging into
the HTTP adapters
and intercept some of
the connections that are
opening and kind of
like all this other stuff.
And you have the
opportunity to do that
with requests.
And, you know,
a really specific example,
we talked a lot about
this monitoring bit.
So we have
different libraries
that you can like,
or sorry,
different like interfaces
that you can basically
attach and say,
hey, this is my
Salesforce session.
And it'll start to do
all the logging
for like this public API
to the proper spots for you.
So you don't have to like
worry so much about like,
oh, am I doing
the right logging here?
Am I doing,
and it'll do all this stuff
where it like kind of
cleans up the secrets
out of it.
And it does the
proper truncating.
So we're not like
dumping like 50 megs
to the logging service.
And it does all this stuff.
The logs are full again.
Yeah.
That you,
you kind of forget that.
Oh yeah, yeah, yeah.
You got to remember
to do that through
a very familiar interface
that you can just pop on
to, you know,
Python requests dot,
you know,
read the docs dot,
you know,
org and read all about
exactly what it's doing.
It's so much easier
to onboard new employees,
new engineers on,
and you can kind of extend
and get kind of the best
of the community,
the best of tools
that exist out there.
So that's kind of way
we've approached using
like things like requests.
And it's been really
quite,
quite useful for us
and very successful.
It's a great library.
I'm wondering if,
you know,
there's some of these
integrations that are,
they basically have
really nice prebuilt
Python packages.
And some of them
are just like,
here's an HTTP endpoint,
knock yourself out
with whatever tech,
right?
So,
you know,
like for example,
GitHub,
there's like GitHub packages
that you can use in Python.
Do you ever use those
or you're like,
it's better for us
to just stay separate
at HTTP access level?
Yeah,
we've gone to the HTTP level.
Those libraries are awesome,
especially for,
if you want to just
spin something up quick,
but we've gotten so good
at talking,
like we talk to REST APIs.
We're good at that.
We've done it a lot,
right?
It's kind of our bread and butter.
So we want to like talk to that
and have kind of the control
over that.
So we used to do
a little bit of that
in the early days.
And then we got to the point
where observability
became tricky.
Some of them used
things like requests
behind the scenes.
Some of them used
HTTP lib too.
Some of them just use
like the built in,
like everyone was doing
different stuff.
And then you had to like rewrite
how your observability
and your monitoring worked.
So for us,
it just became hard to do.
So if you're talking
to lots and lots of APIs
like this,
it really does help
to kind of standardize
your interface on that.
Yeah,
it seems like it must have been
like almost a dependency,
a dependency hell of a sort.
Like if you're like,
we're going to have
500 different packages
and those packages
might go out of favor
and then they're not
quite up to date
or there's like a breaking change.
Like there's just so many layers,
right?
And you're just like,
let's just do the same
for everything.
Yeah.
that's really where we kind of landed.
That was through some like learning,
like especially in the early days,
we did do a couple of those
like kind of like those libraries
that you can just,
you know,
pip install MailChimp,
right?
Yeah,
exactly.
We're good.
Yeah.
Do this.
They did help.
And a lot of them are also,
funny enough,
they're built on little odd behaviors
that seem fine in isolation.
But when you're running,
you know,
something like,
like what we do,
it doesn't really fit.
For example,
some of these libraries,
you'll just import,
like let's say MailChimp.
and then,
you know,
on the global object,
do MailChimp.API key
and just set your API key there,
which you never know
which user's API key
you're going to use.
Yeah.
When you're us,
like that kind of like,
you look at that
and you're like,
oh,
that doesn't,
that doesn't really work.
Like I want to create
like a MailChimp session,
you know what I mean?
So like.
Anytime you're doing multi-tenancy,
like those little gotchas
are super scary.
You're like,
why did I just expose
all of this other customer's data
or something,
right?
Yep,
exactly.
So that's why having control
over that layer
and being really particular
about like when you're opening
and creating new sessions
or you're reusing them or not,
you want that control.
So that's where we've
kind of landed for that
and it works pretty well.
And you know,
when I was talking about
the AWS Lambda side,
a lot of that's
on the JavaScript side.
So we do provide
maybe not the perfect equivalent
to Python requests
on the platform,
what we call our platform
like CLI and SDK,
like that has like
a similarly inspired
sort of a thing.
So use our,
in our case,
it's like the Z object
that has a request
method on it.
If you use that,
then you get all
the magical like logging
and you get all
the magical like introspection
on the calls
that you're doing.
So it supports that.
And if you break out of that,
maybe you have your own
little like JavaScript SDK
for talking to your own service,
you'll lose a little bit of that.
And in that case,
like it'll be harder
for us to debug your services
or help users out.
So we do try to encourage folks
to use that kind of standard thing,
both inside Zapier,
obviously,
but also outside of Zapier.
So when partners are like interacting,
working with us,
we also provide some.
Yeah, interesting.
So you talked earlier
about this asynchronous message
passing and queues
and whatnot and celery.
And that's really important
for scalability.
But another huge,
huge scalability booster
is some sort of
asynchronous web handling,
right?
like G event or async and await,
asyncio,
all those kinds of things.
Does that show up anywhere?
We have experimented a bit with it.
This is like one of those classic,
like you try it out
and then you run the numbers
and see how it like,
how it like,
how much does it save you?
How much headache does it add?
How much more complex is it, right?
Yeah, exactly.
And we tried this maybe,
I'm going to say two years ago
with G event.
It was right on the line,
but probably at that time,
it just wasn't a good enough
move for us to like
transition over to it.
So we actually abandoned
using G event
and we're still using
like pretty like
straightforward,
synchronous like Python,
which is right now
we're still transitioning
to Python 3 at the moment,
making good headway on it.
But a lot of the,
you know,
asyncio you mentioned,
like a lot of that
adds tons of complexity
to your code.
G event does the same thing.
There's all these like
little extra things
you got to remember.
And Python,
I think,
hands you a gun
that's very loaded
when it comes to like
async stuff.
So it is a little bit tricky
and that's not to say
there's,
I mean,
there are other languages
that do a really nice job
with async,
but even like something
like JavaScript,
I think still,
you get a loaded gun,
right?
So it just adds
this layer of complexity
that I don't think
was worth it for us
at the time.
I think there's some cool
things we can probably do
in 2019
to get back to that
and that's through
moving more services
behind like an IO boundary,
just a regular HTTP boundary.
so less thing is happening
like in the process itself
and is sitting across
a very easily
async boundary
where you're talking
over a socket
or something.
I think if we start
to move more workloads
over there
and we have like
kind of this clean interface,
the Geovent equation
will quickly tip
in the other direction.
so I'm excited
to maybe look at that again
in 2019.
But yeah,
we gave it a good,
we gave it the old college try
and it was a bit of a bear
and it was about a wash
and maybe it would have saved us,
you know,
10 or 15%
on our infrastructure costs
because you still had
tons of locking
and database connection stuff
that you had to deal with
that was still kind of
put like some limits
on what you could do.
But I'm excited
to play with it in 2019
because I think
we could probably
drive our costs down
by if we do it really well
by 30 or 40%.
And now that's starting
to be appreciable amounts
of money.
Like before,
that was not that much.
So now I think
it would probably
be getting closer
to worth it.
It's that complexity trade-off
and at some point
when it's $50,000 a month,
you're like,
you know what?
We're going to just
bite the bullet
and solve this complexity problem
and, you know,
give everyone a raise
or whatever, right?
Yeah.
In those cases, yeah.
High-world people.
Yeah, yeah.
And that's, I mean,
you know,
you mentioned that
we also do like profit sharing.
So there's like
there's like a lot of incentive
to do some of these
like cool cost-saving things.
But when you're measuring,
you know,
and this is getting away
from maybe the Python side,
but you're getting into
like the business side
of like,
well, we could get a,
we could trade a dollar
of cost savings
for $2 of revenue.
Like, you know,
do you go for the revenue?
Because it opens up
more opportunities
later on,
serving more customers.
So you're always,
I'd go for the revenue.
Yeah, we're always like,
I'd go for the revenue, man.
Yeah, we're the same way.
Like we've been,
you know,
really fortunate
that the product itself
really connects
with all these users.
So for us,
like expanding that
is really,
really interesting.
Right.
Well, that's a whole,
that's actually basically
a good,
it has analogies to like,
why do you even choose
Python at all, right?
Like you could write this
in C++,
mostly with wrapping it
in Java or something,
and you could save money
on infrastructure, right?
But the rate
at which you generate features
and you make users delighted
would be slower.
And, you know,
it's like people say
Python is slow
and you guys are doing
incredible stuff.
I don't actually think
that's true.
I think it's really,
I think it's like,
it's complicated.
It's more like the right answer.
But at the same time,
it's,
that does not say,
talk about,
well, Python is fast
for features, right?
Which is pretty awesome.
That is pretty awesome.
And you're looking at
companies that are
generally selling
like a digital good.
The margins are really good.
So you have that headroom.
Like you're not getting
squeezed down to
exactly some super tight thing
where you're like,
oh, we have to rewrite this
and see,
we have to like really
go super low level
and get that,
eat that performance out.
Right.
You're not General Motors
at 6% margin, right?
Yeah.
You're,
you're doing,
you got more room
to play with, right?
That equation may change.
Let's say you're the,
I'm going to say
some of these like
security cam software
that exists out there, right?
Maybe you do like
this cool little hardware.
Like we get,
we sell the hardware
and we give away like
the free tier
and you have like
a more complex business model.
And now you may care
a lot about those margins
and Python may not work for you.
And in that case,
yeah, it's too slow, right?
Right, right, right.
It's very context specific.
For us,
it's not too slow
because the value
we can create with it
is, you know,
orders of magnitude
higher than the effort
that goes into it.
And that changes
depending on the business
and the margins
and the context.
for us,
it's been awesome
and it's been really,
really fun.
But that's not to say
that there's maybe
not cool opportunities
to use.
Maybe see,
or maybe for us
might be like more
like Golang stuff.
And if you choose
to rewrite some
kind of like core
tier zero services
in that
to eke out a bit
more performance,
but still use Python
for a lot of the
product facing
and things that you're
going to be touching
on a day-to-day basis
and iterating for users,
it's not one and a
done kind of a thing
either these days.
Yeah, yeah.
Well, one of the things
that seems like
there's probably a place
somewhere for this to fit,
but it might cross
the complexity threshold
on the other direction
and make it not worthwhile.
But, you know,
like with Cython,
you could take
the little bits
that are slow,
write that in Cython
and then use
the no gill
block in Cython
and achieve a ton
of concurrency
and CPU speed
with, you know,
changing like
20 lines of code
potentially.
Like, have you explored
any of that?
We have done
a little bit of that,
especially in some
of our like
ancillary services.
A good example is
we have a email parser
which kind of lets you
tag emails
and then we'll try
to extract pieces of that.
And we've done
quite a bit of stuff
where we've tried
to optimize that engine,
but like it's just
pure Python.
It's just doing
a bunch of like
string slicing
and comparison stuff
and you can wrap
that up in Cython.
Cython was a thing
that we actually used.
We ended up
rewriting a little bit
of it,
like some of the core
pieces in C++
to speed it up.
But again,
a very kind of niche,
very specific use case
which is really
how you should be
thinking about performance
is you should make it work
and then if you need to,
right,
make it fast
and use the tools
at your disposal
and Cython was one of those.
And we had pretty good
success with it.
So I do think
it's a realistic option.
Yeah, I've seen some really good
outcomes from that as well
and I think it's
pretty interesting, right?
It's like when you look
at a huge code base,
there's usually a couple
of spots that if you could
unlock the performance
on just those little bits,
actually that's where
most of the problem is, right?
That's where most
of the latency is.
Yep, 100%.
And there's some
really cool posts out there.
I think Instagram
engineering did some
where they would use
like the C profile
module
and then measure
CPU ticks
for like all the
different stuff that they did
so they could really get
like instead of like
wall clock time
which can be like,
oh, we blocked on this,
we waited on this
and you get a different
like look at the code.
You can do some
really cool stuff
with Python that
I don't know if a lot
of folks are like aware
of like the just
the level that you can go
just like straight Python
and get like really,
really high performance stuff.
There's a lot of cool
tricks in headroom
and the language itself.
So I think we got
a lot of headroom
when it comes to like
Python and Zapier.
We're definitely doubling down
on using Python
in 2019 and beyond.
And the conversion
from legacy Python
to modern Python
is it's underway?
Yeah, it's underway.
Yeah, we've got
we're running all of our
like test,
all of our tests run
in both Python 2
and Python 3
and have been
for a little while.
So really what we got
to figure out now
is just make
make that transition
in like production
and make sure
that we don't break
any like caches
or behaviors
or things like that.
So figure out a good way
to try doing some
canaries
and all this stuff.
That's like kind of
the next step for us.
Like we're right there.
That's exciting.
You're on the cusp of it.
Yeah.
Yep.
I suspect
in the next couple months
we'll probably be running
Python 3 in production.
So pretty exciting.
Sweet.
Yeah.
the one place I've heard
really scary stuff in
is around caching
and pickles, right?
People take objects
and they just pickle them up
and cache them
rather than doing
like a JSON 2
from sort of
intermediate thing
and then boom
like, you know,
a different CPython
grabs it.
It looks different.
Yep.
That can cause
a lot of trouble.
All sorts of badness
breaks out.
We try to avoid pickle
as much as we can
but for some stuff
like the convenience
is just so nice
like just throw anything in,
right?
So we try to avoid
and we try to do
JSON serialization
as much as possible
but yeah,
that is a tricky thing
around deploying
between two and three
and testing them
and it just makes it
really operationally complex.
So that's something
that, you know,
I think is sometimes
overlooked whenever
people think about
transitioning their code base
to Python.
Right, right.
You got to be careful there.
That's cool.
All right.
Well, this has been
such a super interesting
look inside.
Let's have to hear
what you guys are doing
and they're really
neat use case.
There's a couple
other questions
I'm going to ask you
before I let you go though.
Sure.
Even though we're running
a little low on time.
First of all,
like what's it like
being the CTO of,
you know,
a big tech company
like this?
There must be a lot
of plate spinning,
a lot of things to do
but I bet it's fun.
Yeah, it's really fun
and you know,
the role has kind of changed
and I feel like one thing,
the journey that I went on
from early CTO
all the way to now
which was doing,
wearing a lot of the hats
is probably a really common one
but it's different
for a lot of folks.
Like I talk to a lot of CTOs
to try to figure out
like what is a CTO
in like a tech company?
I talked to maybe
a dozen different people
and I got a dozen
different answers
so it's really like
a very personal journey
and like what does
a company need
sort of a thing
so for me,
it's been definitely
a part of like discovery
as well
but also for me,
it always comes back
to what does a product do?
Like I really,
like the product resonates
with me like as an engineer
like giving people
the power of engineers
to do all this stuff
even though they don't know
they're kind of like
building automations
and the same things
that we would do,
you're still giving them
that power
and that's really exciting
to me
so I'm excited
to spend a lot more time
in that
and then diving
into the architecture
and the scaling issues
that a service like Zapier has
is really,
really exciting for me.
So yeah,
that's a fun part of software
to think about, right?
Yeah,
you know,
so being a CTO
at a company
like this
is really exciting.
I'm also just like
really excited
to be hiring like engineers
that are like better than me
when it comes to all this
like deep,
interesting stuff
so we have like,
we have some core
like Django contributors
and they're like amazing
to like dive into
and like learn about
some of this stuff
that we use every day
and that's always
really exciting
so it's really humbling
at the same time
so it's just a really fun,
I mean it's super fun,
it's super fulfilling,
we have a lot
that we want to do
so I feel like
I gotta get back
to work too.
I'm sure.
Alright,
another one is
just a couple real quick,
what are your favorite
automations?
Some of my favorite ones
they have a little
Python sprinkled into them
so those are some
of my favorites
especially whenever
I want to do something
that's kind of
a little bit funky
and weird
and I don't want to
deploy any extra code
like I'll often
like write like
a Python step.
A really specific one
that I've done
because I've been running
a lot of like
engineering
like the meetings
the all hand
kinds of things
and we try to like
prep
and have notes
and kind of
have a lot
of the stuff ready
is I have a schedule
set up
and I have a zap
that triggers off
of that event
which is kind of
a recurring event
and it goes out
and it grabs a template
from Google Docs
it copies a bunch
of stuff into that
into a new Google Doc
includes a bunch
of charts
and a bunch
of little things
that we should
be talking about
and then goes
into the Slack room
and posts it
and reminds people
and I have it
set it up
with the delay
so it even
reminds people
periodically
like hey
two days to go
make sure you
add your notes
to this
and like meeting
is coming up
in an hour
so it's really
really helpful
because a lot
of times
many people
are crazy busy
obviously
so I found
myself going
around reminding
folks to get
that stuff ready
and this kind
of just automates
that away
and does a lot
of really cool
stuff there
so I have so
many like zaps
running
we have so many
bots running
around in Zapier
too
like sometimes
they'll bump
into each other
and do crazy
things
and it's really
quite fun
it's pretty exciting
that's super
awesome
and it has
this nice
aspect
where the
thing doing
the reminding
is not a
person
so that person
doesn't get
a little bit
like ah that
person's a
nagger
right
you don't have
that aspect
it's like well
the bot did it
like it's you know
it does it every
Thursday for our
Friday meeting
or whatever right
yep and it's
super handy
and you can go
in and you can
adjust it
so if someone's
like oh that's
the way this is
phrased or you
didn't include
this link
or it'd be great
if you added
this data to it
like it's
in a team
account like
anyone can hop
in and like
play with it
and change it
and swap it
out and tweak
it and we
always include
like the link
to like edit
this app
in the messages
that we send
around so
anyone can go
in and make
it even better
and add
functionality to
it which is
a really kind
of collaborative
way to build
some of these
workflows and
augment these
bots that are
doing stuff
because like
you'll see it
and you'll be
like oh it'd be
great if it
actually included
this link or
it said this
thing and you
just hop in
and change it
it's really
really fulfilling
to be able to
just like see
something change
something and then
next time it
goes it's
there's no
deployment going
on it's just
doing it it's
really cool
yeah that's
super how cool
the singularity
is near okay
so the last two
questions that are
standard not
focused just on
you as if you're
going to write
some python code
what editor do
you use
i use visual
studio code the
new one i think
that microsoft's
new code editor
like some of their
philosophies around
focusing on
developers has been
really great
yielded some
awesome tools so
i use that
and i quite like
it that's cool
yeah it's definitely
got a lot of
momentum and i think
they're doing
nice stuff there
and then you
definitely work with
a ton of
pypi packages
maybe one that's
notable not necessarily
the most popular but
people you're like oh
you should know about
this have you heard
of yeah whatever
i would probably say
and maybe everyone
knows about this but
it was kind of new
for us for a couple
months ago the
python formatter
black yes really
really love that so
we've completely like
blackified our entire
code base and it's
so nice to just like
not think about that
because like we
would always be like oh
do we want to do
commas here do we
want to split these
lines like it was
just a thing we
were i mean we we
worried about and we
cared about and you
know we want our code
to look great and like
be easy to read and
black does an amazing
job at that so we do
that and we really
really like it so if
folks out there aren't
using black you should
check it out it's
really cool super
useful so that would
be mine
that's great you set
up as like a pre
commit hook yeah we
have it as a pre
commit hook we have
all of our ci stuff
running that checks it
all and like yeah so
it's it's a big part of
our workflow now
yeah solid that's
cool all right well
final call to action
people are excited about
this distributed work
zapier all the tech we
talked about what do
you got for them
check us out if you're
interested in building
any of these workflows or
you have colleagues that
could benefit from any
sort of like little
workflows like this let
them know i mean we
can kind of do a little
bit of anything if you
use any sass apps we're
probably got something
useful for you and
also if you're looking
for a new gig and
you're interested in
remote work check us
out not only do we have
a ton of information
on how we work like
we just shared it and
kind of open source a
lot of our processes but
we'd love to maybe work
with you so we have
lots of jobs coming up
in 2019 lots of roles
so definitely keep your
eyes out there and you
can even set up a zap
that will alert you of
new jobs that pop up on
the site and then they
will send you an email
or maybe a slack
message or something
like that so check that
out that's pretty meta
and awesome awesome let
me ask i guess one more
follow-up so you talked
about if people want to
set up these automations
that makes a lot of
sense what if they're
these folks listening are
developers at a company
and they're like why
doesn't zapier integrate
with my company and my
apis like what do they
do then yeah so we do
have an open platform so
anyone can kind of hop on
and i mean we actually
have a lot more like
private apis that people
have added for their own
companies it's really
really straightforward we
have lots of examples
you can either build them
in the browser and just
kind of post your like
rest urls and things
like that and we'll do all
the magic there or you
can write a javascript
node.js package which
fits just kind of a light
spec that we have we have
tons of examples and you
can expose different
endpoints and write little
custom functionality so
you can say here's this
json list that shows all
the most recent leads for
our company that can turn
into a trigger and here's
a post endpoint to create
a lead and you can define
what fields go into that
and then away you go and
it works with you know
1400 different apps out
of the box which is
really powerful that's
pretty cool all right
well this has been a lot
of fun to talk to you
about all these ideas so
much cool stuff you're all
doing thanks for being on
the show of course thanks
michael see ya yeah bye
this has been another
episode of talk python to
me our guest on this
episode was brian helmig and
it's been brought to you by
linode and ting linode is
your go-to hosting for
whatever you're building
with python get four
months free at talk
python.fm slash linode
that's l-i-n-o-d-e ting is
the fast mobile network
custom built for technical
folks use their savings
calculator to see exactly
what you'd pay visit
python.ting.com to get a
twenty five dollar credit
and get started without a
contract want to level up
your python if you're just
getting started try my
python jumpstart by
building 10 apps course
or if you're looking for
something more advanced
check out our new async
course that digs into all
the different types of
async programming you can do
in python and of course
if you're interested in more
than one of these be sure
to check out our
everything bundle it's like
a subscription that never
expires be sure to
subscribe to the show open
your favorite podcatcher and
search for python we
should be right at the top
you can also find the
itunes feed at /itunes
the google play feed at
/play and the direct
rss feed at /rss on
talkpython.fm this is your
host michael kennedy thanks so
much for listening i really
appreciate it now get out
there and write some python
code
you're welcome you're welcome
Thank you.