Learn Python with Talk Python's 270 hours of courses

Automate all the things with Python at Zapier

Episode #199, published Thu, Feb 14, 2019, recorded Mon, Jan 21, 2019

Do your applications call a lot of APIs? Maybe you have a bunch of microservices driving your app. You probably don't have the crazy combinatorial explosion that Zapier does for connecting APIs! They have millions of users automating things with 1,000s of APIs. It's pretty crazy. And they are doing it all with Python. Join me and Bryan Helmig, the CTO and co-founder of Zapier as we discuss how they pull this off with Python.
Zapier: zapier.com
Bryan on Twitter: @bryanhelmig
Jobs at Zapier: zapier.com/jobs
Episode transcripts: talkpython.fm

--- Stay in touch with us ---
Subscribe to Talk Python on YouTube: youtube.com
Talk Python on Bluesky: @talkpython.fm at bsky.app
Talk Python on Mastodon: talkpython
Michael on Bluesky: @mkennedy.codes at bsky.app
Michael on Mastodon: mkennedy

Episode Transcript

Collapse transcript

00:00 Do your applications call a lot of APIs?

00:01 Maybe you have a bunch of microservices driving your app.

00:04 You probably don't have the crazy combinatorial explosion that Zapier does for connecting APIs.

00:10 They have millions of users automating things with thousands of APIs.

00:15 It's pretty crazy.

00:16 And they're doing it all with Python.

00:17 Join me and Brian Helmig, the CTO and co-founder of Zapier, as we discuss how they pull this off with Python.

00:23 This is Talk Python to Me, episode 199, recorded January 21st, 2019.

00:28 Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.

00:48 This is your host, Michael Kennedy.

00:50 Follow me on Twitter, where I'm @mkennedy.

00:52 Keep up with the show and listen to past episodes at talkpython.fm.

00:55 And follow the show on Twitter via at Talk Python.

00:59 This episode is brought to you by Linode and Ting.

01:01 Check out what they're offering during their segments.

01:03 It really helps support the show.

01:05 Brian, welcome to Talk Python.

01:07 Thank you, thank you.

01:08 I'm excited to be here.

01:08 I'm excited to have you here.

01:10 It's so cool to have a look inside a company like Zapier and all the interesting ways you're using Python,

01:16 and especially some of the scalability and sort of interoperability stuff you got going on

01:19 is like next level stuff you got there.

01:22 Yeah, yeah.

01:22 It's some really cool stuff.

01:24 It's really fun.

01:24 Yeah, awesome.

01:25 Now, before we get into all that, though, let's start with your story.

01:28 How did you get into programming in Python?

01:29 I got into programming mostly, I want to say, early university years, right?

01:34 So kind of getting into that, hmm, I want to start some startups.

01:37 I want to kind of create some businesses online.

01:39 And I was big into music, so I actually did like online guitar lessons for a while

01:44 and needed to build a site around that.

01:46 And it just kind of all expanded from that.

01:49 But I never trained it like formally.

01:51 I just literally like got addicted to doing it like day in and day out.

01:55 And then all throughout college, like that's basically like all you could find me doing

01:59 was like building things and creating all these little side projects and little businesses that didn't really work.

02:04 Where's Brian?

02:05 He's in the computer lab, man.

02:06 He's down there working on something.

02:08 That's cool.

02:08 I really like it.

02:10 It's like you learn programming because you have these ideas.

02:12 And you're like, well, programming is, you know, it's the hammer I need to build my idea, right?

02:16 The hammer, the saw, and whatever.

02:18 Yeah, exactly.

02:18 And then you, I mean, you learn all the more formal stuff, like after having like practical,

02:23 like applications for it.

02:25 And for me, that always just worked better for, I mean, same thing for music.

02:28 Like I learned like the licks and all the cool things.

02:30 And then I'd learn like the theory of why it works.

02:32 And that was always much more satisfying to me, like as a person.

02:36 So I found it worked well.

02:38 Oh, that's great.

02:38 So did you start out with Python or did they have you doing C++?

02:43 Or where did you start when you're in college?

02:46 With those sites like the WordPresses and the Joomla's and the Drupal's and stuff,

02:50 it was a little bit of PHP.

02:51 But once I found Python and particularly Django back in the very early days of Django,

02:56 I was like addicted to that.

02:58 And that was really, I mean, I would say really Django was like the thing

03:01 that got me into the Python ecosystem and got me stuck on it.

03:05 And it's funny, Zapier's built on Django and I've spent a lot of time in the Django ecosystem since then.

03:10 Yeah, that's really cool.

03:11 It's just like you found it and it's still serving you well.

03:14 You know, I think that's one of the things that's super interesting about Python in general.

03:18 And I think of Python as a full spectrum language, right?

03:23 There's languages that are really good for getting you started to build something simple,

03:27 but then they kind of top out before you can go like pro with them.

03:31 And then there's pro languages like Java.net C++, where it's like you'll start there.

03:35 You start with all the complexity.

03:37 And Python's one of those few ones that you start easy, but as you need more complexity,

03:42 you can start to bring in those features of the language, but they're not forced on you from the beginning.

03:47 Right.

03:47 And the industry support for Python's like outrageous.

03:50 Like a lot of these like more niche sort of things, which are kind of fun and cool,

03:53 or like they just don't have as much like adoption throughout the industry.

03:57 And when you look at like, you know, employment opportunities and ease of starting

04:01 and what the ecosystem looks like community, like Python's really hard to beat in both a practical

04:07 and I also think it's a really well-designed, nicely done language as well.

04:11 So you kind of like, it's firing on all cylinders.

04:13 It's really quite incredible.

04:15 Yeah, absolutely.

04:15 I totally agree with you.

04:16 All right.

04:17 So let's talk about just really quickly your role day to day, you know, just so folks know where you're coming from, right?

04:23 So right now you're the CTO of Zapier, right?

04:26 Correct.

04:26 Yeah, yeah.

04:27 I'm CTO.

04:27 I was one of three founders and we've grown, man, it's Zapier is a little over seven years old now.

04:34 So we've grown from just us three, Wade, Mike and myself to 200 people.

04:39 So obviously the things I'm doing day to day are a lot different than they used to be.

04:42 It used to be a lot of hacking and then it quickly turned into a lot of hiring.

04:47 And that's really been where a lot of my time has been.

04:50 You know, some of the cool things are like, once you get to certain scales,

04:53 you can start to really hire amazing, amazing people to take over different slices

04:59 who are way better than you.

05:00 And we recently did something similar.

05:02 We just hired a VP of engineering who I'm hoping will be able to kind of take us

05:06 to that next level for organizational and hiring and all this stuff that's so critically important

05:11 to like building a team.

05:12 And maybe I can spend a little more time on the code side and the product and architecture side.

05:17 So I'm really excited in 2019 to spend a bit more time on that.

05:20 That's awesome.

05:20 A little bit back to your roots.

05:21 Yeah, yeah.

05:22 No, it'll be really, it's going to be really fun.

05:24 I'm really excited.

05:24 Yeah, I think in the early days, if it's the three of you starting a company,

05:29 you all have to do so many things that you just, you are unprepared and unqualified to do, right?

05:36 Like, you're like, all right, who's going to do accounting and taxes?

05:38 You're like, wait a minute.

05:39 Nobody knows about this.

05:41 Who's going to do, you know, deployment?

05:43 Like, well, we've never deployed it.

05:44 Like, you know, there's just so many things, but now you can find somebody who's like,

05:48 say, super good at DevOps and you don't have to worry about that.

05:51 And that's a big transition, right?

05:52 Yeah, it is.

05:53 And you kind of, you know, whenever you're just starting out, you can kind of get away with the mess, right?

05:57 Like if you mess up a deploy, it's like, well, only 10 people were using your site.

06:01 So no one noticed, you know, you have little things like that, that really help.

06:05 And same thing.

06:06 I mean, you mentioned like accountant hiring and like, you're not doing so much of it.

06:10 So if you're not world-class at it, you know, you can kind of get away with it.

06:13 But at the heart of it, like you got to build that product that really connects to people.

06:18 And if you do that well in the beginning, the rest kind of comes and you can get amazing people

06:23 who know all the, know how to build a great deployment system, know how to run taxes in 50 different states,

06:28 know how to do all this other stuff.

06:30 So yeah, it changes.

06:31 Yeah, yeah, I'm sure.

06:32 All right, cool.

06:33 So we talked a little bit about it.

06:35 You work at Zapier and stuff, but you know, I'm sure most folks listening

06:39 have more or less heard of it, but maybe they don't really know.

06:42 They're like, oh, I think that's some kind of online startup thing.

06:45 Tell us what is Zapier.

06:46 Yeah, so you can think of Zapier as sort of the glue between all the different SaaS services out there.

06:52 So if you use Gmail, if you use, especially like Google Sheets, if you use things like Dropbox or Slack,

06:59 like all these different services that people use on a day-to-day basis,

07:02 they have data in them and you want to trigger on that data and do other things and other services.

07:09 Zapier is that like mechanism that does that, which is very much, you know,

07:13 I think of it a little bit like, you remember all the, and I guess there still exists,

07:17 some of the languages like Scratch who are kind of those visual programming

07:21 and they never really caught on, but they kind of were like, oh, we're going to give the power of automation

07:25 and the power of coding to the masses, right?

07:29 Yeah.

07:29 It was always, it's an awesome goal, but it was always really difficult.

07:33 And I think we're kind of doing that and kind of coming in through the back door.

07:37 So you can do these cool things where when you, you know, star a message in Slack,

07:42 you can add it to your to-do app and you can choose one of, you know, a dozen to-do apps that are supported on Zapier.

07:49 And then you can think, oh, well, if I say like super important in that message,

07:54 I had an extra filter and I send a message to this room in Slack or that room in Slack,

07:59 maybe, or I send, you know, my colleague a DM, like you can start to build these like workflows.

08:05 And that's really the power of Zapier is that you can kind of start to string these actions together

08:10 and do some really complex stuff that normally couldn't do unless you had a fleet of engineers

08:15 to like custom build this bespoke software.

08:18 This is interesting.

08:18 So, you know, you talked about Scratch and some of these other like edgy block

08:22 type of application.

08:23 Right.

08:24 I'm not going to call it application, but programming frameworks that are these visual things.

08:28 Right.

08:28 And I think the idea of those is not terrible.

08:32 I think the problem is they work at too low of level.

08:34 Right.

08:35 They're like, you issue a print statement, you do a for loop.

08:38 You, I mean, like they literally work almost like line by line as you would with text.

08:42 Whereas what you guys have is like an email came into Gmail and that's like an integration with an API

08:48 and OAuth and like tons of other stuff.

08:50 And then, you know, you've condensed it down to the right size of the draggy droppy thing.

08:55 You know what I mean?

08:56 So it's like, there's a few blocks, not 50,000 blocks, which makes no sense.

09:00 Yeah.

09:00 And you get away from the idea of like trying to teach people control structures.

09:05 Right.

09:05 And how data structures work and like all this stuff is like really heady stuff.

09:12 Like once you get it, it's fine and it doesn't bother you.

09:15 But like until you get it, it's another world, right?

09:18 It's completely foreign.

09:19 And we avoid that.

09:21 And we try to keep it just simple to like, oh, I get a new email and I add a message to Slack.

09:27 Like, got it.

09:27 Like I can connect those two together.

09:29 And that's always like the beginning point.

09:31 And then I think the really powerful stuff comes where you have a team in there

09:36 and they're like building zaps together or they're adding their own like little flavors of,

09:40 oh, I want an extra, you know, filter here.

09:43 Only do it if this matches or add an extra step to this and don't just send it to Slack,

09:48 you know, send it to my ToDo app or send it to this, send it to my CRM.

09:52 Like we have lots of business folks and marketers who are doing all this like cool stuff as well.

09:56 So this is, this is not for like, only for engineers.

09:59 We do have some, some of my favorite stuff to do is like you can literally embed Python

10:03 into these workflows, which of course like is very topical for this podcast.

10:07 I love it.

10:08 But this is for like marketers as well.

10:10 So they can build these things.

10:11 This is for real estate agents who have, you know, form software and they want to like trigger off

10:17 of people submitting that and get e-signatures and like all of this stuff

10:20 is like in this ecosystem.

10:22 So it's not just engineers or devs that use this.

10:25 It's kind of regular folks, right?

10:27 People who are just trying to get work done, right?

10:29 It's cool.

10:30 And you can bring, bring the level down a little bit.

10:32 You don't have to be, you know, if you're kind of good with computers, but you're not a developer,

10:37 right?

10:37 Like this could be something you could maybe really make what, maybe like a developer would go,

10:43 I'm just going to write an app that'll just call this API than this and write.

10:45 But like, there's so many people who, even if they could write that, they're not going to say,

10:50 deploy it as some kind of background process in like on Linux and put it in the cloud.

10:55 Like they're just not going to do that.

10:56 Yeah.

10:56 So yeah, this is great.

10:57 Yeah.

10:58 Yeah.

10:58 And you think about, that's a great example.

11:00 It's like we often get engineers or devs who are like, I really don't want to build

11:04 that thing you just asked me to build.

11:05 I'm just going to use Zapier because if you think of like, oh, I got to pull this service

11:09 and now I have to come up with this stateful store of which things I've seen

11:12 and haven't seen, I got to deploy it to like, I don't know, maybe we'll put it on a Heroku.

11:17 Maybe we'll like, being a DevOps person, we'll get it still, you know, like it just starts to spider

11:21 into this whole thing.

11:22 Whereas like with Zap, you just like create a trigger, create this, and then you can hand it off

11:26 to the marketer.

11:27 You can hand it off to your colleague and they can just dig into it.

11:30 So that's, that's a powerful pattern.

11:31 Exactly.

11:32 It's also like, I think this is a good idea to build this little integration,

11:36 but I don't want another puppy that I have to take for a walk and maintain.

11:40 I didn't want to, I don't want a thing to care for.

11:42 So if I build it myself, it's my problem.

11:44 If I build it here and hand it off, like you guys, that could be your problem.

11:47 You'll keep it running if the API changes.

11:49 Yeah, we do.

11:50 And there's a lot of things that change in the API world that we stay on top of.

11:53 So it just kind of, it just takes that whole layer of like what's going on away

11:58 because we have, and I'm sure we'll get into, we have tons of tooling that like watches this.

12:03 Yeah, yeah, yeah.

12:03 It's really cool.

12:04 I definitely want to talk about that because, you know, you have thousands

12:07 of integrations and these integrations integrate together, which is even more

12:10 like, you know, combinatorially insane.

12:12 So that's going to be fun.

12:14 And I want to talk about that, but let's just talk briefly about this at a high level

12:17 while we're talking about Zapier.

12:19 You know, a lot of folks, they get up and they get in the car and they drive

12:23 through angry drivers and traffic for 30 minutes.

12:26 Right.

12:26 And they all have to live around their company and they go to work, right?

12:29 But as developers and tech companies and things like that, right?

12:33 Like a lot of times, even if you're sitting next to somebody, you're working

12:36 in asynchronous modes, you're making a GitHub issue, you're doing a PR, you're throwing a message

12:41 in Slack, you're like, there's no reason you've got to sit next to the person, right?

12:44 So that's kind of what you guys decided as well.

12:46 And with your 200 people, you're more or less 100% remote, right?

12:50 That's correct.

12:50 Yeah, 100% remote.

12:51 We don't have a single office.

12:53 We're not geographically bound.

12:55 We have people in, I'm going to say, I think it's like 18 or 19 different countries

12:59 right now that work for Zapier.

13:01 And we've been that since the very beginning.

13:03 So we started 100% distributed.

13:05 We've been distributed throughout the entire time.

13:08 So every single person, it's funny, like basically everyone we've hired,

13:12 we had not met in person, right?

13:15 There's like a few exceptions to that rule, but like everyone we've hired

13:18 has been just like a Zoom call or a Skype call to kind of do the interview

13:22 or interacting over email or interacting over, you know, text communications in some way,

13:28 which really mimics the way that most of us work anyways, right?

13:31 That's been just kind of a core part of Zapier since day one.

13:36 I think that's really, it's pretty exciting, right?

13:38 I mean, there's certainly drawbacks to working from home, right?

13:41 You know, people can get a little bit lonely.

13:43 You have to be a little more self-directed, I think, because some people see

13:48 working from home means I have no boss.

13:49 No boss means I can just do whatever the heck I want.

13:52 Like if I want to just watch football for the second half of the day, I'm doing that.

13:55 Like those types of people don't thrive, but I feel like a lot of developers

14:00 and technical folks are pretty driven anyway.

14:03 Yeah.

14:03 And if they have a place for a proper outlet that's really, they feel like they can contribute to,

14:07 like it's pretty easy to work from home, honestly, as a developer.

14:10 Yeah, I think so too.

14:11 And, you know, everything you're, everything, all the artifacts you're creating

14:15 in a tech company, like you're writing code, you're fixing bugs, you're working on features.

14:20 It's all, like the output's like all there.

14:22 You can see it either shows up in the repo or it doesn't.

14:25 It either lands on the site or it doesn't.

14:27 And so you can, it's not that complicated to like track the work.

14:32 I often get folks who are, tend to be a little bit more, especially in the early days of Zap,

14:36 you're very skeptical about the idea of like remote.

14:39 Like how do you know people are doing their work?

14:41 And how do you, like that has not played out to be the biggest problem for us.

14:46 Many of the problems around, and you mentioned like these are trade-offs,

14:49 like a lot of the challenges you get into are just challenges that you hit in a normal company.

14:55 You just hit them sooner, right?

14:57 Like that communication challenge doesn't happen, you know, when you're in the hundreds

15:01 of like employees.

15:02 It happens when you're in the dozens of employees, right?

15:05 Where everybody's off doing all this like cool stuff and it starts to get hard

15:09 to coordinate, right?

15:10 So we had to invest in coordination and communication a lot earlier in the org

15:15 because you couldn't do that implicitly by picking it up around the water cooler,

15:18 so to say, right?

15:19 So that's always been a big focus for us in the early days was like what's breaking now

15:24 that probably wouldn't break until, you know, we were double or triple the size

15:28 in a normal company.

15:30 So what are some of the tools and things you put in place?

15:33 You know, to me, it feels like distributed teams 2018 is 10 times easier

15:37 than distributed teams 2006, 2008, but still like there's a lot of tools

15:43 to pick from.

15:43 Like what's important part of that whole structure there?

15:46 I mean, they're the same tools I think a lot of companies are using.

15:49 I mean, they're the GitHubs, right?

15:50 They're the Slacks.

15:51 We use Zoom for like calls.

15:53 We do so many video calls, constantly hopping on calls to like chat over stuff.

15:58 We've even built a couple like internal tools that are custom to us.

16:03 So we've built an internal tool called Async modeled around the idea of asynchronous communication,

16:08 right?

16:08 Where, you know, you look at something like Slack, it's very synchronous, right?

16:12 Like you drop information in, it scrolls away and it's gone.

16:15 It's not gone, but it's like basically gone.

16:17 It feels ephemeral, right?

16:19 It does definitely feel ephemeral, right?

16:21 It's hard to keep permanent actions that you need to do like somewhere in your Slack history.

16:25 Yeah, that's a poor, it's a poor idea.

16:27 You want to push that out.

16:28 So what we've done is created this tool called Async and you can kind of think of it

16:32 like WordPress meets Reddit meets Twitter kind of a thing.

16:37 So it's basically a blog mechanism that has some structured ways to describe, for example,

16:43 we require each post have like a TLDR, right?

16:46 Like where you can, under a hundred words or less, give or take, kind of like summarize

16:52 what your decision is that's enclosed in that Async post.

16:56 Man, I love it.

16:56 That's such a good idea because then you can tell do you need, how much attention

17:01 you need to pay to it and like just an email, like that would be a great

17:04 inter-office communication way because I mean, I personally get so much email

17:09 that I'll periodically have to take, you know, eight hour days and just answer email

17:13 just so it doesn't get too bad.

17:14 And it's like, those were those little short vets?

17:17 Like that would help sometimes.

17:18 That's awesome.

17:19 I like it.

17:19 Yeah, it really would.

17:20 And we've also added things like depending on how many people view it or comment or like it

17:25 or interact with it, it gets hotter.

17:27 Like it's a very like Reddit style or Hacker News style thing.

17:31 Yeah, yeah.

17:31 Which helps like surface just important information.

17:34 So earlier this year, Wade posted a whole bunch of like essays on like what we're trying

17:38 to do in 2019 and everybody was interacting and like diving in and chatting about them

17:42 and they'd popped right to the top.

17:44 So you don't even have to like really work super hard to get that information.

17:48 You just kind of hit the homepage of Async and you get a flavor of what's going on

17:53 in the company.

17:53 You can read those TLDRs and you get a good idea.

17:55 You can click in, you can go deeper if you want, you can interact.

17:59 So this is something, as an example of something we built internally for this,

18:03 but it's not crazy software, right?

18:05 It's not super complicated.

18:06 This is something that in conjunction with other tools like the classic,

18:10 like, yeah, like we use lots of Google apps and then the GitHub and the Elysians

18:14 and the Jira's and all that stuff.

18:15 So we do do some custom stuff.

18:17 Yeah, that sounds really awesome and I think it's great.

18:19 You know, I've worked for 10 years for a company that was remote and we had people

18:24 all over the world and there are definitely trade-offs, but I really felt like

18:28 the people I was working with were some of the very best people because they were chosen

18:33 from the population of the world, not from the population of the town I happen to be in,

18:39 which maybe is a small town or whatever, right?

18:41 Like, how has that either been like a positive or negative for you?

18:44 I've seen like a lot of tension between some companies are like, oh, we have to have somebody local

18:48 because it's super important we have this like sort of control or like view over them

18:53 and interaction with them, but I don't know that that's always the best.

18:56 No, I don't think so.

18:57 We believe, and we've seen it in reality, we believe that like talent is like evenly distributed

19:04 throughout the world, but opportunity isn't, right?

19:06 So in a large way, we see this as an incredible opportunity that even just like talk about America,

19:11 a small town in America, right?

19:13 And you have an amazing engineer that lives there.

19:15 They live there because they have family and it's a great cost of life and they love where they're at.

19:20 The opportunities as an engineer are likely pretty slim.

19:24 It's one of those opportunities where those amazing folks, those really talented folks

19:29 can actually contribute at the level that they can contribute at.

19:32 That's pretty special.

19:34 Like that's really, really hard to do when everybody's geographically located

19:37 and you require someone sit there in an office that just wouldn't really like scale out.

19:42 So for us, we think of it as something that is good for business, but also just really good

19:48 for people as well.

19:49 It's just a better way to work.

19:50 It's a better way to live.

19:51 You can live where you want.

19:52 You can have a better, like just look at like the cost of living for your family,

19:56 for everything.

19:57 It's just so much better in every way.

20:00 And you just have to like work through the little challenges that pop up

20:04 in a remote environment and it really helps to be 100% distributed.

20:07 I think when you get that mix thing, yeah, it gets a little trickier.

20:11 The mix is tricky because you got the insiders who are at the office and they don't necessarily

20:16 fully trust the remote workers, but maybe the remote workers are really talented

20:20 so they have to work with them.

20:21 But there's always these little undercurrents of weirdness.

20:24 Like I've been in those situations as well.

20:28 This portion of Talk Python To Me is brought to you by Linode.

20:31 Are you looking for hosting that's fast, simple, and incredibly affordable?

20:35 Well, look past that bookstore and check out Linode at talkpython.fm/Linode.

20:40 That's L-I-N-O-D-E.

20:42 Plans start at just $5 a month for a dedicated server with a gig of RAM.

20:46 They have 10 data centers across the globe, so no matter where you are or where your users are,

20:51 there's a data center for you.

20:52 Whether you want to run a Python web app, host a private Git server, or just a file server,

20:57 you'll get native SSDs on all the machines, a newly upgraded 200 gigabit network,

21:02 24-7 friendly support even on holidays, and a seven-day money-back guarantee.

21:07 Need a little help with your infrastructure?

21:08 They even offer professional services to help you with architecture, migrations,

21:13 and more.

21:13 Do you want a dedicated server for free for the next four months?

21:16 Just visit talkpython.fm/Linode.

21:19 You were just talking about having a company retreat in New Orleans, and the company I worked for

21:25 before when I was working remotely before I started my own.

21:27 Certainly, we had the same challenges of how do you connect, and so having some kind

21:33 of yearly or bi-yearly get-together physically, and also you talked about Zoom.

21:37 The video aspect of the Zoom and Skype actually makes a big difference, I think.

21:42 It really does, and the internet connections are getting so good now that that video connection

21:47 can feel very real, can feel very, like you can interact, you can see someone's body language,

21:51 so you do get that.

21:52 Yeah, you and I are talking, and it's almost as if we're sitting across the table

21:56 from each other.

21:56 It basically is good.

21:57 Yeah, it works really well.

21:58 So we found it to be exceptionally good.

22:02 Yes, I do think having the visual interaction is important, like video calls,

22:06 but then periodically these actual get-togethers, right?

22:09 Yes, the retreats.

22:11 Yeah, like these retreats, I always look forward to them because it was like

22:15 these people I really liked and I would have more of a work.

22:17 Like you wouldn't hang out on Skype.

22:19 You'd be like, hey, I'm just, what are you up to, man?

22:21 Let's hang out on Zoom, right?

22:22 Like that's kind of weird, but you would have these meetings.

22:24 So this is your chance to be more friendly with like your coworkers, right?

22:28 So I think it's a good combo.

22:29 I think it works really well.

22:30 Yeah, I do too.

22:31 And when it comes to the retreats, we've done those since the early days.

22:35 I remember our first retreat was seven people and it's grown up to 185 or 190 people.

22:42 I think we're at the last retreat, which brings about a whole set of, you know,

22:46 unique logistics challenge, but we found them to be really, really useful

22:50 to kind of help set the stage for the stuff that we want to talk about.

22:53 So you get that, you know, one of the cool things is, at least for me, is when you're in an office,

22:58 you kind of are pretty much like flatline energy level.

23:01 Like it's just kind of day in, day out.

23:03 It's what you do every day.

23:04 When you meet the team, you get them together.

23:06 For us, it's like twice a year.

23:08 That energy level's through the roof.

23:09 Everyone's really excited to see each other.

23:11 You're diving into things.

23:12 You're hacking on stuff.

23:14 You're going, like there's some really cool stuff that comes out as a result.

23:17 So you get a little bit of the best of both worlds where you get to go back.

23:21 You get to be in your space.

23:22 You get to like zone in.

23:24 You get to like have your kind of flow state, but you also get to have that really high energy

23:29 connecting with people, spending time with them, which works really well.

23:33 And we try to emulate, some of that serendipity things.

23:36 Like we've used, there's this product called Donut for Slack and it will like pair you up

23:41 with folks.

23:42 So we have these like two by, two people like pairings that happen every week,

23:46 three people pairings that happen every week.

23:48 And they just let you get together and just chat about whatever.

23:51 And you get matched up with random folks on the team.

23:53 So you get a little bit of that as well.

23:55 Oh, nice.

23:56 And throw in little jokes or just stuff just for more personal things.

23:59 Yeah, yeah, you definitely do.

24:00 And you get to hear a little bit about what people are up to and like, you know,

24:03 hear about their family and hear about this or hear about that.

24:06 And it's those sorts of unstructured conversations that kind of bring out

24:10 that humanness, which, you know, you do lose a little bit of when you're remote.

24:14 So we often have like advice for folks to try to plan one day of your week

24:18 where you go to a coffee shop or you go somewhere just to get out and move around.

24:23 And, you know, some people don't need that.

24:25 Some people need more of that.

24:26 And just be aware of that for yourself because you're not often going to get that directly

24:30 from working on Slack or something like that all day.

24:32 I think that's great advice.

24:33 Like people can go kind of stir crazy.

24:35 Like I work 100% from my home office or somewhere else, right?

24:38 Everyone I work with is some kind of contractor or author or something who's also not

24:43 in any form of office with me.

24:44 I know all the coffee shops around, especially in the summer, you know, and sometimes hit the libraries,

24:50 but just getting out that makes a big difference.

24:52 So I guess one more question that people out there might find interesting

24:57 and maybe they will more so after what we talk about in just a moment, but it seems like

25:01 it fits better here is they're out there listening.

25:04 They're like, you know, I have this Java thing I'm doing and I'm not really super happy.

25:08 I love a Python job.

25:09 You guys are pretty much doing tons of Python and they could work from where they are.

25:14 Are you hiring?

25:15 Yeah, we're doing a lot of hiring in 2019.

25:17 We've been really fortunate that, you know, we've talked about this publicly.

25:20 Zapier has been profitable for years and years and we've been growing at a really respectable pace

25:25 and we have a lot of ambitions for 2019.

25:28 A lot of cool stuff we got to build.

25:30 So yeah, we're 100% hiring.

25:32 We're looking for folks who have Python background, have JavaScript background.

25:36 We're going to be hiring slightly different, you know, on the manager track,

25:39 we're going to be hiring folks on that side as well, which we kind of think of

25:42 as like a different skill set.

25:43 We're also hiring like security folks.

25:45 We're hiring like test automation, you name it.

25:47 We're hiring all the things in 2019.

25:50 Lots of cool stuff.

25:51 All right.

25:51 Awesome.

25:52 Well, people can, I'm sure there's a career page at zapier.com, right?

25:55 Slash jobs.

25:56 Yeah.

25:56 Zapier.com slash jobs and we got them all on there.

25:59 All right.

25:59 Awesome.

25:59 Nice.

26:00 All right.

26:00 So let's dig into some of the technology because I think there's two sides of things

26:04 that are super interesting here.

26:05 One, you've got your infrastructure and just the amount of people running jobs

26:11 and triggered actions on your site is really, really interesting.

26:15 There's so much happening there.

26:16 And then also you're calling thousands of APIs and these APIs are not totally stable

26:22 and all sorts of stuff.

26:23 So let's dig into that a little bit.

26:25 Like maybe just tell us real quick, like what kind of tech is involved and things along that line.

26:29 We've tried to keep it pretty simple because we have plenty of complexity

26:31 when it comes out on the other side with talking to APIs and some of the

26:36 interesting combinations users make.

26:38 So for us, we're Python.

26:40 I mentioned earlier, we're Django based.

26:41 So we're still using a lot of the ORM, a lot of the kind of standard tooling

26:45 that comes with Django, a lot of the caching bits, a lot of the session bits,

26:48 all that stuff we still use really, really heavily.

26:51 What kind of database back in are you guys talking to with that ORM?

26:54 Yeah, yeah.

26:54 We're talking to MySQL right now.

26:56 We're starting to kind of shard off and we've done and not sharding in the sense of like

27:00 sharding over consistent sharding over IDs or anything like that, but like functional sharding

27:05 where we try to extract this service from kind of the monolith and say, okay, well,

27:09 now this is an opportunity.

27:10 Maybe we'll use a more specialized database.

27:12 Either maybe we'll stay relational and we'll use Postgres or upgrade the MySQL

27:15 to something a little different or maybe we'll use some sort of NoSQL or specialized

27:19 read and write patterns that make that use case even better.

27:23 So for that, we're generally, I mean, we're looking at like mostly like vanilla Python

27:28 like on the back end, the workflow engine side, right?

27:31 which is probably, which is probably, you think of like that big behemoth

27:34 that's running in the background doing all this like crazy stuff, talking to all these APIs,

27:38 kind of your little robots running around the internet and that's all like Python

27:42 and that's usually doing a lot of, I mean, you think of like celery, we use a ton of celery,

27:47 right?

27:47 We use a ton of that and we use RabbitMQ for a lot of the message passing between that.

27:52 We're still using Django on the HTTP front, I mentioned.

27:56 we use a lot of just pretty like vanilla like ingest for like webhooks.

28:01 Like I know we do a lot of pulling, which is a lot of back end stuff, but we also pull in a lot of webhooks.

28:06 So this is great when services send us those webhooks and let us know about things changing.

28:10 Right.

28:10 Webhooks have to be basically the best possible outcome, right?

28:14 Because something like an email or an action happens and that service tells you

28:19 rather than just you got anything for me, you got anything for me, you got anything for me.

28:22 Yeah, yeah, yeah.

28:23 That's, and that's, it's huge.

28:24 We did a big analysis and we actually even shared and try to get more partners

28:28 on board with like webhooks and what we call rest hooks, which is just a way

28:31 to subscribe to webhooks.

28:33 Because I think for a while, especially in the early days of webhooks, it was like,

28:36 hey user, please copy this URL into this other service.

28:40 And then, whereas like now that we have rest APIs, like we can just post

28:43 to a subscription or a webhook endpoint, it'll create a webhook and send you those events

28:48 and then you can tear it down.

28:49 So we've done a lot there and we've done, and we did a bunch of analysis

28:53 in the early days and it's something like 66 times more efficient, like,

28:56 because like on average, you're asking hundreds of times for new data and there's nothing there.

29:02 And then the one time there is, right?

29:03 Yeah.

29:04 Just super, super wasteful.

29:05 So we try to push towards like webhooks and event-driven sorts of things

29:09 as much as we can.

29:10 Yes.

29:10 So you go to these partners, you're like, we would like to not call you 100,000 times a day.

29:14 Yes.

29:15 Could we make it so we just call you 100 or 50 or 1,000 or whatever the right number is?

29:19 Yeah.

29:19 Yep, exactly.

29:20 And some partners are really like into that and they have like some services

29:24 and some endpoints set up to kind of create those webhooks and subscriptions

29:29 and some don't.

29:29 And in those cases, we do fall back kind of that polling architecture.

29:34 But yeah, we really like it when partners do add those.

29:38 I'm sure.

29:38 So I feel like this asynchronous, cueing, message passing, message bust type of world

29:44 must be pretty important for you guys.

29:46 Like, you know, a new trigger has come in from like a new Gmail, but then you want to talk

29:51 to Dropbox.

29:51 Like, you don't necessarily want to block that up.

29:53 You're like, okay, well, kick off a message.

29:54 Like, you need to take this thing from Gmail and move it over to Dropbox

29:57 when you have a moment.

29:58 All right.

29:58 And just sort of pass it down the flow.

30:00 You want to talk a little bit about when that happens and like how that helps?

30:03 Yeah.

30:03 I mean, if you go into Zapier and you go into build a Zap, like a workflow,

30:07 you'll see multiple steps.

30:08 Maybe it's a, I'll use a starring, a message in Slack, and then you add it to,

30:13 let's say, Todoist, right?

30:14 Those are two distinct steps that we've broken apart on the back end.

30:17 And on the back end, they're stored in kind of a self-relational, sort of a tree.

30:22 You can kind of think of it as a tree.

30:24 We try not to, there's no real ways to do cyclical, right?

30:29 Sorts of like things because you get into, now you're into control flow stuff

30:33 and that's, you know, a little bit crazy.

30:34 For most users.

30:35 Recursion with drag and drop, probably not.

30:37 Yeah, yeah.

30:37 It gets tricky.

30:38 So we avoid that.

30:39 So you can kind of think of this as like, that's a two-step Zap, Slack, Todoist,

30:44 and we just pull that data out of Slack, we pass it on to the next one and we have a whole like

30:50 templating system that kind of like passes in that data into the inputs of the next step

30:54 in a particular way and users can define that however they want, which is kind of this

30:58 cool customizability.

30:59 So you can think of that task would say, you know, the message and maybe the time

31:04 and who like actually made that message in Slack and you put that all in the description

31:09 field for Todoist action, which would be maybe create a task or something

31:14 or a to-do item.

31:15 So that would trigger on and push down to a totally separate step that's going through

31:20 RabbitMQ in that case.

31:22 And then once that, you call into Todoist and you're complete with that,

31:26 you get a 200, everything's hunky-dory.

31:28 You look for any other children you have and in that case, you have nothing

31:31 and you insert no child task and you're done.

31:34 This app is finished running.

31:36 That's pretty much kind of the workflow and you can see how that would expand

31:39 and you could have dozens of steps and it walks down through a dozen steps

31:42 until it gets to the last one.

31:43 Nothing to do, so it's finished.

31:45 Yeah, yeah.

31:45 How do you deal with like the poison message equivalent, right?

31:49 So like the third step is I want to call the Todoist API and I'm a big fan

31:53 of Todoist by the way, but you know, maybe that day the Todoist API is down

31:57 for like two hours and you're like, well, do you like throw it back in the queue?

32:02 Do you put it in some other place like we're going to get to this maybe when things are better?

32:05 Because I'm sure you're like, there's probably an API permanently at any moment in the day

32:11 there's one of your APIs that is like not behaving that great, right?

32:14 That you got to interact with.

32:15 There's so many.

32:16 Yeah, definitely.

32:16 And there's a couple ways that we handle that.

32:19 So we do a ton, since we do so many requests and we log all these requests

32:23 and we track all these requests just for understanding the health of APIs.

32:27 When we see one starting to fail, we see if it's failing for a lot of users

32:31 or if it's just one user.

32:32 If it's just one user, that's an error.

32:34 Maybe, you know, something about the to-do item they were adding is actually invalid, right?

32:40 Right.

32:40 They deleted the project they're trying to add it to or whatever, right?

32:43 Yeah, exactly.

32:44 It could be anything like that and that's a user-specific thing.

32:47 So that would pop up as an error.

32:49 We'd email them, hey, heads up, something funny is going on with your Zap.

32:51 Click here to kind of learn more.

32:53 But if it's something that's happening across a high cardinality of users,

32:57 a bunch of different users are hitting this, now we know, okay, something looks

33:01 a little fishy here.

33:02 And at that point, we'll start to hold things back and periodically like kind of let them through

33:07 to see, do we have things going from, you know, maybe a 403, we're getting like

33:11 nonstop 403s, someone shipped a bug and da-da-da.

33:14 It's a mess.

33:15 And we'll retry those until we start to see 200s and then we'll let the rest of them go.

33:20 We'll try to trickle them out until we see them clear up.

33:23 And that's a really common pattern for us.

33:25 And we have, I mean, we've invested a lot in just trying to understand when APIs go down

33:30 because I joke that we're like an edge case catching machine for the internet

33:33 and anything that will go wrong, we will find out about it in some way.

33:37 So we have to have all this like really intelligent monitoring to kind of stop that

33:41 from happening, really keep it from impacting users or at least like give them the awareness

33:46 that, hey, heads up, there's some internet stuff going on.

33:50 you know, maybe AWS is having some sort of an outage and look at all these services

33:53 that are 503.

33:54 And we can kind of tie that together.

33:56 Yeah.

33:56 Cross your fingers and hope that a US East 1A doesn't go down.

34:01 Right.

34:01 Yeah.

34:02 Like how much stuff runs on the Virginia data center of AWS.

34:05 So what?

34:06 Yeah.

34:06 A lot of ours does.

34:07 And we can often tell when, like if we see any funniness with AWS, we can always just go look

34:12 at our like partner dash partner status health dashboard and see if anyone else

34:18 is having issues.

34:19 And if it's just us, it's like, okay, so we must be doing something funny.

34:22 But if everybody's like having issues, we're like, okay, so Amazon is having an issue

34:26 and they haven't quite updated their status page yet.

34:28 Right.

34:28 Exactly.

34:29 You're like a little bit of a canary on that.

34:31 Yeah, definitely.

34:32 Interesting.

34:33 This portion of Talk Python is brought to you by Ting.

34:38 Let me tell you about Ting, a new mobile service available in the US that's targeted developers

34:43 and other technically savvy folks.

34:45 First of all, their average customer only pays $23 a month, but they're no discount provider.

34:50 Their service runs over T-Mobile's and Sprint's fast nationwide network.

34:54 If you don't use that much data because you're usually on Wi-Fi like many of you are,

34:58 then Ting will save you a ton of cash.

35:00 But don't worry, you can still use as much data as you like for just $10 per gig.

35:04 One mobile feature I use all the time is tethering.

35:06 And with Ting, you get unlimited tethering at the same data rate with your account.

35:11 $6 a month for a phone line, $10 a gig, $3 a month for text if you usually chat

35:16 over iMessage or WhatsApp.

35:17 Think about it, no contracts and super clear and fair billing.

35:21 Visit python.ting.com, that's python.ting.com, and check out their savings calculator.

35:27 Enter your usage and see exactly what you'd pay.

35:30 Use that link and you'll get a $25 credit to try them as well.

35:34 That's python.ting.com or just click the link in the show notes.

35:39 I guess it sounds like you host a lot of your stuff at AWS and you probably don't have

35:43 like a big in-house data center.

35:45 Like that made more sense if you have a big office and like part of it could be the data center.

35:49 But if you have no office, like who's going to offer up their bedroom to get fiber

35:53 and like cooling and like, all right, that just doesn't make sense.

35:56 So are you running AWS?

35:57 Where's your services hosted?

35:59 We're predominantly AWS.

36:00 We have a lot US East, of course, but we also have a couple things that fail over

36:04 to multiple different regions.

36:05 So yeah, I mean, we've just leaned into AWS since I think the early, I think maybe we did

36:10 like Linode or Linode back in the early days for a hot minute and then that went over

36:14 to AWS since then.

36:16 We try to use some of the managed services where we can.

36:19 So you're looking at the RDSs and the Elastic Caches and of course, like the ELBs and ALBs

36:25 that exist out there, tons of S3.

36:27 So we try to lean on some of the managed services where we can to just knock down

36:31 the amount of like deep, intricate knowledge we have to have about like Postgres

36:35 or MySQL or whatever, right?

36:37 Yeah, let them do the operational bits of that.

36:39 Interesting.

36:39 So to me, it feels like the cloud, especially AWS and Azure and Google, those three clouds

36:46 feel like that's the new lock-in, right?

36:49 And I don't necessarily mean that in a bad way, right?

36:52 Like when you take full advantage of a platform, you get all of the features

36:56 of it, right?

36:57 Like you could treat AWS as a fancy Linux hosting environment, right?

37:03 Or you could use all the databases and all the files.

37:06 all of the APIs and really get a lot more out of it, but then you're kind of permanently

37:10 stuck to it.

37:11 Like how do you guys think about like lock-in and cloud and all that?

37:14 We kind of like tread the line a little bit in between.

37:16 So all of our workloads that you think of or a large majority of our workloads

37:20 that you think of just like running tasks and performing these actions for users,

37:25 like they're running pretty bare bones like EC2 instances.

37:28 We found that that generally has the best performance characteristics.

37:33 we have the most kind of control over it.

37:35 So in that case, like it's not really like locked in a whole lot.

37:38 We're starting to lean into things that are like open source orchestration frameworks,

37:42 the classic Kubernetes like transition that I think a lot of companies are like trying to do now.

37:46 Again, we're talking about pretty open source standard sort of a thing, which we really like,

37:51 but we're also not afraid that on an occasion using some of these specialized services,

37:56 if it makes our lives a lot better, we'll use it.

37:59 Like RDS is a great example of kind of them splitting the middle.

38:02 like it's standard MySQL.

38:04 So in the future, if you decide you want to like swap out, it's not the end of the world,

38:08 but you don't have to worry about it in the meantime, right?

38:11 You get all the goodies kind of a little bit, certainly not for free, but with less work

38:16 on your side and I can focus on the product.

38:18 So we've definitely leaned into that a lot.

38:20 We haven't done a whole lot of like hyper specialized stuff.

38:23 Maybe the only thing I think that we do a lot of is use a ton of AWS Lambda.

38:28 In fact, like back in the early days, we built our own version of AWS Lambda.

38:35 It was, I forget what we called it, but where we could more securely run

38:39 a bunch of partner provided scripts and user provided scripts.

38:44 That way they could do this kind of Python and his app thing, or they could like

38:48 stand up their own partner apps.

38:49 So this allows partners to kind of deploy code to us, to talk to APIs and integrate with

38:55 any of our, you know, 1400 different apps that was running on our own little flavor.

39:00 And then as soon as AWS came along, it was like, actually, we got this and it's way better.

39:05 We immediately swapped it all over to that as fast as we could.

39:09 So we do use things like that, but even that has been commoditized by, to a large degree.

39:14 I mean, both Google and Microsoft's cloud offerings have, you know, the serverless

39:19 sort of vibe going on too.

39:21 So it's kind of interesting.

39:23 we've tried to balance it on the whole to where we can focus more time on the product,

39:28 but not go like too hardcore, use really specialized stuff.

39:32 Right, right.

39:33 It's that kind of stuff that lets you do what you're doing on a 200 person team

39:38 instead of a 500 or a thousand person team, right?

39:40 There's not like a fleet of database engineers and other infrastructure people

39:45 and yeah, right.

39:46 It's like, they're really big benefits to this.

39:48 And I kind of do the same thing.

39:49 Like there's certain things where like, okay, that's really not available elsewhere

39:52 or so much more work.

39:54 I'm going to just, you know, plug into that API or this API, but yeah, definitely.

39:57 Yeah.

39:58 So you did talk about calling a few APIs and if you're doing that with Python,

40:05 like the most common way to do that probably is the request, the module by Kenneth,

40:09 right?

40:09 Right.

40:09 Are you guys using requests for all this?

40:11 How's this work?

40:12 Yeah.

40:12 Yeah.

40:12 We use requests, but we use a bunch of our own.

40:15 I mean, the great thing about requests is how like extensible it is.

40:18 Like you can swap in and out your own HTTP adapters.

40:21 And then from there, you can swap out almost everything.

40:23 So we have custom DNS resolvers.

40:26 We have custom request adapters or HTTP adapters.

40:30 We have custom like response and request types that all kind of encapsulate

40:34 some of the behaviors and the logging and the monitoring that we want to have.

40:38 And we have like different classes.

40:39 Give us some concrete examples.

40:41 Like, so you have a custom DNS thing.

40:42 Is that like, so you can do better caching and not like, like make the calls

40:46 faster?

40:46 Like what's the idea there?

40:47 The idea there is if you, for us, like we want to be able to have a lot of kind

40:53 of control over what DNS services we have with which like adapter we're using.

40:59 For example, we use requests for internal communications, which is like a different

41:02 DNS sort of provider.

41:04 And then when it comes to like user outbound user stuff, we would want to use

41:09 either like the Google 8.8.8.8 or maybe like open DNS or even maybe some sort of a

41:15 custom thing for a user.

41:16 So in those cases, we want to be able to swap that out.

41:19 And that's a fairly low level.

41:21 I mean, it's kind of automagically done for you a lot of times.

41:24 But if you want to control that, you have to kind of start digging into the HTTP adapters

41:28 and intercept some of the connections that are opening and kind of like all this other stuff.

41:32 And you have the opportunity to do that with requests.

41:35 And, you know, a really specific example, we talked a lot about this monitoring bit.

41:40 So we have different libraries that you can like, or sorry, different like interfaces

41:45 that you can basically attach and say, hey, this is my Salesforce session.

41:48 And it'll start to do all the logging for like this public API to the proper spots for you.

41:54 So you don't have to like worry so much about like, oh, am I doing the right logging here?

41:58 Am I doing, and it'll do all this stuff where it like kind of cleans up the secrets

42:02 out of it.

42:03 And it does the proper truncating.

42:04 So we're not like dumping like 50 megs to the logging service.

42:07 And it does all this stuff.

42:09 The logs are full again.

42:09 Yeah.

42:10 That you, you kind of forget that.

42:12 Oh yeah, yeah, yeah.

42:12 You got to remember to do that through a very familiar interface that you can just pop on

42:16 to, you know, Python requests dot, you know, read the docs dot, you know,

42:21 org and read all about exactly what it's doing.

42:23 It's so much easier to onboard new employees, new engineers on, and you can kind of extend

42:29 and get kind of the best of the community, the best of tools that exist out there.

42:34 So that's kind of way we've approached using like things like requests.

42:37 And it's been really quite, quite useful for us and very successful.

42:41 It's a great library.

42:42 I'm wondering if, you know, there's some of these integrations that are,

42:45 they basically have really nice prebuilt Python packages.

42:49 And some of them are just like, here's an HTTP endpoint, knock yourself out

42:54 with whatever tech, right?

42:55 So, you know, like for example, GitHub, there's like GitHub packages that you can use in Python.

43:02 Do you ever use those or you're like, it's better for us to just stay separate

43:06 at HTTP access level?

43:08 Yeah, we've gone to the HTTP level.

43:10 Those libraries are awesome, especially for, if you want to just spin something up quick,

43:15 but we've gotten so good at talking, like we talk to REST APIs.

43:19 We're good at that.

43:20 We've done it a lot, right?

43:21 It's kind of our bread and butter.

43:22 So we want to like talk to that and have kind of the control over that.

43:25 So we used to do a little bit of that in the early days.

43:28 And then we got to the point where observability became tricky.

43:31 Some of them used things like requests behind the scenes.

43:35 Some of them used HTTP lib too.

43:37 Some of them just use like the built in, like everyone was doing different stuff.

43:41 And then you had to like rewrite how your observability and your monitoring worked.

43:45 So for us, it just became hard to do.

43:47 So if you're talking to lots and lots of APIs like this, it really does help

43:51 to kind of standardize your interface on that.

43:53 Yeah, it seems like it must have been like almost a dependency, a dependency hell of a sort.

43:58 Like if you're like, we're going to have 500 different packages and those packages

44:02 might go out of favor and then they're not quite up to date or there's like a breaking change.

44:06 Like there's just so many layers, right?

44:07 And you're just like, let's just do the same for everything.

44:09 Yeah.

44:10 that's really where we kind of landed.

44:11 That was through some like learning, like especially in the early days, we did do a couple of those

44:16 like kind of like those libraries that you can just, you know, pip install MailChimp,

44:19 right?

44:20 Yeah, exactly.

44:21 We're good.

44:21 We're good.

44:22 Yeah.

44:22 Do this.

44:23 They did help.

44:23 And a lot of them are also, funny enough, they're built on little odd behaviors

44:29 that seem fine in isolation.

44:30 But when you're running, you know, something like, like what we do, it doesn't really fit.

44:36 For example, some of these libraries, you'll just import, like let's say MailChimp.

44:40 and then, you know, on the global object, do MailChimp.API key and just set your API key there,

44:44 which you never know which user's API key you're going to use.

44:48 Yeah.

44:49 When you're us, like that kind of like, you look at that and you're like,

44:52 oh, that doesn't, that doesn't really work.

44:53 Like I want to create like a MailChimp session, you know what I mean?

44:56 So like.

44:56 Anytime you're doing multi-tenancy, like those little gotchas are super scary.

45:01 You're like, why did I just expose all of this other customer's data or something,

45:05 right?

45:05 Yep, exactly.

45:06 So that's why having control over that layer and being really particular

45:10 about like when you're opening and creating new sessions or you're reusing them or not,

45:14 you want that control.

45:15 So that's where we've kind of landed for that and it works pretty well.

45:18 And you know, when I was talking about the AWS Lambda side, a lot of that's

45:22 on the JavaScript side.

45:23 So we do provide maybe not the perfect equivalent to Python requests on the platform,

45:29 what we call our platform like CLI and SDK, like that has like a similarly inspired

45:35 sort of a thing.

45:35 So use our, in our case, it's like the Z object that has a request method on it.

45:40 If you use that, then you get all the magical like logging and you get all

45:44 the magical like introspection on the calls that you're doing.

45:47 So it supports that.

45:48 And if you break out of that, maybe you have your own little like JavaScript SDK

45:52 for talking to your own service, you'll lose a little bit of that.

45:56 And in that case, like it'll be harder for us to debug your services or help users out.

46:01 So we do try to encourage folks to use that kind of standard thing, both inside Zapier,

46:06 obviously, but also outside of Zapier.

46:08 So when partners are like interacting, working with us, we also provide some.

46:11 Yeah, interesting.

46:12 So you talked earlier about this asynchronous message passing and queues

46:16 and whatnot and celery.

46:17 And that's really important for scalability.

46:20 But another huge, huge scalability booster is some sort of asynchronous web handling,

46:27 right?

46:27 like G event or async and await, asyncio, all those kinds of things.

46:31 Does that show up anywhere?

46:32 We have experimented a bit with it.

46:34 This is like one of those classic, like you try it out and then you run the numbers

46:39 and see how it like, how it like, how much does it save you?

46:42 How much headache does it add?

46:43 How much more complex is it, right?

46:44 Yeah, exactly.

46:45 And we tried this maybe, I'm going to say two years ago with G event.

46:50 It was right on the line, but probably at that time, it just wasn't a good enough

46:55 move for us to like transition over to it.

46:57 So we actually abandoned using G event and we're still using like pretty like

47:02 straightforward, synchronous like Python, which is right now we're still transitioning

47:06 to Python 3 at the moment, making good headway on it.

47:10 But a lot of the, you know, asyncio you mentioned, like a lot of that adds tons of complexity

47:16 to your code.

47:17 G event does the same thing.

47:18 There's all these like little extra things you got to remember.

47:20 And Python, I think, hands you a gun that's very loaded when it comes to like

47:25 async stuff.

47:26 So it is a little bit tricky and that's not to say there's, I mean, there are other languages

47:30 that do a really nice job with async, but even like something like JavaScript,

47:35 I think still, you get a loaded gun, right?

47:37 So it just adds this layer of complexity that I don't think was worth it for us

47:41 at the time.

47:42 I think there's some cool things we can probably do in 2019 to get back to that

47:46 and that's through moving more services behind like an IO boundary, just a regular HTTP boundary.

47:52 so less thing is happening like in the process itself and is sitting across

47:57 a very easily async boundary where you're talking over a socket or something.

48:01 I think if we start to move more workloads over there and we have like kind of this clean interface,

48:06 the Geovent equation will quickly tip in the other direction.

48:10 so I'm excited to maybe look at that again in 2019.

48:13 But yeah, we gave it a good, we gave it the old college try and it was a bit of a bear

48:17 and it was about a wash and maybe it would have saved us, you know, 10 or 15%

48:23 on our infrastructure costs because you still had tons of locking and database connection stuff

48:27 that you had to deal with that was still kind of put like some limits on what you could do.

48:31 But I'm excited to play with it in 2019 because I think we could probably

48:35 drive our costs down by if we do it really well by 30 or 40%.

48:39 And now that's starting to be appreciable amounts of money.

48:42 Like before, that was not that much.

48:44 So now I think it would probably be getting closer to worth it.

48:47 It's that complexity trade-off and at some point when it's $50,000 a month,

48:52 you're like, you know what?

48:53 We're going to just bite the bullet and solve this complexity problem and, you know,

48:57 give everyone a raise or whatever, right?

48:59 Yeah.

49:00 In those cases, yeah.

49:01 High-world people.

49:01 Yeah, yeah.

49:02 And that's, I mean, you know, you mentioned that we also do like profit sharing.

49:06 So there's like there's like a lot of incentive to do some of these like cool cost-saving things.

49:10 But when you're measuring, you know, and this is getting away from maybe the Python side,

49:14 but you're getting into like the business side of like, well, we could get a,

49:17 we could trade a dollar of cost savings for $2 of revenue.

49:20 Like, you know, do you go for the revenue?

49:23 Because it opens up more opportunities later on, serving more customers.

49:27 So you're always, I'd go for the revenue.

49:29 Yeah, we're always like, I'd go for the revenue, man.

49:30 Yeah, we're the same way.

49:32 Like we've been, you know, really fortunate that the product itself really connects

49:36 with all these users.

49:37 So for us, like expanding that is really, really interesting.

49:40 Right.

49:41 Well, that's a whole, that's actually basically a good, it has analogies to like,

49:45 why do you even choose Python at all, right?

49:47 Like you could write this in C++, mostly with wrapping it in Java or something,

49:51 and you could save money on infrastructure, right?

49:54 But the rate at which you generate features and you make users delighted

49:58 would be slower.

49:59 And, you know, it's like people say Python is slow and you guys are doing

50:04 incredible stuff.

50:05 I don't actually think that's true.

50:06 I think it's really, I think it's like, it's complicated.

50:08 It's more like the right answer.

50:09 But at the same time, it's, that does not say, talk about, well, Python is fast

50:16 for features, right?

50:17 Which is pretty awesome.

50:18 That is pretty awesome.

50:19 And you're looking at companies that are generally selling like a digital good.

50:24 The margins are really good.

50:26 So you have that headroom.

50:27 Like you're not getting squeezed down to exactly some super tight thing where you're like,

50:31 oh, we have to rewrite this and see, we have to like really go super low level

50:35 and get that, eat that performance out.

50:37 Right.

50:37 You're not General Motors at 6% margin, right?

50:40 Yeah.

50:41 You're, you're doing, you got more room to play with, right?

50:43 That equation may change.

50:44 Let's say you're the, I'm going to say some of these like security cam software

50:49 that exists out there, right?

50:50 Maybe you do like this cool little hardware.

50:53 Like we get, we sell the hardware and we give away like the free tier and you have like

50:57 a more complex business model.

50:58 And now you may care a lot about those margins and Python may not work for you.

51:03 And in that case, yeah, it's too slow, right?

51:04 Right, right, right.

51:05 It's very context specific.

51:07 For us, it's not too slow because the value we can create with it is, you know,

51:11 orders of magnitude higher than the effort that goes into it.

51:14 And that changes depending on the business and the margins and the context.

51:20 for us, it's been awesome and it's been really, really fun.

51:22 But that's not to say that there's maybe not cool opportunities to use.

51:26 Maybe see, or maybe for us might be like more like Golang stuff.

51:30 And if you choose to rewrite some kind of like core tier zero services in that

51:34 to eke out a bit more performance, but still use Python for a lot of the

51:38 product facing and things that you're going to be touching on a day-to-day basis

51:41 and iterating for users, it's not one and a done kind of a thing either these days.

51:46 Yeah, yeah.

51:47 Well, one of the things that seems like there's probably a place somewhere for this to fit,

51:52 but it might cross the complexity threshold on the other direction and make it not worthwhile.

51:57 But, you know, like with Cython, you could take the little bits that are slow,

52:02 write that in Cython and then use the no GIL block in Cython and achieve a ton

52:08 of concurrency and CPU speed with, you know, changing like 20 lines of code

52:13 potentially.

52:14 Like, have you explored any of that?

52:15 We have done a little bit of that, especially in some of our like ancillary services.

52:19 A good example is we have a email parser which kind of lets you tag emails

52:24 and then we'll try to extract pieces of that.

52:25 And we've done quite a bit of stuff where we've tried to optimize that engine,

52:30 but like it's just pure Python.

52:31 It's just doing a bunch of like string slicing and comparison stuff and you can wrap

52:36 that up in Cython.

52:37 Cython was a thing that we actually used.

52:39 We ended up rewriting a little bit of it, like some of the core pieces in C++

52:43 to speed it up.

52:45 But again, a very kind of niche, very specific use case which is really how you should be

52:50 thinking about performance is you should make it work and then if you need to,

52:54 right, make it fast and use the tools at your disposal and Cython was one of those.

52:59 And we had pretty good success with it.

53:01 So I do think it's a realistic option.

53:03 Yeah, I've seen some really good outcomes from that as well and I think it's

53:05 pretty interesting, right?

53:06 It's like when you look at a huge code base, there's usually a couple of spots that if you could

53:12 unlock the performance on just those little bits, actually that's where most of the problem is, right?

53:17 That's where most of the latency is.

53:18 Yep, 100%.

53:19 And there's some really cool posts out there.

53:22 I think Instagram engineering did some where they would use like the C profile

53:25 module and then measure CPU ticks for like all the different stuff that they did

53:30 so they could really get like instead of like wall clock time which can be like,

53:35 oh, we blocked on this, we waited on this and you get a different like look at the code.

53:39 You can do some really cool stuff with Python that I don't know if a lot

53:43 of folks are like aware of like the just the level that you can go just like straight Python

53:48 and get like really, really high performance stuff.

53:50 There's a lot of cool tricks in headroom and the language itself.

53:53 So I think we got a lot of headroom when it comes to like Python and Zapier.

53:58 We're definitely doubling down on using Python in 2019 and beyond.

54:02 And the conversion from legacy Python to modern Python is it's underway?

54:06 Yeah, it's underway.

54:07 Yeah, we've got we're running all of our like test, all of our tests run

54:11 in both Python 2 and Python 3 and have been for a little while.

54:14 So really what we got to figure out now is just make make that transition

54:18 in like production and make sure that we don't break any like caches or behaviors

54:23 or things like that.

54:24 So figure out a good way to try doing some canaries and all this stuff.

54:27 That's like kind of the next step for us.

54:29 Like we're right there.

54:30 That's exciting.

54:30 You're on the cusp of it.

54:32 Yeah.

54:32 Yep.

54:32 Yep.

54:33 I suspect in the next couple months we'll probably be running Python 3 in production.

54:37 So pretty exciting.

54:38 Sweet.

54:38 Yeah.

54:39 the one place I've heard really scary stuff in is around caching and pickles, right?

54:44 People take objects and they just pickle them up and cache them rather than doing

54:48 like a JSON 2 from sort of intermediate thing and then boom like, you know,

54:52 a different CPython grabs it.

54:53 It looks different.

54:54 Yep.

54:54 That can cause a lot of trouble.

54:56 All sorts of badness breaks out.

54:57 We try to avoid pickle as much as we can but for some stuff like the convenience

55:01 is just so nice like just throw anything in, right?

55:04 So we try to avoid and we try to do JSON serialization as much as possible

55:09 but yeah, that is a tricky thing around deploying between two and three and testing them

55:16 and it just makes it really operationally complex.

55:18 So that's something that, you know, I think is sometimes overlooked whenever

55:22 people think about transitioning their code base to Python.

55:24 Right, right.

55:25 You got to be careful there.

55:26 That's cool.

55:27 All right.

55:27 Well, this has been such a super interesting look inside.

55:31 Let's have to hear what you guys are doing and they're really neat use case.

55:35 There's a couple other questions I'm going to ask you before I let you go though.

55:37 Sure.

55:37 Even though we're running a little low on time.

55:39 First of all, like what's it like being the CTO of, you know, a big tech company

55:43 like this?

55:43 There must be a lot of plate spinning, a lot of things to do but I bet it's fun.

55:47 Yeah, it's really fun and you know, the role has kind of changed and I feel like one thing,

55:52 the journey that I went on from early CTO all the way to now which was doing,

55:57 wearing a lot of the hats is probably a really common one but it's different

56:00 for a lot of folks.

56:02 Like I talk to a lot of CTOs to try to figure out like what is a CTO in like a tech company?

56:06 I talked to maybe a dozen different people and I got a dozen different answers

56:10 so it's really like a very personal journey and like what does a company need

56:14 sort of a thing so for me, it's been definitely a part of like discovery

56:17 as well but also for me, it always comes back to what does a product do?

56:22 Like I really, like the product resonates with me like as an engineer like giving people

56:27 the power of engineers to do all this stuff even though they don't know they're kind of like

56:30 building automations and the same things that we would do, you're still giving them

56:34 that power and that's really exciting to me so I'm excited to spend a lot more time

56:38 in that and then diving into the architecture and the scaling issues that a service like Zapier has

56:43 is really, really exciting for me.

56:45 So yeah, that's a fun part of software to think about, right?

56:47 Yeah, you know, so being a CTO at a company like this is really exciting.

56:52 I'm also just like really excited to be hiring like engineers that are like better than me

56:56 when it comes to all this like deep, interesting stuff so we have like, we have some core

57:00 like Django contributors and they're like amazing to like dive into and like learn about

57:05 some of this stuff that we use every day and that's always really exciting

57:09 so it's really humbling at the same time so it's just a really fun, I mean it's super fun,

57:13 it's super fulfilling, we have a lot that we want to do so I feel like I gotta get back

57:19 to work too.

57:20 I'm sure.

57:20 Alright, another one is just a couple real quick, what are your favorite

57:25 automations?

57:25 Some of my favorite ones they have a little Python sprinkled into them so those are some

57:30 of my favorites especially whenever I want to do something that's kind of

57:33 a little bit funky and weird and I don't want to deploy any extra code like I'll often

57:38 like write like a Python step.

57:39 A really specific one that I've done because I've been running a lot of like

57:43 engineering like the meetings the all hand kinds of things and we try to like

57:47 prep and have notes and kind of have a lot of the stuff ready is I have a schedule

57:51 set up and I have a zap that triggers off of that event which is kind of

57:55 a recurring event and it goes out and it grabs a template from Google Docs

57:59 it copies a bunch of stuff into that into a new Google Doc includes a bunch

58:03 of charts and a bunch of little things that we should be talking about and then goes

58:07 into the Slack room and posts it and reminds people and I have it set it up

58:10 with the delay so it even reminds people periodically like hey two days to go

58:15 make sure you add your notes to this and like meeting is coming up in an hour

58:18 so it's really really helpful because a lot of times many people are crazy busy

58:23 obviously so I found myself going around reminding folks to get that stuff ready

58:27 and this kind of just automates that away and does a lot of really cool stuff there

58:31 so I have so many like zaps running we have so many bots running around in Zapier

58:35 too like sometimes they'll bump into each other and do crazy things and it's really

58:40 quite fun it's pretty exciting that's super awesome and it has this nice

58:45 aspect where the thing doing the reminding is not a person so that person

58:49 doesn't get a little bit like ah that person's a nagger right you don't have

58:52 that aspect it's like well the bot did it like it's you know it does it every

58:55 Thursday for our Friday meeting or whatever right yep and it's super handy

58:58 and you can go in and you can adjust it so if someone's like oh that's the way this is

59:03 phrased or you didn't include this link or it'd be great if you added this data to it

59:07 like it's in a team account like anyone can hop in and like play with it

59:11 and change it and swap it out and tweak it and we always include like the link

59:15 to like edit this app in the messages that we send around so anyone can go

59:19 in and make it even better and add functionality to it which is a really kind

59:23 of collaborative way to build some of these workflows and augment these bots that are

59:27 doing stuff because like you'll see it and you'll be like oh it'd be great if it

59:30 actually included this link or it said this thing and you just hop in and change it

59:34 it's really really fulfilling to be able to just like see something change

59:37 something and then next time it goes it's there's no deployment going on it's just

59:41 doing it it's really cool yeah that's super how cool the singularity is near okay

59:46 so the last two questions that are standard not focused just on you as if you're

59:51 going to write some python code what editor do you use i use Visual Studio Code the

59:55 new one i think that microsoft's new code editor like some of their philosophies around

01:00:00 focusing on developers has been really great yielded some awesome tools so

01:00:04 i use that and i quite like it that's cool yeah it's definitely got a lot of

01:00:07 momentum and i think they're doing nice stuff there and then you definitely work with

01:00:11 a ton of pypi packages maybe one that's notable not necessarily the most popular but

01:00:16 people you're like oh you should know about this have you heard of yeah whatever

01:00:19 i would probably say and maybe everyone knows about this but it was kind of new

01:00:23 for us for a couple months ago the python formatter black yes really really love that so

01:00:29 we've completely like blackified our entire code base and it's so nice to just like

01:00:33 not think about that because like we would always be like oh do we want to do

01:00:36 commas here do we want to split these lines like it was just a thing we were i mean we we

01:00:41 worried about and we cared about and you know we want our code to look great and like

01:00:45 be easy to read and black does an amazing job at that so we do that and we really

01:00:50 really like it so if folks out there aren't using black you should check it out it's

01:00:54 really cool super useful so that would be mine that's great you set up as like a pre

01:00:58 commit hook yeah we have it as a pre commit hook we have all of our ci stuff

01:01:01 running that checks it all and like yeah so it's it's a big part of our workflow now

01:01:06 yeah solid that's cool all right well final call to action people are excited about

01:01:10 this distributed work zapier all the tech we talked about what do you got for them

01:01:15 check us out if you're interested in building any of these workflows or you have colleagues that

01:01:20 could benefit from any sort of like little workflows like this let them know i mean we

01:01:24 can kind of do a little bit of anything if you use any sass apps we're probably got something

01:01:28 useful for you and also if you're looking for a new gig and you're interested in

01:01:33 remote work check us out not only do we have a ton of information on how we work like

01:01:37 we just shared it and kind of open source a lot of our processes but we'd love to maybe work

01:01:42 with you so we have lots of jobs coming up in 2019 lots of roles so definitely keep your

01:01:47 eyes out there and you can even set up a zap that will alert you of new jobs that pop up on

01:01:52 the site and then they will send you an email or maybe a slack message or something

01:01:55 like that so check that out that's pretty meta and awesome awesome let me ask i guess one more

01:02:02 follow-up so you talked about if people want to set up these automations

01:02:05 that makes a lot of sense what if they're these folks listening are developers at a company

01:02:09 and they're like why doesn't zapier integrate with my company and my apis like what do they

01:02:13 do then yeah so we do have an open platform so anyone can kind of hop on

01:02:17 and i mean we actually have a lot more like private apis that people have added for their own

01:02:22 companies it's really really straightforward we have lots of examples you can either build them

01:02:26 in the browser and just kind of post your like rest urls and things like that and we'll do all

01:02:32 the magic there or you can write a javascript node.js package which fits just kind of a light

01:02:38 spec that we have we have tons of examples and you can expose different endpoints and write little

01:02:43 custom functionality so you can say here's this json list that shows all

01:02:47 the most recent leads for our company that can turn into a trigger and here's

01:02:51 a post endpoint to create a lead and you can define what fields go into that

01:02:55 and then away you go and it works with you know 1400 different apps out of the box which is

01:03:00 really powerful that's pretty cool all right well this has been a lot of fun to talk to you

01:03:04 about all these ideas so much cool stuff you're all doing thanks for being on

01:03:07 the show of course thanks michael see ya yeah bye this has been another episode of talk python to

01:03:12 me our guest on this episode was brian helmig and it's been brought to you by

01:03:17 linode and ting linode is your go-to hosting for whatever you're building

01:03:21 with python get four months free at talkpython.fm/linode that's l-i-n-o-d-e ting is

01:03:28 the fast mobile network custom built for technical folks use their savings

01:03:33 calculator to see exactly what you'd pay visit python.ting.com to get a twenty five dollar credit

01:03:39 and get started without a contract want to level up your python if you're just

01:03:43 getting started try my python jumpstart by building 10 apps course or if you're looking for

01:03:49 something more advanced check out our new async course that digs into all

01:03:53 the different types of async programming you can do in python and of course

01:03:56 if you're interested in more than one of these be sure to check out our everything bundle it's like

01:04:01 a subscription that never expires be sure to subscribe to the show open your favorite podcatcher and

01:04:06 search for python we should be right at the top you can also find the itunes feed at /itunes

01:04:11 the google play feed at /play and the direct rss feed at /rss on talkpython.fm this is your

01:04:18 host michael kennedy thanks so much for listening i really appreciate it now get out

01:04:22 there and write some python code you're welcome you're welcome Thank you.

Talk Python's Mastodon Michael Kennedy's Mastodon