#199: Automate all the things with Python at Zapier Transcript

00:00 Do your applications call a lot of APIs?

00:01 Maybe you have a bunch of microservices driving your app.

00:04 You probably don't have the crazy combinatorial explosion that Zapier does for connecting APIs.

00:10 They have millions of users automating things with thousands of APIs.

00:15 It's pretty crazy.

00:16 And they're doing it all with Python.

00:17 Join me and Brian Helmig, the CTO and co-founder of Zapier, as we discuss how they pull this off with Python.

00:23 This is Talk Python to Me, episode 199, recorded January 21st, 2019.

00:28 Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.

00:48 This is your host, Michael Kennedy.

00:50 Follow me on Twitter, where I'm @mkennedy.

00:52 Keep up with the show and listen to past episodes at talkpython.fm.

00:55 And follow the show on Twitter via at Talk Python.

00:59 This episode is brought to you by Linode and Ting.

01:01 Check out what they're offering during their segments.

01:03 It really helps support the show.

01:05 Brian, welcome to Talk Python.

01:07 Thank you, thank you.

01:08 I'm excited to be here.

01:08 I'm excited to have you here.

01:10 It's so cool to have a look inside a company like Zapier and all the interesting ways you're using Python,

01:16 and especially some of the scalability and sort of interoperability stuff you got going on

01:19 is like next level stuff you got there.

01:22 Yeah, yeah.

01:22 It's some really cool stuff.

01:24 It's really fun.

01:24 Yeah, awesome.

01:25 Now, before we get into all that, though, let's start with your story.

01:28 How did you get into programming in Python?

01:29 I got into programming mostly, I want to say, early university years, right?

01:34 So kind of getting into that, hmm, I want to start some startups.

01:37 I want to kind of create some businesses online.

01:39 And I was big into music, so I actually did like online guitar lessons for a while

01:44 and needed to build a site around that.

01:46 And it just kind of all expanded from that.

01:49 But I never trained it like formally.

01:51 I just literally like got addicted to doing it like day in and day out.

01:55 And then all throughout college, like that's basically like all you could find me doing

01:59 was like building things and creating all these little side projects and little businesses that didn't really work.

02:04 Where's Brian?

02:05 He's in the computer lab, man.

02:06 He's down there working on something.

02:08 That's cool.

02:08 I really like it.

02:10 It's like you learn programming because you have these ideas.

02:12 And you're like, well, programming is, you know, it's the hammer I need to build my idea, right?

02:16 The hammer, the saw, and whatever.

02:18 Yeah, exactly.

02:18 And then you, I mean, you learn all the more formal stuff, like after having like practical,

02:23 like applications for it.

02:25 And for me, that always just worked better for, I mean, same thing for music.

02:28 Like I learned like the licks and all the cool things.

02:30 And then I'd learn like the theory of why it works.

02:32 And that was always much more satisfying to me, like as a person.

02:36 So I found it worked well.

02:38 Oh, that's great.

02:38 So did you start out with Python or did they have you doing C++?

02:43 Or where did you start when you're in college?

02:46 With those sites like the WordPresses and the Joomla's and the Drupal's and stuff,

02:50 it was a little bit of PHP.

02:51 But once I found Python and particularly Django back in the very early days of Django,

02:56 I was like addicted to that.

02:58 And that was really, I mean, I would say really Django was like the thing

03:01 that got me into the Python ecosystem and got me stuck on it.

03:05 And it's funny, Zapier's built on Django and I've spent a lot of time in the Django ecosystem since then.

03:10 Yeah, that's really cool.

03:11 It's just like you found it and it's still serving you well.

03:14 You know, I think that's one of the things that's super interesting about Python in general.

03:18 And I think of Python as a full spectrum language, right?

03:23 There's languages that are really good for getting you started to build something simple,

03:27 but then they kind of top out before you can go like pro with them.

03:31 And then there's pro languages like Java.net C++, where it's like you'll start there.

03:35 You start with all the complexity.

03:37 And Python's one of those few ones that you start easy, but as you need more complexity,

03:42 you can start to bring in those features of the language, but they're not forced on you from the beginning.

03:47 Right.

03:47 And the industry support for Python's like outrageous.

03:50 Like a lot of these like more niche sort of things, which are kind of fun and cool,

03:53 or like they just don't have as much like adoption throughout the industry.

03:57 And when you look at like, you know, employment opportunities and ease of starting

04:01 and what the ecosystem looks like community, like Python's really hard to beat in both a practical

04:07 and I also think it's a really well-designed, nicely done language as well.

04:11 So you kind of like, it's firing on all cylinders.

04:13 It's really quite incredible.

04:15 Yeah, absolutely.

04:15 I totally agree with you.

04:16 All right.

04:17 So let's talk about just really quickly your role day to day, you know, just so folks know where you're coming from, right?

04:23 So right now you're the CTO of Zapier, right?

04:26 Correct.

04:26 Yeah, yeah.

04:27 I'm CTO.

04:27 I was one of three founders and we've grown, man, it's Zapier is a little over seven years old now.

04:34 So we've grown from just us three, Wade, Mike and myself to 200 people.

04:39 So obviously the things I'm doing day to day are a lot different than they used to be.

04:42 It used to be a lot of hacking and then it quickly turned into a lot of hiring.

04:47 And that's really been where a lot of my time has been.

04:50 You know, some of the cool things are like, once you get to certain scales,

04:53 you can start to really hire amazing, amazing people to take over different slices

04:59 who are way better than you.

05:00 And we recently did something similar.

05:02 We just hired a VP of engineering who I'm hoping will be able to kind of take us

05:06 to that next level for organizational and hiring and all this stuff that's so critically important

05:11 to like building a team.

05:12 And maybe I can spend a little more time on the code side and the product and architecture side.

05:17 So I'm really excited in 2019 to spend a bit more time on that.

05:20 That's awesome.

05:20 A little bit back to your roots.

05:21 Yeah, yeah.

05:22 No, it'll be really, it's going to be really fun.

05:24 I'm really excited.

05:24 Yeah, I think in the early days, if it's the three of you starting a company,

05:29 you all have to do so many things that you just, you are unprepared and unqualified to do, right?

05:36 Like, you're like, all right, who's going to do accounting and taxes?

05:38 You're like, wait a minute.

05:39 Nobody knows about this.

05:41 Who's going to do, you know, deployment?

05:43 Like, well, we've never deployed it.

05:44 Like, you know, there's just so many things, but now you can find somebody who's like,

05:48 say, super good at DevOps and you don't have to worry about that.

05:51 And that's a big transition, right?

05:52 Yeah, it is.

05:53 And you kind of, you know, whenever you're just starting out, you can kind of get away with the mess, right?

05:57 Like if you mess up a deploy, it's like, well, only 10 people were using your site.

06:01 So no one noticed, you know, you have little things like that, that really help.

06:05 And same thing.

06:06 I mean, you mentioned like accountant hiring and like, you're not doing so much of it.

06:10 So if you're not world-class at it, you know, you can kind of get away with it.

06:13 But at the heart of it, like you got to build that product that really connects to people.

06:18 And if you do that well in the beginning, the rest kind of comes and you can get amazing people

06:23 who know all the, know how to build a great deployment system, know how to run taxes in 50 different states,

06:28 know how to do all this other stuff.

06:30 So yeah, it changes.

06:31 Yeah, yeah, I'm sure.

06:32 All right, cool.

06:33 So we talked a little bit about it.

06:35 You work at Zapier and stuff, but you know, I'm sure most folks listening

06:39 have more or less heard of it, but maybe they don't really know.

06:42 They're like, oh, I think that's some kind of online startup thing.

06:45 Tell us what is Zapier.

06:46 Yeah, so you can think of Zapier as sort of the glue between all the different SaaS services out there.

06:52 So if you use Gmail, if you use, especially like Google Sheets, if you use things like Dropbox or Slack,

06:59 like all these different services that people use on a day-to-day basis,

07:02 they have data in them and you want to trigger on that data and do other things and other services.

07:09 Zapier is that like mechanism that does that, which is very much, you know,

07:13 I think of it a little bit like, you remember all the, and I guess there still exists,

07:17 some of the languages like Scratch who are kind of those visual programming

07:21 and they never really caught on, but they kind of were like, oh, we're going to give the power of automation

07:25 and the power of coding to the masses, right?

07:29 Yeah.

07:29 It was always, it's an awesome goal, but it was always really difficult.

07:33 And I think we're kind of doing that and kind of coming in through the back door.

07:37 So you can do these cool things where when you, you know, star a message in Slack,

07:42 you can add it to your to-do app and you can choose one of, you know, a dozen to-do apps that are supported on Zapier.

07:49 And then you can think, oh, well, if I say like super important in that message,

07:54 I had an extra filter and I send a message to this room in Slack or that room in Slack,

07:59 maybe, or I send, you know, my colleague a DM, like you can start to build these like workflows.

08:05 And that's really the power of Zapier is that you can kind of start to string these actions together

08:10 and do some really complex stuff that normally couldn't do unless you had a fleet of engineers

08:15 to like custom build this bespoke software.

08:18 This is interesting.

08:18 So, you know, you talked about Scratch and some of these other like edgy block

08:22 type of application.

08:23 Right.

08:24 I'm not going to call it application, but programming frameworks that are these visual things.

08:28 Right.

08:28 And I think the idea of those is not terrible.

08:32 I think the problem is they work at too low of level.

08:34 Right.

08:35 They're like, you issue a print statement, you do a for loop.

08:38 You, I mean, like they literally work almost like line by line as you would with text.

08:42 Whereas what you guys have is like an email came into Gmail and that's like an integration with an API

08:48 and OAuth and like tons of other stuff.

08:50 And then, you know, you've condensed it down to the right size of the draggy droppy thing.

08:55 You know what I mean?

08:56 So it's like, there's a few blocks, not 50,000 blocks, which makes no sense.

09:00 Yeah.

09:00 And you get away from the idea of like trying to teach people control structures.

09:05 Right.

09:05 And how data structures work and like all this stuff is like really heady stuff.

09:12 Like once you get it, it's fine and it doesn't bother you.

09:15 But like until you get it, it's another world, right?

09:18 It's completely foreign.

09:19 And we avoid that.

09:21 And we try to keep it just simple to like, oh, I get a new email and I add a message to Slack.

09:27 Like, got it.

09:27 Like I can connect those two together.

09:29 And that's always like the beginning point.

09:31 And then I think the really powerful stuff comes where you have a team in there

09:36 and they're like building zaps together or they're adding their own like little flavors of,

09:40 oh, I want an extra, you know, filter here.

09:43 Only do it if this matches or add an extra step to this and don't just send it to Slack,

09:48 you know, send it to my ToDo app or send it to this, send it to my CRM.

09:52 Like we have lots of business folks and marketers who are doing all this like cool stuff as well.

09:56 So this is, this is not for like, only for engineers.

09:59 We do have some, some of my favorite stuff to do is like you can literally embed Python

10:03 into these workflows, which of course like is very topical for this podcast.

10:07 I love it.

10:08 But this is for like marketers as well.

10:10 So they can build these things.

10:11 This is for real estate agents who have, you know, form software and they want to like trigger off

10:17 of people submitting that and get e-signatures and like all of this stuff

10:20 is like in this ecosystem.

10:22 So it's not just engineers or devs that use this.

10:25 It's kind of regular folks, right?

10:27 People who are just trying to get work done, right?

10:29 It's cool.

10:30 And you can bring, bring the level down a little bit.

10:32 You don't have to be, you know, if you're kind of good with computers, but you're not a developer,

10:37 right?

10:37 Like this could be something you could maybe really make what, maybe like a developer would go,

10:43 I'm just going to write an app that'll just call this API than this and write.

10:45 But like, there's so many people who, even if they could write that, they're not going to say,

10:50 deploy it as some kind of background process in like on Linux and put it in the cloud.

10:55 Like they're just not going to do that.

10:56 Yeah.

10:56 So yeah, this is great.

10:57 Yeah.

10:58 Yeah.

10:58 And you think about, that's a great example.

11:00 It's like we often get engineers or devs who are like, I really don't want to build

11:04 that thing you just asked me to build.

11:05 I'm just going to use Zapier because if you think of like, oh, I got to pull this service

11:09 and now I have to come up with this stateful store of which things I've seen

11:12 and haven't seen, I got to deploy it to like, I don't know, maybe we'll put it on a Heroku.

11:17 Maybe we'll like, being a DevOps person, we'll get it still, you know, like it just starts to spider

11:21 into this whole thing.

11:22 Whereas like with Zap, you just like create a trigger, create this, and then you can hand it off

11:26 to the marketer.

11:27 You can hand it off to your colleague and they can just dig into it.

11:30 So that's, that's a powerful pattern.

11:31 Exactly.

11:32 It's also like, I think this is a good idea to build this little integration,

11:36 but I don't want another puppy that I have to take for a walk and maintain.

11:40 I didn't want to, I don't want a thing to care for.

11:42 So if I build it myself, it's my problem.

11:44 If I build it here and hand it off, like you guys, that could be your problem.

11:47 You'll keep it running if the API changes.

11:49 Yeah, we do.

11:50 And there's a lot of things that change in the API world that we stay on top of.

11:53 So it just kind of, it just takes that whole layer of like what's going on away

11:58 because we have, and I'm sure we'll get into, we have tons of tooling that like watches this.

12:03 Yeah, yeah, yeah.

12:03 It's really cool.

12:04 I definitely want to talk about that because, you know, you have thousands

12:07 of integrations and these integrations integrate together, which is even more

12:10 like, you know, combinatorially insane.

12:12 So that's going to be fun.

12:14 And I want to talk about that, but let's just talk briefly about this at a high level

12:17 while we're talking about Zapier.

12:19 You know, a lot of folks, they get up and they get in the car and they drive

12:23 through angry drivers and traffic for 30 minutes.

12:26 Right.

12:26 And they all have to live around their company and they go to work, right?

12:29 But as developers and tech companies and things like that, right?

12:33 Like a lot of times, even if you're sitting next to somebody, you're working

12:36 in asynchronous modes, you're making a GitHub issue, you're doing a PR, you're throwing a message

12:41 in Slack, you're like, there's no reason you've got to sit next to the person, right?

12:44 So that's kind of what you guys decided as well.

12:46 And with your 200 people, you're more or less 100% remote, right?

12:50 That's correct.

12:50 Yeah, 100% remote.

12:51 We don't have a single office.

12:53 We're not geographically bound.

12:55 We have people in, I'm going to say, I think it's like 18 or 19 different countries

12:59 right now that work for Zapier.

13:01 And we've been that since the very beginning.

13:03 So we started 100% distributed.

13:05 We've been distributed throughout the entire time.

13:08 So every single person, it's funny, like basically everyone we've hired,

13:12 we had not met in person, right?

13:15 There's like a few exceptions to that rule, but like everyone we've hired

13:18 has been just like a Zoom call or a Skype call to kind of do the interview

13:22 or interacting over email or interacting over, you know, text communications in some way,

13:28 which really mimics the way that most of us work anyways, right?

13:31 That's been just kind of a core part of Zapier since day one.

13:36 I think that's really, it's pretty exciting, right?

13:38 I mean, there's certainly drawbacks to working from home, right?

13:41 You know, people can get a little bit lonely.

13:43 You have to be a little more self-directed, I think, because some people see

13:48 working from home means I have no boss.

13:49 No boss means I can just do whatever the heck I want.

13:52 Like if I want to just watch football for the second half of the day, I'm doing that.

13:55 Like those types of people don't thrive, but I feel like a lot of developers

14:00 and technical folks are pretty driven anyway.

14:03 Yeah.

14:03 And if they have a place for a proper outlet that's really, they feel like they can contribute to,

14:07 like it's pretty easy to work from home, honestly, as a developer.

14:10 Yeah, I think so too.

14:11 And, you know, everything you're, everything, all the artifacts you're creating

14:15 in a tech company, like you're writing code, you're fixing bugs, you're working on features.

14:20 It's all, like the output's like all there.

14:22 You can see it either shows up in the repo or it doesn't.

14:25 It either lands on the site or it doesn't.

14:27 And so you can, it's not that complicated to like track the work.

14:32 I often get folks who are, tend to be a little bit more, especially in the early days of Zap,

14:36 you're very skeptical about the idea of like remote.

14:39 Like how do you know people are doing their work?

14:41 And how do you, like that has not played out to be the biggest problem for us.

14:46 Many of the problems around, and you mentioned like these are trade-offs,

14:49 like a lot of the challenges you get into are just challenges that you hit in a normal company.

14:55 You just hit them sooner, right?

14:57 Like that communication challenge doesn't happen, you know, when you're in the hundreds

15:01 of like employees.

15:02 It happens when you're in the dozens of employees, right?

15:05 Where everybody's off doing all this like cool stuff and it starts to get hard

15:09 to coordinate, right?

15:10 So we had to invest in coordination and communication a lot earlier in the org

15:15 because you couldn't do that implicitly by picking it up around the water cooler,

15:18 so to say, right?

15:19 So that's always been a big focus for us in the early days was like what's breaking now

15:24 that probably wouldn't break until, you know, we were double or triple the size

15:28 in a normal company.

15:30 So what are some of the tools and things you put in place?

15:33 You know, to me, it feels like distributed teams 2018 is 10 times easier

15:37 than distributed teams 2006, 2008, but still like there's a lot of tools

15:43 to pick from.

15:43 Like what's important part of that whole structure there?

15:46 I mean, they're the same tools I think a lot of companies are using.

15:49 I mean, they're the GitHubs, right?

15:50 They're the Slacks.

15:51 We use Zoom for like calls.

15:53 We do so many video calls, constantly hopping on calls to like chat over stuff.

15:58 We've even built a couple like internal tools that are custom to us.

16:03 So we've built an internal tool called Async modeled around the idea of asynchronous communication,

16:08 right?

16:08 Where, you know, you look at something like Slack, it's very synchronous, right?

16:12 Like you drop information in, it scrolls away and it's gone.

16:15 It's not gone, but it's like basically gone.

16:17 It feels ephemeral, right?

16:19 It does definitely feel ephemeral, right?

16:21 It's hard to keep permanent actions that you need to do like somewhere in your Slack history.

16:25 Yeah, that's a poor, it's a poor idea.

16:27 You want to push that out.

16:28 So what we've done is created this tool called Async and you can kind of think of it

16:32 like WordPress meets Reddit meets Twitter kind of a thing.

16:37 So it's basically a blog mechanism that has some structured ways to describe, for example,

16:43 we require each post have like a TLDR, right?

16:46 Like where you can, under a hundred words or less, give or take, kind of like summarize

16:52 what your decision is that's enclosed in that Async post.

16:56 Man, I love it.

16:56 That's such a good idea because then you can tell do you need, how much attention

17:01 you need to pay to it and like just an email, like that would be a great

17:04 inter-office communication way because I mean, I personally get so much email

17:09 that I'll periodically have to take, you know, eight hour days and just answer email

17:13 just so it doesn't get too bad.

17:14 And it's like, those were those little short vets?

17:17 Like that would help sometimes.

17:18 That's awesome.

17:19 I like it.

17:19 Yeah, it really would.

17:20 And we've also added things like depending on how many people view it or comment or like it

17:25 or interact with it, it gets hotter.

17:27 Like it's a very like Reddit style or Hacker News style thing.

17:31 Yeah, yeah.

17:31 Which helps like surface just important information.

17:34 So earlier this year, Wade posted a whole bunch of like essays on like what we're trying

17:38 to do in 2019 and everybody was interacting and like diving in and chatting about them

17:42 and they'd popped right to the top.

17:44 So you don't even have to like really work super hard to get that information.

17:48 You just kind of hit the homepage of Async and you get a flavor of what's going on

17:53 in the company.

17:53 You can read those TLDRs and you get a good idea.

17:55 You can click in, you can go deeper if you want, you can interact.

17:59 So this is something, as an example of something we built internally for this,

18:03 but it's not crazy software, right?

18:05 It's not super complicated.

18:06 This is something that in conjunction with other tools like the classic,

18:10 like, yeah, like we use lots of Google apps and then the GitHub and the Elysians

18:14 and the Jira's and all that stuff.

18:15 So we do do some custom stuff.

18:17 Yeah, that sounds really awesome and I think it's great.

18:19 You know, I've worked for 10 years for a company that was remote and we had people

18:24 all over the world and there are definitely trade-offs, but I really felt like

18:28 the people I was working with were some of the very best people because they were chosen

18:33 from the population of the world, not from the population of the town I happen to be in,

18:39 which maybe is a small town or whatever, right?

18:41 Like, how has that either been like a positive or negative for you?

18:44 I've seen like a lot of tension between some companies are like, oh, we have to have somebody local

18:48 because it's super important we have this like sort of control or like view over them

18:53 and interaction with them, but I don't know that that's always the best.

18:56 No, I don't think so.

18:57 We believe, and we've seen it in reality, we believe that like talent is like evenly distributed

19:04 throughout the world, but opportunity isn't, right?

19:06 So in a large way, we see this as an incredible opportunity that even just like talk about America,

19:11 a small town in America, right?

19:13 And you have an amazing engineer that lives there.

19:15 They live there because they have family and it's a great cost of life and they love where they're at.

19:20 The opportunities as an engineer are likely pretty slim.

19:24 It's one of those opportunities where those amazing folks, those really talented folks

19:29 can actually contribute at the level that they can contribute at.

19:32 That's pretty special.

19:34 Like that's really, really hard to do when everybody's geographically located

19:37 and you require someone sit there in an office that just wouldn't really like scale out.

19:42 So for us, we think of it as something that is good for business, but also just really good

19:48 for people as well.

19:49 It's just a better way to work.

19:50 It's a better way to live.

19:51 You can live where you want.

19:52 You can have a better, like just look at like the cost of living for your family,

19:56 for everything.

19:57 It's just so much better in every way.

20:00 And you just have to like work through the little challenges that pop up

20:04 in a remote environment and it really helps to be 100% distributed.

20:07 I think when you get that mix thing, yeah, it gets a little trickier.

20:11 The mix is tricky because you got the insiders who are at the office and they don't necessarily

20:16 fully trust the remote workers, but maybe the remote workers are really talented

20:20 so they have to work with them.

20:21 But there's always these little undercurrents of weirdness.

20:24 Like I've been in those situations as well.

20:28 This portion of Talk Python to me is brought to you by Linode.

20:31 Are you looking for hosting that's fast, simple, and incredibly affordable?

20:35 Well, look past that bookstore and check out Linode at talkpython.fm/Linode.

20:40 That's L-I-N-O-D-E.

20:42 Plans start at just $5 a month for a dedicated server with a gig of RAM.

20:46 They have 10 data centers across the globe, so no matter where you are or where your users are,

20:51 there's a data center for you.

20:52 Whether you want to run a Python web app, host a private Git server, or just a file server,

20:57 you'll get native SSDs on all the machines, a newly upgraded 200 gigabit network,

21:02 24-7 friendly support even on holidays, and a seven-day money-back guarantee.

21:07 Need a little help with your infrastructure?

21:08 They even offer professional services to help you with architecture, migrations,

21:13 and more.

21:13 Do you want a dedicated server for free for the next four months?

21:16 Just visit talkpython.fm/Linode.

21:19 You were just talking about having a company retreat in New Orleans, and the company I worked for

21:25 before when I was working remotely before I started my own.

21:27 Certainly, we had the same challenges of how do you connect, and so having some kind

21:33 of yearly or bi-yearly get-together physically, and also you talked about Zoom.

21:37 The video aspect of the Zoom and Skype actually makes a big difference, I think.

21:42 It really does, and the internet connections are getting so good now that that video connection

21:47 can feel very real, can feel very, like you can interact, you can see someone's body language,

21:51 so you do get that.

21:52 Yeah, you and I are talking, and it's almost as if we're sitting across the table

21:56 from each other.

21:56 It basically is good.

21:57 Yeah, it works really well.

21:58 So we found it to be exceptionally good.

22:02 Yes, I do think having the visual interaction is important, like video calls,

22:06 but then periodically these actual get-togethers, right?

22:09 Yes, the retreats.

22:11 Yeah, like these retreats, I always look forward to them because it was like

22:15 these people I really liked and I would have more of a work.

22:17 Like you wouldn't hang out on Skype.

22:19 You'd be like, hey, I'm just, what are you up to, man?

22:21 Let's hang out on Zoom, right?

22:22 Like that's kind of weird, but you would have these meetings.

22:24 So this is your chance to be more friendly with like your coworkers, right?

22:28 So I think it's a good combo.

22:29 I think it works really well.

22:30 Yeah, I do too.

22:31 And when it comes to the retreats, we've done those since the early days.

22:35 I remember our first retreat was seven people and it's grown up to 185 or 190 people.

22:42 I think we're at the last retreat, which brings about a whole set of, you know,

22:46 unique logistics challenge, but we found them to be really, really useful

22:50 to kind of help set the stage for the stuff that we want to talk about.

22:53 So you get that, you know, one of the cool things is, at least for me, is when you're in an office,

22:58 you kind of are pretty much like flatline energy level.

23:01 Like it's just kind of day in, day out.

23:03 It's what you do every day.

23:04 When you meet the team, you get them together.

23:06 For us, it's like twice a year.

23:08 That energy level's through the roof.

23:09 Everyone's really excited to see each other.

23:11 You're diving into things.

23:12 You're hacking on stuff.

23:14 You're going, like there's some really cool stuff that comes out as a result.

23:17 So you get a little bit of the best of both worlds where you get to go back.

23:21 You get to be in your space.

23:22 You get to like zone in.

23:24 You get to like have your kind of flow state, but you also get to have that really high energy

23:29 connecting with people, spending time with them, which works really well.

23:33 And we try to emulate, some of that serendipity things.

23:36 Like we've used, there's this product called Donut for Slack and it will like pair you up

23:41 with folks.

23:42 So we have these like two by, two people like pairings that happen every week,

23:46 three people pairings that happen every week.

23:48 And they just let you get together and just chat about whatever.

23:51 And you get matched up with random folks on the team.

23:53 So you get a little bit of that as well.

23:55 Oh, nice.

23:56 And throw in little jokes or just stuff just for more personal things.

23:59 Yeah, yeah, you definitely do.

24:00 And you get to hear a little bit about what people are up to and like, you know,

24:03 hear about their family and hear about this or hear about that.

24:06 And it's those sorts of unstructured conversations that kind of bring out

24:10 that humanness, which, you know, you do lose a little bit of when you're remote.

24:14 So we often have like advice for folks to try to plan one day of your week

24:18 where you go to a coffee shop or you go somewhere just to get out and move around.

24:23 And, you know, some people don't need that.

24:25 Some people need more of that.

24:26 And just be aware of that for yourself because you're not often going to get that directly

24:30 from working on Slack or something like that all day.

24:32 I think that's great advice.

24:33 Like people can go kind of stir crazy.

24:35 Like I work 100% from my home office or somewhere else, right?

24:38 Everyone I work with is some kind of contractor or author or something who's also not

24:43 in any form of office with me.

24:44 I know all the coffee shops around, especially in the summer, you know, and sometimes hit the libraries,

24:50 but just getting out that makes a big difference.

24:52 So I guess one more question that people out there might find interesting

24:57 and maybe they will more so after what we talk about in just a moment, but it seems like

25:01 it fits better here is they're out there listening.

25:04 They're like, you know, I have this Java thing I'm doing and I'm not really super happy.

25:08 I love a Python job.

25:09 You guys are pretty much doing tons of Python and they could work from where they are.

25:14 Are you hiring?

25:15 Yeah, we're doing a lot of hiring in 2019.

25:17 We've been really fortunate that, you know, we've talked about this publicly.

25:20 Zapier has been profitable for years and years and we've been growing at a really respectable pace

25:25 and we have a lot of ambitions for 2019.

25:28 A lot of cool stuff we got to build.

25:30 So yeah, we're 100% hiring.

25:32 We're looking for folks who have Python background, have JavaScript background.

25:36 We're going to be hiring slightly different, you know, on the manager track,

25:39 we're going to be hiring folks on that side as well, which we kind of think of

25:42 as like a different skill set.

25:43 We're also hiring like security folks.

25:45 We're hiring like test automation, you name it.

25:47 We're hiring all the things in 2019.

25:50 Lots of cool stuff.

25:51 All right.

25:51 Awesome.

25:52 Well, people can, I'm sure there's a career page at zapier.com, right?

25:55 Slash jobs.

25:56 Yeah.

25:56 Zapier.com slash jobs and we got them all on there.

25:59 All right.

25:59 Awesome.

25:59 Nice.

26:00 All right.

26:00 So let's dig into some of the technology because I think there's two sides of things

26:04 that are super interesting here.

26:05 One, you've got your infrastructure and just the amount of people running jobs

26:11 and triggered actions on your site is really, really interesting.

26:15 There's so much happening there.

26:16 And then also you're calling thousands of APIs and these APIs are not totally stable

26:22 and all sorts of stuff.

26:23 So let's dig into that a little bit.

26:25 Like maybe just tell us real quick, like what kind of tech is involved and things along that line.

26:29 We've tried to keep it pretty simple because we have plenty of complexity

26:31 when it comes out on the other side with talking to APIs and some of the

26:36 interesting combinations users make.

26:38 So for us, we're Python.

26:40 I mentioned earlier, we're Django based.

26:41 So we're still using a lot of the ORM, a lot of the kind of standard tooling

26:45 that comes with Django, a lot of the caching bits, a lot of the session bits,

26:48 all that stuff we still use really, really heavily.

26:51 What kind of database back in are you guys talking to with that ORM?

26:54 Yeah, yeah.

26:54 We're talking to MySQL right now.

26:56 We're starting to kind of shard off and we've done and not sharding in the sense of like

27:00 sharding over consistent sharding over IDs or anything like that, but like functional sharding

27:05 where we try to extract this service from kind of the monolith and say, okay, well,

27:09 now this is an opportunity.

27:10 Maybe we'll use a more specialized database.

27:12 Either maybe we'll stay relational and we'll use Postgres or upgrade the MySQL

27:15 to something a little different or maybe we'll use some sort of NoSQL or specialized

27:19 read and write patterns that make that use case even better.

27:23 So for that, we're generally, I mean, we're looking at like mostly like vanilla Python

27:28 like on the back end, the workflow engine side, right?

27:31 which is probably, which is probably, you think of like that big behemoth

27:34 that's running in the background doing all this like crazy stuff, talking to all these APIs,

27:38 kind of your little robots running around the internet and that's all like Python

27:42 and that's usually doing a lot of, I mean, you think of like celery, we use a ton of celery,

27:47 right?

27:47 We use a ton of that and we use RabbitMQ for a lot of the message passing between that.

27:52 We're still using Django on the HTTP front, I mentioned.

27:56 we use a lot of just pretty like vanilla like ingest for like webhooks.

28:01 Like I know we do a lot of pulling, which is a lot of back end stuff, but we also pull in a lot of webhooks.

28:06 So this is great when services send us those webhooks and let us know about things changing.

28:10 Right.

28:10 Webhooks have to be basically the best possible outcome, right?

28:14 Because something like an email or an action happens and that service tells you

28:19 rather than just you got anything for me, you got anything for me, you got anything for me.

28:22 Yeah, yeah, yeah.

28:23 That's, and that's, it's huge.

28:24 We did a big analysis and we actually even shared and try to get more partners

28:28 on board with like webhooks and what we call rest hooks, which is just a way

28:31 to subscribe to webhooks.

28:33 Because I think for a while, especially in the early days of webhooks, it was like,

28:36 hey user, please copy this URL into this other service.

28:40 And then, whereas like now that we have rest APIs, like we can just post

28:43 to a subscription or a webhook endpoint, it'll create a webhook and send you those events

28:48 and then you can tear it down.

28:49 So we've done a lot there and we've done, and we did a bunch of analysis

28:53 in the early days and it's something like 66 times more efficient, like,

28:56 because like on average, you're asking hundreds of times for new data and there's nothing there.

29:02 And then the one time there is, right?

29:03 Yeah.

29:04 Just super, super wasteful.

29:05 So we try to push towards like webhooks and event-driven sorts of things

29:09 as much as we can.

29:10 Yes.

29:10 So you go to these partners, you're like, we would like to not call you 100,000 times a day.

29:14 Yes.

29:15 Could we make it so we just call you 100 or 50 or 1,000 or whatever the right number is?

29:19 Yeah.

29:19 Yep, exactly.

29:20 And some partners are really like into that and they have like some services

29:24 and some endpoints set up to kind of create those webhooks and subscriptions

29:29 and some don't.

29:29 And in those cases, we do fall back kind of that polling architecture.

29:34 But yeah, we really like it when partners do add those.

29:38 I'm sure.

29:38 So I feel like this asynchronous, cueing, message passing, message bust type of world

29:44 must be pretty important for you guys.

29:46 Like, you know, a new trigger has come in from like a new Gmail, but then you want to talk

29:51 to Dropbox.

29:51 Like, you don't necessarily want to block that up.

29:53 You're like, okay, well, kick off a message.

29:54 Like, you need to take this thing from Gmail and move it over to Dropbox

29:57 when you have a moment.

29:58 All right.

29:58 And just sort of pass it down the flow.

30:00 You want to talk a little bit about when that happens and like how that helps?

30:03 Yeah.

30:03 I mean, if you go into Zapier and you go into build a Zap, like a workflow,

30:07 you'll see multiple steps.

30:08 Maybe it's a, I'll use a starring, a message in Slack, and then you add it to,

30:13 let's say, Todoist, right?

30:14 Those are two distinct steps that we've broken apart on the back end.

30:17 And on the back end, they're stored in kind of a self-relational, sort of a tree.

30:22 You can kind of think of it as a tree.

30:24 We try not to, there's no real ways to do cyclical, right?

30:29 Sorts of like things because you get into, now you're into control flow stuff

30:33 and that's, you know, a little bit crazy.

30:34 For most users.

30:35 Recursion with drag and drop, probably not.

30:37 Yeah, yeah.

30:37 It gets tricky.

30:38 So we avoid that.

30:39 So you can kind of think of this as like, that's a two-step Zap, Slack, Todoist,

30:44 and we just pull that data out of Slack, we pass it on to the next one and we have a whole like

30:50 templating system that kind of like passes in that data into the inputs of the next step

30:54 in a particular way and users can define that however they want, which is kind of this

30:58 cool customizability.

30:59 So you can think of that task would say, you know, the message and maybe the time

31:04 and who like actually made that message in Slack and you put that all in the description

31:09 field for Todoist action, which would be maybe create a task or something

31:14 or a to-do item.

31:15 So that would trigger on and push down to a totally separate step that's going through

31:20 RabbitMQ in that case.

31:22 And then once that, you call into Todoist and you're complete with that,

31:26 you get a 200, everything's hunky-dory.

31:28 You look for any other children you have and in that case, you have nothing

31:31 and you insert no child task and you're done.

31:34 This app is finished running.

31:36 That's pretty much kind of the workflow and you can see how that would expand

31:39 and you could have dozens of steps and it walks down through a dozen steps

31:42 until it gets to the last one.

31:43 Nothing to do, so it's finished.

31:45 Yeah, yeah.

31:45 How do you deal with like the poison message equivalent, right?

31:49 So like the third step is I want to call the Todoist API and I'm a big fan

31:53 of Todoist by the way, but you know, maybe that day the Todoist API is down

31:57 for like two hours and you're like, well, do you like throw it back in the queue?

32:02 Do you put it in some other place like we're going to get to this maybe when things are better?

32:05 Because I'm sure you're like, there's probably an API permanently at any moment in the day

32:11 there's one of your APIs that is like not behaving that great, right?

32:14 That you got to interact with.

32:15 There's so many.

32:16 Yeah, definitely.

32:16 And there's a couple ways that we handle that.

32:19 So we do a ton, since we do so many requests and we log all these requests

32:23 and we track all these requests just for understanding the health of APIs.

32:27 When we see one starting to fail, we see if it's failing for a lot of users

32:31 or if it's just one user.

32:32 If it's just one user, that's an error.

32:34 Maybe, you know, something about the to-do item they were adding is actually invalid, right?

32:40 Right.

32:40 They deleted the project they're trying to add it to or whatever, right?

32:43 Yeah, exactly.

32:44 It could be anything like that and that's a user-specific thing.

32:47 So that would pop up as an error.

32:49 We'd email them, hey, heads up, something funny is going on with your Zap.

32:51 Click here to kind of learn more.

32:53 But if it's something that's happening across a high cardinality of users,

32:57 a bunch of different users are hitting this, now we know, okay, something looks

33:01 a little fishy here.

33:02 And at that point, we'll start to hold things back and periodically like kind of let them through

33:07 to see, do we have things going from, you know, maybe a 403, we're getting like

33:11 nonstop 403s, someone shipped a bug and da-da-da.

33:14 It's a mess.

33:15 And we'll retry those until we start to see 200s and then we'll let the rest of them go.

33:20 We'll try to trickle them out until we see them clear up.

33:23 And that's a really common pattern for us.

33:25 And we have, I mean, we've invested a lot in just trying to understand when APIs go down

33:30 because I joke that we're like an edge case catching machine for the internet

33:33 and anything that will go wrong, we will find out about it in some way.

33:37 So we have to have all this like really intelligent monitoring to kind of stop that

33:41 from happening, really keep it from impacting users or at least like give them the awareness

33:46 that, hey, heads up, there's some internet stuff going on.

33:50 you know, maybe AWS is having some sort of an outage and look at all these services

33:53 that are 503.

33:54 And we can kind of tie that together.

33:56 Yeah.

33:56 Cross your fingers and hope that a US East 1A doesn't go down.

34:01 Right.

34:01 Yeah.

34:02 Like how much stuff runs on the Virginia data center of AWS.

34:05 So what?

34:06 Yeah.

34:06 A lot of ours does.

34:07 And we can often tell when, like if we see any funniness with AWS, we can always just go look

34:12 at our like partner dash partner status health dashboard and see if anyone else

34:18 is having issues.

34:19 And if it's just us, it's like, okay, so we must be doing something funny.

34:22 But if everybody's like having issues, we're like, okay, so Amazon is having an issue

34:26 and they haven't quite updated their status page yet.

34:28 Right.

34:28 Exactly.

34:29 You're like a little bit of a canary on that.

34:31 Yeah, definitely.

34:32 Interesting.

34:33 This portion of Talk Python is brought to you by Ting.

34:38 Let me tell you about Ting, a new mobile service available in the US that's targeted developers

34:43 and other technically savvy folks.

34:45 First of all, their average customer only pays $23 a month, but they're no discount provider.

34:50 Their service runs over T-Mobile's and Sprint's fast nationwide network.

34:54 If you don't use that much data because you're usually on Wi-Fi like many of you are,

34:58 then Ting will save you a ton of cash.

35:00 But don't worry, you can still use as much data as you like for just $10 per gig.

35:04 One mobile feature I use all the time is tethering.

35:06 And with Ting, you get unlimited tethering at the same data rate with your account.

35:11 $6 a month for a phone line, $10 a gig, $3 a month for text if you usually chat

35:16 over iMessage or WhatsApp.

35:17 Think about it, no contracts and super clear and fair billing.

35:21 Visit python.ting.com, that's python.ting.com, and check out their savings calculator.

35:27 Enter your usage and see exactly what you'd pay.

35:30 Use that link and you'll get a $25 credit to try them as well.

35:34 That's python.ting.com or just click the link in the show notes.

35:39 I guess it sounds like you host a lot of your stuff at AWS and you probably don't have

35:43 like a big in-house data center.

35:45 Like that made more sense if you have a big office and like part of it could be the data center.

35:49 But if you have no office, like who's going to offer up their bedroom to get fiber

35:53 and like cooling and like, all right, that just doesn't make sense.

35:56 So are you running AWS?

35:57 Where's your services hosted?

35:59 We're predominantly AWS.

36:00 We have a lot US East, of course, but we also have a couple things that fail over

36:04 to multiple different regions.

36:05 So yeah, I mean, we've just leaned into AWS since I think the early, I think maybe we did

36:10 like Linode or Linode back in the early days for a hot minute and then that went over

36:14 to AWS since then.

36:16 We try to use some of the managed services where we can.

36:19 So you're looking at the RDSs and the Elastic Caches and of course, like the ELBs and ALBs

36:25 that exist out there, tons of S3.

36:27 So we try to lean on some of the managed services where we can to just knock down

36:31 the amount of like deep, intricate knowledge we have to have about like Postgres

36:35 or MySQL or whatever, right?

36:37 Yeah, let them do the operational bits of that.

36:39 Interesting.

36:39 So to me, it feels like the cloud, especially AWS and Azure and Google, those three clouds

36:46 feel like that's the new lock-in, right?

36:49 And I don't necessarily mean that in a bad way, right?

36:52 Like when you take full advantage of a platform, you get all of the features

36:56 of it, right?

36:57 Like you could treat AWS as a fancy Linux hosting environment, right?

37:03 Or you could use all the databases and all the files.

37:06 all of the APIs and really get a lot more out of it, but then you're kind of permanently

37:10 stuck to it.

37:11 Like how do you guys think about like lock-in and cloud and all that?

37:14 We kind of like tread the line a little bit in between.

37:16 So all of our workloads that you think of or a large majority of our workloads

37:20 that you think of just like running tasks and performing these actions for users,

37:25 like they're running pretty bare bones like EC2 instances.

37:28 We found that that generally has the best performance characteristics.

37:33 we have the most kind of control over it.

37:35 So in that case, like it's not really like locked in a whole lot.

37:38 We're starting to lean into things that are like open source orchestration frameworks,

37:42 the classic Kubernetes like transition that I think a lot of companies are like trying to do now.

37:46 Again, we're talking about pretty open source standard sort of a thing, which we really like,

37:51 but we're also not afraid that on an occasion using some of these specialized services,

37:56 if it makes our lives a lot better, we'll use it.

37:59 Like RDS is a great example of kind of them splitting the middle.

38:02 like it's standard MySQL.

38:04 So in the future, if you decide you want to like swap out, it's not the end of the world,

38:08 but you don't have to worry about it in the meantime, right?

38:11 You get all the goodies kind of a little bit, certainly not for free, but with less work

38:16 on your side and I can focus on the product.

38:18 So we've definitely leaned into that a lot.

38:20 We haven't done a whole lot of like hyper specialized stuff.

38:23 Maybe the only thing I think that we do a lot of is use a ton of AWS Lambda.

38:28 In fact, like back in the early days, we built our own version of AWS Lambda.

38:35 It was, I forget what we called it, but where we could more securely run

38:39 a bunch of partner provided scripts and user provided scripts.

38:44 That way they could do this kind of Python and his app thing, or they could like

38:48 stand up their own partner apps.

38:49 So this allows partners to kind of deploy code to us, to talk to APIs and integrate with

38:55 any of our, you know, 1400 different apps that was running on our own little flavor.

39:00 And then as soon as AWS came along, it was like, actually, we got this and it's way better.

39:05 We immediately swapped it all over to that as fast as we could.

39:09 So we do use things like that, but even that has been commoditized by, to a large degree.

39:14 I mean, both Google and Microsoft's cloud offerings have, you know, the serverless

39:19 sort of vibe going on too.

39:21 So it's kind of interesting.

39:23 we've tried to balance it on the whole to where we can focus more time on the product,

39:28 but not go like too hardcore, use really specialized stuff.

39:32 Right, right.

39:33 It's that kind of stuff that lets you do what you're doing on a 200 person team

39:38 instead of a 500 or a thousand person team, right?

39:40 There's not like a fleet of database engineers and other infrastructure people

39:45 and yeah, right.

39:46 It's like, they're really big benefits to this.

39:48 And I kind of do the same thing.

39:49 Like there's certain things where like, okay, that's really not available elsewhere

39:52 or so much more work.

39:54 I'm going to just, you know, plug into that API or this API, but yeah, definitely.

39:57 Yeah.

39:58 So you did talk about calling a few APIs and if you're doing that with Python,

40:05 like the most common way to do that probably is the request, the module by Kenneth,

40:09 right?

40:09 Right.

40:09 Are you guys using requests for all this?

40:11 How's this work?

40:12 Yeah.

40:12 Yeah.

40:12 We use requests, but we use a bunch of our own.

40:15 I mean, the great thing about requests is how like extensible it is.

40:18 Like you can swap in and out your own HTTP adapters.

40:21 And then from there, you can swap out almost everything.

40:23 So we have custom DNS resolvers.

40:26 We have custom request adapters or HTTP adapters.

40:30 We have custom like response and request types that all kind of encapsulate

40:34 some of the behaviors and the logging and the monitoring that we want to have.

40:38 And we have like different classes.

40:39 Give us some concrete examples.

40:41 Like, so you have a custom DNS thing.

40:42 Is that like, so you can do better caching and not like, like make the calls

40:46 faster?

40:46 Like what's the idea there?

40:47 The idea there is if you, for us, like we want to be able to have a lot of kind

40:53 of control over what DNS services we have with which like adapter we're using.

40:59 For example, we use requests for internal communications, which is like a different

41:02 DNS sort of provider.

41:04 And then when it comes to like user outbound user stuff, we would want to use

41:09 either like the Google 8.8.8.8 or maybe like open DNS or even maybe some sort of a

41:15 custom thing for a user.

41:16 So in those cases, we want to be able to swap that out.

41:19 And that's a fairly low level.

41:21 I mean, it's kind of automagically done for you a lot of times.

41:24 But if you want to control that, you have to kind of start digging into the HTTP adapters

41:28 and intercept some of the connections that are opening and kind of like all this other stuff.

41:32 And you have the opportunity to do that with requests.

41:35 And, you know, a really specific example, we talked a lot about this monitoring bit.

41:40 So we have different libraries that you can like, or sorry, different like interfaces

41:45 that you can basically attach and say, hey, this is my Salesforce session.

41:48 And it'll start to do all the logging for like this public API to the proper spots for you.

41:54 So you don't have to like worry so much about like, oh, am I doing the right logging here?

41:58 Am I doing, and it'll do all this stuff where it like kind of cleans up the secrets

42:02 out of it.

42:03 And it does the proper truncating.

42:04 So we're not like dumping like 50 megs to the logging service.

42:07 And it does all this stuff.

42:09 The logs are full again.

42:09 Yeah.

42:10 That you, you kind of forget that.

42:12 Oh yeah, yeah, yeah.

42:12 You got to remember to do that through a very familiar interface that you can just pop on

42:16 to, you know, Python requests dot, you know, read the docs dot, you know,

42:21 org and read all about exactly what it's doing.

42:23 It's so much easier to onboard new employees, new engineers on, and you can kind of extend

42:29 and get kind of the best of the community, the best of tools that exist out there.

42:34 So that's kind of way we've approached using like things like requests.

42:37 And it's been really quite, quite useful for us and very successful.

42:41 It's a great library.

42:42 I'm wondering if, you know, there's some of these integrations that are,

42:45 they basically have really nice prebuilt Python packages.

42:49 And some of them are just like, here's an HTTP endpoint, knock yourself out

42:54 with whatever tech, right?

42:55 So, you know, like for example, GitHub, there's like GitHub packages that you can use in Python.

43:02 Do you ever use those or you're like, it's better for us to just stay separate

43:06 at HTTP access level?

43:08 Yeah, we've gone to the HTTP level.

43:10 Those libraries are awesome, especially for, if you want to just spin something up quick,

43:15 but we've gotten so good at talking, like we talk to REST APIs.

43:19 We're good at that.

43:20 We've done it a lot, right?

43:21 It's kind of our bread and butter.

43:22 So we want to like talk to that and have kind of the control over that.

43:25 So we used to do a little bit of that in the early days.

43:28 And then we got to the point where observability became tricky.

43:31 Some of them used things like requests behind the scenes.

43:35 Some of them used HTTP lib too.

43:37 Some of them just use like the built in, like everyone was doing different stuff.

43:41 And then you had to like rewrite how your observability and your monitoring worked.

43:45 So for us, it just became hard to do.

43:47 So if you're talking to lots and lots of APIs like this, it really does help

43:51 to kind of standardize your interface on that.

43:53 Yeah, it seems like it must have been like almost a dependency, a dependency hell of a sort.

43:58 Like if you're like, we're going to have 500 different packages and those packages

44:02 might go out of favor and then they're not quite up to date or there's like a breaking change.

44:06 Like there's just so many layers, right?

44:07 And you're just like, let's just do the same for everything.

44:09 Yeah.

44:10 that's really where we kind of landed.

44:11 That was through some like learning, like especially in the early days, we did do a couple of those

44:16 like kind of like those libraries that you can just, you know, pip install MailChimp,

44:19 right?

44:20 Yeah, exactly.

44:21 We're good.

44:21 We're good.

44:22 Yeah.

44:22 Do this.

44:23 They did help.

44:23 And a lot of them are also, funny enough, they're built on little odd behaviors

44:29 that seem fine in isolation.

44:30 But when you're running, you know, something like, like what we do, it doesn't really fit.

44:36 For example, some of these libraries, you'll just import, like let's say MailChimp.

44:40 and then, you know, on the global object, do MailChimp.API key and just set your API key there,

44:44 which you never know which user's API key you're going to use.

44:48 Yeah.

44:49 When you're us, like that kind of like, you look at that and you're like,

44:52 oh, that doesn't, that doesn't really work.

44:53 Like I want to create like a MailChimp session, you know what I mean?

44:56 So like.

44:56 Anytime you're doing multi-tenancy, like those little gotchas are super scary.

45:01 You're like, why did I just expose all of this other customer's data or something,

45:05 right?

45:05 Yep, exactly.

45:06 So that's why having control over that layer and being really particular

45:10 about like when you're opening and creating new sessions or you're reusing them or not,

45:14 you want that control.

45:15 So that's where we've kind of landed for that and it works pretty well.

45:18 And you know, when I was talking about the AWS Lambda side, a lot of that's

45:22 on the JavaScript side.

45:23 So we do provide maybe not the perfect equivalent to Python requests on the platform,

45:29 what we call our platform like CLI and SDK, like that has like a similarly inspired

45:35 sort of a thing.

45:35 So use our, in our case, it's like the Z object that has a request method on it.

45:40 If you use that, then you get all the magical like logging and you get all

45:44 the magical like introspection on the calls that you're doing.

45:47 So it supports that.

45:48 And if you break out of that, maybe you have your own little like JavaScript SDK

45:52 for talking to your own service, you'll lose a little bit of that.

45:56 And in that case, like it'll be harder for us to debug your services or help users out.

46:01 So we do try to encourage folks to use that kind of standard thing, both inside Zapier,

46:06 obviously, but also outside of Zapier.

46:08 So when partners are like interacting, working with us, we also provide some.

46:11 Yeah, interesting.

46:12 So you talked earlier about this asynchronous message passing and queues

46:16 and whatnot and celery.

46:17 And that's really important for scalability.

46:20 But another huge, huge scalability booster is some sort of asynchronous web handling,

46:27 right?

46:27 like G event or async and await, asyncio, all those kinds of things.

46:31 Does that show up anywhere?

46:32 We have experimented a bit with it.

46:34 This is like one of those classic, like you try it out and then you run the numbers

46:39 and see how it like, how it like, how much does it save you?

46:42 How much headache does it add?

46:43 How much more complex is it, right?

46:44 Yeah, exactly.

46:45 And we tried this maybe, I'm going to say two years ago with G event.

46:50 It was right on the line, but probably at that time, it just wasn't a good enough

46:55 move for us to like transition over to it.

46:57 So we actually abandoned using G event and we're still using like pretty like

47:02 straightforward, synchronous like Python, which is right now we're still transitioning

47:06 to Python 3 at the moment, making good headway on it.

47:10 But a lot of the, you know, asyncio you mentioned, like a lot of that adds tons of complexity

47:16 to your code.

47:17 G event does the same thing.

47:18 There's all these like little extra things you got to remember.

47:20 And Python, I think, hands you a gun that's very loaded when it comes to like

47:25 async stuff.

47:26 So it is a little bit tricky and that's not to say there's, I mean, there are other languages

47:30 that do a really nice job with async, but even like something like JavaScript,

47:35 I think still, you get a loaded gun, right?

47:37 So it just adds this layer of complexity that I don't think was worth it for us

47:41 at the time.

47:42 I think there's some cool things we can probably do in 2019 to get back to that

47:46 and that's through moving more services behind like an IO boundary, just a regular HTTP boundary.

47:52 so less thing is happening like in the process itself and is sitting across

47:57 a very easily async boundary where you're talking over a socket or something.

48:01 I think if we start to move more workloads over there and we have like kind of this clean interface,

48:06 the Geovent equation will quickly tip in the other direction.

48:10 so I'm excited to maybe look at that again in 2019.

48:13 But yeah, we gave it a good, we gave it the old college try and it was a bit of a bear

48:17 and it was about a wash and maybe it would have saved us, you know, 10 or 15%

48:23 on our infrastructure costs because you still had tons of locking and database connection stuff

48:27 that you had to deal with that was still kind of put like some limits on what you could do.

48:31 But I'm excited to play with it in 2019 because I think we could probably

48:35 drive our costs down by if we do it really well by 30 or 40%.

48:39 And now that's starting to be appreciable amounts of money.

48:42 Like before, that was not that much.

48:44 So now I think it would probably be getting closer to worth it.

48:47 It's that complexity trade-off and at some point when it's $50,000 a month,

48:52 you're like, you know what?

48:53 We're going to just bite the bullet and solve this complexity problem and, you know,

48:57 give everyone a raise or whatever, right?

48:59 Yeah.

49:00 In those cases, yeah.

49:01 High-world people.

49:01 Yeah, yeah.

49:02 And that's, I mean, you know, you mentioned that we also do like profit sharing.

49:06 So there's like there's like a lot of incentive to do some of these like cool cost-saving things.

49:10 But when you're measuring, you know, and this is getting away from maybe the Python side,

49:14 but you're getting into like the business side of like, well, we could get a,

49:17 we could trade a dollar of cost savings for $2 of revenue.

49:20 Like, you know, do you go for the revenue?

49:23 Because it opens up more opportunities later on, serving more customers.

49:27 So you're always, I'd go for the revenue.

49:29 Yeah, we're always like, I'd go for the revenue, man.

49:30 Yeah, we're the same way.

49:32 Like we've been, you know, really fortunate that the product itself really connects

49:36 with all these users.

49:37 So for us, like expanding that is really, really interesting.

49:40 Right.

49:41 Well, that's a whole, that's actually basically a good, it has analogies to like,

49:45 why do you even choose Python at all, right?

49:47 Like you could write this in C++, mostly with wrapping it in Java or something,

49:51 and you could save money on infrastructure, right?

49:54 But the rate at which you generate features and you make users delighted

49:58 would be slower.

49:59 And, you know, it's like people say Python is slow and you guys are doing

50:04 incredible stuff.

50:05 I don't actually think that's true.

50:06 I think it's really, I think it's like, it's complicated.

50:08 It's more like the right answer.

50:09 But at the same time, it's, that does not say, talk about, well, Python is fast

50:16 for features, right?

50:17 Which is pretty awesome.

50:18 That is pretty awesome.

50:19 And you're looking at companies that are generally selling like a digital good.

50:24 The margins are really good.

50:26 So you have that headroom.

50:27 Like you're not getting squeezed down to exactly some super tight thing where you're like,

50:31 oh, we have to rewrite this and see, we have to like really go super low level

50:35 and get that, eat that performance out.

50:37 Right.

50:37 You're not General Motors at 6% margin, right?

50:40 Yeah.

50:41 You're, you're doing, you got more room to play with, right?

50:43 That equation may change.

50:44 Let's say you're the, I'm going to say some of these like security cam software

50:49 that exists out there, right?

50:50 Maybe you do like this cool little hardware.

50:53 Like we get, we sell the hardware and we give away like the free tier and you have like

50:57 a more complex business model.

50:58 And now you may care a lot about those margins and Python may not work for you.

51:03 And in that case, yeah, it's too slow, right?

51:04 Right, right, right.

51:05 It's very context specific.

51:07 For us, it's not too slow because the value we can create with it is, you know,

51:11 orders of magnitude higher than the effort that goes into it.

51:14 And that changes depending on the business and the margins and the context.

51:20 for us, it's been awesome and it's been really, really fun.

51:22 But that's not to say that there's maybe not cool opportunities to use.

51:26 Maybe see, or maybe for us might be like more like Golang stuff.

51:30 And if you choose to rewrite some kind of like core tier zero services in that

51:34 to eke out a bit more performance, but still use Python for a lot of the

51:38 product facing and things that you're going to be touching on a day-to-day basis

51:41 and iterating for users, it's not one and a done kind of a thing either these days.

51:46 Yeah, yeah.

51:47 Well, one of the things that seems like there's probably a place somewhere for this to fit,

51:52 but it might cross the complexity threshold on the other direction and make it not worthwhile.

51:57 But, you know, like with Cython, you could take the little bits that are slow,

52:02 write that in Cython and then use the no GIL block in Cython and achieve a ton

52:08 of concurrency and CPU speed with, you know, changing like 20 lines of code

52:13 potentially.

52:14 Like, have you explored any of that?

52:15 We have done a little bit of that, especially in some of our like ancillary services.

52:19 A good example is we have a email parser which kind of lets you tag emails

52:24 and then we'll try to extract pieces of that.

52:25 And we've done quite a bit of stuff where we've tried to optimize that engine,

52:30 but like it's just pure Python.

52:31 It's just doing a bunch of like string slicing and comparison stuff and you can wrap

52:36 that up in Cython.

52:37 Cython was a thing that we actually used.

52:39 We ended up rewriting a little bit of it, like some of the core pieces in C++

52:43 to speed it up.

52:45 But again, a very kind of niche, very specific use case which is really how you should be

52:50 thinking about performance is you should make it work and then if you need to,

52:54 right, make it fast and use the tools at your disposal and Cython was one of those.

52:59 And we had pretty good success with it.

53:01 So I do think it's a realistic option.

53:03 Yeah, I've seen some really good outcomes from that as well and I think it's

53:05 pretty interesting, right?

53:06 It's like when you look at a huge code base, there's usually a couple of spots that if you could

53:12 unlock the performance on just those little bits, actually that's where most of the problem is, right?

53:17 That's where most of the latency is.

53:18 Yep, 100%.

53:19 And there's some really cool posts out there.

53:22 I think Instagram engineering did some where they would use like the C profile

53:25 module and then measure CPU ticks for like all the different stuff that they did

53:30 so they could really get like instead of like wall clock time which can be like,

53:35 oh, we blocked on this, we waited on this and you get a different like look at the code.

53:39 You can do some really cool stuff with Python that I don't know if a lot

53:43 of folks are like aware of like the just the level that you can go just like straight Python

53:48 and get like really, really high performance stuff.

53:50 There's a lot of cool tricks in headroom and the language itself.

53:53 So I think we got a lot of headroom when it comes to like Python and Zapier.

53:58 We're definitely doubling down on using Python in 2019 and beyond.

54:02 And the conversion from legacy Python to modern Python is it's underway?

54:06 Yeah, it's underway.

54:07 Yeah, we've got we're running all of our like test, all of our tests run

54:11 in both Python 2 and Python 3 and have been for a little while.

54:14 So really what we got to figure out now is just make make that transition

54:18 in like production and make sure that we don't break any like caches or behaviors

54:23 or things like that.

54:24 So figure out a good way to try doing some canaries and all this stuff.

54:27 That's like kind of the next step for us.

54:29 Like we're right there.

54:30 That's exciting.

54:30 You're on the cusp of it.

54:32 Yeah.

54:32 Yep.

54:32 Yep.

54:33 I suspect in the next couple months we'll probably be running Python 3 in production.

54:37 So pretty exciting.

54:38 Sweet.

54:38 Yeah.

54:39 the one place I've heard really scary stuff in is around caching and pickles, right?

54:44 People take objects and they just pickle them up and cache them rather than doing

54:48 like a JSON 2 from sort of intermediate thing and then boom like, you know,

54:52 a different CPython grabs it.

54:53 It looks different.

54:54 Yep.

54:54 That can cause a lot of trouble.

54:56 All sorts of badness breaks out.

54:57 We try to avoid pickle as much as we can but for some stuff like the convenience

55:01 is just so nice like just throw anything in, right?

55:04 So we try to avoid and we try to do JSON serialization as much as possible

55:09 but yeah, that is a tricky thing around deploying between two and three and testing them

55:16 and it just makes it really operationally complex.

55:18 So that's something that, you know, I think is sometimes overlooked whenever

55:22 people think about transitioning their code base to Python.

55:24 Right, right.

55:25 You got to be careful there.

55:26 That's cool.

55:27 All right.

55:27 Well, this has been such a super interesting look inside.

55:31 Let's have to hear what you guys are doing and they're really neat use case.

55:35 There's a couple other questions I'm going to ask you before I let you go though.

55:37 Sure.

55:37 Even though we're running a little low on time.

55:39 First of all, like what's it like being the CTO of, you know, a big tech company

55:43 like this?

55:43 There must be a lot of plate spinning, a lot of things to do but I bet it's fun.

55:47 Yeah, it's really fun and you know, the role has kind of changed and I feel like one thing,

55:52 the journey that I went on from early CTO all the way to now which was doing,

55:57 wearing a lot of the hats is probably a really common one but it's different

56:00 for a lot of folks.

56:02 Like I talk to a lot of CTOs to try to figure out like what is a CTO in like a tech company?

56:06 I talked to maybe a dozen different people and I got a dozen different answers

56:10 so it's really like a very personal journey and like what does a company need

56:14 sort of a thing so for me, it's been definitely a part of like discovery

56:17 as well but also for me, it always comes back to what does a product do?

56:22 Like I really, like the product resonates with me like as an engineer like giving people

56:27 the power of engineers to do all this stuff even though they don't know they're kind of like

56:30 building automations and the same things that we would do, you're still giving them

56:34 that power and that's really exciting to me so I'm excited to spend a lot more time

56:38 in that and then diving into the architecture and the scaling issues that a service like Zapier has

56:43 is really, really exciting for me.

56:45 So yeah, that's a fun part of software to think about, right?

56:47 Yeah, you know, so being a CTO at a company like this is really exciting.

56:52 I'm also just like really excited to be hiring like engineers that are like better than me

56:56 when it comes to all this like deep, interesting stuff so we have like, we have some core

57:00 like Django contributors and they're like amazing to like dive into and like learn about

57:05 some of this stuff that we use every day and that's always really exciting

57:09 so it's really humbling at the same time so it's just a really fun, I mean it's super fun,

57:13 it's super fulfilling, we have a lot that we want to do so I feel like I gotta get back

57:19 to work too.

57:20 I'm sure.

57:20 Alright, another one is just a couple real quick, what are your favorite

57:25 automations?

57:25 Some of my favorite ones they have a little Python sprinkled into them so those are some

57:30 of my favorites especially whenever I want to do something that's kind of

57:33 a little bit funky and weird and I don't want to deploy any extra code like I'll often

57:38 like write like a Python step.

57:39 A really specific one that I've done because I've been running a lot of like

57:43 engineering like the meetings the all hand kinds of things and we try to like

57:47 prep and have notes and kind of have a lot of the stuff ready is I have a schedule

57:51 set up and I have a zap that triggers off of that event which is kind of

57:55 a recurring event and it goes out and it grabs a template from Google Docs

57:59 it copies a bunch of stuff into that into a new Google Doc includes a bunch

58:03 of charts and a bunch of little things that we should be talking about and then goes

58:07 into the Slack room and posts it and reminds people and I have it set it up

58:10 with the delay so it even reminds people periodically like hey two days to go

58:15 make sure you add your notes to this and like meeting is coming up in an hour

58:18 so it's really really helpful because a lot of times many people are crazy busy

58:23 obviously so I found myself going around reminding folks to get that stuff ready

58:27 and this kind of just automates that away and does a lot of really cool stuff there

58:31 so I have so many like zaps running we have so many bots running around in Zapier

58:35 too like sometimes they'll bump into each other and do crazy things and it's really

58:40 quite fun it's pretty exciting that's super awesome and it has this nice

58:45 aspect where the thing doing the reminding is not a person so that person

58:49 doesn't get a little bit like ah that person's a nagger right you don't have

58:52 that aspect it's like well the bot did it like it's you know it does it every

58:55 Thursday for our Friday meeting or whatever right yep and it's super handy

58:58 and you can go in and you can adjust it so if someone's like oh that's the way this is

59:03 phrased or you didn't include this link or it'd be great if you added this data to it

59:07 like it's in a team account like anyone can hop in and like play with it

59:11 and change it and swap it out and tweak it and we always include like the link

59:15 to like edit this app in the messages that we send around so anyone can go

59:19 in and make it even better and add functionality to it which is a really kind

59:23 of collaborative way to build some of these workflows and augment these bots that are

59:27 doing stuff because like you'll see it and you'll be like oh it'd be great if it

59:30 actually included this link or it said this thing and you just hop in and change it

59:34 it's really really fulfilling to be able to just like see something change

59:37 something and then next time it goes it's there's no deployment going on it's just

59:41 doing it it's really cool yeah that's super how cool the singularity is near okay

59:46 so the last two questions that are standard not focused just on you as if you're

59:51 going to write some python code what editor do you use i use Visual Studio Code the

59:55 new one i think that microsoft's new code editor like some of their philosophies around

01:00:00 focusing on developers has been really great yielded some awesome tools so

01:00:04 i use that and i quite like it that's cool yeah it's definitely got a lot of

01:00:07 momentum and i think they're doing nice stuff there and then you definitely work with

01:00:11 a ton of pypi packages maybe one that's notable not necessarily the most popular but

01:00:16 people you're like oh you should know about this have you heard of yeah whatever

01:00:19 i would probably say and maybe everyone knows about this but it was kind of new

01:00:23 for us for a couple months ago the python formatter black yes really really love that so

01:00:29 we've completely like blackified our entire code base and it's so nice to just like

01:00:33 not think about that because like we would always be like oh do we want to do

01:00:36 commas here do we want to split these lines like it was just a thing we were i mean we we

01:00:41 worried about and we cared about and you know we want our code to look great and like

01:00:45 be easy to read and black does an amazing job at that so we do that and we really

01:00:50 really like it so if folks out there aren't using black you should check it out it's

01:00:54 really cool super useful so that would be mine that's great you set up as like a pre

01:00:58 commit hook yeah we have it as a pre commit hook we have all of our ci stuff

01:01:01 running that checks it all and like yeah so it's it's a big part of our workflow now

01:01:06 yeah solid that's cool all right well final call to action people are excited about

01:01:10 this distributed work zapier all the tech we talked about what do you got for them

01:01:15 check us out if you're interested in building any of these workflows or you have colleagues that

01:01:20 could benefit from any sort of like little workflows like this let them know i mean we

01:01:24 can kind of do a little bit of anything if you use any sass apps we're probably got something

01:01:28 useful for you and also if you're looking for a new gig and you're interested in

01:01:33 remote work check us out not only do we have a ton of information on how we work like

01:01:37 we just shared it and kind of open source a lot of our processes but we'd love to maybe work

01:01:42 with you so we have lots of jobs coming up in 2019 lots of roles so definitely keep your

01:01:47 eyes out there and you can even set up a zap that will alert you of new jobs that pop up on

01:01:52 the site and then they will send you an email or maybe a slack message or something

01:01:55 like that so check that out that's pretty meta and awesome awesome let me ask i guess one more

01:02:02 follow-up so you talked about if people want to set up these automations

01:02:05 that makes a lot of sense what if they're these folks listening are developers at a company

01:02:09 and they're like why doesn't zapier integrate with my company and my apis like what do they

01:02:13 do then yeah so we do have an open platform so anyone can kind of hop on

01:02:17 and i mean we actually have a lot more like private apis that people have added for their own

01:02:22 companies it's really really straightforward we have lots of examples you can either build them

01:02:26 in the browser and just kind of post your like rest urls and things like that and we'll do all

01:02:32 the magic there or you can write a javascript node.js package which fits just kind of a light

01:02:38 spec that we have we have tons of examples and you can expose different endpoints and write little

01:02:43 custom functionality so you can say here's this json list that shows all

01:02:47 the most recent leads for our company that can turn into a trigger and here's

01:02:51 a post endpoint to create a lead and you can define what fields go into that

01:02:55 and then away you go and it works with you know 1400 different apps out of the box which is

01:03:00 really powerful that's pretty cool all right well this has been a lot of fun to talk to you

01:03:04 about all these ideas so much cool stuff you're all doing thanks for being on

01:03:07 the show of course thanks michael see ya yeah bye this has been another episode of talk python to

01:03:12 me our guest on this episode was brian helmig and it's been brought to you by

01:03:17 linode and ting linode is your go-to hosting for whatever you're building

01:03:21 with python get four months free at talkpython.fm/linode that's l-i-n-o-d-e ting is

01:03:28 the fast mobile network custom built for technical folks use their savings

01:03:33 calculator to see exactly what you'd pay visit python.ting.com to get a twenty five dollar credit

01:03:39 and get started without a contract want to level up your python if you're just

01:03:43 getting started try my python jumpstart by building 10 apps course or if you're looking for

01:03:49 something more advanced check out our new async course that digs into all

01:03:53 the different types of async programming you can do in python and of course

01:03:56 if you're interested in more than one of these be sure to check out our everything bundle it's like

01:04:01 a subscription that never expires be sure to subscribe to the show open your favorite podcatcher and

01:04:06 search for python we should be right at the top you can also find the itunes feed at /itunes

01:04:11 the google play feed at /play and the direct rss feed at /rss on talkpython.fm this is your

01:04:18 host michael kennedy thanks so much for listening i really appreciate it now get out

01:04:22 there and write some python code you're welcome you're welcome Thank you.