Continuous Integration and Delivery at Codeship

00:00 Have you heard about the It Works on My Machine certification program?

00:03 This is a really awesome certification for developers.

00:06 It was created by Joseph Cooney.

00:07 Here's how it works.

00:08 Step one, compile your application code.

00:11 Getting the latest version of any recent code changes from other developers

00:16 is purely optional and not a requirement for certification.

00:19 Step two, launch the application or website that's just been compiled.

00:22 Step three, cause one code path in the code you're checking in to be executed.

00:27 Now, the preferred way to do this is with ad hoc manual testing of the simplest possible case for the feature in question.

00:33 And you may omit this step if the code change was less than five lines or if in the developer's professional opinion,

00:40 the code change could not possibly result in an error.

00:43 Now, check in your changes to version control.

00:45 Congratulations, you're certified.

00:47 On this episode of Talk Python to Me, you'll meet Florian Motlik from CodeChip.

00:52 He's here to tell us all about continuous integration and continuous delivery.

00:56 Maybe you can help keep you and your team from getting certified as well.

00:59 This is episode number 38, recorded December 10th, 2015.

01:04 Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem,

01:34 and the personalities.

01:35 This is your host, Michael Kennedy.

01:37 Follow me on Twitter where I'm @mkennedy.

01:39 Keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via at Talk Python.

01:46 This episode is brought to you by Hired and DigitalOcean.

01:50 Thank them for supporting the show via Twitter where they're at Hired underscore HQ and at DigitalOcean.

01:56 Hey, everyone.

01:57 Thanks again for listening today.

01:59 I have a few prize winners to announce before we get to the show.

02:02 Two weeks ago, JetBrains and I gave away three PyCharm professional licenses.

02:06 The lucky winners were Drew Kerr, Ben Kelly, and Seref Alvinides.

02:11 And in episode 37 on computer security, Justin Seitz gave away a copy of Black Hat Python.

02:17 The lucky winner there was Arek Fyla Kowalski.

02:20 Congratulations to everyone.

02:22 And if you want to be able to win these giveaways, all you have to do is be a friend of the show.

02:27 Just visit talkpython.fm, click Friends of the Show in the navbar, and sign up.

02:31 Now, let me introduce Flo.

02:33 Florian Motlik is the co-founder and CTO of CodeShip, a continuous delivery service.

02:39 While studying and working in the Technical University in Vienna, he stumbled into running test infrastructure for other teams and hasn't stopped since.

02:46 He sometimes blogs on the CodeShip blog at blog.codeship.com or on his personal blog, flomotlik.me.

02:54 You can also find him on Twitter, where he's at Flo Mollik.

02:57 Flo, welcome to the show.

02:59 Thanks. Thanks for having me.

03:00 Yeah, I'm really excited to talk about building software, testing software, and automating the whole thing with you.

03:05 So it'll be fun.

03:07 Yeah, totally.

03:08 For sure.

03:09 So we're going to talk about continuous integration, continuous delivery, maybe throw in a little Docker, infrastructure, all those kinds of things.

03:17 But before we get into those topics, maybe give me your story.

03:20 How do you get into software and programming and this whole thing?

03:23 Sure.

03:24 Sure.

03:24 So I mean, as most people at some point, I started playing games on my dad's old computer.

03:31 And so that got me into a little bit of scripting and stuff like that and trying to look a little bit under the hood at like 10, 11 or something.

03:39 And then when I was 15.

03:40 So the school that I did had a computer science branch, basically, that I went to.

03:45 And so there I really started to get into programming and what it actually means to develop software.

03:52 So that was in school from like 15 on.

03:59 So I'm originally from Vienna and then started software engineering.

04:07 then, basically, and then basically, and at some level, really, the whole web development side of things really captured my interest.

04:14 And so that's web development and infrastructure is what I've done since, basically.

04:20 Oftentimes, I ask people how they got into Python because, you know, a lot of my guests are explicitly doing Python.

04:25 But you guys are doing all sorts of languages.

04:28 Can you maybe just tell me what areas you work in, what technologies?

04:31 Sure.

04:32 So in the past, basically, like I started with the typical Java, like school and university stack.

04:38 So been doing Java for a long time.

04:40 At some point, we switched.

04:41 So when we started CodeShip, I looked at various technologies out there.

04:45 And since it was a small startup and I was the only developer, like I looked into Ruby and Rails for getting stuff up and running quickly.

04:54 So most of our stack, including our back end and like the build infrastructure that we're running for customers, was written in Ruby.

05:00 And then in terms of production systems, like we've started moving more and more stuff to Go.

05:05 So we've re-implemented a lot of the back end systems.

05:08 A lot of that stuff is now in Go.

05:09 So that's on the part of like developing our own systems.

05:12 But then since we basically rerun software and we run the tests for many, many other people and support a lot of different technologies.

05:19 I've dabbled in Python, in Java, of course, a lot of it, in Ruby, in many other languages that we support in our stack.

05:28 So been playing with and enjoying the ecosystem of many different languages, but mostly programming in either Ruby, Java in the past or a little bit of Go now.

05:39 Although I don't get to code a lot anymore, sadly.

05:42 That is unfortunate.

05:43 So I've worked for very large companies and very small companies.

05:48 And I've been in a situation, like you just described, where I'm the only developer.

05:52 And I have to say, it's pretty cool to start something new where you're the only developer.

05:58 And you kind of get to look across the entire tech landscape and say, I can pick anything.

06:03 Where do we start?

06:04 You know, that's a really fun place to be.

06:08 You work for CodeShip, obviously, and you guys have been a sponsor of the show for a long time.

06:12 So thank you for that.

06:13 Why don't you tell everyone what you guys do at CodeShip?

06:17 Sure.

06:18 So the main idea, I mean, just in terms of the messages, CodeShip is a continuous integration, hosted continuous integration and continuous delivery service.

06:27 The main point there is that basically when you build software, you want to test it as fast and as often as possible to really know when something breaks.

06:35 Like you want to do tests.

06:36 You want to do test-driven development or write tests constantly.

06:39 And then you need a separate system that whenever you change any of your code, push anything into the repo, that it runs all of your tests, all of your builds, basically.

06:48 And doing that on your own local development machine is kind of a hassle.

06:52 And oftentimes it's not done.

06:53 Like we're all humans.

06:54 Like we only think this small change will not break my whole application.

06:57 And then it does.

06:59 Yeah.

06:59 Of course it does.

07:01 And, you know, maybe it doesn't break on your machine.

07:03 It breaks on someone else's machine.

07:05 Exactly.

07:05 Exactly.

07:06 Have you seen the It Works on My Machine certification program?

07:10 No, I haven't actually.

07:11 So everyone out there listening, you should search Google for Works on My Machine certification.

07:16 It's a really great certification program that basically marks you as a bad developer.

07:21 Nice.

07:23 I have to look at that.

07:24 No, I haven't heard of that one before.

07:25 Yeah.

07:25 It's really like that whole thing is a huge problem.

07:28 And so people really like the...

07:30 And that's where the idea came from.

07:33 Like we ran or I ran like CI for a lot of companies or like at university before.

07:37 And it's just such a hassle.

07:39 It's just so painful to run it yourself and maintain that stuff when you actually want to do...

07:45 Like you actually want to get productive stuff done and like work on your customers' projects or like your own product.

07:50 And yeah.

07:51 So that's where it came from.

07:52 Instead of having everybody else like do the boring task of maintaining the test infrastructure,

07:57 we take that for them and then do it much better for everybody basically.

08:01 Yeah.

08:01 That's fantastic.

08:02 When did you guys get started?

08:03 We started in 2010.

08:06 We talked a little bit about continuous integration.

08:09 Maybe we could dig into that a little bit more.

08:12 Like what is continuous integration?

08:14 What are the steps?

08:15 And like what's involved in that?

08:17 Continuous integration as a term is heavily overloaded.

08:20 I think at this point, a lot of people when they talk about continuous integration,

08:23 especially in like small to medium sized teams, are talking about automated testing and running your automated tests on a separate system.

08:32 So I think that's really the main goal of making sure that whenever anything changes on your code base, whenever you push anything anywhere,

08:41 that it's run by a separate system, a separate system that makes sure that everything works and notifies you that the whole build has been working fine and everything has been working correctly.

08:53 That's the main goal of the CI system.

08:55 Right.

08:56 I think a lot of people initially, you know, back when you guys started around 2010, saw that almost as a, here's an automated compile step.

09:05 Maybe we'll run a couple of tests, you know, depending on whether you're working for with a group that's actually written the test.

09:13 Right.

09:14 And people have been adding more and more to it over time.

09:18 Right.

09:19 Yes, absolutely.

09:20 So I think that's, that's something where like, I think continuous integration can be seen as more than just running your unit tests.

09:25 I think that's something where in the end, like the way I would define is the important part is that at every point,

09:32 the team should know that your application works as expected for your customers.

09:39 I think that's the whole goal.

09:40 Like, and that can be unit tests.

09:41 That can be like functional level tests that can be deploying into a staging environment and like testing against that staging environment just to make sure it works there.

09:49 I think it's the, the, the end goal.

09:51 I think it's the definition that we really like is less about like the technical aspect of continuous integration.

09:56 And I don't really like defining the continuous integration as like a technical process.

10:01 It's more of like this thing needs to make sure that whenever I run something, that I never want to change something in my, my system.

10:10 I can actually do that and not break anything for my customers.

10:13 It should be very customer or like user driven.

10:16 Like we want to make sure that we don't break anything because that's, that's actually like, that's why we, we build software.

10:22 Like we want users to use the things that we've built.

10:25 We don't want to break things for them constantly.

10:27 And we want to move fast, like move fast and break things.

10:30 But even better if you just move fast and don't break things.

10:33 Right.

10:33 Maybe you break them, but maybe you don't send the broken pieces to your customers.

10:37 Yes, exactly.

10:38 Exactly.

10:39 Get, get that information very quickly.

10:41 And I think especially like break things maybe in terms of like features that you remove or when you want to change your product, but not really like break things as in show a 500 error page.

10:51 Like that just shouldn't be something that, that actually should be happening.

10:53 And I think it's, I mean, all of us have like experienced that all, all the time.

10:58 Like things when, when features break, when things don't work as expected anymore, when like there's a downtime, just a drain on like the product.

11:08 You don't like the product as much anymore.

11:10 And it just doesn't feel good at that point.

11:13 And I think continuous integration and the process of really making sure from a customer's perspective that your application works is usually important and can really put a lot of, a lot of stop basically to unnecessary work that your team does.

11:27 And has to do that.

11:27 I mean, we've all been in those situations where we broke something.

11:31 And so now the whole team has to jump onto fixing that thing.

11:34 And like, it takes so much resources from the rest of the team.

11:37 If you can capture that early and make sure that it actually works and it, and there's no issue.

11:42 It just, it puts a lot less stress on the team.

11:45 There's a lot less overwork that we need to do.

11:47 So that's, I think the, the really important part of automating all of this and just making sure it works.

11:52 It basically lets you focus on the fun part of software development.

11:56 Writing cool new stuff.

11:58 Yes.

11:58 Not fixing things that were not broken previously.

12:01 Absolutely.

12:02 Absolutely.

12:03 There's no worse thing than regressions that come in where things that are working or have been working are now broken for no particular reason, just because there wasn't anything in place to catch that.

12:13 And so that's, that's, it's really important to make sure that that doesn't happen.

12:17 Yeah.

12:17 Yeah.

12:17 I've noticed a lot of really interesting cultural behavioral shifts that seem to be out of proportion with the effort of setting up continuous integration.

12:26 So for example, I've seen teams that they release every three months or every six months, you know, to me, that sounds insane, but that's how they were working.

12:36 Right.

12:37 Because they wanted to test it.

12:38 They wanted to make sure they wanted to sort of put it in a box and say, okay, now we're ready to send out this next thing.

12:43 And, you know, maybe the next week somebody created some great feature, but well, that's going to come out in three months because it's just too much effort to go through the release test deploy cycle.

12:55 Yes.

12:55 To justify going through that just to get this one feature out.

12:59 But if, if all of that becomes push a button or push to a branch and it just happens, well, maybe all of a sudden when that feature is tested, it's ready to roll and you just push it out.

13:11 Yep.

13:11 And so I've, yeah, I've seen teams that have sort of gone from releasing these big, painful, slow things like, hey, we're all going to come in Sunday night when no customers are there, shut everything down, make sure the integration stuff works.

13:25 It's to just much more fluid work.

13:29 And I think that's great for the software.

13:31 It's also great for the people that have to do it.

13:33 It's way more fun to see your work that you do come right out.

13:37 Yeah.

13:37 Absolutely.

13:38 I think that the whole team productivity and team happiness is a huge part of like automating your, your whole workflows and everything around it.

13:45 Because I mean, in the end, the best people that we have in our team, and if you want to build like a high class engineering team, they don't want to fix broken things.

13:53 They want to build interesting, new and cool stuff.

13:55 So if you have a process that is continuously like bringing down the happiness in the team and continuously having the team have to shift resources between like building new stuff, but then fixing old stuff, building new stuff, fixing old stuff.

14:08 That's just going to drive the best people in your team away.

14:11 And I think that's a really good way to drive the whole engineering team into the ground.

14:15 So I think making sure that automation and how we release processes all really help to strengthen team happiness and putting those into like the perspective of team happiness is, is really crucial because I mean, and also that that's something that we've seen that we've seen with customers that we've seen in our own team.

14:32 Like when the build, like when the tests get unstable or something like that, that's a huge drain on like team productivity, team happiness.

14:40 And just, it takes so much mental energy from the team and it just, it just puts people in a bad position.

14:46 And I think that it really, like, it's not just about the technology.

14:50 It's not just about the customers.

14:51 It's a big part of that is that, but it's also about putting it into perspective of how can I make my team happy by not having them to do any like menial, boring tasks that I really don't want them to do on a daily basis.

15:03 So that's some of the things that I think are really important when setting up all that automation.

15:08 And, and we've seen that ourselves where we have teams basically like their release, even small teams, like their release process was like a week of like going through Excel, like having Excel sheets that had manual test cases in there.

15:20 And like that took a whole week, like that, that took, like they took their backend, like backend, main backend engineers and just had them run through that because they didn't have any more people.

15:29 Like their way of parallelizing all of that was basically getting more people, more engineering people in and like clicking through the application.

15:36 And I mean, that obviously doesn't scale and like, and they've, but, but I think it's, it's so often like that path is just so set in stone for some teams that, that it really has to like, they really have to get the help to like be taken out of it and like see a different path, how it can automate this.

15:51 And I'm sure like the developers were aware of how this could be done differently.

15:55 But I think if you don't put it in terms of like making the team actually happy and just see it as a, that's just more time spent by the team and like not see it as, Hey, if we do this like two more times, like these three people will just leave.

16:07 They just, they just won't keep up with this anymore.

16:10 I think it's a much easier sell to actually do this.

16:12 Yeah.

16:13 That's a really interesting point.

16:15 It seems crazy.

16:16 Like as soon as you said Excel, like I started to get a little bit of a headache and, you know, it was just like, you don't want to do that.

16:24 Right.

16:24 Like that is, that is not what people signed on for, but you know, some places they seem to develop this culture where that's how software is built.

16:33 And, you know, it comes probably from a good place.

16:35 Like we can't break things.

16:36 You know, maybe you're a bank or something and, and if you're down, it's, you lose a hundred thousand dollars an hour.

16:41 Like you obviously want to avoid that.

16:43 But you know, things like continuous integration and continuous delivery and all that sort of stuff can really help there.

16:50 Yeah, totally.

16:50 And I think it's also like, it doesn't necessarily, as you mentioned, we actually like there's like we, we, we worked with teams where they had exactly that, that problem that they were contractually obliged to not release.

17:01 Like at a specific point in day, like they, they cannot release from, I think it was like 10 in the morning to like 6 PM or something like that.

17:07 There is just like in the contract, you cannot release at that time.

17:12 Just, you're not allowed to.

17:13 So there is no continuous delivery for them.

17:15 Like that's just not, or at least not directly to production because that's, that's not what their business is doing.

17:20 But what they, they've done is just simply released to a staging environment, like continuously have something that is pretty much exactly what is, what production is released there on a continuous basis.

17:30 And then run like tests against that staging environment to make sure it actually works then so that they can release like during the night or in the very early morning hours.

17:38 And actually know that, that this works and then they're fine with that.

17:42 So I think there's different ways how to set that up, how to make that work, that, that teams can do and that the teams can use.

17:49 but it's really about like, how can I make my customers happy?

17:53 How can I make my team happy?

17:54 And how can I get the most like productivity and focus out of my team?

17:59 Because I mean, in the end that's, we're in a, we're now in a world with software where like everything is basically software.

18:05 Like every company, like an Uber is taking over the tax industry.

18:08 Netflix and Amazon are taking over like basically everything or like the TV industry.

18:13 So like a lot of these industries were in the past, they had a much stronger, part in like the real world, to call it the real world, are not taking over by companies that are software companies that are just intermediaries between like different people selling stuff.

18:27 And so everything is software competition moves a lot faster in all of those fields because it's just so much easier and cheaper like AWS and all those other cloud services make it so much cheaper to get started with things.

18:38 And then we have so little talent.

18:41 Like there's so few engineers generally out there.

18:44 Like everybody's constantly hiring.

18:46 Everybody's constantly trying to find their people that it's just crazy to spend time of your engineers on doing something that some workflow or automated system could, could do for you.

18:56 Because it's like, you need to get faster.

18:59 Like you cannot slow down.

19:00 Like you're just not allowed to slow down if you're in the software world anymore.

19:03 It's just not, it's just over.

19:04 Somebody else will come in and just eat your lunch.

19:06 This episode is brought to you by Hired.

19:19 Hired is a two-sided curated marketplace that connects the world's knowledge workers to the best opportunities.

19:26 Each offer you receive has salary and equity presented right up front and you can view the offers to accept or reject them before you even talk to the company.

19:35 Typically, candidates receive five or more offers in just the first week and there are no obligations ever.

19:41 Sounds pretty awesome, doesn't it?

19:44 Well, did I mention there's a signing bonus?

19:46 Everyone who accepts a job from Hired gets a $2,000 signing bonus.

19:50 And as Talk Python listeners, it gets way sweeter.

19:54 Use the link Hired.com slash Talk Python To Me and Hired will double the signing bonus to $4,000.

20:02 Opportunity's knocking.

20:04 Visit Hired.com slash Talk Python To Me and answer the call.

20:07 Yeah, that's for sure.

20:18 Speed is definitely a foundation of your business itself, right?

20:22 You mentioned Netflix.

20:23 Those guys have tremendous amounts of software driving and cloud deployments and whatnot driving their whole business.

20:31 And they have the policy that as an engineer, you can release whenever you want.

20:39 Yep.

20:40 You decide, yes, it's done.

20:42 Out it goes.

20:43 And they have all the processes in place.

20:44 So it's that kind of stuff you can sort of make much more doable with continuous integration and delivery, right?

20:51 Totally agree.

20:53 Yeah, I think having those processes in place where I think for once putting a trust in the engineers,

20:57 but also having things in place where maybe you just want to release this to like 1% of your customer base and just to try it.

21:04 And like give those tools to your engineers.

21:05 And it's obviously like at Netflix scale, that's a lot more doable than it is on like other people's scale,

21:11 where like you can divert like 1% of your customer, like your requests to that specific new release.

21:19 But I think in general, like I think teams should really think through like how can we support our team in like testing new things,

21:25 in experimenting with new things, with moving faster.

21:28 And yeah, continuous integration and continuous delivery are definitely an important part there to make sure that you can actually release there.

21:34 And on the other hand, that you don't actually break stuff for your customers and you just don't break that buy button.

21:39 So customers can actually give you money still.

21:41 That's right.

21:42 I think the other thing that would really helps people have confidence moving faster and just going with it is the ability to undo something, right?

21:53 If the ability to say, oh, that's a bad, something broke, right?

21:57 We thought we had the system in place.

21:59 Turns out not quite everything was caught.

22:01 And just push a button and roll it back as well, rather than, oh, we're going to be down for a day if we break this thing.

22:06 Yep, absolutely.

22:08 You need to have a way to either.

22:10 So what we've seen often with continuous delivery, once people move to a point where like you can release so quickly, is that when you release like small batches at a time, then oftentimes people just don't.

22:21 They don't really roll back.

22:23 But oftentimes they just push forward with like, because you just released something that is incredibly small.

22:27 So if there's a bug out there, you find it immediately.

22:30 It's typically pretty quick to fix it.

22:32 You fix it and push that new fix out there.

22:34 So that's something that we've seen a lot of teams go to in terms of the process where you definitely need a way to roll back.

22:40 Like that definitely needs to be in place.

22:43 But we've seen a lot of teams that then just push forward and like just deploy a new version very quickly.

22:49 Because you only deploy so small things that it's really easy to find like what the actual problem is.

22:55 You don't deploy like work of like three weeks where like you have no idea, like something breaks and you have just no idea where it happens.

23:01 It's just a combination of like five different things coming together and then breaking your application.

23:05 But it's just actually a small thing.

23:07 So because you're pushing so often and so fast, you can actually push more often and faster because you can find stuff so quickly and so easily.

23:16 Yeah, I think that's a really good point.

23:17 You talked about releasing on shorter cycles and sort of catching bugs early.

23:23 I think, you know, the way that people run into trouble and the way that deployments become not fun at all is you save all of your work for three months and then you put it out on the real system.

23:35 And then you figure out what works and what doesn't.

23:37 And you find these problems.

23:38 You have to go back and like completely analyze the whole system.

23:43 Right.

23:43 You know, what did you do the last three months?

23:45 Well, let's start here.

23:46 And the opposite of if I've been working for an hour, I put something out right then.

23:51 Well, you probably know exactly what it is.

23:53 Just like you said, it's super easy to fix, right?

23:55 Yep.

23:56 Yep, totally.

23:57 And I think that's even something even for teams that have those three month release cycles, like where like maybe the business for some reason can only ship every three months.

24:05 But then you need to find a way where push it to a staging environment, push it to like a QA environment, like push continuously somewhere.

24:13 And make sure that that somewhere runs correctly and have some like sanity checks, like write a few test cases that just go through the happy paths and like one sad path of that application.

24:23 And yeah, so happy path and sad path is basically like a test that should work and a test that shouldn't work as expected and walking through the application.

24:32 And if you just, even though, you know, you're just releasing this in three months, but you know, today, like all the changes that have been happening over the three months have been pushed somewhere.

24:40 They've been tested somewhere.

24:41 Somebody looked at those somewhere.

24:43 On a continuous basis.

24:44 So once the release comes, there is no surprise anymore.

24:47 There is no, no, nothing new that's coming in.

24:50 It just, it's all been running somewhere.

24:51 And, and I think that's something that every team can do.

24:54 I think that's, that's, that there might be like, maybe somebody has some exotic example where this is absolutely not possible for some reason.

25:01 But I think for the vast majority of teams, that's something that should not be too hard to do.

25:06 And I think most people should be able to do that in some way or another, just to get that feedback early as possible to like make that feedback loop as small as possible and, and, and get going there.

25:18 I find usually when people say that it's not possible, that usually is like some kind of sign that something's not quite put together right in the app.

25:26 Right.

25:27 So there's no way to test this part because it calls into this thing, which goes into this whole infrastructure, which does that other, you know, maybe, you know, you need some, some way to break your app up.

25:38 Some design patterns, dependency injection, things like this, that will actually allow you to break your app into the small pieces that you can test and should be testing.

25:46 Agreed.

25:47 If it's too complex to test it, then it's probably just too complex for anything.

25:50 Yeah, that's, that's a great, great way to put it.

25:53 I have my own opinions on this, but I'll ask, I'll get yours.

25:57 Is there like a size of a team where you should start doing continuous integration?

26:02 Like if there's only two of us, should we do this?

26:05 Is it worth it?

26:06 And is there an upper bound?

26:08 Like, you know, if you have 200 people, it's like just too, too crazy, too complicated.

26:12 Yeah, I think I'm, I'm obviously like totally biased in that regard.

26:17 But I think, yeah, there is definitely, I think you should do CI on, on every level of, on every team size.

26:23 I think it just, the approach to CI becomes different on, on different team sizes.

26:28 I think that especially like if you're one or two people, like early product, just a proof of concept or something,

26:35 you might just want to do something where you write like a few tests and then it depends on like, do you have customers or not?

26:41 But in the end, like you, you don't want to break stuff for your customers, especially not your early customers, because they're the ones, even though they expect it to break.

26:48 They're the ones that can be the most committed evangelist for your product.

26:52 So if you break it for them, then you just lose basically your whole marketing team in the early days.

26:57 Like the evangelist, the people that you, the outside people, the outside users and customers, your early users are your marketing team.

27:04 They are the ones who are going to talk to everybody else about your product.

27:07 So you don't want to break it for them constantly.

27:09 So what you can do there, instead of writing like a million small unit tests, you just make sure that you have like 10, 15, 20, very high level tests that make sure like the most important workflows of your application aren't broken.

27:21 Like, so people can sign up, people can like, whatever is the main business function of your product.

27:26 People can do that.

27:27 Like the main things and, and, and those can be easy to write.

27:30 They, they can be on a very high level.

27:32 So they don't break constantly, even though you're changing your application and they just make sure that like your customers can do at least the most important stuff on a constant basis or like the, especially early users.

27:43 Like if some small feature on the sidelines gets broken at some point, like they'll miss it and they'll tell you, but it's probably not going to break everything.

27:51 For them and they're probably not going to like leave the product or like leave the, the, the, like evangelizing for you that quickly.

27:58 But you should write something where like at least the most important and main paths of your application are covered so that you can push faster.

28:06 I think that's also something like in your early times, like you want to move so fast, like you want to push your product out the door as quickly as possible.

28:12 And then again, like if you push something out the door that breaks other features and you don't know it and you like, you only know, like after a couple of days that it happened,

28:20 and you don't know exactly what broke, like it takes so much time away from, from actually doing something productive that spending a little time on like writing good tests and writing tests from the right perspective of the user on a very high level actually makes a lot more sense.

28:36 So I think the, the larger the team gets, the larger the code base gets, you want to like go lower in, in that stack in terms of like high level tests.

28:43 So you go away from like, maybe in the beginning, you just write tests like with Selenium or something where you actually like, you just point a browser at your application and just click through your application automatically.

28:52 So you actually know that like these things work on a relatively low level as well and not just on the higher level, but you still want to keep that higher level tested very well.

29:13 So you know, because in the end, like that's what customers care about.

29:16 Like a customer doesn't care if the unit test for a validation on a model works or doesn't like, that's not important to them.

29:23 The important part is, does the workflow work?

29:25 Does the product work?

29:26 Does the feature work?

29:27 So that needs to be tested in my opinion at first.

29:30 And once you've tested like those few happy and sad paths on the upper level, then you can go like, okay, I've tested like that a validation in generally fires and, and, and error message gets shown on the web UI.

29:43 So now let's, let me write like 10 more tests about like all these different validations that we run on the model before we save it.

29:50 and, and before that works, because I don't want to test all of them on a very high level because that would be really slow.

29:55 So you want to have that mix of like slow, very high level tests to make sure that from a customer's perspective, it works.

30:01 And then lots and lots of like smaller level tests that verify specific things that you, you wouldn't want to verify on the higher level because it would just take too long.

30:12 So that's at least the mix that I, that I found and that we found works pretty well.

30:17 And that really captures and catches a lot of problems early on that, that might impact your, your customers.

30:23 That's a really pragmatic and powerful way to look at things.

30:28 I guess the way I would kind of summarize that is don't let the perfect be the enemy of the good.

30:33 You know, don't let, don't let the goal for saying we must have a hundred percent test driven development.

30:38 We must have 95% code coverage push you into a point where you say, well, but we have to go faster.

30:43 We're just going to do zero.

30:45 Yeah, no, totally.

30:46 I agree.

30:47 And I think that's, it's, it's a misconception, misconception that that makes you faster, that not writing test makes you faster.

30:53 It just makes you, it only makes you faster.

30:56 If you don't count the times where you have to actually fix things.

30:59 Like if you do that, then it's just not going to make you any faster.

31:03 It's just going to make you slower in, in my opinion or, and in my experience, like that's what we, the experience that we've had from, from the teams that I've worked in and from many, many, many different customers that I've talked to.

31:15 And I've worked with that writing, you have to do it the right way.

31:18 Like, and I've seen teams that have gone up from like early teams that have written thousands and thousands of low level unit tests, but their application still broke constantly.

31:27 Like that's not really the right way to go.

31:30 Like the, in my opinion, the right way to go is really see it from a, from a customer perspective and whatever the customer needs to make sure that it validates well and that it works well.

31:39 That's what you should focus on.

31:40 Yeah, that's a good point.

31:41 And my rule of thumb is kind of when I, when I look at the, the part that I'm trying to test or think about whether I should write a test for it, I think is the customer actually paying me for this part of my application?

31:55 You know, is, is this my core business?

31:57 So like, is this the trading engine?

31:58 If I work at like a, a stock based company or is this, you know, the continuous integration execution part, if I work at CodeShip or something like, does the, the, about our team page work or not?

32:11 Like, you know, I don't really care and I'm not going to put energy into it, but if people are paying me for that part, right?

32:16 My company, then probably that part should have a test.

32:18 Yeah, exactly.

32:19 And I think it can be simple as like, open up the page and see if it, if the page throws an error, like it doesn't have to be like, that's like for, for a lot of things.

32:28 And that's something, for example, like when we deploy something new, like we deploy into a staging environment and just simply open up the application.

32:34 And that captured a lot of different problems from, or like not a lot, but like from time to time, it captures something where maybe we've done something wrong in the configuration of the page and just the whole web app doesn't load.

32:43 And it really pays off to just load one single website or like one single page of the application just to make sure like, does this thing actually boot when we push it somewhere?

32:53 And that definitely captured a couple of times where like, we would have pushed something out that, that wouldn't have, that would have gone bad.

32:58 And it didn't.

33:00 And it was very easy to catch and very easy to deploy a fix for that.

33:02 And, and it never saw, saw the light of a customer.

33:05 But I think that putting that, that in place.

33:08 And I think especially the about page is, I think a good example, because that's something like you wouldn't put a lot of effort into setting up an infrastructure to test your about page.

33:16 So that should just something that should be there because you need it for many other things as well, like to, to just make sure that your about page is actually loaded and can actually be shown is something that shouldn't take more than a few seconds basically to implement in, in a test.

33:30 But it's really critical because you are, you also don't want to like, you don't want your about page.

33:35 Like if somebody looks up your website and the about page breaks, that's kind of like, that's not really a good sign.

33:40 Like, especially for a tech product, like I would at least expect the about page to load.

33:44 And I think that's some of the things where like, it should be easy.

33:48 It should be fast to, to test that.

33:50 Then there should be like a lot of different ways how you can do that.

33:52 But yeah, your developers shouldn't have to spend a lot of time on, on doing that.

33:55 Yeah, I agree.

33:56 And like you said, just making requests to the page and seeing if a 200 rather than a 500 comes back actually can do quite a bit.

34:04 Yeah, that catches a lot of stuff.

34:05 This episode is brought to you by DigitalOcean.

34:23 DigitalOcean offers simple cloud infrastructure built for developers.

34:27 Over half a million developers deployed to DigitalOcean because it's easy to get started, flexible for scale, and just plain awesome.

34:34 In fact, DigitalOcean provides key infrastructure for delivering Talk Python episodes every day.

34:41 When you, or your podcast client, download an episode, it comes straight out of a custom Flask app built on DigitalOcean, and it's been bulletproof.

34:48 On release days, the measured bandwidth on my single $10 a month server jumps to over 900 megabit per second for sustained periods, and there's no trouble.

34:58 That's because they provide great servers on great hardware at a great price.

35:02 Head on over to DigitalOcean.com today and use the promo code TALKPYTHON, all caps, no spaces, to get started with a $10 credit.

35:10 In the early days, back in 2008, 2009, 2010, it was all about continuous integration and making sure that we could check in our code,

35:29 that if we're using a compiled language, it compiles for writing tests, the tests run in an automated sort of way.

35:36 But since then, people have been moving on to actually turning this whole system into actually a deployment mechanism, right?

35:43 Like I can check in and magically stuff is in production.

35:46 And that's called continuous delivery, right?

35:48 The main idea with continuous delivery is that you keep your application in a state which either can be or is deployed somewhere constantly.

35:58 And I think the important part that can be through CI, that can be by deploying it to a staging environment,

36:02 but that can also be by just pushing it out there on a continuous basis.

36:06 How would I go about that?

36:08 Like I totally know how I check into Git and I set up a continuous integration server to pull that and run my tests.

36:13 But how do I get that, say, into my, like if I wanted to do continuous delivery,

36:19 how would I actually make changes to my server in AWS, for example, or DigitalOcean, something like that?

36:25 So I think the important part there is that to take one quick step back and something that we find really important for continuous delivery in general is how you start triggering those changes.

36:36 Basically, I think continuous delivery and the whole deployment part of continuous delivery really needs to be also fully automated and done by like the CI or continuous delivery system for you.

36:49 Because in the end, like what you want, even like on a daily basis, like if you deploy all the time, you don't actually want your developers to think about like, okay, now I actually have to deploy this.

36:58 Like what you want them to do is here is code and like I'm merging code into master and just do stuff to it.

37:06 Like, but, but you want them to like focus on the code all the time, like on a, on a daily basis.

37:11 You, you don't want anybody to have to think about like deployment strategy and how, like which script do I run in which way and where do I get which credentials, all that kind of stuff.

37:19 So I think the important part and the way we see it is in terms of triggering it is through the repository.

37:25 So we think the repository, is a really great way to capture the intent of releasing something.

37:32 Like if you merge something from a feature branch into the master branch, like there is some intent to like test it, but also put it somewhere.

37:39 So you can use that and just say, basically whenever you're on a specific branch, like run all the test, commands and then also release them to, to production.

37:49 But at that point, like the developer can already be working on something completely different.

37:53 and, and just is already jumped into the next task and, and works on the next thing, next thing, or, or gets a coffee or has a chat with somebody else on the team, but they don't have to actually think about this.

38:03 I think that's, that's a core part of continuous delivery.

38:06 because you really only want your developers to focus on code on a daily basis.

38:10 That also helps with a couple of other things because it's easy to move between different parts of the infrastructure then because so for example, it doesn't matter.

38:18 If you're working on like, if you want to release changes to your DNS system or your main web application.

38:22 If all the things you need to do is merge code from a feature branch into the master branch and then it gets released.

38:27 Like that's easy to understand.

38:30 for anybody and anybody can go into any part of the, the infrastructure application.

38:35 And, and actually do it then.

38:36 And from then on, it depends, totally depends on like the specific technologies.

38:39 If you want to use something like a platform as a service, like a Roku or an AWS elastic beanstalk or have your own servers, different ways, different tools, different abilities.

38:49 But I think the important part is really capturing it in an automated fashion, but in a way where like it just happens kind of automatically for, for developers.

38:58 Like they know it happens, but they don't have to consciously think about it.

39:01 Like at some point, like that merge button on GitHub in a pull request is just like you're merging code and then you're off to the next thing.

39:08 And you don't really actually think about the deployment at all at that point anymore.

39:12 Yeah.

39:12 That's, that's really cool.

39:13 Deploying modern software can involve a lot of infrastructure and a lot of pieces, especially if you're doing like microservices, it can involve a whole bunch of them.

39:22 Right.

39:23 And so if you can automate that, even if it takes you a week to automate that, and normally it would take you a half hour to do a deploy, the speed that you get on the other side of that, the ability to, like you said, push a button, merge to a branch and just magically new software is out.

39:37 That's, that's really cool.

39:38 Right.

39:38 Totally.

39:39 I think that's, it's just more of the, I think it's more of that focus and productivity for your engineering team.

39:45 Like that's something that is always important.

39:48 It's really, really, really important that the engineering team can focus as much as possible on getting new stuff out there and building new features and product for, for customers and anything else that is in the way.

40:00 And that is manual testing that is thinking about the release process in terms of like, how do I actually get this out the door?

40:06 That's all wasted time.

40:07 Like that just shouldn't be.

40:09 And it's not just wasted time on like existing engineers who actually know that, but actually when you onboard a new engineer and you have to walk them through like every single way that you deploy all the different systems.

40:19 It, that's just a huge time investment that you need to do.

40:23 And, and on the other hand, if you just tell them like, yeah, open up a feature branch and like that, then just merge it into the master branch.

40:29 And like, this is just a technology, like technologically, you, you know, like it's just a, I don't know, Django or a rails application.

40:36 Like, you know how to work with that.

40:38 Like our infrastructure might be a little bit different than what you've seen in the past, but you don't really have to worry about that at all.

40:44 Like just do your code and like, just merge it in.

40:46 And over time you'll learn more about like how actually the whole system works.

40:51 And I think it takes away some of that stress of onboarding new people.

40:56 And that, like that, that wish that a lot of companies have where like you, you have somebody new and like they should release something on day number one, which I'm not the biggest fan of.

41:04 But at least in the first week, I think that it just puts a lot of pressure, unnecessary pressure on everybody to have them release on day number one.

41:11 But I think it removes a lot of that, that, that, the pressure, like the problem with like showing somebody how that stuff actually works.

41:18 And instead, like you tell them five minutes how the workflow works and then they couldn't do it anywhere from, as I mentioned, the DNS system to like your main application, to your back end application, to whatever.

41:30 Yeah, that makes a lot of sense.

41:32 You talked about earlier having different levels of tests and running them possibly at different times.

41:37 Obviously, you would want to run your tests before you go through and push it through the continuous delivery path.

41:43 Do you see people having a different set of tests for like a regular feature branch check-in versus this final step?

41:52 Would people write, like run more tests if maybe their tests are slow or something?

41:55 Sometimes.

41:56 I think that's mostly due to time constraints, though.

41:59 I think that's, and I think that's also something like that.

42:02 I mean, that's one of the challenges that we try to solve for them.

42:04 It's basically giving them a system that makes it so fast that they can run anything in parallel.

42:08 But yeah, we definitely see that where people like they have a lot of like they run their maybe unit tests on every feature branch and run like their full on like integration or like functional testing suite only on like the master branch.

42:24 This definitely happens.

42:25 But yeah, I think so from what I've seen, I haven't seen a team that says I don't want to run it on like all the tests on my branch.

42:33 It's more I can't run it there because it's just too slow.

42:36 So I think it's a lot of it is just down to like optimizations in the test suite and like making it faster and making it more parallelized, all that kind of stuff.

42:43 Yeah.

42:44 If it takes two hours to give you feedback, even on your feature branch, it's maybe better to save it for later.

42:49 So you know right away whether it's mostly good, right?

42:51 Yeah, exactly.

42:52 Exactly.

42:53 And I think that's so what we've seen.

42:54 I think like a team should get their whole build on like a feature branch.

43:00 I'm not including deployments on a master, but just like running their tests.

43:03 If you can get it down to like two or three minutes, that's below the level of like where people actually have to think about it.

43:09 So when I was thinking about it, like as a developer, like I code something locally, I commit something locally, I push it into the repository.

43:16 And if I get an answer, if that works or not in like two or three minutes or in less than that, I actually, I don't think about it actually again.

43:25 Like I'm still coding on the next thing.

43:27 And at some point, like a notification pops up that tells me, hey, everything worked on that thing that you just pushed or a notification pops up.

43:33 It didn't work.

43:34 And like you have to like fix something that you broke on that feature branch.

43:38 But it's not like while I'm still working on the other thing, I'm not thinking like, hey, did this already finish or is this still running?

43:43 I think that's like two or three minute like level.

43:46 I think anything beyond that.

43:47 Then the problem is that you're sitting there, you're coding on your stuff.

43:51 You're thinking, hey, wait, did my build already finish?

43:54 Let's just look that up.

43:55 So you go to a browser, you open up the build, you open up the build page, and you've just lost a little bit of time.

44:00 You lost focus from your development effort.

44:02 And like over time, like if you have a large team and like a lot of people that have to do this constantly, that just is a lot of focus lost from actually developing.

44:12 So I think we like our goal is really to get any team of any size to a point where you can actually run the builds in like this two or three minute time window where you're not actually actively thinking about the results of your build.

44:27 And you just basically keep working and you just constantly keep working and you just get notified, hey, this thing worked and now this thing worked and now this thing worked without having to go back.

44:36 And I think that's something that in my opinion, people really should strive for.

44:40 Again, it's all down to productivity and focus for your engineers.

44:44 Anything that takes even slight productivity and focus away from your engineers is something that actually hurts your bottom line.

44:50 And so that just you have to get rid of that.

44:54 Yeah, there's a lot of talk in programming about flow and focus and getting into the zone.

44:58 And these things just can really kill that if you're like half still thinking, well, the thing that I did, is it actually okay?

45:05 Or can I go forward assuming it's okay?

45:07 Or, you know, do you know because the build finished that you're fine, right?

45:11 Exactly, exactly.

45:12 And if it just takes 15 minutes, like you're just going to come back, like you're just going to open up the page.

45:16 And like then you open the page, you wait for the build to finish, like you open up Reddit or something.

45:20 And like it's just like all that kind of stuff is just it's taking away that focus.

45:24 And I think it's really important to keep that flow, to keep that focus going.

45:27 Sure.

45:28 So what's the longest or the craziest build you've seen running?

45:32 Definitely people who were like, so where it's been over, like we ran it through the night.

45:39 And at some point, like we just we just stopped because it just took too long, basically.

45:43 I mean, there's definitely things where like it takes hours and hours.

45:45 And I think a lot of it comes down to, especially with the teams that we work with, they're not like huge enterprise, like 48 hours straight, like running through like 100% CPU usage.

45:57 It's often just comes down to optimization.

45:59 I think oftentimes teams, they build their test suite and it gets progressively slower.

46:05 Like no team starts out with like a two minute build.

46:09 And then like three weeks later, it's like at an hour.

46:12 Like it just happens over a longer time.

46:14 Like it's two minutes, then it's three minutes, then it's five minutes, eight minutes, 15 minutes.

46:18 And just it gets slower and slower and slower.

46:21 And the team never really does the decision to like make this faster and look into it.

46:26 So I've certainly seen teams where the tests have been running and they've been running slow, but they've only been running slow because they didn't really look it.

46:34 And I mean, that has been the case with us, like a couple of like or a year ago, like one of our engineers joined and he's really strong on like optimizing test suites.

46:42 Like that's, that was, he's been doing like for a while and he got our test suite down from 15 minutes to like three or something.

46:49 And that's, and we're pretty good with like test suites.

46:53 And I think that's something where like he just invested the time.

46:56 He had the experience, knowledge about it.

46:57 Obviously he then blogged about it on our blog.

46:59 So we always try to keep that up.

47:02 But that really helps like spending time on actually making your test suite faster and seeing where it's slow and how to make it faster.

47:09 And actually like setting an SLA.

47:10 Like I think as a team, you should set an SLA on your test suite.

47:15 And if it just takes longer than that SLA, you just put resources on it to make it faster.

47:20 Because otherwise, if you're just progressively letting it get slower and slower and slower, again, it'll just stop the process.

47:27 Like continuous delivery really relies on your test suite being fast.

47:31 Because if it's not, then your developers have to constantly wait for the results of what's happening.

47:36 And there's nothing worse than having to wait on like the build result getting in before you can open up a pull request.

47:42 And like you can ask other people for like feedback or maybe somebody has already looked into it and like given you feedback and done a code review.

47:50 But then the build comes back half an hour later and tells you, wait, this actually failed.

47:54 Like you have to recode it in some way.

47:56 And then they do have to do another, go over it and do a code review.

48:00 And like that just takes so much productivity away from you.

48:03 Yeah.

48:04 And wait again, right?

48:05 Wait that 15 minutes again.

48:06 Exactly.

48:08 So like once you feel like, and I think having a strict rule of like our build should be that fast.

48:14 And then dedicating resources to it.

48:16 It's so much cheaper than like just waiting, like putting the resources on features and just waiting.

48:21 Like it's not going to magically become faster.

48:24 And it's just going to block your team like crazy all the time.

48:27 And it's just so frustrating if the test suite is slow that you really should dedicate resources to it.

48:33 It's just, it's developer productivity.

48:35 Developer productivity is really important.

48:36 Developers are really expensive and it's really hard to get new ones.

48:40 So you really, really should make sure that your team is productive, your team is happy and can actually work on the most important tasks.

48:47 And yeah, making your test suites fast is definitely something that pays off very quickly.

48:53 Yeah, totally agree.

48:54 When you're talking about continuous delivery, the putting the new version of the source code or the compiled bits on the server and making that run, it's pretty straightforward.

49:03 But usually it gets really sort of tricky when you're talking about databases.

49:07 Yep.

49:08 What do you see people doing there?

49:09 Putting a lot of it into services.

49:12 I think that's something that's a relatively consistent theme that we've seen is like maintaining databases is really, really hard.

49:19 I think continuous delivery is definitely something where you can't, whenever you store state, that's really hard to continuously deliver.

49:28 And so I think a lot of teams, and that's certainly us as well.

49:32 We just put that off to a service and we just let Heroku in our case or Amazon RDS deal with that because they have the infrastructure and the knowledge to actually do that.

49:43 If you run your own data system, it's definitely possible to do that with the follower, like do some level of replication and then switching that over.

49:50 But it's generally really, really hard.

49:52 And the question is, whenever I look at like how often we deploy our web application and how often we would have to upgrade our database, it's not really that big of a deal to deploy.

50:04 Like you don't really have to deploy like some infrastructure parts and parts where you store state that often compared to like the things that you that don't store state and that you can actually push constantly.

50:15 So I think a lot of teams don't even have to really spend a lot of cycle on like, how do I actually continuously deploy like my database?

50:24 Because oftentimes it's just fine enough to do it like on a Sunday morning and like do a one hour window there where you like just do a follower, switch over and then stop out the other one.

50:35 So I think that's something where I think it's again, like don't let perfect be the enemy of good.

50:41 And I think really focusing the continuous delivery on the parts that you actually can control very easily, mainly being like everything that you don't have stayed in makes sense.

50:50 And then I think putting the other systems into like a service, putting it into like a Roku, into an RDS, at least for us, I think that's like, I'm not, I'm not in the business of maintaining databases.

51:02 Like that's not my product.

51:04 I just need a database.

51:06 So if somebody else can provide a really good product around the database, that's totally worth it for me.

51:11 And so that I don't have to do that.

51:12 And again, put my developers on a much more productive for our customers, much more productive and for our business, much more productive path.

51:19 So I think that's how we typically try to deal with it.

51:23 But I think, yeah, for most companies, if you really, really have to do like zero downtime database updates, then you're, you're probably in a position where like you have enough resources to do that.

51:35 Because if you, if you're really small, but you have to do like zero downtime database deployments, you're something on the business sounds a little bit off because that sounds really hard to do.

51:47 Like if you have little customers and you're a small team, but the customers don't pay you a lot.

51:51 But on the other hand, they, they still can require like really hard technical things like continuously updating or like zero downtime deploying new databases.

51:59 Like that's not easy.

52:00 Like then there's probably, you're not making, like you should, you should totally get more money from them.

52:05 But on the other hand, I think that's, yeah, just don't worry too much about it.

52:09 If you can just get away with a like Sunday early morning update.

52:13 I think it's just, it's just really hard for small teams to do that.

52:15 Yeah, it definitely is.

52:16 So one of the things I saw you guys talking about is Docker.

52:20 And I think Docker is going to become increasingly important in the whole way that we host and maintain and evolve our applications.

52:28 What's the story of Docker?

52:30 So generally I fully agree.

52:32 I think it's, it's definitely a very important part of how we will build Docker and like containers and generally like very low level and easy to do virtualization is a big part of how we're going to build infrastructure in the future.

52:46 So for me, a lot of the things that are important there are that when the system can maintain itself, it just, you, you easily push a container out that has everything in there that to actually run your application, which also makes that separation between developers and operations much easier because the developers can actually fully control what's actually running as part of their, their application.

53:08 But it's still fast.

53:09 But it's still fast.

53:09 So that's really important.

53:11 And on the test and continuous integration side, obviously, it's really key for us in the future.

53:17 So we've just rebuilt our whole system basically to run on top of Docker.

53:20 And, and, and the main idea there is that developers should be able to control their build environment fully.

53:27 And each repository is different.

53:29 Each repository and each application needs a different setup.

53:33 And, and Docker can definitely provide that.

53:36 And that's just not limited to like code chip.

53:38 I think that's, many people are using it for many different ways in their, their test and, and, and CI setup, but it's really about giving the development team a lot more power about the infrastructure that the code is running in.

53:50 and so that the operations team, like there's a clear separation between like how to run the whole container.

53:57 Like that's just a clear interface.

53:58 And then whatever's inside of it, the development team has full control over that.

54:02 And I think that's like that separation of concern that you don't just throw code over the wall basically, and just let the operations team or somebody else deal with it, is really key.

54:12 And then you can do a lot of, a lot more automation around it, like a lot more health checks, a lot easier way to like, run several instances of that, get it up and running again.

54:21 When something goes down, really use the most of your resources, in your machines.

54:26 So I think Docker and, and the way I think it's less about just the technology of Docker and like Docker as a company, although I think they're doing great and, and they're providing great, a great product.

54:36 I think the more important part there is that it really enables, developers and the whole team to take full control over their environment and work in a different way where it's not just about like, let's update those three servers that we're running, but just.

54:51 Let's push this artifact into like our environment.

54:54 And then our environment decides where, when, and in how many items this artifact is running.

55:00 So I think that's something that is, is really critical where you have a lot of more separation of concerns there where you don't have to think about everything all the time.

55:08 And there's just a system taking care of, of a lot of this, and nice abstractions in place, to make this, this run and work.

55:16 Yeah, that's cool.

55:17 It seems like Docker and continuous delivery go well together because now you're almost able to deliver the infrastructure.

55:24 As part of this, right?

55:25 Here's the Docker container for this version plus my code, right?

55:29 Yeah, absolutely.

55:30 You just have a bunch of servers running and that can be in a cloud that can be as a cloud service that can be on your own infrastructure.

55:35 And all that thing provides is just a way to run arbitrary Docker containers.

55:40 And I think that's really powerful because again, like the development team can take full charge.

55:45 Like no, no operations person has to know which version of Java runs in there and they don't have to be involved in our Python or Ruby or whatever.

55:52 They don't have to be involved in like updating that or making sure that like all the different projects that are running as part of the company all need to use the same Java version because otherwise like stuff will break all the time.

56:04 Like it totally frees every team, every repository, every developer basically to use whatever is best for that specific problem and not have to deal with like all different complexities of running application, like many different applications on the same kind of infrastructure.

56:20 And virtualization in the past has definitely gone a long way there.

56:23 But then it's just like, it takes a lot more resources to do that.

56:27 And with Docker and containers, it just takes a lot less resources that the underlying technology has been, is proven.

56:32 Like Google runs their operating system on it.

56:34 Facebook runs, runs their operating, like their data centers on it.

56:38 So I think we can say that this actually scales to whatever.

56:42 I think we just need the tools to make it scaled down.

56:44 Like we just need the tools to make it scaled down to like small teams.

56:47 And I think that's what really Docker is providing a great tooling for that we can use containers for like a lot of different stuff, a lot of different workflows, a lot of different processes in our either production systems, but also in, in our whole build environment, in our whole build system.

57:04 So I think that's where it's, it's really about control and giving the developers the utmost level of control about everything they're running as part of their application and not have other people either take away control or have to worry about that control at all.

57:17 So I think you're absolutely right.

57:19 I think Docker just moves a little bit more of that control back to the software team.

57:25 And by virtue of that moves it into this continuous integration story, right?

57:30 So we can build out these Docker containers and I can say, you know, look, if we're going to switch from, let's say Python 3.4 to 3.5 and use some of the new features there, I don't have to make sure I coordinate with the infrastructure team to make sure that the right versions of Python are on the servers we're going to push to.

57:46 When we do continuous delivery, we just say, well, we're just going to push a different version of the container and boom, it works, right?

57:52 Yeah, exactly.

57:53 And on different branches, maybe like on a different branch, you want to test against the new version of Python.

57:58 Because like on the master, you're still running against a different version of Python, but on the feature branch, because you want to update, you're running against new version there.

58:04 So you want the whole build environment and the whole deployment environment to run on a different version.

58:09 So nothing other than that, than just changing it in a config file and using a different one.

58:13 That's a really interesting component to it as well, because, you know, as you move forward, you're sort of moving forward in time on your infrastructure and all the other pieces as well on your, your branch.

58:24 And so if you need to go back one month and run code from back then, if maybe solve a bug that somebody reported on a different version of your code or something, you, you know, to really do that.

58:34 Genuinely, you have to like somehow roll back your system to that level, right?

58:39 But if, if the Docker container specification is part of that, that check-in history, you can literally roll back to what you built it on, right?

58:47 Yeah, absolutely.

58:48 You can have the exact same.

58:50 I think that's, I mean, and as I mentioned before, like that's, that's why we built a new system on top of Docker to be able to exactly support that.

58:56 Like that should be easy.

58:57 Like you should be able to run, to roll back to like your build workflow, your code, your environment that you built, like everything.

59:03 You should be able to roll back everything and like have a complete history of, of everything that happened as part of your, your build and even production system.

59:12 Because only then, like, as you mentioned before, like if you want to do a rollback, if you want to fix something, if you want to look into something, debug something, there isn't really any other way to do it than to be able to go back historically and like really look up what exactly happened there.

59:24 Yeah.

59:24 Yeah.

59:25 Very cool.

59:25 So yeah, we'll see more Docker in the future.

59:28 I'm sure.

59:28 I'm very sure.

59:30 Yeah, definitely.

59:31 Okay.

59:33 So we're getting kind of to the end of the show.

59:35 Do you have a final call to action or something you'd like to make sure people go out and try and do?

59:41 So two things on, on, on, on, on the one side of obviously if, if people are interested in like a Docker based, CI system, give us a call.

59:48 we're happy to show you guys, what we're doing there.

59:51 But I think on a, on a non coaching basis, I think we really, really, really need to need to make sure that as a community, we focus more on the productivity and focus of our engineering teams.

01:00:04 Like that needs to be number one, like customer success, obviously.

01:00:07 And then, but that is driven and done through productivity and focus of, of our engineering team.

01:00:14 So I think that anything, and we deeply need to analyze on a daily basis, like how is our engineering doing things?

01:00:20 What can we change?

01:00:21 What's in their way?

01:00:22 How can we improve the processes?

01:00:24 What can we automate?

01:00:25 How can we automate operations more?

01:00:28 How can we automate automatic health checks so that there's no downtimes and no issues or problems there?

01:00:32 Like how can we do all of that, so that they can actually focus on building stuff for our customers that actually makes us money and is interested in new technology.

01:00:41 So I think that's something that is, is really key and really important.

01:00:45 And that would be my main takeaway for, for people like really, really focus on those processes to make sure that your team can actually focus as much as possible on to building stuff for your customers.

01:00:58 Yeah, that's great.

01:00:58 I really love the focus on team happiness because I think it often gets lost in the technical bits of this story, right?

01:01:06 Yes, absolutely.

01:01:08 Flo, it's been great talking to you.

01:01:10 Thanks for being on the show.

01:01:11 Yeah, thanks for having me.

01:01:13 It was great.

01:01:13 You bet.

01:01:14 Talk to you later.

01:01:14 This has been another episode of Talk Python to Me.

01:01:18 Today's guest with Florian Motlick.

01:01:20 And this episode has been sponsored by Hired and DigitalOcean.

01:01:23 Thank you both for supporting the show.

01:01:25 Hired wants to help you find your next big thing.

01:01:28 Visit Hired.com slash Talk Python To Me to get five or more offers with salary and equity presented right up front and a special listener signing bonus of $4,000.

01:01:36 DigitalOcean is amazing hosting blended with simplicity and crazy affordability.

01:01:42 Create an account and within 60 seconds, you can have a Linux server with a 30 gig SSD at your command.

01:01:48 Seriously, I do this all the time.

01:01:49 Remember the discount code too.

01:01:51 It's Talk Python, all caps, no spaces.

01:01:54 You can find the links from today's show at talkpython.fm/episode slash show slash 38.

01:02:00 Be sure to subscribe to the show.

01:02:03 Open your favorite podcatcher and search for Python.

01:02:05 We should be right at the top.

01:02:06 You can also find the iTunes and direct RSS feeds in the footer of the website.

01:02:11 Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.

01:02:15 You can hear the entire song on talkpython.fm.

01:02:18 Just look for music in the nav bar.

01:02:20 This is your host, Michael Kennedy.

01:02:23 Thank you so much for listening.

01:02:24 Smix, take us out of here.

01:02:26 I'll see you next time.

01:02:47 you I'll put it in.

01:02:48 Thank you.