#166: Continuous delivery with Python Transcript

00:00 Michael Kennedy: We've evolved from, if it builds, ship it, to continuous integration where every check-in is automatically verified by something like Travis CI. Taking that further, some people today are using continuous delivery. This means once a check-in is validated by the CI system, it's deployed automatically. There are many moving parts in these processes. On this episode, you'll meet Cris Medina who has put together a world-class CI/CD system, and he's here to share how he did it and what tools and libraries are involved. This is Talk Python to Me, Episode 166, recorded June 11th, 2018. Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm @MKennedy. Keep up with the show and listen to past episodes at talkpython.fm, and follow the show on Twitter via @talkpython. This episode is sponsored by Linode and Rollbar. Please check out what they're offering during their segments. It really helps support the show. Cris, welcome to Talk Python.

01:17 Cris Medina: Hi, how's it going? Glad to be here.

01:19 Michael Kennedy: It's going great. I'm happy to have you here. It feels like we were just hanging out in Cleveland just a little while ago, right?

01:26 Cris Medina: Just a little bit ago.

01:28 Michael Kennedy: Everybody scattered back to where they came from, and it's sad that PyCon is over, but that was really fun to spend some time together there.

01:35 Cris Medina: For sure, it was great to meet the PyBytes guys, too. I hadn't met them before, so that was pretty cool.

01:39 Michael Kennedy: They traveled quite far to get there.

01:42 Cris Medina: From both sides of the globe.

01:44 Michael Kennedy: I mean PyCon is such a special place, and PyCon US seems to be where the most gravity is. I know Euro Python is also large, but I feel like that PyCon US is probably the biggest. Just the sense that I get. Anyway, I think just, recommend that people next year, if they didn't get a chance to go, definitely go. Like, wouldn't you say? Did you have a good time?

02:04 Cris Medina: Definitely. I started doing PyCon maybe three years ago now, and it's definitely been a lot more interesting, to so much stuff from so many different people doing so many different things with the language. 'Cause Python has such a wide usage. It's just great to just be out there and just see what everybody's doing now. I like just hanging out in the expo hall and just talking to everybody. See like, oh, what do you do with Python?

02:32 Michael Kennedy: Absolutely.

02:33 Cris Medina: Definitely, the open sessions. People don't quite understand how cool that is, versus your average conference where everything is just kind of pre-configured for you.

02:43 Michael Kennedy: Definitely. Both you and I ran some open sessions, right?

02:46 Cris Medina: That's true. Yup.

02:47 Michael Kennedy: What were yours on?

02:48 Cris Medina: I did two. I did one on just blogging and stuff. Folks that are creators for Python. I did one on virtual reality augmented reality to see how experiences with people and Python, and that type of stuff with those environments.

03:03 Michael Kennedy: That's pretty awesome. Now that you're back to doing what you do day-to-day, maybe we could get your story and your background. Tell us how you got into programming Python and what you do day-to-day.

03:12 Cris Medina: My story's more like a classic story. Dad grew up doing software. He had a consulting business. There's stories of two year old Cris sitting in his lap typing into the keyboard somehow. Not that any of that was intelligible, obviously. So when he started stuff with old IBM systems, like monochrome monitor, 8 1/2-inch floppy, integrated keyboard, no hard drive things.

03:34 Michael Kennedy: No hard drive. That still blows my mind that computers came with no hard drives.

03:37 Cris Medina: I remember the first time I actually got an idea of what software was is 'cause my dad wrote something for me to sit in front of his computer and learn the times tables. It's just a short, little program, but I could go into the program and fiddle with it. I was really young for that, and I still remember it was pretty cool. It was an IBM System/23.

04:03 Michael Kennedy: Nice. Would it like, randomly pick two numbers and ask you what the answer was, and say you got it right or wrong, basically?

04:10 Cris Medina: Pretty much.

04:11 Michael Kennedy: That's pretty nice. Like interactive flashcards, basically.

04:12 Cris Medina: I remember doing data entry for him back in the days where you can slurp anything in from any APIs, or anything like that, right? Somebody would just hand-deliver an invoice and you had to type it into the computer.

04:23 Michael Kennedy: When computers were not actually connected to anything? They were just there on a desk? How weird, right?

04:29 Cris Medina: Right. Exactly. Some of the first stuff I did was system System/360 Basic. My hello world was like a menu for opening an invoicing app, or something like that. Back when you had to type line numbers and make sure you left enough room between them in case you needed to add more lines before.

04:47 Michael Kennedy: That's right. Like, if people don't know this, that used to be a big deal. It used to say 10 would be like a line number, and then you'd put a command, and then 20, you'd put another. And the reason it didn't go one, two, three, is you might have to do 11, 12 and 13 someday and you don't want to have to rewrite the whole program. It's so insane that you had to explicitly call out the line numbers, but I guess with the goto navigate, sort of branching mechanism you had to say, goto this line so it had to be really clear what line that was.

05:17 Cris Medina: Yup. And then we added gosub, oh man. That was advanced concepts right there, huh?

05:24 Michael Kennedy: That was amazing. Right, so you started there. Did you go and get a computer science degree?

05:29 Cris Medina: I did computer engineering. Well, I kind of halfway understand at least some of the software stuff, so let me see what all the software's built on, and I went off and figured out how to do hardware. I did computer hardware, computer architecture, that type of stuff. From there, I went into IBM, which was the first guys that hired me. I spent many, many years with IBM doing systems test.

05:54 Michael Kennedy: Were you doing hardware stuff for IBM?

05:56 Cris Medina: I was on the side of we developed a new server. It would come into our organization and we ran validation on it. Before it made it out to the customer, we would go and check a bunch of things. It was checking into organization tens of thousands of tests I would execute, so we had a small piece of add that had integration with, more on the integration level of the server with the firmware and the hardware. I had to do hardware tools that I'd have to code and say some sort of what is today, embedded systems, right? All the way up to high level software tools to maybe interface with that stuff. Or even business apps, which was the biggest thing I wound up doing, which was a management system for keeping track of all our test organization stuff, including status reporting, test execution, procedures, all that type of stuff.

06:47 Michael Kennedy: That sounds pretty interesting. I don't know a whole lot about the hardware side of computers. I mean obviously, I have some concepts, but I couldn't design RAM, or anything like that, right? That's sort of where the RB Dragons aspect to programming is for me. I have a conceptual idea of it, and I don't know how close that is to reality. It's pretty cool that you get to bridge that world.

07:12 Cris Medina: So why some of this stuff is important too, is because you get to understand a little more some of the more recent security things called row hammer, if you've heard about that.

07:22 Michael Kennedy: Tell people what row hammer is.

07:24 Cris Medina: Row hammer is a way of essentially, kind of hacking your machine to run code by accessing certain parts of memory at certain speeds such that you would make an adjacent memory cell have the data that you wanted or the code that you wanted to execute, and then maybe that adjacent memory cell is the one that's in the privileged memory.

07:49 Michael Kennedy: I see. That's so tricky. Is it used like cache hits and misses, and stuff like that? Or the pre-fetch stuff, or where does that come from?

07:58 Cris Medina: So this is actually winds up in your actual, it's a general problem with DDR3, in general. Anything that has a DRAM from DDR3, you could do that in, as long as the refresh rates, you know, memory, your processor kind of controls how often memory is refreshed. It would go in and blank out whatever charge is accumulated. So if your refresh rates are, if you wait a long time between refreshing, you have a larger window in which you can get in and make those changes. It'll actually happen on your DIMMs.

08:31 Michael Kennedy: Oh wow, these are crazy ideas.

08:32 Cris Medina: Mm-hmm.

08:33 Michael Kennedy: We're seeing a couple of these, right? There's the row hammer. There's Spectre, there's Meltdown. I mean, these are not even software problems, right? These are down in the chips.

08:44 Cris Medina: So then the other ones you mentioned are down at the processor level. A lot of those are because optimization is trying and execute instructions faster, and the way that processor pipelines work, you want to pre-fetch some information. I know one of them, I don't remember which one was what, but I know on one of them was more related to branch prediction. If you're going through code and you have an if/then/else kind of thing, it'll pre-fetch both sides of your if, whether it meets the condition, or it doesn't. And then you can play with that a little bit and have it pre-fetch some memory information or data that it shouldn't have because it's at such a low level. Stuff like that.

09:31 Michael Kennedy: It's crazy. It's going to be interesting. I think we'll probably see more of those types of things. It really gets scary when you mix that with Cloud computing, and we're going to talk about Cloud computing a lot, actually. Maybe start with what you do day-to-day. Let people know what you're working on.

09:47 Cris Medina: I'm part of a small group of folks that work for a company called Nimble Storage that was acquired by Hewlett Packard Enterprises last year, and Nimble makes storage arrays, so as in external enclosures where you have a bunch of hard drives that you access through iSCSI or Fibre Channel.

10:06 Michael Kennedy: These are for data centers, right? These are not NAS for your home, are they?

10:10 Cris Medina: Right, these are built for data centers, so these are expensive things that have way higher performance that you'd get something out of a consumer product. They have a bunch of systems management and stuff around that, as well, and a bunch of guarantees in terms of data savings due to de-duplication or compression. Things like that. Enterprise class, features, snapshotting, and things of that nature.

10:36 Michael Kennedy: Okay. Nice, so if you run 100 VMs, most of the OS is probably the same across all of them. You just need that one copy of those files?

10:46 Cris Medina: Right, so we do instant snapshots. If you have a virtual machine, so one of the use cases is, say you have a database in a virtual machine with say, your database in it, and that's all contained in a couple volumes which you can group together. You can go in and say, snapshot that, and it happens instantly, and then spin off a new VM based off of that, those volumes you just made over there, and now you have a copy of your data, essentially. And it keeps track of the diffs, so it's kind of like, if you bring it back to the software world, it feels a little bit like you were playing with Git really, or Docker, where you have special commits which have your data, and then you have the diffs of your data into the next set of commits kind of thing.

11:30 Michael Kennedy: Mm-hmm.

11:31 Cris Medina: When you do your snapshots and your clones and stuff, instead of having an entire duplicate of all of the data, you only have the difference that you write afterwards, so that's also pretty helpful. Those are the arrays that we make, and so one of the things that we're kind of experimenting with Cloud stuff we decided to go off and build what is Cloud Volumes. The product's called HP Cloud Volumes. The purpose is, if you have an Azure or an AWS virtual machine, and you want to tie it to some of our storage arrays, you can go to the website and request volume of certain size, and other characteristics along with it, and configure it so you can plug it into your VM.

12:16 Michael Kennedy: That's pretty interesting. Tell me, why would I pick that, say, instead of just creating a volume in AWS or Azure in their mechanism, right?

12:26 Cris Medina: Right, so one of the main things is you can go across Cloud. You could have your volume that you made with, say, your Mongo data, or something, and it's attached to your AWS. But if you're running a super critical application, and say, your AWS region goes down, you can just clone that volume and attach it to Azure, and you can use it on Azure's side of the world. The other thing you get out of it is, we can do higher IOPs. IOPs is a measure of how many storage operations, IO operations, you do on your drives over the array that you can get out of regular EBS. As far as I understand, the instant snapshots as well, I forget if EBS does instant snapshots, or not. I don't think they do. So you also get that ability. There's also extra stuff that we provide for our actual physical enclosure customers. If you own one of our physical arrays, you can actually replicate your data up into the Cloud through our service, and you don't have to worry about the ingress costs through AWS or Azure. We provide a way of seeding information out to, say, different regions and things like that, as well.

13:37 Michael Kennedy: Oh, that's pretty interesting, 'cause that can get pricey real quick.

13:39 Cris Medina: Yes, it can. Yep.

13:41 Michael Kennedy: Yes, definitely. I know a little bit about bandwidth charges and whatnot. I think last month I paid $600 in AWS bandwidth.

13:48 Cris Medina: Oh, jeez! Man, you're getting lots of downloads! Woo-hoo!

13:51 Michael Kennedy: No, these are good problems to have, but that's a lot of bandwidth.

13:52 Cris Medina: Uh-huh.

13:55 Michael Kennedy: This portion of Talk Python to Me is brought to you by Linode. Are you looking for bulletproof hosting that's fast, simple, and incredibly affordable? Look past that bookstore and check out Linode at talkpython.fm/linode. That's L-I-N-O-D-E. Plans start at just $5 a month for a dedicated server with a gig of RAM. They have 10 data centers across the globe, so no matter where you are there's a data center near you. Whether you want to run your Python web app, host a private Git server or file server, you'll get native SSDs on all the machines, a newly upgraded 200 gigabit network, 24/7 friendly support even on holidays, and a seven-day money back guarantee. Do you need a little help with your infrastructure? They even offer professional services to help you get started with architecture, migrations, and more. Get a dedicated server for free for the next four months. Just visit talkpython.fm/linode. Another thing you do is you spend a little time writing some fairly popular articles on your blog. I know 'cause Brian and I end up copying them often on Python Bytes.

14:59 Cris Medina: I appreciate that, too. I don't know if it's like a chicken before the egg thing. It's like, something comes up a little bit, but then you guys post it up, and then it gets a lot more reads. I have tryexceptpass.org, and we do the posts include a bunch of stuff. Usually things I play with. Most of the articles that get the most views usually are how-to's. I try to do a lot on asyncio stuff 'cause I'm trying to do more asyncio, and that is not an easy concept as it is today in Python.

15:30 Michael Kennedy: The work that you're doing there is really nice because I feel like that's one of the areas where there's really not very much coverage. I'm definitely planning on writing a course on asyncio because I feel like either people are just, they know about it and they're confused. They're like, ah, it's too hard, or they just don't even know, right? They're like, I'm switching to Go because Go has better async than Python. But you don't understand it does too, you're just not using it. I understand it's not as integrated into the web frameworks than it should be, but still.

15:58 Cris Medina: Right. But even that's changing now, so it's getting pretty good, and so I just want to try to keep playing with that and post that up there. I also do a couple things just on general engineering, and software, and a little bit of test, since I spent so many years in test. I got this series going on called Practicality Beats Purity, mostly about that part of Python and how one thing sounds great, but how good is it really when you implement it, kind of thing?

16:28 Michael Kennedy: Right, like one of the popular articles you had was microservices versus monoliths, and all the interesting trade-offs you make there. We'll definitely dig into those, but I kind of want to focus on the whole reason we started talking about having you on the show, which is the continuous delivery you're doing around the work you're doing in HPE, right? Maybe let's just start with what is continuous delivery? I know there's continuous integration, and that's something to watch as my repository and does some sort of build verification on check-ins.

17:00 Cris Medina: Exactly.

17:01 Michael Kennedy: What's continuous delivery? How does that typically work?

17:04 Cris Medina: Continuous delivery expands on top of the continuous integration concept, and say that build I just built, I want to do everything that I need to do to that build to make it deployable to production, and have it available to be deployed to production, if not already deployed in production automatically. So the idea is to be able to deliver a code into production as quickly as possible in a way that's maintainable, while still having a set of status checks around it that makes your job easier, right?

17:36 Michael Kennedy: Okay, so maybe the holy grail is like, I've got a GitHub repository, and it has different branches, so maybe a branch is called production and a branch is called staging. And if you commit into staging after some delay of the builds and creation of the servers, and whatnot, there is now a new staging server or services based on top of that where you did nothing but wait a little bit.

18:00 Cris Medina: And that's kind of how we have our stuff set up. We have two branches, and everything that's on our production stuff, production branch, is things that have gone already out to production. So everything there is a known working build that we can deliver to a customer, and we actually use the master branch for all our staging stuff, which is things that are fully tested, or we can guarantee that all the basic functions are tested and the whole thing is completely built into a deliverable that we can actually go and put out in staging. As we go through our pull request cycle flow, GitHub flow type thing, GitFlow, we can automatically put the build up in our staging environment.

18:46 Michael Kennedy: Right, okay, so you guys use the GitFlow style of work. Now, this is really common in open source. Some random outside person wants to make a contribution to a project. They don't have write privileges, so they'll fork the repository, make the change, do a PR back, and then the people can review and accept it. But some organizations, sounds like you guys, do that even for yourself on your own projects as a way of sort of formalizing it, right?

19:10 Cris Medina: Right, so we use GitHub enterprise internally, but things like GitLab also have a similar concept for this. It's just called differently. Our master and production branches are protected branches, and the only way to get in there is to go through a pull request merge. So we don't necessarily require the developer to have a separate, like a forked repo. They just have permissions to push their own branch into our main repo. Once they push their branch, and they open a pull request, we have a whole set of automation systems in place which receive the web hooks for the pull request, and kick off automated bills, our style checking, linting, and all of the testing that goes around that pull request. And so you can even in GitHub say, I require the following statuses, 'cause the pull request object in GitHub has the concept of statuses. As you run those web hooks, the code that kicks off from those web hooks, the status is reported straight back into that pull request. And in GitHub you can say, well, if all of these have passed, only then is your pull request valid to be merged into whatever branch it is that you want to go.

20:27 Michael Kennedy: That's a really awesome feature. To me, it sounds like this is a really nice way to sort of pre-vet what would be standard code review, right? Instead of going, well, you've done your work, now let's review it and see if it's good. It's going to be like, it's on the verge of being merged, and then you've already had all the tests done and everything is good. You push the button, and when that happens, it automatically deploys with no further work, as well. It's just like that one gate, right?

20:52 Cris Medina: Right, once all our tests are done and once the review is approved, we click our button, we do some other niceties around it, we kind of squash every right R merge commit a little bit so that it's useful, and we press the button.

21:06 Michael Kennedy: Right, maybe talk about that a little bit because like if I'm going to do, so maybe I'll create a branch, I'll do 20 commits. A bunch of little, tiny ones, and then eventually, I'm going to create a PR from that. You might want that to not look like 20 small operations, but one holistic one, right?

21:23 Cris Medina: Right, so we try to follow a system where we abstract more as we go down the production side, so we have as much detail as possible on the branch that the developer put their things together in. All the commits on that branch are going to be the itty-bitty things. Started to work on this feature, went to lunch. Came back, oh, didn't work. Trying it again. There's a bunch of really funny commits that go into that, and then you say okay, so I'm ready, so here's what I'm going to take all that group of 20 commits and merge it as one commit up into my master branch. And so I go back and clean all that stuff up and say, these are the features that are going in. These are the issues that are getting closed 'cause GitHub has all that automation in for us. You can say, close this hashtag issue number, and it automatically closes the issue for me. And then when I press a squash and merge button, all that stuff gets squashed and you only get one bubble in your master branch with the summarized changes.

22:18 Michael Kennedy: That's really awesome. Of course, you can go back to the other branch and see it, right?

22:22 Cris Medina: And see all the details, yup.

22:23 Michael Kennedy: That's really nice. I wonder how many people actually use that GitHub automation around interacting with issues. I use it all the time even just for myself. Hashtag some ID of an issue, or a PR and say, this is related to that, and it's really nice to just get those automatic links in there.

22:40 Cris Medina: Yup, and like I'm very opinionated on issues, so I love the way that GitHub does issues. Not in the UI, or anything like that. Just the idea it's an issue, it has a title, it has a description, some comments in it, and some tags, some labels, right? I don't need anything more than that. Everything else can be described with labels.

22:58 Michael Kennedy: Right. Jira feels like you're swimming in too many UI soup.

23:03 Cris Medina: Yes, and so I've used half a dozen different issue trackers. At the moment, the one we use internally actually, is Jira. We have the usual GitHub versus Jira thing, so we actually wound up writing a bridge to help us out. Follow our business logic in Jira as we do stuff on GitHub. We also listen for issue web hooks and update Jira for us automatically. But this way, I can go open an issue in two seconds by typing it into GitHub, and then all the stuff that goes into Jira gets all rolled into place as it should be.

23:37 Michael Kennedy: Wow, that's a really awesome way. I don't really want to work in Jira, so I'm just going to automate, working in Jira like my robot will.

23:44 Cris Medina: You got to be careful with that stuff 'cause then you'll wind up maintaining it, right?

23:50 Michael Kennedy: Right.

23:51 Cris Medina: So once our pull request is actually merged, more of our web hooks also say, oh, there was this pull request that got merged into master. That means we need to deploy code. We open a new pull request to go to production, and so that's what we call our deploy pull request. That one does a little bit of the different thing where it actually builds so our deliverables are container images.

24:16 Michael Kennedy: Docker containers, right?

24:16 Cris Medina: 'Cause our service, right, our service runs in using Docker and AWS elastic container services. Just to step back on that a little bit, so a lot of people get confused. Docker containers is one thing, but there's really two concepts. There's the image, and then there's the instantiation of that image, which is your actual container, right? So when I go and say I want an image of my REST API, that means I have the file system in place so that when I say docker run, I can instantiate a version of that image and execute my code inside that environment. We deliver two different images. One for our web UI, and one for our core stuff. All of the orchestration that we have to do in order to make our service work, which involves Cloud orchestration, with AWS, and Azure, some third party data center orchestration, switch management, array configuration, and then all resource allocation algorithms, user management, all that stuff, that's all kind of bundled into one container image and we run it with different environment variables to have it perform different functions. We have a microservices architecture, but with two images. The way we manage, the way that AWS works, when you have the container service in it, you define a service, like my core REST API service, and I say, I want this to run several tasks, or one task, and I want it to run this container, but I want this image tagged in such a way. For example, in our repository for that, the REST API back end, when we deploy an image we push up the code, and then we say, okay, we're ready to move to staging so we tag it with staging latest. So then I can go to AWS and my deployment activation work of make this image now the valid one in this environment is just stop the containers. And then AWS will automatically restart them, and when they start to come back up they say, oh, there's a new staging latest image. Let me download that and use that one.

26:35 Michael Kennedy: I see. That's really cool. Basically, the AWS container service just knows, I've got to run out of this Git repository with this tag, and I just always look for that. If necessary, rebuild it.

26:48 Cris Medina: Right, a docker image.

26:49 Michael Kennedy: A docker image.

26:52 Cris Medina: Right, so everything just builds off of that. It's really helpful also, because if you broke things for one reason or another, all you have to do is move your staging latest tag back to your previous one and restart the containers.

27:09 Michael Kennedy: Oh no, undo it, undo it!

27:12 Cris Medina: Yup, and that's all you got to do. You don't have to worry about anything else 'cause you know that was working code, and you're back in time. Now, there's other--

27:18 Michael Kennedy: That's really nice.

27:19 Cris Medina: Complexities when it comes to database migrations and things like that, but 90% of the time you don't have to worry about it.

27:28 Michael Kennedy: Right, okay. That's really nice, because it's one thing to roll the code back, but you're potentially making infrastructure changes and OS changes at that level as well. And the ability to go, oh no! Just put it back like it was, it was working, is pretty cool.

27:44 Cris Medina: So the code underwent all the testing, and all that worked, and the container image that we used to build that code, and I'll get a little bit into that in a minute, is slightly different than the one I actually wind up putting in staging because it has more stuff in it to maintain the test infrastructure, or be able to get to the test infrastructure. The one that actually makes it up into staging, which is equal to the one in production at that point, also has a bunch of other things like an Nginx configuration that's a bit different, or a uWSGI config along with it. So I can easily mess that up, and it's only a problem I see when I go to deploy it.

28:25 Michael Kennedy: Right, right. Just today, I would've liked to have something like this. I mean, I have multiple staging and production servers for my various things.

28:33 Cris Medina: Mm-hmm.

28:34 Michael Kennedy: One of 'em, I got an indication there was an upgrade for IDNA, I think, which is some low level dependency in my system. And then there's requests, and so I upgraded the low level thing, and it said, oh no, requests forces to use an older version of that. So guess what? Your site won't even start. It's just dead. But luckily, it was running on, like I had taken that one out of the load balancer, and they're like, oh my goodness. But I had to do a lot more work than just move the tag back, right? It was like, alright, well now, how do I unravel this? How do I make it know that it's supposed to install the right one, and all that kind of stuff. I can definitely see the advantage here.

29:14 Cris Medina: And that actually happens quite often with a bunch of things. Not just your Python libraries that you have to worry about and their interactions, but also your docker images. So what happened to us in the past couple weeks while we were going through testing, like, in the middle of test are passing, and then all of a sudden everything's failing. What's going on? Like, a couple minutes. We actually hit where whoever maintains the base Python docker image that we depend on, iterated on it.

29:40 Michael Kennedy: They changed it? Oh boy.

29:42 Cris Medina: Obviously, we can go back to the older one, but at this point we wanted to move with the new one. The newer one changed the base. I think it moved major version, so all the aft packages and stuff like that had updated, so I needed to use different names to pull some stuff, so that was fun.

30:02 Michael Kennedy: That's fun. And if you do that on the real machine in production while it's running, not so good.

30:07 Cris Medina: Not so good. Right, that's why all this stuff is in place.

30:10 Michael Kennedy: That's right. This portion of Talk Python to Me has been brought to you by Rollbar. One of the frustrating things about being a developer is dealing with errors. Relying on users to report errors, digging through log files, trying to debug issues, or getting millions of alerts just flooding your inbox and ruining your day. With Rollbar's full stack error monitoring, you get the context, insight and control you need to find and fix bugs faster. Adding Rollbar to your Python app is as easy as pip install Rollbar. You can start tracking production errors and deployments in eight minutes or less. Are you considering self-hosting tools for security or compliance reasons? Then you should really check out Rollbar's Compliant SaaS option. Get advanced security features and meet compliance without the hassle of self-hosting, including HIPAA, ISO 27001, Privacy Shield, and more. They'd love to give you a demo. Give Rollbar a try today. Go to talkpython.fm/rollbar, and check 'em out. One question I did have while you were describing what you're up to, and we talked about the GitHub hooks like #errorcloses, #123, or whatever. Is there a way to make that happen only when it merges into the main branch, or does that PR commit itself trigger the closing of that issue?

31:28 Cris Medina: Yes, I understand what you're asking. Yes, it only happens when you do the merge.

31:33 Michael Kennedy: Oh really? Okay.

31:34 Cris Medina: I type it in my commits all the time.

31:36 Michael Kennedy: Interesting. Okay, well that's awesome. Very, very nice. Alright, I think maybe the most interesting thing to cover is we've now set the stage of what you're building, but all the various pieces. There's so many cool, little libraries, and packages, and things involved in the act of building this whole pipeline that you've created. Do you want to walk us through that?

31:57 Cris Medina: In order to run all the testing, we have a Docker Swarm internally on-premises in our data center where we orchestrate all of this stuff. I have a container running that is my web hook receiver, and I built that REST endpoint for that using Hug. Hug is a Python 3 REST API kind of like Flask, but it's a bit smaller and a little more expressive because it uses annotations when you define your functions to define the input type of your parameters from your rest API. It also automatically generates documentation.

32:35 Michael Kennedy: That's cool. Hug gets really interesting in that it's one of these REST-only frameworks. It's not built, as far as I understand it, mostly for building web applications, but more for building web services.

32:48 Cris Medina: Correct.

32:49 Michael Kennedy: There's a host of these that are really amazing at sort of leveraging Python 3. Hug is definitely in there, which is super cool. API Star.

32:59 Cris Medina: And API Star. That's the new one I'm playing with because API Star fully supports asyncio. I can have an async function, and have API Star serve it up.

33:11 Michael Kennedy: That's awesome.

33:12 Cris Medina: Yup. I can await in the function, and in the meantime, it'll go off and do other things. It's pretty cool.

33:16 Michael Kennedy: How about Hug? Does it do asyncio, do you know?

33:18 Cris Medina: I don't think so. Last I checked, it did not, but I think it was able to handle it better because it was all Python 3.

33:28 Michael Kennedy: It's definitely all Python 3, which is cool. If you look at the performance, Hug is built on a framework called Falcon.

33:34 Cris Medina: Falcon, yes.

33:35 Michael Kennedy: Which is also another cool web framework that probably not many people have heard of, but I had the guys building Falcon on my show, and it's a really low level, high-performance web framework, and then Hug is actually built on top of Falcon, which is pretty cool. But they are definitely, both of those are right near the absolute top of performance in terms of requests per second on some random piece of hardware. Pretty cool.

33:57 Cris Medina: I find that a lot of things like Django, Pyramid or Flask, these things have been along for a while, so they have a bunch of things they do for you. The higher level of abstraction that you get out of a framework, which is what you want from a framework usually, the more careful you got to be with performance because in order to give you that abstraction, they needed to put you through a number over other levels, especially usually function calls, which in Python are a little bit expensive.

34:27 Michael Kennedy: They're surprisingly expensive actually.

34:29 Cris Medina: Yes, yes they are.

34:31 Michael Kennedy: For example, one thing that Hug says, and I think this is partly coming through the Falcon side of things, is it's compiled with Cython to basically get much higher performance, which is a pretty cool aspect, as well.

34:44 Cris Medina: Yup, yup.

34:45 Michael Kennedy: Okay, so you've got this, and this is one of the really important things about this Docker stuff. Is like, it's awesome to have your database in a docker container, and your web framework, and then your back end services, but they all need to know, okay, we just all got rebuilt. Where are you now? Who are you, right? How do I find my back end , right? So that's the role of this thing that you built?

35:06 Cris Medina: Right. That receiver, the web hook receiver also can communicate with the Docker Swarm using DockerPy, and then orchestrate, oh, I need to build a new container. I need you to start a new container. I need to build a new image, or I need you to start a new container with this existing or newly built image. For example, one of the things we do is when the web hook comes in we go in and do use requests to go up to GitHub, grab some information on the repository, search for a file that kind of works kind of like how Travis CI does. We have a yaml that says, oh, to set up for testing, here's your install instructions. The actual test are these things, and there's a bunch of other settings we can do. One of the things in there might say, well, I want all these tests to run in parallel. Then that means I got to orchestrate getting the container image built off of the repository the way that the instructions say they're supposed to be done. Then taking that and committing that new image to an internal registry that we have, and then telling our Docker Swarm to start five to six parallel images to go and execute tests based off of that new image. And then those tests all have to require resources of their own, so we have another infrastructure piece, which is a resource manager. All it does is it sits there and receives, waits for a web socket. In order to pull essentially, check out a resource, I open a web socket connection and I say, oh, I want this type of resource. While that web socket connection is open, I have a reservation on that resource. That makes it so that I can write tests and not worry about releasing the resources when they fail.

37:00 Michael Kennedy: Oh, that's pretty interesting, 'cause when the thing goes away--

37:04 Cris Medina: The socket closes.

37:05 Michael Kennedy: Socket closes, but when it's all done. Huh, I thought you were just doing push notifications. I didn't realize the web socket session had such an important role. That's pretty cool.

37:15 Cris Medina: So we do that. For that, I used the Autobahn, which is something I used inside SoFi, one of my open source modules. But I recently moved it to web sockets. It's a module called websockets. It's built around asyncio a lot better. More Pythonic, using async for and async with, so it makes it a lot easier to interact within a coroutine kind of way.

37:39 Michael Kennedy: Interesting, so your test might just do a async with web socket connection, and then its stuff.

37:44 Cris Medina: Something like that. The receiver will do an async for around the web socket around receiving something in a web socket, and so that's on a server side. On the client side, we just open the socket and do, I think on the client side is an async with where you just sit there and just kind of wait for messages.

38:06 Michael Kennedy: Interesting. That's pretty awesome. Then another thing that you do after the test pass, then you build your artifacts, right? Your packages. You use proper Python packaging as part of this, right?

38:19 Cris Medina: Too many deliverables or container images, but I also have those are built on top of a bunch of other repositories that I have. Two or three of those repositories, their deliverables are actual Python packages, which are internal, and we use an internal Python package index for that, which we later migrate it to a tool called Artifactory.

38:41 Michael Kennedy: I had never heard about Artifactory. This is the thing by Jfrog.

38:44 Cris Medina: Yup, it's by Jfrog.

38:46 Michael Kennedy: Oh my goodness. That is one serious piece of enterprise software management software there.

38:52 Cris Medina: It's a lot of stuff. A lot of stuff. Python package indexes, npm indexes, whatever those are called. Just NFS, docker registries, and then you can mirror. So if you have stuff in the outside world, you can mirror those and you can have it automatically push things for you. You can add tags and properties to things. It's quite complex. It has a REST API too, to get to it. It's pretty interesting.

39:19 Michael Kennedy: It's really interesting. Their website has a bunch of cool little animations. It just kind of draws you in. To me, it looks like you've taken, you guys in general, have taken a lot of the awesome stuff from the public open source, and maybe sort of made your own private version of it. You've got GitHub Enterprise. You've got a private PyPI server. Private Docker repositories, all sorts of stuff, and registries.

39:43 Cris Medina: Yup, it just makes the whole thing easier to work with 'cause you have an existing ecosystem that can work with all of this. You don't have to build your own modules to talk to 'em.

39:52 Michael Kennedy: Super cool. I'd never heard of Artifactory, but it definitely looks like worth checking out. It's no small piece of software, as far as I can tell. It looks like a big thing that does a whole bunch of stuff, but it definitely looks like it's pretty cool.

40:05 Cris Medina: Yup, and it comes with its own complexities. If you just want an internal package index, really, there's a bunch of existing things already you can use, or you can just build your own. I built one with Hug before Artifactory. It's just a web server.

40:19 Michael Kennedy: Right, it's just a web server and a couple of interactions. It's not super complicated, but pretty cool. We talked a little bit about some of what happens next. You have your GitHub hooks and your PRs, and all that kind of stuff. What else is involved? You have your Hug service that you've talked about. That's pretty awesome. You used PyDocker, which you mentioned in passing there. DockerPy, sorry, which is just pip install docker, right?

40:44 Cris Medina: Yes. Yes. pip install docker.

40:47 Michael Kennedy: If you wanted your Python app to say orchestrate creating new containers or spinning them up, that would be what you use?

40:54 Cris Medina: Right. It's got two client layers. One is a lot lower level, which is an API client kind of thing. I think they call it API. I forget what they call it. And then there's one which is the Docker client. The Docker client operates more at an object level, so you point it to where your Docker, your main Docker master is of your Swarm, and you could just do .images, .lists, .create. Stuff like that. Same thing for docker containers. And then with the Swarm in general, things get pretty complicated when you go out to Swarm or Kubernetes, mostly because your containers that are managed by the Swarm are not really containers, they're services. It all makes perfect sense if you're running, say, a web service and you say, I want a web service that needs to always be up, and I want two instances of it. You run one service with two tasks. Each one of this type of container. But for us that are actually creating, essentially, one container or two containers individually to run every time, we have to make a new service for it. There's a lot of layers there that complicate things a little bit, but it's very easy to manage with DockerPy 'cause it's all built by the Docker guys.

42:16 Michael Kennedy: Okay. That's really, really cool. And then another thing that you use, something called ChatOps. What is ChatOps? Is it like something for DevOps?

42:25 Cris Medina: Sure. When our deployed PR is complete, that means it builds a container image and it pushed that image out to our Amazon container registry. Then we could automate this, but I still want to have some manual checks in place. What we did was we have a chat system, and just made a bot. I can tell the bot, hey, I want this image to be my staging image. Go do it. And then the bot will go in and tag that image with staging latest, and it'll go in and stop all my containers in AWS, which will automatically restart and essentially, do my flip over to a new version.

43:12 Michael Kennedy: That's awesome.

43:13 Cris Medina: ChatOps in general, is kind of like a concept of being able to run to manage a bunch of services, or deliverables, or code, or whatever you want to do over a chat system using a bot, essentially.

43:26 Michael Kennedy: That's pretty cool. I mean, we saw Kelsey Hightower's thing at the 2017, it was 2017 PyCon, where he got basically Google's voice assistant to do his Kubernetes stuff, right?

43:41 Cris Medina: Yup, so I was kind of laughing through his presentation a little bit because I was like, yup, that's what I do, except I can't talk to it. Not through voice, right?

43:52 Michael Kennedy: Well, you're not far away from getting some Google Home or some Alexa-- You could use some of the Alexa stuff going. Sorry, if everyone's Alexa's going off. Mine is as well, now. The Amazon assistant. Let's call it that.

44:06 Cris Medina: There's a Python one called Calliope we were just looking at today. There's another one. I forget the name that's also pretty famous in the Python world. We were just laughing at it this morning saying that we should hook all our stuff up together and just say, hey, deploy to staging. Restart our stuff.

44:26 Michael Kennedy: There's some pretty easy ways to do it, actually.

44:28 Cris Medina: Just some random dude walks into the cubes and just kind of kills all our stuff.

44:32 Michael Kennedy: Deploy production! No, no, no, no, no! That's pretty awesome. Another tool that I don't think I've heard of that was really impressive to me is Locust.

44:43 Cris Medina: Yes.

44:44 Michael Kennedy: Tell us about that. Use pytest for your standard level, your automated testing, but Locust is more on the performance side, right?

44:51 Cris Medina: That's right. Locust, the idea of Locust is to test web services. You can write tasks in the forms of scripts or actions that represent users of your service. You can have all set of set up and tear down type of stuff kind of like your regular type of test environment. But then Locust manage that over a large amount of virtual machines to go off and test your API, and then come back and tell you, well, you managed to receive this many requests per second, and this endpoint, and this many in that endpoint, and this one was right the minute at this point, and this one was errored out when you did this thing. And so you get a report.

45:38 Michael Kennedy: That's really cool. You've got all the different parts of your site, and it shows you, here's the number of requests in a big grid. This URL got this number of requests with this many failures. Average response time is this. It's super cool. One of the big problems with the load testing is actually it can be at least, getting enough pressure on your web server, right? If you just do that over say, your broadband connection at home on your laptop, maybe the limit is your outbound network, or something, right? Something like that where it's like if you could put it on 100 VMs, spin them all up and turn those loose in a slowly way, that's awesome, right?

46:19 Cris Medina: Yup, you can have a different amount of virtual machines and it just kind of orchestrates all of them for you. You just got to have Locust installed on them and the scripts that they got run on them.

46:27 Michael Kennedy: This is lookin' super cool. I definitely would like to look more into it. Says define user behavior with Python code and swarm your system with millions of simultaneous users. When tools like this exist, I'm just blown away when there are websites that fail so badly when they get a lot of traffic. I understand there is some limit where it's like, okay, it just is not going to take more. But that limit should be many thousands, not a couple hundred, right?

46:57 Cris Medina: Right, and so another thing we did, we used Locust to figure out where we might break. And then what I do is there's in our Docker image whenever that actually gets executed in a container, there's a few instructions that go in and replace environment variables in our Nginx configuration. I can go and tweak the request per second that I allow. I do it at the Nginx level so I never hit the Python code. If I know I'm going to break at whatever, 10 requests per second for an endpoint, I can put in a limit at 8, or whatever, on my--

47:37 Michael Kennedy: And then it just queues in Nginx until--

47:40 Cris Medina: Correct.

47:41 Michael Kennedy: 'Til uWSGI gets done--

47:42 Cris Medina: You can configure Nginx to do that per IP address, or just in general, and what error codes to return. All that stuff.

47:52 Michael Kennedy: Wow, that's awesome. A couple other tools that are at play here are some of your projects. One is Sofi, and one is Korv. Sofi falls into a pretty interesting realm of Python. Maybe tell people what Sofi is.

48:09 Cris Medina: Some people talk a lot about user interfaces and Python. I don't know any of those. Actually, I think it was after one of your very first, like a long time ago conversations about user interfaces and stuff. I was like, oh, you know, I'm pretty sure this is exactly what I was thinking. I've built so many bootstrap-based, quick interfaces with some just jQuery default stuff. I don't want to write that anymore. I want to just write it in Python. What I did is, essentially, it's a module that lets you do that where I kind of wrapped the widgets that you'd get out of bootstrap, the HTML kind of library to help put that together. But it evolved because the way I do it is through web sockets and asyncio, so in the back end you can kick off a web page that loads a basic JavaScript library that you only have to write once, which tells that web page how to interact with Sofi. You run a Python websocket server, which is Sofi, that actually sends commands out to the web page. You open up a website, and all of your interaction and eventing can go all the way back to your Python code and you can react on that and come back out to the UI. After that evolution, I realized that Sofi is actually really a websocket protocol to help you do all of this, and kind of like a library to help you do all of this. 'Cause you could just drop in and replace other ways of doing these conversations between client and the server. And so what I did after that was I went to a game engine Unity 3D, and dropped in a websocket client written in C#, so now I can from Python, spawn game objects and things like that in Unity.

50:04 Michael Kennedy: That's pretty awesome. It's a tiny bit like Electron JS-type apps, where there's a Python back end and some sort of web front end. At least the first incarnation of it.

50:15 Cris Medina: It's also deployable kind of like that if you want, as well. The original idea was to go down a desktop application type thing, in which case you would want to build it like that, like an Electron JS thing where you distribute Chromium, the browser that Chrome is based on, which is completely open source as your front end, but you could just deploy the back end by itself onto some service in a Docker Swarm, which is what we do. And just open up a web page and talk to it.

50:43 Michael Kennedy: Nice. Your other project, Korv, is about sort of skipping the whole REST API entirely, right? And using actually SSH.

50:52 Cris Medina: So in the process over the years, working on different services, there's always the customer-facing one, but then you always want some data or something you want to have some admin mode for. It's always a risk to put those admin endpoints in the customer-facing one, 'cause if they're there, somebody's going to fiddle around, and bump into 'em, and then you've got to worry about security and all that.

51:15 Michael Kennedy: Right. It just takes one forgotten security check, and all sorts of badness happens.

51:19 Cris Medina: Right, especially when those checks are usually decorators around Python functions, which you could forget to put in, right? The idea is, instead of using HTTP, use SSH. The first time I came up with this was when I ran into async SSH, which is the base library for this, because I wanted an async way of doing SSH calls. They let you do this. So I let SSH take care of the authentication. You've got to allow the client's known hosts as well on both your known host, on your client, and your acceptable public keys for your client on the server. SSH handles your authentication, and then after that, you open a TCP socket over SSH, and just send information back and forth. I just wrapped it in JSON and kind of used HTTP-ish REST-like mechanisms for get, store, update and delete. I do that as my admin interface. It's only accessible to me, but even if I mess it up and it somehow exposes the ports out to the internet, it's still SSH, so you still need the proper keys to get in.

52:30 Michael Kennedy: That's really awesome. I think that's quite a cool idea. Add certain things where you can only get to them through SSH, and you can't access them or interact with them without that. This is kind of of like instead of just exposing, tunneling that through or something, you're like, no, let's just make that the API network layer. Exchange layer.

52:51 Cris Medina: Both of those are async, so you could just have it kick off long-running things. Doesn't matter. You'll get a call back when it's done.

52:59 Michael Kennedy: Really cool. Let's see, another couple things that were really interesting that you were using. One is PyAutoGUI, and that's from Al Swaggert. What's that?

53:07 Cris Medina: These are kind of more of a experimentation thing. We have pytest to execute all of our tests, which is great. We use Selenium for a few things, and then we use some other JavaScript node-specific runners for the web UI stuff, as well. Browser compatibility is always going to be an issue. We want to try to do something more at the, we ran into a couple of issues where you say you open this thing in Firefox, but if you open it in Chrome this one little piece of it is kind of wonky. It's larger than it should be, or it's off the screen, and things like that. PyAutoGUI does a lot of stuff. It's about automating your OS through GUI things. Move the mouse here, click on this, type this from the keyboard, and things like that. Now, it also helps you take screenshots. One of the things that I was thinking was we could put something together that says, open browser, type this into location bar, load webpage, type username, type password, click login. And then take a screenshot of the result, and then use OpenCV to compare that screenshot to an already existing screenshot that I should have, and maybe find the next button I have to click on. And if that button is not on the screen, then error, and move on kind of thing. The advantage of using the OpenCV stuff is you can have a confidence level on it. If things are off, you'll know.

54:44 Michael Kennedy: That's totally cool. There's a project by a friend named Lou Alan Thalko called Approval Test. It will do something similar. Basically, it will go and instead of having a whole bunch of tests, it just says here's the output. Is that good or bad? You say, this is good, and then it records that, and then unless that output changes--

55:05 Cris Medina: Oh, sure.

55:06 Michael Kennedy: Right? I think it does that with pictures, as well, right? You could screenshot something and go, this is the verified version. If this changes, I need to check it out. Otherwise, just keep running the test and saying they pass.

55:16 Cris Medina: Cool. Something like that is what we wanted to do.

55:19 Michael Kennedy: That sounds really cool. And you were talking about using OpenCV as well, huh? That's cool.

55:23 Cris Medina: And I actually tried out a few different things, and OpenCV wound up being the faster one.

55:27 Michael Kennedy: Awesome. Cool, alright, well there's just so many little interesting tools and steps along this whole process I think a lot of organizations are trying to get to, right? Like I said at the beginning, right? I check in, I merge with PR, I wait. Magic appears on the other side, right, with zero downtime? But it sounds like you guys have really got it pretty nearly there. That's awesome. So was it worth it?

55:52 Cris Medina: For me, it is. I mean, but you have to step back. You have to do your engineering work behind it, right? Don't just do it because everybody's doing it kind of thing. We have a service, right? It is in our best interest to make it as fast and as easy as possible to release a fix out to the customer.

56:16 Michael Kennedy: The last thing you want to do is try to release a fix, and then take the whole thing down and make it worse.

56:21 Cris Medina: Exactly. The way to go through all this is you have to step back, you have to look at your process. It's a lot of pieces and a lot of moving parts, so you have to say what checks do I need at which point in time in my delivery flow? Does it make sense to check what? If I say, all these tests need to pass here, that means I've guaranteed this basic function is working. These tests pass here, that means my infrastructure's working. Et cetera, et cetera.

56:52 Michael Kennedy: Right, and it also depends on the quality of your tests, right? You need to know that if the tests pass, pushing to production without further question is okay, right? Whereas if you only test a few things, and maybe they don't test that well, if a lot of stuff slips through, then this isn't so helpful, right? It's got to be a good net.

57:12 Cris Medina: Right, and we made the decision early on to invest into that because we think it's going to bear fruit for us, and it has been very useful. We very, very seldomly have a really broken function into staging. And when we do, the first thing to fix is not the function, it's the test to make sure that you can't push again with it broken.

57:36 Michael Kennedy: That's a really good point. It's like, why did this get through? There's actually a problem in the continuous delivery system that it got this far. Now let's fix that, right? That's awesome.

57:47 Cris Medina: Take advantage of the situation that you're in, which is you have a real failure. Not something that you thought you might have. You have a real one, so make sure that your tests fail when you have a real failure, then go fix the code to make sure that it passes.

58:02 Michael Kennedy: That's awesome. I think that's really great advice. Alright, I wanted to take just a moment, let you maybe list off some of your popular articles that you've written, but don't want to take too much time since we're running out of that.

58:13 Cris Medina: The most viewed article over time from the blog has been Threaded Asynchronous Magic and How to Wield It, which is pretty much an intro into asyncio and what you could do with it.

58:25 Michael Kennedy: It's a really good one.

58:27 Cris Medina: Just how to manage tasks, and stuff like that. The one that's most read, as in the most time people have spent going through all the details was the A Python Ate My GUI, which was the starter article for making Sofi, and kind of the state of GUIs in Python. The most recent one I have, which is now no longer true, the most recent one is about GDPR, and the implications of the new European regulation for software developers. The one before that, which was the one in my list, was Practicality Beats Purity about microservices and monoliths, which we talked a little bit about already.

59:10 Michael Kennedy: The people should check that out if they're considering one or the other, and there's a lot of interesting trade-offs that you highlight there. Alright, so if this was a few weeks ago, I might ask you a little more about the GDPR and get your thoughts on that, but there's something bigger to talk about. You talked about using GitHub Enterprise. I'm super invested in GitHub. I just checked right now at the time of this recording, I have 134 repositories in GitHub. That's a lot, and many of those are private ones like supporting my verisyncs, but a lot of them are public, as well. The big news last week was that Microsoft acquired GitHub.

59:45 Cris Medina: Yes.

59:45 Michael Kennedy: What was your first thought?

59:47 Cris Medina: Oh boy. That was my first thought. I'm not a Microsoft fan, but I am willing to admit that they have a different direction, which I like, which is way more embracing of open source. They are the world's biggest open source contributor today.

01:00:05 Michael Kennedy: Which is think about where we are, just that you said that. Like, that's crazy.

01:00:09 Cris Medina: Yup. They have incorporated Linux into Windows, sort of. They have contributed a significant amount of work to Docker.

01:00:21 Michael Kennedy: And GitHub, or Git itself, actually. The Git Virtual File System.

01:00:25 Cris Medina: Yes, with the Git Virtual File System, which was a huge contribution, especially for folks doing large single repository code bases. That's for me, that's a good direction. Unfortunately, the track record so far hasn't been all that great, even with their most recent acquisitions. The most common one that I hear and that I have problems with is Skype, and where that wound up with. But some folks also complained about how LinkedIn is going. I don't know. I don't know internally, there was a lot of outcry from the community, but equally, there's a lot of Microsoft developers that are like, well, but we at Microsoft love GitHub just as much as you guys do. I genuinely believe that. It's not about whether the folks contributing to the code like it or not, or want to keep it going or not. It's the fact that now you've concentrated the majority of open source projects in a business that has its own languages, its own platforms, its own software, which you could maybe wind up getting biases for. Once it's there, it's just going to be more complicated. There's also questions about IP that people brought up. I don't know how much that's a thing, but technically, everybody had access to all those repos anyway, but I guess now they get access to all the private ones too. I don't know.

01:01:58 Michael Kennedy: So a couple of thoughts. I definitely sort of felt similar to what you were saying. I don't think Microsoft has any sort of bad intentions towards GitHub. I think they do really love it. They're really invested in it. However, they could fumble it and just make it not so nice, right? I don't think they would intentionally shut it down, or do something to make it less good, but they certainly could try to make it better, and make it worse. That is a thing that could happen. The IP part is pretty interesting. What really surprised me is there's an article, it was on Ars Technica. That's one of my favorite places to read this kind of stuff because the comments are great. Says everyone complaining about Microsoft buying GitHub needs to offer a better solution. And they really went through point by point, how GitHub was actually in pretty big trouble. At some point. This is almost like, would you rather not have GitHub, or have a GitHub that Microsoft owns? Rather than, well I want GitHub to be this free thing. To me, the biggest negative here is just consolidation. There was this sort of independent place where open source could go be its thing, and everybody was on sort of equal footing, and now it's been consolidated into one of the big five tech companies. That's just different, and not necessarily better. But after reading this Ars Technica article, I felt much better about it because I didn't realize the alternative was as bad as it could be. I mostly don't like the fact that it's just consolidating further in the whole tech space.

01:03:31 Cris Medina: I think that is the main concern. I agree. I think after going through this, I think it becomes a bit more obvious, I think I said this over Twitter, we kind of need like a Mozilla Software foundation of open source repositories kind of thing. An independent body that's somehow funded, whose sole purpose is to just you put stuff up there, and it's going to stay there, and we'll keep it up kind of thing. That would alleviate the concerns a bit.

01:03:59 Michael Kennedy: It would. I think on the positive side, I think Microsoft has done a good job with Xamarin, and that was open source, sort of. I think that that's stronger now than it has been, as part of them taking it over. There's one check for maybe the win box. Microsoft's part of the Linux Foundation. There's signs that this is going to go well, but there's like you said, Skype examples. There's also signs that it might not go so well. I think it's up in the air. My concern is two things. It could just get fumbled and messed up, but the fact they're running GitHub as an independent organization is really good. The fact that the guy that was one of the co-founders of Xamarin is going to be the CEO of GitHub, where apparently they were struggling to get a CEO at all. There's like, a big problem there. Anyway, I think it's pretty interesting. Fingers crossed for a positive result.

01:04:52 Cris Medina: From the business perspective as well, Microsoft, it makes perfect sense for them. They just spent all this time into the virtual file system, and stuff, and they just moved all their stuff over to that, so they want to secure a future for that, right? And this is their way of doing that.

01:05:09 Michael Kennedy: I guess the other thing that sort of gives me sort of a positive outlook, I guess, is at least the way I've seen it these days, if you want to understand what Microsoft is doing, or why they're doing it, the answer is Azure, and then you got to figure out what the question is. Obviously, it's for Azure, Azure, Azure. They're just trying to grow Azure. They could just care less about Windows, or to some degree, Office, right? They see the new lockin is the Cloud, and how do we go be part of that? All the different technologies run there. I think that that's going to put some pressure to keep it more fair-handed, rather than say it's only .NET, or it's only Windows, or any of these sort of pressures that you kind of hinted at, at the beginning.

01:05:49 Cris Medina: Yup. Agreed.

01:05:50 Michael Kennedy: Alright, well I guess we'll leave it there. We could have a whole show on the thoughts of--

01:05:54 Cris Medina: Yes, we could.

01:05:54 Michael Kennedy: Maybe I will, at some point. At first, I was like, oh boy, this is probably going to get messed up somehow. After spending a week doing more research, I'm kind of like, well, it looked kind of like it was necessary, and it's probably the least bad outcome that we're going to get. You know, fingers crossed.

01:06:11 Cris Medina: For sure.

01:06:12 Michael Kennedy: Cool, alright. Let me hit you with the last two questions before you get out of here, Cris. Notable PyPI package. We covered a bunch, actually.

01:06:19 Cris Medina: The two that I would bring out, out of the whole list, which since we're kind of talking about async stuff, as well, is async SSH. Forget about paramiko. Async SSH does it better. The web sockets module, which it's pretty good if you're doing anything with web sockets.

01:06:37 Michael Kennedy: Nice. And so final call to action. People want to bring continuous delivery into their whole workflow, their life, their team. How do they get started?

01:06:45 Cris Medina: The first thing to is to, we kind of touched on it a little bit earlier, is step back, analyze what benefits you get out of it, and what problems you're trying to solve, and then slowly go through it and put something in place where you know my end result is going to be X, and a delivered package of this here, and the need to guarantee that it works in pieces at different steps of the way. Figure out the effort to make that happen. If the effort is really, really, really, really, really large, then maybe it's not worth it for you.

01:07:16 Michael Kennedy: Cool. Well, I really appreciate you coming and sharing what you guys are up to because you definitely have it pretty dialed in.

01:07:21 Cris Medina: No problem. Glad to be here. It's always fun to have these conversations.

01:07:26 Michael Kennedy: Thanks Cris. This has been another episode of Talk Python to Me. Our guest has been Cris Medina, and this episode is brought to you by Linode and Rollbar. Linode is bulletproof hosting for whatever you're building with Python. Get four months free at talkpython.fm/linode. That's L-I-N-O-D-E. Rollbar takes the pain out of errors. They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until your users complain, of course. As Talk Python to Me listeners track a ridiculous number of errors for free at rollbar.com/talkpythontome. Want to level up your Python? If you're just getting started, try my Python Jumpstart by Building 10 Apps, or our brand-new 100 Days of Coding Python. And if you're interested in more than one course, be sure to check out the everything bundle. It's like a subscription that never expires. Be sure to subscribe to the show. Open your favorite podcatcher and search for Python. We should be right at the top. You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm. This is your host, Michael Kennedy. Thanks so much for listening. I really appreciate it. Now, get out there and write some Python code.