WEBVTT 00:00:00.001 --> 00:00:04.800 Do you have a big monolithic web application or service that's hard to manage, hard to change, 00:00:04.800 --> 00:00:09.420 and hard to scale? Well, maybe breaking them into microservices would give you many more 00:00:09.420 --> 00:00:13.620 options to evolve and grow that app. This week, we'll meet up again with Miguel Grinberg 00:00:13.620 --> 00:00:18.840 to discuss the trade-offs and advantages of microservices. It's Talk Python To Me, 00:00:18.840 --> 00:00:22.680 episode 121, recorded June 2, 2017. 00:00:22.680 --> 00:00:51.980 Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the 00:00:51.980 --> 00:00:56.600 ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter, 00:00:56.600 --> 00:01:01.520 where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm, 00:01:01.520 --> 00:01:04.060 and follow the show on Twitter via at Talk Python. 00:01:04.060 --> 00:01:10.360 Talk Python To Me is partially supported by our training courses. Here's an unexpected question 00:01:10.360 --> 00:01:15.620 for you. Are you a C# or .NET developer getting into Python? Do you work at a company 00:01:15.620 --> 00:01:20.500 that used to be a Microsoft shop, but is now finding their way over to the Python space? 00:01:21.280 --> 00:01:26.720 We built a Python course tailor-made for you and your team. It's called Python for the .NET 00:01:26.720 --> 00:01:31.840 developer. This 10-hour course takes all the features of C# and .NET that you think 00:01:31.840 --> 00:01:37.420 you couldn't live without. Unity Framework, Lambda Expressions, ASP.NET, and so on, and it 00:01:37.420 --> 00:01:42.040 teaches you the Python equivalent for each and every one of those. This is definitely the fastest 00:01:42.040 --> 00:01:50.480 and clearest path from C# to Python. Learn more at talkpython.fm.net. That's talkpython.fm.dotnet. 00:01:50.480 --> 00:02:01.040 Miguel, welcome back to Talk Python. Thank you. Thank you for having me. It's great to have you back. Previously, we talked about 00:02:01.040 --> 00:02:19.440 building Flask web applications and web services, and that was really fun. And I think we're going to take it up a notch in terms of sort of abstraction and talk about a more general idea, which obviously is frequently done in Flask, but could be done in Pyramid, could be done in Django, or, you know, Node.js, right? Microservices. 00:02:19.440 --> 00:02:41.400 Yes. So microservices are really interesting, and there's a couple of ideas vying for this, like how do we decompose our big applications, and we'll dig into that. But first, I just want to let people know, if they want to hear your previous episode, that was episode 48, and maybe we could quickly talk about PyCon. We were just both there, right? You have a good time there? 00:02:41.400 --> 00:02:57.900 Yeah, yeah. I had a lot of fun. It always surprises me that people recognize me, and they stop me, and they say, you know, thank you, your tutorials, your blog, what got me into programming, or helped me advance. So I always end up with a smile. 00:02:57.980 --> 00:03:04.920 It's really amazing how many people I was around who were like, wow, people actually really appreciate what I do. 00:03:04.920 --> 00:03:11.380 Right. Yeah, yeah. Likewise. I mean, I just do it because I enjoy it. I mean, to be honest, I do it for myself. 00:03:11.740 --> 00:03:25.480 Yeah. But it's really great to see that you make a difference, like some of the other podcasters at our podcaster booths were like, wow, people really appreciate what we do. I like, you know, you do this work often, really, very much in isolation. 00:03:25.740 --> 00:03:43.840 I don't mean it in a bad way, but like, you sit down to write your book, or you write your series of blog posts, or you record a course, or even the podcast is kind of, you know, just two people, right? And then you go to one of these events, you're like, wow, there's a lot of people who this actually affects how cool it is that I'm doing this. 00:03:43.840 --> 00:03:45.420 Yeah, that's something that I really enjoy. 00:03:45.420 --> 00:03:48.720 Yeah. And you did a tutorial, right? On microservices, right? 00:03:48.880 --> 00:04:01.120 I did a tutorial. So these are the three hour long tutorials that happened before the main conference. So yeah, this year, this is the fourth consecutive year that I do a tutorial. And this one was on microservices. 00:04:01.120 --> 00:04:04.180 Yeah, excellent. So we'll definitely have to put that into the show notes. 00:04:04.180 --> 00:04:10.020 Sure. Yeah, it's been on YouTube, like the day after I gave it, they put it, they're very efficient. 00:04:10.020 --> 00:04:15.000 It's amazing. They really seem to have the AV work down at PyCon these days. 00:04:15.000 --> 00:04:16.400 They're very good. Yes. 00:04:16.460 --> 00:04:28.040 I think it's also part of their outreach mission, I would guess. Like there's 3,300 people that came to PyCon, but there's, you know, already, I think some of those videos have more people watching it online than actually attended the conference. 00:04:28.040 --> 00:04:29.000 Right. Yeah. 00:04:29.000 --> 00:04:34.700 Yeah. So what were some of your favorite sessions, like some talks or experiences? 00:04:34.700 --> 00:04:43.060 Always dictionaries. Those are always interesting. For example, there were a few this year. So, and I always end up learning something new that I didn't know. 00:04:43.060 --> 00:04:43.680 It's amazing, right? 00:04:43.960 --> 00:04:45.960 Yeah. So, and I always end up learning something new that I've been doing. 00:04:45.960 --> 00:04:46.960 Yeah. So, and I always end up learning something new that I've been doing. 00:04:46.960 --> 00:04:47.960 Yeah. So, and I always end up learning something new that I've been doing. 00:04:47.960 --> 00:04:48.960 I always end up learning something new that I've been doing. 00:04:48.960 --> 00:04:49.960 I always end up learning something new that I've been doing. 00:04:49.960 --> 00:04:50.960 I always end up learning something new that I've been doing. 00:04:50.960 --> 00:04:51.960 I always end up learning something new that I've been doing. 00:04:51.960 --> 00:04:53.960 I always end up learning something new that I've been doing. 00:04:53.960 --> 00:04:54.960 I always end up learning something new that I've been doing. 00:04:54.960 --> 00:04:55.960 I always end up learning something new that I've been doing. 00:04:55.960 --> 00:04:56.960 Yeah, I always end up learning something new that I've been doing. 00:04:56.960 --> 00:04:57.960 Yeah, I always end up learning something new that I've been doing. 00:04:57.960 --> 00:04:58.960 I always end up learning something new that I've been doing. 00:04:58.960 --> 00:04:59.960 I always end up learning something new that I've been doing. 00:04:59.960 --> 00:05:00.960 I always end up learning something new that I've been doing. 00:05:00.960 --> 00:05:01.960 I always end up learning something new that I've been doing. 00:05:01.960 --> 00:05:02.960 I always end up learning something new that I've been doing. 00:05:02.960 --> 00:05:03.960 I always end up learning something new that I've been doing. 00:05:03.960 --> 00:05:04.960 I always end up learning something new that I've been doing. 00:05:04.960 --> 00:05:05.960 I always end up learning something new that I've been doing. 00:05:05.960 --> 00:05:06.960 I always end up learning something new that I've been doing. 00:05:06.960 --> 00:05:07.960 I always end up learning something new that I've been doing. 00:05:07.960 --> 00:05:08.960 I always end up learning something new that I've been doing. 00:05:08.960 --> 00:05:10.960 You know, some people know you, some people don't. 00:05:10.960 --> 00:05:11.960 What do you do day to day? 00:05:11.960 --> 00:05:12.960 Where are you coming from? 00:05:12.960 --> 00:05:13.960 What's your perspective? 00:05:13.960 --> 00:05:16.960 In the last few years, I have been working. 00:05:16.960 --> 00:05:20.960 Well, actually, for a very long time, I've been working as a software engineer. 00:05:20.960 --> 00:05:23.960 In the last few years, I've been building cloud applications. 00:05:23.960 --> 00:05:26.960 Right now, I'm working for Rackspace. 00:05:26.960 --> 00:05:34.960 And I am helping build some of the services that our customers use when they go to the Rackspace 00:05:34.960 --> 00:05:35.960 control panel. 00:05:35.960 --> 00:05:41.960 More specifically, the services that I work on are ones that allow customers to deploy 00:05:41.960 --> 00:05:43.960 applications that then we manage. 00:05:43.960 --> 00:05:47.960 They deploy applications very easily by clicking and dragging stuff. 00:05:47.960 --> 00:05:49.960 Basically, we do all the magic. 00:05:49.960 --> 00:05:52.960 And we, of course, we use microservices for all of this. 00:05:52.960 --> 00:05:53.960 Yeah, that's really cool. 00:05:53.960 --> 00:05:54.960 And you have a book, right? 00:05:54.960 --> 00:05:57.960 That we talked about just a little bit in the previous podcast. 00:05:57.960 --> 00:05:58.960 Right. 00:05:58.960 --> 00:05:59.960 And then I have a book. 00:05:59.960 --> 00:06:01.960 The book is called Flask Web Development. 00:06:01.960 --> 00:06:05.960 It's now, I think, a little bit over three years old. 00:06:05.960 --> 00:06:09.960 And I'm currently working on the second edition. 00:06:09.960 --> 00:06:13.960 So, probably later this year, hopefully before Christmas, we'll see. 00:06:13.960 --> 00:06:15.960 The second edition will be out. 00:06:15.960 --> 00:06:18.960 That's going to basically refresh. 00:06:18.960 --> 00:06:20.960 The books largely are going to be the same. 00:06:20.960 --> 00:06:29.960 It's going to update a few things that changed in Flask or some of the extensions and related projects that I referenced. 00:06:29.960 --> 00:06:30.960 Yeah, progress is great. 00:06:30.960 --> 00:06:38.960 But when you create things like books or video courses, it's really frustrating when they change, actually. 00:06:38.960 --> 00:06:39.960 Yeah. 00:06:39.960 --> 00:06:48.960 I mean, it's really amazing, to be honest, that after three years or a little bit more, large parts of the book are still updated. 00:06:48.960 --> 00:06:53.960 Flask, thanks God, is not a framework that likes to change things in big ways. 00:06:53.960 --> 00:06:54.960 Yeah, that's right. 00:06:54.960 --> 00:06:56.960 It was a pretty mature framework when you got to it. 00:06:56.960 --> 00:07:02.960 If you did like Jopranto or Sanic right now, you might be rewriting that thing in a year. 00:07:02.960 --> 00:07:03.960 Right. 00:07:03.960 --> 00:07:04.960 Yeah. 00:07:04.960 --> 00:07:06.960 Actually, I do have one of my open source projects. 00:07:06.960 --> 00:07:13.960 I have support for Sanic and AIo HTTP, as well as Flask and Django and many others. 00:07:13.960 --> 00:07:16.960 It's a socket IO server. 00:07:16.960 --> 00:07:18.960 It's called Python socket IO. 00:07:18.960 --> 00:07:27.960 And I find that AIo, HTTP and Sanic require more attention than the old friends from the whiskey world. 00:07:27.960 --> 00:07:28.960 Yeah, absolutely. 00:07:28.960 --> 00:07:30.960 Yeah, but it's good. 00:07:30.960 --> 00:07:31.960 Those things are changing. 00:07:31.960 --> 00:07:32.960 Those things are growing. 00:07:32.960 --> 00:07:35.960 Those are the frameworks that are pushing the web forward in the Python space. 00:07:35.960 --> 00:07:36.960 Yes, absolutely. 00:07:36.960 --> 00:07:42.960 So it makes living around their orbit more work, but I think it's going to make it all better for everyone in the end. 00:07:42.960 --> 00:07:43.960 Yeah. 00:07:43.960 --> 00:07:44.960 All right. 00:07:44.960 --> 00:07:47.960 So let's start with what are microservices? 00:07:47.960 --> 00:07:56.960 Like, if I want to just take and create like a keep it on Flask since that's where your book is, if I want to just create like a Flask app, I could just put everything in there, right? 00:07:56.960 --> 00:08:01.960 I could do my user management, I could do my data access, I could do my reporting, all that stuff. 00:08:01.960 --> 00:08:05.960 I could just stick into like one big Flask app and ship that, right? 00:08:05.960 --> 00:08:06.960 Correct. 00:08:06.960 --> 00:08:11.960 And I covered many times, it's in the book and in tutorials that I've done in previous years. 00:08:11.960 --> 00:08:18.960 You know, with Flask, you have a way to organize your application when it starts to become large. 00:08:18.960 --> 00:08:20.960 It gives people some trouble. 00:08:20.960 --> 00:08:26.960 There are sometimes issues with secular dependencies, but you can do it and you can end up with a single application. 00:08:26.960 --> 00:08:33.960 In the context of microservices, we call these types of applications monoliths because they're one big thing. 00:08:33.960 --> 00:08:34.960 Yeah, yeah. 00:08:34.960 --> 00:08:39.960 So maybe compare and contrast with what are microservices relative? 00:08:39.960 --> 00:08:40.960 I can draw a parallel. 00:08:40.960 --> 00:08:47.960 So we all know that if you write your application in a single big function, that's really not good, right? 00:08:47.960 --> 00:08:48.960 It's hard to maintain. 00:08:48.960 --> 00:08:50.960 Yeah, you should at least use two functions. 00:08:50.960 --> 00:08:51.960 Two functions. 00:08:51.960 --> 00:08:53.960 You should use two or three, right. 00:08:53.960 --> 00:08:59.960 So basically what you do when you're talking about functions is you write small functions. 00:08:59.960 --> 00:09:03.960 Each function, at least you should try that it does one thing. 00:09:03.960 --> 00:09:08.960 And then the functions call each other and that's how you achieve the big problem, right? 00:09:08.960 --> 00:09:10.960 The solution to the big problem, right? 00:09:10.960 --> 00:09:10.960 Right. 00:09:10.960 --> 00:09:17.960 And functions, the way I think of it is if I can't give it just a simple name that says what it does, it's wrong. 00:09:17.960 --> 00:09:18.960 There's something wrong with it. 00:09:18.960 --> 00:09:19.960 I need to change. 00:09:19.960 --> 00:09:21.960 I need to change the function so I can name it better, right? 00:09:21.960 --> 00:09:22.960 Correct. 00:09:22.960 --> 00:09:23.960 Right. 00:09:23.960 --> 00:09:28.960 So microservices, it's basically the same idea, but applied to a web service. 00:09:28.960 --> 00:09:37.960 So the traditional way in which you develop a web application in Python, say using Flask, Bottle or Django or anything, pyramid. 00:09:37.960 --> 00:09:42.960 Basically, like you said before, you put all the contents in one application. 00:09:42.960 --> 00:09:48.960 And then without realizing it, you have a coupling between the different subsystems, right? 00:09:48.960 --> 00:09:53.960 You have a user subsystem that keeps track of your users. 00:09:53.960 --> 00:09:55.960 And then you have many others. 00:09:55.960 --> 00:10:00.960 And, you know, they all use the same database models. 00:10:00.960 --> 00:10:10.960 And you don't realize it, but you are basically making it harder for that application to grow and maintain because of all the these references that one subsystem has into the other. 00:10:10.960 --> 00:10:22.960 So solution that microservices bring is that you take all these conceptually separate subsystems and you create a separate web service with each one. 00:10:22.960 --> 00:10:29.960 Right. So maybe you've got like a front end web app that still does the back end server side stuff. 00:10:29.960 --> 00:10:38.960 But instead of going straight to the database or straight to some sub modules in your web app, it calls these related microservices that sort of implement the functionality, right? 00:10:38.960 --> 00:10:44.960 Correct. Right. So that gives you a number of advantages, some disadvantages, too. 00:10:44.960 --> 00:10:48.960 But the clear advantage is that each service is going to be very simple. 00:10:48.960 --> 00:10:53.960 We're going back to, you know, you know, very small code bases for each service. 00:10:53.960 --> 00:10:57.960 For example, with Flask, you can easily write a an entire microservice in a single file. 00:10:57.960 --> 00:11:02.960 Right. So give us some examples of microservices that would be reasonable to create. 00:11:02.960 --> 00:11:05.960 Like, would there be a logging microservice? 00:11:05.960 --> 00:11:11.960 Would there be an authentication or like how would you how have you decomposed this before? 00:11:11.960 --> 00:11:16.960 So the example that I used at the Python tutorial was a chat application. 00:11:16.960 --> 00:11:19.960 So the chat application exists as a monolith. 00:11:19.960 --> 00:11:23.960 And I showed in class how to break it into microservices. 00:11:23.960 --> 00:11:26.960 And basically there were five microservices. 00:11:26.960 --> 00:11:30.960 So one was users. That's the name. Basically registered users. 00:11:30.960 --> 00:11:32.960 The second one was tokens. 00:11:32.960 --> 00:11:37.960 Basically took care of generating authentication tokens for the client side app. 00:11:37.960 --> 00:11:41.960 There was a third microservices was messages. 00:11:41.960 --> 00:11:47.960 This is the one that adds and basically stores messages when a user types a message. 00:11:47.960 --> 00:11:50.960 The fourth was the UI application. 00:11:50.960 --> 00:11:59.960 So basically there was a very simple service that served the JavaScript and CSS and HTML files to the client. 00:11:59.960 --> 00:12:00.960 Super simple. 00:12:00.960 --> 00:12:05.960 And then the final one was the one that did the server push. 00:12:05.960 --> 00:12:09.960 So this was based on WebSocket using my socket IO server. 00:12:09.960 --> 00:12:19.960 And anytime there was a change, either a new user or a user leaving the chat or a new message, the service knew how to push those updates to all the clients. 00:12:19.960 --> 00:12:26.960 So five microservices are all completely independent applications in this case written in Flask. 00:12:26.960 --> 00:12:28.960 Okay. That makes a lot of sense. 00:12:28.960 --> 00:12:30.960 It definitely adds some complexity, right? 00:12:30.960 --> 00:12:39.960 So you're no longer maintaining the configuration for one app, but you're maintaining five or four and then the interplay between them, right? 00:12:39.960 --> 00:12:39.960 Correct. 00:12:39.960 --> 00:12:44.960 So the complexity, I did mention this in the class, the complexity doesn't go away. 00:12:44.960 --> 00:12:47.960 Basically, you're shifting the complexity to a different place. 00:12:47.960 --> 00:12:49.960 And now we have an ops. 00:12:49.960 --> 00:12:50.960 Yeah. 00:12:50.960 --> 00:12:54.960 It's more complicated to deploy an application that's based on five. 00:12:54.960 --> 00:12:56.960 And this was a relatively small app, right? 00:12:56.960 --> 00:13:01.960 Normally you may have dozens or maybe even hundreds of microservices. 00:13:01.960 --> 00:13:05.960 So, yeah, definitely the complexity goes somewhere else. 00:13:05.960 --> 00:13:12.960 What I find that I like to shift the complexity into those places because I'm a software developer, right? 00:13:12.960 --> 00:13:18.960 So from my point of view, I really like clear code that's easy to maintain. 00:13:18.960 --> 00:13:25.960 For example, something that I see done with microservices is if you have a team where you have a beginner, right? 00:13:25.960 --> 00:13:34.960 Usually if you have a big, complex application, you're going to be afraid that this person that doesn't have a lot of experience may inadvertently break something. 00:13:34.960 --> 00:13:35.960 Right. 00:13:35.960 --> 00:13:37.960 And they could break it entirely, right? 00:13:37.960 --> 00:13:38.960 Correct. 00:13:38.960 --> 00:13:39.960 It could be... 00:13:39.960 --> 00:13:41.960 You know, unknowingly, right? 00:13:41.960 --> 00:13:48.960 It's because of all this coupling that, you know, from over time keeps increasing in these types of applications. 00:13:48.960 --> 00:13:49.960 Right. 00:13:49.960 --> 00:13:59.960 And the slightest little problem in like even a trivial part of the app, if it makes it fail to start, like you've taken the entire site down for everyone, for everything, right? 00:13:59.960 --> 00:14:00.960 Right. 00:14:00.960 --> 00:14:00.960 Right. 00:14:00.960 --> 00:14:01.960 It's gone for everybody. 00:14:01.960 --> 00:14:02.960 Right. 00:14:02.960 --> 00:14:07.960 So with microservices, however, you can have a beginner work on one microservice, even own it. 00:14:07.960 --> 00:14:12.960 And if there are any problems with that microservice, that's not going to affect the overall application, right? 00:14:12.960 --> 00:14:14.960 All the other microservices will continue to run. 00:14:14.960 --> 00:14:29.960 So this is in general, not only when a beginner makes a mistake, but in general, if one microservice is sick, it goes down or has problems, that doesn't mean that the whole application goes down. 00:14:29.960 --> 00:14:30.960 It's just that system. 00:14:30.960 --> 00:14:38.960 And many times if you kill that microservice and start a new instance, then you're back up running and you have more time to fix the problem. 00:14:38.960 --> 00:14:40.960 Yeah, that's really an interesting way to think about it. 00:14:40.960 --> 00:14:47.960 And you could probably even just force a rollback to the previous deploy and run that. 00:14:47.960 --> 00:14:58.960 And that could be super hard to do in your regular application because maybe the UI has changed, maybe the database schema in some little part has changed, and SQLAlchemy freaks out or whatever, right? 00:14:58.960 --> 00:14:59.960 Right. 00:14:59.960 --> 00:14:59.960 Yeah. 00:14:59.960 --> 00:15:05.960 Databases are one of the big reasons why deployments for monolithic applications, it's so hard, right? 00:15:05.960 --> 00:15:07.960 Once you migrate the database. 00:15:07.960 --> 00:15:13.960 I mean, yes, migration frameworks have downgrades, but very few people use them. 00:15:13.960 --> 00:15:18.960 And even those that use them, many times they don't test them, so they're usually broken. 00:15:18.960 --> 00:15:18.960 Yeah. 00:15:18.960 --> 00:15:21.960 And they could be remove this column which had data in it. 00:15:21.960 --> 00:15:21.960 Correct. 00:15:21.960 --> 00:15:22.960 Right. 00:15:22.960 --> 00:15:28.960 So, yes, the idea with microservices in particular to databases is that each microservice has its own database. 00:15:28.960 --> 00:15:35.960 So, if you migrate one database for the messages service, that has nothing to do with the users. 00:15:35.960 --> 00:15:39.960 So, it's a much smaller problem if you end up having problems. 00:15:39.960 --> 00:15:40.960 Yeah, that's really cool. 00:15:40.960 --> 00:15:42.960 There's a ton of advantages to that. 00:15:42.960 --> 00:15:46.960 I like the way, gosh, who was it? 00:15:46.960 --> 00:15:52.960 Martin Fowler was referring to these databases as the ones from the monoliths and bigger ones. 00:15:52.960 --> 00:15:54.960 He called those integration databases. 00:15:54.960 --> 00:15:54.960 Right. 00:15:54.960 --> 00:15:56.960 And these called application databases. 00:15:56.960 --> 00:16:07.960 I'm not sure if that's quite the right term, but I really like to think of it as like, you can take this one big complex database that's trying to represent everything from every part of the app or multiple apps. 00:16:07.960 --> 00:16:11.960 So, the user's table is as complicated as it could possibly be, right? 00:16:11.960 --> 00:16:12.960 Right. 00:16:12.960 --> 00:16:17.960 The order history table is as complicated as it could be because it has to support every single possible option. 00:16:17.960 --> 00:16:23.960 But if you break it into these little microservices, you know, you could have a really simple like, here's the microservice that handles orders. 00:16:23.960 --> 00:16:25.960 It has a database that handles orders. 00:16:25.960 --> 00:16:26.960 Right. 00:16:26.960 --> 00:16:27.960 It's just that. 00:16:27.960 --> 00:16:28.960 Correct. 00:16:28.960 --> 00:16:30.960 Now, there's a problem with that. 00:16:30.960 --> 00:16:36.960 You lose the ability to run joins because now you don't have everything in one database. 00:16:36.960 --> 00:16:36.960 Right. 00:16:36.960 --> 00:16:42.960 So, if you need to correlate users or customers with orders, you have to do it in the application. 00:16:42.960 --> 00:16:43.960 Yeah, exactly. 00:16:43.960 --> 00:16:46.960 Like you can't join across ACP requests. 00:16:46.960 --> 00:16:47.960 Correct. 00:16:47.960 --> 00:16:48.960 That doesn't work. 00:16:48.960 --> 00:16:49.960 Not really. 00:16:49.960 --> 00:16:52.960 You have to do it in the Python space in our case. 00:16:52.960 --> 00:16:53.960 Yeah. 00:16:53.960 --> 00:16:55.960 I don't find that terrible. 00:16:55.960 --> 00:16:57.960 That's my first observation. 00:16:57.960 --> 00:17:16.960 My second observation is that even though people that know me know that I'm a big fan of relational databases, when you're working with microservices and your databases are usually one table or two tables, you know, the reason to use relational databases sort of lessens. 00:17:16.960 --> 00:17:21.960 And now it's starting to make more sense to go to a NoSQL database. 00:17:21.960 --> 00:17:21.960 Yeah. 00:17:21.960 --> 00:17:23.960 Especially a document database. 00:17:23.960 --> 00:17:24.960 Yes. 00:17:24.960 --> 00:17:29.960 The one thing that you get kind of contains the pre-joined data as a hierarchy anyway. 00:17:29.960 --> 00:17:31.960 Yeah, that's really interesting. 00:17:31.960 --> 00:17:37.960 So, I can definitely see how that makes like rolling back one of these services if it gets sick much, much easier. 00:17:37.960 --> 00:17:40.960 And the chance that it gets sick is smaller as well, right? 00:17:40.960 --> 00:17:41.960 Because it's simpler. 00:17:41.960 --> 00:17:47.960 There's a lot less chances of making mistakes because you're working with a much simpler code base. 00:17:47.960 --> 00:17:48.960 How about scalability? 00:17:48.960 --> 00:17:49.960 Well, right. 00:17:49.960 --> 00:17:54.960 So, if you have a big monolithic app, you need to scale it. 00:17:54.960 --> 00:17:56.960 You need to scale the whole thing. 00:17:56.960 --> 00:18:05.960 Maybe going back to the chat example, you're probably going to have a lot more activity around messages than around users, even less on tokens. 00:18:05.960 --> 00:18:12.960 So, if you were to scale a monolith, you're going to be basically, you're going to have to provision, you know, for the entire thing. 00:18:12.960 --> 00:18:13.960 Right. 00:18:13.960 --> 00:18:14.960 You have to work. 00:18:14.960 --> 00:18:17.960 You have to aim for the worst case scenario. 00:18:17.960 --> 00:18:18.960 Correct. 00:18:18.960 --> 00:18:20.960 Basically, across any part of it, right? 00:18:20.960 --> 00:18:25.960 So, if you're going to have to do a lot of messages from users, you need to run 10 instances. 00:18:25.960 --> 00:18:29.960 You're going to have to provision 10 instances for everything, right? 00:18:29.960 --> 00:18:31.960 Because it's all one piece. 00:18:31.960 --> 00:18:37.960 Now, when you are doing microservices, you can scale each service independently. 00:18:37.960 --> 00:18:39.960 That's really, really cool. 00:18:39.960 --> 00:18:40.960 So, it's super exciting. 00:18:40.960 --> 00:18:42.960 You can scale. 00:18:42.960 --> 00:18:48.960 I mean, if you use something like Kubernetes, for example, you can scale across different hosts. 00:18:48.960 --> 00:18:53.960 If you have a cluster of container hosts, automatically does it for you. 00:18:53.960 --> 00:19:01.960 So, you can have not only scalability, but reliability by having your instances of the same service distributed across multiple hosts. 00:19:01.960 --> 00:19:19.960 Yeah, that's really, really neat to think that, okay, I might have two or three of the straight-up web front-ins, maybe five of the orders, servers, you know, three of the message senders, and just to be able to configure those independently is really cool. 00:19:19.960 --> 00:19:21.960 And then dynamically, as well, right? 00:19:21.960 --> 00:19:22.960 Yeah. 00:19:22.960 --> 00:19:25.960 I mean, the concept of auto-scaling also applies to this. 00:19:25.960 --> 00:19:33.960 So, you know, the messages can, you know, or orders or whatever, anything that's very active, you can decide, okay, I'm going to start one more. 00:19:33.960 --> 00:19:38.960 And some other components we haven't discussed yet help with that dynamicity. 00:19:38.960 --> 00:19:39.960 Sure. 00:19:39.960 --> 00:19:57.960 One of the things that was striking about the Instagram keynote, which is a really cool story of moving Django from Python 2 to Python 3, while you have millions and millions of users, and doing it on a single branch without going down, was super interesting. 00:19:57.960 --> 00:20:11.960 But one of the things they were really obsessed about was how can we get basically being very aggressive with how they work with memory so they can get the best memory usage out of each server that they work with. 00:20:11.960 --> 00:20:18.960 Like, for example, they went so far as disabling garbage collection in their Python apps, period. 00:20:18.960 --> 00:20:20.960 Which is just crazy. 00:20:20.960 --> 00:20:21.960 That sounds scary. 00:20:21.960 --> 00:20:31.960 Yeah, they have a really interesting blog post they wrote up about it that they were able to get much better memory sharing across processes if they did that, like dramatically better. 00:20:31.960 --> 00:20:33.960 It made it... 00:20:33.960 --> 00:20:34.960 Right. 00:20:34.960 --> 00:20:36.960 It probably makes for a cleaner use of memory, right? 00:20:36.960 --> 00:20:38.960 Memory is not coming and going. 00:20:38.960 --> 00:20:39.960 Exactly. 00:20:39.960 --> 00:20:43.960 And apparently the cycles that were leaked were not sufficiently bad that there was a... 00:20:43.960 --> 00:20:44.960 Surprisingly, it worked. 00:20:44.960 --> 00:20:47.960 So the point is they're really, really focused on this. 00:20:47.960 --> 00:20:53.960 And when you scale the monolith over and over and over, maybe it takes 200 megs per worker process. 00:20:53.960 --> 00:20:54.960 Right. 00:20:54.960 --> 00:20:55.960 Yeah. 00:20:55.960 --> 00:20:56.960 If you want 10 of them, that's a gig. 00:20:56.960 --> 00:21:01.960 But you could get these other ones much, much smaller and only scale the parts that are really hot, right? 00:21:01.960 --> 00:21:02.960 Correct. 00:21:02.960 --> 00:21:04.960 It's also a big savings, right? 00:21:04.960 --> 00:21:11.960 And if you need to buy hosting for 100 instances of a monolith, that's going to be very expensive. 00:21:11.960 --> 00:21:13.960 That's going to be a lot of cloud instances. 00:21:13.960 --> 00:21:14.960 Yeah. 00:21:14.960 --> 00:21:21.960 Now, if you're using microservices, you're scaling up, you know, very little things and only the ones that you need. 00:21:21.960 --> 00:21:21.960 Yeah. 00:21:21.960 --> 00:21:25.960 So you have a lot more knobs and you end up saving a lot of money. 00:21:25.960 --> 00:21:26.960 Yeah. 00:21:26.960 --> 00:21:31.960 And this was not the case with Instagram because they were already in this monolith space. 00:21:31.960 --> 00:21:39.960 But had they been in microservices, they could have done their migration from Python 2 to Python 3 and Django 1.3 to modern. 00:21:39.960 --> 00:21:42.960 They could have done that microservice by microservice. 00:21:42.960 --> 00:21:44.960 And it probably would have been dramatically easier. 00:21:44.960 --> 00:21:45.960 One at a time. 00:21:45.960 --> 00:21:45.960 One at a time. 00:21:45.960 --> 00:21:49.960 And if, so that's one, one of the other benefits you get. 00:21:49.960 --> 00:22:06.960 Let's say, I don't know if this is true, probably not, but let's say one of the services that in this example, if they had microservices, if one of them was, could not be upgraded to Python 3 due to some late time dependency that hasn't been upgraded. 00:22:06.960 --> 00:22:07.960 That's not a problem. 00:22:07.960 --> 00:22:10.960 You can keep that one running Python 2. 00:22:10.960 --> 00:22:11.960 It doesn't matter. 00:22:11.960 --> 00:22:15.960 So you're not constrained to use the same platform in all your services. 00:22:15.960 --> 00:22:25.960 If you find that some service can be benefited if you write it in Go or in Ruby or in Node.js, that's totally fine. 00:22:25.960 --> 00:22:27.960 You can pick the best tool for each service. 00:22:27.960 --> 00:22:29.960 Yeah, that's really cool that you can break it up. 00:22:29.960 --> 00:22:32.960 And it also means, like say the data level, right? 00:22:32.960 --> 00:22:38.960 Like you talked about relational versus NoSQL, like you could do MySQL on some pieces and MongoDB on others. 00:22:38.960 --> 00:22:39.960 Absolutely. 00:22:39.960 --> 00:22:46.960 And you don't have to say, well, this part's going to have to fit into Mongo or that part's going to have to fit into MySQL when it would more naturally live somewhere else. 00:22:46.960 --> 00:22:47.960 Yeah. 00:22:47.960 --> 00:22:53.960 So basically you can pick the best tools for each service and each service is completely independent from the others. 00:22:53.960 --> 00:23:06.960 But basically you are encouraged to keep this coupling that's always bad under control by having these hard boundaries between services. 00:23:06.960 --> 00:23:08.960 Hey everyone, this is Michael. 00:23:08.960 --> 00:23:09.960 Let me tell you about Datadog. 00:23:09.960 --> 00:23:11.960 They're sponsoring this episode. 00:23:11.960 --> 00:23:15.960 Performance and bottlenecks don't exist just in your application code. 00:23:15.960 --> 00:23:18.960 Modern applications are systems built upon systems. 00:23:18.960 --> 00:23:21.960 And Datadog lets you view the system as a whole. 00:23:21.960 --> 00:23:23.960 Let's say you have a Python web app running Flask. 00:23:23.960 --> 00:23:29.960 It's built upon MongoDB and hosted and scaled out on a set of Ubuntu servers running Nginx and Microwisgi. 00:23:29.960 --> 00:23:33.960 At Datadog and you can view and monitor and even get alerts across all of these systems. 00:23:33.960 --> 00:23:37.960 Datadog has a great getting started tutorial that takes just a few moments. 00:23:37.960 --> 00:23:41.960 And if you complete it, they'll send you a sweet Datadog t-shirt for free. 00:23:41.960 --> 00:23:42.960 Don't hesitate. 00:23:42.960 --> 00:23:46.960 Visit talkpython.fm/datadog and see what you've been missing. 00:23:46.960 --> 00:23:48.960 That's talkpython.fm/datadog. 00:23:48.960 --> 00:23:59.960 You know, there are some companies that basically have rules that say you're not allowed to create a web app that has more than 10,000 lines of code in it. 00:23:59.960 --> 00:24:07.960 What you have to do is create a service and then maybe multiple services and then you can construct your app out of these services. 00:24:07.960 --> 00:24:08.960 Right? 00:24:08.960 --> 00:24:13.960 Almost like creating these guidelines that just naturally leads to microservices. 00:24:13.960 --> 00:24:15.960 So we've competed against monoliths. 00:24:15.960 --> 00:24:25.960 The other thing that I feel is like really strongly working in this space, trying to achieve the same thing has some benefits, some trade-off is serverless architecture. 00:24:25.960 --> 00:24:28.960 AWS Lambda, Azure Functions, things like this, right? 00:24:28.960 --> 00:24:29.960 Yes. 00:24:29.960 --> 00:24:32.960 What do you think about those relative to this? 00:24:32.960 --> 00:24:40.960 So glad that you asked because that's actually how we at Rackspace, in my team, that's how we deploy our microservices. 00:24:40.960 --> 00:24:49.960 So we haven't discussed this, but one of the main components in a microservices platform is the load balancer. 00:24:49.960 --> 00:24:58.960 You know, to achieve all these scalability and no downtime upgrades and another benefit that you get, you need to have all the services load balanced. 00:24:58.960 --> 00:25:01.960 Even if you run one instance, it needs to be behind a load balancer. 00:25:01.960 --> 00:25:12.960 So what you get when you go to a serverless platform like Lambda on AWS is that AWS manages the load balancing for you. 00:25:12.960 --> 00:25:16.960 So all you need to do is, you don't even need to have a WSGI server. 00:25:16.960 --> 00:25:27.960 All you need to do is write your microservice as a function and then upload the function with all its dependencies to AWS. 00:25:27.960 --> 00:25:33.960 And then anytime, you know, the function gets called, AWS will somehow figure out how to run it. 00:25:33.960 --> 00:25:36.960 It'll start a container, put the code in it, and then run it. 00:25:36.960 --> 00:25:46.960 If you, in a burst, you make a hundred calls, then AWS is running the load balancer and it'll run a hundred containers for you. 00:25:46.960 --> 00:25:49.960 You don't have to worry about it, which is really nice. 00:25:49.960 --> 00:25:56.960 And then if you make an upgrade, you know, the moment you make the upgrade, any calls from the known will use the new code. 00:25:56.960 --> 00:25:59.960 So you got immediate, no downtime upgrades as well. 00:25:59.960 --> 00:26:01.960 Yeah, that's really neat. 00:26:01.960 --> 00:26:05.960 Do you think that there's good situations to have like a combination? 00:26:05.960 --> 00:26:13.960 It seems to me like there's certain things that would be really well suited for like a serverless Lambda type of thing. 00:26:13.960 --> 00:26:20.960 And others maybe more stateful, like sort of a bigger microservice that would much better fit somewhere else. 00:26:20.960 --> 00:26:30.960 So I'm thinking like if you wanted to say charge someone's credit card with Stripe, like to do that as a, like a single Lambda function that's really stateless, really straightforward. 00:26:30.960 --> 00:26:32.960 Maybe that would make perfect sense. 00:26:32.960 --> 00:26:39.960 Maybe something more complex, like for example, your message push stuff wouldn't necessarily be as appropriate there. 00:26:39.960 --> 00:26:39.960 Right. 00:26:39.960 --> 00:26:44.960 So here's one very important thing that Lambda does not support. 00:26:44.960 --> 00:26:47.960 It does not support WebSocket services. 00:26:47.960 --> 00:26:50.960 So server push, exactly what you just mentioned. 00:26:50.960 --> 00:26:54.960 Basically for that, you need to have a permanent connection with the client. 00:26:54.960 --> 00:26:57.960 So we have a WebSocket connection. 00:26:57.960 --> 00:27:00.960 All the clients have permanent connection to the server. 00:27:00.960 --> 00:27:02.960 The server needs to handle a lot of connections. 00:27:02.960 --> 00:27:08.960 Now the Lambda services are, or functions I should say, really ephemeral, right? 00:27:08.960 --> 00:27:12.960 They run and then they exit and then they don't exist anymore until you make another call. 00:27:12.960 --> 00:27:16.960 So there's no way to have a permanent presence in Lambda. 00:27:16.960 --> 00:27:26.960 So in that case, you will have to host that in a container or something like an instance, a cloud instance, for example. 00:27:26.960 --> 00:27:27.960 Sure. 00:27:27.960 --> 00:27:33.960 And do you get maybe better response time with, if you run it and say your own container that's more permanent? 00:27:33.960 --> 00:27:39.960 Like, so there's probably a little startup to infrequently called Lambda functions or something, right? 00:27:39.960 --> 00:27:43.960 In general, if you're looking for performance, you will not be using Lambda. 00:27:43.960 --> 00:27:45.960 That's my experience. 00:27:45.960 --> 00:27:48.960 It's, in general, slower. 00:27:48.960 --> 00:27:51.960 So I can give you an example from work. 00:27:51.960 --> 00:27:55.960 You know, we don't have, you know, nothing that's extremely complex. 00:27:55.960 --> 00:28:06.960 But typically we see a REST API call that's hosted on Lambda and it goes, it gets to Lambda through another AWS service called API Gateway. 00:28:06.960 --> 00:28:11.960 And we're seeing that nothing is, nothing takes less than half a second. 00:28:11.960 --> 00:28:14.960 So 500 milliseconds for a simple call. 00:28:14.960 --> 00:28:18.960 There's really, we found no way to bring it below that. 00:28:18.960 --> 00:28:21.960 No matter how simple the actual endpoint is. 00:28:21.960 --> 00:28:26.960 There's that much overhead just to get to your function, basically, get it up and running and so on. 00:28:26.960 --> 00:28:27.960 Right. Yeah. 00:28:27.960 --> 00:28:34.960 There's so many layers of that that go through AWS before, you know, your code gets to run that you really, you can optimize all you want. 00:28:34.960 --> 00:28:36.960 You're still not going to make a difference. 00:28:36.960 --> 00:28:37.960 Yeah. 00:28:37.960 --> 00:28:42.960 Whereas in a Flask or other Python and Wizgy app, like 10 millisecond response time would be totally reasonable. 00:28:42.960 --> 00:28:43.960 Correct. Right. 00:28:43.960 --> 00:28:53.960 You can see much faster times and then you have the option to go async if you want, you know, something that's, that's very, very fast and can tolerate a lot of clients. 00:28:53.960 --> 00:29:06.960 Yeah. So I guess the takeaway probably is like these serverless components of microservices, these like serverless building blocks are cool, but you can't just in general, it wouldn't make sense to just go and I'm only doing that for the most part. 00:29:06.960 --> 00:29:07.960 Probably. 00:29:07.960 --> 00:29:10.960 It makes sense in many cases, not every case. 00:29:10.960 --> 00:29:13.960 And you have to keep performance in mind. 00:29:13.960 --> 00:29:23.960 Typically, if you know, the kind of services that we do, these are all requested by a, by a rich client application, something like Angular or React. 00:29:23.960 --> 00:29:26.960 And, you know, those are background requests. 00:29:26.960 --> 00:29:28.960 So, yeah, if it takes half a second, that's fine. 00:29:28.960 --> 00:29:29.960 Yeah. 00:29:29.960 --> 00:29:30.960 Usually not, not a big deal. 00:29:30.960 --> 00:29:36.960 We actually, we, we had one of the little services that we wanted to build at some point was, 00:29:36.960 --> 00:29:37.960 a auto-completion. 00:29:37.960 --> 00:29:38.960 Yeah. 00:29:38.960 --> 00:29:39.960 It's not worth that. 00:29:39.960 --> 00:29:40.960 And was that frustrating on Lambda? 00:29:40.960 --> 00:29:41.960 Yes, exactly. 00:29:41.960 --> 00:29:43.960 That you cannot, you cannot host that in Lambda. 00:29:43.960 --> 00:29:47.960 That's the typing equivalent of hearing your own voice echo back in a call, right? 00:29:47.960 --> 00:29:48.960 Yeah. 00:29:48.960 --> 00:29:49.960 It's not good. 00:29:49.960 --> 00:29:50.960 Yeah. 00:29:50.960 --> 00:29:51.960 That didn't go the way. 00:29:51.960 --> 00:29:52.960 Yeah. 00:29:52.960 --> 00:29:53.960 That was. 00:29:53.960 --> 00:29:57.960 So one of the challenges that I can certainly see, especially if you start throwing containers 00:29:57.960 --> 00:30:04.960 into this as well as like, if I have a monolith, it knows how to find the user, the user interaction 00:30:04.960 --> 00:30:06.960 bit and the credit card bit and so on. 00:30:06.960 --> 00:30:09.960 It just, you know, imports and it works. 00:30:09.960 --> 00:30:15.960 But if you break it across all these different servers, like how do you keep it connected without 00:30:15.960 --> 00:30:17.960 the hardwiring every bit of deployment into the code? 00:30:17.960 --> 00:30:24.960 There's a component that all microservices platform have that's called the service registry. 00:30:24.960 --> 00:30:29.960 So basically there are, you know, each platform does it in a slightly different way. 00:30:29.960 --> 00:30:35.960 But in general, the idea is that when you start a service or an instance of a service, the first 00:30:35.960 --> 00:30:40.960 thing that the service does is talk to the service registry, which is in a known address, 00:30:40.960 --> 00:30:41.960 right? 00:30:41.960 --> 00:30:44.960 So everybody knows where to find the service registry. 00:30:44.960 --> 00:30:48.960 That one, you basically hard code the domain or something in it. 00:30:48.960 --> 00:30:49.960 Right. 00:30:49.960 --> 00:30:50.960 It's hard coded. 00:30:50.960 --> 00:30:53.960 It's usually in production deployments, it's highly available. 00:30:53.960 --> 00:30:56.960 So you are not going to have, you know, a single point of contact. 00:30:56.960 --> 00:31:01.960 Probably you can code a few, you know, addresses to talk to the service. 00:31:01.960 --> 00:31:03.960 And if one of them is down, you try the next one. 00:31:03.960 --> 00:31:06.960 So you want to make sure that, you know, this piece of code is always running. 00:31:06.960 --> 00:31:11.960 Basically, the service starts and then it reports itself to the registry. 00:31:11.960 --> 00:31:12.960 It says, hey, I'm here. 00:31:12.960 --> 00:31:13.960 I'm at this address. 00:31:13.960 --> 00:31:18.960 If you're running containers, the address is going to be basically Docker, for example, 00:31:18.960 --> 00:31:20.960 it's going to come up with some port for you. 00:31:20.960 --> 00:31:23.960 So you find out what port you're running on. 00:31:23.960 --> 00:31:27.960 And then you tell the service, okay, I'm running on this address and this port. 00:31:27.960 --> 00:31:29.960 So I'm ready to start taking jobs. 00:31:29.960 --> 00:31:30.960 Yeah. 00:31:30.960 --> 00:31:31.960 Yeah. 00:31:31.960 --> 00:31:34.960 And that service registry can be very simple code, right? 00:31:34.960 --> 00:31:38.960 It could almost just store like a dictionary in memory or something, right? 00:31:38.960 --> 00:31:39.960 Right. 00:31:39.960 --> 00:31:42.960 Essentially, it's a dictionary, right? 00:31:42.960 --> 00:31:43.960 If you think about it. 00:31:43.960 --> 00:31:47.960 The complication is that you want that to be super robust. 00:31:47.960 --> 00:31:56.960 If the server where the registry is running goes down, then nothing can find, there's no way for microservices to communicate. 00:31:56.960 --> 00:32:03.960 So it's very important that you hosted, you know, multiple locations and, you know, that there is a redundancy. 00:32:03.960 --> 00:32:06.960 Do people sometimes use things like S3? 00:32:06.960 --> 00:32:09.960 Just like, I'm going to write to this bucket. 00:32:09.960 --> 00:32:12.960 And then, that's actually an interesting idea. 00:32:12.960 --> 00:32:13.960 I haven't seen that. 00:32:13.960 --> 00:32:16.960 It might get complicated with multiple accesses. 00:32:16.960 --> 00:32:17.960 Yeah. 00:32:17.960 --> 00:32:20.960 You need to implement some sort of a knocking mechanism. 00:32:20.960 --> 00:32:27.960 I would think to keep the, you know, the file that has all the list of services robust and never get corrupted. 00:32:27.960 --> 00:32:28.960 Yeah. 00:32:28.960 --> 00:32:29.960 Maybe each file could have it. 00:32:29.960 --> 00:32:30.960 Each service could have its own file. 00:32:30.960 --> 00:32:31.960 I don't know. 00:32:31.960 --> 00:32:32.960 Or, right. 00:32:32.960 --> 00:32:33.960 You could write different files. 00:32:33.960 --> 00:32:34.960 That, that actually could work. 00:32:34.960 --> 00:32:35.960 Yeah. 00:32:35.960 --> 00:32:36.960 I think that could work. 00:32:36.960 --> 00:32:37.960 Interesting. 00:32:37.960 --> 00:32:41.960 But basically, there's a really simple thing that just every, every server can go, I'm here. 00:32:41.960 --> 00:32:42.960 The things I need, where are they? 00:32:42.960 --> 00:32:43.960 Right? 00:32:43.960 --> 00:32:44.960 Right. 00:32:44.960 --> 00:32:53.960 So then, just to complete this, they, one of the simplest service registry options that, that, that, the one that I like, it's called etcd. 00:32:53.960 --> 00:32:56.960 It's an open source project from core OS. 00:32:56.960 --> 00:33:00.960 And, basically, yeah, you, you send a, a request. 00:33:00.960 --> 00:33:02.960 You can even do it in a batch script with core. 00:33:02.960 --> 00:33:03.960 Okay. 00:33:03.960 --> 00:33:06.960 Just, it's a key value database, basically. 00:33:06.960 --> 00:33:07.960 That's very fast. 00:33:07.960 --> 00:33:12.960 So then etcd, in this example, will have the list of all the services that are running. 00:33:12.960 --> 00:33:14.960 On the other end, we have the load balancer. 00:33:14.960 --> 00:33:24.960 And the load balancer will go periodically check the contents of this service registry and refresh its own configuration based on that. 00:33:24.960 --> 00:33:28.960 So, a service starts, it writes itself to the service registry. 00:33:28.960 --> 00:33:31.960 Then on the other side, the load balancer says, oh, there's a new service. 00:33:31.960 --> 00:33:32.960 I'm going to add it. 00:33:32.960 --> 00:33:33.960 Yeah. 00:33:33.960 --> 00:33:34.960 Oh, that's, that's cool. 00:33:34.960 --> 00:33:38.960 I didn't think of using the service registry to drive the load balancer, but that's cool. 00:33:38.960 --> 00:33:39.960 Yeah, that, that, that's very nice. 00:33:39.960 --> 00:33:45.960 There are, there's actually, I know of one load balancer that, that has this functionality embedded. 00:33:45.960 --> 00:33:46.960 It's called traffic. 00:33:46.960 --> 00:33:53.960 If you go with a more traditional one, like Nginx or HAProxy, which are the ones that I've used for a lot of time. 00:33:53.960 --> 00:34:04.960 With those, you, you need to add a piece to the side that does the, the, the, watching the service registry and then updating the configuration and restarting the load balancer. 00:34:04.960 --> 00:34:05.960 Right, right. 00:34:05.960 --> 00:34:10.960 And the one that I know about, which is actually written by, by Kelsey, it's called ConfD. 00:34:10.960 --> 00:34:12.960 That's the one that I showed in the class. 00:34:12.960 --> 00:34:13.960 Okay. 00:34:13.960 --> 00:34:14.960 Yeah. 00:34:14.960 --> 00:34:15.960 Yeah. 00:34:15.960 --> 00:34:16.960 That's cool. 00:34:16.960 --> 00:34:17.960 And Nginx resorts pretty quickly. 00:34:17.960 --> 00:34:18.960 So it's, it's pretty good. 00:34:18.960 --> 00:34:19.960 Right. 00:34:19.960 --> 00:34:22.960 Nginx is pretty good about reloading itself cleanly. 00:34:22.960 --> 00:34:26.960 When, when you update the configuration, HAProxy is getting there. 00:34:26.960 --> 00:34:27.960 It's getting better. 00:34:27.960 --> 00:34:33.960 It's a little bit clunky, but basically it starts a new process when, when you want to update the configuration. 00:34:33.960 --> 00:34:42.960 And process by which all the connections are passed from the old process to the new process has been for many years has been problematic. 00:34:42.960 --> 00:34:43.960 It costs some downtime. 00:34:43.960 --> 00:34:45.960 It's much better these days. 00:34:45.960 --> 00:34:46.960 Okay. 00:34:46.960 --> 00:34:47.960 That's great. 00:34:47.960 --> 00:34:56.960 Another challenge I can imagine is if I just start using logging in my monolith app, it will all go to the same file. 00:34:56.960 --> 00:34:59.960 It will go always in order unless I'm doing threading. 00:34:59.960 --> 00:35:00.960 It's a piece of cake, right? 00:35:00.960 --> 00:35:01.960 It's a piece of cake. 00:35:01.960 --> 00:35:06.960 If I have 20 little microservices, how do I like trace through the steps? 00:35:06.960 --> 00:35:07.960 Right. 00:35:07.960 --> 00:35:08.960 You need help. 00:35:08.960 --> 00:35:12.960 You really can't manage 20 log files. 00:35:12.960 --> 00:35:14.960 Sometimes it gets insane. 00:35:14.960 --> 00:35:20.960 So basically you use, you find another piece, but basically a log consolidator. 00:35:20.960 --> 00:35:26.960 And there's one that's pretty good for, I mean, if you're doing Docker containers, that's called log spout. 00:35:26.960 --> 00:35:27.960 Log spout. 00:35:27.960 --> 00:35:28.960 Okay. 00:35:28.960 --> 00:35:37.960 So basically what this does is it grabs all the logs from all the containers in a host, the host where log spout is running. 00:35:37.960 --> 00:35:42.960 And then basically it outputs a stream that you can watch over. 00:35:42.960 --> 00:35:45.960 For example, you can connect with a web browser. 00:35:45.960 --> 00:35:47.960 That's WebSocket, for example. 00:35:47.960 --> 00:35:51.960 So you can connect over WebSocket and then watch the stream of logs in real time. 00:35:51.960 --> 00:36:00.960 Or you can connect log spout to something more useful in a production environment, which will be, for example, an elk stack. 00:36:00.960 --> 00:36:12.960 So Elasticsearch, Logstash, and Kibana, which is this trio of open source apps that basically create a very powerful log storage and search solution. 00:36:12.960 --> 00:36:13.960 Yeah. 00:36:13.960 --> 00:36:13.960 Okay. 00:36:13.960 --> 00:36:17.960 So basically you put in something that brings it back into one place. 00:36:17.960 --> 00:36:18.960 Yes. 00:36:18.960 --> 00:36:23.960 You usually want to have everything in one stream and then you can filter if you want. 00:36:23.960 --> 00:36:34.960 But imagine if you have five instances of the same microservice, you may, even though it's five different ones, you may want to see the entire thing. 00:36:34.960 --> 00:36:41.960 Because if a client is sending requests to this service, requests are going to randomly arrive to any of those five. 00:36:41.960 --> 00:36:42.960 Right. 00:36:42.960 --> 00:36:43.960 It's hitting the load balancer. 00:36:43.960 --> 00:36:44.960 Yeah. 00:36:44.960 --> 00:36:45.960 Right. 00:36:45.960 --> 00:36:52.960 So you probably want to see the sequence regardless of which of the instances get a specific request. 00:36:52.960 --> 00:36:53.960 Yeah. 00:36:53.960 --> 00:36:54.960 Yeah. 00:36:54.960 --> 00:37:00.960 Do you do or do people do things where they somehow identify a request? 00:37:00.960 --> 00:37:10.960 So like flag it with some kind of unique ID at the top and then flow that through so that you can say this, this is the steps that this request went through. 00:37:10.960 --> 00:37:10.960 Yeah. 00:37:10.960 --> 00:37:11.960 That's pretty common. 00:37:11.960 --> 00:37:14.960 Some, some platforms offer that. 00:37:14.960 --> 00:37:17.960 I implemented by hand in some cases myself. 00:37:17.960 --> 00:37:21.960 But basically, yes, the entry point request. 00:37:21.960 --> 00:37:28.960 So the first, the service that receives a request from the outside world assigns an ID to that request. 00:37:28.960 --> 00:37:34.960 And then in any communications that service has with other microservices, it will pass that ID. 00:37:34.960 --> 00:37:45.960 So you always preserve and log the initial ID and then, and then you get a trace of all the services that worked on a single client request. 00:37:45.960 --> 00:37:45.960 Right. 00:37:45.960 --> 00:37:49.960 And there's infrastructure to actually do the communication between microservices. 00:37:49.960 --> 00:37:51.960 Is that typically requests? 00:37:51.960 --> 00:37:52.960 There are different ways. 00:37:52.960 --> 00:37:58.960 So the easiest will be to use HTTP as an interface, as a REST API for each service. 00:37:58.960 --> 00:38:01.960 And then, yeah, you use a Python requests. 00:38:01.960 --> 00:38:05.960 Some people prefer something that's less chatty. 00:38:05.960 --> 00:38:10.960 So HTTP, as you know, you know, you have all these requests, the headers, you know, all that stuff. 00:38:10.960 --> 00:38:13.960 When you are talking to a client, that makes sense. 00:38:13.960 --> 00:38:14.960 Right. 00:38:14.960 --> 00:38:19.960 And besides, that's the only way the browser can, or an HTTP client can talk to the server. 00:38:19.960 --> 00:38:24.960 But when you're talking among services, you may say, well, okay, I want something quicker. 00:38:24.960 --> 00:38:31.960 So some people implement RPC schemes where a service can say to the other server, hey, I need to do, I need you to do this. 00:38:31.960 --> 00:38:38.960 And it's, for example, passing messages through a message queue, which could be a Redis queue or SQS if you're on AWS. 00:38:38.960 --> 00:38:41.960 Do people set up socket servers, maybe? 00:38:41.960 --> 00:38:42.960 You could do a socket server too. 00:38:42.960 --> 00:38:43.960 Yeah. 00:38:43.960 --> 00:38:44.960 Yeah. 00:38:44.960 --> 00:38:45.960 If you're looking for really low latency, low traffic. 00:38:45.960 --> 00:38:58.960 The main idea, what I would consider a good design point, you know, when thinking about how microservices communicate, you want to leave the door open to having different, you know, tech stacks on different services. 00:38:58.960 --> 00:39:02.960 You don't want to go with something that, let me give you an example. 00:39:02.960 --> 00:39:06.960 I would probably not use a Celery worker for this, right? 00:39:06.960 --> 00:39:09.960 Because that will basically restrict me to use Python. 00:39:09.960 --> 00:39:09.960 Right. 00:39:09.960 --> 00:39:12.960 And you probably wouldn't ship data across the wire as pickle. 00:39:12.960 --> 00:39:13.960 Right. 00:39:13.960 --> 00:39:14.960 Yeah. 00:39:14.960 --> 00:39:17.960 As versioning issues, not just, not just Python. 00:39:17.960 --> 00:39:18.960 Yeah. 00:39:18.960 --> 00:39:21.960 But even within Python, that could be a big problem too. 00:39:21.960 --> 00:39:21.960 Right. 00:39:21.960 --> 00:39:22.960 Yeah. 00:39:22.960 --> 00:39:22.960 Yeah. 00:39:22.960 --> 00:39:22.960 Yeah. 00:39:22.960 --> 00:39:23.960 Yeah. 00:39:23.960 --> 00:39:23.960 Yeah. 00:39:23.960 --> 00:39:24.960 Yeah. 00:39:24.960 --> 00:39:24.960 Yeah. 00:39:24.960 --> 00:39:25.960 Yeah. 00:39:25.960 --> 00:39:28.960 So, you know, the way Celery works, I think it's not, you know, it's not friendly to the microservices 00:39:28.960 --> 00:39:29.960 idea. 00:39:29.960 --> 00:39:31.960 So the idea is that microservices try to push. 00:39:31.960 --> 00:39:33.960 So yeah, I would go HTTP. 00:39:33.960 --> 00:39:38.960 Maybe messages are JSON formatted over a queue. 00:39:38.960 --> 00:39:39.960 Mm-hmm. 00:39:39.960 --> 00:39:44.960 All things that you are sure that any technology can easily communicate over. 00:39:44.960 --> 00:39:45.960 Yeah. 00:39:45.960 --> 00:39:46.960 It makes a lot of sense. 00:39:46.960 --> 00:39:49.960 I imagine there's a lot of JSON happening back in the data center. 00:39:49.960 --> 00:39:50.960 Right. 00:39:50.960 --> 00:39:51.960 Yes. 00:39:51.960 --> 00:39:52.960 Absolutely. 00:39:52.960 --> 00:39:53.960 Yeah. 00:39:53.960 --> 00:39:54.960 Nice. 00:39:54.960 --> 00:39:55.960 So that maybe brings us to an interesting place to talk about the tools. 00:39:55.960 --> 00:39:59.960 We talked about requests and some of the other tools that work and don't work, but what 00:39:59.960 --> 00:40:02.960 are the other Python tools that you might involve here? 00:40:02.960 --> 00:40:04.960 Well, of course you need a framework, right? 00:40:04.960 --> 00:40:06.960 And we discussed this. 00:40:06.960 --> 00:40:12.960 I like, or surprise, I like to use Flask, but really you can use any web framework that 00:40:12.960 --> 00:40:13.960 you like, right? 00:40:13.960 --> 00:40:17.960 As long as it knows how to communicate with the other services. 00:40:17.960 --> 00:40:24.960 As far as other Python tools, there are many packages that talk to service registries, for 00:40:24.960 --> 00:40:25.960 example. 00:40:25.960 --> 00:40:31.960 So if you want your Flask-based or Python-based microservice to be able to talk to the service 00:40:31.960 --> 00:40:35.960 registry, there's packages for certainly for etcd. 00:40:35.960 --> 00:40:41.960 If you use another one like console, for example, this one from HashiCorp, there's packages. 00:40:41.960 --> 00:40:46.960 So you're going to find a lot of support for these tasks that you need to do that are sort 00:40:46.960 --> 00:40:50.960 of specific to microservices in the Python ecosystem. 00:40:50.960 --> 00:40:51.960 Sure. 00:40:51.960 --> 00:40:56.960 Besides that, you're going to be doing things, and this is something that I really like, in 00:40:56.960 --> 00:41:00.960 the same way like you work with a monolith, but you're going to be working with a much much 00:41:00.960 --> 00:41:01.960 much smaller code bases. 00:41:01.960 --> 00:41:02.960 code bases. 00:41:02.960 --> 00:41:07.960 So you're going to be doing unit tests the usual way, but you're going to be testing each 00:41:07.960 --> 00:41:08.960 service separately. 00:41:08.960 --> 00:41:12.960 And then you're going to have integration tests if you need, and you probably do. 00:41:12.960 --> 00:41:14.960 But yeah, nothing really changes. 00:41:14.960 --> 00:41:20.960 It's just that the scale goes, you know, you're doing your work on a much smaller scale. 00:41:20.960 --> 00:41:22.960 You're working with smaller applications. 00:41:22.960 --> 00:41:23.960 Yeah. 00:41:23.960 --> 00:41:28.960 It sounds to me like they're easier to work on and maintain and deploy, but maybe more difficult 00:41:28.960 --> 00:41:31.960 to initially set up the infrastructure that wires them together. 00:41:31.960 --> 00:41:32.960 Yes. 00:41:32.960 --> 00:41:33.960 You have more servers to set up. 00:41:33.960 --> 00:41:34.960 You've got the load balancers. 00:41:34.960 --> 00:41:35.960 You've got the service registry. 00:41:35.960 --> 00:41:37.960 Like these things you have to add. 00:41:37.960 --> 00:41:40.960 But once that is in place, it kind of sounds like life gets easier. 00:41:40.960 --> 00:41:44.960 So there's like a bar to cross, but once you cross it, you're good. 00:41:44.960 --> 00:41:45.960 Yeah. 00:41:45.960 --> 00:41:46.960 Right. 00:41:46.960 --> 00:41:47.960 I agree with that. 00:41:47.960 --> 00:41:48.960 Yes. 00:41:48.960 --> 00:41:49.960 It's difficult to set up the platform. 00:41:49.960 --> 00:41:56.960 And of course you can go, if you use Kubernetes, for example, or AWS Lambda, you know, a lot 00:41:56.960 --> 00:41:59.960 of all those pieces are done for you. 00:41:59.960 --> 00:42:04.960 You don't have to worry about load balancers and service registries, right? 00:42:04.960 --> 00:42:05.960 They do it for you. 00:42:05.960 --> 00:42:09.960 This portion of Talk Python To Me is brought to you by us. 00:42:09.960 --> 00:42:10.960 Us. 00:42:10.960 --> 00:42:13.960 As many of you know, I have a growing set of courses to help you go from Python beginner 00:42:13.960 --> 00:42:15.960 to novice to Python expert. 00:42:15.960 --> 00:42:17.960 And there are many more courses in the works. 00:42:17.960 --> 00:42:21.960 So please consider Talk Python training for you and your team's training needs. 00:42:21.960 --> 00:42:26.960 If you're just getting started, I've built a course to teach you Python the way professional 00:42:26.960 --> 00:42:28.960 developers learn by building applications. 00:42:28.960 --> 00:42:34.960 Check out my Python jumpstart by building 10 apps at talkpython.fm/course. 00:42:34.960 --> 00:42:36.960 Are you looking to start adding services to your app? 00:42:36.960 --> 00:42:39.960 Try my brand new consuming HTTP services in Python. 00:42:39.960 --> 00:42:45.960 You'll learn to work with the restful HTTP services as well as SOAP, JSON and XML data formats. 00:42:45.960 --> 00:42:47.960 Do you want to launch an online business? 00:42:47.960 --> 00:42:51.960 Well, Matt Makai and I built an entrepreneur's playbook with Python for entrepreneurs. 00:42:51.960 --> 00:42:55.960 This 16 hour course will teach you everything you need to launch your web based business with 00:42:55.960 --> 00:42:56.960 Python. 00:42:56.960 --> 00:42:59.960 And finally, there's a couple of new course announcements coming really soon. 00:42:59.960 --> 00:43:04.960 So if you don't already have an account, be sure to create one at training.talkpython.fm 00:43:04.960 --> 00:43:05.960 to get notified. 00:43:05.960 --> 00:43:09.960 And for all of you who have bought my courses, thank you so much. 00:43:09.960 --> 00:43:11.960 It really, really helps support the show. 00:43:11.960 --> 00:43:17.960 In this class that I gave at PyCon, I didn't want to just tell, okay, install Kubernetes and 00:43:17.960 --> 00:43:18.960 you're done. 00:43:18.960 --> 00:43:20.960 I wanted to teach what microservices are. 00:43:20.960 --> 00:43:24.960 So I built my own platform, which was a lot of fun. 00:43:24.960 --> 00:43:32.960 And I thank the PSF for approving my tutorial ideas and let me work on this. 00:43:32.960 --> 00:43:33.960 It was a lot of fun. 00:43:33.960 --> 00:43:38.960 And I wanted to demonstrate that really it's not as hard as it sounds. 00:43:38.960 --> 00:43:43.960 You can go pick, you know, the best tools for each of the different tasks, you know, in 00:43:43.960 --> 00:43:46.960 a very flask, you know, fashion, right? 00:43:46.960 --> 00:43:50.960 Where everything is done, you know, you pick the best tool for each task. 00:43:50.960 --> 00:43:55.960 Yeah, it sounds like people who like micro frameworks for their web frameworks might like this as 00:43:55.960 --> 00:43:56.960 well, right? 00:43:56.960 --> 00:43:57.960 Because you kind of get a pick and... 00:43:57.960 --> 00:44:01.960 Yes, they're going to find that there's a lot of affinity, right? 00:44:01.960 --> 00:44:04.960 So I built a platform using Bash. 00:44:04.960 --> 00:44:05.960 So it's all Bash scripts. 00:44:05.960 --> 00:44:06.960 You can do a... 00:44:06.960 --> 00:44:13.960 You've seen Kelsey Hightower do a super cool voice operated demo of a no downtime upgrade, 00:44:13.960 --> 00:44:14.960 right? 00:44:14.960 --> 00:44:15.960 So take away the voice thing. 00:44:15.960 --> 00:44:17.960 I didn't do that. 00:44:17.960 --> 00:44:23.960 But, you know, during class I showed how with a Bash script you can deploy your upgrades 00:44:23.960 --> 00:44:25.960 without the service ever going down. 00:44:25.960 --> 00:44:28.960 You know, your application keeps running while you do the deployment. 00:44:28.960 --> 00:44:29.960 Sure. 00:44:29.960 --> 00:44:30.960 So, yeah. 00:44:30.960 --> 00:44:31.960 Yeah. 00:44:31.960 --> 00:44:38.960 So is there a roadmap or some guidance on how to take maybe a 50,000 line monolith web app 00:44:38.960 --> 00:44:41.960 and turn it into a number of services? 00:44:41.960 --> 00:44:43.960 Well, that's really difficult. 00:44:43.960 --> 00:44:49.960 I think that the best advice I can give you is you probably cannot do it all in one go. 00:44:49.960 --> 00:44:55.960 You're going to have to find a strategy to do, you know, phased migration to microservices. 00:44:55.960 --> 00:45:02.960 You need to have a very good unit tests in place because the task of... 00:45:02.960 --> 00:45:10.960 Basically, what you're going to do in most cases is take the monolith and put it inside a microservices platform as a big piece, right? 00:45:10.960 --> 00:45:15.960 And then over time, you're going to start taking little pieces out of it and write microservices, right? 00:45:15.960 --> 00:45:23.960 So the task of writing a microservice when you have, you know, the monolith is basically involves a lot of copying and pasting, right? 00:45:23.960 --> 00:45:28.960 You have to move endpoints that are in the monolith to a standalone application. 00:45:28.960 --> 00:45:29.960 Sure. 00:45:29.960 --> 00:45:39.960 And that's pretty easy in some aspect, but breaking the tight coupling and the dependencies of code that you're moving around, that sounds to me like it could be pretty challenging. 00:45:39.960 --> 00:45:40.960 Yes. 00:45:40.960 --> 00:45:41.960 It's difficult. 00:45:41.960 --> 00:45:42.960 It's actually hard. 00:45:42.960 --> 00:45:43.960 You know, all that... 00:45:43.960 --> 00:45:46.960 Basically, when you work on a monolith, you accumulate technical debt. 00:45:46.960 --> 00:45:48.960 That's pretty common. 00:45:48.960 --> 00:45:53.960 You're going to find that many times that technical debt is going to inform your decisions. 00:45:53.960 --> 00:45:59.960 You're going to take less than ideal decisions when you design your microservices to keep things the same way. 00:45:59.960 --> 00:46:00.960 I can give you an example. 00:46:00.960 --> 00:46:08.960 In this project that I showed during the PyCon class, I was actually migrating this chat application to microservices. 00:46:08.960 --> 00:46:11.960 And I started and I migrated the UI first. 00:46:11.960 --> 00:46:13.960 That was very easy. 00:46:13.960 --> 00:46:15.960 And then I migrated the users. 00:46:15.960 --> 00:46:17.960 And then I went to migrate tokens. 00:46:17.960 --> 00:46:21.960 And I realized that I could do a much better job with tokens. 00:46:21.960 --> 00:46:27.960 The tokens in the old application were sort of inefficient, were random strings. 00:46:27.960 --> 00:46:34.960 You know, when you're working with microservices, you want tokens that can be verified without calling the token service. 00:46:34.960 --> 00:46:40.960 And when you need that, you usually use JSON web tokens, which you can verify with cryptography. 00:46:40.960 --> 00:46:45.960 So, I had to decide, I mean, do I keep this and make it inefficient? 00:46:45.960 --> 00:46:53.960 Or do I say, okay, I'm going to draw a line in the sand and I'm going to change the token format, but then everything is going to break. 00:46:53.960 --> 00:46:56.960 I'm going to have to migrate all the services, you know, to the new token style. 00:46:56.960 --> 00:46:57.960 Right. 00:46:57.960 --> 00:47:02.960 And those decisions, you know, on a real application, they're going to be, you know, much harder to make. 00:47:02.960 --> 00:47:04.960 Yeah, I can imagine that. 00:47:04.960 --> 00:47:16.960 But, so, one thing I was thinking of while you were speaking of like how you might break this apart, it seems like you could almost do the partition in the web tier and then the data. 00:47:16.960 --> 00:47:23.960 Like, so, for example, if you have a database that obviously the monolith talks to the database, all of it to the same, through the same connections. 00:47:23.960 --> 00:47:32.960 If you could break this out into services, they theoretically could go back and just continue talking to the same database and you could kind of get the service decomposition piece working. 00:47:32.960 --> 00:47:37.960 And then you could say, okay, now how do we move this into the application database that's dedicated to each one of the services. 00:47:37.960 --> 00:47:39.960 So, that could be a valid approach. 00:47:39.960 --> 00:47:47.960 So, when you do it that way, if you're sharing the database, then the zero downtime upgrades are still difficult. 00:47:47.960 --> 00:47:49.960 Yeah, I'm just thinking as a transitional thing. 00:47:49.960 --> 00:47:50.960 So, it's going to be a transition, right? 00:47:50.960 --> 00:47:50.960 Yeah. 00:47:50.960 --> 00:47:52.960 You want to go all the way eventually. 00:47:52.960 --> 00:47:53.960 But, yeah, definitely. 00:47:53.960 --> 00:47:53.960 Okay. 00:47:53.960 --> 00:47:55.960 Yeah, you have to figure it out. 00:47:55.960 --> 00:47:57.960 It depends on the application what's the best route. 00:47:57.960 --> 00:47:59.960 But, yeah, it's difficult. 00:47:59.960 --> 00:48:04.960 What I've seen some people do is they say, okay, I'm going to migrate to microservices, but only from now on. 00:48:04.960 --> 00:48:07.960 I'm not going to change what I have. 00:48:07.960 --> 00:48:10.960 So, basically, they grandfather this big piece. 00:48:10.960 --> 00:48:13.960 You know, they think of it as a microservice, even though it's not. 00:48:13.960 --> 00:48:14.960 It's a big microservice. 00:48:14.960 --> 00:48:15.960 Right. 00:48:15.960 --> 00:48:16.960 It's a big one. 00:48:16.960 --> 00:48:21.960 But then, you know, from then on, any new functionality, they start writing in microservices. 00:48:21.960 --> 00:48:24.960 And that's actually a very valid approach. 00:48:24.960 --> 00:48:26.960 In many cases, it's the only viable way, right? 00:48:26.960 --> 00:48:27.960 Sure. 00:48:27.960 --> 00:48:28.960 Sure, sure. 00:48:28.960 --> 00:48:33.960 How does software development change for a team when they're transitioning to microservices? 00:48:33.960 --> 00:48:35.960 How does their world get different? 00:48:35.960 --> 00:48:38.960 Well, they don't have to all work in the same code, right? 00:48:38.960 --> 00:48:40.960 So, that's a big plus. 00:48:40.960 --> 00:48:42.960 Fewer merge conflicts. 00:48:42.960 --> 00:48:43.960 Right. 00:48:43.960 --> 00:48:45.960 You basically merge conflicts. 00:48:45.960 --> 00:48:49.960 For me, I don't remember when I had a merge conflict last. 00:48:49.960 --> 00:48:50.960 Right? 00:48:50.960 --> 00:48:51.960 Yeah, you don't usually see that. 00:48:51.960 --> 00:48:58.960 So, usually, you're going to find that your team, the members will get specialized in specific services, right? 00:48:58.960 --> 00:49:03.960 For example, at Rackspace, I've been doing a lot of authentication microservices, right? 00:49:03.960 --> 00:49:10.960 So, you know, when there's a new need for authentication, I do it usually. 00:49:10.960 --> 00:49:12.960 Some people may not like that, right? 00:49:12.960 --> 00:49:15.960 May prefer to be generalists. 00:49:15.960 --> 00:49:16.960 So, yeah, it depends. 00:49:16.960 --> 00:49:26.960 But you find that, you know, some people is more, basically, has affinity to certain parts of the system, certain microservices. 00:49:26.960 --> 00:49:27.960 Sure. 00:49:27.960 --> 00:49:29.960 And now they can focus more on it now because it's more explicit. 00:49:29.960 --> 00:49:30.960 Right, yeah. 00:49:30.960 --> 00:49:40.960 And they can do a much better job at, you know, that specific task because they don't have all the baggage of the remaining, the rest of the system. 00:49:40.960 --> 00:49:46.960 That's basically that needs to be, you know, that you have to make sure that you don't break. 00:49:46.960 --> 00:49:47.960 Right. 00:49:47.960 --> 00:49:55.960 So, one thing that seems like it might make sense would be to rotate the team around through the different services, potentially. 00:49:55.960 --> 00:49:58.960 If you want to make sure, like, there's many people that kind of know the whole thing. 00:49:58.960 --> 00:50:01.960 Like, you can say, okay, this month you're on this service, that month you're on. 00:50:01.960 --> 00:50:03.960 Yeah, that's actually a good idea. 00:50:03.960 --> 00:50:17.960 You can find a way for the person that's experienced with microservice to sort of mentor a new member, you know, and basically code review the changes that the new person makes, for example. 00:50:17.960 --> 00:50:18.960 Yeah. 00:50:18.960 --> 00:50:22.960 There are a lot of different ways to make sure that everybody gets a little bit of everything, sure. 00:50:22.960 --> 00:50:23.960 Yeah. 00:50:23.960 --> 00:50:30.960 So, yeah, overall, I find that if you like to code, right, I mean, we can talk about the ops side, right? 00:50:30.960 --> 00:50:37.960 If you like to code, then you're going to be coding more and, you know, fixing bugs a lot less. 00:50:37.960 --> 00:50:38.960 Yeah. 00:50:38.960 --> 00:50:44.960 You're going to find that you're going to be working on small code bases and that leads to less mistakes and errors. 00:50:44.960 --> 00:50:45.960 Yeah, that sounds great. 00:50:45.960 --> 00:50:49.960 And I have to write unit tests that a lot of people don't because it's too complicated. 00:50:49.960 --> 00:50:52.960 And now you're back to a simple application that's very easy to unit test. 00:50:52.960 --> 00:50:53.960 Yeah. 00:50:53.960 --> 00:50:57.960 To be more careful on the boundaries, though, because they all talk to each other, right? 00:50:57.960 --> 00:51:01.960 And then you need, right, this I would probably put an experienced person. 00:51:01.960 --> 00:51:12.960 You need someone that overviews what are the interfaces that all the microservices expose to the rest of the, you know, to the other microservices and sometimes to clients. 00:51:12.960 --> 00:51:13.960 Right. 00:51:13.960 --> 00:51:18.960 You need to make sure that, especially with the public endpoints that are consistent. 00:51:18.960 --> 00:51:25.960 So you need one person that's experienced, at least one person that's experienced in API design to make sure that you get good APIs. 00:51:25.960 --> 00:51:26.960 Yeah, of course. 00:51:26.960 --> 00:51:27.960 That's the other thing. 00:51:27.960 --> 00:51:30.960 It's very difficult to change them once they're out there. 00:51:30.960 --> 00:51:37.960 And this is if you want to have no downtime deployments, you cannot really introduce breaking changes. 00:51:37.960 --> 00:51:38.960 So you cannot remove an API. 00:51:38.960 --> 00:51:40.960 You cannot remove a column in the database. 00:51:40.960 --> 00:51:42.960 You know, there are some rules that you need to follow. 00:51:42.960 --> 00:51:43.960 Right. 00:51:43.960 --> 00:51:55.960 So when you design databases and when you design APIs, you need to have people review that very well, make sure that you like what you are designing, because you're going to have to be with those decisions for a long time. 00:51:55.960 --> 00:51:57.960 Yeah, that's a good point. 00:51:57.960 --> 00:52:05.960 Some of the HTTP frameworks for building services have even built in versioning into them. 00:52:05.960 --> 00:52:07.960 I'm thinking of like Hug and some of these things. 00:52:07.960 --> 00:52:11.960 But obviously, you can add it into your own apps pretty easily. 00:52:11.960 --> 00:52:15.960 Just set up a second endpoint rather than calling the same one. 00:52:15.960 --> 00:52:17.960 Is there something that you have to do that? 00:52:17.960 --> 00:52:18.960 You have to do that. 00:52:18.960 --> 00:52:19.960 Basically, you're forced. 00:52:19.960 --> 00:52:24.960 So imagine you have five instances running of this one microservice. 00:52:24.960 --> 00:52:28.960 And now you want to introduce a breaking change in one endpoint. 00:52:28.960 --> 00:52:31.960 And of course, you don't want to go down for the upgrade. 00:52:31.960 --> 00:52:34.960 So you cannot stop the five instances at the same time. 00:52:34.960 --> 00:52:36.960 You're going to have to do a rolling update upgrade. 00:52:36.960 --> 00:52:37.960 Right. 00:52:37.960 --> 00:52:43.960 So, you know, during a window of time, you're going to have a bunch of instances on the old API and a bunch on the new. 00:52:43.960 --> 00:52:46.960 And then the rest of the system knows nothing of this. 00:52:46.960 --> 00:52:50.960 And they're going to start sending requests, probably assuming that the old API is in place. 00:52:50.960 --> 00:52:53.960 Yeah, until you upgrade that part, which then I'll go to the new one. 00:52:53.960 --> 00:52:57.960 But it'll be sort of moving around through these services as that process happens. 00:52:57.960 --> 00:52:57.960 Right. 00:52:57.960 --> 00:52:58.960 Right. 00:52:58.960 --> 00:53:00.960 So you need to create a new endpoint for the breaking change. 00:53:00.960 --> 00:53:02.960 Keep the old one working. 00:53:02.960 --> 00:53:07.960 And then once you're sure that the whole system is upgraded, it's on the new one. 00:53:07.960 --> 00:53:13.960 Only then you can go ahead and basically deprecate or remove the endpoint that you don't want to use anymore. 00:53:13.960 --> 00:53:14.960 Right. 00:53:14.960 --> 00:53:18.960 That's probably important as well to like eliminate the cruft building up as well. 00:53:18.960 --> 00:53:19.960 Yeah. 00:53:19.960 --> 00:53:19.960 Yeah. 00:53:19.960 --> 00:53:30.960 I think that, you know, this platform encourages you to be clean and to keep things clean, to think about these important decisions, you know, very carefully. 00:53:30.960 --> 00:53:30.960 Yeah. 00:53:30.960 --> 00:53:31.960 Excellent. 00:53:31.960 --> 00:53:37.960 So we talked about how AWS Lambda has this sort of built in latency. 00:53:37.960 --> 00:53:42.960 And when you think about performance, a lot of times there's really two sides to it. Right. 00:53:42.960 --> 00:53:47.960 One is if I go and I hit that endpoint, how quick does it get back to me? 00:53:47.960 --> 00:53:52.960 And then the other one is if a million people hit it, how much does it degrade from that one person hit it? Right. 00:53:52.960 --> 00:53:58.960 Like how scalable is it and how like single request high performance is it? 00:53:58.960 --> 00:54:07.960 I can certainly see how this generates better scalability. Like you can very carefully tune like the individual parts of your app and scale those up. 00:54:07.960 --> 00:54:20.960 But it also seems like it might add some latency like for an individual request. So how much slower, like what's the typical changes? Like would I add 10 milliseconds to a request? Would I add a hundred milliseconds? What? 00:54:20.960 --> 00:54:24.960 Yeah, that's a good question. So even if we take serverless out of the equation, so. 00:54:24.960 --> 00:54:25.960 Yeah, because it's really bad. Yeah. 00:54:25.960 --> 00:54:39.960 Right. So the performance is not the same as in a monolith, just by the mere fact that in many cases, the client is going to send the request and the service that receives that request cannot carry out the request alone. Right. 00:54:39.960 --> 00:54:48.960 So you have to talk to a bunch of other microservices. Right. So there are a lot of communications among the microservices that of course will take time as well. Right. 00:54:48.960 --> 00:54:57.960 So, so, so, so latencies increase no matter what. Right. So microservices is not, if you're looking for performance, it's not really. 00:54:57.960 --> 00:55:06.960 It doesn't necessarily have to be microservices versus monolith. It could be more coarse grained microservices, more fine grained ones potentially. Right. 00:55:06.960 --> 00:55:14.960 Right. The times that I've seen, they're not really, you know, that terrible. I mean, we are talking like tens of milliseconds. 00:55:14.960 --> 00:55:21.960 Yeah. You know, you're making requests over the internet, maybe across the country, it might be a hundred millisecond ping time. So if it's a hundred, 110, who cares? 00:55:21.960 --> 00:55:26.960 It could be in the noise. Right. Yeah. Absolutely. In many cases, it's going to be in the noise. Yeah. 00:55:26.960 --> 00:55:41.960 So, you know, compared to the, all the benefits that we already discussed, I think in my view, it's a no brainer. It makes a lot of sense in many cases, but yeah, it's, you know, it's all this complication of, you know, services talking to each other. 00:55:41.960 --> 00:55:52.960 And you might find that in some cases you need to go async. So you can totally have an asynchronous microservice. So you tell it, you need something and the service says, yeah, okay, I'll do it on my time. 00:55:52.960 --> 00:55:53.960 Right. 00:55:53.960 --> 00:55:56.960 But keep going. Don't mind me. Right. That's totally fine. 00:55:56.960 --> 00:56:01.960 Yeah. For example, send this email. You don't have to wait for the email to be acknowledged to be sent. Right. 00:56:01.960 --> 00:56:09.960 That's a great example. Right. Yeah. Yeah. It can definitely, email can be pretty slow actually, given all the stuff that it does. 00:56:09.960 --> 00:56:11.960 Email, yeah, you count it in seconds. 00:56:11.960 --> 00:56:12.960 Yes, exactly. 00:56:12.960 --> 00:56:13.960 Microseconds. 00:56:13.960 --> 00:56:22.960 You got to mail a group, right? Like, I want to send this mail to 2000 people in my class on my platform. Like, okay, that needs to be a thing. Just let me tell you, I've learned that. 00:56:22.960 --> 00:56:30.960 Yeah. And that's actually, that's a good example in which you may, in many cases, you're not interested if the email bounces. 00:56:30.960 --> 00:56:49.960 Yeah. What are you going to do about it anyway? Right. Right. Exactly. If it bounces, there's nothing to do. If you are, you can have that service that's asynchronous record the addresses that are bouncing in a database that's going to be owned by that service. And then later on, you know, in a cron job or whatever, you can clean up the addresses that are bad. 00:56:49.960 --> 00:57:04.960 So some other service can send a request to the email service and ask, you know, what are the bad addresses that you know? And that that will be another endpoint. You'll return the addresses and then they can be cleaned up or, you know, the accounts can be canceled or whatever. 00:57:04.960 --> 00:57:06.960 Yeah. It sounds, it sounds really promising. 00:57:06.960 --> 00:57:18.960 It's a great way to think about problems, right? It's, it's all, you know, little pieces. So it's a lot easier to think about solutions there. And, and, you know, at the beginning, it's hard to start thinking this way. 00:57:18.960 --> 00:57:22.960 But, but then you get used to it and all the problems become easier. 00:57:22.960 --> 00:57:36.960 Yeah. I can definitely see how that, that would happen. It might be difficult to think, how am I going to build this huge app? But if I can build, well, could you build 10 small apps and then have them help each other out? Right? Right. Exactly. Yeah. Very cool. 00:57:36.960 --> 00:57:48.920 All right, Miguel. I think we're going to leave it there. That's like, I think a great, great conversation on microservices. So let me ask you the two questions. So if you're going to work on your microservices and Python, what, 00:57:48.920 --> 00:58:18.880 editor do you use? So it's getting complicated. It's complicated. Yeah. That's the correct answer for me. I usually iterate over a few editors. So Vim is my go-to editor. Many times I need to edit files on remote hosts. So Vim works anywhere. So that, that, that's, that's the one that I use most of the time. Sometimes I need to debug. And for that, sometimes I use, you know, an IDE. And the two that I've been using, I can't even 00:58:18.880 --> 00:58:26.740 decide on one. I, the two that I'm using is PyCharm and lately Visual Studio Code, which, which is surprisingly good. 00:58:26.740 --> 00:58:38.420 Yeah. The, the Python plugin there is, is doing really quite a bit. Python plugin is, it's not an official plugin, but yeah, this person that I wrote it, he, he did an awesome job. It's a, it's very, very good. 00:58:38.540 --> 00:58:46.020 Yeah. He did a great job. I actually had him on an episode, maybe 20, 20 shows back or something. It's, it's very cool how we took like 10 different 00:58:46.020 --> 00:58:52.200 open source projects and like brought them all together and turned it into the plugin for Visual Studio Code. It was cool. 00:58:52.360 --> 00:59:05.560 Well, he did a great job. It's super powerful. And in particular, I like the way that you, you, you set your configuration for a project, which basically it opens up a text file and you write JSON. 00:59:05.560 --> 00:59:08.080 Yeah. It is quite interesting for sure. 00:59:08.080 --> 00:59:15.020 That's a, you know, contrasting to PyCharm where you have to enter a dialog, you know, a window and find the setting that you want. 00:59:15.220 --> 00:59:19.080 Yeah. Well, it also makes it very source friendly, like a source control friendly. 00:59:19.080 --> 00:59:26.940 Source control friendly. And, you can copy configs from one project to the next. It all becomes much, much easier. 00:59:26.940 --> 00:59:30.440 Yeah. Great. All right. And PyPI package. 00:59:30.440 --> 00:59:39.980 So one package that I'm, I'm sort of ashamed. I didn't know. And I learned about it from my colleagues at Rackspace. It's called Arrow. 00:59:39.980 --> 00:59:52.020 And so, so this is a package that it's a, drop-in replacement for, for the daytime package in the Python library, but it implements all the holes in support that, the daytime has. 00:59:52.020 --> 00:59:53.020 For example. 00:59:53.020 --> 00:59:55.780 Yeah, definitely. I would definitely second that one. Arrow is awesome. 00:59:55.780 --> 01:00:03.580 Yeah. I only knew it from, from a few months since I've been working with this team and I, yeah, it's, I use it all the time now. 01:00:03.700 --> 01:00:10.200 So for example, it has, you know, daytime starts with, with this naive time zone approach where there's no time zone. 01:00:10.200 --> 01:00:15.780 So by default, Arrow will, will use UTC, which is what you always want anyway. Right. 01:00:15.780 --> 01:00:17.200 So you always work with UTC. 01:00:17.200 --> 01:00:19.340 Especially if you're working on servers. 01:00:19.340 --> 01:00:24.580 Right. Yeah. You, you want to have, common units. So, so that, that's the one that everybody uses. 01:00:24.840 --> 01:00:35.900 And then, support to convert to and from the ISO 8601, which is, daytime can output ISO 8601, but cannot import from it. 01:00:35.900 --> 01:00:43.880 So if, something very common. So another thing that I tend to work on is, is the billing microservices in my team. 01:00:43.880 --> 01:00:50.360 And you, you want to know, you have a date and you want to know the first of the month and the last of the month. 01:00:50.680 --> 01:01:01.020 It's like in one line you can get it. You don't have to do strange aerobics or acrobatics to, to get the first and last of, of a given month. So yes. 01:01:01.020 --> 01:01:09.260 Yeah. People should definitely check that out. All right. So people heard this conversation. They're probably excited about microservices. How do they get started? What do they do? 01:01:09.500 --> 01:01:17.000 So what I recommend, if you have, three hours to waste, you can check out the YouTube video of my tutorial. 01:01:17.000 --> 01:01:23.320 I think I've made it very approachable. If, if your experience developing either Django, it doesn't need to be Flask. 01:01:23.320 --> 01:01:29.620 So any web, you know, monolithic web applications, I think you're going to pick up the tutorial really well. 01:01:29.620 --> 01:01:36.260 So the code that comes with the tutorial, which is on GitHub includes a, vagrant setup. 01:01:36.260 --> 01:01:45.820 So you can deploy the system that I show in the class to, to your machine, to your laptop on a vagrant VM, and then you can play with it. 01:01:45.820 --> 01:01:53.880 And even at the end of the class, I listed a list of things that, that will be great ideas to, if you want to practice. 01:01:53.880 --> 01:01:58.360 So you can take the project that I built and then extend it in many different ways. 01:01:58.360 --> 01:02:05.760 that would be my recommendation. you can look into Kubernetes, which is, something that you can also deploy to your laptop. 01:02:05.760 --> 01:02:15.560 If you want to use it for testing. I include it in, in, in the, in the GitHub repositories for this class, the scripting, required to deploy. 01:02:15.560 --> 01:02:23.200 The same application to Kubernetes, if you're into that. And then the other valid option would be to look into AWS Lambda and API gateway. 01:02:23.200 --> 01:02:24.560 Have you seen Zappo? 01:02:24.560 --> 01:02:25.900 You mean Zappa? Yes. 01:02:25.900 --> 01:02:30.820 Zappa. That's Zappa. Yeah. Which is a framework that just like uses Lambda as a backend. 01:02:30.820 --> 01:02:38.580 Yes. I I've seen it. I even wrote a clone of it, which is called Slam, which is in my GitHub, account as well. 01:02:38.580 --> 01:02:45.120 But yes, the idea is that you take a Whiskey application, which is really a function. If you think about it, Whiskey, 01:02:45.300 --> 01:02:50.200 you know, it's a callable, not only a function, but a callable. The API is quite simple. You can think 01:02:50.200 --> 01:02:55.600 about it as a, as a function, right? And, and then Lambda requires a function. So there's a really 01:02:55.600 --> 01:03:01.240 match. The only problem is that the way Lambda expects the function is not the, you know, in the 01:03:01.240 --> 01:03:07.580 way Whiskey applications are formatted. So then Zappa comes in or my slam also. And basically the, 01:03:07.580 --> 01:03:12.700 it's an adapter that sits in between Lambda and your application and makes the conversion between the 01:03:12.700 --> 01:03:18.140 two formats. I see. All right. Well, that's really cool. That that's a really easy way to deploy your, 01:03:18.140 --> 01:03:22.120 your, any Python web application that that's, you know, Whiskey. So Django, 01:03:22.120 --> 01:03:29.280 Plask, Pyramid, Bottle, all those, you can get a deploy to AWS. Yeah. All right. Well, very, 01:03:29.280 --> 01:03:33.800 very cool. So I definitely recommend people check out your tutorial, which I'll put in the show notes. 01:03:33.800 --> 01:03:39.500 And I'm also going to put Kelsey Hightower's talk. Yeah. Those go well together. I think 01:03:39.500 --> 01:03:44.600 that's, that's actually a good thing to watch first. If you like that, then you can learn how 01:03:44.600 --> 01:03:49.660 those things work by watching the tutorial. Yeah. Kelsey's is high level and flashy and 01:03:49.660 --> 01:03:55.100 interesting. And then yours is the detail. Yeah. Right. I wish I could be as good a speaker as he is. 01:03:55.100 --> 01:03:59.900 Yeah. That was really great. Yeah. All right. Well, Miguel, thank you so much for being on, 01:04:00.100 --> 01:04:04.120 on the show once again. And it's great to chat with you. Thank you for inviting me. You bet. Bye. 01:04:04.120 --> 01:04:12.020 This has been another episode of talk Python to me. Our guest was Miguel Grinberg. And this episode 01:04:12.020 --> 01:04:17.840 has been brought to you by Datadog and Talk Python training. Datadog gives you visibility into the 01:04:17.840 --> 01:04:23.560 whole system running your code. Visit talkpython.fm/Datadog and see what you've been missing. 01:04:23.560 --> 01:04:28.100 Don't even throw in a free t-shirt for doing the tutorial. Are you or a colleague trying to learn 01:04:28.100 --> 01:04:33.080 Python? Have you tried books and videos that just left you bored by covering topics point by point? 01:04:33.080 --> 01:04:39.080 Well, check out my online course, Python Jumpstart by building 10 apps at talkpython.fm/course 01:04:39.080 --> 01:04:44.300 to experience a more engaging way to learn Python. And if you're looking for something a little more 01:04:44.300 --> 01:04:50.640 advanced, try my WritePythonic code course at talkpython.fm/Pythonic. Be sure to subscribe 01:04:50.640 --> 01:04:54.680 to the show. Open your favorite podcatcher and search for Python. We should be right at the top. 01:04:54.680 --> 01:05:01.500 You can also find the iTunes feed at /itunes, Google Play feed at /play and direct RSS feed 01:05:01.500 --> 01:05:08.040 at /rss on talkpython.fm. Our theme music is Developers, Developers, Developers by Corey Smith, 01:05:08.040 --> 01:05:12.960 who goes by Smix. Corey just recently started selling his tracks on iTunes. So I recommend you 01:05:12.960 --> 01:05:19.300 check it out at talkpython.fm/music. You can browse his tracks he has for sale on iTunes and listen 01:05:19.300 --> 01:05:24.240 to the full length version of the theme song. This is your host, Michael Kennedy. Thanks so much for 01:05:24.240 --> 01:05:27.840 listening. I really appreciate it. Smix, let's get out of here. 01:05:36.120 --> 01:05:41.060 Thank you. 01:05:41.060 --> 01:05:41.940 Thank you. 01:05:41.940 --> 01:05:44.940 Thank you. 01:05:44.940 --> 01:05:45.000 Thank you. 01:05:45.000 --> 01:05:49.760 Developers, developers, developers, developers, developers, developers, developers. 01:05:49.760 --> 01:05:50.020 . 01:05:50.020 --> 01:05:50.520 you