#118: Serverless software Transcript

00:00 Michael Kennedy: How do we host and run server software? Let's consider the progression we've been on over the past 15 years or so. We've gone from software and operating systems that we manage, running on hardware that we own, and babysit, to virtual machine, to infrastructure as a service, on the cloud, or even platform as a service in the cloud. And then, on from there, we moved to containers, usually Docker, maybe running on someone else's cloud. After that, maybe we want to put these into microservices, which are conglomerates of these containers, working together, managed by something like Kubernetes. Well, where do we go from there? I can't tell you what the final destination of this whole progression is going to be. But I believe we have reached a leaf node in this hierarchy with our topic today. On this Episode 118 of Talk Python To Me, with Ryan Scott Brown, we're going to explore serverless computing. It's an interesting paradigm shift, and I hope you enjoy the conversation. It was recorded on May 24th, 2017. Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter, where I'm @mkennedy, keep up with the show and listen to past episodes at talkpython.fm, and follow the show on Twitter, via @talkpython. This episode's brought to you by Rollbar, and us, at Talk Python Training. Be sure to check out what we're offering during our segments, it really helps support the show. Ryan, welcome to Talk Python.

01:52 Ryan Scott Brown: Hey Michael, glad to be on.

01:53 Michael Kennedy: Yeah, it's going to be super fun. You know, we're here to talk about running code out on the internet, with no servers, which seems kind of impossible, and awesome, all at the same time, so that's going to be exciting.

02:04 Ryan Scott Brown: Yeah, I mean there are still servers. We'll get around to that.

02:07 Michael Kennedy: Of course, of course, of course. Awesome, so before we do, though, let's hear your story. How'd you get into programming in Python?

02:13 Ryan Scott Brown: So I had kind of a pretty vanilla path. I actually started as a, with my interest in networking. So I was majoring in systems administration and network admin, and actually, got brought into the programming world by a Perl class that I took. And then, in a, there was an open source club at my college, that everyone was using Python to write programs for the One Laptop per Child project. And that was my introduction to Python 2.

02:41 Michael Kennedy: Oh, that's really cool, what school was that?

02:43 Ryan Scott Brown: Rochester Institute of Technology.

02:45 Michael Kennedy: Oh yeah, okay, that's a great school. I know a couple of people that have gone there, and it sounds like a pretty neat program.

02:50 Ryan Scott Brown: Yeah, they have, actually, an Open Source Software minor, now. They didn't when I went, but, the program's kind of slowly evolved and grown over time.

02:58 Michael Kennedy: Oh my gosh, I have never heard of a, an actual cert, like, official university program that is, like, focused on Open Source. Tell us just a little bit about that. Do you know anything about it, like, what does a person who takes that degree do, or, that minor do?

03:13 Ryan Scott Brown: It's not a full degree, so usually, you'll be in software engineering, or in computer science. And my case, I actually TA'd for some of the classes that then were added to the minor, so we had a bunch of pilot courses first, and then you get a certification from the university that you're a real minor. And then, you can have real students enrolled.

03:32 Michael Kennedy: Awesome.

03:32 Ryan Scott Brown: And so the, all of the courses focused around either Open Source software, or the idea of open communities in different ways. Like, we had some Wikipedia editors that would come in and talk, and be TAs. We had contributors to the BSD project, one of whom lives in Rochester, and would teach one of our classes on Open Source software. And for course credit, you could do things like contribute to existing Open Source projects.

03:57 Michael Kennedy: All right, so that sounds like a really cool program that you were in, in the university. Now, what do you do now?

04:02 Ryan Scott Brown: Yeah, so now I actually work for Red Hat, who've obviously makes all of their money on Open Source software, and support and consulting around that. And the team that I work on is the Ansible team. So we make a engine, and a kind of language that people write in, so that they can orchestrate all of their servers, and cloud services, and network devices, and all that from a simple YAML language, so that's not a full fledged programming language. So you don't need the kind of experience you'd need to write things like Chef recipes, which are Ruby. And so you'd have to know Ruby to write Chef recipes. So we try and make that as accessible as possible.

04:40 Michael Kennedy: Yeah, I've heard really good things about Ansible, and that sounds fun. Ansible is a thing that recently came to Red Hat, right? Like, recently acquired that project?

04:49 Ryan Scott Brown: Yep, about a year ago.

04:51 Michael Kennedy: Well,

04:51 Ryan Scott Brown: Okay. Little more than a year ago now, Red Hat acquired AnsibleWorks, which was the company that was behind Ansible, the Open Source project. And so now Red Hat sells Ansible Tower, which is a web interface that previously, AnsibleWorks sold on top of Ansible. And the Ansible Core is still Open Source, and just as kind of community focused as ever.

05:12 Michael Kennedy: Nice, which side did you come from, or were you working on Ansible before, or were you at Red Hat, and then it came there?

05:16 Ryan Scott Brown: I was using Ansible, but I was working on OpenStack, which is a series of Open Source projects, also all in Python.

05:24 Michael Kennedy: Yeah, that's quite a big Python project, actually.

05:26 Ryan Scott Brown: Yeah, it's actually, I think, 50 big Python projects. Yeah, exactly. But yeah.

05:32 Michael Kennedy: It's an understatement to call it large, right?

05:34 Ryan Scott Brown: Huge contributors, both to, obviously the Python community, in terms of bringing things back, like PyPI, and other things that become things in the regular Python community, as well as to the core language. There are a lot or core Python contributors that work on OpenStack, and so split their time between making Python better, and then making things on Python better.

05:55 Michael Kennedy: Yep, that's definitely a symbiotic relationship, so that's great. So let's talk about this idea of serverless programming. I feel like we started out with just having, like, machines in data centers, maybe colocating some of our machines in a data center, to virtual machines, to things like Docker. And you know, things getting sort of smaller and smaller. And now, we're reached, I'm not sure you can get much smaller than single functions, almost, running on the internet, right?

06:25 Ryan Scott Brown: Yeah, I mean that's sort of the direction that we're moving, and it's been a direction in programming for a very long time. From when you had a system that just ran one job at a time, like on punch cards, so you'd toss in a stack of punch cards, and it would churn through them, and then churn out results, also on punch cards. You didn't have multitasking at all, so one program was exactly what was running. So you were, you had the whole CPU to yourself. And then you have multitasking operating systems, so you had multiple people providing code that would run in different processes, that would be isolated in a certain way. And the operating system would handle scheduling those. And you can sort of think of function as a service, which is things like AWS Lambda, and mostly called serverless, quote-unquote. You can think of those as like a multi-user scheduling, but where you have untrusted entities sharing compute resources, so you're getting smaller and smaller slices.

07:22 Michael Kennedy: Right, also, that's really interesting to track that trend over time, right? From just sort of OSes to running these functions on other people's computers. So obviously, there really are servers out there, like you said, but when it's not your responsibility, it's not your problem to like babysit these servers. So for example, yesterday I think, I logged in to some of my servers with SSH, and I had to update, like, the unattended install software package on Linux. So even though, of course, your code runs on servers, when you don't have to think about it, or manage it, or balance it, it really makes a difference, right?

08:02 Ryan Scott Brown: Yeah, exactly, it's serverless in the same way that wireless doesn't have any wires involved. There are still wires, they just aren't directly to your device.

08:10 Michael Kennedy: Right, well, you know, in wireless, you never care about the wires. Somebody has to care about the wires, but it's opaque to you, right? And so as far as you're concerned, wireless is wireless.

08:21 Ryan Scott Brown: Same analogy here, as far as you're concerned, you get (a) particular promise from your provider. Like, we will always have such and such version of such and such underlying dependencies, like libc and things like that that you would statically link to. And such and such version of a Python interpreter available, and such and such version of whatever else you need. And you would write your code such that you can rely on that such and such version being there. But you don't install that, you don't deal with the sort of provisioning that you would do to install, let's say, you need a specific Python 3. So if you only can run on 3.5 or up, you don't have to worry about making sure that's there. You just tell your provider, "Give my this runtime, please."

09:05 Michael Kennedy: Right, and it just works. You don't have to worry about it, that's great. So can you compare this to like, say, working with, like, event driven programming, versus like hooking up to cloud events, say you know, something like Erlang?

09:17 Ryan Scott Brown: Yeah, and this is kind of the other side of serverless, the one side is, run this code in the cloud for me, and the other side is, hook my code up to these other things. So, real world or virtual world, I guess, events that you want your code to act upon. So let's say you're, in your podcast world, you have something that you want to happen every time a sponsor contacts you. You could hook up a way that when the sponsor fills in the contact form, it invokes a lambda function, that then maybe calls you if it's super important to get back to them immediately, or, any number of other things. But the idea is, the event source half, and then the run my code somewhere that I don't have to manage half is what kind of makes up serverless.

10:02 Michael Kennedy: Right, right, it's easy to focus on here's a function, run it, but really, the diverse triggers that trigger those functions are super important.

10:11 Ryan Scott Brown: Yeah, exactly, and it maps to a lot of event driven concepts that we might be familiar with in Erlang, where you have processes that are managed by a supervisor. So in serverless, that would be individual functions that are managed by your cloud provider, which, under the hood, deals with scheduling those across diverse hardware, and all that stuff. And then in Erlang, you also have an event routing system where every process has input and output kind of event hooks, and you can get that similar thing in Lambda, with using APIs to send data out. So you might send data out over, like a MailChimp or a Mandrill, or you might send data out into a database. That would be your kind of output. And then your input can be triggers from things like DynamoDB, it can be HTTP events. Any number of things, I'm not going to list them all, because this would be a long podcast.

11:02 Michael Kennedy: Yeah yeah, absolutely, and you know, to bring it back to like, what could I do, you know, you're saying something happens. Like maybe I've, I'm going to up load my MP3 file that I want to release for the week to S3, and that needs to be, you know, maybe have like the artwork and the little, like the description embedded into the MP3 header. And then it needs to be moved over to the content delivery, I got to, like, flip it to publish, all these different things. Like, I could possibly set that up serverless, you think?

11:32 Ryan Scott Brown: Yeah, yeah, and individually, you would have something like the S3 event source. So you would tell a Lambda function that, when a new object appears in S3, and then S3 would notify your function when that happened. It could download from S3 your file, it would put in the, I forget what the name of the MP3 metadata format is.

11:51 Michael Kennedy: Yeah.

11:52 Ryan Scott Brown: It would dump in whatever author metadata, show notes, I don't know if that goes in there.

11:56 Michael Kennedy: Yeah, stuff like that.

11:57 Ryan Scott Brown: And then it could save it to another S3 bucket that would either kick off yet another Lambda, and you'd have this chain, or, that would be the S3 bucket you're serving public content from, to your CDN. And it doesn't have to be linear, either. You can have kind of forking things, so you could have this Lambda see, "Oh, a new show's up, here, I'll put the metadata in and I'll also kick off the job that will add it to the list on the front page. and I will kick off the job that will add it to the RSS feed." And those can be a bunch of Lambda functions that each have a small focused purpose.

12:31 Michael Kennedy: It sounds really amazing to think of how that kind of opens it up. And to me, it feels like almost like the event sources, and the workflow bit, is more important than, "Hey, run this function."

12:42 Ryan Scott Brown: Yeah, because I mean when you think about it, we've been really good at running small bits of code for a very long time. And a lot of the value comes in when you can hook those small bits of code up to something that's a useful thing in the real world, right. Because I could make a little program that simulates a vending machine, that will, you know, take pretend quarters and things, and I can type in on the terminal. But until I'm hooking that up to hardware that's actually giving someone a soda, or whatever, it's much less useful, right?

13:10 Michael Kennedy: Right, absolutely, absolutely, So, you were talking about S3, and I'm assuming that that probably ties right back into AWS Lambda, which is probably the most popular one of these. But it's not the only one, like, I think there's Azure Functions, for example. What are some of the implementations, and places, where we might find this type of programming model? There's a load of different providers. There's, obviously, the big three cloud providers all have their offering, Google Cloud Platform has what they call Cloud Functions. Microsoft Azure has what they call Azure Functions. AWS has Lambda, which is a kind of a pun on function. Right, nothing to do with Python lambdas, right?

13:53 Ryan Scott Brown: No, and I, yeah, that's, been a problem sometimes. Because you'll pull something into Python to automate, so if you want to automate the deployment of a Lambda function, and you pull that into Python, and then you name a variable "lambda,"

14:05 Michael Kennedy: Ah, yeah.

14:06 Ryan Scott Brown: You can have problems if you're not careful, so.

14:09 Michael Kennedy: Why is this keyword gone? I don't know if that actually happens.

14:11 Ryan Scott Brown: Oh, there we go, it does. You can override keywords in Python. So you can override true, false, lambda, all kinds of things that you shouldn't.

14:19 Michael Kennedy: Yeah, these are not good. And recently, I can't speak to the other cloud providers, I just don't track them that closely, but, AWS Lambda recently switched to Python, well not switched, made available Python 3.6, which is pretty cool.

14:32 Ryan Scott Brown: Yes, that is absolutely right, and it has been great.

14:35 Michael Kennedy: Yes, because surprisingly, until like a month ago, this was a Python 2 only option, right?

14:40 Ryan Scott Brown: Yeah, exactly, and I mean, if you think about when it came out, Lambda was out I believe in, towards the end of 2014, when, you know, Python 3 has been out for,

14:50 Michael Kennedy: Six years at that point?

14:52 Ryan Scott Brown: Yeah, that sounds about right. And I would love to be able to go run a second experiment, and really sit with Python 3 only, and see what that does for adoption.

15:00 Michael Kennedy: Yeah, yeah, yeah, that would have been awesome.

15:02 Ryan Scott Brown: These would just be really interesting experiments to run, but for these other providers, there are some smaller providers that only do JavaScript. So Auth0 is an example of one that has a function as a service, but it's JavaScript, Node JS only. If you look at IBM's OpenWhisk, you can actually provide an arbitrary Docker container as the runtime for your function. And so it manages the scheduling, and it will invoke as many as needed to handle events. So it's still got the function as a service going for it. But, you get a lot more control over what runtimes you give it. And so you can not only do Python 3, but you can do entire Ruby installs, you can do Apple Swift, any language that you can fit in a Docker container, which is any language.

15:45 Michael Kennedy: Right, you could probably do something like Fortran, if you really wanted to torture yourself, right? Like, you could do basically, if it can accept a request, you can do it, right?

15:52 Ryan Scott Brown: Yeah, I mean, as long as it accepts, I believe, the rules for OpenWhisk is, it has to accept data on standard in, and provide a result on standard out, when the container is invoked as a process. And so as long as you can do those things, you're good to go, so COBOL in the cloud.

16:07 Michael Kennedy: How interesting, yay. Let's get more COBOL in the cloud. So one of the things that immediately comes to mind when I think of this, in the Docker variant, obviously, would be one way to basically get an escape hatch here, is like, how much do you have control over dependencies and requirements? So for example, what if I want to use, like, NumPy, or Requests, or something that doesn't ship with Python?

16:36 Ryan Scott Brown: Yeah, sure.

16:37 Michael Kennedy: But I want to use that, like, can I go to AWS Lambda and go, oh yeah, and like, here's my requirements.txt, make that happen?

16:43 Ryan Scott Brown: Unfortunately, you can't just upload a requirements.txt, as much as I would love to. You can with Google Cloud Functions. You can give them your code, and a requirements.txt, and they'll do it for you.

16:54 Michael Kennedy: That's pretty cool.

16:55 Ryan Scott Brown: In Lambda, you provide a zip file, which contains, can contain any dependencies you want, whether that's C object libraries, if you're using something that links directly to C, like NumPY. Or, it can just be Python files, like Requests. So you would use pip on either your local machine, or a build server, that would build in all of your Requests, and pack them into a ZIP file. And I've had to do some back flips to get SciPy and scikit-learn working in Lambda.

17:23 Michael Kennedy: Wow, but you've done it?

17:24 Ryan Scott Brown: No, it took, it absolutely works, and the nice thing is that AWS now provides a Docker image that duplicates the Lambda environment. So you can download that Docker image, build stuff in there, so it's all going to be built just right, and then, dump it to a zip file on your machine, and then upload it to Lambda. So you can have exactly the same build environment locally for Lambda, and then, you build all the scikit-learn, and optimize it for that hardware, or for that environment. And then you can strip out everything that isn't needed. So you can do things like compact the .o files, you can strip out a bunch of .txt, and .md, and .rst documentation files to get your size down, because that relates to how fast your function can run.

18:06 Michael Kennedy: Yeah, because it's got to somehow take that thing apart, and work with it, right?

18:11 Ryan Scott Brown: I mean, it's got to unzip it, but it's also got to download it to whatever machine's running your code, because they don't all have it, right.

18:15 Michael Kennedy: Right, absolutely. Does it get, like, warmed up as it runs? Like, how many machines might be involved in executing your code, you know, does it, if it's running a while?

18:27 Ryan Scott Brown: Yeah, it depends on your number of invocations. So the way that it works under the hood is that they spin up, basically, a container that will handle events, and each container will handle one event at a time. So let's say that you have five events per second, and they all take half a second.

18:42 Michael Kennedy: Okay.

18:43 Ryan Scott Brown: Then on average, you're going to have about three hosts running, or three containers handling requests, because if they each take half a second, you've got two that are fully utilized, and one that's half utilized, if that make sense.

18:53 Michael Kennedy: Yeah, yeah, that makes sense.

18:54 Ryan Scott Brown: And so, if you have a bunch of events that come in in a very spiky kind of way, so you have 500 requests this second, and then nothing for 10 minutes, it's going to spin up on a bunch of machines to handle these events, and then they'll all spin down, basically. And eventually, they get evicted if they're not used.

19:11 Michael Kennedy: What's the payment model for this kind of stuff? How much difference does it make from, say, I'll just fire up a VM and just run my stuff there, relatively?

19:19 Ryan Scott Brown: Yeah, that depends a lot on what your utilization level can be. So if you get, let's say you have a VM, and you get one request a day. You're paying for 24 hours of computer, to handle 50 milliseconds, or 100 milliseconds, or something.

19:32 Michael Kennedy: Yeah.

19:34 Ryan Scott Brown: And whereas in Lambda, you pay by the unit, I believe is the megabyte-second of memory, and that also scales your CPU. So they have tiers of anywhere from 128 megabytes of RAM, up to, I believe, a gig and a half. And each of those is billed per hundred millisecond slice, by the tier.

19:56 Michael Kennedy: Yeah, let's suppose I've got a function that doesn't really use much memory, right, it's pretty basic, and it's run five times a day, for 10 milliseconds. Is this, like, a penny, a dollar, five dollars?

20:09 Ryan Scott Brown: Oh, that's in the hundredths of pennies. The way, I had a blog post that I calculated out the cost to, quote-unquote, more sensible unit, the picodollar per byte-second.

20:20 Michael Kennedy: Okay.

20:22 Ryan Scott Brown: And if you run, I believe it's, you get a million invocations for something like 14, or a million milliseconds for something like 1.4 cents.

20:35 Michael Kennedy: Wow.

20:36 Ryan Scott Brown: And so there's a lot of, I mean you get, it's more expensive if you were to run a 24 hour EC2 instance, and then run a 24 hour Lambda function, which you can't, because there's a timeout. But if you ran them one after another for a total of 24 hours,

20:49 Michael Kennedy: Like if it was under a super heavy load that was basically equivalent to a continuous, right?

20:53 Ryan Scott Brown: Yeah, if you were to run 24 Lambda hours, if you will, so if you run, you know, 24 of them all for one hour solid, then that would be the equivalent of a little more expensive than the equivalent EC2. But, that would be assuming that you can fully utilize your EC2 instance, which is not all that common for most workloads, because you've got kind of extra capacity to handle random spikes and variations in what users are doing. So if you're going to get a little bit more traffic, you don't want to spin up a brand new server every time you get just slightly over the threshold.

21:26 Michael Kennedy: Right, yeah yeah, that makes a lot of sense.

21:27 Ryan Scott Brown: Whereas Lambda can match very precisely, you know, one to one with the events that you've got.

21:35 Michael Kennedy: Hey everyone, Michael here. Let me take just a moment, and thank one of our sponsors, who makes this show possible. This portion of Talk Python to Me has been brought to you by Rollbar. One of the frustrating things about being a developer is dealing with errors, ah, relying on users to report errors, digging through log files trying to debug them, or, a million alerts, just flooding your inbox and ruining your day. With Rollbar's full stack error monitoring, you get the context, insight, and control you need to find and fix bugs faster. Adding the Rollbar Python SDK is just as easy as pip install rollbar, you can start tracking production errors and deployments in eight minutes or less. Rollbar works with all the major languages and frameworks, including the Python ones, like Django, Flask, Pyramid, as well as Ruby, .NET, Node, iOS and Android. You could integrate Rollbar into your existing workflow, send error alerts to Slack or HipChat, or automatically create new Jira issues, Pivotal tracker issues, and a lot more. They have a special offer for Talk Python to Me listeners. Visit talkpython.fm/rollbar, sign up, and get the Bootstrap plan free for 90 days. That's 100,000 errors tracked for free. But you know, just between you and me, I hope you don't encounter that many errors. Give Rollbar a try today, just go to talkpython.fm/rollbar. There's all these different event sources. I understand how AWS can invoke our function. If like, say, and S3 thing changes, it's all, like, inside AWS. But you could wire up even, like, and API to this thing, right, if somebody hits this URL with a post, with this JSON body, run this function, right?

23:06 Ryan Scott Brown: Yep, just like you would put a handler in your Pyramid or Django application. You can attach a Lambda function to a, what's called in AWS, an API Gateway. In other serverless platforms, they call it, I believe, a HTTP event. But it's all the same, basically. They take, they have something that's running all the time, that's waiting for HTTP requests. And then, when someone hits it, it will invoke your lambda function, get the output, and then send it back. And you only pay when people are actually using it. So you can run HTTP API that's available, that costs you pretty much nothing, in base cost. It's just for requests.

23:44 Michael Kennedy: Yeah yeah, that's pretty awesome. And can you do things like map, like, SSL certificates, and custom domains and stuff, to those URLs, do you know?

23:52 Ryan Scott Brown: Yep, yep, you can do custom SSL certificates, you can do custom domain names. And in your function, you can figure out what domain name it was sent to, so you can do special things in your template, for example. Let's say you have a short domain and a long domain, but they both show the same site. If your Lambda function is rendering things, or rendering links, it can give the correct URL back.

24:14 Michael Kennedy: Yeah, I see.

24:15 Ryan Scott Brown: By looking in the event and seeing all the information about where it was sent, and all that stuff.

24:19 Michael Kennedy: Okay, that sounds great. And it has access to all the data sent to it, right, like the URL, the query string, the POST body, the headers, is that possible?

24:29 Ryan Scott Brown: Yeah, absolutely, but there's kind of two ways you can do it in Amazon's API Gateway. You can use what's called the Lambda proxy integration, which gives you this giant JSON event that has everything in it. So it'll have the forwarded by, the X-Forwarded-For, all, any other HTTP headers like auth, and the query string, and the path, and everything. Or, you can use subset language that AWS calls Velocity Template Language, and you can select very specific little bits. You shape the event that you get in your function, to only the stuff that you need. And this means that your function runs faster, because it's decoding less data, and there's less kind of a security attack surface area, because you're only letting through one or two little things, and then that's happening before it gets to your code at all.

25:18 Michael Kennedy: Right, you don't want to, there were those mass assignment injection attacks, for example.

25:23 Ryan Scott Brown: Yeah, or you could just have someone who sends a massive regex, or something else that's hard, particularly expensive, to decode, and make your Lambda function slower, and you know, cost you more money in the end, right?

25:36 Michael Kennedy: Right, yeah, it's interesting to think that the more we use these cloud resources on a consumption based model, how distributed denial of service has a direct monetary component.

25:48 Ryan Scott Brown: Yeah, I mean it's more like a banking denial of service.

25:51 Michael Kennedy: Yes, exactly. So if I have like my VM running at, say, Digital Ocean or something, and somebody decides to attack it and pound on it, well, it may degrade or even kill my service, but I'm still going to pay the $10 a month, or whatever I pay, you know what I mean? Whereas this, it could vary, right?

26:08 Ryan Scott Brown: It can vary, but there are limits. AWS puts in place two kinds of limits. They call them, the first ones they call safety limits, which are relatively low, just so that you don't outbill, you don't bill yourself out, right?

26:20 Michael Kennedy: Right.

26:21 Ryan Scott Brown: And then they have what are called soft limits, which are limits that they say, okay, most of our users never hit this limit. But the ones that do can just call us, and we'll raise it right up for you. And then they have hard limits of services that there's some technical limitation, where they just can't go above, you know, 40 gig ethernet, for example.

26:38 Michael Kennedy: Sure, okay, that makes sense. And of course, in AWS, pretty sure the others have this as well, but you have, like, billing alerts.

26:45 Ryan Scott Brown: Yep.

26:45 Michael Kennedy: Right, and so,

26:46 Ryan Scott Brown: Yep, you have billing alerts, you can also monitor specific things about Lambda. And that's another thing that's really nice in serverless, is the provider needs to monitor really well to bill you correctly. And so you also happen to get really good monitoring, because they need good monitoring to bill you.

27:02 Michael Kennedy: Right, and they just service that for you.

27:04 Ryan Scott Brown: So that lines up real nice.

27:05 Michael Kennedy: Oh yeah, that's cool. So maybe that's a good place to look, maybe compare and contrast with traditional web frameworks like Pyramid, Flask, Django, compared to, like, this programming model. Obviously, the way you set up the server's different. Like, you don't deal with nginx, and whatnot. But you know, sort of the paradigms, what do you think?

27:24 Ryan Scott Brown: There's a lot of things that you don't get. So for example, you can't just have some super long-running API request, because for example, Lambda has a five minute timeout maximum. But usually, you'll set that lower, so that you don't bill yourself out, or so you don't cost yourself too much money. Because if you've, you know, if your average web request doesn't terminate in five seconds, the user's gone anyways, so you want to just stop that. And you also get a lot more control over what's shared, and what's not. So in Django, or Flask, or Pyramid, you have sort of a shared state that's internal to the server, that isn't persisted out to a database, for example.

28:01 Michael Kennedy: Right, maybe some in-memory static caching stuff, you pre-compute it at start, and you just can reference that, right?

28:06 Ryan Scott Brown: Yep, so you have, you can build up pretty expensive caches locally, that in Lambda, don't make sense to do quite so much. And so what you would use for that is some other really fast storage system, like DynamoDB, or a Redis, or even Elasticsearch.

28:22 Michael Kennedy: Sure.

28:23 Ryan Scott Brown: All of these things give you a really low-latency way to just get data back, and then that would be accessed by your Lambda functions, and then you would be able to get stuff really quick. And so that's different in that you don't have the persistence. And then the other thing is that you don't have the same limitations on language boundary. And I know this is a Python podcast, so, we'll flip around this example, too. Imagine that if you're writing something in Express, which a Node JS framework, and then you want to write a function in Python, because you'll like, Python more. You're kind of SOL, unless you make a new microservice.

28:55 Michael Kennedy: Right.

28:56 Ryan Scott Brown: Whereas in Lambda, you can go as granular as you want with languages. So I can say, oh, well the user create endpoint is in Python 3.6, but our profile image generator only runs in 2.7 right now, because I haven't gotten around to it. And so you can actually make a more granular migration between languages, because you're doing one feature at a time.

29:17 Michael Kennedy: I see you, you upgrade a function at a time, and execute a function, more or less, in a, its own isolated environment, right?

29:24 Ryan Scott Brown: Yep, exactly, and then, if you compare that to a Django project or something, then you've got one Python interpreter for your whole app, so everything either has to be Python 3, or nothing can use Python 3.

29:34 Michael Kennedy: Right, it's all or nothing, yeah. Okay, that's pretty interesting.

29:38 Ryan Scott Brown: So the granularity is really cool, and then you also get the ability to make your dependencies separate. So if you have certain dependencies that you only want to run against very trusted data, you can make those only in the functions that are invokable by very trusted things. So you get a lot more security firewalling, not literal firewalls, but a lot more compartmentalization between functions, and you can even do AWS resource permission distinctions between functions. So you can do things like, say, this function's allowed to write to S3, but it's the only one, everyone else is denied.

30:11 Michael Kennedy: Right, okay, that's actually pretty awesome. Because again, in like a traditional WSGI app, you have to put those walls up yourself, and it can be tricky, because it's still the same memory in the end anyway, right?

30:24 Ryan Scott Brown: Yeah, I mean regardless of how tricky it is, it's just, it's easy to make a mistake, or to accidentally add an endpoint that probably shouldn't get write to S3, and then, so on and so forth.

30:36 Michael Kennedy: Sounds like serverless code might be a little bit more safe, by default.

30:40 Ryan Scott Brown: It's as safe as you make it. You can just say, oh, I'm just going to give all my functions full admin, and they're just going to execute arbitrary Python that comes in off the internet, and you can just eval every request.

30:52 Michael Kennedy: Here, give me that pickled object, I'll work on that, no problem.

30:55 Ryan Scott Brown: Yeah, I also take arbitrary pickled objects, and just, yeah, let's go.

30:58 Michael Kennedy: That'll be fun, let's try that.

31:00 Ryan Scott Brown: But you can do a lot, there's a couple of really good talks, one is Gone in 60 Milliseconds, as an example of how, even in a serverless context, if you overpermission your functions, attackers can still get things that they shouldn't from your Lambda functions.

31:15 Michael Kennedy: Of course, yeah yeah. Yeah, it just sounds like it might be a little easier to exercise some least privilege.

31:20 Ryan Scott Brown: Yeah, definitely.

31:21 Michael Kennedy: So I have the stuff here, okay. So talking about the dependencies, and persistence, and caching, and things like that, it sounds to me like, to really, if we're going to have kind of complicated programs that are running in this serverless architecture, you kind of need to go a little more all in on the cloud providers, like, let's just stick to AWS, because we've been talking about Lambda. But, right, this applies generally. So AWS has DynamoDB, and some kind of caching, I'm guessing Redis, I haven't played with their Redis option.

31:51 Ryan Scott Brown: Yeah, they offer Redis or Memcached, as a service.

31:54 Michael Kennedy: Yeah, exactly, so there's that, there's RDS, storage would go to S3 instead of the file system. Right, so do you feel like, to be effective with this stuff, if you kind of have to go a little more into the various APIs, whereas I could use EC2, and like, basically forget I'm on AWS?

32:11 Ryan Scott Brown: Yeah, yeah, you could. The downside of doing EC2, and you forget that you're on AWS, is that you've, you know, forgotten Amazon's, you know, zillion man-years, or zillion developer years, that have gone into creating all these higher level services that are, basically, commodities. So things like S3, oh, store this blob for me, and then let me get it later, when you're running just on EC2, and you're storing it to disk, you need a backup strategy. You need to make sure that if that server goes down, it's still available, so you have to do the replication. And S3 is just one example, but, you have all these services that make your life easier, and so, you do have a trade going on. So you can choose to use as few provider specific services as possible, but then you don't get the benefits of using those. So the example that I like to use is, there's a online training company called A Cloud Guru, that they built their prototype over about a week on Lambda, and using Firebase.

33:09 Michael Kennedy: Okay.

33:10 Ryan Scott Brown: And the downside is that they would be locked into that forever. The upside is that if they couldn't have done that, they couldn't have started, and so they wouldn't even exist. Right?

33:18 Michael Kennedy: Right.

33:19 Ryan Scott Brown: And so, every time you get up, you're making a trade between, you know, going and killing your own hand-raised, going out and growing your own food versus getting it from someone else, where you're locked into that provider, somewhat. So you're making a tradeoff between what you're able to do in a short amount of time, versus how easy it would be to switch to another provider, basically.

33:38 Michael Kennedy: Right, of course, and it doesn't necessarily mean you have to stick with serverless, right? Like, you could go and use RDS, and Redis, and S3, and then switch to EC2, and still use those, right? Like, you're just, kind of just stuck to AWS at that point, but not to Lambdas.

33:52 Ryan Scott Brown: Yeah, because a lot of the services that you'll use alongside Lambda, you would use from a traditional application. So you'll see people that write web applications, and then some of the functionality is in Lambda, because they didn't want to deal with something like a resource overrun on that particular item, or they wanted a special event source that wasn't an HTTP type event. Or, they just liked that context better, because it's a language they don't normally work with. There's all kinds of reasons that you would have kind of a hybrid.

34:20 Michael Kennedy: Right, of course. I have a lot of faith in these cloud providers, like, they very rarely go down, and you know, when they do, it's usually really, really short. But what if you wanted to have some flexibility to, say, move, like maybe, could you speak to lock-in a little bit?

34:37 Ryan Scott Brown: Everywhere you're going to go, you're going to have some lock in, whether that's just the time it would take to move your data, or, the time it would take to write your app to make changes. An example of not very lock-in lock-in would be something like file systems on Linux. Because you can switch between ext4 and ZFS, and that's, as far as your code is concerned, the same, because they both provide the same interface. Whereas when you migrate from, like, you've talked about, on previous podcasts, from Postgres to MongoDB, you've got to do change to your code to deal with the different modeling that MongoDB does of your data and queries, versus what Postgres does.

35:15 Michael Kennedy: Right, exactly, yeah. That took a couple days of work, and a few bugs I had to hunt down, right, yeah. So that was different, of course.

35:21 Ryan Scott Brown: Yeah, but you made it, so. And serverless is the same way. You're going to be embedded in whatever cloud provider you're in, because in most contexts, you kind of want to be embedded like that, because then you take advantage of their work, so that you don't have to. But if you're worried about transitioning cloud providers, there are a few things that you can do, like never handling an event directly, is you always transcode it into a sensible format. So instead of relying on the Lambda proxy event format, you have something that transcodes that into just the stuff that you need, in a format that makes sense to you. So that way, at least, you just have to re-write these shims, and then your internal code make sense.

36:00 Michael Kennedy: Sure, that makes a lot of sense.

36:01 Ryan Scott Brown: Or, your internal code still works.

36:02 Michael Kennedy: Right, right.

36:03 Ryan Scott Brown: When you rewrite the shim, and to handle the, say, the Google format of the event, or the Amazon format of the event.

36:08 Michael Kennedy: Yeah, I guess, you know, probably some proper architecture makes a lot of sense here. Like for example, your primary code could directly write to S3, or it could call some other code that says, save this file to my thing, wherever that is. And then that could be implemented to do it to S3, it could be implemented to Azure Blob storage, whatever, right?

36:30 Ryan Scott Brown: Azure actually has a really cool implementation of this that I like a lot, that I hope that more providers will copy. You make a trigger, they call it a trigger, an event source, and you can have Azure Blob Store be the trigger, or, you can have something like Dropbox be the trigger. And what it'll do is, it will pull down that file, put it in a temporary directory, where your function is running, and then invoke your function, and tell it about the local file path.

36:54 Michael Kennedy: Oh, nice.

36:54 Ryan Scott Brown: So you don't actually deal with, like, the Dropbox or Azure Blob Storage APIs to get files. It puts them in a local directory for you, and then you use your language's regular file support. So events like that would be amazing for more providers to implement. And I think they've done a great job there.

37:11 Michael Kennedy: Yeah yeah yeah, that actually sounds really quite interesting. Another thing that I guess I see as a bit of a challenge for serverless is, what if I want to work on this locally? All right, like, I want to just fire up my local CPython, and run this code, and see what happens, you know? Like, maybe I'm on a plane, and I, I want to, you know, do a little bit of work before I get to this conference, for something I'm doing, or some customer demo, or something, right? And it involves this app, like, what's the story on local, or offline, or any of these things?

37:48 Ryan Scott Brown: You've got some options. For local, there's the kind of what I would call the first degree local, which is, you can develop locally, and deploy to a dev environment that just takes your code directly. So that's kind of first degree local, but the code's actually not running on your machine at all. If you want more than that, so that you can have a faster feedback loop, because deploys do take time, not a lot of time, but time, you can use something like, Atlassian has a thing called LocalStack, that will fake out API Gateway, Kinesis, Dynamo, S3, and a bunch of other services on your local machine. So they'll be on local ports, and then, you can run code that would use AWS services, and point them at your local LocalStack.

38:26 Michael Kennedy: Nice, yeah maybe just change, like, have a different, like a dev local config that has different endpoints.

38:31 Ryan Scott Brown: It would be a config change that would point your code at those local services. So you can then develop locally, like on a plane. So once you have LocalStack downloaded, you would be able to do that. Other thing is, Plugin for the Serverless framework, which I'm a contributor to the Serverless framework, but not to this Plugin, that it runs Docker locally, and will use the container image that I talked about, and simulate different event invocations to your function. And so you can either use that with, they have a built in thing for the DynamoDB developer mode, which is a little JAR that you run that puts up a fake DynamoDB, or, you could use that in conjunction with LocalStack to get the runtime and the service simulation together.

39:15 Michael Kennedy: Yeah, I see, yeah. I wonder how much Atlassian came up with LocalStack to, like, solve their own local cloud testing problems, and just Open Sourced it, do you know?

39:25 Ryan Scott Brown: I don't actually know the people that made LocalStack, but it definitely seems like that kind of thing that someone at Atlassian size would say. We have so and so many developers, and it would save us so and so much time to not deploy every time they want to test something. So, let's take that time that we're going to save, and invest it in this tool that's going to, you know, make our developers a lot happier down the line.

39:46 Michael Kennedy: Yeah, oh, that actually made me think of continuous integration, and testing, and verification, and stuff like that, like, how would I basically verify my stuff in a CI style?

39:58 Ryan Scott Brown: Yeah, so there's some things that are hard, like, simulating actual chains of Lambdas. So let's say that you have that podcast Lambda that we used as an example earlier, where it takes an MP3 file, it does stuff, it puts it back in S3, and then the next function picks up the new file, and does something else, and so on. It's hard to simulate that chaining, but because each function is pretty focused, using a regular testing framework to test, given this input, what happens? And there are several libraries that will pretend to be AWS for you, in Python unit tests. I really like Placebo, which you can run it, and it will record your interactions with AWS, the interactions of your code, and save them all in order. And then you can re-run it in playback mode. And so, it will insert itself before your calls go to AWS, and just send back the recorded response, so you can make sure that things are called a certain number of times and that your code handles these responses from AWS the right way.

40:57 Michael Kennedy: Okay, that sounds cool.

40:58 Ryan Scott Brown: You'd use unit level testing for that, and then you'd have an integration level test, where you deploy to a staging environment, and then you have scripts that exercise kind of the full life cycle, with all of those AWS services that you can't simulate locally, or want to make sure that your local simulation isn't different from the current AWS behavior.

41:15 Michael Kennedy: Right, it sounds like mocking might be an important part, as well, for certain parts.

41:20 Ryan Scott Brown: Actually with Placebo, you don't have to do the mocking, because it injects itself in to the AWS client library, and sort of does that mocking under the hood for you, which is pretty cool.

41:29 Michael Kennedy: I see, yeah, it's involved. You just don't have to write it.

41:32 Ryan Scott Brown: Yeah, it's involved, but you don't write it. I guess that's mockless, now?

41:35 Michael Kennedy: Yeah, it's mockless, mockless unit testing. This portion of Talk Python To Me is brought to you by us. As many of you know, I have a growing set of courses to help you go from Python beginner, to novice, to Python expert, and there are many more courses in the works. So please consider Talk Python Training for you and your team's training needs. If you're just getting started, I've built a course to teach you Python the way professional developers learn, by building applications. Check out my Python Jump Start by building 10 apps at talkpython.fm/course. Are you looking to start adding services to your app? Try my brand new Consuming HTTP Services in Python. You'll learn to work with RESTful HTTP services, as well as SOAP, JSON, and XML data formats. Do you want to launch an online business? Well, Matt Makai and I have built an entrepreneur's playbook, with Python for Entrepreneurs. This 16 hour course will teach you everything you need to launch your web based business with Python. And finally, there's a couple of new course announcements coming really soon, so if you don't already have an account, be sure to create one at training.talkpython.fm to get notified, and for all of you who have bought my courses, thank you so much. It really, really helps support the show. We talked about developing locally, but there's also some other tools that just help with things like deployment, and something called Zappa, which basically, as soon as AWS Lambda switched to Python 3, Zappa's like, hey, we're Python 3! Because it's like running more or less on top of Lambda, and there's some others as well. Do you want to talk about those?

43:04 Ryan Scott Brown: Yeah, so Zappa is a project that will take your WSGI-ish app. So it'll take things like api-star, now that it supports Python 3, Flask, Django, Pyramid, and wrap them up in its own kind of fake WSGI that takes the Lambda API Gateway events, and will put them into the request object for that web framework, and then give it to your function that would work as a Django app, but is now inside of Lambda, but it doesn't know.

43:33 Michael Kennedy: Right, so you write code as if it were Django, or something.

43:36 Ryan Scott Brown: So you pretend that it's Django, and then Zappa handles packaging up each endpoint, and associating it with the API Gateway, and then uploading that code to Lambda, and hooking it up. And then you get your Django-ish endpoint, or well, it is a Django endpoint, but now, running without Django actually serving the connection.

43:55 Michael Kennedy: Interesting, yeah, because you basically, when you start these apps normally you say, here's my WSGI app, and that's, the implementation of WSGI is like, you've received a request, right? So they just have to adapt to, hey, you've received a request, it just happened to have come, not from a WSGI server, but from somewhere else, right?

44:13 Ryan Scott Brown: Yeah, that's exactly right, because you're getting the same sort of data about the request, in a different format from API Gateway. And so what Zappa does is it makes that format match the format that WSGI expects, and then gives it off to your code.

44:25 Michael Kennedy: Okay, cool.

44:26 Ryan Scott Brown: And that'll do deployment, as well. So you say zappa deploy, and it'll deploy it up for you, and it will also let you, you know, do some local testing, and things of that nature.

44:36 Michael Kennedy: Yeah, I wonder if that actually makes it more locally testable, because you could just run it as a Django app, or?

44:44 Ryan Scott Brown: I think you're still really bound to those services, so I don't know how much that really helps you.

44:49 Michael Kennedy: Yeah, I guess you're right, because it's really that you're trying to get to that to that S3 bucket, you're trying to get to that RDS instance, and so on, right?

44:55 Ryan Scott Brown: Yeah, and for example, you might have a thing that you're calling, that's only available inside of your VPC, which is Virtual Private Cloud, which is basically a private little subnet. And if you're testing locally, you're just not going to have it. But Zappa is really cool, Chalice is another one that's made by Amazon, as sort of a Labs project. I think it's still point something, so zero point something. But it looks a lot like writing Flask, but it uses those decorators that you put on to autodiscover how it should connect your code to API Gateway, and it has a thing that will try and autodiscover what IAM permissions you need, which is, in my experience, a little hit or miss.

45:33 Michael Kennedy: Okay, and then there's also something called Gordon?

45:35 Ryan Scott Brown: Yeah, Gordon is a Python frame, it's both written in Python, and you can deploy Python with it, but it's just for deployment. So you'd write your code like Gordon doesn't exist, and then you use Gordon to deploy your code. So things like Gordon, Ansible, the Serverless Framework, you use to specify all these resources around your function, but you write your function sort of independent of them. So they don't want kind of anything to do with your internal code, they just want to deploy it.

46:02 Michael Kennedy: Interesting, okay. So let's talk about where people are using this. What are some popular deployments? We already talked about A Cloud Guru, which is that online training place for cloud stuff.

46:16 Ryan Scott Brown: Yep, and then you also probably know of iRobot. They make the Roomba, and they have a new series of Roomba, I think it's the nine something. And that has an associated app, and it will map your home over time as it sort of figures out where things are. And that's actually all backed with Lambda and API Gateway.

46:32 Michael Kennedy: Ah, that's crazy, so it is like, you hook it on your wifi?

46:36 Ryan Scott Brown: Yeah, yeah, it has network.

46:38 Michael Kennedy: It just streams its location and info?

46:40 Ryan Scott Brown: I don't think it streams the location. You'd have to talk to them about that. They're also huge Python users, so they might be a good guest, but...

46:46 Michael Kennedy: Oh yeah, that sounds fun, actually.

46:48 Ryan Scott Brown: The Roomba maps kind of locations in your home, and eventually, it'll figure out, like, well, this is always here, and this chair is only sometimes here, so it must be mobile, and so it can figure out the most efficient way to vacuum your house. But you can also do things like kick it off remotely, so if you're at work and you're like, oh no, I have guests that are going to be in for dinner, and I'm at work, run the Roomba, so I don't look like a slob. And all of that's going through the iRobot API, both on the Roomba side, because they need to be able to tell the Roomba to go, and then on your app side, because your apps needs to tell them, I want this to happen. And all of that's going through API Gateway with Lambda.

47:23 Michael Kennedy: Yeah, it seems like these IOT things would be a really good fit for serverless. They just need to talk back.

47:29 Ryan Scott Brown: Yeah, because you don't know what the usage profile's going to be, and you're really cost sensitive, because people pay you however many dollars for the device, and then they expect it to just keep working.

47:39 Michael Kennedy: Yeah, that's cool. Another one's Nordstrom's, right?

47:42 Ryan Scott Brown: They've been speakers at Serverlessconf several times, and they use Lambda API Gateway and Kinesis, which is an event streaming service. So, you can put as many events as you want on Kinesis, and it'll sort of keep them in order to a certain extent, and invoke Lambdas for batches of them. And they have a project called Hello, Retail! that I'll make sure gets in the show notes, but is a really nice, I want to see how an architecture of a real thing works. So it's, for them, it's a simple retail platform that's an example of all these services, kind of together in one place.

48:15 Michael Kennedy: Mm hmm, okay, great. So, I guess, you know, all this sounds really good. There's a few cases where serverless makes things a little harder, but a lot of places where it makes it much easier. When would you say that we should, maybe, so we talked about a lot of places using it, when would you say, maybe should not use it? Like, if I'm trying to do this thing, like, this is probably not a good use case for this model, what's that?

48:39 Ryan Scott Brown: There's a few, there's things that you have a, right now, things that have a low latency sensitivity. So if you absolutely, positively need a response in x milliseconds.

48:49 Michael Kennedy: Right, so like high frequency trading might not make sense.

48:51 Ryan Scott Brown: And this is like single digit milliseconds, yeah. So you're not going to high frequency trade on Lambda. But even if you have something like an ad marketplace, Lambda's probably not the best, because usually, your responses are sitting around, I believe it's 150 milliseconds to go through API Gateway and back, and then however long your Lambda function wants to run. So if you need 50 millisecond latency, then I think you're going to have problems.

49:16 Michael Kennedy: I see, so there's kind of an assumed, like, 150 millisecond latency, just in the whole system, yeah.

49:22 Ryan Scott Brown: Yeah, just because you're invoking a new container, and it's starting, and it's loading your code into memory, and doing all this stuff. And subsequent invocations are faster, because the code's already in memory and it's just waiting on more events. But you have kind of a base time that you're going to be spending, just communicating over the network between API Gateway and Lambda. Whereas, if you're going from client directly to your Django app, there's no, like, API Gateway in between. And so, you are adding a hop, even though it's a hop inside AWS, so it's pretty quick. The other thing is, for web sockets, there's not a great Lambda web socket story yet. But you can go over to a place like Firebase, and they have awesome web socket support, and they hook into Google Cloud Functions. So it depends on your provider, too.

50:04 Michael Kennedy: I see, okay, well that sounds like good advice. Firebase, definitely worth checking out.

50:09 Ryan Scott Brown: Yeah, and they have really good Google Cloud Functions integrations, now that we're, I think, some of them were just announced at Google Next, a month or two ago.

50:17 Michael Kennedy: Yeah, excellent. So this is probably a pretty good place to wrap things up. One thing I did want to give you a chance to talk about is, you just wrote some video courses on serverless programming, right?

50:27 Ryan Scott Brown: Yeah, that's right, I have two. One from a little while ago that's just on AWS Lambda, and function as a service, and how to write code for kind of this new architecture, and runtime, and all that stuff. And the most recent one I have is using the Serverless Framework, which is one of these deployment options, one of these tools, to deploy GraphQL APIs, which is a query language, it's developed at Facebook, I think you had a show on it, unless it was a different Python podcast.

50:53 Michael Kennedy: Yeah, I think it was on Podcast.__init__, actually, but that's, yeah, so people can check that out, that sounds interesting.

50:58 Ryan Scott Brown: Yeah, so that's the Serverless Framework to deploy GraphQL API, that you would consume from a web front end, like a single page app, maybe using React, or you could use it from a mobile app. Anything that can speak GraphQL. And the idea is that you can then write your own back end with a lot less experience than you needed before, because now you're not managing like a load balancer, and autoscaling groups, and all that stuff. You just deploy your function that talks to your data store, that then gives you access to all your data from the client.

51:27 Michael Kennedy: That sounds really cool. So those are both at A Cloud Guru, and we can link to that in the show notes, so people can check that out. All right Ryan, this is really interesting stuff. Let's close it out, the two questions. If you're going to write some Python code, what editor do you open up?

51:40 Ryan Scott Brown: That would be vim, of course.

51:42 Michael Kennedy: All right, cool, and, most notable but not really super popular PyPI package, you want to draw people's attention to?

51:50 Ryan Scott Brown: I don't know if it's super popular, but I really love structlog, which is a library for structured logging. So you import it, and you put it into your standard library logging configuration, and it intercepts stuff, and it can take keyword arguments. So instead of logging out a formatted line with a bunch of %s or the curly braces, you log out the name of your event, so that can just be a short message, and then as many key-values as you want, as keyword args. And then it'll log them as either JSON, or a prettified kind of CLI thing. And so you can inject a lot more data into your logs, and then you can get a lot more out of them if you're parsing them through like the Elastic stack for log parsing, or using CloudWatch, or anything like that. So in Lambda, you can also use this, and it will get parsed as JSON by CloudWatch, which is pretty cool.

52:37 Michael Kennedy: Yeah yeah, that seems like a really cool addition, all right. Final call to action, how do people get started with this stuff?

52:42 Ryan Scott Brown: I have a blog about this, that has some Python material, as well as Node.js, so whatever people are into, at serverlesscode.com. They can just kind of go through there. I have tutorials, projects, interviews, kind of a mix. Or, hit up those video courses, if you're the kind of person that learns from video.

52:58 Michael Kennedy: All right, very cool. Thanks for sharing all this serverless stuff with us, Ryan, it was very interesting.

53:02 Ryan Scott Brown: Yeah, thanks Mike, my pleasure.

53:03 Michael Kennedy: Yep, you bet, bye. This has been another episode of Talk Python To Me. Today's guest was Ryan Scott Brown, and this episode has been sponsored by Rollbar, and Talk Python Training. Rollbar takes the pain out of errors. They give you the context and insight you need to quickly locate and fix errors, that might have gone unnoticed, until your users complain, of course. As Talk Python To Me listeners track a ridiculous number of errors for free at rollbar.com/talkpythontome. Are you, or your colleagues, trying to learn Python? Well, be sure to visit training.talkpython.fm. We now have year long course bundles, and a couple of new classes released just this week. Have a look around, I'm sure you'll find a class you'll enjoy. Be sure to subscribe to the show. Open your favorite podcatcher, and search for Python. We should be right at the top, you can also find iTunes feed at /iTunes, Google Play feed at /play, and direc t RSS feed at /rss, on talkpython.fm. Our theme music is "Developers, Developers, Developers" by Cory Smith, who goes by Smixx. Cory just recently started selling his tracks on iTunes, so I recommend you check it out at talkpython.fm/music. You can browse his tracks he has for sale on iTunes, and listen to the full length version of the theme song. This is your host, Michael Kennedy. Thanks so much for listening, I really appreciate it. Smixx, let's get out of here.