WEBVTT 00:00:00.020 --> 00:00:06.080 Your site is down. It's 3 a.m. Is it a bug, a bill, or a breach? You can't tell yet, and everyone is 00:00:06.260 --> 00:00:11.500 watching you find out. Matt Lea has spent 15 years being the person companies call when an outage is 00:00:11.720 --> 00:00:15.520 costing them real money per hour. And his sole argument is that everything you want in that 00:00:15.650 --> 00:00:20.800 moment gets decided months earlier, on ordinary afternoons, when someone chose the convenient 00:00:21.100 --> 00:00:27.000 thing. We walk through his top 12 do's and don'ts for AWS. Infrastructure as code, IAM roles instead 00:00:27.020 --> 00:00:32.240 of access keys, private subnets, no wildcards, no public buckets. And I push on which of them 00:00:32.349 --> 00:00:38.840 actually matters if you're one person on a small VPS self-hosting. Then we get to cloud war games 00:00:39.200 --> 00:00:44.920 where Matt breaks things on purpose so your team's first real incident isn't their first incident. 00:00:45.500 --> 00:00:51.440 Let's get into it. This is Talk Python To Me episode 559, recorded July 31st, 2026. 00:01:08.660 --> 00:01:13.080 Welcome to Talk Python To Me, the number one Python podcast for developers and data scientists. 00:01:13.600 --> 00:01:18.920 This is your host, Michael Kennedy. I'm a PSF fellow who's been coding for over 25 years. 00:01:19.480 --> 00:01:20.680 Let's connect on social media. 00:01:21.100 --> 00:01:24.160 You'll find me and Talk Python on Mastodon, Bluesky, and X. 00:01:24.330 --> 00:01:26.260 The social links are all in your show notes. 00:01:27.040 --> 00:01:30.560 You can find over 10 years of past episodes at talkpython.fm. 00:01:30.740 --> 00:01:33.920 And if you want to be part of the show, you can join our recording live streams. 00:01:34.200 --> 00:01:34.760 That's right. 00:01:35.080 --> 00:01:38.220 We live stream the raw uncut version of each episode on YouTube. 00:01:38.800 --> 00:01:43.220 Just visit talkpython.fm/youtube to see the schedule of upcoming events. 00:01:43.410 --> 00:01:47.100 Be sure to subscribe there and press the bell so you'll get notified anytime we're recording. 00:01:47.960 --> 00:01:49.900 This episode is brought to you by Sentry. 00:01:50.220 --> 00:01:51.480 Don't let those errors go unnoticed. 00:01:51.650 --> 00:01:53.260 Use Sentry like we do here at Talk Python. 00:01:53.760 --> 00:01:56.640 Sign up at talkpython.fm/sentry. 00:01:57.220 --> 00:02:00.100 And it's also brought to you by Talk Python Courses. 00:02:00.680 --> 00:02:02.940 Course completion certificates are now live. 00:02:03.120 --> 00:02:07.360 If you finished a course, there's a certificate waiting for you on your account page right now. 00:02:07.840 --> 00:02:11.560 Download it as a PDF or add it to your LinkedIn profile with one click 00:02:12.070 --> 00:02:13.580 under Licenses and Certifications. 00:02:14.280 --> 00:02:15.460 Same section as your formal degrees. 00:02:16.560 --> 00:02:20.480 Visit training.talkpython.fm/account to see what you've already earned. 00:02:21.480 --> 00:02:27.180 Matt, welcome to Talk Python To Me. I'm here for some scary clouds and good clouds. Let's talk about it, huh? 00:02:27.380 --> 00:02:28.480 Yeah, sounds good. 00:02:28.780 --> 00:02:34.200 Yeah, the cloud is easy and fun until things go wrong and then it's often in a panic. 00:02:34.670 --> 00:02:37.300 And yeah, we're going to talk about that in two acts, I suppose. 00:02:37.520 --> 00:02:41.720 We're going to talk about kind of the do's and don'ts of cloud, specifically AWS. 00:02:41.990 --> 00:02:45.400 But I imagine a lot of these have analogies to Azure and GCP and others. 00:02:46.300 --> 00:02:50.600 Then we're going to talk specifically about your Cloud War Games, which I think is really fun. 00:02:50.960 --> 00:02:51.560 You're happy to do it. 00:02:51.800 --> 00:02:56.120 Right on. Before we do any of those things, give yourself a quick introduction. Tell the audience who you are. 00:02:56.380 --> 00:03:00.340 Sure. I'm Matt Lea. That's LEA. I run Schematical.com. 00:03:00.620 --> 00:03:04.480 It's a small consulting tech agency that specializes in AWS. 00:03:05.840 --> 00:03:09.100 Our tagline is we help CTOs running on AWS sleep at night. 00:03:09.640 --> 00:03:23.900 So when we do that, we also have cloudwargames.com, which is a place where we do simulated outages and responses for anybody that's looking to learn or train their team to better handle the bump in the night, the cyber attack, etc. 00:03:24.300 --> 00:03:27.400 You know, do you have a section on where you get the email? 00:03:27.980 --> 00:03:29.320 Hi, I'm a security researcher. 00:03:29.640 --> 00:03:31.900 Where do I report issues I found? 00:03:32.620 --> 00:03:33.480 That'll get your attention. 00:03:33.580 --> 00:03:43.000 I was actually just going back and forth with some security researchers on the vulnerability that allowed the OpenAI to escape. 00:03:43.720 --> 00:03:47.740 So I'm going to try and replicate that one, see if I can make something happen there. 00:03:48.340 --> 00:03:49.640 Oh, that's awesome. Yeah, yeah, yeah. 00:03:50.080 --> 00:03:53.540 For people who don't know, one of the OpenAI bots got loose and hacked Hugging Face. 00:03:53.920 --> 00:03:56.400 I don't know what its intent was. I didn't follow it closely enough. 00:03:56.700 --> 00:03:58.280 But yeah, that's what you're referring to, right? 00:03:58.540 --> 00:04:02.440 From what I hear, it was trying to get the answers to the test it was taking. 00:04:02.720 --> 00:04:06.360 So basically it knew that the data set to pass its test was on Huggy Face. 00:04:06.560 --> 00:04:08.840 So cracked into it and just said, ah, interesting. 00:04:09.100 --> 00:04:11.000 You did so much extra work to be lazy. 00:04:11.680 --> 00:04:13.740 What would be the most accurate way? 00:04:13.860 --> 00:04:15.820 Now, if I could just get the answers. 00:04:16.320 --> 00:04:16.480 Yeah. 00:04:16.579 --> 00:04:17.320 Morals aside. 00:04:18.230 --> 00:04:18.320 Yeah. 00:04:18.760 --> 00:04:20.140 At least that's my understanding of it. 00:04:20.320 --> 00:04:24.160 Not only is it 100% accurate, it's really fast now when it takes the test. 00:04:24.840 --> 00:04:25.000 Yeah. 00:04:25.280 --> 00:04:25.560 Crazy. 00:04:26.060 --> 00:04:26.200 Okay. 00:04:26.360 --> 00:04:26.420 Yeah. 00:04:26.500 --> 00:04:28.940 I'm really excited about this, this Cloud War Games thing. 00:04:29.020 --> 00:04:34.440 because I've been on the receiving end of waking up to an outage for various reasons. 00:04:34.680 --> 00:04:39.000 And honestly, I think my employments and my applications have gotten better because of it. 00:04:39.140 --> 00:04:41.920 But in the moment, I also aged faster than normal. 00:04:42.840 --> 00:04:43.200 Oh, yeah. 00:04:43.500 --> 00:04:46.080 When you get the C-suite, yeah, it gets intense, 00:04:46.360 --> 00:04:48.760 which is, again, why I tried to create Cloud War Games 00:04:48.800 --> 00:04:52.120 to try and simulate that experience and inoculate people from the stress. 00:04:52.540 --> 00:04:53.420 Yeah, I love it. 00:04:53.540 --> 00:04:54.940 Like I said, it super resonated with me. 00:04:55.220 --> 00:04:58.380 You also authored a course, Zero to Hero, on AWS security. 00:04:58.700 --> 00:05:00.300 Yep, an animated guide security in the cloud. 00:05:00.600 --> 00:05:03.480 So if you like pixel art and you want to learn about AWS security, 00:05:04.200 --> 00:05:08.420 I've got my isometric pixel art there that I demonstrate, you know, 00:05:08.560 --> 00:05:10.820 and animate network maps and show how it all works. 00:05:11.280 --> 00:05:12.940 You've got some really – you also do cartoons. 00:05:13.150 --> 00:05:14.820 These are really fun. 00:05:15.460 --> 00:05:16.460 I like the Black Friday one. 00:05:17.280 --> 00:05:18.680 I like Black Friday stories. 00:05:19.200 --> 00:05:21.320 This Black Friday sale will make or break us. 00:05:21.440 --> 00:05:22.560 Are we really ready on strike? 00:05:22.860 --> 00:05:23.920 Oh, I'm sure we'll be fine. 00:05:24.160 --> 00:05:24.260 Great. 00:05:24.420 --> 00:05:25.560 I thought we'll sell this on then. 00:05:25.840 --> 00:05:26.080 Yep. 00:05:27.000 --> 00:05:29.300 So as you know, that's a huge retail day. 00:05:29.340 --> 00:05:34.560 So if you've got an e-commerce client, maybe it's their first year doing it or the first time they've hit it big. 00:05:35.420 --> 00:05:36.780 And maybe like, oh, we'll be fine. 00:05:36.880 --> 00:05:38.840 It's like, no, you probably should provision up. 00:05:39.080 --> 00:05:48.740 So each year with my clients, we check in and make sure leading up to that that we've got everything provisioned accordingly and plans for if it needs extra, if we've got to double in scale or something like that. 00:05:48.980 --> 00:05:49.160 Yeah. 00:05:49.620 --> 00:05:53.240 I've never had anything go down on Black Friday, but of course, they get quite a bit of traffic. 00:05:53.360 --> 00:05:55.960 And certainly I've almost had it go down. 00:05:56.400 --> 00:05:59.160 And it's often not for the reason you expect. 00:05:59.380 --> 00:05:59.940 You're like, really? 00:06:00.460 --> 00:06:00.980 That's the thing? 00:06:01.220 --> 00:06:02.020 That was the weakest link? 00:06:02.060 --> 00:06:02.880 I had no idea. 00:06:03.380 --> 00:06:03.500 Yeah. 00:06:04.300 --> 00:06:10.440 One time it was, I had extra large JavaScript file that was being served and gzipped by 00:06:10.680 --> 00:06:10.960 Nginx. 00:06:11.240 --> 00:06:12.440 And that almost took down the site. 00:06:12.640 --> 00:06:12.860 Really? 00:06:13.060 --> 00:06:13.180 Okay. 00:06:13.440 --> 00:06:13.860 Oh, well. 00:06:14.220 --> 00:06:15.880 But yeah, that was a fun cartoon. 00:06:15.940 --> 00:06:17.420 So how often do you do these cartoons? 00:06:17.540 --> 00:06:18.100 How many you got? 00:06:18.880 --> 00:06:24.560 I was doing them monthly at one point, but now it's more as it comes to me a little bit 00:06:24.640 --> 00:06:24.780 more. 00:06:25.220 --> 00:06:28.300 So we're, you know, but yeah, there's them. 00:06:28.420 --> 00:06:30.260 I mean, there's 30, 40 of them or so. 00:06:31.020 --> 00:06:33.700 Some of them have gotten on Reddit quite a few views. 00:06:33.800 --> 00:06:36.000 Programmer humor, r slash programmer humor. 00:06:36.320 --> 00:06:36.880 They like it there. 00:06:37.080 --> 00:06:39.180 I am all here for r slash programming humor. 00:06:39.300 --> 00:06:40.320 That place is hilarious. 00:06:40.880 --> 00:06:41.080 Yeah. 00:06:41.420 --> 00:06:41.480 Yeah. 00:06:42.000 --> 00:06:42.180 Amazing. 00:06:42.780 --> 00:06:42.940 Amazing. 00:06:43.480 --> 00:06:43.720 All right. 00:06:43.820 --> 00:06:47.740 Well, let's start by just talking about and like get a high level. 00:06:48.080 --> 00:06:49.540 You know, people want to move to the cloud. 00:06:49.920 --> 00:06:52.160 You know, what are some of the benefits that they see? 00:06:52.400 --> 00:06:55.820 Like what are maybe some surprise issues that they run into? 00:06:56.320 --> 00:07:00.580 I'm sure you've had this conversations with a lot of C-suites and dev leads and so on. 00:07:00.700 --> 00:07:01.660 I'd say the complexity. 00:07:02.340 --> 00:07:07.040 AWS, I think I did a post pretty recently, but AWS has a meeting simulator service now 00:07:07.280 --> 00:07:10.040 where you can train yourself on how to take meetings. 00:07:10.240 --> 00:07:11.520 And I'm like, that's too many services. 00:07:12.560 --> 00:07:14.960 So probably the number one problem is the complexity. 00:07:15.100 --> 00:07:17.580 People are like, oh my gosh, I don't understand this. 00:07:17.640 --> 00:07:19.080 I understand what I'm getting billed for. 00:07:19.480 --> 00:07:23.580 And if you can whittle it down to some core services that meet your needs and your goals, 00:07:24.340 --> 00:07:27.460 then a huge amount of these services aren't even necessary. 00:07:27.640 --> 00:07:30.460 There are services for robotics, which is great if you're in robotics, 00:07:30.530 --> 00:07:33.100 but that's not something you really need to worry about if you're just an e-commerce platform. 00:07:33.600 --> 00:07:36.160 So there's just so much, such a variety there. 00:07:36.400 --> 00:07:39.460 So pick your tools and just get really good with the tools. 00:07:39.900 --> 00:07:42.660 There's some basics that you got to use for everything, which we're going to go over in a minute. 00:07:43.100 --> 00:07:43.320 Yeah. 00:07:43.850 --> 00:07:48.320 One of my most recurring experiences with AWS is I open up the console. 00:07:48.640 --> 00:07:50.540 I'm like, what is all of this? 00:07:50.920 --> 00:07:51.600 What is this? 00:07:51.900 --> 00:07:54.160 So you're suggesting that maybe trim that down a little 00:07:54.400 --> 00:07:55.780 and just put the stuff that you really know 00:07:55.840 --> 00:07:56.500 that you're going to need there? 00:07:56.800 --> 00:07:59.680 Yeah, and we can go over kind of some of my core stacks there. 00:07:59.760 --> 00:08:01.780 It depends if you want to be serverless or not serverless. 00:08:01.880 --> 00:08:02.920 If you're just starting out, 00:08:02.940 --> 00:08:04.360 you don't have a crazy amount of traffic. 00:08:04.560 --> 00:08:05.900 I'd go serverless a lot of times. 00:08:06.080 --> 00:08:07.660 It's real easy to set up, less to manage. 00:08:07.960 --> 00:08:08.560 Okay, yeah. 00:08:08.680 --> 00:08:09.680 What are some of the core ideas? 00:08:09.820 --> 00:08:12.740 I mean, certainly Lambda sounds like that's in there. 00:08:13.000 --> 00:08:15.000 Probably EC2, S3, what are we talking? 00:08:16.219 --> 00:08:17.580 Lambda makes a great web server. 00:08:18.080 --> 00:08:19.900 the API gateway plus Lambda. 00:08:20.740 --> 00:08:22.360 So you have the serverless, 00:08:22.960 --> 00:08:24.840 extremely cheap application layer right there. 00:08:25.070 --> 00:08:25.960 So not a data layer, 00:08:26.470 --> 00:08:29.260 but that's a great start to just connect those two together. 00:08:29.580 --> 00:08:31.140 And if anybody wants, 00:08:32.159 --> 00:08:33.500 these are my offer for free. 00:08:33.530 --> 00:08:34.300 I've got them open source, 00:08:34.479 --> 00:08:35.659 schematical.com slash free. 00:08:35.950 --> 00:08:38.360 I have Terraform scripts that'll set this up. 00:08:38.700 --> 00:08:41.180 So I don't know how chronologically ordered we want to go, 00:08:42.080 --> 00:08:44.860 but I just jump it back a step. 00:08:45.140 --> 00:08:47.140 Terraform or CloudFormation. 00:08:47.400 --> 00:08:48.140 I highly recommend. 00:08:48.210 --> 00:08:51.100 So if you're just getting into it, a lot of times, just like code, right? 00:08:51.130 --> 00:08:53.460 You always want to have your code in some type of version control. 00:08:53.610 --> 00:08:55.820 You don't want to just have it FTP or saved to your disk. 00:08:55.870 --> 00:08:57.760 And you're just like, oh, you know, it disappeared. 00:08:57.810 --> 00:09:02.740 I don't know when Bob over here, you know, other developer X made this change. 00:09:03.060 --> 00:09:04.480 You want all that in version control. 00:09:04.760 --> 00:09:10.100 And you can do that infrastructure as code known, you know, which is Terraform, Bintofu, 00:09:11.780 --> 00:09:15.800 and CloudFormation is actually AWS's proprietary one that they made themselves. 00:09:16.340 --> 00:09:20.480 And so I'd recommend you get in that habit because it's very difficult to get into that later. 00:09:21.120 --> 00:09:25.820 Once you're you've established your infrastructure, you've all hand spun it up and you're like, how did I do that again? 00:09:26.320 --> 00:09:29.140 What was that? What would what is such and such that left the company? 00:09:29.280 --> 00:09:32.520 And now, you know, so you're basically describing me 10 years ago. 00:09:33.740 --> 00:09:40.100 I was there. I didn't I didn't get on that game till I was a couple of years after they had cloud formation launched. 00:09:40.130 --> 00:09:43.380 At first, I was like, this could be cool. And then I was like, OK, this is essential. 00:09:43.880 --> 00:09:47.940 Yeah. Yeah. I just, I used to be just like, I'm going to SSH in and I'll just get it working. 00:09:48.380 --> 00:09:52.300 And then, you know, the code will be well-structured, but just kind of the server, 00:09:52.520 --> 00:09:56.280 once I get it set up, it'll be fine. And then I realized actually, no, it's not fine. 00:09:57.040 --> 00:10:00.420 You know, you want to make any changes you want to move, you want to move from one setup to 00:10:00.560 --> 00:10:04.120 another. You're like, Oh gosh, I don't even like, this is a research project to figure out what to 00:10:04.220 --> 00:10:09.820 do. Yeah, exactly. But definitely when you're using those, those terraforms, always read your, 00:10:10.200 --> 00:10:11.000 what you're about to apply. 00:10:11.380 --> 00:10:13.900 I've had too many people that just auto apply 00:10:14.360 --> 00:10:15.120 the Terraform changes. 00:10:15.190 --> 00:10:17.960 And what happens is they'll have a slight renaming 00:10:18.100 --> 00:10:18.460 of a database. 00:10:18.570 --> 00:10:19.520 This actually happened recently. 00:10:19.660 --> 00:10:22.380 Somebody had their AI vibe code, this thing, 00:10:22.700 --> 00:10:24.120 and it wanted to rename the database. 00:10:24.730 --> 00:10:27.340 Well then what happens is Terraform deleted the database 00:10:27.700 --> 00:10:29.560 and then spun up another database right there. 00:10:29.590 --> 00:10:30.780 That was their production database. 00:10:31.230 --> 00:10:31.880 That was a rough day. 00:10:32.880 --> 00:10:34.380 That wasn't the project I have. 00:10:34.550 --> 00:10:36.240 I did a write up on it, but it wasn't my project. 00:10:36.390 --> 00:10:38.320 I was always read your replies though, basically. 00:10:38.640 --> 00:10:40.140 Always know what you're about to send. 00:10:41.320 --> 00:10:43.200 Make sure you think it's an update. 00:10:43.390 --> 00:10:45.820 It's got to be an update, not a delete and replace. 00:10:46.300 --> 00:10:49.720 This is something that has certainly been in the zeitgeist lately. 00:10:50.260 --> 00:10:52.280 It's a little bit, I don't hear about it as much. 00:10:52.310 --> 00:10:53.820 I think it's sort of to be dealt with. 00:10:54.110 --> 00:10:57.840 But there's certainly horror stories of the AI deleted the production database 00:10:58.230 --> 00:11:00.600 because kind of like we opened with like, 00:11:00.830 --> 00:11:02.980 instead of trying to just solve the problems, like, is there a better way? 00:11:03.080 --> 00:11:06.780 Like, well, there's a problem inserting to the database 00:11:07.060 --> 00:11:07.860 because the structure is wrong. 00:11:08.120 --> 00:11:11.400 So if we just recreate the database with the right structure, this code will work, right? 00:11:11.500 --> 00:11:15.320 It's like, well, that technically will solve the problem, but that is not acceptable. 00:11:15.779 --> 00:11:16.620 I've heard of it. 00:11:16.780 --> 00:11:18.000 They didn't like the schema. 00:11:18.100 --> 00:11:21.140 So they said, okay, drop the table and I'll create a new one. 00:11:21.540 --> 00:11:27.620 Yeah, I've literally had that happen to me only in my dev machine where it doesn't really matter. 00:11:27.620 --> 00:11:29.180 I'm like, why are there no records? 00:11:29.280 --> 00:11:30.140 Where's all the data? 00:11:30.560 --> 00:11:33.420 Oh, it was better for like, no, we never do that. 00:11:33.580 --> 00:11:37.000 But I'm not giving my AI access directly to my production database. 00:11:37.200 --> 00:11:37.540 No, thanks. 00:11:38.000 --> 00:11:39.560 No, I would not recommend that. 00:11:39.680 --> 00:11:46.160 I mean, as we're seeing more requests for that from like as C-suite people become more functional with these agents, 00:11:46.660 --> 00:11:49.860 they're like, well, I want direct access to the database so I can get the latest in this and that. 00:11:49.920 --> 00:11:53.480 It's like, hold on, let me code you up a tool call, get your read access. 00:11:53.720 --> 00:11:58.920 And if you want to do any right, they'll code up another tool call that'll make it so it's approved by a human. 00:11:59.340 --> 00:12:01.220 But doesn't that, you know, they could say, hey, I suggest this. 00:12:01.220 --> 00:12:03.840 And then, you know, but we've seen a lot of that lately. 00:12:04.600 --> 00:12:08.860 I mean, it's great that we're empowering more people to interact with the data and do more with it. 00:12:08.940 --> 00:12:13.820 You just, we still got to be careful about InfoSec and these things do make mistakes occasionally. 00:12:14.360 --> 00:12:14.960 Yeah, yeah. 00:12:16.040 --> 00:12:19.520 They work in a narrow context and let's make this code run. 00:12:19.600 --> 00:12:20.420 Like, oh, this will fix it. 00:12:20.480 --> 00:12:22.500 Like, we need the whole company to survive. 00:12:22.780 --> 00:12:31.500 No, I really love your suggestion of maybe creating like an MCP or a tool that's read only and make all the AI access go through that. 00:12:31.600 --> 00:12:38.240 You can observe it and check it out, but you don't have even the capability to destroy it through this mechanism, right? 00:12:38.480 --> 00:12:50.580 Well, there's even an extra layer to that is I just had a post come out about this because I had this request come through is put it pointed at your data lake if you can, not your production database, because those agents can spam like no tomorrow. 00:12:50.700 --> 00:12:58.420 So if they write a very complicated read query that goes through your tool call and they spam it could slow down your production database if it's not, you know, quickly optimized. 00:12:58.880 --> 00:13:02.320 So what we did to make it even more effective is our data lakes. 00:13:02.330 --> 00:13:05.840 A lot of times you don't want to store personally identifiable information and data lakes and all that stuff. 00:13:06.040 --> 00:13:06.900 It's anonymized. 00:13:06.930 --> 00:13:08.500 You can do big data regression on it. 00:13:08.690 --> 00:13:13.820 And so that way, even if it gets data that possibly be personally identifiable, it would be in your production database. 00:13:14.360 --> 00:13:17.020 The data lake theoretically wouldn't have that if you're doing it right. 00:13:17.440 --> 00:13:24.380 And so they couldn't, even if they leaked all your sales data for the last five years, at least you wouldn't have a PII leak, which would cost you even more. 00:13:24.720 --> 00:13:26.680 So that's my recommendation there. 00:13:26.840 --> 00:13:33.320 And I didn't actually, for the outline for today, I didn't talk about data lakes, but I can tell you my favorite data lake stack on AWS, but it's an advanced one. 00:13:33.400 --> 00:13:34.720 I'd save that for number 14, maybe. 00:13:35.020 --> 00:13:36.580 All right, maybe we'll get to that at the end. 00:13:38.200 --> 00:13:38.780 Hey, one second. 00:13:39.680 --> 00:13:43.040 Normally, this would be an ad break from Sentry, but not this time. 00:13:43.400 --> 00:13:46.960 Let's just thank them for supporting the show and get right back to the conversation. 00:13:47.420 --> 00:13:50.840 Also, visit talkpython.fm/sentry after the show. 00:13:51.180 --> 00:13:51.640 Thanks, Sentry. 00:13:52.860 --> 00:14:01.100 Okay, well, let's put a bit of a stake in the ground here and put down the first item as don't hand provision, use Terraform. 00:14:01.300 --> 00:14:11.560 And as I was getting ready for this episode and go through all the stuff we kind of like we'd talk about, I realized that this is acronym packed, an acronym dense and tooled as things. 00:14:11.720 --> 00:14:13.420 So maybe you could help us along the way. 00:14:13.560 --> 00:14:14.620 We've got Terraform. 00:14:15.680 --> 00:14:18.980 And yeah, just maybe a quick summary on this concept. 00:14:19.160 --> 00:14:20.400 Like where we use it and how we use it. 00:14:20.460 --> 00:14:22.520 We've been talking around it, but not as much as an item. 00:14:22.580 --> 00:14:25.360 Safe to assume most of the people listening to this are coders. 00:14:25.960 --> 00:14:26.360 100%. 00:14:26.640 --> 00:14:30.920 Data science coders, yeah, or learners, but, you know, they're still good for them. 00:14:31.080 --> 00:14:31.220 Sure. 00:14:31.240 --> 00:14:37.560 So it's Terraform or IOC infrastructure, IAC, sorry, IAC, infrastructure as code is the category. 00:14:37.800 --> 00:14:42.220 And then under that category, there's Terraform, OpenTofu, and CloudFormation. 00:14:42.420 --> 00:14:48.260 And those are all just ways to define how you want your infrastructure to look basically in code. 00:14:48.480 --> 00:14:52.540 So if I was real simple, I could say I want an EC2 instance, which is... 00:14:52.560 --> 00:14:55.300 just your basic server, virtual server running on Amazon. 00:14:55.900 --> 00:15:00.400 And then I want a database, a MySQL database, this version. 00:15:00.920 --> 00:15:03.100 And I can put those in code in some way. 00:15:03.170 --> 00:15:06.260 And then, like I said, version control it, or I can hit apply. 00:15:06.290 --> 00:15:08.940 So say somebody accidentally deletes my application layer. 00:15:09.040 --> 00:15:10.800 A lot easier to replace than a data layer, of course. 00:15:11.080 --> 00:15:13.840 And I need to spin that back up real quickly. 00:15:13.850 --> 00:15:18.740 You just re-hit Terraform apply, and it re-spins up and provisions the missing area. 00:15:18.830 --> 00:15:21.360 It's really nice, especially for my cloud war games, 00:15:21.500 --> 00:15:25.400 I'm blowing up stuff all the time, intentionally being the, you know, junior that accidentally 00:15:25.820 --> 00:15:26.680 deletes something important. 00:15:27.060 --> 00:15:31.880 A little bit of a code monkey or chaos monkey, rather, running loose kind of thing. 00:15:32.200 --> 00:15:33.360 Yeah, cloud demolition expert. 00:15:33.480 --> 00:15:35.520 I think I had that as my title on LinkedIn for a while. 00:15:35.720 --> 00:15:36.140 I love it. 00:15:36.380 --> 00:15:36.480 Okay. 00:15:36.900 --> 00:15:39.300 Does Terraform do incremental updates? 00:15:39.400 --> 00:15:43.640 Like, let's suppose I've got an EC2 machine and I've got it all set up and it's running 00:15:43.960 --> 00:15:44.820 the app, the database. 00:15:45.240 --> 00:15:50.179 And I'm like, oh, you know, it would be really great if we put these firewall rules specifically 00:15:50.200 --> 00:15:54.680 on the machine or we set up fail to ban or something like that you can define every firewall 00:15:55.000 --> 00:16:00.120 rule which is perfect you wouldn't be able to define as easily what's in the box so to say 00:16:00.340 --> 00:16:05.060 there's you have some control over that um like for example i spin up lambdas and a lot of times 00:16:05.110 --> 00:16:09.340 i have a hello world javascript i throw in there and it's but i also again these are all for free 00:16:09.340 --> 00:16:13.380 on my website if you want them but um it also spins up the build pipeline and everything like 00:16:13.480 --> 00:16:17.259 that so then you can just hook that up to GitHub but at least it throws in a hello world lambda 00:16:17.280 --> 00:16:21.560 the code. But yeah, you can define all the stuff we're about to talk about today. You can define 00:16:21.660 --> 00:16:31.340 every user role, every access key, every username, the exact way you want your networks to talk. 00:16:31.440 --> 00:16:36.800 So the way your VPC, a virtual private cloud, has its subnet set up, all that stuff can be defined 00:16:37.060 --> 00:16:42.140 in there. It's almost crazy what can be defined. I wouldn't recommend committing your code and 00:16:42.360 --> 00:16:46.539 having it go through there. I'd have that go through a build pipeline separately that you'd 00:16:46.560 --> 00:16:51.260 spin up with terraform or whatever but you'd also you just pull from GitHub sure especially for the 00:16:51.480 --> 00:16:56.200 compiled languages and like go or something where there's no reason to deploy the source with python 00:16:56.400 --> 00:17:01.380 it's almost like not much of a difference although ci is still a thing okay so number one don't hand 00:17:01.560 --> 00:17:05.839 provision use terraform set standards stick to them i love it yeah it's it's getting that habit early 00:17:06.180 --> 00:17:12.299 it's it's tough to get into later number two act don't use access keys use iam roles so again 00:17:12.640 --> 00:17:19.439 acronym please so access access keys so i am is identity access management manager what it's it's 00:17:19.439 --> 00:17:26.240 how you define how your users interact with aws as well as your different running services on aws 00:17:26.680 --> 00:17:31.840 and so a lot of people when they first start out they download their first access keys they ideally 00:17:32.120 --> 00:17:35.780 don't put them in version control where it can be passed around or shared or anything like that 00:17:35.800 --> 00:17:41.120 they do a nice dot env and hope their agent doesn't steal it um but uh you know you get so you get those 00:17:41.080 --> 00:17:44.620 access keys but a lot of times when they push things to production if they're very beginner at 00:17:44.720 --> 00:17:49.280 this they'll put like a.env file in the code or something like that that's got access keys 00:17:49.760 --> 00:17:55.380 and those if those get leaked you know again I've got a story about this that somebody actually 00:17:56.180 --> 00:18:02.920 pushed their access keys for sending email luckily luckily it was just sending and receiving email 00:18:03.340 --> 00:18:07.420 and they pushed it up to a public repo as a junior and they just made a mistake there and it ended up 00:18:07.440 --> 00:18:13.760 on GitHub and some bad guys got it. And I believe we sent 16 million emails for some type of knockoff 00:18:13.880 --> 00:18:18.660 Viagra in about eight minutes before AWS shut down the account and said, what are you doing? It was a 00:18:18.700 --> 00:18:22.000 lot more than we normally said, but that's an example of why you don't want to hard. 00:18:22.280 --> 00:18:30.500 Yeah. I have a thing to share. Have you seen skit? Like, no, it's, it's the pronunciation is perfect 00:18:30.540 --> 00:18:38.460 of what it is, but it's like secrets, S-H-H, get, but get like, oh God. And it used to be a website. 00:18:38.680 --> 00:18:42.500 It's no longer here. Let me see if I can open up this image in a full screen for us to see. 00:18:42.880 --> 00:18:49.680 So it was a website that would scan the real time fire hose of changes on GitHub, pull out all 00:18:50.140 --> 00:18:58.960 cryptographic looking things, AWS keys, you know, email keys. And it just had this stream going and 00:18:58.980 --> 00:19:05.240 it updates, I don't know, like once a second or more. It's absolutely terrifying. And so anytime 00:19:05.820 --> 00:19:09.700 that someone's out there listening and you need to make this point to juniors or people who are new 00:19:09.760 --> 00:19:13.340 or whatever, like do not commit stuff to get it. Like, well, it's a, you know, I can just 00:19:14.020 --> 00:19:18.100 uncommit it. It's not a very popular repo. How fast would people find it? Like this fast. 00:19:18.540 --> 00:19:24.620 Fast enough. Yeah. If it's up there, you should assume it's gone immediately. And so, yeah, 00:19:25.120 --> 00:19:26.420 That's pretty scary. 00:19:26.820 --> 00:19:29.000 You know, I don't know if you tracked this, probably not, 00:19:29.600 --> 00:19:33.800 but at PyCon this year, there was a talk by Tristan McKinnon 00:19:34.340 --> 00:19:36.600 called Zero Trust in 200 Milliseconds. 00:19:37.000 --> 00:19:40.980 Now, I didn't watch the talk, so I'm only going off the abstract. 00:19:41.280 --> 00:19:45.480 But basically, the idea is every time you need to make an API call 00:19:45.580 --> 00:19:46.760 or do a transaction or something, 00:19:46.920 --> 00:19:51.380 you can get a separate 200 millisecond duration key. 00:19:51.960 --> 00:19:53.060 What do you think of that idea? 00:19:53.260 --> 00:19:54.100 I like it. 00:19:54.600 --> 00:19:58.060 200 milliseconds is interesting if that's a set one or a variable one. 00:19:58.330 --> 00:19:58.820 Yeah, I'm not sure. 00:19:58.820 --> 00:20:00.900 It might be a little bit shorter, a little bit longer. 00:20:01.320 --> 00:20:03.720 In case there's a little lag, you don't want to pull three. 00:20:04.060 --> 00:20:06.860 I also immediately wonder how long does it take you to provision that key? 00:20:07.420 --> 00:20:09.640 Hopefully it doesn't take you 200 milliseconds to provision it, 00:20:09.640 --> 00:20:11.200 but if you could provision it in two milliseconds. 00:20:13.060 --> 00:20:17.220 I don't like, what do they call it, chocolate candy infrastructure 00:20:17.400 --> 00:20:21.380 where once you pass the outside, then it's free once you get inside. 00:20:21.450 --> 00:20:23.700 I forget the exact term there, but you've got to be careful. 00:20:23.880 --> 00:20:25.620 even once you're past the infrastructure, 00:20:26.360 --> 00:20:27.600 the external infrastructure, 00:20:27.810 --> 00:20:28.280 once you're in there, 00:20:28.310 --> 00:20:29.000 you don't want to have it. 00:20:29.020 --> 00:20:30.080 So anybody's got free reign 00:20:30.200 --> 00:20:31.280 to grab whatever they want. 00:20:31.510 --> 00:20:32.720 They might get that. 00:20:32.730 --> 00:20:33.600 So I like it. 00:20:33.880 --> 00:20:34.820 My other question though, 00:20:34.860 --> 00:20:36.200 we always have to balance is, 00:20:36.720 --> 00:20:38.320 what's the latency effects? 00:20:38.720 --> 00:20:40.520 If that costs 200 milliseconds of latency, 00:20:40.700 --> 00:20:41.360 that's pretty expensive 00:20:41.620 --> 00:20:42.460 from a compute standpoint. 00:20:43.100 --> 00:20:43.820 So 00:20:44.240 --> 00:20:45.860 Yeah, I'm not entirely sure. 00:20:45.860 --> 00:20:47.320 But just a little more background about this. 00:20:47.320 --> 00:20:49.160 This is from FedRAMP, 00:20:49.160 --> 00:20:51.500 which is the US government's, 00:20:51.500 --> 00:20:52.860 they use this on a lot of the US government 00:20:53.200 --> 00:20:56.500 internal protection like cloud or whatever, which is maybe a special case. 00:20:56.830 --> 00:21:01.800 That makes a lot of sense. And I wager the government's latency limitations are a lot 00:21:01.830 --> 00:21:04.420 less than someone trying to be competitive in e-commerce or something like that. 00:21:04.600 --> 00:21:07.160 Yes, exactly. You know what? They can wait for security. 00:21:07.520 --> 00:21:11.540 Yeah. It's like the DMV online. Just stand in line all day. They'll get there. There's nowhere 00:21:11.660 --> 00:21:11.960 else to go. 00:21:12.440 --> 00:21:16.780 It's like just get in line with all the other poor browsers and wait back there for your turn and 00:21:16.790 --> 00:21:22.840 we'll get to you eventually. Unless we closed at four. Unless we closed at four. Okay. So 00:21:22.860 --> 00:21:24.700 Basically, don't use access keys. 00:21:24.940 --> 00:21:25.540 Use IAM roles. 00:21:25.940 --> 00:21:26.960 How do we make this happen, though? 00:21:27.120 --> 00:21:32.060 So IAM, you go in there, there's users, there's groups, which makes a lot of sense. 00:21:32.639 --> 00:21:36.900 The roles are things you assign to the various services you boot up. 00:21:37.080 --> 00:21:41.160 So you can have a role for the EC2 instance or a role that the Lambda has. 00:21:41.240 --> 00:21:47.340