Michael Kennedy: Your site is down. It's 3 a.m. Is it a bug, a bill, or a breach? You can't tell yet, and everyone is
Michael Kennedy: watching you find out. Matt Lea has spent 15 years being the person companies call when an outage is
Michael Kennedy: costing them real money per hour. And his sole argument is that everything you want in that
Michael Kennedy: moment gets decided months earlier, on ordinary afternoons, when someone chose the convenient
Michael Kennedy: thing. We walk through his top 12 do's and don'ts for AWS. Infrastructure as code, IAM roles instead
Michael Kennedy: of access keys, private subnets, no wildcards, no public buckets. And I push on which of them
Michael Kennedy: actually matters if you're one person on a small VPS self-hosting. Then we get to cloud war games
Michael Kennedy: where Matt breaks things on purpose so your team's first real incident isn't their first incident.
Michael Kennedy: Let's get into it. This is Talk Python To Me episode 559, recorded July 31st, 2026.
Michael Kennedy: Welcome to Talk Python To Me, the number one Python podcast for developers and data scientists.
Michael Kennedy: This is your host, Michael Kennedy. I'm a PSF fellow who's been coding for over 25 years.
Michael Kennedy: Let's connect on social media.
Michael Kennedy: You'll find me and Talk Python on Mastodon, Bluesky, and X.
Michael Kennedy: The social links are all in your show notes.
Michael Kennedy: You can find over 10 years of past episodes at talkpython.fm.
Michael Kennedy: And if you want to be part of the show, you can join our recording live streams.
Michael Kennedy: That's right.
Michael Kennedy: We live stream the raw uncut version of each episode on YouTube.
Michael Kennedy: Just visit talkpython.fm/youtube to see the schedule of upcoming events.
Michael Kennedy: Be sure to subscribe there and press the bell so you'll get notified anytime we're recording.
Michael Kennedy: This episode is brought to you by Sentry.
Michael Kennedy: Don't let those errors go unnoticed.
Michael Kennedy: Use Sentry like we do here at Talk Python.
Michael Kennedy: Sign up at talkpython.fm/sentry.
Michael Kennedy: And it's also brought to you by Talk Python Courses.
Michael Kennedy: Course completion certificates are now live.
Michael Kennedy: If you finished a course, there's a certificate waiting for you on your account page right now.
Michael Kennedy: Download it as a PDF or add it to your LinkedIn profile with one click
Michael Kennedy: under Licenses and Certifications.
Michael Kennedy: Same section as your formal degrees.
Michael Kennedy: Visit training.talkpython.fm/account to see what you've already earned.
Michael Kennedy: Matt, welcome to Talk Python To Me. I'm here for some scary clouds and good clouds. Let's talk about it, huh?
Matt Lea: Yeah, sounds good.
Michael Kennedy: Yeah, the cloud is easy and fun until things go wrong and then it's often in a panic.
Michael Kennedy: And yeah, we're going to talk about that in two acts, I suppose.
Michael Kennedy: We're going to talk about kind of the do's and don'ts of cloud, specifically AWS.
Michael Kennedy: But I imagine a lot of these have analogies to Azure and GCP and others.
Michael Kennedy: Then we're going to talk specifically about your Cloud War Games, which I think is really fun.
Michael Kennedy: You're happy to do it.
Michael Kennedy: Right on. Before we do any of those things, give yourself a quick introduction. Tell the audience who you are.
Matt Lea: Sure. I'm Matt Lea. That's LEA. I run Schematical.com.
Matt Lea: It's a small consulting tech agency that specializes in AWS.
Matt Lea: Our tagline is we help CTOs running on AWS sleep at night.
Matt Lea: So when we do that, we also have cloudwargames.com, which is a place where we do simulated outages and responses for anybody that's looking to learn or train their team to better handle the bump in the night, the cyber attack, etc.
Michael Kennedy: You know, do you have a section on where you get the email?
Michael Kennedy: Hi, I'm a security researcher.
Michael Kennedy: Where do I report issues I found?
Michael Kennedy: That'll get your attention.
Matt Lea: I was actually just going back and forth with some security researchers on the vulnerability that allowed the OpenAI to escape.
Matt Lea: So I'm going to try and replicate that one, see if I can make something happen there.
Michael Kennedy: Oh, that's awesome. Yeah, yeah, yeah.
Michael Kennedy: For people who don't know, one of the OpenAI bots got loose and hacked Hugging Face.
Michael Kennedy: I don't know what its intent was. I didn't follow it closely enough.
Michael Kennedy: But yeah, that's what you're referring to, right?
Matt Lea: From what I hear, it was trying to get the answers to the test it was taking.
Matt Lea: So basically it knew that the data set to pass its test was on Huggy Face.
Matt Lea: So cracked into it and just said, ah, interesting.
Matt Lea: You did so much extra work to be lazy.
Michael Kennedy: What would be the most accurate way?
Michael Kennedy: Now, if I could just get the answers.
Michael Kennedy: Yeah.
Michael Kennedy: Morals aside.
Matt Lea: Yeah.
Matt Lea: At least that's my understanding of it.
Michael Kennedy: Not only is it 100% accurate, it's really fast now when it takes the test.
Michael Kennedy: Yeah.
Michael Kennedy: Crazy.
Michael Kennedy: Okay.
Michael Kennedy: Yeah.
Michael Kennedy: I'm really excited about this, this Cloud War Games thing.
Michael Kennedy: because I've been on the receiving end of waking up to an outage for various reasons.
Michael Kennedy: And honestly, I think my employments and my applications have gotten better because of it.
Michael Kennedy: But in the moment, I also aged faster than normal.
Matt Lea: Oh, yeah.
Matt Lea: When you get the C-suite, yeah, it gets intense,
Matt Lea: which is, again, why I tried to create Cloud War Games
Matt Lea: to try and simulate that experience and inoculate people from the stress.
Michael Kennedy: Yeah, I love it.
Matt Lea: Like I said, it super resonated with me.
Michael Kennedy: You also authored a course, Zero to Hero, on AWS security.
Matt Lea: Yep, an animated guide security in the cloud.
Matt Lea: So if you like pixel art and you want to learn about AWS security,
Matt Lea: I've got my isometric pixel art there that I demonstrate, you know,
Matt Lea: and animate network maps and show how it all works.
Michael Kennedy: You've got some really – you also do cartoons.
Michael Kennedy: These are really fun.
Michael Kennedy: I like the Black Friday one.
Michael Kennedy: I like Black Friday stories.
Michael Kennedy: This Black Friday sale will make or break us.
Michael Kennedy: Are we really ready on strike?
Michael Kennedy: Oh, I'm sure we'll be fine.
Michael Kennedy: Great.
Michael Kennedy: I thought we'll sell this on then.
Matt Lea: Yep.
Matt Lea: So as you know, that's a huge retail day.
Matt Lea: So if you've got an e-commerce client, maybe it's their first year doing it or the first time they've hit it big.
Matt Lea: And maybe like, oh, we'll be fine.
Matt Lea: It's like, no, you probably should provision up.
Matt Lea: So each year with my clients, we check in and make sure leading up to that that we've got everything provisioned accordingly and plans for if it needs extra, if we've got to double in scale or something like that.
Michael Kennedy: Yeah.
Michael Kennedy: I've never had anything go down on Black Friday, but of course, they get quite a bit of traffic.
Michael Kennedy: And certainly I've almost had it go down.
Michael Kennedy: And it's often not for the reason you expect.
Michael Kennedy: You're like, really?
Michael Kennedy: That's the thing?
Michael Kennedy: That was the weakest link?
Michael Kennedy: I had no idea.
Michael Kennedy: Yeah.
Michael Kennedy: One time it was, I had extra large JavaScript file that was being served and gzipped by
Michael Kennedy: Nginx.
Michael Kennedy: And that almost took down the site.
Michael Kennedy: Really?
Michael Kennedy: Okay.
Michael Kennedy: Oh, well.
Michael Kennedy: But yeah, that was a fun cartoon.
Michael Kennedy: So how often do you do these cartoons?
Michael Kennedy: How many you got?
Matt Lea: I was doing them monthly at one point, but now it's more as it comes to me a little bit
Matt Lea: more.
Matt Lea: So we're, you know, but yeah, there's them.
Matt Lea: I mean, there's 30, 40 of them or so.
Matt Lea: Some of them have gotten on Reddit quite a few views.
Matt Lea: Programmer humor, r slash programmer humor.
Matt Lea: They like it there.
Michael Kennedy: I am all here for r slash programming humor.
Michael Kennedy: That place is hilarious.
Michael Kennedy: Yeah.
Michael Kennedy: Amazing.
Michael Kennedy: All right.
Michael Kennedy: Well, let's start by just talking about and like get a high level.
Michael Kennedy: You know, people want to move to the cloud.
Michael Kennedy: You know, what are some of the benefits that they see?
Michael Kennedy: Like what are maybe some surprise issues that they run into?
Michael Kennedy: I'm sure you've had this conversations with a lot of C-suites and dev leads and so on.
Matt Lea: I'd say the complexity.
Matt Lea: AWS, I think I did a post pretty recently, but AWS has a meeting simulator service now
Matt Lea: where you can train yourself on how to take meetings.
Matt Lea: And I'm like, that's too many services.
Matt Lea: So probably the number one problem is the complexity.
Matt Lea: People are like, oh my gosh, I don't understand this.
Matt Lea: I understand what I'm getting billed for.
Matt Lea: And if you can whittle it down to some core services that meet your needs and your goals,
Matt Lea: then a huge amount of these services aren't even necessary.
Matt Lea: There are services for robotics, which is great if you're in robotics,
Matt Lea: but that's not something you really need to worry about if you're just an e-commerce platform.
Matt Lea: So there's just so much, such a variety there.
Matt Lea: So pick your tools and just get really good with the tools.
Matt Lea: There's some basics that you got to use for everything, which we're going to go over in a minute.
Michael Kennedy: Yeah.
Michael Kennedy: One of my most recurring experiences with AWS is I open up the console.
Michael Kennedy: I'm like, what is all of this?
Michael Kennedy: What is this?
Michael Kennedy: So you're suggesting that maybe trim that down a little
Michael Kennedy: and just put the stuff that you really know
Michael Kennedy: that you're going to need there?
Matt Lea: Yeah, and we can go over kind of some of my core stacks there.
Matt Lea: It depends if you want to be serverless or not serverless.
Matt Lea: If you're just starting out,
Matt Lea: you don't have a crazy amount of traffic.
Matt Lea: I'd go serverless a lot of times.
Matt Lea: It's real easy to set up, less to manage.
Michael Kennedy: Okay, yeah.
Michael Kennedy: What are some of the core ideas?
Michael Kennedy: I mean, certainly Lambda sounds like that's in there.
Michael Kennedy: Probably EC2, S3, what are we talking?
Matt Lea: Lambda makes a great web server.
Matt Lea: the API gateway plus Lambda.
Matt Lea: So you have the serverless,
Matt Lea: extremely cheap application layer right there.
Matt Lea: So not a data layer,
Matt Lea: but that's a great start to just connect those two together.
Matt Lea: And if anybody wants,
Matt Lea: these are my offer for free.
Matt Lea: I've got them open source,
Matt Lea: schematical.com slash free.
Matt Lea: I have Terraform scripts that'll set this up.
Matt Lea: So I don't know how chronologically ordered we want to go,
Matt Lea: but I just jump it back a step.
Matt Lea: Terraform or CloudFormation.
Matt Lea: I highly recommend.
Matt Lea: So if you're just getting into it, a lot of times, just like code, right?
Matt Lea: You always want to have your code in some type of version control.
Matt Lea: You don't want to just have it FTP or saved to your disk.
Matt Lea: And you're just like, oh, you know, it disappeared.
Matt Lea: I don't know when Bob over here, you know, other developer X made this change.
Matt Lea: You want all that in version control.
Matt Lea: And you can do that infrastructure as code known, you know, which is Terraform, Bintofu,
Matt Lea: and CloudFormation is actually AWS's proprietary one that they made themselves.
Matt Lea: And so I'd recommend you get in that habit because it's very difficult to get into that later.
Matt Lea: Once you're you've established your infrastructure, you've all hand spun it up and you're like, how did I do that again?
Matt Lea: What was that? What would what is such and such that left the company?
Matt Lea: And now, you know, so you're basically describing me 10 years ago.
Matt Lea: I was there. I didn't I didn't get on that game till I was a couple of years after they had cloud formation launched.
Matt Lea: At first, I was like, this could be cool. And then I was like, OK, this is essential.
Michael Kennedy: Yeah. Yeah. I just, I used to be just like, I'm going to SSH in and I'll just get it working.
Michael Kennedy: And then, you know, the code will be well-structured, but just kind of the server,
Michael Kennedy: once I get it set up, it'll be fine. And then I realized actually, no, it's not fine.
Michael Kennedy: You know, you want to make any changes you want to move, you want to move from one setup to
Michael Kennedy: another. You're like, Oh gosh, I don't even like, this is a research project to figure out what to
Matt Lea: do. Yeah, exactly. But definitely when you're using those, those terraforms, always read your,
Matt Lea: what you're about to apply.
Matt Lea: I've had too many people that just auto apply
Matt Lea: the Terraform changes.
Matt Lea: And what happens is they'll have a slight renaming
Matt Lea: of a database.
Matt Lea: This actually happened recently.
Matt Lea: Somebody had their AI vibe code, this thing,
Matt Lea: and it wanted to rename the database.
Matt Lea: Well then what happens is Terraform deleted the database
Matt Lea: and then spun up another database right there.
Matt Lea: That was their production database.
Matt Lea: That was a rough day.
Matt Lea: That wasn't the project I have.
Matt Lea: I did a write up on it, but it wasn't my project.
Matt Lea: I was always read your replies though, basically.
Matt Lea: Always know what you're about to send.
Matt Lea: Make sure you think it's an update.
Matt Lea: It's got to be an update, not a delete and replace.
Michael Kennedy: This is something that has certainly been in the zeitgeist lately.
Michael Kennedy: It's a little bit, I don't hear about it as much.
Michael Kennedy: I think it's sort of to be dealt with.
Michael Kennedy: But there's certainly horror stories of the AI deleted the production database
Michael Kennedy: because kind of like we opened with like,
Michael Kennedy: instead of trying to just solve the problems, like, is there a better way?
Michael Kennedy: Like, well, there's a problem inserting to the database
Michael Kennedy: because the structure is wrong.
Michael Kennedy: So if we just recreate the database with the right structure, this code will work, right?
Michael Kennedy: It's like, well, that technically will solve the problem, but that is not acceptable.
Matt Lea: I've heard of it.
Matt Lea: They didn't like the schema.
Matt Lea: So they said, okay, drop the table and I'll create a new one.
Michael Kennedy: Yeah, I've literally had that happen to me only in my dev machine where it doesn't really matter.
Michael Kennedy: I'm like, why are there no records?
Michael Kennedy: Where's all the data?
Michael Kennedy: Oh, it was better for like, no, we never do that.
Michael Kennedy: But I'm not giving my AI access directly to my production database.
Michael Kennedy: No, thanks.
Michael Kennedy: No, I would not recommend that.
Matt Lea: I mean, as we're seeing more requests for that from like as C-suite people become more functional with these agents,
Matt Lea: they're like, well, I want direct access to the database so I can get the latest in this and that.
Matt Lea: It's like, hold on, let me code you up a tool call, get your read access.
Matt Lea: And if you want to do any right, they'll code up another tool call that'll make it so it's approved by a human.
Matt Lea: But doesn't that, you know, they could say, hey, I suggest this.
Matt Lea: And then, you know, but we've seen a lot of that lately.
Matt Lea: I mean, it's great that we're empowering more people to interact with the data and do more with it.
Matt Lea: You just, we still got to be careful about InfoSec and these things do make mistakes occasionally.
Matt Lea: Yeah, yeah.
Michael Kennedy: They work in a narrow context and let's make this code run.
Michael Kennedy: Like, oh, this will fix it.
Michael Kennedy: Like, we need the whole company to survive.
Michael Kennedy: No, I really love your suggestion of maybe creating like an MCP or a tool that's read only and make all the AI access go through that.
Michael Kennedy: You can observe it and check it out, but you don't have even the capability to destroy it through this mechanism, right?
Matt Lea: Well, there's even an extra layer to that is I just had a post come out about this because I had this request come through is put it pointed at your data lake if you can, not your production database, because those agents can spam like no tomorrow.
Matt Lea: So if they write a very complicated read query that goes through your tool call and they spam it could slow down your production database if it's not, you know, quickly optimized.
Matt Lea: So what we did to make it even more effective is our data lakes.
Matt Lea: A lot of times you don't want to store personally identifiable information and data lakes and all that stuff.
Matt Lea: It's anonymized.
Matt Lea: You can do big data regression on it.
Matt Lea: And so that way, even if it gets data that possibly be personally identifiable, it would be in your production database.
Matt Lea: The data lake theoretically wouldn't have that if you're doing it right.
Matt Lea: And so they couldn't, even if they leaked all your sales data for the last five years, at least you wouldn't have a PII leak, which would cost you even more.
Matt Lea: So that's my recommendation there.
Matt Lea: And I didn't actually, for the outline for today, I didn't talk about data lakes, but I can tell you my favorite data lake stack on AWS, but it's an advanced one.
Matt Lea: I'd save that for number 14, maybe.
Michael Kennedy: All right, maybe we'll get to that at the end.
Michael Kennedy: Hey, one second.
Michael Kennedy: Normally, this would be an ad break from Sentry, but not this time.
Michael Kennedy: Let's just thank them for supporting the show and get right back to the conversation.
Michael Kennedy: Also, visit talkpython.fm/sentry after the show.
Michael Kennedy: Thanks, Sentry.
Michael Kennedy: Okay, well, let's put a bit of a stake in the ground here and put down the first item as don't hand provision, use Terraform.
Michael Kennedy: And as I was getting ready for this episode and go through all the stuff we kind of like we'd talk about, I realized that this is acronym packed, an acronym dense and tooled as things.
Michael Kennedy: So maybe you could help us along the way.
Michael Kennedy: We've got Terraform.
Michael Kennedy: And yeah, just maybe a quick summary on this concept.
Michael Kennedy: Like where we use it and how we use it.
Michael Kennedy: We've been talking around it, but not as much as an item.
Matt Lea: Safe to assume most of the people listening to this are coders.
Michael Kennedy: 100%.
Michael Kennedy: Data science coders, yeah, or learners, but, you know, they're still good for them.
Matt Lea: Sure.
Matt Lea: So it's Terraform or IOC infrastructure, IAC, sorry, IAC, infrastructure as code is the category.
Matt Lea: And then under that category, there's Terraform, OpenTofu, and CloudFormation.
Matt Lea: And those are all just ways to define how you want your infrastructure to look basically in code.
Matt Lea: So if I was real simple, I could say I want an EC2 instance, which is...
Matt Lea: just your basic server, virtual server running on Amazon.
Matt Lea: And then I want a database, a MySQL database, this version.
Matt Lea: And I can put those in code in some way.
Matt Lea: And then, like I said, version control it, or I can hit apply.
Matt Lea: So say somebody accidentally deletes my application layer.
Matt Lea: A lot easier to replace than a data layer, of course.
Matt Lea: And I need to spin that back up real quickly.
Matt Lea: You just re-hit Terraform apply, and it re-spins up and provisions the missing area.
Matt Lea: It's really nice, especially for my cloud war games,
Matt Lea: I'm blowing up stuff all the time, intentionally being the, you know, junior that accidentally
Matt Lea: deletes something important.
Michael Kennedy: A little bit of a code monkey or chaos monkey, rather, running loose kind of thing.
Matt Lea: Yeah, cloud demolition expert.
Matt Lea: I think I had that as my title on LinkedIn for a while.
Matt Lea: I love it.
Michael Kennedy: Okay.
Michael Kennedy: Does Terraform do incremental updates?
Michael Kennedy: Like, let's suppose I've got an EC2 machine and I've got it all set up and it's running
Michael Kennedy: the app, the database.
Michael Kennedy: And I'm like, oh, you know, it would be really great if we put these firewall rules specifically
Michael Kennedy: on the machine or we set up fail to ban or something like that you can define every firewall
Matt Lea: rule which is perfect you wouldn't be able to define as easily what's in the box so to say
Matt Lea: there's you have some control over that um like for example i spin up lambdas and a lot of times
Matt Lea: i have a hello world javascript i throw in there and it's but i also again these are all for free
Matt Lea: on my website if you want them but um it also spins up the build pipeline and everything like
Matt Lea: that so then you can just hook that up to GitHub but at least it throws in a hello world lambda
Matt Lea: the code. But yeah, you can define all the stuff we're about to talk about today. You can define
Matt Lea: every user role, every access key, every username, the exact way you want your networks to talk.
Matt Lea: So the way your VPC, a virtual private cloud, has its subnet set up, all that stuff can be defined
Matt Lea: in there. It's almost crazy what can be defined. I wouldn't recommend committing your code and
Matt Lea: having it go through there. I'd have that go through a build pipeline separately that you'd
Matt Lea: spin up with terraform or whatever but you'd also you just pull from GitHub sure especially for the
Michael Kennedy: compiled languages and like go or something where there's no reason to deploy the source with python
Michael Kennedy: it's almost like not much of a difference although ci is still a thing okay so number one don't hand
Matt Lea: provision use terraform set standards stick to them i love it yeah it's it's getting that habit early
Michael Kennedy: it's it's tough to get into later number two act don't use access keys use iam roles so again
Michael Kennedy: acronym please so access access keys so i am is identity access management manager what it's it's
Matt Lea: how you define how your users interact with aws as well as your different running services on aws
Matt Lea: and so a lot of people when they first start out they download their first access keys they ideally
Matt Lea: don't put them in version control where it can be passed around or shared or anything like that
Matt Lea: they do a nice dot env and hope their agent doesn't steal it um but uh you know you get so you get those
Matt Lea: access keys but a lot of times when they push things to production if they're very beginner at
Matt Lea: this they'll put like a.env file in the code or something like that that's got access keys
Matt Lea: and those if those get leaked you know again I've got a story about this that somebody actually
Matt Lea: pushed their access keys for sending email luckily luckily it was just sending and receiving email
Matt Lea: and they pushed it up to a public repo as a junior and they just made a mistake there and it ended up
Matt Lea: on GitHub and some bad guys got it. And I believe we sent 16 million emails for some type of knockoff
Matt Lea: Viagra in about eight minutes before AWS shut down the account and said, what are you doing? It was a
Matt Lea: lot more than we normally said, but that's an example of why you don't want to hard.
Michael Kennedy: Yeah. I have a thing to share. Have you seen skit? Like, no, it's, it's the pronunciation is perfect
Michael Kennedy: of what it is, but it's like secrets, S-H-H, get, but get like, oh God. And it used to be a website.
Michael Kennedy: It's no longer here. Let me see if I can open up this image in a full screen for us to see.
Michael Kennedy: So it was a website that would scan the real time fire hose of changes on GitHub, pull out all
Michael Kennedy: cryptographic looking things, AWS keys, you know, email keys. And it just had this stream going and
Michael Kennedy: it updates, I don't know, like once a second or more. It's absolutely terrifying. And so anytime
Michael Kennedy: that someone's out there listening and you need to make this point to juniors or people who are new
Michael Kennedy: or whatever, like do not commit stuff to get it. Like, well, it's a, you know, I can just
Michael Kennedy: uncommit it. It's not a very popular repo. How fast would people find it? Like this fast.
Michael Kennedy: Fast enough. Yeah. If it's up there, you should assume it's gone immediately. And so, yeah,
Michael Kennedy: That's pretty scary.
Michael Kennedy: You know, I don't know if you tracked this, probably not,
Michael Kennedy: but at PyCon this year, there was a talk by Tristan McKinnon
Michael Kennedy: called Zero Trust in 200 Milliseconds.
Michael Kennedy: Now, I didn't watch the talk, so I'm only going off the abstract.
Michael Kennedy: But basically, the idea is every time you need to make an API call
Michael Kennedy: or do a transaction or something,
Michael Kennedy: you can get a separate 200 millisecond duration key.
Michael Kennedy: What do you think of that idea?
Matt Lea: I like it.
Matt Lea: 200 milliseconds is interesting if that's a set one or a variable one.
Michael Kennedy: Yeah, I'm not sure.
Michael Kennedy: It might be a little bit shorter, a little bit longer.
Michael Kennedy: In case there's a little lag, you don't want to pull three.
Matt Lea: I also immediately wonder how long does it take you to provision that key?
Matt Lea: Hopefully it doesn't take you 200 milliseconds to provision it,
Matt Lea: but if you could provision it in two milliseconds.
Matt Lea: I don't like, what do they call it, chocolate candy infrastructure
Matt Lea: where once you pass the outside, then it's free once you get inside.
Matt Lea: I forget the exact term there, but you've got to be careful.
Matt Lea: even once you're past the infrastructure,
Matt Lea: the external infrastructure,
Matt Lea: once you're in there,
Matt Lea: you don't want to have it.
Matt Lea: So anybody's got free reign
Matt Lea: to grab whatever they want.
Matt Lea: They might get that.
Matt Lea: So I like it.
Matt Lea: My other question though,
Matt Lea: we always have to balance is,
Matt Lea: what's the latency effects?
Matt Lea: If that costs 200 milliseconds of latency,
Matt Lea: that's pretty expensive
Matt Lea: from a compute standpoint.
Matt Lea: So
Michael Kennedy: Yeah, I'm not entirely sure.
Michael Kennedy: But just a little more background about this.
Michael Kennedy: This is from FedRAMP,
Michael Kennedy: which is the US government's,
Michael Kennedy: they use this on a lot of the US government
Michael Kennedy: internal protection like cloud or whatever, which is maybe a special case.
Matt Lea: That makes a lot of sense. And I wager the government's latency limitations are a lot
Matt Lea: less than someone trying to be competitive in e-commerce or something like that.
Michael Kennedy: Yes, exactly. You know what? They can wait for security.
Matt Lea: Yeah. It's like the DMV online. Just stand in line all day. They'll get there. There's nowhere
Matt Lea: else to go.
Michael Kennedy: It's like just get in line with all the other poor browsers and wait back there for your turn and
Michael Kennedy: we'll get to you eventually. Unless we closed at four. Unless we closed at four. Okay. So
Michael Kennedy: Basically, don't use access keys.
Michael Kennedy: Use IAM roles.
Michael Kennedy: How do we make this happen, though?
Matt Lea: So IAM, you go in there, there's users, there's groups, which makes a lot of sense.
Matt Lea: The roles are things you assign to the various services you boot up.
Matt Lea: So you can have a role for the EC2 instance or a role that the Lambda has.
Matt Lea: So when your Lambda executes, it's acting as if it has the same permissions that this role has.
Matt Lea: So you could say, hey, I want this Lambda to be able to access this one specific S3 bucket.
Matt Lea: you know, and it will basically know these are the permissions it has just because you've assigned it that role.
Michael Kennedy: I see. So maybe it can only read from the database and can only write to a certain bucket and that's it.
Matt Lea: It's not network level, it's service level. There's a separate one we'll go over in a minute for network level, how you restrain a network.
Matt Lea: It's more what services can it call on AWS? Can it boot up instances? Can this instance boot up other instances?
Matt Lea: You don't typically want that.
Matt Lea: You know, in this instance, you know, S3 is right.
Matt Lea: Can it invoke Lambdas?
Matt Lea: Can it this, you know, Lambda trigger other Lambdas?
Matt Lea: In some cases that might make sense.
Matt Lea: In some cases that could just be very expensive.
Matt Lea: Yeah.
Matt Lea: But a lot of it, like you could theoretically give a Lambda the ability to boot up or provision
Matt Lea: hardware.
Matt Lea: I don't think you really want to.
Matt Lea: Maybe if you were running a hosting company, but maybe not.
Matt Lea: But it's not a million though.
Matt Lea: Yeah.
Matt Lea: Yes.
Matt Lea: It's a service to service, not network level permissions thing.
Michael Kennedy: Okay.
Michael Kennedy: I see. So that's right. We're talking, IAM defines what it can do within AWS as infrastructure,
Michael Kennedy: the service type of thing, or code as infrastructure, not what can it do within
Michael Kennedy: your database row level type of things. That's, that's correct. Yeah. The database engines,
Matt Lea: you know, wouldn't specify that Dynamo is a little bit of a Dynamo's serverless has got invokes.
Matt Lea: It's not like something like a Redis or, you know, or MySQL where it expects traffic on a
Matt Lea: certain port, you know? So there's some of their proprietary stuff. It actually does have a little
Matt Lea: extra power there.
Matt Lea: But when they're doing open source stuff and they can't really change the
Matt Lea: innards of it, then it just says, can you start a new one or stop this one?
Michael Kennedy: That's pretty much it.
Michael Kennedy: Okay.
Michael Kennedy: And following on from there, don't use generic IAM permissions.
Michael Kennedy: Use granular ones.
Michael Kennedy: Like, just let it do everything.
Michael Kennedy: That'll be easier.
Michael Kennedy: It works.
Michael Kennedy: That could trade the permission errors.
Michael Kennedy: Oh, yeah.
Matt Lea: So let's go back to that email one.
Matt Lea: What if they said, let it do anything on any service right there?
Matt Lea: They could have deleted everything in a second.
Matt Lea: people are like, okay, that makes sense. But what, what I see is for convenience sake, you're, you're
Matt Lea: banging your head against the wall. And all of a sudden you're like, you know what, we're just
Matt Lea: going to let them access all the S3 buckets. And next thing, you know, a couple, you know,
Matt Lea: months later or whatever, they, they've now got your internal reporting and they pull that out,
Matt Lea: you know, it's like, nope. So don't ever do the asterisk is your enemy. Basically S3 colon asterisk
Matt Lea: gives you all S3 permissions. You know, you could delete that bucket right there and then it's down.
Matt Lea: So avoid using the asterisk whenever possible.
Matt Lea: Same thing with resources.
Matt Lea: So it's not just permissions, but it's also the resources.
Matt Lea: Like I said, you've got 20 S3 buckets.
Matt Lea: Specify only the ones you want to write to.
Matt Lea: You can actually specify down to the path if you want to.
Michael Kennedy: Sounds like if you use the asterisk,
Michael Kennedy: it sounds like a good way to end up on the front page of a computer
Michael Kennedy: or somewhere you don't want to end up on.
Matt Lea: Yeah.
Matt Lea: So, I mean, if you're experimenting for 10 seconds
Matt Lea: and you're going to delete it,
Matt Lea: I mean, I wouldn't commit it into version control.
Matt Lea: Let's put it that way.
Matt Lea: You want your version control,
Matt Lea: if you're like accidentally poke a hole for 10 minutes,
Matt Lea: to be like, okay, how is this not working?
Matt Lea: You want your version control, Terraform, IIC to wipe that out the second you're done with it.
Matt Lea: Makes sense.
Matt Lea: Probably not do that in production.
Michael Kennedy: Yeah.
Michael Kennedy: It's worth repeating a couple of times.
Michael Kennedy: You know, it sounds to me like you're pretty savvy with the agentic stuff and so on,
Michael Kennedy: given your tool calling and things you were talking about.
Michael Kennedy: I feel like five years ago, granular permissions and things like that were really tricky because
Michael Kennedy: you're just like, ah, what do I need?
Michael Kennedy: Like so many boxes for allow or disallow.
Michael Kennedy: Now with agents, I feel like you can just say, look, I need to do this with this service to that service.
Michael Kennedy: Probably be pretty accurate.
Michael Kennedy: What are your thoughts on using that for help?
Matt Lea: Yeah, it depends.
Matt Lea: So just imagine you're not an extremely technical person and you tell the agent, listen, I just need this to talk to that.
Matt Lea: They can achieve that by putting that asterisk in there.
Matt Lea: So, you know, so that's an interesting one where I've seen them to accomplish their goal.
Matt Lea: They don't exactly go about it the way you'd want them to.
Matt Lea: But I've also seen it counteracted by people using multiple agents.
Matt Lea: And AWS does offer its own DevOps agents and FinOps agents that are going to put me out of a job and all that.
Matt Lea: But I would say, I mean, this is kind of, so I think I mentioned beyond Cloud War Games, I'm writing a game that's going to be on Steam.
Matt Lea: And the goal is to make sure that people with the least amount of technical skills understand all the bad things that can happen.
Matt Lea: You know, because you might just, right now we're seeing with the agents is that these people are getting incredible half hours.
Matt Lea: But for 30 years, I've been programming.
Matt Lea: And some people, it looks like Wizardry.
Matt Lea: Now they've got that power.
Matt Lea: They don't know the vulnerabilities.
Matt Lea: It can go really poorly for you.
Matt Lea: And so the agents, I mean, it can be great.
Matt Lea: But man, you got to know what a cross-site scripting attack is.
Matt Lea: Because if it's not validating that, what is a MySQL injection attack?
Matt Lea: And it's rocky.
Matt Lea: So it's great that they can speed you up and make you more efficient.
Matt Lea: But make sure you're double-checking it to work, is what I guess I've got to say.
Michael Kennedy: Yeah, and as I'm hearing you talk, I'm just thinking,
Michael Kennedy: this is just one more reason that being an actual expert
Michael Kennedy: and using these tools is so much more powerful
Michael Kennedy: than just trying to vibe your way through it.
Michael Kennedy: So the using the star thing, that is the easiest way to solve it,
Michael Kennedy: and that might be what the agent does.
Michael Kennedy: But you could say, and your primary goal
Michael Kennedy: is to be absolutely least privileged about this thing.
Michael Kennedy: And so you're going to win this session
Michael Kennedy: by getting the least privilege that you possibly can now go.
Michael Kennedy: Like that would make all the difference, I imagine.
Matt Lea: I would think having competing agents,
Matt Lea: one that's the coder and you give it that instruction,
Matt Lea: but then have a second one do a peer review on it
Matt Lea: might be of some use to be like,
Matt Lea: hey, is there any possible way to break this?
Matt Lea: Just one agent alone, I don't, I'm here.
Matt Lea: From my experience, it'll just say,
Matt Lea: yeah, sure, I did it sometimes.
Matt Lea: Yeah, yeah, yeah, 100%.
Matt Lea: Having competing more skeptical agents seems to help,
Matt Lea: but if you get the part, you know.
Michael Kennedy: Right, ask for an adversarial review or something like that.
Michael Kennedy: Yeah.
Michael Kennedy: All right. Don't put back-end resources on public subnets.
Michael Kennedy: Do you need your database on the internet just open?
Michael Kennedy: Probably not.
Matt Lea: Exactly. This is one of the things where it's not convenient because people are like,
Matt Lea: well, I just want to be able to hit my database for my local machine.
Matt Lea: Well, you still can. You've got a bastion, if anybody's familiar with that.
Matt Lea: I can explain that in a second.
Matt Lea: But let me zoom out to, there's a thing called a virtual private cloud.
Matt Lea: It's basically inside your AWS account.
Matt Lea: It's a way to kind of siphon off resources.
Matt Lea: Some people, personally, I like having a prod AWS account,
Matt Lea: the staging AWS account and the test.
Matt Lea: And that way you're all separate.
Matt Lea: I'll be honest, for my own stuff, I actually have separate VPCs instead.
Matt Lea: So inside of that, you've got your groups of subnets.
Matt Lea: So this is tough to do without diagramming it right now.
Matt Lea: But you know that there's US West, right?
Matt Lea: There's US West 1.
Matt Lea: And that is a giant bunker in the ground.
Matt Lea: Inside of that, there's about six availability zones.
Matt Lea: And so each one of those availability zones is its own bunker inside that facility that
Matt Lea: has completely independent power supply and all that.
Matt Lea: So if one of those gets new, you still got three others or four or five others in that region.
Matt Lea: Honestly, they keep adding them. So I don't know what they're at exactly.
Matt Lea: But and so inside of those, you want to have subnets.
Matt Lea: Now, a lot of times people put everything in big subnet.
Matt Lea: That's not good. You know, if you just have one subnet on one availability zone in one region, that's not reliability.
Matt Lea: That's not redundancy. And those availability zones independently go down actually fairly frequently.
Matt Lea: But you don't see it if you use multiple availability zones or if you got the budget for it, multi-region, which has actually come down and cost quite a bit.
Matt Lea: So I guess I'm going to split this one into two parts.
Matt Lea: One, use multiple availability zones whenever possible because it'll give you a lot more uptime.
Matt Lea: Two, I always have it so there's a public subnet, which is things that should be accessible from the Internet.
Matt Lea: OK, that'd be the load balancer.
Matt Lea: A lot of times the load balancer takes in traffic and then sends it to various instances running in the various AZs.
Matt Lea: So that's always out there. But then privately in the private subnet, you want to have your databases and you're like, OK, well, I should probably should have done the next one before this because you can firewall these things.
Matt Lea: But man, it's so easy to poke a hole in those firewalls when it's convenient.
Matt Lea: So a lot of times I'll come back and do an audit and something that's like poke the hole here so you can get traffic from anywhere on port 22.
Matt Lea: You shouldn't have that going on.
Matt Lea: You want to only have, you know, so just adding that, having a private subnet specifically for that, for the databases, for anything secure,
Matt Lea: it just adds an extra level of inconvenience that even a valid user who might have been compromised accidentally,
Matt Lea: like you installed a wrong game on your computer, you know, and it grabs your SSH keys and pokes through the security wall,
Matt Lea: and it pokes through the firewall because someone put it on 00, basically allowed the whole world in on that port.
Matt Lea: Well, now they still can't get to it because there's not actually a network path that would go from point A to point B.
Michael Kennedy: So it's just another way of adding more security to it.
Michael Kennedy: And listening on the private cloud addresses versus listening on just 000, you know, like certain things like that are definitely worth considering, right?
Matt Lea: Yeah, well, yeah, that makes sense.
Matt Lea: I should have put this one before the next one.
Matt Lea: Let's jump to the next one and I'll kind of circle back on this one.
Matt Lea: I got a little backwards here.
Michael Kennedy: No worries.
Michael Kennedy: So this one?
Michael Kennedy: Well, we went one too far.
Michael Kennedy: One too far.
Michael Kennedy: There we go.
Michael Kennedy: Don't use one security group for everything.
Michael Kennedy: You're smart.
Matt Lea: Yes.
Matt Lea: So security group is how we limit network traffic.
Matt Lea: Think of that as firewall.
Matt Lea: Okay.
Matt Lea: This EC2 instance, this Lambda, whatever it is, can access this other thing, this database
Matt Lea: on 0.336, whatever, you know.
Matt Lea: So that says that only those two things can talk.
Matt Lea: It would be a security group rule.
Matt Lea: So what you have is you assign a security group to your Lambda and a security group to the
Matt Lea: database.
Matt Lea: But a lot of times people will just say anything and all these security groups can take traffic
Matt Lea: other one because you just have security group a and anything in security group a can take traffic
Matt Lea: over any port in there so now you've got potentially you know http traffic uh getting that could access
Matt Lea: something else or hit you know hit your database on a port that's not supposed to be hit you know
Matt Lea: so you've got to be very particular with those security group rules um because you don't want it
Matt Lea: so someone that say say you've got a box an ec2 instance and a nefarious party somehow get something
Matt Lea: on there even something the npm remote exploit stuff that's been going around a lot i don't know
Matt Lea: if you've heard about that but oh yeah let's just say you install it in your application layer now
Matt Lea: that application layer do you want it just to be i mean it's all gonna have to talk to your database
Matt Lea: they'll get access to that that sucks but let's just say that that happens in test at least this
Matt Lea: one you know if you had the security group set up right and the subnets and everything it couldn't
Matt Lea: access production you know so that would save you a little bit of headaches but you right don't want
Matt Lea: so they can crawl every IP within your subnet.
Matt Lea: You want that.
Michael Kennedy: Yeah, I mean, what gets your virtual private cloud network
Michael Kennedy: if once you're in, you get the whole thing.
Michael Kennedy: The term that comes to mind for me is lateral movement.
Michael Kennedy: It's a big thing in security, right?
Michael Kennedy: Once you're in, you find a way in.
Michael Kennedy: Well, now where can you go, right?
Matt Lea: Think about that.
Matt Lea: Exactly.
Matt Lea: Lateral movement is a perfect term.
Matt Lea: You also can limit your outbound as well.
Matt Lea: So you can limit where the box can talk out to.
Matt Lea: So if you really want to get secure,
Matt Lea: lock that down as well.
Matt Lea: If only this IP for this one third-party API vendor, that's it.
Matt Lea: They can't go reach out to somewhere overseas that we don't know about.
Matt Lea: So make sure you lock down the outbound if you need to.
Matt Lea: I don't know if I do that as a beginner day one, but if you're a big company, oh yeah.
Michael Kennedy: This portion of Talk Python is brought to you by Talk Python courses.
Michael Kennedy: Here's the thing that always bug me.
Michael Kennedy: You finish one of our courses, that's hours of video, a pile of code you actually wrote, and real skills you didn't have a month before, and then nothing happens.
Michael Kennedy: No paper, no credential, nothing to show for it.
Michael Kennedy: So we fixed it.
Michael Kennedy: Every Talk Python course now generates a completion certificate automatically.
Michael Kennedy: Go to your account page in your dashboard section,
Michael Kennedy: scroll down to your completed courses, and click Certificate.
Michael Kennedy: That's the whole process.
Michael Kennedy: Two things you can do with these course completion certificates.
Michael Kennedy: Download the full PDF, which is handy if your employer reimburses training
Michael Kennedy: or gives you credit for finishing it.
Michael Kennedy: Or you can make the certificate public and hit Share on LinkedIn,
Michael Kennedy: which adds it to your LinkedIn profile under licenses and certifications,
Michael Kennedy: not a poster that scrolls away in a day,
Michael Kennedy: an actual credential sitting on your profile where your manager and recruiters can see it.
Michael Kennedy: Plus, if you've been taking our courses for a while,
Michael Kennedy: you've probably earned several of these without even knowing they existed.
Michael Kennedy: Just visit training.talkpython.fm/account and collect them.
Michael Kennedy: Thanks to all of you who have taken a Talk Python course.
Michael Kennedy: It's a great way to support the podcast.
Michael Kennedy: What are your thoughts on overlay networks like TailScale and stuff?
Michael Kennedy: Instead of opening up a port, maybe put your employees on a tail scale network that you can see the into the database server or something, but nothing else.
Matt Lea: So in my terms, a lot of times there's Bastion.
Matt Lea: I think I've got it in here somewhere.
Matt Lea: Yeah, tell us.
Michael Kennedy: Yeah, yeah.
Michael Kennedy: Tell us about Bastion.
Michael Kennedy: We can go ahead and jump to that.
Matt Lea: So Bastion's before you boot up a Bastion in a public IP, a public network that could be accessed publicly.
Matt Lea: Ideally, you'd limit it using security groups and saying, hey, this is the only accept traffic on 22 from this IP address, you know, mine.
Matt Lea: And every time I relocate, I'd have to update the IP address.
Matt Lea: And then you'd SSH into that bastion, and then that would be able to then be your door to the rest of everything.
Matt Lea: So we still do port forwarding if you want to do local development, hit the test database or something like that.
Matt Lea: So I do that quite a bit, but they've created a new tool.
Matt Lea: I'm old. I've been doing this since 2010.
Matt Lea: That's called Secure Session Manager.
Matt Lea: it's the SSM and you can use that to enter in using IAM authentication and that's much more
Matt Lea: secure. So I'd consider something more like that. Also, you don't have the cost of running an EC2
Matt Lea: instance 24 seven act as your bastion. So that's a nice tool to have in there. I guess I didn't put
Michael Kennedy: that in. Yeah, no worries. Have you heard of Knock Knock? Knock Knock? No. So Knock Knock is this
Michael Kennedy: interesting thing that the firewalls are blocked like 100% for all the management. Let me see if I
Michael Kennedy: can get this right. So everything is blocked 100% for the IPs. But if you authenticate to a certain
Michael Kennedy: point, then the service opens up just for your IP address for the length of that session. You kind
Michael Kennedy: of like knock on a different way, then it'll let you through the firewall. It's the concept. I
Michael Kennedy: yeah that would be a good one to have in there i would think um so just firewall allow this these
Michael Kennedy: ip addresses or whatever you're like okay every time that port opens temporarily i need it to
Matt Lea: authenticate yeah it's kind of cool i mean that's actually a good feature request for aws to have
Matt Lea: uh durations on their their uh security group rules so you say i only want to do this for an hour
Michael Kennedy: you know right let's open this up so i can i can debug and diagnose this because i gotta get to it
Michael Kennedy: apparently but then just shut it back off without me requiring to like fail safe instead of fail
Michael Kennedy: Yeah, that could be a very useful feature.
Michael Kennedy: We should send that to you.
Michael Kennedy: Yeah, let's do it.
Michael Kennedy: Send it through the podcast.
Michael Kennedy: I'm sure some folks there are listening.
Michael Kennedy: Apologies to vegetarians for this next one, but don't nurse your EC2 instances.
Michael Kennedy: Cattle, not puppies.
Matt Lea: I didn't come up with the phrase, but it sums it up.
Michael Kennedy: No, I know, I know.
Michael Kennedy: It sums it up very well.
Matt Lea: So I'm sure you've, in your career, probably had a box somewhere where you were just nursing.
Matt Lea: I mean, very common before they did serverless.
Matt Lea: Or before they did virtual servers, I should say, when you actually had a rack.
Matt Lea: somewhere before aws and then you'd have to call a guy someone kicked over the plug you're in trouble
Matt Lea: but you'd end up i saw this even on aws with ec2 instances specifically i see it a lot where
Matt Lea: someone boots up a couple ec2 instances and they're just this is my baby i've got i've got it
Matt Lea: set up absolutely perfect and if one of those things gets sick or dies and you got to restart
Matt Lea: now you're sitting there like oh how did i how did we set this up before did we you know what
Matt Lea: scripts did we run all that stuff and with both lambda and ecs ecs so ecs is elastic container
Matt Lea: service and i don't know if people are familiar with docker but docker is a phenomenal tool where
Matt Lea: you can basically build the image it's almost like if we're going back in time you're building your
Matt Lea: os and saving it to a disk you just pop that disk and anytime it'll boot up the exact one so if you
Matt Lea: have to kill it 10 times a day you know your servers you if you had to delete your servers 10
Matt Lea: times a day you could spin them up exactly the same way they were in that perfect state when they
Matt Lea: when they first started taking traffic in a split second and it's it's really nice and same lambdas
Matt Lea: that with some simplicity to it um it's it's a lot like docker but it's not actually docker
Matt Lea: under the hood i found out um but it's it's the same thing your lambdas you know will scale
Matt Lea: infinitely especially if you want auto scaling which is again a huge advantage of the cloud
Matt Lea: you'd want to have this type of technology so you can if you're you know it's the middle of the night
Matt Lea: I only want two instances running, two containers, two tasks.
Matt Lea: It's the middle of the day on Thanksgiving, the day before Thanksgiving,
Matt Lea: the day after Thanksgiving, sorry.
Matt Lea: And now we need 100.
Matt Lea: It'll scale just perfectly.
Matt Lea: So be very weary of just SSH-ing in and customizing stuff.
Matt Lea: On EC2, it's just way too easy to do at first.
Matt Lea: It's tempting.
Matt Lea: Yeah, tempting.
Matt Lea: But then once you get to scale, you want to have those images built out
Matt Lea: that'll just spin up very quickly.
Michael Kennedy: 100%.
Michael Kennedy: long time I resisted using Docker because to me it felt like complexity paired with capacity like
Michael Kennedy: it's more complex to do Docker and also it's harder for me to observe and analyze it and what I
Michael Kennedy: really what I realized not too long after that is well all the commands you put in the Docker file
Michael Kennedy: it's like what you had to write into your server to make it do whatever it does anyway just put run
Michael Kennedy: instead of not run on the front you know copy versus not copy or whatever and then the observability
Michael Kennedy: really you can just map some volumes or go into the container like what you can still observe it
Michael Kennedy: pretty much just as well so i just want to echo your thoughts of like yeah absolutely look into
Matt Lea: docker if you if you want to go fast you know and uptime scalability reliability that's not an issue
Matt Lea: i'm not going to push it you know i i told i was i'm a startup guy i totally get it if you're
Matt Lea: a company that's goes down and loses a hundred thousand dollars an hour and uh you don't have
Matt Lea: the setup commands documented or you know your lead tech gets hit by a bus quote unquote you know
Matt Lea: you want that docker file to be clear as day so you can make modifications spin up whatever so
Michael Kennedy: 100 100 what about kubernetes what do you think on kubernetes versus docker i mean that's another
Michael Kennedy: level complexity i think another dev i respected i was on a podcast with said if you're if you're
Matt Lea: doing kubernetes on anything but gcp you're you're kind of it's not a good fit it's just you're kind
Matt Lea: of messing around. So they tried, AWS tried, and some people have done real well with this EKS.
Matt Lea: That's the Elastic Kubernetes Service, I believe is what it's called. And that is the ECS's brother
Matt Lea: that runs with Kubernetes. But it just, I've never had it feel natural. I've had clients multiple
Matt Lea: times have me pull them off of EKS to ECS just because ECS is a bit more AWS native and they
Matt Lea: are already on it. So I'm not going to say no, but most of the people I see have the best success
Matt Lea: with Kubernetes are probably using GCP. I have no problem with that. It's not my expertise.
Matt Lea: Yeah, that makes a lot of sense. For Docker, you know, a lot of times,
Michael Kennedy: it's running yourself, not necessarily in the container service on AWS. You could even do this
Michael Kennedy: on EC2 for sure. You know, like you can go a long ways with Docker Compose for like people
Michael Kennedy: startup things like my company. I got 33 Docker containers running on something at this point,
Michael Kennedy: different distinct ones.
Michael Kennedy: But anyway, there are other tools like Coolify.
Michael Kennedy: Have you seen this?
Michael Kennedy: I've heard of Coolify.
Michael Kennedy: I haven't dug in much.
Michael Kennedy: It's pretty interesting.
Michael Kennedy: So what you can do is you can sign up with Coolify,
Michael Kennedy: point them at any Linux server.
Michael Kennedy: It could be EC2, it could be DigitalOcean, Hetzner, whatever.
Michael Kennedy: And it will install a little agent back
Michael Kennedy: so they can sort of issue commands.
Michael Kennedy: And then it will manage all the Docker stuff.
Michael Kennedy: And it's got like a listing of different services you can pick,
Michael Kennedy: like thousands of, you know, like I just want to run Hemidol,
Michael Kennedy: the dashboard for organizing and managing my server applications
Michael Kennedy: or Hayform or whatever.
Michael Kennedy: And it'll just, okay, we'll just do whatever Docker things have to happen there.
Michael Kennedy: And it's a little bit more of I want to take a self-hosted type of thing
Michael Kennedy: or some kind of service and just stick it in there.
Michael Kennedy: I don't know.
Matt Lea: I can completely respect that too.
Matt Lea: There's part of me, trust me, there's a part of me that wants to go off
Matt Lea: and live in a cabin by the woods and have my own servers in there and everything.
Matt Lea: And, you know, self-host, it just, it depends, you know,
Matt Lea: from a business standpoint so a lot of times i work directly you know with the business side of
Matt Lea: people to coordinate with and from a tech side that's really cool from a business side you know
Matt Lea: well what if we someone kicks the power plug you know what if this that and the other like
Matt Lea: how are we going to maintain it you know and there becomes it just depends where you want to invest
Matt Lea: your money and your engineer's time and you could probably i mean you could host you could spin this
Matt Lea: up on a dot on a aws instance and not have to worry about all that stuff have it running in there
Matt Lea: but then the question is how many engineering hours is it going to take to maintain something
Matt Lea: like this versus a managed service?
Michael Kennedy: Even though it feels like it's managing it for you, it's still a bit of a puppy because
Michael Kennedy: you still got to do backups and it has a way to do it.
Michael Kennedy: But if something goes wrong, you still got to, you, it's really down to you to like,
Michael Kennedy: well, you upgraded and it corrupted the database.
Michael Kennedy: So now what, you know?
Michael Kennedy: Yeah.
Michael Kennedy: Something weird.
Matt Lea: I'm self-hosting N8N right now.
Matt Lea: And it keeps like, oh, you got to do another update.
Matt Lea: You got to do another update.
Matt Lea: How much of my time could I save if I just had N8N on their native platform?
Michael Kennedy: I know.
Michael Kennedy: I tried it in for a while and it absolutely was just,
Michael Kennedy: eventually became a no for me.
Michael Kennedy: So I've switched over to Hermes,
Michael Kennedy: which is just ironically on the screen right here,
Michael Kennedy: which is so good.
Michael Kennedy: But it's like a sort of self-improvement.
Matt Lea: Oh, I've got Hermes.
Matt Lea: I've got it and used it with mixed results.
Matt Lea: It made me a little angry when I found it was resizing the context.
Matt Lea: It has its own custom AI for changing the context window.
Matt Lea: And so I was losing my context.
Matt Lea: And then this is more of the model's fault.
Matt Lea: The model, I could see the memories it was putting in it.
Matt Lea: You know, it's trying to save in plain text.
Matt Lea: And it was saying the user gets agitated when I,
Matt Lea: it's like a user gets agitated when you hallucinate.
Michael Kennedy: Yes, but.
Matt Lea: Yeah, exactly.
Matt Lea: When you're wrong.
Michael Kennedy: It's like a user is agitated.
Michael Kennedy: Yeah, I have codex and 5.6 soul back in mind.
Michael Kennedy: So it's pretty smart.
Michael Kennedy: All right, let's talk logs.
Michael Kennedy: Because I think when you go away from the single server,
Michael Kennedy: this starts to be one of the things you worry about is like,
Michael Kennedy: well, it used to be,
Michael Kennedy: I could just look at the logs on the machine.
Michael Kennedy: And now they're all over these different places.
Michael Kennedy: What do you say about that?
Matt Lea: It takes some getting used to, but it can be your best friend.
Matt Lea: They have incredible tools.
Matt Lea: I was just chasing down a big DDoS attack from one of my biggest customers.
Matt Lea: Almost three times the amount of our normal traffic.
Matt Lea: It was brutal.
Matt Lea: And we're trying to track down like a needle in a haystack there for some other stuff as well.
Matt Lea: And they've got really powerful tools.
Matt Lea: CloudWatch Insights allows you to query those.
Matt Lea: Now, you can spend a lot of money on that.
Matt Lea: Be careful.
Matt Lea: I had a, we taught a guy how to do queries and CloudWatch insights.
Matt Lea: Well, they, it's a, it's a serverless type thing.
Matt Lea: It just charges you for every byte it ingests or the log.
Matt Lea: So if you tell it to go back five years and read through all the logs,
Matt Lea: you're going to have a pretty decent bill there.
Matt Lea: So don't do that.
Matt Lea: Be careful.
Matt Lea: Get your queries figured out in the five minute range and then run them,
Matt Lea: you know, so you're not debugging query over and over again with a long date range.
Matt Lea: But the, and the metrics are super powerful as well.
Matt Lea: You can set logs or you can, they pipe in pretty much everything from your services.
Matt Lea: I mean, I couldn't even try and label all of them, but the obvious ones are CPU usage, memory usage, latency on requests for your load balancers, volume of requests, et cetera.
Matt Lea: So you can set up these really beautiful dashboards that help me.
Matt Lea: When a client comes to me and says, hey, we've got an extra X seconds of latency spiked on this day, I can go in there and zoom in real deep and just go through layers and go all the way down to the database and be like,
Matt Lea: okay, this database didn't have enough swap or, you know, this Redis eviction shot through the sky,
Matt Lea: all that stuff. So it takes a while to find it, but it pays off so much. And this also goes back
Matt Lea: to the Lambda logs. So say you set up everything on EC2, it's not going to pump everything that's
Matt Lea: on EC2 automatically to those logs, right? You'd have to specify it. It pumps some base stuff.
Matt Lea: But with an EC2 or sorry, an ECS task, it'll pump everything that's getting that you'd see normally if you ran it from Docker, you know, Docker and same thing with Lambdas.
Matt Lea: And so just as long as you're not telling some process to write to disk and some log file, which can happen, you know, then you'd have to tell it to pipe that out to CloudWatch logs.
Michael Kennedy: So you're saying like if I basically did Docker or Docker Compose Logs-F type of thing where it streams out whatever's coming out of the container, it'll just send that over to Watch?
Matt Lea: And those are all by default, super easy to set up.
Matt Lea: Just the only problem would be if you said, hey, application layer, write this secondary log to this file.
Matt Lea: And that wasn't getting piped out somehow.
Matt Lea: I've successfully had it write to a file separately and then told the Docker process to pipe that out to be a tail.
Matt Lea: That was the end of the Docker run command was tail.
Matt Lea: It's a little sloppy, but you can do it.
Matt Lea: A better way to do it would be to use AWS CLI commands
Matt Lea: or some of the AWS SDK to pump it to a CloudWatch log stream.
Matt Lea: And then you'd have that ready to go.
Matt Lea: And a lot of my clients do that.
Matt Lea: It's a little cleaner than the first one.
Michael Kennedy: Sure.
Michael Kennedy: Another option might be set up two destinations for the logging
Michael Kennedy: because you can set multiple, you could say this file and standard out
Michael Kennedy: and just let it rip.
Michael Kennedy: And then just that becomes part of the Docker logs.
Matt Lea: Yeah, exactly. So just, you know, it's pretty simple once you get in there. Talked about insight. But yeah, the metrics. Another thing about metrics is you can set alarms, not just to wake you up if something bad's happening, but also to trigger auto scaling, which is really nice. And so you could scale up and down based on not just. We had something. We were pulling from queues. We had a worker. It was not serverless. It was provisioned. And we had a worker. And when the queue got too high, you know, got too big, we wanted it to boot up another one.
Matt Lea: one but that's normally if you just use a default one it's just like cpu or memory or something like
Matt Lea: that but we were able to configure it so it would pull from the count of messages in flight and use
Matt Lea: that to auto scale up and down and that was really convenient that wasn't that was a client request
Michael Kennedy: for that one and that's cool yeah it seems like it makes way more sense than just cpu yeah because
Michael Kennedy: maybe you've got some process that spin off like an analytics thing pin in the cpu but it's not
Michael Kennedy: really that busy right and scaling won't make a difference but if you've got something like maybe
Michael Kennedy: average response time if it gets too big.
Michael Kennedy: Just scale up.
Michael Kennedy: If it's over 300 milliseconds on average or median,
Michael Kennedy: we just need more.
Matt Lea: Yeah, and you can pop all that stuff into your chat of choice.
Matt Lea: We've got all that for my clients.
Matt Lea: It goes right into a channel and on-call people jump on it.
Matt Lea: I don't do on-call work anymore.
Matt Lea: I just set it up so I train the new guys to do it.
Michael Kennedy: Yeah, that's the way to do it.
Michael Kennedy: Tell me in the morning what you did, what went wrong.
Michael Kennedy: Yeah, excellent.
Michael Kennedy: Okay, so that's logs.
Michael Kennedy: I feel like people probably got this.
Michael Kennedy: don't hand deploy, but also a little bit of a, well, then what? You know, CI/CD, right?
Matt Lea: Yep. So I'm going to go take a time machine back about 20 years, a long time ago, we would
Matt Lea: basically FTP up, you know, stuff from, from our computers to a server or something like that.
Matt Lea: And then it would run. So if you didn't, depending on how your deployment was,
Matt Lea: kind of have to hand go through it. Well, nowadays people want to be moving super fast.
Matt Lea: So we've created these systems that you just commit it gets approved from the pull request
Matt Lea: from whomever. And then we rebuild the image like we talked about. If it's Docker, we rebuild the
Matt Lea: image. If it's Lambda, we build whatever and upload the file as we need. But you also can do database
Matt Lea: migrations as well. I've got clients that use MySQL and their table updates all run through there.
Matt Lea: So you can have that all run there. You can have automated tests, test-driven development. If that
Matt Lea: is something that suits your need, I'd encourage it for anybody that's big enough and has something
Matt Lea: They don't want to go down.
Matt Lea: And if you're just starting, I'm a little looser on it.
Matt Lea: But having those build pipelines in place saves you a ton of time.
Matt Lea: And of course, if they fail, have them send you a message.
Michael Kennedy: So do you recommend CodeCommit from AWS?
Matt Lea: No, I've actually, a long time ago, I had my clients migrate off CodeCommit.
Matt Lea: Is CodeCommit even running?
Matt Lea: CodeCommit is different than CodeBuild and CodePipeline.
Matt Lea: It was their GitHub.
Matt Lea: Oh, I think it might be shutting down.
Matt Lea: Okay, sorry, I got this backwards.
Matt Lea: Yeah, so CodePipeline and CodeBuild.
Matt Lea: So pipeline is more of an orchestration layer.
Matt Lea: Okay.
Matt Lea: It listens for the commit and then you can tell it to do a bunch of stuff, upload S3,
Matt Lea: you know, or deploy or whatever.
Matt Lea: Code build is actually kind of ECS.
Matt Lea: It's like, it spins up a Docker container or a Lambda now.
Matt Lea: That's actually kind of new.
Matt Lea: And that you can have that Docker container.
Matt Lea: That's just a build container, build your application container.
Matt Lea: It's a little, it sounds a little weird, but basically you could just have it run your Docker
Matt Lea: build in the cloud, you know, and it'll just do it.
Matt Lea: And you say Docker push and pushes it out to ECR.
Matt Lea: the elastic container repo that's basically where the images are stored for docker and then then the
Matt Lea: next step that it'll throw it back to code pipeline you sell code pipeline do a red green deploy do a
Michael Kennedy: regular deploy do whatever interesting i never really thought about it but it makes perfect
Michael Kennedy: sense that they wouldn't be going docker pull from docker hub why do you leave their cloud right
Matt Lea: well and you can but they've also created this and you've got your granular iam permissions like
Matt Lea: we talked about before iam roles that you can now super secure without having to have two different
Matt Lea: things in the mix. They do that a lot. They take whatever's working out there and they just rebuild
Michael Kennedy: it. And that's probably why they have billion services at this point. Exactly. That's why the
Michael Kennedy: console looks like it does. All right. Now we've got to pick up some speed. We're still on our,
Michael Kennedy: on a thing. So maybe give us a little bit of a lightning round in the last couple here.
Matt Lea: Okay. I knew I put it in here. I was looking at my notes and I've got it in small text here. Yes.
Matt Lea: So SSM session manager. Okay. So like we talked about before, so if you boot up an ECS,
Matt Lea: sorry, EC2, EC2, that's the old school virtual machines, the big chunky ones that people like
Matt Lea: to nurse, you have the option to put an SSH key on there.
Matt Lea: So that way you can get into it and then do all that hand tuning you're doing.
Matt Lea: Again, I kind of touched on this.
Matt Lea: We'll go fast on this one.
Matt Lea: Use SSM session manager instead of the bastion.
Matt Lea: If you can, it'll make it a lot easier for you.
Matt Lea: So now we just caught some time up there because I already covered it.
Michael Kennedy: Yeah, perfect.
Michael Kennedy: Yeah, I love it.
Michael Kennedy: Okay, this is cool.
Michael Kennedy: That's really good advice.
Michael Kennedy: I mean, S3 public, why not?
Matt Lea: This one, I have seen so many clients stunned by it.
Matt Lea: I did an assessment, which is like my entry, like the first thing I do with people a lot of times is a short-term assessment to make,
Matt Lea: to give them an idea where they're at, and I map out the whole system.
Matt Lea: And they were just telling me, they were like, we really can't have these files public.
Matt Lea: They're binary files.
Matt Lea: Like, they would be big trouble for us if they were, you know, if anybody could hack it, could you check that out?
Matt Lea: And sure enough, the bucket was public to anybody on the planet.
Matt Lea: They just grabbed it out of there.
Matt Lea: It would have been tough to crawl it because they don't have listing publicly, but it would have, could have,
Matt Lea: The pattern wasn't so bad that you couldn't have had something brute force.
Matt Lea: You could like enumerate, attack it.
Matt Lea: Yeah.
Matt Lea: So S3 is a wonderful binary storage.
Matt Lea: It is not a CDN.
Matt Lea: It is not cost effective to serve up things at mass very fast through that.
Matt Lea: So you want to use the right tools for the job.
Matt Lea: And it's also secure-ish.
Matt Lea: But what you can do is you can do cloud front signed URLs.
Matt Lea: So you could say you could be very granular with your permissions.
Matt Lea: this like you said with the service earlier i could say this url is valid for exactly this
Matt Lea: amount of time and no longer and it goes to exactly this one file and no other files and
Matt Lea: i can send that out there you can do the same thing with uploads actually the uploads would go
Matt Lea: quickly to s3 but this you signed uploads so that way you're very that you've got this url lasts for
Matt Lea: five minutes you know two minutes 30 seconds whatever and you can get the upload done that
Matt Lea: that way so just i'm going to reiterate signed uploads with s3 make the bucket private use a
Matt Lea: signed url and then sign um use cloudfront and then if you need to lock it down use signed urls
Matt Lea: on the cloudfront side um but cloudfront is going to be so much more cost effective to serve up a
Matt Lea: lot high volume of binary or you know any any assets javascript right it's optimized for it yeah
Michael Kennedy: yeah so just um definition wise cloudfront they're cdn yeah that's right okay cost do people care
Michael Kennedy: about cost aws i don't know is that a thing uh so i i've been doing an experiment proactive
Matt Lea: engagement where i've got agents crawling around the internet uh looking for people complaining
Matt Lea: and cost is pretty much the number one thing why the heck did this cost me this much why the heck
Matt Lea: did it cost me that much you know and if you don't understand the costs it's really it's ugly but
Matt Lea: learn to use cost explorer because they're going to have a new service that comes out next we use
Matt Lea: there i wrote about this in march march they released a new charge if you didn't upgrade
Matt Lea: from older database versions they had a past end of life fee so all of a sudden my clients were
Matt Lea: getting smacked with about four grand a month and end of life fees and you're just like huh that
Michael Kennedy: where did that come from so but we didn't know that's terrible they're not even using it it's
Matt Lea: good yeah well they were using the database they just hadn't they should they should have moved
Matt Lea: up they should yeah yeah on IWS's side it's expensive to keep that older stuff running so
Matt Lea: it kind of makes sense but I wish they would have given us a better heads up but either way so I we
Matt Lea: go in there in the cost explorer and i was able to find that exact one it's really if you know what
Matt Lea: you're doing and you know how to switch between services and usage types and breakdowns you can
Matt Lea: find exactly where that jump is where it came from and then you can usually track that back to a very
Matt Lea: specific resource um and it also allows you to do tagging which is great you can tag production you
Matt Lea: can tag staging you can tag microservice x whatever you know so you can tag them and you can say look
Michael Kennedy: qa is actually costing us twenty thousand dollars a month can we just if you had hardware tagged as
Michael Kennedy: QA, yeah.
Michael Kennedy: Nice.
Michael Kennedy: Just to kind of bring this home here,
Michael Kennedy: and it's a beautiful loop back in a terrible or morbid way,
Michael Kennedy: but there was this story of this woman who created this,
Michael Kennedy: Cara, it's the app, but she created this.
Michael Kennedy: She really disliked AI-generated art because she was an artist,
Michael Kennedy: so she created this app that would use AI to tell
Michael Kennedy: if a piece of art was legit or AI.
Michael Kennedy: And she's coolified to host it at Vercel.
Michael Kennedy: Got a $95,000 Vercel bill in like a couple of days.
Michael Kennedy: As a student or just an independent artist or something like that, it was really not good.
Matt Lea: Yeah, that's not good.
Matt Lea: I mean, I don't want to down on anybody, but that's probably a token cost.
Matt Lea: And as we're finding out, the costs can be prohibitive.
Michael Kennedy: I think it was just CPU, actually.
Michael Kennedy: I don't think it actually was.
Michael Kennedy: It became like number one on the App Store or something like that.
Michael Kennedy: And Vercel had sent her a warning like, hey, your bill is blowing up.
Michael Kennedy: You need to either approve or disapprove.
Michael Kennedy: there was no upper bounds on cost set at all.
Michael Kennedy: And she didn't respond to the warning emails
Michael Kennedy: for like three or four days or something like that.
Michael Kennedy: It was like a cascading set of lessons
Michael Kennedy: that should be taken from this.
Michael Kennedy: Wow.
Matt Lea: Yeah, that's, I mean, definitely set up cost alerts.
Matt Lea: Keep an eye on it.
Matt Lea: You can set up budgets in AWS.
Matt Lea: And I mean, I, every week,
Matt Lea: so I have office hours with my clients.
Matt Lea: It's like an advisory session
Matt Lea: and I kick it off typically budget sheet
Matt Lea: right in front of me.
Matt Lea: So I was like, okay, here we're at,
Matt Lea: we saw this spike, what happened there?
Matt Lea: Oh, you know.
Michael Kennedy: Sometimes spikes are amazing.
Michael Kennedy: I mean, that means people are using your stuff, but sometimes they're not.
Michael Kennedy: Yeah.
Michael Kennedy: All right.
Michael Kennedy: You know, another time that they're not is when you're getting DDoSed.
Matt Lea: Yeah, this brings it right in.
Matt Lea: So DDoSs are everywhere.
Matt Lea: Cyber attacks all the time.
Matt Lea: The bigger you are, once you get to that big league, they're coming for you from the front.
Matt Lea: They're coming for phishing attacks from the side, from the back, from everything you can.
Matt Lea: The way we deal with that on AWS, there's a couple different ways.
Matt Lea: WAF is my tool of choice typically.
Matt Lea: There's some that you can,
Matt Lea: they've got some tools
Matt Lea: where they do some really fancy analytical stuff
Matt Lea: to block stuff.
Matt Lea: And that can work, but at volume,
Matt Lea: it's not really the most cost effective.
Matt Lea: You know, if you're,
Matt Lea: every request that comes through,
Matt Lea: you're running through some AI model,
Matt Lea: it's going to add up real quick.
Matt Lea: But there's a lot of stuff you can do that's obvious.
Matt Lea: Like, okay, are they looking for the WordPress admin?
Matt Lea: And you're not running WordPress.
Michael Kennedy: That should just be an absolute flag.
Michael Kennedy: Like if it's slash WP, you know,
Michael Kennedy: WP admin.php, Instaban is what it should be.
Matt Lea: Yep, you got it.
Matt Lea: You can do rate limiting as well, you know, and then you do rate limiting by route.
Matt Lea: We just did that for a client.
Matt Lea: There's a lot of different, yeah, there's a lot of different fun stuff you can do there.
Matt Lea: And they've got some of those things already kind of packaged in there.
Matt Lea: So you don't always need to be using like the smartest, like, you know, an AI model to evaluate.
Matt Lea: It's real simple.
Matt Lea: If the URL is, you know, something in the well-known thing that shouldn't be there, then you know it.
Matt Lea: Sorry, I'm getting sidetracked.
Matt Lea: You can also do challenges.
Matt Lea: So that's a JavaScript challenge that runs beneath the scene, like in your browser, and the user won't even really notice it.
Matt Lea: Or you can do a captcha if you really want to get into it.
Matt Lea: Captchas are a bit more expensive.
Matt Lea: Day challenges are much more cost effective.
Matt Lea: But there's a time and a place for them.
Matt Lea: You just got to kind of phenomps your way to how much you want to spend on it.
Michael Kennedy: Yeah, I've found that turnstile flair has gone a long ways towards keeping the bots away at the same time not being, you know, I just can't take another find the fire hydrant.
Michael Kennedy: I can't do it.
Michael Kennedy: You know, it's like, find all the fire hydrants like I did.
Michael Kennedy: And then one will fade in later.
Michael Kennedy: And it'll say, well, you didn't find all of them.
Michael Kennedy: Now you're banned because you didn't find them.
Michael Kennedy: So now you're going to do like fire hydrants.
Michael Kennedy: I can't take it.
Michael Kennedy: Or bicycles, you know.
Matt Lea: Yeah, well, that's where the challenges come into play.
Matt Lea: If they're using something like API-FI, which is if I'm trying to crawl something at scale,
Matt Lea: that's my tool to go to.
Matt Lea: It's not going to find that.
Matt Lea: But, you know, it'll do pretty good.
Matt Lea: You also can block entire countries, it turns out.
Matt Lea: I've got clients that just decided to block France once.
Matt Lea: I'm like, why?
Matt Lea: We can figure out this traffic.
Michael Kennedy: They beat their team in the World Cup and that's it.
Michael Kennedy: No more access for France.
Michael Kennedy: Sorry, our French friends.
Michael Kennedy: It's crazy.
Michael Kennedy: All right.
Michael Kennedy: Well, let's spend a few minutes as we kind of wind down the show.
Michael Kennedy: Now people got the idea.
Michael Kennedy: I mean, nobody's calm at 3 a.m.
Michael Kennedy: I think this is true.
Michael Kennedy: I've gone through the experience of multiple times, like your website's down.
Michael Kennedy: Oh, no, surely not.
Michael Kennedy: Oh, my gosh.
Michael Kennedy: And then why?
Michael Kennedy: And then there's just all this hack.
Michael Kennedy: Did I do something wrong?
Michael Kennedy: It's never fun when you're in it.
Michael Kennedy: And that's kind of your Cloud War Games thing, right?
Michael Kennedy: To help people through that.
Matt Lea: Yeah, you'd rather learn, you know, get the stress inoculation during a simulation, a 45-minute game,
Matt Lea: than at 3 a.m., you know, or when the entire board of directors is standing behind you
Matt Lea: because you're the guy that's supposed to fix it.
Matt Lea: So I saw, I mean, many times I saw it and I would be working with juniors and I'd try and,
Matt Lea: I'd be, you know, doing my, in a real incident, you know, trying to solve the problem.
Matt Lea: I'd pass something out to them, you know, so, okay, you look at this.
Matt Lea: trying to get them to learn and they just you know they'd freeze up and just like i don't know and
Matt Lea: then you'd ask them later like hey what was the problem you know you do that now so i mean another
Matt Lea: part of the war games is afterwards we always post-mortem and if you ever have an outage and
Matt Lea: especially if you got people freezing up you should post-mortems every time uh that's the way you learn
Matt Lea: and that's actually where i got most of the content for things i run people through on cloud war games
Michael Kennedy: from things i've actually suffered through so this is a program i can go sign up for and is it like a
Matt Lea: paid course or a community or what's the experience no so so it's it's actually evolving right now
Matt Lea: um we were running it where i did monthly ones and just the scheduling was complex so we're we're
Matt Lea: adapting a little bit um the you can sign up and you can get access to a bunch of what i call
Matt Lea: asynchronous challenges for free completely free there's just we'll send you one every week for
Matt Lea: i don't remember how long it goes um but the thing where we're trying to move into is trying to
Matt Lea: move it into, the Schematical has got a community offering basically where we, it's kind
Matt Lea: of more like coaching and a little bit more ongoing. So we're trying to move it in there
Matt Lea: where we can have it. I don't know, trying, I'm trying to get people in discord so we can
Matt Lea: plan these things a little bit better. It's a bit, you know, email list is great, but there's
Matt Lea: not like an organic, like, Hey, let's run a game right now. You know? Yeah. Discord is nice. Yeah.
Matt Lea: Yeah. And I'm also working this in my, we, we brought this into my consulting as well for the
Matt Lea: big companies is I do this for training, but also it's a great way to figure out, especially with AI
Matt Lea: resumes and all that stuff. It's a great way to figure out if someone actually knows their stuff.
Matt Lea: And even if they do know their stuff, are they a team player? So you can throw, you know,
Matt Lea: a couple of your applicants in there, candidates, and you'll see if one of them's my comics,
Matt Lea: the lone wolf programmer who doesn't talk to anybody and just like goes heads down and,
Matt Lea: or are they a team player? And they're like communicating, they're shouting out,
Matt Lea: Hey, you try this. And that those soft skills. I didn't know this when I first started it,
Matt Lea: but it really lets you find the people that are soft skills that you want being a leader of a team.
Matt Lea: And so we're working with bigger companies to run that,
Matt Lea: both to train their people internally, but also as a recruiting tool.
Michael Kennedy: Yeah, that's super neat. I like it.
Michael Kennedy: Thinking back of why did I freak out,
Michael Kennedy: a lot of times it's because I did something a couple of years ago
Michael Kennedy: and I had forgotten how I did it and even where the thing I needed was.
Michael Kennedy: Like one of the most annoying ones I remember is the SSH or just the TLS certificate.
Michael Kennedy: I was using for MongoDB to encrypt expired.
Michael Kennedy: I'm like, why?
Michael Kennedy: First of all, I'm like,
Michael Kennedy: why did I set it to be a one year expiry
Michael Kennedy: for an internal cert?
Michael Kennedy: I really should have made it longer
Michael Kennedy: or I should have written down what I did.
Michael Kennedy: But then I couldn't even remember,
Michael Kennedy: like, well, what was the command I ran?
Michael Kennedy: And like, how just,
Michael Kennedy: so I was just running around,
Michael Kennedy: like checking all these old logs and projects.
Michael Kennedy: Where is it?
Michael Kennedy: What are the things that you see
Michael Kennedy: that drive people crazy that they could maybe,
Michael Kennedy: what are some of the learnings, I guess,
Michael Kennedy: you're getting out of this?
Michael Kennedy: Like for me, it would be like, write the stuff down in a consistent place so I can get to it calmly.
Matt Lea: Right. Well, and so I hate to say some people don't like it, but a problem you just had there would be well fixed by a managed service of some type.
Matt Lea: AWS is KMS is a certificate manager.
Matt Lea: You know, these days in the age, you know, it saves you how many, how much money would you have paid that night?
Matt Lea: Or did it cost the company that now you could just pay a AWS KMS?
Matt Lea: You know, that's it would have saved you probably twice as much or a lot more than you spent that night.
Matt Lea: headaches you know yeah yeah so um i'd have more hair at least um yeah as far as the common ones
Matt Lea: it's the most common ones i see is misconfigured im you know misconfigured bpc public s3 happens
Matt Lea: oh too much i can i can't do an audit without finding something in s3 that's not supposed to
Matt Lea: be public you know sales numbers something like that you don't want your competitor to have so
Matt Lea: those are the most common ddos is like i said every all the time um but then it's the it's the one
Matt Lea: off. There's one I do. I don't want to disclose it because I love it, but there's a ransomware
Matt Lea: attack. It's my advanced one. I call it the Kobayashi Maru because it's pretty brutal.
Matt Lea: Luckily, I've never had a client hit with it, but it is a brutal attack that's specific to AWS,
Matt Lea: and it can be run one command if you've got the right privileges, and it's brutal.
Michael Kennedy: That's not good. That's not good. I saw that you've got a couple of these live events on video,
Michael Kennedy: like a beginner one and a more advanced one.
Michael Kennedy: So maybe I'll link to these for people in the show notes.
Matt Lea: Sounds good.
Matt Lea: And they're on youtube.com slash grammatical if you need it.
Michael Kennedy: All right.
Michael Kennedy: Well, maybe bring it home to people.
Matt Lea: Sounds good.
Matt Lea: Closing advice.
Matt Lea: Since we've been talking, I came up with like three more.
Matt Lea: So if anybody wants my other ones, you know, reach out.
Michael Kennedy: Want a lightning round?
Michael Kennedy: Let's want a lightning round them real quick.
Matt Lea: Sure.
Matt Lea: Well, yeah.
Matt Lea: So AI, SageMaker for your Python crew, SageMaker.
Matt Lea: Let's check that service out.
Matt Lea: It's if you want the GPUs, have more granular control than you can on basically any other service.
Matt Lea: And it's Python first, data science first.
Matt Lea: So if you're data science, you know, you like it's check out SageMaker.
Matt Lea: If you're not that stuff, Bedrock or if you just generic stuff, Bedrock's amazing as well.
Matt Lea: It's their Netflix for AI. Phenomenal. I'd check that out.
Matt Lea: The other one, the Data Lake Stack I was talking about earlier.
Matt Lea: I need to do some Terraform scripts for this, but they have I didn't talk about adventure in architecture, but great cues that you can use.
Matt Lea: Kinesis is a Firehose queue, meaning that you can have a lot of different things consuming from it.
Matt Lea: So if you've got 10 different worker services, they all can consume from it.
Matt Lea: It's great.
Matt Lea: Pump that.
Matt Lea: So whenever your application layer fires off an event, hey, this was updated, a user was created, a sale was made, pump it into Kinesis.
Matt Lea: And then you can have Firehose save that as a parquet file to S3.
Matt Lea: It's extremely cheap.
Matt Lea: It doesn't cost fractions of a penny.
Matt Lea: It's so cheap.
Michael Kennedy: There's super interesting Python libraries that you can mount S3 parquet files and stuff and do query.
Michael Kennedy: There's really a lot of cool integrations with Python and Parquet, yeah.
Matt Lea: Yeah, and you can query it with Glue,
Matt Lea: which I believe has some Python in there.
Matt Lea: But if you didn't want to do that, you're just an SQL guy,
Matt Lea: Athena can talk and query to it as well.
Matt Lea: So I need to do some writing down on this,
Matt Lea: but it's a phenomenal data-like thing.
Matt Lea: Don't try and keep sales from 2003 in your production database.
Matt Lea: Move all that stuff, that old stuff, off of it.
Matt Lea: Keep your production database nice and lean and moving fast.
Michael Kennedy: Awesome.
Michael Kennedy: Well, Matt, thank you for being here.
Michael Kennedy: It's been really awesome.
Michael Kennedy: I'll put all your contact info in the show notes
Michael Kennedy: for people who want to get in touch.
Michael Kennedy: And yeah, thanks for keeping our clouds a little healthier.
Matt Lea: Yeah, sounds good.
Matt Lea: Happy to help.
Matt Lea: Feel free to reach out to me.
Matt Lea: And, you know, thanks for having me.
Michael Kennedy: Yeah, you bet.
Michael Kennedy: Bye-bye.
Michael Kennedy: This has been another episode of Talk Python To Me.
Michael Kennedy: Thank you to our sponsors.
Michael Kennedy: Be sure to check out what they're offering.
Michael Kennedy: It really helps support the show.
Michael Kennedy: Take some stress out of your life.
Michael Kennedy: Get notified immediately about errors and performance issues
Michael Kennedy: in your web or mobile applications with Sentry.
Michael Kennedy: Just visit talkpython.fm/century and get started for free.
Michael Kennedy: Be sure to use our code talkpython26.
Michael Kennedy: That's talkpython, the numbers two, six, all one word.
Michael Kennedy: And it's also brought to you by Talk Python Courses.
Michael Kennedy: Course completion certificates are now live.
Michael Kennedy: If you finished a course, there's a certificate waiting for you on your account page right now.
Michael Kennedy: Download it as a PDF or add it to your LinkedIn profile with one click under licenses and certifications.
Michael Kennedy: Same section as your formal degrees.
Michael Kennedy: Visit training.talkpython.fm/account to see what you've already earned.
Michael Kennedy: If you or your team needs to learn Python,
Michael Kennedy: we have over 270 hours of beginner and advanced courses on topics
Michael Kennedy: ranging from complete beginners to async code, Flask, Django, HTML, and even LLMs.
Michael Kennedy: Best of all, there's no subscription in sight.
Michael Kennedy: Browse the catalog at talkpython.fm.
Michael Kennedy: And if you're not already subscribed to the show on your favorite podcast player,
Michael Kennedy: what are you waiting for?
Michael Kennedy: just search for Python in your podcast player
Michael Kennedy: we should be right at the top
Michael Kennedy: if you enjoyed that geeky rap song
Michael Kennedy: you can download the full track
Michael Kennedy: the link is actually in your podcast blur show notes
Michael Kennedy: this is your host Michael Kennedy
Michael Kennedy: thank you so much for listening
Michael Kennedy: I really appreciate it
Michael Kennedy: I'll see you next time
Matt Lea: I think is the norm.