12 Things You Should (and Shouldn't) Do in AWS
Episode Deep Dive
Guest Introduction and Background
Matt Lea runs schematical.com, a small consulting agency that specializes in AWS. Their tagline says almost everything you need to know about the work: "we help CTOs running on AWS sleep at night." Matt has been doing this since 2010, has been programming for roughly thirty years, and describes his entry point with most clients as an assessment where he maps out their whole system and tells them where they actually stand. He is the person companies call when production is down and the board of directors is standing behind somebody's chair.
Beyond the consulting, Matt built cloudwargames.com, where he runs simulated outages and cyber attacks so teams can practice incident response before they have to do it for real at 3am. He authored an animated course, Zero to Hero on AWS Security, which teaches cloud security through isometric pixel art and animated network maps. He also draws tech comics at schematical.com/comics, somewhere in the range of 30 to 40 of them at this point, several of which have done well on r/ProgrammerHumor. He mentioned that "Cloud Demolition Expert" was his LinkedIn title for a while, which is a fair description of a guy who blows up infrastructure on purpose for a living.
- schematical.com
- cloudwargames.com
- schematical.com/comics
- oreilly.com/videos/zero-to-hero
- github.com/schematical/sc-terraform
- youtube.com/schematical
- linkedin.com/in/schematical
What to Know If You're New to Python
This episode is less about Python syntax and more about the environment your Python code eventually has to survive in. A little familiarity with how a web app gets from your laptop to a server, and with the vocabulary of cloud infrastructure, will make the whole conversation land much harder.
- Infrastructure as code: Instead of clicking buttons in a web console to create servers and databases, you write text files that describe what you want, then check those files into git. This is the single idea the entire episode is built on, and Matt says it is very hard to adopt after the fact.
- The AWS acronym soup: EC2 is a virtual server, S3 is file storage, Lambda runs a function without a server, IAM controls permissions, VPC is your private network. Michael and Matt stop repeatedly to define these, so you do not need them going in, but knowing that they are just products with jargon names removes a lot of the intimidation.
- Least privilege: The principle that every piece of software should be able to do exactly what it needs and nothing more. Most of the twelve items on Matt's list are really applications of this one idea, and the wildcard character in a permissions policy is its opposite.
- Containers and Docker: A container packages your app and its operating system setup into an image you can start identically anywhere. The "cattle not puppies" advice in the episode only makes sense once you see that a Dockerfile is just the list of commands you used to type into the server by hand.
- Giving AI agents access to real systems: A lot of this episode deals with what happens when an agent has permission to change infrastructure or a database. If you use AI coding tools, the discussion of read-only tool calls, human-approved writes, and pointing agents at a copy of your data rather than production is directly applicable.
Key Points and Takeaways
Everything you want at 3am is decided months earlier, on an ordinary afternoon
The through-line of the whole conversation is that incident response is not a skill you deploy in the moment, it is a bill that comes due for decisions made when nothing was on fire. The convenient choice, the wildcard permission to unblock a deploy, the one big subnet, the server you SSH into and hand-tune, all of them feel harmless on a Tuesday. They only reveal themselves when the site is down and you cannot tell whether you are looking at a bug, a bill, or a breach. Matt's twelve items are essentially a list of the afternoons that go wrong. Michael makes the point that his own deployments got better because of outages, but that he also "aged faster than normal" in the process. The premise of Cloud War Games is that you can get the learning without the aging.
Don't hand-provision, use infrastructure as code
Matt's number one item is that you should never build your infrastructure by clicking around in the console. Infrastructure as code (IaC) is the category, and the tools are Terraform, OpenTofu, and AWS's own proprietary CloudFormation. The argument is the same one you already accept for application code: you would not FTP your source to a server and hope it stays there, so why treat your servers, roles, firewall rules, and networks any differently. Matt says he came to CloudFormation a couple of years after it launched, thought "this could be cool," and shortly after concluded "this is essential." Michael's confession is that he used to just SSH in and get it working, then discovered that moving from one setup to another turned into a research project. The critical detail is that this is very difficult to retrofit, because once everything is hand-spun nobody remembers how it was done and the person who did it has left the company. Matt offers his own Terraform scripts for free at schematical.com/free, including a setup that wires up API Gateway plus Lambda with a build pipeline.
- terraform.io
- opentofu.org
- aws.amazon.com/cloudformation
- schematical.com/free
- github.com/schematical/sc-terraform
Always read the plan before you apply it
The counterweight to loving Terraform is that Terraform will do exactly what you told it to, at machine speed, with no hesitation. Matt has seen too many teams auto-apply changes without reading them. His example, which he wrote up but did not personally live through, involved an AI vibe-coding a change that included a slight rename of a database. Terraform interpreted that as delete the old one and create a new one, and the old one was production. His rule is to always know what you are about to send, and specifically to confirm that what you are looking at is an update and not a delete-and-replace. Michael connects this to the broader class of AI-deleted-the-production-database horror stories, where the agent technically solves the problem it was given and destroys something irreplaceable on the way. The failure is not that the tool is dangerous, it is that nobody read the diff.
Don't use access keys, use IAM roles
IAM stands for Identity and Access Management, and it governs how your users and your running services interact with AWS. Most people start by downloading their first access keys, dropping them in a .env file, and hoping for the best. Matt's alternative is roles, which you attach to the service itself: an EC2 instance gets a role, a Lambda gets a role, and when that Lambda executes it simply has the permissions the role grants. No key material to leak, no secret to accidentally commit. He is careful to draw the boundary here, though. IAM is service-level, not network-level, and it is not row-level either. It answers questions like "can this Lambda write to this one S3 bucket" or "can this instance boot up other instances," not "what rows in the database can this query see."
Leaked keys are found immediately, not eventually
This is where the episode's best cautionary tale lives. A junior on one of Matt's teams pushed access keys for sending email up to a public repo. Bad actors found them and sent roughly 16 million knockoff Viagra emails in about eight minutes before AWS shut the account down and asked what was going on. Michael adds the tooling side of the story with shhgit, which scanned the real-time firehose of public GitHub commits for anything that looked cryptographic and streamed the finds live. The lesson to pass along to anyone new: "I'll just uncommit it" is not a remediation. If a credential touched a public repo, assume it is gone immediately and rotate it. Michael also brings up a PyCon talk by Tristan McKinnon called "Zero Trust in 200ms," built around issuing a fresh, extremely short-lived credential per transaction, which comes out of the FedRAMP and government cloud world. Matt likes the idea and immediately asks the practitioner questions: how long does it take to provision the key, and what does that do to your latency budget if you are trying to compete in e-commerce.
Don't use generic IAM permissions, the asterisk is your enemy
Granular permissions are the difference between a leaked credential being an incident and being an extinction event. Matt's framing is blunt: go back to the email keys story and ask what would have happened if that policy had said "do anything on any service." They could have deleted everything in a second. The realistic path to a wildcard is not malice, it is fatigue. You have been banging your head against a permissions error for an hour and you decide to just let it access all the S3 buckets, and months later somebody has pulled your internal reporting out of a bucket you forgot was in scope. His guidance is to scope both the actions and the resources, and to remember that you can specify all the way down to a path inside a bucket, not just the bucket name. If you genuinely need a wildcard for ten seconds of experimentation, do not commit it to version control, so that your next apply wipes it out for you.
Agents will happily take the shortcut, so review their work adversarially
Michael raises the optimistic case: five years ago writing granular permissions meant an ocean of checkboxes, and now you can describe what you need in a sentence and let an agent draft the policy. Matt's response is the sharpest practical AI advice in the episode. If a non-technical person tells an agent "I just need this to talk to that," the agent can absolutely satisfy that request by putting an asterisk in there. It accomplished the goal, just not the way you wanted. What he has seen work is competing agents: one writes the policy, a second does a peer review with the explicit instruction to find any possible way to break it. A single agent will often just say "yeah, sure, I did it." Michael's version is to tell the agent that its primary objective is least privilege and that it wins the session by granting as little as possible. Both agree the real lesson is that domain expertise makes these tools far more powerful, and that vibing your way through security is how you end up in a headline.
Point AI agents at a data lake, not your production database
As executives get comfortable with agents, Matt is fielding more requests for direct database access so leadership can pull the latest numbers themselves. His answer is to build a tool call with read access rather than hand over a connection string, and if writes are genuinely needed, a second tool call that requires human approval so the agent can suggest a change instead of making one. Then he adds the layer most people miss. Agents can spam a tool call relentlessly, so a complicated read query fired in a loop can degrade a production database that is not perfectly optimized. Better to point the agent at your data lake, which is a separate analytical copy that is typically anonymized anyway. If something does leak, you leaked sales figures instead of personally identifiable information, and the PII leak is the one that costs you far more.
Don't put backend resources in public subnets
Matt zooms out to explain the shape of an AWS network before giving the advice. A region like US West is a giant facility, and inside it are several availability zones, each its own bunker with completely independent power. A virtual private cloud (VPC) is your own slice of network inside your account, and inside that you carve out subnets. Two problems show up constantly. First, people put everything in one subnet in one availability zone in one region and call it redundancy, when individual availability zones go down fairly frequently and you only avoid noticing if you are spread across several. Second, people leave databases somewhere reachable from the internet. His layout is a public subnet holding only what genuinely faces the world, typically the load balancer, and private subnets holding the databases and anything sensitive. Yes, you can firewall a public resource, but Matt's audits keep turning up holes somebody poked for convenience. A private subnet means that even when a firewall rule is wrong, there is no network path from point A to point B.
Don't use one security group for everything
Security groups are AWS's firewall layer, and the common failure is putting everything into one group where any member can reach any other member on any port. That is not segmentation, it is a flat network with extra steps. Michael supplies the security term for what that enables: lateral movement, where an attacker who gets a foothold anywhere can then wander. Matt's example is topical, invoking the wave of npm supply chain exploits. If something malicious lands in your application layer, that layer has to be able to reach your database, and that is bad enough. But with correct security groups and subnets, at least the compromised test environment cannot reach production, and the attacker cannot crawl every IP in your subnet looking for what else is listening. He also points out the half of this that almost nobody configures: outbound rules. If you really want to lock things down, restrict what your boxes are allowed to call out to, so a compromised instance cannot phone home to an address you have never heard of.
Use SSM Session Manager instead of a bastion host
The classic pattern for getting into a private network is a bastion, a small publicly reachable box, ideally locked down with a security group that only accepts SSH from your own IP address, which you then have to update every time you change locations. You SSH into the bastion and it becomes your door to everything else, including port forwarding so you can hit a test database from your laptop. Matt still does this, but says the better modern answer is AWS Systems Manager Session Manager. You authenticate with IAM rather than a key, it is more secure, and you are not paying to keep an EC2 instance running 24/7 just to act as a door. Michael offers two adjacent approaches from outside AWS: overlay networks like Tailscale, which put your employees on a private network that can see the database server and nothing else, and KnocKnoc, where management ports are closed to everyone and only open for your specific IP for the duration of an authenticated session. That leads to a genuine feature request the two of them agree AWS should ship: time-limited security group rules, so the hole you open to debug something closes itself instead of relying on you to remember.
Don't nurse EC2 instances, cattle not puppies
Matt did not coin the phrase but says it sums things up perfectly. The pattern he sees is somebody booting a couple of EC2 instances, tuning them until they are exactly right, and treating them as irreplaceable, so when one gets sick everybody is left asking which scripts were run and in what order. The alternative is images. Docker lets you build the machine's setup as a file, and Matt's analogy is burning your OS to a disk you can pop in any time to boot the exact same thing. If you had to delete your servers ten times a day you could bring them back in their perfect first-traffic state in a split second. On AWS that means ECS (Elastic Container Service) or Lambda, and it unlocks autoscaling: two containers in the middle of the night, a hundred the day after Thanksgiving. Michael admits he resisted Docker for years because it felt like complexity paired with opacity, until he realized the Dockerfile is just the commands he was typing into the server anyway, and that he could still observe the container by mapping volumes or shelling into it.
Kubernetes, Coolify, and the honest cost of self-hosting
Asked about Kubernetes, Matt relays advice from another developer he respects: if you are doing Kubernetes anywhere other than GCP, it is probably not a great fit. AWS has EKS, the Elastic Kubernetes Service, and some people do well with it, but he has pulled multiple clients off EKS and onto ECS because ECS is more AWS-native and they were already there. He is explicit that Kubernetes is not his expertise and that the teams having the best time with it tend to be on GCP. Michael pushes the conversation toward the small end, noting you can go a long way with Docker Compose and that he runs 33 distinct containers, and brings up Coolify, which installs an agent on any Linux box (EC2, DigitalOcean, Hetzner) and manages the Docker side of a large catalog of self-hosted services, so that standing up something like the Heimdall application dashboard is a pick-from-a-list operation rather than a Docker exercise. Matt's response is the most business-minded moment in the episode. He admits he would love to live in a cabin with his own servers, but the questions that decide it are what happens when someone kicks the power cord and how many engineering hours per month the thing will eat versus a managed service. Michael's counterpoint is that even a tool that manages Docker for you is still a bit of a puppy, because backups and a corrupted upgrade are still yours to handle. Matt's live example is self-hosting n8n and constantly being told to run another update, which prompts the obvious question of how much of his time he would get back on n8n's own platform. Michael tried n8n too and it eventually became a no for him, so he switched to Hermes. Matt has used Hermes with mixed results, and his specific complaint is a good one for anyone building on agents: it has its own AI for managing the context window, so he was losing context without meaning to. He also caught the model writing plain-text memories about him, including the note that the user gets agitated when you hallucinate.
- kubernetes.io
- aws.amazon.com/eks
- coolify.io
- github.com/linuxserver/Heimdall
- n8n.io
- github.com/NousResearch/hermes-agent
Learn CloudWatch logs, metrics, and alarms, but watch what queries cost
Moving off a single server means your logs are no longer one file you can tail, and this is the only entry on Matt's list that is a "do" rather than a "don't." CloudWatch collects logs and metrics from essentially everything, and CloudWatch Insights lets you query them, which he was using recently to chase a DDoS that pushed a client to nearly three times normal traffic. The cost warning is specific and useful: Insights charges by bytes ingested, so a colleague they had just trained pointed a query at five years of logs and produced a memorable bill. Get your query right over a five-minute window, then widen the range. On metrics, the payoff is being able to answer a client's "we saw extra latency on this day" by zooming through layers all the way down to whether a database had enough swap or Redis evictions spiked. Alarms are not just for waking you up, they can drive autoscaling, and Matt's favorite example is a worker fleet that scales on the count of messages in flight in the queue rather than on CPU. Michael notes why that matters: an analytics process can peg the CPU without the service being busy, and adding instances would not help. Logging is close to automatic for ECS tasks and Lambdas, since anything that would show up in docker logs gets piped through; the gap is an application that writes to its own log file on disk, which you need to explicitly pipe out.
Don't hand-deploy, build a pipeline
Twenty years ago you FTP'd files up to a server and hoped. Today the shape is: commit, get the pull request approved, rebuild the image, ship it. On AWS, CodePipeline is the orchestration layer that listens for the commit and sequences the steps, and CodeBuild is effectively a container that runs your build, pushing the resulting image to ECR, the Elastic Container Registry. Then control returns to CodePipeline for the deploy, blue-green or otherwise. Matt notes you can run database migrations through the same pipeline, and automated tests if that suits your team, though he is looser about it for people who are just starting. Michael's observation, which Matt confirms, is that AWS builds its own version of whatever is already working out there, partly so everything can be secured with the same granular IAM roles rather than stitching credentials across two vendors, and partly why the console has a billion services in it. Matt is not a fan of CodeCommit, AWS's Git hosting, and moved clients off it long ago; CodeBuild and CodePipeline are a separate matter and he uses them.
Don't make S3 public, put CloudFront in front of it and sign the URLs
Matt says he cannot do an audit without finding something in S3 that is not supposed to be public. His story here is a client who told him during an assessment that it would be very bad if certain binary files were reachable, and sure enough the bucket was open to anyone on the planet. Public bucket listing was off, so crawling it would have been awkward, but the file naming pattern was guessable enough to enumerate. There are two separate reasons to fix this. The security reason is obvious. The cost reason is that S3 is excellent binary storage and is not a CDN, so serving high volumes of files or JavaScript through it directly is not cost effective. The correct shape is a private bucket with CloudFront in front of it, and signed URLs when you need to lock access down. A signed URL can be valid for exactly one file for exactly two minutes and no longer, and the same mechanism works for uploads, so browsers can upload straight to S3 through a short-lived signed URL without your app touching the bytes.
- aws.amazon.com/s3
- aws.amazon.com/cloudfront
- docs.aws.amazon.com/AmazonCloudFront/private-content-overview
Learn Cost Explorer, set budgets, and tag everything
Matt has agents crawling the internet looking for people complaining about AWS, and cost is far and away the number one complaint. His advice is to actually learn Cost Explorer, because if you know how to pivot between services, usage types, and the various breakdowns, you can trace a spike back to a specific resource. His example: in March, AWS introduced a past-end-of-life fee for older database versions, and clients who had not upgraded were suddenly eating about $4,000 a month with no idea where it came from. Tagging is what makes the whole thing legible, so you can say that QA specifically is costing $20,000 a month rather than staring at one undifferentiated number. Set budgets and cost alerts. Michael brings the cautionary tale: an artist who built an app to detect AI-generated art, watched it hit the top of the app store, and received a roughly $95,000 Vercel bill within days. There was no upper bound on spend configured at all, and the warning emails went unanswered for three or four days. Matt runs weekly office hours with clients and kicks them off with the budget sheet in front of him, asking about every spike. Sometimes a spike means people love your product. Sometimes it means you are being attacked.
Use WAF and cheap rules before you reach for expensive ones
Once you are big enough, attacks come from the front, the side, and the back, and Matt says he is dealing with DDoS constantly. His tool of choice on AWS is WAF, the web application firewall. There are fancy analytical options that evaluate every request, but at volume, running every request through a model gets expensive fast. His preference is to exhaust the obvious rules first: if a request is looking for the WordPress admin and you do not run WordPress, that is not a judgment call, that is an instant ban. Rate limiting is available, including rate limiting per route, which he had just implemented for a client. For bots, JavaScript challenges run invisibly in the browser and are much more cost effective than CAPTCHAs, which have their place but cost more. Michael mentions that Cloudflare's Turnstile has gone a long way toward keeping bots out without subjecting humans to another round of finding all the fire hydrants. And in the category of blunt instruments that work, Matt has clients who simply blocked an entire country once they determined the traffic was worthless.
Cloud War Games: practice the incident before the incident
Nobody is calm at 3am, and Matt's answer is stress inoculation. Cloud War Games runs simulated outages, roughly a 45-minute game, where he intentionally plays the junior who deletes something important and a team has to diagnose and fix it. The scenarios come from things he has personally suffered through. The postmortem afterward is not optional and is where the actual learning happens, and he argues you should be running postmortems after every real outage too, especially if you watched people freeze. Because his own environments are defined in Terraform, he can blow them up and re-apply them all day. Right now you can sign up and get a series of asynchronous challenges by email for free, and he is working on moving the live format into the Schematical community and a Discord so games can be organized more spontaneously. There is a second use he did not anticipate: in an era of AI-assisted resumes, putting a few candidates through an incident together reveals both whether they actually know the material and whether they communicate under pressure, which turns out to be exactly the soft skill you want in someone leading a team. He also has an advanced ransomware scenario he calls the Kobayashi Maru, an AWS-specific attack that can be run with a single command given the right privileges, which he declined to describe in detail.
Lightning round: SageMaker, Bedrock, and a cheap data lake stack
For the Python and data science crowd specifically, Matt points at SageMaker, which he describes as Python-first and data-science-first, and the place to go if you want GPUs with more granular control than any other AWS service gives you. For general AI work, Bedrock is his pick, which he calls their Netflix for AI. Then he sketches the data lake stack he had been hinting at all episode: your application fires events (a user was created, a sale was made) into Kinesis, which is a firehose queue that many consumers can read from at once, and Firehose writes those events out to S3 as Parquet files for fractions of pennies. From there you query with Glue, which has some Python in it, or with Athena if you would rather write SQL. Michael notes there is a lot of good Python innovation around mounting and querying Parquet files on S3 directly. The closing advice attached to this: do not keep sales records from 2003 in your production database, move the old data out and keep production lean and fast.
- aws.amazon.com/sagemaker
- aws.amazon.com/bedrock
- aws.amazon.com/kinesis
- aws.amazon.com/glue
- aws.amazon.com/athena
- parquet.apache.org
Complexity is the number one problem, so pick a small stack and get good at it
When Michael asks what surprises people about moving to the cloud, Matt's answer is not security or cost, it is complexity. He jokes that AWS now has so many services there is one to train you on how to take meetings. His advice is to whittle it down to the core services that meet your goals and ignore the rest, because there are services for robotics that are wonderful if you are in robotics and irrelevant if you sell things online. For someone starting out without crazy traffic, he would often go serverless, and his minimal application-layer stack is API Gateway plus Lambda, which is easy to set up, easy to manage, and extremely cheap. That is not a data layer, but it is a great start, and it is exactly what his free Terraform scripts spin up.
Interesting Quotes and Stories
"Our tagline is we help CTOs running on AWS sleep at night." -- Matt Lea
"Somebody had their AI vibe code this thing, and it wanted to rename the database. Well, then what happens is Terraform deleted the database and then spun up another database right there. That was their production database. It was a rough day." -- Matt Lea
"I believe we sent 16 million emails for some type of knockoff Viagra in about eight minutes before AWS shut down the account and said, what are you doing?" -- Matt Lea
"The asterisk is your enemy." -- Matt Lea
"If it's up there you should assume it's gone immediately." -- Michael Kennedy, on credentials pushed to a public repo
"It's great that we're empowering more people to interact with data and do more with it. You just, you still got to be careful about InfoSec, and these things do make mistakes occasionally." -- Matt Lea
"Having competing more skeptical agents seems to help if you've got the token budget for it." -- Matt Lea
"If you're a company that goes down and loses $100,000 an hour and you don't have the setup commands documented or your lead tech gets hit by a bus, quote unquote, you want that Docker file to be clear as day." -- Matt Lea
"For a long time, I resisted using Docker because to me, it felt like complexity paired with opacity. And what I realized not too long after that is, well, all the commands you put in the Docker file, it's like what you had to write into your server to make it do whatever it does anyway." -- Michael Kennedy
"The model, I could see the memories it was putting in it. It's trying to save it in plain text, and it was saying the user gets agitated when you hallucinate." -- Matt Lea, on self-hosting an agent
"S3 is a wonderful binary storage. It is not a CDN." -- Matt Lea
"Public S3 happens too much. I can't do an audit without finding something in S3 that's not supposed to be public." -- Matt Lea
"Nobody's calm at 3am." -- Michael Kennedy
"You'd rather get the stress inoculation during a simulation, a 45-minute game, than at 3 a.m., or when the entire board of directors is standing behind you because you're the guy that's supposed to fix it." -- Matt Lea
"That's actually where I got most of the content for things I run people through on Cloud War Games, is from things I've actually suffered through." -- Matt Lea
"AWS has a meeting simulator service now, like where you can train yourself on how to take meetings. And I'm like, you've got too many services." -- Matt Lea
Two smaller stories worth the listen. Michael's Black Friday cartoon moment, where the punchline is a CTO saying "oh I'm sure we'll be fine," pairs with his own near-outage caused by an extra-large JavaScript file being gzipped on the fly by Nginx, which is a perfect illustration of his line that the weakest link is never the one you expected. And late in the episode, Michael describes chasing down expired TLS certificates he had generated a year earlier for MongoDB, unable to remember which command he had run or where he had written it down, digging through old scripts and projects while the clock ran. Matt's answer was gentle but firm: that is exactly the problem a managed service exists to solve, and the hours you spent that night cost more than the service would have.
Key Definitions and Terms
- IaC (Infrastructure as Code): The practice of defining servers, networks, permissions, and other infrastructure in text files that live in version control. Terraform, OpenTofu, and CloudFormation are the three tools named in the episode.
- IAM (Identity and Access Management): AWS's permission system, covering both human users and running services. Service-level, not network-level, and not row-level inside your database.
- IAM role: A set of permissions attached to a service rather than a person. When a Lambda or EC2 instance runs, it acts with the permissions of its role, which removes the need for long-lived access keys.
- Least privilege: Granting exactly the permissions needed and nothing more. The wildcard character in a policy is the opposite of this.
- VPC (Virtual Private Cloud): Your own isolated network inside an AWS account. Matt keeps separate VPCs for production, staging, and test, though he notes separate accounts are also a reasonable approach.
- Availability zone: An independent facility within an AWS region with its own power supply. Individual zones go down fairly often, which you only survive without noticing if you are spread across several.
- Public subnet vs private subnet: A public subnet holds things that should be reachable from the internet, typically the load balancer. A private subnet holds databases and anything sensitive, so that even a mis-set firewall rule leaves no network path in.
- Security group: AWS's firewall layer, controlling which resources can talk to which others on which ports, in both the inbound and outbound directions.
- Lateral movement: The security term for an attacker who has gained a foothold in one place moving sideways through your network to reach more valuable targets. Flat security groups make this easy.
- Bastion host: A small publicly reachable server used as a controlled entry point into an otherwise private network, usually via SSH with port forwarding.
- SSM Session Manager: AWS Systems Manager's session tool, which lets you get a shell on an instance using IAM authentication rather than SSH keys and an always-on bastion.
- Cattle not puppies: The philosophy that servers should be interchangeable and disposable rather than hand-tuned and irreplaceable. The test is whether you could terminate one right now and be fine.
- CDN (Content Delivery Network): A distributed network optimized for serving files fast and cheaply at high volume. CloudFront is AWS's CDN; S3 by itself is storage, not a CDN.
- Signed URL: A time-limited, scope-limited URL that grants access to exactly one object for exactly as long as you specify. Works for downloads and for direct-to-S3 uploads.
- WAF (Web Application Firewall): A filtering layer in front of your application that can block requests by pattern, rate limit by route, issue JavaScript challenges or CAPTCHAs, and block by geography.
- Data lake: A separate analytical store of your data, usually cheaper, often anonymized, and the right target for both big-data queries and AI agents that would otherwise hammer production.
- PII (Personally Identifiable Information): Data that can identify a specific person. Matt's point is that a leak of anonymized sales data is survivable in a way a PII leak is not.
- Parquet: A compact columnar file format well suited to storing analytical data cheaply on S3 and querying it later with tools like Athena, Glue, or Python libraries.
- Postmortem: The structured review after an incident. Matt runs one after every war game and argues you should run one after every real outage, especially when people froze.
- Stress inoculation: Deliberately exposing people to a simulated high-pressure incident so that the real one is not their first. The core idea behind Cloud War Games.
Learning Resources
If this episode made you want to shore things up, here are places to go deeper. The first two are directly about the security and delivery concerns Matt raised, and the third is about working safely with the agents that keep showing up in these stories.
- Python Web Security: OWASP Top 10 with Agentic AI: Works through every category of the 2025 OWASP Top 10 with vulnerable-and-fixed examples in Flask, Django, and FastAPI, then builds a custom security-review agent you can turn loose on your own codebase. This is the application-layer half of what Matt is describing at the infrastructure layer.
- Python Web Apps that Fly with CDNs: A hands-on guide to putting a CDN in front of your Python app for static content, images, and large user files. Directly relevant to Matt's "S3 is not a CDN" advice and the CloudFront pattern he recommends.
- Agentic AI Programming for Python: Covers the guardrails and workflows that turn agentic coding tools into a force multiplier rather than the thing that renamed your production database. Pairs well with Matt's advice about adversarial review agents and least-privilege instructions.
- Cloud War Games: Matt's simulated outage platform. You can sign up for free asynchronous challenges delivered by email, and there are recorded sessions on YouTube if you want to watch a game before joining one.
- Zero to Hero on AWS Security: Matt's animated AWS security course on O'Reilly, taught with isometric pixel art and animated network maps, with the accompanying Terraform in the sc-terraform repo.
- AWS Well-Architected: AWS's own framework covering security, reliability, cost optimization, and operational excellence. A useful map of the territory the twelve items in this episode sit inside.
Overall Takeaway
The most useful thing about Matt's list is that not one item on it is exotic. There is no zero-day, no nation-state actor, no clever exploit chain. There is a junior who pushed a key to a public repo, a wildcard somebody added at hour two of a permissions fight, a bucket that was public since before anyone remembered, a database in a subnet it should never have been in, and a server that one person knew how to rebuild. Every one of those was somebody choosing convenience on an afternoon when nothing was wrong, and every one of them cashed out later at the worst possible hour. That is the real argument for infrastructure as code, for roles instead of keys, for private subnets and tight security groups: not that they make you safe, but that they make the convenient path and the correct path the same path, so you stop having to be disciplined at 2am.
The part that should stick with you, though, is the Cloud War Games premise. You can configure everything on this list once and be done. Incident response is the one thing you cannot configure, because it lives in people, and the hidden cost of an outage is rarely the downtime itself, it is that nobody in the room has ever done this before. Matt's whole business rests on the observation that the first ten minutes of an incident go completely differently for a team that has practiced than for one that has not. So pick one thing this week. Put your infrastructure in version control. Delete a wildcard. Check whether a bucket is public. Then, at some point, deliberately break something while the stakes are low, and find out what your team actually does. That is a much cheaper way to learn than the 3am version.
Links from the show
Matt Lea: linkedin.com
Talk Python Certificates: training.talkpython.fm/certificates
Schematical: schematical.com
CloudWarGames.com: cloudwargames.com
Zero to Hero on AWS Security: www.oreilly.com
Repo: github.com
Custom Wheel Offset: customwheeloffset.com
2012 TechCrunch Disrupt Hackathon: techcrunch.com
tech comics: schematical.com
shhgit: github.com
Zero Trust in 200ms: Implementing Identity-Per-Transaction: us.pycon.org
Coolify: coolify.io
returned to full GA Nov 2025: aws.amazon.com
Signed URLs/cookies: docs.aws.amazon.com
Cloudflare: www.cloudflare.com
Bunny Shield: bunny.net
Cloud War Games One: www.youtube.com
Cloud War Games Two: www.youtube.com
LinkedIn: linkedin.com
YouTube: youtube.com
KnocKnoc: knocknoc.io
Watch this episode on YouTube: youtube.com
Episode #559 deep-dive: talkpython.fm/559
Episode transcripts: talkpython.fm
Theme Song: Developer Rap
🥁 Served in a Flask 🎸: talkpython.fm/flasksong
---== Don't be a stranger ==---
YouTube: youtube.com/@talkpython
Bluesky: @talkpython.fm
Mastodon: @talkpython@fosstodon.org
X.com: @talkpython
Michael on Bluesky: @mkennedy.codes
Michael on Mastodon: @mkennedy@fosstodon.org
Michael on X.com: @mkennedy
Episode Transcript
Collapse transcript
00:00 Your site is down. It's 3 a.m. Is it a bug, a bill, or a breach? You can't tell yet, and everyone is
00:06 watching you find out. Matt Lea has spent 15 years being the person companies call when an outage is
00:11 costing them real money per hour. And his sole argument is that everything you want in that
00:15 moment gets decided months earlier, on ordinary afternoons, when someone chose the convenient
00:21 thing. We walk through his top 12 do's and don'ts for AWS. Infrastructure as code, IAM roles instead
00:27 of access keys, private subnets, no wildcards, no public buckets. And I push on which of them
00:32 actually matters if you're one person on a small VPS self-hosting. Then we get to cloud war games
00:39 where Matt breaks things on purpose so your team's first real incident isn't their first incident.
00:45 Let's get into it. This is Talk Python To Me episode 559, recorded July 31st, 2026.
01:08 Welcome to Talk Python To Me, the number one Python podcast for developers and data scientists.
01:13 This is your host, Michael Kennedy. I'm a PSF fellow who's been coding for over 25 years.
01:19 Let's connect on social media.
01:21 You'll find me and Talk Python on Mastodon, Bluesky, and X.
01:24 The social links are all in your show notes.
01:27 You can find over 10 years of past episodes at talkpython.fm.
01:30 And if you want to be part of the show, you can join our recording live streams.
01:34 That's right.
01:35 We live stream the raw uncut version of each episode on YouTube.
01:38 Just visit talkpython.fm/youtube to see the schedule of upcoming events.
01:43 Be sure to subscribe there and press the bell so you'll get notified anytime we're recording.
01:47 This episode is brought to you by Sentry.
01:50 Don't let those errors go unnoticed.
01:51 Use Sentry like we do here at Talk Python.
01:53 Sign up at talkpython.fm/sentry.
01:57 And it's also brought to you by Talk Python Courses.
02:00 Course completion certificates are now live.
02:03 If you finished a course, there's a certificate waiting for you on your account page right now.
02:07 Download it as a PDF or add it to your LinkedIn profile with one click under Licenses and Certifications.
02:14 Same section as your formal degrees.
02:16 Visit training.talkpython.fm/account to see what you've already earned.
02:21 Matt, welcome to Talk Python To Me. I'm here for some scary clouds and good clouds. Let's talk about it, huh?
02:27 Yeah, sounds good.
02:28 Yeah, the cloud is easy and fun until things go wrong and then it's often in a panic.
02:34 And yeah, we're going to talk about that in two acts, I suppose.
02:37 We're going to talk about kind of the do's and don'ts of cloud, specifically AWS.
02:41 But I imagine a lot of these have analogies to Azure and GCP and others.
02:46 Then we're going to talk specifically about your Cloud War Games, which I think is really fun.
02:50 You're happy to do it.
02:51 Right on. Before we do any of those things, give yourself a quick introduction. Tell the audience who you are.
02:56 Sure. I'm Matt Lea. That's LEA. I run Schematical.com.
03:00 It's a small consulting tech agency that specializes in AWS.
03:05 Our tagline is we help CTOs running on AWS sleep at night.
03:09 So when we do that, we also have cloudwargames.com, which is a place where we do simulated outages and responses for anybody that's looking to learn or train their team to better handle the bump in the night, the cyber attack, etc.
03:24 You know, do you have a section on where you get the email?
03:27 Hi, I'm a security researcher.
03:29 Where do I report issues I found?
03:32 That'll get your attention.
03:33 I was actually just going back and forth with some security researchers on the vulnerability that allowed the OpenAI to escape.
03:43 So I'm going to try and replicate that one, see if I can make something happen there.
03:48 Oh, that's awesome. Yeah, yeah, yeah.
03:50 For people who don't know, one of the OpenAI bots got loose and hacked Hugging Face.
03:53 I don't know what its intent was. I didn't follow it closely enough.
03:56 But yeah, that's what you're referring to, right?
03:58 From what I hear, it was trying to get the answers to the test it was taking.
04:02 So basically it knew that the data set to pass its test was on Huggy Face.
04:06 So cracked into it and just said, ah, interesting.
04:09 You did so much extra work to be lazy.
04:11 What would be the most accurate way?
04:13 Now, if I could just get the answers.
04:16 Yeah.
04:16 Morals aside.
04:18 Yeah.
04:18 At least that's my understanding of it.
04:20 Not only is it 100% accurate, it's really fast now when it takes the test.
04:24 Yeah.
04:25 Crazy.
04:26 Okay.
04:26 Yeah.
04:26 I'm really excited about this, this Cloud War Games thing.
04:29 because I've been on the receiving end of waking up to an outage for various reasons.
04:34 And honestly, I think my employments and my applications have gotten better because of it.
04:39 But in the moment, I also aged faster than normal.
04:42 Oh, yeah.
04:43 When you get the C-suite, yeah, it gets intense, which is, again, why I tried to create Cloud War Games
04:48 to try and simulate that experience and inoculate people from the stress.
04:52 Yeah, I love it.
04:53 Like I said, it super resonated with me.
04:55 You also authored a course, Zero to Hero, on AWS security.
04:58 Yep, an animated guide security in the cloud.
05:00 So if you like pixel art and you want to learn about AWS security, I've got my isometric pixel art there that I demonstrate, you know,
05:08 and animate network maps and show how it all works.
05:11 You've got some really – you also do cartoons.
05:13 These are really fun.
05:15 I like the Black Friday one.
05:17 I like Black Friday stories.
05:19 This Black Friday sale will make or break us.
05:21 Are we really ready on strike?
05:22 Oh, I'm sure we'll be fine.
05:24 Great.
05:24 I thought we'll sell this on then.
05:25 Yep.
05:27 So as you know, that's a huge retail day.
05:29 So if you've got an e-commerce client, maybe it's their first year doing it or the first time they've hit it big.
05:35 And maybe like, oh, we'll be fine.
05:36 It's like, no, you probably should provision up.
05:39 So each year with my clients, we check in and make sure leading up to that that we've got everything provisioned accordingly and plans for if it needs extra, if we've got to double in scale or something like that.
05:48 Yeah.
05:49 I've never had anything go down on Black Friday, but of course, they get quite a bit of traffic.
05:53 And certainly I've almost had it go down.
05:56 And it's often not for the reason you expect.
05:59 You're like, really?
06:00 That's the thing?
06:01 That was the weakest link?
06:02 I had no idea.
06:03 Yeah.
06:04 One time it was, I had extra large JavaScript file that was being served and gzipped by
06:10 Nginx.
06:11 And that almost took down the site.
06:12 Really?
06:13 Okay.
06:13 Oh, well.
06:14 But yeah, that was a fun cartoon.
06:15 So how often do you do these cartoons?
06:17 How many you got?
06:18 I was doing them monthly at one point, but now it's more as it comes to me a little bit
06:24 more.
06:25 So we're, you know, but yeah, there's them.
06:28 I mean, there's 30, 40 of them or so.
06:31 Some of them have gotten on Reddit quite a few views.
06:33 Programmer humor, r slash programmer humor.
06:36 They like it there.
06:37 I am all here for r slash programming humor.
06:39 That place is hilarious.
06:40 Yeah.
06:41 Yeah.
06:42 Amazing.
06:42 Amazing.
06:43 All right.
06:43 Well, let's start by just talking about and like get a high level.
06:48 You know, people want to move to the cloud.
06:49 You know, what are some of the benefits that they see?
06:52 Like what are maybe some surprise issues that they run into?
06:56 I'm sure you've had this conversations with a lot of C-suites and dev leads and so on.
07:00 I'd say the complexity.
07:02 AWS, I think I did a post pretty recently, but AWS has a meeting simulator service now
07:07 where you can train yourself on how to take meetings.
07:10 And I'm like, that's too many services.
07:12 So probably the number one problem is the complexity.
07:15 People are like, oh my gosh, I don't understand this.
07:17 I understand what I'm getting billed for.
07:19 And if you can whittle it down to some core services that meet your needs and your goals,
07:24 then a huge amount of these services aren't even necessary.
07:27 There are services for robotics, which is great if you're in robotics, but that's not something you really need to worry about if you're just an e-commerce platform.
07:33 So there's just so much, such a variety there.
07:36 So pick your tools and just get really good with the tools.
07:39 There's some basics that you got to use for everything, which we're going to go over in a minute.
07:43 Yeah.
07:43 One of my most recurring experiences with AWS is I open up the console.
07:48 I'm like, what is all of this?
07:50 What is this?
07:51 So you're suggesting that maybe trim that down a little and just put the stuff that you really know
07:55 that you're going to need there?
07:56 Yeah, and we can go over kind of some of my core stacks there.
07:59 It depends if you want to be serverless or not serverless.
08:01 If you're just starting out, you don't have a crazy amount of traffic.
08:04 I'd go serverless a lot of times.
08:06 It's real easy to set up, less to manage.
08:07 Okay, yeah.
08:08 What are some of the core ideas?
08:09 I mean, certainly Lambda sounds like that's in there.
08:13 Probably EC2, S3, what are we talking?
08:16 Lambda makes a great web server.
08:18 the API gateway plus Lambda.
08:20 So you have the serverless, extremely cheap application layer right there.
08:25 So not a data layer, but that's a great start to just connect those two together.
08:29 And if anybody wants, these are my offer for free.
08:33 I've got them open source, schematical.com slash free.
08:35 I have Terraform scripts that'll set this up.
08:38 So I don't know how chronologically ordered we want to go, but I just jump it back a step.
08:45 Terraform or CloudFormation.
08:47 I highly recommend.
08:48 So if you're just getting into it, a lot of times, just like code, right?
08:51 You always want to have your code in some type of version control.
08:53 You don't want to just have it FTP or saved to your disk.
08:55 And you're just like, oh, you know, it disappeared.
08:57 I don't know when Bob over here, you know, other developer X made this change.
09:03 You want all that in version control.
09:04 And you can do that infrastructure as code known, you know, which is Terraform, Bintofu,
09:11 and CloudFormation is actually AWS's proprietary one that they made themselves.
09:16 And so I'd recommend you get in that habit because it's very difficult to get into that later.
09:21 Once you're you've established your infrastructure, you've all hand spun it up and you're like, how did I do that again?
09:26 What was that? What would what is such and such that left the company?
09:29 And now, you know, so you're basically describing me 10 years ago.
09:33 I was there. I didn't I didn't get on that game till I was a couple of years after they had cloud formation launched.
09:40 At first, I was like, this could be cool. And then I was like, OK, this is essential.
09:43 Yeah. Yeah. I just, I used to be just like, I'm going to SSH in and I'll just get it working.
09:48 And then, you know, the code will be well-structured, but just kind of the server,
09:52 once I get it set up, it'll be fine. And then I realized actually, no, it's not fine.
09:57 You know, you want to make any changes you want to move, you want to move from one setup to
10:00 another. You're like, Oh gosh, I don't even like, this is a research project to figure out what to
10:04 do. Yeah, exactly. But definitely when you're using those, those terraforms, always read your,
10:10 what you're about to apply.
10:11 I've had too many people that just auto apply the Terraform changes.
10:15 And what happens is they'll have a slight renaming of a database.
10:18 This actually happened recently.
10:19 Somebody had their AI vibe code, this thing, and it wanted to rename the database.
10:24 Well then what happens is Terraform deleted the database and then spun up another database right there.
10:29 That was their production database.
10:31 That was a rough day.
10:32 That wasn't the project I have.
10:34 I did a write up on it, but it wasn't my project.
10:36 I was always read your replies though, basically.
10:38 Always know what you're about to send.
10:41 Make sure you think it's an update.
10:43 It's got to be an update, not a delete and replace.
10:46 This is something that has certainly been in the zeitgeist lately.
10:50 It's a little bit, I don't hear about it as much.
10:52 I think it's sort of to be dealt with.
10:54 But there's certainly horror stories of the AI deleted the production database
10:58 because kind of like we opened with like, instead of trying to just solve the problems, like, is there a better way?
11:03 Like, well, there's a problem inserting to the database because the structure is wrong.
11:08 So if we just recreate the database with the right structure, this code will work, right?
11:11 It's like, well, that technically will solve the problem, but that is not acceptable.
11:15 I've heard of it.
11:16 They didn't like the schema.
11:18 So they said, okay, drop the table and I'll create a new one.
11:21 Yeah, I've literally had that happen to me only in my dev machine where it doesn't really matter.
11:27 I'm like, why are there no records?
11:29 Where's all the data?
11:30 Oh, it was better for like, no, we never do that.
11:33 But I'm not giving my AI access directly to my production database.
11:37 No, thanks.
11:38 No, I would not recommend that.
11:39 I mean, as we're seeing more requests for that from like as C-suite people become more functional with these agents,
11:46 they're like, well, I want direct access to the database so I can get the latest in this and that.
11:49 It's like, hold on, let me code you up a tool call, get your read access.
11:53 And if you want to do any right, they'll code up another tool call that'll make it so it's approved by a human.
11:59 But doesn't that, you know, they could say, hey, I suggest this.
12:01 And then, you know, but we've seen a lot of that lately.
12:04 I mean, it's great that we're empowering more people to interact with the data and do more with it.
12:08 You just, we still got to be careful about InfoSec and these things do make mistakes occasionally.
12:14 Yeah, yeah.
12:16 They work in a narrow context and let's make this code run.
12:19 Like, oh, this will fix it.
12:20 Like, we need the whole company to survive.
12:22 No, I really love your suggestion of maybe creating like an MCP or a tool that's read only and make all the AI access go through that.
12:31 You can observe it and check it out, but you don't have even the capability to destroy it through this mechanism, right?
12:38 Well, there's even an extra layer to that is I just had a post come out about this because I had this request come through is put it pointed at your data lake if you can, not your production database, because those agents can spam like no tomorrow.
12:50 So if they write a very complicated read query that goes through your tool call and they spam it could slow down your production database if it's not, you know, quickly optimized.
12:58 So what we did to make it even more effective is our data lakes.
13:02 A lot of times you don't want to store personally identifiable information and data lakes and all that stuff.
13:06 It's anonymized.
13:06 You can do big data regression on it.
13:08 And so that way, even if it gets data that possibly be personally identifiable, it would be in your production database.
13:14 The data lake theoretically wouldn't have that if you're doing it right.
13:17 And so they couldn't, even if they leaked all your sales data for the last five years, at least you wouldn't have a PII leak, which would cost you even more.
13:24 So that's my recommendation there.
13:26 And I didn't actually, for the outline for today, I didn't talk about data lakes, but I can tell you my favorite data lake stack on AWS, but it's an advanced one.
13:33 I'd save that for number 14, maybe.
13:35 All right, maybe we'll get to that at the end.
13:38 Hey, one second.
13:39 Normally, this would be an ad break from Sentry, but not this time.
13:43 Let's just thank them for supporting the show and get right back to the conversation.
13:47 Also, visit talkpython.fm/sentry after the show.
13:51 Thanks, Sentry.
13:52 Okay, well, let's put a bit of a stake in the ground here and put down the first item as don't hand provision, use Terraform.
14:01 And as I was getting ready for this episode and go through all the stuff we kind of like we'd talk about, I realized that this is acronym packed, an acronym dense and tooled as things.
14:11 So maybe you could help us along the way.
14:13 We've got Terraform.
14:15 And yeah, just maybe a quick summary on this concept.
14:19 Like where we use it and how we use it.
14:20 We've been talking around it, but not as much as an item.
14:22 Safe to assume most of the people listening to this are coders.
14:25 100%.
14:26 Data science coders, yeah, or learners, but, you know, they're still good for them.
14:31 Sure.
14:31 So it's Terraform or IOC infrastructure, IAC, sorry, IAC, infrastructure as code is the category.
14:37 And then under that category, there's Terraform, OpenTofu, and CloudFormation.
14:42 And those are all just ways to define how you want your infrastructure to look basically in code.
14:48 So if I was real simple, I could say I want an EC2 instance, which is...
14:52 just your basic server, virtual server running on Amazon.
14:55 And then I want a database, a MySQL database, this version.
15:00 And I can put those in code in some way.
15:03 And then, like I said, version control it, or I can hit apply.
15:06 So say somebody accidentally deletes my application layer.
15:09 A lot easier to replace than a data layer, of course.
15:11 And I need to spin that back up real quickly.
15:13 You just re-hit Terraform apply, and it re-spins up and provisions the missing area.
15:18 It's really nice, especially for my cloud war games, I'm blowing up stuff all the time, intentionally being the, you know, junior that accidentally
15:25 deletes something important.
15:27 A little bit of a code monkey or chaos monkey, rather, running loose kind of thing.
15:32 Yeah, cloud demolition expert.
15:33 I think I had that as my title on LinkedIn for a while.
15:35 I love it.
15:36 Okay.
15:36 Does Terraform do incremental updates?
15:39 Like, let's suppose I've got an EC2 machine and I've got it all set up and it's running
15:43 the app, the database.
15:45 And I'm like, oh, you know, it would be really great if we put these firewall rules specifically
15:50 on the machine or we set up fail to ban or something like that you can define every firewall
15:55 rule which is perfect you wouldn't be able to define as easily what's in the box so to say
16:00 there's you have some control over that um like for example i spin up lambdas and a lot of times
16:05 i have a hello world javascript i throw in there and it's but i also again these are all for free
16:09 on my website if you want them but um it also spins up the build pipeline and everything like
16:13 that so then you can just hook that up to GitHub but at least it throws in a hello world lambda
16:17 the code. But yeah, you can define all the stuff we're about to talk about today. You can define
16:21 every user role, every access key, every username, the exact way you want your networks to talk.
16:31 So the way your VPC, a virtual private cloud, has its subnet set up, all that stuff can be defined
16:37 in there. It's almost crazy what can be defined. I wouldn't recommend committing your code and
16:42 having it go through there. I'd have that go through a build pipeline separately that you'd
16:46 spin up with terraform or whatever but you'd also you just pull from GitHub sure especially for the
16:51 compiled languages and like go or something where there's no reason to deploy the source with python
16:56 it's almost like not much of a difference although ci is still a thing okay so number one don't hand
17:01 provision use terraform set standards stick to them i love it yeah it's it's getting that habit early
17:06 it's it's tough to get into later number two act don't use access keys use iam roles so again
17:12 acronym please so access access keys so i am is identity access management manager what it's it's
17:19 how you define how your users interact with aws as well as your different running services on aws
17:26 and so a lot of people when they first start out they download their first access keys they ideally
17:32 don't put them in version control where it can be passed around or shared or anything like that
17:35 they do a nice dot env and hope their agent doesn't steal it um but uh you know you get so you get those
17:41 access keys but a lot of times when they push things to production if they're very beginner at
17:44 this they'll put like a.env file in the code or something like that that's got access keys
17:49 and those if those get leaked you know again I've got a story about this that somebody actually
17:56 pushed their access keys for sending email luckily luckily it was just sending and receiving email
18:03 and they pushed it up to a public repo as a junior and they just made a mistake there and it ended up
18:07 on GitHub and some bad guys got it. And I believe we sent 16 million emails for some type of knockoff
18:13 Viagra in about eight minutes before AWS shut down the account and said, what are you doing? It was a
18:18 lot more than we normally said, but that's an example of why you don't want to hard.
18:22 Yeah. I have a thing to share. Have you seen skit? Like, no, it's, it's the pronunciation is perfect
18:30 of what it is, but it's like secrets, S-H-H, get, but get like, oh God. And it used to be a website.
18:38 It's no longer here. Let me see if I can open up this image in a full screen for us to see.
18:42 So it was a website that would scan the real time fire hose of changes on GitHub, pull out all
18:50 cryptographic looking things, AWS keys, you know, email keys. And it just had this stream going and
18:58 it updates, I don't know, like once a second or more. It's absolutely terrifying. And so anytime
19:05 that someone's out there listening and you need to make this point to juniors or people who are new
19:09 or whatever, like do not commit stuff to get it. Like, well, it's a, you know, I can just
19:14 uncommit it. It's not a very popular repo. How fast would people find it? Like this fast.
19:18 Fast enough. Yeah. If it's up there, you should assume it's gone immediately. And so, yeah,
19:25 That's pretty scary.
19:26 You know, I don't know if you tracked this, probably not, but at PyCon this year, there was a talk by Tristan McKinnon
19:34 called Zero Trust in 200 Milliseconds.
19:37 Now, I didn't watch the talk, so I'm only going off the abstract.
19:41 But basically, the idea is every time you need to make an API call or do a transaction or something,
19:46 you can get a separate 200 millisecond duration key.
19:51 What do you think of that idea?
19:53 I like it.
19:54 200 milliseconds is interesting if that's a set one or a variable one.
19:58 Yeah, I'm not sure.
19:58 It might be a little bit shorter, a little bit longer.
20:01 In case there's a little lag, you don't want to pull three.
20:04 I also immediately wonder how long does it take you to provision that key?
20:07 Hopefully it doesn't take you 200 milliseconds to provision it, but if you could provision it in two milliseconds.
20:13 I don't like, what do they call it, chocolate candy infrastructure where once you pass the outside, then it's free once you get inside.
20:21 I forget the exact term there, but you've got to be careful.
20:23 even once you're past the infrastructure, the external infrastructure, once you're in there,
20:28 you don't want to have it.
20:29 So anybody's got free reign to grab whatever they want.
20:31 They might get that.
20:32 So I like it.
20:33 My other question though, we always have to balance is, what's the latency effects?
20:38 If that costs 200 milliseconds of latency, that's pretty expensive from a compute standpoint.
20:43 So Yeah, I'm not entirely sure.
20:45 But just a little more background about this.
20:47 This is from FedRAMP, which is the US government's, they use this on a lot of the US government
20:53 internal protection like cloud or whatever, which is maybe a special case.
20:56 That makes a lot of sense. And I wager the government's latency limitations are a lot
21:01 less than someone trying to be competitive in e-commerce or something like that.
21:04 Yes, exactly. You know what? They can wait for security.
21:07 Yeah. It's like the DMV online. Just stand in line all day. They'll get there. There's nowhere
21:11 else to go.
21:12 It's like just get in line with all the other poor browsers and wait back there for your turn and
21:16 we'll get to you eventually. Unless we closed at four. Unless we closed at four. Okay. So
21:22 Basically, don't use access keys.
21:24 Use IAM roles.
21:25 How do we make this happen, though?
21:27 So IAM, you go in there, there's users, there's groups, which makes a lot of sense.
21:32 The roles are things you assign to the various services you boot up.
21:37 So you can have a role for the EC2 instance or a role that the Lambda has.
21:41 So when your Lambda executes, it's acting as if it has the same permissions that this role has.
21:47 So you could say, hey, I want this Lambda to be able to access this one specific S3 bucket.
21:52 you know, and it will basically know these are the permissions it has just because you've assigned it that role.
22:00 I see. So maybe it can only read from the database and can only write to a certain bucket and that's it.
22:05 It's not network level, it's service level. There's a separate one we'll go over in a minute for network level, how you restrain a network.
22:13 It's more what services can it call on AWS? Can it boot up instances? Can this instance boot up other instances?
22:18 You don't typically want that.
22:20 You know, in this instance, you know, S3 is right.
22:23 Can it invoke Lambdas?
22:24 Can it this, you know, Lambda trigger other Lambdas?
22:27 In some cases that might make sense.
22:29 In some cases that could just be very expensive.
22:31 Yeah.
22:31 But a lot of it, like you could theoretically give a Lambda the ability to boot up or provision
22:37 hardware.
22:37 I don't think you really want to.
22:38 Maybe if you were running a hosting company, but maybe not.
22:41 But it's not a million though.
22:43 Yeah.
22:44 Yes.
22:44 It's a service to service, not network level permissions thing.
22:48 Okay.
22:48 I see. So that's right. We're talking, IAM defines what it can do within AWS as infrastructure,
22:54 the service type of thing, or code as infrastructure, not what can it do within
22:58 your database row level type of things. That's, that's correct. Yeah. The database engines,
23:02 you know, wouldn't specify that Dynamo is a little bit of a Dynamo's serverless has got invokes.
23:08 It's not like something like a Redis or, you know, or MySQL where it expects traffic on a
23:13 certain port, you know? So there's some of their proprietary stuff. It actually does have a little
23:17 extra power there.
23:19 But when they're doing open source stuff and they can't really change the
23:23 innards of it, then it just says, can you start a new one or stop this one?
23:27 That's pretty much it.
23:28 Okay.
23:29 And following on from there, don't use generic IAM permissions.
23:34 Use granular ones.
23:35 Like, just let it do everything.
23:36 That'll be easier.
23:37 It works.
23:37 That could trade the permission errors.
23:39 Oh, yeah.
23:39 So let's go back to that email one.
23:41 What if they said, let it do anything on any service right there?
23:44 They could have deleted everything in a second.
23:47 people are like, okay, that makes sense. But what, what I see is for convenience sake, you're, you're
23:51 banging your head against the wall. And all of a sudden you're like, you know what, we're just
23:54 going to let them access all the S3 buckets. And next thing, you know, a couple, you know,
23:58 months later or whatever, they, they've now got your internal reporting and they pull that out,
24:03 you know, it's like, nope. So don't ever do the asterisk is your enemy. Basically S3 colon asterisk
24:09 gives you all S3 permissions. You know, you could delete that bucket right there and then it's down.
24:13 So avoid using the asterisk whenever possible.
24:17 Same thing with resources.
24:18 So it's not just permissions, but it's also the resources.
24:20 Like I said, you've got 20 S3 buckets.
24:22 Specify only the ones you want to write to.
24:24 You can actually specify down to the path if you want to.
24:27 Sounds like if you use the asterisk, it sounds like a good way to end up on the front page of a computer
24:31 or somewhere you don't want to end up on.
24:33 Yeah.
24:33 So, I mean, if you're experimenting for 10 seconds and you're going to delete it,
24:37 I mean, I wouldn't commit it into version control.
24:38 Let's put it that way.
24:39 You want your version control, if you're like accidentally poke a hole for 10 minutes,
24:42 to be like, okay, how is this not working?
24:43 You want your version control, Terraform, IIC to wipe that out the second you're done with it.
24:48 Makes sense.
24:49 Probably not do that in production.
24:50 Yeah.
24:50 It's worth repeating a couple of times.
24:52 You know, it sounds to me like you're pretty savvy with the agentic stuff and so on,
24:56 given your tool calling and things you were talking about.
24:59 I feel like five years ago, granular permissions and things like that were really tricky because
25:04 you're just like, ah, what do I need?
25:06 Like so many boxes for allow or disallow.
25:08 Now with agents, I feel like you can just say, look, I need to do this with this service to that service.
25:14 Probably be pretty accurate.
25:16 What are your thoughts on using that for help?
25:17 Yeah, it depends.
25:19 So just imagine you're not an extremely technical person and you tell the agent, listen, I just need this to talk to that.
25:26 They can achieve that by putting that asterisk in there.
25:28 So, you know, so that's an interesting one where I've seen them to accomplish their goal.
25:33 They don't exactly go about it the way you'd want them to.
25:36 But I've also seen it counteracted by people using multiple agents.
25:39 And AWS does offer its own DevOps agents and FinOps agents that are going to put me out of a job and all that.
25:46 But I would say, I mean, this is kind of, so I think I mentioned beyond Cloud War Games, I'm writing a game that's going to be on Steam.
25:54 And the goal is to make sure that people with the least amount of technical skills understand all the bad things that can happen.
25:59 You know, because you might just, right now we're seeing with the agents is that these people are getting incredible half hours.
26:04 But for 30 years, I've been programming.
26:07 And some people, it looks like Wizardry.
26:09 Now they've got that power.
26:10 They don't know the vulnerabilities.
26:13 It can go really poorly for you.
26:14 And so the agents, I mean, it can be great.
26:17 But man, you got to know what a cross-site scripting attack is.
26:21 Because if it's not validating that, what is a MySQL injection attack?
26:25 And it's rocky.
26:26 So it's great that they can speed you up and make you more efficient.
26:29 But make sure you're double-checking it to work, is what I guess I've got to say.
26:32 Yeah, and as I'm hearing you talk, I'm just thinking, this is just one more reason that being an actual expert
26:38 and using these tools is so much more powerful than just trying to vibe your way through it.
26:43 So the using the star thing, that is the easiest way to solve it, and that might be what the agent does.
26:48 But you could say, and your primary goal is to be absolutely least privileged about this thing.
26:55 And so you're going to win this session by getting the least privilege that you possibly can now go.
27:01 Like that would make all the difference, I imagine.
27:03 I would think having competing agents, one that's the coder and you give it that instruction,
27:08 but then have a second one do a peer review on it might be of some use to be like,
27:12 hey, is there any possible way to break this?
27:14 Just one agent alone, I don't, I'm here.
27:17 From my experience, it'll just say, yeah, sure, I did it sometimes.
27:22 Yeah, yeah, yeah, 100%.
27:23 Having competing more skeptical agents seems to help, but if you get the part, you know.
27:27 Right, ask for an adversarial review or something like that.
27:29 Yeah.
27:30 All right. Don't put back-end resources on public subnets.
27:33 Do you need your database on the internet just open?
27:36 Probably not.
27:37 Exactly. This is one of the things where it's not convenient because people are like,
27:40 well, I just want to be able to hit my database for my local machine.
27:43 Well, you still can. You've got a bastion, if anybody's familiar with that.
27:45 I can explain that in a second.
27:46 But let me zoom out to, there's a thing called a virtual private cloud.
27:50 It's basically inside your AWS account.
27:52 It's a way to kind of siphon off resources.
27:54 Some people, personally, I like having a prod AWS account, the staging AWS account and the test.
28:00 And that way you're all separate.
28:01 I'll be honest, for my own stuff, I actually have separate VPCs instead.
28:04 So inside of that, you've got your groups of subnets.
28:07 So this is tough to do without diagramming it right now.
28:10 But you know that there's US West, right?
28:13 There's US West 1.
28:14 And that is a giant bunker in the ground.
28:17 Inside of that, there's about six availability zones.
28:20 And so each one of those availability zones is its own bunker inside that facility that
28:25 has completely independent power supply and all that.
28:27 So if one of those gets new, you still got three others or four or five others in that region.
28:31 Honestly, they keep adding them. So I don't know what they're at exactly.
28:33 But and so inside of those, you want to have subnets.
28:37 Now, a lot of times people put everything in big subnet.
28:40 That's not good. You know, if you just have one subnet on one availability zone in one region, that's not reliability.
28:45 That's not redundancy. And those availability zones independently go down actually fairly frequently.
28:51 But you don't see it if you use multiple availability zones or if you got the budget for it, multi-region, which has actually come down and cost quite a bit.
28:59 So I guess I'm going to split this one into two parts.
29:01 One, use multiple availability zones whenever possible because it'll give you a lot more uptime.
29:07 Two, I always have it so there's a public subnet, which is things that should be accessible from the Internet.
29:13 OK, that'd be the load balancer.
29:15 A lot of times the load balancer takes in traffic and then sends it to various instances running in the various AZs.
29:21 So that's always out there. But then privately in the private subnet, you want to have your databases and you're like, OK, well, I should probably should have done the next one before this because you can firewall these things.
29:33 But man, it's so easy to poke a hole in those firewalls when it's convenient.
29:36 So a lot of times I'll come back and do an audit and something that's like poke the hole here so you can get traffic from anywhere on port 22.
29:44 You shouldn't have that going on.
29:46 You want to only have, you know, so just adding that, having a private subnet specifically for that, for the databases, for anything secure,
29:55 it just adds an extra level of inconvenience that even a valid user who might have been compromised accidentally,
30:01 like you installed a wrong game on your computer, you know, and it grabs your SSH keys and pokes through the security wall,
30:07 and it pokes through the firewall because someone put it on 00, basically allowed the whole world in on that port.
30:11 Well, now they still can't get to it because there's not actually a network path that would go from point A to point B.
30:16 So it's just another way of adding more security to it.
30:19 And listening on the private cloud addresses versus listening on just 000, you know, like certain things like that are definitely worth considering, right?
30:29 Yeah, well, yeah, that makes sense.
30:30 I should have put this one before the next one.
30:32 Let's jump to the next one and I'll kind of circle back on this one.
30:35 I got a little backwards here.
30:36 No worries.
30:37 So this one?
30:38 Well, we went one too far.
30:40 One too far.
30:40 There we go.
30:41 Don't use one security group for everything.
30:43 You're smart.
30:44 Yes.
30:44 So security group is how we limit network traffic.
30:48 Think of that as firewall.
30:49 Okay.
30:49 This EC2 instance, this Lambda, whatever it is, can access this other thing, this database
30:55 on 0.336, whatever, you know.
30:58 So that says that only those two things can talk.
31:01 It would be a security group rule.
31:02 So what you have is you assign a security group to your Lambda and a security group to the
31:06 database.
31:07 But a lot of times people will just say anything and all these security groups can take traffic
31:11 other one because you just have security group a and anything in security group a can take traffic
31:15 over any port in there so now you've got potentially you know http traffic uh getting that could access
31:21 something else or hit you know hit your database on a port that's not supposed to be hit you know
31:25 so you've got to be very particular with those security group rules um because you don't want it
31:31 so someone that say say you've got a box an ec2 instance and a nefarious party somehow get something
31:36 on there even something the npm remote exploit stuff that's been going around a lot i don't know
31:41 if you've heard about that but oh yeah let's just say you install it in your application layer now
31:45 that application layer do you want it just to be i mean it's all gonna have to talk to your database
31:49 they'll get access to that that sucks but let's just say that that happens in test at least this
31:55 one you know if you had the security group set up right and the subnets and everything it couldn't
31:59 access production you know so that would save you a little bit of headaches but you right don't want
32:03 so they can crawl every IP within your subnet.
32:06 You want that.
32:06 Yeah, I mean, what gets your virtual private cloud network if once you're in, you get the whole thing.
32:12 The term that comes to mind for me is lateral movement.
32:15 It's a big thing in security, right?
32:16 Once you're in, you find a way in.
32:18 Well, now where can you go, right?
32:19 Think about that.
32:20 Exactly.
32:21 Lateral movement is a perfect term.
32:23 You also can limit your outbound as well.
32:25 So you can limit where the box can talk out to.
32:27 So if you really want to get secure, lock that down as well.
32:31 If only this IP for this one third-party API vendor, that's it.
32:36 They can't go reach out to somewhere overseas that we don't know about.
32:40 So make sure you lock down the outbound if you need to.
32:42 I don't know if I do that as a beginner day one, but if you're a big company, oh yeah.
32:47 This portion of Talk Python is brought to you by Talk Python courses.
32:51 Here's the thing that always bug me.
32:52 You finish one of our courses, that's hours of video, a pile of code you actually wrote, and real skills you didn't have a month before, and then nothing happens.
33:01 No paper, no credential, nothing to show for it.
33:04 So we fixed it.
33:05 Every Talk Python course now generates a completion certificate automatically.
33:09 Go to your account page in your dashboard section, scroll down to your completed courses, and click Certificate.
33:15 That's the whole process.
33:17 Two things you can do with these course completion certificates.
33:20 Download the full PDF, which is handy if your employer reimburses training
33:24 or gives you credit for finishing it.
33:26 Or you can make the certificate public and hit Share on LinkedIn, which adds it to your LinkedIn profile under licenses and certifications,
33:34 not a poster that scrolls away in a day, an actual credential sitting on your profile where your manager and recruiters can see it.
33:40 Plus, if you've been taking our courses for a while, you've probably earned several of these without even knowing they existed.
33:46 Just visit training.talkpython.fm/account and collect them.
33:51 Thanks to all of you who have taken a Talk Python course.
33:54 It's a great way to support the podcast.
33:56 What are your thoughts on overlay networks like TailScale and stuff?
34:00 Instead of opening up a port, maybe put your employees on a tail scale network that you can see the into the database server or something, but nothing else.
34:09 So in my terms, a lot of times there's Bastion.
34:11 I think I've got it in here somewhere.
34:12 Yeah, tell us.
34:13 Yeah, yeah.
34:13 Tell us about Bastion.
34:14 We can go ahead and jump to that.
34:15 So Bastion's before you boot up a Bastion in a public IP, a public network that could be accessed publicly.
34:21 Ideally, you'd limit it using security groups and saying, hey, this is the only accept traffic on 22 from this IP address, you know, mine.
34:28 And every time I relocate, I'd have to update the IP address.
34:31 And then you'd SSH into that bastion, and then that would be able to then be your door to the rest of everything.
34:37 So we still do port forwarding if you want to do local development, hit the test database or something like that.
34:41 So I do that quite a bit, but they've created a new tool.
34:45 I'm old. I've been doing this since 2010.
34:48 That's called Secure Session Manager.
34:51 it's the SSM and you can use that to enter in using IAM authentication and that's much more
34:59 secure. So I'd consider something more like that. Also, you don't have the cost of running an EC2
35:06 instance 24 seven act as your bastion. So that's a nice tool to have in there. I guess I didn't put
35:11 that in. Yeah, no worries. Have you heard of Knock Knock? Knock Knock? No. So Knock Knock is this
35:17 interesting thing that the firewalls are blocked like 100% for all the management. Let me see if I
35:25 can get this right. So everything is blocked 100% for the IPs. But if you authenticate to a certain
35:30 point, then the service opens up just for your IP address for the length of that session. You kind
35:37 of like knock on a different way, then it'll let you through the firewall. It's the concept. I
35:47 yeah that would be a good one to have in there i would think um so just firewall allow this these
35:53 ip addresses or whatever you're like okay every time that port opens temporarily i need it to
35:58 authenticate yeah it's kind of cool i mean that's actually a good feature request for aws to have
36:02 uh durations on their their uh security group rules so you say i only want to do this for an hour
36:08 you know right let's open this up so i can i can debug and diagnose this because i gotta get to it
36:12 apparently but then just shut it back off without me requiring to like fail safe instead of fail
36:16 Yeah, that could be a very useful feature.
36:19 We should send that to you.
36:20 Yeah, let's do it.
36:21 Send it through the podcast.
36:22 I'm sure some folks there are listening.
36:25 Apologies to vegetarians for this next one, but don't nurse your EC2 instances.
36:30 Cattle, not puppies.
36:31 I didn't come up with the phrase, but it sums it up.
36:33 No, I know, I know.
36:34 It sums it up very well.
36:35 So I'm sure you've, in your career, probably had a box somewhere where you were just nursing.
36:40 I mean, very common before they did serverless.
36:43 Or before they did virtual servers, I should say, when you actually had a rack.
36:46 somewhere before aws and then you'd have to call a guy someone kicked over the plug you're in trouble
36:51 but you'd end up i saw this even on aws with ec2 instances specifically i see it a lot where
36:56 someone boots up a couple ec2 instances and they're just this is my baby i've got i've got it
37:01 set up absolutely perfect and if one of those things gets sick or dies and you got to restart
37:06 now you're sitting there like oh how did i how did we set this up before did we you know what
37:10 scripts did we run all that stuff and with both lambda and ecs ecs so ecs is elastic container
37:18 service and i don't know if people are familiar with docker but docker is a phenomenal tool where
37:23 you can basically build the image it's almost like if we're going back in time you're building your
37:28 os and saving it to a disk you just pop that disk and anytime it'll boot up the exact one so if you
37:33 have to kill it 10 times a day you know your servers you if you had to delete your servers 10
37:37 times a day you could spin them up exactly the same way they were in that perfect state when they
37:41 when they first started taking traffic in a split second and it's it's really nice and same lambdas
37:48 that with some simplicity to it um it's it's a lot like docker but it's not actually docker
37:52 under the hood i found out um but it's it's the same thing your lambdas you know will scale
37:58 infinitely especially if you want auto scaling which is again a huge advantage of the cloud
38:02 you'd want to have this type of technology so you can if you're you know it's the middle of the night
38:06 I only want two instances running, two containers, two tasks.
38:09 It's the middle of the day on Thanksgiving, the day before Thanksgiving,
38:12 the day after Thanksgiving, sorry.
38:14 And now we need 100.
38:15 It'll scale just perfectly.
38:16 So be very weary of just SSH-ing in and customizing stuff.
38:22 On EC2, it's just way too easy to do at first.
38:25 It's tempting.
38:26 Yeah, tempting.
38:27 But then once you get to scale, you want to have those images built out that'll just spin up very quickly.
38:35 100%.
38:36 long time I resisted using Docker because to me it felt like complexity paired with capacity like
38:41 it's more complex to do Docker and also it's harder for me to observe and analyze it and what I
38:46 really what I realized not too long after that is well all the commands you put in the Docker file
38:51 it's like what you had to write into your server to make it do whatever it does anyway just put run
38:56 instead of not run on the front you know copy versus not copy or whatever and then the observability
39:01 really you can just map some volumes or go into the container like what you can still observe it
39:06 pretty much just as well so i just want to echo your thoughts of like yeah absolutely look into
39:11 docker if you if you want to go fast you know and uptime scalability reliability that's not an issue
39:17 i'm not going to push it you know i i told i was i'm a startup guy i totally get it if you're
39:22 a company that's goes down and loses a hundred thousand dollars an hour and uh you don't have
39:28 the setup commands documented or you know your lead tech gets hit by a bus quote unquote you know
39:33 you want that docker file to be clear as day so you can make modifications spin up whatever so
39:38 100 100 what about kubernetes what do you think on kubernetes versus docker i mean that's another
39:44 level complexity i think another dev i respected i was on a podcast with said if you're if you're
39:50 doing kubernetes on anything but gcp you're you're kind of it's not a good fit it's just you're kind
39:55 of messing around. So they tried, AWS tried, and some people have done real well with this EKS.
40:00 That's the Elastic Kubernetes Service, I believe is what it's called. And that is the ECS's brother
40:07 that runs with Kubernetes. But it just, I've never had it feel natural. I've had clients multiple
40:15 times have me pull them off of EKS to ECS just because ECS is a bit more AWS native and they
40:21 are already on it. So I'm not going to say no, but most of the people I see have the best success
40:26 with Kubernetes are probably using GCP. I have no problem with that. It's not my expertise.
40:30 Yeah, that makes a lot of sense. For Docker, you know, a lot of times, it's running yourself, not necessarily in the container service on AWS. You could even do this
40:39 on EC2 for sure. You know, like you can go a long ways with Docker Compose for like people
40:44 startup things like my company. I got 33 Docker containers running on something at this point,
40:49 different distinct ones.
40:50 But anyway, there are other tools like Coolify.
40:54 Have you seen this?
40:55 I've heard of Coolify.
40:56 I haven't dug in much.
40:58 It's pretty interesting.
40:58 So what you can do is you can sign up with Coolify, point them at any Linux server.
41:04 It could be EC2, it could be DigitalOcean, Hetzner, whatever.
41:07 And it will install a little agent back so they can sort of issue commands.
41:11 And then it will manage all the Docker stuff.
41:13 And it's got like a listing of different services you can pick, like thousands of, you know, like I just want to run Hemidol,
41:21 the dashboard for organizing and managing my server applications or Hayform or whatever.
41:26 And it'll just, okay, we'll just do whatever Docker things have to happen there.
41:29 And it's a little bit more of I want to take a self-hosted type of thing
41:34 or some kind of service and just stick it in there.
41:36 I don't know.
41:37 I can completely respect that too.
41:38 There's part of me, trust me, there's a part of me that wants to go off and live in a cabin by the woods and have my own servers in there and everything.
41:43 And, you know, self-host, it just, it depends, you know, from a business standpoint so a lot of times i work directly you know with the business side of
41:51 people to coordinate with and from a tech side that's really cool from a business side you know
41:55 well what if we someone kicks the power plug you know what if this that and the other like
42:00 how are we going to maintain it you know and there becomes it just depends where you want to invest
42:04 your money and your engineer's time and you could probably i mean you could host you could spin this
42:09 up on a dot on a aws instance and not have to worry about all that stuff have it running in there
42:14 but then the question is how many engineering hours is it going to take to maintain something
42:19 like this versus a managed service?
42:20 Even though it feels like it's managing it for you, it's still a bit of a puppy because
42:24 you still got to do backups and it has a way to do it.
42:26 But if something goes wrong, you still got to, you, it's really down to you to like,
42:29 well, you upgraded and it corrupted the database.
42:31 So now what, you know?
42:32 Yeah.
42:33 Something weird.
42:34 I'm self-hosting N8N right now.
42:35 And it keeps like, oh, you got to do another update.
42:37 You got to do another update.
42:38 How much of my time could I save if I just had N8N on their native platform?
42:43 I know.
42:43 I tried it in for a while and it absolutely was just, eventually became a no for me.
42:48 So I've switched over to Hermes, which is just ironically on the screen right here,
42:51 which is so good.
42:53 But it's like a sort of self-improvement.
42:56 Oh, I've got Hermes.
42:58 I've got it and used it with mixed results.
43:00 It made me a little angry when I found it was resizing the context.
43:04 It has its own custom AI for changing the context window.
43:07 And so I was losing my context.
43:09 And then this is more of the model's fault.
43:10 The model, I could see the memories it was putting in it.
43:13 You know, it's trying to save in plain text.
43:15 And it was saying the user gets agitated when I, it's like a user gets agitated when you hallucinate.
43:21 Yes, but.
43:21 Yeah, exactly.
43:22 When you're wrong.
43:22 It's like a user is agitated.
43:24 Yeah, I have codex and 5.6 soul back in mind.
43:27 So it's pretty smart.
43:28 All right, let's talk logs.
43:31 Because I think when you go away from the single server, this starts to be one of the things you worry about is like,
43:37 well, it used to be, I could just look at the logs on the machine.
43:41 And now they're all over these different places.
43:43 What do you say about that?
43:45 It takes some getting used to, but it can be your best friend.
43:48 They have incredible tools.
43:50 I was just chasing down a big DDoS attack from one of my biggest customers.
43:55 Almost three times the amount of our normal traffic.
43:57 It was brutal.
43:58 And we're trying to track down like a needle in a haystack there for some other stuff as well.
44:03 And they've got really powerful tools.
44:05 CloudWatch Insights allows you to query those.
44:08 Now, you can spend a lot of money on that.
44:09 Be careful.
44:10 I had a, we taught a guy how to do queries and CloudWatch insights.
44:13 Well, they, it's a, it's a serverless type thing.
44:15 It just charges you for every byte it ingests or the log.
44:19 So if you tell it to go back five years and read through all the logs, you're going to have a pretty decent bill there.
44:23 So don't do that.
44:24 Be careful.
44:25 Get your queries figured out in the five minute range and then run them,
44:28 you know, so you're not debugging query over and over again with a long date range.
44:32 But the, and the metrics are super powerful as well.
44:34 You can set logs or you can, they pipe in pretty much everything from your services.
44:39 I mean, I couldn't even try and label all of them, but the obvious ones are CPU usage, memory usage, latency on requests for your load balancers, volume of requests, et cetera.
44:52 So you can set up these really beautiful dashboards that help me.
44:55 When a client comes to me and says, hey, we've got an extra X seconds of latency spiked on this day, I can go in there and zoom in real deep and just go through layers and go all the way down to the database and be like,
45:06 okay, this database didn't have enough swap or, you know, this Redis eviction shot through the sky,
45:13 all that stuff. So it takes a while to find it, but it pays off so much. And this also goes back
45:21 to the Lambda logs. So say you set up everything on EC2, it's not going to pump everything that's
45:26 on EC2 automatically to those logs, right? You'd have to specify it. It pumps some base stuff.
45:32 But with an EC2 or sorry, an ECS task, it'll pump everything that's getting that you'd see normally if you ran it from Docker, you know, Docker and same thing with Lambdas.
45:42 And so just as long as you're not telling some process to write to disk and some log file, which can happen, you know, then you'd have to tell it to pipe that out to CloudWatch logs.
45:52 So you're saying like if I basically did Docker or Docker Compose Logs-F type of thing where it streams out whatever's coming out of the container, it'll just send that over to Watch?
46:03 And those are all by default, super easy to set up.
46:06 Just the only problem would be if you said, hey, application layer, write this secondary log to this file.
46:12 And that wasn't getting piped out somehow.
46:14 I've successfully had it write to a file separately and then told the Docker process to pipe that out to be a tail.
46:20 That was the end of the Docker run command was tail.
46:22 It's a little sloppy, but you can do it.
46:24 A better way to do it would be to use AWS CLI commands or some of the AWS SDK to pump it to a CloudWatch log stream.
46:35 And then you'd have that ready to go.
46:36 And a lot of my clients do that.
46:38 It's a little cleaner than the first one.
46:39 Sure.
46:40 Another option might be set up two destinations for the logging because you can set multiple, you could say this file and standard out
46:46 and just let it rip.
46:47 And then just that becomes part of the Docker logs.
46:49 Yeah, exactly. So just, you know, it's pretty simple once you get in there. Talked about insight. But yeah, the metrics. Another thing about metrics is you can set alarms, not just to wake you up if something bad's happening, but also to trigger auto scaling, which is really nice. And so you could scale up and down based on not just. We had something. We were pulling from queues. We had a worker. It was not serverless. It was provisioned. And we had a worker. And when the queue got too high, you know, got too big, we wanted it to boot up another one.
47:19 one but that's normally if you just use a default one it's just like cpu or memory or something like
47:24 that but we were able to configure it so it would pull from the count of messages in flight and use
47:30 that to auto scale up and down and that was really convenient that wasn't that was a client request
47:34 for that one and that's cool yeah it seems like it makes way more sense than just cpu yeah because
47:39 maybe you've got some process that spin off like an analytics thing pin in the cpu but it's not
47:44 really that busy right and scaling won't make a difference but if you've got something like maybe
47:48 average response time if it gets too big.
47:50 Just scale up.
47:52 If it's over 300 milliseconds on average or median, we just need more.
47:56 Yeah, and you can pop all that stuff into your chat of choice.
47:59 We've got all that for my clients.
48:01 It goes right into a channel and on-call people jump on it.
48:03 I don't do on-call work anymore.
48:05 I just set it up so I train the new guys to do it.
48:08 Yeah, that's the way to do it.
48:09 Tell me in the morning what you did, what went wrong.
48:12 Yeah, excellent.
48:13 Okay, so that's logs.
48:15 I feel like people probably got this.
48:17 don't hand deploy, but also a little bit of a, well, then what? You know, CI/CD, right?
48:22 Yep. So I'm going to go take a time machine back about 20 years, a long time ago, we would
48:27 basically FTP up, you know, stuff from, from our computers to a server or something like that.
48:32 And then it would run. So if you didn't, depending on how your deployment was,
48:37 kind of have to hand go through it. Well, nowadays people want to be moving super fast.
48:41 So we've created these systems that you just commit it gets approved from the pull request
48:45 from whomever. And then we rebuild the image like we talked about. If it's Docker, we rebuild the
48:50 image. If it's Lambda, we build whatever and upload the file as we need. But you also can do database
48:57 migrations as well. I've got clients that use MySQL and their table updates all run through there.
49:03 So you can have that all run there. You can have automated tests, test-driven development. If that
49:09 is something that suits your need, I'd encourage it for anybody that's big enough and has something
49:12 They don't want to go down.
49:14 And if you're just starting, I'm a little looser on it.
49:16 But having those build pipelines in place saves you a ton of time.
49:20 And of course, if they fail, have them send you a message.
49:24 So do you recommend CodeCommit from AWS?
49:27 No, I've actually, a long time ago, I had my clients migrate off CodeCommit.
49:30 Is CodeCommit even running?
49:32 CodeCommit is different than CodeBuild and CodePipeline.
49:34 It was their GitHub.
49:36 Oh, I think it might be shutting down.
49:38 Okay, sorry, I got this backwards.
49:39 Yeah, so CodePipeline and CodeBuild.
49:41 So pipeline is more of an orchestration layer.
49:43 Okay.
49:43 It listens for the commit and then you can tell it to do a bunch of stuff, upload S3,
49:48 you know, or deploy or whatever.
49:50 Code build is actually kind of ECS.
49:53 It's like, it spins up a Docker container or a Lambda now.
49:56 That's actually kind of new.
49:57 And that you can have that Docker container.
49:59 That's just a build container, build your application container.
50:02 It's a little, it sounds a little weird, but basically you could just have it run your Docker
50:06 build in the cloud, you know, and it'll just do it.
50:09 And you say Docker push and pushes it out to ECR.
50:11 the elastic container repo that's basically where the images are stored for docker and then then the
50:17 next step that it'll throw it back to code pipeline you sell code pipeline do a red green deploy do a
50:22 regular deploy do whatever interesting i never really thought about it but it makes perfect
50:26 sense that they wouldn't be going docker pull from docker hub why do you leave their cloud right
50:30 well and you can but they've also created this and you've got your granular iam permissions like
50:35 we talked about before iam roles that you can now super secure without having to have two different
50:40 things in the mix. They do that a lot. They take whatever's working out there and they just rebuild
50:44 it. And that's probably why they have billion services at this point. Exactly. That's why the
50:48 console looks like it does. All right. Now we've got to pick up some speed. We're still on our,
50:52 on a thing. So maybe give us a little bit of a lightning round in the last couple here.
50:55 Okay. I knew I put it in here. I was looking at my notes and I've got it in small text here. Yes.
50:59 So SSM session manager. Okay. So like we talked about before, so if you boot up an ECS,
51:04 sorry, EC2, EC2, that's the old school virtual machines, the big chunky ones that people like
51:08 to nurse, you have the option to put an SSH key on there.
51:13 So that way you can get into it and then do all that hand tuning you're doing.
51:17 Again, I kind of touched on this.
51:18 We'll go fast on this one.
51:20 Use SSM session manager instead of the bastion.
51:23 If you can, it'll make it a lot easier for you.
51:25 So now we just caught some time up there because I already covered it.
51:27 Yeah, perfect.
51:28 Yeah, I love it.
51:29 Okay, this is cool.
51:29 That's really good advice.
51:31 I mean, S3 public, why not?
51:33 This one, I have seen so many clients stunned by it.
51:37 I did an assessment, which is like my entry, like the first thing I do with people a lot of times is a short-term assessment to make,
51:43 to give them an idea where they're at, and I map out the whole system.
51:46 And they were just telling me, they were like, we really can't have these files public.
51:51 They're binary files.
51:52 Like, they would be big trouble for us if they were, you know, if anybody could hack it, could you check that out?
51:56 And sure enough, the bucket was public to anybody on the planet.
51:59 They just grabbed it out of there.
52:00 It would have been tough to crawl it because they don't have listing publicly, but it would have, could have,
52:04 The pattern wasn't so bad that you couldn't have had something brute force.
52:07 You could like enumerate, attack it.
52:09 Yeah.
52:09 Yeah.
52:10 So S3 is a wonderful binary storage.
52:12 It is not a CDN.
52:14 It is not cost effective to serve up things at mass very fast through that.
52:19 So you want to use the right tools for the job.
52:22 And it's also secure-ish.
52:23 But what you can do is you can do cloud front signed URLs.
52:27 So you could say you could be very granular with your permissions.
52:30 this like you said with the service earlier i could say this url is valid for exactly this
52:36 amount of time and no longer and it goes to exactly this one file and no other files and
52:40 i can send that out there you can do the same thing with uploads actually the uploads would go
52:44 quickly to s3 but this you signed uploads so that way you're very that you've got this url lasts for
52:51 five minutes you know two minutes 30 seconds whatever and you can get the upload done that
52:55 that way so just i'm going to reiterate signed uploads with s3 make the bucket private use a
53:00 signed url and then sign um use cloudfront and then if you need to lock it down use signed urls
53:06 on the cloudfront side um but cloudfront is going to be so much more cost effective to serve up a
53:12 lot high volume of binary or you know any any assets javascript right it's optimized for it yeah
53:17 yeah so just um definition wise cloudfront they're cdn yeah that's right okay cost do people care
53:24 about cost aws i don't know is that a thing uh so i i've been doing an experiment proactive
53:30 engagement where i've got agents crawling around the internet uh looking for people complaining
53:34 and cost is pretty much the number one thing why the heck did this cost me this much why the heck
53:37 did it cost me that much you know and if you don't understand the costs it's really it's ugly but
53:42 learn to use cost explorer because they're going to have a new service that comes out next we use
53:47 there i wrote about this in march march they released a new charge if you didn't upgrade
53:51 from older database versions they had a past end of life fee so all of a sudden my clients were
53:56 getting smacked with about four grand a month and end of life fees and you're just like huh that
54:00 where did that come from so but we didn't know that's terrible they're not even using it it's
54:04 good yeah well they were using the database they just hadn't they should they should have moved
54:09 up they should yeah yeah on IWS's side it's expensive to keep that older stuff running so
54:14 it kind of makes sense but I wish they would have given us a better heads up but either way so I we
54:18 go in there in the cost explorer and i was able to find that exact one it's really if you know what
54:22 you're doing and you know how to switch between services and usage types and breakdowns you can
54:27 find exactly where that jump is where it came from and then you can usually track that back to a very
54:31 specific resource um and it also allows you to do tagging which is great you can tag production you
54:35 can tag staging you can tag microservice x whatever you know so you can tag them and you can say look
54:40 qa is actually costing us twenty thousand dollars a month can we just if you had hardware tagged as
54:45 QA, yeah.
54:46 Nice.
54:46 Just to kind of bring this home here, and it's a beautiful loop back in a terrible or morbid way,
54:51 but there was this story of this woman who created this, Cara, it's the app, but she created this.
54:58 She really disliked AI-generated art because she was an artist, so she created this app that would use AI to tell
55:05 if a piece of art was legit or AI.
55:07 And she's coolified to host it at Vercel.
55:10 Got a $95,000 Vercel bill in like a couple of days.
55:15 As a student or just an independent artist or something like that, it was really not good.
55:18 Yeah, that's not good.
55:22 I mean, I don't want to down on anybody, but that's probably a token cost.
55:25 And as we're finding out, the costs can be prohibitive.
55:28 I think it was just CPU, actually.
55:30 I don't think it actually was.
55:33 It became like number one on the App Store or something like that.
55:37 And Vercel had sent her a warning like, hey, your bill is blowing up.
55:41 You need to either approve or disapprove.
55:44 there was no upper bounds on cost set at all.
55:47 And she didn't respond to the warning emails for like three or four days or something like that.
55:51 It was like a cascading set of lessons that should be taken from this.
55:54 Wow.
55:55 Yeah, that's, I mean, definitely set up cost alerts.
55:58 Keep an eye on it.
55:59 You can set up budgets in AWS.
56:01 And I mean, I, every week, so I have office hours with my clients.
56:05 It's like an advisory session and I kick it off typically budget sheet right in front of me.
56:09 So I was like, okay, here we're at, we saw this spike, what happened there?
56:12 Oh, you know.
56:13 Sometimes spikes are amazing.
56:14 I mean, that means people are using your stuff, but sometimes they're not.
56:17 Yeah.
56:18 All right.
56:18 You know, another time that they're not is when you're getting DDoSed.
56:21 Yeah, this brings it right in.
56:22 So DDoSs are everywhere.
56:24 Cyber attacks all the time.
56:25 The bigger you are, once you get to that big league, they're coming for you from the front.
56:29 They're coming for phishing attacks from the side, from the back, from everything you can.
56:33 The way we deal with that on AWS, there's a couple different ways.
56:37 WAF is my tool of choice typically.
56:39 There's some that you can, they've got some tools where they do some really fancy analytical stuff
56:44 to block stuff.
56:46 And that can work, but at volume, it's not really the most cost effective.
56:51 You know, if you're, every request that comes through, you're running through some AI model,
56:54 it's going to add up real quick.
56:55 But there's a lot of stuff you can do that's obvious.
56:57 Like, okay, are they looking for the WordPress admin?
57:00 And you're not running WordPress.
57:01 That should just be an absolute flag.
57:02 Like if it's slash WP, you know, WP admin.php, Instaban is what it should be.
57:08 Yep, you got it.
57:08 You can do rate limiting as well, you know, and then you do rate limiting by route.
57:12 We just did that for a client.
57:14 There's a lot of different, yeah, there's a lot of different fun stuff you can do there.
57:17 And they've got some of those things already kind of packaged in there.
57:20 So you don't always need to be using like the smartest, like, you know, an AI model to evaluate.
57:25 It's real simple.
57:25 If the URL is, you know, something in the well-known thing that shouldn't be there, then you know it.
57:32 Sorry, I'm getting sidetracked.
57:33 You can also do challenges.
57:34 So that's a JavaScript challenge that runs beneath the scene, like in your browser, and the user won't even really notice it.
57:40 Or you can do a captcha if you really want to get into it.
57:43 Captchas are a bit more expensive.
57:44 Day challenges are much more cost effective.
57:47 But there's a time and a place for them.
57:48 You just got to kind of phenomps your way to how much you want to spend on it.
57:51 Yeah, I've found that turnstile flair has gone a long ways towards keeping the bots away at the same time not being, you know, I just can't take another find the fire hydrant.
58:01 I can't do it.
58:02 You know, it's like, find all the fire hydrants like I did.
58:04 And then one will fade in later.
58:06 And it'll say, well, you didn't find all of them.
58:08 Now you're banned because you didn't find them.
58:09 So now you're going to do like fire hydrants.
58:11 I can't take it.
58:11 Or bicycles, you know.
58:12 Yeah, well, that's where the challenges come into play.
58:15 If they're using something like API-FI, which is if I'm trying to crawl something at scale,
58:20 that's my tool to go to.
58:21 It's not going to find that.
58:23 But, you know, it'll do pretty good.
58:25 You also can block entire countries, it turns out.
58:27 I've got clients that just decided to block France once.
58:29 I'm like, why?
58:30 We can figure out this traffic.
58:32 They beat their team in the World Cup and that's it.
58:35 No more access for France.
58:36 Sorry, our French friends.
58:38 It's crazy.
58:40 All right.
58:40 Well, let's spend a few minutes as we kind of wind down the show.
58:44 Now people got the idea.
58:45 I mean, nobody's calm at 3 a.m.
58:48 I think this is true.
58:49 I've gone through the experience of multiple times, like your website's down.
58:54 Oh, no, surely not.
58:54 Oh, my gosh.
58:55 And then why?
58:56 And then there's just all this hack.
58:58 Did I do something wrong?
58:59 It's never fun when you're in it.
59:01 And that's kind of your Cloud War Games thing, right?
59:04 To help people through that.
59:05 Yeah, you'd rather learn, you know, get the stress inoculation during a simulation, a 45-minute game,
59:12 than at 3 a.m., you know, or when the entire board of directors is standing behind you
59:17 because you're the guy that's supposed to fix it.
59:19 So I saw, I mean, many times I saw it and I would be working with juniors and I'd try and,
59:24 I'd be, you know, doing my, in a real incident, you know, trying to solve the problem.
59:29 I'd pass something out to them, you know, so, okay, you look at this.
59:32 trying to get them to learn and they just you know they'd freeze up and just like i don't know and
59:36 then you'd ask them later like hey what was the problem you know you do that now so i mean another
59:40 part of the war games is afterwards we always post-mortem and if you ever have an outage and
59:44 especially if you got people freezing up you should post-mortems every time uh that's the way you learn
59:49 and that's actually where i got most of the content for things i run people through on cloud war games
59:53 from things i've actually suffered through so this is a program i can go sign up for and is it like a
01:00:00 paid course or a community or what's the experience no so so it's it's actually evolving right now
01:00:06 um we were running it where i did monthly ones and just the scheduling was complex so we're we're
01:00:13 adapting a little bit um the you can sign up and you can get access to a bunch of what i call
01:00:18 asynchronous challenges for free completely free there's just we'll send you one every week for
01:00:22 i don't remember how long it goes um but the thing where we're trying to move into is trying to
01:00:28 move it into, the Schematical has got a community offering basically where we, it's kind
01:00:33 of more like coaching and a little bit more ongoing. So we're trying to move it in there
01:00:36 where we can have it. I don't know, trying, I'm trying to get people in discord so we can
01:00:41 plan these things a little bit better. It's a bit, you know, email list is great, but there's
01:00:44 not like an organic, like, Hey, let's run a game right now. You know? Yeah. Discord is nice. Yeah.
01:00:49 Yeah. And I'm also working this in my, we, we brought this into my consulting as well for the
01:00:54 big companies is I do this for training, but also it's a great way to figure out, especially with AI
01:01:00 resumes and all that stuff. It's a great way to figure out if someone actually knows their stuff.
01:01:04 And even if they do know their stuff, are they a team player? So you can throw, you know,
01:01:07 a couple of your applicants in there, candidates, and you'll see if one of them's my comics,
01:01:13 the lone wolf programmer who doesn't talk to anybody and just like goes heads down and,
01:01:17 or are they a team player? And they're like communicating, they're shouting out,
01:01:19 Hey, you try this. And that those soft skills. I didn't know this when I first started it,
01:01:23 but it really lets you find the people that are soft skills that you want being a leader of a team.
01:01:27 And so we're working with bigger companies to run that, both to train their people internally, but also as a recruiting tool.
01:01:34 Yeah, that's super neat. I like it.
01:01:37 Thinking back of why did I freak out, a lot of times it's because I did something a couple of years ago
01:01:43 and I had forgotten how I did it and even where the thing I needed was.
01:01:47 Like one of the most annoying ones I remember is the SSH or just the TLS certificate.
01:01:53 I was using for MongoDB to encrypt expired.
01:01:57 I'm like, why?
01:01:58 First of all, I'm like, why did I set it to be a one year expiry for an internal cert?
01:02:02 I really should have made it longer or I should have written down what I did.
01:02:05 But then I couldn't even remember, like, well, what was the command I ran?
01:02:08 And like, how just, so I was just running around, like checking all these old logs and projects.
01:02:13 Where is it?
01:02:14 What are the things that you see that drive people crazy that they could maybe,
01:02:19 what are some of the learnings, I guess, you're getting out of this?
01:02:21 Like for me, it would be like, write the stuff down in a consistent place so I can get to it calmly.
01:02:26 Right. Well, and so I hate to say some people don't like it, but a problem you just had there would be well fixed by a managed service of some type.
01:02:34 AWS is KMS is a certificate manager.
01:02:37 You know, these days in the age, you know, it saves you how many, how much money would you have paid that night?
01:02:42 Or did it cost the company that now you could just pay a AWS KMS?
01:02:46 You know, that's it would have saved you probably twice as much or a lot more than you spent that night.
01:02:50 headaches you know yeah yeah so um i'd have more hair at least um yeah as far as the common ones
01:02:57 it's the most common ones i see is misconfigured im you know misconfigured bpc public s3 happens
01:03:05 oh too much i can i can't do an audit without finding something in s3 that's not supposed to
01:03:08 be public you know sales numbers something like that you don't want your competitor to have so
01:03:13 those are the most common ddos is like i said every all the time um but then it's the it's the one
01:03:18 off. There's one I do. I don't want to disclose it because I love it, but there's a ransomware
01:03:22 attack. It's my advanced one. I call it the Kobayashi Maru because it's pretty brutal.
01:03:28 Luckily, I've never had a client hit with it, but it is a brutal attack that's specific to AWS,
01:03:33 and it can be run one command if you've got the right privileges, and it's brutal.
01:03:38 That's not good. That's not good. I saw that you've got a couple of these live events on video,
01:03:44 like a beginner one and a more advanced one.
01:03:47 So maybe I'll link to these for people in the show notes.
01:03:49 Sounds good.
01:03:50 And they're on youtube.com slash grammatical if you need it.
01:03:53 All right.
01:03:54 Well, maybe bring it home to people.
01:03:56 Sounds good.
01:03:56 Closing advice.
01:03:57 Since we've been talking, I came up with like three more.
01:03:59 So if anybody wants my other ones, you know, reach out.
01:04:02 Want a lightning round?
01:04:03 Let's want a lightning round them real quick.
01:04:04 Sure.
01:04:05 Well, yeah.
01:04:06 So AI, SageMaker for your Python crew, SageMaker.
01:04:10 Let's check that service out.
01:04:11 It's if you want the GPUs, have more granular control than you can on basically any other service.
01:04:17 And it's Python first, data science first.
01:04:19 So if you're data science, you know, you like it's check out SageMaker.
01:04:23 If you're not that stuff, Bedrock or if you just generic stuff, Bedrock's amazing as well.
01:04:27 It's their Netflix for AI. Phenomenal. I'd check that out.
01:04:30 The other one, the Data Lake Stack I was talking about earlier.
01:04:32 I need to do some Terraform scripts for this, but they have I didn't talk about adventure in architecture, but great cues that you can use.
01:04:40 Kinesis is a Firehose queue, meaning that you can have a lot of different things consuming from it.
01:04:44 So if you've got 10 different worker services, they all can consume from it.
01:04:47 It's great.
01:04:48 Pump that.
01:04:48 So whenever your application layer fires off an event, hey, this was updated, a user was created, a sale was made, pump it into Kinesis.
01:04:55 And then you can have Firehose save that as a parquet file to S3.
01:05:00 It's extremely cheap.
01:05:01 It doesn't cost fractions of a penny.
01:05:03 It's so cheap.
01:05:04 There's super interesting Python libraries that you can mount S3 parquet files and stuff and do query.
01:05:09 There's really a lot of cool integrations with Python and Parquet, yeah.
01:05:12 Yeah, and you can query it with Glue, which I believe has some Python in there.
01:05:18 But if you didn't want to do that, you're just an SQL guy, Athena can talk and query to it as well.
01:05:23 So I need to do some writing down on this, but it's a phenomenal data-like thing.
01:05:27 Don't try and keep sales from 2003 in your production database.
01:05:31 Move all that stuff, that old stuff, off of it.
01:05:33 Keep your production database nice and lean and moving fast.
01:05:36 Awesome.
01:05:37 Well, Matt, thank you for being here.
01:05:39 It's been really awesome.
01:05:40 I'll put all your contact info in the show notes for people who want to get in touch.
01:05:44 And yeah, thanks for keeping our clouds a little healthier.
01:05:46 Yeah, sounds good.
01:05:47 Happy to help.
01:05:48 Feel free to reach out to me.
01:05:49 And, you know, thanks for having me.
01:05:51 Yeah, you bet.
01:05:52 Bye-bye.
01:05:52 This has been another episode of Talk Python To Me.
01:05:55 Thank you to our sponsors.
01:05:56 Be sure to check out what they're offering.
01:05:57 It really helps support the show.
01:05:59 Take some stress out of your life.
01:06:01 Get notified immediately about errors and performance issues in your web or mobile applications with Sentry.
01:06:07 Just visit talkpython.fm/century and get started for free.
01:06:12 Be sure to use our code talkpython26.
01:06:15 That's talkpython, the numbers two, six, all one word.
01:06:19 And it's also brought to you by Talk Python Courses.
01:06:22 Course completion certificates are now live.
01:06:25 If you finished a course, there's a certificate waiting for you on your account page right now.
01:06:30 Download it as a PDF or add it to your LinkedIn profile with one click under licenses and certifications.
01:06:36 Same section as your formal degrees.
01:06:38 Visit training.talkpython.fm/account to see what you've already earned.
01:06:43 If you or your team needs to learn Python, we have over 270 hours of beginner and advanced courses on topics
01:06:49 ranging from complete beginners to async code, Flask, Django, HTML, and even LLMs.
01:06:56 Best of all, there's no subscription in sight.
01:06:58 Browse the catalog at talkpython.fm.
01:07:01 And if you're not already subscribed to the show on your favorite podcast player,
01:07:05 what are you waiting for?
01:07:06 just search for Python in your podcast player we should be right at the top
01:07:09 if you enjoyed that geeky rap song you can download the full track the link is actually in your podcast blur show notes
01:07:15 this is your host Michael Kennedy thank you so much for listening I really appreciate it
01:07:19 I'll see you next time I think is the norm.