Do we talk about running Python in production enough?
I can tell you the Talk Python infrastructure, courses, podcasts, APIs, and so on get a fair
amount of traffic, but they look nothing like what Google or Instagram or insert big tech
name here's deployments look like.
Yet, mostly, we hear about interesting feats of engineering at massive scale that, while
impressive, often is outside the world of most Python devs.
On this episode, we have three great guests who do think we should talk more about small
to medium-sized Python deployments.
Emily Morehouse, Hennick, and Glyph.
I think you'll enjoy the conversation.
They each bring their own interesting perspectives.
This is Talk Python to Me, episode 352, recorded January 12, 2022.
Welcome to Talk Python to Me, a weekly podcast on Python.
This is your host, Michael Kennedy.
Follow me on Twitter, where I'm @mkennedy, and keep up with the show and listen to past
episodes at talkpython.fm.
And follow the show on Twitter via at Talk Python.
We've started streaming most of our episodes live on YouTube.
Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming
shows and be part of that episode.
This episode is brought to you by SignalWire and Tonic.ai.
Please check out what they're offering during their segments.
It really helps support the show.
Emily, Hennick, Glyph, welcome to Talk Python to Me.
It's so good to have you here on the show.
And I'm super excited about talking about running Python in production.
And I'm glad that it's more than just one of you, because I think everyone has their own
perspective and their own context and whatnot.
You know, do you work for a small company or a large company or do you have many people on
the team or just a couple, right?
And I think that really influences what you might cover, some of the decisions you might
make and so on.
But before we get to that, let's just start off with a little bit of background.
Emily, it's been a while since you've been on the show.
Maybe, you know, tell people a bit about yourself.
Yep.
So I'm Emily Morehouse.
I am the director of engineering at Cuddlesox.
We are a digital product development company.
So we kind of touch on anything from web, mobile, IoT, DevOps.
We really touch the whole stack and get to work with a lot of different industries.
And then I also am a Python core developer and the PyCon chair for this year.
Yeah.
You just happened to grab that role right when, you know, COVID hit and conferences got insanely
complicated and uncertain.
How did you juggle all that?
It was a really big bummer.
I think because PyCon is really such an important piece of our community and getting to see people
in person and connect on a regular basis.
But of course, we wanted to do the thing that was going to keep everyone the most safe.
So kind of last minute going online and then doing a full online conference and then still
hoping that things will settle down this year so that in person is something that feels very
safe for us to do.
But I was grateful.
I actually wound up staying on for this third year as chair just so I could have that sort
of like last hurrah of hopefully getting to be in person.
Yeah.
I hope so.
Awesome.
Well, thanks for all your hard work on it, even if we didn't get a meet in Pittsburgh.
Glyph, you want to go next?
I'm Glyph.
I am probably best known for Twisted.
I've worked on a variety of production Python environments in my career.
I am currently in the process of exploring some options for something independent, all
of which are secret and I can't really say anything about here yet.
Most recently of Pilot.com.
They are a fantastic company and I very much recommend people work there.
And we also ran a ton of Python in production over there.
I'm sure that folks in the Python podcasting space have probably heard me occasionally before,
so I won't belabor my introduction.
Yeah.
Sounds good.
Well, good to have you here again.
Henrik, welcome.
Welcome.
Yes.
Thank you for having me.
I think more or less my first Python podcast in my life.
Well, it's awesome to have you.
You do so much writing and you have such an influence on through like conferences and presentations.
You should definitely be here.
So it's long overdue.
Welcome.
It's great to hear because I don't think about it like myself.
But so I'm Henrik.
I work for a small web hosting company.
We are like a traditional web hoster.
I've been working there for now like 14 years, which is a very long time, which is like a thousand
years in Silicon Valley time, I think.
Everything I do, I do in Python basically there.
Like from web services to simple web pages over proxies, applications, you name it.
We use Python for it.
Oh, fantastic.
Do you like control VMs and provision VMs and stuff and that kind of thing?
Or what type of work are you doing?
We are more on the, no, we don't sell VMs.
We don't sell anything where you can have root on it.
So we're selling a platform basically.
Like mostly PHP.
Like it is what it is.
Yeah, sure.
At least you can control it all in Python.
Yeah.
Well, fantastic.
And so the reason we're all here today actually is it's your fault, Henrik.
So you wrote this article back about a year ago.
Yeah, check the date.
Yeah, I know.
We've been talking about getting the four of us together for a while, haven't we?
It's been a bit of a journey.
So we didn't mean for it to be that long, but I don't think there's anything that is dated
about this.
So it's totally fine.
What I meant is that it was like right before Corona.
Like when I was, I was writing this article on my phone while listening to a podcast in
a van going to a husky farm in Finland.
A husky farm like the dogs?
Yeah.
I did a husky safari basically.
Like riding husky sleds, mushing.
Yeah.
Oh, that sounds interesting.
It was.
It was my last trip before Corona.
So, which makes it kind of funny because I talk about conferences later on, which, yeah.
Yeah.
That kind of got put on hold, but well, we can replace it with things like this and other
conversations.
So in this article, you talked about the title is Python in production.
Of course, we'll link to it in the show notes.
And you said you were missing a key part from the public Python discourse and would like to
help change that.
Basically, we should have more conversations about running in productions.
So let me see if I get this right.
You're a huge proponent of microservices.
Everything should just be a bunch of small microservices and everyone else is doing it
wrong.
I'm just kidding.
You actually don't take a different view than that.
But we're going to go through, you know, sort of some of the themes in your article, some
of the other ones out there.
And then also just all of us talk about sort of what we're doing in production, some of the
considerations and trade-offs.
So a lot of fun there.
Let's talk about cloud computing first.
I'll pick one of these out of here and then we can touch on the whole microservice monolith
thing as well.
You know, Glyph, I'll just pick you.
We have this trade-off, right?
We could go and create a VM, set up our own system, and it's totally portable.
It could run in a data center.
It could run in the cloud or whatever.
Or we could go all in with Lambda and other, you know, very specialized services tied to AWS
or to Azure or whatever.
What are your thoughts on this whole running in the cloud story?
There's as many dimensions to that question as there are services in AWS, which is to
say like a higher order than ALF null infinity.
The way that I look at that question is you have to look at each service kind of one at
a time.
And the way that I would generally suggest people decide whether or not it's a good idea to adopt
a particular service has to do with the way that their operational footprint works in terms
of overhead.
So for example, should you run your own Postgres or should you use RDS?
Almost always you should just use RDS because the operational footprint of that is Amazon configures
and manages the tremendous complexity of operating a relational data store in production.
And when you need to set that up for development, because like part of production is having a
development environment that matches closely enough that it works well and that you can test
things and know for sure that you're getting like adequate test coverage is you can just run a
Postgres Docker container.
And functionally for your infrastructure, it's probably the same.
You rarely need to do like deep configuration that is like meaningful to your application
between your local development Postgres container and your RDS instance.
As that slider moves further and further towards like, well, actually our data store configuration
is really part of our application.
Well, actually we need to like tune things very closely.
Even running your own data store might make sense.
That's a pretty unusual circumstance, but that you can look at that for other
aspects of your platform as well.
Routing, like load balancing, caching.
Is that part of your app?
Do you care about it in development?
If it breaks in an interesting way, is that going to like take you down?
And look at each of those issues for each cloud service.
How much, how expensive is it going to be for you to replicate it in development?
And how accurate is that reflection going to be of what's in production?
And always try to pick the answer that is the lowest overhead for your particular team
configuration.
That said, the tricky part is this changes as your team grows.
Right.
And as your service gets more complex.
One of the things that I imagine I'm going to touch on a few times is that probably the
most interesting Python in production experience I had was running pilot.com's application, which
started in a very different place than it ended up.
And I'm very happy with the way that that went.
But we started in an incredibly like super macro service, just one big Python container
and like nothing else towards picking up application load balancers and lambdas and all kinds of
other stuff that eventually became part of that infrastructure as the team grew.
We had more of a dedicated infrastructure, like operational footprint that we could actually
manage because it was staffed.
Yeah, that makes a lot of sense.
It definitely changes as your team changes.
Emily, what do you think?
What's your view on this?
How much of the cloud should you bite off?
And where's the trade-offs?
I agree with a lot of the things that Gliff said.
I think that once you get to the point where you're adding complexity to your local development
environment, that's usually a red flag for me.
And you have to be doing something that gives you a lot of value.
So for example, we'll use like Firestore as an offline database for mobile applications.
And whether you're using that or you're using CouchDB, whatever choice you're making there,
you're going to have some sort of added complexity locally.
So yeah, I totally agree with like RDS is a really easy swap.
And if it's RDS on AWS or moving to GCP or Peroku or whatnot, your portability between
those different ecosystems is going to be pretty straightforward.
But if you want to rewrite your Lambda functions to use Cloud Run instead, like it's not just a
drag and drop sort of thing.
Right.
You don't just change a connection string and now it's all good again.
Yeah.
And I think it's going to depend on, you know, like where the company is at.
Again, for us, like we're often looking at what our client needs.
So if they're a client that doesn't have a technical team that they just kind of want to
like let this run on its own and they don't want to manage it, like that's going to impact
whether we choose Heroku or AWS.
So like really looking at where you're at now and what sort of support you need in the future
is a big question to ask for that decision.
Which way does that gauge go for you if they are kind of hands-off and not super technical?
Do you give them Heroku or what do you give them?
Yep.
If they're non-technical, we give them Heroku.
If they are technical, even then that kind of depends.
A lot of times it's AWS or GCP, just kind of depending on what other pieces of the ecosystem
that they need.
Or since we work with a lot of existing tech companies, a lot of times they say,
hey, we are a GCP company or an AWS company.
And then we just say, cool, we can do either.
Right.
And if they're already there, you might as well just keep going with that, right?
Yeah, definitely.
Yeah, absolutely.
But I will say that like 95% of the time, anything that's like web application or API based,
it's going to be in a Docker container anyway, just to give us that portability.
Sure.
Hynek, thoughts?
I have very, very little first-hand experience with cloud services because we use none.
It would be kind of odd for a cloud hosting company to use somebody else's cloud.
Although I know that people are doing it.
It totally happens.
People are more or less reselling the big cloud.
That's true.
Yeah.
Wrapper.
Yeah.
We run our own hardware in own cages in a shared data center, like military grade.
I once almost got tasered because I took a photo from the outside.
Security came running immediately.
So the one thing we run outside is Sentry because that makes sense.
And I didn't want to run it anymore myself.
But there's also the thing that Europeans in general and Germans in particular are not very
excited when you put their data on other people's servers, particularly of US companies.
So often it is also like a competitive advantage for us to just say our data literally does not leave Berlin.
Right.
That's certainly something that here in the US, it's easy to kind of forget about.
Right.
Oh, these just the thing of, oh, it's just the big clouds.
Right.
These are big cloud companies.
But really, those are US cloud companies.
Right.
And if you're in Europe or somewhere else, that's another angle to think about.
Right.
I have maybe one thing to add as a user.
That as a user, I don't care that US East 1 is down.
But you will know.
You'll still know.
Yeah.
This is what I see happening.
Right.
Again, people rely too much on cloud services.
that half of the internet is down when some of that many people are not talking about.
Yeah, for sure.
So maybe we can wrap this up real quick.
But I do want to add just one thing.
A roller out there in the audience says, I prefer my $5 DigitalOcean VM and just my Nginx,
MongoDB, Postgres, Python.
So how often do you think about, you know, especially, I guess, you, Emily, how often do you think about
price?
You know, if you look at the price for AWS, it can add up.
And for some companies, that doesn't matter.
Right.
Just having infrastructure is great.
But for others, you know, maybe they don't have revenue yet or they're not profitable and
they're really trying to squeeze by.
Like, where do you land on like, let's save you some money.
We could do this for $10, but we're going to do it for $500 because it's the right architecture
to run this distributed thing.
You know, what are your thoughts?
What do you tell your customers?
Sure.
The cost benefit often comes down to picking an ecosystem that's going to be super stable.
for them where we can say, hey, yeah, this is going to cost you a few hundred dollars
a month.
But if you need somebody to step in and start to manage your system for you, you know, a
couple of developer hours will easily outshine their cost for their cloud hosting.
So that kind of puts it in perspective.
A lot of these clients are already people who have spent multiple thousands, you know,
tens to hundreds of thousands of dollars actually building their software in the first place.
So as long as you're, you know, on a much lower order of magnitude, you're typically okay.
Yeah, that's true.
If you, especially if you're not technical and you've got to hire someone in to kind of
fix it, then yeah.
Then any sort of failure is a big problem.
This portion of Talk Python to Me is brought to you by SignalWire.
Let's kick this off with a question.
Do you need to add multi-party video calls to your website or app?
I'm talking about live video conference rooms that host 500 active participants, run in the
browser and work within your existing stack and even support 1080p without devouring the
bandwidth and CPU on your user's devices.
SignalWire offers the APIs, the SDKs and edge networks around the world for building the
realest of real-time voice and video communication apps with less than 50 milliseconds of latency.
Their core products use WebSockets to deliver 300% lower latency than APIs built on REST,
making them ideal for apps where every millisecond of responsiveness makes a difference.
Now, you may wonder how they get 500 active participants in a browser-based app.
Most current approaches use a limited but more economical approach called SFU or selective
forwarding units, which leaves the work of mixing and decoding all those video and audio
streams of every participant to each user's device.
Browser-based apps built on SFU struggle to support more than 20 interactive participants.
So SignalWire mixes all the video and audio feeds on the server and distributes a single unified
stream back to every participant.
So you can build things like live streaming fitness studios where instructors demonstrate
every move from multiple angles.
Or even live shopping apps that highlight the charisma of the presenter and the charisma of
the products they're pitching at the same time.
SignalWire comes from the team behind FreeSwitch, the open-source telecom infrastructure toolkit
used by Amazon, Zoom, and tens of thousands of more to build mass-scale telecom products.
So sign up for your free account at talkpython.fm/signalwire.
And be sure to mention Talk Python to me to receive an extra 5,000 video minutes.
That's talkpython.fm/signalwire.
And mention Talk Python to me for all those credits.
One thing that I'd love to add on to that is, well, two things.
First of all, on the notion of price and sort of related to which technologies you should use,
I think the question you constantly need to be asking is really never ask about one side of
the equation.
Never ask about features or price.
You always want to be looking at a price-performance ratio.
And that performance shouldn't necessarily, in fact, usually should not be like metrics like gigabytes
or throughput or anything like that.
You should be looking at, do you need what the product offers?
And what is it going to save you time on?
So like when Emily says, if it's going to save you development time, that is gold.
You always want to go err on the side of saving development time.
Developers are very expensive.
We're very finicky.
And anything that you develop, you also need to maintain.
So one of the big benefits of cloud services is think of that price, not just in terms
of like development cost, but are you going to need to maintain it?
And on the flip side, do you need this thing at all?
Quite like Hinnick was talking about having, you know, just not using cloud at all.
And yet he successfully develops and deploys many services in production and they all seem
fine.
And so I think that there's often kind of this tool obsession where we look at features,
features, features, and assume that features equals benefits.
But features are only benefits to you if you need them.
If you don't need them, they're additional costs.
You have to learn stuff about every single one of those features.
The time your developers spend learning the surface of the AWS API is a cost you have to
think about.
And so like, it's quite often just better to not use the cloud because it's cheaper to
not figure it all out.
If you know how expensive it's going to be in your own infrastructure.
Or as Emily pointed out, if it's in a Docker container, you may run it in the cloud now,
but it's fairly portable and not super tied to it.
Paul Everett out in the audience has a quick question.
He says, we talk about pinning dependencies to control change.
But when we talk about cloud computing, we say let things like AWS handle it.
You know, you're sort of, that's kind of a dependency on their compatibility of each one
of those services you take on.
What do you think about, you know, stability of those APIs and those services over time?
I think Azure maybe is a little more changing than AWS.
Things come and go, but still it's something to consider, right?
It's another thing you've got to deal with churn on, I guess.
Yeah.
And I think it depends on like how quickly you're jumping on new services.
So we actually started using ECS right after it was released and it's come a long way since
then.
And both like the product and our perficiency has come a long way.
So I think that taking any sort of like new service with a heavy dose of skepticism and
making sure that it's something that is really stable, that's going to fulfill the need.
It's an important thing to look at.
Yeah.
That's a really good point.
All right.
Let's move on.
We touched on this enough.
I think let's talk about microservices.
I was joking with Hennick about that at the beginning.
So we have this spectrum.
We can have just one code base, one process that runs in micro WSGI or G Unicorn or whatever,
where we could have a little Lask API that does user management and login, another part
that does, you know, the catalog or whatever.
We could have a bunch of these little services, these microservices, and then put them all
together.
I have thoughts on this, but where are your thoughts?
My opinion at this point is, I think, quite public.
I think you should start with a monolith first for this simple reason that microservices come
with a lot of adjacent complexity.
We basically just talked about with cloud services, right?
Like you cannot have microservices without service discovery.
You cannot have microservices without tracing because every error becomes like a, I thought
it's once called a distributed murder mystery.
When you're trying to find the fault of it, there are things that people don't think about, like retries need to be kept.
Otherwise, you're going to get exponential growth and you denial service yourself and stuff like that.
So, I mean, it's a general good idea to have as few moving parts as possible, which already someone said about cloud computing.
And because more moving parts are always harder to make reliable, right?
And Martin Foller calls it the microservice premium, which I like.
And it's again, it's a trade-off.
Is this premium worth it to you?
And I think you need a lot more experience to make this trade-off than many people think.
That's my experience.
Yeah.
You probably start simple and then with just two or three pieces and then you end up with 20 and you're like, well, how do we get here?
Yeah.
And they don't really know why, right?
Like if people want boundaries, like I cannot speak for huge teams because one of the peculiarities of my job is that our team is very small with a lot of responsibility.
But there are big teams that have boundaries that do not come with a network partition.
So, yeah, it's a trade-off.
And I think that most people need more experience to actually make this trade-off.
Yeah, I would, I very much agree with that.
particularly the people who tend to be asking this question are often asking it because they're at a small company.
They have a small team.
They saw a really cool white paper or a presentation by somebody at Google or Netflix.
And they're like, wow, all this stuff about microservices sounds great.
Fault isolation and distributed tracing and, you know, use whatever language you want.
And I'll get back to that one in a second.
And again, like I said before, you will need to think about everything in terms of cost benefit and not just benefit.
And the folks at Google and Netflix are talking about 10,000 person teams and like, how do you manage complexity at that scale?
How do you deal with the problems that come with that type of organization?
And so asking like, I also, even the shape of this debate has long bothered me because the terms that we're using, right?
Do we want monoliths or microservices?
Asking that question is like saying, well, when we have a wheel, do we want it to be like an 18-wheeler, like truck wheel?
Or do we want it to be like one of those wheels that comes on like a micromachines car?
And the answer is, I don't know.
Like, what are you putting on the vehicle that this wheel is going to be attached to?
Like, it really depends.
The question of like, how big do you want your service to be?
Which is, I think, a better question than do you want a microservice or a monolith?
It comes down to what is the surface area to volume ratio of your team?
It takes about 12 people to run a microservice, I would say.
Like a dozen people per service is roughly what you should be thinking about.
Plus, there's the fixed overhead of a microservice architecture where you need service discovery and tracing and logging and like a whole bunch of other stuff,
which you probably need in today's modern fancy cloud environment if you're going to be able to leverage services like Honeycomb and CloudWatch and like just manage all your logs and not have to deal with that yourself.
Then you're talking about maybe a six-person team that can just do infra.
And if you are at a three-person startup and thinking, huh, that sounds like a lot of people, you want a monolith.
You want one piece of code you can maintain because the benefit of having something like a very fine-grained microservice architecture is that you can say, we're going to have each team own its little piece.
And we're going to move a lot of the complexity of orchestrating from code to configuration because our operations team can like deal with the configuration and manage their process around testing configurations.
But if you're a small development team, you want code you can write unit tests for.
You want code you can run and understand the way that you run and understand everything else.
And if you push everything into YAML files and INI files and cloud APIs, what does your development environment look like anymore?
How do you test those changes?
Right. Even developing on your own machine becomes tricky.
Yeah, exactly. Yeah.
Yeah. I think the big takeaway is that the team needs to match what microservices are optimized for.
Yeah, exactly.
That's a really good point.
Emily, I suspect the people you suggest use Heroku probably are not receiving 10 microservices over there, are they?
No, definitely not.
I think one of the things that we typically have built in is a certain amount of ability to scale independently.
And so we typically do have two different workers on Heroku.
So we're going to have, you know, a Django application and a Celery task worker.
And then we know that we can kind of scale those two independently.
But that's where we really see the division of load.
So that's my lens that I look at monolith versus microservices with is really, do I need to scale them independently?
Yeah. And I certainly think having some kind of, we're going to kick off the long running work over there.
So it's not happening as part of a web request.
That certainly makes sense.
Even things just like sending a group of people an email often takes too long if the group is large enough before the request will time out and wreck your servers and all sorts of stuff.
So yeah, it makes sense.
But I think developer experience is a really big one.
We have had the pleasure of working with another like very large tech company, like multiple hundreds of employees, and they had a monolithic Rails app.
And it was absolute hell to work with as a developer because you're constantly like having hundreds of PRs open at a time.
You open a PR and within 20 minutes, you have a merge conflict with somebody else.
And you really, at that point, it makes sense from a developer perspective too, to be able to divide it up and say, okay, like, this is your area of expertise.
This is your area of expertise.
And at least set up those partitions.
But you can do that in a monolith as well.
It just takes a little bit of like awareness and slice in the problem a little differently.
Yeah.
So let me think of a, let me throw out an idea that I just thought of that I'm not sure I'm advocating this or even that it is a good idea.
But one of the problems you have is sort of you have this big monoliths, like you were touching on, Emily, is a lot of people are changing the same bits of code and they're kind of in all over the place.
So the microservices is one way to solve that problem.
What about trying to think about, you know, could we package up some of the functionality of what our application does or our area into Python packages that we can then install and use in the main app?
That idea?
Good idea?
I mean, it's interesting, but it's also another level of complexity, right?
Yeah.
You got to run your own private server or something.
I would actually say that that is a prerequisite to microservices.
Specifically, one of the things that people often forget about the whole service architecture thing, and this comes down to the future.
Remember, I said I was going to talk about choose your own language kind of thinking before.
One of the things that people often choose microservice architecture for badly is this idea that they want to experiment with different programming languages because they want to use the right tool for the right job.
And number one, just the cognitive overhead of jumping between Haskell and OCaml and Python and Rust on your back end is way, way higher than most people think.
But even forgetting about the sort of human cognitive overhead, your service architecture, your service fabric needs to be logging and recording metrics and dealing with load balancers and dealing with data stores in a consistent way.
And that task by itself is complex enough that you probably need a library, which means every supported language in your environment needs to have packages installed that are maintained by your infrastructure team and not by your application teams.
And that means before you can even think about microservices, you have to be able to split your workflow into multiple different package repositories, different source control, different teams, different CI.
Like you need to be able to do that first before you can reasonably split things across like multiple actual services.
That doesn't necessarily mean you need to like not have a quote unquote monolith because you can put a monorepo into that kind of multiple package, multiple library workflow.
And that's also fine, but you do need to be able to have multiple work streams going that end up in the same service package.
This portion of Talk Python To Me is brought to you by Tonic.ai.
Creating quality test data for developers is a complex, never-ending chore that eats in the valuable engineering resources.
Random data doesn't do it.
And production data is not safe or legal for developers to use.
What if you could mimic your entire production database to create a realistic dataset with zero sensitive data?
Tonic.ai does exactly that.
With Tonic, you can generate fake data that looks, acts, and behaves like production data because it's made from production data.
Using their universal data connectors and a flexible API, Tonic integrates seamlessly into your existing pipelines
and allows you to shape and size your data to scale, realism, and degree of privacy that you need.
Their platform offers advanced subsetting, secure de-identification, and ML-driven data synthesis
to create targeted test data for all your pre-production environments.
Your newly mimicked datasets are safe to share with developers, QA, data scientists, and, heck, even distributed teams around the world.
Shorten development cycles, eliminate the need for cumbersome data pipeline work,
and mathematically guarantee the privacy of your data with Tonic.ai.
Check out their service right now at talkpython.fm/tonic or just click the link in your podcast player's show notes.
Be sure to use our link, talkpython.fm/tonic, so they know you heard about them from us.
Another thing is versioning across the services, right?
The definition, like if you're using SQLAlchemy, the user shape has to match all the parts that talk to the database that might touch a user object or something like that.
So I guess I'll wrap it up with a quick thought that for me, microservices feel like I'm trading code and developer complexity for operational and DevOps complexity.
And I personally feel much more comfortable managing and working with code complexity than like infrastructure complexity.
But if your team and your organization is all about managing infrastructure complexity and DevOps, and you have a bunch of junior devs, maybe microservices make sense.
I don't know.
It kind of depends.
But my vote's for the monolith side because I'd rather manage the software complexity.
Let's talk about security.
I mean, we've had some crazy stuff.
I think it's log4jmeme.
There's a .com.
I think that is a .com.
Yeah.
So like security is clearly on the, it's something we always have to think about, but something, right, that recently log4j obviously being a Java, not Python thing.
But it's the one thing that makes me nervous about running stuff in production.
Honestly, we've got stuff like status cake or various other things that you can fire up or uptime or whatever that will tell you if your site is down and send you notifications and things like that.
But the security one is a little bit scary.
So how do you approach thinking about this?
You know, Henrik, maybe you go first hosting other people's code is like a next level is like meta security.
Yeah.
The one flag I want to wave here is defense in depth, which is something that's very dear to my heart and which I feel is, yeah, which could be a bit more popular.
Sure.
Because every significant attack nowadays is a multi-stage one.
It doesn't matter if it's like owning your Chrome or owning a server.
It's usually multiple stages.
So you shouldn't make it as hard as possible to the attackers, even though they have entered your infrastructure at this point.
So for me, it means that I treat our own network, which is very private.
You need a VPN to get in and everything as if it has intruders inside it.
And that's our standard.
So you should hash your passwords as the maintainer of Argon2CFFI, of course, using Argon2, but use whatever.
We use TLS even in private networks, because I know that cloud providers have virtual LANs, which are also often encrypted.
But still, it's just another layer.
You cannot have enough layers to protect yourself at this point, because if someone intrudes you, you don't want them to sniff passwords out of your traffic and things like that.
The list goes on.
Yeah, it definitely does.
Emily, security thoughts?
Yeah, luckily, this isn't one that we've necessarily had to worry too much about.
I think the worst thing that's happened for us is DDoS attack, and that's about it.
So for me, I think definitely staying on top of dependency management, keeping things up to date, which is not necessarily always the easiest thing.
Especially the Python 2.3 transition, upgrading Django is sometimes a little bit more complex than you'd want it to be.
But I think that getting those security updates and making sure that you're on an LTS version is really important.
LTS being long-term support.
Yeah, absolutely.
Yeah.
So one of the things that I think, you know, I made the, I was picking on the log4j thing.
But one of the problems that made this a little bit harder, I mean, this is going to be something we live with for a long time.
It's going to be a nightmare.
But the consequences of it.
But one of the problems was that the fixed version of log4j, which for those who didn't know, the log4j problem is if you can basically get a server to print a message with any of your input, like this URL was invalid, or this email tried to log in and failed, you can own the computer.
Which is really, really bad.
So it has to be fixed, like straight away.
But the problem was, the fixed version was on a newer version, like Java 8.
So if you were running Java 7, you had to both not just upgrade your library, but then upgrade your whole runtime, which might be problematic.
So Emily, that makes me think that, you know, you probably, one of the good rules to go by is don't let your frameworks get too far out of date.
Like you don't have to be on the latest Django, but don't stay on Django 1 when Django 4 is about to be released.
Or don't stay on Python 3.6 when you could be on the newer one or something like that.
How do you all feel about that?
Yeah, and I think that, like looking at it from an open source maintainer's perspectives, making sure that you have the ability to kind of hotfix previous versions and be very clear about which versions you're supporting and which ones you're not.
That way, you make it easy for a user to know, like, yeah, I'm not going to get the security update and I need to upgrade and have that done ahead of time to stay on top of things.
Yeah.
And AgraGliff, old frameworks, what do you think?
I think that this is a, I don't think I can find this right away, but I remember one of my more popular tweets was one of those sort of two buttons memes where the guy can't choose.
And on one side, you've got get owned because your dependencies are out of date and you have no way to immediately update them or get owned because you're automatically updating from an upstream that you don't know if you can trust or not.
And that's kind of, so the way that I sort of split that difference in practice is you really want to make sure that it's not just about like regularly upgrading because you can always say like, oh yeah, we'll regularly upgrade.
But you've only got a fixed budget for security, right?
Like it's possible to spend all of your time spinning your wheels trying to increase the cost for attackers across every possible access.
I'm sorry, access.
Yeah.
But eventually you got to ship features and deliver stuff.
Like that's really.
Exactly.
It's a balance.
Yeah.
Thinking about Hinnick's suggestion of defense in depth, the way that I like to think about that is you want to raise the bar to a certain minimal level in a bunch of different areas.
And then not get too obsessed with getting that last 5% of security on each possible access.
So for dependencies, the way that I think about that is have the, you know, very widely available automation that already does this stuff like depend a bot.
Make sure you are getting those PRs automatically pin everything so that you're never dealing with a library upgrade in the middle of future development.
Your library upgrade work should always be, I am upgrading this library now.
And 90% of the time, if you have that set up where you've got a build that is running on every PR that builds your whole dependency structure, that's got everything pinned and hashes pinned and everything.
And then is also regularly receiving these PR updates.
Most attackers who are doing things with supply chain attacks aren't all that clever.
And so they will just end up trying to pop your CI and you'll see that you'll get some kind of error.
You'll probably notice.
Not necessarily attackers can be very sophisticated, but like you want to have everything, every library running in your nicely isolated CI environment on your GitHub actions or whatever first.
And again, you want that those changes to all be like same code with the old dependencies, same code with new dependency.
And every so often you'll get one very expensive upgrade that you really got to do and you got to make time for.
If you're not upgrading all the super easy, almost free, like just hit the green button on the PR that's working.
If you're not doing that all the time, then when you do get to those big upgrades, you will be upgrading 50 dependencies at a time.
Yeah, it's going to be rough.
Yeah.
You really need to think about the cost to you as the defender.
And the way that you reduce that cost on dependency upgrades is spending just a little bit of time every week, tending that automation and looking for those upgrades that are really going to take some development work.
And there are fewer than you might think, like quite often feels like a really big task because most people get stuck in this, oh no, it's time to do the dependency upgrades this quarter.
And you have like a seven week project that you're like trying to figure out which dependency is the one that's making all your tests fail.
And you're like changing 50 lines in your requirements.txt all at once.
If you don't do that, and by the time you're upgrading the one library that really does have a big breaking API change, it's not actually that hard as long as everything else is already up to date.
Yeah.
And usually there's one or two libraries that are massive, like Gingo or Flask.
And then it's all the little dependencies that probably require no effort on your part.
One other thing that I switched to, you talked about Dependabot, which is really great.
And I have Dependabot turned on for my stuff, but I started using pip compile where it'll go and basically you give it an in file and it'll build your requirements.txt.
And it'll even say like, this thing is here because it depends.
You know, you installed Flask.
That's why it's dangerous is here, for example.
And I really like having that because it's just, you know, this week I'll go and I'll run it and it'll tell me what the new requirements are.
I'm actively knowing that's happening.
I'll sort of process it and go with it.
What do you all think?
Emily, are you using Dependabot or something else?
Yeah, we do use Dependabot.
I think another interesting like integral piece of being able to upgrade your dependencies is having tests that give you the confidence that you can.
So do you feel confident to say, yes, our upgraded dependencies passed our test suite, therefore we can upgrade?
Or is there something that is going to require somebody else to go in and look at it?
But yeah, I think we've come a really long way in terms of dependency management.
I do not miss the giant requirements.txt files that had all the dependencies of dependencies of dependencies all the way down and you didn't know what you would install versus another library.
Yeah, exactly.
I also like things like pip compile because it lets you have a little bit more control over the child dependencies versus relying on a top level library like Django to specify their own requirements in a way that works for your organization.
Yeah, absolutely.
So I also use Dependabot, like I said, and if you go and run pip-tools and then you commit that back to the repo, Dependabot will notice that that's been upgraded and it will automatically close the PR.
So there's kind of a nice match between them as well.
Yeah, the answers to all these questions are really are so much simpler than they used to be, you know, five, six years ago.
I've been using requires.io for probably almost a decade and Dependabot is much better, largely because it defaults just sending you those individual upgrades and you can really tune how much stuff it's going to try to do at once.
So yeah, mostly the answer on this is like, make something that builds your Docker container or whatever your fully realized application artifact is, run it all the time and have part of that process be like, freeze your dependencies, pin everything.
You need to make sure that everything is pinned so that you get very reliable, repeatable builds.
But having done that, you can really just bask in the, you know, cornucopia of tools that we have available to do this now that make it all pretty easy.
Yeah, it's definitely getting easier and easier.
Let's talk about performance a little bit.
I guess maybe I'll show just really quickly, I'll throw out there that at sneak.io, S-N-Y-K.io, they've got some stuff where you can put in like a package or something like that.
Let's see.
I don't remember where you go to do it, but you can basically put in like a Python package.
It'll give you a security report for it.
So that's, I don't know how accurate that turns out to be for everything, but it's something.
But let's talk about performance.
And one thing I'd like to touch on is.
Actually, can we just say one more thing about security before we move on?
Yeah, go for it.
Because Hennick mentioned encryption and not trusting your local network at the beginning.
And one other thing I wanted to mention, speaking of tools that are much better than they used to be.
One popular idiom is to like have a production mode that has all your encryption on and a development mode where you like just turn it all off for convenience.
And I'm a big fan of setting up like an entry in your hosts file for each developer and having some infrastructure for provisioning certificates for individual developer machines.
So that encryption is just always on.
Nobody ever makes a connection without TLS on it.
Even your like local API stubs still have some kind of TLS on them.
Because it's actually not all that hard, particularly with Let's Encrypt.
You get a couple of DNS plugins and you can easily vend those certificates to your dev team.
It takes a couple of days of work at most.
And having done that, not only are you more secure because you just don't even have that switch anymore to like accidentally be sending everything in plain text.
But also you spot a surprising number of configuration issues.
And like you get to see how your like for real certificates work while you're doing development.
And that can really help a lot of developers like understand what's going on with the somewhat tricky world of HTTPS.
So it's just running a little bit closer to production.
Yeah.
All right.
Let's talk about performance.
And I'll just put this up on the screen because I think it's a fun thing.
Have you all seen Locus.io?
Yeah, we use it regularly for load testing.
Do you?
Yeah.
Maybe tell people about it real quick.
I've used it once or twice and it's fantastic.
Yeah, definitely.
So it's a tool that allows you to essentially write a script that will emulate requests to your server.
And then you can give it a variety of, wow, mom brain with no sleep makes me forget words.
But basically just like different parameters that you can specify.
So it makes it really easy to say like, here is the approximate randomized behavior for a single user.
And then scale it up to hundreds or thousands of users and see how your server handles it.
Right.
You give it a Python script, which is really interesting, right?
You create a class and you say, here's a typical user of this type.
And it does different things like it'll call the index page.
It'll call the about page.
It'll go do a search.
And you can say things like, there's going to be a certain amount of delay between pages.
And then you can, as you said, scale that up.
Say like, I want 2,700 regular users and 50 admin users and let them go crawl around on the site and do what they do.
Right.
Yeah.
So it's super cool.
And the, like the real, a real time dashboard is neat.
So if you want to know about your performance, you could use this, but you know, how do you all think about performance for the apps, either you're running or delivering, you know, what's fast enough?
What's just wasting your time, getting that last millisecond of response time?
So I'm going to make this the bold statement that for the vast majority of developers, Python is fast enough.
Oh yeah, absolutely.
It is fast enough to saturate a database.
And once your database is saturated, you have different problems.
Like there's definitely things for what it is, where it is a problem.
Like you cannot saturate an LDAP server, for example.
I know that because I tried and it's one of our Go services.
It would be nice if Python could be faster.
And I'm very excited about Guido's performance task force at Microsoft.
It's great.
Yeah.
I feel like the indifference that the no-jill movement has been shown kind of shows that nobody really thinks that intensively about it anymore.
Like for most people, it's fine.
It's not good.
It's not great.
It's fine.
And I mean, Instagram is printing money with it.
So they absolutely are.
Yeah.
You know, you can go and put your site into PageSpeed Insights and see what you get.
And I think once you get down to several millisecond response time per page, it just, you know, there's not a whole lot you can do to make it faster.
There's not a lot of benefit to doing that work, right?
It's mostly tuning a database with the right indexes and caching.
That's the two things.
Yeah.
And everything else is like the last 10% or something.
Yeah, absolutely.
So what I wanted to definitely emphasize here and hear your thoughts.
I feel like there's so many times I go to some page and it doesn't necessarily, some site doesn't necessarily have to be Python.
It's just some database backed page.
And you go there and it's sitting on what seems like one of their primary pages.
And it's like five seconds until it responds or several seconds even.
It's like, what is this thing doing?
And I just, I know they don't have indexes in their database.
They just, they can't.
So, you know, I just want to put out a plea for please use like some database profiling feature.
Please look at your queries and please put an index.
It's so incredibly easy.
Like any of you all want to rant on that with me?
I need to bring up one tool before we talk about this.
And it's called PG mustard.
Okay.
I've never been able to fully understand an explained statement.
Like I learned it many times and then I forgot it again.
And PG mustard is amazing.
You just take an explain.
You copy paste it into a web app and it tells you exactly what's going wrong.
Oh, nice.
And I've shaved, like I had like one query, which is like very big.
It's from a financial system.
And I've, I think like 66% of the query at runtime, I was able to shave off.
Yeah.
Just because there was an index, but it was set up wrong.
It was amazing.
Yeah.
So here's a really beautiful visual thing.
And it's also a bit of like a profiler.
So it says on this part of the query statement, you spent 0.4% of the time.
And this part you spent 58% of the time, right?
Here's an index scan.
That's the best part.
It doesn't just tell you what's going wrong, but do this, make this index.
Things like that.
Okay.
I've never seen this before.
This is fantastic.
Emily, thoughts on indexes?
Join my plea.
Yeah.
So I think that that piece of it's definitely very important.
Like making sure that you're doing those sort of basic things to get you to that, you
know, 80 to 90% performance potential.
But I would also argue that these days in a vast majority of applications, you're going
to have a decoupled front end from your back end.
And I would argue that making sure that you're implementing best practices for user experience
on your front end is going to give you so much more payoff than trying to optimize that
API call by, you know, a few milliseconds.
So previously I had pulled up the PageSpeed Insights before, right?
And at PageSpeed.web.dev from Google, I believe.
Yeah.
And what's really interesting about that is if you go and put your site into there,
it doesn't feel fantastic to do it, by the way.
So if you go and you put your site in here, you at some point might get a good number.
I'm getting 100 out of 100 on the TalkByThan training site right now.
But that's because I spent three days addressing every little thing.
Like this image is not sized right.
This JavaScript is not bundled with that JavaScript.
This element is being resized by the browser.
You should make it the same size by default.
And just all of these little things.
And it wasn't even about the server response time, which was always pretty low.
It's about all the other stuff.
It's like, how does it feel to the user to get to the page rather than what is the, you
know, HTML out of the server response time?
And that's what you're talking about, right?
That kind of stuff?
Yep.
Yeah.
Yeah, definitely.
Awesome.
All right.
So I have so many feelings about this.
This is probably the number two topic for me behind packaging.
Okay.
Those of you not listening on the live stream, if this next part sounds choppy, it's because
I talked for an hour and a half and Michael had to edit it down.
We just cut him off.
We just had to cut him off when he lost his voice.
But seriously, the thing that I think is most important is the, quite often the parts of
your application that end up being the performance issues are once you're past this step.
Now there are lots of like, you all have been mostly talking about websites, except for Hennek
mentioning LDAP, which is an interesting.
I've never really thought about scaling that one.
No.
I have also scaled LDAP.
So the two major applications that I've dealt with performance issues on are an internal
calendaring service that I maintained at a large company that you can figure out which
one it is by reading my Wikipedia page and pilot.com's internal bookkeeping automations.
And in both of those, so number one, the calendar service was an API with no front end.
The front end was maintained by the client teams that were not even doing web stuff.
And the way that we had to performance test that involved standing up our own custom load
generation tool and running it as kind of a qualification process for our deployments.
And the reason that I bring up that one is it's interesting because we had to figure out
what our actual interesting load patterns were.
We couldn't use any of these like standard HTTP load generation things because we needed very
specific data.
And that often ends up being the problem that you're facing.
We, on that service in particular, we had performance problems that arose because we
added too many indexes.
So we were having problems on the right side of the equation.
Every time you add an index, you're optimizing read at the expense of write.
And usually it's like a lot of read performance for a little write performance, but eventually
it does add up.
It does.
Yeah.
Kivo out in the audience says, yeah, you've got to be careful adding too many indexes as
well.
There's two parts to that, right?
One part is when you write something, the indexes have to be computed for that thing that's going
in.
And the other is more indexes mean more stuff in memory.
And so like another really important aspect is do your, does the totality of your indexes
reside in memory or does it have to get paged out?
Right.
And so you would, you would hit that problem.
Both of those problems you would run into by having too many indexes.
Right.
And you really need to, so you need to be ready to measure things because you don't necessarily
know.
Like, I mean, this is that, you know, chestnut about not really, you know, premature optimization,
not knowing what the hotspots are until you run them.
But that also means, so there's two things about that.
One, the tools for doing this are not great.
Like the one that I really wish were good and just isn't is SpeedCenter, which Twisted and
PyPy used to monitor their performance over time.
Like what the performance of each revision of the code is.
And there's nothing like, you know, GitHub Actions or Travis CI, there's no sort of leader
in that field that will just tell you like, hey, your performance regressed by 10% on this
commit.
And that is the tool which I desperately want to exist.
And so most of the things that I do are trying to approximate that.
Part of that is making sure you have metrics in production that are telling you so that you
notice when things are slow.
You don't want to be having users telling you, or even really like if you're getting the bad
performance metrics out of your load testing tool, and that's a surprise to you, that means
you're probably not instrumenting enough in prod to know like, oh, users are seeing some
slowness here.
Because you're also going to get things where like your database is doing great.
Everything seems like it's super fast, but your queries are actually really slow.
They're just all running in memory.
And then you hit the cliff where suddenly you're hitting the disk.
And now everything's much, much slower.
And none of your code changed.
And your data only grew by like 10%.
And being able to spot stuff like that means you have to be looking at perf in prod.
You can't do synthetic tests for everything.
And particularly if you have a large site with a lot of users, it's very easy to miss if your
95th percentile is falling off a cliff, right?
Like you have to be looking at your core tiles and like all of these different things, not just
like average performance.
And the second part of that is the custom data generation for your synthetic tests.
So for example, on the calendar service had those issues that I just mentioned.
And pilot service had this issue where most of the performance stuff was not the database.
It was talking to APIs to pull in financial transactions and analyze them.
And it was those APIs being slow, us being silly and not talking to those APIs in parallel,
data volumes just being huge, like thousands and thousands and thousands of transactions
in a single call.
And that you have to know that that's going to happen.
And you have to be able to on demand add to your test suite or your performance test suite
new types of data.
And that performance tool with where you like write Python code looks like a great way to
do that.
I actually had never used that one.
This thing is glorious.
Yeah.
What was it?
Locust.io?
Yeah.
And it has the dynamic sort of graphs and dashboards show you sort of as it ramps up
and as you change it.
And it's, I think that might give you like the right, because you basically structure with
Python how it hammers on the server, which is pretty neat.
Yeah.
And the one thing I'd say about that was you need to do that.
You will need to write custom stuff.
Don't just assume you can like add a couple indexes.
Like you should just add a couple indexes at first if performance is not your primary concern.
But having done that, you have to know you're going to need to think about perf and like
write code to monitor it.
Absolutely.
All right.
I think we're just about out of time, although we have barely scratched the surface of all
the stuff we could talk about.
Let's close out with this question.
Do any of you have, sounds like maybe you've thought about this.
You have CI performance checks or failures or anything like that, right?
Like we have run our tests.
The build doesn't pass if the tests don't pass.
But do you have something like that for performance?
That's the thing I want to exist.
And I've never managed.
I've done things that approximate it within tests, but never gotten it really.
I have nothing like that either.
Emily?
No, I mean, I think the closest approximation that we get is we focus a lot on like Cypress
tests for front end.
So actually the user going through and working with the application.
And I think the closest thing that we could get at this point is just setting our max time
out in our like HTTP service, bumping that down and saying like, if anything's taking over
10 seconds to respond when we run the tests, then it should fail.
But no, I don't think we don't do any like regular performance testing.
Yeah.
Hynek?
We do not know.
Yeah.
But we all kind of are like, that was kind of nice to have, I think.
But yeah.
There's a lot to set it up though, right?
You've got to have enough data in the database for it to be meaningful.
And that's tricky to do in CI.
And it would be cool though.
I think Glowth makes a really good point that like the load testing and the manual testing
of performance is great, especially when it's a prerequisite to launch.
But there's no way that you're going to be able to replicate anything in production.
And the best thing that you can do is monitor prod as closely as you can.
Yeah, absolutely.
Some of that real-time monitoring.
Fantastic.
All right.
Well, thank you all for being here.
This has been super interesting to chat about.
Before you got out of here, let me, I think I'll just put it down to one of the final
two questions so that we don't take too long.
But you're going to write some Python code, what editor to use?
Let's go.
Blockwise, Hynek, how about you?
What are you writing code with these days?
Well, I have a long story of editors.
I've used almost all of them at some point.
I usually stopped using because I got crippled, like Emix Pinky.
Nowadays, I usually use either Vim in a console or VS Code.
Okay.
Right on.
Cliff?
There's an implicit thing in this question where it sounds like you're recommending the
thing that you use.
So I want to be clear that I'm not doing that.
Don't do as I do.
Yeah.
I use Emix, but with about 10 megabytes of custom Elisp, which I'm never going to share
with anyone.
Fantastic.
Because you shouldn't use it.
Just use VS Code.
Awesome.
Emily?
VS Code.
All day.
Okay.
Big thanks to Brett Cannon, who sat me down at a Python and forced me to use it because
the first time I used it, I hated it and I went back to Sublime.
But yeah, I don't think there's anything that competes these days.
Yeah.
He's such a good ambassador for that.
So it's good to have him working on it.
All right.
Thank you all for being on the podcast.
It's been great to chat about the stuff and really insightful to get your experience.
Thanks so much for having us.
Awesome.
Thanks for having us.
Yeah.
You bet.
Bye.
This has been another episode of Talk Python to Me.
Thank you to our sponsors.
Be sure to check out what they're offering.
It really helps support the show.
Add high-performance, multi-party video calls to any app or website with SignalWire.
Visit talkpython.fm/SignalWire and mention that you came from Talk Python to Me to get started
and grab those free credits.
Tonic.ai creates quality test data that does not contain personally identifiable information.
Your generated data sets are safe to share with developers, UA, and data scientists.
Most importantly, they behave like production because they're made from production data.
Check them out at talkpython.fm/tonic.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at Talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show.
Open your favorite podcast app and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play,
and the direct RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air,
be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
Bye.
you
Thank you.