Does it seem like your Python projects are getting bigger and bigger?
Are you feeling the pain as your code base expands and gets tougher to debug and maintain?
Patrick Viafor is here to help us write more maintainable, longer-lived, and more enjoyable
Python code.
This is Talk Python to Me, episode 332, recorded August 30th, 2021.
Welcome to Talk Python to Me, a weekly podcast on Python.
This is your host, Michael Kennedy.
Follow me on Twitter, where I'm @mkennedy, and keep up with the show and listen to past
episodes at talkpython.fm.
And follow the show on Twitter via at Talk Python.
We've started streaming most of our episodes live on YouTube.
Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming
shows and be part of that episode.
This episode is brought to you by Clubhouse, soon to be known as Shortcut, by masterworks.io,
and the transcripts are brought to you by assembly.ai.
Please check out what they're offering during their segments.
It really helps support the show.
Pat, welcome to Talk Python to Me.
Thank you.
I'm so honored to be here.
Long-time listener.
Oh, that's fantastic.
I really, really appreciate that.
It's an honor to have you here.
We're talking about one of these subjects that I really enjoy.
I feel like it's one of these more evergreen type of topics.
You know, it's super fun to talk about the new features in Flask 2.0, but that's only relevant
for so long and for so many people.
But writing software that's reliable, that can be changed over time, doing so in a Pythonic
way, that's good stuff to learn no matter where you are in your career.
Oh, I absolutely agree.
You've written a book called Robust Python, which caught my interest.
I, as you can imagine, get a couple of books and ideas said to me periodically per day.
And so they don't usually appeal to me.
But this one really does for the reasons that I exactly stated in the opening.
And we're just going to sort of riff on this more broad idea of what is robust Python?
What is clean code?
Like, how do you do this in Python and make it maintainable for both you in the future
and other people and so on?
Yeah, sounds good.
Before we get to that, though, let's start with you.
What's your story?
So I got into programming like many males my age.
I loved video games as a kid.
And I was 12 or 13.
And some ad in one of my video game magazines caught my eye for video game creator studio.
Oh, nice.
And it was this C++, just C++ engine, but it provided easy ways to do sprites and animations
and particle effects.
And they gave you step-by-step instructions.
Here's how you build Pong.
Here's how you build Frogger.
And here are the assets.
I started riffing on it.
I really came to love it.
I then took a programming class in high school, True Basic, which I don't remember much of other
than I don't think I'd ever go back to it now.
But it interested me enough that I decided to go to computer science college.
I learned more about the video game industry, especially some of the hours they work and said,
you know what, maybe the video game industry is not for me, but I still loved programming
at its heart.
So I continued on and just found a few jobs after college and just went from there.
As far as Python went, I worked in telecom for a little bit.
And early 2010s, we had some test software.
It was all written in the Tickle programming language.
And that is an interesting programming language.
As far as types go, everything's treated as a string.
And it led to some very unmaintainable code.
So a few sharp engineers and myself, we decided, you know, what would it look like if we wrote
this in Python?
And a whole test of the automation was born from there.
And we just went, wow, Python makes everything nicer.
Like, this is so much pleasurable to work with.
And I've done like C++ and Java before then.
So Python was this breath of fresh air.
And I just fell in love with the language.
Yeah.
Those are both very syntax symbol heavy languages.
I've done a ton of C++.
I did professional C++ development for a while.
I have never done much Java.
But yeah, they're really interesting.
I remember when I came to Python, I came from C#, but also from C++.
And just in my mind, those symbols, all the parentheses on the if statements and the semicolons
at the end of the line and all that business, it was just required.
That's just what language is needed to be.
And then they weren't there.
And I'm like, I just, I'm having a hard time with this.
Right.
And I felt it was weird.
But then when I went back just to work on like a project that was still ongoing, it seemed
more weird.
I'm like, wait a minute.
My eye, you know, like the wool has been taken off my eyes.
I know now these are not required and still here I am continuing to write them.
And it just drove me crazy once I realized I don't have to put these things here.
It's just this language that makes me.
And so, yeah, it was really nice.
Yeah.
Python kind of captured that magic for me.
And it's funny because going back to C++ every once in a while, my C++ got a lot better
because of Python.
You start focusing on the simple.
That's nice.
Yeah.
Yeah, exactly.
You're not trying to flex with your double pointers and all that kind of stuff.
It's like, we don't need this.
Let's not do this.
Come on.
Very cool.
So how about now?
What are you doing?
So right now I am a software engineer at Canonical.
I'm on their public cloud team.
So we're building the Ubuntu operating system for public clouds, AWS, Azure, GCE, a couple others.
And so we customize the images for clouds, work with a lot of other canonical teams to make sure that their changes are reflected in our software.
We maintain the CICD pipelines to deliver those as often as we can.
Most of the toolings are in Python, so I get to use Python day to day, which makes me happy.
Oh, that's really cool.
What kind of features are we talking about here?
Or aspects?
Yeah, general hardware enablement.
Clouds may have new instances, new processor types.
Working to make sure that that's enabled.
Do a lot of close work with our kernel team.
Features that the cloud themselves might request.
Maybe, you know.
Like something the cloud needs to set up and maintain the VM aspect.
Yeah.
Okay.
More on the infrastructure side.
Maybe how do they hibernate their VMs?
Okay.
How do they improve boot speed?
Those sort of changes.
Yeah.
Or how do they improve the security profile?
That's something I'm working on right now with some of our clouds.
Yeah.
Super, super important.
Yeah.
That's part of the promise to go into the cloud is often there's better security and better durability.
But if it fails, it fails for potentially all of some big chunk of the internet, right?
Like that's the consequence of failure is also higher.
So.
Yeah.
You have to be very aware of how many people are using your software when you deploy to the cloud.
Does it make you nervous to work on that kind of stuff?
Oh, absolutely.
Yeah.
I wouldn't, I'd be lying if I said it didn't, but you, you learn to be careful.
You learn to really focus on making everyone, you know, having good communication between your teams, making sure there's no surprises for anyone, that sort of thing.
I remember that as well, just working on certain things.
Like the first e-commerce system I wrote, I'm like, this thing's going to be charging thousands of dollars per transaction.
And I might screw this up.
And I'm really worried for deploying it for the whole company.
There's like big purchases people were doing.
And it made me really nervous.
But at the same time, one of the things I've come to learn over my career programming is it's one of the things that will really put a sad, sad look on your face is if you spend a lot of time and create beautiful software and no one uses it.
That's right.
Yeah.
So the, even though it may be stressful that it's getting used a lot and it's really central, that's also amazing, right?
Yeah.
But it's also made me a better programmer because you just start thinking about explicit error cases more.
And my last job, I mentioned I did telecom and 911 calls get routed through your equipment.
You don't want to fail that.
And yeah.
Wow.
It is scary at first, but you learn to develop disciplines over time of, okay, how do we make sure we're not making mistakes?
You know, speaking of that, I think that's maybe a good segue in to talk about this idea of robust Python.
First of all, like, let's just talk about your book for just a minute and then we'll just get into the general idea of it.
Like what motivated you to write this book?
So I mentioned I've done C++ development in the past.
I started diving heavy into modern C++ and how it uses a lot more type safety features and some earlier versions of C++.
Did a quick dabble into Haskell just from a learning perspective and really started to love static types.
But my day-to-day language was Python.
So I really, really dove deep on, you know, how do I make my Python safer moving forward?
So I pitched to a few publishers.
I want to write a book about, you know, type system best practices in Python.
Let's talk type checkers.
Let's talk about design types and so on and so forth.
O'Reilly, who published my book, they bit and they said, oh, that sounds interesting.
Can we expand the scope a little bit more?
And I thought, okay, well, why do I write types in Python?
Why do I, you know, advocate type systems?
It's to make things more maintainable, make things live longer, to make things clearer.
And the idea of robust Python came out of that.
Well, I think it's really a pretty interesting idea.
I do think when you come from these languages like C++ or Java or something, there is something to be said for it literally is verified to be clicked together.
Just like a perfect jigsaw puzzle.
Yeah.
Right.
It won't catch every error, but it can catch more errors.
And that's a good thing.
Yeah.
Well, I remember how excited I was when I got my first non-trivial C++ thing to compile.
And I'm like, yes, I've done it.
Little did I know, like I was in for it.
I mean, it was out of compiler errors and into like real errors.
Yeah.
At that point.
Right.
Yeah.
The fun errors.
Right.
Yeah.
Exactly.
The why, what is it?
What is it?
Seg fault?
Whatever.
I don't know.
We're going to work on that.
So anyway, that was, that was really interesting.
I, I do think that it's not required, right?
I do think that there is really interesting code being written in Python.
And you can see that how it's, it's being adopted in all these ways.
The biggest, maybe example I have in mind of these two things side by side is, you know,
YouTube versus Google video and how Google video was in Python with 20 engineers.
Google video was a hundred C++ engineers.
And eventually Google just said, all right, we're going to put that project aside and buy
YouTube because they just keep out running us in features.
Yeah.
I think that's the beauty of Python is that you can build things fast.
You can see them working faster.
And you don't have to fight the compiler as much of the way.
So it's almost like these two things are at odds.
And the answer is just Python's gradual typing.
You just type as you need to.
You add typing where it makes sense.
And I won't be prescriptive and say you need types everywhere, but there are certain places
where it makes a lot of sense and save you money and time.
I do find it pretty interesting how they've implemented types in Python.
And to be clear, this idea of robust Python that you're focused on, types is just one part
of it, but it sounds like it was the genesis of it.
It was absolutely the genesis.
Yeah.
So we have sort of two other realities that we might look at.
We could look at C++ or C# or Java.
They're all the same in that regard.
They're like static compiled languages.
And then we've got something that's much closer to Python with TypeScript, where they said,
we have the same problem.
We have JavaScript, which is even less typed because the runtime types are all just weird
dictionary prototype things anyway.
And we want to make that reliable for larger scale software and integrations and stuff.
So we create TypeScript.
But the real fork in the road, I think that was interesting is the TypeScript people said,
we're going to apply the ideas of absolute strict typing.
Once you start down that path, it's all down that path.
And they have to match up.
Whereas in Python, it's like, let's help you down this path, but not give up the zen of Python,
where you can easily put things together and you're not restricted by this type system.
But if you run the right tools, be that PyCharm, VS Code, or mypy or something like that,
it'll be there to help you make it better in lots of ways.
Oh, absolutely.
I think that's a really creative and interesting aspect of it.
Yeah.
I absolutely agree with that.
A lot of things that I discuss throughout the book is the idea that these are tools in your toolbox.
And so I talk about types.
I talk about API design.
I talk about sensibility.
They're all tools and they shouldn't be applied everywhere.
I really want to start focusing on first principles.
Why do we do the things we do in software engineering?
And what are the most appropriate for them?
You had a really interesting quote in your book about software engineering.
And I've always sort of struggled to have this conversation with people like, oh, are you a programmer?
Are you a coder?
Are you an engineer?
Like, where are you on that?
Like, is that even a meaningful distinction?
And to a large degree, I've always felt like it was just sort of whatever sort of culture you're in.
You know, if you're in a startup, it's one thing.
If you're in like a giant enterprise, they may value a different title in a different way.
It's all kind of the same.
But you had this cool quote that said something like software engineering is programming integrated over time.
Yes.
And I wish I could claim credit for that, but that actually came from Titus Winters of Google and C++ Con or CPP Con Talk.
And it's just really resonated with me.
We program, but software engineering is the efforts of that programming over years or decades.
Your code will live decades.
The more valuable it is, the longer live it probably will be.
I've worked on code that's been 15 years old when I started on it.
And there's code that I wrote 12 years ago that's still running operationally.
And that scares me to some degree.
Because like, oh, did I know what I was doing back then?
But it just reemphasized like why it's so important to think towards the future of what your audience.
You're going to have maintainers that come after you.
And do you want them to curse your name?
Or do you want them to be like, oh, thank goodness, this person wrote something that I can use.
It's really easy.
And I always prefer the latter on that.
I also like to say.
And the nicer code that you write and the more durable and long live code that you write,
the more it can continue to have a life even after you've stepped away from the project, right?
It's like, this thing is still working well.
We can keep growing it rather than it's turned into a complete pile of junk that we've got to
throw away and start over, right?
So that's a goal that you might want.
And then also that long-term maintainer might be you.
Yep.
You might be working out five years from now and you go, why did I make the decisions that I did?
I don't remember this.
Maybe you've jumped to a different project and came back.
It absolutely can be you.
This portion of Talk Python To Me is brought to you by Clubhouse.io.
Happy with your project management tool?
Most tools are either too simple for a growing engineering team to manage everything or way
too complex for anyone to want to use them without constant prodding.
Clubhouse.io, which soon will be changing their name to Shortcut, is different though, because
it's worse.
No, wait.
No, I mean, it's better.
Clubhouse is project management built specifically for software teams.
It's fast, intuitive, flexible, powerful, and many other nice positive adjectives.
Key features include team-based workflows.
Individual teams can use Clubhouse's default workflows or customize them to match the way
they work.
Org-wide goals and roadmaps.
The work in these workflows is automatically tied into larger company goals.
It takes one click to move from a roadmap to a team's work to individual updates and back.
Type version control integration.
Whether you use GitHub, GitLab, or Bitbucket, Clubhouse ties directly into them so you can
update progress from the command line.
Keyboard-friendly interface.
The rest of Clubhouse is just as friendly as their power bar, allowing you to do virtually
anything without touching your mouse.
Throw that thing in the trash.
Iteration planning.
Set weekly priorities and then let Clubhouse run the schedule for you with accompanying burndown
charts and other reporting.
So give it a try over at talkpython.fm/clubhouse.
Again, that's talkpython.fm/clubhouse.
Choose Clubhouse.
Again, soon to be known as Shortcut, because you shouldn't have to project manage your project
management.
Let's talk about some of the core ideas that you have about making software maintainable,
reliable.
One of the things you talk about is the separation of time and how code has to communicate or the
artifacts that we all produce around code has to communicate with people both maybe almost
immediately.
Right?
Like, I'm working on this project with a couple of other developers and we need to,
like, keep it going forward.
The other one is, like, way down in the future, someone comes back and they're like, oh, I'm
new here.
The person who created it left.
Maybe talk about some of those ideas that you've covered.
Yeah.
And so it feels a little weird because what we're talking about is asynchronous communication.
And it's weird to talk about that on a Python podcast and not talk about async away.
Yeah, exactly.
But it's asynchronous communication in real life, which I actually think is much harder.
You have to think about your time traveling to some degree.
You have to think about the future and you have to communicate to them.
You probably will never meet them.
You'll never talk to them.
The only thing that lives on are the artifacts you create.
So your code, your documentation, your commit messages, that's what people in the future are
going to construct this mental model of your work from.
They're going to be doing archaeology when things go wrong.
Why is this code the way it is?
Is it safe to change?
What were the original intentions?
And so the more you can embed that into code you write and the surrounding commit messages,
documentation, the more robust your code base is going to be, the more you're communicating
intent future.
That's one of the things that was so important for 2020, 2021 for all of us, right?
Like, yeah, we didn't know how much we were going to need it because there's always been
this kind of tension.
Well, there's the open source world and these other projects.
And there's those weird remote teams, but we come to our cubicles and we all sit down
and we have our stand up in the morning and we write our software together.
And like that's, and we use, you know, Perforce or something internal where we lock the file.
No one else can edit the file till I unlock the file, right?
Like there's these, these different ways.
And we've been moving more and more towards this sort of everything.
Even if the person is right next to you, the way we work is as if they were across the
world.
Yeah.
And, you know, it's, it's been really, I guess, lucky for us as an industry that that
was mostly in place.
Yes, absolutely.
That actually became true.
Yep.
And so I think you touched on an interesting point of a lot of developers think, oh, if
we're close in space, we can collaborate.
You know, I don't have to worry about my remote teams and I don't have to worry about, you
know, someone global.
But by thinking of that, your collaboration in that terms, it sets you up to think about
the future as well, because you could be asynchronous in space or asynchronous time.
Right.
And basically the same tools are there to, to address it.
The same strategies help with both.
Yeah.
So one of the things that you talked about was this principle of least surprise.
Yeah.
Tell us about that principle.
Yeah.
The principle of least surprise also known as the principle of least astonishment.
I feel like it's safe to say most developers have gone through a code base and been legitimately
surprised and went, that function does that?
Oh my goodness.
I would never have thought that.
Once worked on a nasty bug where the get event function was setting an event.
And I kept overlooking it because I'm like, what's a get event?
It's just a getter.
I can ignore this.
And like two days later.
This will have no side effects.
This will be fine.
Two days later.
I'm like, let me go step through this.
And I was floored.
I'm like, of course this explains my bug.
So your goal when developing software is you don't want to surprise your future readers.
I mean, I think this is why people say avoid clever code, favor clear code over clever code.
You don't want to surprise readers.
Many, many people may not be as well versed in the languages.
Maybe they're coming from a different language, maybe from TypeScript to Python.
The more you rely on clever tricks or poor naming or just the wrong patterns, you're throwing people off for a pee and they need.
There's an added cognitive burden that they must then carry to understand what you've read.
One of the beautiful things about different abstractions are here's a function.
I read the function name.
That's all I need to know.
Here's a class.
I understand what the class does.
I don't need to go and look into the details.
And it lets you build these more larger building blocks of conceptual models.
But if that's not true, right?
Like if a getter is a setter, well, then all of a sudden those things are all out the window and that's bad.
Yeah.
I mean, we work by building mental models and you need trust to build those mental models.
And as soon as that trust is violated, it just starts taking time to do everyday tasks.
You say, I want to implement a single API endpoint.
Well, if I have to go dig through 10 different files just to make sure I'm doing everything right, that's going to slow me down.
If I can trust that my mental model is correct and that it's been shown to be correct time and time again, I can put more faith in that code and I can feel safer to start changing.
Yeah, for sure.
Another thing that I really liked about some of your ideas was that you talked about people having good intentions, even if they write bad code.
Like they're trying their best and they're for the most part.
I mean, there might be people who are just lazy or whatever, but for the most part, like they tried to write this well.
Even if it came out bad, they probably tried to write it well and it just didn't turn out as good as they hoped.
Or even they wrote it well and it's just changed over time and those original assumptions got lost.
Right, right.
It made sense in the early days and the assumptions or the context changed.
Now it's no longer accurate.
Yeah.
Or it's inflexible and it's really hard to use to extend to your current use case.
I see that all the time.
One of the ideas that I thought was interesting is this idea of legacy code.
And I've always been kind of fascinated with what does legacy code mean?
Yeah.
Because legacy code for one person could be COBOL.
Legacy code for another person could be Python 2.6.
That's pretty old.
It could even just be Python 3.6 that's been around for a while, right?
And there's different people who have different definitions.
Michael Feathers has a cool book called Working Effectively or Effectively Working with Legacy Code.
Something like that.
That's an interesting book.
I believe it comes from maybe a slightly different time, but still some of the ideas will make you think.
And I was going to quote that book actually because his is a legacy code base is a code base that doesn't have tests.
And I used to think that for a long time.
I love tests, but I've come to evolve my understanding of legacy code.
And here's where the definition I've settled on.
It's a legacy code, a legacy code base.
It's a code base in which you can no longer communicate to the original authors or maintains.
So the length of time doesn't matter as much.
If you've lost contact from the original authors, all you have is the code base and its surrounding documentation to understand why it's doing the things it does.
So that's become my favorite definition for legacy code as a place.
Yeah, I like that one too.
I don't really like the tests one.
I see where it comes from, but I feel like it's judging a world from too strict of a place.
Because not every piece of code is written in a way that it has to be absolutely correct.
Yeah.
Right.
So I think this is also a good way to sort of scope this conversation.
Because a lot of times people hear, oh, I have to use protocols and I have to use mypy and I have to use X, Y, and Z.
And I have to do all these fancy things because Pat and Michael said so because they were awesome.
And it's a big hassle.
I don't think I need it.
But here I'm trying to be a good software developer, right?
So I'll give you an example.
Years ago, I switched all the Talk Python stuff, especially the Talk Python training stuff, from relational databases and SQLAlchemy over to MongoDB and MongoEngine.
And so I had to write a whole ton of code that would take a couple of tables and then put them into a structure and then put them in Mongo.
And that was all fine and good.
But here's the thing.
The moment that code ran successfully, I never wanted to run it again.
It only had to run once.
It had to move the data one time.
When it was done, there was no scenario where I cared about its typing or I cared about its continuous integration.
I could have deleted it.
I just kept it because, hey, source control.
But there's these scenarios, right?
On the other hand, you talk about if you're an online reservation system, like the reservation booking engine, that part needs an entirely different bit of attention and mindset than my little migration script, right?
Yeah.
And what I see is what value are the things delivering?
Some things have one-off value, and that's perfectly okay.
Your migration script, it's service value, but there won't be much value derived from it in the future.
Maybe from an archaeology standpoint of how did I go do this?
Yes, exactly.
Maybe.
With something that's core to a business, a reservation booking engine, it delivers value when you built it, but you want it to keep delivering value throughout its lifetime.
And furthermore, the people who are working on it want to deliver value just as fast as you did in the beginning.
So you don't want to slow down the future.
You start getting into where you get product managers saying, why is this taking so long?
This is super easy.
Why do you have to spend three weeks just adding this one little field?
And the answer is often, oh, you know, we didn't think about how to enable value faster when we built it.
And there's a tricky line there because you can't just go play everything and say, I'm going to make everything super flexible.
That often has the reverse effect.
It makes things too flexible and that becomes unmaintainable.
But there's a fine line between saying, okay, I'm going to think for the future and deliver value now.
Yeah.
If every dependency can be replaced and everything can be configured from a file, some settings file and like you never, you know, eventually that becomes a nightmare.
It sounds cool.
It's not cool.
I worked on some of those.
Yeah.
It's just painful.
And so here's the advice I give to people who want to think about how to make their code base more maintainable.
Target your moneymakers, the things that produce value, because those are the things you want to protect.
And whatever value means to you.
Target things with high churn.
So you can look in your Git history.
You can see what files change the most.
Chances are those are the files that are being read the most.
They're the files that people are working in the most.
Putting more safeguards in those files, making them more extensible will pay off just because more people are using them.
Look for areas where you do large swaths of changes.
It's called shotgun surgery, where if you want to add a single thing, you have to touch 20 files.
The same 20 files keep getting changed again and again in a grouping.
That tells you that, you know, if I were someone coming in new to the project, how do I know it's 20 and not 19, 21?
That's a place we can simplify.
Those are the things that are super easy to forget a case.
Oh, yeah.
Like, oh, we added this feature and every if statement had an else if that covered the new thing, except for that one where we did the auditing or except for that one where we checked if they were an admin.
Oh, that one.
Now everyone's an admin.
Whoops.
Yep.
And so that's the sort of places where I think type hinting is and other strategies are super useful because you can start encoding that those ideas of I want to catch this when I miss a case.
You can start encoding that into your checks, into linters, type checkers, static analysis, and so on and so forth.
Important.
I think people are pretty familiar with the typing system these days.
I think it's really cool.
The new type system is coming along with more things like 3.9 now lets you write lowercase set bracket integers or whatever rather than from typing import capital set.
And then you can say it in parallel, right?
That's really nice.
With the pipe union definition, you can do like none, pipe, a thing instead of optional.
Yeah.
All of those are really nice and so on.
I suspect that a lot of people are using types for their editor but are not going any further than that with anything like mypy or continuous integration or any of those.
Do you want to maybe speak to like the use case of both of those?
Yeah.
So the use of editors alone is valuable.
You get autocomplete.
I mean, autocomplete alone.
You get squiggly if you do it wrong.
Yeah.
I mean, that's right.
Autocomplete alone is so good.
And I was just thinking when you were talking about that getter that was actually a setter, there's a really good chance that whoever wrote that code knew that was bad.
And yet their tooling was such that it was so error prone for them to change the name that they were willing to live with a getter that changed values.
Because they're like, I could break so much stuff in ways I don't understand if I don't have like a proper set of tools, like a proper editor that'll do multifile refactoring and continuous integration and all of those things.
Right.
And so this is sort of like in that vein of your tools now do more for you.
Yeah.
And so if you take a look at how costly errors are, an error of the customer is incredibly costly when you factor in support and testing and, you know, field engineering, whatever you need to resolve that customer context, not to mention loss of customer faith.
Yeah, that's a big one.
It's expensive for, you know, tests in the later stages of QA to catch an error too, because now our development hasn't been planning to go fix this test or this code.
We have to stop what we're doing and go back and fix this test that maybe worked on three weeks ago.
The best time to catch an error is immediately after you write it.
And that's where that tooling comes in with your editors.
As you're typing, if you can find an error, great.
You is the least amount of costs you could have spent.
And then the second least amount, in my opinion, is letting some sort of stack analysis tool catch it.
Something like my pie.
So using types, you can say, you know, I want to encode some assumptions to my code base.
This value will never be none.
This value may be none.
This value may be an enter string or this string.
If you're right that it's never none, you never have to check it for none.
Right.
But if you're wrong, you always have to check it for none.
Right.
Which is it?
Yeah.
We don't want to do defensive programming of checking is none.
every single variable we create in every function in vacation.
That would just, it wouldn't be fun.
But your alternative, if you don't have that tooling is, all right, does this function return
none?
Let me go look at its source code.
Oh, it calls five other functions.
Let me go look at their source code.
Oh, this calls something to the database.
Is that a nullable feel?
And anytime you're making someone trawl through your code base to try to answer a question of,
can this value be none?
You're wasting their time.
And they're either going to delay delivering features.
That just adds up over time.
Or they're going to make some incorrect assumptions.
And that's going to cause mistakes, which will lead to time and waste money.
You've already talked about focusing your attention to put things like type annotations on the
parts that matter and not necessarily stress about the parts that don't, especially for
code that's being retrofitted.
Oh, absolutely.
I think that's not right.
Like the thing that logs, you know, it's going to turn whatever to a string.
And if it comes out as like, you know, some object at some address, like we'll catch it
later and figure it out.
But it's the core thing that you want that stuff to be right.
But one of the things that can be challenging is the interest and the buy-in and the love for
this idea might not be uniform across your team.
No, no.
You know, and I've seen the same thing for testing and I've seen the same thing for
continuous integration, not necessarily the same person in the same thing, but you know,
it's like if there's a person on your team that just doesn't care about the continuous
integration and it turns off all the notifications that the continuous integration fails and then
they keep checking in stuff and failing the build, you know, like not again, I, I got to
go fix this because this person doesn't bother to check their thing and it just gets super
frustrated.
And I feel like probably typing has a similar analogy to that.
It does.
And that's why if you're in a legacy code base or even a maintained code base that doesn't
have a whole lot of typing in it, there's alternatives you can have just beyond being strategic where
you pick.
There's some fantastic tooling like monkey type, which can annotate your code base for you.
There's Google's type checker, pi type.
It can do type checking without type annotations.
And it does it in a little bit different philosophy than my pi, which I think is capital.
It tries to infer based on just the values throughout your function bodies, the types should be without type
annotations.
So there might be ways to get the benefits without actually making the full commitment to those type checkers for just type
annotations in general.
There's also, I mean, tie your value to it.
Look through your bug reports.
If you find out that, hey, you know, we've had 12 dereferences of none in the past, you know,
month.
None type does not have such attribute, whatever.
Right.
And it's cost us X amount of dollars.
I can now go to someone and say, look, this is costing us real money and cut like our time.
Sometimes data speaks volumes.
Um, if you're having a tough time convincing people, I often say, find the data to back it up.
Look through your bug reports.
If I, what would we have caught?
Oh, how much faster could we go?
Oh, how much I'll do, do a, just even an informal survey around your developer base.
How much more confident do you feel working in your code?
Um, and use that data to decide, yes, this is working or no, we need to look at alternative
strategies.
How much autocomplete do you get?
I mean, that might be a winner, right?
That would for me, actually.
This portion of Talk Python is brought to you by masterworks.io.
You have an investment portfolio worth more than a hundred thousand dollars.
Then this message is for you.
There's a $6 trillion asset class that's in almost every billionaire's portfolio.
In fact, on average, they allocate more than 10% of their overall portfolios to it.
It's outperformed the S&P, gold, and real estate by nearly twofold over the last 25 years.
And no, it's not cryptocurrency, which many experts don't believe is a real asset class.
We're talking about contemporary art.
Thanks to a startup revolutionizing fine art investing, rather than shelling out $20 million
to buy an entire Picasso painting yourself, you can now invest in a fraction of it.
Few realize just how lucrative it can be.
Contemporary art pieces returned 14% on average per year between 1995 and 2020, beating the S&P
by 174%.
Masterworks was founded by a serial tech entrepreneur and top 100 art collector.
After he made millions on art investing personally, he set out to democratize the asset class for
everyone, including you.
Masterworks has been featured in places like the Wall Street Journal, the New York Times,
and Bloomberg.
With more than 200,000 members, demand is exploding.
But lucky for you, Masterworks has hooked me up with 23 passes to skip their extensive waitlist.
Just head over to our link and secure your spot.
Visit talkpython.fm/masterworks or just click the link in your podcast player's show notes.
And be sure to check out their important disclosures at masterworks.io slash disclaimer.
I do want to move on to some other ideas because it's not all about typing, but I think typing
unlocks a lot of these sort of durability ideas that you're covering there.
So another one that you talked a lot about, I think is really interesting in this context,
has to do with collection types and knowing the right data type.
Yes.
And that matters so much, right?
Like somebody might use a dictionary where they should have used a set or something.
And you're like, well, you used a dictionary, so you made me to look this up by value.
Like, no, no, I just want to have one of everything.
Okay, but why do we use the dictionary?
But people, when they're new, they don't necessarily know that.
They find the first thing that works.
They're like, oh, a dictionary worked for this.
We're using dictionaries.
But beyond that, it means something for certain container types and other things, right?
Yeah.
I think if you look at this on a Python of, you know, there should only be one way to do
it.
Most people say, but there's multiple ways to do that.
Like, I can use a dictionary.
I can use a set.
I can use a list and just search for unique values.
And it's encoded in a string and you can parse it every time.
Yeah.
The choices you make, the abstractions you choose, communicate a certain intent.
When you choose to use a set, that tells me I can iterate over it.
There won't be any duplicates.
And I don't, I won't be looking things up by key.
When I think of a dictionary, I think of a mapping from key to value.
The keys must be unique.
But if all I care about was the keys and no values, like there's added cognitive,
just dissonance of why do I have values for this dictionary?
They're all serious.
It doesn't make sense.
Yeah.
It goes back to the principle of, Lisa Sashman.
You get surprised like, oh, this dictionary is being used as a set.
I get it now.
And if you're not addressing that as you find it, you're just kicking the can onto a future
maintainer who then has to add that to the 20 other things he's trying to keep track of
throughout their maintenance of the program.
Um, yeah.
Yeah.
The, you, you mentioned that one of my big pet peeves is, so a dictionary, I
say is a mapping from key to value.
That's typically a homogenous mapping.
Every key is the same type and every value is the same type.
But we really, really, really love dictionaries for things like JSON responses or relationships
of data.
And that can be so detrimental to maintain ability of code bases.
If you're not careful.
The problem is a type checker.
Isn't the greatest at saying, oh, this is a dict.
Some of the keys are strings.
Others are bits.
Others are decimals.
And the values are all over the type.
Uh, you can use a type stick to try to get around that.
But really what you're talking about is a relationship of data.
Uh, when you talk about dictionaries, you're getting into the, well, you, if I see, let's
say I'm code review.
Not only were, put this in concrete terms.
I'm code reviewing some code.
I see someone accessing dictionary and the key is foot.
I have to go look at all of where that dictionary was created and modified to make sure that
who is actually a valid field in that dictionary.
I have no guarantees.
Just if I see dict and a type checker, I'm sorry, a type annotation.
So again, it's that trawling through the code base.
Oh, this is actually coming from an API.
Now I have to go read the API and I can't effectively code review code or maintain code.
If I'm just reading that through without doing that every time to make sure something hasn't
changed.
So in this case, I'd say use a data class.
You know, you have explicit fields.
You can, you can lean on your tooling.
If you mess up the field, you aren't expecting to see new fields get created and probably 95,
99% of the time.
Or, and I'm like, I will prefer a data class to a dictionary.
If I have heterogeneous data almost all the time.
Yeah.
And maybe get it back like, like a flask API call called JSON.
And then just jam that star star of that thing into the data class.
Something like that.
Yep.
And I, I know you've talked about Pydantic on the show.
Oh, a lot.
I love Pydantic.
Just, you know, define a model, let it parse that JSON response and just build that data
class for me.
Throw an error if it's invalid.
And I really liked that model of, of attacking programs like that.
I love Pydantic.
I love how it, it tries to sort of be flexible.
It's like, if we, if I think we can fix this, if you had a thing that is a string, but in
the string, it really is parsable to a number and it's supposed to be a number.
I'll just go ahead and do that type conversion for you.
If not, give you a decent error message.
It's really, really lovely.
Yep.
So if I'm, I'm working with APIs, I love Pydantic for that, that reason that you just outlined,
but I'll often convert it to a data class or Pydantic data class so that I can say, this
is a relationship of data and I can kind of shape how a user uses that relationship of
data throughout the lifetime of the code.
Yeah.
It's an interesting tension on how much those models get used throughout all the tiers of
your app and how much you want to keep them separate.
Have you seen, SQL model?
I believe it's called.
Oh, I have not.
I have not.
Okay.
Uh, here we go.
You probably heard a FastAPI, right?
Yeah, of course.
Obviously that's where Pydantic got, I think it's, it's a big boost.
So Sebastian Ramirez came out a few days ago with, this thing called SQL model and it
already, I think it's less than a week old and has 4,000 GitHub.
stars, which is amazing, but basically it's a merging of Pydantic and SQLAlchemy.
Oh, fantastic.
It has a SQLAlchemy, you didn't work model and it underlying has all the SQLAlchemy
stuff, but it's models are actually Pydantic models.
Yeah.
And I think this is just another illustration of why thinking about what types things are,
even if you're not doing type annotations, why that's so important.
It goes back to how do we build mental models?
How do we build these, these abstractions in our brain that we can rely upon as we
work through our code base?
I do think this is interesting in that you could use the same model at the data level
internal to your app, and then you could even use the API level, but there's also people
are saying, but maybe that's not a good idea.
Maybe you want to separate those in, in interesting ways for like one can change and the other doesn't
have to change.
But yeah, I've, I see a lot of value to this thing.
It's, it looks exciting.
And I mean, the answer to that's going to probably be, it depends if your application needs them
to be the same.
Lovely surprise, make them the same.
If they have different reasons to change.
This is something I see a lot too.
We all get the dry principle.
Don't repeat yourself and rain in our head.
And we think, oh, source code looks the same.
We must deduplicate it.
But if that source code can change for different reasons, you're going to add more headaches
by deduplicating it.
You're going to start adding special cases.
Well, this thing needs to change, but the other thing doesn't.
How do I reconcile that?
Special cases.
And soon you, you've become the very thing you've sworn to destroy as you build that out.
It's just littered with special cases as you're trying to tie together two things that have
two separate contexts.
So, you know, if you want to use something like SQL model for a API level and your internal
data model, ask yourself, do these have different reasons for change?
Maybe, maybe in the future.
And like a lot of this is just empathy for the future.
When I talk about maintainable code, robust code, you have to have empathy for those future
maintainers, put yourself in their shoes.
Are they going to want to migrate sometime and maybe change the API that they present to
customers or users or other developers?
Do you want to change your database at the same time?
If so, keep them together.
If you want to keep that separate so that you have certain migration paths, maybe you keep
them separate.
And so it's going to depend case by case.
But again, everything comes down to those first principles of what are you communicating
with your intent when you make that decision in your code?
Like we've touched on, you're building different things at different times under different
constraints.
Are you building Instagram's API, which millions of people and apps are using?
Or are you building something really quick so you can get that app to work for marketing
for the next thing for the week?
Right?
Yeah.
If, if it's that the second one, then you, you don't want to worry about too much abstraction
and you just want to go like, I just need these to be in sync.
The data goes here.
We're going to send over the API.
We're going to be good.
Right?
This is going to fly.
On the other hand, if you're building something incredibly consumed and long lived, then maybe
you have different design patterns and care about it.
And I think it's important to think about organizational boundaries is the, the users, the consumers,
the actors in your use cases are, is there an organizational boundary separating in the Instagram
case?
The people who are going to be using my API, I have no control over in my organization.
It's general public as far as I'm concerned.
If it's just someone on my team using it, we can work through, okay, you know, I'm changing
this API.
Let me help you through that.
But if it's someone outside your team or outside your organization, you're going to have a backwards
compatibility to think about.
You're going to have all these vast amount of things that people are going to complain about.
And you need to think about that and say, okay, can I make these changes?
And is it good for my user base?
Even though I can't control them, the best I can do is entice them.
You'll never be able to force someone to use something they don't want to.
If you don't have that control over them, that's why we have open source forks.
One of the ideas that I think comes up in this, this whole story, and you talk a lot about
like inheritance, both inheritance in terms of class hierarchies and even multiple inheritance,
but also maybe the more traditional interface style with protocols, just so that you can express
pipe stuff separately, right?
In interesting ways without coming up with inheritance beasts and whatnot.
Yeah.
So one thing I did want to give people a quick shout out.
So we've all heard of solid principles, right?
And I've, I really enjoyed the solid principles and I thought they were super interesting.
I still do.
I think they're pretty great.
I recently came across a presentation and it was, I kind of over on Richard Campbell's
show.net rocks, which I know people probably don't care about that now, but they actually
talked about this thing called Cupid, which is an alternative to solid.
I'm going to go check that out.
And it was super, it has nothing to do with.net.
It's just like software in general patterns.
Like what, what do we know now, 20 years later, that doesn't necessarily make so much sense
for solid.
And what is the alternative?
And he has this really cute name.
So as people are thinking about maybe those kinds of levels, like that's a pretty interesting
to check out.
Yeah.
And I like thinking of solid again, going back to first principles, there are things that
I like about solid.
There are things I also think get a bad rep because they're associated with heavy, heavy
OO code bases that have class hierarchies that are just unmanageable.
Probably multiple templates in there somehow.
Yeah.
And we've kind of grouped these things together.
So an example, like one of the most misunderstood ones, in my opinion, is the list of substitution
principle of the solid principles.
And like it applies when you're doing duck typing in Python, which has nothing to do with
class hierarchies.
It doesn't have anything to do with interfaces or things like that.
It's a turnaround substitutability.
And so when we talk about duck typing in Python, it's, well, multiple types can represent some
behavior.
Maybe they support an addition method and a subtraction method.
Can they be substituted into this function that's expecting addition and subtraction?
And so when we think about-
I do addition and subtraction on dates and times.
Another might do it on like imaginary numbers.
But the concept of adding and subtracting things, you can reuse that, right?
Something like that.
And so things like list of substitutability, excuse me, list of substitutability principle
talks about how do you think about substitutability from your requirements and from your behaviors?
And you can get value from that without ever touching a class.
So I think all too often we lump those solid principles to a strict OO of the 90s or the knots and think that they aren't as useful.
I'm going to have to check out Kupin and see how that's changed as well.
I would expect that a lot of the same first principles are true, but they've kind of reframed it in a way that is more applicable to how we program today.
I don't remember all the details exactly.
So, but yeah, something roughly along those lines.
I think that makes sense.
So one thing I do also want to just sort of get your thoughts on before we leave it completely in the dust is you talked about a lot of times you've got a dictionary and it's supposed to represent some stuff.
Maybe it's a response from an API where it says, you know, here was what you requested.
Here was the status code.
Here's what the cloud cost of that is.
And here's like the, you know, some, something that looks like that maybe doesn't make sense as a dictionary.
Maybe it gets moved into like a Pydantic model or something like that.
Right.
But the other one is I have 10,000 users and I want to put their email address as the key and their user object as the value.
And given an email address, I want to know super fast, which user is that?
Or do I have that user at all?
In that case, the dictionary makes a lot of sense.
Absolutely.
But there's a super big difference to say the keys represent like the same thing across different results.
Whereas the same data structure, a dictionary represents this heterogeneous, like not really a class, but kind of a class.
How do you position those so that beginners understand that those are completely unrelated things that need to be thought about separately, but they kind of appear the same in code?
Because when I'm talking to people about dictionaries and I'm teaching, they're like,
Oh, well, why would you use a dictionary here?
Because over here you were doing it this other way, but now it's like a database.
Like, what are you doing with it?
This is weird.
Why are these not, why are they the same but different?
So here's how I find it.
So for the key mapping value, I would say there's a few use cases you're going to have.
You're going to either be iterating over the dictionary to do something on every element, or you're going to be looking up a specific element.
And that element that lookup is dynamic, typically passing in some value.
Maybe it's a variable that contains the email address in your example.
For something like an API response, something that might be more heterogeneous, something that should be a data class or a Pydantic model, what have you.
You typically aren't iterating over every key.
You're looking up specific keys, but it's static to the index, meaning you're passing in string literals between the square brackets.
This is the key I want on this circumstance.
You know, the name, the age, the date of birth.
Right.
And so you're looking up specific fields and you're building a relationship between different key value pairs.
A name, age, and date of birth is a relationship called purse.
With your dictionary, you're doing, or with your mapping, your key value mapping, there's no relationship between email one to user object one and email two to user object two.
The relationship is just from key to value.
And that's typically what I explain to people trying to discern the difference.
That's a really interesting way to think about it because, yeah, you will almost always use static literal strings when it's the heterogeneous API response style.
And you will almost never do that.
Why would you say, quote, type in the email address?
You would just have that object.
You wouldn't need to type in, like, you would just never create that structure in the first place, which I think is really interesting.
So, yeah, if you're dynamically passing in the keys, it's probably all the same object, but different ones of them as opposed to an API response.
And I think that's the heart of why I really wanted to write about robust Python.
So I've been programming for a while now, and I make a lot of decisions as a senior engineer.
I really started to step back and ask myself, why do I make these decisions?
Why do I choose a dictionary over a data class?
Why do I choose a class over a data class?
Why do I choose an acceptance test over a unit test?
Why do I choose to do a plug-in here, but a dependency injection somewhere else?
And I just wanted to start documenting.
Here's why I make the decisions I did.
Here's why, what intent I'm trying to communicate through in the hopes that we start normalizing that conversation more in our field.
Why are we doing the things we do?
As a beginner, it's frustrating.
Oh, wow.
You use a dictionary here.
Why?
How did you ever know to do that?
Yeah.
And so I wanted to really capture, again, I've said it a lot.
The first principle, why am I doing the things I'm doing?
What are the things I often give as advice to junior programmers, intermediate programmers, too, and senior programmers?
It's just, it was really enlightening for me to really step back and try to dissect why I do the things I do in Python and come up with, here are the valid engineering reasons behind it and try to frame that in terms of maintainability.
Yeah.
I think that's very valuable.
I think it'll help people learn.
Because it's not enough to see it by example.
You're like, well, these both kind of work.
It's like, yeah, but they mean something different.
They totally communicate something different.
A couple of thoughts from the live stream I want to share.
Kim Van Wick out there says fantastic advice about having too much configurability.
Far too often, I prematurely de-dupe two similar blocks of code and then find myself adding special cases.
I wouldn't need, yeah.
That sounds like what you were saying as well.
Yeah.
And so what I would say is that there's a difference between policies and mechanism.
The policies are your business logic and the mechanisms are how you go to do something.
Logging module in Python is fantastic.
You're logging.
The logger module doesn't care what you're logging, how you're logging it, when you're logging it.
It's just the mechanism.
To dev null, who cares?
Yeah.
It's all the same.
But the what you're logging is your policy.
And so very often I try to find a way to say, okay, how do I make my mechanisms de-duplicate?
Because those are what I have to reuse.
My business rules are going to change.
They're going to change for different reasons.
I want to make that simple to define business rules, but keep the mechanisms reusable.
You want to be able to compose those mechanisms together.
Also on the live stream, Mr. Hypermagnetic says, is it faster to look up a user or whatever
from a dictionary rather than iterate over a list?
And I think, boy, oh boy, if you've been down that path, you know there's a difference.
This is coming from an advice from a C++ programmer who really cared about performance.
Measure it.
Use cases can be surprising.
If there are times where a contiguous block of memory with a binary search is faster than
a dictionary lookup.
But you will not know until you measure it.
There are also things that need to be faster than your program.
And there are many things that don't fast in your program.
I think it's important being aware of which one those are.
You don't want to, not to sound cliche, the whole premature optimization is the root of all
evil.
Breaking that down a bit, you don't want to just optimize everything because it takes time to
optimize things.
And it can obfuscate your code to optimize.
Why am I entering over a list when a dictionary lookup can be just as useful?
If you measure the parts that need to be fast and you say, this is slow and this way is
faster.
As long as you have a comment and a good commit message that says why you're doing the slightly
obfuscated way or breaking that law of least surprise.
As long as you have those breadcrumbs for someone to follow and go, oh, it's for speed.
This is critical performance loop of my application.
Maybe it's the web request handler or something in a database engine.
You want to get those breadcrumbs so people can learn why you made the choices you did.
The last thing you want someone in five years to say, oh, they were iterating over lists because
they don't know that a dictionary exists.
Let me go change that and slow down your app.
But without that measurement, without that data, it's a fruitless endeavor.
I mean, for a lot of data in general, a dictionary lookup would be a lot faster.
But if what you have to do is every time create the dictionary and then do the lookup, probably
that act of creating the dictionary erases the speed up of, because you would have just,
you're kind of looping it over anyway, right?
So it certainly, it depends.
But yeah, if it's pre-created.
Yeah, don't take what I've said as prescriptive.
Measure, measure, measure, measure.
Yeah, absolutely.
Absolutely.
And the invoking just your command line application with dash MC profile is super illuminating.
How long have I spent in a function?
How long, how many times did I call this function?
Where is most of my execution time spent in?
And it might be that the bottleneck you think you have is somewhere else completely.
Well, yeah, that's actually a really important point as well.
If you do that C profile and you find you're spending 5% of your time in that part you're trying
to optimize, if you could make it go to zero, it's still going to be 95% as fast, right?
So it might be able to be sped up significantly, but it might not actually be germane to making
your program feel any faster because it's like optimizing nothing, basically.
And I don't want to misrepresent.
To go back to the question on the live stream, typically a dictionary lookup will be faster than
entering or row list.
I expect that most of the time.
I would not rely on that all of the time.
Yeah.
That's why I advocate measuring.
Yeah.
Measuring, measuring, measuring.
I've definitely been in a place where I'm like, oh, this 6,000 line, super complicated
thing that is hard to understand and work on.
This is where it's slow.
And no, it was just some other like wrong data structure and some other very simple part.
But if I didn't measure, I would have gone and tried to rewrite the part.
I was like barely understanding.
I would have probably broken it and still would have been slow.
Yeah.
I forget who mentioned it.
There's the order of writing things is you make it correct so that, and you have tests
around it so that you know that it stays correct.
You then work on making it clear and then you make it fast if it needs to be.
And that last part is optional based on, is this slowing, actually slowing me down?
Is there a business value I'm losing?
If it, you know, there are certain things I work with.
Um, I, we're publishing something once a day.
Something takes another five minutes.
I don't really care.
Like my time is spent better on focus on parallelization or things of that nature.
And so it's just really understanding why things are slowing down and where a little bit.
There's a little bit of a tangent of robust Python, but it's an interesting question to
bring up.
Number less, you know, is that five minutes?
Are you waiting on that five minutes?
No, no, no.
You kick it off and then you go.
Yeah, exactly.
Then you probably don't care.
I care at all.
But speaking of, time, I'm sure there are a ton of people listening that are like,
I would love to take some of these ideas, maybe using my pie or putting more types or
looking at stuff like Pydantic and so on.
But at the same time, I, there's a lot of pressure on me to just from the time I receive the request
to get some feature done until the feature is done.
And the people making that request, managers, business owners, clients, whatever, they don't
care that much about robust Python.
No.
They want minimum input of money and time for output of features.
But anyone who has done that for any extended period of time knows that that is a net negative.
Yes.
How do you address that?
I'll give you some thoughts of what I've done, but how do you address that for people who
the way I feel receiving that I want to frame the conversation for a sub it is your duty
to deliver value as quickly as you can.
It is also your duty to make it so that your maintainers can deliver value at the same space
in the same space.
So if you're making, if you're delivering value today, but hindering your ability to deliver
a month, three months from now, that's a problem.
Now I'm not saying jeopardize your MVP or go-to market, but making sure that the business
people understand, I can do this, I'm not going to be able to do this again.
The other part that I say that is just practice.
Practice with these sort of concepts and apply them in small places.
You can do things incrementally.
Maybe there's a couple hundred lines of a library you have.
Just write type annotations for that or make it more extensible.
Find small little wins here and there.
And as you keep working out, just this Boy Scout rule, leave the code base cleaner than you
found it.
So if I'm, if I'm going into something and said, you know what, these assumptions don't
hold up anymore.
I can't fix it all right now, but I can do one incremental step that makes it easier.
The next time I'm in here, I take those little bites over time and slowly morph into something
as like, okay, as we start slowing down delivery, I'm starting to make those improvements.
And then we start speeding up back again.
Because there is a real tension between deliver it now and how making sure that you are going
to be able to deliver hastily in the future.
And by no means am I advocating gold plating where, oh, make everything maintainable now.
Because there's typically a business behind this and you don't want the business to fail.
You need to be able to be able to ship early and often.
And it's that often part that is often the tricky part.
It is tricky.
Practice, practice, practice, and target where you're applying these ideas.
You don't have to put all that flexibility in there first, right?
You write it one way, you get it out.
But then you have refactoring tools.
You have continuous integration.
You have things that allow you to make those changes as long as you keep using them and as
long as the code doesn't get too bad.
And here's one of the things I've always liked seeing.
I forget who said this.
There's a great quote.
It's like, to make a hard change, first make the hard change easy.
That's the hard part.
And then go make the easy change.
Yeah.
When you see, like a lot of teams do some sort of estimation of their stories.
How big will this effort take?
If you see consistently high estimates, this is always a large.
We're always doing larges and extra larges.
Ask yourself, what would it take to make this a small?
Sometimes you use Sivanachi numbers.
If this is an eight, what would it take for me to make this a three?
And some of your listeners may be like, what are you talking about?
But estimation, we're estimating effort.
Right.
Yeah.
Traditionally, way, way back in the day, people used to estimate in hours.
How many hours is this going to take you?
And it's like, that is a granularity that we don't have.
I can't tell you is it 17 or 18 hours.
I can tell you is it two hours or two days, but I can't tell you whether it's 17 hours.
And so when you get large estimates again and again, ask yourself, are these estimates because
of necessary complexity, complexity that's inherent to my domain, or is it accidental complexity
that's built up over time?
I have to talk.
So necessary complexity is, you know, the deep workings of a neural net or flight controller
software.
Like these are inherently hard problems.
I can't make that simpler, not without making a whole lot of money revamping an industry,
but things like tax preparation, right?
Like there is a minimum set of complexity to just figure out what that answer is because
the rules are complex.
Yeah.
But if let's take that tax, if I have to, you know, maybe add a new rule and it's just a
simple multiplication somewhere, I shouldn't have to add 20 to 30 files of changes just to
implement that one thing, because it's going to happen again and again and again.
And so you look through these common cases and say, you know, we keep saying this is a large
effort.
How do I make that smaller?
And then you say, oh, you know, if we just restructured things here, you often find that
the same amount of effort to go make that hard change easy will pay off in one to two
cycles of like, okay, now that we're doing this again and again and again, like, yes,
I took an extra day longer or week longer.
And again, be mindful of your schedules.
I'm not saying so schedules intentionally, but find that wiggle room where you can band together
with your team and find those areas.
Oh, this is a place that we can get some gains back.
I often like to set junior engineers on the team at these sort of tasks.
They get more familiar with the code.
They often don't have as many responsibilities as a senior engineer.
And they can just get that understanding and they feel like, oh, I've done something that
really, really matters.
I really like being an engineer.
Right, so you're made on a cleanup job.
Yeah.
And that has worked wonders for me as well.
And like, it's just worked.
It's a win-win.
And then you're delivering those features, those same things you're doing again and again
at a much greater speed.
And that saves you time to go do the next thing after.
My approach to this, I mean, it only works with certain groups of stakeholders, I suppose.
But when somebody would ask me, how long does this feature take?
It wasn't like, well, here's the essence of the feature.
Like it will work and you can click the button.
Here's how much it'll take to add error handling.
Here's how much it'll take to add logging.
Here's how much it'll add to make that work in continuous integration and have tests.
And here's how much we got to do to like not add technical debt while adding this feature.
Right.
Instead of presenting like, well, here's your menu.
Well, all I want is the feature.
Just give me the feature.
Right.
I just say, okay, well, if the feature is three hours, the other two hours, like five hours,
it can take me five hours or whatever like metric used.
That is what delivering it means in a professional setting is that it is done, not that it's minimal
path, zero errors and zero bad data will execute.
Right.
They're not the same, but it's easy to get sucked into the minimum.
How quick can I do this?
I bet I can do it an hour if I really fly.
Right.
And I think that's a trap we often fall into.
Like we're as engineers, it is our, it's our responsibility to provide accurate estimates
for what's best for the business.
And like, if you say, oh, I can do this without tests, what you're really telling people are
I can do it, but I can't guarantee it will keep working after I deploy it.
Cause I will have no way of knowing I've removed that visibility of tests.
Tell me, oh, this important thing that you really wanted now.
No idea if it's working now.
Sorry.
And like, once you start framing it like that, people go, okay, yeah.
Yeah.
Okay.
Let's see what we can do.
And you might need to have some hard conversations with the people who manage us.
It might be need more headcount.
We don't need us.
We can't take as many projects.
We can't take, you know, we can't take this much complexity.
Maybe, you know, maybe a worldwide pandemic is happening and are, you know, we've been severely
impacted and I mean, you can't, to some degree, you can't fix poor management of something,
but you can coach your way up to try to get out of it.
Some people are very receptive to that.
Some people are not.
And in the places that aren't, I mean, you have to ask yourself, is this something that
I'm going to live with or is this something that bothers me and I need to think about other
things?
Well, I think you're probably in a pretty good space if you're making that clear to people.
So I think if you're proposing a feature or how long will it take to do this feature,
then you just, you just include what that means for a feature to be done.
Like it has, it, it's not like zero bad data and it works sometimes.
Like it's, it works for real.
The harder story is we've gotten into a bad place.
We need to not add features and just refactor improvement.
Like that is a different story than like, what does that mean?
And that's a really tough one.
Cause if I'm a business person, I'm never going to say, yes, please stop feature delivery
and making money for us to go build the same thing we already have.
Like that's a tough sell to a business.
And I really think just an incremental approach.
And again, I mentioned earlier targeting strategically.
What are your most used libraries or parts of code?
What are the areas with the highest bug count?
Put data to that.
Hey, we've had five customer complaints about a crash in your system.
They're linked to this one piece.
If we start doing this, we save X amount of money and we would have kept that one customer.
That's the sort of thing that speaks when you're getting against those somewhat unrealistic business
pressures.
You have to speak on their, on their level.
And I, what I'm saying is a little wrong because it's not an us versus them type mentality.
You're in this together.
You feel us versus them at some times, but you need to work together, make them understand
here are the business implications of the choices we're making.
Exactly.
I mean, you're the engineer and they're the business people, but you've got to just put
the situation in terms that they appreciate and they can decide if it makes sense for them.
Yeah.
I would also add people out there listening that if the answer is almost never, can we clean
up technical debt and make this better?
You probably will end up with a team over time who is full of people who don't care about
erasing technical debt and making it better.
And it's only just going to get worse and worse in combinatorial ways.
It's, it's not where the top engineers want to be and they don't get excited about coming
to work where they have to like, you know, sneak in something that's unreliable and cram
new features into broken, ugly code.
And something I've learned over a long time programming is almost every technical solution,
there's a people or process problem at the root of it.
Like there, there's a lot of things that we think, oh, code can solve this.
If I just refactor this, everything will be great.
But there's a people aspect of this.
And I'm not putting down those people.
Like everyone around you is just living, breathing human being.
They all have their own hopes, their own dreams, their own obstacles, their own places in life.
And we, we have to recognize that sometimes, sometimes you have to work with the people
in order to solve that technical problem.
So much for the, mythos of the software engineer in the basement by themselves, huh?
And I think that, that, stereotype is long dead for the most part.
I agree.
All right.
Well, we go on and on, but I think we're out of time.
So thanks for all your thoughts on this and, and the book's interesting and we'll talk
more about it in a second.
But before we do, if you're going to write some of this code, work on some of these ideas,
you know, what editor are you up to?
Yeah.
I use VS Code, cause I still jump between languages quite a bit.
And I like the flexibility of that.
Um, for a onesie twosie, I'll use a van.
Cool.
Then notable PI PI package.
I'm going to call it Stevedore.
Cause, a package used for, plugins, doing plugin development.
I think it's really neat how it uses a, the packages entry points so that you can deploy
plugins as separate pip installable packages.
Uh, just something I wish more people knew about.
That's awesome.
Yeah.
One of the things we don't put, we didn't cover, but it's also part of this robust story
that you tell is about extensibility, about plugins, about all those kinds of things.
Right.
Yeah.
And that just goes back to making the common cases simple.
If you, if you're doing something again and again and again, make it easy for people to
do that.
And extensible code is a way to do that.
Yeah.
Like that tax example, right?
If, if you could plug in the formulas that are applied in this place and then like you just
add it in and maybe it just picks it up and goes, that would be ideal.
Yeah.
All right.
Awesome.
Good recommendation.
One that I had not heard of.
All right.
So people are interested in this idea.
I assume that they can check out your book.
Maybe tell people about that real quick.
And then like, what else would you provide as resources or places to get started?
Yeah.
I think just talking to senior members on your team, talking to junior members on your team,
learning how people use the code around you, keeping an ear for your, your stakeholders,
your code developers, think about the code you write, think about the implications it has.
I mean, just listen.
Most of this is empathy.
I thought I was writing a technical book.
I wrote a book about empathy instead.
Just didn't know it.
Put yourself in other people's shoes and think about how they're going to be receiver code
a month from now, a year from now, five years from now.
And again, just, I mean, a lot of talks you'll see at Python or the other Python conferences,
you'll see these nuggets of truth throughout them.
Ask yourself, why are they making the decisions they do?
And see if you can really understand the kind of the root cause of why they're using this
feature, especially as new features come out with Python.
A lot of great ideas in here and I had a fun time talking about them.
Oh, me too.
Yeah.
Thanks for being here, Pat.
Been an absolute pleasure.
Thank you so much.
You're welcome.
Bye.
This has been another episode of Talk Python to Me.
Our guest in this episode was Patrick Viafor, and it's been brought to you by Clubhouse,
Masterworks, and Assembly AI.
Choose clubhouse.io for tracking all of your work because you shouldn't have to project manage
your project management.
Visit talkpython.fm/clubhouse.
Make contemporary art your investment portfolio's unfair advantage.
With Masterworks, you can invest in fractional works of fine art.
Visit talkpython.fm/masterworks.
Do you need a great automatic speech-to-text API?
Get human-level accuracy in just a few lines of code.
Visit talkpython.fm/assemblyai.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at Talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show.
Open your favorite podcast app and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play,
and the direct RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air,
be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I'll see you next time.