#246: Practices of the Python Pro Transcript
00:00 When can you call yourself a professional developer?
00:02 Sure, getting paid to write code is probably part of the formula, but when is your skill set up to that level?
00:08 Many folks in the industry suffer from imposter syndrome or other types of uncertainty.
00:13 Yet, there are real techniques and skills you should know before you meet this bar.
00:18 Dane Hillard is here to share his take on the practices of the Python Pro.
00:23 We'll discuss concrete steps as well as abstract design concepts to help your code make the jump to pro level.
00:30 This is Talk Python to Me, episode 246, recorded December 5th, 2019.
00:34 Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.
00:53 This is your host, Michael Kennedy.
00:55 Follow me on Twitter where I'm @mkennedy.
00:58 Keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via at Talk Python.
01:04 This episode is brought to you by Tidelift and Linode.
01:07 Please check out what they're offering during their segments.
01:09 It really helps support the show.
01:10 Dane, welcome to Talk Python to Me.
01:13 Hey, thanks.
01:14 It's great to have you here.
01:15 I'm really excited about this topic, actually.
01:16 I think you've got something good going on here.
01:18 It'd be fun to talk about.
01:19 Yeah, thanks a lot.
01:19 Thanks for having me.
01:20 Yeah, we're going to learn what a pro means.
01:23 I mean, there's a lot of debate about titles and when are you no longer a beginner or you're
01:28 actually a professional programmer and all these things.
01:30 So that's going to be a lot of fun to dig into.
01:31 But before we get to that, let's start with your story.
01:33 How'd you get into programming in Python?
01:35 For programming in general, I followed a fairly traditional path.
01:39 I went to school for computer engineering, mostly learning C++ at the time, MATLAB, things
01:45 like that.
01:46 And then computer engineering is kind of this mix of hardware and software disciplines, which
01:52 I've mostly not made use of since school.
01:55 So I've stuck very much to the software side.
01:58 But since I got out of school, I've been doing everything from Perl and PHP and C++ and C
02:06 to Python and JavaScript as well.
02:09 That sounds fun.
02:10 You know, the whole hardware side of computers, it's kind of a black box to me.
02:15 And it's just something that I've never really learned.
02:18 I mean, obviously, I understand what RAM is and a disk is and buses and all that.
02:22 But I couldn't build one or design one or even understand what the little parts mean when
02:27 I take it apart other than this goes in the PCI slot.
02:29 So that's probably fun to know that level, right?
02:33 It is.
02:34 And I've admittedly forgotten more than I would like.
02:37 But, you know, I think I still try to keep with it for things like Raspberry Pi and Arduino
02:42 and things like that from time to time.
02:44 So it's sort of fun to keep as a hobby.
02:46 Yeah.
02:47 It's a bit of a renaissance for that, right?
02:48 I mean, it used to be you had to be Dell or somebody like this to make these things.
02:52 And now you can spend five bucks and get a little microchip.
02:55 Yeah, absolutely.
02:55 It's kind of cool.
02:56 It's an empowering time in that regard.
02:59 Yeah, absolutely.
03:00 So you got out of school and you're working with all these different languages.
03:02 How did you come to Python?
03:04 I want to say that there was a short precursor to my time with Python that was learning Ruby.
03:10 And that was kind of at the behest of a friend of mine who was doing a hackathon and wanted
03:15 to build some, I think like a domain specific language of some kind on top of Ruby for something.
03:20 And I had never used sort of a dynamically typed language or anything like that.
03:26 And Ruby was kind of a refreshing experience from what I knew at the time.
03:30 And I think around that time, I was also looking for a new language to write my personal website in.
03:36 The current iteration was in PHP at the time.
03:39 And I had tried Java Spring a bit and didn't like that too much.
03:43 So I decided to try Django.
03:46 And I think Django was sort of my first major foray into Python because I remember learning a lot of Python stuff at the same time I was learning Django.
03:54 Right, right, right.
03:54 That's cool.
03:55 Well, I think a lot of people come into Python because of Django.
03:58 Yeah, I would agree.
03:59 I want to do something on the web.
04:00 I heard Django is easy and cool.
04:02 Let's try that.
04:03 Oh, I guess I have to learn Python.
04:04 Well, that wasn't too hard.
04:06 It's kind of easy language to learn.
04:08 But, you know, that also touches a little bit on the subject of what we're going to cover, right?
04:12 Some people say, oh, you can learn Python in a weekend.
04:14 It's so easy.
04:16 And yet, just weekly, I'm learning new things on Python.
04:20 And I spend all day diving into it and looking at all the edge cases.
04:24 And you can't learn Python in a weekend.
04:26 But you can sort of get it together.
04:27 And I think there's a really interesting spectrum there from, hey, I made this website.
04:32 And I kind of know the language, too.
04:33 I'm really good at this language and the ecosystem and so on.
04:37 It's broader than it first seems or harder than it first seems.
04:39 Yeah, absolutely.
04:40 And I mean, I think there's plenty of people out there, too, who don't want or need to get to that depth for a given language.
04:48 Especially people who are kind of polyglot, if you will, and need to know a bevy of languages.
04:54 Sometimes just knowing enough of that language to solve the problems in those various domains is totally, totally good.
05:01 So, yeah, there's such a spectrum of situations and experiences there.
05:06 Yeah, for sure.
05:06 And then there's the folks who are doing something else.
05:09 Economics, biology.
05:11 And they just need enough to get through their biology problem or to run that economic simulation.
05:16 And that they don't care about the nuances of generators.
05:20 They just want to know whether, like, they can get pandas to spit out the answer they need or map plot to, like, show the graph that they need so they can carry on.
05:29 Yeah, exactly.
05:29 For sure.
05:30 All right.
05:30 So, before we get to the main topic, maybe just tell us what you do day to day so people know where you come from.
05:35 Yeah.
05:35 So, I'm still very much into the web side of things these days.
05:40 I'm working on jstor.org.
05:42 It's an academic research platform, kind of a journal database.
05:46 And I've been there for about five years now.
05:49 Before that, I was actually in biology to some degree.
05:53 I was doing some bioinformatics work.
05:55 And I did do a little bit of Python scripting there, but a lot of database administration and things like that also.
06:01 And before that, I was doing some radar telemetry.
06:05 So, it's quite the spectrum of stuff.
06:07 Yeah.
06:07 I feel like bioinformatics would actually be really interesting.
06:11 Yeah, it really was.
06:12 And I think, to me, that's often, like, the joy of programming is that you're both solving problems with code directly, but also kind of picking up these secondhand areas of knowledge about some different subject.
06:27 Right?
06:27 So, people have a chance to kind of explore beyond just knowing how to write code.
06:32 Right.
06:33 If you get bored of writing bank software, you can go write, you know, renewable energy software or take your bank.
06:38 Yeah, exactly.
06:39 Yeah, that's actually pretty awesome.
06:40 Because unlike many jobs, developers are in demand across all the industries.
06:47 Basically, there's this quote by Satya Nadella that every company is a software company these days.
06:52 And I kind of feel like that's right.
06:53 Yeah, that resonates with me, too.
06:55 I think it's, you know, every company has some area, whether it's, I don't know, it ranges from mundane to interesting.
07:01 But every company has this area where they can use software to augment whatever it is that they're doing, whether it's to do directly what they need for their business, or if it's just to make their business better in some way, faster, more efficient.
07:16 If they're thinking, well, we need to hire 10 people to like manually do this thing, because it's really slow and tedious.
07:22 Well, maybe people shouldn't be doing that thing.
07:25 Maybe they should be overseeing the machines doing that thing or something to that effect.
07:28 Yeah.
07:28 Yeah.
07:28 So, let's get into this topic.
07:30 We're going to talk about your book, Practices of the Professional Python Pro.
07:35 And let's just start with that last word there.
07:37 Like, what makes a developer a pro?
07:39 I mean, you've had a lot of experiences, a lot of different types of companies.
07:42 What do you think?
07:43 What's the indicator for you?
07:45 Yeah.
07:45 So, the title is almost kind of got this double entendre to it.
07:49 It gets at this idea that there are practices that can make you a more effective developer.
07:56 There are things you can think about as you're developing code to improve the outcomes that you get.
08:02 And that is stuff that over time, sometimes decades, right, whole careers, you can build and learn and never run out of stuff to start learning.
08:12 But at the same time, I think a lot of these things are things anybody writing code could start to pick up bits and pieces of right away.
08:22 And so, in that regard, everyone who's paid to do something is a professional, right?
08:28 Right.
08:29 Right.
08:29 Like, if that's you, congrats, you've made it.
08:32 And so, I think some of what the book is about also is just like, hey, if you're new and you just want to know what people in the industry are kind of thinking about,
08:41 maybe you haven't done software professionally before, maybe you've used it in your biology work or something like that, or you're switching fields altogether.
08:51 It's kind of a way to get some insight into the kinds of things you might want to be thinking about.
08:57 Yeah, absolutely.
08:57 And what I like the title, I do like how it sort of puts that out there is,
09:01 these are the things that you would expect somebody who really cares about their craft, either because they love it or because they're paid enough that they're supposed to pay attention to that.
09:10 Whatever reason, these are the techniques and things that they do.
09:14 You probably are not doing all of them.
09:16 Probably you've come from some other industry or you're new to the industry, or maybe even you've been a professional C++ developer for 20 years.
09:24 And all of a sudden, somebody told you compilers are out and you're going to go write this funny language with no semicolons.
09:30 What do these people do in their pro side of things, right?
09:33 Right.
09:33 And as experts here, I think there's a lot to take away for a lot of folks.
09:38 But yeah, this idea of just what is a pro or maybe, you know, more broadly, so we're not hung up on the money side of it, just like an expert in a language or, you know, in Python.
09:49 It's an interesting question of when you can call yourself a developer, when you call yourself a professional developer or something like that.
09:57 I don't really know where that is.
09:58 We can think a little bit more about that.
10:00 But I do know that beginners get frustrated really easily when they're trying to work.
10:05 And I'm thinking more beginner programmers, not beginner just in Python.
10:08 They've got to solve some problem.
10:10 They're like, oh, my gosh, I just heard I have to use a database for this.
10:13 I don't even know what a database is.
10:15 Not really.
10:15 So what do I do now?
10:17 And it can be really frustrating.
10:18 So I don't know.
10:19 What are your thoughts for folks who are not at this level?
10:22 But, you know, maybe things like this book would help them get there.
10:25 What would you tell them?
10:26 Yeah, I think that a lot of the source of frustration as a developer or as a beginning developer is that sometimes you aren't not only do you not know the answer, but sometimes you don't even know the question.
10:38 Yeah.
10:38 So you're often kind of trying to think about what it is you even are trying to know, whether it's because an error you saw is too obscure or you just can't quite ask the right question.
10:51 And so, like my general advice outside the book or sort of tangential to the book is to start figuring out how to search better, like for your problems.
11:01 Yes.
11:01 So now let me just let me stop you for a second, because I agree with that.
11:05 But I feel like a lot of people who are new probably perceive searching for a fix as giving up or I failed to like get this understanding correctly.
11:16 Yeah.
11:17 I think and I don't mean this in a negative way, but I think that there's this whole sort of hidden realm that a lot of people don't see where it's Googling like crazy.
11:26 As a professional software developer, you're like, okay, I got to use this library.
11:29 I don't know anything about it.
11:30 Let me take a shot.
11:31 Nope, that didn't work.
11:32 Let's take that error.
11:33 Let's search the heck out of that.
11:34 Oh, no, I'm over here.
11:35 Oh, wait, this is another thing that seems better.
11:37 I'm going to search.
11:37 It's just there's a lot of searching.
11:40 And really soon, at least in my world, when I'm like, if this is not working and I don't have an answer in two minutes, I'm on the Internet trying to find it because often that works.
11:48 Yeah, I think that's a really great thing to dispel because it doesn't go away and perhaps only increases as your sort of breadth of knowledge grows or at least breadth of awareness of what's out there and your search skills grow.
12:05 You kind of have a bigger surface area to explore and you have to figure out which piece to hone in on to maybe find your problem.
12:12 And so it's hard to say just go learn how to search better, I guess.
12:17 Yeah, that's one of the elements, but I think it's part of it.
12:20 Yeah.
12:20 Yeah.
12:21 And there is also that danger, right, that you land on either the incorrect answer or something as a beginner that you wouldn't necessarily know how to assess the validity of.
12:31 Yes, that's true.
12:33 From just plain wrong to malicious or something, right?
12:36 Or even just it's a perfectly good answer, but it's not the same situation.
12:41 Yeah.
12:42 But you don't have the nuance to know that's not actually what I'm looking for.
12:46 It's the next Stack Overflow one I got to go after.
12:49 Yeah.
12:50 So unfortunately, the answer there sometimes is that there's a lot of trial and error.
12:55 But again, that sort of sticks with you throughout your career, I think.
12:59 Yeah.
13:00 Yeah.
13:00 I did a show back on 203 with Ned Batchelder and some other folks about beginners and experts in this panel talking about a bunch of beginners.
13:09 And I had Ned as the expert.
13:11 And it was a really interesting conversation.
13:13 And a few of the things that I sort of condensed out of that conversation was there's a lot of things that beginners perceive as a, this is me doing it wrong or this is me not good at it.
13:25 But it's just that's how it feels.
13:27 Like, for example, being stuck and trying to get a library to work when it's not or trying to get some, like the compiler to work if you're in a compiled language or whatever.
13:35 Just being stuck or feeling kind of lost is just part of the journey.
13:39 And just like, as a professional, as a pro, you're like, oh, we're in this part of the journey.
13:43 We're in the next half hour is going to suck.
13:45 And I'm going to be on Google a lot.
13:46 But here we go.
13:47 And then you just know if you just keep hitting that problem, eventually you'll be gone through it.
13:52 Another one, I think knowing data structures is really, really good for people.
13:58 And then, like you said, kind of having these battle scars of, yeah, it looks like they're talking about your problem, but that's not the problem.
14:05 It's this other thing.
14:06 Because I've tried four times that way, and that was really a bad week.
14:09 And that didn't work.
14:10 So now we're over here.
14:11 And some of those you can learn, like just knowing that, well, if you're stuck, you just pound through it.
14:16 And that's just part of the journey.
14:17 It's fine.
14:18 The building up the battle scars, like that just takes time, I think.
14:21 Yeah, absolutely.
14:22 And like you said, when you see a Stack Overflow answer and you think it might be the one to solve your problem, there's a similar thing as you're using like third-party libraries.
14:31 Yes, true.
14:32 You can't get it to work.
14:34 You can't get it to work, but you don't yet know if it's because of something you did or if it's really because there's a bug.
14:42 And getting to uncovering that bug can sometimes be an exciting moment, even though it's frustrating getting there because you're like, oh, I actually exercised a problem that other people might be experiencing.
14:55 And then you can sort of contribute back to that discussion.
14:59 And if that bug's already been filed or something, you can say, hey, I ran into this too.
15:03 And it just kind of grows the community in that way.
15:06 Yeah, it's interesting.
15:07 I just got an email today from GitHub.
15:09 I had filed an issue for PyInstaller.
15:12 I was trying to combine it with something else.
15:14 And it wasn't quite working right.
15:16 I was doing something weird.
15:17 Or no, it wasn't PyInstaller.
15:18 It was GUI.
15:19 I was using those two together.
15:20 And I got a message today.
15:22 Somebody else said, I'm having exactly the same problem.
15:24 What can we do about it?
15:26 I'm like, oh, I still can't solve it for you.
15:28 But I started the conversation six months ago or whatever it was.
15:31 Yeah.
15:32 So let me ask.
15:34 It's serendipitous sometimes.
15:35 Yeah, it definitely is.
15:36 It definitely is.
15:38 It's aha moments.
15:38 Yeah, it's funny.
15:42 This portion of Talk Python to Me is brought to you by Tidelift.
15:45 Tidelift is the first managed open source subscription, giving you commercial support and maintenance
15:50 for the open source dependencies you use to build your applications.
15:54 And with Tidelift, you not only get more dependable software, but you pay the maintainers of the exact
15:59 packages you're using, which means your software will keep getting better.
16:03 The Tidelift subscription covers millions of open source projects across Python, JavaScript,
16:07 Java, PHP, Ruby, .NET, and more.
16:10 And the subscription includes security updates, licensing, verification, and indemnification,
16:15 maintenance and code improvements, package selection and version guidance, roadmap input,
16:20 and tooling and cloud integration.
16:22 The bottom line is you get the capabilities you'd expect and require from commercial software.
16:27 But now for all the key open source software you depend upon, just visit talkpython.fm
16:33 slash Tidelift to get started today.
16:38 Another bad feeling is when you're searching for something that you don't quite understand how it works.
16:42 And the best answer you can find is something you've written before.
16:44 That's also bad.
16:45 The cup lost.
16:49 All hope is lost now.
16:50 So let me ask you, thinking back, I'm not sure if you can remember clearly to this,
16:55 but when was the first time you felt like, I'm actually a professional software developer.
17:00 Like, I'm a pro now.
17:01 I fit in with all these people who used to seem way smarter than me and I couldn't figure out what they were talking about.
17:06 It's a great question.
17:07 I don't know that I have sort of this decisive moment.
17:12 So I guess one place that I had a kind of clear division line was from my first job,
17:19 everyone was kind of a generation older than me at my office.
17:22 And I mean, they were all great mentors and I learned a ton of stuff, but I definitely realized my own shortcomings often.
17:31 And then sort of at my next position, people were more my peers in age and experience.
17:37 And I think when mostly it was that kind of process of solving things together.
17:45 And like collectively, we all knew about the same amount of stuff.
17:49 And suddenly people are asking you to like, not only execute on a solution,
17:56 but actually figure out what the solution is supposed to be.
17:59 Right.
17:59 I think that's really where it starts to make you grow as a developer.
18:04 So I would say that's, to me, that's kind of the hinging moment.
18:08 Yeah, that's interesting.
18:09 Like, Dane, we've elected that you're going to write the e-commerce part of the website.
18:13 You're like, I've never done that one.
18:15 Neither have we.
18:15 So someone's got to figure it out.
18:16 So we've elected you.
18:17 We got trusted you, right?
18:18 I guess for me, there's probably, I guess there's probably two experiences.
18:22 One, the first time I got paid a good salary to set and work on software
18:27 and create applications for this company was so awesome.
18:31 I was so excited.
18:32 But I kept thinking, man, I better figure some of this stuff out before they realize,
18:36 I don't know what I'm doing.
18:37 They're going to kick me out of here.
18:38 You know what I mean?
18:40 So that was sort of the first step in the door.
18:43 Like, wow, I'm really getting paid to write and create software.
18:45 And it's fantastic, right?
18:48 But I didn't really feel that totally solid.
18:51 And I guess where I really felt like, okay, I guess I'm a pro, I guess, was when I was giving user group talks on subjects, design patterns or testing or whatever it was.
19:05 And I went to, you know, all these people are enthusiastic.
19:07 They go to meetups and stuff and gave a talk.
19:10 And they're like, that was really great.
19:11 I learned some stuff.
19:11 And they didn't laugh at me.
19:13 I'm like, that was great.
19:15 Apparently, I, you know, present ideas to people.
19:18 And just, you know, there's a couple of those milestones I can think of.
19:21 But yeah, it's really hard to put a clear demarcation on it.
19:24 Yeah, I mean, going back to your point about feeling like people are going to find you out, right?
19:28 Like, that is the epitome of imposter syndrome, which I think is, especially in knowledge work and like that kind of area.
19:36 I think people experience that to a large degree.
19:39 So that's another thing to call out as like a beginning developer.
19:42 If you have this constant feeling that somebody is going to call you out for being a fake, like that's real.
19:50 That's a thing and that sometimes doesn't really go away either.
19:54 I know I certainly feel it at least from time to time still, right?
19:57 Yeah, for sure.
19:58 It's hard for that to go away.
19:59 But at the same time, you know, if you're making steady progress and learning a little bit more and learning a little bit more,
20:04 if you're delivering what people want from you, just because you don't know how to factor that database into a third normal form,
20:09 if that's not your job at the moment, you're probably fine.
20:12 You know what I mean?
20:13 Yeah, exactly.
20:13 Cool.
20:13 So speaking of topics and things like third normal form you should know or design patterns or software design and all that,
20:20 let's dig into some of the ideas behind your book because I thought you did a good job laying out some of these things.
20:24 And again, I don't think it's professional development, but more like what do you really need to be kind of an expert in software development in general and Python in particular, right?
20:34 Probably the right focus, huh?
20:36 Yep, I would say so.
20:37 Yeah.
20:37 So why do you write this?
20:38 The impetus for this, I think, was seeing a few of my friends who were not in sort of the traditional path to software switching into a career in software.
20:49 And seeing kind of how they thought about problems and the kinds of questions they were asking me at the time,
20:56 that sort of insight into the kinds of things people were experiencing made me kind of interested in helping other people who were in that same situation.
21:06 One of my friends was using Python to solve computational biology problems, computational chemistry problems, if I remember correctly.
21:17 And it was sort of asking me to review his code from time to time and ask like what he was doing wrong or what he could do better.
21:26 And it's sometimes hard to answer that question, right?
21:29 If it's someone who isn't in software, you don't necessarily want to dump the entire world of, you know, you should format your code this way or you should optimize for readability or you should comment your code for these reasons.
21:43 Like it can overload people.
21:45 So I figured a book could be a more digestible thing that people can kind of jump in and out of as needed.
21:52 And I tried to organize it in a way that they could sort of pick areas that they want to get better at and read about that topic without having to know everything.
22:01 Right. It does seem like it's a little bit of this cookbook feel like you can drop in and I need to work on design.
22:06 I need to work on testing or whatever it happens to be.
22:08 Yeah. Nice.
22:09 So let's start at the beginning, I guess.
22:12 And you talk a lot about design and planning and software architecture to get started.
22:17 And I guess one of the challenges, I guess it's probably two challenges that are really tricky for people right away is first, like what the heck is software design?
22:25 You know, like if you know, when you see that it's not designed, if there's just a, there's no functions, there's no comments, there's bad variable.
22:33 Like, you know, this is not designed and this is not professional.
22:36 And probably this other thing, if I go to GitHub and, you know, pull down Flask and I look at its code, it probably is designed, but that's a little bit tricky.
22:44 But knowing, you know, how much energy to put into it, what are the first steps that you take?
22:49 What do you think about that?
22:50 Yeah. I mean, I definitely emphasize in the book that to the point about energy, like things,
22:55 are always trade-offs as you develop software, right?
22:59 You only have so much time in your day.
23:01 And realistically, you do need to ship software sometime.
23:04 There's always a trade-off between how much time you spend upfront designing things versus how much time you spend actually implementing things and getting them out the door.
23:14 So that is worth stating, I think, explicitly here.
23:17 And then sort of generally, I think about design often the way you sort of think about solving problems in general, right?
23:26 Like the problem is often too big to solve in one go.
23:30 And so we typically break problems down into smaller chunks that we can solve.
23:34 And that is a lot of what software design is too, right?
23:38 Like this idea of separation of concerns and things like that, encapsulation, abstraction.
23:42 They're all about not doing the whole world at one time.
23:48 Sure. I've tried to boil the ocean before.
23:49 It's not so fun.
23:50 Like I remember there's this project that a whole team of us worked on for years, at least a year.
23:58 And I somehow discovered this cool linting tool that would come along and tell you all the issues, even security issues, performance issues, and ran it against that.
24:08 And it found like a thousand problems.
24:10 And I started to work on a couple and it took like 10 minutes per one.
24:14 Like, yep, we're not doing this.
24:15 Just throw that away.
24:16 We're just going to just keep going the way it's going.
24:18 We can't take the next half year off and just make it the same, but slightly better.
24:25 So there is this real tension of, I know this could be better.
24:29 Maybe I should refactor this to use not deep inheritance, but I should use composition.
24:33 Or I should refactor this so this is a separate package and then we can all use that and things along those lines.
24:39 But one of the challenges with software design, I think, is it's almost got this creativity, this art aspect to it, where you could just keep polishing or keep sculpting indefinitely, right?
24:51 I could continually refactor this code or restructure it basically indefinitely.
24:57 And so, you know, you need to know maybe some reasons or triggers that say it's time now to go put a little effort into this or it's time to stop.
25:06 Yeah, I could probably could refactor that as well.
25:08 But you know what?
25:08 Nobody touches that part.
25:10 It doesn't really matter.
25:11 Yeah, that's a great point.
25:12 And I think a lot of newcomers, as they learn sort of new tricks, if you will, tend to try to use them as much as they can, which is it's a great way to continue learning that subject and understand more of the nuances around it.
25:25 But sometimes if a nuance around, like you said, when to stop or when things are good enough is missed, you can certainly continue much too far down that track.
25:35 Yeah, so let me throw out a couple of ideas on what I might think of as like clues there and you can comment or add to it.
25:41 Yeah.
25:41 So if I'm trying to add a feature and it's super hard to add the feature without causing a bug, I need to make a change and I got to touch a bunch of places or something like that.
25:51 That seems like I need to redesign something so that doesn't happen too much.
25:54 Or there's a part that bugs keep coming back to or something like that.
25:59 If it's really problematic and every time you need to make a change, I got to spend 10, 15 minutes on this section just to make sure it doesn't break or to make sure it's in sync over and over and over again.
26:11 I guess if you feel annoyed at your code, you're like, this is really, you're like, wait a minute, maybe that's a signal that I should take that frustration and fix rather than just endure this code.
26:22 Yeah, that's exactly right.
26:23 I think the squeaky wheel gets the grease in that sense, right?
26:27 Like as you go about your day-to-day work, if you find an area of code that's commonly the thing causing you grief, that is a pretty strong signal.
26:35 And to the point about going too far, I think if it is giving you grief at that time and you choose to attack it, like it's very easy to get tunnel vision and want to solve it to its extremity, but try to focus on solving the thing that was causing you pain.
26:52 And, you know, if you need to change some peripheral things to solve that direct problem, that's okay.
26:58 But try not to fix every problem until they really jump out at you.
27:03 Yeah.
27:04 And you probably also need some guide rails, maybe safety nets is a better analogy here.
27:10 If you're going to start tearing stuff up, maybe you want to have tests to make sure that it still works or something like that, right?
27:15 Like it's not every bit of code needs testing, but if you're going to start really, really pounding on it, you know, might be worth putting some basic rough.
27:23 Does this kind of still sort of work tests in place before you do that?
27:28 Right.
27:28 These are like these characterization tests, right?
27:30 Like you want to make sure that the most important aspects of your code continue to stay.
27:36 Right.
27:36 If this breaks, it's going to be really bad.
27:39 In line with your expectations.
27:40 Right.
27:41 Or the MetaBank, the core trading engine that decides buy or sell, that part can't break.
27:46 So we better fuck this up now.
27:48 I'm sure I'm going to hear of it if I switch a minus to a plus the wrong way.
27:53 Yeah.
27:55 That's not one you want to hear about with a production error going up.
28:00 No, no, it's definitely not.
28:02 I guess when it comes to this design and like restructuring your code.
28:06 So I kind of try to follow what I guess I'd call the Boy Scout rule.
28:10 I was never a Boy Scout.
28:11 So maybe I'm mischaracterizing them, but just leave the code slightly nicer than I found it.
28:16 You know, if I come in here, instead of making it worse, what could I do in a little bit here and there to make it better than when I, if I got to touch it anyway.
28:25 How do I make it a little bit better?
28:26 Yeah.
28:27 And I think the corollary of that is that that's a great place to start incrementally improving your code if you're not sure where to start.
28:34 Like as you're working on something already, whether it's to add a feature or to fix a bug, why not use that place where you're already working as the place to improve?
28:44 So if it feels like there's too much and you need to fix everything, start by fixing where you're already going.
28:51 Right.
28:51 Yeah, that's a good point.
28:52 So you call it some interesting things, some considerations of making software good as a user and as a developer.
29:00 And I think these can also guide this design thinking and this refactoring story.
29:06 So some of the considerations as a user of how the software is designed.
29:10 I mean, you as a user, hopefully you don't see the code, you know, you see the buttons and the widgets and whatnot, but you still get a sense of like, I can tell you there's certain times where I've worked with certain websites.
29:20 I'm like, whoa, this is one shady, crappy website right here.
29:24 Or other ones are like, this thing is super slick and maybe it's crummy underneath, but it feels like at least pretty well put together.
29:30 And some of those for the user or like speed, you know, how quickly does it respond to me?
29:35 Or data integrity, does the data get corrupted?
29:39 And is it accurate in terms of resources?
29:42 Maybe if it's a website, you don't see these, but certainly if it's a desktop app or something, you would.
29:47 And, you know, does it feel secure, right?
29:50 So these are certainly things that people can think about as well, right?
29:53 I talk a little bit in the book about when is good enough to the point of speed.
29:57 Things are often down to your user's perception, right?
30:01 You don't necessarily need to continue making something faster from 100 milliseconds to one millisecond.
30:08 If it's, for instance, some sort of UI interaction, like human perception doesn't go much faster than about 100 milliseconds, right?
30:17 So there's heuristics, if you will, that can kind of help guide where you should shoot for.
30:22 Yeah, and that's interesting.
30:23 Certainly, there's the single user speed and then there's the scalability speed, you know.
30:29 On Friday, nobody can use the website because everyone's entering their timesheet or something like that, right?
30:34 But on Monday, it's really fast.
30:36 That's also another one to consider.
30:40 At least in the web world, maybe even in apps for APIs and stuff.
30:44 There's a lot of stuff that doesn't, it seems like it's sort of some weird side case.
30:51 I don't really care about this.
30:52 For example, if I'm working on a website and I want to make it fast and the page loads in 150 milliseconds, I could optimize indices.
31:00 I could refactor databases.
31:03 I could do cluster.
31:04 I could do all sorts of stuff.
31:05 But if there's a bunch of JavaScript and a bunch of CSS and a bunch of images that don't have the right cache setting and the page appears to me as a user in one and a half seconds, whether I shave 10, 20 milliseconds off it, I don't care.
31:19 The thing is super, super slow.
31:21 I'm not going to call any companies out, but there's so many professional sites out there.
31:25 Big companies are like, why is this so bad?
31:27 It's inconceivable to me that it is this bad.
31:31 And I actually have a recommendation for people.
31:34 I don't know if you're familiar with this, but you know PageSpeed Insights from Google?
31:38 Yeah.
31:38 So that's a really great place.
31:39 You can just put your domain or actually put any particular page in there and it'll tell you how fast your site is.
31:46 So, for example, the training site that I run, the Talk Python training, its response time is super fast.
31:51 I've got that dialed in so well and it was responded in like 15 milliseconds, right?
31:57 Out the other side of the web server.
31:58 That seemed like, well, I can't do much better than that.
32:00 No one's going to perceive that last five milliseconds.
32:02 I throw it into PageSpeed Insight and it says, your pages, it's average to slow.
32:07 I'm like, wait, wait, wait, wait, wait, wait.
32:09 What is it?
32:10 Like, it's 15 milliseconds.
32:11 What are you talking about?
32:11 And then there's all these other things like, well, this type of caching does this.
32:15 And these images are like 1,200 pixels.
32:18 But when you put them into a phone, the phone has to resize it and that makes it render more slowly and stuff.
32:23 So I did actually a ton of work and now I've got it down to 95 on mobile and 99 out of 100 on its speed.
32:31 Higher is better, right?
32:32 So there's actually different levels, right?
32:35 Like, you can get really focused in on, well, let me make the database query fast and it should be fast.
32:39 But there might be bigger problems that are actually easier to solve.
32:43 That's a really good point.
32:44 That process that you described sounds very familiar to me.
32:47 Did you put something into PageSpeed Insight?
32:49 You're like, I thought this was good.
32:51 What is the problem here?
32:52 This is not what I expected.
32:53 Yeah, exactly.
32:54 And like Google Webmaster Tools start to incorporate some of that data too.
32:59 So you can see like how often you're terrible.
33:01 Yeah.
33:02 Well, and to put a little more fire under people to like try this out, Google is starting to use that for rank.
33:10 Absolutely.
33:11 Right?
33:11 So it's an advantage.
33:13 Yeah.
33:13 They're including things like on mobile, if you have an icon-based navigation menu or something, if the icons are too close together, they'll ding you because it's difficult to touch the icon you meant to, right?
33:28 Especially for people with mobility issues or things like that.
33:31 So they're really starting to, like you said, incorporate that back into how well you're actually ranked.
33:37 Yeah.
33:37 Yeah.
33:37 So I guess to wrap it up is like, you definitely want a fast website, fast app, whatever, but make sure you're viewing it from the user perspective.
33:46 What does fast really mean?
33:48 You know, if something is running quick, but another part is really slow, make sure that the way the user feels it is actually the way you want them to feel it, I guess.
33:58 Yeah.
33:58 Well said.
33:59 Yeah.
33:59 Yeah.
34:01 This portion of Talk Python to Me is brought to you by Linode.
34:04 Whether you're working on a personal project or managing your enterprise's infrastructure, Linode has the pricing, support, and scale that you need to take your project to the next level.
34:13 With 11 data centers worldwide, including their newest data center in Sydney, Australia, enterprise-grade hardware, S3-compatible storage, and the next-generation network, Linode delivers the performance that you expect at a price that you don't.
34:28 Get started on Linode.
34:30 And you'll be able to take your time to have a lot of data center in the next level.
34:32 And you'll be able to take your time to learn.
34:33 And you'll be able to learn.
34:34 And you'll be able to learn.
34:34 And you'll be able to learn.
34:35 And you'll be able to learn.
34:35 And you'll be able to learn.
34:36 And you'll be able to learn.
34:36 And you'll be able to learn.
34:37 And you'll be able to learn.
34:38 And you'll be able to learn.
34:38 And you'll be able to learn.
34:39 And you'll be able to learn.
34:40 And you'll be able to learn.
34:41 And you'll be able to learn.
34:42 And you'll be able to learn.
34:43 And you'll be able to learn.
34:44 And you'll be able to learn.
34:45 And you'll be able to learn.
34:46 And you'll be able to learn.
34:47 So that was considerations as a user.
35:03 Speed, integrity, resources, security.
35:05 But as a developer who's working on this project.
35:08 Be it a library or an application or whatever.
35:11 There's a different set of considerations, right?
35:13 Yeah.
35:14 I mean, from the developer perspective.
35:15 In the book.
35:17 Sort of a famous idiom.
35:18 It's like, make it work.
35:19 Make it right.
35:20 Make it fast.
35:21 So often we get to the make it work, right?
35:25 Like that's the software we often ship.
35:27 Or the software we often write to start.
35:30 Yeah.
35:30 And I think part of the problem is also the project managers.
35:32 They're like, oh, I see it working.
35:35 Our sprint is almost over.
35:36 You can pick another item out of the sprint.
35:38 You're like, no, no, no, no.
35:39 You don't understand the technical debt that is deep within this beast.
35:42 I have nowhere near done.
35:44 Right?
35:45 I actually ended up at some point, not lying, but just when somebody says, how long is this
35:51 going to take?
35:51 It's not like how long to minimal viable feature.
35:54 It is how long to feature plus unit test plus refactoring.
35:59 That's what I'm going to tell them in the estimate it is because otherwise I'm going to hate life
36:03 in six months.
36:03 It's like that builds up.
36:05 Yeah.
36:05 In my experience, the best time to incorporate those things is as you're writing the code,
36:11 because at least then you can't, you have some leverage because things aren't shipped
36:16 yet.
36:16 Right?
36:16 Yeah.
36:18 And, you know, six months from now, if you didn't address the tech debt back then,
36:23 it's really often difficult.
36:24 And, you know, it's understandable, right?
36:26 Because at the end of the day, most of us work for some sort of business and they do have
36:30 to have value to keep the lights on.
36:32 And so that's why I kind of encourage people to bring this in as an iterative part of their
36:38 day-to-day programming process, because again, that's kind of the best time to do it, especially
36:43 because you have all the context and everything that's easy to lose after, you know, even a
36:49 couple hours after leaving some code.
36:51 So yeah, it's hard to get that back.
36:53 It's like, it's much easier to keep your credit card paid off than it is to deal with
36:58 it when it's maxed out.
36:59 Yeah, exactly.
37:01 I mean, you've had the same amount of money over the same period of time, but it doesn't
37:04 feel the same.
37:05 Yeah.
37:05 And so like, as you start shipping stuff, you also then, as a developer, you've hopefully
37:11 already worked on the ideas of speed and security and things like that as part of the feature
37:16 you've developed.
37:17 But sort of under all of that is the actual code that you wrote.
37:22 If ever you need to return to it to add another feature or fix a bug or fix performance, the way
37:28 you've designed your code is sort of when it becomes really important.
37:32 Because I often find if I haven't put a lot of thought into some code, when I come back
37:37 to it later, I'm kind of viewing it for the first time, even if I was the one who wrote
37:43 it, you know?
37:43 What is that saying?
37:45 Like code as if a person inheriting this code in six months is a psychopath and they know
37:50 where you live or something like that?
37:51 Right.
37:51 Yeah.
37:51 I mean, but that sort of applies to you as well, because you'll forget, right?
37:55 You'll forget.
37:56 And it seems so clear and simple at the time when it was all perfectly balanced in your mind.
38:02 But then when you haven't picked it up for a long time, you're like, why is this still
38:05 not working?
38:06 It's like a whack-a-mole game.
38:08 Yeah.
38:09 The best is when you like get blame a line and you're like, oh, I did that.
38:13 This is wrong.
38:15 Who wrote this?
38:15 Get blame.
38:16 Oh, yeah.
38:17 All right.
38:17 I mean, the worst to me is when I'm like trying to understand some stanza of code and I'm like,
38:24 I need to go ask someone what this does.
38:27 It's like, who should I ask?
38:29 Well, I guess I need a time machine.
38:33 Yes, exactly.
38:34 If it says you wrote it, then well, you're kind of out of luck because the buck stops
38:39 there.
38:39 That's right.
38:40 Yeah.
38:40 Yeah.
38:40 So I think these are all really interesting ideas.
38:44 I do feel like one of the areas you should focus most on is make it easy to add new features.
38:49 And some of the things that come out of that are maybe loose coupling and extensibility.
38:54 Because if one little change means I have to change five other parts of the application,
38:59 well, then you do have to hold a lot of those things in your mind.
39:03 But if you've got a very clear structure, like, well, here's all the strategies we have
39:08 for this part of our app.
39:09 And at another one, you just kind of clone this thing and derive from that and register
39:14 it there.
39:14 And now there's another option in the UI or whatever, right?
39:18 You can definitely help yourself or hurt yourself in that regard.
39:21 Yeah.
39:22 I mean, we often talk about software complexity as a sort of like this inherently bad thing,
39:28 which I wouldn't necessarily agree is true.
39:31 I mean, you can sort of argue the semantics of it, I suppose.
39:34 But to me, like the danger of complexity is that people, not software and code, but people
39:42 have trouble understanding it and keeping it all in their head.
39:46 And they will inevitably not have the full context when they make some change.
39:51 And so bugs become an inevitability and security failures become an inevitability.
39:59 And that's where things really break down.
40:01 So the value in kind of breaking down complexity is often for people's benefit.
40:07 It's nothing about the code itself, if that makes sense.
40:10 Right.
40:10 Absolutely.
40:11 And with the right design, you can control how much of that complexity has to be cached
40:16 in your brain.
40:17 Yeah.
40:17 Right.
40:18 Like the software may be super complex, but you just got to work with this little part.
40:21 And you know, if I'm working this part, it interacts with this other thing through this
40:25 clear way.
40:26 And long as I have that in my mind, I can't mess up the rest of it.
40:29 Or you design it poorly and you change it 10 places because somebody's hard coded a string
40:34 duplicated in four places.
40:35 That means something.
40:36 And now you better not forget that fourth place or you're, you know, in trouble.
40:40 Yeah.
40:40 And so I think new people often pretty readily understand the value of writing a function,
40:47 right?
40:47 Like here, you need this six times.
40:50 And instead of writing it six times, you can write it one time.
40:53 And so they get the value of that kind of statement.
40:55 And then this book kind of tries to bring that concept along to layers of abstraction and encapsulation
41:03 and things as concepts that you can further use to kind of get that same benefit at a larger
41:08 scale on more complex systems, right?
41:10 Yeah, that's a good point.
41:11 Obviously, functions are probably the lowest level that people generally have that thought
41:15 process.
41:16 But yeah, you have a whole section where you talk about separation of concerns and talk about
41:21 functions, but there's also classes and there's modules and there's packages.
41:25 And probably the whole step from different modules, I feel like not everyone, but there's a lot of
41:33 folks in the Python space that are hesitant to have many files that make up their application.
41:38 They'd prefer to have just one like 2000 line file, even if it's got a bunch of functions and
41:43 classes in it.
41:44 And I don't know, there's just this reticence to do that.
41:48 So certainly focusing on modules and then even maybe packages for broader reuse makes a lot
41:55 of sense.
41:55 Yeah.
41:56 And I think it's easy to go crazy with those as well.
41:59 You could go to the other extreme and have like one function per module and have a thousand
42:04 modules, right?
42:06 Again, it's kind of this balance that you have to strike and it's whatever.
42:10 At the end of the day, it's always like the goal is always whatever helps people understand what they need to
42:19 understand when they need to understand it and no more, no less.
42:23 Right.
42:23 If I could say group five actions and a few pieces of data into a module, and then if all
42:30 the people have to think about is I use this module, it solves this problem.
42:34 It takes this input.
42:35 It has this output, right?
42:37 They no longer have to worry about the details or what changes there or anything.
42:41 And while that's useful on one level, if you can break your whole way of thinking your program
42:46 about that, you can just think at a different level and the building blocks become much more
42:51 easy to think about how you position them and how you use them and so on.
42:55 And I think changing the way people think about problems is a really important part of this
43:01 separation of concerns and this design thinking.
43:03 If you can change the building blocks as people perceive it to be large and clear rather than many,
43:11 many small pieces, that's huge.
43:12 Even through the course of this book, my sense of that concept has definitely grown.
43:19 It's very easy to think of the value of refactoring or pulling out functions and things as strictly
43:26 for reuse purposes.
43:28 Or testing purposes or something.
43:29 Yeah.
43:29 Yeah.
43:30 And so I've come to kind of feel that instead the real value definitely is that sort of cognitive
43:36 load aspect.
43:38 Yeah.
43:38 I find the similar value with design patterns.
43:42 If instead of thinking of these 10 lines of code, I can think of it as a strategy or a
43:48 something else, right?
43:49 It's even though design patterns are not used that often in the Python space, it's just a
43:54 little bit simpler.
43:54 So many of the design patterns are about solving the inflexibility of static languages.
44:00 And so you don't have to really think about them as much, but there's certainly many cases
44:04 where design patterns are super valuable.
44:07 You know, even just things like passing a Lambda function to a list to sort it rather than
44:13 getting in there and trying to deal with it yourself or whatever.
44:15 It's pretty nice.
44:16 My experience is that it just takes a lot of trial and error to get there.
44:20 I don't know.
44:20 I think there's a lot of design patterns applied from things like Java and C++ and C# that
44:24 are just, they just don't belong.
44:26 Right?
44:26 Like really complicated singleton patterns where, you know what, that could just be a global
44:31 variable in a module.
44:32 All right.
44:32 What's next?
44:33 You know what I mean?
44:34 Yep.
44:34 It's the same thing.
44:35 Yeah.
44:36 And so I kind of, in the book, I deliberately kind of skirt around this idea of design patterns
44:42 explicitly.
44:43 I do kind of mention it at the end as something to learn more about for sure.
44:48 But I think like the value they provide is taking some concept and naming it.
44:55 Like once you have a name for a thing and everyone agrees what that name is, like you
44:59 can start to have conversations about it and whether it's the right thing or the wrong thing.
45:03 Right.
45:04 And it has, it has clear trade-offs, you know, well, like for singleton, it has the trade-off
45:09 that you have shared data and it could be really tricky to test it or other things, but it also
45:13 is really clear how you use it.
45:14 And so I think it's not just the name, it's that the name plus known trade-off, like these
45:20 are the positives.
45:21 These are the pros and these are the cons.
45:23 Do we use that here?
45:24 Like that's really valuable to bring in quickly.
45:26 Yeah.
45:27 And so like, if you didn't have a name for that, you couldn't even start to apply those
45:32 traits and trade-offs.
45:33 Right.
45:33 So even if you aren't using design patterns per se in your code, it's good to at least
45:38 understand.
45:39 I mean, there will be patterns of some kind, right?
45:42 Whether they're design patterns that have a name or not.
45:44 Right.
45:45 Right.
45:45 Are they organic or are they like well-known ones?
45:48 Yeah.
45:48 Right.
45:48 So they give you the name just gives you kind of the hook around which you can, you can
45:54 explore more.
45:54 Yeah.
45:54 Yeah.
45:55 The nomenclature.
45:55 That's cool.
45:56 Yeah.
45:56 Yeah.
45:56 So another thing you talk about in the book is designing for performance.
46:00 Do you want to touch on some of the highlights from that?
46:03 Yeah.
46:03 I think that goes back to what you said earlier about using, knowing and using the right data
46:08 types for jobs, right?
46:10 Right.
46:10 And understanding a little bit about complexity and about how to do things as iterative chunks
46:18 of work rather than trying to read, you know, large files into memory and things like this.
46:23 That in part is where you do become so-called more professional in a language at least.
46:29 Yeah.
46:29 Is when you, I think, have a good handle on those data types and when they're useful or expensive.
46:35 You know, I think there's a lot of people who in the industry these days who are just kind
46:41 of, they don't know how to address this kind of broader conversation in hiring people or
46:47 assessing skills.
46:49 And they kind of fall back to just testing algorithms.
46:51 And, you know, a lot of the technical interviews seem to be like, well, implement quick sort
46:55 on a whiteboard.
46:57 Like, you know what?
46:57 Nobody needs to implement quick sort on a whiteboard.
47:00 But one thing I do think really is an important skill and it's an indicator of how effective you're
47:07 going to be is, do you know what a dictionary is?
47:10 Do you want a set is?
47:11 Do you want a list is?
47:12 Do you know when to use a dictionary over a set?
47:14 When should I use a list over a dictionary?
47:16 Like those types of things can just dramatically change the complexity of the code you got to
47:21 write, the speed that you write it in or speed that it runs in, maybe both.
47:25 But those seem like these data, these simple common data structures seem just so valuable.
47:31 Yeah.
47:31 And I think that's a thing you can translate to most any language, right?
47:36 You can carry that with you wherever you go.
47:38 So you just, most times if you have a background in one language with those in another language,
47:43 it's really just learning the syntax to produce those.
47:46 And so, I mean, sometimes they vary names, whether it's a hash or a map or a dictionary,
47:50 that kind of thing, but largely, largely the same.
47:54 So, yeah, I definitely remember an experience where we were storing some data in a list and
47:58 we had to look up some feature about it.
48:00 So we would iterate through the list and then go find it and then work with it.
48:04 And that was happening over and over and over.
48:07 And the code was supposed to be something kind of like real time.
48:10 And it was just too slow.
48:12 The other thing you talk about in there is profiling.
48:14 So I actually threw into a profiler and I found, oh my gosh, it's spending 80% of its time in
48:21 this list finding the item.
48:23 80%.
48:24 It was a huge complicated thing.
48:25 80% was one line trying to find the item in the list to go do something with it.
48:29 And I switched it to a dictionary and that 80% went to like 1% or something.
48:34 Because you know what?
48:35 Dictionaries are awesome looking up by feature and getting the real thing back.
48:39 Right?
48:39 And it's just like, ever since that, I'm just like, oh boy, I'm going to pay so much better
48:44 attention to data structures.
48:45 I think that was 20 years ago or something, but it really still stuck with me because, wow.
48:49 Yeah.
48:50 Yeah.
48:50 It's pretty interesting.
48:51 I emphasize the benefits, I guess, of profiling in the book, because I do think the situation
48:57 you describe is sometimes difficult to find, even if you're familiar with your code.
49:02 And certainly as someone maybe less familiar with data types and things like that, it would
49:07 sort of be a needle in the haystack kind of search.
49:10 And so profiling really gives you that.
49:12 People are really bad at estimating what, if you show them code, like where is this slow?
49:17 Even if you're pretty experienced, you could still get it quite wrong.
49:20 Yeah.
49:20 We build up these very subjective views of these things in our minds, I think.
49:25 And the profiler is a way to kind of strip away all of that.
49:29 And it's nice too, because it's much easier to take feedback from a computer, I find.
49:35 That's right.
49:36 It removes a lot of the guesswork.
49:38 And so I emphasize definitely shoot for, always profile and gather some kind of evidence for
49:45 your changes that are performance related, at least.
49:48 And as you're changing things and looking at the profiling data, shoot for things that are
49:52 like order of magnitude changes and not just beating it with a hammer bit by bit until
49:58 it gets roughly where you want, right?
50:00 Yeah.
50:02 Because there's certainly different things that you can do.
50:05 And knowing, I guess the first thing is, if you've got, let's say a thousand lines of code
50:10 and you look at it and you say, I think this five lines of this function is where it's slow.
50:15 If it turns out that that's only 2% of the time, it doesn't matter if you can make it infinitely
50:21 faster.
50:21 You're only going to make it 2% faster, right?
50:24 That's the upper bound.
50:26 If you can make it go to zero, that's as fast, that's as good as it gets.
50:29 So knowing where to put that energy, that's what I find the value of the profiling is like,
50:34 maybe not saying this algorithm is bad or whatever, but just like, we should focus here because
50:39 80% of the time is on this little bit.
50:42 And I don't know what the fix is.
50:44 Maybe it's to use a different library.
50:46 Maybe it's to rewrite this part in Cython.
50:48 Maybe it's to just use a different data structure, but this is where the energy and effort should
50:53 go because we can optimize the rest all day.
50:56 Maybe introduce a bunch of bugs and it won't get much faster.
50:58 Yeah.
50:59 It's often true too, that trying to eek performance out of something necessarily makes the code
51:06 more difficult, right?
51:07 Because you are either going closer to the middle or doing these kinds of
51:12 hoop jumping that doesn't really result in clear code.
51:15 Right.
51:16 You're like, well, this library that we could just call and get the answer to, well, now
51:20 we've got to rewrite it because we need to have more control over it.
51:22 But now, you know, all of a sudden, what's what one or two lines is now many, many lines
51:27 that is your baby to take care of.
51:29 Right.
51:29 Yeah.
51:29 Yeah.
51:30 Cool.
51:30 So definitely profiling.
51:31 I think it's really good.
51:32 Another thing to think about, I guess, is test data versus real data.
51:36 You know, how many times have there been website launches where, oh, it seemed like
51:42 it was fine, but now it's down because there are actually more than, you know, 10 rows in
51:47 the database and they forgot to put indexes.
51:49 I don't know how people can forget to put indexes because they're like magic to databases.
51:54 They're like speed magic.
51:55 You can just give them and it's like a thousand times faster.
51:58 Like, so as soon as you see it slow, put an index on it.
52:01 Now, now let's go worry about what's next.
52:03 Right.
52:03 But still, this happens all the time.
52:05 Yeah.
52:06 And I don't know.
52:07 Indexes are sort of notoriously, notoriously hard, even when you are thinking about them.
52:12 You can have situations where you're like, you know, it did this index didn't cover it
52:15 right because we were sorting this way.
52:17 It actually used a different index.
52:19 We had to give it a hint.
52:19 I'm not talking about that.
52:21 I'm talking about we've got a million rows that we're doing a query with no index, which
52:25 I think happens more than people want to admit, you know?
52:27 Yeah, definitely.
52:27 And I guess to the point about real data also sort of at the, at the product and user
52:34 and social level, there's a bunch of videos where people of color can't use a product because
52:39 they didn't test it on people with that.
52:42 Like those automatic hand dispenser, sand soap dispenser things and all sorts of stuff.
52:46 Yeah, exactly.
52:46 It's fresh.
52:47 It's got to be super frustrating.
52:48 Those things already frustrate me.
52:50 And yeah.
52:51 So I mean, I'm not only, yeah.
52:53 Yeah.
52:53 It not only like manifests as a cost to you as the business, but absolutely like impacts
52:59 people in the real world in very negative ways.
53:01 Yeah.
53:02 Well, I can speak to a hardware testing of that sort.
53:05 But one thing that I've, I found Bob Belderbos introduced me to this is this place called
53:10 Mockaroo.
53:11 You know Mockaroo?
53:13 I'm not familiar.
53:13 Mockaroo.
53:14 Make sure I spell it right for people.
53:16 Yeah.
53:17 Mock A-R-O-O.
53:19 Yeah.
53:19 And what you can do is they have all these different types of data and you can say,
53:23 go generate me certain amounts.
53:25 So you can ask for like first name, last name, email, but you can also thing, ask for all different
53:30 kinds of stuff.
53:31 There's so many.
53:32 So I could ask for like cars, VIN numbers.
53:35 I could ask for, let's see, for commerce.
53:39 I could ask for currency codes or stock names or, you know, and it'll generate as many of
53:45 those as you want.
53:46 I want a million of those in a JSON format like this.
53:49 Give it to me.
53:49 And it looks real and it feels real.
53:51 So, you know, you can definitely generate some quick data with Mockaroo and it's, it's pretty
53:56 awesome.
53:56 Yeah.
53:56 That is really cool.
53:57 There's a couple of plugins I know for Django that do something similar meant exactly for
54:03 that, that same reason.
54:04 Like testing, testing often is people espouse like testing in isolation and testing just
54:10 the, just this part or that part.
54:11 And there's like unit testing and integration testing and all sorts of varieties of that stuff.
54:17 But I don't think this point is talked about enough, which is like test it in isolation,
54:23 but also use values that make sense.
54:26 Right.
54:27 Maybe don't just test with a first name and a last name.
54:29 Some names, it's like a three part name or something like that, you know?
54:34 And if you're splitting on the space, that's probably not going to come out.
54:38 Okay.
54:38 Right.
54:38 Yep.
54:39 So yeah, yeah, definitely testing.
54:40 Yeah.
54:41 Or if you, you're always using like foo, the string as your test variable, like maybe you
54:46 won't uncover your bugs that are throughout your code when there's Unicode characters and
54:51 things like that.
54:52 Yeah.
54:52 True.
54:52 Test with emojis, test with emojis as well, I guess.
54:55 That's pretty awesome.
54:59 So we're kind of getting near the end of our time to talk about all this.
55:03 Let me throw out a quick rule of thumb that I use a lot of times around this.
55:07 And I just want to hear your thoughts on it.
55:09 You can disagree with it if you want.
55:10 But one of the rules of thumbs that I have is that if a function or a class or some part of
55:16 code needs a comment, it's probably broken.
55:19 Yeah.
55:19 That's interesting.
55:20 I'm not talking to doc string, right?
55:21 Like if you want to document it, that's a different thing.
55:23 But if you're like, well, we got to do this weird thing because, or the function is named
55:28 so badly, there's a comment saying what the function name should have really just been,
55:32 you know, something like that.
55:33 Yep.
55:33 So I would agree that like comments that are redundant to what the code around it could
55:40 have been or could have said are definitely an anti-pattern.
55:44 I don't mind when a comment says the why of the code.
55:51 So like code often expresses simply the how or the what.
55:55 But if you can't get all of that just by your variable names or your class names and stuff
56:01 like comments, giving you that extra context are okay.
56:04 Yeah.
56:05 Maybe you're calling some API and it returns a dictionary and you're like, this looks weird,
56:09 but we have to do this thing because sometimes it does this and sometimes we don't control it.
56:14 We got to live with it, but just please leave this check here because I know it looks like it's
56:18 not necessary, but on the first month, first day of the month, it always goes weird.
56:22 So we're going to just need it.
56:23 Right.
56:24 That's cool.
56:25 Postal's law is like this idea that you should be liberal in what you accept and conservative
56:29 in what you send.
56:30 Right.
56:31 Right.
56:31 Sometimes being liberal in what you accept means exactly that.
56:36 We have this API we have no control over, but we want to use it because it provides us other
56:40 value.
56:42 And so we're going to accept this reality and we wrote this code to deal with it.
56:46 And here's the situation.
56:48 Yeah.
56:48 Yeah.
56:48 Pretty cool.
56:49 And those comments make perfect sense, right?
56:51 Like, please don't take this out.
56:52 It's I'm going to hide that away in a function.
56:54 So hopefully you don't look at it, but if you got to get in there, here's what we're doing.
56:58 So look out.
56:59 Yeah.
56:59 Yeah.
56:59 Pretty cool.
57:00 All right.
57:00 Well, it's been really fun to think about these design concepts.
57:04 And I think a lot of what we talked about is actually pretty broadly applicable, although
57:08 we did talk about some stuff that very much is Pythonic, right?
57:11 We talked about comprehensions and whatnot a little bit.
57:14 Yeah.
57:14 So it's great.
57:15 Yeah.
57:16 I think this major chunk of the book, I would say, is very translatable to other languages.
57:21 I use Python as sort of the vehicle of teaching the concepts, if you will.
57:26 Yeah.
57:27 Yeah.
57:27 Sure.
57:27 Not that people shouldn't use Python.
57:29 It's an awesome language, right?
57:30 But, you know, even I think it's just, these are just, I guess the point I'm trying to
57:35 make is these ideas transcend individual languages.
57:38 Knowing this kind of stuff will help you across if now you're a Python developer and later you're
57:45 going to go learn C#, or you're going to learn Java, or you're going to do C++, knowing
57:49 a lot of the stuff is really applicable.
57:51 Yeah, absolutely.
57:52 And I think that's also maybe one of the pro things, right?
57:55 When you're a beginner, you're very much focused on the syntax of the language and just making
57:59 the tools do their thing and not worried about these higher order general ways of thinking.
58:05 So that's probably another little indicator there, like we started at the beginning.
58:08 Yeah.
58:09 And I think sort of parallel to design patterns, there's just these patterns of languages and
58:14 patterns of ways of thinking about systems that only come sort of with time.
58:18 And as you learn the names for those concepts, you can, again, have those conversations.
58:23 Yeah, absolutely.
58:24 All right.
58:25 Cool.
58:25 Well, let me ask you the two last questions before you get out of here.
58:29 If you're going to write some Python code, what editor do you use?
58:32 So I would say I split my time fairly evenly between PyCharm and VI.
58:37 Okay.
58:37 I have, I guess when I'm working in a larger context and I need to be flipping between a lot
58:42 of files and jumping through code a little more often, or if I want to attach a debugger,
58:47 I usually will use PyCharm.
58:49 And then if I'm just doing some quick things, I prefer VI because I like moving around the text
58:55 that way.
58:55 There are like VI bindings for PyCharm and I might be happy with that.
58:59 Can you bring those worlds together?
59:00 Yeah.
59:01 I don't know.
59:02 I don't know.
59:03 You know, speaking of debuggers, I suspect, I don't know if you cover it anywhere in the
59:07 book.
59:07 I don't recall.
59:07 But using like consider a debugger is probably not a bad piece of recommendation.
59:12 Yeah.
59:13 That's a really great advice.
59:14 If you can sort of understand what it's doing and why you would want to step through your
59:20 individual lines of code.
59:22 Like it's a great way to solve problems.
59:25 There is a tendency for some people to use it as a crutch almost to just not really think
59:30 it through.
59:31 It is just like, well, the debugger will show me.
59:33 But I think there's a bigger tendency of people to put 20 print statements, not because it really
59:38 has had some timing issue and you really have to have the print statements.
59:41 It's just like, well, I'm not in a thing that debugs easily or I didn't set it up right
59:45 in VS Code.
59:46 So there's no button to press, even though like you could just go and set up a run configuration,
59:50 whatever they call it.
59:51 So yeah, I think definitely that's probably something in there as well.
59:54 Yeah.
59:54 And it can help you like kind of bisect where the problem is.
59:58 Like if you, if you have a suspicion about where the problem is, but don't know exactly
01:00:01 what's going wrong, picking a couple of like starting points and then slowly moving your
01:00:07 break points until you really find like the line that was mucking things up.
01:00:12 It is a really nice way to solve problems.
01:00:14 It comes back to just having less in your head at once.
01:00:16 You're like, well, it's down to these three lines.
01:00:17 What do I have to think about for these three lines, right?
01:00:19 Yeah, sure.
01:00:20 Cool.
01:00:21 And then notable PyPI package, not necessarily something super popular, but something you
01:00:26 came across.
01:00:26 You're like, oh, this thing was so sweet.
01:00:27 I can't believe I found it.
01:00:28 We've developed a couple of services in FastAPI recently.
01:00:32 Okay.
01:00:32 Yeah.
01:00:32 I've heard tons of good things about FastAPI.
01:00:34 It seems like there's a lot of momentum around it.
01:00:36 It's really kind of a joy to work with.
01:00:39 There's sort of some underlying packages that it uses, Starlette and things like that.
01:00:44 But there's also some that it works in tandem with.
01:00:47 Pedantic is one.
01:00:48 Pedantic is really cool.
01:00:50 Yeah.
01:00:50 Yeah.
01:00:51 It's like a data modeling related library and just love them all.
01:00:54 Yeah.
01:00:54 You can put a little pedantic constraints on it.
01:00:57 Decorators and then decorators, I think.
01:00:59 Right.
01:00:59 And then it's a little bit like data classes with validators, if you will.
01:01:03 Yes, exactly.
01:01:03 It's, it's looks super cool.
01:01:05 I haven't had a chance to use it, but I just really kind of got the zen of it a while ago
01:01:08 and I want to know.
01:01:09 Yeah.
01:01:09 Especially if you're working in microservices and things like that.
01:01:12 Asynchronous microservices kind of where it shines.
01:01:16 Yeah.
01:01:16 That's super cool.
01:01:17 Awesome.
01:01:17 Well, those are some good recommendations as well.
01:01:19 All right.
01:01:19 So final call to action.
01:01:20 People are inspired about this design stuff.
01:01:22 What's your advice to them?
01:01:24 What do they do?
01:01:24 I mean, there's a couple of other sort of seminal works that I would say to read.
01:01:29 There's like the Gang of Four book and other things like that.
01:01:32 If you're interested in Python, I would certainly hope you'll take a look at this book.
01:01:36 Yeah.
01:01:36 And I'll put your book, I'll link to your book in the show notes so people can get to it easily.
01:01:40 Yeah.
01:01:40 Fantastic.
01:01:41 And I think once you get beyond that or maybe even in parallel, this idea about testing and
01:01:47 refactoring are both super valuable, super valuable bits.
01:01:51 So Martin Fowler has a good book about refactoring.
01:01:54 It's got a pretty straightforward title, Refactoring from 1999.
01:01:58 It's still totally valid though.
01:01:59 It's a great book.
01:02:00 Yeah.
01:02:00 And there's Working Effectively with Legacy Code.
01:02:03 I think Michael Feathers maybe is the name of the author.
01:02:06 Yeah.
01:02:07 Michael Feathers wrote that.
01:02:07 That's a really good book.
01:02:08 If you've got a large code base and you're like, you know, the problem I described where
01:02:13 there's a thousand issues and we're just like, well, we're not fixing these.
01:02:16 Like if you're in that situation, but you want to carve out a part that behaves better,
01:02:20 his book is beautiful for that.
01:02:21 Yep.
01:02:22 So those are even the ones I'm reading to some degree now.
01:02:24 So super.
01:02:25 All right.
01:02:25 Well, Dane, thanks for being on the show.
01:02:27 It was really great to have this chat with you.
01:02:28 Yeah.
01:02:29 I really appreciate it.
01:02:29 It was a lot of fun.
01:02:30 Yep.
01:02:30 You bet.
01:02:31 Bye.
01:02:31 Yeah.
01:02:31 Take care.
01:02:32 This has been another episode of Talk Python to Me.
01:02:35 Our guest on this episode is Dane Hillard, and it's been brought to you by Tidelift and Linode.
01:02:40 If you run an open source project, Tidelift wants to help you get paid for keeping it going
01:02:45 strong.
01:02:45 Just visit talkpython.fm/Tidelift.
01:02:49 Search for your package and get started today.
01:02:51 Start your next Python project on Linode's state-of-the-art cloud service.
01:02:56 Just visit talkpython.fm/Linode, L-I-N-O-D-E.
01:03:00 You'll automatically get a $20 credit when you create a new account.
01:03:03 Want to level up your Python?
01:03:05 If you're just getting started, try my Python Jumpstart by Building 10 Apps course.
01:03:10 Or if you're looking for something more advanced, check out our new Async course that digs into
01:03:15 all the different types of Async programming you can do in Python.
01:03:18 And of course, if you're interested in more than one of these, be sure to check out our
01:03:22 Everything Bundle.
01:03:23 It's like a subscription that never expires.
01:03:25 Be sure to subscribe to the show.
01:03:27 Open your favorite podcatcher and search for Python.
01:03:30 We should be right at the top.
01:03:31 You can also find the iTunes feed at /itunes, the Google Play feed at /play,
01:03:36 and the direct RSS feed at /rss on talkpython.fm.
01:03:40 This is your host, Michael Kennedy.
01:03:42 Thanks so much for listening.
01:03:43 I really appreciate it.
01:03:44 Now get out there and write some Python code.
01:03:46 I really appreciate it.