Monitor performance issues & errors in your code
#411: Things I Wish Someone Had Explained To Me Sooner About Python Transcript
00:00 What advice would you give someone just getting into Python?
00:03 What did you learn over time through hard work and a few tiers that would have really helped you?
00:07 It's a fun game to play and we have Jason McDonald on the podcast to give us his take.
00:12 This is Talk Python to Me, episode 411, recorded March 2nd, 2023.
00:18 (upbeat music)
00:21 - Welcome to Talk Python to Me, a weekly podcast on Python.
00:34 This is your host, Michael Kennedy.
00:36 Follow me on Mastodon, where I'm @mkennedy and follow the podcast using @talkpython, both on fosstodon.org.
00:43 Be careful with impersonating accounts on other instances, there are many.
00:47 Keep up with the show and listen to over seven years of past episodes at talkpython.fm.
00:53 We've started streaming most of our episodes live on YouTube.
00:56 Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.
01:04 This episode is brought to you by Sentry and us over at Talk Python Training.
01:09 Please check out what we're both offering during our segments.
01:11 It really helps support the show.
01:14 Jason, welcome to Talk Python to me.
01:16 Thank you for having me.
01:17 Yeah, it's great to have you here on the show.
01:19 I'm looking forward to talking about a bunch of things I wish people taught you about Python sooner.
01:26 I think there are many, many things that fall into that category.
01:29 And part of that, I guess it's true with all languages, but I think it's maybe a little extra true in Python because getting started is so easy, but getting really good at it takes a lifetime sort of thing, you know?
01:41 And so there's like this big mismatch of how accessible it feels, But then what you can get out of it if you go further.
01:47 Right.
01:47 So I think that we'll touch on that going further apart, maybe.
01:49 Absolutely.
01:50 Yeah, for sure.
01:51 Now, before we get into all those things, let's start with your story.
01:55 How'd you get into programming and Python?
01:56 I, well, I got into programming by a route that I don't recommend to anybody.
02:00 I fell down a staircase, hit my head on the banister and ended all hopes of ever becoming a doctor.
02:05 Oh, so went from 4.0 straight A student, college level reading in high school to failing pre-game material, two extra years to climb back up where I was academically a bit longer after that for other things.
02:19 But somewhere along the way, I discovered that I had a natural inclination towards an enact for coding and got into that because I wanted to make games.
02:28 I wanted to make educational games.
02:29 They had been part of my recovery.
02:30 So I'm like, I want to make games.
02:31 So I learned how to code.
02:33 And so I taught myself VB.net and then felt very constrained by the .net ecosystem and broke out into Python.
02:42 found my home, stayed put until I wound up going to a bunch of other languages.
02:47 I used Flash for quite some time.
02:49 I've used C++.
02:50 I muck about in whatever handy, but Python is definitely near to dear to my heart.
02:55 Yeah, absolutely.
02:55 Well, if you started out in VB.net, I think it's a little less true these days, but in the earlier days, that was a really different ecosystem than Python, right?
03:05 There was like, well, what is the framework for this thing that Microsoft recommends?
03:09 Like, how do I do the web?
03:10 I do ASP.net because that's what they provide me.
03:12 How do I do database?
03:13 I do this because that's what they provide me.
03:15 And you come into Python, you're like, wait, there's a thousand different ways to do this.
03:18 And I have to pick and decide and I can mix and match.
03:21 And it's both amazing and like terrifying.
03:23 Kind of like, yeah, it's terrifying.
03:24 Like, what do I make the wrong choice?
03:26 I've, how do I decide?
03:28 I have no basis, right?
03:29 I'm both new.
03:30 So I have no basis for deciding, but I also, I've got to decide to get started.
03:33 So where do we go?
03:34 Right.
03:34 Like it's, it's a different world.
03:36 It is where I first started running into trouble with the paradox that I wound up addressing in my book.
03:42 I'm back to that, but the paradox that a lot of people find themselves in, it's like, I know all this stuff, but I don't know enough to do this thing.
03:50 And so you wind up with this weird little gap in a lot of training material because you have like the absolute basic elementary and the examples there are almost insultingly simplistic.
03:58 It's like, oh, you just have to type this stuff into, into, into this file and you run the file and look at that.
04:04 You have a hello world.
04:05 Aren't you smart?
04:06 And then you go all the way to the ever extreme and it's something like super complicated and I'll understand any of it.
04:10 And nobody sat and wrote down, here's how you structure a multi-Python project in a way that's going to, at least when I started, that wasn't documented anywhere, like no one had ever written it down.
04:20 So it was just like, what do I do now?
04:22 Yeah, it's really tricky.
04:23 A lot of the tutorials and stuff are like, you just put all the stuff into one file.
04:27 And then it's magic and, but you should never ever do that.
04:30 Well, maybe in simple cases, but yeah, super realistic apps, like no realistic app belongs in one file, like just like you wouldn't have one function or just like no functions, right?
04:39 You're like, not really.
04:40 It's not how you do it.
04:41 Yeah.
04:42 But so there is this, even when you're sort of hunting through tutorials, there's this mismatch of the simple way to show it the simplest way to show it and then like maybe how you should.
04:52 Yeah.
04:52 Oh yeah, exactly.
04:54 It's it's the example of given is that there's two types of books/articles/tutorial out there.
04:59 There's here's basics of variables, boys and girls.
05:02 And then the next level after that is here's how to program a natural language recognition, OCR, AI, machine learning on a Raspberry Pi at your grandmother's base or running on a potato battery and there's nothing in between.
05:13 Yeah.
05:14 Yeah, that's true.
05:16 It does make it quite challenging.
05:18 All right.
05:18 How about now?
05:19 What are you up to?
05:19 Oh, these days.
05:20 So gosh, I'm actually kind of veering these days more into, actually more into kind of the, facilitation business analyst side of things more.
05:28 Interestingly, the last, last couple of years, that's become a point particular interest, but you know, in terms of coding, I'm working on, I'm working on some interesting projects that, you know, at work building, you know, an API for an internal project.
05:39 And that's been, that's been fun getting to get my hands dirty with fast API and SQLAlchemy and that's been, that's been fairly enjoyable.
05:47 So that's been the majority of my programming bandwidth lately, but it's fun.
05:51 It's, it's the sort of project you hope you're going to get at work because it's an interesting topic.
05:56 You have enough love, leeway.
05:59 That's where we're looking for enough leeway to do things right.
06:02 And do things to some degree your way, if you can justify it well enough.
06:05 And, you know, other tech lead on it.
06:07 So I'm able to make sure like, you know, we're making good decisions early and paying down technical debt and building something that's really, really cool.
06:14 So it's, it's a fun project.
06:15 I'm enjoying it.
06:16 Yeah.
06:16 Those kinds of projects, they're, they're fun, but they're also important, right?
06:20 Like if your work, if you work, if you work at a place where they say, you know, you need to use 10 year old technology and there's no opportunity to try new things like fast API and other others.
06:31 What you end up with is a whole bunch of people who that work there who don't want to learn any new technologies and don't are not deeply passionate about programming. And it's kind of the self-fulfilling prophecy.
06:42 And I think it's cool that the place you're working at allows that, right?
06:45 Because I think it's, it's one of the signs of a healthy tech, not necessarily a startup or whatever, but all these companies have like small software teams within it.
06:54 It's a sign of a healthy, small software team, even if it's just building internal software.
06:58 Well, definitely.
06:59 Yeah.
06:59 And you know, and there's a place for older technology as well.
07:01 It's good to be well-versed in both and be able to move between it.
07:04 But I mean, yeah, if you're, if you're just stuck in the past, then you either play it, like you said, get a lot of people that are afraid of change or are really wanting that change and they're dissatisfied with the fact that they can't get it.
07:15 But on the other hand, if you just, if you're only new technology, you'll just spontaneously combust because you can't, you know, there's just, there's so much constant invention and reinvention and research and unexpected bugs and whatever, it's like, sometimes the nice thing about a 10 year old library is that there's less bugs because it's eight years old.
07:33 >> Or you can find documentation about issues or whatever.
07:37 You know, they have a term.
07:38 It's like a term in the software industry about this, this constant churn that's like going too quick and especially not appreciating stuff that's been around, it's called JavaScript.
07:47 >> You're not wrong.
07:51 JavaScript is the embodiment of shiny object syndrome.
07:53 >> It is.
07:54 I'm not bashing on JavaScript per se, but the life cycle of some of these frameworks is just so brief, you know, it's nuts.
08:01 >> Everybody wanted to use Deno for about 10 minutes.
08:04 I remember that.
08:05 There was, there were a thousand articles a minute on Deno and now it's, now we've moved on to other things and it's, well, I mean, it's funny because you get a new, new technology, like, okay, the, the, the topic of the minute, ChatGPT and everybody on LinkedIn is an expert in ChatGPT, you know, have you noticed that?
08:20 Every other post is about ChatGPT.
08:22 It's like, well, it's interesting, but it's just a bit early.
08:26 And the early adopters have a lot of egg they're still trying to wash off their face in the last six fads.
08:30 So it's like, let's see what happens.
08:33 But chances are, this is just an early hiccup.
08:36 Yeah, for sure.
08:37 So you mentioned FastAPI and SQLAlchemy and Yasser out there says SQLAlchemy is the best library.
08:42 So SQLAlchemy is pretty great.
08:44 And it's, you know, it's cool to see my Bayer and crew updated to 2.0 and add async and await to it and stuff.
08:50 But there's also, especially when you're in Pydantic world, say with fast API, you know, there's a SQL model for also from Sebastian Ramirez that says, Hey, we are already doing Pydantic for the API or front end exchange.
09:05 Like, what about, what about making that your database?
09:08 Did you all evaluate or consider a SQL model?
09:10 I did.
09:11 It's one of those things like this would be great.
09:13 And I think we can move towards that later.
09:15 But sometimes when you're paying down technical debt, you have to make a distinction between do I want to make, you know, the monthly payment as it were more than a month, a monthly payment, or do I want to try and pay it all off in one thing?
09:27 And sometimes when you're having to budget that time, you know, for us, it made the most sense to migrate from the 1.0 that we were working with earlier on to using the, you know, SQLAlchemy 2.
09:38 And so there was a relatively smaller set of changes that we needed to make.
09:41 Moving to something like a SQL model would have, would be great, but it would also require us to rewrite some major things.
09:49 And so that's one of those things is like, well, this might be a good option.
09:53 I mean, it looks really cool, but let's defer that for right now so that we can actually get some stuff done.
09:58 It's a constant balancing act.
09:59 Like do we pay down, pay down technical debt or build new features?
10:02 And the answer is yes.
10:03 You have to do both, but you have to keep it in.
10:05 You have to keep it in balance, pay down the credit card, but also make sure you're buying your groceries, you know?
10:09 Yeah, absolutely.
10:10 I feel like if you wanted to go that way, the change is pretty straightforward, but if you already have it all set up in the other way, again, it's like, eh, What is the value?
10:18 Just one thing that kind of timely as of a couple days ago, let's see.
10:23 As of two days ago, the Pydantic folks, Samuel Colvin and Terrence Dorsey posted, Hey, we're excited to announce the first alpha of Pydantic V2.
10:33 And I had Samuel on like a year ago to talk about what they're doing there.
10:36 But the big news is the headline news in here is they rewrote much of the core of Pydantic in Rust and now Pydantic version two is five to 50 times faster.
10:48 Nice.
10:49 That's cool.
10:50 Right?
10:50 That's gorgeous.
10:51 Thank you.
10:52 This is just put more work on my plate.
10:54 That's okay.
10:54 No, this will be good.
10:57 I'll have to bring this up at work.
10:58 It's like, Hey, we should look at this later.
11:00 Yeah.
11:00 I mean, I don't think that there's any, I don't think anything will, obviously nothing will change for your SQL alchemy side, but even for fast API, I'm sure that that's, this is going to get rolled in in a smooth way, but it does mean that all that data exchange should just go way, way faster, which for basically no effort on users of things like FastAPI, SQL model, and Beanie, and all those.
11:21 So that's cool news.
11:22 - Yeah, definitely.
11:23 That is neat.
11:24 - Other than the fact that if you do a lot of work in directly in Pydantic models, there might be a few small breaking changes.
11:30 But let's talk about our things, the things we wish someone had explained to us sooner.
11:36 - Yes.
11:37 - And I guess we can start with this.
11:38 Like these things were motivated.
11:40 We'll come back and talk a bit more about this at the end, but they're motivated.
11:43 This whole conversation was motivated.
11:44 This big, long book that you wrote here, dead simple Python, give people the elevator pitch and then we'll get to it.
11:50 Exactly.
11:50 So dead simple Python is the book I wish I had.
11:53 And, yeah, it's, I get teased a lot.
11:55 It's like dead simple and it's that thick, but it's dead simple in the same way that Python is obvious.
12:01 It's obvious retrospective.
12:03 Things are dead simple.
12:04 Once you understand it, you look back and go, Oh, okay.
12:07 That clicks.
12:07 That just makes sense.
12:08 A lot of things in Python are like that.
12:10 You look into it, it's like, that looks really complicated.
12:12 What, you know, how is this Async supposed to work?
12:14 And then you look back and go, oh, that's all.
12:16 Okay.
12:17 No, that's.
12:17 Yeah, exactly.
12:18 I was, why was I afraid?
12:19 Why did I put learning, put off learning this for six months?
12:21 Exactly.
12:22 It's, you know, things look more intimidating than they, than they are in practice.
12:28 But basically this is the book for those persons who already know another programming language, or maybe they've even been working with Python for a while.
12:37 Like they're familiar with the basics.
12:39 They do not want to have their hand held through the, you know, this is what a variable is and this is what a function is.
12:45 I got the, they've been through this probably more times than they want to admit to, and they want to get into what's different about Python and how to use Python as Python.
12:56 I think that's kind of the core of the book is there's a big difference between Python code that works and Python code that makes the most of the language and its patterns, what we would refer to as Pythonic code.
13:08 And there's a big difference between the two, because it's very tempting to come in and write Python as BB dot that in my case, because that's where I started or write Python as C or write Python as Java, write Python as Ruby.
13:19 And we wind up holding ourselves back and having a really convoluted and hard to maintain code because we don't understand the patterns.
13:25 So the approach I took with the book is let me explain why we do things this way.
13:30 Let's go all the way down into the internals and how this actually works under the hood.
13:37 So by the time you get to the for loop, the web statements, the meta class or whatever, you're like, "Oh, I understand what this is, why it's there, how it's used, and when not to use it." Yeah, that's a great service.
13:48 So you're saying if I create a class called like, user factory and it implemented I user factory, and then I would use dependency injection, I might be doing that wrong?
13:59 >> You might be writing C# and Python.
14:01 >> Yeah, indeed.
14:04 This portion of Talk Python to Me is brought to you by Sentry.
14:09 Is your Python application fast?
14:12 Or does it sometimes suffer from slowdowns and unexpected latency?
14:16 Does this usually only happen in production?
14:18 It's really tough to track down the problems at that point, isn't it?
14:22 If you've looked at APM, application performance monitoring, products before, they may have felt out of place for software teams.
14:28 Many of them are more focused on legacy problems made for ops and infrastructure teams to keep their infrastructure and services up and running.
14:36 Sentry has just launched their new APM service.
14:40 And Sentry's approach to application monitoring is focused on being actionable, affordable, and actually built for developers.
14:47 Whether it's a slow running query or latent payment endpoint that's at risk of timing out and causing sales to tank, Sentry removes the complexity and does the analysis for you, surfacing the most critical performance issues so you can address them immediately.
15:01 Most legacy APM tools focus on an ingest everything approach, resulting in high storage costs, noisy environments, and an enormous amount of telemetry data most developers will never need to analyze.
15:14 Sentry has taken a different approach, building the most affordable APM solution in the market.
15:19 They've removed the noise and extract the maximum value out of your performance data while passing the savings directly onto you, especially for Talk Python listeners who use the code TalkBython.
15:30 So get started at talkpython.fm/sentry and be sure to use their code, talkpython, all lowercase, so you let them know that you heard about them from us.
15:40 My thanks to Sentry for keeping this podcast going strong.
15:44 So I like the idea of it.
15:48 Let's talk, let's go through the topics.
15:51 We both contributed a little bit, mostly.
15:53 I'll throw in a few if we got extra time.
15:54 - Yeah, yeah.
15:55 - The first one has to do with this, I guess, like really early stage writing code, like how to create variables, what are the deals with types and variables, but also types and objects.
16:05 Tell us about that.
16:06 - Yeah, exactly.
16:07 How names and values actually work.
16:08 And this is something that I had to learn early, but I didn't fully understand.
16:13 So when I first logged into the Python IRC room back when I learned this language, you know, back in the Cretaceous period, I logged in and I, you know, I'm coming from a strongly type background, so I'm saying, okay, so how do I declare a data type on a variable because I'm used to VB.
16:28 I was like, yeah, DM and the name of the variable, type equals whatever.
16:31 I don't want to call it type.
16:33 It's before type hints existed.
16:34 And one of the guys in the room goes, you're a data type.
16:37 And I'm like, sorry, what?
16:38 And he goes on to explain, well, it doesn't quite work this way in Python, but I didn't realize just how different it was.
16:44 So because I just sort of had this vague, like, oh, you just have the name equals the value.
16:48 Cool. Type still seems kind of important.
16:50 Maybe I should start doing Hungarian notation.
16:52 Don't do that.
16:53 - SZ user name, no.
16:57 - Exactly, exactly, int age or whatever.
17:01 - Yeah, yeah.
17:01 - And I thought I was being super clever and I wasn't, but eventually someone explained to me, I think it was actually Ned Batchelder, someone gave me the link to Ned Batchelder's talk on this, is that Python's variables work very differently than other languages, so much so that some smart alex like to say, well, Python doesn't really have variables.
17:18 Yes, it does, but they're variables in a different sense.
17:21 You have names which are just like, they're like labels and you can call it whatever you want but a name has a scope.
17:27 You know, so the name exists in the context of your whole program or in the context of your function or whatever.
17:32 And a name can be bound to whatever you want.
17:34 Imagine drawing, you know, basically having this label on a corkboard, think like a conspiracy theorist corkboard.
17:41 You know, you have the tack and the label and you put a little string around it and you tie that string to this other thing over here and that's your value.
17:49 value has a type. It has, it is one thing. It's a Boolean or it is an integer or it is a string or it is a list.
17:56 >> We've all seen the error, you cannot combine integer and string with plus operation. Like they know what they are, right?
18:02 >> Exactly. Those values know what they are and that's something that throws people off coming from JavaScript where everything's weakly typed. It'll just try to coerce them together. Python says, "Hey, these two values are different. I'm not going to try and figure this out." That's because it's working with the values. The values have type, but the the values don't have scope. And this is where people get thrown off. Because if you say, you know, let's say, I'm gonna use example here, x equals spam. If you were to say z equals x, you're binding z to the same value as x. And then if you decide to change z, that works x is unchanged, z is changed, you might think, Oh, cool. Okay, this is just it just made a copy. Not exactly. Because we have this whole thing about mutable and immutable types.
18:44 Some things can't be changed.
18:46 Boolean and integer string, a tuple, they're not changed.
18:49 The value is never modified in memory.
18:51 A new thing is made instead.
18:54 But then some, and by some I mean most, sorry about my clock, most types in Python like lists and dictionaries and the objects that you're creating with your fancy little classes are immutable.
19:05 So if you create a list or a dictionary and you bind that to two different names.
19:11 You modify, you add a value to that list.
19:14 It doesn't matter which of the names you use, you're modifying the value in memory, in place.
19:19 And now that is accessible from either of those names.
19:22 And that can be confusing, especially when you start to understand that functions don't pass by value, they don't pass by reference.
19:28 They just bind the value to the name of that argument.
19:32 Pass by assignment is what we usually call it.
19:34 So you're just having one more handle, one more alias pointing to that value in memory.
19:38 So if you change on that argument, you're just changing your spot in memory.
19:43 - Many languages have values that are like local and copied types of values.
19:49 So you mentioned VB.NET, which like the integer and float values, numbers and things like that.
19:56 If you assign one, it's a copy of a thing.
19:58 Whereas Python, literally everything is a pointer, right?
20:02 This corkboard conspiracy theory, thing, right?
20:07 There's, there's no concept of, Oh, I created this on the stack.
20:09 And when the function returns, it's gone.
20:11 It's everything is a, you know, even like a, just an integer is a PI long star down in the runtime.
20:18 Right.
20:18 That's a bit of a mental thing.
20:20 People got to get used to just like how much, because when you come from a language like C++ or something, pointers and these reference type type of ideas are like right in your face.
20:30 Right?
20:31 Like you see this, like I just said, PI long star, like the star means it points at things without the star it's on the stack, right?
20:38 That it's, you are way down in those details.
20:41 Whereas in Python, you never see pointers.
20:43 Like you could imagine, like, there's no such, like, we don't even have pointers in Python.
20:46 Like, but the irony is like, literally everything is basically managed in that on the heap reference type style of work.
20:54 So that's certainly something I wish I knew.
20:56 Yeah.
20:57 And, and of course then people come to C++ to see like, oh, cool.
21:00 That's easy.
21:01 So it's all pointers.
21:02 Well, once again, Python being Python, not quite because it's actually technically called a reference, not a pointer.
21:08 And that's important because a Python reference or Python value has two things.
21:15 It has, yes, it does have a pointer, at least we're talking about CPython, but it also has a reference counter, which is itself an integer and that's adding overhead.
21:22 So if you ever do actually go down and look at the sizes of things, you will notice that your base value has this additional overhead.
21:31 Yeah.
21:32 of two numbers, one of them is your pointer and the other one is your reference counter because it has a reference count in Garbage Collection.
21:40 - Yeah, often a lot more.
21:41 I think, trying to, I guess I could check, but either an integer or a single character, I can't remember which, is 28 bytes, right?
21:51 And that's in C, the single character would be one or two bytes, and the integer would be two, four, or eight, right?
21:58 And so there's just a big difference in the way you got to sort of, like the implications of that.
22:02 So related to that, kind of moving on to the next, oh no, before we move on, the other thing that you mentioned, and I think is also a really big deal in this area, is the scope of a variable.
22:13 And so in some languages, you have very fine-grained control of the lifetime of a variable by where you declare it.
22:21 So for example, if you declare a variable in C++ or VB.NET or whatever, in a if block, when you leave the if block, the thing goes away.
22:31 Like if there's curly braces around it in a lot of languages, like that is the lifetime of the variable innermost bracket of curly braces, like outside of a loop.
22:41 The thing goes away in Python.
22:43 We don't have that.
22:43 And that's, you know, kind of the variable persists and lives after afterwards, right.
22:49 Which is a interesting difference.
22:51 Yeah.
22:51 Because the garbage collection is something you're never supposed to have to deal with.
22:55 So even if you called delete on something, it's not necessarily guaranteed to be cleaned up when you expect it.
23:01 Like the language is going to handle things when it, when it sees fit for whatever reason.
23:06 So yeah, there's only to my knowledge, there's only three, there's only actually three scopes in Python.
23:12 There's global level, there's module level, and there's I need to back.
23:18 It's a little pydantic.
23:20 And then you have function level classes are not their own scope.
23:23 Classes actually do not a class exists in the global level.
23:28 And whenever you have an attribute sitting on a class or sitting on an instance is actually in a dictionary, unless using slots, I'm not going into that, actually sitting in a dictionary that is a part of that object in memory.
23:41 Yep.
23:41 It's special, but it works out.
23:43 It's just something you got to kind of get used to.
23:44 Definitely.
23:45 All right.
23:45 Next one.
23:46 Ducks.
23:47 Wank duck typing.
23:49 Yeah.
23:49 And, and it's, this is where I went.
23:52 Oh, Heywire.
23:53 When I, when I was doing that, that Hungarian notation, I mentioned I'm being so clever, I'm just putting the name of the type in the name, and then I know it.
24:02 But unfortunately, you're coming from a strongly typed language, you're actually, it's easy to underestimate just how much strong typing can actually hold us back in some cases, because we start thinking about what a thing is rather than what it can do.
24:15 And we see this in Java, like you have to have these complex inheritance patterns, because you want to be able to say, well, this thing is, it is this matches this interface, you know, it does this particular thing.
24:25 Python's take is, I really don't care what it is.
24:29 As long as I can interact with it according to a particular kind of set of assumptions, then I don't care what it's really doing. I'm good.
24:37 And this can be really beneficial because it allows you to create new things that act like integers, even though they're not integers, or new things that act like Booleans that aren't Booleans, new things that act like strings that aren't strings.
24:47 Python doesn't care.
24:48 It's just interfacing with those objects on the basis of how it can use them and what certain things are going to return.
24:58 That's all it cares about.
24:59 So the concept of duck type in being it doesn't care whether it's a duck or a picture of a duck or a statue of a duck or a moose in a duck suit.
25:05 As long as it can interact with it in the way it needs, it's good.
25:08 And that's very freeing.
25:10 >> It walks like a duck and talks like a duck and quacks like a duck and it's probably a duck.
25:13 >> Yeah, exactly.
25:14 And it's very freeing once you get that because you don't have to muck about with all this weird inheritance anymore.
25:19 It's like, well, it does what I need.
25:22 And that's the important part.
25:23 There is certainly less inheritance and less complex hierarchies in Python.
25:27 Not that there's not an important place for object-oriented programming and classes.
25:31 They're just often simpler.
25:33 A lot more straightforward, definitely.
25:35 And it frees us up to then the saying I coined a couple of years ago during a conference talk was that you need to define object by every constituent data.
25:44 So when you're writing a class or you're designing an object, you're thinking about what is it storing?
25:50 Like what, what sort of state, if you think in a functional way, or what sort of data am I really putting in here?
25:57 And you can design based off of that.
26:01 And then you can think about, well, how do I want, how does the system need to interact with that data?
26:07 But you don't really have to sit there and think about encapsulating behavior.
26:10 Cause that's not really the point.
26:11 The point is you can think about, like with example, I use this, we always see the example of, I'm going to create a class called cat and a class called dog and they inherit from animal. And that's actually a really lousy example we need to stop using because that's starting from the concept of behavior. Well, it barks, it meows, it eats, whatever. And in real systems, what we think about is I have a, you know, someone design a game, I have a monster. Okay. What is a monster? Well, it's, it's a bad guy. No, no, no, no, not what it does. What is it? Well, it's, that's a collection of statistics about the character, you know, like how much strength it has and how much health it has.
26:49 Okay, that is not a monster, that's an entity.
26:53 You don't have to put what it's doing into that code, it is this collection of information.
27:01 And when you shift your thinking to a data-first mindset about object-oriented, then you find that you can actually use it with functional paradigms that are sometimes a lot easier to reason about.
27:12 - Yeah, the grouping together of data is an important part.
27:15 I would say the most extreme example of duck typing in Python, maybe a couple areas, and they all seem to revolve around the magic methods or the Python data types, like the dunder methods, right?
27:28 - I love those. - So I could, yeah, I know.
27:30 So I could have a function, which I could call, or I could have a class that implements dunder call, is a callable, but is actually an instance of an object, Right. And, but as far as Python's concerned, it doesn't matter.
27:43 It can call it and pass parameters to the thing it called.
27:46 Right.
27:47 And there's no real type that disambiguates those two things.
27:50 Like one is a function type.
27:51 One is a, you know, a class with a particular method on it.
27:54 Right.
27:55 But they work just fine.
27:56 And that's a pretty extreme example.
27:58 You know, another one would be, I have a list I can iterate, or I have something that implements the iterator pattern through the Dunder methods, right?
28:07 Like the Dunder iterator and so on.
28:09 Right.
28:09 Well, yeah, one of the classes I, one of the examples of the book I gave was like a, a cafe queue, so a queue of customers in a line.
28:17 And yeah, I wrote it as an iterable, but there was all this additional stuff in there, it wasn't just a basic list.
28:23 There was more to it than that.
28:24 And, but because I implemented as an iterable, I didn't care.
28:28 I could use it like I use a list, but it was able to do this additional work with data behind the scenes that made it just easier to use the, to reason about this line of customers.
28:41 And then you get generator methods with the yield keyword and yield from and like all sorts of stuff.
28:46 You, you get Py Test fixtures that allow you to set up and tear down without actually writing try finally blocks and all these things.
28:55 A lot, a lot of interesting places.
28:57 While I was thinking about this conversation and this topic in particular, I ran across an article called static duck typing in Python with protocols.
29:05 How's that to like crash two ideas together?
29:08 Yeah, it's well, and it's, well, when we look at data types, it's like the way Python even does its data types is yeah, you can say that you want to say, okay, this is a list and you can type it as a list, but you can also just actually type it as a iterable and basically tell Python what you care about.
29:27 Do you really care that it's a list?
29:30 Probably not.
29:31 You care that it's an iterable that you can work with similar to a list?
29:36 How are you going to use it?
29:37 That's what you ask yourself.
29:38 And you base your annotations around that.
29:41 And so you can say, well, like here, measurement-like, I want it to be usable as a measurement.
29:47 I don't really care what it is.
29:48 It doesn't need to be this particular type of class.
29:51 And so that's moving from, like we're talking about, moving from Java thinking into Python thinking is, well, it's not what it is, it's what it does.
29:58 Yeah, a lot of the types people put down are like, I need to treat it like some duck typing situation.
30:04 So it needs to have these attributes about it.
30:07 Let's try to put that aspect of it into the formal static typing rather than earlier.
30:13 Yeah.
30:13 Okay.
30:14 Maybe, maybe a quick audience question or a bit of feedback here before we move on to the next one.
30:18 It's relevant.
30:19 Nick says early in my Python journey, I wish I had gotten better understanding of all the intricacies, differences and use cases for various data structures like set versus list.
30:28 Right.
30:28 Yeah.
30:29 Versus dictionary.
30:30 And those are certainly, even if you've done programming before, maybe that environment language didn't deeply embrace these data structures.
30:37 So certainly those are table stakes for being good at Python.
30:41 Absolutely.
30:41 Use cases hits the nail on the head because it's like, what's the difference between them?
30:45 Well, you know, it's a collection.
30:46 And so maybe that's all you care about, but I find myself using all of them in different cases because it's like, all right, I never need to add or remove things from it, I probably have too.
30:55 Or I need to, or I need to be able to hash it, use it as an immutable.
30:59 That's a tuple.
31:00 Or I need to be able to see what's in group A and not group B.
31:03 That's set.
31:04 Or I care about the order.
31:05 That's a list.
31:06 Or I want to be able to associate two things.
31:08 That's a dictionary.
31:09 So knowing your different data structures is helpful.
31:12 And you know, it's tricky because Python's data structures aren't really all that unique, but they have different names than maybe you use whatever languages.
31:19 So, you know, you come across list and what is that?
31:21 well, it's a C++ vector or a JavaScript array, you know, it's like, it's, it's, which is where it's helpful to come back to Python kind of with a fresh mind is like, what is this?
31:32 Not assuming that things have the same names, but being prepared to get a different vocabulary.
31:36 This portion of talk Python me is brought to you by us over at Talk Python Training with our courses.
31:43 I want to tell you about a brand new one that I'm super excited about.
31:47 Python web apps that fly with CDNs.
31:51 If you have a Python web app, you want it to go super fast.
31:55 Static resources turn out to be a huge portion of that equation.
31:59 Leveraging a CDN could save you up to 75% of your server load and make your app way faster for your users.
32:06 And this course is a step-by-step guide on how to do it.
32:09 And using a CDN to make your Python apps faster is way easier than you think.
32:14 So if you've got a Python web app and you would like to have it scaled out globally, If you'd like to have your users have a much better experience and maybe even save some money on server hosting and bandwidth, check out this course over at talkpython.fm/courses.
32:30 It'll be right up there at the top.
32:31 And of course the link will be in your show notes.
32:34 Thank you to everyone who's taken one of our courses.
32:36 It really helps support the podcast.
32:38 I'm back to the show.
32:39 Iteration, iterables, iterators, Looping.
32:45 Yes.
32:46 So I was working in some code last year and I came across something that just, I, was, I cringed and then I laughed out loud because this was written.
32:55 So this code was written by who I best guests university data scientists.
33:01 So, you know, a, to their favor, this is not a software engineer.
33:05 What they had written and they could tell it had been a refactor a few times, but I am not kidding.
33:10 They had said, all right, so I have this thing that it's iterable.
33:13 It was already something that we could use in a loop.
33:15 I'll come back to what that is for anyone who's curious, but we have this iterable.
33:18 I'm going to convert that into a list.
33:20 By the way, we're talking about in this case, tens of thousands of items in that list.
33:25 I'm going to convert that into a list.
33:26 Then on the next line, they say for index comma value in enumerate, enumerate is a thing that lets you get the indexes as well, enumerate that list.
33:37 And then it says, you know, and then it accessed the value again, inside the loop by going to the original list and accessing using the index.
33:46 And I just sat there and I kind of half laugh, half cries.
33:50 It's just like there is so much, this is so convoluted.
33:54 And all I had to do was take all that logic out and just say for item in original iterable, and then item was the thing.
34:02 But what had happened was that they didn't understand what was really happening in a for loop.
34:06 They were still thinking C or C++.
34:09 And the thing is that Python has these two magic methods called iter and next.
34:13 Iter just says, give me the thing I can use that has the next operator that knows where it is in the list.
34:19 I'm gonna leave it at that.
34:20 And the next just says, give me the next item.
34:22 That's all it is. Just give me the next item.
34:24 All a for loop ever does is it grabs that, is it calls that next.
34:28 It's like, give me the next thing.
34:29 All right, good. I'm done. Give me the next thing. Give me the next thing. Give me the next thing.
34:32 You can write entire classes that behave like this, even generate the value at that moment.
34:38 It's called lazy iteration.
34:40 You can actually generate the next value when it's wanted, not a moment sooner, which means you only ever have one value in memory sometimes.
34:47 And then you can just put that in the for loop.
34:49 You come up with any arbitrary name for when you access it for each thing in iterable and name thing, whatever you want, and then use it directly in your code.
34:57 And I wish someone had told me that a lot sooner, discover that about three years into my journey, I'm like, Oh my gosh, I've been making lists out of everything and I've been doing it.
35:05 It's like a waste.
35:07 - Yeah, and you know, the yield keyword, obviously is a super easy way to do that lazy iteration that you talk about, but it also is really important for things like databases, right?
35:16 - Yeah.
35:17 - If you talk SQLAlchemy, right?
35:18 I'm gonna do a query, I could call the all method and turn it into an in-memory list, or I could, if I don't need that, I could just loop over it, right?
35:25 And kind of read it cursor style, potentially stop talking to the database halfway through.
35:29 - Exactly.
35:30 If I only need it, I don't need the rest of the data.
35:31 - Exactly.
35:32 - Yeah, exactly.
35:33 Yeah, interesting.
35:34 Cool.
35:35 I think iteration is quite, it's settled because there's only two types of loops in Python and usually we're using the for loops, but the variations, there's just so many of them, right?
35:48 You can have your for loop that just goes over a thing, but what if you need the index like you described, not for this improper use, but sometimes you actually still do need the index.
35:55 So you could do enumerate of that thing and get a tuple, but you could take it to another level do tuple unpacking and say like index comma value in enumerate of things.
36:04 Right.
36:05 And you get it back and you can unpack a dictionary too.
36:08 You can unpack a tuple with that in the dictionary.
36:10 There's this whole library and it built in a Python called inner tools.
36:13 And if you ever have, you know, 10 minutes over coffee, just pull up the documentation for inner tools and just kind of scroll through there's some neat stuff, like you can, you can like find every possible combination of three different lists, I just give it three different lists and this is one that It'll take all three of them, just give you every single possible combination of it, actually use it in the book.
36:32 And it's just, there's all these neat things you can do with iterables.
36:35 And some of them don't even have to stop.
36:36 There are infinite iterables where you can just say, I want to get this sequence repeating infinitely.
36:42 One, two, three, one, two, three, one, two, three, one, two, three, one, two, three, one, two, three, and two lines of code.
36:46 And you can have that repeating forever.
36:47 And if you're using an old fashioned while loop or an old fashioned C++ style for loop, that would be multiple lines of code.
36:53 Like, you don't have to keep track of the state and all that stuff.
36:56 I think it's two lines.
36:57 - Yeah, it's amazing.
36:58 Yeah, so there's a lot of depth, even though it seems so simple when you first look at it.
37:02 Closely related to that are comprehensions.
37:05 - Yes. - Like a list comprehension.
37:07 People often see a list comprehension or a set comprehension and go, especially a list comprehension, they're like, "Oh, that's the same thing as a for loop." Where I create, before the for loop, I create a list, and then in the for loop, I add the thing to the list, and these are the same.
37:21 May have the same outcome, but they're not actually the same thing.
37:24 - They're very different. - The runtime performance is not the same.
37:27 The bytecode implementation of those two things is not the same.
37:30 This is like a nonstop debate I have on YouTube whenever I put a video and mention something about list comprehensions, they're like, that's just a for loop.
37:37 It's not actually just a for loop.
37:38 It's like a kind of accomplishing the same thing, but we now have for potentially 312, Carl Meyer put out a PEP 709 called inlined list comprehensions, which are supposed to take some of the mostly invisible behind the scene implementation and make comprehensions two times faster.
37:59 I believe they're already faster than this looping story I talked about, but this would make them faster still, which is pretty cool.
38:06 Yeah.
38:06 Wow. How does this syntax work?
38:08 I'm curious. It's just like, it's just an internal change?
38:12 Okay.
38:12 So the syntax is unchanged.
38:14 Yeah. So here's the magic of list comprehensions that's funky that people maybe aren't aware of.
38:19 Is it, it used to actually create a list comprehension object that implemented the loop in a function and then it called that function as if you'd like nested a function to create isolation of the list comprehension.
38:32 So that created a stack frame, that created an object, then it did function in direction and all those things made it slower than it needs to be. So now they create a new load fast and clear byte code operation that like stores the state of the function then does the list comprehension and then pops it back off the stack to unwind it. So Anyway, comprehensions are interesting.
38:55 I just wanted to, this is like kind of a brand new thing.
38:57 That's cool.
38:57 That's coming along.
38:59 That might be interesting for people.
39:00 Well, I think it's worth mentioning then that what a list comprehension really is.
39:05 So like if I unpacked it, because people think, oh, people use that term very loosely a lot of times.
39:09 They say, well, it's a list comprehension, but what is a list comprehension?
39:13 And if I unpack that, what it really is, is it is a generator expression that is being unpacked into a list. So what's a generator expression? Generator expression is a type of generator. What's the generator? A generator is a function or type of function that is itself iterable. So that's where this is. That's where this is critical is that a generator you can you can write a generator that what wouldn't call it that it has you mentioned that earlier this yield keyword and so you can do whatever you want in this function. You can write it however much logic you want, every time you hit the yield statement, it will return that value and then it's going to wait for the next call, then the next iteration of that, like the next call to next on that generator, and then it's going to run through again until it hits a yield. And then it's going to stop returning that value and wait again. And you can do some incredible things with these. But once you understand the generator, which are incredibly powerful tools, then you can write a generator expression, which is I had discovered later, a generator expression is to a generator as a lambda as to a function. It's an anonymous generator. That's all it is. A lambda is just a very small inline anonymous function. Has the logic, it can take in some values, return some values. It's a lot simpler, a lot shorter, and you do it right in place. So you're not having to go off somewhere and write a different function and come back. That's why we have anonymous functions is so that we can just do it like, like you see there at the bottom, reduce and just some, some rules right there instead of writing a separate function, generate expression, same sort of thing. We're doing writing some generator logic, it's shorter, it's creating a generator, but it's just a lot smaller. And if you just wrap it in in parentheses, you can use it as a generator. But when you change those parentheses to be square brackets, what happens is you're running that generator, and then you're loading all the values into the list. You do it in in curly braces, you're creating a set or if you're if you're doing the same thing, but you are the return value in each step is a key value pair, then you're going to create a dictionary.
41:19 But all you're really doing is unpacking that generator into a data structure.
41:23 So sometimes you want to think when you're mucking about with these, do you really want the data structure when you're done, or do you just want to go over the values?
41:32 Maybe you can just write a generator expression.
41:34 >> Yeah, the difference is super subtle.
41:35 Like do you have square brackets or do you have parentheses around the expression, but the effect or consequence of that is very significant at runtime.
41:44 It's kind of like that example you talked about, putting everything in a list and looping over it or just looping over it.
41:49 Right?
41:50 Exactly.
41:51 And because these one-liners and we as coders tend to get addicted to these one-liners.
41:55 It's like, oh, look at all the cool things I can do.
41:58 I have a couple of examples in the book.
41:59 Some real list comprehensions I've come across or some colleagues of mine have come across where it's like, it's multiple lines and that's a bad list comprehension.
42:08 That's a bad generator expression.
42:09 Just like you don't want to have a multi-line lambda.
42:11 At that point, write a function.
42:13 Generator expressions and lambdas are supposed to be these really small, short in place bits of logic.
42:18 And if it's getting complicated, write the full size thing.
42:22 It's a lot easier to debug.
42:23 If nothing else, it's easier to debug, but it's also a lot easier to read.
42:26 It is. It is.
42:28 Just while we're kind of ranching on these things, like, I wish there was a way to specify a sort in these list comprehensions, generator expressions, and maybe not the generators, because that gets tricky, but so often I find myself, I'm doing a list comprehension, And then I got to turn around and sort it.
42:44 Like if it could just, just add that in there, that would make me happy, but all right, moving on packaging, deciding on packaging and all that right up front.
42:54 Actually, before we get to that, I guess the other one that you threw in here is virtual environments.
42:58 Indeed.
42:59 Yeah.
42:59 Virtual environments use them.
43:01 There'll be only, I avoided them for years because I thought, oh, these are scary.
43:04 I don't understand them.
43:06 They're not scary.
43:07 I'm actually gonna, I'm actually gonna hijack something I learned from a, one of I guess on my own podcast recently is what a containers actually are.
43:14 By the way, Docker containers, this is where this is relevant.
43:17 You know what a Docker container actually is?
43:19 Is it's the folder on your operating system and it's just changing the root.
43:22 So when you're working inside of it, it's just changing the root.
43:24 That there's a bunch of magic logic around that, but that's all it is.
43:27 It's like, it's pretending that that folder is the root and it's running things accordingly.
43:32 That's all a virtual environment really is doing either.
43:35 It's so there's not a lot of difference between.
43:37 It lies to Python about the path.
43:39 Yeah.
43:40 It's not a lot of difference in a virtual environment and a Docker image.
43:43 Like they're the same fundamental concept.
43:45 So all you're doing is you're creating this little spot on here, this little folder, and you're saying, "These are my Python packages." And unless you tell it otherwise, "Don't go looking anywhere else. These are the ones I want to use." And they're super easy to create.
43:59 It's like one line to create a virtual environment.
44:01 And it lets you maintain exactly the packages you want for that particular situation.
44:07 And it's great.
44:08 It is, yeah. Once you get used to it, it's great.
44:11 I remember it being frustrating at the beginning.
44:13 You're like, "Ah, I don't remember exactly how do I create the environment?
44:17 How do I activate it? How do I know? Do I really need this?" And yeah, I don't start any projects without having them as just straight up, use them right away. Like, absolutely.
44:26 I probably have created them for projects that don't even have external dependencies because I was just like on autopilot.
44:31 Like, "Oh, actually I didn't need that, but whatever. It's still good to have." Yeah, absolutely. It's a control which Python version you're using and then you can install whatever packages you want.
44:38 You don't not worry about messing up your system or user scope packages.
44:42 And a coworker recently said, I dream of the day when I can actually remember how to activate this, but it's actually really easy.
44:48 So once you, so assuming you know the name of the folder, you can pick the name of the folder you use for your virtual environment.
44:55 Traditionally, it's called like VENV.
44:57 And it's, here's the thing to remember, you don't actually have to activate your virtual environment.
45:02 Put that in your mind for a moment.
45:03 You can run things directly in the virtual environment.
45:06 So in Unix systems like Linux and Mac, there's a bin folder.
45:10 So them/bin/ and then whatever Python by whatever Python executable you want to run.
45:16 It could be Python, it could be pip, it could be your own package.
45:19 You can actually just navigate into that bin folder.
45:22 And so I can say them venv pip, and I've got the pip and that knows to only work within its virtual environment.
45:29 It's not going to break out of that container.
45:30 Once you understand that, here's where the activate script is.
45:33 Then bin activate.
45:35 It lives in the bin file.
45:36 Yeah.
45:37 And if you're on windows, the only thing you're changing is you're going from bin to capital S scripts.
45:42 And then the things have an ending of like either dot bad or dot PS one, depending on what you're, what you're, you know, what show you're using, but that's the structure of a virtual environment.
45:52 And once you understand that they're a little over scary.
45:54 Yeah.
45:54 My RC file, my, for my user profile on my Mac is just full of, here's the Python executable in some virtual environment, and here's the script and and commands to send to it for like little aliases and stuff.
46:06 And like none of those activate the virtual environments.
46:09 They just run the Python in the virtual environment to make all the magic happen.
46:12 Another thing I would recommend is people think about creating aliases that make virtual environments happier and easier to use.
46:20 So for example, and you could do this in PowerShell on Windows, you can obviously do it in your RC files on Mac and Linux.
46:27 So for example, I have VNV, which is for VENV, but I don't want it to accidentally change the directory and if you haven't, like whatever.
46:34 I just have a short version, so I know it's something slightly different, but it'll go run the virtual environment and it'll always remember to add --prompt=.
46:42 which will make the virtual environment not just be called VENV, but it'll be called the name of the project that it's in.
46:48 So for example, if it's in like Talk Python, like the virtual environment, when I activate it, it will say it's the virtual environment Talk Python, right?
46:55 And then it activates it.
46:57 And then I think this is almost a bug in Python that most of the time, I would say like 49 weeks out of the year, when I create a virtual environment, the first thing I hear is, "Warning, your pip is out of date." You know, like, okay, well, I'm starting a new project.
47:13 How about we start with a new one?
47:14 Let's just go with that right now, right?
47:16 And so my, just that is, I always want the new one.
47:19 I don't ever want the old one because this is brand new.
47:22 So I want the new one.
47:24 Why don't we fix that?
47:24 So my alias then says, pip install -upip, setup tools, wheels, pip-tools, etcetera, and just ditches that output.
47:31 So like, right, immediately, like I have, when I create a virtual environment, it's immediately active and it's immediately named the project and it's immediately up to date.
47:39 So I don't see those silly warnings, at least for a while.
47:41 And all I got to remember is type VNV, boom.
47:44 And that's it, right?
47:45 So if you get into this, if you just do a few little steps.
47:47 - I do this too.
47:49 - Yeah, yeah, I imagine you did.
47:50 And so it's, I just strongly encourage people to somehow either do this in PowerShell or do this in their terminal setup because like that big long thing to type, that's a hassle.
48:00 Three letters is not a hassle and they just makes it, there's something about lowering the friction that makes it really nice.
48:07 - And there's tools you can also install that do it, but like I recommend just writing your own aliases too because everyone's got a sub-different workflow instead of perfecting it anyway, that's why VIMP has so many options.
48:16 So writing this yourself, it's super easy and it teaches you a bit about how it's working so that you're not surprised at the results.
48:24 - Yeah, yeah, indeed.
48:26 Okay, now packaging.
48:28 And, I I'm going, I'm going to keep this one super short because it's a deep, deep, deep, deep, deep, deep, deep rabbit hole.
48:35 Figure out your packaging day one.
48:37 Like you get, I will figure out packaging for even figure out what I'm, you know, how I'm building stuff.
48:42 I am just going to set up a basic hello world, and then I'm going to get the packaging work because I actually, and I learned this the hard way.
48:51 I built a project once it took me one month.
48:53 I was very happy with it.
48:55 It was gorgeous.
48:56 It was a game.
48:56 You can actually find it online now.
48:58 I was very happy with myself.
49:00 And then I spent the next two years trying to package the dark thing
49:03 Turned out it was impossible to package because of certain assumptions that one of the libraries I was using made.
49:08 Had I actually tried to package it from the beginning, I could have saved myself a heck of a lot of time and just used a different library.
49:12 So, you figure it out from day one.
49:16 And then the benefit you get to this is that then you are constantly, especially if you're doing out of place testing, which I strongly recommend, run your tests, your tests should not be in your source folder, put them separate.
49:28 that's going to force you to actually install your package into a virtual environment and then run the test against that package.
49:34 It's not as scary as it sounds. I have a whole, by the way, if you go to my website code mouse92.com, I've got a whole talk on this called Escaping the Cardinal Cult.
49:41 And I explain how these patterns work. But when you do it that way, that means from day one and ever onward, you are testing your packaging, you're testing your assumptions about the working directories and all that. And you are testing it in the same way that it's going to be used on someone else's computer.
49:58 So you never again have to have the thing of, well, it works on my machine, because you are, all of your tests are depending on actually installing your package.
50:07 So if something breaks, you bring in a dependency that just doesn't work with your packaging scheme, you know right away and you can make a decision about it.
50:15 Yeah, that's good advice.
50:17 There's a lot of depth to the packaging thing. I had a whole panel on it on talkPython not long ago.
50:21 Definitely a lot there.
50:22 No, yeah.
50:23 Deep rabbit hole, but you know what?
50:24 Don't copy it.
50:25 Don't, don't copy, pasta.
50:27 Don't copy paste.
50:27 Find out what the things you're using are doing because.
50:30 All right.
50:33 Last one.
50:34 Yeah.
50:35 Concurrency.
50:35 Concurrency.
50:36 Yes.
50:37 Which for years I heard concurrency, asynchrony, threading, parallelism, multiprocessing.
50:42 I heard those all used as, as synonyms for one another.
50:46 They're completely different things.
50:50 So, and actually I have to thank the Go Language for teaching me this.
50:55 Thank you Rob Pike for his talk, concurrency and parallelism are not the same thing.
51:00 But here's the thing to understand, concurrency is doing, you're switching between tasks.
51:05 You're checking your phone and you're talking to your friend.
51:07 Are you doing both at once? No.
51:09 You're not reading stuff on Twitter and listening to what your friend says.
51:13 You're actually switching between the two contexts rapidly.
51:16 You're not really multitasking, you're just changing your focus fast enough that no one ever notices the difference. That's concurrency. So you have one CPU or potentially, potentially multiple remote this where people get confused. Python's nice because you're locked the one CPU for concurrency. So it's I'm gonna work on this, then I'll work on this and work on this, then work on this, it doesn't make your code faster.
51:38 It'll never make your code faster, just like multitasking will not make you get through your workday faster. What it's going to do is it's going to slow you down. But it doesn't look like is in some cases, it'll make your code look faster. Because while you're waiting on the user to type in their answer, this example I use the book, or you're waiting on the user to type in their answer to a question, you're doing some math in the background, your IO blocked, you're waiting on something over which you have no control, you can do some other work while you wait, that's concurrency. And you can do that with a synchrony where Python's handling that multitasking, or you can do with threading where the operating system is handling that multitasking. That's different than parallelism, which is where you're actually using different CPU cores and you're doing two things at the same time, because you have two different CPUs, two different cores working in parallel. And they're not depending on one another. Not they're not waiting. You're doing multiple tasks at once. Both of them have overhead near one is a silver bullet for performance, you're really going to have to think a lot about it. But I actually like the GIL for this reason, global interpreter lock the much aligned thing that stops us from getting free multiprocess of free parallelism, free multiprocessing with every thread.
52:48 Because this forces us to actually stop and think, which thing do I need?
52:53 Do I need concurrency or do I need parallelism?
52:56 Because you can get both in Python, but the gil makes sure that you have to make a decision.
53:01 Because if you want multiprocessing, you're going to have to create different processes, spin them off, manage them separately.
53:07 There's more overhead, but you're going to have to address all of those things of how do we share resources, which is a really snarly topic.
53:14 And Python doesn't let you pretend that that's not going to be an issue.
53:17 It makes you think about that.
53:19 It does make you think about it.
53:20 While you can't get more done, you can't make an individual thing go faster by using concurrency on it.
53:26 You can start a bunch of things that run outside of your control and outside of your program, and then get them back.
53:32 Right?
53:33 Like I could issue a web request to three API endpoints, all of them, and then start gathering up the answers.
53:38 The first one comes back, right?
53:39 Like you have to think about, I think the main thing to think about here is Where am I waiting on something else when you're looking for the async and await type of benefits?
53:50 Like, am I waiting on a thing?
53:51 Could I be doing something while I'm waiting?
53:53 And async and await amount, it makes it super easy to do stuff while you're waiting, but your program itself is still constrained by the gil and often with async and await, there's not even a second thread involved, right?
54:06 It's just slicing up time, slicing up the current thread.
54:09 Yeah.
54:09 Which I mean, technically is all that's going on with the threads anyway.
54:12 like when you have multiple threads, the operating system is now just doing the slicing up.
54:16 You're still constrained to one process.
54:18 Then you need a separate process if you want to do multiple things.
54:21 - Yeah, but like my machine has 10 cores.
54:23 It will go across the cores according to the OS, but the effect in Python, because of the GIL, means it doesn't actually make any difference.
54:30 It's still the same, right?
54:31 But if you get multi-processing, then for sure.
54:33 So yeah, super, super interesting.
54:34 I think one of the things people tell, I think people say two things.
54:39 One, asynchronous programming is so hard, you should never try it.
54:42 I hear that all the time. It's, it's so complicated. It's so hard.
54:45 You're going to end up with race conditions. Just don't try it.
54:47 Mere mortals don't dare. They only regret.
54:51 Maybe if you're doing like signaled events in C++ across, you know, hardware notification, like there are scenarios that are super, super tricky using async and await generally don't fall into that.
55:06 Like you write regular code without changing the structure.
55:09 You don't have to have callbacks and signals and locks.
55:12 You just write regular code or you have the async stuff you put away.
55:15 It's kind of the end of the story.
55:17 And it's not necessarily as simple as straight up regular programming, but it's, you know, it's like 10% more.
55:22 It's not 10 X more.
55:23 Yeah.
55:23 I think a lot of people for some, I don't understand why, but a lot of people, I think they, they want to feel like I've conquered this dragon.
55:32 And if I'm going to tell other people how to do it or what to do, the first thing I'm going to tell them is how hard it is and how daunting it is.
55:39 And like, yet I've conquered it and now I will help you.
55:41 Right.
55:41 That doesn't do people a favor.
55:43 I feel that overall.
55:44 I guess so.
55:46 But I, I feel like threading and parallelism lands deeply in that space where people are like, Oh, it's super hard.
55:53 And it's not, it really isn't.
55:55 But where it's hard is you, you have to be prepared to really think about why do I need it?
55:59 And if I can briefly say, I have two entire chapters in this book on that topic.
56:05 Yeah.
56:05 Cool.
56:05 And I pick one example and it has nothing to do with waiting for a webpage to download because that is both overused and something that most of us don't need.
56:12 And I apply a symphony.
56:13 I apply threading.
56:14 I apply parallelism and you actually see all those little concerns of how do you handle deadlocking, live locking?
56:22 How do you handle starvation?
56:24 How do you handle the producer consumer problem?
56:27 How do you handle the fact that, huh, I just had a 15 multi, multi processing workers and now this thing is like way slower.
56:33 Why?
56:33 I go into all of that.
56:35 And I think that's one thing I wanted to say is when I was writing this book, that was the thing that was my chief delight was coming up with examples for all of the topics from the simplest, all the way to the most complicated, where I did not hold back on the relevant complexity.
56:49 My list comprehensions were the starlist list comprehensions you're ever going to find in this tutorial.
56:54 You know, my, you know, I have, I go into recursion.
56:57 I go into, yes, I go into meta classes, whichever one's afraid of, I go into all of these things.
57:02 I dig into them, but the complexity is always relevant.
57:06 So you're only going to get the complexity from the topic and you're going to understand where all these bits and pieces interplay.
57:11 But the complexity coming in from other irrelevant topics, I try to keep to a minimum so you can really focus on understanding, not just the happy path of coding, not just the happy path of like, okay, well, you can async this.
57:25 Look how easy that was.
57:26 Well, it's not easy, but it's also not painfully hard.
57:30 you just have to be prepared to think through those different factors.
57:35 And so I focused a lot of time on really helping you learn how to think through those factors as it relates to your code.
57:41 Excellent.
57:41 It sounds like a bit of the Einstein philosophy, simple as possible, but not simpler.
57:45 Right.
57:45 Exactly.
57:46 Yeah.
57:46 Cool.
57:47 I know I'm last thoughts on the async stuff.
57:49 Like I think a sprinkling of async and await makes massive difference for a lot of scenarios.
57:53 And that's pretty straightforward.
57:54 If you're going to try to write trio or some full on framework that deeply coordinates and understands it. Like, yeah, that's super, super hard. Again, some of the signaling stories I talked about with like C code, that's super hard, but this stuff, people, it's pretty approachable. So don't, don't forget to check it out. All right. Well, I think that's it for our things we wish we knew if we had more time, we could wish upon more things, but this pretty much definitely covers it. Doesn't it? Yeah, I think so. Nice. All right. Well, before you get out of here, let me ask you the final two questions. If you're going to write some code, what editor to use?
58:28 Yeah, you know, for the longest time I was a VS code guy.
58:31 I still love it, but I broke down and I bought PyCharm professional.
58:34 And I love that because I do a lot of refactoring and PyCharm saves my butt so much when it comes to refactoring that I fell in love with it.
58:43 I can't imagine working without it now.
58:44 I hear you.
58:45 So I'm spoiled.
58:46 It does surprise me how people, not just a PyCharm comment, but in general, There's a lot of people are like, Oh, I don't really want to get like a new computer or I don't really want to pay 20 bucks for this thing that would save you thousands of dollars worth of time as you go for, you know, like, so I'm glad.
59:06 Marks on your tools.
59:07 Hey, as far as your budget will allow you.
59:10 And I I've been on both ends of that, that spectrum.
59:12 And as your budget allows you to support on your tools, because third, what you're going to be spending most of the day using anyway, and the last thing you want is something that you're going to be getting splitters on.
59:21 So Indeed.
59:22 And then notable PyPI package, something you're like, all this thing's awesome.
59:26 I just ran across XYZ.
59:28 I guess I have two.
59:30 One would be, I think my favorite, and this has certainly got plenty of press, but I love it.
59:34 It's the hypothesis testing library.
59:36 I love hypothesis.
59:38 Hypothesis is fantastic.
59:40 It will revolutionize the way you write tests.
59:41 It's not as scary as the documentation makes it look.
59:44 If you click on what you can generate and how start there, but I'd go for the quick start, grind and then go right to this.
59:49 And it's so cool because you just write these strategies on like, okay, this is what my data is kind of sort of supposed to look like.
59:56 And then your test is going to try and break your code.
59:59 I love hypothesis is going for all sorts of garbage at you until it finds a way to break your code.
01:00:04 And then it's going to go, Hey, here's the simplest thing I could find that blew up your code.
01:00:08 It's it lovely.
01:00:09 Nice.
01:00:09 So much fun.
01:00:10 The other favorite package I have is one of my own actually branch detective.
01:00:14 I built this at two jobs ago.
01:00:17 And basically, we had the problem where we had these diverging branches where we had no choice but to completely diverge them and we were cherry picking back and forth. And there was always this which commits are in this one that aren't in this one, when you're doing cherry picking your hashes are complete garbage. So I built this tool that allows you to see what's in one branch, it's not in another branch, any branch arbitrarily, you can specify day range, it knows how to handle cherry picks, it knows how to handle merge revisions or filter them out and you can even pass the dash M flag and just get this nice.
01:00:50 Markdown for using conventional commits.
01:00:52 We'll just spit out this markdown that you can then copy and paste into your PR and say, Hey, here's, here's all, here's the summaries of the 26 commits I'm putting in here.
01:01:00 So I use this every single day and I've improved it for every project I'm on so far and I, I just, I love it.
01:01:08 It's my favorite.
01:01:09 One of my favorite things ever built.
01:01:10 So you kind of wrote set forget, right?
01:01:13 Like you can say what's in the set, subtract that set.
01:01:15 Show me those things. Yeah. Speaking of data structures.
01:01:18 Exactly. Like what's here. That's not there. Yes.
01:01:20 Yeah. Cool. All right, Jason. Well, thanks for being here.
01:01:23 Final call to action. People will get their Python better. What do you tell them?
01:01:27 Yes. So that's a book I thought is available from wherever you buy books, but especially if you go to no starch.com, you can get this comes in ebook or in physical book form.
01:01:38 And, Hey, you know what? I wrote it and I refer to it every single day because this is the book I wish I had had.
01:01:43 And my favorite little feature of this is that I have a thing in the back with every single command for the debugger.
01:01:49 Also, I have every single Dunder method is also documented in here.
01:01:53 Which, even the Python docs didn't do that.
01:01:55 This is the entire core language, bar nothing, explained for the existing coder. So, check this out.
01:02:01 From nostarch.com.
01:02:03 Well, thanks for taking the time for being here and enjoy the chat.
01:02:05 Thank you so much. It's been a pleasure.
01:02:07 You bet. See you later.
01:02:09 This has been another episode of Talk Python to Me.
01:02:13 Thank you to our sponsors.
01:02:14 Be sure to check out what they're offering.
01:02:15 It really helps support the show.
01:02:18 Take some stress out of your life.
01:02:19 Get notified immediately about errors and performance issues in your web or mobile applications with Sentry.
01:02:25 Just visit talkpython.fm/sentry and get started for free.
01:02:30 And be sure to use the promo code talkpython, all one word.
01:02:34 Want to level up your Python?
01:02:35 We have one of the largest catalogs of Python video courses over at Talk Python.
01:02:39 Our content ranges from true beginners to deeply advanced topics like memory and async.
01:02:44 And best of all, there's not a subscription in sight.
01:02:47 Check it out for yourself at training.talkpython.fm.
01:02:50 Be sure to subscribe to the show, open your favorite podcast app, and search for Python.
01:02:55 We should be right at the top.
01:02:56 You can also find the iTunes feed at /iTunes, the Google Play feed at /play, and the Direct RSS feed at /rss on talkpython.fm.
01:03:06 We're live streaming most of our recordings these days.
01:03:09 If you want to be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
01:03:17 This is your host Michael Kennedy.
01:03:18 Thanks so much for listening.
01:03:19 I really appreciate it.
01:03:21 Now get out there and write some Python code.
01:03:22 [MUSIC]