#441: Python = Syntactic Sugar? Transcript

00:00 You've probably heard the term syntactic sugar.

00:02 That is syntax within a programming language that is designed to make things easier to read or to express.

00:08 It makes the language sweeter for humans to use.

00:12 It turns out Brett Cannon has spent two years diving into and writing about Python's sweet language features and how they really work down inside CPython.

00:22 He joins me on the show today to dive into a few of the more relevant posts he's written about.

00:27 This is Talk Python to Me, episode 441, recorded November 1st, 2023.

00:32 Welcome to Talk Python to Me, a weekly podcast on Python.

00:50 This is your host, Michael Kennedy.

00:52 Follow me on Mastodon, where I'm @mkennedy and follow the podcast using @talkpython.

00:57 Both on mastodon.org.

00:59 Keep up with the show and listen to over seven years of past episodes at talkpython.fm.

01:04 We've started streaming most of our episodes live on YouTube.

01:08 Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.

01:15 This episode is sponsored by us over at Talk Python Training.

01:21 Did you know that we have over 250 hours of Python courses?

01:26 Yeah, that's right.

01:27 Check them out at talkpython.fm/courses.

01:29 Brett, welcome back to Talk Python to Me.

01:34 You've been here a time or two.

01:35 Always great to have you here.

01:37 Well, thanks for having me yet again, Michael.

01:38 I'm glad I've not worn out my welcome quite yet.

01:40 You always have so much cool stuff going on and so many interesting perspectives.

01:45 So.

01:45 Some would call them interesting, whether that's a positive or negative is still open to interpretation.

01:51 May you program in interesting times as the old ancient proverb goes.

01:56 So been here a bunch of times, but real quick, just tell people about bit about yourself before we jump in.

02:01 Just in case they haven't, they don't know you.

02:02 Yeah.

02:03 name's Brett Cannon.

02:04 I live in Vancouver on the unceded territories of the Squamish, Tsleil-Waututh and West Green First Nations.

02:10 I am currently the dev manager for the Python extension in VS Code.

02:14 I am serving my fifth and as of now final term on the Python steering council.

02:20 I'm stepping down just because when we first created the steering council, I said five years seemed like a good term limit, we actually don't have term limits, but figured I'd stick to my word.

02:29 And I'm also the social core developer.

02:32 I've been doing it for just over 20 years.

02:35 Lucky for me, I got it my first commit bit right after the, sorry, my first commit bit after the first PyCon.

02:40 So I get to use the PyCon anniversaries as a way to keep track of that.

02:43 So that's convenient.

02:44 That's kind of the biggies.

02:46 I have a cat.

02:46 Yeah.

02:47 And that's usually the other thing people like.

02:48 Something about VS Code.

02:49 I've heard that word.

02:50 It's an editor, right?

02:51 Yeah, I know.

02:52 Someday you'll say you use it and I'll be very happy that day.

02:55 I had it open today.

02:57 Oh, all right.

02:57 Well, there I'm happy today then.

02:59 Yes, indeed.

03:00 No, awesome.

03:01 So many cool things you got going on.

03:03 And despite all of that background in steering council core dev here, this time you're here to give us like diet advice, eating advice, like something about sugar, what is it?

03:15 Yeah.

03:16 Just like story behind all this.

03:18 My wife was taking the certificate in data science course at the university of British Columbia, which is actually our Alma mater and where we met.

03:26 And she was doing her homework in a JupyterLab notebook through the browser via JupyterHub and she stopped doing the homework to go, I can't remember if it was lunch or dinner, but basically stepped away for like an hour or so.

03:40 And then she came back and the run buttons on the cells weren't working anymore.

03:44 I was like, well, what's going on?

03:45 And I looked and it's like, oh, well you got disconnected from the server.

03:48 And she said, well, isn't that happening in the browser?

03:51 Like, no, Python doesn't run in the browser right now.

03:54 It's not a, it's not a thing.

03:55 Oh, okay.

03:56 And then I think about a day or two later after she thought about it, we were in the car and I still remember the exact location in Vancouver where this occurred.

04:04 My wife said, you know, you should probably fix that problem of Python not working in the browser.

04:08 And I explained like what WebAssembly was and how big of a thing that would be to make that all happen and all that.

04:13 And she literally took about a beat to wait and said, no, you should go fix that.

04:18 Part of that led me down the road of trying to figure out what the minimum viable Python was, right?

04:24 Like what is the minimal amount of Python I would have to implement in the browser or in, or in WebAssembly in this case to make Python work.

04:32 And I had realized that, you know, a decent amount of Python is actually what's called syntactic sugar, which is really just syntax that you could actually devolve and unravel into other syntax of a language and have it do the exact same thing.

04:47 Right.

04:47 It's just basically syntax as a shortcut.

04:49 And so I ended up going on this very long journey unexpectedly long of trying to go through all of Python syntax and trying to figure out which parts of the syntax was like crucial and you couldn't actually re implement it in pure Python itself and which parts you could actually just make it work with Python code itself with the idea that at the end it would have basically a list of the bits of Python that an interpreter would have to implement and the other bits of syntax where you could run some tool over it ahead of time, it would just translate from Python to different Python code.

05:21 And then that way I would just know what the exact target of would have to be.

05:25 Might not run very fast.

05:26 Don't get me wrong, but it would at least be very clear definition of kind of what you have to have to really call something Python.

05:33 It's everything else you have to just work around.

05:36 Right.

05:36 Yeah.

05:36 So the standard Python syntax we expect still works, but how much do you have to work behind the scenes to make that happen?

05:43 So examples that come to mind for me are context managers, the width block.

05:47 Awesome.

05:48 I love them, but we have try finally, and you could simulate a context manager effectively with maybe a try finally, maybe an accepts in there as well, but you don't actually need with in the language to accomplish what it does.

06:02 Right?

06:02 Yeah.

06:03 Decorators are also another really good example, right?

06:05 Like where the concepts for that syntax predates it.

06:09 And it happened to just be something where Python core development team looked at where people were in terms of development and what they needed from Python and realize, you know what, that common pattern that people are doing would actually be, but would be very well served via piece of syntax.

06:27 So why don't we just do that?

06:28 Why don't we introduce that piece of syntax, make these steps.

06:30 And as you said, with context managers were a perfect example, right?

06:33 Where before, if you wanted to do the right thing, when you opened a file, is you call, call the open function for your file, assign it to a variable, do your try block of everything you want to do with the file.

06:46 And then in the finally part of the try finally close that file, which I mean, if you just think about it compared to a with statement is four lines, assuming you're doing reasonable formatting for the assignment, the try, the finally, then the close call versus the single with open to get the exact same effect.

07:04 And then you've also got the possibility that the context manager, the thing that's used in the with block can do something different if there's an exception, right?

07:12 Like if there's an exception, it could say, if it's a database transaction, it could roll back the transaction automatically.

07:17 Rather, if there's no exception, it could commit it automatically.

07:19 But there's no real magic there.

07:21 If you think about it in the end, you could have written that all up by hand.

07:23 It just would have been cumbersome, but that makes the with statement very much a piece of syntactic sugar, where you could totally read it out, get the exact same semantic outcome.

07:32 It might be a little slower.

07:33 Probably similar performance as well.

07:35 And maybe, yeah, similar.

07:37 It really depends on the syntax, right?

07:39 Like how complicated it is, how much of it's going to call into, at least in CPython's case, into the C code and stay in the C code versus not like with context managers probably won't just because the point is you're going to call that dunder enter method or that dunder exit method.

07:53 So you're calling Python code.

07:54 So no real, probably big shift.

07:56 But for stuff that we'll probably cover in this podcast, there are other bits that go into the C code and stay in the C code.

08:03 And when that happens, stuff can go really, really fast.

08:05 Yeah, especially with all the work, the faster CPython teams done around stuff and all that.

08:09 So the point is you can just start to just kind of leap into C code, stay there and get stuff done way faster.

08:14 Comparative to writing out by hand where you're staying in the Python world, it can just take a bit longer.

08:19 Yeah.

08:19 So the word sugar, I think it's a good one here.

08:23 I love like these ideas, like code smells, code sugar, all the things.

08:27 It makes it sound a little frivolous, maybe like little, ah, it's, it's nice to have it sweetens it up.

08:34 They don't really need it.

08:34 But I do think having these different constructs, like when you see a with block, it instantly conveys meaning.

08:41 It means we're going to create a thing.

08:43 We have a cleanup section.

08:44 We don't have to deal with it.

08:45 As opposed to if you see a try, except finally you have to read it and process it and see, okay, does it actually do the things that would be accomplished?

08:53 I think these, these sugars let you also think not just write less code, but think in different levels of abstraction.

09:00 Yeah.

09:00 I mean, as you said, the use of the word sugar is meant to mean something sweet, something nice.

09:05 It's a treat.

09:06 It's a bonus.

09:07 So from the perspective of if you didn't have it, you could still accomplish it.

09:11 But that can potentially in a way, as you, as you've suggested, not quite denote the benefit of it, right?

09:19 Like if we think of sugar in terms of not something good for your diet.

09:23 Yeah.

09:23 What, what is your perspective on sugar?

09:24 Right.

09:25 Are you a fan of desserts or are you trying to cut it out?

09:27 Yeah.

09:27 But yeah, I mean, the hope is whenever we add syntax to Python, it's for everyone's general benefit, right?

09:34 This is why we don't do it very often.

09:35 And we do, when we do do it, it's a very thoughtful discussion, very long discussion, sometimes contentious discussion.

09:41 Walrus operator is the famous one for that.

09:45 I don't understand the contention, the Walrus operator.

09:47 It's such a small change to the language.

09:50 There's so many other things that are bigger changes that could have been battles, but I guess it just came to a head at that point or something.

09:56 Right.

09:56 For those of you who don't know the Walrus operators, the thing that pushed Guido kind of over the edge to finally step down as being the bit of a dictator for life.

10:04 And it was just due to the veracity of the opposition to it and just how basically angry it made people.

10:11 And you're right.

10:13 It's very small.

10:13 And to be honest, I've used it and I like it.

10:16 I get use out of it.

10:17 It means it, I use it for its purpose, which was to add some basic syntax for those, those situations where you always have that assignment and then you do something like an F or something, and then you're going to do something that was like, just be able to inline certain things to just do the right thing.

10:32 It's not a huge thing, but it's a nice thing.

10:35 And in that case, I don't know, honestly, a lot of people got really concerned that they didn't see the usefulness enough that they thought it was going to be a slippery slope or people are always very concerned about Python getting too big and that's no longer fitting in their brain.

10:53 Yeah.

10:53 Maybe it was kind of the last straw sort of thing, rather than like, maybe stuff came before and they're like, all right, that's it either.

10:58 And now the walrus, the walrus, like, you know, I don't know.

11:02 I don't know.

11:03 I mean, I don't think people realize that we actually cut syntax out when we moved from Python two to Python three.

11:07 Yeah.

11:08 That was all kind of brewing.

11:09 Yeah.

11:09 Like the back ticks went away.

11:11 Like no one knows about the back ticks anymore, but we still have syntax for using back ticks and that's now that's gone now, so it's not like it's always been a add, add, add, but I think it's just one of these things where it's just people were very concerned that Python would get too big, too complicated, and it wouldn't be fully understandable.

11:27 I will say, whenever we do this, we very much consider how easy it would be to understand if you saw it for the first time and never received any instruction about it, right?

11:35 Like I would argue that walrus operator is a bit obvious if you look at the code around it of what it does.

11:41 And so you can probably figure out what it does without being told ahead of time versus other syntax and other languages where you go, I have no clue what this does.

11:49 Right.

11:49 Exactly.

11:50 Right.

11:50 Like why does C++ and C do different things if it's in the front or the back and what's the difference?

11:56 Like, unless you know, prefix versus suffix increment and the difference in terms of referencing the pointer and all this crazy stuff, like you're not going to say pick up on the difference unless you run your code versus something where you can just look at it and just learn on the spot.

12:09 Well, I want to give you all a bit of a compliment, I guess, or encouragement.

12:14 I think the Python language is not changing too fast.

12:18 I think it's the stability over time is really good.

12:21 You compare that to other languages.

12:23 I mean, here's one that's kind of close to home for you since you're at Microsoft, like C#.

12:27 I feel like that language, pretty nice, at least in the early days.

12:30 And now it's, there's a whole team who are employed.

12:33 You don't have to make any comment.

12:35 I know you work there, but there's a whole team who's employed to, to like shepherd that language.

12:41 And I feel like there's a feeling they always need to have some new language features because it's their job to have language features.

12:47 There's just like a constant kind of turn, like for properties, like we have properties, they have properties, but I think there's four to five different ways that properties in C# that have evolved over time.

12:57 And you're just like, yeah, it's a little better, but it's why do we have so many ways if you're going to be that right.

13:02 And so when I at least look at Python compared to like my background in C++ or in C# or JavaScript, I feel like it's comfortable.

13:11 It's not stale, but it's not, we don't have five ways to do properties.

13:16 That's a good thing.

13:17 Even if there's a better way to do it, it might not be worth having two to three ways just for that, for a slight advantage.

13:23 Well, thank you for the kind words.

13:24 Yeah.

13:25 I mean, we do have the Zen of Python does kind of outline the general guidelines we try to follow when doing, making these discussions decisions for those who've never read the Zen of Python, look, just run, import this from the rep bowl and you'll, you'll find it.

13:38 And yeah, it's just the way it's designed, right?

13:40 We don't, hopefully there's one in hopefully only one obvious way to do anything, and so it's just something we've kind of leaned into now, granted things shift, like I'm sure someone will point out the myriad of ways you can format a string now, but a lot of that's backwards compatibility and just realizing, you know what, we just have to accept the fact that the first way we got it is no longer the best way we can think of.

14:02 And it's enough of a jump from using string interpolation with the percent sign to using f-strings that it was worth adding a second way.

14:11 Now maybe someday we'll take out string interpolation.

14:14 I doubt it because it just works and it's just sitting there and it's not worth breaking really old code, but I mean, it's just the way we approach things and luckily most people like it.

14:22 And I think that's kind of why some people always freak out is.

14:24 The words are afraid somehow we're going to go downhill from here and it's just the start of things or somehow Python is going to be harder to learn.

14:31 Well, we very much try to make sure Python is a gradual curve of learning, right?

14:35 Yeah.

14:35 The basics are very straightforward and good.

14:37 And as you get into more advanced things like the match statement, right?

14:40 Like you'll be able to see it and understand it and it'll make sense, but not something you'll necessarily learn the first month or two, but it's there for when you do need it, right?

14:48 Like we very much try to make sure that everything makes sense on a curve for from beginner to advanced and try to make sure the language is useful to everyone from both a beginner to advanced user.

14:57 It's tricky, but like to think we're doing a decent job.

15:00 Yeah.

15:01 One of Python's really, one of its powers is that you can be super effective with it, with a really partial understanding of what it is, right?

15:08 You don't have to understand namespaces, classes, async, et cetera, et cetera.

15:12 You just like, well, I think I can write these four lines in a, without even a function and I'll get a cool graph that'll show my, my work or whatever.

15:18 Right.

15:18 Yep.

15:19 Exactly.

15:19 Yep.

15:19 And I think I have a pretty partial understanding of the match statement.

15:23 I know it can get pretty advanced in like how it captures and changes values, but I wrote one yesterday with a partial understanding.

15:29 It went great.

15:29 Nice.

15:30 Yeah, indeed.

15:31 All right.

15:31 Before we move on though, got to go back to the, the, your opening, because you said, I created this to understand, like if I were to solve that problem, you know, I had about like, you got to fix that thing about the browser.

15:44 And I know there's some folks in Texas who have thoughts on this as well and the PyScript team, but is this project you're actively working on or just, let me gather up, do some research and deep thinking about what is actually in the language that would have to carry over or where's this coming from, I guess.

16:01 No, it's not an active project of mine at the moment.

16:04 It ended up being mostly just an exercise and just snowballed into a thing where I just didn't want to stop until I completed it.

16:11 Yeah.

16:12 How do you feel about the, the MicroPython PyScript angle of approaching this?

16:18 Yeah.

16:18 I mean, I think it's great that, I mean, that was the other hopeful benefit of all of this is trying to nail down the definition of what Python is.

16:26 Yeah.

16:27 Cause I've heard those arguments made against MicroPython that it's not quite Python cause it doesn't quite run all the syntax or it doesn't have the entire standard library.

16:36 I've gotten into arguments with people online where they got very upset when I suggested that the REPL was an optional part of Python and actually not a requirement.

16:45 And I just viewed it as a learning experience for me, hopefully a learning experience for others as well.

16:51 And then after that was just trying to hopefully clarify like, like, look, this is truly what you have to have to really beat Python.

16:57 You literally cannot implement the rest of the language without this, but everything else, you could totally just do it yourself.

17:02 You could totally just write your tool and make it all work.

17:05 And as long as this existed, you could totally get a buy with it.

17:08 And so that's kind of where I ended up.

17:10 I luckily the tooling around WebAssembly got far enough along.

17:14 Pyodides continue to do well in the browser.

17:16 I've been working on the WebAssembly, the system interface, Wazzy side of things, and that's been going pretty well.

17:22 So that's at least taking care of itself.

17:24 So I've ultimately not tackled the huge problem of trying to re-implement Python from scratch, probably in Rust, just in a minimal fashion to try to make it work better in the browser.

17:36 That's probably a project for when I retire.

17:39 And I suspect that's at least a decade out, if not more.

17:42 So that's probably going to be put on the back burner for a while.

17:44 Right.

17:45 Yeah.

17:45 We can all install B Python and run Brett Python built in Rust.

17:49 There you go.

17:49 It, Rust does seem to be having quite the influence, an outsized influence for how popular Rust is in terms of the number of developers, but it seems to be really leaving its mark on Python and other areas as well these days.

18:05 So.

18:05 Yeah.

18:06 Well, I mean, I will fully admit I have bias cause I am a fan and I do like using the language, but it's one of those things where if you've been living in the CNC++ world and it never was that satisfying to you, but you had to be there for whatever reason, and then you discover Rust and it does things that saves you from very chronic set of problems.

18:24 Yeah.

18:25 Including yourself.

18:25 Yeah.

18:26 You would hope that you, people would notice and latch onto it.

18:29 And I mean, I think that's basically what's happened is people have just discovered that, oh yeah, this is useful and good and just there's enough CNC++ code in the Python community due to CPython C API that it, I think they're just a nice cross-section, right?

18:44 I think it's just one of these things where people have just gone like, well, I've got to be, I got to do the system level programming anyway, so I might as well use something that's seems like a better, more modern language.

18:53 Yeah.

18:53 Yeah.

18:53 Yeah.

18:53 Yeah.

18:54 More modern.

18:54 And so I think that's kind of what's happened here.

18:56 Yeah.

18:56 That makes sense.

18:57 All right.

18:58 Let's talk about the sugar.

18:59 Okay.

18:59 So I think we covered pretty much what syntactic sugar is.

19:02 And you started this whole thing out saying the goal of this is really to kind of nail down that minimum Python.

19:09 And I agree that the REPL is optional.

19:11 I like the REPL, but just, just so I can also get mail about it.

19:16 I like the language.

19:17 Like you can't take away these features once you add them.

19:21 Right.

19:21 And so it's like, what could you survive with as a minimum?

19:25 And then you can kind of like, what are the, what's the extras?

19:28 What could you build up from there?

19:29 Yes, exactly.

19:30 And the final destination may be being, you know, CPython proper.

19:34 Yeah.

19:34 So you've got a whole bunch of different aspects of Python that you are unraveling in this syntactic sugar.

19:43 And basically to highlight what's actually happening, how much of this is, is going on as you've already laid out.

19:48 So I want to point out that I'll link to how many articles you've got here, but two years, like 20, I think.

19:55 Yeah.

19:55 Like looking like close to 30, maybe.

19:57 I don't know.

19:58 Just eyeball on it.

19:59 There's a lot there.

19:59 So I think I've picked us, you know, half of those hours, some, some reasonable subset that we can talk about and there's some code and stuff in here.

20:08 So we'll kind of have like, my thought was we could just talk about kind of like, what is the essence of this?

20:13 Like what's really happening when you do attribute access, what's really happening when you write the word pass or use a, a with block and we've talked about a little bit.

20:22 So the first one here is unraveling attribute access in Python.

20:26 You want to tell us a bit about what actually happens if you just say, you know, like object dot attribute, it seems so simple, Brett, there's just one dot.

20:35 What's the big deal?

20:36 Yeah.

20:36 Why doesn't this just, what's there to talk about?

20:38 I mean, when I just do something dot something magically, I get back what I expected, but there's actually a lot going on there, right.

20:46 And it, it comes down to multiple layers, but it's all ties around Python's object model, right?

20:51 How do all, cause I always find this weird to say, but Python's more object oriented, my opinion in Java, right?

20:58 Java has these primitive types, like in quotes that have to get boxed and unboxed to fake to make them look like an object to Java itself.

21:06 Python doesn't have that.

21:07 Everything is an object.

21:08 A function is an object in integers and object like true is an object.

21:12 Seven is an object.

21:13 Exactly.

21:14 There's nothing you can name in Python.

21:16 That's not somehow an object.

21:17 This idea of attributes and all this permeates all the way down into the object model, which once again, defines kind of everything in a way.

21:24 So the key thing here is there's kind of two parts to it.

21:27 It's how do you look up based on inheritance?

21:30 What thing is going to service the call asking what is this attribute value?

21:36 Basically what object is bound to this name for the attribute.

21:39 And after that is what methods are going to get called to make that happen?

21:43 Cause effectively almost everything in Python at the end of the day leads to a method call or a function call.

21:49 Yeah.

21:49 First of all, it leads to a lot of times to an op code, which leads to a method call, right?

21:55 A Python byte code thing.

21:57 And I think it's maybe also worth pointing out, like have you point out to everyone listening is Python compiles to this intermediate language like Java and .NET compiled to intermediate languages.

22:08 It's just what happens then, right?

22:11 Like in those other ones, they, they jit compile to machine instructions.

22:14 Here it goes through like the big C eval loop and delegates to some internal C operations.

22:19 And a lot of those, these unravelings, like kind of look at what happens at that step, right?

22:23 Yeah.

22:24 So CPython itself is an interpreter, right?

22:27 So what happens is when you load up your Python code is Python parses it.

22:32 What's called an abstract syntax tree or AST.

22:34 It then takes that AST does some stuff with it, and then it compiles it down to Python byte code.

22:41 I actually gave a talk on this at PyCon Canada way back when, for those who are interested, that basically goes from the steps of literally source code all the way to execution if you want more detail, but effectively, yeah, we compile it down to byte code of our own design to then execute in more or less what's a big for loop in C.

23:01 That is a really big for loop.

23:03 Very big for loop.

23:03 And it's gotten a bit fancier thanks to the faster CPython team, right?

23:08 Like there's now kind of like lower level op codes that are very type specific, and we actually auto-generate the loop now thanks to them so that it's easier to maintain and make tweaks to.

23:18 But essentially it's literally just a set of instructions that are extreme, that kind of encompass key bits of semantics in Python.

23:27 So for instance, for attribute access, there's literally a op code called load adder where on the execution stack, you basically push the object you care about, the name of the thing you want, and then you just call load adder on it and it'll just look it up.

23:42 Actually, technically, I guess it doesn't push the name on it.

23:44 That's technically stored in the function.

23:47 There's little details on kind of hand waving over, literally hand waving.

23:51 If you're watching the live stream, but effectively you push the object you want, you call load adder, give it an argument of what it should, what attribute it wants, and then load adder op code behind the scenes is the thing that does all the magic of calling everything you'd expect, et cetera, et cetera.

24:04 Yeah.

24:04 Excellent.

24:05 And to view that a lot of times what you'll do is you'll just use the dis module.

24:10 So people haven't used dis, you can just dis.dis, a lot of disrespect on this function, and then it tells you, it tells you what the op codes are that it's made of, or yeah.

24:19 Yeah, exactly.

24:19 It'll, I mean, dis is short for disassemble.

24:21 So literally you, you can pass it a function and it will effectively disassemble what the function does into the byte code, which is just sort of as an attribute of bytes on the function object and print it out nicely because it, the dis module knows how to map the bytes to what the op code's name is and just gives it a nice way to look at and actually read and understand it.

24:42 Yeah.

24:43 Okay.

24:43 And this example, you said, I look, here's the case statement in that tremendous for loop switch thing that does the execution.

24:51 What do we do if we slow C load adder?

24:53 And ultimately it comes down to this high object underscore get adder.

24:58 So it seems so simple.

25:00 You just object.attribute object.field, whatever.

25:03 But it turns out there's just so many variations and so much going on down here, and I don't necessarily want you to go through all the details cause it's pretty, some of it's pretty intense C and like a lot of it, you know, optimizations and stuff, but like just give people a sense of what actually happens down, down in the guts when you try to do that.

25:21 Well, so the, so in the case of attribute access, right, that it effectively ends up calling the guilt get adder built-in function.

25:29 Right.

25:29 And in that case, it needs to check how many arguments it gets to start, right?

25:34 Did you give it two arguments or did you give it three arguments?

25:37 So you return a default value if the attribute doesn't exist.

25:40 And then at the C level, we have to check whether or not what you gave it, right?

25:44 Because there's like a Python level, the C code, all types to what's called the two Py object, which is basically a massive struct in C that represents any and all Python objects.

25:55 So even at the C level, we don't know if you gave us a string or an integer or what did you give us for the name of the attribute that you want?

26:05 So we got to do that.

26:07 Right.

26:07 And so there's just basic, like just safety checks to go like, Nope.

26:12 Like when you, if you call get adder with you wanting the attribute 42, which is a number, it's going to throw a type error.

26:19 Well, that code's got to exist somewhere and this is where it exists.

26:22 Yeah.

26:23 Yeah.

26:23 Right.

26:24 And then effectively just bubbles all down to the C API where a lot of this stuff gets exposed in its own way.

26:29 And specifically in this case, it ends up calling Py objects, get adder.

26:33 Yeah.

26:33 And I think another thing that's pretty interesting here is it's not always just a value, right?

26:38 That object.adder, that adder could be a property.

26:41 That adder could be a descriptor, which I guess is a generalization of a property.

26:46 It could be on the class type, but not on the instance type.

26:49 So you got to find that it could be on the instant type overriding, but only there.

26:52 Right.

26:52 Like it could not exist.

26:54 All of these scenarios that you got to go through.

26:57 And so it really, you know, it really emphasizes how much is actually going on in the runtime while this single simple line seems to be running.

27:05 Right.

27:06 Yeah.

27:06 I mean, in the simple case, hitting the dots, not complicated, like in general, what you end up doing is you have your object, you hit dot.

27:15 And what it does is it looks in the Dunder decked attribute of the object.

27:19 And there's a key matching that to rename and just gets the thing in the dictionary and that's it.

27:23 You're done.

27:24 But as you alluded to, how the heck do you even get to that basic case or, or those more complicated cases, as you said, descriptors, right.

27:32 Which are how we actually implement properties or if it's on the class versus the instance and if there's inheritance or multiple, multiple inheritance or Dunder get at, or was defined, right?

27:45 Like basically almost all of this ends up flowing through the get outer built in, which effectively ends up calling the appropriate Dunder get attribute method off of an object.

27:55 And that this is one of those barrier level kind of things, right?

27:58 Where you have to implement this to make Python work.

28:01 You have to basically implement the, the get outer built in.

28:03 And it has to understand how objects are structured underneath the hood to know how basically methods and stuff are attached to an object.

28:10 You really can't fake that.

28:12 There's no way to not kind of have that as a low level detail that you can just unravel.

28:17 It has to be implemented by the interpreter.

28:19 And in this case, effectively the way Python does things is it knows the order.

28:24 You can always actually call the MRO method on any object in order to return you the method resolution order.

28:29 And effectively what Python is doing is if it checks on something to see if it has it or not, it will effectively end up calling the Dunder get attribute object of method, sorry, on object.

28:41 That's usually where it bottoms out.

28:42 And unless you happen to have a Dunder get outer, if that fails, but that's only the failure case, but really in the end, almost everything passes through object dot Dunder get attribute.

28:52 And that's really where all this ends up going.

28:54 But as you said, there's a lot of little twists and turns.

28:56 Yeah.

28:56 A lot of these internals, you'll, you'll see the method resolution order calls and it's all about, well, there's a lot of classes and a lot of objects.

29:05 They can override things and they can override operators.

29:08 And so it's, there's a lot of navigation to just figure it out.

29:11 Not just calling these get outer type of things, but what level does that even supposed to happen on given what I'm working with, right?

29:20 Yeah, exactly.

29:21 And I think this is a good example.

29:23 And the reason I started with is it's so fundamental, right?

29:26 Like how do you unravel something if you don't know how to even get an attribute off of something?

29:31 So this is why this was the very first post in this series.

29:33 And partly why I went into such detail on how to figure all this out, right?

29:37 Like kind of one, this is almost a template for anyone else who wanted to go exploring on their own.

29:41 But I think it's also a good example of how much complexity and flexibility Python hides from users in order to make the simple work, but give you the flexibility to make the complicated possible, right?

29:53 Yeah.

29:54 Like for the vast majority of any of us for our code, it's not just as what you would think it would do.

30:00 And it's pretty straightforward.

30:01 But as soon as you get into the fancy world of descriptors or dendrogrammable attributes.

30:06 Yeah, SQLAlchemy.

30:07 I can both have a value or I can do a database query with the same thing.

30:10 How is that possible?

30:11 Right?

30:11 Yeah.

30:12 You want that magic?

30:13 Language has to support it somehow.

30:14 And it requires a lot of finesse and thinking through and a lot of mechanisms that generally get hidden from you, but someone's got to write that code somewhere.

30:24 And in this case, it's CPython and this kind of outlines how that all happens.

30:30 And it's surprisingly complicated.

30:32 It's not horrible.

30:33 That was my main takeaway too.

30:35 Yeah.

30:36 I mean, you can totally read it and follow it.

30:37 I don't want to scare anyone saying it's so complex.

30:39 You can't follow it.

30:40 It's just, it's not just a simple like two if statements and you're done.

30:45 It's no, there's subtlety to it to make sure the common, the common semantics makes sense to people.

30:51 It's not going to trip people up because it's doing something weird.

30:54 There's a lot of thought that goes into these semantics to make sure it makes sense for the common case.

31:00 This portion of talk Python to me is brought to you by us.

31:03 Have you heard that Python is not good for concurrent programming problems?

31:07 Whoever told you that is living in the past because it's prime time for Python's asynchronous features.

31:13 With the widespread adoption of async methods and the async and await keywords, Python's ecosystem has a ton of new and exciting frameworks based on async and await.

31:22 That's why we created a course for anyone who wants to learn all of Python's async capabilities.

31:28 Async techniques and examples in Python.

31:30 Just visit talkpython.fm/async and watch the intro video to see if this course is for you.

31:36 It's only $49 and you own it forever.

31:38 No subscriptions.

31:39 And there are discounts for teams as well.

31:42 Another takeaway I got from looking at this is wow.

31:47 It's nice that dictionaries are fast and how fast are they to make this possible to lean on them so, so much to make this happen.

31:53 Yeah.

31:54 I mean, I will say when Guido chose to use dictionaries as the name of dictionaries as the namespace, it was kind of a unique decision at the time.

32:01 It might still be, honestly, but it did simplify some things conceptually and also made it so that any wins in how dictionaries work is just a massive win universally across the language.

32:15 Right?

32:15 Like if you make dictionaries work in any way better, you not only make it work when you just, you personally use it as a data structure, as a container, but also literally every attribute access will get faster.

32:28 Right?

32:28 Like there's a reason why it's so fine tuned and why we don't really touch it very much because it's been tweaked over the decades by a lot of people to be extremely fast and for good reason.

32:41 Yep.

32:41 Indeed.

32:41 One final thought on this, you know, you think of optimizing both the memory and also the performance a little bit by doing things like slots on your classes to maybe like short circuit some of this.

32:51 It feels like somewhere in this world lives like some kind of massive optimization that maybe that if you said I'm going to, for this class, I'm going to give up some flexibility.

33:01 Is there some way attribute reads and writes could get lots faster?

33:05 I don't know.

33:05 What are you, I'm sure this has gone around and around because it's such a hot, it's just involved in almost every line of Python code and stuff like this.

33:13 Right?

33:13 Oh yeah.

33:14 I mean, and this plays into what the faster CPython team has been doing, right?

33:18 Like, how do you make things faster by looking at what the patterns are?

33:23 And if you can just kind of skip stuff, right?

33:26 Like how do you short circuit this thing?

33:27 Can you go like, well, I just know this objects always look this way.

33:30 Like this was some of the stuff that the self programming language team at UC Santa Barbara did, I think in the seventies, maybe eighties of just like, well, if we know the layout of the object is going to be consistent.

33:43 Yeah.

33:43 Can we go seven bytes in and get four bytes?

33:45 That kind of thing, right?

33:46 Exactly.

33:47 And you start to be able to calculate their shortcuts, you know, like, Hey, did anything change from the last time I made this assumption?

33:54 No.

33:55 Okay.

33:55 Well then I can use my assumption.

33:56 That's way fast.

33:57 Right.

33:57 Nobody's mocked it.

33:59 Nobody's dynamically messed with it.

34:01 Like maybe it is just a plain thing until somebody puts it into an edge case.

34:05 Is there a short circuit?

34:07 Right.

34:07 Yeah.

34:08 And I mean, I don't want to go off on a tangent, but.

34:11 Well, I'm one of 17 topics here, Brett.

34:14 Like, no problem.

34:15 We got lots of time.

34:16 I have thought about like, after doing this, what is there something syntactic here to add, right?

34:22 Is there a way to kind of come up with a effectively a class definition?

34:28 That's way simpler for the really common cases that could be optimized extremely fast because you give away some flexibility and then the interpreter can make assumptions.

34:41 PyPI can make assumptions, et cetera, et cetera.

34:44 And would there be enough wins on that to warrant doing it right?

34:48 Like, could we take the concept of kind of a data class that was just data roughly or something along those lines and just what could we get out of it?

34:57 And what kind of perf wins could we get out of it?

34:59 And would it be worth it?

35:00 Would it be worth adding syntax to the language for that case?

35:04 If you look at Mojo, for instance, they have a struct and that struct works in a very specific way for performance, because when you say like this thing can't have new attributes, which you can do with slots.

35:15 And for those of you who don't know what slots are, if you define a thunder slots attribute on your class, you're effectively telling Python, don't create a dictionary for me.

35:23 Just create effectively a list underneath it's technically a C array.

35:27 And when I need to access something, just get it off this array.

35:31 So there's no dictionary.

35:32 You can skip all that dictionary overhead.

35:34 Plus you shrink the memory because now only amount of memory you need is that array for pointers to the PI objects and not a dictionary, which grows and shrinks.

35:43 And who knows what size is going to be.

35:44 If we had that built in syntactically, would that be beneficial to people or not?

35:49 And I have a prototype in Python code of what I'm thinking of.

35:53 I did a blog post on this earlier this year to try to feel out people in the community and what they thought in terms of restrictions and would that be too much or too little?

36:02 I subsequently prototyped what I wanted.

36:04 And there'll be a followup blog post about this is where it sits now.

36:08 What do people think?

36:09 And do people think this is worth trying to get syntax for just don't worry about it's too close to data classes or what?

36:16 Yeah.

36:16 Maybe it could be a keyword to the data class decorator that just makes it behave differently.

36:21 Who knows?

36:22 Yeah.

36:22 But so unfortunately the tricky bit with that is it's still flowing through Python code, right?

36:26 Data classes is still just a module in the standard library.

36:28 So you can't, it's really hard to optimize that stuff without syntax to specifically say, Hey, this will be different.

36:35 You get to treat it differently and we can literally lock it down by literally creating a different type of object at the C level that just literally doesn't give you access.

36:43 Right?

36:43 Like tuples are a good example.

36:45 Tuples at the C level very much just don't implement the stuff that you mutate it.

36:50 Right.

36:50 What happens if we want to do that for classes?

36:52 Like how do you do frozen data classes?

36:54 It's a ton of properties.

36:55 But what happens if we had a way to just say, yep, nope.

36:58 We're just literally not giving you access.

36:59 Would that, how much faster could that be?

37:01 I, I don't know.

37:02 But I suspect it would be decently faster.

37:04 Yeah.

37:05 Computers are surprisingly fast.

37:07 Exchangers are surprisingly fast, but there's a lot of stuff here that could be skipped and I think might, might be, to me, I had the same thought at least.

37:15 Probably not with the level of sophistication you did, but like, wow, there's some way to sort of do a more of a, a C style, just like we know the structure and the offsets like for a restricted set, because that restricted set is mostly what people do actually do in Python.

37:28 They don't go crazy dynamic with stuff coming and going like sometimes, but not usually.

37:33 Yeah.

37:34 I mean, you can go read my original blog posts and if you follow me on master, you can see the follow up conversations, but like the original thing, if you've been a long time Java user, you'll, you'll know the acronym POJO, plain old Java object.

37:47 It was effectively the equivalent for Python.

37:49 Some people love that idea.

37:51 Some people like, I really need methods.

37:53 You, I can't not have methods kind of thing.

37:56 It really depends on what people are after.

37:58 And what, what they think makes sense, but there will be a follow up blog post.

38:03 So if you follow my blog, you'll eventually see what this hopefully all leads to, which may be nowhere beyond what I put up on PyPI, but who knows, maybe, maybe it'll become Zendak someday.

38:11 It really depends on how the community reacts to it.

38:14 Cool.

38:14 Well, I didn't know you were looking into that.

38:17 So very interesting.

38:17 Next one.

38:18 Let's move on to something that also seems pretty straightforward is A minus B.

38:23 Yeah, right.

38:26 Yeah, exactly.

38:27 You think it makes, yeah.

38:28 How hard can A minus B be right?

38:31 Like two objects just subtracting from one another, right?

38:35 We all know how this works.

38:36 There's, I mean, there's an op code just called binary subtract.

38:39 Just like there's a binary add a binary multiply.

38:41 You put the numbers on the register, you do the instruction set on the CPU and you're good to go.

38:48 Yeah.

38:48 Hey, I know what five minus three is.

38:50 How much work can it be to make that work in Python?

38:51 Oh yeah.

38:52 The numbers happen to be arbitrarily large and sometimes they're not numbers.

38:57 Sometimes they're matrices or strings or oh boy.

38:59 Okay.

39:00 Exactly.

39:00 And subtraction is not communicative.

39:02 So, or matters here.

39:04 So yeah, it gets surprisingly complicated very quickly.

39:07 Yeah.

39:07 So yeah, from, from a conceptual level.

39:10 Yeah.

39:10 It seems simple, but when you figure out what that op code actually does, it becomes way fancier.

39:14 Yeah.

39:15 And so as you dig into this, you start to realize like, okay, so this is actually controlled by the Dunder sub operator on the type.

39:23 Right.

39:24 And then how do you get that method?

39:25 Well, this is the method resolution order on that type.

39:29 So you got to start navigating that and that gets pretty interesting.

39:32 Right?

39:33 Yeah.

39:33 Well, and let's be clear here.

39:35 You're a bit foreshadowing because you said, oh, you called the Dunder sub.

39:38 Yeah.

39:39 Just start if that works.

39:41 Right.

39:42 Like for a lot of these things, there's lots of fallback to once again, give you that flexibility to make things work as necessary.

39:49 So for a lot of things in Python, you can return the not implemented Singleton, which you may have, some people may have noticed and wonder what the hell is this thing for.

39:57 And it's a Singleton signal to Python that, Hey, there's not, it's not that an error happened.

40:03 It's just, I just don't know how to handle this.

40:05 Right.

40:06 So for instance, if let's say you have left-hand side minus right-hand side, the first step is you check to see if the left-hand side has a Dunder sub method.

40:17 If it does, you can call that with left-hand side dot Dunder sub and pass in right-hand side.

40:23 Well, that method has the option to do something in return to value.

40:28 And that's what left-hand side, right-hand side becomes, or it can return not implemented and say, just tell Python, Nope, I don't know how to do this.

40:35 Probably should ask someone else.

40:36 Right.

40:36 And then this adds an extra little wrinkle now, because suddenly when you do that, that says, okay, well, you know what, let's give the right-hand side to try.

40:45 Maybe the right-hand side knows how to do left-hand side minus right-hand side.

40:48 It might not be the front thing, but it might just magically know.

40:51 Right.

40:52 So in this case, for instance, you could have something of your fancy object minus four, which would be your Dunder sub method.

41:00 And so, you know how to handle integers and it works.

41:02 What happens if it's four minus your object?

41:05 Yeah.

41:06 Four doesn't know what to do.

41:07 Yeah.

41:07 Four has never seen your thing before.

41:09 I don't know what the heck to do here.

41:10 So in that case, four would return not implemented thing.

41:14 I don't know what to do here.

41:15 Good luck to you.

41:16 Maybe you'll figure it out.

41:17 And in this case, your object Dunder, our sub would get called with four.

41:22 And then maybe you'll be able to say like, Hey, I might not be on the left-hand side, but I know how to handle myself on the right-hand side and I can make this work.

41:28 So we try to make sure that even if some other objects don't know how to work with you, as if you're involved, there's at least the opportunity for you to make it work.

41:36 Yeah.

41:37 That's pretty interesting.

41:38 Yes.

41:38 An example could be like the number four, a constant minus a vector that's meant to be like a mathematical vector.

41:45 And it's like, okay, we just subtract that from every dimension.

41:47 Right.

41:47 And that would be totally reasonable, but the four has no idea how to do that.

41:51 Yes.

41:51 Yeah.

41:51 And this is still the simple case.

41:52 Yeah.

41:53 So if I'll need a ref, I need a slight refresher on this one, but effectively there's scenarios here where you call the right-hand side first.

42:08 If it's the left-hand side, one can say, I don't know what to do with this object, I think in return and not implemented from the subtract.

42:15 Right.

42:15 Something like that.

42:16 Right.

42:16 There's also a subtle optimization in these binary operators where if it is a direct subclass of the other object, you get first dibs.

42:28 Oh, right, right, right.

42:29 So that's the thing.

42:30 Yeah.

42:30 There's a lot of spam and bacon and vegetable spam and stuff that you did in here that people can check out.

42:36 It gets surprisingly complicated.

42:37 But the thing here, basically, this is the C of a left-hand side and a right-hand side, right?

42:42 Once again, left-hand side, right-hand side, left-hand side might be an integer.

42:46 Your right-hand side though, might be a subclass of integer, like fancy integer.

42:51 Okay.

42:51 Now, if you do int minus fancy int, it will understand how to do it because you're a subclass.

42:57 So it's just going to do it and it'll just do the math and return you whatever it is.

43:01 So like if you did five minus fancy three, five is going to return the integer two.

43:07 But maybe that's not what you want with your fancy int.

43:09 Maybe you always, maybe you'd rather get fancy int two.

43:12 So how does Python make sure that you get that opportunity if you happen to be on the right-hand side instead of the left-hand side?

43:17 Yeah.

43:17 Well, in that case, Python checks first.

43:20 Okay.

43:21 I have a left-hand side and right-hand side for subtraction.

43:24 Is the right-hand side a direct subclass of the left-hand side?

43:28 If that's true, then you know what?

43:29 I'm going to give the right-hand side a chance first so that they can return their subclass and not have the left-hand side kind of just blindly not realize that it's a different type and return something else.

43:40 So that when it's, as I said, five minus fancy three, we actually get fancy three the first chance instead of five so that you have the chance to return fancy two like you wanted and not have five self-return.

43:52 Boring two.

43:53 Yeah.

43:53 So I think one example that comes to mind that I think might be relevant would be working with units.

44:01 So we've got cool libraries like Pint and others where you can say, this is not just the number five, but this is five kilograms.

44:09 And if I subtract, you know, a hundred milligrams from it, it's not negative 95.

44:14 It's, you know, four point whatever, nine, five or whatever it turns out to be.

44:18 And so in this case, if you took a regular number and one of these numbers with units, you could get a number with units back potentially.

44:25 Yeah.

44:26 No, that's a very good example.

44:27 Right.

44:27 Yeah.

44:28 This is where those, once again, very simple syntax that in the common case and in your general use day to day makes total sense and happens has some really important, but subtle, if you don't know about them, semantics to make sure that those reasonable expectations are met.

44:44 Right.

44:44 Like I totally didn't even think about this when I was doing, when I was writing this blog post that like, oh yeah, God, that's a good point.

44:51 Like I do want to make sure that I get my special version of everything when I happen to be on the right hand side, just cause I happen to be written on the right hand side.

44:59 But plus is a really good example in this one, right?

45:01 Like you could have written a plus B or B plus a makes zero difference.

45:05 I mean, that operator is supposed to be communicative, so it can be totally just dumb luck based on how I just happened to be thinking that day, what I put on the left side and what I put on the right side.

45:13 And that would suddenly make a difference.

45:15 And in this case, it just helps make sure that what you would think is the priority order of who you'd want to handle this for you gets that priority.

45:24 But once again, it's a thing you just never think about day to day.

45:27 But in actuality, when you think about the really key ways that Python operates, makes a surprising difference.

45:33 Yeah, that seemingly leakiness of the language of like, well, sometimes it gives you the right answer, some it doesn't, depending on how you add.

45:39 People wouldn't love that.

45:40 Yeah. And then the last bit of subtlety on that one is this only happens if, by the way, dunder sub and dunder sub are different.

45:48 So that's another check we do, because at the C level, they're actually equivalent.

45:53 Like there is no left side versus right side thing here.

45:56 So we actually check to make sure we don't waste our time asking you, hey, can you do this and get back not not implemented and then calling it a reverse and getting the exact same result because underneath it all is the exact same method.

46:07 Right.

46:07 So it's one of those fun things of once again, lots of little things here to try to optimize, as you said, as best we can.

46:13 Where and in this case is one of those.

46:15 Well, if we know you're not going to work this way with the exact same method, we just swap the order of the stuff.

46:19 Why are we going to ask you again?

46:20 So we just skip that.

46:22 So it's a minor thing.

46:23 But yeah, semantics are there to try to help help with the performance.

46:26 I think people as people go through your series here, I think they're going to get a deep appreciation for edge cases.

46:33 Yeah.

46:33 Subtlety.

46:34 Yeah.

46:35 Yeah.

46:35 I mean, the last few days.

46:36 I learned a lot.

46:37 I mean, it was one of the reasons I finished it is I finished the series as well, even though I partway through and like, yeah, you know what, I'm not going to implement Python from scratch.

46:46 But I was learning enough and it was fun.

46:50 Like, I got to dive a bit deep.

46:52 I got to ask the Python core team is like from people primarily Guido, who obviously were there when these decisions were like, why this way?

46:59 Like, why do we choose this and just trying to understand somewhat historically, how did these things happen?

47:06 Right?

47:07 Like, I think a lot of people don't realize how old Python is, right?

47:11 Like February of 1991 is when Python came out.

47:14 It predates Linux going public.

47:16 Like people really forget that fact.

47:18 It's been around for over 32 years.

47:20 Yeah.

47:21 Decisions were made a long time ago.

47:23 And I know everyone got all upset over the two to three transition.

47:26 And some people still get upset whenever we push new changes or whatever.

47:29 But do realize, yeah, I can't believe you just canceled three seven.

47:33 I like that version.

47:33 But the key thing here is the core concepts still work.

47:39 Subtraction still works right from the way it worked back then.

47:43 There's not been a massive upheaval here.

47:45 It's just a lot of us end up using stuff that's kind of fancy.

47:48 And sometimes the fancy stuff gets a little tweaks and stuff.

47:51 But the really core low level stuff really hasn't shifted very much.

47:54 So if at all, depending on your people's sense of the timing, like the release of Python predates by a couple of years, the release of the web, right.

48:02 The web was 93, I think.

48:03 And so, yeah.

48:05 Yeah.

48:05 One of the early browsers was actually, Guido was writing in Python and TK.

48:10 Incredible.

48:10 Great.

48:10 So we talked about simple things like thing dot value, A minus B.

48:15 The next simple one is the import statement because imports are simple.

48:18 Honestly, I think imports are, they, if you come from a compiled language, they're really, you've really got to change your mindset.

48:26 Right.

48:27 And so I think this one actually is maybe the opposite end of that spectrum a little.

48:30 Oh yes.

48:31 I will fully out my co some of my coworkers who are because VS Code is written in TypeScript as an electron app.

48:38 And so they live in TypeScript all day long.

48:41 The fact that Python's imports work the way they are in terms of flexibility and just the way it works and you're not just specifying file paths to everything, just.

48:48 Really throws them for a loop and they go like, why is it so complicated?

48:52 Once again, the flexibility that Python gives you where the base case, normal case of if you just do things the way just kind of just it's Python code and directory kind of thing just works versus I can also import something from a URL.

49:06 If I want, that's totally optional.

49:08 You know, I could keep all my core code in a SQLite database and import from there, totally possible.

49:13 Right.

49:14 So it's one of these things, once again, where if you're coming from a Python mindset of just what makes sense, it all just works.

49:21 I realized when you come in with an outside perspective, it can look kind of crazy, but you also have to understand that a lot of the designs were made a decades ago and B, there's a lot of flexibility there that you may or may not be taking advantage of, but there are others who are.

49:36 And so that's kind of why it exists that way.

49:38 Yeah, indeed.

49:38 And yes, this posts.

49:40 Yeah.

49:41 I, so for those of you who don't know, I actually am responsible for import lib and re-implementing Python's import system in Python itself.

49:49 And so I know these details a little too well.

49:53 Yeah.

49:54 So still have nightmares about some of them.

49:56 No, luckily, it luckily import lib in terms of actual import itself.

50:03 Like this I'm ignoring import lib.resources or import lib.metadata.

50:07 Jason Coons mainly manages those modules.

50:09 It's been stable enough for long enough now that there's no real crazy surprises anymore.

50:14 Honestly, the biggest headache is people doing imports in their threads.

50:18 So my word of advice to everyone, don't do imports and threads.

50:22 If you're going to do imports, do them on your main thread up front, then spawn your threads.

50:26 Don't do imports as a side effect of anything either.

50:29 It's circular imports.

50:31 Same thing.

50:32 It's just, just, just don't.

50:34 Your life will be so much easier, but that's honestly where any of the bug reports that ever come in come from is due to either circular imports and people wanting why or weird race conditions with threads on imports and just trying to get the locking down such that things lead to the right outcome.

50:49 Actually 3.12 almost got held up a little bit due to a slight memory leak from the newest version of tracking imports via threads.

50:59 There was a slight leak of it.

51:00 Basically an empty dictionary every time you did an important thread and we didn't want to have that leak.

51:04 So we cleaned it up, but yeah, it was a lot of subtlety.

51:08 Oh, we got lots of memory, bro.

51:09 We don't need to worry about those things anymore.

51:10 No, just kidding.

51:11 Yeah.

51:12 There's no reason everyone gets excited every time Apple launches a new M chip.

51:17 Right.

51:18 We've had enough processing power since we got someone on the moon with how many, how many Hertz.

51:22 Exactly.

51:23 No one needs more than seven Hertz.

51:25 You know, the favorite, famous saying.

51:27 No, just kidding.

51:27 You know, one of the real big differences I think here is that like import runs code, right?

51:32 Like rather than tells the compiler, these things are in scope if you want them, but no, it's like running code.

51:39 And if people are wondering where that weird dunder name equals dunder main.

51:45 So you don't run too much code during imports and those sorts of things.

51:49 Right.

51:49 Yeah, actually.

51:50 And this is an interesting case where it's very much syntactic sugar, but it's also very much syntactic sugar actually for the bytecode.

51:58 Yeah.

51:58 So the thing here is when you do import spam, right.

52:02 Or whatever module you want to import, it effectively ends up being a call to the dunder import built-in function where it passes in spam as the name, and then it passes on all the globals and the locals for where it gets called for name resolution and stuff like this is where you do relative imports with dot, you know, like dot, dot something.

52:20 Yeah, exactly.

52:21 When I first saw it, like, why are all these arguments going to import?

52:23 You just need to know where a, a.py or the package a lives and you're good to go until you say, you know, dot, dot this or from dot.

52:33 Yeah.

52:33 From dot, dot a import lib kind of stuff.

52:35 Like there's a surprising amount of information you need to pass in to do those kinds of resolutions and such from imports, right?

52:42 All that kind of thing to make sure that those things all exist.

52:44 Cause what you're effectively doing in a, from import is if you list multiple ones, you're basically importing the module that those things are in and making sure that after we get set on the thing you're wanting so that you can then grab it later and all sorts of stuff.

52:57 But the other key thing is, is the dunder import function is structured to kind of make the upcode easier, right?

53:03 Cause it's very structured to just push things onto the stack.

53:07 The Python's executing and then make the call to dunder import where it just like, yeah, everything's already there.

53:12 Right.

53:12 But it's also why it's so funky.

53:14 You should never call it directly in your own code, right?

53:17 Like dunder imports very much.

53:18 It's borderline implementation detail on this kind of thing.

53:21 This is why you should use import lib.import module, right?

53:24 That function exists specifically and I designed it specifically for those cases where you need to do dynamic imports and you want a very clean, simple API because like the return type for dunder import very much is designed for the upcode so that whether you're doing an import something or from something, import other thing, right?

53:44 Like the return value makes sense or that for the bytecode level does not make sense for human beings.

53:49 Right.

53:50 So once again, do not call dunder import directly.

53:52 You should always be calling import lib dot import module.

53:55 If you need something dynamically to import.

53:58 Or just use the word import.

53:59 Yeah.

53:59 Or just import.

54:00 Yeah.

54:01 Just import.

54:02 Yeah.

54:03 In your right up here, you have all the different variations.

54:05 Like what does it mean to say from thing, import something as, or just import something or relative imports.

54:12 So people can see all, all the different, again, back to that flexibility that you talked about.

54:17 Exactly.

54:17 So it's definitely one of these things though, where in the end it effectively just boils down to a function call, right?

54:23 Like that's a piece of syntax of import really is just a function call with an assignment in the end.

54:29 Yeah.

54:29 And it's really all it is.

54:31 It's the unraveling it, all the trickery is really behind the dunder import function, but the actual syntax is not, it's not crazy actually, but it literally is a function call.

54:40 You can fully re-implement import.

54:42 Yeah.

54:43 But just calling dunder import and just doing the proper assignment.

54:45 Nice.

54:45 Some confirmation of your advice out there.

54:48 And Nathan says, my data science professor always says the same thing.

54:50 Import first, which is good, but also is there a way to remove syntactical sugar on an example, but not fully through just disassembly.

55:00 I can actually write a tool that will take Python syntax and unravel it.

55:04 I actually started it, but it was just too much of a headache because there's so many variations and edge cases and all that stuff to handle.

55:10 And there was no package out there that, that was going to, that quite did what I needed to do to make my life easier.

55:16 And I had other things too.

55:17 Yeah.

55:17 What's the canonical destination, what's the destination, right?

55:20 Like if I undo a, a with statement on a lock, like how far do I want to go back?

55:25 You know?

55:26 Yeah.

55:26 Well, and I can understand people wanting to do it to maybe learn, but I mean, it's going to be a learning thing, not really a performance thing or It might be slower, right?

55:37 Because some of this stuff is happening at the C level.

55:39 No, yeah, exactly.

55:40 Like, like if you go back to our A plus B example, it actually definitely will be slower as things progress, as things happen in Python, because once again, with the, the speed performance improvements that have been going in, like, like the, the byte code is able to go like, all right, is this an int and an int?

55:59 Okay.

55:59 Well then we're, we know exactly how to do integer math.

56:02 Let's just do it.

56:02 Right.

56:03 And then it can add cards in and go like, okay, quick check.

56:06 A B yes.

56:06 Okay.

56:07 Do the binary int add thing, right.

56:11 Where it just very, very obviously does that.

56:14 But if you unravel it, you can't do that because now you're just doing method calls.

56:17 Right.

56:18 You can't hide those details anymore and put all that stuff.

56:20 Exactly.

56:20 But yeah, but to answer the question, you totally could write a tool to do this.

56:24 And I kind of started one and it just wasn't fun.

56:27 So I just stopped because I just had a lot of, you know, sometimes you see this in the editor as well, when like editors will give you previews.

56:35 So for example, if you use optional of a thing or thing, pipe, none, and then you hover over it in the editor, it'll say, even if you used optional, it'll say thing, pipe, none, because it, it's like, those mean the same thing.

56:47 I'm going to just represent them in one way.

56:49 And yeah, yeah.

56:51 Interesting.

56:51 Okay.

56:52 We have a bunch more things covered, but we don't have really any time.

56:54 Like my goal with this, having you on here to talk about this stuff, it's really kind of like the highlight, the whole series and people should go.

57:00 And we've been kind of hand-waving cause reading C code on air is not ideal.

57:05 No, I think it'd be a whole an idea, but really to highlight your whole series and people can just dive into this pretty deep and, and I think they'll have a much better appreciation for when you write this one line.

57:16 Oh my goodness.

57:17 This is actually what's happening.

57:19 And how I actually found a bug in Python.

57:21 Thanks to it.

57:22 Okay.

57:22 Tell us quick about that.

57:23 Yeah, it's covered in the in-place binary operator, augmented assignment post.

57:29 So, you know how you can do A plus equals B and it's like doing A equals A plus B.

57:34 Yeah.

57:34 Yeah.

57:35 So it turned out it was broken for call.

57:36 So when you know, when you could do a star, star B that's a to the B power.

57:41 Yeah.

57:42 It turned out the semantics were busted for star, star equals.

57:45 Wow.

57:45 Okay.

57:46 And no one had ever noticed because obviously people do not write custom implementations of thunder pal because effectively what turned out was, and someone actually discovered it and reported it, I just didn't know about it.

57:57 That basically when you did that, so the in-place augmented assignment, there's I versions of everything.

58:04 I'm like, I add a dunder.

58:06 I add dunder.

58:07 I mall dunder.

58:08 I pal.

58:09 Right.

58:09 Turns out for all of them, they did the right thing except for power where it would check for I pal.

58:14 And I mean, in all these scenarios, because a plus equals B is the same as a equals a plus B.

58:20 If the I version returns, not implemented, it falls back to just doing the binary operator and then doing the assignment.

58:27 Right.

58:27 So like if I plus, so I plus equals B unravels to a.i pap.

58:33 I hate dot dunder.

58:34 I pal with B as an argument.

58:36 If that returns on implemented, then it devolves into a plus B a equals the result of that, right?

58:42 In the power case, if you return, not implemented on thunder, I pal, it didn't fall back to pal.

58:47 Okay.

58:48 He just crashed.

58:49 He said, I don't know what to do here.

58:50 It was just an exception.

58:51 It was like, yep.

58:52 Nope.

58:52 I suppose that's better than the wrong answer.

58:54 Like actually the square of nine is 18.

58:56 Carry on.

58:57 It threw me for a loop because I wrote code to verify all my unraveling and how did the augmented assignment for power just kept not working.

59:05 And it's like, what the heck's going on?

59:07 And then I had to dig into the C code.

59:08 And I was like, what am I doing wrong here?

59:10 Yeah.

59:11 Wait a second.

59:11 That code unravels calling this C code, calling this C code.

59:14 And it didn't check that return value on calling it under I pal.

59:18 It's like, well, did a return not implemented?

59:20 And if it did, then we've got to try the other version.

59:22 It just didn't.

59:23 It's just like, Oh, no, it's just the value just got returned.

59:26 It just didn't even check.

59:27 I think we figured out it happened when the third argument to pal got added.

59:31 And it was just someone overlooked it, but I see it been sitting there for years.

59:36 Like I think it was a decade.

59:37 Yeah.

59:38 You're looking through these things.

59:39 You can tell that they're pretty subtle.

59:41 There's a lot going on in the different special cases.

59:43 So I can see it wouldn't hit you in the face that you necessarily miss something right away.

59:47 Exactly.

59:47 Yeah.

59:48 Yeah.

59:48 I don't fault anyone at all.

59:50 It was, I think we were just missing a test case somewhere to make sure that that happened because, and once again, I don't think people define their own custom pal operator for their types.

59:59 There's a whole bunch of mathematicians going, don't we do, we do or physicists or something.

01:00:04 Yep.

01:00:05 All right.

01:00:05 We're pretty much out of time to go any deeper on this.

01:00:08 And I think that that's right.

01:00:09 I think people got a really good sense of what's happening.

01:00:12 You know, like what you mean with your series here and hopefully they're inspired to go check it out.

01:00:16 But I do, while I have this import stuff on the screen and you've done so much work with import lib, will there ever be a time where like hot reload type of stuff is a thing, you know, like I edit this file in this web app and it's got this project it's been running and I just, just maybe could we have a file watcher trigger this module to re-import not necessarily saying it's a good idea, but have you thought of that?

01:00:40 Have you thought of that?

01:00:41 You know, you want automatic import lib dot reload?

01:00:43 Yes.

01:00:44 Or, you know, even like the, the lib reload stuff, a lot of times like you can do it, but it's kind of discouraged because I think largely because of the side effects, what can happen there?

01:00:52 No.

01:00:53 So to be very specific, if you call a module, if you call import lib dot reload on a module, it effectively reruns import on that module.

01:01:02 The problem is, is all the references you have to objects don't change.

01:01:07 Yeah, exactly.

01:01:08 So if you got something out of that module, like let's say there was a global dictionary that you stored a reference to a way that you expect.

01:01:15 Right.

01:01:16 Or a function or a class.

01:01:17 Maybe.

01:01:18 Exactly.

01:01:18 Yeah.

01:01:19 And then you call reload.

01:01:20 Everyone else in the future is going to get the new module and they'll get the new version of all the things that were in that module, but all that stuff, you still have reference to do that.

01:01:28 And it's magically taken away or garbage collected.

01:01:30 And we can't really swap it underneath you either.

01:01:33 Cause the whole type could have changed.

01:01:34 Right.

01:01:35 Right.

01:01:35 So, I mean, if you really wanted to set up your own import system that does file monitoring and omnic calls import lib dot reload, you could totally do that today.

01:01:43 There's, there's nothing stopping you.

01:01:44 But you have to just be very aware that.

01:01:48 Only sometimes will it have a good effect.

01:01:50 Yeah.

01:01:51 It'll have an effect.

01:01:52 Yeah.

01:01:52 It's just whether it's going to do what you expect.

01:01:55 And that's, and that's why import lib dot reload is discouraged because people often don't quite understand the side effects and the things you have to watch out for to understand that it's going to do what you expect.

01:02:06 And especially when you import do from imports, right.

01:02:11 Cause then you're getting the direct objects off the module instead of the module itself.

01:02:14 Cause like if it doesn't in place re-import, so like if you did import spam and you reloaded spam, the attributes off of spam will now be the new stuff.

01:02:24 Cause we actually, we actually changed that dictionary in place, but if you did from spam import function, function is not going to change.

01:02:31 There's another reason why I always tell everybody don't import the objects from a module import to the module.

01:02:35 I find it.

01:02:36 Yeah.

01:02:36 I personally find it more readable cause I can look at the code no matter where I am in the file.

01:02:39 I know where that function came from.

01:02:41 It's not from this module.

01:02:42 It's from somewhere else.

01:02:43 I am always with you.

01:02:43 I'm almost never from thing import other thing.

01:02:46 Yeah.

01:02:47 Unless it's a deep module in a package, in which case that makes sense.

01:02:50 Cause you'll still do module dot thing.

01:02:53 Right.

01:02:53 Right.

01:02:53 I was like, well, even though I have that, if it's like four levels deep, I'll do the first three and then from the first three import the last, last step of the module, so you can say like data layer dot, you know, exactly update rather than just update, like, well, what does update mean?

01:03:06 I don't know.

01:03:07 Exactly.

01:03:07 Yeah.

01:03:08 I mean, the canonical examples, random, you see the function random in your code.

01:03:12 Whose random is that?

01:03:13 Is that from the random module?

01:03:14 Is that from NumPy?

01:03:15 Did you implement random?

01:03:17 Whose random is that?

01:03:18 Yeah.

01:03:19 You just know.

01:03:19 But if you saw np.random, you'd know.

01:03:21 Exactly.

01:03:22 Yeah.

01:03:22 If you knew what np was.

01:03:23 Yeah.

01:03:24 If you did, of course.

01:03:25 Right.

01:03:26 I guess, give you a chance to kind of summarize final, final thoughts here on your syntactic sugar series and, you know, tell people how to get it and stuff.

01:03:35 You can go to my blog.

01:03:36 It's on snarky.ca.

01:03:37 It has the tags and syntactic sugar.

01:03:39 I'm sure Michael included in the show notes.

01:03:41 Don't feel the need to read all of it, but do look at least the first one or two, because I do go into more detail on how you can kind of go exploring on your own in way more detail in those posts than I do towards the end.

01:03:53 Cause I'll be honest.

01:03:54 I lost a bit of steam and going diving into all the deep layers.

01:03:59 I wanted to get the blog post series done after two years.

01:04:01 So it was one of those.

01:04:02 Okay.

01:04:02 Go look at the earlier ones.

01:04:03 They'll explain how to, how to figure out how to go from syntax to the byte code, to the C code, and then know where to look in the C code, which by the way, you don't have to understand the C code.

01:04:12 It's not a, it's not crazy C code, but it's also not necessarily critical.

01:04:16 I do try to write down the Python equivalents for everything.

01:04:19 So it's there if you do understand C, but don't feel intimidated if you don't.

01:04:23 And hopefully you just find it interesting, right?

01:04:25 Like you've always wondered how the heck does this work?

01:04:27 If it's in that, if it's in the blog post, you'll build a, have a better understanding, hopefully of how things work.

01:04:33 Cause as I said, I honestly even learned some details that I forgot or never even knew about some of the semantics behind Python and how, like how things actually work underneath the hood and it just gave me a better understanding to understand.

01:04:45 When things do and don't happen the way I expect and just honestly appreciate all the work everyone's put into language over the decades to make it all seemingly seem simple and yet it's surprisingly complex.

01:04:55 Yeah.

01:04:55 I don't think you need to read them all either as somebody who read most of them the last couple of days.

01:04:59 No, you don't.

01:05:00 That I do.

01:05:01 If you're a completionist, go for it, but you definitely do not need to.

01:05:04 But I do think going through it, you know, pick out the ones, there's a whole bunch of different topics.

01:05:09 Pick out the ones that are, you're like, Oh, I did always wonder how that one worked and pretty much with the exception of pass, I think you'll be like, Oh my gosh, I had no idea what was involved here.

01:05:19 Yeah.

01:05:19 Pass was very much just a, I had a checklist of every piece of syntax and every single keyword, and that was that the blog post on pass, I think is like three sentences and half of it saying this is going to be the shortest, shortest post in this whole series.

01:05:32 Yeah.

01:05:33 Passes.

01:05:33 Yeah.

01:05:34 It was contending with the dot, dot, dot, which we haven't had a chance to talk about ellipsis versus pass, but I think we're out of time, Brett.

01:05:42 So thanks for being here.

01:05:44 Thanks for all you do.

01:05:45 I mean, steering council, core dev, author, VS Code, et cetera.

01:05:50 A lot of contributions.

01:05:51 Thanks.

01:05:52 Thanks for having me back on yet again.

01:05:53 And hopefully I've not worn out my welcome still.

01:05:55 No, you're already thinking about what next WebAssembly rust thing we'll get together on.

01:06:01 Thanks for being here.

01:06:02 See you later.

01:06:03 Thanks.

01:06:03 This has been another episode of talk Python to me.

01:06:07 Thank you to our sponsors.

01:06:09 Be sure to check out what they're offering.

01:06:10 It really helps support the show.

01:06:12 Want to level up your Python.

01:06:14 We have one of the largest catalogs of Python video courses over at talk Python.

01:06:18 Our content ranges from true beginners to deeply advanced topics like memory and async, and best of all, there's not a subscription in sight.

01:06:26 Check it out for yourself at training.talkpython.fm.

01:06:28 Be sure to subscribe to the show.

01:06:31 Open your favorite podcast app and search for Python.

01:06:33 We should be right at the top.

01:06:35 You can also find the iTunes feed at /itunes, the Google play feed at /play and the direct RSS feed at /rss on talkpython.fm.

01:06:44 We're live streaming.

01:06:45 Most of our recordings these days.

01:06:47 If you want to be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at talkpython.fm/youtube.

01:06:55 This is your host, Michael Kennedy.

01:06:57 Thanks so much for listening.

01:06:58 I really appreciate it.

01:06:59 Now get out there and write some Python code.

01:07:02 [Music]

01:07:17 [Music]