You've probably heard the term syntactic sugar.
That is syntax within a programming language
that is designed to make things easier to read or to express.
It makes the language sweeter for humans to use.
It turns out Brett Cannon has spent two years diving into
and writing about Python's sweet language features
and how they really work down inside CPython.
He joins me on the show today to dive into a few of the more relevant posts
he's written about.
This is Talk Python to Me, episode 441, recorded November 1st, 2023.
Welcome to Talk Python to Me, a weekly podcast on Python.
This is your host, Michael Kennedy.
Follow me on Mastodon, where I'm @mkennedy,
and follow the podcast using @talkpython.
Both on fosstodon.org.
Keep up with the show and listen to over seven years of past episodes
at talkpython.fm.
We've started streaming most of our episodes live on YouTube.
Subscribe to our YouTube channel over at talkpython.fm/youtube
to get notified about upcoming shows and be part of that episode.
This episode is sponsored by us over at Talk Python Training.
Did you know that we have over 250 hours of Python courses?
Yeah, that's right.
Check them out at talkpython.fm/courses.
Brett, welcome back to Talk Python to Me.
You've been here a time or two.
Always great to have you here.
Well, thanks for having me yet again, Michael.
I'm glad I've not worn out my welcome quite yet.
You always have so much cool stuff going on
and so many interesting perspectives.
Some would call them interesting.
Whether that's a positive or negative is still open to interpretation.
May you program in interesting times,
as the old ancient proverb goes.
Mm-hmm.
So, been here a bunch of times,
but real quick, just tell people a bit about yourself before we jump in.
Sure.
Just in case they don't know you.
Yeah.
Name's Brett Cannon.
I live in Vancouver on the unceded territories
of the Squamish, Tsleil-Waututh, and Musqueam First Nations.
I am currently the dev manager for the Python extension in VS Code.
I am serving my fifth and, as of now, final term
on the Python Steering Council.
I'm stepping down just because when we first created the Steering Council,
I said five years seemed like a good term limit.
We actually don't have term limits, but figured I'd stick to my word.
And I'm also, thus, also a core developer.
I've been doing it for just over 20 years.
Lucky for me, I got it, my first commit bit right after the,
sorry, my first commit bit after the first PyCon.
So, I get to use the PyCon anniversaries as a way to keep track of that.
So, that's convenient.
That's kind of the biggies.
Have a cap.
Yeah.
That's usually the other thing people like.
Something about VS Code?
I've heard that word.
It's an editor, right?
Yeah, I know.
Someday, you'll say you use it,
and I'll be very happy that day.
I had it open today.
Oh, all right.
Well, there, I'm happy today then.
Yes, indeed.
No, awesome.
So many cool things you got going on.
And despite all of that background in steering council, core dev, here, this time, you're
here to give us, like, diet advice, eating advice, like something about sugar.
What is it?
Yeah.
Just a slight story behind all this.
My wife was taking the certificate in data science course at the University of British Columbia,
which is actually our alma mater and where we met.
And she was doing her homework in a JupyterLab notebook through the browser via JupyterHub.
And she stopped doing the homework to go, I can't remember if it was lunch or dinner, but
basically stepped away for like an hour or so.
And then she came back and the run buttons on the cells weren't working anymore.
I was like, well, what's going on?
And I looked and I was like, oh, well, you got disconnected from the server.
I said, well, isn't that happening in the browser?
Like, no, Python doesn't run in the browser right now.
It's not a thing.
Oh, OK.
I think about a day or two later after she thought about it, we were in the car and I
still remember the exact location in Vancouver where this occurred.
My wife said, you know, you should probably fix that problem of Python not working in the
browser.
And I explained like what WebAssembly was and how big of a thing that would be to make
that all happen and all that.
And she literally took about a beat to wait and said, no, you should go fix that.
Part of that led me down the road of trying to figure out what the minimum viable Python
was, right?
Like, what is the minimal amount of Python I would have to implement in the browser or in
WebAssembly in this case to make Python work?
And I had realized that, you know, a decent amount of Python is actually what's called syntactic
sugar, which is really just syntax that you could actually devolve and unravel into other
syntax of a language and have it do the exact same thing, right?
It's just basically syntax that's a shortcut.
And so I ended up going on this very long journey, unexpectedly long, of trying to go through
all of Python syntax and trying to figure out which parts of the syntax was like crucial
and you couldn't actually re-implement it in pure Python itself and which parts you could
actually just make it work with Python code itself.
With the idea that at the end, it would have basically a list of the bits of Python that an
interpreter would have to implement and the other bits of syntax where you could run some
tool over it ahead of time.
It would just translate from Python to different Python code.
And in that way, I would just know what the exact target would have to be.
It might not run very fast.
Don't get me wrong.
But it would at least be a very clear definition of kind of what you have to have to really call
something Python.
It's everything else.
Right.
kind of just work around.
Right.
So the standard Python syntax we expect still works.
But how much do you have to work behind the scenes to make that happen?
So examples that come to mind for me are context managers, the width block.
Awesome.
I love them.
But we have try finally.
And you could simulate a context manager effectively with maybe a try finally, maybe an accepts in
there as well.
But you don't actually need width in the language to accomplish what it does.
Right.
Yeah.
Decorators are also another really good example, right?
Like where the concepts for that syntax predates it.
And it happened to just be something where Python core development team looked at where people
were in terms of development and what they needed from Python and realized, you know what?
That common pattern that people are doing would actually be would be very well served via
a piece of syntax.
So why don't we just do that?
Why don't we introduce that piece of syntax?
Make things happen.
As you said, with context managers were a perfect example, right?
Where before, if you wanted to do the right thing when you opened a file is you call the
open function for your file, sign it to a variable, do your try block of everything you want to
do with the file.
And then in the finally part of the try finally close that file, which I mean, if you just think
about it compared to a with statement is four lines, assuming you're doing reasonable formatting for
the assignment, the try, the finally, then the close call versus the single with open to get the exact
effect.
And then you've also got the possibility that the context manager, the thing that's used in
the with block can do something different if there's an exception, right?
Like if there's an exception, it could say if it's a database transaction, it could roll back the
transaction automatically.
Rather, if there's no exception, it could commit it automatically.
But there's no real magic there.
If you think about it in the end, you could have written that all out by hand.
It just would have been cumbersome.
But that makes the with statement very much a piece of syntactic sugar where you could totally
write it out, get the exact same semantic outcome.
It might be a little slower.
Probably similar performance as well.
And maybe similar.
It really depends on the syntax, right?
Like how complicated it is, how much of it's going to call into, at least in CPython's
case, into the C code and stay in the C code versus not.
Like with context managers, probably won't just because the point is you're going to call
that dunder enter method or that dunder exit method.
So you're calling Python code.
So no real, probably big shift.
But for stuff that we'll probably cover in this podcast, there are other bits that go into
the C code and stay in the C code.
And when that happens, stuff can go really, really fast.
Yeah, especially with all the work the faster CPython team's done around stuff and all that.
So the point is you can just start to just kind of leap into C code, stay there and get
stuff done way faster.
Comparative to writing out by hand where you're staying in the Python world, it can just take
a bit longer.
Yeah.
So the word sugar, I think it's a good one here.
I love like these ideas like code smells, code sugar, all the things.
It makes it sound a little frivolous, maybe like little, it's nice to have.
It sweetens it up.
You don't really need it.
But I do think having these different constructs, like when you see a with block, it instantly
conveys meaning.
It means we're going to create a thing.
We have a cleanup section.
We don't have to deal with it.
As opposed to if you see a try except finally, you have to read it and process it and see,
okay, does it actually do the things that would be accomplished?
I think these sugars let you also think, not just write less good, but think in different
levels of abstraction.
Yeah.
I mean, as you said, the use of the word sugar is meant to mean something sweet, something
nice.
It's a treat.
It's a bonus.
So from the perspective of if you didn't have it, you could still accomplish it.
But that can potentially, in a way, as you've suggested, not quite denote the benefit of it,
right?
Like if we think of sugar in terms of not something good for your diet.
Yeah.
What is your perspective on sugar, right?
Are you a fan of desserts or are you trying to cut it out?
Yeah.
But yeah, I mean, the hope is whenever we add syntax to Python, it's for everyone's
general benefit, right?
This is why we don't do it very often.
And when we do do it, it's a very thoughtful discussion, very long discussion, sometimes
contentious discussion.
Walrus operator is the famous one for that.
I don't understand the contention in the Walrus operator.
It's such a small change to the language.
There's so many other things that are bigger changes that could have been battles.
But I guess it just came to a head at that point or something, right?
For those of you who don't know the Walrus operator is the thing that pushed Guido kind
of over the edge to finally step down as being the bit of a dictator for life.
And it was just due to the veracity of the opposition to it and just how basically angry
it made people.
And you're right.
It's very small.
And to be honest, I've used it and I like it.
I get use out of it.
It means it.
I use it for its purpose, which was to add some basic syntax for those up those situations
where you always have that assignment and then you do something like an if.
for something and then you're going to do something like just be able to inline certain
things to just do the right thing.
It's not a huge thing, but it's a nice thing.
And in that case, I don't know.
Honestly, a lot of people got really concerned that they didn't see the usefulness enough that
they thought it was going to be a slippery slope or people are always very concerned about
Python getting too big and it's no longer fitting in their brain.
Yeah.
Maybe it was kind of the last straw sort of thing rather than like maybe stuff came before
and they're like, all right, that's it.
And now the walrus, the walrus, like, you know, I don't know.
I don't know.
I mean, I don't think people realize that we actually cut syntax out when we moved from Python
two to Python three.
Yeah.
That was all kind of brewing.
Yeah.
Like the back ticks went away.
Like no one knows about the back ticks anymore, but we used to have syntax for using back
ticks and that's now that's gone now.
So it's not like it's always been a add, add, add.
But I think it's just one of these things where it's just people were very concerned that
Python would get too big, too complicated, and it wouldn't be fully understandable.
I will say whenever we do this, we very much consider how easy it would be to understand
if you saw it for the first time and never received any instruction about it.
Right.
Like I would argue that walrus operator is a bit obvious if you look at the code around
it of what it does.
And so you can probably figure out what it does without being told ahead of time versus
other syntax and other languages where you go, I have no clue what this does.
Right.
Exactly.
Right.
Like why does plus plus and C do different things if it's in the front or the back?
And what's the difference?
Like unless you know prefix versus suffix increment and the difference in terms of referencing the
pointer and all this crazy stuff, like you're not going to necessarily pick up
on the difference unless you run your code versus something where you can just look at
it and just learn on the spot.
Well, I want to give you all a bit of a compliment, I guess, or encouragement.
I think the Python language is not changing too fast.
I think it's the stability over time is really good.
You compare that to other languages.
I mean, here's one that's kind of close to home for you since you're at Microsoft, like
C#.
I feel like that language is pretty nice, at least in the early days.
And now there's a whole team who are employed.
You don't have to make any comment.
I know you work there, but there's a whole team who's employed to shepherd that language.
And I feel like there's a feeling they always need to have some new language features because
it's their job to have language features.
There's just like a constant kind of turn, like for properties, like we have properties,
they have properties.
But I think there's four to five different ways that properties in C# that have evolved
over time.
And you're just like, yeah, it's a little better, but it's why do we have so many ways if you're
going to be there?
Right.
And so when I at least look at Python compared to like my background in C++ or in C#
or JavaScript, I feel like it's comfortable.
It's not stale, but it's not.
We don't have five ways to do properties.
That's a good thing.
Even if there's a better way to do it, it might not be worth having two to three ways just
for that for a slight advantage.
Well, thank you for the kind words.
Yeah.
I mean, we do have the Zen of Python does kind of outline the general guidelines we try to
follow when doing making these discussions decisions.
For those who've never read the Zen of Python, look, just run, import this from the REPL and
you'll find it.
And yeah, it's just the way it's designed, right?
We don't, hopefully there's one and hopefully only one obvious way to do anything.
And so it's just something we've kind of leaned into now, granted things shift.
Like I'm sure someone will point out the myriad of ways you can format a string now, but a
lot of that's backwards compatibility and just realizing, you know what?
We just have to accept the fact the first way we got it is no longer the best way we
can think of.
And it's enough of a jump from using string interpolation with the percent sign to using
f-strings that it was worth adding a second way.
Now, maybe someday we'll take out string interpolation.
I doubt it because it just works and it's just sitting there and it's not worth breaking
really old code.
But I mean, it's just the way we approach things.
And luckily, most people like it.
And I think that's kind of why some people always freak out is the words afraid somehow
we're going to go downhill from here and it's just the start of things or somehow Python
is going to be harder to learn.
Well, we very much try to make sure Python is a gradual curve of learning, right?
The basics are very straightforward and good.
And as you get into more advanced things like the match statement, right?
You'll be able to see it and understand it.
It'll make sense, but not something you'll necessarily learn the first month or two,
but it's there for when you do need it, right?
Like we very much try to make sure that everything makes sense on a curve from beginner to advanced
and try to make sure the language is useful to everyone from both a beginner to advanced
user.
It's tricky, but I like to think we're doing a decent job.
Yeah.
One of Python's really, one of its powers is that you can be super effective with it with
a really partial understanding of what it is, right?
You don't have to understand namespaces, classes, async, et cetera, et cetera.
You're just like, well, I think I can write these four lines in a, without even a function
and I'll get a cool graph that'll show my, my work or whatever, right?
Yep.
Exactly.
Yep.
And I think I have a pretty partial understanding of the match statement.
I know it can get pretty advanced in like how it captures and changes values, but I wrote
one yesterday with a partial understanding.
It went great.
Nice.
Yeah.
Indeed.
All right.
Before we move on though, got to go back to the, the, your opening because you said,
I created this to understand, like, if I were to solve that problem, your wife had about like,
you got to fix that thing about the browser.
And I know there's some folks in Texas who have thoughts on this as well in the PyScript
team, but is this project you're actively working on or just, let me gather up, do some
research and deep thinking about what is actually in the language that would have to carry
over or where's it coming from?
I guess.
No, it's not an active project of mine at the moment.
It ended up being mostly just an exercise and just snowballed into a thing where I just
didn't want to stop until I completed it.
Yeah.
How do you feel about the, the micro Python, PyScript angle of approaching this?
Yeah.
I mean, I think it's great that, I mean, that was the other hopeful benefit about all of this
is trying to nail down the definition of what Python is.
Yeah.
Cause I've heard those arguments made against micro Python that it's not quite Python cause
it doesn't quite run all the syntax or it doesn't have the entire standard library.
I've gotten into arguments with people online where they got very upset when I suggested that
the REPL was an optional part of Python and actually not a requirement.
And I just viewed it as a learning experience for me, hopefully a learning experience for
others as well.
And then after that was just trying to hopefully clarify and like, like, look, this is truly
what you have to have to really be Python.
You literally cannot implement the rest of the language without this, but everything else,
you could totally just do it yourself.
You could totally just write your tool and make it all work.
And as long as this existed, you could totally get it by with it.
And so that's kind of where I ended up.
I luckily the tooling around WebAssembly got far enough along.
Pyodide's continued to do well in the browser.
I've been working on the WebAssembly, the system interface, WASI side of things, and that's
been going pretty well.
So that's at least taken care of itself.
So I've ultimately not tackled the huge problem of trying to reimplement Python from scratch,
probably in Rust, just in a minimal fashion to try to make it work better in the browser.
That's probably a project for when I retire.
And I suspect that's at least a decade out, if not more.
So that's probably going to be put on the back burner for a while.
Right.
Yeah.
We could all install Bpython and run Brett Python built in Rust.
There you go.
Rust does seem to be having quite the influence.
an outsized influence for how popular Rust is in terms of the number of developers, but
it seems to be really leaving its mark on Python and other areas as well these days.
Yeah.
Well, I mean, I will fully admit I have bias because I am a fan and I do like using the
language, but it's one of those things where if you've been living in the C and C++ world
and it never was that satisfying to you, but you had to be there for whatever reason.
And then you discover Rust and it does things that saves you from very chronic problems.
Yeah.
Including yourself.
Yeah.
You would hope that you, people would notice and latch onto it.
And I mean, that's, I think that's basically what's happened is people have just discovered
that, oh yeah, this is useful and good.
And just there's enough C and C++ code in the Python community due to CPython C API that
I think there's just a nice cross section, right?
I think it's just one of these things where people have just gone like, well, I've got to
be, I got to do the system level programming anyway.
So I might as well use something that's seems like a better, a little more modern language.
Yeah.
A bit more modern.
And so I think that's kind of what's happened here.
Yeah.
That makes sense.
All right.
Let's talk about the sugar.
Okay.
So I think we covered pretty much what syntactic sugar is.
And you started this whole thing out saying the goal of this is really to kind of nail
down that minimum Python.
And I agree that the REPL is optional.
I like the REPL, but just, just so I can also get mail about it.
I like the language.
Like you can't take away these features once you add them.
Right.
And so it's like, what could you survive with as a minimum?
And then you can kind of like, what are the, what's the extras?
What could you build up from there?
Yes, exactly.
And the final destination maybe being, you know, CPython proper.
Yeah.
So you've got a whole bunch of different aspects of Python that you are unraveling in
this syntactic sugar.
And basically to highlight what's actually happening, how much of this is going on as you've already
laid out.
So I want to point out that I'll link to how many articles you've got here, but two years.
20, I think.
Yeah.
Like looking like close to 30, maybe.
I don't know.
Just eyeball it.
There's a lot there.
So I think I picked a, you know, half of those out or some, some reasonable subset.
That we can talk about.
And there's some code and stuff in here.
So we'll kind of have like, my thought was we could just talk about kind of like, what is
the essence of this?
Like what's really happening when you do attribute access?
What's really happening when you write the word pass or use a width block as we've talked
about a little bit.
So the first one here is unraveling attribute access in Python.
You want to tell us a bit about what actually happens if you just say, you know, like object
dot attribute.
It seems so simple, Brett.
There's just one dot.
That's the big deal.
Yeah.
Why doesn't this just, what's there to talk about?
I mean, when I just do something dot something magically, I get back what I expected, but there's
actually a lot going on there.
Right.
And it, it comes down to multiple layers, but it's all ties around Python's object model.
Right.
How do all, because I always find this weird to say, but Python's more object oriented in
my opinion than Java.
Java has these primitive types like instant quotes that have to get boxed and unboxed to
fake, to make them look like an object to Java itself.
Python doesn't have that.
Everything is an object.
A function is an object.
An integer is an object.
True is an object.
Seven is an object.
Exactly.
There's nothing you can name in Python.
That's not somehow an object.
Yep.
This idea of attributes and all this permeates all the way down into the object model, which
once again, defines kind of everything in a way.
So the key thing here is there's kind of two parts to it.
It's how do you look up based on inheritance, what thing is going to service the call asking
what is this attribute value?
Basically, what object is bound to this name for the attribute?
And after that is what methods are going to get called to make that happen?
Because effectively, almost everything in Python at the end of the day leads to a method call
or a function call.
Yeah.
First of all, it leads to a lot of times to an op code, which leads to a method call, right?
A Python bytecode thing.
And I think it's maybe also worth pointing out, like have you point out to everyone listening
is Python compiles to this intermediate language like Java and .NET compiled to intermediate languages.
It's just what happens then, right?
Like in those other ones, they JIT compiled to machine instructions.
Here it goes through like the big C eval loop and delegates to some internal C operations.
And a lot of those, these unravelings, like kind of look at what happens at that step, right?
Yeah.
So CPython itself is an interpreter, right?
So what happens is when you load up your Python code is Python parses it, what's called an abstract
syntax tree or AST.
It then takes that AST, does some stuff with it, and then it compiles it down to Python bytecode.
I actually gave a talk on this at PyCon Canada way back when, for those who are interested,
that basically goes from the steps of literally source code all the way to execution, if you
want more detail.
But effectively, yeah, we compile it down to bytecode of our own design to then execute
in more or less what's a big for loop in C.
That is a really big for loop.
Very big for loop.
And it's gotten a bit fancier thanks to the faster CPython team, right?
Like there's now kind of like lower level opcodes that are very type specific.
And we actually auto-generate the loop now thanks to them so that it's easier to maintain and
make tweaks to.
But essentially, it's literally just a set of instructions that are extreme, that kind of
encompass key bits of semantics in Python.
So for instance, for attribute access, there's literally a opcode called load adder, where on
the execution stack, you basically push the object you care about, the name of the thing you
want, and then you just call load adder on it, and it'll just look it up.
Actually, technically, I guess it doesn't push the name on it.
That's technically stored in the function.
There's little details I'm kind of hand-waving over, literally hand-waving if you're watching
the live stream.
But effectively, you push the object you want, you call load adder, give it an argument of
what it should, what attribute it wants, and then load adder opcode behind the scenes is
the thing that does all the magic of calling everything you'd expect, et cetera, et cetera.
Yeah, excellent.
And to view that, a lot of times what you'll do is you'll just use the dis module.
So if people haven't used dis, you can just dis.dis, a lot of disrespect on this function,
and then it tells you what the opcodes it's made of are, yeah?
Yeah, exactly.
I mean, dis is short for disassemble.
So literally, you can pass it a function, and it will effectively disassemble what the
function does into the bytecode, which is just stored as an attribute of bytes on the
function object and print it out nicely because the disk module knows how to map the bytes
to what the opcode's name is and just gives it a nice way to look and actually read and
understand it.
Okay.
In this example, you said, all right, look, here is the case statement in that tremendous
for loop switch thing that does the execution.
What do we do if we see load adder?
And ultimately, it comes down to this pyobject underscore get adder.
So it seems so simple.
It's just object dot attribute, object dot field, whatever.
But it turns out there's just so many variations and so much going on down here.
And I don't necessarily want you to go through all the details because it's pretty, some of
it's pretty intense C and like a lot of it, you know, optimizations and stuff.
But just give people a sense of what actually happens down in the guts when you try to do it.
Well, so in the case of attribute access, right, that effectively ends up calling the
get adder built-in function, right?
And in that case, it needs to check how many arguments it gets to start, right?
Did you give it two arguments or did you give it three arguments so you return a default value
if the attribute doesn't exist?
And then at the C level, we have to check whether or not what you gave it, right?
Because there's like a Python level, the C code all types to what's called the to py object,
which is basically a massive struct in the C that represents any and all Python objects.
So even at the C level, we don't know if you gave us a string or an integer or what did you
give us for the name of the attribute that you want.
So we got to do that, right?
And so there's just basic, like just safety checks to go like, nope.
Like when you, if you call get adder with wanting the attribute 42, which is a number,
it's going to throw a type error.
Well, that code's got to exist somewhere and this is where it exists.
Yeah.
Right.
And then effectively just bubbles all down to the C API where a lot of this stuff gets exposed
in its own way.
And specifically in this case, it ends up calling py object get adder.
Yeah.
And I think another thing that's pretty interesting here is it's not always just a value, right?
That object dot adder, that adder could be a property.
That adder could be a descriptor, which I guess is a generalization of a property.
It could be on the class type, but not on the instance type.
So you got to find that.
It could be on the instant type overriding, but only there, right?
Like it could not exist.
All of these scenarios that you got to go through.
And so it really, you know, it really emphasizes how much is actually going on in the runtime while
this single simple line seems to be running, right?
Yeah.
I mean, in the simple case, hitting the dots, not complicated.
Like in general, what you end up doing is you have your object, you hit dot.
And what it does is it looks in the dunder decked attribute of the object.
And there's a key matching the attribute name.
And it just gets the thing in the dictionary and that's it.
You're done.
But as you alluded to, how the heck do you even get to that basic case?
Or those more complicated cases, as you said, descriptors, right?
Which are how we actually implement properties.
Or if it's on the class versus the instance.
And what happens if there's inheritance or multiple inheritance?
Yeah.
Or dunder get adder was defined, right?
Like basically almost all of this ends up flowing through the get adder built in, which effectively
ends up calling the appropriate dunder get attribute method off of an object.
And this is one of those bare level kind of things, right?
Where you have to implement this to make Python work.
You have to basically implement the get adder built in.
And it has to understand how objects are structured underneath the hood to know how basically methods
and stuff are attached to an object.
You really can't fake that.
There's no way to not kind of have that as a low level detail that you can just unravel.
It has to be implemented by the interpreter.
And in this case, effectively, the way Python does things is it knows the order.
You can always actually call the MRO method on any object in order to use the method resolution
order.
And effectively what Python is doing is if it checks on something to see if it has it or not,
it will effectively end up calling the dunder get attribute object of method, sorry, on
object.
That's usually where it bottoms out.
And unless you happen to have a dunder get adder, if that fails, but that's only in the
failure case.
But really, in the end, almost everything passes through object dot dunder get attribute.
And that's really where all this ends up going.
But as you said, there's a lot of little twists and turns.
Yeah, a lot of these internals, you'll see the method resolution order calls.
And it's all about, well, there's a lot of classes and a lot of objects.
They can override things and they can override operators.
And so there's a lot of navigation to just figure it out.
Not just calling these get adder type of things, but what level does that even supposed to happen
on, given what I'm working with, right?
Yeah, exactly.
And I think this is a good example.
And the reason I started with this is so fundamental, right?
Like, how do you unravel something if you don't know how to even get an attribute off of something?
So this is why this was the very first post in this series.
Partly why I went into such detail on how to figure all this out, right?
Like, kind of one, this is almost a template for anyone else who wanted to go exploring on their own.
But I think it's also a good example of how much complexity and flexibility Python hides from users in order to make the simple work, but give you the flexibility to make the complicated possible.
Right.
Yeah.
Like for the vast majority of any of us for our code, it's not just does what you would think it would do.
And it's pretty straightforward.
But as soon as you get into the fancy world of descriptors.
SQLAlchemy.
Yeah.
I can both have a value or I can do a database query with the same thing.
How is that possible?
Right.
Yeah.
You want that magic?
Language has to support it somehow.
And it requires a lot of finesse and thinking through and a lot of mechanisms that generally get hidden from you.
But someone's got to write that code somewhere.
And in this case, it's a Python.
And this kind of outlines how that all happens.
And it's surprisingly complicated.
It's not horrible.
That was my main takeaway, too.
Yeah.
I mean, you can totally read it and follow it.
I don't want to scare anyone saying it's so complex you can't follow it.
It's just it's not just a simple two if statements and you're done.
It's no.
There's subtlety to it to make sure the common semantics make sense to people.
It's not going to trip people up because it's doing something weird.
There's a lot of thought that goes into these semantics to make sure it makes sense for the common case.
This portion of Talk Python to Me is brought to you by us.
Have you heard that Python is not good for concurrent programming problems?
Whoever told you that is living in the past because it's prime time for Python's asynchronous features.
With the widespread adoption of async methods and the async and await keywords,
Python's ecosystem has a ton of new and exciting frameworks based on async and await.
That's why we created a course for anyone who wants to learn all of Python's async capabilities,
async techniques and examples in Python.
Just visit talkpython.fm/async and watch the intro video to see if this course is for you.
It's only $49 and you own it forever.
No subscriptions.
And there are discounts for teams as well.
Another takeaway I got from looking at this is, wow, it's nice that dictionaries are fast.
And how fast are they to make this impossible to lean on them so much to make this happen?
Yeah.
I mean, I will say when Guido chose to use dictionaries as the namespace, it was kind of a unique decision at the time.
It might still be, honestly.
But it did simplify some things conceptually and also made it so that any wins in how dictionaries work is just a massive win universally across language.
Right.
Like if you make dictionaries work in any way better, you not only make it work when you just you personally use it as a data structure, as a container,
but also literally every attribute access will get faster.
Right.
Like, yeah, there's a reason why it's so fine tuned and why we don't really touch it very much because it's been tweaked over the decades by a lot of people to be extremely fast and for good reason.
Yep.
Indeed.
One final thought on this.
You know, you think of optimizing both the memory and also the performance a little bit by doing things like slots on your classes to maybe like short circuit some of this.
It feels like somewhere in this world lives like some kind of massive optimization that maybe that if you said I'm going to for this class, I'm going to give up some flexibility.
Is there some way attribute reads and writes could get lots faster?
I don't know.
What are you?
I'm sure this has gone round and round because it's such a hot.
It's just involved in almost every line of Python code, stuff like this.
Right.
Oh, yeah.
Yeah.
I mean, this plays into what the faster CPython team has been doing, right?
Trying to make things faster by looking at what the patterns are and if you can just kind of skip stuff.
Right.
Like, how do you short circuit this thing?
Can you go like, well, I just know this object's always looked this way.
Like, this is some of the stuff that the self programming language team at UC Santa Barbara did, I think, in the 70s, maybe 80s.
I'm just like, well, if we know the layout of the object is going to be consistent.
Yeah. Can we go seven bytes in and get four bytes?
That kind of thing, right?
Exactly.
And you just kind of have to, and you start to be able to calculate the shortcuts.
You know, like, hey, did anything change from the last time I made this assumption?
No? Okay.
Well, then I can use my assumption.
That's way fast.
Right.
Nobody's mocked it.
Nobody's dynamically messed with it.
Like, maybe it is just a plain thing.
Until somebody puts it into an edge case, is there a short circuit, right?
Yeah.
And, I mean, I don't want to go off on a tangent, but...
Well, I'm one of 17 topics here, Brett.
Like, I know probably we got lots of time.
I have thought about, like, after doing this, what...
Is there something syntactic here to add, right?
Is there a way to kind of come up with effectively a class definition that's way simpler for the really common cases?
Yeah.
That could be optimized extremely fast because you give away some flexibility and then the interpreter can make assumptions.
PyPy could make assumptions, et cetera, et cetera.
And would there be enough wins on that to warrant doing it, right?
Like, could we take the concept of kind of a data class that was just data, roughly, or something along those lines?
Yeah.
And just what could we get out of it?
And what kind of perf wins could we get out of it?
And would it be worth it?
Would it be worth adding syntax to the language for that case?
If you look at Mojo, for instance, they have a struct.
And that struct works in a very specific way for performance because when you say, like, this thing can't have new attributes, which you can do with slots.
And for those of you who don't know what slots are, if you define a dunder slots attribute on your class, you're effectively telling Python, don't create a dictionary for me.
Just create effectively a list underneath.
It's technically a C array.
And when I need to access something, just get it off this array.
So there's no dictionary.
You can skip all that dictionary overhead.
Plus, you shrink the memory because now the only amount of memory you need is that array for pointers to the PyObjects and not a dictionary, which grows and shrinks and who knows what size it's going to be.
If we had that built in syntactically, would that be beneficial to people or not?
And I have a prototype in Python code of what I'm thinking of.
I did a blog post on this earlier this year to try to feel out people in the community and what they thought in terms of restrictions and would it be too much or too little?
I subsequently prototyped what I wanted.
And there'll be a follow-up blog post about this is where it sits now.
What do people think?
And do people think this is worth trying to get syntax for?
Just don't worry about it.
It's too close to data classes or what?
Maybe it could be a keyword to the data class decorator that just makes it behave differently.
Who knows?
Yeah.
But so unfortunately, the tricky bit with that is it's still flowing through Python code, right?
Data classes is still just a module in the standard library.
So you can't, it's really hard to optimize that stuff without syntax to specifically say, hey, this will be different.
You get to treat it differently and we can literally lock it down by literally creating a different type of object at the C level that just literally doesn't give you access, right?
Like tuples are a good example.
Tuples at the C level very much just don't implement the stuff that you mutated.
What happens if we want to do that for classes?
Like how do you do frozen data classes?
It's a ton of properties.
But what happens if we had a way to just say, yep, nope, we're just literally not giving you access.
Would that, how much faster could that be?
I don't know.
But I suspect it would be decently faster.
Yeah.
Computers are surprisingly fast.
Dictionaries are surprisingly fast.
But there's a lot of stuff here that could be skipped that I think might be.
To me, I had the same thought at least.
Probably not with the level of sophistication you did.
But like, wow, there's some way to sort of do a more of a C style, just like we know the structure and the offsets.
Yeah.
Like for a restricted set, because that restricted set is mostly what people do actually do in Python.
They don't go crazy dynamic with stuff coming and going like sometimes, but not usually.
Yeah.
I mean, you can go read my original blog posts.
And if you follow me on Mastodon, you can see the follow up conversations.
But like the original thing, if you've been a longtime Java user, you'll know the acronym POJO, plain old Java object.
It was effectively the equivalent for Python.
Some people love that idea.
Some people were like, I really need methods.
I can't not have methods kind of thing.
It really depends on what people are after and what they think makes sense.
But there will be a follow up blog post.
So if you follow my blog, you'll eventually see what this hopefully all leads to, which may be nowhere beyond what I put up on PyPI.
But who knows?
Maybe it'll become Zindax someday.
It really depends on how the community reacts to it.
Cool.
Well, I didn't know you're looking into that.
So very interesting.
Next one.
Let's move on to something that also seems pretty straightforward is A minus B.
Yeah, right.
Yeah, exactly.
You think it makes.
Yeah.
How hard can A minus B be, right?
Like two objects just subtracting from one another, right?
We all know how this works.
There's I mean, there's an opcode just called binary subtract.
Just like there's a binary add, a binary multiply, binary divide.
You put the numbers on the register.
You do the instruction set on the CPU and you're good to go.
Yeah.
Hey, I know what five minus three is.
How much work can it be to make that work in Python?
Oh, yeah.
The numbers happen to be arbitrarily large and sometimes they're not numbers.
Sometimes they're matrices or strings or.
Oh, boy.
Okay.
Exactly.
And subtraction is not communicative.
So what matters here.
So, yeah, it gets surprisingly complicated very quickly.
Yeah.
So, yeah.
From from a conceptual level.
Yeah, it seems simple.
But when you figure out what that opcode actually does, it becomes way fancier.
Yeah.
And so as you dig into this, you start to realize like, OK, so this is actually controlled
by the Dunder sub operator on the type.
Right.
And then how do you get that method?
Well, this is the method resolution order on that type.
So you got to start navigating that.
And that gets pretty interesting.
Right.
Yeah.
Well, and let's be clear here.
You're a bit foreshadowing because you said, oh, you called the Dunder sub.
Yeah.
To start.
If that works.
Right.
Like for a lot of these things, there's lots of fallback to once again, give you that
flexibility to make things work as necessary.
So for a lot of things in Python, you can return the not implemented singleton, which you may
have some people may have noticed and wonder what the hell is this thing for?
And it's a singleton to signal to Python that, hey, there's not it's not an error happened.
It's just I just don't know how to handle this.
Right.
So for instance, if let's say you have left hand side minus right hand side, the first
step is you check to see if the left hand side has a Dunder sub method.
If it does, you can call that with left hand side dot Dunder sub and pass in right hand side.
Well, that method has the option to do something and return a value.
And that's what left hand side minus right hand side becomes.
Or it can return not implemented and say, just tell Python, nope, I don't know how to do this.
Probably should ask someone else.
Right.
And then this adds an extra little wrinkle now, because suddenly when you do that, that
says, OK, well, you know what?
Let's give the right hand side a try.
Maybe the right hand side knows how to do left hand side minus right hand side.
It might not be the front thing, but it might just magically know.
Right.
So in this case, for instance, you could have something of your fancy object minus four,
which would be your Dunder sub method.
And so you know how to handle integers and it works.
What happens if it's four minus your object?
Yeah.
Four doesn't know what to do.
Yeah.
Four's never seen your thing before.
I don't know what the heck to do here.
So in that case, four would return not implemented thing.
I don't know what to do here.
Good luck to you.
Maybe you'll figure it out.
And in this case, your object Dunder, our sub would get called with four.
And then maybe you'll be able to say like, hey, I might not be on the left hand side,
but I know how to handle myself on the right hand side.
And I can make this work.
So we try to make sure that even if some other objects don't know how to work with you,
as if you're involved, there's at least the opportunity for you to make it work.
Yeah.
That's pretty interesting.
Yes.
An example could be like the number four, a constant minus a vector that's meant to be
like a mathematical vector.
And it's like, okay, we just subtract that from every dimension.
Right.
And that would be totally reasonable.
But the four has no idea how to do that.
Yes.
And this is still the simple case.
Yeah.
So if I'll need a, I need a slight refresher on this one, but effectively there's scenarios
here where you call the right hand side first.
If it's the left hand side, one can say, I don't know what to do with this object, I think
in return and not implemented from the subtract, something like that.
Right.
There's also a subtle optimization in these binary operators where if it is a direct subclass
of the other object, you get first dibs.
Oh, right, right, right.
So that's the thing.
Yeah.
There's a lot of spam and bacon and vegetable spam and stuff that you did in here that people
can check out.
It gets surprisingly complicated.
But the thing here, basically, this is, let's see, you have a left hand side and a right hand
side, right?
Once again, left hand side minus right hand side.
Left hand side might be an integer.
Your right hand side, though, might be a subclass of integer, like fancy integer.
Okay.
Now, if you do int minus fancy int, it will understand how to do it because you're a subclass.
So it's just going to do it.
And it'll just do the math and return you whatever it is.
So like if you did five minus fancy three, five is going to return the integer two.
But maybe that's not what you want with your fancy int.
Maybe you would always, maybe you'd rather get fancy int two.
So how does Python make sure that you get that opportunity if you happen to be on the right
hand side instead of the left hand side?
Yeah.
Well, in that case, Python checks first.
Okay.
I have a left hand side and a right hand side for subtraction.
Is the right hand side a direct subclass of the left hand side?
If that's true, then you know what?
I'm going to give the right hand side a chance first so that they can return their subclass
and not have the left hand side kind of just blindly not realize that it's a different type
and return something else.
So that when it's, as I said, five minus fancy three, we actually give fancy three the first
chance instead of five so that you have the chance to return fancy two like you wanted and
not have five self-return.
Boring two.
Yeah.
So I think one example that comes to mind that I think might be relevant would be working with
units.
So we've got cool libraries like pint and others where you can say this is not just the
number five, but this is five kilograms.
And if I subtract, you know, a hundred milligrams from it, it's not negative 95.
It's, you know, four point, whatever, nine, five or whatever it turns out to be.
And so in this case, if you took a regular number and one of these numbers with units, you
could get a number with units back potentially.
Yeah.
No, that's a very good example, right?
Yeah.
This is where those, once again, very simple syntax that in the common case and in your
general use day to day makes total sense and happens has some really important, but subtle,
if you don't know about them semantics to make sure that those reasonable expectations are met,
right?
Like I totally didn't even think about this when I was doing, when I was writing this blog
post that like, oh yeah, God, that's a good point.
Like I do want to make sure that I get my special version of everything when I happen to be on the
right-hand side, just because I happen to be written on the right-hand side.
Yeah.
Plus is a really good example.
This one, right?
Like you could have written a plus B or B plus a makes zero difference.
I mean, that operator is supposed to be communicative.
So it can be totally just dumb luck based on how I just happened to be thinking that day,
what I put on the left side and what I put on the right side.
And that would suddenly make a difference.
And in this case, it just helps make sure that what you would think is the priority order of who you'd
want to handle this for you gets that priority.
But once again, it's a thing you just never think about day to day.
But in actuality, when you think about the really key ways that Python operates, makes a surprising
difference.
Yeah.
That seemingly leakiness of the language of like, well, sometimes it gives you an answer.
Some it doesn't.
Depends on how you add.
People wouldn't love that.
Yeah.
And then the last bit of subtlety on that one is this only happens if, by the way, Dunder
Sub and Dunder R Sub are different.
So that's another check we do because at the C level, they're actually equivalent.
Like there is no left side versus right side thing here.
So we actually check to make sure we don't waste our time asking you, hey, can you do this
and get back not implemented and then calling it the reverse and give me the exact same result
because underneath it all is the exact same method.
Right.
So it's one of those fun things of, once again, lots of little things here to try to optimize,
as you said, as best we can.
And in this case, it's one of those.
Well, if we know you're not going to work this way with the exact same method, we just swap
the order of this stuff.
Why are we going to ask you again?
So we just skip that.
So it's a minor thing, but semantics are there to try to help with the performance.
I think people, as people go through your series here, I think they're going to get a deep
appreciation for edge cases.
Yeah.
Subtlety.
Yeah.
I learned a lot.
I mean, it was one of the reasons I finished it is I finished this series as well, even though
I part way through went like, yeah, you know what?
I'm not going to implement Python from scratch, but I was learning.
And it was fun.
Like I got to dive a bit deep.
I got to ask the Python core team is like from people, primarily Guido, who obviously
were there when these decisions were like, why this way?
Like, why do we choose this?
And just trying to understand somewhat historically, how did these things happen?
Right?
Like I think a lot of people don't realize how old Python is, right?
Like February of 1991 is when Python came out.
It predates Linux going public.
Like people really forget that fact.
It's been around for over 32 years.
Yeah.
Decisions were made a long time ago.
And I know everyone got all upset over the two to three transition.
And some people still got upset whenever we push new changes or whatever.
But do realize.
Yeah.
I can't believe you just canceled three, seven.
I like that version.
But the key thing here is the core concepts still work.
Subtraction still works, right?
From the way it worked back then.
There's not been a massive upheaval here.
It's just a lot of us end up using stuff that's kind of fancy.
And sometimes the fancy stuff gets a little tweaked and stuff.
But the really core low level stuff really hasn't shifted very much.
So if at all, depending on your view.
Give people a sense of the timing.
Like the release of Python predates by a couple of years, the release of the web.
Right?
The web was 93, I think.
And so, yeah.
Yeah.
One of the early browsers was actually Guido was writing in Python and TK.
Incredible.
Gradle.
So we talked about simple things like thing dot value A minus B.
The next simple one is the import statement because imports are simple.
Honestly, I think imports are, if you come from a compiled language,
they're really, you've really got to change your mindset.
Right?
And so I think this one actually is maybe the opposite end of that spectrum a little.
Oh, yes.
I will fully out my code.
Some of my coworkers who are, because VS Code is written in TypeScript as an electron
app.
So they live in TypeScript all day long.
The fact that Python's imports work the way they are in terms of flexibility and just the
way it works.
And you're not just specifying file paths to everything just really throws them for a loop
and they go like, why is it so complicated?
Once again, the flexibility that Python gives you where the base case, normal case of if you
just do things the way just kind of just it's Python code and directory kind of thing just
works versus I can also import something from a URL if I want.
That's totally optional.
You know, I could keep all my code in a SQLite database and import from there.
Totally possible.
Right?
So it's one of these things, once again, where if you're coming from a Python mindset
of just what makes sense, it all just works.
I realize when you come in with an outside perspective, it can look kind of crazy.
But you also have to understand that a lot of the designs were made A, decades ago.
And B, there's a lot of flexibility there that you may or may not be taken advantage of.
But there are others who are.
And so that's kind of why it exists that way.
Yeah, indeed.
And yes, this posts.
Yeah.
So for those of you who don't know, I actually am responsible for Import Lib and re-implementing
Python's import system in Python itself.
And so I know these details a little too well.
This one you live deep in this world.
Yeah.
So.
You still have nightmares about some of them?
No, luckily.
I'm just kidding.
Luckily, Import Lib, in terms of actual import itself, like this, I'm ignoring Import Lib.resources
or Import Lib.metadata.
Jason Coons mainly manages those modules.
It's been stable enough for long enough now that there's no real crazy surprises anymore.
Honestly, the biggest headache is people doing imports in their threads.
So my word of advice to everyone, don't do imports in threads.
If you're going to do imports, do them on your main thread up front.
Then spawn your threads.
Don't do imports as a side effect of anything either.
It's circular imports.
Same thing.
It's just, just, just don't.
Your life will be so much easier.
But that's honestly where any of the bug reports that ever come in come from is due to either
circular imports and people wondering why or weird race conditions with threads on imports
and just trying to get the locking down such that things lead to the right outcome.
Yeah.
Actually 3.12 almost got held up a little bit due to a slight memory leak from the newest
version of tracking imports via threads.
There was a slight leak of it.
Wow.
Basically an empty dictionary every time you did an important thread and we didn't want to have
that leak.
So we cleaned it up.
But yeah, it once had a lot of subtlety.
Oh, we got lots of memory, bro.
We don't need to worry about those things anymore.
No, just kidding.
Yeah.
There's no reason everyone gets excited every time Apple launches a new M chip, right?
We've had enough processing power since we got someone on the moon with how many hertz?
Exactly.
No one needs more than seven hertz.
You know the favorite thing.
You know, one of the real big differences I think here is that like import runs code,
right?
Like rather than tells the compiler these things are in scope if you want them.
But no, it's like running code.
And if people are wondering where that weird dunder name equals dunder main, so you don't
run too much code during imports and those sorts of things, right?
Yeah, actually.
And this is an interesting case where it's very much syntactic sugar, but it's also very much
syntactic sugar actually for the bytecode.
Yeah.
So the thing here is when you do import spam, right?
Or whatever module you want to import, it effectively ends up being a call to the dunder
import built-in function where it passes in spam as the name, and then it passes in all
the globals and the locals for where it gets called for name resolution and stuff like this
is where you do relative imports with dot, you know, like dot, dot something.
Yeah, exactly.
When I first saw it, like, why are all these arguments going to import?
You just need to know where a, a dot pi or the package a lives and you're good to go
until you say, you know, dot, dot this or from dots.
Yeah.
From dot, dot a import B kind of stuff.
Like there's a surprising amount of information you need to pass in to do those kinds of resolutions
and such from imports, right?
All that kind of thing to make sure those things all exist.
Because what you're effectively doing, you know, from import is if you list multiple ones, you're
basically importing the module that those things are in and making sure that attribute gets set
on the thing you're wanting so that you can then grab it later and all sorts of stuff.
But the other key thing is, is the Dunder import function is structured to kind of make the
up code easier, right?
Because it's very structured to just push things onto the snack that Python's executing and then
make the call to Dunder import where it's just like, yeah, everything's already there.
Right.
But it's also why it's so funky.
You should never call it directly in your own code, right?
Like Dunder imports very much.
It's borderline implementation detail on this kind of thing.
This is why you choose import lib dot import module, right?
That function exists specifically.
And I designed it specifically for those cases where you need to do dynamic imports.
And you want a very clean, simple API because like the return type for Dunder import very much
is designed for the op code so that whether you're doing an import something or from
something import other thing, right?
Like the return value makes sense or that for the bytecode level does not make sense for
human beings, right?
So once again, do not call Dunder import directly.
You should always be calling import lib dot import module if you need something dynamically to
import.
Or just use the word import.
Yeah.
Or just import.
Yeah.
Just import.
Yeah.
In your right up here, you have all the different variations.
Like what does it mean to say from thing import something as or just import something or
relative imports?
So people can see all the different, again, back to that flexibility that you talked about.
Exactly.
So it's definitely one of these things though where in the end, it effectively just boils
down to a function call, right?
Like that piece of syntax of import really is just a function call with an assignment in
the end.
Yeah.
And it's really all it is.
It's the unraveling.
All the trickery is really behind the Dunder import function, but the actual syntax is
not, it's not crazy actually.
But it literally is a function call.
Yeah.
You can fully re-implement import.
Yeah.
But just calling Dunder import and just doing the proper assignment.
Nice.
Some confirmation of your advice out there.
Nathan says, my data science professor always says the same thing.
Import first, which is good.
But also, is there a way to remove syntactical sugar on an example, but not fully through disassembly?
You can actually write a tool that will take Python syntax and unravel it.
I actually started it, but it was just too much of a headache because there's so many variations
and edge cases and all that stuff to handle.
And there was no package out there that was going to, that quite did what I needed it
to do to make my life easier.
And I had other things to do.
Yeah.
What's the canonical, what's the destination, right?
Like if I undo a with statement on a lock, like how far do I want to go back?
You know?
Yeah.
Well, and I can understand people wanting to do it to maybe learn, but I mean, it's going
to be a learning thing, not really a performance thing.
It might be slower, right?
Because some of this stuff is happening at the sea level.
Oh yeah, exactly.
Like, like if you go back to our A plus B example, it actually definitely will be slower
as things, as things happen in Python, because once again, with the speed performance improvements
that have been going in, like, like the, the byte code is able to go like, all right, is
this an intent and an end?
Okay.
Well then we're, we know exactly how to do integer math.
Let's just do it.
Right.
And then it can add cards in and go like, okay, quick check AB.
Yes.
Okay.
Do the binary int add thing.
Right.
Where it just very, very obviously does that.
But if you unravel it, you can't do that because now you're just doing method calls.
Right.
You can't hide those details anymore and put all that.
Exactly.
But yeah, but to answer the question, you totally could write a tool to do this.
And I kind of started one and it just wasn't fun.
So I just stopped because I just had a thing.
You know, sometimes you see this in the editor as well, when like editors will give you previews.
So for example, if you use optional of a thing or thing pipe none, and then you hover over
it in the editor, it'll say, even if you used optional, it'll say thing pipe none, because
it, it's like, those mean the same thing.
I'm going to just represent them in one way.
And yeah.
Yeah.
Interesting.
Okay.
We have a bunch more things covered, but we don't have really any time.
Like my goal with this, having you on here to talk about this stuff, it's really kind of
like the highlight, the whole series and people should go.
And we've been kind of hand waving because reading C code on air is not ideal to give people
an idea, but really to highlight your whole series and people can just dive into this pretty
deep.
And, and I think they'll have a much better appreciation for when you write this one line.
Oh my goodness.
This is actually what's happening.
And how I actually found a bug in Python.
Thanks to it.
Okay.
Tell us quick about that.
Yeah.
It's covered in the in place binary operator, augmented assignment post.
So, you know how you can do a plus equals B and it's like doing a equals a plus B.
Yeah.
So it turned out it was broken for pal.
So when, you know, when you could do a star, star B, that's a to the B power.
Yeah.
It turned out the semantics were busted for star, star equals.
Wow.
Okay.
And no one had ever noticed because obviously people do not write custom implementations of
thunder pal.
Yeah.
turned out was, and someone actually discovered it and reported it.
Uh, I just didn't know about it.
that basically when you did that.
So the in place augmented assignment, there's I versions of everything.
Like I add a dunder.
I add dunder.
I mall dunder.
I pow, right?
It turns out for all of them, they did the right thing except for power where it would
check for I pow.
And I mean, in all these scenarios, because a plus equals B is the same as a equals a plus
B.
If the I version returns, not implemented, it falls back to just doing the binary operator
and then doing the assignment.
Right.
So like if I plus, so I plus equals B unravels to a dot I pap, I, a dot dunder I pow with
B as an argument.
If that returns not implemented, then it devolves into a plus B a equals the result of that.
Right.
In the pow case, if you return not implemented on dunder I pow, it didn't fall back to pow,
dunder pow.
Okay.
It just crashed.
You said, I don't know what to do here.
It was just an exception.
It was like, yeah, no.
I suppose that's better than the wrong answer.
Like actually the square of nine is 18.
Carry on.
It threw me for a loop because I wrote code to verify all my unraveling and how did the
augmented assignment for power just kept not working.
It was like, what the heck's going on?
And then I had to dig into the C code.
I was like, what am I doing wrong here?
Yeah.
Wait a second.
That code unravels calling this C code, calling this C code.
And it didn't check that return value on calling it under ipow.
It's like, well, did it return not implemented?
And if it did, then we've got to try the other version.
It just didn't.
It's like, oh, no, it's just the value just got returned.
It just didn't even check.
I think we figured out it happened when the third argument to pow got added and it was just
someone overlooked it.
But I see it been sitting there for years.
Like I think it was a decade.
Yeah.
You're looking through these things.
You can tell that they're pretty subtle.
There's a lot going on in the different special cases.
So I can see it wouldn't hit you in the face that you necessarily miss something right
away.
Exactly.
Yeah.
I don't fault anyone at all.
I think we were just missing a test case somewhere to make sure that that happened.
Because and once again, I don't think people define their own custom pow operator for their
types.
There's a whole bunch of mathematicians going, we do.
We do.
Or physicists or something.
Yeah.
All right.
We're pretty much out of time to go any deeper on this.
And I think that I think people got a really good sense of what's happening, you know,
like what you mean with your series here.
And hopefully they're inspired to go check it out.
But I do.
While I have this import stuff on the screen and you've done so much work with import lib,
will there ever be a time where like hot reload type of stuff is a thing?
You know, like I edit this file in this web app and it's got this project.
It's been running.
And I just, just maybe could we have a file watcher trigger this module to re-import?
Not necessarily saying it's a good idea, but have you.
Do you mean like you want automatic import lib.reload?
Yes.
Or, you know, even like the lib reload stuff.
A lot of times like you can do it, but it's kind of discouraged because I think largely because
of the side effects.
What can happen there?
No, so to be very specific, if you call a module, if you call import lib.reload on a
module, it effectively reruns import on that module.
The problem is, is all the references you have to objects don't change.
Yeah, exactly.
So if you got something out of that module, like let's say there was a global dictionary
that you stored a reference to a way that you expected.
Right.
Or a function or a class.
Maybe.
Yeah.
And then you call reload, everyone else in the future is going to get the new module
and they'll get the new version of all the things that were in that module.
But all that stuff you still have reference to do, that didn't get magically taken away
or garbage collected.
And we can't really necessarily swap it underneath you either because the whole type could have
changed.
Right, right, right.
So, I mean, if you really wanted to set up your own import system that does file monitoring
and Omnic calls import.reload, you could totally do that today.
There's nothing stopping you.
But you have to just be very aware that.
Only sometimes will it have a good effect.
Yeah, it'll have an effect.
Yeah.
It's just whether it's going to do what you expect.
And that's why import.reload is discouraged because people often don't quite understand
the side effects and the things you have to watch out for to understand that it's going
to do what you expect.
And especially when you import, do from imports, right?
Because then you're getting the direct objects off the module instead of the module itself.
Because if it doesn't in place reimport.
So, if you did import spam and you reloaded spam, the attributes off of spam will now be
the new stuff.
Because we actually changed that dictionary in place.
But if you did from spam import function, function is not going to change.
There's another reason why I always tell everybody, don't import the objects from a module, import
to the module.
I personally find it more readable because I can look at the code no matter where I am
in the file and know where that function came from.
It's not from this module.
It's from somewhere else.
I am always with you.
I'm almost never from thing import other thing.
Yeah.
Unless it's a deep module in a package, in which case that makes sense.
Because you'll still do module dot thing.
Right.
Well, even though I have that, if it's like four levels deep, I'll do the first three.
And then from the first three, import the last step of the module.
So, you can say like data layer dot, you know.
Exactly.
Update rather than just update.
Like, well, what does update mean?
I don't know.
Exactly.
Yeah.
I mean, the canonical example is random.
You see the function random in your code.
Whose random is that?
Is that from the random module?
Is that from NumPy?
Did you implement random?
Whose random is that?
Yeah.
You just know.
But if you saw np.random, you'd know.
Exactly.
Yeah.
If you knew what np was.
Yeah.
If you did.
Of course.
Right.
I guess give you a chance to kind of summarize.
Final, final thoughts here on your Syntactic Sugar series and, you know, tell people how
to get it and stuff.
You can go to my blog.
It's on Snarky.ca.
It has the tag Syntactic Sugar.
I'm sure Michael will include it in the show notes.
Don't feel the need to read all of it, but do look at at least the first one or two because
I do go into more detail on how you can kind of go exploring on your own in way more detail
in those posts than I do towards Anne because I'll be honest, I lost a bit of steam and
going diving into all the deep layers.
I wanted to get the blog post series done after two years.
It was one of those.
Okay.
Go look at the earlier ones.
They'll explain how to how to figure out how to go from syntax to the bytecode to the C code
and then know where to look in the C code, which, by the way, you don't have to understand
the C code.
It's not a is not crazy C code, but it's also not necessarily critical.
I do try to write down the Python equivalents for everything.
So it's there if you do understand C, but don't feel intimidated if you don't.
And hopefully you just find it interesting, right?
Like you've always wondered how the heck does this work?
If it's in that, if it's in the blog post, you'll be able to have a better understanding
hopefully of how things work.
Because as I said, I honestly even learned some details that I forgot or never even knew
about some of the semantics behind Python and how, like how things actually work underneath
the hood.
And it just gave me a better understanding to understand when things do and don't happen
the way I expect.
And honestly, appreciate all the work everyone's put into language over the decades to make it
all seemingly seem simple.
And yet it's surprisingly complex.
Yeah.
I don't think you need to read them all either as somebody who read most of them the last couple
of days.
No, you don't.
I mean, if you're a completionist, go for it.
But yes, you definitely do not need to.
But I do think going through it, you know, pick out the ones, there's a whole bunch of
different topics.
Pick out the ones that are like, oh, I did always wonder how that one worked.
And pretty much with the exception of pass, I think you'll be like, oh my gosh, I had no
idea what was involved here.
Yeah.
Pass was very much just a, I had a checklist of every piece of syntax and every single keyword.
And that was that the blog post on pass, I think is like three sentences and half of
it saying this is going to be the shortest, shortest post in this whole series.
Yeah.
Pass is.
Yeah.
It was contending with dot, dot, dot, which we haven't even had a chance to talk about
ellipses versus pass, but I think we're out of time, Brett.
So thanks for being here.
Thanks for all you do.
I mean, steering council, core dev, author, VS Code, et cetera.
A lot of contributions.
Thanks.
Thanks for having me back on yet again.
And hopefully I've not worn out my welcome still.
No, you're already thinking about what next WebAssembly Rust thing we'll get together on.
Thanks for being here.
See you later.
Thanks, Michael.
This has been another episode of Talk Python to Me.
Thank you to our sponsors.
Be sure to check out what they're offering.
It really helps support the show.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at Talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show.
Open your favorite podcast app and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play,
and the direct RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air,
be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I'll see you next time.