A recent Twitter poll went around the web, and it asked,
what percentage of the Python standard library do you think you know?
Someone copied me on it, maybe expecting some really high percentage answer like 80-90%.
In reality, what I did answer, and my rough estimate still is, that's probably around 50%.
This episode with Alessandro Molina definitely helped confirm that estimate for me.
He just published a book entitled Modern Python Standard Library Cookbook,
and it's full of these great little corners of the standard library that you might not have bumped into,
but you'll be super glad to hear about them on this episode.
It's Talk Python to Me, episode 197, recorded January 10th, 2019.
Welcome to Talk Python to Me, a weekly podcast on Python,
the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy.
Follow me on Twitter, where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm,
and follow the show on Twitter via at Talk Python.
This episode is sponsored by Linode and Rollbar.
Please check out what they're offering during their segments.
It really helps support the show.
Alessandro, welcome back to Talk Python.
Hi, Michael. Thank you.
Yeah, it's great to have you on the show again.
It's been a long time since back on episode 35 when we talked about Turbogears,
and you're back with a new project that I think is really, really cool,
a look at the standard library through a modern Python lens,
which I'm excited to dig into with you.
But first, maybe just tell us, what have you been up to the last couple of years
since we heard from you on the show?
Yeah, actually, I've been mostly doing Python development as usual.
That's my most prominent interest in open source development.
I started a bunch of open source projects aside of Turbogears,
but I'm still moving Turbogears forward,
and we actually just released version 2.4,
which had a major rewrite of the framework,
which is great because we are trying to keep it modern.
The standard library is doing a great job at that.
Yeah, that's really cool.
What are some of the other projects that you've been working on,
the other open source ones you've released?
One of the most interesting projects, I think, was DuckPy,
which is, I will say, JavaScript execution environment for Python,
because one of the problems that I faced
was integrating the assets pipeline in web projects,
and that always requires you to have Node.js involved
and Webpack or tools like that.
And I really wanted to have a solution
that worked in the Python environment.
Like you did pip install,
and you got not only all the Python dependencies,
but also the JavaScript dependencies
and your ECMAScript compiled and things like that.
That's pretty cool.
So it's kind of like what people are using NPM for
on Python web apps to manage Bootstrap
and AngularJS and stuff along those lines?
Yeah, it's a suite of various tools, I would say,
because it provides a JavaScript interpreter,
which is used to run things like the TypeScript compiler
or the Babel compiler.
So to translate your JavaScript from more recent versions
to the oldest ones,
you don't need to install Node.js anymore.
You can use DuckPy and have Python do that for you.
And of course, it provides a command
which is compatible with NPM.
So you can install NPM packages from Python
without having the need to have NPM or Node.js.
It's not super easy to explain
because I think that people frequently get confused
about its purpose when I try to explain it.
But the one idea is that you can have,
like, run just run your pip install
and you can have your whole application compiled,
JavaScript files compiled back to plain old JavaScript
and all your dependencies installed,
even if they are from JavaScript
instead of being from Python and things like that.
So it makes your life a lot easier
because you don't have to care
about maintaining two different packaging environments
and more to different interpreters and things like that.
I really like it.
That sounds awesome.
Cool.
So let's maybe talk about this idea
of the modern standard library.
So, you know, there's a whole range of people listening
who have different levels of experience with Python.
Let's just start with what is the standard library?
Standard library is practically everything
that ships with Python itself.
I mean, it's known that Python ships,
but it is included.
And the standard library have those batteries.
Yeah, exactly.
They're the batteries that come in the box
when you get Python.
You can get other batteries,
but these are the built-in batteries
or the ones that come included, right?
Yeah, I would say that they have far more than batteries
because there is tons of things that it can do for you.
There is a lot inside the standard library.
And I think it's also something that is not super frequently covered around
in blog posts or things like that.
We always tend to have, like, blog posts on the most recent new cool project
or things like that that is outside of the standard library, of course,
because things get included in the standard library after years they are around.
But that's actually good because it means it's something you can really rely on.
Absolutely.
It's important that that has to stay there.
And there's actually a huge bar for bringing things into the standard library.
I recall a year or two ago,
there was a debate about whether requests should be brought into the standard library
to more or less supersede the built-in HTTP client capabilities.
And they decided no, not because they felt requests wasn't good enough,
but because requests was changing more quickly than the standard library could really facilitate, right?
Because it's released with new features really every 18 months and things like that.
Yeah, absolutely.
That's one of the problems of the standard library,
but also one of the reasons why it's very good.
Because the things you have there, you can rely on them for the years to come.
It was especially clear in the recent, in the last years,
where Python changed very quick.
We released many versions of Python in a very quick time.
And it was clear that even if you are on Python 3.2,
and you upgrade to Python 3.7,
99% of what you use from the standard library is still there,
works exactly like before, and you can be fine.
Of course, between Python 2 and Python 3, there were major changes,
but that was expected due to the B.
That was intentional, right?
That was like, okay, we finally have to just bite the bullet and make these changes.
But other than that, you're right, it's really stable.
There was some blog post or something about somebody wrote how they hated Python,
and one of their reasons they claimed was,
well, if you have 3.5 and you upgrade to 3.6, it might not work.
I'm like, no, that's actually exactly how it works.
And I'm really impressed with the stability of Python as it changes.
I haven't seen any problems.
The only problems I've encountered is I've used features too new on my dev machine,
and then I pushed it to production where I didn't realize,
oh, yeah, that feature is not yet on my server.
So that's my own fault.
Yeah.
That happened to me from time to time too, of course.
Yeah.
Not often, but unfortunate when it does.
So I think actually when people talk about how amazing Python is,
or when they judge any programming language and they compare it against another programming language,
they might compare the syntax.
They might say, well, look how much easier like a foreign loop is in this language versus that language,
or this exception handling block is cleaner than that exception handling block.
But while that matters, I think actually what most people have in mind when they think about how they feel about a language
is the standard library of the two languages and maybe the broader ecosystem as well.
When I think of like, why is Python awesome?
I don't think, well, because the way the language works with numbers is great.
I think, well, I can import all these things and solve all these problems right away.
And that's really the standard library, not the language, right?
There are many things that are built in Python.
They're like whenever you use the dictionary or list or things like that.
They are built into the language itself.
But every time you do an import, it means you are going to the standard library.
Unless it's a package you installed explicitly, of course.
Of course.
So let's talk about your book.
And your book is called the Modern Standard Library, Python Modern Standard Library Cookbook, right?
And I like these cookbook ideas because you're like, well, I'm trying to solve some networking problem.
Oh, here's two little recipes that I can use to solve that problem.
So let's just start with what you mean by modern in the standard library cookbook here.
And then we've selected a handful of specific recipes that we'll talk about that are pretty fun.
Yeah, that's the modern part was the hardest part for me to find out to best decline in the implementation of the book, I would say.
Because it's, of course, modern because it covers Python 3.
And most of the things that are there are Python 3 specific.
But I didn't want to go for, like, the only latest Python version.
At the time where I started writing the book, 3.6 was just released.
So we are now at 3.7.
And it's probably going to happen pretty soon, 3.8.
So I didn't want to follow the most recent thing because I know that in the real world, on your job, you are probably not going to be allowed to upgrade Python every single time when your release happens.
So you probably are going to need the recipes that you can apply on your daily life on version of Python, which are modern, but not the most recent one.
So I will say that I tried to think to things that work on Python 3.5.
And I think that that's actually clear on some recipes because there are different ways to do those things in, like, more recent Python 3.6 or 3.7.
But I still went for the way that you can do that in the works with Python 3.5 and the subsequent version.
So I tried to balance between being modern and covering as many users as possible.
You know, that's always...
Yeah, it's a tough balance to strike.
But I do think the choice of targeting Python 3.5 is pretty good.
Like, really, if you're using Python 3, most people are on 3.5 or higher at this point.
And so, I mean, you do give up a few cool things.
You give up data classes and you give up f-strings and a couple of other things that would be really nice to mix in here.
But at the same time, it's a little more timeless, a little more broad this way.
All right, well, let's just talk about some of the recipes you have in here.
So we pulled out some of the more interesting ones.
You know, what I liked about going through these in your book was a lot of times I'm like,
oh, I didn't know that class or that function existed and did this.
And whenever I'm surprised like that, I'm like, oh, you can send emails out of the logging framework?
That's pretty awesome.
I didn't know that, right?
So I think these will be pretty interesting to folks.
And let's just start with that, like reporting errors in production.
So I guess we should maybe frame this a little bit.
There's lots of ways to report errors in production or do the other things we're going to talk about
if you depend on some external library, right?
But the goal of your book is how much awesome stuff can you do without installing or depending on other libraries,
unless you absolutely have to, right?
Yeah, absolutely.
I think that you say that the right thing.
One of the reasons why I wanted to start this book is exactly because there are a lot of things that people don't know that are available in the standard library,
or they don't know that they can easily be changed to work in a different way or things like that.
So like, for example, you mentioned the logging module.
And we probably all know that it exists.
And many of us are using it on the daily world to log messages on things like that.
But the interesting thing is that it has many different handlers.
So you can send the output of what you are logging in many different places.
And one of those handlers actually sends the output to an SMTP server, so by mail.
So you could send your logging messages by mail.
Right.
So you've got a logging.handlers.smtphandler class, right?
Yeah.
You just plug that in.
Yeah, it's built in.
Exactly.
And, of course, sending all your logged messages as in an email doesn't make too much sense because it will make your life very hard.
But for some specific messages, it might make sense.
And as you can configure the logging method to filter only some messages, like in this case, we are talking about logging errors, so logging exceptions.
You can use the logger to only report exception by mail.
So every time an exception happens in your code, you get notified and you are aware.
You don't have to wait for the user to come to you and complain that the software is not working.
You can answer, I already know.
I've already fixed it.
Yeah, and that's really cool.
And in your example, you do this a lot, actually, and I like this, is you create decorators that you can use to, say, decorate a method and say if there's an error here, you know, exception here, email it to us and things like that, right?
Yes, exactly.
In the recipe, it explains how to achieve that with decorator, which usually you will throw the decorator at the main function of your program so that every exception that happens is reported to your user.
But if you are, for example, writing a web application, you might want to decorate the whiskey main callable because the main application is actually the application server, not your own code.
But the core idea is that if you apply that decorator to the beginning of your code base, then everything that crashes within your code base will be reported to you and you'll know when the program fails.
This portion of Talk Python to me is brought to you by Linode.
Are you looking for hosting that's fast, simple, and incredibly affordable?
Well, look past that bookstore and check out Linode at talkpython.fm/Linode.
That's L-I-N-O-D-E.
Plans start at just $5 a month for a dedicated server with a gig of RAM.
They have 10 data centers across the globe, so no matter where you are or where your users are, there's a data center for you.
Whether you want to run a Python web app, host a private Git server, or just a file server, you'll get native SSDs on all the machines, a newly upgraded 200 gigabit network, 24-7 friendly support, even on holidays, and a seven-day money-back guarantee.
Need a little help with your infrastructure?
They even offer professional services to help you with architecture, migrations, and more.
Do you want a dedicated server for free for the next four months?
Just visit talkpython.fm/Linode.
It looks like you could probably take this idea and extend it.
Like, if you wrote a web service of some sort that you could call and report your errors, maybe log it in your own database, you could create your own custom handler, stick it into the login framework, and do the same thing.
Instead of sending mail, it logs it remotely to your service, which goes in your database for reporting and whatnot, right?
Yeah, absolutely.
You can actually write your own custom handler and log the messages and the exceptions wherever you prefer.
But I think that, as usual, most of the modules available in the standard library are very good when you have an average, medium level of complexity of your needs.
Because when you're using the standard library, it's a great way to report your errors.
So, you might want to switch to a national external service that is dedicated to that.
Sure, but maybe you have some level of data protection or you don't want to share your tracebacks and all that kind of stuff, right?
So, you might want to keep it custom for privacy or IP reasons.
All right.
So, that's really cool.
And just to give people a sense, this is like three or four pages in the book.
It's not a huge, long chapter on it, right?
These are really quick little things out of many, many recipes.
So, that's very cool.
Another one has to do with temporary objects.
So, what people often do is if they've got to load up a bunch of arbitrary things, they'll stuff them in a dictionary, put the values in there, and pass them around.
But those don't behave like objects.
I can't say container.value.
I have to say container.bracket value or container.get quote value, things like that, right?
So, it kind of breaks this idea of these truly flexible objects that you can just pass around in Python, right?
Yeah.
That's surprisingly one of the most frequent complaints that I hear from, like, new Python users.
They're still not well-versed in the language.
Maybe they are approaching Python for the first time.
And they frequently came to me saying, hey, but I could do these in maybe JavaScript when I can just declare an object on the fly and assign attributes.
Yeah, probably a C++ and a C# and a Java developer wouldn't have this complaint.
No, yeah.
Right?
Yeah, absolutely.
And actually, in Python, that's not hard at all.
You can do the same exact thing in Python.
And there is a fast way to implement an object that allows you to do the same exact thing.
You can implement your own bunch objects, which are able to store anything in a dictionary.
Because in the end, it is nearly the same as a standard Python object works.
Because we store attributes in a dictionary.
But you can access them to the dot notation.
So by saying object dot whatever attribute you want.
One of the really interesting things is that if you don't have too many, too much problems of using a model that is made for doing totally different things,
you can even use the arc parse dot namespace class, which does exactly that.
But it won't be immediately obvious to other people that read your code what are you doing and why you import a namespace for parsing arguments.
But it allows you to accept any property to the dot notation.
Yeah.
So in here, you define a class.
You call it a bunch, which is like just an object that can be extended with arbitrary values.
And the trick is to have it derived from dictionary, have it implement dunder get attribute and dunder set adder.
And then it just reads from and writes to the internal dictionary and converts key errors to attribute errors.
And you have these little anonymous objects you can just create.
Yes, exactly.
And one interesting benefit is also that it inherits from a dictionary.
You can set all the attributes at the beginning in a single step if you want, because you can just provide them as arguments to the dictionary.
Right.
It basically has like a built-in keyword argument initializer you could just use, right?
Yeah.
Which is cool.
The one trick that I thought was nice, I had seen the get attribute and set adder before.
But one of the problems is if you go and ask what type it is, you know, type of or type, parentheses, the thing.
Yeah.
It'll always say bunch, right?
But you showed a way to like extend this just a little bit so it'll actually report whatever name you want it to like in a traceback or in a, you know, rep or things like that.
Yes, you can extend the bunch to report the name you need so that when you receive a traceback or something like that, as you pointed out, you'll read that it's actually a user, for example, and not just a bunch that you never know what's contained.
And you can actually extend that even further to pass type checks.
If you are, for example, trying to use them to emulate some other object or things like that, you can actually extend it to pass the type checks for the other objects.
So it can be pretty flexible and convenient.
Yeah, I really like it.
So what are some of the use cases where you might choose to use that over a dictionary or over a custom, more fixed structure class or something like that?
Well, for example, the bunch is something that I use frequently when experimenting, when writing prototypes or things like that.
I don't want to go and declare the whole hierarchy of classes before I even have clear the idea of what I'm trying to do.
So I frequently end up relying on bunch for that purpose.
And also, for example, in TurboGear itself, they are used to keep around some user-provided values that you don't know what the user is going to store there.
You don't know what the user is going to need to keep around and things like that.
And you want to allow the user a quick access to those attributes without having to look up in addition or things like that.
So it's generally very convenient when you really don't know yet what you are going to store in that object.
Right.
Or it's determined at runtime in bizarre ways, right?
Like, you know, I think of like a CSV file.
And maybe that's some common stuff.
But other things you don't know if they're there and you just want to put it all, you know, load it all up, right?
You just gave me a pretty interesting idea, actually.
It would be easy to extend anything that loads additionally, like a JSON parser, for example, to output bunches.
So you can access the properties in a normal way instead of having to look up in addition.
I like it.
And, you know, we'll talk, you know, at least in your book, you cover like default values and avoiding all the tests, you know, is this key in this dictionary?
And you could probably combine those to come up with a pretty clean API for interacting with data exchange of all sorts.
Yeah.
Cool.
So one of the next ones that you talked about is templating.
Now, Python has lots of templating outside the standard library.
We've got Jinja 2.
We've got Chameleon templates.
You can, like, write your own if you really want to go crazy.
But all those are both external packages with many dependencies and often just found in the web and things like that.
But what I was surprised to see is you can actually extend what string.format means with your own implementation and sort of create your own little mini templating language like Jinja 2 or something.
Yes, absolutely.
That's actually one of the recipes that I used for many years.
And now I was not sure about which way to propose the best implementation because there are new ways you can achieve the same result in Python 3.6.
For example, to the f-strings, you can actually write code in an f-string itself and it would be evaluated.
But for the previous version of Python, that was not available.
And also the kind of code that you can write in f-string is limited to expression so that you can evaluate.
While it's not really suggested, but by subclassing the string formatter and specifying it, you can actually make it able to run any kind of code that you provided in the template.
Like even define a function and call it from your template and things like that.
The trick is interesting because I didn't know myself up to like recent years that the string formatter can actually, whenever you write something between the brackets, it calls the method to look up that value.
So you can actually write anything within the brackets and it will always ask to that method, what's the value of this thing that the user brought within the brackets.
And at that point, by subclassing it and driving that method, you can make anything happen.
Like you can even take code that was written within the brackets and run it and put back in place of the brackets the result of the code execution and things like that.
Yeah, that's a really great trick.
So you basically have a little miniature template language just in a string and you just say format, here are the values and boom, it comes out just like you would.
And you even have what is effectively like a loop, right?
You give it a list of messages and it can basically iterate over them using string.join.
Yeah, so in the specific example that I did in the book for the recipe, it tries to showcase that you can even achieve loops because you actually have list compressions or you can have join or depends on the kind of output that you need to do.
But you can run a list compression within the template or you can run a join within the template and so output a result for multiple entries that depend on the container that you provided.
It's really cool.
I think this is a great example.
And so often you see people sort of imperatively building up strings in code and it's just, it doesn't seem like a great way to do it.
It can't be great for performance.
It's not clear really what the output is supposed to be sometimes.
And this way I think is a lot nicer.
So quite cool.
Yeah, of course, because you can have the wall, you can have a better idea of the wall output that you are going to generate because you see it on the fly instead of having to go through tons of if cases and string concatenations and things like that.
You could have basically a, you know, a multi-line string with more or less the exact shape of what's going to be outputted.
Yeah.
It's just the place is going to be filled in.
It's really nice.
Much like a chameleon template or something like that, right?
A simplified version.
It's not as powerful.
Yeah.
But it's already can satisfy most of the needs that you might have.
Yeah.
And, you know, this is one of those cases where I'm like, I had no idea you could do this.
This is quite cool.
There was a conversation on Twitter recently where somebody sent out a poll and said, what percentage of the standard library do you think you know?
And it was, you know, zero to 20, 20 to 40%, 40 to 60, and so on.
And they copied me thinking, I don't know, maybe they're thinking I would check like, oh, yeah, like 90%.
But it's because of this stuff.
I'm thinking like 40% is the right answer.
Because I know how to do the really common stuff super well.
But there's all these little extra amazing things that it's like, I didn't even know that existed.
And so, for example.
Yeah, I totally agree with you.
Maybe I'll ask you, what do you think?
What percentage of the standard library do you think you know?
And after writing this book, maybe that number went up a lot.
Oh, yeah, absolutely.
I think that that's a really hard question because I believe I probably don't go over the 70%, you know?
And I wrote a book about it because there are so many things that are hidden inside.
There's probably not even people that wrote the standard library know every way you can use it, you know?
Yeah.
It's pretty incredible.
Anyway, back to the topics.
This next one also falls into that, oh, that is so cool.
I didn't realize this was around for us.
So I know about working with memory.
I know about working with files.
But you have this cool example of I would like to basically cache something in memory or load it into memory.
But if it gets too big, I need to switch what I'm doing, maybe save it to a file and start reading it from there.
Because, well, if it's 20 gigs, that's probably not going to work out super well most of the time, right?
So tell us about this.
So that's one of my favorite tools in the standard library.
When I need to, like, keep around some data, temporary data, maybe a file that the user uploaded to me or something,
or something I'm generating, like think of an image you are reciting or things like that.
It makes sense to do that in memory because it's the fastest option, so it's usually the best way to go.
But at a certain point, if you really don't know the size of the input that you are going to receive,
if you are reciting, like, 300 pixels image, it can fit in the memory of every computer.
But if you are reciting 5 gigabytes of JPEG, it can start to be a big problem.
And that's the point where having something, a data structure that automatically switches from memory to disk
on a specified threshold can be very convenient.
And that's exactly what the temporary file spooled temporary file class does.
You create one, and at the beginning, everything is in memory up to a point where it grows so big
that it goes over a threshold that you choose.
And when the threshold is surpassed, everything switches to disk, and you don't consume memory anymore.
That is so cool.
So this is the spooled temporary file, and you just give it a maximum size, and you work with it like a regular file.
You can write from it, seek on it, read from it.
And either it stays in memory, or if it turns out that it was got to be too big,
then it just, you know, writes itself to disk and streams off the disk.
That's cool.
Yeah, absolutely.
Because I frequently saw this pattern implemented by many projects, like use byte.io if the size is small,
and switch to temporary file if it's big.
But it can actually be done for you without having to write any code if you use a spooled temporary file.
This is cool.
To me, it feels like that's something you see a lot in Python when people come from another language
or some other technology where they're like, oh, I need to do this thing, so I'm going to implement it from scratch
when it could just be temp file dots, well, temporary file.
You don't need to implement it.
It's done, and it's already tested, and it's fast, right?
I think that's part of knowing the standard library well, right?
Yeah, and it cannot only make your life easier because it's, of course, probably more tested
and been around for years and more robust and things like that.
But it can usually be actually faster because many parts of the standard library are implemented in C,
so they can be far faster than the code that you've wrote yourself in Python.
So it's usually a very good idea to look up into the standard library before trying to write something from scratch.
That touches on this idea of is Python fast or slow?
And I feel often that the answer is both or either.
Something like this, right?
Like you can write code that just runs in pure Python, and it can be very inefficient.
But as soon as you work with something that just hands off something down to a C layer,
either that's in the standard library in CPython or if it's in, say, NumPy or something like that,
like all of a sudden that whole conversation changes.
And this is a little bit of like you may pick up that advantage automatically
by just using the standard library stuff better.
Yeah, I think that it's the only, if not one of the few languages,
where the pattern on performance is reversed.
The farther away you stay from the machine, the faster it goes, you know?
Yes.
The more you try to work with low-level data, the slower it will be.
The more you try to work with higher-level data and functions, the faster it will go.
That's a really good perspective.
I like it.
This portion of Talk Python to Me is brought to you by Rollbar.
Got a question for you.
Have you been outsourcing your bug discovery to your users?
Have you been making them send you bug reports?
You know there's two problems with that.
You can't discover all the bugs this way.
And some users don't bother reporting bugs at all.
They just leave, sometimes forever.
The best software teams practice proactive error monitoring.
They detect all the errors in their production apps and services in real time and debug important
errors in minutes or hours, sometimes before users even notice.
Teams from companies like Twilio, Instacart, and CircleCI use Rollbar to do this.
With Rollbar, you get a real-time feed of all the errors so you know exactly what's broken
in production.
And Rollbar automatically collects all the relevant data and metadata you need to debug
the errors so you don't have to sift through logs.
If you aren't using Rollbar yet, they have a special offer for you, and it's really awesome.
Sign up and install Rollbar at talkpython.fm/Rollbar, and Rollbar will send you a $100
gift card to use at the Open Collective, where you can donate to any of the 900-plus projects
listed under the Open Source Collective or to the Women Who Code organization.
Get notified of errors in real time and make a difference in open source.
Visit talkpython.fm/Rollbar today.
The next pattern that you talk about is with displaying progress bars.
And for all the Python developers, you know, think when you type pip install a thing and
you see the little downloading progress bar going across in the terminal or command prompt,
like those kinds of progress bars, right, on the terminal.
That's a very common need.
And actually, I came up with this recipe because I wanted to showcase a more general need that
frequently you have when writing terminal text-based software, which is you really don't know the
environment where you are going to run.
You know, maybe my user has a terminal which is very small or maybe a full screen or things
like that.
So when you try to provide your output, it's always hard to find the right balance of how
much I should write, how much I should write on a single line, when should I go to a new line,
and things like that.
And the progress bar is a perfect example for that because it should always consume as much
space as available and write the progress on top of that, you know.
So it was a perfect case to showcase that in the final library, you have tools that allow
you to inspect the terminal, understand what's the size.
And we very, like, I don't remember the specific length of the recipe, but I think we are talking
about, like, less than 10 lines of code.
You can write a fully functional progress bar that fits and adapts to the size of the screen
and things like that.
Yeah, it's a great little example.
And of course, there are things like TQDM if you want to go outside, right, which is a
cool progress bar.
But that's not the same as I have no dependencies and I still have a progress bar, which is pretty
cool.
Yeah, exactly.
Yeah, if you want to cover all the cases, like things like TQDM are very cool when I use
it in some projects, I think it works great.
But, you know, one thing is using five lines of code function and the other is bringing
in a whole library.
For sure.
One of the things I liked about this recipe is you used a lot of cool parts of Python.
So you have a nice decorator that you put onto a function.
And that function long as it is a generator that returns numbers that can drive the progress
bar.
So basically, you do your work and you're looping through it and the function can just periodically
yield out a number, which is one to 100, which is the progress, right?
That's a great pattern.
Yeah, absolutely.
I think it's very convenient compared to, like, the most common solution that I saw around
when implementing progress bar is actually to go for a territory that you, like, throw up
around the original data source.
And at that point, it computes the site itself, but that requires that you already have all the
data available.
Like, I can apply that pattern to a list because I know that a list has five elements, so I can
compute how much progress I should do for each element.
But I cannot apply that to a generator because I might not know how much data is going to
generate a generator.
And with this pattern is the best balance, in my opinion, because you just yield the progress
yourself.
It doesn't matter what you are iterating.
It can be in the network input.
It can be on a generator.
It can be on a list.
It doesn't really matter.
It's just the great you're financial with the progress bar decorator and yield the progress
that you want to report.
I really like it.
And it's quite cool.
So the next one has to do with the overall safety or consistency of a block of code.
Now, a lot of times people think if they have a try and they have an accept catching block,
they've handled the exception correctly and everything is fine.
But that's sometimes true.
But a lot of times there's an iterative or multi-step process where a bunch of changes
have to be made and either all the changes should be applied or none of the changes should
be applied.
You know, think of a database transaction, right?
That's very common.
Either you get to the end or you commit it or roll it back.
But that same thing applies in other persistent things.
Even in memory, actually, if you're changing different parts in the memory of your app, you should
consider this.
But this next recipe has to deal with files.
So if I'm going to be making multiple changes to a file or I want to make sure I write all
the file or none of the file, how do I do that?
That's actually a very good question because whenever you handle exceptions, it's obvious that you are
handling the exception in your own code.
But it's less obvious that there might be side effects from that exception.
Like in the case you were writing a file and you fail at a certain point, maybe it's a deep space or something
like that.
What you've wrote so far, it's already there, was already written up to the point where the
candle flashed the bright and so on.
So you are not really handling the exception.
Yeah.
And even if it's not like you run out of disk space, it could be I would need to write 20
things to a file in the 10th wall.
Yeah.
Something is none and I didn't expect it to be and it crashed.
Right.
So it sort of just bailed out halfway through.
Right.
Which is how do you know that's going to happen?
How do you recover from it?
The way you recover is actually not super complex and it can be made a very small tool.
And in the book, there is this recipe which is called Safe Open.
So that allows you to open a file in a safe way for writing.
And whenever you Safe Open, all the writes that you do to the file happen all or nothing.
So if everything succeeded, all the writes happen.
If something fails, like you have an exception in the middle between three lines of code that
draw two pieces, then nothing was written because everything is recovered to the previous
phase.
And that's actually done based on...
I was just going to say, I love it.
That is such a great thing.
Like you write, you just write exactly the same code to write to a file, use Safe Open
instead of Open.
And then either the file doesn't even exist if there's an error or it's completely consistent
at the end.
And this, just to be clear, the Safe Open is a thing that you created in your recipe, but
it's pretty straightforward, right?
Yeah.
It's like four lines of code or something like that.
And it actually was like having a database transaction.
You start the transaction and you can roll back on committees and the middle nothing happened.
It's super cool.
So basically the way it works is it writes to a temporary file and then it uses...
It is itself a context manager.
So you put it in a with block.
If you exit the with block without error, then it will just rename the file to the real one,
what you're targeting.
If it exits with an error, it'll just remove the temporary file, which is perfect.
It's so clean and nice.
And that works because the rename operation is guaranteed to be atomic.
So it's all or nothing.
You replace the old file with the new one or you didn't.
It's not like when writing that you could have written only part of the output.
Yeah, it's super.
Now, this other one is an interesting problem in a more complicated area, but still really good.
So in Python, we have now, in modern Python, you have basically three ways to do concurrency, right?
You've got asyncio, which actually uses only one thread and basically releases that thread to do other work
while you're waiting on like a network or a database or something.
So that's the asyncio and async and wait keywords.
Then we have threads and we have multiprocessing.
And the multiprocessing really exists to get around the limitation of the GIL, the global interpreter lock,
for computational type stuff, right?
Mostly.
Yeah, absolutely.
That's the only way you can currently work around the GIL and Python
because all the other solutions that you mentioned suffer from that problem that you can actually run only one operation at a time,
which is obvious in the case of async because that's the inner behavior of the feature itself.
But it's less obvious for new users for threads because they expect to be really parallel.
And that's the way frequently when talking about parallel operations in Python,
multiprocessing is suggested because that way you can actually go concurrent for real.
So the way you get around the GIL, the way multiprocessing works is in your code,
you say, I want to run these functions with these 10 different values in parallel on different,
you know, on different basically processes.
And Python will create 10 sub processes or it can pull it up or whatever.
But the point is there are multiple other Python sub processes doing the work.
And then you get the answer back.
But the challenge can be if I'm doing threading or asyncio,
I can just have an object and share it and change it with all the different parts of my code.
In multiprocessing, that memory is not shared by default.
So then how do you interact with like if different parts of my multiprocessing thing are generating data
and they need access to it?
Like what do I do?
Yeah, there are many tools in Python for doing that.
But I think that the most powerful one is the multiprocessing manager.
And it's also there are very few people that use it on every day and then know it exists and things like that.
And even between those that know it exists,
even less people know that you have some very cool features.
Like you can actually replace a whole database system with multiprocessing manager
because not only it allows you to share the values that you want across the processes that all forked from the same parent.
So when like using writing a multiprocess program that forks multiple children to do some job.
But it also allows you to share data across processes that have nothing in common.
They were started on totally different times, different places, on different machines even.
Because the way the multiprocessing manager works is that it actually allows you to listen on a port.
So it works on a plain TCPIP network protocol.
And you fetch and store values in the multiprocessing manager by picking them to this object and sending them across the network.
So if you think about that, it's practically like having something like Redis or any other key value store in Python itself.
It's already there.
That's a great analogy.
That's exactly what I was thinking.
I'm like, this is like a little baby Redis that your process creates, right?
Yeah, exactly.
Yeah.
It's pretty interesting.
And the big benefit is the shared data can be changed and created over time.
So, for example, the other things like queues and whatnot that you can use for multiprocessing, you have to create these values up front.
And they can get more tricky, right?
So this lets you start and stop processes and the values sort of persist across it and they can be created after by the subprocesses.
It's like basically a little sub database or a key value store that is just live, right?
It's like having a dictionary that is shared by all your processes and they can look for values there or new values there.
And I think that it's very cool that it can be easily adapted to work across the network.
So you can actually buy really multiprocessing tools and even distribute it across machine and continue to work using the same multiprocessing manager.
Yeah, that's pretty interesting.
So the next one that we're going to cover is a little bit in this realm.
Like I want this thing that I know I could go create a real server for, but I just want to keep it nice and simple, right?
We could go get real Redis, but now you have a big infrastructure thing instead of a single Python file, right?
And similarly here, like obviously, you know, you have Turbogears, we've got Flask, Pyramid, Django, all these different frameworks that we could go.
And they're big, huge dependencies, right?
They have dependencies of dependencies of dependencies to run.
But this next example shows you can actually create a non-trivial but somewhat basic WSGI HTTP server, right, in Python.
Yes, absolutely.
That's actually, in my opinion, one of the reasons why Python has so many web frameworks.
You cited like four or five, but there are like, I don't know, a hundred web frameworks in Python or something like that.
Yes, there's many more.
We could go for a while.
Yeah, exactly.
And the reason, in my opinion, is that because you have so many building blocks already in the Python library that it takes no more than a weekend to create your own web framework, you know?
And so when you start adding more complex features, of course, it gets far more complex than that.
But for rolling out a framework framework that it's able to route requests to functions or classes and send them back, it takes no more than a few hours of work.
And actually, if you already know how to do that, it's like a very few lines of code, like 10, 20 lines of code.
You can achieve everything you need, including routing, requests, handling, serving responses, and everything.
Yeah, that's interesting because when I thought of what built-in HTTP servers are there for the standard library, I was thinking, okay, so I can create up and just listen for an HTTP request.
That's pretty easy with sockets and stuff.
But the whole routing and all that kind of response stuff, I didn't realize was that easy to add on, right?
Like, much of it is already built in and things like that.
Yeah, absolutely.
And not only, you don't even need to go as far as handling the requests yourself because actually there is a fully working application server in the standard library itself that by default works to single-threaders.
So you will not be able to serve more than one request at a time.
But there is actually another mix-in in the standard library that you can apply to that class to make it multi-threaded.
So you get a fully functional multi-threaded application server with just one single line of code and only using what's available in the standard library.
That's really cool.
At that point, the only part you need to add on top of that is the routing, which is very easy to roll out using regular expressions lookup.
Absolutely.
And you cover that in this simple little one.
I guess one other thing that it doesn't do is it doesn't serve static files necessarily, right, in this first example.
But your very next recipe is, and here's how you serve static files.
Yeah, absolutely.
That's very easy.
So you can actually be extended in just a bunch of lines, of course.
Quite cool.
All right.
So another one that we might want to do in the same general area is creating HTML.
So if you're doing basic stuff like this and you have any user input or you just are going to stick it into some other template system,
basically if you are generating HTML, you need to be really careful about untrusted HTML.
Like if you accept user input or the data is based on user input, they could, instead of putting their first name,
they could put, you know, script, hackable JavaScript or other horrible stuff.
And if you are not careful and escape it, it'll just go in and execute, right?
That's one of the most boring parts of writing web application in the past,
that you have to like to manually go and escape everything yourself.
I saw, like I still remember those huge files of code where like every single string was wrapped in an escape code.
And that's horrible because you have to do that for every single thing.
And if you forget even one, that's a big security issue in your code, you know?
It is.
And it's not obvious, right?
Because like you could have regular text or you could have like Unicode characters that mean the text,
but they don't look like the text.
There's just like all sorts of weird ways that people could try to sneak through.
So you don't want to try to do that yourself.
That's for sure.
You don't want to have to care.
I mean, that's something that I always thought.
There should be some easy way that does that for me always unless I explicitly want to write some HTML in the output.
And that's essentially what the recipe can do because, again, the string formatter can be extended in many different ways.
And one of the ways you can modify it is actually by doing the escaping of everything you provide to the formatter.
So whenever you're bidding in your string, all the variables that you inject in the string can be escaped for you.
So you don't have to care.
And so when you're writing your webpage output or your email output or whatever you're trying to send out to the whole world,
you don't have to care about properly escaping everything yourself.
The formatter will do that for you.
Yeah, I really like it.
You basically just use standard string format and wherever the input, the variable values go, they either get escaped or not escaped based on your pattern.
You can say you can either mark them as like safe HTML because you want to dynamically generate them, but you need to stick it in there because it's your code.
Or you're taking user input and that needs to definitely be escaped.
Yeah.
It's a cool pattern.
Yeah, cool.
All right.
The very last one that we're going to talk about is tracing code.
So understanding what code is executed.
And you had an interesting comment about how it's not just useful for debugging, but it's also really interesting to just understand what a new library does.
Like if you want to say, I'm going to run this function, what does it do?
If it could actually show you the sequential Python that it executed, that'd be kind of neat, right?
Yes, exactly.
That's one way I most frequently end up using this recipe, actually.
When I want to see what's going on in a new piece of code, the PDB debugger is not always the...
It's very good when you have to pinpoint a specific point and understand what's happening right there at that moment.
But when you want to get a general overview of what that whole package or library or set of functions is doing and how, where things go when you do something.
So you want to follow the flow of the code itself.
It's not as easy.
You end up like spending hours just writing next into the PDB to show you where it goes next.
Yeah.
And what this recipe can do for you is leverage what's available in the standard library to trace the code that Python executed and tell you the output of exit if each code was run and show you the source code that was executed so you can understand.
Hey, I called this method.
And by calling that method, I also ended up calling all these other methods and they executed these branches.
And so that's why I got that answer.
Now I see, now I understand how I ended up having that response.
Yeah, I think it's great.
And you could even see, oh, this library is actually calling into this other library.
Yeah.
And then maybe even why.
Why is this a dependency?
Oh, I see what it's doing.
It's using it here.
Yes, exactly.
You end up discovering a lot.
And it's like having something that goes through the source code for you.
You know, whenever you want to understand a new library, you usually end up going to GitHub or something like that, open the source code and start reading, you know.
And you see that here it calls that, so I go and look into the source code to where that function is implemented and so on.
You do all this work yourself.
But the tracing module can actually do that for you.
And so it generates a single flow of everything that happens.
I think that's really cool, a cool way to think of it, because when you open up somebody else's code that you've never seen before, you're like, all right, well, what is important?
What isn't?
I'm going to have to sort of sift through this and figure out, okay, it looks like this is where the action is, and I'm going to pay attention to this.
And it's kind of a little bit of a detective job.
Whereas this, it only will show you what executed.
So you can kind of ignore all the other stuff and just see the part that it actually used.
That's pretty cool.
Yes, and you just have like to nearly decorate the function that you want to trace, and you will get the output printed.
So it's very easy to apply.
That's cool.
So definitely, we're going to try playing with this as well.
All right, well, those are the 10 recipes that we chose to talk about because we thought they were pretty cool.
But there's a bunch of other ones, many, many more.
How many are in the book?
Do you know?
I have to admit that I don't remember the exact number, but there are around like from 10 to 15 recipes for each chapter, and there are 15 chapters.
So we are more than 100 for sure.
Definitely more than 100.
So there's a lot of these types of little things in here.
And I think this is a really great book, and I'm happy to highlight it because, like I said, I learned a lot just going through this here.
And I'm sure everyone who checks it out will learn even more because they'll go through all of them, not just the 10.
Yeah, I really hope that at least a single person by reading the book will say, wow, I didn't know this.
That was my whole purpose for the whole time I was writing the book.
That's cool.
I think after going through all of this book, I'm going to change my answer from 40 to 50% of how much of the standard library I know.
It's cool.
It's really great.
I appreciate the topic.
So let's leave it there for that.
But I do have the two final questions for you, and I'm going to change it up just a little bit.
So if you're going to write some Python code, what editor do you use?
I usually use PyCharm for most of the big projects editing.
And for small lacking around, I started using Visual Studio Code recently.
Yeah, nice.
That's exactly what I do.
Yeah, that sounds strange when you tell people.
They always look at you, Visual Studio.
No, it's a pretty good editor.
They are doing a really good job, and they're putting so much energy into the Python space these days.
So, yeah, I think it's a great answer.
Yeah.
All right, that's the editor.
And like I said, that's basically the same way that I am and using mine.
Now, I would normally ask you a notable PyPI package, but let's mix it up and talk about a notable standard library module package that you want to just highlight.
Okay, that's an interesting question.
There are so many great modules within the standard library that it's really hard to pick one.
But if I really have to pick a single module, I would say that the logging module is one of the most fascinating ones.
Not because it can be the one that you will use most often or it's more feature-rich, but because there's so many ways it can be set up, configured.
There are so many side effects on the things you do that you can go on learning about logging for years, and you will never know everything that the logging module can do, how it interacts, or the configuration format that it supports, and things like that.
Yeah, that's a great answer.
I totally agree with you on that, by the way, that just like you could learn it forever.
You're never done learning it.
Yeah.
All right.
So people are excited about standard library.
How do they learn more?
Final call action, what should they do?
Oh, just go and read the Python source code.
That's the way that I learned most things.
You know, you just open the Python repository on GitHub and start reading modules whenever you want to see what's going on and things like that.
That's really the best way, and it's worked a lot for me, because that's the only way you can actually see the hidden side effects or those functions that you use for years, importing them from the sign library, but you never really know why they're working that way.
And that's the way you also discover that something like the formatter can be subclassed with its behavior in different ways or things like that.
Because those more advanced views are not really documented because they are more in internal detail, I would say, of the standard library.
But once you know that they exist, they've been there for years, and it's pretty safe to leverage them.
Yeah, that's great.
I definitely feel like there's a whole bunch of stuff to explore, and even more so after talking with you about this.
So thanks for being on the show.
Absolutely.
Thank you for having me.
You bet.
Bye.
This has been another episode of Talk Python to Me.
Our guest on this episode was Alessandro Molina, and it's been brought to you by Linode and Rollbar.
Linode is your go-to hosting for whatever you're building with Python.
Get four months free at talkpython.fm/linode.
That's L-I-N-O-D-E.
Rollbar takes the pain out of errors.
They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed, until users complain, of course.
Track a ridiculous number of errors for free as Talk Python to Me listeners at talkpython.fm/rollbar.
Want to level up your Python?
If you're just getting started, try my Python Jumpstart by Building 10 Apps course.
Or if you're looking for something more advanced, check out our new async course that digs into all the different types of async programming you can do in Python.
And of course, if you're interested in more than one of these, be sure to check out our Everything Bundle.
It's like a subscription that never expires.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I really appreciate it.
Thank you.