The Python core developers recently released Python 3.7 and are now busy planning what's coming in 3.8.
That makes right now a great time to dig into what's included in Python 3.7
and see what's on deck for this next great release of CPython.
On this episode, we have Anthony Shaw back on the podcast to tell us all about it.
This is Talk Python to Me, episode 180, recorded October 1st, 2018.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm.
And follow the show on Twitter via at Talk Python.
Hey everyone, Michael here.
Just a couple of things to share before we get to our chat with Anthony.
First, you might notice I sound slightly different this week.
I've been experimenting with a new audio setup.
Your feedback on whether this is actually an improvement is welcome.
Just send me a mention on Twitter via at mkennedy or at Talk Python.
Second, I recently released a new course and this is one I've wanted to create for a long time.
It's called Asynchronous Techniques and Examples in Python.
This is the definitive course on asynchronous programming in Python.
It starts out right away digging into the asyncio and the magical async and await keywords.
We also discussed the time-tested techniques such as threading and multiprocessing.
But we don't stop there.
Once you're using threads, you need thread safety.
Covered.
Want to write some async web apps with Flask using async and await.
Covered.
We even discussed some ways to combine these techniques for when you have some mixed CPU-based work and some IO-based work.
We round out the course with leveraging Cython to make our threaded code go truly fast.
If you're interested in this course, just visit talkpython.fm/async.
And if you act quickly enough, you can still grab the early bird discount.
Now, let's talk Python present and future with Anthony Shaw.
Anthony, welcome back to Talk Python.
Hi, Michael.
It's great to be back.
Yeah, it's super to have you here.
It's great to have you on as a regular.
You're doing so many cool things.
Thanks.
Yeah.
So just to let people know, you were on way back in episode 155 for Practical Steps Moving
to Python 3 and also in 168 for 10 Security Holes in Python.
Those were both fun episodes.
But we're kind of not just upgrading to Python 3.
We're cutting edge at this point, right?
Yeah, we're going all the way up to 11.
This one goes to 11.
Awesome.
Well, I'm definitely looking forward to doing that.
But before we get into all the details of Python 3.7 and 3.8 and the speculation around there,
let's just talk really quickly about what you do day to day so people know where you're coming from.
Great.
Yeah, sure.
I work at Dimension Data.
It's a big IT company.
And I actually work in HR, which might surprise a few people.
As I came from an R&D background and a couple of years ago, I made a big career change.
So I focus on learning now.
Similar to you, I guess.
And skills development within the organization.
Yeah.
And just so people know, that's not like a team of 10, right?
There's more than 10 people, I think, that take some of these programs you put together and stuff.
Yeah.
So we're about 31,000 employees.
And I've been running a whole bunch of learning programs and stuff like that,
trying to teach people Python, for example.
About 4,000 people are learning Python that work for us across the world, which has been fantastic.
And running a whole bunch of other programs as well.
We're basically trying to make sure that the skills of the people who work at the company
are up to date with all the latest technologies.
Wow, that's really cool.
And you've done some good write-ups.
Maybe I'll even link to them in the show notes.
So that's your day-to-day.
But then when maybe you're on an airplane or you're traveling,
you decide to break out some massive open source contribution, something like that?
Yes.
So I have to travel quite a lot for my job, which means I spend a lot of time on the planes,
really quite bored.
And there's only really so many Marvel movies you can watch.
So really, I kind of feel some time working on open source projects involved in quite a few at the moment.
So all of them are Python.
They were a range of different languages a few years ago, but now I'm 100% Python contributions only.
So that's been excellent.
And the Apache Software Foundation just got named as a PSF fellow a couple of months ago, which was awesome.
Yeah, congratulations on that.
That's really awesome.
Only five people per quarter get that honor.
And so that's a big deal.
That's cool.
Yeah, I don't really know what I did to get that.
But it was really great to be among the amazing list of names of people who are on the list.
Yeah, that's awesome.
Cool.
Well, our topic for today is Python current and beyond, right?
At least the time of this recording, people go back far enough in time.
Maybe they'll hit 3.7 or something in the past.
But we're going to start by talking about what is cool in Python 3.7 people should pay attention to, they should learn about.
And then we're going to talk about what's coming in the next version of Python 3.8.
So let's start with Python 3.7.
Like, is it a big change?
What do you think?
I guess you kind of have to bring in the whole history of, well, not the whole history of Python 3.
But you have to kind of take a bigger lens to this because Python 3 has been out for 11 years now.
Is it 10 years, 11 years?
Since 2008, I believe.
It was just 3,000 days, like a year ago or something like that.
So yeah, it's been about 10 years.
But no offense to the developers who worked on it.
But the first few releases weren't really production ready.
So 3.0 to 3.3 were really where they introduced a lot of the major changes in Python 3.
And there were a lot of stability issues.
And there were memory leaks and stuff like that that needed to be fixed.
Also performance, right?
Like a lot of people said, well, I need to do this thing.
And this thing is like 50% slower in Python 3.
So I'm not doing it.
Definitely.
There's some major changes in Python 3 which made things significantly slower in some of the earlier releases.
But really, the focus was, let's not just spend 10 years and then ta-da, we've come out with something that's production ready.
They wanted to get something out there and get the feedback earlier on.
So even though Python 3 has been out for a decade, I guess the stable releases have only been out for the last four years, five years now.
So 3.4 was pretty stable.
But 3.5, definitely, that's where you sort of get picked up by some of the major Linux distributions.
For example, it starts to get packaged with operating systems and stuff like that.
Yeah, that's a great point because I feel like 3.5 is really where it was clearly the right choice.
Like 3.4, there was a debate at that time.
It was like, well, maybe, maybe not.
At 3.5, it was like, what are you talking about?
Come on, let's go.
Yeah, exactly.
And then 3.6 comes out.
So 3.5 included the async await, a lot more functions with asynchronous programming.
And also 3.6 came out with more async await features, like the generators, for example, async with statements.
So context managers, basically.
And the f strings came out in 3.6, which is yet another way of formatting strings in Python, which is pretty cool.
Because I don't know what, five wasn't enough or I don't know if they were.
And once you get to use it, actually, it's definitely my favorite way to format strings now.
I'm with you.
You know, the problem I run into with f strings is I'm like halfway through a string and I decide, oh, yeah, now I want to put this variable here.
I'm going to go back to the beginning and put the f and then go back and do it.
And, you know, I don't know.
That's something I just got to train myself to put the f at the front more often.
Yeah, so 3.7 came out.
And, yeah, I guess really they're an opportunity now where they can start adding some good new features.
For people, it's not just bug fixes and performance improvements.
So, yeah, it's been a pretty significant change.
Yeah, that's right.
Because Python 3 is really either as close to, you know, same as Python 2 or faster, right?
There's a lot of performance improvements where the argument used to be, well, I can't switch to Python 3 because it's too slow.
Now it's, well, if I don't, I'm going to be behind, right?
Yeah.
So 3.7 was the first release of 3 that basically across all the benchmarks was quicker than Python 2.7.
With one exception, which is the startup time.
But that's still been an ongoing issue.
Yeah, and it really depends on what kind of apps you're building, whether that matters.
If you're building, like, a background thing that runs in a queue or a web app, like, who cares about startup time?
If you're building a little command line app, that might matter, right?
Yeah.
But it's still milliseconds.
It's not.
It's not .NET or something, right?
Yeah.
Not WPF.
I just want to, like, while we're on this little micro subject of Python 2 to 3, I just want to put it out there one more time.
Let's all call Python 2 Legacy Python.
And I think that's going to start to make a little dent with the managers.
Yeah, we're still using the Legacy Python.
Can we stop doing that?
Yeah, it came out in June, which is winter for us.
So I'm on the other side of the world.
Yeah.
So, yeah, it came out in June this year, so 2018.
The first sort of beta versions came out earlier in the year, sort of February time.
And there's a couple of alphas out very early on in the new year, this 2018.
So, yeah, it's been out for, what are we, six months now, nearly.
So that's, yeah, it's kind of had the chance to pick up speed.
Obviously, it's not been included with any of the major distributions yet.
So you have to go out of your way to download and install or upgrade to Python 3.7.
So I'd say it's not as widely used as 3.6 or 3.5 yet.
But it's definitely a major step forward.
Yeah, it definitely is.
And, you know, if I log into my Ubuntu machines, it's Python 3.6.6, I think, which is already
really awesome, right?
That's way better than, too.
Like, that ships with the operating system.
So that's really cool.
But 3.7 has a bunch of great features.
Now, the reason that I thought, hey, I should have Anthony on the show is because you wrote
a pretty sweet course on specifically what's new in Python 3.7.
And it's really quite consumable.
It's like an hour, right?
Do you want to just tell us quickly about your course?
And we'll get into the details of the stuff.
Yeah, sure.
So the course is on Pluralsight.
And it's called What's New in Python 3.7.
And basically, I go through all of the new features and functions and show you how to
use them, show you any gotchas and stuff like that.
It's only about an hour.
So it's not going to take much of your time to watch.
But yeah, it's been fun to put it together because, you know, with each alpha and beta
release, I was running all the upgrades or pretty much just running on a copy of the master
branch on CPython development for a few months.
Now that it's on GitHub, right?
Yeah, it's finally on GitHub.
Yeah, yeah.
So it's good to kind of keep on the bleeding edge and also to read all the peps and get
all the background and details and stuff.
So I put the course together to condense all of that information down into a way that's
a bit easier for people to just watch, get a gist of and get going with all the cool new
stuff.
Yeah, sweet.
So we'll link to that in the show notes as well.
I guess it's worth pointing out, like this came out in June and Python right now is on
an 18-month shipping schedule.
not next June, but next Christmas, December or something like that.
It should be Python 3.8.
Right.
So that's the current schedule anyway.
Yeah.
So about in around August, maybe September, there is actually a Python 3.8 release schedule
already.
I think in about September 2019, we're going to start to see some beta releases of 3.8.
You can actually test some of the new features already if you go and download from the source.
If you live at the tip of the master branch.
Yeah, exactly.
There was a Python meetup in London last month.
And I gave a quick lightning talk showing some new 3.8 features.
And it basically segfaulted halfway through the demo.
Don't do this in production, Jess.
Don't run it in production.
Maybe hold off a little bit.
It's not even alpha ready yet.
It's very, very, very early.
Yeah, that's pretty cool though.
There's some really great stuff in there.
But let's start with 3.7.
We'll talk about 3.8 in a little bit.
So I guess probably the big star of the show for 3.7 is data classes, right?
That's the thing that is totally new and pretty obvious that most everyone could use, right?
Yeah, so Python has always come batteries included, which is in the form of the standard library.
And in the standard library, there's a whole load of things you can do.
You can talk to HTTP endpoints.
You can work with the file system.
You can even manipulate audio files, like all from the standard library.
There's all this great functionality built in.
And really, there's been a bit of a gap in terms of working with data structures, like APIs, for example,
or working with document databases.
The gap's been filled by a couple of third-party packages.
Django, for example, has its kind of built-in ORM, which helps you basically create a class called a Django model.
And you can have attributes on the class.
And then when you save that to a database, it automatically creates all the fields and stuff for you.
So Django is not the only one to do that.
There's lots of other ways of doing it.
There's also packages like attrs, A-T-T-R-S.
which is great, which does a very similar thing.
But the concept is that you have a class in Python.
You put some fields on it.
And basically, you can use that to represent data that's in some other system.
So that could be an API you're talking to or a database or it could be a memory store, for example.
So it's a whole bunch of border plate code you have to create.
If you do this from hand, you have to create all the Dunder methods that you want to use.
So Dunder init, for example, the Dunder EQ, Dunder Reper, Dunder STR, etc.
Right.
So it seems like you're kind of done when you've done the Dunder init and you put your fields in there.
But if you're going to, say, try to hash those things or compare them or sort them or you want them read only,
all of those things require a pretty good understanding of the Python data model and the magic methods and all that kind of stuff.
And data classes are just, you know, frozen equals true type thing.
Yeah.
So there's some great features in data classes.
So really what you have to do is you create a new class and that implements a data class.
And within that, then you basically have attributes on the class.
So these are class level attributes.
You have to give them type annotations.
Otherwise, they .gitignored.
If you're not aware what type annotations are, it's a colon after the variable name.
And then the type you want to use.
But there's another change in type annotations in 3.7 as well, which we'll come back to maybe later.
And basically, you put the fields on there and then you're done.
And then you can basically call that class.
You can call the initializer or the constructor on the class and pass it all the fields that you've assigned.
And then there's a whole bunch of really cool features you can use.
For example, you can set default values.
You can have default factory values.
You can have it implements done the EQ for you, done the rep, done the STR.
So basically, it just gets rid of a lot of boilerplate of having a class which basically just stores data and represents some of that kind of data structure.
Yeah, it's really sweet.
And that initializer, it writes that for you, right?
Yeah, it writes everything for you.
So all you need to do is declare the class and sign the fields and then you're done.
That's cool.
Can you inherit them?
Can you create like a hierarchy of data classes?
You can.
I'm not saying you should.
I'm just wondering if you can.
You definitely can.
And there's a lot of reasons why you want to do that.
Now, if you want to inherit from a base class, it's important to note that within data classes, you can have fields with default values and fields with basically require a value.
So if you mix those things together, so if you have a base class that has both non-default and default fields and then you have a inherited class, you can't mix them in the inherited class.
This is probably one of the bigger limitations of the data classes package is to do with the way that it builds the done the init method.
But it's, yeah, that's pretty, I guess, a bit of an edge case, but it's something to watch out for.
Yeah, that's cool.
And so if I want to have some kind of check in my initializer, like this number has to be between 0 and 10, but it's written that for me, like where does that happen?
Yeah, so you can do it in a couple of places.
One would be a post init method.
So after the auto-generated dunder init has been called, you can have your own method which does extra stuff at the end.
So that could be a value check, for example.
It doesn't do type checking.
So even though it uses type annotations, I guess the Pythonic way of doing it is that they're just a friendly reminder of the type and they're not guarded in any way.
They're meant for linting and like build time checks, not run time checks, right?
Yeah, exactly.
Another great feature is that you can use, basically you can create immutable types.
So by default, they're mutable, which means you can change the values.
You can also create immutable types by setting frozen equals true when you basically instantiate them.
The frozen basically means that they're immutable, you can't change their values.
And also they'll implement a dunder hash method as well.
So they're hashable.
So you can use them in things like sets.
Also, you can use them as dictionary keys, which is pretty cool.
Yeah, that's really cool because you shouldn't be able to put a thing into the dictionary and then change the value of it and then make it no longer have the same key.
That would definitely drive you insane, which is what it effectively means if the hash value changes, right?
Yeah, so it's a really cool way of creating more complex dictionary keys, I guess.
And also moving away from just representing everything in dictionaries all the time.
I think when you're working with APIs and stuff like that and you're passing data around, it's convenient to just stick with dictionaries because you don't have to worry about what field you have and what types they are, etc.
But it's definitely nicer and it's also a lot more testable to introduce a bit more concreteness, I guess, to the types.
Yeah, it's cool.
And data classes kind of makes that less effort to do so correctly.
Yeah, really easy.
This portion of Talk By The Dermade is brought to you by Tidelift.
Open source software is everywhere.
How can we as a community ensure that the open source software we rely upon continues to get even more awesome and more dependable?
Tidelift believes the solution is hiding in plain sight.
Pay the maintainers.
They're dedicated to creating an effective way to do just that.
And today, they're excited to share that they've reached over $1 million in committed payments for maintainers available via the Tidelift platform.
And yes, that includes Python maintainers.
Income for a project grows based on usage, not the number of hours spent on the project.
So you can build significant income around your open source projects.
Tidelift is offering a guaranteed minimum $10,000 payout over the next 24 months to select projects in the Python ecosystem.
If you maintain a Python project, visit talkpython.fm/Tidelift to find out if your project is included.
That's talkpython.fm/Tidelift.
The next one doesn't actually sound very exciting to me, but I'm sure it's super important, is that the locales and UTF-8 got a little bit of a change, right?
What's the story around that?
How much do we need to worry about it?
I think we won't spend too much time on this because it's really confusing.
Any form of encoding is confusing.
It's super confusing.
Yeah.
So a local means it's, I guess, the settings on your operating system, which describe what country you're from, for example, and the way you like things to be represented.
So, you know, in...
Like, digit grouping or the order of month, date, year when you print out a date, things like that?
Yeah.
So, like, in mainland Europe, most countries would represent the number 1,000 with 1.000, whereas in the UK and the US, there would be 1,000.
So that's one change, I guess, and that would be in the locale.
Other ones would be the encoding type, which is where UTF-8 comes in, the keyboard mapping, and there's a whole bunch of other things.
So, basically, there's a default, which is the sort of NCC locale, which was default like 20 years ago, maybe.
But, you know, since computing has moved on a bit and we've had to support more countries than just the US.
So I would like to think that this is like the world going, let's just, let's accept everybody and let's support all of those languages.
But how much of this has to do with emojis?
It definitely helps with emojis.
And hopefully that's not the main...
My thumbs up is a square, come on.
Hopefully that's not the main driver.
But yeah, there's quite a few reasons.
Now, I guess Python 2 doesn't really, or say legacy Python, this problem is a bit more invisible because they use byte strings by default, whereas in Python 3, we use unicode strings by default.
So I guess the thing is, if you're reading from a shell pipe, for example, so you want to pipe the output of one command into your Python script, then Python will look at the locale.
So look at the operating system to see, you know, what settings you've got locally.
Now, the problem is that not everyone configures their operating system correctly.
There's also other scenarios like people creating Docker containers and not bothering to set the locale.
Exactly.
Like all the cloud stuff, right?
Yeah.
All the cloud stuff.
You just fire it up and you're like, oh, look, it works.
But I passed in my data file, which was in one locale, and I tried to read it in another, and there it went.
Exactly.
So if you just had that set to the, I guess, the most basic one, which would be the NCC, then that would not be with UTF-8.
So Unicode would not work by default.
So basically, in Python 3.7, they've made more assumptions about people misconfiguring their operating system to make it more friendly for Unicode.
Nice.
So the next one that you have on your list, I think, is really interesting.
And it's interesting because I'm like, oh, this is super.
This is not like, why would I ever use it?
I would never use this.
What is this about?
And then at the end, I'm like, oh, no, this is really useful.
And I feel like I really learned something.
And I always love it when that happens.
So the next one is breakpoint, right?
We used to import PDB, and we'd do a set trace thing.
Or we would fire up PyCharm or Visual Studio Code or something and just click in the side.
So why does Python need a breakpoint?
Is it just like an alias for PDB set trace?
Yes and no.
So Python 3.7 introduces a new built-in function called breakpoint.
So if you just write breakpoint, open, close, parentheses, anywhere in your code, when that line executes, it will jump into a debugger.
By default, that will be the one that comes in the standard library called PDB.
So anyone who's done debugging in Python today, I guess, would have learned that to jump into a debugger, you basically type import PDB semicolon.
Wait, wait, wait.
What did you say?
Semi-colon.
Oh my gosh.
All right.
Yeah.
So you can put it on one line and comment it out really easily, right?
Yeah.
So basically, to insert a breakpoint in Python requires two separate statements.
And having two lines of code to do a breakpoint kind of is confusing because then you've basically just changed your code in the way that you didn't really need to.
It's also, I guess, not super intuitive for people coming from other languages, people who are just used to working in an IDE and clicking on the red circle on the left-hand side.
You know, how do you do that in Python?
How do you step into a breakpoint and work in a debugging console within Python?
So what they've introduced is this new built-in function.
By default, it will just call import PDB, PDB.setTrace.
So it behaves the same way that you would expect.
But you don't have to write a semicolon.
That's better.
Yeah, exactly.
And it doesn't cause the linter to throw up its arms in anger that you've dared to include a semicolon in your code.
And so if that's all it was, I don't think actually it has that much value.
I'm not against having the language, but it's not that huge of a thing.
But then what if I don't want to use PDB?
Like, I don't like it.
There's other, even in the terminal, there's other nice sort of semi-graphical debuggers that are pretty cool, right?
Yeah, so what it actually does is it will eventually call breakpoint hook, which is a global variable within the sys module.
And like I said, by default, that's set to PDB.setTrace.
But you can change it to something else.
So there are loads of other debuggers of Python, which are a lot more friendly to use.
Also, there's browser-based ones like WebPDB, for example.
So WebPDB, when you hit that instead of as your breakpoint, it'll actually pause the code and it'll actually start up a little web server.
And you can open up your browser and you can navigate to it.
And it's just a lot easier to see and explore through the local scope and the global scope in a browser with a whole tree and everything than it is to do it in a little text console.
Yeah, that's awesome.
And basically, the breakpoint lets you customize what debugger gets called when you type breakpoint open close parentheses.
Yeah, so you can do it either in code.
So in Python, you can change the value of sys.breakpoint hook.
Or alternatively, you can set it using an environment variable, which is also pretty cool.
So if, for example, if you wanted to have breakpoints in a certain part of your code because you wanted to debug something that was running across multiple machines,
and one of the debuggers would be a local one and one of them might be a remote one, then you could have an environment variable set to basically change which debugger you want to use whenever breakpoint gets called.
Yeah, that's pretty awesome.
So even remote debugging.
I love it.
Yeah, it's great.
I think it's a great feature.
Yeah.
The next one is pretty cool.
So time seems pretty accurate in Python already.
So even like if I subtract two date times and I get a time delta, there's a lot of accuracy to like dt.total seconds or whatever it is.
But in Python 3.7, there's more seconds or more parts of seconds.
Yeah, this one really confused me when I was doing the research.
I think anyone who's worked with time accuracy, this would totally make sense.
But if you have put it in the I don't really care, I just want the time and it sounds pretty accurate, camp, which is where I was definitely from.
Then when the announcement came that they were introducing these nanosecond resolution time functions, it's like, okay, great.
Why do I need that much accuracy in my application?
So basically what they've changed in 3.7 is they've introduced some new functions into the time standard library module.
And they have the append xdunda underscore ns for nanoseconds.
And basically you've got a lot more accurate representation of time.
Right.
So it's not like time now returns better numbers.
You have to call time underscore ns, right?
Yeah.
So it's a separate function.
The different functions.
And for whatever reason, the old functions that would give you the time epoch.
So the time epoch was the 1st of January 1970.
Apparently nothing happened before then.
But that's basically the beginning of time.
In Python, it'll give you a floating point number back.
And anyone who's worked with floating points before will know there can be all sorts of inaccuracies.
So the number of seconds since the 1st of January 1970 is the major number.
And then the actual floating point itself is the parts after that.
So it's, you know, the milliseconds, for example.
Now, the problem is that you get all sorts of rounding issues.
And also they're not particularly accurate.
So it's not necessarily in milliseconds.
It's almost like the closest number that it could find depending on the operating system and the CPU that you're using.
So there can be these gaps and issues with accuracy in Python.
Anyone who's used Python before to try and measure time very accurately will probably know about these limitations.
So in 3.7, they've introduced a new nanosecond function, which is awesome.
And it will give you that in basically a long, long integer, which is the Python integer.
Nice. Well, I know that in that there's a time NS.
What else are there other underscore NS functions I can use?
It says get time NS.
There's set time NS, which you can only use in certain situations.
And there was a couple of others as well.
But those were the major ones.
Yeah, nice.
For this next one, what do you think, before we get into the details of what's new, let's just talk about type annotations.
You hinted at this in the data classes bit because it's very interesting that it's even required, although not checked at runtime.
How do you feel about type annotations?
So I haven't really used them very much up until 3.7 for exactly the reason that we'll talk about this next feature, actually.
So in 3.6, they're kind of a bit clunky to use.
If you want to represent anything remotely complicated, other than just a string or an int, you might have extra code you have to add to your Python modules.
I think it's just cumbersome.
And I haven't got a huge amount of value out of it so far.
So I think...
Right.
If you want to say a list, you'd have to say import list from typing like capital L list, not lowercase l list, things like that, right?
Yeah, because a lot of my code that I would write would return a tuple, for example, and the first entry would be an integer and the second one would be a dictionary.
And in the dictionary, then there'd be a certain structure.
And by the time you've had to describe all of that, it's just such a pain.
And it adds so much code to your...
And the type annotations themselves, the way to represent complex types is not particularly intuitive.
And if you're not familiar with type annotations either, then it's quite a bit of a blocker.
Yeah.
You know, my take on them is I really like them, but I don't think they belong everywhere.
But I really love type annotations on the boundaries.
So if I have like a data access layer and I've got some functions you call, like those functions you call, I love to put like annotations on, well, you get a list of these things back.
Or this one could be an optional user because they might not exist in the database.
And then that kind of flows enough through the editors for me that like everything else kind of picks it up.
But I don't turn it into like C# and put, you know, types on everything that exists, right?
Yeah.
One of the other big limitations is that if you're, you know, the type is actually a class that you've declared somewhere, then you have to import that in order to add it as an annotation.
Yes.
Yeah.
And that's partly what drove the changes for 3.7, right?
Like, so the place that made me the most crazy was it cannot be solved in the most straightforward way is that I have like, say, some kind of method on a class, like a class method on a class that returns an instance of that class.
I cannot say in the type system, it returns that class because it's not yet defined.
I mean, you could use like the string sort of cheating style, but you can't, you have to go realize there's this other way to like talk about it when it's not important and it's not as well as enforced and so on.
Yeah.
So the example I use in the course is, is sort of fictional, but you know, you could have a theater class, for example, and then in the theater, it has a number of seats and you have a seat class, which is in a separate file.
So in a separate submodule, and then you've got an attribute on the seat, which says which theater it belongs to, or you've got an attribute in the theater, which says how many seats there are.
And you can reference the seats.
Now, in order to do that as a type annotation, you need to import either one from each other, which creates a circular import, which Python is pretty good at handling.
It's not, it's not as bad as it was in C, but what it's kind of led to is kind of similar to what you see in Java and C#, which is the first, you know, 30 lines of every file is just import statements.
Which is a bit of a limitation really, because you're not using them for anything other than to add a type hint to the method in terms of what it returns or what it expects as parameters.
You're not doing anything with it.
And the imports obviously, you know, add quite a bit of time as well.
It slows the application down.
It's not a compile time thing.
It's a runtime thing.
So when you import all those things to just declare, here is the return type, that actually slows down the startup of that program, because now it's doing more stuff just to get to that one function.
Yeah.
Yeah, exactly.
All right.
So what's 3.7 add for us?
Like, how does it change this or make it better?
So what they've introduced is delayed evaluation type annotations, which is a very fancy way of saying that.
Basically, what happens is when your code gets parsed.
So in Python, to create the, I guess, the executable code, the Python files, the text files, or the code that you've given it needs to be parsed first, and then it needs to be compiled.
So at the parser stage, if you've imported from Dunder future annotations, and it will enable these delayed evaluations, then what it does is it goes and looks at all your type hints.
And if you'd imagine, let's say you've got a method, and you want to set the return type.
So you do a colon, and then the type hint, the type that you want to return.
Basically, what the parser will do is equivalent to just putting double quotes either side of that name.
So it's basically a string literal as a type hint instead of the actual reference to the type.
Right, which already works in the previous version.
Like, you can do that, sort of put it in a quote, and that actually works for, say, like, the class self-reference thing I talked about.
But then you don't get, like, say, refactoring help and things like that, right?
Yeah, so it causes issues with IDEs and other things like that.
So basically what this does is it supports a lot of stuff you get in IDE.
So the IDE would know, or should know, how to import those and reference them.
But it doesn't slow down your application at runtime.
And it's a lot easier to use because you don't have to worry about all the imports and things like that.
Yeah, very interesting.
The IDEs and the linters, they know how to still check for those things, even though Python at execution time doesn't necessarily do the imports, right?
Yeah, it basically pushes the problem down the path a bit, though.
Okay.
Because if you actually want to use the type annotations to find the concrete types, then you need to call a special new function called get type hints.
And you need to pass it the local and global scope.
And when you call that function, the thing, the type hints that you, if they'd referenced a class, for example, then you would have need to imported that class by that point.
Otherwise, that function won't work.
So you still have to import stuff if you want to use the type hints.
But it basically assumes that most people just want to use them as general annotations.
They don't want to use them to actually reference concrete types.
Right.
Yeah, very interesting.
So while we're on this type annotations thing, there's some very interesting stuff happening around type annotations in the later versions of Python.
And they all seem to be about trying to make Python faster, actually.
I mean, obviously, it was introduced to make it more understandable and help the tooling and stuff.
But so Cython has recently started adopting, if you have regular Python 3 type annotations, it'll use those instead of its funky way of declaring variables.
So it can actually take just annotated code and compile it to see better, which is pretty cool.
That's awesome.
I didn't know that.
Yeah, I know.
I just learned it as well.
And do you know about mypy, which is like the type verifiers, like a linter, but like deeply for typing?
Yeah, it's like an introspection tool, isn't it?
Yeah.
So that's pretty interesting.
And there's some tools to like generate the type annotations based on like runtime stuff and various interesting things.
But what also is interesting is Dropbox is releasing this thing called mypyC, which is a compiler that takes annotated statically typed Python code and compiles them to C extensions.
Wow.
Okay.
That's really cool.
So these are both kind of interesting things that people are playing with.
So anyway, those are worth maybe looking at.
Yeah.
And the important thing with this feature is that in the pep, it says that it will be the default behavior in Python 4.
That's pretty awesome.
There's going to be a Python 4 pretty soon.
Well, we don't know how long, but.
Yeah, well, a couple of 18 month iterations.
There's actually talk Łukasz Langa is overseeing the release for Python 3.7, 3.8, I believe.
Definitely for 3.8, maybe 3.8, 3.9.
I don't want to give me the wrong numbers.
Yeah, thank you.
And he's trying to get the release cycle down to yearly, but that's kind of on hold for a little bit.
Okay.
So maybe it'll be quicker than it otherwise would have been by, you know, six month segments.
All right, so we opened this section, the whole podcast a little bit by talking about performance in Python 2 and 3.
So 3.7 brings some interesting new performance benefits, right?
Yeah, so I guess calling methods or calling functions in Python definitely has an overhead or an associated cost.
I got to say, when I first learned Python, I was blown away at how slow a function call was compared to if I inlined it effectively.
Right, yeah.
And I think it's a big deal.
And this leads to people writing code in certain funky ways, copying and pasting things a bit more once they learn about this quirk of Python.
Especially if you're using some of the frameworks because the number of, I guess, turtles deep or the number of layers that it has to go through function scrolling functions, it can really kind of slow down the application.
Yeah, yeah.
I know in SQLAlchemy that Mike Bayer very carefully worked on the architecture to minimize the function call depth, actually, for performance reasons.
Like, there's a lot.
So in 3.7 calling functions gets somewhat faster.
Not quite like inline assembly fast, but faster, right?
Yeah.
So basically, they've introduced some new opcodes.
And there's a different implementation for calling.
And it's quite specific, actually.
It's not just functions and methods in general.
It has to be on a bound method, which means if you've got a class and you've got a method on the class, you've instantiated it.
When you call it, if that class doesn't have any keyword arguments, then it will be faster than before.
So like I said, it's quite specific, but that probably covers quite a lot of cases.
But that's a really common case, right?
There's a lot of times when you have some kind of object and it has a function and you want to call it.
Like, if objects don't have functions, why would you create them, right?
Just put them in a data structure or something, right?
So it's obviously nice that those are there.
And I wouldn't say the majority of functions have keyword arguments.
So it's a good improvement.
Yeah.
No, I think it's a good step forward.
So it's 10% to 15% faster.
And in benchmarks, there's basically a speed boost because of that knock-on effect.
Yeah.
That's really cool.
So maybe that might even affect like what you've pip installed, right?
Even if you have no classes, you might be interacting with like some turtle at layer 3 that uses classes, right?
Yeah.
So even just by moving to Python 3.7, you can actually see a speed improvement on a lot of applications.
I've upped 10% to 15%.
Yeah, that's really sweet.
So after going through all this, doing all this research, if somebody's on 3.6, is it worth upgrading to 3.7, you think?
I'd definitely be worth upgrading to 3.7.1.
Point anything other than 0?
Yeah.
I would never upgrade to a .0 release in a production environment.
But definitely, I think data classes are really cool.
I'd been using the Attrs module for a few projects and really liked it.
But obviously, adding yet another dependency to your application is not always desirable.
So it's nice to have that in the standard library.
The speed improvements are always welcome.
I think anything that can make Python faster is great.
The type annotations change, I think, actually makes it a lot more appealing to me because you don't have to worry so much about the import statements and things like that if you're not actually going to use them in that way.
So, yeah, there's some big changes.
I think the one that I've actually used the most is Breakpoint.
Just since I've switched to 3.7, just got so used to just writing Breakpoint instead of the import PDB.
And also worked with the pytest team to get Breakpoint supported in pytest, which was pretty fun.
So I got to kind of dig into some of the details.
Although somebody, Anthony Sotile, the other Anthony who works on pytest, there's quite a few of us actually,
but he raised a pull request basically saying that he proved that my pull request basically did nothing.
So I was a bit disheartened.
Oh, dear.
But it has a lot of test coverage.
That's good.
I hear you with the .0 release in production.
Right now, what I'm doing is I'm still using 3.6, 6, whatever the latest on Ubuntu is, right?
I keep that up to date as it ships with there.
But I'm doing 3.7 in dev and just making sure that I don't use any of the features yet that are going to cause trouble when I ship it.
So I've definitely taken my site down by accidentally using an F string in a utility function that was never called in the web app,
but was actually parsed by 3.5.
And they wouldn't start the web app.
So it can sneak in there.
But maybe you could start testing in dev or Q&A or some sort of staging environment and then someday switch pretty soon.
Yeah.
And then someday move to 3.8.
Yeah.
You could move to 3.8.
Although the segfault thing you're talking about, maybe not yet, maybe wait a little bit.
But yeah, let's do a quick preview of what's coming in 3.8.
Yeah, sure.
So there's a couple of quick, easy ones.
So for example, AsyncIO on Windows gets a different event loop, a proactor event loop.
I actually have no idea what that is.
I now want to go research it and see what that is.
But that's pretty cool that AsyncIO has these configurable, extendable event loops like UV loop and so on.
There's a few simple language things like a continue statement used to be illegal in a final clause, but now it's not.
But then there's a few other major ones.
You want to maybe touch on the big ones and we can hit a few small ones after that.
Yeah, there's been, well, there's, I guess, for every major release of Python, there's all the proposals for new features in the form of these documents called PEPs.
So I've been reading, I guess, some of the proposed PEPs in detail and trying to understand what they do.
There's a couple that have been accepted already.
The big controversial one, I guess, was PEP 572, which is something called assignment expressions.
The easiest way to understand it is that if you just do in a REPL, X equals one, enter, and then X, enter, it will return one.
And assignment expressions basically introduce a new syntax, which is X colon equals one, which will return one.
So it basically squashes those two statements together into one.
All right.
A lot of time you might be thinking, well, I would never use that.
Like, what is the value of this?
But if you're doing, well, expressions, right?
If I'm doing like a list comprehension and I want to do both a test and a creation of a variable or things like that, you know,
like there's a few interesting use cases where it does simplify the language a bit.
Yeah.
And that syntax is in other languages.
So people might be used to it in other places.
JavaScript is an example.
And Perl would be another one.
So I guess people, some people are more familiar with using that kind of syntax.
And yeah, if you're doing statements, like if statements, this comprehensions while statements, for example, then yeah, you can introduce this new syntax and you can basically get the value back and then use it for other things.
All right.
Maybe like a lambda expression where you've got to squeeze it into one line, possibly.
I don't know.
Yeah.
It's basically because you can, yeah, you can't write return statements inside where something should be a statement itself.
Yeah.
Even with semicolons, right?
Even with semicolons, you can't do that now.
Right.
So this was controversial partly because let's say close to half of the community really you thought like this just does not need to be added to the language.
We don't do this.
And there was like maybe a minority.
I'm not sure.
But another part of the community that really did want it.
And so there was actually quite a bit of a disagreement about whether it should have been accepted.
And the fallout of that is sort of the straw that caused Gita Van Rossum to step down as BDFL.
Yeah.
I think that some of the comments got a bit nasty, to be honest, and some of the pushback.
But I don't know.
I've seen places where it could be used.
And I think, yeah, that could be useful.
But I don't think I'd use it every day.
Like this is a once every now and again, you'd use it kind of thing.
Yeah.
That's kind of how I feel about it.
Like it's not a huge deal, but I'm not like looking forward to doing amazing stuff with it yet.
Yeah.
Like an async context manager.
Like, you know, that's kind of cool.
But it's not something you do use in everyday code.
Yeah, for sure.
So the one that's coming that I'm really positive about is the none aware or null aware operators.
Tell us about that.
Yeah.
So this one has been proposed, but not accepted.
I think it's in draft state at the moment.
So it's basically, again, a new syntax.
So anyone who works with Python a lot will know the sort of famous error that X is not an attribute of none type or something along those lines.
So like, I think it's one of the most common exceptions is that attribute error.
Something is not an attribute of none type.
Yeah.
So it's basically like it's the equivalent of a null pointer or a null reference exception in Java or C# or C++.
It's the Python equivalent of that.
So none, I guess, is our special type to represent nothing instead of null.
But it doesn't technically have a value similar to null.
It's actually a special type in itself, which means that in this PEP and their proposal, what they're proposing is basically a special set of syntax based around the question mark to allow you to basically follow branches depending on whether something is null or not.
So, for example, if you had a data class, if you had a data class with some fields and one of the values of the fields was null, you could reference it.
And then if that field itself, you wanted to call a method on it.
Like, I don't know, let's assume it was a string.
You wanted to run the upper on it.
If it was none, then you'd get upper as not a property of none type.
But basically, instead, you could do question mark dot as the method accessor.
Right.
Instead of object dot like field dot upper, you'd say object dot field question mark dot question mark upper.
Yes.
It also introduces something called a ternary operator, which is a double question mark.
But that already exists in Python, but it's pretty long form.
You have to say something equals something if the thing is that else, not this.
Yeah.
And it is a little bit longer.
The sort of double question mark null class none class operator thing does not excite me very much.
But this fluent style of these chaining function calls that at any step may be none is super exciting to me because you might have to check.
First, is the original object none?
OK.
Then if that's not the case, now is its name, you know, maybe it contains like a rich object.
You know, is that like an address?
Like, is the address none?
OK.
The address is not none.
So is the street not none?
OK, now we can call to upper on it.
That's like one line, right?
Object question mark dot address question mark dot street question mark dot upper.
And boom, you get either none or you get an uppercase street name.
I love it.
Yeah.
So if anyone works with like highly nested types, I guess, then and you have to write all these.
If that's not none, do this.
Then, yeah, this is super helpful.
Could be if it gets accepted.
Yeah, it could be.
It's one possible branch of the future.
This will be very helpful.
Yeah, I just one thing that I dislike a lot in my code is like sort of seesaw nested jaggedy code.
And this can like turn that into just like a one liner, which I think is great.
You also have called out runtime audit hooks.
What might that be useful for?
This is probably a security feature mainly.
But again, it's a proposal.
It's not been accepted yet.
But the idea is that for any built in functions.
Or as things in the sys module or some of the low level functions within Python standard library,
whenever they got called by either your code or somebody else's code, you've got the option to set like a audit hook.
So you might write a special application like a profiler, for example, if you're writing a profiler and you want to see every time somebody tries to open a network socket or calls a URL or something.
Or if you wanted to develop security application that checked anyone who is, you know, opening file handles in the operating system or...
Right.
You could build a really cool package that you can just import into your app at the beginning.
And it says, these operations are allowed.
These are not like no network access, no file access.
And with this, maybe you could just stop that globally in the runtime, right?
Yeah.
So kind of similar to what you can do with SE Linux, if anyone's familiar with SE Linux.
You can kind of lock down the kernel effectively for certain users and certain method calls can't be run.
So this would be similar.
So you could basically have a Python interpreter.
And when certain things are done inside Python, then your method gets called as part of an audit.
You could watch for the eval function.
Yeah, exactly.
It is a little bit light on the details.
The implementation is not finished either.
But I know that Steve Dower is working on this PEP.
And he has said if anyone wants to help contribute to the implementation or do some testing, then the door is open.
Nice to get in touch with Steve Dower.
Perfect.
So maybe I'll just throw out a few real quick ones also that are like more minor.
So for virtual environments, it now has an ability to activate this through PowerShell on Windows, which if you use PowerShell on Windows, that's pretty awesome.
Yeah, that's awesome.
I don't know why you wouldn't use PowerShell if you're using Windows anymore.
If you want to use MS-DOS, that's fine.
Yeah.
It's not the greatest shell.
Yeah, yeah.
I like Commander.
C-M-D-E-R.
I think it's commander.net on Windows.
So optimizations around file copies.
That's been improved on the Unix-based systems, right?
Yeah.
I don't think that makes a pretty big difference, actually.
Yeah.
There's some built-in, like, sysplatform-specific fast copy.
Like, I wonder if it uses the new Apple file system sort of replication stuff, which would be a lot faster.
It'd be very cool.
They also removed pyvnv, the script.
So the expected way that you create a virtual environment is python3-m spacevnv, right?
Yes.
Nice.
I don't really use that, but...
Nice.
Well, that's what they tell you in 3.8.
Other minor changes.
And I guess one thing that's interesting is python3.8 is temporarily...
Like, it's pretty far out, so this might not be a problem forever.
But right now, with Guido stepping down and the governance decision, like, the choice of how Python governs its decision-making is not decided.
No new features are going into 3.8 until they can decide how to decide on those new features.
Yeah.
That's a bit of a weird catch-22 thing there.
I bump into one of the core developers last month, and they said that it's likely that there'll be fewer features in 3.8 than there have been in previous releases, unless...
Well, until they can work this thing out, and then even then, having, I guess, faster, sort of mainly bug fix and performance improvement releases is a lot more likely to be the way forward.
Yeah.
I suspect it's going to take a little while before this gets dialed in and it becomes a smooth process again, but I'm sure they'll figure it out.
Yeah.
Well, there's a few major things that kind of need to happen in future releases.
I guess one thing is the startup time is still slower than legacy Python.
I think there's ways for them to optimize that in terms of what it loads at import at startup and how it does all the importing and caching and stuff.
Hopefully, that'll be quicker in newer releases.
And then there's a C API, which has, over time, got a little bit messy.
So I think there's a few proposals for Python 4 for that to have a bit of a revamp.
Yeah, that'll be interesting.
That's a little far out, but definitely be interesting.
Maybe it'll be rewritten in Rust.
Yeah, maybe.
Yeah, I don't know about that.
Anyway, that's a ways out, but definitely something to look forward to.
Well, I think that's probably a good place to leave it.
We're probably low on time, kept you long enough.
But before you go, I know I've asked you this at least back in episode 155, but that was a while ago.
I'll ask you again.
If you're going to write some Python code, maybe your answer has changed.
What editor do you use?
It's still Visual Studio Code.
Right, with the Python plugin.
Yeah, it's definitely a nice thing.
And it's like, I don't know how many times, millions, like 5, 10 million.
I don't know how many millions there are downloads of that Python extension for Visual Studio Code, but it's pretty remarkable.
Yeah, it's great.
Although I'm using Windows at the moment with Windows subsystem Linux, which is causing me problems because Linux creates different virtual environments to Windows.
So the IDE is using a different virtual environment to my actual application, which is causing me a source of pain.
There's all sorts of process isolation and system isolation between the Windows subsystem for Linux and Windows itself, right?
Yeah.
So that's causing me.
You're going to be an expert pretty soon.
That'd be awesome.
You'll be able to help people who are also doing this.
Nice.
All right.
So notable PyPI package?
Yeah, I'd recommend people check out Black if they haven't already as a code formatter.
It's basically, if you ever check in code and then if the build fails because you've, you know, put an extra space at the end of a line or something, and you've had that frustration, then you can use this basically as a tool to go through your code and format it for you so that it passes most code linters and style checkers and stuff like that.
And it's basically no options, no configuration.
You just run the command, give it a directory, and it will just go through and change your code for you.
I absolutely love that.
I think the code it produces is really nice, really easy to read, and it's just really easy to get set up with.
Yeah, most linters, they just complain to you, you should get rid of that space at the end or whatever.
This one just goes, yeah, I'll fix that.
Yeah, it just fixes it up for you and you check it in.
Right on.
All right, final call to action.
People are excited about 3.7.
What do they do?
They need to download it first, install it, test it out, run their applications on it, see how they're improved, and then check out some of the new features.
Yeah, maybe check out your course.
Yeah, and absolutely check out my course.
Yeah, like I said, it'll take less than an hour of your time.
It's 57 minutes.
And you can find out all the details on Python 3.7.
Yeah, awesome.
I definitely agree.
People should check it out.
There's some great features.
And Anthony, thanks for being on the show to share them with everyone.
Thanks, Michael.
It's been great to be back.
Yep.
Bye.
This has been another episode of Talk Python to Me.
Our guest on this episode was Anthony Shaw, and it was brought to you by Tidelift.
If you run an open source project, Tidelift wants to help you get paid for keeping it going strong.
Just visit talkpython.fm/Tidelift, search for your package, and get started today.
Want to level up your Python?
If you're just getting started, try my Python jumpstart by building 10 apps, or our brand
new 100 days of code in Python.
And if you're interested in more than one course, be sure to check out the Everything Bundle.
It's like a subscription that never expires.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct
RSS feed at /rss on talkpython.fm.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now, get out there and write some Python code.
Thank you.