#155: Practical steps for moving to Python 3 Transcript

00:00 Since 2008, there's been this tension in Python where much of the effort to improve it has been focused on Python 3,

00:06 and many of the developers were left stuck in Python 2, primarily because important packages were not yet Python 3 capable.

00:13 We've moved into an era where most of the packages anyone is using are fully Python 3 enabled, and many are Python 3 only.

00:21 The latest Django framework, for example.

00:23 There are many carrots and a number of heavy sticks encouraging us all to move to Python 3.

00:29 But what if you have a large code base that needs to be migrated?

00:32 What are the concrete steps and the gotchas in this whole process?

00:35 This week, we welcome back Anthony Shaw to the show.

00:38 He just published a new course on migrating Python 2 code, and he's here to share his tips with us.

00:44 This is Talk Python to Me, episode 155, recorded February 13, 2018.

00:49 Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.

01:09 This is your host, Michael Kennedy.

01:11 Follow me on Twitter, where I'm @mkennedy.

01:13 Keep up with the show and listen to past episodes at talkpython.fm, and follow the show on Twitter via at Talk Python.

01:20 This episode is sponsored by Linode and Rollbar.

01:23 Please check out what they're offering during their segments.

01:25 It really helps support the show.

01:27 Before we get to the interview with Anthony, I just want to let you all know, I recently published a brand new course at Talk Python Training called Eve, Building RESTful APIs with MongoDB and Flask.

01:39 This course is written by Nicola Hiroshi, who was actually the very first guest on Talk Python.

01:45 He's episode number one.

01:47 He also happens to be the creator and maintainer of the Eve framework.

01:51 Eve is this framework that lets you take Flask and MongoDB and turn your collections into a RESTful API with validation, schema verification, permissions, all sorts of cool stuff.

02:04 It's a powerful, powerful framework.

02:06 The course is really well done by Nicola.

02:08 Check it out at training.talkpython.fm.

02:11 Now, let's chat with Anthony about getting that Python 2 code out of here.

02:15 Anthony, welcome to Talk Python.

02:18 Hi, Michael. How are you?

02:18 Hey, it's great to catch up with you. It's been a while.

02:21 Yeah, definitely. It's great to be on the show.

02:23 Doing well and looking forward to this conversation.

02:26 I think the time really has come for this whole conversation about like, why are we not on Python 3?

02:33 And then, you know, maybe what are the actual concrete steps that'll help you get over the blockers?

02:38 I think we keep talking about how it's been, you know, 10 years since Python 3 has been out and trying to lay guilt on people to move across the Python 3.

02:46 But there's really, it's really a bit more complicated than that.

02:49 So I'm looking forward to kind of diving into some detail.

02:51 Yeah, absolutely.

02:52 So we had you on the panel show a while ago on Contribute to Open Source.

02:57 And that was really, really great.

02:58 But I don't know that we got a chance to get your whole story and how you got into programming Python.

03:02 So why don't you let us know?

03:04 Yeah, sure.

03:04 So I got into programming at a really early age.

03:07 I think I was 12.

03:10 And I guess that probably would have been mid-90s.

03:13 And in those days, there was not really a free access to compilers and online knowledge and things like that.

03:19 So my programming experience was based off of boxes of software and floppy disks that I could get my hands on.

03:27 So I learned some pretty odd programming languages to start off with.

03:30 One of those was Dbase 4, which is sort of an accounting programming language.

03:34 And I tried to make a Robot Wars simulator in ASCII, which is pretty challenging.

03:39 So yeah, that's kind of as I evolved from there.

03:42 Over the last seven years, I've been working in software teams in C#.net and got into Python a few years ago.

03:51 I was in Seattle.

03:53 Over the weekend, I became pretty sick.

03:56 I was stuck in my hotel and I decided to pick up a new programming language.

03:59 So I was contributing to the Apache Lib Cloud project, which I've contributed a lot to actually over the last three years.

04:07 And last year, joined the Apache Software Foundation as a member, which is great.

04:12 So yeah, just kind of fell in love with the Python language, the simplicity and the speed at which you can get something working.

04:19 And it still scales pretty well.

04:21 So that's great.

04:22 Yeah, that's really interesting.

04:23 So you kind of healed yourself with Python.

04:25 Yeah, definitely.

04:26 You're really making me think back to when I was younger as well.

04:30 When you used to get software tools literally in boxes.

04:33 Like I think I remember going to the store and buying Visual C++ 1.52 and getting all the DVDs, like the six or seven DVDs and just flipping them in one after another.

04:44 Those are the times, right?

04:45 Yeah, but the best thing about the box was it always came with a manual.

04:47 And, you know, as a kid, I used to sit there and read the manual to understand everything the language could do and how it worked.

04:54 And, you know, see who didn't want a copy of this particular language or Visual Studio.

04:59 Yeah, and leveraging the books, which is great.

05:01 But nowadays, we have the internet.

05:02 So we don't need that anymore.

05:05 It's interesting.

05:06 It's much more JIT style, right?

05:08 Like it used to be, here's the book.

05:10 This thing contains everything.

05:11 So you can just go through it, right?

05:13 You would never try to do that now because it's just too comprehensive.

05:16 Pretty funny.

05:16 Okay.

05:17 Well, that's really cool.

05:18 And maybe tell people really quickly about Apache Lib Cloud.

05:21 Like that's a pretty big project and you've done a lot with it.

05:23 Yeah, sure.

05:24 So it's a cloud abstraction library in Python.

05:28 So you've got some major cloud platforms like AWS, Azure, Google Cloud.

05:34 And really what it does is it provides a common class and a common set of models in Python that you can connect to all these different cloud platforms and get lists of servers or your DNS records or your storage objects.

05:49 And the responses you get back from the API will be consistent across all the different clouds.

05:53 It's a great way of writing compatibility code essentially across different cloud platforms.

05:59 We don't just support the big four.

06:00 Yeah, it's about 50 different cloud platforms.

06:03 There's some pretty niche regional ones, especially, you know, there's like a great cloud platform in Switzerland, for example, that's really popular over there.

06:11 Yeah, that's really cool.

06:11 Also, if you're going to do some sort of DevOps type thing, you could say right against Lib Cloud, like spin up the servers and you could just point it at EC2, point it at Azure or whatever, right?

06:25 It just does it.

06:26 It just does it.

06:27 There's a lot of spoon bending under the covers.

06:29 Yeah, the APIs were never really consistent.

06:33 And that's been on purpose, really.

06:34 You know, they're trying to offer the full functionality of what their cloud can through the API, but also portability is not really in their best interest because they want to kind of keep you locked into the platform.

06:45 Yeah, do you feel like the cloud is like the next great lock-in, maybe even larger than, say, Windows and Mac?

06:52 Absolutely.

06:52 And that's kind of really why I've been dedicated to the Lib Cloud project for the last few years.

06:57 I mean, even in my current job, I guess I'm, you know, working pretty late hours most days.

07:02 And I'm just trying to find whatever time I can to keep contributing to it because I think it's really important, especially when I think back to how I learned programming.

07:10 It was, you know, spending whatever pocket money I could on books and trying to find whatever software I could and assembling PCs out of old bits of hardware that nobody wanted so I could keep learning.

07:21 And my concern is that if cloud becomes the way of running computing, and all of it costs money, then it's going to push out all these people who don't have access to cash, like students, young people, people in areas of geographies where they don't necessarily have access to.

07:40 Typically, US dollars to pay for all these cloud resources.

07:43 And then also countries where network connectivity is not that great.

07:47 I do a lot of work in Africa, for example.

07:49 And there's some countries in Africa where their network connectivity can be pretty sporadic, as well as the sort of power as well.

07:56 So that can cause all sorts of challenges.

07:58 And yeah, I really care about people having access to learn and run computing.

08:03 So I don't want them to be pushed out of that.

08:04 That's a really cool project.

08:05 So you're like a cloud freedom fighter, keeping the cloud free for everyone.

08:09 Awesome.

08:10 Yeah.

08:10 So maybe talk about what you do day to day.

08:13 You have this kind of mission of teaching people, getting people into programming in Python on a grand scale, right?

08:19 Yeah, no, definitely.

08:20 So I'm basically head of talent development at a company called Dimension Data.

08:25 So it's a 30,000 people organization, operates in 50 countries, so pretty much all over the world.

08:32 And my role is to look after the skills of the people within the organizations that can include lots of things.

08:40 On the technology side, one of the big pushes we've been doing is to try and convince as many people as possible to learn Python.

08:47 So we're up to 4,000 people now.

08:50 Yeah.

08:50 And we partnered with Talk Python Training.

08:52 You're far too modest about the quality of those courses.

08:55 They've been so popular.

08:57 The Python Jumpstart by Building 10 Apps is the single most popular course we have within the whole company now on our learning management system.

09:05 Yeah, and it's really cool to visit offices all over the globe and see the little Python logo popping up in all these different places.

09:12 So it's been great.

09:13 That's awesome.

09:13 Why don't you talk just really quickly about how you got people to do this?

09:18 Because so many organizations, whether it's Python, my classes, some other random classes,

09:23 they have problems of getting people to actually want to learn, to be excited about it, and so on.

09:29 You have a lot of cool gamification and sort of social aspects to it, right?

09:33 The first bit is really about confidence.

09:35 And curiosity.

09:36 So if you ask people, well, how did you get into programming?

09:39 There's always typically a story about how they were curious and how they explored things in an environment where it's quite safe.

09:45 And I think people think of programming, I blame them.

09:49 I blame Hollywood.

09:49 You know, you see these whizzing zeros and ones going by on a green screen, and they make it look complicated.

09:56 And it kind of puts people off because they think that's maybe not something that's for me,

10:00 or that could be something I'm going to struggle with.

10:03 So the first thing you're going to do is build up people's confidence to just give it a try,

10:06 and then put them in an environment where they can just learn the very basics and build them up from there.

10:12 So that's really what we've been trying to do.

10:14 First of all, the course is super important.

10:16 But then also encouraging people to sit with each other in offices and different places and just help each other and collaborate with each other and work through some of the hurdles they might have.

10:28 So in different countries, you've got different programs depending on what food is popular.

10:32 So South Africa, for example, we've got there's a donut morning that runs every week, where people to get together and talk about their Python challenges and work through the course and things like that.

10:41 Thailand, as an example, they have a challenge to do one of the apps in the jumpstart course every week.

10:47 And they have a little celebration every time someone finishes one of the apps.

10:51 So that's kind of why we've seen so much popularity in Thailand.

10:53 It's all about people, really, and getting them together and helping them learn.

10:57 Yeah, that's awesome.

10:58 Hopefully a lot of organizations can adopt some of that because that's a really big difference.

11:03 I mean, just think of a company of 30,000 people where they all have some capability to automate parts of their job.

11:10 It could really make a big difference.

11:11 Yeah, definitely.

11:12 It's not that we're trying to make everyone developers because that's not what we need to do.

11:17 It's that basic coding skills is going to be almost like a requirement going into the future.

11:22 So just as I can use Microsoft Outlook or Excel, that's kind of assumed when you go into a professional job nowadays.

11:31 And I think programming, now that it's being taught in schools to pretty much all students in some countries, it's going to be really useful.

11:38 And especially in our space in technology, everything is now software-defined and automated and has APIs.

11:44 So coding is now the new way of interacting with it.

11:47 So people need to learn that.

11:48 It's definitely a superpower.

11:49 It's awesome that you're doing that.

11:50 So speaking of courses, we were talking a while ago.

11:53 You said, hey, I'm working on this really cool course on moving from Python 2 to Python 3 and helping people with that.

11:59 So I'm really excited that you built this.

12:01 This is definitely something that should exist out there.

12:03 Why don't you tell people quickly about your course and how they can get to it?

12:06 Then we'll get into the details of why and how.

12:09 This is a course really talking about how to move from Python 2 to Python 3.

12:13 And it goes through a whole range of things, some of which we'll talk about today.

12:17 So that's, you know, what changed?

12:19 How should you move?

12:20 How should you plan for the change?

12:22 How to test it?

12:23 Really kind of all the things that you'd need to know to move a library or an application from 2 to 3.

12:29 So it's a pretty big challenge.

12:31 But I've tried to really break it down into small pieces.

12:33 So I think, you know, if you can break something down to small enough pieces and just start with the first one, then it really makes any problem a lot easier.

12:40 So this will be a course that will be available on Pluralsight.

12:43 So if you have a Pluralsight subscription, if you're a professional developer, your company should have already provided you a Pluralsight account if you've asked.

12:51 And if they haven't, you should push them a bit harder, especially access to some of the talk Python courses as well.

12:56 I'd say, you know, really kind of lean on your employer to give you access to this stuff.

13:00 So, yeah, that will be out on Pluralsight.

13:02 It's about two hours long.

13:03 So it's not too bad.

13:05 You can watch it over, you know, maybe like a short afternoon or something and play with some of the examples.

13:09 Yeah, maybe you could even have people do something really awesome, kind of like what you're doing with some of your teams, where you could literally get the entire team together and sit and experience the course like in a projector in a conference room in an afternoon.

13:24 And then or maybe in the morning, in the afternoon, you go and actually start the conversion and plan that out.

13:29 Right. That'd be awesome.

13:29 Yeah, definitely.

13:30 A lot of the stuff in the course is actually about having conversations with your team about how to move the application.

13:37 I think I've kind of assumed that what you're trying to move to Python 3 is typically a medium to large application.

13:44 If it was a simple script, it would be pretty straightforward.

13:47 So, yeah, it's kind of assuming it's a large one.

13:49 It's not just you working on the project.

13:51 So, yeah, I mean, let it out.

13:54 If it's like 500 lines, like you could just rewrite it.

13:57 Right.

13:57 It's not a big deal.

13:59 But if it's Instagram with a million lines of code, if it's Dropbox, if it's Bank of America, like it is the essence of your product and hundreds of people work on it.

14:09 That is an entirely different undertaking.

14:11 Right.

14:12 Yeah, definitely.

14:12 For sure.

14:13 So, before we get into the details of sort of how, like three years ago, it really felt to me like there was this debate of whether Python 2 or Python 3 was the right thing to do for your projects.

14:27 It seems to me as if that's more or less settled in Python 3's favor.

14:32 What do you think?

14:33 Is that debate still worth having or is it kind of done?

14:36 I'm still having the debate with people on different projects and things like that.

14:42 I say, you know, like I said earlier, it's been out for a decade, but that's not really a fair statement.

14:47 I mean, 3.4 has been out for four years now.

14:50 And I'd say that's probably the first kind of really stable release of the 3 series.

14:55 So, we've really had the last four years to kind of work with Python 3 and some of the changes.

15:01 And Python 2.7 specifically is such a stable release of Python 2 that it's been a real challenge for people, I think, to have a reason to move across.

15:12 So, Python 3.4 is as good as Python 2.7.

15:16 There's a few more extra features.

15:18 Futures came out there.

15:19 I think the AsyncIO library came in 3.4.

15:22 So, there's kind of like a couple of reasons to move.

15:24 But definitely with 3.5, 3.6, and now 3.7, there are lots of reasons to move across.

15:30 So, I think that's settled that part of the debate.

15:32 And then also, we've got the upcoming, you know, end of support for Python 2.7, which people are going to say, you know, well, it works fine.

15:41 Maybe if security holes and things like that come out in the future, we can resolve them.

15:45 But lots of people don't actually realize there are existing holes in Python 2.7.

15:49 In the XML modules, for example, there are some pretty well-documented flaws.

15:54 In some of the temp file standard library modules, there are some security holes.

15:58 So, this is going to happen that there are reasons to move across to Python 3 from a security point of view.

16:03 But then also, I don't think we should keep trying to divide the community because we've seen time and time again in other languages like Perl 5 to 6, for example.

16:13 And then, you know, .NET's going through it at the moment, moving to .NET Core, where they're kind of, there's been a huge overhaul trying to fix all the big issues that they had in the old version.

16:22 And it's kind of separating the community until they can all agree to move forward and get on with the new.

16:28 Well, I think that's a really good point.

16:29 And one of the major frustrations and challenges of that, I think, is you mentioned the .NET Core stuff.

16:34 I think you see it there.

16:35 You certainly see it with the core developers in Python, is that the people who create and maintain and improve Python are putting all of their energy and effort into Python 3, right?

16:47 That's where the work is being done.

16:50 And you're only going to, you know, you're only going to get the benefit and realize all those things that they've added in terms of performance and memory and language features and everything if you make the jump, right?

17:03 Yeah, definitely.

17:04 Yeah.

17:04 So the sooner that gap is sort of over with and everybody's back focused on the same thing, I think it's really interesting.

17:10 I was at PyCascades recently, a conference in Vancouver, and Guido was there, and he gave the keynote, which was a retrospective on Python 3.

17:20 And he talked about why he felt there were challenges moving to Python 3, why this whole thing began, right?

17:26 Obviously, there's the benefits, but like why, what could they have done to avoid this gap, basically?

17:33 It's a really interesting talk.

17:34 I'll link to it in the show notes.

17:35 One of them I think that was interesting was basically underestimating the importance and significance of third-party libraries, right?

17:45 So how do you think we're doing there, right?

17:46 There's places you can go check and say, does my library support Python 3 for the popular ones on PyPI?

17:52 Like what ones are converted and so on, right?

17:54 Yeah, I think we're definitely up there now.

17:56 There's a few tools you can use that check for Python 3 compatibility for your modules.

18:02 So if you're pulling stuff from PyPI, but from what I checked last time, of the 200 most popular modules on PyPI, I think 95% of them are now compatible with Python 3.

18:13 And the ones which are left over have Python 3 equivalents.

18:18 So I think, you know, for the big packages, we're pretty much done.

18:21 It's kind of a lot of the really niche applications and more obscure packages that have been a bit of a challenge.

18:28 And yeah, there's definitely ways to, you know, upgrade third-party packages if you can't find the original developer.

18:37 Some of that has really been because people publish them and then, you know, say, oh, I'm done with this and move on to something else and it never gets updated.

18:43 Yeah, for sure.

18:44 And you actually wrote a nice article on like, I depend on a package that's Python 2.

18:49 Now what?

18:49 How do I fix this, right?

18:50 Yeah, I've had that issue many times where I've pulled it in a package that only works in Python 2 and I've had to update it to 3.

18:57 So that's kind of where some of my experience in writing this course has come from has been, you know, updating other people's applications, but also maintaining some pretty big packages, you know, like Apache Lib Cloud, which supports both 2 and 3, which is the sort of portability approach.

19:11 Pretty interesting.

19:12 So before we move on to the goodness in Python 3, I want to try to spread this concept a little farther in the community.

19:22 I talk about it all the time with Brian Okken on Python Byte.

19:25 And if you say there's Python 2 and there's Python 3, they kind of feel like equivalent.

19:29 Oh, there's these two things and they're similar.

19:31 But if we all start calling Python 3 just Python and Python 2 legacy Python, when we have these conversations with managers and stuff, they're like, wait, our system is written in legacy Python.

19:42 That doesn't sound good.

19:43 What is that?

19:43 So, you know, hopefully that term resonates with people.

19:47 Yeah, I think so.

19:48 Yeah, and definitely.

19:49 I mean, some of the tools I work with a lot is Ansible, SaltStack, and another tool called StackStorm, which are about infrastructure automation.

19:58 All of those were written in Python 2 originally, and they still primarily support Python 2.7.

20:03 And these are huge code bases, and they're all trying to go through the migration at the moment to Python 3.

20:08 So, yeah, it's definitely, they're seeing, I think, in the last 18 months, I think there's been a big shift.

20:14 You know, people really starting to realize that this Python 3 thing isn't going to go anywhere.

20:18 It's not going to go away.

20:20 They're not going to go and say, you know what?

20:22 We made a mistake.

20:23 We're going to move back to byte strings instead of unicode strings.

20:27 Yeah, yeah, exactly.

20:28 All right, so let's maybe a good place to talk about what you get by moving to Python 3.

20:33 What's better?

20:34 Actually, what are the tradeoffs?

20:35 What's worse?

20:36 Things like that.

20:37 The most wide-reaching shift is that Python 3 is faster.

20:42 And I put that in air quotes, not that you can see them.

20:45 If you go on speed.python.org, they run a continuous speed test against all the different versions of Python.

20:53 And you can go and compare specific things between Python 2 and Python 3.

20:58 And you can even compare different versions, say, like 3.5 versus 3.6.

21:02 So in pretty much all of the cases that I've used, Python 3.5 and above is faster than Python 2.7.

21:11 So you're probably going to see a speed improvement in moving to Python 3.

21:16 Then you've got the async and await support in AsyncIO.

21:21 And if you start to implement some of those features, you're definitely going to see a big speed improvement.

21:25 So that's a big shift.

21:27 That could just completely unlock it.

21:29 You know, just one story on that.

21:30 There was a guy who we talked about AIo HTTP client, which is a little like request, but lets you await while you're waiting on the server and do other requests.

21:40 This guy was downloading a whole bunch of stuff off a server and it was taking like a day or something really, really long.

21:46 And he switched from request to the async and await version.

21:50 And then his machine crashed because he ran out of memory going so fast.

21:53 Then he had to like throw out a little bit.

21:55 But it was like it was like a minute versus a day.

21:58 And it was something really super dramatic like that.

22:01 So when you think about performance, it's never so simple as just it's always faster in this way or that.

22:06 But it could be, well, there's this new algorithm or technique that's unlocked that is so much better that, you know, you shouldn't compare apples to apples.

22:15 You should like pick the new shiny apple and compare that to the old, you know, the old unpolished apple.

22:20 I don't know what the metaphor there is.

22:22 But there's a bunch of great stuff in there.

22:24 And then, you know, you talked about performance in terms of like speed.

22:27 There's also improved memory, right?

22:29 Yeah, definitely.

22:29 There's been some significant improvements under the covers in Python 3.

22:34 So, I mean, if you're just writing with Python 3, you'd never really see most of those changes.

22:39 But in the actual execution, they did a lot of work to clean up a lot of the C code and definitely a lot of the optimizations around how it handles memory consumption and garbage collection and things like that.

22:50 So, I've definitely seen some big improvements.

22:53 There's some more coming in 3.7 in terms of how you call methods, which are part of a class instance.

23:00 So, that's going to get a bit of a speed improvement, about 20%.

23:03 So, yeah, there's lots of things coming and there's lots of reasons to move across.

23:06 Yeah, there's actually a lot of stuff still coming.

23:09 And I think it's interesting to, while we're on this performance bit, is I think this is around the 3.5 timeframe where the core developers were like, one of the actual things we need to focus on, one of the features is actually performance.

23:23 Because it's great that we have async and await or type annotations or something like that.

23:28 But if we can say your code uses half the memory or runs twice as fast, like that will solve the Python 2.3 divide right there, right?

23:35 So, they've actually been putting a lot of effort into 3.5, 3.6 and 3.7 around that goal.

23:42 Yeah, definitely.

23:43 And also, Python 3 has kind of embraced iterators more in some of the built-ins.

23:48 So, when you run filter, for example, or zip in Python 3, it will return an iterator instead of a list, which it would in Python 2.

23:56 And now, when you're working with dictionaries, if you're calling keys, values, and items on a dictionary, it's going to return a view, which has an iterator built into it.

24:06 Whereas in Python 2, that would be a list.

24:08 So, yeah, they definitely kind of embraced iterators a lot more in the built-in types.

24:12 And you'll get definitely memory, better memory consumption by using those.

24:16 What's cool about that is a lot of those is you don't do anything.

24:20 You could just, you already were like four in over in the keys or something, but now it just does it better.

24:26 Really nice.

24:27 Yeah.

24:28 Just don't modify the dictionary.

24:29 Oh, yeah, there's that.

24:35 This portion of Talk Python to me is brought to you by Linode.

24:38 Are you looking for bulletproof hosting that's fast, simple, and incredibly affordable?

24:41 Look past that bookstore and check out Linode at talkpython.fm/Linode.

24:46 That's L-I-N-O-D-E.

24:48 Plans start at just $5 a month for a dedicated server with a gig of RAM.

24:53 They have 10 data centers across the globe, so no matter where you are, there's a data center near you.

24:59 Whether you want to run your Python web app, host a private Git server, or file server, you'll get native SSDs on all the machines, a newly upgraded 200 gigabit network, 24-7 friendly support, even on holidays, and a seven-day money-back guarantee.

25:13 Do you need a little help with your infrastructure?

25:16 They even offer professional services to help you get started with architecture, migrations, and more.

25:22 Get a dedicated server for free for the next four months.

25:25 Just visit talkpython.fm/Linode.

25:28 Let's talk a little bit about the carrots.

25:31 You talked about the stick.

25:33 Python 2 is going to go end of life in 2020, just to put a date on it.

25:37 It's unclear.

25:38 Have you heard anything about the actual day?

25:40 I only know the year.

25:41 I've heard it's PyCon, so it'll be PyCon 2020.

25:45 It should be PyCon 2020.

25:47 I hope they do that.

25:48 I think there's going to be a burning of the source code or something.

25:51 We'll see what happens on the day.

25:53 That'd be a great party to celebrate what it's done for us, and like a goodbye party, basically.

25:57 That'd be awesome.

25:58 All right.

25:58 So that's the stick.

25:59 If you don't move and there's a security vulnerability or something bad, you're in a bad place.

26:04 And it's not just the runtime, right?

26:06 It's not just Python.

26:07 It's the stuff that you might depend upon on top of it that hasn't moved.

26:11 We'll get into that.

26:11 But let's talk about the carrots.

26:13 Like, what are some of the good things you get from moving to Python 3, other than the performance, which we already talked about?

26:18 Yeah.

26:18 So there's a lot of work with async and await being introduced as new keywords.

26:23 Anyone who's worked with C# before will definitely be familiar with that syntax.

26:28 So I think that's actually where it was inspired by was from the C# language.

26:33 So really, this is a way of making it a bit easier to, I guess, work with asynchronous programming and to specify a bit more upfront that your functions can be awaited on.

26:44 Especially when you're waiting on something, like you're calling a web service or a database.

26:48 Your code could be doing so much more while you're just waiting for that network response, right?

26:52 And this makes it drop dead simple.

26:53 Yeah, definitely.

26:54 I think there's still a bit of confusion with the sync and await.

26:57 You know, we've not seen it that widely adopted because it really takes a bit of time to get around the concept.

27:03 You can't just stick it in front of every method and expect your code to just run faster.

27:07 It doesn't quite work like that.

27:10 It's more about scalability than it is about pure performance increase for a single request, right?

27:14 Yeah, definitely.

27:15 And if you're working with REST APIs a lot, which in the modern world is pretty typical,

27:20 then waiting for a response back from an API can keep your code, you know, hanging, essentially.

27:26 So this is a great way to kind of speed that up.

27:29 Dictionaries have got a bit of an overhaul.

27:31 Especially around microservices, right?

27:33 If you're calling more than it's not just one thing, you're calling a bunch of little services that you could really like clog that up.

27:38 So this will unlock it.

27:39 Yeah, definitely.

27:39 Yeah, dictionaries.

27:40 Go ahead.

27:41 There's been a separation before between the order in which the dictionary keys have been set up based on how they were hashed.

27:49 And in the newer versions of Python 3.6, they will be ordered by default.

27:53 However, you should not assume that that will always be the case.

27:57 So whilst it's a great feature, you know, you should probably still import from collections and get ordered dicks that way.

28:04 So the performance improvements are pretty great.

28:07 Also, there's the yield from keyword.

28:09 So if you're working with generators, iterators a bit more, then, you know, you can leverage that to make sure you don't have to exhaust the generator before you yield it back.

28:19 It really makes like that sort of consuming generators and passing them on or the really good and recursive sorts of situations.

28:25 You can sort of looping over and yielding an individual item.

28:29 You just say, here's a generator.

28:30 Give me all this, you know, return all the stuff out of there.

28:32 It's nice.

28:33 Yeah, definitely.

28:34 Type annotations is a great feature.

28:36 And there's some really interesting ways to work with type annotations.

28:40 Data classes is coming out in 3.7, which heavily leverages type annotations.

28:46 So this is really a way of creating a class which represents a model of data, like the response you would get back from an API.

28:54 And you can actually say, here are the types.

28:57 And if you're working with a good IDE, it's going to give you a lot of hinting as to what methods are going to be on that instance and how to leverage it.

29:04 So that's another great, great feature.

29:06 Yeah, type annotations are really, really great.

29:08 I was a little unsure what they would do to the language.

29:11 Would they make it like C# or Java or something like that by, you know, making you say the types everywhere?

29:17 And it's, I think a little sprinkling of them around is really working out quite nice from my experience.

29:22 There's some really cool stuff that Facebook and Instagram folks are doing type annotations.

29:27 So when we were talking earlier about the third-party packages that were not converted to Python 3, there was maybe a couple that you said, well, these aren't converted, but there's Python 3 equivalents.

29:40 But what I'm noticing a lot more recently over the last year or so is things that are only available in Python 3.

29:47 So really awesome third-party libraries you only get, right?

29:51 So it's kind of almost the reverse, right?

29:53 Like, well, if you don't move, you know, the world is moving on without you.

29:56 So some of those come to mind.

29:58 You already mentioned the REST full services.

30:00 So there's API Star, which is the newfangled thing from Tom Christie.

30:05 He does Django REST framework.

30:06 But this is like it uses type annotations to inject parameters from the API and all sorts of really interesting stuff.

30:14 There's Quart, which is a async and await enabled Flask API compatible derivative.

30:21 And maybe the biggest one of all is Django 2.

30:23 Yeah, Django 2 is a great step forward, I think, in terms of getting people to move to Python 3.

30:28 And, you know, even using Python 3 as the default on their tutorials and stuff like that, you know, contributed a lot to the uptake.

30:37 Looking at PyPI download stats for some of the packages that I maintain or work with, you still see like a large number of people using Python 2.

30:47 And I guess the question is, do they know which version of Python they're consuming?

30:52 Or are they just, you know, running some commands in a shell and it's pulling in packages from somewhere else and they don't really question it?

30:58 So I think that's a really interesting point.

31:00 And how much of that is continuous integration systems that happen to be set up to test on all the versions of Python?

31:06 So they also are pulling Python 2.7.

31:09 Yeah, definitely.

31:10 And there's also still a challenge.

31:12 I mean, Django, I think, have stepped forward and said, you know, from Django 2, we're just not going to support Python 2.

31:18 And if you do pip install Django from a Python 2 distribution, it will just give you an error.

31:23 Unfortunately, the error is not this doesn't support Python 2.

31:27 It's something really obscure.

31:28 It seems like that'd be pretty easy to put in the setup.py for that package.

31:32 Just go, if version, you know, major is two or less, just raise this error.

31:38 Sorry, wrong version or something, right?

31:39 But it doesn't matter.

31:41 The fact that Django, all Django work going forward is focused on Python 3 only is going to make such a huge difference.

31:47 It let them forget trying to write two different versions of code for everything they're doing.

31:52 They can just focus on new features, which is really good.

31:55 You touched on data classes, but I don't know that everyone necessarily knows exactly what they are.

32:00 Could you give us just like the quick summary of what those things are?

32:03 There are three, seven.

32:04 They're coming soon.

32:05 It's very similar to the Attrs project.

32:07 And this is if you have a class in Python, which only has really one purpose, and that is to store some data to reflect on some data that you might have in a database or in Mongo, or it gets responded from a REST API.

32:24 So this would have a number of fields like ID and name and things like that.

32:28 And you want to store them in an object so that you can write the fields and you can instantiate them quickly.

32:34 So there's a lot of boilerplate code you typically have to write to do that sort of thing.

32:38 If you use Atters, it definitely becomes a lot easier.

32:41 But I can't think of how many times I've had to write a class with just a long list of, you know, init variables and then just basically storing each one of those in the class instance.

32:50 So this basically gets rid of all that boilerplate for you and just automates it.

32:54 So you declare a data class, you specify all the fields you want to have, and then it implements dunder init for you, dunderrepper, dunderstr, dundereq.

33:03 And it's got some great other features as well.

33:06 You can make it immutable by saying it's frozen, in which case it also implements dunder hash, which actually can be kind of challenging to do.

33:13 Pretty cool.

33:13 So yeah, these are all sorts of things that are in Python 3 only, right?

33:19 We kind of have to wait for a little while because 3.7 is not yet out.

33:24 We'll talk more about that in a bit.

33:25 But maybe let's talk about some of the strategies of actually doing it.

33:29 Okay, so people think they sound great.

33:31 They're like, awesome.

33:32 Data classes, I definitely want that.

33:34 Ace to get away, I want that.

33:35 But I have all this code that's in Python 2.

33:38 How do I move it?

33:39 What are my strategies?

33:40 In the course, the way I try to explain this problem is, so I live near the beach on the east coast of Australia.

33:47 And I went down to the beach and basically shot a series of photos of me jumping from one rock to another one with some water in the middle.

33:55 And what I was trying to illustrate is that you're on Python 2 and you're on solid ground and you want to move to Python 3.

34:02 And it's extremely tempting to just say, okay, let's just go and rewrite the whole thing and jump from one side to the other.

34:08 But the real challenge with that, I think, is if it's a large application, then the amount of time you're going to spend suspended in midair where your application neither works in Python 2 or Python 3 could be quite a long period of time.

34:24 And if any customers come to you or you need to make any changes to the code base or whatever during that time, it's going to be really challenging.

34:31 So, you know, you could go and create a separate branch of your application and just rewrite everything in Python 3.

34:37 But definitely you're going to get also the temptation to go, oh, well, since we're rewriting in Python 3, you know, let's also go and rewrite everything in microservices and use the latest design patterns.

34:49 So, you know, let's just move to the newest version of Python 2, you know, let's re-engineer the whole application from scratch.

34:58 So, you know, if you've got a big enough team and a big enough budget, then great, you know, knock yourself out.

35:03 I think a more realistic approach is to start to introduce Python 3 support in specific modules or parts of your application.

35:13 So this is kind of like the portability approach.

35:15 Now, you probably don't want to support Python 2 for very long after you've done the move, but it's a lot easier to introduce Python 3 support to an already working application.

35:26 I think that's the most important thing is that your application today, or at least I hope it does, works.

35:31 So, you know, really introducing Python 3 support piece by piece means that you can maintain a working application whilst moving to Python 3.

35:40 That's going to be a lot easier to, you know, land with the managers.

35:43 Yeah, it definitely is to say our code, we're making it better and eventually we'll be off of legacy Python.

35:49 But at any moment we could ship it.

35:51 And it's just as we work with it, it gets closer and closer to that goal, which is really awesome.

35:57 I think one of the most insane stories and insane in an awesome way stories of this is what Instagram was doing.

36:05 And they really documented this at PyCon 2017 with one of the keynotes there.

36:10 Yeah, the Instagram story is amazing.

36:12 And I think it's a great case, you know, to talk about the power of Python and the scalability of Python 3 as well.

36:19 You know, whenever I hear someone say Python's not scalable, I just say, oh, how many users do you have?

36:24 Is it more than, you know, half a billion?

36:25 Well, you probably don't have an issue.

36:28 Exactly.

36:29 What response time do you need?

36:31 Six milliseconds is not fast enough, right?

36:33 I mean, maybe, maybe very rarely, but yeah, it's amazing.

36:37 So at PyCon 2017, Instagram, a couple of the engineers from there got up and they gave this really compelling presentation of they were on an old version of Django.

36:47 And I don't even think it was necessarily Python 2.7.

36:49 It was like a pretty old version of Python.

36:52 It definitely was some variant of two.

36:55 And they talked about how they moved all of Instagram over to Python 3 without taking down the site, without creating a separate branch and continuously shipping and running their program.

37:09 And so some of the techniques were things like, we're going to make our code run on both Python 2 and 3 instead of just rewriting it.

37:17 So we're just going to make it compatible with both.

37:19 And then they started running unit tests on both versions, Python 2 and Python 3.

37:24 And they'd only deployed a production if it would pass on both.

37:27 But for a long time, production was Python 2.

37:29 And then slowly they started like running production Python 3 versions, but only giving those to the internal people and, you know, just slowly growing up.

37:38 But it was really great.

37:39 They did this major conversion.

37:40 They didn't branch.

37:42 They didn't, like, kill the momentum for a while.

37:44 It's pretty cool.

37:45 Yeah, I think it kind of goes back to some other examples.

37:48 I think of Windows Vista as well as a great example of where to be cautious of reintroducing bugs that you solved, you know, decades ago.

37:57 I remember the first time they released the beta, they'd say, oh, we've rewritten the TCP IP stack from scratch.

38:04 And they'd actually re-implemented something called ping of death, which is like a late 1980s security hole in the stack.

38:13 So you should definitely not dismiss a lot of the robustness of your old application, which is why testing is so important in the migration story.

38:20 So, you know, as I talk about in the course, Python is a very, very dynamic language.

38:25 You know, you can pretty much override anything and all sorts of crazy things can happen at runtime.

38:30 So the better testing coverage and, you know, the more robust test suite you have, the easier it's going to be to move.

38:37 And then also making sure that you don't reintroduce bugs that you've already solved, which is why I'm more of a fan of the sort of gradual migration process than the completely rewrite.

38:48 Because I'd be concerned that you'd introduce bugs, which you either didn't have before or you'd already solved.

38:54 Yeah, there's always so the temptation is right there to just say, you know, we could rewrite this.

38:59 We could do it better.

39:01 And that may be a great thing.

39:02 But I would say, like you pointed out before, separate that from the concept of moving to Python 3.

39:07 Do the move and then think about a restructuring of some part, if that makes sense.

39:12 But there's no reason to conflate them and make them both riskier.

39:15 Yeah, definitely.

39:16 Yeah.

39:16 So how about some of the tools that people might use?

39:19 Right.

39:19 There's some some linters and other things that might work, right?

39:22 Yeah.

39:23 So probably the easiest one to use is when you're running Python, you can use the dash three flag.

39:28 So if it's Python 2, you can use the dash three flag and it will actually give you warnings on the shell where you've used something which is not compatible in Python 3.

39:37 So that's kind of like the easiest way of doing things.

39:39 You've also got some linters.

39:42 So if you're using pilot as an example, if you're using pilot from Python 2, it will only give you linting warnings that are relevant to Python 2.

39:51 And if you're calling it from Python 3, it will give you newer warnings.

39:54 So linters can be really useful in terms of understanding or at least trying to guess what types certain variables are.

40:02 Tox is a tool which I highly recommend.

40:06 So let's say you already, at least I hope, you've got some sort of tests written in unit test or pytest or something like that.

40:13 Basically just introducing Tox, which isn't actually that complicated.

40:18 It's just pip installs Tox and then you can run Tox hyphen quick start.

40:23 And it will just ask you a couple of questions about which Python versions you want to test.

40:27 So Tox, basically, each time you run it, it will create a virtual environment for each version of Python and it will run your test suite for those different versions.

40:36 So if you just wanted to target.

40:38 That's really cool.

40:39 Yeah, it's fantastic.

40:40 I think people have been a little bit put off because it takes it takes longer to run the test suite on multiple versions.

40:47 If you install a tool called Detox, it will actually run the multiple versions in different threads.

40:53 So it takes the same amount of time as it would just to run your normal test suite.

40:57 Yeah, that's pretty cool, actually.

40:58 Detox.

40:59 I love it.

40:59 Yeah.

41:00 So the idea is you can basically have your continuous integration or just your unit test execution basically automatically run on all the versions that you check.

41:10 And it could even be things like PyPy, right?

41:13 It doesn't have to be just Python 2 versus Python 3 and CPython.

41:15 Yeah, definitely.

41:17 PyPy runs on a two equivalent syntax.

41:21 And then PyPy 3 has still been a bit of a challenge, actually.

41:25 We've had challenges in pytest support.

41:28 I'm not sure if they've fixed that yet with PyPy 3.

41:30 It's been a bit of a thing to get over.

41:32 So yeah, I do caution people who are using PyPy and moving to PyPy 3 and the supportability of third-party packages.

41:40 Yeah, that's definitely a little bit tricky.

41:42 So it sounds like you definitely recommend continuous integration.

41:45 Yeah, continuous integration.

41:46 So start off simple, which is just run your tests in Python 3.

41:51 And another thing you can do is Python actually has a compile-all method, which is in the standard library, which will go through a directory that you have, look at all the Python source files, and try and just compile them all up front.

42:06 That's another handy trick you can use just to check, just to do a quick sense check of the syntax that you've got and make sure you've not done anything blaringly obvious.

42:14 Right, because normally, because normally, you've got to do a quick sense of the Python files, and you've got to do a quick sense of Python 3.

42:22 But you can sort of force Python to parse all the files and turn them into PYCs, right?

42:27 Yeah, definitely.

42:27 And then as you start to introduce unit testing, think about, in particular, the string change in Python 3, which we'll probably talk about in a minute, and having more testing on your application for Unicode strings.

42:41 So that's definitely going to bring up some challenges in your existing code.

42:43 This portion of Talk Python to Me has been brought to you by Rollbar.

42:49 One of the frustrating things about being a developer is dealing with errors.

42:53 Relying on users to report errors, digging through log files, trying to debug issues, or getting millions of alerts just flooding your inbox and ruining your day.

43:02 With Rollbar's full-stack error monitoring, you get the context, insight, and control you need to find and fix bugs faster.

43:09 Adding Rollbar to your Python app is as easy as pip install Rollbar.

43:12 You can start tracking production errors and deployments in eight minutes or less.

43:17 Are you considering self-hosting tools for security or compliance reasons?

43:21 Then you should really check out Rollbar's Compliance SaaS option.

43:24 Get advanced security features and meet compliance without the hassle of self-hosting, including HIPAA, ISO 27001, Privacy Shield, and more.

43:34 They'd love to give you a demo.

43:35 Give Rollbar a try today.

43:37 Go to talkpython.fm/Rollbar and check them out.

43:43 Yeah, when I hear people working with challenges in their code or problems they have converting, number one is there's some package I depend upon and it's not working.

43:54 But number two is strings.

43:57 It seems like that catches a lot of people, especially if they're working at the network layer where they're getting bytes off the network and then they want to do string likes.

44:05 It's really ultimately supposed to represent a string or something?

44:08 Yeah, I hugely support the decision in Python 3 to change the core types.

44:13 I think it was the right choice.

44:15 So kind of going back to, I guess, what changed.

44:19 In Python 2, there's a type called string or STR.

44:24 And I think people have used it for two different purposes.

44:28 And this is kind of coming back to some of the issue.

44:30 It's an 8-bit string type.

44:33 And it just happens that the ASCII characters, which are the English alphabet plus things like, you know, basically everything you would find on an American or an English keyboard fits in the ASCII character set.

44:45 Now, that set, because it doesn't have that many characters, fits inside 7-bits.

44:51 So you can use it as a way of transferring, sending strings, which only include American English characters.

45:00 So people would use the STR type for text and also use it for binary data.

45:07 So if they were reading images, as an example, you'd still use the STR type.

45:11 So I think part of the issue has been that there was also a Unicode type.

45:16 So if you're a really good citizen, you know, you would try and use the Unicode type, but it would introduce all these weird issues in Python 2.

45:23 So one of those issues is that Python 2 will try its hardest if you concatenate a string type and a Unicode type.

45:32 So this is the byte string and the Unicode string.

45:34 It would basically try and convert the byte string to Unicode and then bring the two things together.

45:40 Now, what that leads to is runtime errors specific to the values that you have in the strings.

45:48 So this is kind of the Achilles heel of Python 2's Unicode support is that it just tries to basically run either an encode or a decode at runtime.

45:58 And then if you've got specific values in the strings, then your whole application can just suddenly crash.

46:04 So, yeah, that's been a big challenge.

46:06 And it may well be another culture that you don't typically interact with.

46:11 So it's a really hard edge case to catch, right?

46:13 Like Mandarin or something like that, that I would never type that into my program.

46:19 But other people would.

46:20 It totally makes sense to them, right?

46:21 The case I talk about in the course is, you know, the argument has always been, oh, we don't need Unicode because, you know, all of our customers use English.

46:30 But there's been this new culture called emojis, which is introducing Unicode characters and everything.

46:35 People want to put smiley faces and snakes in their display names or their profile names or, you know, put all sorts of weird things in their addresses.

46:44 So actually, I think the culture of emoji has pushed forward and people trying to think about Unicode support.

46:52 And that's uncovered a lot of the challenges in Python 2.

46:54 So really what they did in Python 3 is to make it a lot more explicit about binary data and text data.

47:04 So the, what was the Unicode type in Python 2 is now the string type in Python 3.

47:10 And if you want to use binary data from reading and writing to say images or network sockets, there's a new type called bytes.

47:19 And those things are very, very different.

47:21 And if you have a string literal, so if you just say hello world in double quotes or single quotes in Python 3, that will be a Unicode string by default.

47:31 And I think that's caught a lot of people out.

47:33 If you're just passing strings around your application, you probably won't see the difference.

47:37 But if you're reading and writing from sockets or files or interacting with libraries which return binary data, then that's when you start to see these Unicode encode errors and things like that.

47:48 Yeah, definitely right at the boundaries of your app where stuff comes in and out.

47:52 That's the danger zone.

47:53 So one other tool I wanted to just throw out there while you're on this topic is mypy.

47:59 So mypy is an optional static typing, linter typing for Python 2 and for Python 3.

48:06 And if you're going to move your code over, one thing you can do is you can add typing to it in the Python 2 style to actually verify that it's working the way you think and everything.

48:18 And so then when you do the switch, you can make sure that, for example, you haven't switched to string for bytes or something weird like that, right?

48:24 It's still all the same types that you were thinking about.

48:28 This is a major project that Guido von Rossum is working on, so it's pretty significant.

48:33 Yeah, that sounds really useful.

48:34 Another thing you can do is just to think about when you've got strings in your code, whether they're text or whether they're binary.

48:42 And I think if you just, instead of thinking about the actual types, if you think about the purpose of the string, then it's a lot easier to pull across to Python 3.

48:50 Yeah, for sure.

48:51 Do you want to talk about just some of the things that change, like, say, the language level?

48:55 So we talked about strings.

48:57 There used to be longs.

48:59 Are there still longs?

49:00 Long in Python 2 is a very, very, very long number.

49:04 You know, almost infinitely long, depending on how much memory you have on the computer.

49:09 And then int was a fixed length numeric type in Python 2.

49:15 So what they've actually done is they renamed the old long type to the int type.

49:21 And then the old int type is no more.

49:22 So if you ever refer to long in your code, then it will give you an error in Python 3.

49:29 There's ways around that.

49:29 You can use the future package, and you can actually import the long type from there, which just redirects to the new int type.

49:35 That's a way of safeguarding.

49:37 Also, if you're working with numbers and you're using the division operator, so, you know, the forward slash, if you divided two integers in Python 2, it would return an integer.

49:47 So if you said one slash two in Python 2, it would give you zero.

49:51 In Python 3, the division operator will give you a floating point number back if that's the actual result.

49:59 So if you do one slash two in Python 3, it will give you 0.5.

50:04 So that's another thing that catches a lot of people out.

50:07 That's got to be super subtle as well.

50:09 If you're saying, do this division, then if it equal equals zero, we do this.

50:13 Otherwise, we do that.

50:13 And of course, no, no, not so much.

50:15 We have a couple of tools that we'll go through and adjust some of this stuff, right?

50:20 Yeah, they can at least make it easier to expect the new behavior when you're running Python 2.

50:26 So I mentioned the future module that definitely does that.

50:29 So you can actually import the new behaviors of certain operators and types into Python 2 using packages like future.

50:37 Have you done anything with 2 to 3 or 6?

50:40 So I know those are not the same purpose, but they're both sort of tools or packages to help you adjust.

50:46 There's three ways of doing it.

50:47 So there's the 2 to 3 application, which comes with Python 3.

50:51 So it's maintained by the core team.

50:55 And that will do its best to automatically make your code portable or executable in Python 3.

51:01 I think when Python 3 initially came out, they said, if you pip install a package and you want it to work in Python 3,

51:09 then inside setup.py, you could actually run 2 to 3 over the source code.

51:15 That's pretty ambitious.

51:16 Yeah, it's ambitious.

51:17 And people pretty quickly realized it's not actually that straightforward.

51:21 So, you know, what I recommend to people is that you can use these tools to fix really mundane things,

51:29 like the use of the print statement and changing it to a print function.

51:33 So, you know, that's something that's not really, you're not really adding much value by doing it by hand.

51:38 So 2 to 3, and there's also two other tools called Futurize and Modernize.

51:42 I'd actually recommend using Futurize instead of 2 to 3.

51:46 It bases itself on the future package.

51:49 And it has a lot of the logic that 2 to 3 has for fixing your code for Python 3.

51:54 But it also has a number of other features, basically, for catching common things that are going to cause issues with portability.

52:02 So if you install Futurize and you run it over your code base, it was going to give you a bunch of changes that it wants to make.

52:11 Those would be in the form of a patch file.

52:13 So it's going to say, I'm going to replace this line with that line.

52:16 The biggest issue with those tools, I think, is that the code that it creates can be pretty messy.

52:20 You know, we talked about dictionaries earlier and how keys, values, and items now return views instead of lists in Python 3.

52:28 So the way those tools will try and get around that is that whenever you call, you know, dot items in a dictionary, it will wrap that in a list constructor.

52:37 So the tool will add that for you.

52:39 And then if you do something in the same line to use another iterator, it will do it again.

52:45 So the code that it can create sometimes can be, you know, eye-wateringly ugly and actually not what you intended.

52:51 Yeah, it could actually change the performance adversely as well.

52:55 Absolutely.

52:55 And there's a reason that they moved to views and there was a reason that people would typically use iter items in Python 2.

53:01 So I kind of warn against that.

53:03 All of these tools, 2 to 3, Futurize and Modernize, you can toggle which fixes, which is the name for the logic that changes the code you want to use.

53:14 So what I recommend is that when you install them, you run in Futurize-L and it will give you a list of all the fixes you can use.

53:22 And then test out different ones, see which ones you're happy with, and then explicitly turn them on or explicitly disable the ones that you definitely don't want.

53:31 You know, I mentioned the dictionary one as an example.

53:33 I'd be really cautious of that because it can introduce, you know, some pretty ugly code and also impact the performance.

53:39 There are some others as well in terms of the way it handles strings and tuples, which can introduce a bit of mess.

53:46 So the tools, I think, are good for starting off by fixing some of the really mundane replacements, such as print statements.

53:54 Yeah, like if you could just have that happen, that would be nice, right?

53:58 It's not going to completely move your application for you, so don't expect it to do that.

54:02 Yeah, but it can do some of the low-level work.

54:04 I also brought up 6, the package SIX.

54:08 That's not a tool that you'd run that changes stuff, but it's like a package you can use to help create compatible code, right?

54:13 Some of the big changes in Python 3 are on the core types.

54:16 And let's say, for example, you've got some code which says, is this variable that I've just received in my function a string?

54:26 So in Python 2, it could be an str type or it could be a unicode type.

54:31 Both of those inherit from something called base string.

54:33 So you can say, if is instance, you know, my variable, base string.

54:38 So that doesn't work in Python 3 because base string is no more.

54:42 And the string type and the bytes type are very, very distinct.

54:47 So what SIX can do is it basically just introduces a redirect.

54:51 So if you want to check if something is a string, you can say if this variable is instance of SIX.string types or SIX.numeric types.

55:01 And then it's a really simple module.

55:04 It's only about 600 lines of code, I think.

55:06 And I recommend that people actually read the source code.

55:09 So it's not magic.

55:10 It's actually pretty straightforward.

55:11 It's just a series of if statements that says, if Python 3 do that, if Python 2 do this.

55:15 But it's pretty handy for the thing you really want to avoid when you're doing portability is having lots of, if it's Python 3, go and run this piece of code.

55:24 If it's Python 2, go and run this piece of code.

55:26 Because in the long term, you know, that's going to be really hard to maintain.

55:30 It's hard to read.

55:31 It's hard to follow.

55:32 And it's going to introduce all these weird edge cases in your testing as well.

55:35 Yeah, you definitely want to stay away from that if possible.

55:38 Very cool.

55:38 All right.

55:39 So let's sort of wrap this up a little bit and talk about maybe what do you do after you've migrated?

55:46 Now what?

55:47 After you've migrated, if you're doing the portability approach, then you've got an application that works on 2 and 3.

55:52 And then what I recommend you do is to basically start dropping 2 support.

55:56 So the ways that you can do that is to remove some of the portability code that you have.

56:01 And then you can, you know, if you're using Python 3 as your execution.

56:05 So in production, basically you're calling Python 3, not Python 2.

56:09 Then you can start to use some of the new features.

56:12 So you can actually start to implement async and await and data classes and f-strings.

56:17 Yeah.

56:18 F-strings are great.

56:18 Yeah, we haven't even talked about those.

56:20 And those are really, really awesome, both in from a performance and a syntax perspective.

56:25 So one thing that I wanted to throw in here at the end is I can see it's a challenge.

56:29 It's like if you're writing some code that's going to go run on a server, really the one that comes to mind is macOS because they ship with Python 2, but they don't ship with Python 3, right?

56:39 So if you're writing some code, you could say, well, I'm going to write in Python 2 and just it'll work on the Mac with nothing.

56:44 But if I write in Python 3, they've got to install Python.

56:47 There's all these challenges.

56:48 So you can do things like use PyInstaller and package up an independent version of your app, right?

56:54 Sort of make it easier to deploy anyway, but also get around some of those challenges, right?

56:59 Even in the Python 3 support, which version of Python 3 you get with different Linux distributions, as an example, is, you know, it's kind of all over the place.

57:07 So expecting that people have already installed Python 3 is then, okay, which version?

57:12 Is it 3.4, 3.5, or 3.6?

57:14 Yeah, exactly.

57:15 So yeah, that's definitely been a challenge, I think, because 2.7 has been out for so long.

57:19 You know, you can pretty much assume that people have already got 2.7 installed.

57:23 But yeah, if your package runs on 3, then you've got to figure that out as well.

57:27 Yeah, I've accidentally taken down, I think the training website, maybe the podcast website, by using F-strings in a little side utility.

57:35 It's not actually run by the website, but the web framework scans all the files to look for like views and stuff.

57:42 And it hit a F-string section, and it couldn't start the web app.

57:45 And I'm like, oh, really?

57:46 This is what takes it down?

57:48 Because it was 3.5, not 3.6 on Ubuntu or something silly like this.

57:51 So yeah, I guess it's still a modern challenge to some degree, right?

57:54 There's still one thing I think which is missing at the moment.

57:57 And that's dropping support for Python 2 in a nice way for packages that you push to PyPI.

58:03 So, you know, I've got a number of packages that I maintain.

58:07 I'd love to drop Python 2 support.

58:09 But I want people who are still using Python 2 to end up installing the version which still supported it.

58:15 And I think something we're still missing is, you know, if you pip install some package, and you don't pin the version, and it detects that you're running Python 2, it says, oh, okay, you want the legacy version, not the new version.

58:26 And it installs the legacy version.

58:28 That would be nice.

58:29 So you can all maintain.

58:29 Yeah, I think that would be really helpful, you know, to help people start to drop Python 2 support.

58:34 Because at the moment, if you know that 70% of your downloads are in Python 2, you still have to kind of maintain both.

58:41 Yeah, and you don't necessarily want to do like a super hard break and say, we're going to rename the package like request3 or boto3 or something like that.

58:49 Yeah, definitely not.

58:50 Don't want that.

58:51 All right.

58:51 So maybe final thoughts.

58:54 We have Python 3.7 coming out already just recently came out in a beta.

58:59 And we can start playing with things like data classes, right?

59:02 I recommend people go and download the beta and give Python 3.7 a go, try some of your existing code.

59:09 The reason it's in beta is because they're asking people in the community to actually go and test it and give them feedback.

59:14 So, you know, if there's something that's not working quite right, or you've somehow managed to find some bugs, yeah, then definitely raise those.

59:21 And now that Python's source code, CPython's source code anyway, is on GitHub, then you can, you know, you can work with pull requests and issues through a nice friendly interface.

59:31 Yeah, I think the move to GitHub is really actually, we're going to see benefits from that for a long time.

59:36 I think that's a big deal.

59:37 Yeah, definitely a huge deal.

59:38 Yeah, definitely.

59:39 All right, Anthony, this is really, really insightful.

59:41 Thank you for sharing everything you put together around this course.

59:44 People should check it out over on Plutalsight, right?

59:46 Will you give us a link to it?

59:47 Yeah, definitely.

59:48 I'll give you that.

59:48 All right, through that in the show notes.

59:49 Awesome.

59:50 You're going to be at PyCon in the US, 2018, Cleveland?

59:53 Yes, I guess I'm flying all the way from Oz over to Cleveland.

59:57 Halfway around the world to come hang out with all our Python friends, right?

01:00:01 Yeah, it's going to be like a remake of planes, trains, and automobiles.

01:00:04 So I'm quite looking forward to it.

01:00:06 Yeah, very cool.

01:00:06 I'm looking forward to that whole event as well.

01:00:08 And I don't think it's sold out yet.

01:00:09 So people should definitely get their tickets because it's going to.

01:00:12 All right.

01:00:13 Let's do the final two questions before you get out of here.

01:00:16 If you're going to write some Python code, what editor do you use?

01:00:18 At the moment, I'm using Visual Studio Code.

01:00:20 It's kind of lightweight enough not to take 10 minutes when I want to open it.

01:00:25 And it's got some more advanced features.

01:00:28 Like it highlights all of my mistakes for me, which is nice.

01:00:31 Yeah, very cool.

01:00:31 It's definitely pretty cool with the Python plugin, which is now part of like the official

01:00:35 team, right?

01:00:37 They hired Don, who was working on it.

01:00:38 That's also a cool story.

01:00:39 So we talked a lot about PyPI packages.

01:00:42 I've got one that is notable you want to recommend to people that they maybe don't know.

01:00:46 Yeah, sure.

01:00:47 I'm going to cheat since this is one of the ones that I maintain.

01:00:51 But if you're using documentation written in restructured text and you would love to use

01:00:58 Read the Docs, but the documentation that you want to publish is proprietary to your company

01:01:02 and you have Atlassian Confluence as a wiki, then install Sphinx-Confluence

01:01:08 Builder.

01:01:08 And it will actually publish your Sphinx output to Confluence for you.

01:01:12 Wow.

01:01:12 Okay.

01:01:13 Very cool.

01:01:13 All right.

01:01:14 So final call to action.

01:01:15 People out there with Python 2 code, what do they do?

01:01:18 So people out there with Python 2 code, start to run some linters, start to run some of the

01:01:23 automated tools to see what you need to change first.

01:01:25 Then introduce your test suite to Python 3.

01:01:29 See what things you need to change and just keep working through the application.

01:01:32 Follow the best practices.

01:01:34 And of course, watch the course.

01:01:36 Right on.

01:01:37 All right.

01:01:38 Well, it's really great to catch up with you.

01:01:39 And thanks for being on the show again.

01:01:40 Yeah.

01:01:41 Thanks very much, Michael.

01:01:41 Yeah.

01:01:42 Bye.

01:01:42 This has been another episode of Talk Python to Me.

01:01:47 Today's guest was Anthony Shaw.

01:01:49 And this episode has been brought to you by Linode and Rollbar.

01:01:52 Linode is bulletproof hosting for whatever you're building with Python.

01:01:56 Get four months free at talkpython.fm/Linode.

01:02:00 That's L-I-N-O-D-E.

01:02:02 Rollbar takes the pain out of errors.

01:02:05 They give you the context and insight you need to quickly locate and fix errors that might

01:02:10 have gone unnoticed until your users complain, of course.

01:02:13 As Talk Python to Me listeners, track a ridiculous number of errors for free at rollbar.com slash

01:02:19 talkpythontome.

01:02:20 Are you or a colleague trying to learn Python?

01:02:22 Have you tried books and videos that just left you bored by covering topics point by point?

01:02:27 Well, check out my online course, Python Jumpstart by building 10 apps at talkpython.fm/course

01:02:33 to experience a more engaging way to learn Python.

01:02:36 And if you're looking for something a little more advanced, try my Write Pythonic code course

01:02:40 at talkpython.fm/pythonic.

01:02:43 Be sure to subscribe to the show.

01:02:46 Open your favorite podcatcher and search for Python.

01:02:48 We should be right at the top.

01:02:49 You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct

01:02:55 RSS feed at /rss on talkpython.fm.

01:02:59 This is your host, Michael Kennedy.

01:03:00 Thanks so much for listening.

01:03:02 I really appreciate it.

01:03:03 Now get out there and write some Python code.

01:03:05 I'll see you next time.

01:03:25 Thank you.