#185: Creating a Python 3 Culture at Facebook Transcript

00:00 Do you or your team maintain a large Python 2 code base?

00:02 Would you like to move to Python 3, but there's just too much in place keeping you on legacy Python?

00:08 Then you will definitely enjoy the story from Jason Fried.

00:11 He created a grassroots campaign to move Facebook's massive Python 2 code base over to Python 3.

00:17 And he made Python 3 part of Facebook's culture today.

00:21 There are lessons here for every listener.

00:23 This is Talk Python to Me, episode 185, recorded November 10th, 2018.

00:28 Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.

00:48 This is your host, Michael Kennedy.

00:50 Follow me on Twitter where I'm @mkennedy.

00:52 Keep up with the show and listen to past episodes at talkpython.fm.

00:56 And follow the show on Twitter via at Talk Python.

00:59 This episode is sponsored by Linode and Rollbar.

01:02 Please check out what they're offering during their segments.

01:04 It really helps support the show.

01:05 Jason, welcome to Talk Python.

01:07 Thank you for having me.

01:08 It's great to have you here.

01:09 We've been working on this for a while, haven't we?

01:11 It's been a long time.

01:12 It's one of the more drawn out, sort of getting a guest on the show.

01:17 But I think people are going to be really happy about it.

01:19 I think it's a great story.

01:21 And the timing is perfect around the move to Python 3.

01:24 Yeah.

01:25 Yeah, it's going to be super fun.

01:26 So we're going to talk about how you've done this at Facebook.

01:29 And you didn't just start.

01:30 It's a long journey.

01:31 But before we get into all that, let's start with a little bit of your background.

01:35 How do you get into programming in Python?

01:36 I didn't have anything to do with Python until I moved to Facebook.

01:40 So before that, I was a Perl programmer.

01:42 I'm kind of ashamed now.

01:44 But I used to love Perl back in the day.

01:46 And then that was not socially acceptable when I moved to Facebook in 2011.

01:50 So I had to learn Python.

01:54 And surprisingly enough, I picked it up very easily.

01:56 And because I'm a little obsessive, so I dug into it enough that I started helping people.

02:02 And that kind of started...

02:04 I guess I thought of myself as an expert.

02:06 But even today, you always get better and better as you get.

02:11 And you look back at some of your old stuff.

02:12 And you're like, yeah, I used to write code like that.

02:15 It doesn't matter how much of an expert you are.

02:17 There's still plenty you don't know.

02:19 And there's still stuff that you look back.

02:21 You're like, whoa, I did that.

02:22 That was bad.

02:22 Yeah, you're like, don't look at that old code, please.

02:25 It's like my skeleton, so my closet.

02:27 Exactly.

02:28 Well, it shows you're making progress, right?

02:31 Yeah.

02:32 I do think it's highly interesting that you were one of the driving forces for this entire

02:39 sort of transformation at Facebook.

02:42 And yet, you walked in the door going, Python?

02:45 Is that the one with the space?

02:47 Or is that a different language?

02:48 Yeah, I played with it before.

02:50 But I didn't really know how to write it.

02:52 You didn't come in as like...

02:53 It's not like Guido von Rassum came in.

02:55 It's like, all right, we're going to change things.

02:56 We're going to fix it.

02:57 Like, basically, more or less a beginner, a newcomer, was the driving force.

03:02 And I think that's an interesting part of the story.

03:03 Yeah, I don't think it's like...

03:05 So, I guess I came in like the end.

03:07 It was October of 2011 on Halloween.

03:10 So, I really didn't get involved in the whole community of Python at Facebook until 2012.

03:16 And we have internal Facebook.

03:18 And there's like groups and stuff.

03:20 So, there was a Python at Facebook group.

03:23 And there was a lot of people learning Python.

03:26 And because I picked it up, I guess, quicker than they did, I was able to help them out.

03:30 And because I was helping them out, people just assumed I knew what I was talking about.

03:36 That's interesting how that happens, right?

03:37 Yeah.

03:38 They don't know you that well, right?

03:40 They're like, oh, this guy's telling me stuff.

03:41 So, he must know it.

03:42 And because I'm like, you know, I think I know what I'm talking about.

03:45 I go and fix things.

03:46 And I start fixing enough stuff that I start touching stuff all over the code base and different projects that I'm not involved in.

03:54 Because I have to touch like either one thing and I don't like the way it looks.

03:58 So, I'm going to go fix it.

03:59 And, you know, you somehow turn into before you know what people think of you as the subject matter expert.

04:07 Yeah.

04:08 That's really interesting.

04:09 And I guess in some sense, you sort of fulfill that prophecy by doing it enough until you know.

04:16 But there is that weird middle ground where people see you that way and you don't really see yourself that way.

04:20 But you kind of are.

04:21 Yeah.

04:21 Yeah.

04:22 Yeah.

04:22 Interesting.

04:23 So, what do you do day to day these days at Facebook?

04:26 So, I'm on a team we lovingly call the Ministry of the Silly Walks.

04:31 Initially, it's called the PE Python Foundation.

04:34 But it's an internal team.

04:37 And we are taking care of Python as a runtime at Facebook.

04:41 So, it means ensuring that we update it in a timely manner that we're going to have 3.7 something released in the next couple weeks.

04:48 That we're going to look at those shared libraries that have existed for 10 years.

04:53 And nobody like to make sure that we're doing best practices instead of, oh, yeah, well, Ted wrote this in 2008.

05:01 So, it's still good, right?

05:03 And you're like, no.

05:03 Yeah.

05:04 Maybe we shouldn't be rewriting our own thread pools everywhere.

05:07 Maybe we should just use concurrent futures or stuff like that.

05:10 Yeah.

05:11 Yeah, exactly.

05:12 Like, that's built in.

05:13 Let's not reinvent the world.

05:14 That's quite interesting.

05:15 I'm sure there's some old stuff like that.

05:18 So, I guess one question to the extent that you're allowed to answer it.

05:22 I don't want you to talk about anything you're not allowed to.

05:25 But when I think of Facebook, I think of PHP.

05:28 What's the story with Python at Facebook versus, I know Instagram joined, but there was Python at Facebook before Instagram was part of Facebook.

05:36 The software engineers that run the front end, it's all PHP.

05:40 But in the back end, people who are like SREs and these system administrators and ops people, they love Python.

05:47 Or they did back then.

05:48 So, they wrote all these scripts in Python.

05:50 They were all these things to, like, push the front end in Python.

05:53 So, you end up having this infrastructure to keep the site running all written in Python.

05:57 We've talked about this all the time.

05:59 So, like, if you have, if your service throws an alarm of some kind, there's another service that reads that alarm and starts running these remediations people have written in code.

06:09 Well, that code is Python.

06:11 And it's all written, the service to do it is Python.

06:15 And the service that distributes that is Python.

06:18 It's like, basically, it's we filled in the glue was with Python in the back end.

06:23 Where all the cracks were, it's been filled with Python.

06:25 Exactly.

06:25 Interesting.

06:26 Okay.

06:27 How interesting.

06:27 So, I guess that's probably a good segue to your main topic.

06:32 And because one of the things you started creating in Python 3 there was this infrastructure stuff.

06:40 So, maybe let's get started by, you know, I just wanted to ask you, I have my own strong opinions about this.

06:46 But do you believe that there's any debate about whether you should use Python 2 or Python 3 these days?

06:51 Or is that, I mean, several years ago, that was a solid debate that was raging.

06:56 Do you think it still is?

06:56 I don't think it's a debate anymore.

06:58 Like, there's a site, I don't happen to know, but you could probably find it, that it's the scientific Python packages.

07:06 And it's like them signing off to drop support for Python 2.

07:10 Yes, that was, it's like called the Python 3 statement.

07:13 Yeah, it's crazy.

07:15 It's done by Matthias Ossier or something like that.

07:19 Yeah.

07:19 And there's tons of these projects.

07:20 I also don't know the name of the URL.

07:21 Sorry.

07:22 Yeah.

07:22 But there's tons of these projects that said that, oh, people just assumed they would always be Python 2.

07:26 And they're like, no, we're going to get rid of our Python 2 support.

07:29 Yeah.

07:29 I think it really changed when, I think when the Python death clock went up.

07:35 Also, I don't happen to know.

07:38 But Guido was like, look, let's just make it January, January 1st of 2020.

07:44 Yeah.

07:44 Let's not.

07:45 Yeah.

07:45 I think it was on the mailing list.

07:46 He said, let's don't miss words about when this might be.

07:49 We're going to make it January 1st, 2020.

07:51 Not, you know.

07:52 Yeah.

07:53 I think the only other really good option.

07:54 Yes, exactly.

07:55 The other possible one could have been PyCon.

07:57 So it could have been like a party like right then.

07:59 But I feel like the sooner the better, honestly.

08:02 Yeah.

08:02 It's like, I know we don't have this debate at Facebook anymore.

08:05 We've won that debate.

08:07 I do a new hire training class.

08:09 And it's like, well, if you're already on the fence of whether you're going to use 2 or 3, you're a little late to the debate here at Facebook.

08:15 We won that debate in 2014.

08:18 Exactly.

08:20 You need a time machine to beat that one at this time.

08:24 Yeah.

08:24 And there had been a lot of conversations around, well, this library doesn't quite support Python 3, so you can't quite move yet.

08:32 And all I've seen in the last year and a half is here's one more amazing library that's Python 3.5 or 3.6 because it's using certain types of type annotations and async and await and so on.

08:42 It really feels like the argument and the evidence has shifted that you're being left behind with all these cool libraries, not stuck because of these cool libraries.

08:52 So the URL that you're looking for is python3statement.org.

08:56 That's a good one about the scientific ones.

08:59 Okay.

08:59 I agree with you.

09:01 I think that's great.

09:02 I remember, I think this was you.

09:04 I saw this around on Twitter to the effect of when Python 3.6 shipped, you said within two days, Facebook was running Python 3.6 in production.

09:13 Was that you?

09:15 Am I remembering that right?

09:16 It may have been Lukash.

09:18 It may have been Lukash, yeah.

09:20 I may have said that, but we were very aggressive on 3.6.

09:24 We've slowed down a bit because it's gotten more complicated, but we're going to be 3.7.

09:30 And I'm trying to push internally that we get these things released quicker than the external release schedule so that we don't build up that layer of, oh, people have snuck in internal patches or something.

09:43 One of the things you talked about that I thought was interesting is basically people get stuck on the idea of this is going to be an impossible problem to solve and we just can't do the switch.

09:55 But once you've made the switch from 2 to 3 and then 3, 4 to 3, 5, like this starts to become a habit.

10:02 You're like, oh, yeah, we just do that when a new one comes out.

10:04 Is it sort of feeling that way now?

10:06 Yeah, we've gotten to the point because we don't support multiple versions of 3 at Facebook and we never supported multiple versions of 2.

10:12 You just kind of you got moved along whatever the slow pace that the compilers on the team would push things out.

10:20 But for 3, it's just if we break you going to 3.7, it means you need to fix your code.

10:26 It's your problem, right?

10:28 Yeah, it's your problem.

10:28 We're going to let you, we're not going to do it by purpose.

10:31 Like on purpose, we're not going to like, we'll let you know, we'll try to fix it.

10:35 But if we can't, we're going to involve you because we don't support multiple versions of 3.

10:39 We're all going to be pushing forward.

10:40 You know, we don't want to be, we don't ever want to be stuck in a situation like we were with 2.

10:45 Right.

10:45 I think it's a really interesting lesson.

10:47 You were like, well, we were badly stuck.

10:48 And so now we're going to be very aggressive and never getting stuck again.

10:52 So your original talk, so you've given two presentations at various PyCons about this, one at PyOhio 2016 and one at the latest PyCon US in 2018.

11:02 Your original one was entitled Rules for Radicals.

11:07 Why the radicals identifier?

11:08 What was radical about all this?

11:10 You know, when you're trying to think of a topic, the name of your talk, you try to want to get something that will make the conference team get it.

11:18 And then you have to kind of get it approved internally.

11:20 And I think I had three different topics.

11:24 I think one of them was, we used to call it like lies.

11:27 And there's basically about lying to people to get for their own good.

11:32 But like, yeah, Tom's like, you can't say that.

11:34 I was like, okay, well, how about Rules for Radicals?

11:38 Because it's like that.

11:39 I've never read the book.

11:40 I've read Epsirps from a little ago.

11:43 But it's like, it's all about making change.

11:47 So I was like, what?

11:47 It's kind of perfect for my talk.

11:49 My talk's kind of about like driving change from a position of no power.

11:52 Right.

11:53 I think that the no power aspect is one of the super interesting aspects.

11:57 So many companies and people tell me that like at our company, we can't, we're still on Python 2 because no one has decided.

12:03 And one of the really interesting things is the Facebook engineering culture and this aspect of sort of earned authority, I guess.

12:14 Like you don't really have authority, but you start answering questions and helping people.

12:17 And you're like, oh, that person knows.

12:18 So I'll listen to them.

12:19 So maybe talk a little bit about the culture and how that worked there or still does.

12:25 Yeah.

12:26 It's a very, it's a great place to work.

12:28 So if you're on the fence about working for Facebook, just work for Facebook.

12:32 It's great.

12:32 But the thing is, it's like it's engineer led.

12:36 So a manager or someone in leadership never comes and says, oh, we're going to use this technology.

12:40 We're going to use Twisted or we're going to use Django.

12:43 No, it's usually, it's engineers say, you know what?

12:46 If we use this module and they convince their engineers next to them that cross the hall down the street in Dublin, Ireland, and they all start using this thing, we start using it.

12:56 And it's like you have to build up consensus by having a good idea because nobody has the like dictatorial thing.

13:06 Like we're going to do it this way.

13:07 Right.

13:07 There's not an on high statement that we're going to be using this technology or that technology.

13:13 It's just like, all right, you all are smart engineers.

13:15 These are the problems.

13:17 These are our goals.

13:18 Make it happen.

13:19 Exactly.

13:19 It's like this is, and then I go to these engineers and be like, look, I can have an argument.

13:24 This is what I think we need to do.

13:25 And if, and you're like, we should, you know, we should go to Python three because it is, it is looking this way.

13:32 It's going to be the future.

13:33 And at the time it was very sketchy.

13:35 Like it may not have been the future, but now we're no, we've, we clearly made the right decision.

13:42 It's like, once we like, look, do we want to be stuck in the old age or do we want to be able to keep moving forward?

13:49 Cause it's like, if we don't make a decision now, we need to like prepare for the other case.

13:54 Right.

13:54 And by the time that argument gets settled, it could be really painful to turn a big ship of technology like Facebook.

14:02 Right.

14:03 Yeah.

14:03 So we needed to get started early.

14:07 This portion of talk Python to me is brought to you by Linode.

14:10 Are you looking for bulletproof hosting?

14:12 That's fast, simple, and incredibly affordable.

14:14 Look past that bookstore and check out Linode at talkpython.fm/Linode.

14:19 That's L I N O D E.

14:21 Plans start at just $5 a month for a dedicated server with a gig of Ram.

14:25 They have 10 data centers across the globe.

14:28 So no matter where you are, there's a data center near you.

14:30 Whether you want to run your Python web app, host a private Git server or file server, you'll get native SSDs on all the machines, a newly upgraded 200 gigabit network, 24 seven friendly support, even on holidays and a seven day money back guarantee.

14:46 Do you need a little help with your infrastructure?

14:48 They even offer professional services to help you get started with architecture, migrations and more.

14:54 Get a dedicated server for free for the next four months.

14:57 Just visit talkpython.fm/Linode.

15:02 What are some of the things that you felt like came out in Python 3 that subsequently came out and you're like, oh, this is one of the things that were really important to us that we didn't know, but we knew things like this would be available if we did this.

15:17 Well, you know, it's like, I think early on, because like when we were wanting to go to Python 3, it was still Python 3.3.

15:25 It was still, most people don't consider Python 3 worth anything at all until like 3.4 or 5 came out.

15:31 And some would probably say 3.6.

15:33 But I think I was even on the fence to a certain point.

15:38 I was very much in the two only school.

15:40 But I went, I was really in, because, you know, we all know Python, we have this GIL.

15:45 We, unless we're doing IO bound work, we're really handicapped in the single thread or multi-thread for CPU bound tasks.

15:53 And I was, being a new person in Python, I wanted to solve this solution.

15:57 So it was like async.

15:58 Async's the solution.

15:59 And at the time, I was a very big G event person.

16:01 You know, I looked a little bit about Twisted, but Twisted, I was still new to the language, and it was just too confusing for me.

16:07 Right.

16:08 It is a little different, yeah.

16:09 But I went to a talk by Guido when he talked about Tulip, which is what we call asyncio, or what was called asyncio now.

16:15 And I just like had this epiphany, like all this stuff, because like I'm writing, I'm writing libraries for people to use internally.

16:23 And if they have not, if they have decided to use Twisted, they couldn't use my library if I wrote it in G event.

16:28 Or if I wrote my library in Twisted, they couldn't use it in G event.

16:31 Or they had to like kind of make a religious decision about which framework they were going to use, because they were very opinionated.

16:38 Right.

16:38 So all of Python's async capabilities was like these random grassroots projects.

16:43 Yeah.

16:43 And none of them had anything to do with each other.

16:45 None of them talked to each other.

16:46 Right.

16:47 Exactly.

16:47 And I had this, it was like, wait, if this is in the standard lib, and people can use it, and they will, because it's easy enough to use, that means that we could all standardize around it.

16:57 And I think the day after, like Tornado, which has got an async framework around it for doing these websites, it posted abilities to tie into the event loop.

17:08 And then it was like, well, if Twisted did it, and now we have Twisted, you can run your Twisted application or run async.io on a Twisted event loop and vice versa.

17:16 So you can like, we're in this magic world where I, as the library writer, don't have to make the religious decision for the application writer, that they can decide to be like natively twisted.

17:27 Right.

17:28 Because that's the wrong direction.

17:30 Right.

17:30 And I was like, we need to go to Python 3.

17:33 We have to have async.io.

17:35 You know, because I knew async was the future, but there wasn't a clear path in Python 2.

17:40 But in Python 3, there was a path.

17:44 Yeah, and I feel like the async story in Python is getting a lot better.

17:49 I mean, you talked about async.io, and then we have like a cleaner API with async and await.

17:53 Yeah.

17:54 We have a bunch of these cool web frameworks coming along based on things like UV loop, which, you know, replace the async.io event loop and all sorts of cool stuff happening there.

18:02 Yeah, there's like even reimagining of async.io using async await.

18:05 There's two competing frameworks for async.io.

18:09 And someone has already written compatibility drivers so you can basically patch them to each other.

18:15 So if you have a library written in the email, you can use them.

18:17 And I was like, that is the future that I saw back in 2013.

18:21 Right.

18:22 And I think it's coming to play.

18:26 And that was what I really, I was like, oh, we have to do this because, you know, I'm tired of making these janky geovint libraries nobody can use.

18:34 I want to make so that people can actually start working with.

18:36 Yeah.

18:37 And I can definitely see that at Facebook.

18:38 Like you're building these libraries and you want people to adopt them.

18:41 Well, you know, you don't want to have that be a real small group.

18:44 You had an interesting.

18:45 So I think some of those additional reimaginings are probably Trio, Curio by David Beasley.

18:51 Trio, Curio.

18:52 Someone has already written compatibility to make them talk to async.io applications.

18:56 And I don't know if they can talk to each other, but I've seen one that talks to either or.

19:00 Right.

19:00 You can probably chain it through like.

19:01 Yeah.

19:02 Go to async.io to Curio or whatever.

19:05 Right.

19:05 Yeah.

19:06 All right.

19:06 So I think one of the really interesting parts of this journey is to sort of go through the years because kind of like we started off, like the view of Python 3 now is really different than the view of it five years ago.

19:20 Exactly.

19:20 There's absolutely advocates and people pulling for it, but it was a different world.

19:25 So want to take us back to 2013 and sort of start there?

19:29 Yeah.

19:30 I think like it was like back in 2013, there was a big movement of the let's just stay in Python 2 forever.

19:39 Like I guess that I think there was this undercurrent that the core team of Python, of CPython had lost their minds.

19:46 And then Python 3 was ridiculous.

19:48 And it was a very, I think I may have, I was new to the language.

19:52 So I just like, I just picked whatever everybody else was using at Facebook, which was 2.6.

19:57 Well, and you look around, like on Stack Overflow, there were people saying never move to 3.

20:01 Like that's just a waste.

20:02 I mean, there was a lot of that sentiment happening.

20:05 People were like, oh, don't bother learning Python 3.

20:07 And you're just like, you know, and then when I learned it, it was just like, why did we, you know, it's not that bad.

20:14 Honestly, I feel like the differences are so small.

20:17 Sort of modern 2.7 and Python 3 is like, well, you put parentheses on the print statement and like a few other handful of things that they're, none of them are like massive conceptually differences.

20:28 Right.

20:28 It was the libraries.

20:30 Yeah.

20:30 The libraries.

20:31 Cause like, especially cause in, in 2.6, like we had to change the way you did exception catching.

20:39 The change, I don't, I don't even remember what it was, but I remember we had to fix it because it was like, it got deprecated by the time 2.7 came out.

20:45 It no longer worked.

20:47 So it had to fix stuff, but it was like, there were so many old crusty things that had been running on like Python 2 for like 10 years.

20:54 So people were just like, no, I don't want to do it again.

20:58 You know, they were just like.

20:59 Well, and maybe the person who wrote that part is not even at the company anymore.

21:03 No one wants to touch it.

21:04 Exactly.

21:05 They don't want to, they don't want that thing to become their problem.

21:08 Right.

21:09 They just want to work on their stuff.

21:10 It's the same for a lot of third party, third party libraries that the person maintaining it just disappeared.

21:16 We don't know if he's dead or she's dead.

21:19 We just, they don't respond to emails or, you know, and their project is just kind of withered away.

21:25 Yeah.

21:26 People use it.

21:26 Retired.

21:27 They don't care about it.

21:28 Maybe they've moved on to Node.js or whatever.

21:31 Right.

21:32 But for whatever reason, they don't care.

21:33 And the crazy part, some of these modules are super popular.

21:36 Yeah.

21:37 Like the MySQL driver.

21:38 Yeah.

21:38 That one, that one, that one became basically unattended.

21:42 Yeah.

21:42 Wow.

21:42 How interesting.

21:44 I can certainly see like maybe some burnout as well.

21:47 Right.

21:47 You're just like, I just got to take a break.

21:50 If no one else is going to help, this is just going to sit here.

21:52 Cause you know, you gotta, gotta live your life.

21:55 Right.

21:56 Your goal is not to just provide software to the world.

21:58 You're a human.

21:59 Right.

21:59 Exactly.

21:59 All right.

21:59 So you said back in 2013, you're kind of like, I would love for this to be a thing to use Python 3, but I don't think it's going to happen.

22:07 Right.

22:08 You actually had access to Python 3, theoretically, as an option.

22:12 But in practice, it was all messed up, right?

22:14 Yeah, like the binaries were there for Python 3.

22:17 You could run them and get the interpreter.

22:20 But we have kind of a complicated environment.

22:23 We have build systems that go and build C++ dependencies for compiled modules.

22:29 And they kind of make PEXs, which is basically, it's like an archive of all your code that

22:36 you can distribute to a machine and just kind of run it like a binary, but for Python.

22:39 Right.

22:40 It's like a hacked zip file that's executable.

22:43 Exactly.

22:43 And it bundles up all your definencies.

22:45 But it does not bundle up the Python runtime, right?

22:48 It's effectively like a virtual environment, kind of.

22:50 Yeah, it's a deployable virtual environment.

22:52 And the system that we had that built that didn't work.

22:55 So it would call out to Python 2 to run, to basically import your code and do a .compile

23:01 using the compile module to get the PYC.

23:05 And then we would just pack up the PYCs to make this PEX thing.

23:07 And of course, it was like, you would give it Python 3 code and the PyTor compiler would

23:12 blow up on it.

23:13 And you're just like, great.

23:14 And then once you fix all that, it's like, you get this PEX and you run it and it just

23:18 seg faults.

23:19 Or not just seg fault.

23:20 It just stack traces like in the early guts where we're setting up the environment.

23:25 You know, you're like, oh, great.

23:27 It's deeply down broken.

23:30 This is not my problem, but it's not going to run, right?

23:32 It's like deeply broken.

23:33 And it's like, I could have just thrown my hands up at that point, but like, this is

23:37 never going to work.

23:38 And I may have for a week or so, but like, once you figure out that nobody's going to fix

23:43 it.

23:43 You do talk about that sort of be the change you want to see in the world.

23:47 And eventually you're just like, well, if no one else is going to do it, it's going to

23:49 have to be me, right?

23:50 And it turns out, yeah, like if you want something to change and you wait around for

23:54 someone to do it, it may never happen.

23:56 It likely will never happen.

23:57 Yeah.

23:57 Likely will never happen.

23:58 So I started fixing it and got like, I would get it to compile and then I would, I put up

24:04 a diff and have to like go chase down people to like, who knew kind of the system I was

24:08 messing with to do the building.

24:10 Like, Hey, could you review this code?

24:12 Be like, here's my unit test.

24:13 You know, I built some Python three stuff and it ran like, okay, let's land it.

24:18 You know, and then, then the thing that would happen is like the next week, somebody would

24:22 have committed some other Python two only code and you're just like, Oh God, this is never

24:26 ending.

24:26 Yeah.

24:27 Yeah.

24:27 So you talked a little bit about one of the challenges is you've got to convert the libraries

24:32 you depend upon that are internal Facebook libraries to be two and three compatible.

24:35 Yeah.

24:36 And then you also talked about like this regression problem, like constantly challenges you.

24:42 I guess that wasn't until the next year, but that was a problem, right?

24:46 Like you would, you would make your stuff work.

24:48 And then somebody would change a base library that wouldn't work.

24:51 So in 2014, like 13, like people didn't care about Python three.

24:55 So like, they knew that we've kind of, like, we kind of said that it's probably the future.

25:00 We should move to it.

25:01 Like, I think people all kind of had this sense that eventually we'll have to move.

25:04 Like, if you want to Python three support, they would be like, well, we don't have the

25:08 roadmap for that.

25:09 Like, we don't have the manpower to work on that right now.

25:12 So you just go and fix it.

25:13 Right.

25:13 We can't stop for six weeks and like rewrite or six months and rewrite this.

25:18 Right.

25:18 Well, most libraries, it wasn't, it was like, it was a handful of things you would fix.

25:21 Yeah.

25:22 And then just like, oh, they're handling, it's like bytes and string doing wrong, but

25:26 you would fix it.

25:27 And people would, no, people aren't running against Python three, so they just break you.

25:31 So I figured out, it's like, well, you know, that's what I think I really got.

25:34 I really accepted the, the concept of unit test driven development was that when I went

25:40 out and fixed these things, I immediately inject that, I immediately put a test to make sure

25:44 they stayed fixed.

25:45 Yeah.

25:46 I'm not going to fix this again.

25:48 And that, that really stemmed the bleeding because when the next, when the next person

25:54 came to write a code against this library and change something, they would immediately break

25:58 my unit test.

25:58 They're like, oh, well, let me fix this.

26:00 Right.

26:00 Yeah.

26:01 So that's one of the things that you talked about is basically, instead of you playing

26:06 whack-a-mole and constantly chasing these problems around, you, you sort of found a way to get

26:12 the build to work better.

26:14 And somehow you managed to get the build system to, to verify Python three behavior without really

26:22 the authority, just kind of like, yeah, this is for code quality.

26:26 We're going to do this as well.

26:27 And people are like, okay, great.

26:28 Yeah.

26:29 So we had like, everybody knows about flake eight.

26:32 It lints your code and it, you can run, like we already had a PEP 8 linter.

26:36 We actually had our, you know, we were kind of organized.

26:40 We actually made sure that all of our Python code kind of looked the same.

26:42 We would hit, pepe.

26:45 People may not, but argue, heck, how do I fix this?

26:48 But, to make the campaign happy, but some various teams would just run their own linters.

26:53 And I noticed myself, I would commit code and maybe like I had to fix something during code

26:58 review and I forgot to run the linter on it and it would make it into, into production.

27:03 And it would break a test immediately because it was like a syntax error or something silly.

27:08 And, you're like, oh, it's so stupid.

27:11 How did I miss this?

27:12 I was like, well, we're not running linters automatically, but there are linters rather than language like

27:17 PHP or P bus plot.

27:18 There's no space that these linters that ran.

27:20 I was like, well, like we're using fabricator, which is this open source code review tool.

27:25 And it had a thing to tie into, flake eight.

27:29 So I, I went ahead and set it up and I was like, well, if I can run flake eight under Python three and under Python two,

27:38 I would know if people, if I made like an obvious break to two or three.

27:43 And I did that and it was probably, it was really hell for a while for a lot of people.

27:48 Cause you were getting double the amount of flint warnings in your code review.

27:52 You'd get them for both two and three.

27:55 And I spent a lot of time to tell people it's like, okay, explain to people what's going on and why they shouldn't go secretly disable my linter.

28:03 And, because it was like, this will make our code better quality and people like, they were already running these linters.

28:10 They were like, oh yeah, this is great.

28:11 Can we actually make them to do?

28:13 Woo gosh, did that.

28:14 And that was, that didn't happen to like 2016.

28:17 I don't think.

28:17 But right.

28:19 And now you're almost to the point where you do it by turning off the version two one.

28:23 Well, no, like I think, well, gosh, basically if the messages were the same, he didn't print it.

28:27 They're like, he didn't print the version.

28:28 He said, pie flakes.

28:29 This is the message kind of thing.

28:31 Yeah.

28:32 So another thing that you talked about doing is using the dunder future module to at least stop writing only Python two code, right?

28:42 Like you imported the print function and division and so on.

28:45 But this was like one of the early, early things, even when there was not actually a path, right?

28:49 Yeah.

28:49 We had a linter, someone, he wanted to prolong the use of his two eight.

28:54 He wanted to keep writing two eight and hopefully it live longer than he thought.

29:00 Like maybe we switched to three and like the next year.

29:03 He didn't really know, but he wanted to, he wanted Python two to behave more like three.

29:07 And that was pretty wise actually.

29:09 And that got us very far at Facebook.

29:12 And he set up a linter to force everybody to do it.

29:14 And it caused a lot of pain at the time, but people just fixed.

29:17 Like if you make it like this, like a commit hook or something, people just try to make it happy.

29:22 And if they can't make it happy, then they complain.

29:24 That's pretty interesting.

29:25 Yeah.

29:26 There was no real message from on high to say we're using Python three.

29:30 But when the linter says, oh, this doesn't work right because of Python three.

29:33 Well, all right, we'll just fix that.

29:34 Yeah.

29:34 It's like, oh, why, why does my import not work?

29:37 Oh, well you're using, you can't use these relative imports without being explicitly relative.

29:41 And yeah, it took a while.

29:42 The Unicode literals thing probably ate up a lot of time, but we solved that back in like Python two.

29:47 And people internalized it.

29:49 It's like, if you wrote good Python two, you were using the future imports.

29:53 Right.

29:53 Yeah.

29:53 That's really cool.

29:54 Another thing you did back in 2012, sorry, 2014, when this was kind of a still just a little micro project was in the build system or just all the environments.

30:06 You made sure that six and two to three were sort of omnipresent.

30:09 What was the story of that?

30:11 Well, we had these linters in place and people were always like, well, how do I fix this thing?

30:15 That would spend a ton of, a lot of churn on the message board was like, okay, I understand this is a problem, but how do I fix it?

30:22 And it's still working two and three.

30:24 And it was like, well, you can do this extra code or, hey, there's this thing, this module called six or this module called future that you can import.

30:33 And it does.

30:33 So, and people like, well, okay, so I have to go add this to my build dependencies.

30:37 And you're like, or they would get mad.

30:39 It's like, oh, it's not there automatically.

30:41 So it's like to make it easier to like lower that barrier.

30:44 I just, I tripped the system into injecting it to everybody's build configs so that your pecs that when it gets produced would have six and the future module.

30:55 That's really cool.

30:56 So that was, they could just do this code and then they have to change anything about their system.

31:00 It was like, it was just, that was the way you could write it.

31:02 It was like, it was part of the standard lib.

31:04 That's great.

31:04 Maybe for folks who don't necessarily have battle scars from the Python two days, tell them what six is.

31:11 So six is like the first link, the first library for converting two to three.

31:17 It allows you to do things that are completely different from either language.

31:21 Like if you're setting a meta class on a class, it's different.

31:25 Like the syntax is different, Python two and three.

31:28 So six provides you a decorator you can call that you can wrap your class with and it'll, it'll set the meta class.

31:34 And six has a thing where it like, cause there are modules that moved between two and three.

31:39 And six has a thing where you can import from a certain move location.

31:43 And there was also this thing called Python.

31:45 It's called the future module.

31:47 From that, you can, it's, it's kind of like six, but it's a little different.

31:50 Sometimes it's, sometimes it's nicer code.

31:52 Sometimes it doesn't.

31:53 I use both.

31:54 So I don't have to force that decision on people.

31:56 You know, it's like, I just want to make people happy.

31:58 Give them all the tools.

31:59 All the tools.

32:00 You want a crescent wrench.

32:01 You want a socket wrench.

32:02 Whatever.

32:02 Just get it to run a load.

32:03 Before six, you would have to write these if statements where you would say like, if sys.version.

32:09 Or info, like, cause it's the main topic, like offset zero equals three.

32:15 You would do some code.

32:16 Like, this is a lot of garbage.

32:18 And six is like, oh, just say from six import pi three.

32:21 And you say, if pi three.

32:23 And you're like, oh, that's better.

32:24 You know, so.

32:25 Yeah, it's much better.

32:26 This portion of Talk Python to Me is brought to you by Rollbar.

32:32 Got a question for you.

32:33 Have you been outsourcing your bug discovery to your users?

32:36 Have you been making them send you bug reports?

32:39 You know, there's two problems with that.

32:40 You can't discover all the bugs this way.

32:42 And some users don't bother reporting bugs at all.

32:45 They just leave, sometimes forever.

32:47 The best software teams practice proactive error monitoring.

32:50 They detect all the errors in their production apps and services in real time and debug important errors in minutes or hours, sometimes before users even notice.

32:59 Teams from companies like Twilio, Instacart and CircleCI use Rollbar to do this.

33:04 With Rollbar, you get a real-time feed of all the errors so you know exactly what's broken in production.

33:10 And Rollbar automatically collects all the relevant data and metadata you need to debug the errors so you don't have to sift through logs.

33:17 If you're using Rollbar yet, they have a special offer for you and it's really awesome.

33:21 Sign up and install Rollbar at talkpython.fm/Rollbar.

33:26 And Rollbar will send you a $100 gift card to use at the Open Collective, where you can donate to any of the 900 plus projects listed under the Open Source Collective or to the Women Who Code organization.

33:38 Get notified of errors in real time and make a difference in open source.

33:41 Visit talkpython.fm/Rollbar today.

33:44 You had sort of summarized each year your thought about how likely this Python 3 world might be.

33:53 In 2013, you said, not going to work, not going to be a thing probably.

33:57 Yeah.

33:57 2014, you're like, highly unlikely.

34:00 So then one of the really interesting things that caught my attention about your talk was in 2015, you started getting involved with teaching Python at Facebook to new hires.

34:10 Yes.

34:11 Talk about that.

34:12 So, yeah, we had this class because we've already talked about we have this kind of complicated environment we build.

34:18 We have the PEC system, the fake zip file with the binary on top of it.

34:23 And we have all the C++ that's compiled in with you.

34:26 And we have a whole process for getting external libraries into it so it can be linked with those C++.

34:32 There's a lot.

34:34 And coming with just an open source knowledge of Python, you would be lost.

34:38 So we have this class that all boot campers, at least for touch engineering, can learn about Python and learn, hopefully get rid of all these roadblocks so that they can hit the ground and just produce some Python code that does some more work.

34:51 Here's what Python at Facebook looks like.

34:53 Here's how you work with our infrastructure from Python.

34:55 Stuff like that, right?

34:56 Because that's where they're going to learn that, right?

34:58 Yeah, we'd warn them about the linter.

35:00 They'd be like, oh, yeah, the linter's going to hit you.

35:02 And this is how you fix it.

35:03 Like, we talk about the four futures.

35:06 And that was already in there when I started teaching classes.

35:09 Like, what if we start teaching people that we expect them to write Python 3?

35:13 And secretly, that was my secret.

35:16 Because it wasn't really true.

35:18 But the new hires didn't know that.

35:21 Right.

35:21 And it was feasible.

35:23 Yeah.

35:24 Like, they could write Python 3 at this point.

35:25 You'd done enough work to get the infrastructure in place.

35:28 So it would work.

35:29 Yeah.

35:29 So it's like we say, oh, you're expected to write Python 3.

35:32 And if you have any issues, feel free to fix them yourself if you can.

35:37 Otherwise, reach out to us.

35:39 And it was like there was like a couple friends, friends of Python 3.

35:42 That was like a secret group that we were trying to push it forward at this point.

35:47 That you could reach out to us and we'll help you.

35:50 Or post in the message board.

35:51 And we'll help you with your problem.

35:54 That's awesome.

35:54 We were gathering like a wiki.

35:56 Yeah.

35:56 And to be clear, when you say if you run into a problem, just fix it.

35:59 That isn't with your code necessarily.

36:01 That's like you try to use one of the libraries.

36:02 It doesn't work in Python 3.

36:04 Well, you're expected to go kind of try to make that dependency that we have work in Python 3.

36:10 And just get them to sort of own the foundation of all the Python, right?

36:15 Yeah.

36:15 At Facebook, there's this saying.

36:17 It's like nothing is nobody else's problem.

36:19 Or it's not something along those lines.

36:22 But it's like there's this mentality that something's broken.

36:26 You don't just immediately go and find out, oh, well, who owns this code?

36:29 I'm going to go complain.

36:30 Like, no.

36:31 See if you can fix it.

36:32 Because if you fix it within like five minutes, you'll be ahead of your schedule if you had to go and like track down a person.

36:38 And like complain to them and get them to release something on their schedule.

36:42 You can just fix it.

36:43 Put up a little diff and keep working.

36:45 And if we taught people to just do that, like we kind of were tricking them to do work for us.

36:51 Yeah.

36:52 Yeah, there's a lot of lessons of sort of distributed responsibility, distributed work in here through the linters and through this education component.

37:00 And it more or less worked, right?

37:02 People started writing Python 3 code.

37:03 Yeah, they would join their teams after boot camp.

37:06 And they would say, hey, how come we're not using Python 3?

37:09 And the people there, they didn't have a good answer.

37:11 And the new people would start writing Python 3.

37:14 And so, like all these teams started coming out with like, hey, we're putting out Python 3 for our new project.

37:22 Because that's all the new people want to write.

37:24 I think that's great.

37:27 So, 2015, you dubbed the year of maybe.

37:30 Maybe it's going to work.

37:31 Starting to look good.

37:32 And then 2016, something pretty major happened with Instagram.

37:37 Instagram is one of or not the biggest Django deployment.

37:42 And they actually did a whole keynote about their move to Python 3.

37:47 But that was a big deal, right?

37:48 Yes, that was huge.

37:49 The thing was like, well, Instagram was Python 2 at the time.

37:53 And it was like, do we, does Instagram stay on Python 2?

37:57 And we were, it was like me and Lukash Longa were like, he really fought hard for us.

38:03 Like, no, you guys have to move to Python 3.

38:06 And if you don't move to Python 3, you should just give up the ghost and go to PHP.

38:10 And they're like, whoa, whoa, whoa.

38:12 They're like, just a second.

38:13 We can find it.

38:13 Actually, you know, it's at Facebook.

38:15 That's a good sound business argument to say.

38:17 It's like, you should probably use PHP.

38:19 We've invested a lot of money in that.

38:20 And it runs like the biggest website in the world.

38:22 Like, I think that makes sense.

38:24 And they're like, oh, let's just stay with Python.

38:29 You know, we don't want to like lose all of our developers.

38:32 Yeah, that's really interesting.

38:33 There was a keynote, I think, at the 2016, 2017, PyCon, about this.

38:39 And it was really, really just such a step-by-step practical way for large organizations to move.

38:47 Because it wasn't just Instagram upgraded to Python 3.

38:51 They also changed Django, major Django versions.

38:53 And they did all of that without branching in production.

38:58 Yeah.

38:58 Right?

38:59 Like, that is just incredible.

39:01 It talks to the power of tests as well.

39:03 Like, this is all unit test-driven, where they would take a test and they would make it work on Python 3.

39:10 Or like, change the code enough to where it would still pass on 2 and 3.

39:13 Right.

39:15 And I don't know how they did Django.

39:17 That was like, that's crazy.

39:20 That's another level.

39:21 But I think it also involves unit tests.

39:23 It must have, right?

39:24 But still, that's impressive.

39:26 Without branching in production.

39:28 Yeah.

39:28 That's crazy.

39:29 That's really super impressive.

39:31 And every time I hear someone say, oh, we can't move to Python 3.

39:35 Or it's totally unpractical.

39:36 Yeah.

39:37 We're too big to move to Python 3.

39:38 Yeah.

39:38 It's like, well, here's a roadmap and a pretty insane example.

39:44 You know, the rest of the stuff is not as visible what you're working on because it's infrastructure level.

39:49 But still, you know, also impressive.

39:51 Another thing that you said happened around 2016 is it became Python 3 by default.

39:55 What does that mean?

39:56 Whenever you would, like I talked about their build system, and you had a little script that you would tell it, oh, these are the third party dependencies I need.

40:02 These are the dependencies that someone at Facebook has written.

40:05 These are the C++ dependencies I need.

40:07 And it would, the build system, you would say build, and it would give you the little pecs that you could take to any system of Facebook and it would run your Python code.

40:14 That can fit if you just, like, copied what the wikis told you to do.

40:18 It would give you a Python 2 binary.

40:20 And we had been telling people in the class, a new hard class, that, oh, you should add this thing that says pi underscore version equals three.

40:28 And you'll get a Python 3.

40:29 And people are like, oh, why do I have to say that?

40:32 And we just say, oh, because two, two, I always do default.

40:35 It was like, there were so many people every day were saying, oh, we'll release this to Python 3.

40:40 That it was like, the only people that were still writing Python 2 code, they didn't have, they didn't really, they weren't like staunch Python 2 supporters.

40:47 They were just using the defaults.

40:49 So I had this like, well, what if we just switch the default, see if anybody cares?

40:53 And so I sent a message out like, hey, I'm going to switch the default to Python 3.

40:57 If you think this is a bad idea, say something.

40:59 And nobody said anything.

41:02 And people were all like, thumbs up, you know, I like this, this is good.

41:05 And so I did the switcheroo and I made everybody who's currently in Python 2 explicitly say that they're Python 2 and everybody's Python 3 get rid of their version line completely.

41:15 And it was just the default was the Python 3 version.

41:17 And nobody got broken.

41:20 There was no like major side outages.

41:23 It just, that was a very smart decision.

41:26 It was like, look, we were probably at like, probably less than 20%.

41:30 I think I have a graph somewhere in my talk where it was like, oh yeah, we were way less than 20%.

41:34 We were not even like 10, but it felt like we were really high because all the new stuff was coming out for Python 3.

41:41 Right.

41:42 And then this just, just flipped it, right?

41:44 All the people who were not setting things explicitly, they didn't care.

41:46 It just, it changed, changed the tide, right?

41:49 People who didn't care what version they're using.

41:50 They just wanted a version of Python that they could, you know, because as long as it wasn't C++, these guys were happy.

41:55 They just, you know, not having to, they didn't care and they weren't part of this fight.

42:00 So we just, whatever the default was, they're going to use it.

42:03 That's really excellent.

42:03 And a lot of the future work, right?

42:05 The import, the print function and stuff is kind of what made that work, right?

42:09 Oh yeah.

42:09 And like, yeah.

42:10 So like a lot of the existing modules of people who had already fixed all that future stuff,

42:14 some of them just imported fine into Python 3.

42:17 Right.

42:18 I mean, how often are you like doing crazy, like meta class stuff, right?

42:21 Like that's not that common.

42:22 More than we should is the case.

42:26 Everybody's like, I don't use meta class here.

42:28 Like maybe you shouldn't.

42:29 But yeah, it's like, I think in most of the major, like heavily used code paths had already

42:35 been cleaned up by like me or somebody else.

42:38 Cause if, if it's a, if it's a major use dependency, it's already been fixed for Python 3.

42:44 Cause by 2016, we already had a services in production, right?

42:48 Python 3.

42:49 Yeah.

42:49 That's great.

42:50 So one of the things I think is interesting, there's a quote that you talk about from one

42:55 of the engineers that sort of touches on this conception that Python 3 is slower.

43:00 And it says something to the effect of, we just did a very basic two to three on our service,

43:05 which means they ran the two to three program on the service to just automatically upgrade

43:08 it.

43:09 Everything just worked.

43:10 Now we're using half as much memory and it's 40% faster.

43:14 Yes, this is true.

43:15 That's a, that's a true quote.

43:16 I saw the, I think I may have stamped the diff as being approved and it was like simple.

43:21 Like it was just like changing the way they catch exceptions.

43:25 Right.

43:26 And, cause it's like, I think before it involved, I didn't remember at this point,

43:31 I've, I've, I've, I've, that's, that's just lost left on my head space, but it was like,

43:34 it was just basic things that you're like, Oh, that that's easy.

43:38 Like you didn't even need the tool to change it.

43:40 It was like, it wasn't, they didn't really have a lot of code that needed to be changed.

43:44 But, so it was like surprising.

43:46 There's like, Oh, that's all you changed.

43:48 And you, you're 40% faster.

43:50 Really?

43:51 Yeah.

43:51 Which speaks to the different runtime, right?

43:53 The different CPython.

43:54 Yeah.

43:55 And for a long time, Python three was a lot slower.

43:58 And I think still there might be some edge cases that are faster and Python two seven.

44:04 A lot of it probably has to do with strings, right?

44:06 Like the, the different behaviors around strings and bytes and all that stuff.

44:09 I think it's gotten faster and three six, like they, they, they really like pulled ahead

44:14 to use less memory and stuff with all the different fixes.

44:17 But I think that when that thing came out, people were still comparing.

44:21 It's like three, two and three, three.

44:23 And they just had, right.

44:24 That was their argument.

44:25 And they just hadn't actually updated their benchmarks to use new version.

44:30 Cause they'd already given up a little Python three.

44:31 And they're like, Oh, Python three is just too slow.

44:33 And then it's like, by the time three, five came out, I think that, or that might've

44:38 been out three, four.

44:39 But, it was like, it's kind of snuck up and be like, actually the core dev has been

44:44 working a lot to make this really fast.

44:46 And that work continues today, right?

44:48 Yes.

44:49 Yeah.

44:49 Python three seven bound function calls.

44:52 So function methods basically are 20% faster.

44:55 Oh, I can't wait.

44:56 In most cases.

44:57 Right.

44:57 Like, and that's just three, six to three, seven.

45:00 This is great.

45:00 So yeah, it's, it's all good.

45:02 I'm hoping for next week.

45:03 If, if Lucas hears this, I'm hoping for next week.

45:08 Okay.

45:08 So you had deemed 2016 Python three by default.

45:12 That was good.

45:13 And then 27 is sort of overtaking Python two.

45:16 And one of the things you highlighted then that happened was now Instagram is running on

45:21 Python three.

45:22 It has access to this amazing new features that were locked, you know, blocked off like

45:27 asyncio and in the typing static, you know, type hints and all that stuff.

45:31 Right.

45:31 Yeah.

45:32 It's like, you can say it's like, Oh, you can use typing in Python two, but it's like

45:35 the types in top two or three are semantically different.

45:39 Like things that you can do in two, like you may have, they may have different types or

45:43 they may have the same type, but they're behave differently.

45:45 So it's like your type formation may not be the same between versions.

45:48 So we told them to hold off to the winter three.

45:51 Cause it just made, we'd have to double the work, but yeah.

45:55 Right.

45:55 And you don't have it like in the function.

45:57 You've got it like in, in doc string type format, right?

46:00 Which is not nearly as nice.

46:02 Yeah.

46:02 It's not as nice.

46:03 And or whatever.

46:04 Right.

46:04 Yeah.

46:05 And people want it to be in line.

46:06 They want it because if it's a comment, it's gonna like, it's not always going to be correct.

46:11 So at least with the type annotations, like you would know, like it would, it's like, Oh,

46:16 I changed the name of the import.

46:18 It's all going to go fix all the types.

46:19 But if it was a comment, you wouldn't know that until you decided to actually run the type

46:23 checker in the early days before we even had a type checker.

46:26 It was like, we're just putting these types for humans to read.

46:28 Yeah, exactly.

46:29 So one of the things that you said around this time is that working in Python was fun

46:34 again, right?

46:35 People were super excited about all the new stuff and the new capabilities and it just kind of

46:40 revitalized people's joy, right?

46:41 Yeah.

46:42 It's like, like, I guess because of the whole two, three problem, like people who were, who

46:47 were like fans of Python were like, they were really negative.

46:51 They're like, Python's dead.

46:53 You know, I guess kind of like the pearl is dead.

46:55 It kind of in, in their, in your mind though, if, if you're in your mind, you're not switching

46:58 to Python three.

47:00 It is dead.

47:01 Yeah.

47:01 For them.

47:02 Well, people have stopped working on this for, since 2008, right?

47:06 Like it's, it's just nothing is happening anymore.

47:09 But once you're over that hurdle, all of a sudden like, Oh, look, this pool is warm.

47:14 It's fun here.

47:15 So like all this for a while, it's like, let's go to go.

47:18 Let's rewrite everything and go.

47:19 And I guess if we had decided to do that, we would still be working on it for, till the

47:23 next four years.

47:24 But, luckily it's like, people were like, Oh wait, there's all this good stuff.

47:30 All this cool stuff in Python three, like Instagram found out with ACCO.

47:34 It's like, this is good stuff.

47:36 And people would convert just to have that stuff.

47:39 Yeah.

47:39 And I think that's the main argument now is there was so much that is inaccessible to Python

47:44 two.

47:44 That is so exciting and interesting these days.

47:47 It's just crazy to not just do the simple work to, to get onto that side where you have

47:52 that as an option.

47:52 Yeah.

47:53 And what I would tell people in my bootcamp, I was like, if you're going to write a library

47:56 for Python users, don't write Python two support for one, it's going to waste your time.

48:01 Cause it's just, it's extra load for you.

48:03 You have to make sure it works in both languages.

48:05 You won't be able to use the new stuff.

48:07 And yeah, I'm not telling them this, but it means that those Python two users will have

48:11 to move to three to use your new good stuff.

48:13 Yeah, absolutely.

48:13 And there's some realistic, reasonable examples for that.

48:18 Like the Django team talked about how they, they were able to drop a ton of code.

48:22 And the Jupyter team is a NumPy.

48:25 I can't remember, but some stuff talked about in the Python statement that we talked about

48:30 earlier.

48:31 There's significant amount of simpler code and not just less code, but really, truly simpler.

48:38 And they say that that helps bring on new people as contributors, right?

48:42 They don't have to figure out how to write it in two different languages, sort of versions.

48:46 They're like, there's one thing to write.

48:47 This is how you do it.

48:48 Oh yeah.

48:48 The compatibility layers in some of these modules were horrific.

48:52 You know, that's why a lot of people were angry.

48:55 It's like, because the languages were in certain low level cases, languages were different enough

49:00 that you would have something that would be simple.

49:02 A couple lines would turn into like 20 lines, you know?

49:06 So it worked in both languages.

49:07 Yeah.

49:08 Like, why can't I just type import this thing?

49:10 It's like, well, you see, you got to do the sign.

49:12 We decided to rename it.

49:14 Yeah.

49:16 Yeah, exactly.

49:16 I understand.

49:18 Okay.

49:18 So that was 2017.

49:19 And 2018, you have declared victory.

49:22 Is that right?

49:22 Yes.

49:23 Yeah.

49:23 At this point, we know what Python 3 has won at Facebook.

49:27 And I think it's won in the world.

49:28 Like, there's...

49:30 I don't think it...

49:32 Even like with Guido's retirement from the core team, I think it's a done deal.

49:37 You know, nobody is...

49:39 There's no strong figure for the Python 2 camp at this point.

49:43 Yeah.

49:44 I know there are plenty of people hanging on, but I think there's just so many reasons

49:49 to move now.

49:50 It's much harder to make that as a reasonable argument.

49:53 It's just like, well, I just haven't done it yet.

49:56 It's kind of a...

49:57 It's a hard place to be in.

49:58 It's like, oh, it's like, we're going to be in a debt thing.

50:01 And, you know, maybe somebody will take care of it.

50:04 That's not a good vision for your future.

50:07 If you're staking your entire project, somebody has a kind of their heart is going to take care

50:12 of.

50:12 Yeah.

50:12 It's really rough.

50:14 So let's talk about this tweet that Lukas Lenga had sent out.

50:18 And you had mentioned in your presentation, and it's a graph, like a performance graph of

50:22 an app that was written there on Python 2.7.

50:26 And then when it switched to Python 3.6, could you just give us the quick summary of that?

50:30 Like it's...

50:31 Yeah, I probably won't talk about it.

50:32 I don't know what this is, but...

50:33 Yeah, no, it's not super important.

50:35 Yeah.

50:35 We traveled to Dublin to help this team out.

50:37 We're with them for two weeks, just helping them fix some, like, issues because their code

50:43 was pretty old.

50:44 Like, it was older than my tenure at Facebook.

50:46 So, like, once at the end of the week, we switched, we fixed all their little problems.

50:52 They switched over and immediately saw this drastic drop in their memory usage.

50:57 There was some CPU.

50:59 Oh, wait, no, they had...

51:00 Oh, it was half the memory.

51:01 Oh, yeah.

51:01 It was just half as much memory.

51:02 Yeah.

51:03 It looks like four times improvement in CPU and half the memory.

51:06 So, 2x in memory.

51:08 Yeah.

51:08 They were just like...

51:09 We were just...

51:10 I think Wukash immediately copied the graph and tweeted it.

51:13 You know, it was just like, this is insane.

51:14 Yes, exactly.

51:15 And this goes back to what we were saying before.

51:16 Yeah, the world seemed to like it.

51:18 There's like 1,800 likes or something just like, yep.

51:21 This is the kind of stuff that people need to see because you can tell them, like, well,

51:25 it's better or whatever.

51:26 But it's a whole other thing to say, well, that fleet of servers can be one server that

51:31 you don't have a hard time maintaining anymore, right?

51:33 Yeah.

51:34 Like, that's a much stronger story.

51:35 And you don't have to do anything but just make it run here.

51:38 Yeah, so this means that they were able to get rid of machines.

51:40 Yeah.

51:41 Because they're not doing as much work anymore.

51:42 And that makes everybody happy.

51:44 That makes your bosses happy because they're not spending as much money, especially if you're

51:48 like a cloud deployment and you're paying for each one of those nodes that costs you

51:52 money.

51:53 And it's, yeah, it's like you were saying nobody was saying these things.

51:56 It's like there was just this common understanding that, oh, Python 3 is slower than Python 2.

52:00 And we had to, like, break that lie that was still just kind of floating out there.

52:06 And we, and Lukash, he's talked about this a lot, was that a lot of the, like, core people

52:14 that, like, I don't know core people, but, like, people that wrote, like, really hard problems

52:18 in Python who had solved, like, kind of the low level, they were very negative.

52:21 And it was because they were solving the really low level problems.

52:25 But it was like the average, like, Joe or Jane writing Python code.

52:30 They're never going to see those issues.

52:33 So that when, when somebody comes out and be like, oh, I wrote this, this database driver

52:37 and it was a pain to do for Python 3.

52:40 It's like they do a disservice to everybody who will never see those issues.

52:44 Right.

52:45 Because you're, almost no one is natively, deep down at the native network layer.

52:50 Yeah.

52:51 Except for people writing web frameworks or data access.

52:54 You know, like, there's a few folks.

52:56 And they take a bit of a beating, right?

52:59 But they're also highly respected folks because they're the head of these major web frameworks

53:05 or database projects.

53:07 And they're like, this was super hard.

53:09 This is the hardest thing I've done.

53:10 It's like, well, okay, that sucked and we're sorry.

53:13 But, like, that is not what it's going to be like for other people.

53:17 They still import your thing and call query and it's all good.

53:20 Exactly.

53:21 Yeah.

53:21 That's pretty interesting.

53:22 So, I think these graphs, you know, also the story around Instagram, right?

53:27 Like, Instagram as a whole was able to turn off servers, was able to use less memory.

53:32 Like, they went pretty deep into that in the keynote that they covered that as well.

53:36 So, one other thing that you talked about in the 2018 timeframe, which I guess is now,

53:40 is this end of life of Python 2 internally.

53:43 Like, we know about the Python 2 death clock.

53:47 At least it exists.

53:48 I don't know if everyone knows about it.

53:50 But we know that there's the official external deadline for Python 2 at January 1st, 2020.

53:56 But you've also internally announced a deadline for this, right?

54:00 Yeah.

54:00 We want to make sure that we don't want to push past January 1st.

54:03 So, next year is going to be the year that I go around and help kind of, like, go through

54:10 and, like, find out those people that are still stuck on 2.

54:14 And maybe they've been in a cave all these years and they haven't known that 3 is a thing.

54:18 But, or maybe the team just doesn't exist anymore.

54:20 The thing's just running on automation.

54:23 Because we're...

54:24 Why is that website still up?

54:25 It's, well, you know, sometimes some things are just written well enough and they have enough

54:30 automation around it to keep it up.

54:32 It's just, you know, you figure out that, oh, wait, there is no team supporting this.

54:36 You know, like, oh, wow.

54:38 That's a success story, right?

54:39 But, yeah, it's a challenge.

54:40 It auto-builds and auto-deploys every month and nobody touches it.

54:43 So, yeah, it's like, so I think we're already internally, we're well over 80% of the libraries

54:50 are all Python 3.

54:52 And I think we're well over 60-something for, like, actually all those PEXs.

54:57 Like, the amount of PEXs we ship or Python 3.

55:00 So, it was figured, like, well, how long do we want to support this thing?

55:03 Because we don't want to get into the situation where we're having to support Python 2 past

55:07 the death clock, the deadline at the end of life.

55:10 So...

55:11 You don't want to be in 2020 doing anything with...

55:13 Yeah, so we're telling people that...

55:15 Trying to keep it going.

55:15 By the end of 2019, at Facebook, we're going to want to make it impossible to ship new Python 2 code.

55:23 Just so it's like, it's got, there's got to be some cutoff at some point.

55:27 And we're still going to support things that were already written, I think,

55:30 until, like, I think we said we were giving people, like, you know, maybe until 2022.

55:34 But it's like, at that point, you won't even be able to run it anymore.

55:38 It's like, we're going to get Python 2, the actual run times off the servers.

55:43 We're going to get them off the, out of the compilers.

55:46 We're going to make sure it doesn't exist.

55:48 Yeah, all the PEXs, they won't work anymore, right?

55:51 Yeah.

55:51 They're missing their CPython.

55:52 If it's just something that has been built in, like, five years, people still use it,

55:55 and it somehow miraculously works, it will stop working.

55:58 Right.

55:59 And then there'll probably be a little bit of a scramble to patch that stuff up, make it work.

56:02 We're hoping we catch all that in 2019.

56:04 And I think we will.

56:05 Yeah.

56:05 Yeah, yeah.

56:06 You guys have a lot of automated builds and whatnot.

56:08 Yeah, we have a lot of automation that can tell us immediately.

56:10 If I say that, oh, people that said they want two are now getting three,

56:14 I'll immediately start getting test failures if it's not working.

56:18 Exactly.

56:18 Just a little reporting, I'll tell you all that.

56:20 Okay.

56:21 So, Jason, that's a super interesting story.

56:24 And I think it's a real success.

56:26 And one of the things I don't think we necessarily covered as well as you did in your talk here is just at no point along the way did some manager or organization come along and say to you, like, okay, make this happen.

56:41 It's just like you wanted to see this happen and you just started working cooperatively with these different teams until it became a thing, right?

56:50 Yeah, at some point, like this was in 2016, when Instagram was moving, it was kind of like, hey, maybe we need a team to help people move or help, like, Instagram.

57:01 We kind of escaped the bullet of being the team that helps you move from two to three.

57:07 Because we did that for Instagram.

57:09 We did it for a couple other people.

57:10 But we were like, no, let's be the team that stewards Python for the future.

57:14 It's like we want something more lofty than just being the converter, the converting janitors or something.

57:19 Right.

57:20 That's very much more lofty in the long term, right?

57:22 Yes.

57:23 Yeah.

57:23 Yeah.

57:23 Yeah.

57:24 Very nice.

57:25 So, I guess transitioning, like, this is what you guys did.

57:29 This is what you were able to sort of germinate within Facebook.

57:34 What would you say to other companies that are stuck on Python 2?

57:38 Well, I was saying if they're engineer-led, like Facebook, and there's a lot of – we're in this internet economy, and there's a lot of startups that are out there.

57:46 They're very much engineer-led.

57:48 The same model works.

57:50 You just basically, you know, you can go and attack it through code quality.

57:53 I was telling somebody, it's like, I didn't always work in the Bay Area and Silicon Valley and these startups.

57:59 I used to work what I used to call real corporations in consulting.

58:04 And it's like, there you've got a manager, and instead of convincing other engineers, you just convinced a manager.

58:09 And it was sometimes easier.

58:10 Right.

58:11 There's one person to convince, and if you win that battle, it's done.

58:14 Yeah.

58:14 Right.

58:15 It's coming from on high, and it's just said.

58:17 Yeah.

58:17 If you can attack it with, like, some of the things we talked about, it's being faster.

58:20 Maybe it's going to use less money.

58:22 Maybe we'll use less servers.

58:23 And you can give them the example of Instagram.

58:26 They were able to do it without slowing down.

58:28 They didn't stop releasing features to convert to Python 3.

58:32 They did it all in line.

58:33 Yeah.

58:33 I really think that Instagram example, I'm so glad they did it as a keynote with all the clear steps.

58:39 Because it's not just an inspiring example, but it's a step-by-step roadmap to how it happens.

58:45 Exactly.

58:46 It's very inspirational.

58:47 So I guess, you know, this is a five-year journey, more or less.

58:51 What was easier than you expected, and what was harder?

58:54 There were several times where I made prognostications that were completely wrong.

58:59 Like, I thought that we would always have Python 2 as, like, the default, and we would still be a French thing.

59:05 But it's like, we quickly overtook Python 2 faster than we thought.

59:10 Like, I remember we actually had a message in our internal, like, IRC, like, hey, when Python 3 gets to 29%, let's have a party.

59:17 And, like, I just noticed it.

59:19 Nobody, like, had seen it for a while.

59:21 I noticed, like, oh, wait a minute.

59:22 We're at, like, 80.

59:23 Wow.

59:25 So I updated the messages.

59:27 Have a party.

59:27 We have, like, 90%.

59:28 There you go.

59:29 Yeah, and that's really something to celebrate, right?

59:32 Like, you're pretty much there at that point.

59:34 Yeah, so I definitely think we're going to have an internal funeral for Python 2.

59:37 And I've already, I was like, oh, no, we'll get, we'll, I was like, we'll get something, we'll make it look like a casket.

59:43 And have people talk about the good things that Python 2 did.

59:47 And we'll get rid of it.

59:49 Python 2 got us to where we are today, right?

59:52 Like, it was a major part of that journey, so.

59:55 Yeah, we can acknowledge it.

59:56 We just don't have to be stuck there.

59:57 Exactly.

59:59 What was harder?

01:00:00 You didn't actually expect it to work that well.

01:00:03 It sounds like in the early days, you're kind of like, this is a great try, but I'm not so sure it's going to work.

01:00:09 So maybe you just, you saw it as hard, and it actually wasn't, but.

01:00:12 Yeah, it wasn't as hard as I thought it was going to be.

01:00:15 Like, there's still the long tail of stuff.

01:00:17 Like, you know, we're getting to smaller, smaller values.

01:00:20 And I think it's just because we have, like, you know, in every company, you get more, as the company grows, you have more people writing code.

01:00:27 If that code is Python 3, your percentages of two just get pushed down automatically.

01:00:31 Right.

01:00:31 So I think that that is the, so the hardest thing is just to convince teams that, like, give them the, like, there are teams that say they want to move.

01:00:39 Like, everybody has the, like, oh, we want to move.

01:00:42 We just don't have the manpower.

01:00:43 We don't have it on a roadmap to move.

01:00:47 And they have it now because we've said, hey, it's 2019's the year.

01:00:51 You have to do it now or never.

01:00:53 It's going to stop running.

01:00:54 It's going to be a problem.

01:00:55 So that is probably, it's like, there are, yeah.

01:00:58 So it's just, there's some teams that feel hopeless.

01:01:00 Yeah, I can imagine.

01:01:01 It's like, kind of like what I thought.

01:01:03 Yeah, do you have companies that you acquire that bring, like, new Python 2 as another challenge every now and then?

01:01:10 Or does that not happen too much?

01:01:11 I don't know if I'm sure you can talk about that.

01:01:12 But then, like, sure, there's got to be some things.

01:01:14 But I think we kind of, like, there's enough people internally that, like, I guess when you're a good leader, like, and people don't know you're a leader, is that when they start repeating the same arguments that you were giving years ago, it's like coming out of their mouths instead of your own mouth.

01:01:30 I think, you know, you've won.

01:01:31 So it's like these people show up, like, hey, I've got this Python 2 app.

01:01:34 It's not working the way I want it to.

01:01:37 And what's the first from some random person I've never even met?

01:01:40 I'll get some things like, why don't you switch to three?

01:01:43 It has a defect.

01:01:44 Let's fix it.

01:01:44 Yeah, it's like switch to three and then we can help you.

01:01:47 Nobody knows how to do two anymore.

01:01:49 Yeah, yeah, that's great.

01:01:50 It's great.

01:01:51 I like it.

01:01:51 I mean, more organizations should be here.

01:01:53 All right, so I want to get your thought on one final thing and give a quick shout out to something that could help people as well.

01:01:58 So the same guy that put together, was involved in putting together the Sunsetting Python 2 support, the Python 3 statement, started calling Python 2 legacy Python.

01:02:10 Oh, I do like that.

01:02:11 Python 3 modern or just straight Python.

01:02:14 There's Python and then there's legacy Python.

01:02:16 And so the question is, Noah.

01:02:17 We do that internally.

01:02:18 You like this idea?

01:02:19 Yes, we call it legacy Python.

01:02:21 I love it.

01:02:21 I do too.

01:02:22 I have actually used that to identify other things I'm trying to sunset.

01:02:26 That's beautiful.

01:02:27 And I think when you spoke about talking to managers, like if you go to a manager and you say like, look, our entire infrastructure, our entire system is built on this legacy thing that's going to be unsupported in a year.

01:02:38 Like they're like, whoa, whoa, whoa.

01:02:40 That's scary.

01:02:40 That sounds horrible.

01:02:41 We got to fix this, right?

01:02:42 Yeah.

01:02:42 Like if you say Python 2 versus Python 3, you're like, ah, techno mumbo jumbo, leave me alone, right?

01:02:46 But legacy, they know that's bad.

01:02:48 They don't want that.

01:02:49 Yeah.

01:02:49 It's like sometimes we took jabs at Python 2 staunch supporters.

01:02:54 We would just to like make like we would call them blood heights because they didn't like the future.

01:03:00 They didn't like new technology.

01:03:03 So we were like, hey, are you like, are you like, is that how you got to work today?

01:03:09 Exactly.

01:03:09 Like how do you power that computer, right?

01:03:11 Yeah.

01:03:11 Plug it in or you got to puddle the thing or what?

01:03:13 It's like seeing your soul machine base.

01:03:16 Exactly.

01:03:16 Interesting.

01:03:17 Cool.

01:03:18 I'm glad.

01:03:19 That's definitely something I'm trying to push on the community as well.

01:03:21 And I think it's, I think it's resonates with people mostly.

01:03:24 All right.

01:03:25 So then a quick shout out, friend of the show, Anthony Shaw, wrote a cool course on converting from Python 2 to Python 3 over a Pluralsight.

01:03:32 So people should check out his course as well if they want to go along.

01:03:35 But certainly they should check out the keynote at PyCon 2016 or 2017 from Instagram because that really is a, you know, this is the roadmap of how you do it.

01:03:44 2017.

01:03:45 Thanks.

01:03:46 Yeah.

01:03:46 So that's a great one.

01:03:47 All right.

01:03:49 Well, we could talk for a long time, but I want to be respectful of your time.

01:03:52 So we'll just leave it here, I guess.

01:03:53 So I'll ask you two quick questions before we get out of here.

01:03:56 First of all, if you're going to write some Python code, what editor do you use?

01:03:59 Well, I'm Nuclide.

01:04:00 So Adam.

01:04:01 Okay.

01:04:01 The editor.

01:04:02 But it's with Fim mode because I still use a lot of Fim.

01:04:05 And, but if I, if I used to be a real big fan of PyCharm, it's just setting it up all the time was kind of a, it takes a lot of time.

01:04:13 So I always use Adam.

01:04:15 Adam's quicker.

01:04:16 Yep.

01:04:17 All right.

01:04:17 Cool.

01:04:17 And then a notable PyPI package.

01:04:20 Maybe not the most popular one, but something you're like, oh, people should know about this.

01:04:24 Wow.

01:04:25 Let's see.

01:04:25 Click.

01:04:25 Click.

01:04:26 All right.

01:04:27 Tell people quickly about it.

01:04:28 What's the, what's the elevator pitch on click?

01:04:29 Well, like internally people write these, they, they want to write these nice, like get style applications that are these sub commands.

01:04:36 And the.

01:04:37 Get add dot.

01:04:39 Get.

01:04:39 Yes.

01:04:39 Exactly.

01:04:40 That kind of stuff.

01:04:41 Right.

01:04:41 the art parse or opt parse to make that happen is horrendous and it's disgusting.

01:04:48 And people have written modules around these things to make it.

01:04:52 And all of them are awful.

01:04:53 And, and it was like, I was like, look, this is this thing.

01:04:57 I wrote, I used click for one of my projects.

01:04:59 I think the project that was the first Python three project used it.

01:05:03 I was so happy with it.

01:05:04 I just kept telling people you have to use this thing because it means that somebody I get off the street, who's been programming Python and open source, that it will feel natural for them to write an application with us without having to use some internal like CLI parsing.

01:05:17 Yeah.

01:05:18 That's really great.

01:05:19 Yeah.

01:05:19 It's definitely one of the most popular ways to build CLI apps, which is, you know, quite a few of the Python ones.

01:05:24 So very cool.

01:05:26 All right, Jason.

01:05:27 So people are excited.

01:05:29 They want to take their organization on the same journey.

01:05:32 What's the final call to action?

01:05:33 Like, what do they need to do?

01:05:34 You just got to set up, set some goals, set a plan, like, like, you know, demand in the future.

01:05:39 Like, look, we're going to be, we're going to go to Python three.

01:05:42 We're not just going to stay there.

01:05:43 We're like, that's, this is a stop.

01:05:45 We're going to go to, we're going to, it's a stop over to Python three seven.

01:05:48 It's a stop over to Python three eight.

01:05:50 We want to be modern so that when the new things happen, we can move with them.

01:05:54 Yeah.

01:05:55 And they're happening fast.

01:05:56 Exactly.

01:05:56 And they're happening in large numbers.

01:05:59 So you definitely want to be there.

01:06:00 All right.

01:06:00 Thank you so much for sharing the story.

01:06:02 I think it's going to inspire a lot of folks to take some action.

01:06:06 Yeah, I think so.

01:06:07 It's been fun.

01:06:07 Yeah, it sure has.

01:06:08 Nice chat with you and goodbye.

01:06:10 Bye.

01:06:10 This has been another episode of Talk Python to Me.

01:06:15 On this episode, our guest was Jason Fried, and it's been brought to you by Linode and Rollbar.

01:06:20 Linode is bulletproof hosting for whatever you're building with Python.

01:06:24 Get four months free at talkpython.fm/Linode.

01:06:28 That's L-I-N-O-D-E.

01:06:30 Rollbar takes the pain out of errors.

01:06:33 They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until your users complained, of course.

01:06:41 Track a ridiculous number of errors for free as Talk Python to Me listeners at talkpython.fm/rollbar.

01:06:48 Want to level up your Python?

01:06:49 If you're just getting started, try my Python jumpstart by building 10 apps or our brand new 100 days of code in Python.

01:06:56 And if you're interested in more than one course, be sure to check out the Everything Bundle.

01:07:00 It's like a subscription that never expires.

01:07:03 Be sure to subscribe to the show.

01:07:04 Open your favorite podcatcher and search for Python.

01:07:07 We should be right at the top.

01:07:08 You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm.

01:07:17 This is your host, Michael Kennedy.

01:07:19 Thanks so much for listening.

01:07:20 I really appreciate it.

01:07:22 Now get out there and write some Python code.

01:07:23 I'll see you next time.

01:07:44 Thank you.