#401: Migrating 3.8 Million Lines of Python Transcript

00:00 At some point, you've probably migrated an app from one framework or major runtime version to another.

00:05 For example, Django to Flask, Python 2 to 3, or even Angular to Vue.js.

00:11 This can be a big challenge.

00:13 If you had hundreds of active devs and millions of lines of code, it's a huge challenge.

00:19 We have Ben Barito from Yelp here to recount their story of moving 3.8 million lines of code from Python 2 to 3.

00:28 But this is not just a 2 to 3 story.

00:30 It has many lessons on how to migrate code in many situations.

00:34 There are plenty of gems to take from his experience.

00:36 This is Talk Python to Me, episode 401, recorded January 18, 2023.

00:56 Welcome to Talk Python to Me, a weekly podcast on Python.

01:00 This is your host, Michael Kennedy.

01:01 Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython, both on fosstodon.org.

01:09 Be careful with impersonating accounts on other instances.

01:11 There are many.

01:12 Keep up with the show and listen to over 7 years of past episodes at talkpython.fm.

01:18 We've started streaming most of our episodes live on YouTube.

01:21 Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.

01:30 This episode is brought to you by Cox Automotive.

01:33 Join their team and use your technical skills to transform the way the world buys, sells, and owns cars.

01:39 Find an exciting position that's right for you at talkpython.fm/cox.

01:42 And it's also brought to you by User Interviews.

01:45 Earn extra income for sharing your software development opinion at User Interviews.

01:51 Head over to talkpython.fm/userinterviews to participate today.

01:55 Ben, welcome to Talk Python to me.

01:58 Thank you.

01:58 Thank you so much for having me, Michael.

02:00 We're going to talk a little bit of legacy code, a little bit of very, very large code bases,

02:05 and how you might not have to permanently live in the past, which I think would be really welcome to a lot of people.

02:12 I just talked a little bit about this before I hit record, but even though your topic is specifically

02:17 how the story of move from Python 2 to 3 and this, like, making your whole code base modern,

02:23 I do think that this idea of how do I move from one code base to another code base is super relevant to lots of folks

02:29 who might not be going from Python 2 to 3, but maybe from Flask to FastAPI or vice versa, or those types of things.

02:36 So I think the techniques that you're going to talk about here are more broadly applicable than just a 2 to 3 migration.

02:42 And it's really cool how you all migrated 3.8 million lines of code without interrupting development.

02:49 That's kind of nuts.

02:50 Yeah, I did it, and it still seems ridiculous.

02:52 You lived it, and it seems like a dream. Amazing.

02:55 Before we get to all that, though, let's start with your story.

02:57 How did you get into programming and Python?

02:59 Took a job at Yelp. Yelp was a Python shop.

03:01 Before that, I had a couple internships, and I went to Georgia Tech, and I mostly did Java.

03:06 So it was sort of a new experience for me.

03:08 You know, Python is one of those, like, beginning languages that everyone loves to throw around.

03:12 So I had done, you know, I dabbled in a little bit, but when I first started really getting, like, deep into the language when I started it at Yelp,

03:22 and I've definitely made it, it's sort of become, like, I've become sort of a local expert on it.

03:28 So I've been able to build up a lot of knowledge about, like, you know, a lot of weird edge cases and, you know, stuff like that.

03:34 You may be familiar with it.

03:36 There's a t-shirt that's kind of a joke, a meme.

03:39 It says, I learned Python.

03:40 It was a great weekend.

03:41 Yeah.

03:42 And yet, I've been doing Python for many years, and I'm still learning new stuff.

03:46 Even today, I learned some interesting new Python things.

03:50 So which is it?

03:51 Do you learn it in a day, or is it, like, this deep journey?

03:54 I think most programming languages have some amount of, you know, width and depth.

03:59 I think, you know, Python definitely has the advantage of being, you know, a relatively straightforward language.

04:04 One of the nice things, obviously, is that, like, instead of using a lot of weird keywords, it has, like, you know, words.

04:11 Like, you know, instead of being, like, oh.

04:13 Like, or a pipe.

04:15 Yeah, or, yeah, or instead of double pipe and stuff like that.

04:18 Like, those are the things that I think definitely help people.

04:21 Personally, you know, me coming into it, that wasn't, like, as big of a deal for me because, like, I was already familiar with all that stuff.

04:27 But, like.

04:27 You're coming from a very symbol-heavy world of Java, which is not as symbol-heavy as C++, but it's got a lot of abstractions in what it builds, for sure.

04:36 Yeah.

04:36 I will say, like, I think there are certain things that Python does.

04:40 I will say Python is not, like, a perfect language by any stretch of the imagination.

04:44 But I will say one thing about Python that I think is really cool is it did, sort of, before many other languages, managed to integrate a lot of, like, functional paradigms.

04:53 Like, list comprehensions or comprehensions in general are, I think, one of those features where it's just, like, this doesn't exist in a lot of other, sort of, more popular languages.

05:02 Yeah.

05:02 And they're really, really fluent and powerful in a way that, like, you kind of miss when you don't have it, right?

05:08 And so, like, I think that's something that's really cool about Python.

05:11 But Python itself, you know, being a language with the legacy that it has, you know, I mean, we're going to be talking about the two to three differences which have their own nuances to them.

05:20 But, like, it's always going to have some weirdnesses to it.

05:22 And some of those things are, like, just like, oh, someone made a decision, you know, 30 years ago that, like, still kind of reverberates today.

05:29 And so that means that, like, it has this depth to it, you know, you have to really learn the depth in order to fully understand all of the problems that exist.

05:38 Like, dealing with not necessarily problems with the language, but, like, you know, when you're building software, you run into problems that you have to solve.

05:44 Right.

05:44 And so that's, like, the main, I think that's true of all languages to an extent, especially, you know, popular ones and older ones.

05:51 But, yeah, I do think that Python, more than a lot of languages, really is able to straddle the line of, like, being like, oh, it's approachable, but also you can do a lot of, you know, really interesting and powerful stuff with it.

06:02 Yeah, you compare that with, like, Java.

06:04 Java, you've got to understand functions.

06:06 You've got to understand classes, possibly namespaces, like, just to write the first line of code.

06:13 Whereas Python, you can work with it for, like, you know what, this is really clumsy to repeat this.

06:18 Maybe I'll learn what a function is.

06:20 And then I can start using that.

06:21 But you don't know what a class is.

06:22 You don't care about it.

06:22 You kind of, like, slowly layer on the stuff as you need it, rather than you've got to jump in and go with it all at once.

06:30 Yeah, for sure.

06:31 Yeah.

06:31 Interesting.

06:31 So you're still at Yelp?

06:33 Yes.

06:33 Yeah.

06:34 I'm in our, in a conference room in our San Francisco office right now.

06:38 Excellent.

06:38 And what are you doing there?

06:40 So I work on a team that's called Core Services.

06:42 Our team is responsible for a lot of infrastructure, mostly Python-focused, though not exclusively.

06:50 We do a lot of our sort of, like, internal Python infrastructure.

06:54 So, you know, we'll be responsible for making sure that we can upgrade to new Python versions.

07:00 We've got, we own our internal PyPI.

07:02 We recently, this is a cool thing that has happened since my talk, so I didn't mention it, is we recently built a system to automatically import certain packages from a public PyPI.

07:16 And that has saved us some headaches.

07:18 And then we own some other stuff, like, you know, we've dealt a lot with sort of the general service contract at Yelp.

07:24 So, like, being like, okay, what, what does it mean to be a service?

07:26 How do you, how do you be a good citizen there?

07:29 And a lot of other things, like, testing tools, a lot of, like, sort of, like, oh, I need to test against multiple services.

07:36 We have a testing tool that, like, automates a lot of the steps to, like, sort of get those all connected together so you can test against them.

07:42 Yeah, that sounds like a really fun set of task you're doing there.

07:45 So you said you have an internal, we're going to dive into code and stuff, all this whole migration.

07:50 But, you know, kind of sidebar, like you said, you have this internal private PyPI server.

07:55 What's the details around that?

07:57 Like, how, obviously, you're whitelisting things that can be brought inside saying, we're going to put those onto our server.

08:04 And you can, like, request, and you can choose when to let the new one in and so on.

08:08 But, you know, what's the, what's the software and how do you put that together?

08:12 Right.

08:12 A number of years ago now, we switched to a piece of software that I don't think is used pretty much anywhere else,

08:19 which was built by one of my teammates who named Chris Keel.

08:24 So Chris Keel built essentially a PyPI implementation called Dumb PyPI.

08:30 It's called Dumb PyPI because unlike some other PyPI servers, the way that it works is you just sort of give it a list of distributions

08:39 and then it just generates all of the HTML pages.

08:42 So instead of, instead of being like, oh, I'm a server and I'm like, you know, going to handle this request and blah, blah, blah,

08:48 and like run some code, it's literally like, okay, here are the HTML pages.

08:51 And I see, it's like a, like a static site generator for PyPI backend.

08:56 Yeah.

08:57 Yeah.

08:58 And we've been using that for a really long time and maybe like four or five years now.

09:03 And it's, you know, it's serviced pretty well.

09:06 It's really nice in my opinion, because it's the only service that my team actually owns.

09:11 So it, and it never pages us.

09:13 So that's great.

09:13 I love that it never pages us.

09:15 It can't go down.

09:16 Yeah.

09:17 Not really.

09:18 Well, it can't, it would be good that if it goes down, but also it doesn't.

09:22 That's the great part.

09:23 It just, it just doesn't.

09:24 So Chris's, Chris's software works great.

09:26 Yeah.

09:26 That's a really cool idea.

09:28 And so you tell it certain versions or, or do you just limit it to the libraries and

09:33 let it pick the latest versions of whatever's on real PyPI?

09:36 So the way that we do it is we have a whole system, which like imports packages.

09:42 We actually rebuild all of our wheels.

09:44 It's kind of for kind of hard to explain reasons.

09:48 So what we'll do is we'll like, someone will say, Hey, I want this version of this package,

09:52 or maybe they'll just say, Hey, I want this package.

09:54 And then we'll just pull down the newest one at the time and we'll do some like security

09:59 bedding on it.

10:00 So we have some like automated security stuff and basically just make sure that it's like

10:03 not malicious.

10:04 And then we build the wheels and then we upload those to the S3 bucket that like backs our PyPI.

10:12 Right.

10:12 So we just do that in terms of like how we decided it's basically just sort of like, we, you know,

10:19 make sure, you know, we do the security check.

10:21 We do like, there's a few other things.

10:23 Like we make sure we have all the dependencies.

10:24 We make sure we have a, that it has like a license that we're okay with, with using internally.

10:30 And so all of those things are checked.

10:32 And then we have, as I mentioned, we have this sort of like automated import system.

10:37 So like certain packages, we'll just, we'll try to download them.

10:40 They might fail, you know, one of those checks and then we won't upload it.

10:43 But like, you know, so we'll just like import it.

10:45 We'll try to import it.

10:46 And so certain packages, we'll try to get the newest one.

10:49 Some packages, you know, how we haven't set that up for one reason or another.

10:53 Some packages, there's certain packages that are just like difficult to build.

10:56 And so we avoid importing them.

10:58 Right.

10:58 You're just, we got this one working.

11:00 It's fine.

11:01 Yes.

11:02 Yes.

11:03 And so like some are difficult to build.

11:05 Some are just like, oh, this is a package we've never used before.

11:08 So we just like, don't use it.

11:10 Yeah.

11:10 So, or we don't have it.

11:11 Yeah.

11:11 You talked about how many dependencies your projects have and stuff and that'll be fun.

11:16 But let's maybe take a step back and just talk about, you know, Python at Yelp.

11:20 You've, this main project that you have, it was running on Python too.

11:25 It's kind of obvious, but some of the reasons are obvious.

11:28 Some are not.

11:28 Like why did you care what version of Python it's on?

11:31 That's a good question.

11:32 I mean, I think the main reason was just sort of like we saw the writing on the wall,

11:37 the running on the wall was the end of life or Python too.

11:39 Right.

11:40 And I think everyone else, we knew that other people were going to follow that.

11:43 Right.

11:43 There was, I remember in 2019, when I was looking into this, there was a thing, I think it was

11:49 called like the Python pledge or something like that, where basically like packages would

11:53 like open source packages would say like, Hey, we're going to drop Python three, you know,

11:57 after end of life at some like, you know, either the day of or, or a few months later,

12:02 or something like that.

12:03 And so we were sort of looking at that and being like, well, we use some of those packages,

12:06 you know, and eventually we might want to upgrade them.

12:09 Yeah.

12:09 You're about to get frozen in time around mid 2020.

12:12 So you're, you maybe don't want that.

12:15 By the time that I did my, my talk, I remember, I think it was early 2021 or something, you know,

12:21 pip had dropped support for Python too.

12:23 So that was like one of those things where it was just sort of like, yeah, there's not,

12:27 there's not a realistic ecosystem in which you are able to use like open source and upgrade

12:32 your stuff, you know, for security patches or whatever, you know, I want this new feature.

12:37 Oh, sorry.

12:38 That's Python three only, you know, kind of thing.

12:40 So that was like the main motivation.

12:42 Right.

12:42 And then I think some secondary stuff was just sort of like, as you build, you know,

12:47 as time marches on and like people stop being familiar with like Python two and it has some

12:52 quirks, you know, for Python three, you definitely have the problem of like, okay, now you have

12:56 to like, if you're hiring people and they're working on Python two, you have to train them

12:59 up on those quirks in a way that like you wouldn't necessarily have to do if you're using a modern

13:04 language that other places are using.

13:06 So those are the, I think the main motivations personally, I think I had like a small motivation

13:11 myself, which was just sort of like, I hate seeing things like be left behind like this,

13:16 you know?

13:16 Yeah, sure.

13:17 Very emotional thing, but yeah, that's part of the reason I pushed for it.

13:21 Well, there's the training thing.

13:22 I mean, there's obviously the, just the infrastructure stopping, stopping the updates, but there's the

13:28 training side of helping people who are new come, but there's also the, how do you hire

13:33 the very best engineers?

13:35 It's really hard to get an amazing Python engineer to come and say, you're going to do

13:40 amazing work from 2008.

13:42 You're going to love it.

13:43 You know what I mean?

13:44 Right?

13:44 Like if they're working on some new package that they're inspired about, instead of trying

13:48 to bring that in and like, you know, help make that better and also boost what you're

13:52 doing.

13:52 It's like, well, we can't use that because that you only don't Python three.

13:55 Like, well, of course I created it this, you know, two years ago.

13:57 Why wouldn't it be Python three only?

13:59 And there's a lot of knock on effects like that, right?

14:01 Yeah.

14:01 Did you see the performance stuff from 2.11 or even from 2.11?

14:06 Sorry, 2.11.

14:07 3.11 or even 3.10?

14:08 Where, where you're like, you know, there might actually be fewer servers as well if we do this?

14:14 That's definitely something that we are, you know, we want to do.

14:18 That specific issue is something that we're, we're sort of, we're trying to move towards

14:22 being able to use those versions of Python right now.

14:25 It's always a process just because of various, you know, internal things, but it's definitely

14:30 something that has been, that has been talked about as well.

14:32 Like, yeah, if we could use new versions of Python, maybe things will be faster.

14:36 Things will be more efficient.

14:38 Trying not to spend too much money is definitely a thing that we, we think about.

14:42 So that's definitely exciting.

14:43 Yeah.

14:44 When I did the episode on 3.11, we talked a lot about the performance there and it's, it's

14:49 impressive.

14:49 It's, you know, 40, 50, 60%.

14:51 And I won't steal your thunder.

14:53 I know at the end, you've got some nice performance boosts that you got even from the changes that

14:58 you made.

14:59 But there was somebody in the audience that pointed out, like, not only is this faster,

15:02 which is nice for us, right?

15:04 It's nice that we have to pay less for servers.

15:06 It's nice that our code runs a little bit faster, but it's also good for the planet, right?

15:11 If we just all start using newer, faster foundations, then necessarily we just use less energy to

15:18 do the same thing that we're already doing, right?

15:19 Yeah, that's definitely, I like that.

15:22 This portion of Talk Python Nemy is brought to you by Cox Automotive.

15:27 With brands like Kelly Blue Book, AutoTrader, Dealer.com, and more, Cox Automotive flips

15:33 the script on how we buy, sell, own, and use our cars.

15:38 And now the team at Cox Automotive is looking for software engineers, data scientists, scrum

15:44 masters, and other tech experts to help create meaningful change in the industry.

15:48 Do you want to be part of a collaborative workplace that values your time and work-life balance?

15:54 Consider joining Cox Automotive.

15:56 Visit talkpython.fm/cox today.

16:00 Thank you to Cox Automotive for sponsoring the show.

16:03 Let's talk about Python at Yelp.

16:06 So you've got this repo, this big project called Yelp Main.

16:11 Let's start there.

16:12 Sure.

16:12 Yelp Main is what it sounds like.

16:15 It is sort of the original repo at Yelp.

16:18 That's when you're a startup in 2004.

16:20 You kind of just make a repo, right?

16:23 And it's your web app.

16:24 You probably made a Subversion repo.

16:26 Because you didn't know that CVS stuff.

16:29 We're doing Subversion.

16:30 I don't remember when we switched from Subversion.

16:33 But it was Subversion.

16:35 I don't know if it was before Subversion.

16:37 I don't know if we actually started out in Subversion.

16:40 Okay.

16:41 Because I didn't start until 2014.

16:42 Sure.

16:43 But yeah, it's definitely Subversion was, I know there was some old Subversion stuff.

16:47 But so you have this one, you know, sort of web app.

16:49 And the web app is a server that serves, that originally served everything.

16:56 So, you know, there's a bunch of stuff.

16:58 Like, you know, you can sort of think of as like Yelp.com, right?

17:01 Like, if you go to www.yelp.com, then you're looking at what you think of as Yelp, right?

17:07 It's like, oh, I can search for businesses.

17:09 I can look at their reviews.

17:10 I can write my own reviews, that kind of stuff.

17:12 So it's that, but it's also other stuff.

17:14 It's also our business owner site.

17:17 So biz.yelp.com, which is where business owners, like, look at their own businesses and, like,

17:21 are able to see the metrics and, like, you know, buy ads and stuff like that.

17:25 There's our admin site, which is, you know, where a lot of anybody, we have our sort of user operations people whose job is at least partially to do some moderation

17:34 and stuff like that.

17:35 So, like, we need to be able to have those tools.

17:37 And then there's also what we call internal API.

17:42 And internal API is a way for internal stuff to get the data that's in Yelp name.

17:49 So that's what that is.

17:51 And that's like its own separate sort of site.

17:53 And, but these are all in the same repo.

17:55 They all run in the same process.

17:57 That's, and yeah, sorry.

17:58 No, I was just going to ask, is this kind of the, the monorepo style or it's, it's truly

18:03 a monolith in the sense that it's kind of all the same app.

18:06 It's truly a monolith.

18:07 There is some amount of stuff where it's like, oh, we have like different containers running

18:12 like different entry points.

18:13 But like the code is all kind of tangled up together.

18:16 So there's not really a meaningful delineation between different components in a way that you

18:22 could really separate them out in any meaningful way.

18:24 So like my understanding of like what I would define as a monorepo, I wouldn't really call

18:29 it that.

18:29 I would just call it, I would call it just a large app.

18:32 Yeah.

18:33 It's a huge, huge, huge app, huge repo.

18:35 Yeah.

18:35 Yeah.

18:36 Okay.

18:36 So in your talk, you said that you have six different sites with 2000 different endpoints,

18:42 which it's a lot.

18:44 I don't think it's completely excessive or anything like 2000 URL endpoints for, for all

18:49 those different services and like all those different admin apps.

18:52 So it seems it's a lot, but it's not insane.

18:54 And then you have these background batch services.

18:57 What story of those?

18:59 It's just sort of anything that you need done.

19:01 As I said, this was just sort of like, this is the one repo, right?

19:04 And so there's a lot of things that you want done that aren't necessarily done in the context

19:09 of a web request or don't make sense to do synchronously.

19:12 So a lot of that is just sort of like, okay, you know, I need to like do this like really

19:17 complex report or something, right?

19:19 You know, I want to get some metrics that like involve collating a bunch of data, doing a

19:23 bunch of joins against a bunch of tables.

19:25 Okay.

19:26 Well, I'm not going to have just like a web request do that.

19:28 I'm going to put that in like a separate process.

19:30 And we originally, as you can imagine for that type of application, we just called them

19:36 just batches.

19:36 Yeah.

19:37 Like a patch, a batch job, right?

19:39 That name is stuck despite the fact that now batches don't necessarily do that type

19:43 of work.

19:44 They're just sort of anything that you want to do in the background.

19:46 And that could be something like, oh, the first of the month we do our like ad billing,

19:51 or we might have some process where it's just sort of like, oh, we want to like update this

19:58 cache, you know, based on like data like stuff, but we don't want to do it inline in a web

20:03 request.

20:04 We can do it asynchronously.

20:05 So it's really anything that is not in the context of a web request.

20:10 I suspect most major apps, most companies have that kind of stuff too, right?

20:15 They've got to.

20:16 I mean, everyone has some version of it.

20:18 Whether or not they do it exactly the way that we do it is a separate question, but I'm

20:22 not really sure.

20:23 But yeah, I think part of the story is, do you deploy them all out of the same code base?

20:26 Or are they, you know, a bunch of different jobs and repos?

20:30 Or how's that fit together?

20:32 That's probably where the, it varies.

20:33 Yeah.

20:33 I mean, for us, we have, I mean, I said 800 batches and I was referring specifically to

20:38 the batches that are still in the Yelp main repo.

20:40 And like I said, all of these things are kind of tangled together.

20:43 So it's not like, oh, you can just like pull a batch out.

20:45 Like that's like talking about like, well, how do you get the data that it needs?

20:49 And like, what does that look like?

20:51 And blah, blah, blah.

20:52 How does it get the data access layer?

20:53 And how does it get a hold of the logging thing that's over here and all that kind of stuff?

20:57 Right?

20:57 Exactly.

20:58 So like, so those 800 batches, but we also have tons and tons of batches that are in services.

21:04 So they live in service repos and they run and they're, they're in totally separate code base.

21:09 I don't know what that number is.

21:10 I'd have to figure it out.

21:11 Yeah.

21:12 It's definitely a lot.

21:12 It's probably, it's almost certainly more than that, than we have in Yelp main at this point.

21:16 But that paradigm exists all over Yelp and not just in this repo.

21:20 Yeah.

21:20 Well, I think these are a lot of value to having that code together, right?

21:25 If you break this out into a whole bunch of different repos, you've got dependency management,

21:29 versioning, deployment, like there is some value to just saying like, just let it live together.

21:34 We'll upgrade it together.

21:35 But it, it does make for some striking headlines when you talk about how many lines of code got

21:40 upgraded at once, right?

21:41 Yeah.

21:42 I mean, I think that when you're talking about like, why do we solve them all?

21:45 The answer is mostly just because it's really hard to not have one.

21:49 Once you have one, you have to do all the work to move it out.

21:53 And there are disadvantages.

21:54 Like you said, it's sort of like, okay, now, as soon as you have a new repo, it has its own

21:58 set of dependencies that you have to keep up to date.

22:00 And, you know, you have to do other sort of maintenance on it.

22:04 Generally speaking, we consider that better though, still, because it's sort of like, it's always

22:09 better.

22:09 Like imagine if I'm in, in this giant monolith and I'm like, oh, I need to upgrade this package.

22:14 And it's like, okay, well you want to do a major upgrade.

22:17 And this package is imported in a thousand places.

22:20 How you need to deal with that migration.

22:22 Whereas if it's like, oh, I'm in my service and this, and I need to do this package upgrade

22:28 and it's imported in 10 places.

22:29 That's like an afternoon instead of like, you know, a quarter of a year or something.

22:35 Right?

22:35 Yeah.

22:36 So there's definitely advantages to that.

22:38 It does add, it's sort of like more work overall, but you can do it in a more granular way.

22:43 So it allows you to unblock people faster, essentially.

22:47 So like, we definitely want to move away from the monolith and we have been doing that.

22:51 Like compared to when I started at Yelp, we have way, way less code that is important running

22:56 in Yelp main.

22:57 There is still a ton that's important in there.

22:59 Like I mentioned in my talk, like almost inevitably someone has to like call into an internal

23:03 API to get our, get data out of it.

23:05 So that's something that like, we definitely want to fix at some point, but it is a process

23:10 and that process is generally speaking, like we're getting to a point where like some people

23:15 who work at Yelp don't really work in Yelp main anymore.

23:17 Like they just don't have to deal with it, especially not on a day-to-day basis.

23:20 Right.

23:21 Sure.

23:21 And you mentioned your talk.

23:22 I don't know if I said this beginning, but you gave a talk at PyCon 2022, which is definitely

23:27 it was a very popular one and highlights some of these things there as well.

23:30 So I'll be sure to link to that so people can check it out.

23:33 And you talk about people developing in Yelp main, some of them not, but there's still a

23:38 lot happening there.

23:38 You said 20 pushes a day, 800 simultaneous developers.

23:44 And yeah, that's, that's no joke.

23:47 That's a lot of traffic on a repo.

23:49 Yeah.

23:49 I think since I did that talk where we've been going, we've been trending down in terms of

23:54 number of changes per day, but it would still probably close.

23:57 Like somewhere in the eight, like 15 to 25 a day.

24:01 So it's less like that's an appreciable percentage less, but it's still a lot of changes per day.

24:06 Yeah.

24:07 Yeah.

24:08 And you also said you have 700 Python package dependencies.

24:11 We talked about the private PyPI.

24:13 So when you say you have 700 dependencies, that's if I go into the virtual environment and type,

24:18 you know, pip list, I see 700 things.

24:20 Yep.

24:20 Okay.

24:21 It's a lot.

24:21 It's a lot.

24:22 It was an ordeal dealing with me.

24:24 Especially coming from a long time ago until present, right?

24:28 In terms of code, code compatibility, right?

24:31 Some of those things you depended on.

24:33 Maybe their new versions have moved to Python 3, but maybe with breaking changes.

24:37 Others, they might just not have a Python 3 version.

24:40 Mm-hmm.

24:41 And how'd you deal with that?

24:42 There were basically, in terms of like open source stuff, there were basically like three

24:47 ways that we dealt with that.

24:48 So one is just like upgrade and like deal with whatever the upgrade entails.

24:53 I don't think we really ran into any issues where we were like, oh no, we have to do this

24:58 like massive breaking change, you know, migration.

25:01 That wasn't really a problem that we ran into, thankfully.

25:03 So a lot of those were just sort of like figuring out what packages need to be upgraded and just

25:08 like sort of doing the upgrade, making sure that they test pass and that kind of stuff.

25:11 So that wasn't too bad.

25:12 The other one, which was a little bit more annoying was, like you said, some packages just

25:17 stopped updating before they got Python 3 support.

25:23 And we were relying on them.

25:24 So we had to be like, okay, well, can we replace these with something that, you know, fixed?

25:29 And there were a few examples of packages where it sort of stopped getting development.

25:34 And then someone was like, oh, that I see where the problem, like that's a problem for me.

25:39 So I'm going to fix that.

25:40 And so luckily a lot of people had already done that work and they, there were like forks

25:44 or sort of drop-in replacements.

25:46 Sometimes not exactly drop-in replacements, but like, you know, close enough that we could

25:50 like do, do the small amount of work that was needed.

25:52 It's one of the advantages of being a little bit later to the party is let other people

25:56 bump into those problems and maybe they fixed them for you, right?

26:00 That probably happened most of the time, honestly.

26:01 That was definitely a good chunk of the time.

26:03 I couldn't tell you I'd have to like go back and like run the numbers on like what percentage

26:07 of the time that was.

26:08 But like, we definitely, yeah, there was definitely a good chunk of things where we're just sort

26:11 of like, oh, someone already made the fork or whatever.

26:13 And we can just use that.

26:14 And that was, that was nice.

26:16 It was okay.

26:17 That's, that's, you know, that one checked off.

26:19 And then the final sort of grouping was stuff where there, what that wasn't available.

26:25 So it was like, oh, this package is Python 2 only.

26:28 And no one ever made a replacement.

26:30 So we need to deal with that.

26:32 Luckily, we, none of those were in a position where we were completely unable to deal with

26:38 it.

26:38 Like we didn't run into anything where we were like, oh, this is just like, this is like

26:42 a blocker.

26:42 But there were things where we were like, oh, this thing needs to be replaced with something

26:46 else that does something similar or maybe right after right away.

26:51 Like very often we ran into code where it was like, oh, this is using this thing.

26:55 And then you'd start looking into it and you're like, oh, actually this code is like this like

26:59 branch or whatever that uses this package isn't actually used anymore.

27:04 So we can just delete all that code and like not have to think about it.

27:07 So that's, that's how we dealt with it.

27:09 Yeah.

27:10 That's a nice way to upgrade it.

27:11 Let's just get rid of it.

27:12 Yes.

27:12 Were there any packages that you're out there that didn't have Python 3 support and

27:17 you're like, yeah, really, we really depend on this one that you upgraded and contributed

27:22 back or were you able to just move on?

27:24 There was nothing that we ran into that was like an absolute blocker like that.

27:29 So we didn't end up contributing anything in terms of open source other than there were

27:35 some packages that are like on our GitHub, like the Yelp GitHub that we did do upgrades

27:40 for.

27:41 Sure.

27:41 That was, you know, that was the only sort of open source work that we, I think we really

27:45 ended up doing.

27:45 Yeah.

27:46 So luckily, I mean, I don't know if this is lucky or not, but it's definitely, it happened

27:50 so that we didn't have to do that.

27:52 Yeah.

27:52 That's good.

27:52 I mean, it would be nice if you ran across that and helped solve it for someone, but you

27:57 don't have to even better.

27:58 Testing.

27:59 One of the challenges of, well, first it's good to have tests, but one of the challenges

28:04 of these upgrades is you wanted to do this without disrupting development.

28:09 You wanted to keep adding new features.

28:11 You didn't want to say, hey, everyone, stop making any progress or bug fixes for six months

28:18 and we're all just going to do this until we're done.

28:19 Right.

28:20 You wanted to keep it moving.

28:21 But in order to do so, you've got to run the test because you're making wholesale changes

28:25 to millions of lines of code.

28:27 So that's pretty nerve wracking.

28:28 Right.

28:28 And you're swapping out its dependencies in big ways.

28:32 And yet running tests, you all have a lot of tests and they take a while to run.

28:35 Right.

28:35 Yeah.

28:36 We have about a hundred thousand tests in Yelp main, a little under.

28:40 And yeah, if you were to run them serially, at least when I wrote my talk, it was about

28:45 35 hours total.

28:46 But we have a test runner framework called Jolt that we run internally.

28:51 And what it does is it basically like puts those tests up into bundles and then runs those

28:56 across a bunch of machines.

28:57 And so you're basically able to get all of the tests run for Yelp main in about, give or

29:04 take an hour and a half.

29:05 Okay.

29:05 That's pretty good for running a hundred thousand tests.

29:07 That's still a long time to have a test run though.

29:09 Right.

29:10 So you probably need it.

29:11 You can't just get immediate feedback.

29:12 Minor change.

29:13 How'd that go?

29:14 Minor change.

29:14 How'd that go?

29:15 You kind of, it'd be a little more thoughtful than that.

29:17 Right.

29:17 Yeah.

29:18 I mean, I think that in terms of, and this sort of gets into like testing theory is that

29:22 like you start to get an idea of like what changes are like affect what other things

29:29 like sometimes you're not, you're not going to have a perfect idea, but like if you're

29:33 like, oh, this is a thing.

29:35 That just affects everything.

29:36 Then you're going to run all the tests.

29:38 But we did have the ability to run tests.

29:40 If we were like, okay, we want to just run tests under Python three.

29:43 We could do like, oh, I'm just going to run this, you know, test module under Python.

29:47 Right.

29:47 I can do that.

29:48 And so like, if you were literally just like, oh, I'm checking, I'm like fixing this test

29:53 under Python three, then you could just do that.

29:55 You could just be like, oh, I'm iterating very quickly by like changing the code and then

29:59 running the test under Python three.

30:01 And then, you know, oh, it passes.

30:03 Okay.

30:04 Let me double check.

30:04 It passes under Python two as well.

30:06 And then you can commit that and then like put that into PR.

30:10 And then we do require.

30:12 So one of the things is we do require running all doing a full jolt run for every pull request

30:17 to Yelp me.

30:18 So in order to do so, you have to run that anyway.

30:20 But like, while you're waiting for that to run, you can work on something else.

30:24 Sure.

30:24 And you're high confidence.

30:26 You run a couple.

30:27 Okay.

30:27 That makes sense.

30:27 So run a couple of local tests, 10, 100, 500, whatever.

30:31 Once you're happy with that, then you put it as a PR and CI figures out what happens.

30:36 Yeah.

30:36 Okay.

30:37 And I think that ultimately, like there, when we were really early on and we were working

30:41 on like the really foundational stuff, but that was like the causing the most issues.

30:45 That was the time when we're like, oh, we really got to run all the tests.

30:48 But once you get down to the nitty gritty pretty early on, actually, you really don't

30:53 need to think about how it affects other things.

30:55 Like it's mostly just sort of like, yeah, you know, this module affects its own tests and

30:59 that's pretty much it.

30:59 Okay.

31:00 Yeah.

31:01 I'm sure you get a feel for it over time.

31:02 Like these are the kinds of far reaching changes and these are the kinds of things I

31:06 can stay really focused on.

31:07 This portion of Talk Python to Me is brought to you by User Interviews.

31:13 As a developer, how often do you find yourself talking back to products and services that

31:18 you use?

31:19 Sometimes it may be frustration over how it's working poorly.

31:22 And if they just did such and such, it would work better.

31:26 And it's easy to do.

31:28 Other times it might be delight.

31:30 Wow.

31:30 They auto-filled that section for me.

31:32 How did they even do that?

31:33 Wonderful.

31:34 Thanks.

31:34 While this verbalization might be great to get the thoughts out of your head.

31:38 Did you know that you can earn money for your feedback on real products?

31:42 User Interviews connects researchers with professionals that want to participate in research studies.

31:47 There is a high demand for developers to share their opinions on products being created for

31:53 developers.

31:53 Aside from the extra cash, you'll talk to people building products in your space.

31:58 You will not only learn about new tools being created, but you'll also shape the future of the

32:03 products that we all use.

32:04 It's completely free to sign up and you can apply to your first study in under five minutes.

32:09 The average study pays over $60.

32:11 However, many studies specifically interested in developers pay several hundreds of dollars

32:17 for a one-on-one interview.

32:18 Are you ready to earn extra income from sharing your expert opinion?

32:22 Head over to talkpython.fm/userinterviews to participate today.

32:27 The link is in your podcast player show notes.

32:30 Thank you to User Interviews for supporting the show.

32:32 The other requirement you said that you had was that any changes must be rollback safe.

32:40 Can you speak to that?

32:42 I'm thinking like database migrations or that, right?

32:46 What are you thinking here?

32:47 Yeah, I mean, it's I think database migrations are a good example of that type of thing.

32:51 We didn't really run into a situation where we actually had to do any schema changes to databases,

32:57 although there was a thing where we had to do we had to make some some changes to some data such that it would be parsed properly under both Python 2 and 3.

33:06 But yeah, what you always want to do is you want to say like, okay, if I undo this later, maybe like a week later, someone realizes, oh, this change made a problem, has a problem.

33:16 We don't want to be in this vision where we say, oh, we can't undo that.

33:19 Something else, you know, it depends on it and we can't we can't undo it.

33:23 And so that was like a main thing.

33:25 It was just sort of like, don't do these things where you're just sort of like, oh, once we do this, we can't go back.

33:30 Like, no, don't do that.

33:31 Right.

33:31 If you need to like do some extra work where you like build up scaffolding or whatever, then like do that work instead.

33:38 And it might take a little bit longer in the in the in the long run, but it makes us have less risk.

33:43 Yeah, I'll save diving into this for later in our conversation.

33:47 But one of the things that you were able to do because of that is you were able to run apps simultaneously in two and three and use URL reverse proxy like Nginx or something to say this part of the web app runs Python 3.

34:00 And this one over here is running Python 2 and filter the traffic and switch it based on how it's performing or behaving.

34:07 If it goes wrong, you can switch it back quick.

34:09 If you didn't have that compatibility, it would be like, all right, today we pull the switch junk and then like you deal with the consequences for how long Yelp is down.

34:18 Right.

34:18 So that's an interesting consequence of this idea that it should be able to be roll back rollbackables.

34:25 You can actually run both versions and then sort of migrate more cautiously.

34:29 You had a cool picture and let me put it on the screen for us here where you talked about the four different steps, the phases and timeline.

34:39 And how much time you spent in there.

34:42 Want to talk us through this?

34:43 Yeah, sure.

34:43 So this is just sort of like if you want to think about, OK, you've got some Python 2 code and you want it to get it to Python 3.

34:52 It's very easy to think about it in a sort of atomic way as you just sort of like, oh, make it Python 3 compatible.

34:57 And it's like, OK, it makes sense on small stuff.

35:00 You know, if you're like, oh, I got my 500 lines back and I'm going to migrate to Python 3 today, you know.

35:05 But on big stuff, when you're talking about millions of lines of code, you want to think about it in terms of in sort of level of compatibility.

35:11 And so the three levels that we had to deal with here were parseability, which basically just means if you try and run this module with Python 3, will it fail with a syntax error or not?

35:24 OK. And so that's the main thing.

35:27 And parseability, it turns out, is pretty easy to fix because there were not a lot, huge number of syntax changes.

35:32 And they're pretty easy to detect and fix in an automated way.

35:35 Yeah. Did you use some tooling like PyUpgrade or any of those types of things?

35:40 PyUpgrade we used a little bit.

35:41 It's not super designed for this, but there was a one specific thing that was really nice about it, which is that it could detect octal literals.

35:50 So if you put like zero and then a number that's an octal literal in Python 2, that's not a lot in Python 3.

35:56 So you have to do zero O number.

35:59 And it was able to detect those really easily and like fix them, which was really nice.

36:04 Those things, it sounds like, oh, well, that's not that much work or that much help.

36:07 But when you're doing it across millions of lines of code, anything you can automate, it's got to be really welcome, right?

36:13 There's a relatively common pattern.

36:15 The reason that I remember that one is there was a relatively common pattern where like people would create like date time.

36:20 objects and then they would write year, month, day.

36:23 And if the month or a year was single digit, they would prefix it with a zero, which works in Python 2.

36:30 They probably didn't mean to make it octal, but that's what they did.

36:33 Yeah.

36:33 And so it kind of worked.

36:36 And so people and so that that existed in a lot of places and it was like a popular pattern.

36:40 But yeah, so PyUpgrade was useful in that way.

36:43 It was useful later on when we were like blowing away all the sick stuff because it's able to fix all those things automatically, which is nice.

36:50 Or most of them.

36:51 But Python modernized was where a lot of most of our automation went because it could fix a lot of this stuff.

36:57 Yeah.

36:57 Yeah.

36:57 So that was priceability.

36:58 Importability is similar is that you try to import it and then you say, you say like, okay, this is failing with an import error.

37:06 Like, or something is making it fail to import, like usually running code at the top level.

37:12 And that was a little bit longer.

37:16 A lot of that was fixing standard lib imports.

37:19 You may be making most of those use six shims.

37:22 If they change the six shim for that.

37:24 And then some of it was also upgrading the packages so they could be used under Python 3 and like imported.

37:31 But there was a little bit of like top level stuff where it was like, oh, this like top level thing is like calling like dict.iter items or something.

37:39 And you got to fix that.

37:40 Yeah.

37:40 So that's, that probably gets maybe a little into the functional parity, which if people look at your talk, they'll see there's a couple weeks of the parsability, maybe a month or two of the importability.

37:52 Then a whole bunch of the functional parity.

37:55 And it reminds me of when I was learning C++ way, way, way back.

37:58 And I got really excited because I finally got some complicated code to compile.

38:03 Not really knowing like, oh, no, no, no, no, no.

38:06 You're only the beginning of figuring out what's wrong with this.

38:09 The compile is the part where it shows you what's wrong.

38:11 Now it's like the mystery tour.

38:13 And like, this is after that, right?

38:15 This is like kind of once you get past parsing and importing, then you're into the how are they different behaviorally?

38:21 Yeah.

38:21 And this is, it's just sort of like, I alluded to this earlier, but basically the idea of you run all of your tests.

38:29 And luckily we had already built up a lot of infrastructure that was really useful to us.

38:33 So one of the things that Jolt was able to do is it was able to do some like normalization of like tracebacks and then be like, oh, these tracebacks are like similar enough that I'm going to group them together as like a single error and say like, oh, this many tests are failing this sort of traceback.

38:51 Okay.

38:52 That was really useful because it was, we were able to just sort of be like, okay, here's where the error is and it's going to fix this many tests or at least unblock this many tests.

39:01 Yeah.

39:01 So that was about a year of basically going through all of those test failures and figuring out, okay, why do they file fail under Python three and just fixing them?

39:12 So a lot of it was like, oh, this thing's supposed to be a string, but it's bytes or vice versa or.

39:19 They're calling dot items, but it's, that used to be a list and it's not a list anymore.

39:23 Yeah.

39:24 So you can index it.

39:25 Yeah.

39:26 So there's all sorts of nitty gritty things that you just have to go through and fix them.

39:30 Some of them are automatable, but like you really need to, but not everything is.

39:35 And some of them are more subtle.

39:37 Yeah.

39:37 What was your target Python three version?

39:39 So we originally targeted three, six at the time it was the newest version.

39:45 When we started the project, it was like the newest version that we had available.

39:48 That was that we were like, we're like, you know, we're sort of ready for, if you will.

39:53 Yeah.

39:53 During, during the project was obviously a long project.

39:56 We were able to get three, seven available.

39:58 And it was actually really great because we were like, I don't know, I'm less than a month

40:02 out from when we were like, oh, we're going to start doing the rollout.

40:05 And my coworker, Chris, who wrote Dump API was working on this project at the time.

40:10 And he was like, you know what?

40:11 I bet we could migrate this to Python three, seven.

40:13 And I'm like, go for it.

40:14 Let's see.

40:14 Let's see how hard it is.

40:15 And he did it in like a day.

40:17 So it was just like, it was just like, oh, he just upgraded it.

40:20 It was like, I think there were like maybe a few three, seven does have like that

40:23 one backwards incompatible incompatibility where it makes asymptote keywords.

40:26 So there were like a few packages where he needed to upgrade.

40:29 But like he was able to do it like really quickly.

40:31 And we were just sort of like, okay, and now we're going to roll out to three, seven.

40:34 So that was nice that it was, it was sort of like we were working on three, six for most

40:38 of it.

40:38 We were, we switched to three, seven near the end and it just sort of worked.

40:41 And it says the foundation for going to the next version after that, right?

40:44 It's actually really weird.

40:45 Chris is working on that again.

40:47 He's going to be trying to upgrade us to three, eight like this week, basically.

40:51 Okay, cool.

40:52 That's really, really excellent.

40:53 What was the emotional state of you and the team as you were going through that year of

40:59 fixing?

41:00 Nope.

41:00 It's list of dick dot items, not list dot items or dictionary dot items.

41:05 And probably excited in the beginning, but you know, six months.

41:09 And what was that like?

41:10 Or making progress or like, oh God, it's still here.

41:12 We're not done.

41:13 I knew what it was going to be like going into it.

41:15 Like I was like, I mean, not exactly, but like, I was like, I know that it's going to be

41:19 this thing where it's like, we're going to make some progress and then we're going to,

41:21 it's going to taper off because of the way that these things work.

41:24 And, but it was definitely like, it was sort of like, you were just sort of doing your tasks

41:30 every day and each task in and of itself was not valuable.

41:33 Right.

41:33 It was sort of like, oh, well, I fixed three tests today, you know, kind of thing.

41:37 But ultimately I was able to see like where the end was.

41:42 So for me, I was like, I was like, yeah, we're going to do this.

41:45 We're going to do it.

41:45 I think not everyone on my team was necessarily like as sort of buying the prize as I was, which

41:51 is fine.

41:52 I think we ended up, we ended up swapping out.

41:54 Basically everyone on the team ended up working on it at one point or another, but it was only

41:58 me and, and another one of my colleagues who, who worked on it basically the whole time.

42:03 So I think, you know, part of it was that some people were like, okay, I'll work on that a

42:08 little bit, but I don't want to only work on that.

42:10 And that's totally understandable.

42:11 Like, I think that this type of work is kind of tedious and this is sort of like, this is

42:16 another argument against monoliths is sort of saying like, if you have to do this, when

42:21 you need to do these, these type of migrations, it becomes really punishing on software engineers.

42:25 Like if you haven't done linting ever, and then five years into it, you like, oh, let's

42:30 get, see what's wrong with you.

42:31 We're at the lunch, you're like a hundred thousand years.

42:33 Like, you know what?

42:33 We're not doing that.

42:34 We're just, we're just going to ignore those.

42:35 Let's just stay.

42:36 Cause you can't just stop and go do a hundred thousand fixes.

42:40 And this is, there's more value on the other side of this.

42:42 So it's, it makes a lot of sense, but it must've been, felt pretty good to get it all done though.

42:47 It really did.

42:47 I mean, it was, it's sort of weird how these projects work is that like, you're sort of

42:52 like, you're doing the work, you're doing the work.

42:54 And then like one day you're just sort of like, and we're done.

42:56 And it's been a year and a half of my life, you know, like, but exciting.

43:01 It was, I had multiple people tell me, they said to me, Hey, it's so cool that you, that,

43:06 you know, we were able to do that because I never thought it would happen.

43:10 Yeah.

43:10 It's kind of amazing to do something that some people are like, this won't ever happen.

43:15 But I did, it happened and we made it happen.

43:17 And I think that that was, you know, really great.

43:20 Let's talk a little bit about how you were able to, to run this on Python two and three.

43:25 What do you do?

43:25 You basically create two virtual environments, one from each setup and then each version and

43:31 then run test there, try it out there.

43:34 Yep.

43:34 That's basically it.

43:35 I mean, there is a technique to having like code that runs under Python two and three, which

43:41 is that, you know, you basically have to make sure that you're using compatibility layers.

43:45 And we use six for that.

43:47 I, that was something that me and most of the people on my team had some pretty significant

43:52 and experience doing, because that's basically how we wrote all of our like libraries or internal

43:57 libraries.

43:58 And, and actually a lot of our like source ones as well, because, you know, you want

44:02 for a long time there, you were like, okay, I want to have Python two and three compatibility.

44:06 So having code that works under both was like pretty normal.

44:09 Yeah.

44:09 Making sure that that code is sort of can run under Python two and three and then building

44:14 the virtual one, there was a little bit of nuance or like a, there was a snag there, which

44:18 is that like something that comes up every once in a while when you're doing this kind of stuff.

44:23 And this still, this is still a thing that happens to this day is you have, you have to

44:27 deal with backport packages.

44:28 So there's like the futures backport, which was the concurrent, which like concurrent futures

44:35 was added in might've been three.

44:36 Oh, I don't remember exactly what version of Python three Python it was added in.

44:40 And then there was like a few other backports, functools 32 backport for adding some of the

44:46 stuff in Python three, three twos, functools, like LRU cache, which is something we used a

44:51 lot of.

44:51 So those were both packages where we needed to actually install them in Python two, because

44:56 like there are packages somewhere in our, you know, depth tree that needs them.

45:01 So what we had, what we ended up doing is we, it was, we made this like silly little, little

45:06 script that it just sort of like took our requirements file and then filtered out the things that don't

45:11 under install under Python three and just spit out a new one.

45:14 And that's the one builder Python three virtual input.

45:17 And so that's how we have Python two and Python three virtual and they're like really similar, not exactly the same, but close enough.

45:24 And then we can run, run the tests against either one of them.

45:27 Nice.

45:27 Or, and then eventually we would do, you know, do the rollout.

45:30 It doesn't like one of the challenges had to do with caching and you have a, a way in which you were using pickle to stuff some results into memcache.

45:40 Is it memcached D or memcached like past tense?

45:43 I never know.

45:44 I never know how to pronounce that, that one.

45:46 I looked at their website like a year ago or like, I guess two years ago or something when I was actually working on this.

45:52 And I'm sure that I know, I'm sure that I read it because it said it there.

45:57 I remember, but I don't remember what the answer is.

45:59 Yeah, no worries.

45:59 All right.

46:00 So let's go with memcached.

46:01 I'll call it memcached.

46:02 So you were, you were previously, you were pickling things.

46:06 You were cpickling, but then that just became pickling.

46:09 But at some point, it's one thing to say at the database query level, we'll deserialize and serialize an ORM object to match the schema.

46:17 It's a whole nother to say the binary shape of this thing is the same across Python versions, which is highly unlikely, right?

46:24 Which is pickle.

46:24 It's basically impossible.

46:25 Yeah.

46:26 That was like the, yeah, that was one of the big problems that we had.

46:28 So we're basically taking up, we were basically like, so we'd pickle, there's like a cache key and then there's a cache value.

46:33 We'd pickle both and then the cache key, we would, we would like hash so that it could be like, you know, a specific binary sequence.

46:41 And then that would, and then we'd, we'd key into that, you know, to get stuff out of the, out of the cache.

46:49 And, but it turns out that for a multitude of reasons, both the key, like you said, the key is not going to be binary the same.

46:56 So like, that's one of the problems.

46:57 And the other problem is that there's a lot of like weirdnesses when you end up like either reading Python two pickles in Python three or reading Python three pickles in Python two.

47:08 It seems like pickles are kind of meant to be transient.

47:11 They're not meant to sort of be long-term storage because there's not a lot of guarantees around their parsability.

47:16 Yeah.

47:17 We were like, okay, well, what's, what's a thing that we can do where we don't have to start being like, okay, now we have to write complicated serialization and stuff.

47:26 And we were like, well, probably JSON, JSON will work.

47:29 And so this is something I worked on for about three months or something was just migrating all of our caches to, to use JSON instead of pickles.

47:37 Yeah, you had this kind of fallback mechanism or this slow upgrade mechanism that said, try to get the JSON version from memcache.

47:46 And if you got it, awesome, go with that.

47:48 But then fall back and try to get the binary pickle, but then immediately replace it with the JSON version so that it just grows over time.

47:57 I mean, thinking about that much code and that many services, there must just be a ton of startup cost if you just kick all the servers over and clean the cache.

48:05 We've never tried it.

48:06 I think everyone's a little bit too scared, but it's definitely like not something we wanted to do.

48:11 And we wanted to be able to be like, okay, for when we cut over to Python 3, we're not just going to lose all of our caches.

48:16 I think this is actually a really great example of something we were discussing before the recording started of like doing a sort of like incremental upgrade.

48:24 And one of the other things I didn't super get into with in my talk is that like one of the things that I felt was a really cool technique, and this really depends on whether or not this is worth it.

48:36 It depends on like how you end up, what the value of your like uptime is basically compared to your dev time.

48:42 But what I did is I sort of logged.

48:45 What I would do is I would like for every cache, I'd be like, okay, I'm going to try to log this to JSON.

48:50 And then if it failed, I wouldn't just fail.

48:52 I'd like do the normal stuff.

48:54 I'd do all the pickle stuff, whatever.

48:56 But then I'd log it somewhere.

48:57 And so that way, I could just like look at this log and be like, oh, here's where my errors are.

49:01 So it wasn't just like, oh, I would do I would like ship changes and then like, see if there were actual errors on prediction is like, there's no errors on prediction.

49:10 There's just errors in this log that I can like fix and like iterate on and no, you know, user ever sees a sees a 500.

49:19 Not everything can fit into that.

49:20 But I think that that's a really I think that's a really useful.

49:23 Yeah, that is really cool because no matter how much testing you do on something this big, it's not until you really put it out there, you see that 100% sure it's going to hang together.

49:31 But if it can fail silently in a way that people don't see, but you you get notified about this and can start working on it.

49:39 That's yeah, that's really valuable.

49:41 Another thing that you did that I thought was pretty clever was the way that you did the rollouts where you were able to say, even though this is one huge monolith of code, that doesn't mean it breaks.

49:53 Evenly, right?

49:54 Once you get it past the parsability stage, there could be some URL endpoint that's going to fail if you request it and another that works totally fine.

50:01 All right.

50:02 So what you were what you all did is you created a reverse proxy.

50:06 And I was imagining Nginx.

50:08 What were you actually using here?

50:10 So it's kind of Nginx.

50:11 It's open Resty, which is a framework where you can write plugins for Nginx in Lua.

50:20 So you can you can do some sort of, you know, general logic in that.

50:24 So you were basically able to say, you know, when you go to Yelp dot com slash something or API dot Yelp dot com or whatever it is, as far as a user, it's the same.

50:32 But some of those URLs are hitting the Python 3 version of this large monolith app running and some are hitting the Python 2.

50:39 And you could move it URL, URL endpoint endpoint at a time.

50:44 Right.

50:44 Yeah.

50:45 Talk to us about that.

50:45 That's pretty clever.

50:46 Yeah, this is a super cool technique.

50:49 So we already had the reverse proxy layer.

50:52 We had the routing service.

50:53 This is something that we had built for just sort of consolidating a bunch of logic in a general place where like everything could rely on it.

51:01 But it was a really great place for us to be able to put this logic as well.

51:06 And I'm going to say say him again.

51:09 Chris Keel on my my colleague on my team came up with this idea as well.

51:14 So it's it's such a great idea.

51:16 And it applies, I think, really generally.

51:18 Like you can just sort of say like, OK, anytime I'm doing some sort of rollout where the setup is such a what in such a way that like I can't like do it within my application.

51:27 Like there's something about the application setup you can if you have this external layer, then you can then you can pretty easily do it.

51:34 And yeah, it was basically just sort of like, you know, we would have a configuration and it would say like, OK, this like prefix endpoint or like this endpoint prefix would go to Python 2 or this one would go to Python 3.

51:47 And we can actually even be a little bit more granular that we could actually give it a percentage of the time.

51:51 So basically like, you know, 20 percent of the time it goes to Python 2, 80 percent of the time it goes to Python 3.

51:57 And so we could do these sort of slower rollouts if people wanted to be more careful.

52:01 I see.

52:02 So maybe it goes like it's on Python 2.

52:04 Now 1 percent of the traffic goes to Python 3.

52:07 Is it dying or no?

52:08 It seems OK.

52:09 It seems OK.

52:10 All right.

52:10 Now 20.

52:11 Now 80.

52:12 Like you could like slowly move it over.

52:14 So if it fails, at least it fails just for a few people.

52:18 And you can you don't even roll it back.

52:20 You just stop sending traffic there and fix it, which is really good.

52:24 Yeah.

52:24 Yeah.

52:24 Very clever.

52:25 And it certainly makes sense for large projects.

52:27 But what's great is it lets you start getting your Python 3 version in production way earlier.

52:32 Right.

52:33 You're not waiting on the last endpoint.

52:34 You just need the first endpoint.

52:36 I mean, probably you didn't do this very like one URL works.

52:38 Put it out there.

52:39 But like you could do it much sooner than you would otherwise.

52:42 Right.

52:42 For various sort of practical reasons, we didn't do we didn't want to actually start

52:48 the rollout until we were like, oh, all of the tests pass under Python 3.

52:52 Because we didn't want people to be like, oh, I'm running my tests and they're not passing

52:57 and that's bad.

52:58 And I mean, they're going to like ignore them or try to fix them in a bad way and stuff like

53:02 that.

53:02 But like, I mean, it was like a two month process where we were like from the first endpoint

53:07 to the last endpoint.

53:08 It was like two months.

53:08 And so that was able that was really nice because it was like, oh, we would dissect

53:14 issues and then we would but we would keep rolling out other stuff.

53:17 And then, you know, the teams or we could try and fix the issues.

53:21 And so.

53:22 Very neat.

53:22 That's great.

53:23 All right.

53:24 Let's wrap this up.

53:25 We're getting short on time here.

53:27 You had some some clear benefits, even though you went to Python 3.6, which I think you'll

53:32 see those benefits again if you go to 3.11.

53:34 All right.

53:35 But even so, going from where do you go from 2.7 to 3.6?

53:40 We went from 2.7 to 3.7.

53:42 2.7 to 3.7.

53:43 Right on.

53:43 OK, cool.

53:44 And you said it got faster and used less memory.

53:47 That's pretty good.

53:48 I don't remember the exact numbers.

53:49 It was in my talk.

53:50 I stole from your talk.

53:51 15 to 20 percent speed up and 20 percent less memory.

53:54 That's that's tangible.

53:56 That's right.

53:56 That's right.

53:57 Yeah.

53:57 I remember this is something I didn't mention in my talk, but I thought I think it's kind

54:00 of interesting.

54:01 So we have some stuff that is, you know, more CPU heavy, which we send to what we call VIP

54:11 instances.

54:12 So VIP like containers have more memory and more CPU allocated towards them.

54:19 And so I remember I talked to someone who was involved in like doing a lot of that sort

54:24 of like operational stuff.

54:26 And after that migration, they like looked at numbers and they were like, oh, we can

54:31 now scale down the VIP to what the old normal one was.

54:36 And the normal one is now scaled down even more.

54:38 Oh, that's cool.

54:38 Yeah.

54:39 Which was super cool.

54:40 So it was like super good to do that.

54:42 And I think that beyond just sort of like, oh, this like gave us this immediate or this

54:48 gave us this like outcome.

54:50 It's like we weren't going for this outcome, but I think it really shows how like this type

54:54 of work, like if you're like, you know, I think very often it's easy to look at like

54:58 base level infrastructure work is like, oh, well, it's just maintenance and you just need

55:02 to do it and blah, blah, blah.

55:03 And, you know, it's not really benefiting anything.

55:05 And it's like, no, like we did this and it like saved money on our bottom line, you know,

55:10 and not necessarily everything is going to be like that.

55:12 But like, I think that thinking about base level infrastructure is like it does have a

55:17 benefit.

55:18 It might not necessarily be obvious before you do it.

55:22 But, you know, this is an example of, OK, if you're doing your upgrades, you get to take

55:27 advantage of all the really hard work that all of the people on the Python language team

55:31 have done to make the, you know, make it more efficient and faster.

55:35 Yeah.

55:35 And it probably opens up other possibilities.

55:37 The previous show I just did, which not out yet, so you wouldn't know, but was talking

55:41 about rough the linter written in Rust for Python.

55:45 But, you know, you have this ability to integrate with more modern tools and modern language.

55:49 It's like, oh, if we got to rewrite this section and Rust for that computation, it's trivial

55:54 now where probably it wasn't before.

55:56 I would imagine.

55:57 I haven't tried it in grading by the seven with, you know, things like that.

56:01 But I bet it's not as easy as the new tools, you know?

56:04 Yeah, for sure.

56:04 All right.

56:05 Well, let's leave it here.

56:06 I think that's all the time we got to talk about it.

56:09 But, you know, you must be enjoying it, enjoying work on the projects and the features more now.

56:14 You can just kind of the world is your oyster again.

56:17 Yeah.

56:17 I love using the project that I've been working on actually lately is we've been adding a lot

56:22 of type annotations internally.

56:24 That's, you know, a Python 3 feature right there.

56:26 Yeah, absolutely.

56:27 You can use f-strings.

56:28 You can use type annotations.

56:30 You can start using tools like mypy, not just standard type annotations just for editors.

56:35 But yeah, Pydantic, for example, all those things, right?

56:40 Very cool.

56:40 All right.

56:41 Now, before you get out of here, I got two questions.

56:43 I always ask at the end of the show, if you're going to write some Python code, what editor are

56:48 you using these days?

56:48 I'm a VIN person.

56:49 Do all my development in VIN.

56:52 Right on.

56:52 And then notable PyPI package.

56:55 Give a shout out to Python Modernize.

56:57 But anything that stands out you want to give a shout out to like that?

57:00 I mentioned Python Modernize.

57:01 Great, excellent tool for what it is.

57:03 I also give a shout out in my, I give a shout out to a couple other things in my talk.

57:08 But I think they're definitely worth giving a shout out to still, which is PyUpgrade, which

57:13 we mentioned earlier.

57:14 It has a lot of really nice features.

57:17 But one of the sort of other things is kind of the other half of Python Modernize is that

57:22 it can take your sort of six shim filled code and turn it into sort of normal Python recode.

57:28 And then another tool by the same, same former colleague of mine, Anthony Stille, is pre-commit.

57:35 They're modernized as a pre-commit hook.

57:36 Packrate is a pre-commit hook.

57:38 That's a, that's a thing that we use extensively internally.

57:40 Super, super nice to be able to like, you know, do all those sorts of things and, and, and

57:45 do it in an incremental way, which is something that was really valuable.

57:48 And then the last one, this is just completely random one, but I just love it is more iter

57:54 tools.

57:54 One of my favorite packages.

57:56 We have like an internal package that has a lot of the like functions that are in more

58:01 iter tools, but they're worse or like quirky in some way that I, that I don't like.

58:05 And so that's something that was like, when I first found out about it, I was like, oh man,

58:10 this is great.

58:10 And, and I think it's pretty popular now, but like, I think it's, it's one of those things

58:14 where it's just sort of like, oh, these, all these little functions that you're like, oh,

58:17 I could write that.

58:18 And it's like, you could, but you'd probably write it, you know, with bad edge cases or

58:21 something.

58:22 It's just, it's a great, great library.

58:24 And it's better if you don't have to write it.

58:26 That's for sure.

58:26 Yes.

58:27 All right.

58:27 Well, Ben, thanks for being on the show.

58:29 Final call action.

58:29 Maybe some other people are out there facing this transformation they got to do.

58:34 Like I said, not actually Python two to three, but you know, some major foundation in their

58:38 code base to another, what do you tell them?

58:40 Figure out a way to make it incremental.

58:42 That's really, I think the main takeaway for me is that incremental changes.

58:47 changes have multiple benefits.

58:49 They're making you less risky.

58:51 You're able to do these types of changes in a way like where you don't necessarily have

58:56 to be like, oh, we have to schedule two years of work.

58:58 It's like, no, you can do, you can do it, you know, a chunk at a time when you have time.

59:03 And also it just like generally makes your, you have less errors.

59:08 Absolutely.

59:09 All right.

59:10 Well, thanks so much for being here.

59:11 It's been great to have you on the show.

59:12 Appreciate it.

59:13 Thanks so much for having me.

59:15 This has been another episode of Talk Python to Me.

59:18 Thank you to our sponsors.

59:20 Be sure to check out what they're offering.

59:21 It really helps support the show.

59:23 Join Cox Automotive and use your technical skills to transform the way the world buys,

59:28 sells, and owns cars.

59:30 Find an exciting position that's right for you at talkpython.fm/cox.

59:36 Earn extra income from sharing your software development opinion at user interviews.

59:41 Head over to talkpython.fm/user interviews to participate today.

59:45 Want to level up your Python?

59:47 We have one of the largest catalogs of Python video courses over at Talk Python.

59:51 Our content ranges from true beginners to deeply advanced topics like memory and async.

59:56 And best of all, there's not a subscription in sight.

59:59 Check it out for yourself at training.talkpython.fm.

01:00:02 Be sure to subscribe to the show.

01:00:04 Open your favorite podcast app and search for Python.

01:00:07 We should be right at the top.

01:00:08 You can also find the iTunes feed at /itunes, the Google Play feed at /play,

01:00:13 and the direct RSS feed at /rss on talkpython.fm.

01:00:17 We're live streaming most of our recordings these days.

01:00:21 If you want to be part of the show and have your comments featured on the air,

01:00:24 be sure to subscribe to our YouTube channel at talkpython.fm/youtube.

01:00:28 This is your host, Michael Kennedy.

01:00:30 Thanks so much for listening.

01:00:32 I really appreciate it.

01:00:33 Now get out there and write some Python code.

01:00:35 I'll see you next time.

01:00:35 Bye.

01:00:35 Bye.

01:00:36 Bye.

01:00:36 Bye.

01:00:36 Bye.

01:00:36 Bye.

01:00:37 Bye.

01:00:37 Bye.

01:00:37 Bye.

01:00:37 Bye.

01:00:38 Bye.

01:00:38 Bye.

01:00:38 Bye.

01:00:38 Bye.

01:00:38 Bye.

01:00:39 Bye.

01:00:39 Bye.

01:00:40 Bye.

01:00:41 Bye.

01:00:42 Bye.

01:00:43 Bye.

01:00:44 Bye.

01:00:45 Bye.

01:00:46 Bye.

01:00:47 Bye.

01:00:48 Bye.

01:00:49 Bye.

01:00:50 Bye.

01:00:51 Bye.

01:00:52 you Thank you.

01:00:55 Thank you.