#421: Python at Netflix Transcript
00:00 When you think of Netflix as a technology company, you probably imagine them as cloud innovators.
00:05 They were one of the first companies to go all in on a massive scale for cloud computing,
00:10 as well as throwing that pesky chaos monkey into those servers.
00:13 But they have become a hive of amazing Python activity.
00:18 From their CDN, their demand predictions and failover, security, machine learning,
00:23 executable notebooks, and lots more, the Python at play is super interesting.
00:28 And on this episode, we have Zoran Simic and Amjith Ramanujan on the show to give us this rare look inside.
00:34 This is Talk Python to Me, episode 421, recorded June 8th, 2023.
00:40 Welcome to Talk Python to Me, a weekly podcast on Python.
00:56 This is your host, Michael Kennedy.
00:58 Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython,
01:03 both on fosstodon.org.
01:06 Be careful with impersonating accounts on other instances.
01:08 There are many.
01:09 Keep up with the show and listen to over seven years of past episodes at talkpython.fm.
01:14 We've started streaming most of our episodes live on YouTube.
01:19 Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.
01:26 This episode is brought to you by JetBrains, who encourage you to get work done with PyCharm.
01:32 Download your free trial of PyCharm Professional at talkpython.fm/done dash with dash PyCharm.
01:39 And it's brought to you by InfluxDB.
01:42 InfluxDB is the database purpose built for handling time series data at a massive scale for real-time analytics.
01:49 Try them for free at talkpython.fm/InfluxDB.
01:52 Hey, Zorin.
01:54 Hey, I'm G.
01:55 Hello, Michael.
01:55 Hello, Michael.
01:56 Welcome to Talk Python to me, you guys.
01:58 It's excellent to have you here.
01:59 Thank you very much.
02:00 I'm a big fan, so it's very nice to be on the show, actually.
02:04 Awesome.
02:04 Yeah.
02:05 We've got to meet a couple times at PyCon, which is honestly one of my favorite purposes of PyCon,
02:10 is to meet people and just hang out and have those experiences, you know?
02:14 Yeah, absolutely.
02:15 Yeah.
02:15 Yeah.
02:16 And nice to have you on the show, Zorin.
02:17 Yeah.
02:18 Big fan as well.
02:19 Thank you very much.
02:20 That's very kind to both of you.
02:21 So we're going to talk about a pretty awesome tech company, I think.
02:26 Netflix.
02:27 You both work at Netflix, and people who are watching the video, you're coming to us from
02:31 the Netflix headquarters, which I've got the chance to be there for some Python stuff going
02:36 on there before as well.
02:37 Got cool posters and sort of movie studio feel.
02:40 So that's the backdrop you both have going on, which is excellent.
02:44 Yeah.
02:44 Yeah.
02:45 Yeah.
02:45 It's pretty nice to work at Netflix.
02:48 It's a very good company.
02:49 I'm very happy.
02:50 A lot of Python we're going to learn.
02:52 Yes.
02:53 Yeah.
02:53 We do use a lot of Python.
02:54 Yeah.
02:55 Excellent.
02:56 So we're going to talk about Python and Netflix, a wide ranging sort of survey of a lot of projects
03:01 you all have created.
03:02 How are you using it?
03:03 Some other ones that you both of you personally created, either tied to or not tied to Netflix.
03:09 But I think people are going to really enjoy this look inside what you all got going on.
03:13 Before we get to that, though, let's start with your stories.
03:15 Quick introduction.
03:17 How do you get here working on Python?
03:19 Soren, you want to go first?
03:20 Yeah.
03:21 So I was hooked into programming ever since I saw my first computer.
03:26 Age of 13 in middle school.
03:28 I was in Amstrad CPC.
03:30 Right.
03:30 I was like, yeah, that was the thing I wanted to do.
03:33 So yeah, I started programming as a hobby at first.
03:36 And fun fact, way back then, later on in high school, one of my math teachers told me, hey,
03:42 do something real, don't do programming.
03:44 It's like a dead end, you know, you won't be able to find a job.
03:48 Did they tell you things like these drag and drop visual tools are going to replace all the
03:55 programmers and like the low code of the 80s and 90s, maybe?
04:00 Yeah.
04:00 You know, yeah.
04:02 Back then, I guess it was very, well, didn't seem that obvious.
04:06 Yeah.
04:07 Yeah.
04:07 And then, yeah, I decided to go computer science anyway, because that's what I wanted to do.
04:13 And then I spent the vast majority of my career in a language that is not very much known or
04:19 used, I think, iPhone.
04:21 So I spent more than a decade on doing iPhone mostly.
04:25 And then I discovered Python once I joined LinkedIn in 2011.
04:29 And that's when I kind of, well, got hooked and decided to do more and more things Python.
04:35 And now at Netflix, even more so trying to support NetPython across the board.
04:41 You're kind of doing MetaPython in the sense that your team does a lot of stuff to facilitate
04:47 other people doing Python too, right?
04:49 Exactly.
04:50 Yes.
04:50 Yeah.
04:51 That's our team at Netflix.
04:53 Like we enable other Python developers to be more productive by building tools or building
04:58 the infrastructure necessary to ship their code faster or build their products sooner, things
05:04 like that.
05:04 Yeah.
05:05 Cool.
05:05 How about you, Amjith?
05:06 Oh, I got introduced to programming in high school.
05:10 We had like one hour of computer lab every week.
05:13 I got to learn GW Basic.
05:15 That was my first language.
05:17 It was fantastic.
05:18 I still have fond memories of like trying to draw circles on the screen.
05:22 And then I went to college.
05:23 I learned C and C++.
05:25 I liked those.
05:26 But then after I got a job, I wanted to learn, you know, how to be a better programmer.
05:30 And somebody mentioned, you know, oh, functional programming is the bee's knees.
05:33 You should actually, you know, if you learn how to do functional programming, your general
05:36 programming will get better.
05:37 And the best language to learn functional programming is Haskell.
05:40 And so I took a book called Learn You a Haskell.
05:44 And then I went through like the first few chapters and it was mind blowing.
05:47 It was like a really fantastic language.
05:49 And I got first introduced to a concept of REPL and like trying out like little snippets in
05:55 the interpreter and getting answers.
05:56 And it was fantastic.
05:57 And I got introduced to list comprehension in Haskell and it was just mind blowing.
06:02 It's like, you know, without having to write a, write like five lines of for loop, you could
06:06 just, it's a single line thing.
06:08 And I quickly realized that, you know, you can't find actual jobs writing Haskell or at
06:14 least, you know, not.
06:15 So I figured out like, what's a language that has list comprehension that is actually employable,
06:22 you know, that I could find jobs.
06:24 And that's how I found Python because I came to Python because of list comprehension.
06:28 Oh, awesome.
06:29 Yeah.
06:29 Okay.
06:30 Learn You a Haskell for Great Good, a beginner's guide.
06:33 Is that the book?
06:33 That is the book.
06:34 Yeah.
06:35 And it's actually still available online for free that anybody could read.
06:39 I'm fairly certain.
06:40 And I actually bought like a paper copy of the book.
06:42 It's, it's a good book.
06:43 It's fun one to go through.
06:45 Yeah.
06:46 It looks like it's really got a playful nature to it.
06:48 Yeah, exactly.
06:49 Yeah.
06:50 You know, your thoughts about list comprehension really connects with me as well.
06:55 I guess my first exposure to something like that was LINQ, L-I-N-Q and C-sharp.
07:00 Mm-hmm.
07:00 Which is, it's, honestly, I think it's better than Python list comprehensions.
07:04 I wish Python had just a little bit more.
07:06 Nice.
07:06 A little bit.
07:07 Just one or two things more.
07:09 For example, wouldn't it be nice in a list comprehension if you could specify a sort?
07:13 Mm-hmm.
07:14 Because I find myself often doing a list comprehension and then sorting the thing in the end afterwards.
07:19 But if you could just say order by and give it an expression like you would to lambda, pass a lambda over to a, you know.
07:25 So there's room for more.
07:27 What is it?
07:27 What PEP do I need to write to get sort in a list comprehension?
07:30 I don't know, but I want it.
07:31 Yeah.
07:32 Yeah.
07:32 So I really think that that's a cool language feature.
07:35 And, you know, it's also one of the areas that they're applying some of these speedups in the faster CPython work that's coming.
07:42 And they're doing, you know, list comprehensions for isolation purposes.
07:46 And Python 3 are basically hidden function calls with their own stack frame and variables that you don't actually see, right?
07:53 Mm-hmm.
07:54 You don't write it, but that's kind of the execution level.
07:56 And now they're inlining those to make them a little bit faster.
07:58 Nice.
07:59 I don't know.
08:00 Yeah.
08:00 I think the faster Python team is doing like a fantastic job.
08:04 Like there was a talk that I attended at PyCon, not this year, but the previous year, where they introduced like switch case, how they were doing the case statements.
08:11 It's not the exact switch case, but, you know, coming from C and C++, I knew what switch cases are.
08:15 And when I saw what is possible with the pattern matching, like structural pattern matching in Python, it's like take switch case and then like turn it up to 11.
08:24 And that's what this is.
08:25 And you're right.
08:26 I mean, there is always more that can be done, but I think it's going in a great direction.
08:30 I think it's fantastic.
08:31 Yeah.
08:32 Let's talk about that.
08:33 I mean, we're going to dive into the details of Netflix and stuff.
08:37 But just, you know, this whole Python 3.11, 3.12, these are really big performance improvements coming along.
08:45 Yeah.
08:45 Are you able yet to take advantage of those at Netflix?
08:48 And is that making a big difference?
08:49 You know, like, are you guys still running, you know, 3.8 or are you more closer to the cutting edge in terms of releases?
08:56 So I think one of the advantages here at Netflix is that every team has the freedom to choose the tools that they need to use.
09:03 And it's great and also kind of painful for central teams because now, you know, there is like a bifurcation of all kinds of different versions out there.
09:10 But where I'm going with this is that every team is allowed to choose what is what they need to use in order to get their job done.
09:17 And so my previous team, we were at the cutting edge, like we used 3.11 or we still use 3.11 in the projects that we built and the services that we used.
09:25 And it is a nice boost.
09:26 Like we could certainly see.
09:28 So, for instance, there is like a periodic job that runs and it's like a sort of a cron job that runs every five minutes or so.
09:35 And we had put in like so much optimization so that it will actually finish within the five minutes because we were doing a lot of data crunching and so forth.
09:42 And just so we don't like stack up the cron tasks.
09:45 But when we switched from, I think from like we did jump from 3.9 to 3.11 directly, we didn't we did not like go to 3.10.
09:52 But then when we jumped, it felt like, you know, things that were taking like four minutes were now finishing in like two minutes.
09:58 And it was it was like it was like a huge improvement that you could see.
10:02 And like it felt very rewarding to see that.
10:04 So, yeah, absolutely.
10:05 So every team gets to choose what they want to use.
10:08 And our job as a central Python team that Zoran and I are currently part of is to try and enable people to use that use whatever is the latest that is available.
10:17 So, you know, whatever internal tools that we have, we have to make sure that it actually gets exercised in the latest Python version that got released and make sure that everything is building and deploying as they are supposed to do.
10:29 And so on.
10:29 OK, excellent.
10:30 That's pretty cool.
10:31 That story is speeding up your cron jobs.
10:33 That's.
10:34 Yes.
10:34 That's a non-trivial.
10:35 And it's probably wasn't a lot of work to move from 3.9 to 3.11.
10:39 I know my upgrade path was rebuild some virtual environments on the server.
10:43 And now we're good to go.
10:44 Exactly.
10:45 Yeah.
10:45 Yeah.
10:46 Yeah.
10:47 Yeah.
10:47 So, Ren, do you want to add about that 3.11, faster CPython side?
10:51 Oh, yeah.
10:52 So, yeah, absolutely.
10:54 It's so faster.
10:54 So much faster.
10:56 Yeah.
10:56 The main issue when upgrading is the lack of wheels if you're like stuck on older libraries.
11:02 But we do have like a few numbers.
11:05 Like the most used right now is Python 3.10 across the board, right?
11:08 It will depend on the team, right?
11:10 Everybody is upgrading at their own pace.
11:13 And 3.11 is starting to grow a bit.
11:15 But yeah, most used right now is 3.10 statically.
11:19 Honestly, that sounds really quite good for a company the size of Netflix and how much Python you're doing.
11:26 That's pretty close to pushing the envelope.
11:28 Yeah.
11:29 And there are still some teams that are sort of stuck on 3.8 or 3.7, I want to say.
11:34 Simply because they provide a platform that allows data scientists to write their code.
11:40 And they have this pre-built image with all of the necessary libraries pre-installed in there.
11:47 And so they have like a pretty tight control over which libraries will get upgraded on what cadence and so on.
11:53 And so for them, I think they're still running on 3.7.
11:57 And I'm sure when they switch to 3.10 or 3.11, it's going to be like a screaming fast improvement.
12:03 So I'm looking forward to that migration to happen.
12:06 Yeah, excellent.
12:07 This number is very static, right?
12:09 It's a number of like pre-short Pythons across repos.
12:13 But yeah, dynamically, right?
12:15 Like you may have lots of instances who still run on 3.7 and they will massively move to a...
12:20 So that team is moving from 3.7 to 3.10, for example.
12:24 Right.
12:24 Yeah.
12:25 Yeah, so upgrade paths.
12:28 This portion of Talk Python to me is brought to you by JetBrains and PyCharm.
12:32 Are you a data scientist or a web developer looking to take your projects to the next level?
12:37 Well, I have the perfect tool for you, PyCharm.
12:40 PyCharm is a powerful integrated development environment that empowers developers and data scientists like us to write clean and efficient code with ease.
12:49 Whether you're analyzing complex data sets or building dynamic web applications, PyCharm has got you covered.
12:55 With its intuitive interface and robust features, you can boost your productivity and bring your ideas to life faster than ever before.
13:02 For data scientists, PyCharm offers seamless integration with popular libraries like NumPy, Pandas, and Matplotlib.
13:09 You can explore, visualize, and manipulate data effortlessly, unlocking valuable insights with just a few lines of code.
13:16 And for us web developers, PyCharm provides a rich set of tools to streamline your workflow.
13:20 From intelligent code completion to advanced debugging capabilities, PyCharm helps you write clean, scalable code that powers stunning web applications.
13:29 Plus, PyCharm's support for popular frameworks like Django, FastAPI, and React make it a breeze to build and deploy your web projects.
13:38 It's time to say goodbye to tedious configuration and hello to rapid development.
13:43 But wait, there's more!
13:44 With PyCharm, you get even more advanced features like remote development, database integration, and version control, ensuring your projects stay organized and secure.
13:53 So whether you're diving into data science or shaping the future of the web, PyCharm is your go-to tool.
13:58 Join me and try PyCharm today.
14:00 Just visit talkpython.fm/done-with-pycharm, links in your show notes, and experience the power of PyCharm firsthand for three months free.
14:11 PyCharm.
14:12 It's how I get work done.
14:16 Let's start by talking about kind of the broad story of Python at Netflix.
14:23 Maybe we could start with what you all do day-to-day in terms of what's your role?
14:29 Because you kind of support other people's Python, as I hinted before.
14:33 So maybe we could get a sense of what you all do day-to-day.
14:35 And then we'll...
14:36 Omg, you wrote a nice blog article that's a big, broad, pure survey of how Python's being used in all these different places.
14:43 So maybe start with what you all do day-to-day on your team, and then we'll go into that.
14:47 Yeah, sure thing.
14:48 I've been with Netflix for about six years now.
14:51 And previously, I was in a different team, and we were doing failovers, which was a way of running...
14:57 You know, if Netflix ever goes down in one of the AWS regions, we are the team that gets paged in,
15:02 and we go and move all the traffic from that region to another...
15:05 Other two regions that we run in.
15:06 So that's what I was doing up until, like, February of this year.
15:10 And let me just take a step back real quick with you.
15:13 Netflix is kind of all in on AWS, right?
15:16 Like, there's been a lot of stories about how you all have set loose the chaos monkey into your data centers
15:22 and how you worked on failover from AWS regions.
15:26 And so I don't know if you all are the largest users of AWS, but certainly one of the more interesting, complicated deployments out there, right?
15:34 Yeah, so I think we were the earliest adopters of cloud computing when AWS first came out.
15:40 And so AWS has used as the poster child for, you know, see, big companies can run in cloud,
15:46 and you don't have to be on-prem.
15:48 And so we think of them as partners, not so much as, you know, like this client-owner relationship or anything like that.
15:55 So we consider AWS as our business partners.
15:58 And yes, we are full in on AWS.
16:00 And Chaos Monkey, even now, yes, it functions in AWS.
16:05 Like, it goes around, and just inside our VPC, it does terminate instances,
16:10 occasionally, or not occasionally, like once every day, one instance every day on every service.
16:15 That is so wild.
16:17 I mean, obviously, you don't want to set it loose on other people's AWS instances, right?
16:22 Yeah.
16:23 But no, that's a really interesting way to force people to think about developers and infrastructure folks to think about what happens if the cloud somehow, your server dies.
16:34 And maybe it's not even the cloud's fault, right?
16:35 It's just like, okay, there's a Linux machine running, and that thing died.
16:38 It could have been running anywhere.
16:39 It happened to be in AWS.
16:41 But to force them to think about it going, like, we will.
16:43 It's not an eventuality.
16:45 This will happen.
16:46 And so you plan for it.
16:47 Yeah.
16:48 It's even more than just the idea of, like, it will happen, so we plan for it thing.
16:53 It's more like, you know, it's a way of building software where you need to build software that's resilient and has enough fallbacks built in.
17:00 And so, for instance, if you are not able to reach the database, do you have a cache in front that can sort of, you know, keep the thing going for the few network calls that are failing to reach the database?
17:09 Those are, like, basic common things, paradigms that have become commonplace nowadays in software development where, you know, building fallbacks automatically is like standard practice these days.
17:20 But when Chaos Monkey was created, which was about 10 years ago, you know, these were, like, new concepts that people were not using.
17:26 And it was assumed that, you know, once you have a server and you put your software on the server and you run it, it's basically done, like, until you do the next deploy, which takes another month or so to refresh that server, refresh that code.
17:37 But that all changed once we went to cloud where, you know, we started doing deployments on a daily basis or maybe even more hourly basis and things like that.
17:44 And so when you are doing that, you know, when you are shutting down one server with old version and bringing up the new server with a new version, how are you going to make sure that the connections are not going to fall?
17:54 And how are you going to make sure that the network continuity continues and so forth?
17:57 So, yeah, Chaos Monkey was just introduced as a way to ensure that people are building software in a way that is resilient.
18:04 And this is just a way to sort of test that on an ongoing basis.
18:09 Yeah, it's quite an operational challenge.
18:11 I mean, I don't recall seeing Netflix saying, our scheduled maintenance is coming up on Sunday.
18:15 We'll be down for five hours.
18:17 Not acceptable, is it?
18:19 It just makes you laugh when you think about it.
18:21 Especially not on a Sunday.
18:23 Okay.
18:23 I've even seen government sites.
18:26 I can't remember which government it was saying that the website was closed.
18:29 Like the website had business hours.
18:31 That's a different deal.
18:32 Like you came at night, like, oh, you can't come here right now.
18:34 It's like, what?
18:35 Yeah.
18:35 It's the web.
18:36 I don't understand what's going on.
18:37 All right.
18:38 So let's go through this blog post that you wrote here entitled just Python at Netflix on the Netflix technology blog.
18:45 Yeah.
18:45 So you wrote this in preparation of PyCon.
18:48 This is PyCon 2023?
18:49 No, this was 2019, actually.
18:52 So this is old by at least two or three days, three years now.
18:56 Okay.
18:56 Yeah.
18:57 You had pointed out before we press record that some of these projects mentioned here that used to be internal things are now also open source.
19:03 So there's a little more access to these than the blog posts might indicate.
19:07 Yeah.
19:07 Some of the things that are mentioned here, yes, they have been open source since then.
19:11 So specifically the one that I remember right now is Metaflow, which is a infrastructure.
19:16 It's like a platform orchestration infrastructure framework that is used by our machine learning organization where, you know, scientists would try and build their model or they use existing models from like XGBoost or like tons of other Python libraries.
19:33 And their interest and their expertise lies in, you know, crafting those models, training those models and building the correct algorithm to do the predictions and so on.
19:43 They are not so interested in like, you know, making sure that enough compute is available to run these models or they're not interested in making sure that the plumbing works or this model's data is now going to the next step of this algorithm or even like getting it deployed and making it available in the production environment.
20:00 So that's all that abstraction is taken care of by Metaflow.
20:04 So Metaflow is the project that was mentioned here and that allows you to make it easy for machine learning folks to get their system running and as well as deploying it out to production.
20:14 And now that is now open sourced and it is available for folks to use.
20:19 And I think some other companies have actually adopted to using that as well.
20:22 So, yeah.
20:23 Kind of operate like a DevOps-y automation for machine learning.
20:28 So the people they're writing, creating the models and the data scientists don't have to also be DevOps people.
20:34 Right.
20:34 It's slightly more than DevOps as well because it also does the pipelining work to make it possible for someone to, you know, bring the data from this database and load it in.
20:45 All of that work is already taken care of or at least there are libraries that are built into Metaflow that makes it possible to bring those in.
20:50 And then it allows you to also do orchestration.
20:53 So, for instance, machine learning models typically happen in multi-steps and multi-stages.
20:57 And so the data gets processed by this function and then it gets moved on to this other function and then it gets moved on to this other thing and so forth.
21:04 And so it does the plumbing to make sure that the data can flow through this topology and actually produce results and so on.
21:10 Yeah.
21:11 Yeah.
21:11 You probably have enough data that that's a lot of data to move.
21:14 All right.
21:16 A quick question from the audience before we dive into the topics here is Diego asks, on such a big platform with so many software engineers with different coding practices, do you all get together and follow some set norms by Netflix or is it more team by team basis?
21:30 It is very much team by team basis.
21:32 So each team has their style and their areas that they focus on.
21:35 So, for instance, like machine learning engineers are not going to care too much about like, how do I make this like, you know, production grade, like super heavily fortified or whatever.
21:45 And security engineers might be focusing on completely different things.
21:48 So it is different.
21:49 But at the same time, I do want to mention that there are certain norms that are common across the entire company where, you know, so for instance, Chaos Monkey is one of those things where since since Netflix operates in a way,
22:00 where, you know, every team is given the freedom to choose and operate the way they see fit, there is no edict that can come from a VP or a president that says, like, you must write code in this way, like that doesn't happen.
22:12 And so what that means is, how are you going to enforce, like, you know, you have to write resilient software or how are you going to make sure that your software will continue to run if one of the servers out of the hundred servers has gone down?
22:24 And so there is not a good way to enforce that.
22:26 And Chaos Monkey was created as a way to enforce that, which is, yes, we're not going to be able to tell you how to write software, but this particular service that exists, it's going to go around killing servers.
22:36 And so you better make sure that your software is actually resilient to servers going down.
22:40 So that's a way in which we influence people to write the, to produce the right outcome without telling them how to do it.
22:48 I see. So sort of, you agree on a common principle of design for failure and design for resiliency, and then it's up to people how to make that happen.
22:58 Yes. And also we have the concept of paved paths, our paved road, which is we have certain libraries that are made to operate within our infrastructure.
23:08 So there is an internal discovery tool and there is an internal metrics collection tool and there is an internal, you know, like a failure recovery tool and so forth.
23:15 And these libraries that are provided in these languages, they make it really that simple to just integrate with these, with these services.
23:23 And so it makes it the obvious choice for people to start using those libraries rather than, you know, paving their own path, for instance.
23:30 Right.
23:30 So we try and make it as easy as possible to do the right thing.
23:34 And so people generally like fall into that paved road solutions that we have.
23:38 Excellent.
23:39 And we try to make it also now, especially as a central Python team to promote good practices, right?
23:46 Like you should have a pipeline, you should choose a release strategy.
23:50 You should have tests and we, we help.
23:53 If you don't, we can help you set that up and choose a good relevance release strategy for you, et cetera.
24:00 Excellent.
24:00 Yeah, that's, that's really good.
24:01 So let's dive into this blog post.
24:03 Now it was written by Amjit, but Soren, jump in as well as, as we talk about, please.
24:08 So the first one is related to bandwidth.
24:12 To some what?
24:14 Like delivering the content.
24:15 And there's some interesting articles and stuff.
24:18 How much of the internet's bandwidth does Netflix use?
24:21 And I don't know how accurate this is, but maybe give us a sense of like, you got to have a lot of traffic, right?
24:26 Yes.
24:27 So I think when I first joined Netflix, I was told that we use about one third of all of internet's bandwidth, but that was back in 2017.
24:35 So things have changed quite a bit since then.
24:38 Our use of bandwidth is slightly interesting in the sense the actual, when somebody goes to their website and they're browsing around, all of that data is served directly from AWS servers.
24:50 And so we have servers running in AWS that does the search functionality, the thumbs up, the thumbs down, the, you're selecting something and reading the review or looking at related things and whatnot.
25:00 But as soon as they click on the play button on a particular video, the actual video file itself is not streaming from AWS, but instead it's coming from a CDN called Open Connect.
25:12 And this is a proprietary thing that we built where we ship these CDNs to various internet exchanges that are already, you know, filled with the right videos and they get populated with the correct, correct videos that are getting released like overnight or on a regular basis.
25:30 The reason we do that is because we want the videos to stream from the closest possible place for the end user.
25:36 And so when a end user in Florida clicks on it, it's coming from an internet exchange that is located in Florida.
25:42 And that's why you don't see a lot of buffering when videos are playing from Netflix is because there's, you know, it's inside their, their network to a large extent.
25:50 That's our Open Connect team.
25:51 And that's, that's what they do.
25:53 And yeah.
25:53 Yeah.
25:54 That's CDNs are awesome.
25:56 And they really are just, they're kind of a bit of magic performance.
25:59 performance dust you can sprinkle on, on sites that works for SS, for CSS and JavaScript and stuff.
26:08 But, but when it comes to large content, then it makes all the difference.
26:11 Yeah.
26:12 So in the blog post you write, let's see.
26:15 Yeah.
26:15 It says various software systems are needed to design and build and operate the CDN infrastructure.
26:20 And a big part of them are written in Python.
26:22 The network devices that underlie a large portion of it are mostly managed by Python and so on.
26:28 Give us a sense of where Python fits in this.
26:30 Right.
26:31 Open Connect CDN that you all run.
26:33 Sure.
26:33 Yeah.
26:33 So the CDNs themselves run like high performance code to, to stream the video.
26:38 That obviously that software is not written in Python, but the software, all the software that orchestrates
26:43 and makes sure that these CDNs are remaining healthy, getting metrics out of them, as well as,
26:48 you know, managing them and forecasting, like what sort of videos are going to be going into these,
26:53 these CDNs and so forth, those are all orchestrated using Python applications.
26:57 So these are all internal tools.
26:58 There's like an OC tools team.
27:00 OC stands for the Open Connect, which is the name of the CDN.
27:03 And OC tools team is the one that builds that.
27:05 And they use quite a lot of Python for not just tracking our CDNs, but also for projecting, you know,
27:12 which videos and what shapes they should be going into.
27:14 So for instance, like to give you a quick example, like if we are launching, let's say like Stranger Things,
27:18 like the newest season, we know for a fact that these videos are going to be, you know,
27:22 they're either going to be streamed like 90% of the time from television, like a 4K definition television,
27:28 or people are going to be watching on their iPhone.
27:31 So all these videos get encoded in different formats, like for different resolutions.
27:36 And how much do we put into the CDNs and how do we get them prepared?
27:40 Do we need like multiple copies so that multiple streams can be read without having to have contention and so on?
27:46 Things like those kinds of projections, those are all done using Python applications.
27:50 You probably can't put every version of every video at every location all the time, right?
27:56 I don't know how much that is, but that's a large amount of video content, large amount of files.
28:00 You probably got to predict, right?
28:02 Like these, we can fall back to, you know, letting them stream from some higher upstream thing,
28:08 but then it'll get cached after it gets viewed a little bit.
28:10 But these were preloading, right?
28:12 Yeah, yeah.
28:13 Actually, Zoran used to work on the team that did all the encoding in different shapes and sizes,
28:18 and they used quite a bit of Python as well.
28:21 He'd be able to tell you more about that stuff.
28:23 Yeah, did you just have like a huge office, like a whole building full of GPUs and just go in the whole time?
28:29 Well, encoding is a lot of work.
28:31 Yeah, tell us about this.
28:32 Yeah, encoding is a lot of work.
28:33 So that was my original start here, and we do a lot of Python as well.
28:37 And yeah, we sum it up.
28:39 We kind of try and scour, scavenge as many instances that we can put our hands on.
28:44 So if we have any, say, AWS reservations that it so happens that nobody's using right now,
28:51 we come and grab them and spawn our workers dynamically on it as much as we can.
28:59 Interesting.
28:59 Almost like grid computing, like a steady at home.
29:02 Yeah, exactly.
29:03 Like steady at home.
29:04 Exactly.
29:04 Yeah.
29:04 And if we do have something like that is high priority, well, you know, there's not enough
29:09 like kind of workers laying around, then we can go and get some on the spot, you know, market
29:15 or, well, get to grab more reservations if need be.
29:18 So that is the encoding is basically we take these big master files, right?
29:24 Like we have these originals, and we encode them for every single variation where it makes
29:29 sense.
29:29 Like for this TV, for that phone, for, you know, Android phone, iOS phone.
29:33 What is the product of all the different resolutions and different platforms?
29:36 How many video files do you have to make for how many formats do you have to have for one
29:41 movie?
29:42 Do you know?
29:43 That changes per lead.
29:44 And, you know, we kind of keep fine tuning how we want the smallest files with the best
29:51 quality, right?
29:51 So that keeps evolving.
29:53 And sometimes we re-encode the full catalog because now we have like a better way of encoding,
29:58 say, anime things versus, you know, action movies versus like it gets to us.
30:04 I see.
30:04 You might choose a different encoder for a cartoon-like thing versus the planet Earth type
30:10 of video.
30:11 Yeah.
30:11 Okay.
30:11 Yeah.
30:12 Yeah.
30:12 And all of this, basically by way of a product of all of this, ends up on Open Connect.
30:17 I mean, S3, but also Open Connect.
30:20 Yep.
30:20 Yep.
30:20 Excellent.
30:21 One thing in there that is mentioned on my team, very interesting project called vMath.
30:26 So that is written in Python.
30:27 It's machine learning.
30:28 And once you have encoded, right?
30:31 Like, let's say you're trying a new way of encoding to make the files even smaller, right?
30:36 You want to know during while you're researching, right?
30:40 You want to know, did you come up with a very good, better encoder than before?
30:44 So vMath is like a little bot that will look at encode a new file and give it a human-like,
30:51 score.
30:52 Like, what quality would the human assess this to be?
30:56 And it has to be, you know, basically excellent quality.
30:59 Get a high score.
31:01 I think 90 out of 100, roughly, to pass.
31:06 And then this is better, right?
31:07 Like, we have a smaller file, but the quality is still excellent.
31:10 And perceptibly, it's as good as before.
31:13 It was just a fairly smaller.
31:14 Then we could decide and re-encode the full catalog with that.
31:18 Yeah.
31:18 I see.
31:18 That's really interesting.
31:19 So what you're telling me is you have an AI that you just make watch Netflix movies all
31:24 the time.
31:25 All the time.
31:25 All the time.
31:26 All the time.
31:26 And we have other AIs that watch the whole catalog, for example, and find where text appears,
31:33 say.
31:33 You know, so that when we put subtitles, we can move them up or down, you know, to not put
31:37 text on text.
31:38 And all kinds of metadata, like where can we find landscapes?
31:42 Where does Brad Pitt show up?
31:43 Things like that.
31:44 Incredible.
31:46 I had no idea.
31:47 People are always full of a lot of surprises.
31:49 This portion of Talk Python to Me is brought to you by Influx Data, the makers of InfluxDB.
31:56 InfluxDB is a database purpose built for handling time series data at a massive scale for real-time
32:03 analytics.
32:04 Developers can ingest, store, and analyze all types of time series data, metrics, events,
32:09 and traces in a single platform.
32:11 So, dear listener, let me ask you a question.
32:14 How would boundless cardinality and lightning-fast SQL queries impact the way that you develop
32:19 real-time applications?
32:20 InfluxDB processes large time series data sets and provides low-latency SQL queries, making
32:26 it the go-to choice for developers building real-time applications and seeking crucial insights.
32:31 For developer efficiency, InfluxDB helps you.
32:34 Create IoT, analytics, and cloud applications using timestamped data rapidly and at scale.
32:40 It's designed to ingest billions of data points in real-time with unlimited cardinality.
32:46 InfluxDB streamlines building once and deploying across various products and environments from
32:52 the edge, on-premise, and to the cloud.
32:54 Try it for free at talkpython.fm/InfluxDB.
32:58 The link is in your podcast player show notes.
33:01 Thanks to InfluxData for supporting the show.
33:06 And I think the VMAP software that's written in Python, I believe that is open source, right,
33:11 Zoran?
33:11 It is.
33:12 It is open source, yes.
33:13 And I think it's one of the Emmy Award winning software.
33:16 I did not know that software could win Emmy Awards before this one.
33:20 And it apparently won an Emmy Award for something.
33:25 Videography or something?
33:28 Probably, yeah.
33:29 Wow.
33:29 That's awesome.
33:30 All right.
33:31 The next major section is demand engineering.
33:34 Yeah.
33:34 This is kind of like DevOps type stuff, right?
33:37 Keeping things running, capacity planning.
33:39 Yes.
33:40 That is exactly right.
33:40 Yeah.
33:41 That was the team that I was in previously.
33:43 And the regional failover is the one where I mentioned where you could traffic from one
33:47 of the AWS regions into the other two regions.
33:49 And so we run in three separate AWS regions.
33:51 And anytime any of those regions is having a difficulty, we can easily move the traffic
33:56 to the other two regions without users even noticing that there was a glitch or any kind
34:00 of issue there.
34:01 How long does it take?
34:03 If you say you've got to move 50% of the traffic out of U.S. East Virginia to somewhere else,
34:10 is that hours, minutes?
34:12 So the fastest we have done is, so on average, it takes about seven minutes.
34:18 Do all of that.
34:19 And that was our target.
34:20 So when I first joined, I was given as a target.
34:22 It used to be around 45 minutes at the time.
34:24 And we built some, you know, interesting things to make it possible to run it inside seven minutes.
34:29 But the fastest we've done is like around five minutes in like an emergency where, you know,
34:33 oh God, the entire region is tanked and people in the U.S. are not happy about this.
34:38 Let's move as fast as we can.
34:40 And we can do it in five minutes.
34:41 Doesn't happen often, but you know when it happens, especially, you know, when AWS Virginia
34:47 goes down because a quarter of the internet stops working.
34:50 Sure.
34:51 But it's not just AWS that goes down.
34:54 Sometimes we shoot our spells in the foot.
34:58 One of the interesting things to make sure that we release software that is safe is we do something
35:02 called regionally staggered releases.
35:05 And so when a new software or when a new version gets released, since it's like hundreds of microservices
35:10 that are running inside of Netflix to make it all possible, every service will deploy.
35:14 And when they start to deploy, they deploy it into a single region, wait about like five to 10 minutes
35:20 to make sure that, you know, nothing bad has happened.
35:21 And then they proceed to the next one and then the next one.
35:24 And so when they release it to the first region, they can either, if they find out that it's
35:29 bad, they can either quickly roll it back or we could just evacuate out of that region because
35:34 we can do that in like under seven minutes.
35:36 And so if the rollback takes longer than seven minutes, then a call will be made by the core
35:41 team, which will say, let's evacuate out.
35:43 We haven't figured out what the problem is.
35:45 So and then, you know, we evacuate and then we'll debug, you know, oh, which service did
35:49 a release and what do we need to roll back and so on.
35:53 Because there are like hundreds of services that are simultaneously releasing at the same
35:57 time.
35:57 So it's like quickly trying to identify which service that we need to roll back can sometimes
36:01 be tricky.
36:01 So we have used failovers for that as well.
36:04 Yeah.
36:04 So it's not just AWS's fault.
36:06 Yeah, sure.
36:06 And I don't mean to pick on AWS because all these data centers go down.
36:11 The difference is when AWS goes down, like the internet goes down, you know, the observability
36:17 of it is so high because so much runs on there.
36:20 It's like that in Cloudflare when they go down too.
36:23 You're like, oh, I see everything's broken.
36:24 Okay.
36:24 Yeah.
36:25 When sites go down in production, even for places way smaller than Netflix, it's really
36:30 stressful and you might make it worse by trying to fix it.
36:34 So the ability to just go, let's buy ourselves some time to figure this out and just get everyone
36:39 out and then we're going to look at it.
36:40 And then we'll get bring them back.
36:41 That's, that's pretty cool.
36:42 You did write an article called how Netflix does failovers in seven minutes flat, which
36:47 I'll put in the show notes so people can read more about that if they want.
36:50 Thanks.
36:50 So this demand engineering side, talk about obviously tools are primarily built in Python
36:57 there.
36:57 You got some NumPy and SciPy and even the B Python shell.
37:02 Tell us about some of the Python stuff going on here.
37:04 Before I joined Netflix, like when I actually first started learning Python, I loved the REPL,
37:09 but I always felt like the REPL did not have auto-completion in it.
37:13 And that like B Python is a, is a alternate REPL for Python that provides you with like
37:18 auto-completion and syntax highlighting and all that stuff.
37:21 So I am a, I'm a huge fan of B Python.
37:23 One of the things that we have done, like demand engineering specifically is, you know, we get
37:28 paged and we have to go in and try and rescue our traffic out of that region into the other
37:33 two regions.
37:34 And sometimes our software itself will not work because if an entire region is down,
37:39 let's say it's because of a network connectivity issue or something, then the things that we
37:43 call out to in order to make these, you know, changes to, to scale up the other regions and
37:48 like evacuate and make the NS changes or whatever, that itself might be broken.
37:51 And when that's broken, like literally SSH into the box and we will open up like a shell,
37:57 Python shell and do whatever we need to do.
38:01 That has not happened in like the last four years, I would say, but six years ago, yeah,
38:05 that was the thing that we used to do.
38:06 And I wanted to call out B Python specifically in this, in this particular case, because it was so
38:11 much more useful than trying to remember, oh, I remember I wrote this function.
38:15 Like, what is it?
38:16 Like, instead of like opening my IDE to try to find out what that function is, like I just
38:19 import the module and then I do the module dot and it lists me all the functions and I could
38:24 invoke it.
38:24 And yeah, it's such a time saver.
38:26 Yeah.
38:26 The Python REPL is cool, but it, it leaves a lot to be desired in terms of.
38:30 Right.
38:31 History.
38:31 Or even if you want to edit a function that is five lines long, it's, it's, it's, it's
38:38 hard to go through.
38:39 It becomes cumbersome.
38:39 Another one is PT Python that I'm.
38:42 Yeah.
38:42 I'm also a fan of that one.
38:44 Yes.
38:44 They're kind of the same category, right?
38:46 Yeah.
38:47 Prompt Toolkit, the one that powered PT Python, written by Jonathan Slenders, actually.
38:51 And it's like a fantastic library.
38:54 I just like kudos to, to Jonathan for doing that.
38:57 That was, it's a fantastic library.
38:58 Yeah.
38:59 Awesome.
39:00 So are you, you got a particular enhancement there for your, your REPL?
39:05 I'm not like that big of a user of REPL.
39:07 In the terminal, we do like, you know, ask questions for generating new projects, et cetera.
39:12 I'm much more of a PyCharm user myself.
39:14 Like I go in the computer over there.
39:16 As you bring that up, you know, one of the really nice Python REPLs is the, what I guess
39:20 it's called probably the Python console in PyCharm, right?
39:23 Because if you go to that, you get the Python REPL, but you get PyCharm's autocomplete and,
39:28 you know, type consistency and it automatically modifies the path to import your project.
39:33 So yeah.
39:33 And like, you got one in there.
39:34 Yeah.
39:35 Just that one's yours.
39:36 All right.
39:36 Let's see the core team alerting and statistical work.
39:41 What's this one about?
39:42 Core team is our frontline SRE.
39:45 So demand team is like building tools that the core team will leverage to, to get us out
39:50 of trouble.
39:50 So core team is the one that, you know, anytime there is like, they monitor a lot of metrics,
39:54 not just streaming metrics, but also things like error rates between services that are happening
40:00 and how many requests are successfully coming back and so forth.
40:03 They obviously use Python to kind of keep tabs on, like obviously a person can't be sitting
40:08 in front of a dashboard, just monitoring it themselves.
40:11 So they use quite a bit of Python to analyze the data from all of the hundreds of microservices
40:16 and between them, the inter-process communication that actually happens and the metrics that
40:20 come through and so forth.
40:21 So they use Python for alerting.
40:23 And so actually they use the monitoring.
40:25 The next section that I was, that's right there is a monitoring, alerting and auto remediation.
40:29 We have an internal observability organization that has built our own time series database that's
40:36 not in Python, but you know, it's open source called Atlas.
40:39 And that uses, that collects all of the time series data from all of these services.
40:44 And then they try and do alerting and remediation, auto remediation.
40:48 So when a particular alert condition is met, they can, you can run a small Python script inside
40:54 of a framework called Winston.
40:56 That's again, internal that allows you to do more complicated things.
40:59 So for instance, if you have like this one bad instance in like this, this collection of
41:04 20 instances, instead of a user going and terminating that instance, you can now automate that by
41:09 writing a script that says, you know, automatically restart that, that instance or just kill it.
41:14 And so on.
41:15 That's, that's our.
41:16 Cool.
41:16 That's part of the auto, auto remediation of it.
41:18 And it says it's built on G unicorn flask and flask rest plus.
41:24 I'm familiar with the first batch, but the flask rest plus, this is new, an extension for flask that adds
41:30 support for quickly building rest APIs.
41:33 Okay.
41:33 Interesting.
41:34 Because flask itself already does rest.
41:36 I don't like, so rest plus I think provides things like a swagger endpoints automatically.
41:40 So you could, you know, try it out on the browser and so on.
41:44 I have not used flask rest plus myself, but that team uses it quite a bit.
41:48 Yeah.
41:48 Cool.
41:49 Probably some of the, some, some similarities to like what FastAPI kind of brings in addition
41:54 to standard flask, I'd imagine.
41:56 Exactly.
41:56 Yeah.
41:57 Yeah.
41:57 You use more FastAPI nowadays.
41:59 Yes.
42:00 Oh yeah.
42:00 Yeah.
42:01 We're using quite a bit of FastAPI in most of our internal tools actually.
42:04 Yeah.
42:05 Just from reading through this article, it sounds like there's a lot of APIs and just a lot of
42:10 connectivity through.
42:11 There's probably a lot of JSON going around Netflix.
42:14 Yes.
42:14 Yeah.
42:14 So the, some of the heavy, heavier data stuff or like high streaming services like that
42:21 are in the streaming path are all typically written in Java and they use for inter-process
42:25 communication.
42:25 They use gRPC and that uses protobuf to communicate and so forth.
42:30 But most of our internal tools that are written in Python either use JSON directly or sometimes
42:36 they need to talk to a gRPC service.
42:38 And so they use Python gRPC to get the work done.
42:41 Maybe we'll have some time to come back to gRPC.
42:44 I'm not sure.
42:44 We got a lot of things to talk about here.
42:46 Yeah.
42:47 We don't have to go through every section here.
42:49 I mean.
42:50 No, I know.
42:50 But there's, there's just so many interesting angles.
42:52 Right.
42:53 And so.
42:53 Yeah.
42:53 The next one here is information security, which obviously if you just put anything on
42:58 the internet and just tail the log of it within minutes, you'll see a request for,
43:03 you know, WP admin.php.
43:05 You know, like it's just, it's already just constantly being, you know, people are just after it.
43:11 Right.
43:11 Yep.
43:12 Yep.
43:12 One of the things you have here that looks interesting is security monkey written in Python,
43:17 which is, I guess, like chaos monkey, but.
43:20 It is kind of like chaos monkey.
43:21 I think this project may have been archived or it's not actively in development.
43:28 It tries to scan our infrastructure for unsafe practices.
43:31 That's like a, an umbrella term to try to add like whatever is like good practices that
43:37 should exist for, for, from the security standpoint.
43:39 Yeah.
43:40 Okay.
43:40 So people can check it out.
43:41 Maybe it's not totally active anymore, but they can take it as inspiration.
43:44 Right.
43:45 Yeah.
43:45 Yeah.
43:45 Yeah.
43:45 Yeah.
43:46 Like back in 2019, it was one of our most active projects.
43:48 But apparently 2023 is a different world.
43:52 It is a different world.
43:53 And one of the areas in which 2023 is a different world is really the, the AI ML side.
43:59 And you all are doing a lot of stuff with personalization algorithms, recommendation engines, machine learning.
44:06 And you talked about Metaflow, which is now available.
44:09 Yeah.
44:09 The personalization one, I think we've just mentioned a bunch of things that we use from
44:14 the open source world here.
44:14 So I think XGBoost is a, is a library that does machine learning.
44:19 So personally, I am not, I am not in this field.
44:21 So I just went and interviewed the team and asked them to give me a blurb.
44:25 So I wouldn't be able to talk in detail about any of the, the personalization stuff here,
44:30 but yeah, this is just a showcase of how much this team relies on Python and the, and the
44:37 open source ecosystem that comes with Python in general.
44:40 So it's like heavy users of Panda TensorFlow and PyTorch and so on.
44:45 Yeah.
44:46 So let me ask you, is it your, both of your, your team supports Python developers and Python
44:53 applications indirectly in that way, but is it different to support the data scientists than
44:58 it is to support, say software developers?
45:00 Like, do you have to think about that differently?
45:02 How so?
45:03 Yes.
45:04 Yes.
45:04 We do have like a, a team that is dedicated to supporting all the data scientists and we
45:08 are like the team that supports the team who supports the data scientists.
45:11 They should.
45:12 Right now.
45:13 So, so yeah, we were, definitely like now in 2023, you know, betting more on
45:19 Python before Python was more like, if it makes sense for you because of freedom and responsibility,
45:24 if it makes sense to use Python in your team, you use Python, you know, and now we're trying
45:28 to provide basically like a better base path.
45:31 This is, me and Amjit with this new team that we started, we're trying to kind of enhance
45:36 this base path better and better for all these teams.
45:39 And we, you know, it's hard to know all the specifics in every single team, but we're
45:46 trying to provide them with as good practices and automation as possible.
45:50 So I think you asked like, how is it different, supporting one versus the other?
45:55 I think we built, so when we first started the team, we met with 10 different organizations
46:00 inside of Netflix to find out like how they use Python.
46:03 And we found that there were some commonalities, but the way, for instance, like algorithms
46:08 engineering uses Python is very different from the way a SRE team uses Python.
46:12 And it's very, very different from how our animation studio uses Python.
46:16 So our VFX animation uses Python in a way where they, once they start, like this is apparently
46:23 common in all of, the movie industry, which is once they start a particular project, whatever
46:28 they have chosen at the start of that project, they will stick to it until that project is
46:32 completed.
46:32 So if that movie takes two years to finish, you cannot upgrade anything inside of that particular
46:37 hermetically sealed, environment, development environment that you have.
46:42 So that is very different from like another, like a machine learning person who's interested
46:46 in like, you know, I just want to write my algorithm.
46:48 Like I don't, I don't care about how pip works or like how I pip install.
46:52 Like I don't want to worry about like virtual environments and things like that.
46:55 Whereas a person who's writing internal tools, they want to like main, like, you know, own
46:59 the entire tool chain.
47:00 It's like, I not only want to maintain virtual environment, I also want this thing to work
47:05 with a front end that is written in react.
47:07 And so I would like you to be able to like make it possible to do like NPM and, and, and
47:12 pip to coexist and, and live together.
47:14 That's not like a hard thing to do, but it's one of those things where it's like, if I'm
47:19 trying to solve a problem, let's say I'm, I'm bringing in like Python dependency locking
47:23 as a, as a mechanism to help these web developers, right?
47:26 Because they, they don't want to automatically upgrade anytime they build their system and,
47:30 and suddenly break in production.
47:31 Now that might be completely useless for someone who's working in machine learning.
47:36 And so they're like, you know, why are you solving that problem?
47:38 This, you know, you bringing like locking to packaging doesn't help me in any way.
47:43 Like, why are you wasting your time?
47:44 And so we had to sort of build personas for various ways in which Python is used inside
47:50 of Netflix.
47:50 So that when we are working on a particular feature, we can tell them we are now targeting
47:54 this persona.
47:55 We are working towards making life easy for animation engineers.
47:58 So if it doesn't work for you, that's fine.
48:01 You know, that's fine.
48:02 We will get to you.
48:03 It's just that our persona that we're targeting right now is not yours.
48:06 So that's, that's how it's different.
48:08 I'd say.
48:08 Yeah.
48:09 Yeah.
48:10 Data scientists have a lot less legacy code.
48:13 That's just still cranking along because a lot of times once they get, they discover
48:16 an insight, they don't need to run it again.
48:18 Right.
48:18 They, or the algorithms are changing so fast.
48:20 They can just, well, now we're using large language models instead of whatever, you know?
48:26 Yeah.
48:26 There you go.
48:26 Yeah.
48:27 Yeah.
48:27 Whereas once you get a web app running, you might not touch that thing if it doesn't
48:32 need touching.
48:32 Right.
48:32 So you just.
48:33 Exactly.
48:33 Stability is what you need there.
48:35 So anything else you want to call out out of this article before we move on?
48:40 We don't have a ton of time left, honestly, but.
48:42 No, no.
48:43 I think this was a great article, but yeah, two things.
48:47 With regard to this, let's just leave people with this idea that we only touched on a small
48:53 part of what is laid out here and all the projects and all the ways in which it's being
48:58 used.
48:58 So certainly check out the article just called Python at Netflix that I'll put in the show
49:02 notes.
49:03 It's hard to cover it all in just one hour.
49:05 It sure is.
49:06 It sure is.
49:08 So let's maybe talk for a minute here about this project that you're involved with, Soren,
49:13 called Portable Python.
49:14 You know, I not long ago had Nathaniel Smith on to talk about PEP 7-11, distributing Python
49:21 binaries and maybe treating like CPython runtimes as wheels almost.
49:28 And you guys also have a way that you've been using for a while internally to package up
49:33 Python into something that can run as well called Portable Python, which is open source.
49:37 You want to talk a bit about that?
49:38 Yes.
49:39 That is in PEP 7-11.
49:41 I discovered it by listening to your podcast.
49:44 Right around PyCon, I think.
49:45 Yes.
49:46 It would be very interesting to see if we could partner up once this is.
49:50 So Portable Python is, we want to provide Python, of course, to all Python developers inside.
49:56 Right?
49:57 You can always grab your own Python via all kinds of ways.
50:00 Right?
50:01 PyN, Docker image, et cetera.
50:03 But we also provide builds of Python inside to be used internally.
50:07 So Portable Python is trying to solve, adjust that.
50:11 Well, one particular issue, how do you go and distribute Python on laptops?
50:15 So the end goal is we want to provide a tarball, just like that PEP says, you know, like a wheel.
50:23 A tarball that you can download and drop somewhere, typically in, you know, a user's own folder,
50:30 tilde slash, you know, mypythons.
50:33 And we want it to work from there.
50:34 So you could use PyN for that, but with PyN, we need to wait for it to build.
50:39 Right?
50:39 And we want to basically build it ahead of time.
50:42 And as soon as it's available and, you know, make it available internally.
50:46 So what Portable Python is designed to do is to do such a build, we call it Portable,
50:51 and drop it in Artifactory.
50:53 And then our tooling can just go fetch that real quick, unzip, and it's ready to go.
50:58 So your tooling, the Portable Python tooling, basically says, I'm on this platform.
51:03 So I'm on macOS and it's Apple Silicon.
51:06 So here's the, and they want this version of Python.
51:09 So that means this binary.
51:11 Let's go grab it.
51:12 Right.
51:12 Right.
51:13 So Portable Python is invoked by our building machinery.
51:16 There's one worker on macOS, x86, macOS M1, Linux, x86, Linux, ARM64.
51:24 And there's some external internal tooling that kind of detects that the new open source version
51:31 is available using Portable Python.
51:33 So Portable Python can report you.
51:35 What is the latest?
51:36 3.11, for example.
51:38 By looking at the ftp.python.org, basically.
51:41 Okay.
51:41 So the latest is 3.11.3.
51:43 Let's see.
51:44 Do we have it internally?
51:45 No.
51:45 Okay.
51:45 Let's kick off a build.
51:46 So we kick off one build for M1, one build for Linux, et cetera.
51:51 And with Portable Python, we fit the configuration.
51:55 We say we want OpenSSL, that version.
51:57 We want SQLite, that version.
51:59 And Portable Python goes ahead and does the build, produces a tarball.
52:02 We take that tarball and publish it internally.
52:05 That's interesting.
52:06 So you can control a little bit some of the internals as well, like the OpenSSL version
52:10 and SQLite version, maybe a bit more carefully.
52:12 Yes.
52:13 Yes.
52:13 And since it's written in Python, then it's able to also inspect, say, any Python.
52:20 Like you could run Portable Python, inspect path to this installation, and it will tell
52:24 you, okay, it has a SSL, that version, SQLite, that version.
52:27 It does it use like homebrew, shared libraries or what.
52:32 It can report on that.
52:34 And, oh yeah, it generates a thing that I find very important, like a little file that says
52:40 it's called manifest.yaml.
52:42 So every time it builds anything, it generates a dot manifest.yaml where it says, well, I did
52:48 the build with dash dash, LTO optimization, dash dash, like it says everything that was
52:53 used to kind of inform what the build had and which worker it ran on, what time, what was
52:59 the platform.
53:00 Like a little bit of metadata, which sometimes.
53:02 So you could even see things like what C compiler optimization flags were enabled when you created
53:07 it, for example.
53:08 Yes.
53:09 Okay.
53:09 And there is one additional thing.
53:11 So Portable Python does not install Python on your system for you.
53:15 So it is a builder.
53:16 So it builds them and produces tarballs that can be used in a standalone manner.
53:21 And so if you want to bring Python onto your system, you just download the tarball from our
53:26 internal artifact storage and then expand it.
53:29 And that we have another tool that automatically does that.
53:32 And so when somebody bootstraps a brand new Python project and they say, I would like to
53:37 use 3.11.3, which are 3.11.4, I think that got released yesterday.
53:41 Then we will already have a binary ready for them that is in the artifactory, in our internal
53:46 artifactory.
53:47 And when they run their build for the very first time, it will bring the appropriate Python
53:52 version that they have specified in either pyproject.toml or in their tox.ini or somewhere.
53:57 And it will bring that appropriate Python, install it or expand it in a known location, and it
54:02 will use that for their building their project and so forth.
54:05 So this is, it's a way to make it easy for people to not have to manage their Python on their
54:10 laptop individually.
54:11 And also this can build Python with a specific prefix.
54:15 So on servers, on our internal servers, what we do is we install Python in a specific location.
54:21 Like we always put it inside, let's say, for example, slash app slash Python, for example.
54:25 It will build it in a way that it makes it easy for Debiants to be built.
54:29 And when you install the Debian, it will put the Python in a specific location.
54:33 And also it has other benefits, such as it tries to make the Python binary as small as
54:40 possible because we're trying to deploy it out to like hundreds of thousands or a hundred
54:44 thousand servers.
54:45 So we would try to reduce the amount that we need to put on that server.
54:49 It does that.
54:50 The final product that Zoran checked yesterday, I believe it was only 50 megabytes compared to
54:55 what other like buy-in on other things are producing, which was 200 megabytes.
54:59 It does it by a few tricks.
55:01 It looks like it removes the test folder because, you know, once you have built it, like, you
55:05 know, having the test folder as part of your final artifact makes no sense.
55:08 That was like a hundred megabytes savings right there.
55:10 So things like that, some optimizations that we do that is custom for our work.
55:15 Yeah.
55:15 That's a really interesting system.
55:16 I think there's increasing momentum around having some kind of tool that is outside of
55:23 Python for managing Python, right?
55:24 So far, primarily what we've had is things like Pip, PipX, Soarin' You have a project called
55:30 Pickley.
55:31 And it's all about like, okay, you have Python.
55:33 Now, how do you go forward?
55:35 But I think a lot of people are realizing like, wait, that assumption that I have Python, now
55:39 what, is not a great assumption, right?
55:41 And so people are starting to look at tools like RustUp, which actually is kind of like Pip,
55:46 but it brings Rust into it, yeah.
55:48 Over, yeah.
55:48 So we're going to see something there, I think.
55:50 I don't know what it is, but it'll be interesting.
55:52 Yeah.
55:52 Did you see the one Rai?
55:54 Rai is the package manager that Armin wrote, which...
55:58 Yeah, from Armin Roenker.
55:59 Yeah.
56:00 That brings Python for you.
56:02 His inspiration is from RustUp, apparently.
56:05 So Rai is actually written in Rust, and it does all the things that Poetry and PDM and
56:11 other package managers does.
56:12 But in addition to that, it also brings Python for you.
56:15 And it's using a different Python called StandalonePython or something that you already had a link
56:20 for, I forgot.
56:21 But it brings Python from there to expand it into your system.
56:25 Yeah, Python Build Standalone, that's the project that it uses.
56:27 Yeah, I've heard of that.
56:29 I haven't done anything with it, but it looks interesting.
56:31 Yeah.
56:31 All right.
56:32 I think we have time.
56:33 We're getting short on time here.
56:34 I think we have time for one more really quick thing, something that you're participating in,
56:38 Amjith.
56:39 I'm sure I don't know if you are as well.
56:41 But Command Line Database Clients with Autocomplete and Syntax Highlighting.
56:46 Tell us about this.
56:47 This looks cool.
56:47 Yeah.
56:48 This is just my personal project that I wrote before.
56:51 This was a while back.
56:52 But the idea is, at the time, I was trying to learn Postgres, and I didn't know how to do,
56:58 like, I was learning Postgres, and I was using PSQL to do this.
57:01 And every time I come to, like, a table, I'd be like, you know, oh, what columns were there?
57:06 I forgot the exact name of the column, and I tried to find it and so forth.
57:10 And so, finally, you know, I broke down and decided to write, like, a shell for Postgres called PGCLI
57:17 that uses, actually, PromToolkit, like the same toolkit that's used by PtPython.
57:21 I was going to say, it looks a lot like PtPython.
57:24 It's got that Emacs mode.
57:25 Yep.
57:26 You've got Autocomplete for basically the whole SQL language, but also the database schema that
57:31 you're connected to, right?
57:32 Yes, that is correct.
57:33 So, it reads the tables and the columns in that database, and then it tries to Autocomplete
57:38 as part of the SQL statement.
57:39 So, after a where clause, it'll only suggest columns.
57:42 And after a from clause, it'll only suggest tables and so on.
57:45 Wow.
57:46 Okay.
57:46 So, after PGCLI, people wanted something for MySQL.
57:50 So, I created MyCLI, and then Microsoft came over and said, like, we would like to fork
57:55 PGCLI to make one for Microsoft's MS SQL server.
57:58 So, they did that themselves.
58:00 Like, we didn't.
58:01 So, they took PGCLI source code, and then they created that.
58:03 And then another person created Lite CLI, which is for SQLite.
58:07 And, yeah.
58:08 And there's other things now.
58:09 iRedis is, like, for a Redis client that's similar to these things, but there's a lot more,
58:14 like, more friendlier shells for databases in general.
58:17 Excellent.
58:17 All right.
58:18 This looks really cool, I think.
58:20 Yeah, this has got nothing to do with Netflix.
58:22 It's mostly just, like, hey, it's my personal project, and, you know, just what I do in my
58:27 free time sort of a thing.
58:28 Yeah.
58:28 Well, it looks really helpful for people, because talking to databases just in your terminal,
58:34 it can be tricky, right?
58:35 And having autocomplete, especially not so much, you know, the select and where people get that
58:40 pretty quick, but the database schema understanding keeps you in your flow pretty well.
58:45 Right.
58:45 Yeah.
58:45 Yeah.
58:45 Again, inspired by Bpython, actually.
58:47 Took the inspiration from them.
58:49 Yeah.
58:49 Excellent.
58:50 All right.
58:50 Well, that'll be in the show notes as well.
58:52 Guys, I think that is it for time that we have today.
58:55 So we're going to have to wrap it up with the final two questions here and recommendations.
58:59 Let's start with a PyPI project.
59:02 Not necessarily the most popular one, but something that you're like, oh, this is awesome.
59:05 People should know about it.
59:07 Soren, got a recommendation for folks?
59:08 I'm going to say Pickly.
59:09 Go check out Pickly.
59:11 Pickly.
59:11 Okay.
59:12 So give us the elevator pitch on Pickly.
59:14 It's a CLI tool that allows you to install other CLI tools, very similar to Pipex in that sense.
59:21 The main difference is being that if you Pickly install Poetry, every time you run Poetry,
59:26 it will keep itself up to date in the background.
59:29 So it will keep self-upgrading by default.
59:32 You can tell it also not to do that, but that's its main useful thing.
59:36 Cool.
59:36 So when you launch it, basically you're launching like a shim that says run this and then the
59:41 background check for update.
59:42 And when it exits, if there's an update, just update it.
59:44 Yes.
59:45 You can take a look at the little shell script, shell wrapper that it wraps it with.
59:49 Yes.
59:49 All right.
59:50 Pickly.
59:50 Awesome.
59:50 Amjith.
59:51 Oh, I guess I could plug again for Bpython.
59:54 Like good design aesthetics.
59:55 I think, yeah, it's an overall better shell than Python shell.
59:59 Yeah.
59:59 Oh, actually PDB++.
01:00:01 That's the one that I would actually recommend.
01:00:03 So if you ever use PDB and you wish that PDB had auto-completion, it's PDBPP in PyPy.
01:00:10 You don't have to change your thing at all.
01:00:13 All you have to do is pip install PDBPP.
01:00:15 And then anytime you do a breakpoint and it stops you there, you can do like a variable
01:00:21 dot and it'll give you auto-completion.
01:00:23 And yeah.
01:00:24 I don't know.
01:00:24 I'm a huge fan of auto-completion.
01:00:26 Yeah.
01:00:26 I was going to say, you and I are kindred spirits.
01:00:29 I am all about the auto-completion.
01:00:31 I'm like, this tool is broken if it doesn't give me auto-complete.
01:00:33 Because it sends you into the documentation.
01:00:36 You'll be like, oh, I need to create one of these client libraries.
01:00:39 What does it take?
01:00:40 Oh, star args, star, star, kwrgs.
01:00:42 Great.
01:00:42 Now what am I supposed to do?
01:00:43 Right?
01:00:44 Like, you know, the auto-complete, it really makes you more productive.
01:00:49 All right.
01:00:49 And then if you're going to write some Python code, what editor, if you're not in the Ripple,
01:00:54 are you using?
01:00:54 For me, it's PyCharm.
01:00:56 PyCharm mostly sublime text and VI if I'm SSHing somewhere.
01:01:02 Excellent.
01:01:03 And Amjith?
01:01:03 Vim all the way.
01:01:05 You know, even if I don't know how to quit it, I can restart my computer.
01:01:11 That is the source of, the endless source of jokes.
01:01:14 You know, like, I saw some laptop, a picture of a laptop, and it was just smashed to pieces.
01:01:20 And it said, finally figured out how to quit Vim.
01:01:22 For the longest time, actually, I had colon Q as a way to quit out of PGCLI because I, by
01:01:31 instinct, just kept hitting colon Q.
01:01:33 And yeah.
01:01:34 That's amazing.
01:01:35 All right, you guys.
01:01:37 Well, it's been great to have you on the show.
01:01:39 Thanks for being here.
01:01:40 Thanks for giving us this look at what you're all doing up over at Netflix and your personal
01:01:44 projects.
01:01:44 Yeah.
01:01:45 Thank you, Michael.
01:01:45 I just would like to mention that we have a lot of jobs at Netflix that require Python.
01:01:50 So if you are at all interested, please go to jobs.netflix.com and type in Python, and you
01:01:55 should get all of the Python job openings that are available.
01:01:58 There's a wide variety.
01:01:59 If you want to do infrastructures of, there's that.
01:02:02 If you want to do data science, there's that, right?
01:02:04 Like a lot of coolers.
01:02:05 Yes, absolutely.
01:02:06 All right.
01:02:07 Have a great day, guys.
01:02:08 Thank you.
01:02:09 Bye.
01:02:09 Bye.
01:02:10 Bye.
01:02:10 This has been another episode of Talk Python to Me.
01:02:13 Thank you to our sponsors.
01:02:15 Be sure to check out what they're offering.
01:02:17 It really helps support the show.
01:02:18 The folks over at JetBrains encourage you to get work done with PyCharm.
01:02:24 PyCharm Professional understands complex projects across multiple languages and technologies,
01:02:29 so you can stay productive while you're writing Python code and other code like HTML or SQL.
01:02:35 Download your free trial at talkpython.fm/done with PyCharm.
01:02:41 Influx data encourages you to try InfluxDB.
01:02:44 InfluxDB is a database purpose-built for handling time series data at a massive scale for real-time
01:02:50 analytics.
01:02:51 Try it for free at talkpython.fm/InfluxDB.
01:02:55 Want to level up your Python?
01:02:56 We have one of the largest catalogs of Python video courses over at Talk Python.
01:03:00 Our content ranges from true beginners to deeply advanced topics like memory and async.
01:03:05 And best of all, there's not a subscription in sight.
01:03:08 Check it out for yourself at training.talkpython.fm.
01:03:11 Be sure to subscribe to the show, open your favorite podcast app, and search for Python.
01:03:16 We should be right at the top.
01:03:17 You can also find the iTunes feed at /itunes, the Google Play feed at /play,
01:03:22 and the direct RSS feed at /rss on talkpython.fm.
01:03:26 We're live streaming most of our recordings these days.
01:03:30 If you want to be part of the show and have your comments featured on the air,
01:03:33 be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
01:03:38 This is your host, Michael Kennedy.
01:03:40 Thanks so much for listening.
01:03:41 I really appreciate it.
01:03:42 Now get out there and write some Python code.
01:03:44 Why?
01:03:45 Why?
01:03:46 Why?
01:03:47 I'll see you next time.