#296: Python in F1 racing Transcript
00:00 Quick, name the three most advanced engineering organizations you can think of.
00:04 Maybe an aerospace company such as SpaceX or Boeing came to mind.
00:08 Maybe you thought about CERN and the LHC.
00:10 But in terms of bespoke engineering capabilities, you should certainly put the F1 race teams on your list.
00:16 These organizations appear as 20 or 30 people on a race day shown on TV.
00:21 But in fact, the number of people back at the home base doing the engineering work can be well over 500 employees.
00:26 Almost every tiny part you see on these cars, as well as the tools to maintain them, are custom built.
00:32 The engineering problems solved are immense.
00:34 Would it surprise you to know that Python is playing a major role here?
00:37 On this episode, you'll meet Joe Borg, who helped pioneer Python's adoption at several F1 teams.
00:43 This is Talk Python to Me, episode 296, recorded November 16th, 2020.
00:49 Wait, wait, wait, wait.
00:52 Normally, I'd play a little guitar riff right here to kick off the episode.
00:56 But this episode is about racing and race cars.
00:59 So let's kick it off with a different kind of instrument and note.
01:02 Let's jump into a 2017 Indy Pro Mazda.
01:05 The furlap around the storied Brands Hatch circuit.
01:08 And let the engine notes be the music.
01:10 The furlap around the storied Brands Hatch circuit.
01:13 The furlap around the storied Brands Hatch circuit.
01:14 The furlap around the storied Brands Hatch circuit.
01:15 The furlap around the storied Brands Hatch circuit.
01:17 The furlap around the storied Brands Hatch circuit.
01:19 The furlap around the storied Brands Hatch circuit.
01:19 The furlap around the storied Brands Hatch circuit.
01:20 The furlap around the storied Brands Hatch circuit.
01:21 The furlap around the storied Brands Hatch circuit.
01:22 The furlap around the storied Brands Hatch circuit.
01:23 The furlap around the storied Brands Hatch circuit.
01:24 The furlap around the storied Brands Hatch circuit.
01:25 The furlap around the storied Brands Hatch circuit.
01:26 The furlap around the storied Brands Hatch circuit.
01:43 Welcome to Talk Python to Me.
01:44 A weekly podcast on Python.
01:46 The language, the libraries, the ecosystem, and the personalities.
01:49 This is your host, Michael Kennedy.
01:51 Follow me on Twitter where I'm @mkennedy.
01:53 And keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via @talkpython.
02:00 This episode is brought to you by Linode and Talk Python Training.
02:04 Please check out the offers during their segments.
02:05 It really helps support the show.
02:08 At Talk Python, we run a bunch of web apps and web APIs.
02:12 These power the training courses as well as the mobile apps on iOS and Android.
02:16 If I had to build these from scratch again today, there's no doubt which framework I would use.
02:21 It's FastAPI.
02:23 To me, FastAPI is the embodiment of modern Python and modern APIs.
02:27 You have beautiful usage of type annotations.
02:29 You have model binding and validation with Pydantic.
02:32 And you have first class async and await support.
02:35 If you're building or rebuilding a web app, you owe it to yourself to check out our newest course.
02:40 modern APIs with FastAPI over at Talk Python Training.
02:45 You have a first course in a series we're building on FastAPI.
02:47 And for just $39, it'll take you from interested to production with FastAPI.
02:52 To learn more and get started today, just visit talkpython.fm/fastapi or click the link in your podcast player show notes.
03:00 Joe, welcome to Talk Python to me.
03:01 Joe, welcome to Talk Python to me.
03:02 Joe Pyshko: Thank you.
03:03 Nice to be here.
03:04 Nice to be here.
03:05 Oh, man.
03:05 It's exciting to have you here.
03:06 I get to come back to one of my favorite topics, racing, which is always cool.
03:10 And we're going to hit the peak of racing and engineering, I think, with F1 here.
03:16 Joe Pyshko: I think so.
03:17 I probably shouldn't admit this is an F1 guy, but I've always kind of preferred WEC.
03:22 But F1 certainly up there with the high performance.
03:26 Joe Pyshko: Sating.
03:27 So the world endurance champion cars, yeah, like Ferraris and stuff like that, right?
03:31 Joe Pyshko: Exactly.
03:32 All the Porsches, Toyota, I think are the biggest one-store racing in it.
03:35 Joe Pyshko: Yeah, yeah.
03:36 Well, now that you've stepped away, I got a little distance between F1, but you work for
03:41 two different F1 teams doing some really neat stuff with Python, and we're going to dig into
03:45 that.
03:46 And I'm super excited about it.
03:47 Before we get to that, though, let's just start with your story.
03:50 How did you get into programming and what brought you to Python?
03:52 Joe Pyshko: I've always been kind of a techie guy since I was a young, young kid.
03:57 And my dad was doing a university course when I was like five or six, just about young enough
04:01 to remember.
04:02 He got a computer as part of the course and sort of, I was introduced at quite a young age.
04:08 It was just, it wasn't anything special at that point.
04:11 It was Windows 95 or whatever.
04:13 Showing my age there, I guess.
04:14 I don't know which way.
04:15 Joe Pyshko: Windows 95 was kind of special.
04:17 Joe Pyshko: Yeah.
04:18 Joe Pyshko: I think Windows 95, like it was the first operating system that really felt like
04:22 welcoming and good.
04:23 Joe Pyshko: I mean, I think you could debate about some of the early Mac ones, but the
04:26 early Mac ones were like super bizarre in some ways.
04:29 And Windows 3, one, like that was a neat time actually.
04:32 Joe Pyshko: Yeah.
04:33 Oddly, I was introduced to Windows 3.1 after 95 because of school, which seems like a weird
04:38 order to do it in.
04:39 Joe Pyshko: What is this thing?
04:41 Joe Pyshko: I sort of vaguely remember that though with my dad choosing the themes because
04:46 I think it came with plus or whatever it was called, the big theme pack on Windows 95.
04:50 And we spent more time doing that than about anything else.
04:53 But then he started showing me how Excel works and how you can do not just the kind of conditions
04:57 on each cell, but then into VB and stuff like that.
05:01 So that's how the seed kind of got planted.
05:03 Joe Pyshko: I never went to university, but I did an apprenticeship with British Telecom,
05:07 quite big in the UK with that kind of thing.
05:09 And I actually did network engineering for a few years there and got my qualification.
05:14 I think that would be an associate's degree in the US.
05:16 Joe Pyshko: Yeah.
05:17 At that time, there's probably not a lot of programming in the network engineering, but
05:21 there's a lot of scripting and automation.
05:23 So it's like on the cusp of programming, right?
05:25 Joe Pyshko: Exactly.
05:26 Joe Pyshko: Okay.
05:27 Joe Pyshko: So I much preferred that to sit manually subnetting stuff for tests, et cetera.
05:31 Like it was much nicer to be doing mostly like it was just scripts for provisioning routers
05:36 and things like that.
05:37 And we had a few systems internally that were, that we wrote because the commercial stuff just
05:41 wasn't very suited for us.
05:43 And then I got quite a lucky break with my, the first team I worked for.
05:48 Dropped in for an interview, which took several hours, which I kind of assume was a good thing.
05:52 Joe Pyshko: And they haven't kicked me out yet.
05:55 Maybe I'm here.
05:56 Joe Pyshko: Exactly.
05:57 Which was, it wasn't really a technical interview at all.
06:00 It was more, how would you approach this on a high level?
06:03 And I think at the time there wasn't really a developer focus for especially the aerodynamics
06:09 side in Formula One.
06:10 So literally just saying, you know, I'd have a web front end for all of the users and I'd
06:15 store stuff in a database and I'd save things on a, on a network file share, et cetera.
06:20 That was enough to, to pique an interest.
06:23 Joe Pyshko: Cool.
06:24 And did you learn Python there?
06:26 Joe Pyshko: Yeah.
06:27 So it was, I think I had a few months in between interviewing there and then actually starting
06:32 the job.
06:32 So I started learning at home.
06:35 I was already a Linux user.
06:37 In my high school, we had a Linux user group, which especially in the UK, I think was a very
06:42 rare thing.
06:43 It just so happened that one of the IT guys was into it and a couple of the six formers
06:47 were into it.
06:49 It was Fedora Core One.
06:50 I think at that time had just come out.
06:52 Yeah.
06:52 And that's what we were all in.
06:54 We'd found some old computers that might be being dropped or whatever.
06:58 And we were allowed to reprovision them with, with Fedora Core One and get them a new lease
07:03 of life as it were.
07:04 So I'd, I'd kind of got...
07:06 Yeah.
07:07 They were probably blazing.
07:08 They probably had thrown away because they were too slow to run.
07:09 Exactly.
07:10 And they were blazing, right?
07:11 Joe Pyshko: Yeah.
07:12 It was nice.
07:13 We did a couple of like LAN parties.
07:15 I can't remember what game.
07:16 Probably some terrible free game that no one actually played in real life.
07:20 But just to prove that we could get them running on the network.
07:23 I had a bit of background of sort of scripting and like provisioning, especially Linux nodes
07:27 at that point.
07:28 So the biggest thing was learning Python.
07:30 And so I literally sat down with Python for dummies at that point and just started basically.
07:35 Yeah.
07:36 Cool.
07:37 Neat place to like start this new venture and you can learn this new language and kind
07:40 of like grow into this whole tech world that you were jumping into, right?
07:45 Like sort of, we'll start with almost IT, but like, let's go farther.
07:48 Exactly.
07:49 I'd always sort of wanted to be a software developer.
07:52 I just needed that kind of prod, I think, to focus on that rather than networking and IT.
07:58 Yeah.
07:59 I had a similar experience.
08:00 You know, I feel like in the abstract, before you get there, like being a software developer
08:05 is this big daunting thing and it's, there's like so much to know and you don't really know
08:09 where to go with it.
08:10 But then someone comes to you and says, could you do this one thing?
08:13 Could you make this happen with programming?
08:15 Like, I could do that one thing.
08:16 Sure.
08:17 I can do that.
08:18 I'm not a programmer, but that I can do.
08:19 Let me work on that for a while.
08:20 And like, eventually you're like, wait a minute, I could do a lot of stuff.
08:23 I'm a programmer.
08:24 How do I get here?
08:25 Especially with the first job was mostly converting to start with anyway, the first kind of six
08:30 months was converting existing stuff into Python, mostly Bash at the time.
08:34 So you've already got at that point, you're not focused massively on sort of architectural.
08:40 Right, right.
08:41 Side of things more just getting comfortable with Python, trying to do everything in a
08:45 Pythonic way.
08:46 So obviously with Bash, there's going to be a lot of repeated code when you're trying
08:49 to modularize that.
08:51 And yeah, that was a really nice way to kind of settle in, I think.
08:55 Yeah.
08:56 Cool.
08:57 So I mentioned you had stepped a little bit away from F1 these days.
08:59 So what are you doing now?
09:00 So now I work for Canonical.
09:02 People, if they don't know the company Canonical, probably have heard of Ubuntu.
09:06 So Canonical are the kind of corporation that back Ubuntu.
09:09 And yeah, I'm working for the Kubernetes team.
09:12 We work on a couple of projects.
09:14 One is what we call charm Kubernetes, which is a kind of very modular Kubernetes distribution.
09:20 You can sort of pick and choose how you want your cluster to look and you're pick and choose
09:25 out all the different components of how you want to put it together.
09:28 That's all done in Python.
09:29 So we have these, the individual components in our distribution are called charms and they're
09:33 literally blocks of Python code to define what to do.
09:36 what to do if it bumps into another component.
09:39 If one of those components go away, how to react to that, et cetera.
09:42 So imagine something like Terraform, but with a constant controller that's watching over the
09:48 infrastructure.
09:49 Yeah.
09:50 That sounds super cool.
09:51 It is nice.
09:52 We're currently working on a new revision of the actual Python framework that's written
09:56 in to basically make it more Pythonic.
09:58 It was perhaps a bit more kind of scripty before.
10:01 So that's, we're quite excited about that.
10:02 I think that's going to be releasing in a few months.
10:04 Nice.
10:05 What's the relationship of that and OpenStack?
10:07 Is there any?
10:08 I used to work for the OpenStack team as a field software engineer.
10:11 So at that time I was sort of going to customers and helping them either decide to go down the
10:16 OpenStack route or not.
10:18 OpenStack as well through Canonical is delivered in the same way through what we call charms.
10:22 So all the individual components of OpenStack basically defined in Python and then connected
10:28 together with what we call interfaces, which again is sort of Python defined functions of
10:33 executed at the time that your infrastructure comes up and then mutates.
10:37 Nice.
10:38 So a lot of people listening out there, I would imagine everyone has heard of Kubernetes.
10:42 Yeah.
10:43 And probably Docker as well.
10:44 Maybe less degree OpenStack.
10:46 Yeah.
10:47 You know, I've had some guys from OpenStack on before, Flavio.
10:50 Mm-hmm.
10:51 But maybe just give us the elevator pitch of like Kubernetes, Docker, and then what this thing
10:57 that you guys are building, like why is that better than just like random Kubernetes or your
11:01 Docker compose or something like that?
11:03 Sure.
11:04 So a kind of on a single machine level, you have Docker and Docker is basically just there
11:09 to run containers.
11:10 You can have either a single executable in that container or multiple, but obviously there's
11:15 an amount of isolation that goes on with that container, namely network resource and disk
11:21 base.
11:22 If you want to orchestrate something a bit bigger and especially across several nodes, perhaps even
11:27 several networks, you would use Kubernetes.
11:29 Kubernetes is basically just there to decide when and where to place your Docker, in this
11:35 case containers, it can be others, but we'll keep it Docker at this point.
11:38 And then also what to do if there's any issue in either provisioning those or once they've
11:43 started running.
11:44 So it's just Kubernetes is sat there to basically ensure that what you've asked for remains the
11:49 case.
11:50 So if you've asked for five Nginx servers inside Docker to keep them running and if one of them
11:54 fails to replace it with a new one.
11:57 Yeah.
11:58 And Kubernetes is pretty good for like rolling out new versions, right?
12:00 It can do basically zero downtime deployments.
12:02 Yeah, exactly.
12:03 So for example, if I had like a web app, I'll have the actual, let's say in this case, the
12:07 Django app and I release a new version of that.
12:10 I can actually use Kubernetes even to start rolling out the new version of Django app and then get
12:17 the, for example, Nginx load balancer to start firing 10% of the traffic at the new instance of our web app.
12:25 And if there's a problem, it can just remove that part of our Nginx application.
12:30 Sorry, our Django application and then continue how it used to be.
12:34 Or indeed, if it's successful, we can keep changing that 10% to 100 in time.
12:38 So it's very flexible like that.
12:39 So you've kind of come back to the software side of where you started with like network engineering, right?
12:44 Yes.
12:45 Yes.
12:46 There's definitely been quite a few instances where I've had to sit and remember how to work out subnets by hand again, which is something that I was hoping never to use again.
12:56 But certainly, certainly appreciating that I had to before.
13:00 Yeah, absolutely.
13:01 All right.
13:02 So let's talk about this, your Python journey through a couple of F1 teams.
13:07 But I do want to set the stage just by setting the, giving people a sense of the scale, because when I learned about F1, it seemed to me like, okay, well, here's a, there's like a race organization and you watch it on TV or you watch it somehow.
13:23 You'll see like, there's a team of 20 people in the little pit garage and the pit wall there.
13:28 And you're like, okay, well, these are the people that do the racing and the race car.
13:32 And they talk about engineering.
13:33 But then as you pay more attention, you're like, oh, and then this is where they actually went back and redesigned the carbon fiber tub so that they could like change some setting.
13:43 And then they redid this, you know, like you start to realize like there's a huge engineering, like super advanced engineering organization that looks almost like an aerospace company, maybe.
13:53 Yeah, these days, aerodynamics is very much the biggest factor in Formula One.
13:58 And as you say, you watch the Formula One on the television, you maybe see, as you say, 20 engineers and mechanics sat either on the pit wall itself or in the garage ready to service the car.
14:09 But in reality, there's another 400 people, 500 people back at base who have been working on the tiniest details on that car to make sure that every bit of performance possible is being extracted.
14:23 These days, it's not uncommon to see aerodynamic departments alone with 100 people working in it, which, you know, is a large user base, you know, even for a fairly modest software company, I'd say.
14:36 Yeah, yeah, that's quite an incredible thing to realize just that many people are working on, you know, like you said, just the aerodynamics.
14:44 And that was the part that you were associated with most, right?
14:47 That's right. So I was a aerodynamic software engineer slash aerodynamic systems engineer, depending on who was giving me the title at the time.
14:54 So effectively, I was the main person writing software and sort of ensuring the quality of it for these aerodynamic departments.
15:02 It did bleed and especially in the second team into the vehicle dynamics department as well, because obviously what you're simulating, it's like shock springs, that kind of stuff.
15:14 Is that what that would be?
15:15 Exactly. Even usually nowadays, all of that is driven mostly by aero.
15:19 So even the suspension on the Formula One car isn't about having this.
15:23 It never was about having the smoothest ride, but it's mostly about keeping the car in the correct attitude for to exploit the aerodynamics.
15:29 So you'll see an F1. Yeah. And I guess it's probably worth also putting out, like you said aerodynamics are important.
15:35 It's an important part. It's it's an insanely important part.
15:38 And like you watch those cars go and it looks like slot cars, like those toy cars that had like literally a little peg that would stick them to the track.
15:45 Right. They just zoom, zoom, zoom, zoom. And it seems like impossible.
15:48 And that's because of all the aerodynamic force.
15:50 And you listen to them talk about it's like multiple times the weight of the car in aerodynamic force. Right.
15:57 Yeah. I would need to double check this, but I'm pretty sure around the time that I was there, you're talking 4000 kilos.
16:03 So four metric tons at the kind of 200 mile an hour mark.
16:07 I know I'm mixing metric and imperial, but at that kind of speed, it's in that ballpark of downforce.
16:13 So it's multiple times, maybe not quite an order of magnitude, but not far off in downforce.
16:18 Yeah. And obviously it's not too difficult to just do that in downforce, but you need to do that without adding a huge amount of drag to the car as well.
16:24 So the real research is right. No drag, no, no, no extra weight, not much extra weight, all those kinds of things. Yeah, it's exactly.
16:32 That's where you could just have a huge wing. Yes. Like a 747 on it. Right. It'd be fine.
16:38 Exactly. Massive angle of attack. It's something else worse.
16:42 And then it could only do. Yeah. And I get, yeah, there's also like some interesting history in Formula One.
16:46 Like one of the very, very cars didn't used to have these aerodynamics.
16:50 And in the early days, one of the first ones was called a fan car.
16:55 And it was this bizarre car where they put like a rubber skirt on the car and literally put a fan that just sucked the air out and just like it was like a vacuum to the surface.
17:03 And that that was around for a little while. So it was banned.
17:06 Yeah. But it was they actually withdrew it before it was officially banned.
17:10 There was some odd politics going on. Interestingly, the guy that designed that has just come out with a road car with the same system in place.
17:16 So if you want to, if you want to buy the experience that you can buy it now.
17:20 Amazing.
17:22 This portion of Talk Python to me is sponsored by Linode.
17:26 Simplify your infrastructure and cut your cloud bills in half with Linode's Linux virtual machines.
17:30 Develop, deploy and scale your modern applications faster and easier.
17:34 Whether you're developing a personal project or managing large workloads, you deserve simple, affordable and expensive.
17:39 Cloud computing solutions. As listeners of Talk Python to me, you'll get a $100 free credit.
17:46 You can find all the details at talkpython.fm/linode.
17:50 Linode has data centers around the world with the same simple and consistent pricing, regardless of location.
17:56 Just choose the data center that's nearest to your users.
17:59 You'll also receive 24/7/365 human support with no tiers or handoffs, regardless of your plan size.
18:05 You can choose shared and dedicated compute instances, or you can use your $100 in credit on S3 compatible object storage, managed Kubernetes clusters, and more.
18:16 If it runs on Linux, it runs on Linode.
18:18 If it runs on Linux, visit talkpython.fm/linode or click the link in your show notes, then click that create free account button to get started.
18:25 All right. So it's in this context of working in these really intense aerodynamic requirements that the cars have today that you started working in the first team that you work for.
18:36 At the time you were working for it, it was called Forced India.
18:39 That's right.
18:40 And now it's called Racing Point.
18:42 Yep.
18:43 If people listen to this episode next year, it's going to be called Austin Martin, I believe.
18:48 That's right.
18:49 Is that right?
18:50 Yeah.
18:51 Yeah. Okay.
18:52 So that group, and at least so far, has always been the pink car.
18:55 So that makes it pretty obvious, right?
18:57 All right.
18:58 So yeah, you said you started there and you had that interview.
19:01 Tell us about like, you showed up, there's a bunch of bash scripts.
19:04 Yeah.
19:05 Then what?
19:06 So then basically, the first job I had was to sit down and to convert as much of that into Python.
19:11 There was an aerodynamicist who had since moved on, who was kind of also interested in Python.
19:16 I'm not sure how he got interested in Python.
19:18 So he'd done a small amount of work, like a sort of proof of concept, as it were.
19:23 Yeah. Before we get too much of the details, let's take a step back and just like, what kind of problems were you solving?
19:28 Because I know a lot of people probably don't know like the standard workflow of aerodynamicists and so on, right?
19:32 Sure.
19:33 So effectively, the name of the game there is to allow aerodynamicists to draw parts CAD and then within a certain amount of time, be told whether that was a good thing or a bad thing.
19:44 So obviously, we can do that with a physical wind tunnel, which is how Formula One teams have been doing it for decades.
19:51 And then more recently, as technology has caught up, we can do that with simulations, which are called computational fluid dynamic simulations.
19:58 So when I think back to my math experiences, like basically the hardest math seemed to be around fluid dynamics.
20:04 So that sounds like there's a lot of computation.
20:07 There's a lot of a lot of things going on there.
20:10 It's not easy to do that kind of stuff, right?
20:12 Sure.
20:13 So like laminar flow, very basic flow is fairly easy to model.
20:16 It's when you start talking about the turbulence that goes around the car, which is what a lot of teams exploit, what all the teams exploit to really get a lot of that downforce.
20:25 And that's when you sort of get away from realized sort of mathematics into more sort of guesswork.
20:31 I guess it's come from lots of years of getting as close as possible to having something that's real, but it's still a bit of guesswork.
20:39 Right, right.
20:40 Okay.
20:41 So in this context, you showed up and you're working.
20:43 There's a bunch of bash scripts that like piece together like the CFD computational fluid dynamics pools and maybe data coming out of the wind tunnels.
20:50 And you're like bash scripts.
20:52 Come on.
20:53 Exactly.
20:54 Which is, it was very common.
20:56 Most of the guys that set the stuff up from university and that's how they did their projects at university.
21:01 Right.
21:02 Probably didn't come as developers, but probably came out of engineering disciplines.
21:05 Exactly.
21:06 Yeah.
21:06 So they had the same attitude from university, which was, we just need to get this done.
21:09 We know bash, which is perfectly fair enough.
21:12 The goals really were to get something that was a bit more reliable because you obviously you change a line in bash and you've got no real accountability for that change.
21:21 It could, if one part of this sort of pipeline fails, then the whole thing would fail.
21:26 So it needs to be made more robust.
21:29 And we wanted to start sharing code because it was, there was these mammoth bash files for each sort of discrete part of the process.
21:36 And can imagine trying to debug those things.
21:39 Or if you want to change something, you're like, oh, we really don't want to touch that part.
21:42 That part's bad.
21:43 There were days where we'd sit there just staring at thousands of lines of bash, trying to, trying to find, you know, very, very small problems.
21:51 Is there a debugger for bash?
21:53 I don't think, not that I've ever heard of.
21:55 So we were using Eclipse at the time, the IDE.
21:58 You had the theme, the bash theme, which helped a lot.
22:01 Yeah.
22:02 Just, just even for counting brackets.
22:04 It's syntactically correct.
22:05 It's sort of.
22:06 Yeah.
22:07 Exactly.
22:08 But I don't think there was specifically a debugger.
22:09 Yeah.
22:10 Yeah.
22:10 And not that I'm sure someone will send us a message in the show notes.
22:13 Like here's the debugger, but yeah, that it's not easy to work through those things.
22:16 Yeah.
22:17 We've done stuff with minus X, obviously to get a step-by-step printout of what was going on, which we had to resort to quite often.
22:23 But the level of, verbosity that you'd get back would be far too much for what you really needed that, you know, it would have just been a variable that hadn't been set correctly.
22:30 And yeah.
22:31 Yeah.
22:32 That take a day to debug.
22:33 Okay.
22:34 So you're going to convert these over into Python and you also wanted to share more code, started using databases, things like that.
22:39 Yep.
22:40 So there was already some use of a, SQL, my SQL in the first instance database that was being written to in order for an existing web app to be able to display mostly headline numbers could then be drilled down into.
22:53 So once you sort of change the part on the car, it would tell you the kind of effects that that had.
22:58 And so, yeah, it was about really getting more into the database than just, you know, summary, as it were things like we would need to produce images.
23:08 And that was all file name based to begin with, which is fine until you suddenly have a new variable, for example, on the, in the image.
23:16 And then suddenly your whole naming convention falls apart.
23:19 So it was basically just lots of little bringing it up to some grade of sort of, I guess, enterprise grade rather than the kind of university thesis grade that it was at.
23:30 Sure.
23:31 How did people interact with it?
23:32 It was basically like a website or some kind of GUI app or what was it?
23:36 Yeah, the biggest kind of the front page as it were, was a web app at the first company was all written in PHP.
23:41 And there was, there was an existing site when I started work there and sort of ended up doing quite a lot of modifications to that.
23:49 It was all static pages when I first started.
23:52 So trying to make them more interactive so that people didn't have to keep hitting refresh every sort of two minutes.
23:58 And there's a lot of image manipulation because you're talking about an entire car.
24:02 So we would make thousands of images per car, per simulation.
24:07 And to be able to browse those, you know, statically was not nice either.
24:11 You're literally loading a full image each time you click next or up or down or however you're trying to navigate.
24:17 So yeah, that was the kind of main entry point for someone.
24:21 If they wanted to drill down at that point, they would usually open some specialized application where you can basically open the 3D model of the car plus all of the simulation data.
24:31 So, you know, I'm sure people have seen like a picture, like a 3D model of a car in CAD.
24:36 Now imagine the actual aerodynamic simulation data on top of that and that you can turn on and off and choose how that's being depicted, etc.
24:44 Yeah.
24:45 And that just required quite a lot of hardware to do.
24:47 So that was, that was something that was sort of saved for when you really needed to drill down.
24:52 Yeah, it's still, it sounds like a really empowering thing, right?
24:56 I mean, instead of doing physical testing and building physical parts, plugging them into a wind tunnel.
25:01 Exactly.
25:02 All right.
25:03 You can piece these tools together and dream them up.
25:06 See what the math says.
25:08 Exactly.
25:09 And look at them, right?
25:10 It really streamlined.
25:11 You know, if you think about it for the wind tunnel, the part would have to be designed in CAD.
25:15 It would then have to be effectively 3D printed, not in the way that your home 3D printer would do, but it's a similar concept that would then have to go and be finished.
25:24 Because if it was the home printer, like the 150 mile an hour wind would just blow it apart, right?
25:28 Yeah.
25:29 And all the kind of small defects you get kind of render anything you've done useless as well, sadly.
25:34 Okay.
25:35 And then these model parts have to be finished by hand as well.
25:38 So they were a team of probably 10 or 20 people literally just finishing small plastic-ish parts.
25:43 And then the aerodynamicists would have to convince their team leader that this warranted some time in the wind tunnel.
25:50 So then that part would be physically has to then be affixed to the model in the wind tunnel.
25:54 That run is then performed.
25:56 And then you get the data from that.
25:58 Whereas with the CFD approach, computer simulation approach, at that point, just making a change in CAD and then getting the result back, it would still take a few hours to run.
26:08 But in comparison for the time and labor that goes into the physical parts, it's very empowering.
26:13 As you say, it's a much quicker.
26:15 It's way better.
26:16 Yeah.
26:17 What kind of compute did you guys have?
26:18 Just like machines you could send it off to?
26:20 Or did you have like grid computing or like high end clusters or?
26:23 So the Formula One teams tend to be quite, I say, paranoid is probably the word that comes to mind when it comes to compute.
26:30 So there is no cloud computing done, which I think is a shame because we could really exploit that.
26:36 So the teams have physical clusters on site due to a quirk in the rules.
26:42 Teams had very specific clusters at the time as well.
26:45 So there were limits on how many floating point operations we could perform within a time.
26:50 No way.
26:51 So I mean, I'm sure a lot of people don't know.
26:53 There's actual limits on how much time you're allowed to do all sorts of testing.
26:57 How much wind tunnel time are you allowed to spend per year?
27:00 How much time on a test track?
27:02 That's right.
27:03 All these different things are highly, highly regulated, but down to the down to the floating point operation?
27:08 Literally.
27:09 And as I'm sure some people are aware, like my gosh, modern CPUs tend to do multiple floating point operations per clock cycle.
27:17 But the simulation code that we were using, which most teams use because it was written quite a while ago, will only exploit a single.
27:26 Some I think would exploit two.
27:28 But it meant that the number that you were giving to the regulating body, it had to be pegged to the CPU.
27:34 If your CPUs were performing eight floating point operations per cycle, that's what you had to tell the governing body.
27:42 So the most important thing wasn't necessarily the speed.
27:45 It was like your resource was number of CPU operations in a sense.
27:49 Yeah, exactly.
27:50 And like, how do you minimize that?
27:52 Okay.
27:53 And literally AMD sponsored one of the teams in the early days of this rule.
27:58 And they pulled an old, I can't remember which, I think it was called the bulldozer spec at the time.
28:04 If I recall, it was an old Opteron, I think.
28:06 And literally.
28:07 Yeah, yeah.
28:08 It was one of the old server, server things from AMD server CPUs.
28:12 Uh huh.
28:13 So nothing special at all off the shelf.
28:14 It would be almost free because they were trying to get rid of them.
28:17 And they basically sort of through hardware killed off all but one of those floating point operations per cycle.
28:24 And then went, oh, we've got a perfect Formula One chip here.
28:27 And then sold it for obscene amounts of money to the teams.
28:30 So most teams were running, we're running with those for a while.
28:34 That's since been rectified, which is good to see.
28:36 But yeah, it was, I'm trying to recall how many, we probably had several thousand cores.
28:41 Each job would run on a few hundred cores spread across several machines, several hundred gigabytes of memory, if not getting into the terabytes of memory per simulation.
28:50 Yeah.
28:51 So yeah, they were quite hefty, quite hefty jobs.
28:54 And they would take anything between four to eight hours to process depending on the size of the, of the model that you were using.
29:01 I'm just still like blown away that that's the metric you got to worry about is like the number of CPU operations.
29:07 Yes.
29:08 It really just must drive a lot of non-intuitive or non-obvious decisions or choices or trade-offs, right?
29:15 Like, oh, this part, we could probably do this part in Python.
29:18 Oh, but there's a lot of operations there.
29:20 We're going to write that in assembly or something weird like that, right?
29:22 So we were quite lucky in that sense because the actual simulation software itself, we bought in.
29:28 And so at that point we just said, this is what it says on the box.
29:32 So we'll assume this is correct.
29:34 All right.
29:35 Although things like any like time loading the data in was a big cost because that was on the clock effectively.
29:41 As soon as the job started, the, the clock's running.
29:44 So the actual kind of the software itself, how many can some, how many clock cycles it was consuming?
29:51 Wasn't a big issue.
29:52 It was just about getting stuff done as quickly as possible.
29:54 Like the isn't simulation time.
29:57 So, right.
29:58 You know, if, if you have a bit of Python sat there loading some artifact that's taking five minutes, that's a big problem.
30:05 So anything that was blocking, we had to pay real special attention to.
30:09 Oh, how interesting.
30:10 That's such an interesting constraint.
30:12 Maybe give us a sense of some of like the, some of the libraries and stuff that you were using there.
30:16 I mean, before you answer, I just, one of the things that never, and you know, never ends to blow me away.
30:23 Blown away by, I guess I should get it out that way is on one hand, things feel so different.
30:30 So I think that's a high-end custom engineering company that is a high-end custom engineering company that is F1 racing teams compared to like a grocery store, a software team that optimizes like what offers do we send to people?
30:42 You look at the tools, you look at some of the programming.
30:43 You look at some of the programming.
30:45 It sounds real similar a lot of the times, right?
30:47 Yeah.
30:48 Even though like the special sauce is absolutely different.
30:51 Internally, it looks a lot of similar.
30:52 All we were really doing was gluing bits of existing software together and then trying to kind of optimize around that.
31:00 So we use like NumPy, for example, for handling all of these, all the numerical data that comes in after the simulation and just trying to, for example, average over it.
31:09 So these would...
31:10 One thing that is really, I was wondering about when I saw you sort of make a note of this was if you had terabytes of data.
31:17 Yeah.
31:18 How do you load that up?
31:19 Right?
31:20 Like, where does that go?
31:21 We're using some sort of distributed computing or like processing it in little parts.
31:25 I mean, Dask might make it work, right?
31:28 Across different machines.
31:29 But what was the story there?
31:30 So we had dedicated infrastructure for loading the cases.
31:34 So these things would have around a terabyte of RAM each.
31:39 And so we would load, obviously, the jobs that had come off the distributed, like actual compute side of the cluster.
31:46 It would then write out a, let's say, terabyte file.
31:50 They then were loaded on to another node, which isn't on the regulation clock, as it were, because we finished the actual compute part, the actual simulation part.
31:59 And then we can start actually decimating this data and getting what we want out of it.
32:04 So this bit can take a bit longer.
32:05 We obviously still want it to be performant because we don't want stuff waiting in the queue.
32:10 So these were huge nodes, the really fancy Nvidia graphics cards in them, pretty big CPUs as well.
32:17 But the RAM was the big thing.
32:18 I mean, especially back in the early 2010s, like a terabyte of RAM was pretty serious.
32:23 And, you know, far bigger than the hard disk space.
32:27 And the power supply for all these computers, right?
32:30 Like there must have been like a big, huge power system.
32:34 Even though the cooling in what was a fairly small server room was huge because these things are just lit up all the time.
32:40 Okay.
32:41 So you had machines that were like heavy enough that could just basically load it up anyway.
32:44 Yeah.
32:45 All onto one node.
32:46 And then we would use a mixture of some open source applications that are designed to load in these sort of big files.
32:53 PowerView is the kind of main one.
32:56 We replaced some commercial software with PowerView because PowerView was free and open source.
33:00 So we could actually a develop against it and be not have to pay for the commercial side of it.
33:06 And we did actually pay them for support, which ended up being really nice, actually.
33:10 And so, yeah, we would use software like that as well as some Python libraries like NumPy and Matplotlib and use these two in combination to make lots of images of the car, lots of plots.
33:21 Because, of course, we need to see as many plots as possible.
33:24 Yeah.
33:25 And sort of try and average out a lot of the data into something meaningful because no one can sit back and look at a terabyte of data.
33:32 We need to pull out a kind of headlines from that, but be average over a lot of that data just to get meaningful numbers.
33:39 Well, it sounds like a really interesting thing that you guys put together there.
33:43 So you did that for a couple of years or five years and then you moved on to another team who decides to change its name.
33:50 Yes.
33:51 Because that's your history, apparently.
33:53 To Scuderia Toro Rosso, which is now AlphaTauri.
33:59 Yes, that's right.
34:00 Is that right?
34:01 So it's the Red Bull sister team.
34:02 Exactly.
34:03 So Toro Rosso is just Red Bull in Italian, which was nice because I got to go to Italy quite a lot.
34:08 So, yeah, I spent quite a few years at Force India and we did some great work there.
34:14 I worked with some really, really good people.
34:16 And we spent a long time converting what was there into a much more sort of streamlined and enterprise.
34:23 I keep using that word process.
34:25 Did you look back at that time and just have like a lot of pride and like look at that transition?
34:29 Yes.
34:30 We helped them on the technical side of engineering there.
34:33 Definitely.
34:34 I think that there's still a great team there that are carrying that on.
34:37 You know, it wasn't a case of me leaving and it all stopped.
34:40 It's still very much all continuing.
34:42 They're still making big strides with stuff there.
34:44 So that's really nice to see.
34:46 And in fact, I think they've done a few of the rewrites that I was hoping to do there,
34:49 which is why I ended up going to Toro Rosso is because rewrote a lot of the bash into Python.
34:54 I really wanted to rewrite that PHP web app because kind of I've done as much as I could by slapping a kind of single page application on the top of an existing PHP site.
35:05 And I really wanted to actually use Django.
35:08 I'd made the proposal there to replace it with Django because we were already very invested with Python at that point.
35:14 So, yeah, the I got a phone call one night from someone at Toro Rosso.
35:18 So it's amazing how this works in Formula One.
35:20 Like there's a lot of churn between the teams.
35:22 So I got a call saying, oh, we heard that you've translated a lot of stuff from bash to Python.
35:27 We're looking to replace our processes as well.
35:30 Would you be interested?
35:31 I sort of said, well, yeah, I've been waiting for a while to get the go ahead to do this at Force India.
35:37 So I'll come and I'll try it here instead.
35:40 So, yeah.
35:41 Yeah, that's really cool.
35:42 What I think is really interesting about that transition is like you've got to bump up against all the challenges.
35:48 And, you know, you made a lot of progress at the first company, but it was still stuck in the kind of the way it was before.
35:55 And here's a chance to say, all right, if I could reinvent that world and do it the way I now know it should be done.
36:03 Exactly.
36:03 Like that was your chance here.
36:04 Right.
36:05 Which most people, I don't really think, get a chance to kind of put those two things side by side.
36:09 So what was that like?
36:10 Yeah, I agree.
36:11 I mean, I fully appreciate why Force India didn't just want to start everything from scratch because everything's much better than it was four years ago.
36:17 So why do we need to keep going?
36:19 So, yeah, it was it was really a good opportunity.
36:22 I really had been that sort of planning how I might do it from scratch in my head, especially facilitating Django.
36:29 Like at the time Django was becoming quite popular.
36:31 And I'd kind of got fed up with writing inline SQL in Python and PHP at that point.
36:38 It's not really great either in any place.
36:40 No, luckily it's everything's fairly sort of secure and tied down in that environment.
36:46 Very much air gapped.
36:47 There's no there's no Internet connectivity.
36:49 So no little Bobby tables.
36:50 Exactly.
36:51 Do you know that?
36:52 Yeah, exactly.
36:53 Yeah, yeah.
36:54 It would have to be an aerodynamicist doing it.
36:56 It would be interesting to see if they could put SQL injection into a CAD model.
36:59 That would be quite a feat.
37:01 Indeed.
37:02 So tell us about like when you had this perspective, this chance to kind of redo things.
37:05 Tell us tell us about that journey.
37:07 So, yeah, my main goal really was to kind of bring together both the web app side and the side that was running all of the simulation side.
37:16 So as we've said at 14 year, it was very much you had a bunch of code that was running these discrete steps through the cluster and then spat out some data at the end, which was then picked up by the web front end.
37:27 What I really wanted to do was make that much more of a single application, probably behemoth.
37:32 And to leverage a lot of the stuff that Django gave you like the ORM, for example, is a real interest from my perspective.
37:40 You know, having your database written out in code.
37:43 Yeah.
37:43 Or sometimes get a bad rap.
37:44 I did a bad rap, but I feel like 80% of the time it's absolutely like no, no contest.
37:49 Like it should just obviously be the thing used.
37:51 And every now and then it maybe it's not the right answer, but that's not the main case.
37:55 It's so nice to work with those things.
37:57 I agree.
37:58 Especially when you have a lot of relational data like we would have.
38:02 So, of course, we have an overarching object, which is the simulation you just run.
38:06 And that's connected to hundreds of discrete fields and rows all over different tables, which to manage in your mind, something that's not committable, as it were, is not nice.
38:17 Having it all nicely laid out in Python.
38:20 And of course, your Python linter can check that what you're doing is saying passing objects around on the face of it.
38:26 So, yeah, it makes that not just easier, but in my mind, a lot, a lot safer as well.
38:31 Yeah, there was some, I don't know where they got the data, if it's like some internal research or just internal keeping a record of how many bugs there were.
38:41 When I talked to Lucas Lange from his time at Facebook and Instagram, and they converted a whole bunch of stuff to have type hints, basically so the system could know what's going on.
38:50 And they said they dropped the number of bugs that they run into quite a bit down by just having that kind of how does this hang together type of analysis in there like you're talking about.
39:00 Yeah, I can definitely see that. Like the moment that type hinting became a first class citizen in Python, I was straight on it.
39:07 Because it does just, as you say, all the silly things that people do it.
39:11 I mean, not all of them, but it gets rid of a lot of the silly mistakes you make a lot more than you first think as well.
39:16 Yeah, many of them. Yeah, for sure. Yeah, I'm with you. It's fantastic.
39:21 Yeah, and the ORM is doing that for you on a database level. And it just means you don't have to keep creating DB connections in code all over the place.
39:30 If you've got part of your application that needs to access the database, it's just an object to Python's view.
39:36 So it just to me, it simplifies so much of that environment that I was very happy to use it.
39:41 Nice. So you rebuilt a lot of that workflow over that PHP sort of static site thing.
39:46 Exactly.
39:47 In Django. And that sounds like it was a really good experience, right? Like what was the contrast between like a PHP site and Django?
39:55 To me, I think PHP has moved on quite a lot now. At the time when I was using it, it was mostly just a templating language, or at least we were using it as such.
40:04 Right. In 2010, it probably was already not brand new at that point, right? It probably had been...
40:09 Exactly.
40:10 ...around for a little while. So we're talking like 2005 PHP or something along those lines, right?
40:14 Probably, yes. So it...
40:16 Yeah, okay.
40:17 It was a massive difference. So it was all of the front end was defined effectively in PHP.
40:22 I then slapped on a bunch of JavaScript to try and make it a bit more of an SPA, but it never...
40:28 Obviously, that's... there's a limit to that.
40:30 I wanted to make this new sort of management site completely first class SPA, because when you're dealing with so many different objects on your webpage,
40:41 being updated constantly, the simulation is spitting out new data every few seconds.
40:45 And if you want to track that, you don't want to be sat there refreshing the page every couple of minutes.
40:50 Right. Yeah, it sounds super cool.
40:52 What JavaScript framework did you pick?
40:54 In the end, Durandal, which isn't that well known. The guy who wrote it now works at Microsoft and has...
41:00 I've heard of it, but I've never used it. Yeah.
41:02 So it's now... it's been sort of superseded by Aurelia.
41:05 I think of the guy's name. I think it's Eisenberg is that name?
41:09 Mm-hmm.
41:10 But just look up Aurelia.
41:11 Rob Eisenberg. Is that right?
41:12 That's the one. Yeah, exactly.
41:14 Yeah.
41:14 And the reason why I liked it is because it's not very intrusive. If you know JavaScript, you'll get along very well with Aurelia or Durandal, because it just looks like JavaScript.
41:24 You're just making observable variables effectively. Whereas we played around with Angular for a bit when it just felt like I was writing something completely new and didn't really want to spend time learning completely new world effectively.
41:38 Yeah, for sure. I had the same sort of feeling about Angular.
41:41 Yeah. So it was like, especially for what we were doing, I found it a great front end toolkit, honestly. And all the stuff I'd done at Force India, I'd written pretty much from scratch.
41:52 A lot of jQuery to Polyfill because we had a mix of Internet Explorer on the Windows machines and Firefox on all the Linux machines. But mostly it was all from scratch, which I didn't want to do again, because again, that has limitations pretty quickly.
42:06 Of course.
42:07 So we didn't use much of the Django templating itself. So we weren't creating that many static pages with Django. We were mostly using the views as an API server, and we would just serve up this SPA on first.
42:20 Right, just basically teach the views to return JSON and you're good.
42:23 Exactly.
42:24 Something like that.
42:25 Yeah.
42:25 Okay.
42:26 So one thing that you put in the notes here that I think is super interesting is that you had some of the views were like performance critical, and they were backed by CPI object.
42:35 Yes.
42:36 So what is this?
42:37 What is this?
42:37 Tell me about this.
42:38 So I'm trying to think precisely what it was doing.
42:40 It was basically just collecting a bunch of numbers and then either multiplying them together or doing some function on a bunch of numbers.
42:47 And yeah, that was taking too long for Python to do on its own. So we ended up basically writing that in C using py object. And then obviously, you can just import that.
42:58 Right, exactly.
42:59 You can use the code to do on your Django view, which is really, really nice. And it sped up the function massively. And that was a function that was called multiple times in a page load, which isn't the nicest thing to do, but it had to be done.
43:10 So like the page load speed was improved dramatically.
43:13 Absolutely.
43:14 I think that's a really interesting escape hatch because Python is so nice for so many things. But there are certain things where it's just kind of slow out, right? And a lot of the libraries that get used in those places like NumPy and whatnot, they actually just fall back to C.
43:26 Exactly.
43:27 Internally, you just don't have to think about it, right? But this ability to say, Oh, I'm going to rewrite this in C or in Rust or something just for this little tiny bit, like you don't have to write the whole thing in it. Just this, you know, 10 lines or 20 lines or whatever it is.
43:40 Yeah, and that's all it was. I think we wrote it once, piled it, shipped it. And then I think there's one alteration done a month later. And that was it. It just sat there for years. And we never touched it again.
43:50 Beautiful. Yeah, the reason it's so interesting to me is because I often think about this in the terms of like a data science computation, right around those tools. Sure, they would do but like in terms of Django, it's just interesting to see it there. It's cool.
44:02 Yeah, because we were using Django for pretty much everything except the actual driving of the simulations themselves. So we used Celery quite extensively for queuing and running micro tasks. So the simulations at this point, we're actually kicking off a bunch of Celery tasks to go and extract data, etc, etc.
44:19 etc, etc. So it was we had two very big, not quite as big as the nodes that I'd mentioned before, but two very big nodes running this Django app across the two of them. And just having Django manage all of that with the fact that we can just import the ORM to dump data into the database was really, really nice.
44:36 Sounds super cool. Another thing this time sounds like you got to displace was a little bit of MATLAB.
44:41 Yeah, so that's the sort of small work I did with the vehicle science side of things. Basically, I think any software developers job in F1 is either replacing Bash, Excel or MATLAB, because
44:53 Yeah, one of the people can replace one of the three pick one, right?
44:56 I did. Yeah, I managed to do all three, actually. Believe it or not, quite a few teams as of recently were running wind tunnels off out of Excel, which is I still whoever wrote the VB to make that work is insane and a genius at the same time.
45:12 Like it's a mad genius.
45:14 It's on the line for sure between the two.
45:17 Yeah, I was I interviewed Kane Replical from Richard Childress Racing, the NASCAR team and talked about the stuff he was doing with wind tunnels there as well. And yeah, it was like an insane amount of Excel they are doing over there in NASCAR as well.
45:31 Like, yeah, I mean, if you have the tool, and you don't really know programming that well, but like, if we put it here, we can like, ease it together. I could see how you kind of put yourself into that corner of like, well, it works over here. But exactly.
45:43 Oh my gosh, is it slow and hard, right?
45:44 Again, it was when you had people who'd only ever used Excel, who were starting the teams, or at least coming into the teams when technology was picking up, you know, this is what they picked, that's what they knew.
45:54 Yeah.
45:54 And you soon outpace that. I mean, you know, obviously, Excel gets quite flaky when you put this level of data in. But then also you have the problems, you've got one instance that you can open, and that's it. So you had the situation where someone in the office next door would open your Excel sheet, and then the wind tunnel would stop
46:12 adding data to the sheet because someone else had opened it.
46:16 Because Excel took a lock on that file, and that can't be written to or something like that, right? Yeah.
46:20 Oh, gosh.
46:20 Exactly. It works while it worked. And then we were very happy to overwrite it.
46:25 I mean, as much fun as it is to like make fun of Excel, and there's like all these examples of minor Excel errors leading to really catastrophic decisions, like large investments that were very bad and all sorts of stuff. But it's also I think it's also worth just admiring, like you said, okay, these people came from here, and they just knew Excel.
46:42 Yeah.
46:43 But they built this, this simulation thing, when they didn't really know programming or programming language, and they still did it like that's pretty awesome.
46:51 Yeah, no, I agree. And you know, that applies to the bash stuff as well. I don't think I've met many developers that could sit down and write 10,000 lines of bash.
46:59 You know, and I don't know I could do it. I might quit before I got to 10,000.
47:03 And it worked pretty well for five or 10 years. It's certainly impressive in its own right.
47:10 Yeah. So a lot of these things are both Yeah, like a lot of these are both like, triumphs, but also it's like time to move on, right?
47:17 Exactly.
47:17 Yeah.
47:18 So I think Matt, I'm so sorry, I sort of derailed you from this MATLAB thing. So tell me about it.
47:22 No, no, that's good.
47:24 I'd say that the MATLAB side, unlike the other two, we wanted to replace bash and XL because of the obvious limitations. But the reason that we wanted to replace MATLAB was, I mean, purely the cost and the cost for what you gain from MATLAB.
47:38 You know, MATLAB can be very good when you're doing very complex things. It can simulate discrete or even bits of electronics if you want it to. So we have people that will make a model for how a tire deforms.
47:50 And then that becomes a component in MATLAB. And, you know, it's nice that you can then share that with other people in the company.
47:55 Right.
47:56 But a lot of the stuff they, you know, do is just analyzing the how many times a tire heats up and cools down during a lap and what window of temperature it sits in. And you just don't need to be paying several thousand dollars a year to do to work that out.
48:11 Well, and it also gets harder to run in situations like, say, with Docker or other stuff, right?
48:16 Like, yeah, if you want to put it on a server or scale it out, all of a sudden you're like, well, we got to get approval to run it on 10 machines instead of one.
48:23 It's not even necessarily a cost. It's just like, why is this friction here?
48:27 Yes.
48:28 If I can just as equally do it somewhere else.
48:30 Yeah, exactly. And you had times where race engineer might drop their laptop and get a new one set up at a race weekend.
48:37 But their MATLAB license was tied to the old laptop, so they couldn't get that immediately.
48:41 So that was a problem.
48:43 So the job was basically just to work with the vehicle science department to make a few Python scripts.
48:50 Effectively, they were just, you know, it weren't much more than just scripts to replace as much of MATLAB as we could.
48:54 And really, it was just me teaching them the basics of sort of NumPy, Matplotlib.
48:58 And then they just took it and ran with it.
49:01 Yeah, beautiful.
49:01 And you can do, there's so much of NumPy that replaces Matplotlib.
49:06 And as you say, not only is it something that they can do on any laptop, you can then start actually scheduling it as well.
49:12 So rather than having to manually run it every lap or whatever, they can just schedule it to do that.
49:16 Yeah. So you replace MATLAB with NumPy and Excel with Pandas and you're kind of good, right?
49:22 Exactly. You're at the races, quite literally.
49:25 You're off to the races. Beautiful.
49:27 Yeah. So that sounds like a really fun project you did for like three or four years to just go, okay, what if we started over?
49:34 What if we really built what we wanted instead of what we were able to do?
49:38 And yeah, how did people in your department and at the team react to seeing that sort of transform?
49:44 So it was a good mix.
49:46 One of the things I really enjoyed about working in F1 is you're sat next to your customers, which I know might sound off-putting, but honestly, it was great.
49:53 No, it's super cool.
49:54 If somebody came over and like, this is beautiful what you built, but if you could just do it slightly just like this, instead of, you know, well, next release, like, okay, give me five minutes.
50:03 Yeah.
50:03 Let me make that tweak.
50:05 And then they're like, yeah, this is fantastic.
50:07 Exactly.
50:07 Right. That's a cool experience.
50:08 I've had that too.
50:09 And just seeing someone's face light up when you've gone from a very static web page where you're, you know, having to load literally an x-axis by x-axis of plots to being able to literally just add a click of a button, change the variable on one axis, change what position in the car that plot is looking at, then say, save that, add another one to the page, compare the two, you know, overlay them, etc.
50:33 It's great to sit and see that, you know, everyone in Formula One is under a lot of pressure.
50:37 So anything, anything that you can do to help take that pressure off people, they're going to appreciate it very quickly.
50:43 Yeah.
50:43 And anything that helps them find performance because then they look good.
50:46 Sadly, I'm not that they're getting a pat on the back for someone designing a great new bit of aero that's adding time to the car.
50:53 But it is nice to know that the processes I've put in place has enabled them to do it.
50:58 Yeah, super cool.
50:59 Are you still a fan of F1?
51:00 Do you watch it these days?
51:02 Not as much as I just found out, but I have, I think it's ESPN in the US.
51:06 So I moved to the US recently.
51:08 So I'm still getting my bearings on how to find various TV shows and things I want to watch.
51:14 So I watched Imola and Portuguese Grand Prix, mainly because they're two very exciting tracks for me.
51:21 The Toroso factory is very close to Imola.
51:25 All right.
51:26 So I've walked the track quite a few times.
51:28 I've been to coffee outside the track many times.
51:31 Or I always wanted to do a track day there.
51:33 And sadly, the stars never aligned.
51:35 That would have been fun.
51:36 But it would have been.
51:37 I probably would have got a bit too excited and ended up crashing.
51:40 But it was a particularly nice track.
51:44 So one that I was very keen to get some time on.
51:46 There's still hope.
51:47 I still have friends that live near there.
51:49 So maybe I'll pay and do a proper track day.
51:52 You might make it back there and get there.
51:54 Yeah.
51:55 Yeah.
51:55 I've been back since leaving.
51:57 I've been back there a couple of times, actually, for sort of mini holidays and things.
52:01 And you can go and rent budget race cars there to do a few laps in, which I might do hopefully after the pandemic clears up.
52:08 Yeah.
52:09 That sounds fantastic.
52:09 I'd love to do that, too.
52:10 All right.
52:11 So one final question.
52:12 I kind of want to put all these pieces together and get your thoughts on it.
52:16 So this racing world, my sense is like, it's pretty high pressure.
52:20 It's high stress.
52:21 It demands a lot of your time.
52:22 And you talked about these events, right?
52:24 Like the software has to be ready to deliver on a date.
52:28 And it's not like, well, we'll push the release a week.
52:30 Right.
52:31 It's got to be there for that thing.
52:32 Right.
52:33 And now you're working for a tech company building almost like an operating system layer for deployment.
52:40 Yeah.
52:41 What's that comparison look like?
52:43 Like if people are in one and they want to think about the other, like what your two worlds look like in those times.
52:48 So certainly my work life balance has shifted towards life quite a lot, which is nice.
52:54 I mean, it was a great seven odd years that I spent.
52:58 I learned a lot.
52:59 And the pressure was a good thing to learn a lot.
53:01 It was the right amount to help there.
53:02 But it's the kind of the any way I can think to describe it was move fast, but don't break things, basically.
53:09 So we had to anything that went wrong, we had to fix quick.
53:12 Right.
53:13 We weren't towards the latter time at Toroso.
53:16 We tried to get some sort of like real sprint planning going forward rather than just kind of picking stuff that was being shouted at the loudest or shouted for the loudest.
53:27 So we tried to get some more calm and some stability.
53:31 But ultimately, if something's broken, unfortunately, the things that would break wouldn't usually be our tools.
53:37 It would be maybe the company that we'd paid to write the simulation software.
53:40 Something would change in that.
53:42 And we had to rewrite the function that read data from it.
53:45 So that stuff had to obviously happen very quickly when it does go wrong because you just can't waste a minute.
53:51 Or if there's, you know, a node goes down because there's a problem in the data center or something, we have to mitigate that really, really fast, as well as delivering all of these fancy new toys that we've promised.
54:01 And they can't break either.
54:04 So it is nice.
54:05 Oddly, we did barely any integration testing.
54:08 It was never really on our agenda.
54:10 It was more sort of get as many features out as possible.
54:14 And I have to admit, I do much prefer now being somewhere where testing is mandatory, you know.
54:20 A little more of a safety net.
54:23 Exactly.
54:24 It's a lot more reassuring to hit the release button when it's been through at least some agreed upon tests rather than just being like, well, I've tested this on my laptop.
54:31 You know, my work laptop.
54:33 Hopefully it will work on everyone else's.
54:34 Exactly.
54:35 It works on my machine.
54:36 We're good.
54:37 Exactly.
54:37 Chip it.
54:38 Nice.
54:39 Cool.
54:40 Yeah, if anyone's thinking about going into, you know, that industry, then it's going to be intense for sure.
54:47 If you can thrive off that environment, then you're going to love it.
54:50 I think a lot of people are in the kind of mindset that they can thrive on it for a while and then they just need to have a bit of a break for a while.
54:57 Some people have put up with it for 20, 30 years.
55:00 You know, I've worked with some people who have been in F1 for that kind of length of time and we're still loving it, which is great to see.
55:07 Yeah, good for them, right?
55:08 I'm a big believer that careers go in sort of seasons, right?
55:13 There's part of your life where you might be young and energetic and you don't have a large family time commitment and something like this would just be perfect.
55:21 But then that same person 10 years later might actually hate it, right?
55:25 They don't want to be away from the family.
55:26 They don't want to be away from a home or like there's just...
55:30 It depends where you are, not just who you are, I think.
55:32 Like where you are in your career, yeah.
55:34 Exactly.
55:35 And it's been nice working with different things.
55:37 So we never really exploited Kubernetes in either of the teams that I work for.
55:41 It was something that we were starting to look at, but most of our provisioning was bare metal provisioning.
55:46 So to have a software layer on top of that, we were talking earlier about the operations per cycle limits, like having a bunch of virtual networks that Kubernetes put together on top of that is just going to suck cycles away.
55:58 Right.
55:58 So we were thinking about doing that with the Django app, actually, to put that in a Kubernetes cluster across those two nodes.
56:04 But, you know, in the end, Docker Swarm was plenty for that.
56:08 There's no need to complicate that further.
56:09 Right.
56:10 There's always the opportunity to, but...
56:12 That's right.
56:13 There is.
56:13 Well, that's a super interesting look inside of your time in F1 and the Python.
56:19 It sounds like Python played a super significant role there.
56:21 Yeah, definitely.
56:22 I'm fairly confident that the majority of teams are pretty deep into Python now, purely because it works great.
56:30 It's the kind of blue code that you need to be driving these kind of commercial softwares that are used for each stage of simulation.
56:37 You can then, with the same bit of code, start making a bunch of images and plots.
56:42 You can have your web server running in it.
56:44 It's so kind of diverse in that sense and fairly easy to pick up.
56:49 I should mention that although I was the kind of primary software engineer at both teams, for the aerodynamics department at least, you know, I had a lot of input from people who trained as aerodynamicists but wanted to learn Python.
57:01 And, you know, for the most part, found it quite easy.
57:03 What is this magic that you're wielding over here?
57:06 Exactly.
57:09 For example, if you wanted a certain picture to be made that we weren't currently making, an aerodynamicist could just sit down, change a few lines of code, push it into a Git branch, and we could have that running in a couple of hours, which is really powerful.
57:22 To give someone the ability just to say, right, I want this to be added to this big chain of steps, and it just works.
57:31 Yeah.
57:31 Super cool.
57:33 All right.
57:33 Well, I think we're out of time, but it's definitely a neat look inside.
57:36 Now, before I let you hire you, though, I've got to ask you the two questions that go at the end, of course.
57:40 Sure.
57:41 If you're going to write some Python code, what editor do you use?
57:44 So now I use VS Code.
57:45 I'll admit that I used PyCharm through my whole F1 career.
57:48 Yeah.
57:49 All right.
57:49 And I've been pulled over to the VS Code side.
57:52 Interesting.
57:53 Yeah.
57:53 And I think it's down to those two these days.
57:55 Like, those are the most popular answers.
57:57 It used to be a lot more variety when I started the show, but now it seems to be settling into those two camps.
58:02 So, yeah.
58:03 And then a notable PyPI package, something that's cool that you've run across that maybe people should know about.
58:08 Yeah.
58:09 So I've been writing some tests recently, and I had a function that I didn't really that would run a part of this code.
58:16 And basically, it would take a while for the application running behind it to settle.
58:21 And I didn't really want to put a bunch of sleeps in there.
58:23 That just seemed bad.
58:25 And I didn't want to write my own back off loop.
58:27 So basically, I found that there's a package called back off.
58:30 And you could just decorate methods or functions that you want to be backed off and then provide a bunch of parameters, like how many times you want it to try.
58:38 How long you want it to wait before the next attempt.
58:41 It was really nice.
58:42 You just literally put the one decorator around your function and it's done.
58:46 Yeah, that's really cool.
58:47 And you can do things like back off on exception of these types of exceptions, then just slow down and try it again.
58:53 And so on.
58:54 Yeah, it's really nice.
58:55 Especially for writing tests.
58:57 Some kind of like too many requests.
58:59 You know, what is that?
59:01 Some 400 error, right?
59:02 Like there's someone that says too many requests.
59:04 That one, let's just slow it down a little.
59:06 Yeah, so if anyone's dealt with the AWS API and has got that, the rate limiter from there, this is a great library for that.
59:12 Okay, cool.
59:12 Yeah, it's a great one.
59:13 And I hadn't heard of that before.
59:14 Very neat.
59:14 All right, final call to action.
59:16 I'll give you two angles here.
59:18 One, if people want to get into some kind of racing career with software development skills, what should they do?
59:23 And then maybe getting started with your Kubernetes charms, charmed Kubernetes stuff.
59:28 Sure.
59:29 So if someone wants to get into, I mean, the Formula One software development side, there's really two approaches.
59:35 I think most of the aerodynamic and the sort of wind tunnel, well, both the CFD and the wind tunnel side will be Python these days.
59:44 A lot of C# is being used in Formula One now as well.
59:47 So if there's anyone who's got a taste for C#, anything that's done on Windows will be written in C# now.
59:53 So for example, the driver in the loop simulators, you know, the actual simulators the drivers sit in will be all written in C#.
01:00:00 So yeah, you don't even really have to be that big into F1.
01:00:04 I happen to be because I've been racing since a kid, since I was a kid.
01:00:07 But yeah, as long as you've got some sort of basis in Python and or C#, just apply.
01:00:13 I mean, just try.
01:00:14 I really wasn't that experienced.
01:00:15 I got a, you know, a lucky break and exploited it to as much as I could.
01:00:19 Beautiful.
01:00:19 And then Charmed Kubernetes, people are interested in that?
01:00:23 Yeah, sure.
01:00:23 So we at Canonical currently have two Kubernetes products that we are working on at the moment.
01:00:29 One is called MicroKates, which is a very opinionated distribution of Kubernetes.
01:00:34 It started off as a single node distribution.
01:00:36 So if you just had your dev laptop and you wanted to test some Kubernetes application that you were writing,
01:00:43 you could just literally in one command install MicroKates.
01:00:45 Now with there's a lot of demand coming from sort of IoT and Edge devices,
01:00:50 you can now run them in a proper Kubernetes cluster.
01:00:53 So with literally one command to install Kubernetes and then another one to add it to your cluster,
01:01:01 you've got it.
01:01:02 So if anyone's used kubeadm, it's similar, except for you've got everything in your MicroKates package.
01:01:06 You've got the container runtime and the API server and everything else.
01:01:10 Oh, yeah, that's neat.
01:01:11 I mean, normally I think of Kubernetes being a server side, big server type of infrastructure, right?
01:01:16 Exactly.
01:01:17 So one of the things that also put me off using Kubernetes in the Formula One team was basically testing on it was quite tricky then.
01:01:24 So with stuff like MicroKates, it's really easy.
01:01:27 You can just define the infrastructure that you want to test and you can just run it in MicroKates,
01:01:31 whether that be on your local dev machine or in CICD.
01:01:34 And now if you've got a bunch of Raspberry Pis running on some Edge platform, you can do the same.
01:01:39 It's just a single command to install it.
01:01:41 You have everything you need in that single package.
01:01:43 And yeah, you just add them to the cluster with a single command as well.
01:01:47 So a total of two commands to install it and enroll it.
01:01:51 And then if you want a less opinionated Kubernetes distribution, all of the kind of the actual code behind it is all written in Python.
01:01:58 It's called Charm Kubernetes.
01:01:59 It runs off a product called Juju, which is basically the actual program that deploys and then monitors your Kubernetes cluster
01:02:08 and then enforces that what you've asked for stays.
01:02:10 So, you know, you can choose between the different run times.
01:02:14 So if you want a Docker or a container D runtime, you can swap out your CNIs, so Flannel, Calico, etc.
01:02:21 All with single line commands.
01:02:23 And you can do it on a running cluster.
01:02:25 And because of this sort of Python infrastructure code that we have running in the background, it's handling all of that for you.
01:02:32 You're not having to either reapply it or gracefully bring down your cluster first.
01:02:37 It will all do this on a running cluster.
01:02:40 Wow.
01:02:40 That sounds neat.
01:02:41 Awesome.
01:02:41 So, yeah.
01:02:42 Two very different paths for people to check out depending on what they're interested in.
01:02:45 So let me just ask you real quick, since you mentioned it, what kind of racing did you do?
01:02:49 I did endurance karting.
01:02:51 So there was a series called EPEC, which is European Pro Kart Endurance Championship.
01:02:57 So we did 24-hour races across Europe.
01:03:00 No way.
01:03:01 24-hour kart races?
01:03:02 Oh, my goodness.
01:03:03 Yeah.
01:03:04 So there were three or four of us per team.
01:03:06 We do two or three hours each and just keep rotating like that.
01:03:10 People might think of karts as like things they go ride at little adventure parks.
01:03:15 But the racing karts are beasts.
01:03:17 Like those things are really fast and intense.
01:03:19 So, yeah, it's an interesting world.
01:03:21 They're cool.
01:03:21 Yeah.
01:03:22 They're not the kind of bachelor party things that I think you'd imagine.
01:03:25 They're capable of doing 100 miles an hour.
01:03:27 So you do have to have your, you know, it's not good to be sleepy.
01:03:30 Two inches off the ground.
01:03:31 Oh, if that.
01:03:32 Two inches off the ground with no seatbelt, right?
01:03:34 Yeah.
01:03:35 Even.
01:03:35 Two millimeters, I think.
01:03:36 Oh, my goodness.
01:03:38 Well, Joe, this is a super cool look at what you've been up to.
01:03:41 And thanks for sharing another cool Python story.
01:03:44 Thank you very much for having me.
01:03:45 Yeah, you bet.
01:03:46 Bye.
01:03:46 This has been another episode of Talk Python to Me.
01:03:50 Our guest in this episode was Joe Borg.
01:03:52 And it's been brought to you by us over at Talk Python Training and our friends at Linode.
01:03:56 Simplify your infrastructure and cut your cloud bills in half with Linode's Linux virtual machines.
01:04:01 Develop, deploy, and scale your modern applications faster and easier.
01:04:05 Visit talkpython.fm/Linode and click the create free account button to get started.
01:04:10 Want to level up your Python?
01:04:12 If you're just getting started, try my Python Jumpstart by Building 10 Apps course.
01:04:16 Or if you're looking for something more advanced, check out our new async course that digs into all the different types of async programming you can do in Python.
01:04:25 And of course, if you're interested in more than one of these, be sure to check out our Everything Bundle.
01:04:29 It's like a subscription that never expires.
01:04:31 Be sure to subscribe to the show.
01:04:33 Open your favorite podcatcher and search for Python.
01:04:36 We should be right at the top.
01:04:37 You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm.
01:04:46 This is your host, Michael Kennedy.
01:04:48 Thanks so much for listening.
01:04:50 I really appreciate it.
01:04:51 Now get out there and write some Python code.
01:04:52 I really appreciate it.