WEBVTT 00:00:00.001 --> 00:00:03.840 When you run your code in the cloud, how much do you know about where it runs? 00:00:03.840 --> 00:00:08.220 I mean, the hardware it runs on and the data center it runs in. 00:00:08.220 --> 00:00:11.440 There are just a couple of hyperscale cloud providers in the world. 00:00:11.440 --> 00:00:16.720 This episode is a unique chance to get a deep look inside one of them, Microsoft Azure. 00:00:16.720 --> 00:00:24.300 Azure is comprised of over 200 physical data centers with hundreds of thousands of servers in each one of those. 00:00:24.300 --> 00:00:27.420 A look at how code runs on them is fascinating. 00:00:27.420 --> 00:00:30.340 Our guide for this journey will be Mark Rusunovic. 00:00:30.340 --> 00:00:36.960 Mark is the CTO of Microsoft Azure and a technical fellow, Microsoft's senior most technical position. 00:00:36.960 --> 00:00:39.300 He's also a bit of a programming hero of mine. 00:00:39.300 --> 00:00:43.160 Even if you don't host your code in the cloud, I think you'll enjoy this conversation. 00:00:43.160 --> 00:00:44.320 Let's dive in. 00:00:44.320 --> 00:00:53.040 This is Talk Python To Me, episode 445, recorded on site at Microsoft Ignite in Seattle, November 16th, 2023. 00:00:53.040 --> 00:01:10.660 Welcome to Talk Python To Me, a weekly podcast on Python. 00:01:10.660 --> 00:01:12.380 This is your host, Michael Kennedy. 00:01:12.380 --> 00:01:19.880 Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython, both on fosstodon.org. 00:01:20.080 --> 00:01:24.960 Keep up with the show and listen to over seven years of past episodes at talkpython.fm. 00:01:24.960 --> 00:01:28.760 We've started streaming most of our episodes live on YouTube. 00:01:28.760 --> 00:01:36.300 Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode. 00:01:37.560 --> 00:01:41.680 This episode is sponsored by Posit Connect from the makers of Shiny. 00:01:41.680 --> 00:01:46.180 Publish, share, and deploy all of your data projects that you're creating using Python. 00:01:46.180 --> 00:01:52.020 Streamlit, Dash, Shiny, Bokeh, FastAPI, Flask, Reports, Dashboards, and APIs. 00:01:52.020 --> 00:01:54.420 Posit Connect supports all of them. 00:01:54.420 --> 00:01:58.780 Try Posit Connect for free by going to talkpython.fm/Posit. 00:01:58.780 --> 00:02:00.080 P-O-S-I-T. 00:02:00.560 --> 00:02:03.700 And it's brought to you by the PyBites Developer Mindset Program. 00:02:03.700 --> 00:02:10.160 PyBytes' core mission is to help you break the vicious cycle of tutorial paralysis through developing real-world applications. 00:02:10.160 --> 00:02:16.440 The PyBites Developer Mindset Program will help you build the confidence you need to become a highly effective developer. 00:02:16.440 --> 00:02:19.840 Check it out at talkpython.fm/PDM. 00:02:21.140 --> 00:02:22.860 Mark, welcome to Talk Python To Me. 00:02:22.860 --> 00:02:23.400 Thanks. 00:02:23.400 --> 00:02:24.040 Thanks, Michael. 00:02:24.040 --> 00:02:25.680 Yeah, it's fantastic to have you here. 00:02:25.680 --> 00:02:28.680 I've been a fan of your work for a really long time. 00:02:28.680 --> 00:02:31.200 We're going to have a really cool look inside of Azure. 00:02:31.200 --> 00:02:34.520 And there's not very many hyperscale clouds in the world. 00:02:34.520 --> 00:02:36.260 You can probably count them on your hands, right? 00:02:36.260 --> 00:02:45.740 And so I think as developers, Python developers generally, we'll be really interested to just kind of get a sense of when we run on the cloud, what exactly does that mean? 00:02:45.740 --> 00:02:47.020 Because it's been a journey. 00:02:47.020 --> 00:02:48.000 Yeah, for sure. 00:02:48.560 --> 00:02:53.980 Before we dive into that, though, and some other cool things you're up to, tell people a bit about yourself. 00:02:53.980 --> 00:02:54.340 Sure. 00:02:54.340 --> 00:02:57.820 I'm a CTO and technical fellow at Microsoft, CTO of Azure. 00:02:57.820 --> 00:02:59.340 I've been in Azure since 2010. 00:02:59.340 --> 00:03:00.900 Prior to that, I was in Windows. 00:03:00.900 --> 00:03:07.920 I joined Microsoft in 2006 when it acquired my software company and my freeware website, Winternals and Sysinternals, respectively. 00:03:07.920 --> 00:03:16.200 And since 2010, I've effectively been in the same role the entire time, which is overseeing technical strategy and architecture for the Azure platform. 00:03:16.200 --> 00:03:18.800 And the skill of that is quite something. 00:03:18.800 --> 00:03:20.240 So it'll be great to get in that. 00:03:20.240 --> 00:03:20.940 That's awesome. 00:03:21.980 --> 00:03:29.100 You first came on my radar, probably in the late 90s, early aughts, through the Sysinternals thing, not through Microsoft. 00:03:29.100 --> 00:03:30.300 And you brought that up. 00:03:30.300 --> 00:03:32.200 So tell people a bit about Sysinternals. 00:03:32.200 --> 00:03:38.080 It was like if you wanted to see inside your what your app is doing on Windows, go to Sysinternals, right? 00:03:38.080 --> 00:03:38.820 Tell us about that. 00:03:38.820 --> 00:03:42.340 Sysinternals grew out of my just love of understanding the way things work. 00:03:42.500 --> 00:03:55.340 And I was doing a lot of work on Windows internals actually in my PhD program where I was trying to figure out how to get operating systems to be able to save their state and then come back in case of a failure. 00:03:55.340 --> 00:04:06.860 So I learned the internals of Windows 3.1 and then Windows 95 and then Windows NT and started to think about cool ways that I could understand the way things worked underneath the hood. 00:04:06.860 --> 00:04:16.800 So actually, the first Sysinternals tool was something called Control to Cap, which swaps the caps lock and key, you know, the control key because I came from a Unix background. 00:04:17.180 --> 00:04:18.440 Who needs caps lock these days? 00:04:18.440 --> 00:04:19.100 Yeah, exactly. 00:04:19.100 --> 00:04:19.720 No one should be that. 00:04:19.720 --> 00:04:21.500 Yeah, I'm not yelling at people very much. 00:04:21.500 --> 00:04:28.400 So the second tool that I wrote was actually called NTFS-DOS to bring NTFS to DOS. 00:04:28.400 --> 00:04:36.040 But the native Windows NT tools that Bryce Cogswell, who I met in grad school, and I co-wrote together were Regmon and Filemon. 00:04:36.040 --> 00:04:37.520 They were like the originals. 00:04:37.520 --> 00:04:41.540 And Regmon allowed you to watch registry activity, Filemon, file system activity. 00:04:41.540 --> 00:04:45.360 We later merged them into Process Monitor after we joined Microsoft. 00:04:45.760 --> 00:04:51.200 But we decided to make those tools available for free and available. 00:04:51.200 --> 00:04:56.520 So we started the ntinternals.com website, which then Microsoft's lawyer said, don't use NT. 00:04:56.520 --> 00:04:59.520 So we switched over to renaming it Sysinternals. 00:04:59.520 --> 00:05:03.640 And then Bryce was like, hey, some of the tools that we've made, we should sell. 00:05:03.640 --> 00:05:10.620 So we wrote a tool that would allow you to mount a dead NT system through a serial cable as if it was a local drive on the recovery system. 00:05:10.620 --> 00:05:13.980 And he said, if we make a read-write version, we should sell it. 00:05:13.980 --> 00:05:20.300 So he started, he went and set up a credit account, you know, credit card account on e-commerce. 00:05:20.300 --> 00:05:21.820 Authorized.net or something like that. 00:05:21.820 --> 00:05:23.140 It was like weird APIs. 00:05:23.140 --> 00:05:23.440 Yeah. 00:05:23.980 --> 00:05:29.980 And we started selling the software and we started selling the software and that grew into what became Winternals, a commercial software company. 00:05:29.980 --> 00:05:33.560 But Sysinternals and Winternals, like I said, were both acquired at the time. 00:05:33.560 --> 00:05:35.960 I joined Microsoft in 2006. 00:05:36.500 --> 00:05:39.080 But Bryce and I continued to work on Sysinternals. 00:05:39.080 --> 00:05:44.420 He worked on them until he retired from Microsoft and retired just in general four years later. 00:05:44.620 --> 00:05:45.960 And then I've continued. 00:05:45.960 --> 00:05:56.100 I have now a couple people, three people working on Sysinternals engineering system, keeping them healthy and the build pipelines working and then adding features to them. 00:05:56.100 --> 00:05:58.380 And I still code on them and still add features. 00:05:58.380 --> 00:06:01.040 Like just did a ZoomIt release a few days ago. 00:06:01.040 --> 00:06:03.880 I added some, I blur and highlight to ZoomIt. 00:06:04.100 --> 00:06:06.000 Yeah, people are doing presentations on Windows. 00:06:06.000 --> 00:06:12.780 With ZoomIt, you can say, let me quickly draw on the screen, not PowerPoint, but just whatever happens to be on your screen, which is really nice. 00:06:12.780 --> 00:06:16.620 I love my Macs these days, but boy, I wish ZoomIt existed on the Mac. 00:06:16.620 --> 00:06:21.100 Well, actually, people have asked for ZoomIt for Mac and I'd like to make a ZoomIt for Mac. 00:06:21.100 --> 00:06:30.600 And now with Copilot, I wonder how good it is at writing Mac apps because I don't want to spend all the time to learn how to write Mac apps just to write ZoomIt. 00:06:30.600 --> 00:06:35.200 But if Copilot can help, maybe it'll, you know, something that I can do in my, you know, you know, spare time. 00:06:35.200 --> 00:06:37.040 I don't know if it'll do it, but it'll get you close. 00:06:37.040 --> 00:06:40.360 It's crazy how these LLMs are writing code for us these days. 00:06:40.360 --> 00:06:42.800 And we're going to talk a bit about maybe how some of those run. 00:06:42.800 --> 00:06:43.140 Yeah. 00:06:43.140 --> 00:06:43.720 And so on. 00:06:43.720 --> 00:06:48.580 I mean, Azure is doing tons of stuff with large language models. 00:06:48.580 --> 00:06:51.500 And you all have some, you know, we're here at the Microsoft Ignite conference. 00:06:51.500 --> 00:06:52.820 You've got some big announcements. 00:06:52.820 --> 00:06:59.720 But yeah, I was such a fan still of Sysmon and I use that for all sorts of things still. 00:06:59.720 --> 00:07:00.700 So super cool. 00:07:00.700 --> 00:07:04.320 Now, before we jump in, I kind of want to talk about some of your big announcements. 00:07:04.320 --> 00:07:04.780 Yeah. 00:07:04.780 --> 00:07:07.000 Because they really caught me off guard here. 00:07:07.000 --> 00:07:08.540 I'm like, yes, this is exciting. 00:07:08.540 --> 00:07:15.400 But maybe since we're going to talk a decent amount about Azure, the internals of hardware, how our code runs, just give us a quick history of Azure. 00:07:15.400 --> 00:07:17.540 You know, when did this whole thing get started? 00:07:17.540 --> 00:07:20.460 Azure started right as I joined Microsoft in 2006. 00:07:20.580 --> 00:07:29.300 There was a group of people, including Dave Cutler, one of the people that I've looked up to because Dave was the original architect behind the VMS operating system. 00:07:29.300 --> 00:07:31.980 And then Windows NT, which is now underlying Windows. 00:07:31.980 --> 00:07:37.380 He and some other people would just at the suggestion of Ray Ozzy. 00:07:37.560 --> 00:07:41.520 This is back when services was a big thing. 00:07:41.520 --> 00:07:48.180 And Ray sent a memo to the company kind of echoing Bill Gates' internet memo saying it's software and services now. 00:07:48.180 --> 00:07:55.220 And they said, how can we build a data center scale type platform to make it easier for Microsoft to develop services? 00:07:55.600 --> 00:07:59.220 And so this was called Project Red Dog, which was incubating for a while. 00:07:59.220 --> 00:08:06.540 And then 2008, they publicly launched it because Steve said, we need to make this available to third parties as well as Windows Azure. 00:08:06.540 --> 00:08:09.800 And I'm sorry, 2008, they announced the preview of it. 00:08:09.800 --> 00:08:13.120 2010, it commercially launched publicly in February. 00:08:13.120 --> 00:08:14.660 And I joined in July. 00:08:14.660 --> 00:08:23.120 And a few years later, with the rise of open source software and so many enterprise customers wanting to have Linux, we rebranded it Microsoft Azure. 00:08:24.080 --> 00:08:25.660 And we also... 00:08:25.660 --> 00:08:27.220 Can I run Linux on Windows Azure? 00:08:27.220 --> 00:08:28.200 I don't know if that makes any sense. 00:08:28.200 --> 00:08:28.360 Yeah. 00:08:28.360 --> 00:08:36.880 And one of the first things I'd done, worked on with Corey Sanders was being asked, hey, we've got platform as a service. 00:08:36.880 --> 00:08:40.100 We have this thing called cloud services, this model for how you write apps. 00:08:40.100 --> 00:08:48.220 But our enterprise customers were saying, I can't move my existing IT stuff to Azure because it just needs VMs. 00:08:48.220 --> 00:08:53.700 And so the first thing we did was, hey, we should get IaaS capability in Azure. 00:08:53.700 --> 00:08:57.300 And so in 2012, we launched the preview of IaaS for Azure. 00:08:57.300 --> 00:09:02.500 And that's really when the business started to take off because enterprises then could, with minimal effort, start to move. 00:09:02.500 --> 00:09:04.920 Oh, well, that's like doing what we do in the... 00:09:04.920 --> 00:09:05.100 Yeah. 00:09:05.100 --> 00:09:06.120 In their own data center. 00:09:06.120 --> 00:09:07.000 Our data center, but in your data center. 00:09:07.000 --> 00:09:07.420 Yeah. 00:09:07.420 --> 00:09:08.000 Yeah, exactly. 00:09:08.000 --> 00:09:09.360 Now no one even thinks about it. 00:09:09.360 --> 00:09:10.080 Exactly. 00:09:10.420 --> 00:09:12.440 So IaaS has continued to evolve. 00:09:12.440 --> 00:09:13.460 PaaS has continued to evolve. 00:09:13.460 --> 00:09:16.820 Cloud services was designed in a world without containers. 00:09:16.820 --> 00:09:22.060 Now we've got containerization, the rise of Kubernetes, and then application models on top of containers. 00:09:22.060 --> 00:09:23.300 And so Azure has evolved. 00:09:23.300 --> 00:09:29.480 And actually, I think, led some of that evolution of cloud native computing up into containers and abstractions. 00:09:30.100 --> 00:09:32.060 But it's been a long, long journey towards that. 00:09:32.060 --> 00:09:39.480 I mean, I think one of the things is I've always believed that ultimately cloud should be about making it easy for developers to say, here's what I want. 00:09:39.480 --> 00:09:41.120 And then the cloud takes care of the rest. 00:09:41.120 --> 00:09:47.020 And we're moving towards it relentlessly, that time when you'll really be able to do that. 00:09:47.040 --> 00:09:49.100 Yeah, so you don't have to know DevOps. 00:09:49.100 --> 00:09:52.100 You don't have to know distributed architectures. 00:09:52.100 --> 00:09:54.320 You just give you guys a good idea. 00:09:54.320 --> 00:09:55.740 Yeah, which is beautiful. 00:09:55.740 --> 00:09:56.160 It's beautiful. 00:09:56.160 --> 00:09:59.220 Now, real quickly, just give us a scale. 00:09:59.220 --> 00:10:04.760 Like, think of how many data centers, how many servers, how many miles, fiber. 00:10:04.760 --> 00:10:05.100 Yeah. 00:10:05.100 --> 00:10:06.460 It's kind of astonishing. 00:10:06.460 --> 00:10:07.700 It is pretty flabbergasting. 00:10:07.700 --> 00:10:10.740 And the numbers continue to grow exponentially. 00:10:11.160 --> 00:10:20.560 I'll just give you, because I remember when I first started in Azure, I was asked to give a talk at the Azure All Hands about architecture and some of the announcements we had coming. 00:10:20.560 --> 00:10:30.200 And the All Hands was two rooms with the partition removed in our on-campus conference center, meetings room, meeting center. 00:10:30.200 --> 00:10:32.100 We totaled about 500 people. 00:10:32.100 --> 00:10:34.680 That was all of the Azure team in 2010. 00:10:34.680 --> 00:10:38.340 And really, nobody outside the Azure team knew anything about Azure. 00:10:38.340 --> 00:10:38.760 Yeah. 00:10:38.760 --> 00:10:41.100 It was kind of a secret even inside, right? 00:10:41.100 --> 00:10:41.580 Yeah. 00:10:41.580 --> 00:10:47.440 So effectively, that was like most, at least half the people in the world that knew anything about Azure was in those two rooms. 00:10:47.440 --> 00:10:52.700 And today, you know, Scott Guthrie's organization, Cloud and AI, all of it's working on Azure. 00:10:52.700 --> 00:10:54.940 And that's tens of thousands of people. 00:10:54.940 --> 00:11:04.560 So at least, you know, a good percentage, majority percentage even of the company is working directly on things that come under the Azure umbrella. 00:11:04.560 --> 00:11:06.920 So it's come a long way from that perspective. 00:11:06.920 --> 00:11:08.620 And you talked about physical scale. 00:11:08.940 --> 00:11:16.360 Back then, when we originally launched Azure in two regions, it was like 40,000 servers, like 20,000 in one, 20,000 in the other. 00:11:16.360 --> 00:11:17.100 That's still a lot of servers. 00:11:17.100 --> 00:11:17.340 Yeah. 00:11:17.660 --> 00:11:18.020 Yeah. 00:11:18.020 --> 00:11:22.940 But, you know, it was like, that is kind of cloud scale back then. 00:11:22.940 --> 00:11:25.500 Now we are at millions of servers. 00:11:26.080 --> 00:11:31.420 And when it comes to data centers, we've got 60 regions around the world, 60 plus regions. 00:11:31.420 --> 00:11:37.120 And each of those consists of one, in many cases, multiple data centers. 00:11:37.120 --> 00:11:39.000 And we're still building out. 00:11:39.000 --> 00:11:43.240 We're launching a data, like two data centers every week, I think is the number that we're launching. 00:11:43.500 --> 00:11:43.720 Wow. 00:11:43.720 --> 00:11:44.640 That's crazy. 00:11:44.640 --> 00:11:47.940 And these could be slotted into one of these regions or it could be. 00:11:47.940 --> 00:11:48.120 Yeah. 00:11:48.120 --> 00:11:48.540 A new region. 00:11:48.540 --> 00:11:48.860 Totally new. 00:11:48.860 --> 00:11:49.060 Yeah. 00:11:49.060 --> 00:11:49.260 Yeah. 00:11:49.260 --> 00:11:49.720 Uh-huh. 00:11:49.720 --> 00:11:50.440 Incredible. 00:11:50.440 --> 00:11:50.960 Incredible. 00:11:50.960 --> 00:11:50.960 Incredible. 00:11:50.960 --> 00:11:54.620 And so the big announcement that I wanted to ask you about, just before we run out of time, 00:11:54.620 --> 00:11:59.520 and then we'll dive into some of that, that sort of how does your code run story is Azure Cobalt. 00:11:59.520 --> 00:12:00.100 Yeah. 00:12:00.100 --> 00:12:02.420 That's a new processor you guys announced. 00:12:02.420 --> 00:12:10.760 And, you know, listeners know, I'm a big fan of Apple Silicon and how it sort of changed the computing landscape for power and speed on like little laptops and stuff. 00:12:10.760 --> 00:12:13.620 And this is kind of that idea, but for the data center, right? 00:12:13.620 --> 00:12:14.080 It is. 00:12:14.080 --> 00:12:14.620 Tell us about that. 00:12:14.620 --> 00:12:15.240 It is that idea. 00:12:15.240 --> 00:12:15.320 Yeah. 00:12:15.320 --> 00:12:20.160 I think having a processor that can be designed really with our specifications. 00:12:20.500 --> 00:12:24.340 If you take a look at Intel and AMD processors, they're fantastic processors. 00:12:24.340 --> 00:12:25.620 They're very versatile. 00:12:25.620 --> 00:12:28.860 They're taking requirements from lots of different sources. 00:12:28.860 --> 00:12:31.480 And so we're just, we're a voice. 00:12:31.480 --> 00:12:35.920 We're a significant voice when it comes to saying we'd like your processors to do these things. 00:12:35.920 --> 00:12:44.520 When we have our own, we've got the ability to just decide unilaterally what we'd like it to do based off of what we see and can vertically integrate it into our systems. 00:12:44.520 --> 00:12:48.480 We can put it on SSEs and integrate it with memory and GPUs. 00:12:48.600 --> 00:12:54.620 And so that is kind of the reason that we've done that verticalization for processors. 00:12:54.620 --> 00:12:58.840 That's not to say that the other processors aren't going to be significant. 00:12:58.840 --> 00:13:00.800 It's going to be probably a blend of these. 00:13:00.800 --> 00:13:01.360 It's going to be a blend. 00:13:01.360 --> 00:13:03.620 They'll have different capabilities that ours won't have. 00:13:03.620 --> 00:13:10.780 There are customers that want the specific features that they've got or performance speeds and feeds that they've got because they're not all going to look the same. 00:13:10.780 --> 00:13:14.320 And so I think it's just better optionality for everybody. 00:13:14.320 --> 00:13:19.160 Well, I can tell you as somebody who tries to run Linux on that thing. 00:13:19.160 --> 00:13:19.480 Yeah. 00:13:19.480 --> 00:13:22.500 It's hit and miss if there's even an ARM version available. 00:13:22.500 --> 00:13:23.060 Right. 00:13:23.060 --> 00:13:24.240 More often than not, there's not. 00:13:24.240 --> 00:13:30.320 And so there's certainly not going to be an insane rush to just let drop everything because there's a lot of code that's written for. 00:13:30.320 --> 00:13:30.720 Exactly. 00:13:31.240 --> 00:13:31.480 Yeah. 00:13:31.480 --> 00:13:31.760 Yeah. 00:13:31.760 --> 00:13:32.500 And optimized. 00:13:32.500 --> 00:13:33.660 That's the other thing too. 00:13:33.660 --> 00:13:34.100 Yeah. 00:13:34.100 --> 00:13:35.100 Yeah, for sure. 00:13:35.100 --> 00:13:39.340 But yeah, so on my list here of things I was going to ask you is, well, what about ARM in the data center? 00:13:39.340 --> 00:13:40.360 Well, that is ARM. 00:13:40.360 --> 00:13:41.060 I know, exactly. 00:13:41.060 --> 00:13:43.520 I'm like, well, okay, so you guys beat me to the punch. 00:13:43.520 --> 00:13:44.240 Exactly. 00:13:44.240 --> 00:13:53.100 This portion of Talk Python To Me is brought to you by Posit, the makers of Shiny, formerly RStudio, and especially Shiny for Python. 00:13:53.100 --> 00:13:55.020 Let me ask you a question. 00:13:55.020 --> 00:13:56.720 Are you building awesome things? 00:13:56.720 --> 00:13:57.800 Of course you are. 00:13:57.800 --> 00:13:59.360 You're a developer or a data scientist. 00:13:59.360 --> 00:14:00.280 That's what we do. 00:14:00.280 --> 00:14:02.380 And you should check out Posit Connect. 00:14:02.380 --> 00:14:09.280 Posit Connect is a way for you to publish, share, and deploy all the data products that you're building using Python. 00:14:09.940 --> 00:14:12.460 People ask me the same question all the time. 00:14:12.460 --> 00:14:15.620 Michael, I have some cool data science project or notebook that I built. 00:14:15.620 --> 00:14:18.900 How do I share it with my users, stakeholders, teammates? 00:14:18.900 --> 00:14:23.700 Do I need to learn FastAPI or Flask or maybe Vue or React.js? 00:14:23.700 --> 00:14:24.960 Hold on now. 00:14:24.960 --> 00:14:29.740 Those are cool technologies, and I'm sure you'd benefit from them, but maybe stay focused on the data project. 00:14:29.740 --> 00:14:32.220 Let Posit Connect handle that side of things. 00:14:32.220 --> 00:14:36.940 With Posit Connect, you can rapidly and securely deploy the things you build in Python. 00:14:37.360 --> 00:14:43.420 Streamlit, Dash, Shiny, Bokeh, FastAPI, Flask, Quattro, Reports, Dashboards, and APIs. 00:14:43.420 --> 00:14:45.720 Posit Connect supports all of them. 00:14:45.720 --> 00:14:51.540 And Posit Connect comes with all the bells and whistles to satisfy IT and other enterprise requirements. 00:14:51.540 --> 00:14:55.940 Make deployment the easiest step in your workflow with Posit Connect. 00:14:55.940 --> 00:15:02.060 For a limited time, you can try Posit Connect for free for three months by going to talkpython.fm/posit. 00:15:02.060 --> 00:15:05.700 That's talkpython.fm/P-O-S-I-T. 00:15:05.700 --> 00:15:07.580 The link is in your podcast player show notes. 00:15:07.580 --> 00:15:10.820 Thank you to the team at Posit for supporting Talk Python. 00:15:12.220 --> 00:15:12.620 Awesome. 00:15:12.620 --> 00:15:23.580 Now, one of the things I wanted to kind of maybe have you go through for listeners that I think is just super interesting is sort of the evolution of the hardware where our code runs throughout the data center. 00:15:23.580 --> 00:15:31.300 So you talked about in some of your talks like the data center generations and what is that, like six or seven, eight maybe different variations. 00:15:31.460 --> 00:15:33.540 I'll kind of give you some prompts from them and all. 00:15:33.540 --> 00:15:43.520 But one of the things that I was thinking about when I was looking at this is, you know, do you have a bunch of small servers or do you have like, or do you partition up really large servers, right? 00:15:43.520 --> 00:15:45.680 What's the right flow for that? 00:15:45.680 --> 00:15:53.260 So one of the things that you've seen since the start of cloud, back when we launched Azure, there was one server type. 00:15:53.260 --> 00:15:53.640 Yeah. 00:15:53.640 --> 00:16:00.080 And we had different virtual machines offerings, but they were just all different sizes that could fit on that one server. 00:16:00.340 --> 00:16:04.120 It was a 32 core Dell Optron with, I think, 32 gig of RAM. 00:16:04.120 --> 00:16:04.320 Yeah. 00:16:04.320 --> 00:16:06.560 And so that was the server back then. 00:16:06.560 --> 00:16:10.920 What we've seen is more workloads come to the cloud that have different requirements. 00:16:10.920 --> 00:16:12.420 Some require large memory. 00:16:12.420 --> 00:16:13.640 Some require more compute. 00:16:13.640 --> 00:16:15.140 Some require GPUs. 00:16:15.140 --> 00:16:19.240 Some require InfiniBand backend networking for high performance computing. 00:16:19.240 --> 00:16:29.000 And so there's been a drastic diversification of the server hardware in Azure that's being offered and current at any one point in time. 00:16:29.560 --> 00:16:31.280 And I think you'll continue to see that. 00:16:31.280 --> 00:16:34.100 So the old, you know, it's just a pizza box. 00:16:34.100 --> 00:16:37.160 It's a low end commodity server. 00:16:37.160 --> 00:16:40.380 Kind of that's the cloud vision back in 2010. 00:16:40.380 --> 00:16:44.980 Now it's the cloud contains specialized servers for specific applications. 00:16:45.540 --> 00:16:51.780 And when it comes to large servers back in 2014, we started to introduce very large servers. 00:16:51.780 --> 00:16:57.580 The kind that, you know, people that were cloud purists back in 2010 would have been like, no, don't allow that. 00:16:57.580 --> 00:16:58.700 It's all about the cheap. 00:16:58.700 --> 00:17:00.320 It's all about the cheap and scale out. 00:17:00.660 --> 00:17:05.360 Is scale up servers for SAP workloads in memory database workloads. 00:17:05.760 --> 00:17:15.680 So we introduced a machine that we nicknamed Godzilla, which had 512 gig of RAM in 2014, which was like an astonishing number. 00:17:15.680 --> 00:17:24.540 And we've continued to, as SAP workloads have gotten bigger and bigger and more has migrated to the cloud, created bigger and bigger and bigger and bigger and bigger machines. 00:17:24.680 --> 00:17:29.360 In fact, I'm showing here at Ignite the latest generation of the SAP scale up machines that we're offering. 00:17:29.360 --> 00:17:38.380 It's not yet, they're not yet public, but I'm going to show a demo of them, of one of them call that I'm calling, nicknaming super mega Godzilla beast. 00:17:38.380 --> 00:17:41.080 because we've gone through so many iterations. 00:17:41.080 --> 00:17:44.500 So this, this one is super is the new, yeah. 00:17:44.500 --> 00:17:46.760 You're running low on adjectives here. 00:17:46.920 --> 00:17:53.820 And I don't know what I'll come up with next, but anyway, we're at super mega Godzilla beast as the current generation, which has 1790 cores. 00:17:53.820 --> 00:17:54.460 Wow. 00:17:54.460 --> 00:17:58.220 And 32 terabytes of RAM, 32 terabytes of RAM. 00:17:58.220 --> 00:17:59.000 Incredible. 00:17:59.000 --> 00:18:09.320 So do you do, things to like pin VMs to certain cores so that they get better cash hits and stuff like that rather than let it just kind of mosh around? 00:18:09.320 --> 00:18:15.380 That's especially important with NUMA architectures where you've got memory that has different latencies to different sockets. 00:18:15.560 --> 00:18:23.160 As you want to have the VM that's using certain cores on a socket, have memory that is close to it, close to that socket. 00:18:23.160 --> 00:18:29.980 So that's, and that's just part of hyper V scheduling, is doing that kind of assignment, which we have under the hood. 00:18:29.980 --> 00:18:35.360 And again, it's like the control plane at the very top says launch a virtual machine of this size of this skew. 00:18:35.560 --> 00:18:41.800 Then there's a resource manager, the Azure allocator that goes and figures out this is the best server to put that on. 00:18:41.800 --> 00:18:47.440 It has enough space and will reduce minimize fragmentation and places it on there. 00:18:47.440 --> 00:18:51.920 And then hyper V underneath is saying, okay, these are the cores to assign it to. 00:18:51.920 --> 00:18:52.980 Here's the RAM to give it. 00:18:52.980 --> 00:18:53.460 Excellent. 00:18:53.660 --> 00:18:55.240 And how much of that can you ask for? 00:18:55.240 --> 00:18:56.600 you can ask for the whole machine. 00:18:56.600 --> 00:19:01.260 You can, you can, you can, you can, for a full machine, you know, full, full server virtual machine 00:19:01.260 --> 00:19:01.640 sizes. 00:19:01.640 --> 00:19:02.420 Wow. 00:19:02.420 --> 00:19:03.000 Okay. 00:19:03.200 --> 00:19:06.380 There's probably not too many of those in, but, but some people using them. 00:19:06.380 --> 00:19:06.820 Yeah. 00:19:06.820 --> 00:19:09.640 Like on the SAP ones, because they're designed for SAP. 00:19:09.640 --> 00:19:16.960 I think for those kinds of the current generations, I think we offer just two sizes, like either half of it or the whole, or the whole thing. 00:19:16.960 --> 00:19:17.520 Incredible. 00:19:17.520 --> 00:19:18.140 Wow. 00:19:18.140 --> 00:19:18.700 Okay. 00:19:18.700 --> 00:19:22.400 How, how much of a chunk of a rack does that take? 00:19:22.400 --> 00:19:24.460 It's basically the whole rack. 00:19:24.460 --> 00:19:24.660 Yeah. 00:19:24.660 --> 00:19:25.920 Pretty much top to bottom. 00:19:25.920 --> 00:19:26.160 Yeah. 00:19:26.160 --> 00:19:27.700 It's like a 10 kilowatt server. 00:19:27.700 --> 00:19:28.240 Yeah. 00:19:28.240 --> 00:19:29.940 A little power plane on the side there. 00:19:30.840 --> 00:19:38.540 As you kind of talk through the history of sort of how your code ran, it was more, more colo, as you said, like the more smaller, smaller ones. 00:19:38.540 --> 00:19:49.400 And then as you got bigger and bigger on some of this, you started working on things like, well, how do we let the servers run hotter and have the air cool them rather than more actively cooled? 00:19:49.400 --> 00:19:55.480 And then it even gets to a almost more, just remove big chunks of it and let them fail. 00:19:55.480 --> 00:19:57.640 And then when enough of it has failed, take them out. 00:19:57.640 --> 00:19:59.500 You want to kind of talk about some of that? 00:19:59.500 --> 00:20:01.300 We're still, yeah, good question. 00:20:01.300 --> 00:20:07.840 Cause we're, we're still exploring this space towards higher efficiency, lower energy consumption, more sustainable. 00:20:07.840 --> 00:20:19.800 One of the experiments that came out of Microsoft research was project Natick, which is taking a bunch of servers, putting it in a rack of servers, putting it in a container that has nitrous oxide gas in it. 00:20:20.000 --> 00:20:26.380 So it's an inert gas and dropping it into the ocean floor and letting it be cooled ambiently through the water there. 00:20:26.380 --> 00:20:29.580 Not water on the inside, but the outside of the container. 00:20:29.580 --> 00:20:29.860 Yeah. 00:20:29.860 --> 00:20:30.900 So giant heat sink. 00:20:31.000 --> 00:20:33.980 And the, there was potential benefits of that. 00:20:33.980 --> 00:20:36.400 And it's still something that might get revived at some point. 00:20:36.400 --> 00:20:46.640 But what we've found coming out of that was if the parts are in an inert environment, they have one eighth the failure rates as ones that are in air environment. 00:20:46.880 --> 00:20:50.400 And with particulate matter and corrosive materials in the air. 00:20:50.540 --> 00:20:59.120 So we started exploring liquid cooling, both for that, as well as potential energy savings and more sustainable cooling and then air cooled. 00:20:59.120 --> 00:21:01.720 We've explored a two phase liquid immersion. 00:21:01.720 --> 00:21:03.160 We had a pilot running. 00:21:03.160 --> 00:21:09.520 There's some regulations that's changed around the kinds of fluids that have made us take a look at a different direction. 00:21:09.520 --> 00:21:10.180 So we use. 00:21:10.180 --> 00:21:14.940 Is that kind of like the core, the, the, what you would get in like an air conditioner or some of the stuff they'd replace? 00:21:14.940 --> 00:21:17.340 They're called forever chemicals or materials. 00:21:17.340 --> 00:21:22.500 The ones we're using actually aren't, but the regulation is a little broad. 00:21:22.500 --> 00:21:32.560 And so we're just steering clear and it might be revisited at some point, but we're also have been exploring liquid cooling, kind of traditional liquid cooling, cold, cold plate called. 00:21:32.560 --> 00:21:42.320 And some people listening, probably like me are gamers and have liquid cooled GPUs or, or CPUs at home in their gaming rigs, which allow them to get overclocked. 00:21:42.320 --> 00:21:44.520 And it's the same thing we're doing in our data centers. 00:21:44.620 --> 00:22:02.860 In fact, one of the things Satya showed in the keynote was something called sidekick, which is a cabinet that allows us to take liquid cold plate, liquid cooling into an existing data center, air cooled data center, where the Maya 100 AI accelerators are in the cabinet sitting right next to it. 00:22:02.860 --> 00:22:09.240 And the cooling pipes are going into the Maya cabinet to cool the accelerators themselves. 00:22:09.240 --> 00:22:10.620 And so that is. 00:22:10.620 --> 00:22:14.600 So they manage like some big metal plate and then the metal plate is liquid cool. 00:22:14.600 --> 00:22:20.600 Yeah, it's effectively that there's a plate on top of the processor and then liquid is going through that. 00:22:20.600 --> 00:22:27.140 So I'm going to actually show pictures of the inside of the Maya system tomorrow in my AI innovation closing keynote. 00:22:27.500 --> 00:22:42.200 But that is, I think the takeaway here is that at the scale we're at and with the efficiency gains that you might get from even a few percentage, we're exploring everything at the same time, like single phase liquid immersion cooling, still exploring that. 00:22:42.700 --> 00:22:44.700 And then how to do cold plate more efficiently. 00:22:44.700 --> 00:22:56.020 I'll also be showing something called microfluidics we're exploring, which is much more efficient than just pure liquid cold plate, which cold plate is just putting the plate, like you just said, right on top of the processor. 00:22:56.420 --> 00:22:59.660 And so the water's taking the heat away. 00:22:59.660 --> 00:23:04.420 But if we can put the liquid right into the processor, like... 00:23:04.420 --> 00:23:06.180 Are we talking channels in the processor? 00:23:06.180 --> 00:23:07.740 Channels around the processor. 00:23:07.740 --> 00:23:07.960 Okay. 00:23:07.960 --> 00:23:09.800 Just flow it right on top of it. 00:23:09.800 --> 00:23:12.440 And so that's something we're calling microfluidics. 00:23:12.440 --> 00:23:15.360 And I'll show that and talk a little bit about that tomorrow too. 00:23:15.740 --> 00:23:17.640 Offers much more efficient cooling. 00:23:17.640 --> 00:23:21.800 And it's not prime time yet, but looks incredibly promising. 00:23:21.800 --> 00:23:22.800 That looks awesome. 00:23:22.800 --> 00:23:30.040 This portion of Talk Python To Me is brought to you by the Pybytes Python Developer Mindset Program. 00:23:30.040 --> 00:23:35.180 It's run by my two friends and frequent guests, Bob Belderbos and Julian Sequeira. 00:23:35.180 --> 00:23:39.040 And instead of me telling you about it, let's hear them describe their program. 00:23:39.040 --> 00:23:40.420 Happy New Year. 00:23:40.420 --> 00:23:43.680 As we step into 2024, it's time to reflect. 00:23:44.020 --> 00:23:45.320 Think back to last year. 00:23:45.320 --> 00:23:46.760 What did you achieve with Python? 00:23:46.760 --> 00:23:52.760 If you're feeling like you haven't made the progress you wanted and procrastination got the best of you, it's not too late. 00:23:52.760 --> 00:23:54.420 This year can be different. 00:23:54.420 --> 00:23:58.000 This year can be your year of Python mastery. 00:23:58.000 --> 00:24:01.780 At Pybytes, we understand the journey of learning Python. 00:24:01.780 --> 00:24:07.560 Our coaching program is tailor-made to help you break through barriers and truly excel. 00:24:07.560 --> 00:24:11.200 Don't let another year slip by with unmet goals. 00:24:11.640 --> 00:24:16.160 Join us at Pybytes and let's make 2024 the year you conquer Python. 00:24:16.160 --> 00:24:22.120 Check out PDM today, our flagship coaching program, and let's chat about your Python journey. 00:24:22.120 --> 00:24:25.420 Apply for the Python Developer Mindset today. 00:24:25.980 --> 00:24:28.020 It's quick and free to apply. 00:24:28.020 --> 00:24:30.420 The link is in your podcast player show notes. 00:24:30.420 --> 00:24:32.740 Thanks to Pybytes for sponsoring the show. 00:24:33.780 --> 00:24:45.980 One of the things I saw in the opening keynote, I don't know if it fits into what you were just talking about or if it's also another thing where it actually had the whole motherboard submerged and then even just the entire computer is just underwater. 00:24:46.500 --> 00:24:51.840 So that was two-phase liquid immersion cooling, like I mentioned, is just dunking the whole thing in the dielectric fluid. 00:24:51.840 --> 00:24:58.720 And you had it boil at a low temperature, I guess, because the phase change is like an extremely energy-intense, aka heat exchange. 00:24:58.720 --> 00:25:02.640 Yeah, and it's actually two-phase because of the boiling. 00:25:02.640 --> 00:25:07.200 It phase changes into gas and then condenses again back into liquid. 00:25:07.600 --> 00:25:10.220 So that was the idea behind two phases. 00:25:10.220 --> 00:25:10.640 I see. 00:25:10.640 --> 00:25:15.440 Instead of just running a radiator, you almost condense it back somewhere else and then bring it back around? 00:25:15.440 --> 00:25:15.820 Yeah. 00:25:15.820 --> 00:25:16.300 Okay. 00:25:16.300 --> 00:25:17.120 Wow, cool. 00:25:17.120 --> 00:25:22.820 So if we go and run our Python codes, whether it's PASS or IaaS or whatever, what's the chance? 00:25:22.820 --> 00:25:28.460 It's hitting that or is this kind of cutting-edge stuff reserved for high-energy AI training? 00:25:28.460 --> 00:25:32.240 Is it more like if we ask ChatGPT, it's liquid cooled? 00:25:32.240 --> 00:25:35.820 Our standard data centers right now are air-cooled. 00:25:35.820 --> 00:25:36.760 So it's air-cooled servers. 00:25:37.380 --> 00:25:39.180 This Maya part is liquid cooled. 00:25:39.180 --> 00:25:44.660 So in our first summer, we've got some of our own workloads now starting to leverage Maya. 00:25:44.660 --> 00:25:47.320 Do you put on your own workloads first just in case? 00:25:47.320 --> 00:25:48.680 Well, it's just to see it. 00:25:48.680 --> 00:25:49.400 It's just we're proving it out. 00:25:49.400 --> 00:25:50.040 Shake it out, yeah. 00:25:50.040 --> 00:25:51.720 Yeah, it's kind of piloting it, yeah. 00:25:51.720 --> 00:25:55.860 You're not offering that up to the big customers just right away. 00:25:55.860 --> 00:25:59.680 And so Maya, I don't know how many people know about this either. 00:25:59.680 --> 00:26:04.460 This is one of the GPU training systems you guys have. 00:26:04.460 --> 00:26:12.120 I mean, for those who don't know, OpenAI and ChatGPT run on Azure, which probably takes a couple of cores, a couple of GPUs to make happen. 00:26:12.120 --> 00:26:12.320 Yeah. 00:26:12.320 --> 00:26:13.520 I want to talk about that. 00:26:13.740 --> 00:26:21.300 Yeah. So right now our fleet, a large-scale AI supercomputing fleet is made up of NVIDIA parts. 00:26:21.560 --> 00:26:29.000 So the previous generation was, well, the original generation that we trained GPT-3 on with OpenAI was NVIDIA V100s. 00:26:29.000 --> 00:26:32.340 Then we introduced A100s, which is what GPT-4 was trained on. 00:26:32.340 --> 00:26:38.000 And these are graphics cards, like 4080s or something, but specifically for AI, right? 00:26:38.000 --> 00:26:38.120 Yeah. 00:26:38.120 --> 00:26:38.480 Okay. 00:26:38.480 --> 00:26:38.740 That's right. 00:26:39.000 --> 00:26:47.120 And then the current generation of supercomputer we're building for OpenAI training, their next generation of their model, that's NVIDIA H100 GPUs. 00:26:47.120 --> 00:26:50.200 Then Maya is our own custom AI accelerator. 00:26:50.200 --> 00:26:51.220 So it's not a GPU. 00:26:51.220 --> 00:26:56.540 You know, one of the aspects of NVIDIA's parts has been their GPU base. 00:26:56.620 --> 00:26:59.820 So they also can do texture mapping, for example. 00:26:59.820 --> 00:27:03.260 But you don't need that if you're just doing pure AI workloads. 00:27:03.260 --> 00:27:04.400 So- 00:27:04.400 --> 00:27:05.540 Back to that specialization, right? 00:27:05.540 --> 00:27:05.780 Yeah, exactly. 00:27:05.780 --> 00:27:09.300 Like, so if you could build it just for the one thing, maybe you'd build it slightly differently. 00:27:09.300 --> 00:27:09.660 That's right. 00:27:09.660 --> 00:27:10.160 Okay. 00:27:10.160 --> 00:27:18.980 So Maya is just designed purely for matrix operations used in, in fact, low-precision matrix operations used for AI training and inference. 00:27:18.980 --> 00:27:25.020 And so that is the specialized part that we've created called Maya 100, the first generation of that. 00:27:25.340 --> 00:27:35.160 Well, if I think of like some of the stuff presented at the opening keynote and stuff here, I think the word AI was said a record number of times, right? 00:27:35.160 --> 00:27:36.240 Yeah, I don't think there was a topic there. 00:27:36.240 --> 00:27:37.060 Oh my goodness. 00:27:37.060 --> 00:27:38.020 I wasn't a part of. 00:27:38.020 --> 00:27:40.680 And so how much is this changing things for you guys? 00:27:40.680 --> 00:27:45.580 Like 12 months ago or something, Chattapiti appeared on the scene and- 00:27:45.580 --> 00:27:46.400 I mean, it's changing. 00:27:46.400 --> 00:27:48.600 It's literally changing everything. 00:27:48.600 --> 00:27:52.060 You know, Jensen was saying this is the biggest thing since the internet. 00:27:52.060 --> 00:27:52.580 Yeah. 00:27:52.580 --> 00:27:53.620 Jensen being the CEO of- 00:27:53.620 --> 00:27:54.140 Jensen, yeah. 00:27:54.140 --> 00:27:54.680 Yeah, yeah. 00:27:54.680 --> 00:27:56.740 Who was on stage with Satya at the keynote. 00:27:56.740 --> 00:27:59.060 It is changing everything. 00:27:59.060 --> 00:28:01.680 It's changing not just the product offerings. 00:28:01.680 --> 00:28:11.180 So the way that we, you know, have integrate AI into the products using Copilot, it's changing the way we develop the products as well and the way that we run our systems inside already. 00:28:11.180 --> 00:28:16.800 So for example, incident management, we've got Copilot built, you know, our own Copilot internally built into that. 00:28:16.880 --> 00:28:22.600 So somebody that's responding to an issue in our production systems can say, okay, so what's going on? 00:28:22.600 --> 00:28:24.280 What's the, what happened with this? 00:28:24.280 --> 00:28:24.640 Yeah. 00:28:25.040 --> 00:28:26.460 Show me the graph of this. 00:28:26.460 --> 00:28:30.020 You know, just be able to use human language to get caught up in what's going on. 00:28:30.020 --> 00:28:30.400 Yeah. 00:28:30.400 --> 00:28:32.700 People tell me it's just statistics, just prediction. 00:28:32.700 --> 00:28:35.300 It doesn't feel like, it doesn't feel like prediction. 00:28:35.300 --> 00:28:41.360 At some, you know, the people that say that, I think are missing the scale of the statistics. 00:28:41.960 --> 00:28:46.100 And because we're probably predicting a little bit, like thinking about what are you going to say next? 00:28:46.100 --> 00:28:46.480 Exactly. 00:28:46.480 --> 00:28:48.040 That's what we're, we're statistical. 00:28:48.040 --> 00:28:48.700 Yeah. 00:28:48.700 --> 00:28:49.140 Yeah. 00:28:49.140 --> 00:28:58.980 And so, so it's just, once you get statistics, that at a large enough scale that you start to see something that looks like what we call intelligence. 00:28:58.980 --> 00:28:59.520 Yeah. 00:28:59.520 --> 00:29:00.120 Yeah. 00:29:00.120 --> 00:29:01.300 It's, it's really incredible. 00:29:01.300 --> 00:29:09.980 I'm starting to use it to just write my, git commit logs for me, you know, push a button and it says, oh, you added error handling to the background task. 00:29:09.980 --> 00:29:13.120 So in case this fails, you'll be more resilient and it'll keep running. 00:29:13.120 --> 00:29:14.920 I'm like, that's better than I could have got it. 00:29:14.920 --> 00:29:17.240 Just thought that might crash, you know? 00:29:17.240 --> 00:29:20.560 And you just push the button and it's just, it's magic. 00:29:20.560 --> 00:29:21.120 It's magical. 00:29:21.120 --> 00:29:21.320 Yeah. 00:29:21.320 --> 00:29:22.080 It really is. 00:29:22.080 --> 00:29:28.520 I mean, it's not, it, it's called co-pilot for a reason because we're not at the point yet where you can just let it do what it does autonomously. 00:29:28.520 --> 00:29:29.960 You need to check its work. 00:29:29.960 --> 00:29:33.020 Like you need to look at it and say, oops, you know, that time it screwed it up. 00:29:33.020 --> 00:29:33.920 It didn't get it quite right. 00:29:33.920 --> 00:29:38.460 Or I need to add more context to this than it had, or extracted. 00:29:38.880 --> 00:29:42.980 So, but as far as accelerating work, it's just a game changer. 00:29:42.980 --> 00:29:44.240 Yeah, it really, really is. 00:29:44.240 --> 00:29:48.540 So before we run out of time, I want to ask you just a couple more things, a bit of a diversion. 00:29:48.540 --> 00:29:52.060 So Python, we saw Python appear in the keynote. 00:29:52.060 --> 00:29:54.680 They were showing off, I can't remember who it wasn't. 00:29:54.680 --> 00:29:56.000 Satya was whoever followed him. 00:29:56.000 --> 00:30:01.840 They look, we want to show off our new sharing of the insanely large GPUs for machine learnings. 00:30:01.900 --> 00:30:05.120 Let's just pull up some Python and a Jupyter notebook and I'll just check that out. 00:30:05.120 --> 00:30:06.640 And you're like, wait, where are we again? 00:30:06.640 --> 00:30:08.020 Really interesting. 00:30:08.020 --> 00:30:11.540 So, you know, you said you're using a little bit of Python yourself. 00:30:11.540 --> 00:30:13.180 Like what's Python look like in your world? 00:30:13.180 --> 00:30:16.700 Well, so the reason that I'm using Python is I took a sabbatical this summer. 00:30:16.700 --> 00:30:19.720 And so I just, I was like, I'm going to do some AI research. 00:30:19.720 --> 00:30:21.160 So I got connected with an AI researcher. 00:30:21.160 --> 00:30:23.600 In fact, I'm going to talk about this at my keynote tomorrow. 00:30:23.600 --> 00:30:24.920 some of the work that came out of it. 00:30:24.920 --> 00:30:28.940 The obviously AI is completely Python at these days. 00:30:28.940 --> 00:30:29.140 Yeah. 00:30:29.140 --> 00:30:29.900 Almost entirely. 00:30:29.900 --> 00:30:30.140 Yeah. 00:30:30.140 --> 00:30:35.140 So I was, I spent the whole summer and I still am spending my time in Python, 00:30:35.140 --> 00:30:39.220 Jupyter notebooks, and then Python scripts when you want to do some, some run for, 00:30:39.220 --> 00:30:40.360 for final result. 00:30:40.360 --> 00:30:44.420 so I hadn't used really Python before other than in passing. 00:30:44.420 --> 00:30:45.320 I mean, it's a very language. 00:30:45.320 --> 00:30:46.720 It's very easy to pick up. 00:30:46.720 --> 00:30:47.000 Yeah. 00:30:47.000 --> 00:30:49.520 There's a, there's a t-shirt that says I learned Python. 00:30:49.520 --> 00:30:50.320 It was a good weekend. 00:30:50.320 --> 00:30:50.600 Yeah. 00:30:50.600 --> 00:30:51.980 It's a bit of a joke, but it's a good joke. 00:30:51.980 --> 00:30:52.160 Yeah. 00:30:52.500 --> 00:30:57.000 And I think that's what makes it so powerful is that it's so easy to pick up, 00:30:57.000 --> 00:30:59.380 but what's made it even easier for me to pick it up. 00:30:59.380 --> 00:31:04.660 I'll, I'd say that I'm a, my, I'm a mediocre Python programmer, but I'm using Copilot. 00:31:04.660 --> 00:31:07.440 And that's made me an expert Python coder. 00:31:07.440 --> 00:31:08.960 How do I do this? 00:31:08.960 --> 00:31:09.260 Yeah. 00:31:09.260 --> 00:31:13.700 And it's like, I've never, I don't go to stack over, you know, it's a question. 00:31:13.700 --> 00:31:15.240 I don't go for stack overflow for questions. 00:31:15.240 --> 00:31:17.780 I haven't had to get a book on Python. 00:31:17.780 --> 00:31:22.000 I basically just either ask Copilot explicitly, like, 00:31:22.100 --> 00:31:26.760 how do I do this or write me this, or I put it in the function or in the comment and it 00:31:26.760 --> 00:31:27.640 gets it done for me. 00:31:27.640 --> 00:31:31.760 And there's a, occasionally I'll have to go hand met, edit it and figure out what's going 00:31:31.760 --> 00:31:32.100 on. 00:31:32.100 --> 00:31:35.180 But for the most part, it is writing almost all my code. 00:31:35.600 --> 00:31:39.100 And so my goal is how can I just not have it write everything for me? 00:31:39.100 --> 00:31:41.880 So that has kind of become the way that I program in Python. 00:31:41.880 --> 00:31:48.260 And I think Python and AI and it, the knowledge of Copilot for Python, because OpenAI, obviously 00:31:48.260 --> 00:31:55.400 for their own purposes has made GPT-4 and GPT three, five before it really know Python. 00:31:55.480 --> 00:31:59.040 I hadn't really thought of that connection, but of course they wanted to answer Python 00:31:59.040 --> 00:31:59.720 questions, I'm sure. 00:31:59.720 --> 00:32:00.140 For themselves. 00:32:00.140 --> 00:32:00.460 Yeah. 00:32:00.520 --> 00:32:07.640 So I think when it comes to seeing what AI can do for programming, Python is at the forefront 00:32:07.640 --> 00:32:08.180 of that. 00:32:08.180 --> 00:32:09.740 What was your impression of it? 00:32:09.740 --> 00:32:12.720 I mean, I'm sure you've probably seen it before, but like, what's your impression of working in 00:32:12.720 --> 00:32:17.340 it coming from a curly brace semicolon type language like C++ or something? 00:32:17.340 --> 00:32:22.260 Because they drop a bunch of parentheses, they have these tab space, these four spaces rules 00:32:22.260 --> 00:32:22.660 and stuff. 00:32:22.660 --> 00:32:25.340 Well, it's, you know, the YAML versus Jason. 00:32:25.340 --> 00:32:27.240 It is kind of a debate, isn't it? 00:32:27.240 --> 00:32:30.460 But I mean, I've gotten used to it. 00:32:30.460 --> 00:32:31.480 It's not a, it's not a big deal. 00:32:31.480 --> 00:32:35.180 And I find it's, it's less verbose than C. 00:32:35.180 --> 00:32:36.700 There's less typing. 00:32:36.700 --> 00:32:38.000 Less symbol noise. 00:32:38.000 --> 00:32:39.140 You can just kind of get the essence. 00:32:39.140 --> 00:32:39.420 Yeah. 00:32:39.420 --> 00:32:39.700 Yeah. 00:32:39.700 --> 00:32:40.160 Yeah. 00:32:40.160 --> 00:32:42.380 I kind of had that experience as well, coming from a C based language. 00:32:42.380 --> 00:32:43.600 I'm like, wow, this is really weird. 00:32:43.600 --> 00:32:47.500 And then after I went back to C#, I'm like, but this is also weird. 00:32:47.500 --> 00:32:49.660 And I kind of like the clarity over here. 00:32:49.660 --> 00:32:51.500 So now what do I do with life? 00:32:51.500 --> 00:32:54.360 And then you go back and like semicolons are annoying now. 00:32:54.360 --> 00:32:55.140 Yes, exactly. 00:32:55.140 --> 00:32:56.380 I like, I thought they were needed. 00:32:56.380 --> 00:32:56.980 They're not needed. 00:32:56.980 --> 00:32:57.680 What's going on? 00:32:57.680 --> 00:33:02.780 Another thing that I think, you know, maybe people really enjoy hearing a bit about, and 00:33:02.780 --> 00:33:09.340 I'm a big fan of, you wrote a three part series of novels about computer hackers called Zero 00:33:09.340 --> 00:33:09.560 Day. 00:33:09.560 --> 00:33:10.780 Really good. 00:33:10.780 --> 00:33:16.200 I read all of them back when they came out and so much of this like computer mystery stuff 00:33:16.200 --> 00:33:18.060 is like, oh, they're using VB6. 00:33:18.060 --> 00:33:19.060 I'm going to get their IP address. 00:33:19.060 --> 00:33:19.780 You're like, wait, what? 00:33:20.140 --> 00:33:23.580 I mean, those words are meaningful, but the sense is not right. 00:33:23.580 --> 00:33:28.080 And you know, your books are like a lot of sort of spy stuff, but also a lot of really 00:33:28.080 --> 00:33:31.940 cool, legit, reasonably possible computer stuff. 00:33:31.940 --> 00:33:32.100 Yeah. 00:33:32.100 --> 00:33:34.340 Tell people a quick bit about that if they want to catch up. 00:33:34.340 --> 00:33:37.940 I love cyber, I love thrillers growing up, techno thrillers. 00:33:37.940 --> 00:33:42.460 I read Dramatistrain when I was like in seventh grade and I was like, this book is so cool 00:33:42.460 --> 00:33:48.040 because it's like, I'm learning science plus it's, you know, it's really exciting. 00:33:48.040 --> 00:33:49.740 So I've always wanted to write one. 00:33:49.780 --> 00:33:56.760 And then coming into after, into the late 1990s, when you started to see some of these large 00:33:56.760 --> 00:34:01.920 scale viruses, I was just thinking, this is such a powerful weapon for somebody to cause 00:34:01.920 --> 00:34:02.380 destruction. 00:34:02.380 --> 00:34:03.500 And then 9-11 happened. 00:34:03.500 --> 00:34:09.540 I'm like, all right, logical next step is leveraging a cyber weapon to do something with the same 00:34:09.540 --> 00:34:09.940 goals. 00:34:09.940 --> 00:34:16.460 And so that's what led me to write Zero Day, which is taking that idea of using a cyber 00:34:16.460 --> 00:34:17.380 weapon for terrorism. 00:34:17.380 --> 00:34:20.160 Then I was like, oh, that book was really well received. 00:34:20.160 --> 00:34:21.380 I had a lot of fun doing it. 00:34:21.380 --> 00:34:22.380 So let me write the next one. 00:34:22.380 --> 00:34:26.820 And I wanted to continue in this theme with the same characters, Daryl, Hagen and, and, 00:34:26.820 --> 00:34:29.940 um, Jeff Akin and say, what, what's something else? 00:34:29.940 --> 00:34:34.360 What's another cyber security angle that I can take a look at in the second one. 00:34:34.360 --> 00:34:37.580 So the second one was state sponsored, cyber espionage. 00:34:37.700 --> 00:34:41.380 And I actually, the ironic thing is I'd already had Iran in the story. 00:34:41.380 --> 00:34:42.740 I'd already had China in the story. 00:34:42.740 --> 00:34:46.480 I had people trying to figure out how to get Iran a nuclear weapon. 00:34:46.480 --> 00:34:49.820 And then Stuxnet happened right when I was still writing the book. 00:34:49.820 --> 00:34:52.960 And I'm like, okay, this is like a small part of my plot line. 00:34:52.960 --> 00:34:53.180 Yeah. 00:34:53.180 --> 00:34:54.460 Line it up for you. 00:34:54.460 --> 00:34:57.460 So I had to change the book a little to acknowledge Stuxnet happening. 00:34:57.460 --> 00:35:01.440 And then the third one was about insider threats, which I think is one of the toughest 00:35:01.440 --> 00:35:02.260 threats to deal with. 00:35:02.260 --> 00:35:07.660 In this case, it was a long range plot from some people that wanted to compromise, 00:35:07.660 --> 00:35:12.640 stock exchange and kind of a mixture of high frequency trading and insider threat with 00:35:12.640 --> 00:35:15.860 cybersecurity systems, was the third one called rogue code. 00:35:15.860 --> 00:35:16.760 Yeah. 00:35:16.760 --> 00:35:17.780 So they were all really good. 00:35:17.780 --> 00:35:18.460 I really enjoyed it. 00:35:18.460 --> 00:35:19.960 Were you a fan of Mr. Robot? 00:35:19.960 --> 00:35:21.040 Did you ever watch that series? 00:35:21.040 --> 00:35:21.420 I did. 00:35:21.420 --> 00:35:22.080 I love that series. 00:35:22.080 --> 00:35:22.340 Yeah. 00:35:22.340 --> 00:35:23.220 Oh my gosh. 00:35:23.220 --> 00:35:25.380 Again, it's, it seemed pretty plausible. 00:35:25.380 --> 00:35:26.760 Yeah. 00:35:26.760 --> 00:35:27.580 I really liked that. 00:35:27.580 --> 00:35:30.780 Imagine a lot of people out there listening have seen, seen Mr. Robot as well. 00:35:30.780 --> 00:35:34.780 If they want, you know, that kind of idea, but in just a series, they can binge. 00:35:34.900 --> 00:35:35.220 Yeah. 00:35:35.220 --> 00:35:35.420 Cool. 00:35:35.420 --> 00:35:39.440 Maybe we should wrap up our, our chat here, but just a quick of like some of the future 00:35:39.440 --> 00:35:44.980 things you talked about, like rapidly deploying some of these data centers and some of these 00:35:44.980 --> 00:35:46.220 Ballard systems. 00:35:46.220 --> 00:35:50.920 Maybe just give us a sense of like, even, like disaggregated rack architecture. 00:35:50.920 --> 00:35:57.060 Do you have, instead of having a GPU alongside a server, like a rack of GPUs and then optical 00:35:57.060 --> 00:35:58.840 connections to a rack of servers? 00:35:58.840 --> 00:36:01.040 Like give us a sense of some of the stuff where it's going. 00:36:01.280 --> 00:36:01.400 Yeah. 00:36:01.400 --> 00:36:03.340 So that's some of the stuff that we've exploring. 00:36:03.340 --> 00:36:08.100 Like I mentioned, we're taking a look at lots of different ways to re-architect the data 00:36:08.100 --> 00:36:09.000 center to be more efficient. 00:36:09.000 --> 00:36:14.100 And one of the ways that you get efficient is by, and in reduced fragmentation is by having 00:36:14.100 --> 00:36:16.320 larger pools to allocate resources from. 00:36:16.320 --> 00:36:20.500 If you think about allocating a virtual machine on a server, how much RAM can you give it at 00:36:20.500 --> 00:36:20.880 most? 00:36:20.880 --> 00:36:24.120 Well, as much as sitting on the server, how many GPUs can you attach to it? 00:36:24.120 --> 00:36:26.580 Well, as most as are attached to that server. 00:36:26.680 --> 00:36:26.860 All right. 00:36:26.860 --> 00:36:27.800 How many PCI slots? 00:36:27.800 --> 00:36:27.960 Yeah. 00:36:27.960 --> 00:36:28.440 Yeah. 00:36:28.440 --> 00:36:34.400 So, but if you think about, I've got a large resource pool, it's a whole group of GPUs 00:36:34.400 --> 00:36:37.440 that I can be able to give it as many GPUs as 50. 00:36:37.440 --> 00:36:37.860 Yeah. 00:36:37.860 --> 00:36:38.340 Ask for 50. 00:36:38.340 --> 00:36:39.020 Ask for 50. 00:36:39.020 --> 00:36:39.300 Exactly. 00:36:39.300 --> 00:36:44.140 The benefits of pooling for resource allocation are that you reduce fragmentation and you 00:36:44.140 --> 00:36:44.800 get more flexibility. 00:36:45.060 --> 00:36:49.940 So we've been trying to explore how we can do this from rack scale disaggregation of saying 00:36:49.940 --> 00:36:54.000 there's a whole bunch of SSDs at the top of the rack, then there's a bunch of GPUs, and 00:36:54.000 --> 00:36:56.220 then there's a bunch of CPU cores. 00:36:56.220 --> 00:36:58.860 And let's just compose the system dynamically. 00:36:58.860 --> 00:37:02.340 There's a bunch of challenges from a resiliency perspective. 00:37:02.340 --> 00:37:07.380 Like how do you prevent one failure of the GPU part of the system, bringing down the whole 00:37:07.380 --> 00:37:07.780 rack? 00:37:07.780 --> 00:37:11.340 For example, there's latency and bandwidth challenges. 00:37:11.740 --> 00:37:15.420 Like how do you, when you're sitting there on the PCI bus, you get a whole bunch of bandwidth 00:37:15.420 --> 00:37:16.760 and you get very low latency. 00:37:16.760 --> 00:37:20.460 If you're going across the rack, you might have the same bandwidth. 00:37:20.460 --> 00:37:24.640 You might have lower bandwidth just because you can't deliver that much bandwidth out of 00:37:24.640 --> 00:37:25.680 the GPUs. 00:37:25.680 --> 00:37:25.820 Right. 00:37:25.820 --> 00:37:29.440 All the systems are optimized to make assumptions about these numbers. 00:37:29.440 --> 00:37:29.900 Exactly. 00:37:29.900 --> 00:37:31.640 And your latency is going to be higher. 00:37:31.640 --> 00:37:33.360 And so some workloads can't tolerate their latency. 00:37:33.360 --> 00:37:36.960 So we've been exploring disaggregated memory, disaggregated GPUs. 00:37:36.960 --> 00:37:38.220 I've shown demos of both of them. 00:37:38.220 --> 00:37:40.120 We're still exploring those. 00:37:40.620 --> 00:37:42.580 We're not, you know, it's not ready for production. 00:37:42.580 --> 00:37:42.780 Yeah. 00:37:42.780 --> 00:37:44.820 Disaggregated memory or GPUs? 00:37:44.820 --> 00:37:45.020 Yeah. 00:37:45.020 --> 00:37:48.020 I would guess memory, but I have a zero experience. 00:37:48.020 --> 00:37:53.180 Memory is challenging because there are certain GPU workloads that aren't so latency sensitive, 00:37:53.180 --> 00:37:54.400 like AI training. 00:37:54.400 --> 00:37:54.620 Sure. 00:37:54.620 --> 00:37:56.920 Like a batch job sort of thing. 00:37:56.920 --> 00:37:57.460 Yeah. 00:37:57.460 --> 00:38:02.360 But when it comes to memory, you almost always see the latency. 00:38:02.640 --> 00:38:07.480 And so what we think we can do is get remote memory down to NUMA, you know, speaking of 00:38:07.480 --> 00:38:10.960 non-uniform memory architecture latency down to that level. 00:38:10.960 --> 00:38:13.560 And a lot of applications can tolerate that. 00:38:13.560 --> 00:38:14.040 Okay. 00:38:14.040 --> 00:38:18.180 And so we have a memory tiering where you've got close memory that's on the system and then 00:38:18.180 --> 00:38:19.680 remote memory, which is like NUMA latency. 00:38:19.900 --> 00:38:22.840 Kind of like a L2 cache, but like a bigger idea of it. 00:38:22.840 --> 00:38:23.860 Very, very cool. 00:38:23.860 --> 00:38:26.240 I think because you're doing such neat stuff. 00:38:26.240 --> 00:38:33.220 And when you see these hyperscale clouds, I think a lot of what people see is the insane 00:38:33.220 --> 00:38:34.580 dashboard of choices. 00:38:34.580 --> 00:38:35.520 Like, do I do? 00:38:35.520 --> 00:38:35.920 Yeah. 00:38:35.920 --> 00:38:36.780 Do I do routing? 00:38:36.780 --> 00:38:37.780 Do I do firewalls? 00:38:37.780 --> 00:38:39.640 Do I do VPCs? 00:38:39.640 --> 00:38:41.280 Do I do like paths? 00:38:41.280 --> 00:38:42.360 I ask, what do I do? 00:38:42.360 --> 00:38:46.600 But oftentimes I don't really think about like, well, you're getting a slice of this giant 00:38:46.600 --> 00:38:50.600 server and, you know, maybe someday it'll live under the ocean or whatever. 00:38:50.600 --> 00:38:50.880 Right. 00:38:50.880 --> 00:38:51.800 So it was really cool to. 00:38:51.800 --> 00:38:52.120 Yeah. 00:38:52.120 --> 00:38:56.100 And I think what you're seeing is the cloud, it started with a few basic building blocks. 00:38:56.100 --> 00:39:01.180 And then we started to explore lots of different directions of creating lots of different paths 00:39:01.180 --> 00:39:05.080 services and paths services for compute and then different data offerings. 00:39:05.080 --> 00:39:09.080 I think the space, and again, coming to the workload, you get this. 00:39:09.400 --> 00:39:10.620 Is it high? 00:39:10.620 --> 00:39:11.960 Do you need key value store? 00:39:11.960 --> 00:39:13.440 Do you need a vectorized database? 00:39:13.440 --> 00:39:18.100 Do you need, and do you need any of those to be extreme performance? 00:39:18.100 --> 00:39:23.740 Because then if you need extreme performance, go for the design for purpose vectorized database. 00:39:23.740 --> 00:39:30.140 If you want key value with vectorization, but it's okay if the vectorization isn't the fastest 00:39:30.140 --> 00:39:32.960 possible, you know, you can go use this other offering. 00:39:32.960 --> 00:39:39.380 So that's why the list of options has continued to expand is just because every workload says, 00:39:39.380 --> 00:39:40.100 I need this. 00:39:40.100 --> 00:39:41.600 And that's the most important thing to me. 00:39:41.600 --> 00:39:43.000 And the other one says, no, I need this. 00:39:43.000 --> 00:39:43.980 That's the most important thing. 00:39:43.980 --> 00:39:45.280 And others are like, I don't care. 00:39:45.280 --> 00:39:49.460 Well, as it becomes the mainframe of the world, right? 00:39:49.460 --> 00:39:51.260 There's a lot of different types of apps running on it. 00:39:51.260 --> 00:39:51.520 Yeah. 00:39:51.520 --> 00:39:52.020 Yeah. 00:39:52.020 --> 00:39:52.760 Awesome. 00:39:52.760 --> 00:39:53.600 All right, Mark. 00:39:53.600 --> 00:39:54.400 Final call to action. 00:39:54.660 --> 00:39:58.000 People maybe want to learn more about some of the stuff we saw here, see some pictures, 00:39:58.000 --> 00:39:59.960 but maybe also just do more with Azure. 00:39:59.960 --> 00:40:00.440 What do you say? 00:40:00.440 --> 00:40:01.640 So a couple of things. 00:40:01.640 --> 00:40:06.300 One is I've been doing a series of Azure innovation talks at build and ignite sessions. 00:40:06.300 --> 00:40:09.200 So go back to the last build and you'll see the most recent one of those. 00:40:09.200 --> 00:40:13.660 And then at this ignite, I'm doing one that's just looking at AI related innovation. 00:40:13.940 --> 00:40:19.500 So that's on Friday, tomorrow here at ignite, and it'll be available on demand. 00:40:19.500 --> 00:40:20.460 So that's awesome. 00:40:20.460 --> 00:40:20.820 Yeah. 00:40:20.820 --> 00:40:23.060 I'll grab the links to some of those and put them in the show notes for people. 00:40:23.060 --> 00:40:23.760 Excellent. 00:40:23.760 --> 00:40:24.980 Well, thanks so much for being on the show. 00:40:24.980 --> 00:40:25.260 All right. 00:40:25.260 --> 00:40:25.860 Thanks for having me. 00:40:25.860 --> 00:40:26.080 Yeah. 00:40:26.080 --> 00:40:30.280 This has been another episode of talk Python to me. 00:40:30.280 --> 00:40:32.100 Thank you to our sponsors. 00:40:32.100 --> 00:40:33.700 Be sure to check out what they're offering. 00:40:33.700 --> 00:40:35.120 It really helps support the show. 00:40:35.120 --> 00:40:41.080 This episode is sponsored by Posit Connect from the makers of shiny publish, share, and deploy 00:40:41.080 --> 00:40:44.080 all of your data projects that you're creating using Python. 00:40:44.080 --> 00:40:50.560 Streamlit, dash, shiny, bokeh, FastAPI, flask, quattro, reports, dashboards, and APIs. 00:40:50.560 --> 00:40:52.940 Posit Connect supports all of them. 00:40:52.940 --> 00:40:58.620 Try Posit Connect for free by going to talkpython.fm/posit, P-O-S-I-T. 00:40:58.620 --> 00:41:01.880 Are you ready to level up your Python career? 00:41:01.880 --> 00:41:06.780 And could you use a little bit of personal and individualized guidance to do so? 00:41:07.220 --> 00:41:13.600 Check out the PyBytes Python developer mindset program at talkpython.fm/PDM. 00:41:13.600 --> 00:41:15.540 Want to level up your Python? 00:41:15.540 --> 00:41:19.620 We have one of the largest catalogs of Python video courses over at Talk Python. 00:41:19.620 --> 00:41:24.720 Our content ranges from true beginners to deeply advanced topics like memory and async. 00:41:24.720 --> 00:41:27.380 And best of all, there's not a subscription in sight. 00:41:27.380 --> 00:41:30.280 Check it out for yourself at training.talkpython.fm. 00:41:30.640 --> 00:41:35.160 Be sure to subscribe to the show, open your favorite podcast app, and search for Python. 00:41:35.160 --> 00:41:36.480 We should be right at the top. 00:41:36.480 --> 00:41:41.640 You can also find the iTunes feed at /itunes, the Google Play feed at /play, 00:41:41.640 --> 00:41:45.840 and the direct RSS feed at /rss on talkpython.fm. 00:41:45.840 --> 00:41:48.800 We're live streaming most of our recordings these days. 00:41:48.800 --> 00:41:52.220 If you want to be part of the show and have your comments featured on the air, 00:41:52.220 --> 00:41:56.600 be sure to subscribe to our YouTube channel at talkpython.fm/youtube. 00:41:57.320 --> 00:41:58.700 This is your host, Michael Kennedy. 00:41:58.700 --> 00:42:00.000 Thanks so much for listening. 00:42:00.000 --> 00:42:01.160 I really appreciate it. 00:42:01.160 --> 00:42:03.060 Now get out there and write some Python code. 00:42:03.060 --> 00:42:23.940 I'll see you next time.