#191: Python's journey at Microsoft Transcript

00:00 When you think about Microsoft, do you think about Python? Maybe not, but you probably should.

00:05 They've been doing an incredible amount of work to improve Python for folks on Windows,

00:09 as well as the broader community. You can, of course, look at the wild growth of Visual Studio

00:15 Code, but did you know that five of the core developers work there and that the majority of

00:20 Python development actually happens on Windows? Join me along with Steve Dower, a core developer

00:25 working at Microsoft who published an amazing retrospective on Python at Microsoft entitled

00:31 Python at Microsoft, Flying Under the Radar. This is Talk Python to Me, recorded December 6th, 2018.

00:37 Welcome to Talk Python to Me, a weekly podcast on Python, the language, the library,

00:55 the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter,

01:00 where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm,

01:04 and follow the show on Twitter via at Talk Python. Steve, welcome back to Talk Python.

01:09 Thanks for having me back.

01:10 Yeah, it's great to have you back. There's so many interesting things that you're doing,

01:13 and you've been really instrumental in pushing Python on its most popular platform, actually,

01:20 on Windows. And I think you actually have a few interesting surprises for the listeners. So I'm

01:26 really excited about what we're going to talk about. But let's set the stage by just talking about what

01:31 you do day to day. You're a Python core developer, and you work at Microsoft. Take it from there.

01:36 Yeah, so I really get two hats in most conversations. I'm either wearing my Microsoft hat or my CPython hat.

01:42 And I feel like most times when I come and chat with you, I'm normally wearing the Microsoft hat,

01:47 which is fun. And that's certainly the one I have on today. So for CPython, I'm one of the Windows

01:52 experts, do a lot of the Windows support and builds and everything. At Microsoft, I'm one of the Python

02:00 experts. And so I get to kind of roam around the company working with a lot of groups as we really

02:06 try and ramp up on Python in a big way, getting to help out various teams, make sure they're doing

02:12 a good job. Like, we don't have a huge base of, you know, Python culture, Python experts throughout

02:19 the company. So there's a lot of really, really good engineers, like expert engineers in all of

02:24 their languages. And bridging that gap between, you know, I'm an expert C# programmer, I'm an expert

02:30 C++ programmer. And how do we make you look like an expert Python programmer is a really big part of

02:37 what I'm doing right now. But it's a lot of fun. Yeah, I guess two things really strike me there. One is,

02:41 you do kind of live in this middle ground where the context matters so much, right? Like when you're

02:47 at, say, PyCon, people are like, Oh, my gosh, you're a C core developer, and you work at Microsoft.

02:52 What's that like? Right? Whereas when you're at Microsoft, they're like, what do you mean you're

02:56 a Python person? Why don't you do C# and C++? What's going on here? Right? So there's like this

03:01 really big context switch, I bet. Oh, yeah, I have virtual hats that I will literally mime putting on and

03:07 off. And I feel like I just need to get real ones. Yeah, absolutely. So there's a couple of things we're going to

03:12 talk about. And the first one really want to dig into is Python's journey at Microsoft, because Microsoft's one of

03:20 these companies that started out not super open in terms of its open source contributions, and its

03:26 culture around open source, right? Famously, Balmer had not super positive things to say about Linux,

03:33 for example. But I feel like Microsoft has really made this transformation. And you've written this

03:40 really cool article essay, maybe on your personal journey that sort of charts some of those things

03:47 as well. So maybe we'll start there. Yeah, it felt a bit like writing like the start of my memoirs,

03:53 which I feel like I'm too young to start doing that. But, but no, I was I was invited. We've got

03:59 a series of people's kind of open source journeys, open source stories coming out right now. We've already

04:05 had one about C#, and rewriting the compiler for that one about the dotnet foundation, if I'm

04:13 recalling correctly. And yeah, now one about Python's journey. And it was, I was thrilled to get to write

04:19 it, because it has been an exciting time. And it's taken a number of years. But it's one of those things

04:25 where I've at least been able to observe it along the way and participate in a few places.

04:29 Yeah, so where does this journey start? You know, traditionally, Microsoft's been a kind of,

04:35 we're going to build it here, even if it exists, we're going to create our own version of the thing,

04:41 right? There's famously Java and C#. In the early days, you know, C# was something or a

04:46 reaction to Java, I think that was more of a legal issue on around Sun and Oracle than it was more

04:51 the not invented here thing. But there has been that kind of stuff around, say, like source control and

04:56 other things that in the early days, I feel like maybe made Python not as welcome as it could have

05:03 been. Yeah, yeah, there's, there's certainly when a company gets to a certain size, there's always a

05:07 lot of that going on, even within the company, like we still have teams all over the place inventing the

05:13 same thing, not realizing that another team at the same company is doing it at the same time,

05:17 which is really interesting. And one of the great things that I get to do, because I bounce around

05:22 a lot of teams is actually connect some of these up and say, Hey, you know, this other team is doing the

05:25 same thing as you. But we have, we're definitely a lot better at looking out than we used to be, because

05:32 there was certainly a time that we'd hear about problems, or we'd recognize problems, or we'd have

05:36 problems ourselves, and go, Well, the only people who can solve this is us. So let's build a solution. And

05:43 that's dramatically changed. Like the first thing now is, let's look out and see how other people have

05:48 solved it. And let's help them, you know, let's help our developers use it. Let's help developers

05:56 outside the company be able to use it as well. And it's also just hiring. I mean, yes, people change at

06:02 the company, people, you know, the older people leave, the younger people join. And they're coming in with a

06:08 totally different experience these days. All the open source stuff is everywhere throughout academia,

06:14 it's everywhere, obviously, throughout open source. And that's what people are coming into the company

06:17 with, right? Even scientists are now, you know, doing pull requests and stuff on GitHub, right? Like,

06:23 or writing Python code instead of MATLAB or Mathematica, like it's all aspects of academia seem to be

06:30 embracing open source, not just the computer science side.

06:33 Yeah, no, absolutely. And I mean, some of the biggest things that we look at and say, oh,

06:39 this is a computer science thing were created by scientists in the first place. Like,

06:43 Travis Oliphant is quite happy to stand up and say he's not a computer scientist. And yet,

06:48 where did NumPy come from? Where did SciPy come from?

06:50 Exactly. Exactly. And it's made such a massive, massive difference. Yeah. So I guess some of the

06:56 things that I see that sort of highlight some of this contrast, and I only know from the outside,

07:01 right? I don't bounce around inside the engineering teams there, are things like Microsoft created

07:06 CodePlex as sort of an alternative to places like GitHub. I don't know exactly the timing of when

07:12 GitHub came out versus CodePlex, but then eventually I decided, you know what, everybody's at GitHub.

07:16 Let's just move things like the .NET open source projects over to GitHub properly, ASP.NET famously,

07:23 for example. Then, well, let's just buy GitHub because that's like where the action is,

07:28 right. And really embracing this place that is, you know, has plenty of Microsoft stuff there,

07:35 but plenty of Python and other others as well. And, you know, there's other examples of those types of

07:41 things where it's like, well, it started out maybe this private build your own thing, and now we're

07:45 going to go do our, you know, another one is like the source control story, like TFS and all that,

07:50 and now Git and GitHub and so on. So I think those are the outside changes I see.

07:54 But, you know, on the inside, you've seen maybe, I guess one of the biggest stories is Python is

08:01 really starting to gain true traction, right? It's like, it's starting to show up in lots of products

08:06 and get some real legitimacy, not just the stepchild thing that we have to care for because some people

08:12 demand it.

08:12 Yeah, no, it absolutely is. One of the mandates that came down a little while back is that all of,

08:18 like, one of the must support languages for APIs and, you know, anything that you might want to

08:24 manage from outside the company, anything you might want to manage on Azure. So if you want to create

08:29 virtual machines or create new storage accounts, be able to push, pull files from wherever, Python is

08:34 one of the required languages. Like, you can't call your service ready until it has Python support for it,

08:40 along with a handful of other languages. But that hitting that point was was really exciting. And also a little

08:47 bit terrifying, because suddenly there were huge, huge code bases that were suddenly deemed not ready. And of

08:54 course, we didn't pull anything, right? Yeah, it was it was an internal. The next thing you need to do is add your

09:00 Python support, right? And so then all these teams with this complicated code base come to you and your couple

09:06 colleagues and say, we need some help. Yeah. And luckily, a lot of luckily, a lot of them did. There was certainly early on a few that that

09:13 would happily just go their own way. And, you know, we'd find out later on, this hasn't happened for a few years now. But we did

09:21 occasionally get emails saying, Have you seen this JavaScript code that someone wrote in Python and then published and called it

09:26 Microsoft? We're like, Well, we'll go find them. Thank you. That hasn't happened recently. But there's definitely some transition to get there. And it has reached the point, like I mentioned in that story on medium, the Azure CLI team, actually coming to us and saying, We think Python's a good choice for this. Now, for writing a command line tool, that is going to get huge. And we have to be able to manage everything.

09:51 Yeah, that's really great. And in your story, you talk a little bit about, you know, they expected this to take a long time. And you're like, No, no, we can knock this out pretty quickly. And they're like, Uh huh, sure. Well, just give it a try. Right? Like, you want to talk about that bit?

10:05 Yeah. Well, I mean, any, I feel like most people who've built a command line in Python using argpars, not going to be too surprised at how quickly you can get something working. But I think they were previously doing it in Node.js. And they'd started it much, much earlier on and basically ended up with their entire command line parsing library written by themselves. And so by comparison to that, being able to take something out of the standard library for a language and just use it, and be able to use it so dynamically,

10:33 like they have thousands and thousands of commands, which Python can handle because you haven't statically declared everything like you haven't had to write them all out in full, you can loop over a list of files and read stuff out of it and build up your command line that way.

10:48 And so it was, I mean, it was, it was an intense few days. Like, don't, don't get me wrong.

10:53 Of course.

10:54 But, but yeah, it, the first version came out, you can dive back into the Git history and, and see my commits. And I was basically going straight to the repo from the start. So they're all there.

11:05 And yeah, it came out with something that had kind of a scattering of features that they said they were interested in.

11:11 And then got to stand up and present it and hand it off, which is a really nice feeling to, to be able to give some code to a team and, and walk through it and spend an hour showing off the bits and pieces and have them say, you just saved us months of work.

11:26 Thank you so much.

11:27 Yeah.

11:28 Like that, that's definitely one of the best feelings that I've had in, in what I've gotten to do with Microsoft.

11:32 Yeah. That's a crazy, crazy experience. And I, you know, you've heard stories about Python having this effect in different ways, but that's, that's a pretty stark contrast right there. That's great.

11:43 Is it still in the Azure CLI? Is that still in Python? Is that shipping now?

11:49 Yeah, it's absolutely still in Python. It bears, I think there's maybe two lines of code that are still there from what I wrote because they've

11:56 gone and, and rewritten all of it. They've actually eventually migrated off argpars because it turns out that when you are getting up to thousands of sub commands that has some performance problems.

12:07 Whoever wrote that is like, Oh, what's the upper limit? Like what's the most commands you've ever seen in an app? 24. Okay. Well, this is going to be fine. Right?

12:16 I stumbled into a conversation at a conference between two people working on other command line libraries. I forget exactly what the libraries were, but they were

12:26 discussing how their performance was kind of falling off around the 50, 60 sub commands level. And, you know, was that okay? That's probably okay. And I'm like, Hey guys, we've, we've got a few more than that.

12:39 But the library they, they went and turned it into, as I said, Microsoft is full of amazing software engineers. And a few of them learned Python, didn't necessarily know it to begin with and created what was eventually refactored out into this library called knack.

12:54 That is a highly scalable command line parser. The amount of work it takes to get started is higher than argpars. But the result that you get is much, much better.

13:08 How interesting. I'd never heard of knack actually. That's cool.

13:11 I hadn't actually heard of it either until I was going to the team and saying, Hey, there are people who would love your command line library. Have you thought about refactoring out? And they just sent me a GitHub repo back and said, here it is.

13:21 Oh, beautiful.

13:51 thousands of sub commands. Wow.

13:53 And I don't actually know that he realizes how much of an achievement he's already made to get it down that far with something so big.

14:03 That is so cool. I mean, it's projects like that, that kind of push the boundaries and make it better for everyone. If that gets pushed back into CPython.

14:10 I, yeah, I hope so.

14:11 Yeah. Let's talk about the early days. I guess the, the very first recollection I have of Python and maybe similarly Ruby making its way around Microsoft is first. I heard of iron Ruby and then iron Python. I think that was the order they were created. And then maybe shortly after that was Python tools for visual studio. This is not VS Code that people know today, right? This is like traditional.

14:39 So that's the original Windows only visual studio. You had these tools you could plug in for Python, right?

14:44 Yeah. So that's basically the timeline there as well. I came in just after Python tools for visual studio was getting started. I think I was there for the 1.0 ship party. I don't remember exactly. I can probably go back and figure that out, but I was definitely there for a party, which was one of the earliest releases. But yeah, before that, there was a project to make the CLR, the common language runtime work well for dynamic languages.

15:10 The thing that .NET runs on top of to make it work well for dynamic languages, which have, you know, a whole different style of, you know, when you, you have the code and you compile it into this intermediate language, but you still don't know what your name lookups are going to find, which we're totally used to in Python.

15:29 But the common language runtime was not initially designed for that. And so there was this project to make a dynamic dynamic one that could handle all of that.

15:38 And IonPython and IonRuby were kind of the test cases for let's show this working on real languages. And both of them got to very good states in terms of supporting the language at that point in time. IonPython is still going in bits and pieces. I do occasionally bump into people at quite large companies with very serious uses of it still. They're actually quite happy with it.

16:02 That's cool. There's some interesting integration cases, right? Like you can do things like the UI framework WPF in Python on top of Iron Python, I think. And stuff like that that you just, you're like, wait, you can put these technologies together?

16:17 Yeah. And it works fairly nicely provided you're happy to live in that 2.7 world with severely restricted library support at this point. There were a number of projects to try and bridge that gap, which have largely fallen away, unfortunately. So the alternate implementations besides CPython are a little bit weaker in terms of extra library support, which is fairly off-putting. The ecosystem is just so critical to Python.

16:44 Yeah, without it, it's just not even hardly the same thing. As a core developer, what are your thoughts on these other runtimes, right? Like PyPy, Jython, Iron Python? Do you look at them and go, ah, the energy would be better placed just like trying to make CPython itself better or they're good experiments? What are your thoughts?

17:06 My thoughts are very big and complicated on that. And I think we'll probably go a little bit deeper in the next topic when we get to it. But certainly there's been a huge amount of value come out of the fact that they exist and that people have worked on them and that there's been collaboration between the reference implementation, CPython, and the other implementations. Everyone has gotten better as a result of that.

17:30 Now, whether the energy is better redirected, honestly, there's not that much energy in the other projects, which is a real shame. But it also means that redirecting it doesn't necessarily add a lot.

17:42 And I'll point out for the dynamic language runtime, Microsoft did two proof of concepts because having Iron Ruby and Iron Python flushed out things that just having one would not. And in the same way, I think having CPython, Jython, PyPy, Iron Python flush out issues in what Python the language is that you would never find out if you only had one implementation.

18:05 Oh, that's interesting. So, you know, if they say run the test suite, you're like, actually, this part's really vague, we don't actually know what this means. That could help make CPython better for sure, right?

18:14 Yeah, there were a lot of language semantics that were clarified, particularly while Iron Python was growing up. PyPy has been coming up a little bit later. It has a lot more support. There's a lot more people working on it than the other projects, but they're trying a lot harder to be compatible with whatever CPython does. And so there isn't as much kind of push and pull on what does the language actually mean here as when the other languages were really going.

18:40 Interesting.

18:41 Interesting.

18:41 This portion of Talk Python to me is brought to you by Linode. Are you looking for hosting that's fast, simple, and incredibly affordable? Well, look past that bookstore and check out Linode at talkpython.fm/Linode. That's L-I-N-O-D-E. Plans start at just $5 a month for a dedicated server with a gig of RAM.

19:02 They have 10 data centers across the globe, so no matter where you are or where your users are, there's a data center for you. Whether you want to run a Python web app, host a private Git server, or just a file server, you'll get native SSDs on all the machines, a newly upgraded 200 gigabit network, 24-7 friendly support even on holidays, and a seven-day money-back guarantee.

19:22 Need a little help with your infrastructure? Need a little help with your infrastructure?

19:24 They even offer professional services to help you with architecture, migrations, and more.

19:29 Do you want a dedicated server for free for the next four months? Just visit talkpython.fm/Linode.

19:35 Let's talk a little bit about your being a core developer and Microsoft employing you, so basically, in some sense, being a sponsor of your open source contributions in that sense.

19:48 Maybe I'll kick it off with a joke that Brian Okken over on Python Bytes showed me.

19:55 So it was, you know, it's around Christmas time.

19:57 So there was this cartoon, and there was like one of these Santas at the mall, right?

20:02 And people, kids go and sit on the lap of the Santa, and the Santa asks, what do you want for Christmas?

20:06 And the Santa says, now, what do you want for Christmas, girl?

20:08 Come on, be realistic.

20:10 He says, I want to know if enough donations that I can just work on my open source project.

20:14 He goes, okay, what color do you want your dragon to be?

20:17 But in a sense, right, like you found this place in the world where, and there's other companies as well, but where Microsoft is basically supporting your work to make CPython better, right?

20:30 Yeah.

20:30 And the story of that coming about is very different from what everyone gets now.

20:36 I feel like now people just, they are going through the interview process to get a job at Microsoft and just kind of tack on, hey, you should give me, you know, one, two days a week to work on this open source project that matters.

20:46 And we just go, okay, sure.

20:47 Back when I did it, I spent weeks negotiating with lawyers.

20:52 I had patent attorneys basically interviewing me to find out what I was going to do.

20:57 Yeah.

20:57 Would you somehow taint all of Microsoft by like interacting with open source code and you've seen some GPL and then all badness breaks loose, right?

21:06 Yep.

21:06 A whole lot of advice on, you know, don't have other code up on the screen while you're working on it.

21:11 And just the, like the risk management that we were doing at the time makes total sense.

21:17 It really does.

21:18 And when you understand that the aim of our legal team is not to help us win the eventual court case, it's to avoid it in the first place, you can see why they're so protective.

21:31 Because basically once any kind of IP litigation goes to court, everyone's already lost, really.

21:40 A couple of lawyers have won, but they're not our lawyers.

21:42 So they're not interested in it going to court.

21:44 They, this is other people get paid for that time.

21:47 So really it's, it's take as much care as possible so that it, whatever comes up just gets thrown out as quickly as possible.

21:54 And so there was a lot of caution there.

21:57 Yeah.

21:57 But again, it's risk management.

21:58 So those of us who were doing that kind of work at that time, and we had at like the same time I was doing that, we had MS Open Tech still, which was the totally separate company.

22:09 That just happened to work on the same campus and some of the same buildings as Microsoft and beyond the same email network and everything.

22:17 Who were legally allowed to do to work on these projects because there, there was some legal boundary where you couldn't sue all of Microsoft if you decided to sue them.

22:26 So I was doing it from within the company.

22:28 Like people like Dino Veland had already done before me and a handful of others.

22:34 Yeah.

22:34 I guess it's worth mentioning.

22:35 There's more than just you as the core developer at Microsoft, right?

22:39 There's Brett Cannon and who else?

22:41 Barry Warsaw is at LinkedIn, but we count him because we like LinkedIn now.

22:44 Eric Snow and Trent Nelson is starting whenever immigration gets their act together, I believe.

22:51 Right on.

22:53 The fun of being an Australian working at Microsoft in the US.

22:55 Yeah.

22:56 A lot of hoops to jump through, right?

22:57 But yeah.

22:58 Okay.

22:59 So it's, it's not just you who have found this place.

23:01 There's this group of folks inside the company.

23:04 And again, the transition within Microsoft over the last six years has gone to the point where we aren't forcing open source contributors to sit down with lawyers to describe what they're going to do ahead of time.

23:16 That's really nice.

23:17 It means we have a lot more contributors to projects.

23:19 We're able to support and help develop projects in, in ways that benefit.

23:25 Obviously they benefit us.

23:26 Otherwise, you know, we wouldn't get work time to do it, but they, they benefit the community as well.

23:32 So it's, we're trying to find that balance between kind of straight up altruistic development on open source projects and being able to resolve the issues that, that people are having that are kind of specific to us.

23:44 Yeah.

23:44 Well, like some of this, you know, it's kind of like basic scientific research.

23:48 Like some of this comes back to help the world in nice ways.

23:51 Like for example, the Git stuff that you all contributed back to Git so that you can move windows to Git because kind of like command line argument thing.

24:00 Like the number of files, like the virtual, basically virtualizing the Git file system had to be done for windows to be part of Git, right?

24:08 Or to be on Git.

24:09 Yeah.

24:10 We had a number of repositories internally that were just the working directory was being counted in gigabytes of code.

24:17 And we had one.

24:18 So one of the repositories I worked on through the transition from TFS to get the, like one checkout.

24:25 When I first checked out that repository, it was something like 180 gigabytes, just the, the working directory.

24:32 Just the main branch.

24:33 Just the main branch.

24:34 And not even the history, like just the latest up to date.

24:37 We had a whole lot of stuff checked in that didn't need to be.

24:39 Yeah, yeah, sure.

24:40 Some massive work went on and I think we got it down to six gig and said, that is the best we can do.

24:46 And part of that six gig is a script that downloads the other hundred gig of tools into a different directory.

24:53 So.

24:53 Wow.

24:53 That's incredible.

24:54 Yeah.

24:54 There's some really, really huge code bases that a lot of the big corporations have inside that simply never get heard of outside.

25:02 Like open source community run projects are never going to consider those cases because they sound utterly ridiculous.

25:09 And yet all of these really big companies have nothing but utterly ridiculous cases.

25:16 Yeah.

25:17 And they've been around for so long and so many people have worked on it.

25:20 Yeah.

25:20 It's, it's pretty insane.

25:22 Just the number of developers continuously contributing to that stuff.

25:26 So I feel like one of the big turning points in Python's journey probably hinges around Azure.

25:32 What do you think?

25:33 Certainly Azure has created a place for Microsoft to really care about developers in, in a way that is kind of different from when it was, when Windows was the main thing.

25:45 Right.

25:45 Because it used to be, well, long as it works well on Windows and we get that app on Windows and then, you know, like that was the key.

25:53 But it's hard to say, well, all the Python developers are going to come over and run on Windows potentially because maybe they're hosting on Linux or whatever.

26:02 But, but now if they're all going to run on the internet, right?

26:07 Exactly.

26:08 If they're running in our data centers, then we're happy to have them.

26:12 And, you know, we want to do whatever we can to roll out the red carpet for whichever developers want to come in and work with us.

26:18 And it's so much less about the end user platform because that's essentially the browser.

26:24 Now it's like everyone's hiding behind the browser.

26:27 The browsers work everywhere.

26:28 The browser and the phone.

26:29 Yeah.

26:30 Yeah.

26:30 So, yeah, it really has created that opportunity to do production apps in Python, which, which I mean, Python was never really designed for rich user facing applications.

26:41 That's never been one of its use cases.

26:43 It can certainly do it.

26:44 There's other languages that have been designed for that kind of thing.

26:47 And so Python's place has always really been more developers and command line, happy people and adding the web to that and being able to run your code on Azure enables Python developers in a way that want to encourage.

27:02 And I don't want to say it's forced us because most of us are happy about it.

27:07 It's broken down the barriers or the walls to pursuing the stuff you'd like to do anyway, maybe.

27:12 How's that?

27:12 That's basically it.

27:13 We have somewhere to put our Python developers that we want to be working with now.

27:17 Which is come and run your stuff on Azure or, you know, multi-cloud, hybrid cloud, whatever it is.

27:22 But we have somewhere for Python developers to run production code, which Windows never really had a place for before.

27:28 Yeah, for sure.

27:29 So when I go to Azure these days, I feel like it's like opening up an encyclopedia.

27:33 It's just like there's so much stuff in here.

27:36 What do I even do?

27:37 How many services are there?

27:39 This actually came up while I was writing the story because I have a mention in there of Laurent as well, who manages the Azure management SDK in Python, which basically has modules for every single Azure service there is.

27:52 And when I drafted it, I'm like, it's probably around 50.

27:56 I'll write around 50.

27:57 And then at one point I went and checked and it's over 100.

28:00 Wow.

28:00 And so I checked again.

28:01 I'm like, no, it's still over 100.

28:02 So I asked him and he's like, yeah, it's definitely over 100.

28:05 I don't even know what they all are.

28:07 Yeah.

28:07 I feel like there needs to be a song about it.

28:10 Like there's that song that all the US kids learn in school for all the states.

28:13 Yeah.

28:14 Like we need that song for all the Azure services just so we can remember what's going on.

28:20 So bizarre how music does that.

28:22 Yeah.

28:22 Yeah.

28:22 No, in terms of getting started, what I think was a really good move, and I'm not sure who made it, but I'll go give them a high five at some point.

28:28 There's basically a set of free services on Azure.

28:32 There's like a free sign up, credit card verification or like .edu email address verification.

28:37 And you get like a limited set.

28:40 And the thing about it being a limited set is there's one obvious choice for whatever you want to do within that set.

28:46 And so it's like, I want to push up a website.

28:47 It's like, well, app services here.

28:49 That's the obvious choice.

28:50 You don't have six things that you could make work that are all subtly different and serve different use cases better, which, you know, you can grow up into.

28:58 Once you're at the point where it's like, this is a really serious workload that my company or my users are depending on.

29:05 Let me optimize it for certain use cases.

29:09 It really just is.

29:10 You get one choice.

29:11 You know, you want a virtual machine?

29:12 Have a virtual machine.

29:14 Oh, you want to choose from the 50 different sized virtual machines and different drive options and different GPU options and different memory options.

29:23 Step up to the paid thing for that.

29:26 But having that kind of filter on this is what you need to just do something.

29:30 Taking away that choice is one of those things that I think is actually a really good user experience at time.

29:36 Yeah, sometimes less is more in that sense.

29:38 That's pretty cool that the free tier starts that way.

29:40 So let's spend just a little bit more time on this journey, maybe focus on the areas that you've worked on.

29:46 I don't know how many listeners out there know, but you actually, you know, when you interact with Python on Windows, they are interacting with a lot of your work, right?

29:55 Yeah, so I actually, the way I got into core development was I was at PyCon.

30:01 I think I was at my second PyCon ever, PyCon US, that is, and had some annoyances with the installer for Windows.

30:09 And so I basically found some of the core developers and said...

30:12 It was cool that it was there, but it definitely was a little bit not...

30:16 Tell us some of the challenges that you ran into because of it.

30:18 It was a simple installer for a simpler time.

30:20 Because there was a time on Windows where you just drop files anywhere and they run, and anyone who has physical access to the machine can do whatever they like.

30:30 Then the internet happened, and it turns out that even people who don't have physical access to the machine have access to the machine.

30:35 And we really ought to be locking things down a little bit better.

30:38 And that sent off the whole security kick that eventually brought us to where we are today, where we get all the way to store packaged apps that are managed in a really, really, really secure and isolated way that didn't exist at the time when the previous Python installer was written.

30:57 Which I think came straight out of the DOS days.

31:00 So it absolutely served a very important purpose for making Python available on Windows.

31:05 But it just hadn't grown up at the same rate as Windows was growing up.

31:11 So I basically found some core developers and said, hey, I would like to contribute some work to improve this.

31:17 Who do I talk to?

31:18 How do I go about suggesting ideas, proposing the work, contributing the work?

31:22 And they basically said, you know, give us your contact, we'll chat and we'll get back to you.

31:27 And they got back to me the next day because it happened that the person who was currently maintaining Windows, Martin Von Lois, had decided to step away and they were looking for a replacement.

31:36 Good timing.

31:37 Very good timing.

31:39 And so I was basically co-op.

31:42 I'm one of the rare people who basically gets dropped into core development without having contributed.

31:48 That's not the usual way people become core developers.

31:52 But I feel like it turns out when you're very much in a minority like that.

31:57 I'm such an affirmative action hire for the CPython core development team.

32:00 It's not even funny because there's very few Windows developers there.

32:05 And so if someone comes along and says, I would like to do Windows and I'll let you all worry about Linux.

32:11 I don't care about it.

32:12 I just want to do Windows.

32:13 They're like, are you even real?

32:15 Yeah.

32:15 Who's this person?

32:16 They're like, yes, please come and help.

32:19 And so I basically just got to rewrite it as I wanted to do things like moving the default install so that it's not writable by anyone on the machine.

32:27 Because it was installing and Python 2.7 still installs to this default directory where any user on the machine can add files that the administrator will then automatically run.

32:38 And it's something out of place like c colon backslash Python, right?

32:41 Yeah.

32:42 It's unconventional now.

32:43 At the time, it certainly wasn't.

32:45 It made a total sense originally.

32:47 But these days, yeah, programs go into program files or they go under the user app directory if it's restricted to the user.

32:55 And that helps keep things secured.

32:57 It sets the access control right so that people aren't modifying files that other people are going to use.

33:02 And being able to put it in a per user location means that we don't have to require users to be administrator to install anymore.

33:10 And I feel like that's been the most exciting feature about the new installer for people is enabling that don't make me elevate.

33:18 For a lot of people, it's don't make me hit yes on that box.

33:21 But for others, it's don't make me go get the administrator to type in their password.

33:25 Yeah, exactly.

33:26 When you're at home and it means the UAC kicks in and you click OK, like that's OK.

33:30 That's I mean, it's not super, but it's not the end of the world.

33:32 But if you're at a company, like some of the places where it would make a lot of sense at like investment places and whatnot, like they don't even get to ask.

33:41 It's just the answer is no.

33:43 All right.

33:43 No, you don't get to install stuff.

33:44 No, you don't get to raise permissions.

33:46 No.

33:46 Yeah.

33:46 And then pip install something becomes challenging as well if it's trying to install into like restricted directories.

33:52 And so you basically made it so you can all that stuff potentially happens into your user profile where you can control almost all the time.

34:00 Yeah. And at this point with 3.7, I made the final change, which was to do with the C runtime that would still try and install as an admin if you needed it.

34:09 We're down to very few people who needed that already.

34:12 So 3.7, I just kind of cut that cord completely and said, if you're still running on a Windows 7 machine that isn't receiving updates for whatever reason, then we won't try and install this properly.

34:24 We'll just give it to you so it works and assume that you're not ever going to run into the trouble that we were concerned about previously.

34:30 But I think my favorite story about people enjoying this feature, I had a teacher who spends a lot of time teaching Python throughout Africa come up to me.

34:40 It was actually at Europython this year and absolutely gushing with praise and thanks for making it possible for his students who are coming into like evening classes.

34:50 They've borrowed their mom's work laptop.

34:52 It's a 10-year-old laptop.

34:54 The keyboard's broken.

34:55 So they've got a USB keyboard plugged into this laptop because they don't get to buy the latest MacBook Pro every year, right?

35:03 Yeah, of course.

35:04 And it's running Windows 7 and it's totally locked down and they can still install Python on it.

35:09 That's great.

35:09 Because of that change.

35:10 It's a little mind-blowing that that change, which feels like it's technically a good thing to do, but it also enables kids in Africa to learn to program.

35:20 And that's just mind-blowing.

35:22 Yeah, that is so awesome.

35:23 That's great.

35:24 And I think actually that sort of speaks a little bit to the reach of talking about Python on Windows and this journey.

35:31 I mean, we talked about, like I talked about, say, oh, Azure is what's like unlocking a lot of the sort of commercial side of the things around Python for you.

35:38 But if you look at who is using Python, like what operating system people are using Python on, you go to the conference, maybe it doesn't feel this way.

35:47 But general studies and surveys say the majority of them is like massively on Windows, right?

35:53 I've actually been going around speaking to this at conferences recently, which I think we'll have a link to at least one of the recordings of the talks in the notes for this.

36:02 But my recent presentation I've been giving at conferences is called Python on Windows is OK, actually, which is a bit of a bait and switch title, to be honest, because most of the talk is not actually about Python on Windows because Python on Windows is OK.

36:17 And one of the best things when I joined the team was it was already OK.

36:20 I think this installer actually makes a big difference.

36:23 I think getting it installed properly is a big deal.

36:27 It's the first experience, right?

36:28 And the first experience matters because if you have a bad first experience, then you may never look twice.

36:33 The big kind of controversial thing in that talk and a lot of the reason that I wanted to give it was I started looking at the stats and, you know, I looked around the conferences and was like, yeah, the entire Python world is using Macs mostly.

36:47 And occasionally people that have taken a Windows laptop and installed Linux on it instead.

36:51 And then I'd get up to give a talk and I've got a Windows machine and I can look out and just see all these glowing fruits staring back at me.

36:58 Taunting you.

37:00 Yeah. And but then we start looking at actual numbers and it's so skewed in the other direction, looking at things through downloads from PyPI, downloads from Conda, usage in the PSF survey, usage in IDEs like VS Code and PyCharm.

37:15 So PyPI accepted.

37:17 Windows is a huge, if not the biggest chunk or just a true majority of Python usage.

37:24 Yeah. So let me just just to put some numbers behind this.

37:26 Let me just read the operating system usage numbers from the JetBrains and PSF survey, the one that you just mentioned, which was not promoted by JetBrains.

37:36 They just happened to do some of the analysis.

37:38 So promoted by PSF.

37:40 We've got other operating systems, whatever that means, at 17 percent.

37:44 We have macOS at 15 percent and Linux at 19 percent.

37:47 Right.

37:48 I feel like everyone has a Mac at the conferences, but 49 percent people said Windows.

37:53 Right. That's like not just a little bit more than others.

37:56 That's more than Mac and Linux combined.

38:00 Yeah. And as you say, it was not promoted by JetBrains so much.

38:04 It was like this was promoted in regular Python channels.

38:07 And so regular Python users are seeing and responding to this and doing it from a Windows machine, which we just don't have that kind of perception in the community that that's the case.

38:19 And so it was really exciting.

38:21 Like my favorite part about giving talks in general is looking at the tweets afterwards and the things that people have been most shocked by or most excited by.

38:28 And the amount of photos of me standing in front of the slide with the 14 million downloads per month of Python for Windows is one of my favorite photos now.

38:39 That's a ridiculous number of downloads.

38:41 That's awesome.

38:41 It's huge.

38:42 And a lot of that is going to be CI systems, to be fair.

38:46 But most of the downloads from Python.org are CI systems.

38:49 We don't actually think there are 14 million Python developers anywhere in the world right now.

38:55 I think the estimates are closer to six to seven million.

38:57 So there's automatic downloads.

38:59 At the same time, if people are automatically downloading Python for Windows, that means they're running Python code on Windows.

39:04 Yeah.

39:04 So it doesn't actually spoil the point at all for it to be, you know, oh, it was a script.

39:10 It's like, well, every single download of the source for Linux was a script.

39:13 Every single download of the Mac installer was also a script.

39:17 That's just the reality is no matter how you slice the numbers, I describe it as at least half of the Python community is actually running on Windows.

39:26 And we just don't realize that.

39:30 And a lot of time we don't act like it.

39:32 So the rest of the talk is basically around things that Python projects or Python packages do or have done that make them not work well on Windows.

39:44 Likewise.

39:45 Things like using string dot split on forward slash to separate paths.

39:50 So paths on Windows use backslashes.

39:52 And we have a couple of great modules for this.

39:55 So I strongly recommend pathlib, which is in the standard library as of 3.4.

39:59 But we've had OS dot path module for dealing with paths.

40:03 And yet people will still just use the normal string functions to split on a forward slash.

40:07 And if you do that to a path on Windows, you get one element with the entire path still in it because there's no forward slashes anywhere.

40:14 It's a weird root directory.

40:15 And it's things like that.

40:16 And my big concern with that is you get, you know, these 50% of Python developers will install your package, try and use it, and it won't work.

40:25 And it's just a very unwelcoming experience.

40:28 Yeah, it can be.

40:29 And, you know, it can be indirect as well.

40:31 Like, for example, Sanic is a cool async web framework, right?

40:35 Python.

40:35 But I believe right now, at least last I heard, it didn't work on Windows because UVlib doesn't work on Windows.

40:43 It doesn't install properly on Windows.

40:45 And there's just these little glitches just like, ah.

40:47 Yeah.

40:47 And there's always a very long and reasonable chain of things that would have to work perfectly for the final result to work, which I understand.

40:55 Yeah.

40:55 Which I understand.

40:56 This portion of Talk Python to Me is brought to you by us.

41:01 Have you heard that Python is not good for concurrent programming problems?

41:06 Whoever told you that is living in the past because it's prime time for Python's asynchronous features.

41:11 With the widespread adoption of async methods and the async and await keywords, Python's ecosystem has a ton of new and exciting frameworks based on async and await.

41:21 That's why we created a course for anyone who wants to learn all of Python's async capabilities.

41:26 Async techniques and examples in Python.

41:28 Just visit talkpython.fm/async and watch the intro video to see if this course is for you.

41:34 It's only $49 and you own it forever.

41:37 No subscriptions.

41:38 And there are discounts for teams as well.

41:42 Part of the problem is that of those 50% of Python users on Windows, I'm pretty well convinced that 49% of them work at places where they're not actually allowed to release their libraries.

41:53 Yeah.

41:54 I know for a fact that they have fast networking libraries, that they've re-implemented things for their platforms that make Python run great, that would be amazing to have out there in the community.

42:05 That'd be a real enabler for a lot of the rest of us that are not kind of restricted to within these companies, but they just can't do it.

42:13 Yeah, it's probably built to drop a half a millisecond on a trading engine for algorithmic trading or something like this where it's like, well, this is our advantage.

42:22 We're not putting this out.

42:23 Yeah, absolutely.

42:23 And that advantage is worth a few millions of dollars per second as well for a lot of these places.

42:29 Yeah, it's not unreasonable for them from a self-interested perspective.

42:33 But yeah, I hear you.

42:34 I would actually say it in reverse, maybe, like how many of the core developers or maybe not even the core developers, let's just say people working on popular packages like UVlib, not calling them out in particular, really, but just those types of things where the developer doesn't even own a Windows machine.

42:52 So it just doesn't bother.

42:54 And ultimately, I think that's the biggest point is when you don't have the machine to run and test on, then you stop thinking about these things.

43:03 And works on my machine is, for volunteers, I think totally reasonable by asking people to invest in new hardware, invest in a new software installation, especially the matrix of things that you have to worry about.

43:17 Even for Linux, I would suspect that most Python packages are not tested on more than one variation of Linux.

43:24 Yeah, for sure.

43:25 For exactly the same reason.

43:26 Like no one, very few people are going to be running on FreeBSD and Ubuntu, for example, to make sure that it works on both.

43:34 So that is a really valid reason for not doing it.

43:38 But at the same time, it doesn't take a lot of work these days to set up a CI system that is going to run on Windows for you.

43:47 And it doesn't actually take any money at all.

43:48 Azure Pipelines is a fairly recent one that a lot of people are really excited about, myself included, because the open source thing.

43:58 So if you mark your project as public so that anyone can see the results, you automatically get unlimited build time, up to 10 concurrent builds, and Windows, Mac, Linux, a choice.

44:10 So you can choose, I think there's two Ubuntu images and two Windows images right now.

44:15 It has Docker support if you want to be running builds inside containers of whatever container you like.

44:21 That's pretty cool.

44:22 Can I plug it into like my GitHub repo?

44:24 Yeah, absolutely.

44:25 You could do that before we bought GitHub.

44:27 In fact, that's always been one of the primary uses for it.

44:30 Yeah, that's pretty awesome.

44:31 We have CPython builds running on that.

44:34 Those are not the binding required checks yet.

44:37 Pandas has switched over most recently.

44:39 NumPy is there.

44:41 Condor Forge is investigating moving all of their builds over to it.

44:45 Oh, that would be huge.

44:46 I mean, I know some people know what Condor Forge is, but maybe not everyone.

44:50 And I think they're really interesting in enabling Windows to not have these problems of like building these weird tools because they deliver binaries on the right OS.

45:01 Maybe tell people real quick what Condor Forge is so they'll appreciate that if they don't know.

45:05 So Anaconda is a distribution and normally how people think about getting to the Condor tool.

45:10 So you'll get this big fat installer from Anaconda and install that and get a whole lot of packages.

45:15 And you have this Condor tool for creating environments, installing packages.

45:18 But Condor is really just a pip equivalent.

45:20 And you can, rather than getting the packages from the Anaconda channel, you could get them from other people's channels as well.

45:28 Condor Forge is a community organization that basically fills in the gaps.

45:34 Like they have something like 5,000 GitHub repositories that are basically the build steps for 5,000 packages.

45:42 And they have a set of packages that are built for, I think they're doing five different platforms now.

45:49 Just automatically building all of these packages.

45:53 So if you've ever run into like trying to pip install something and there's no wheel, Condor Forge has probably done some of the work or someone's contributed the steps to build the pre-built Condor package so that you can Condor install a whole lot of stuff from there that would otherwise not be available.

46:11 And they've been doing that through this incredible system of basically abusing as many free CI providers as possible for the various Mac and Windows and Linux builds.

46:24 We're doing it for the good of the community.

46:26 Yeah.

46:28 And I think we timed things right when I mentioned this to one of the Azure Pipelines team members.

46:34 And I don't actually know exactly what our financial motivation would be, but I don't really care.

46:39 If they're excited to move as many thousands of builds onto one system that does everything that they want and the Condor Forge team is excited to do that and everyone's just excited to do it.

46:51 I don't actually know what anyone's getting out of it, but Condor Forge is getting builds, users are getting packages.

46:57 And so I'm happy.

46:58 It can at least make the story for installing some of these packages on Windows better by ensuring they have a good infrastructure and it's not too much work, right?

47:06 So there's maybe something there, yeah?

47:08 And it makes installing the packages on Mac and Linux better as well.

47:11 Anaconda's popular on Windows, but not like 90% of their usage.

47:17 Yeah, yeah, it's true.

47:18 They still have really significant use on Linux and Mac because they'll give you far more reproducible installs across various platforms than using Building From Source or even Wheels on pip are going to give you.

47:30 Sure, I've used it on my macOS because something wasn't installing right.

47:34 I'm just like, I'm just going to get the pre-built version.

47:36 Let me ask you a question about something that's bugged me for a while.

47:40 And as somebody who creates courses and tutorials, I often have to have like a couple of steps.

47:46 Like, okay, here's the command that you type when you register this package so you can develop it in development mode on Mac or Linux.

47:55 And, oh, here's the separate command that you run on Windows so that you can do that.

47:59 So, like, Python 3 set up PY develop, right, for example.

48:03 But, well, Python 3 was not a command on Windows.

48:06 And that can be super annoying.

48:08 You're like, well, on Windows, what you do is you say Python, but you've got to make sure which Python it is in the right path so that Python 3 comes first.

48:14 And you and I, we've had these conversations before, right?

48:16 Yeah.

48:17 So this, I think, is one of those things.

48:19 But you actually have like a super early announcement of something that's amazing around this.

48:23 Is that right?

48:24 It's not exactly a secret project, but it's, you know, if you're not watching the Python bug tracker, then you won't have noticed.

48:30 But hopefully when we start promoting this a little bit more, you'll see it.

48:33 With Windows 10, we actually added, so Microsoft added a store, an app store, basically.

48:40 Oh, and in fact, it came in Windows 8.

48:41 But in Windows 10, some of the recent updates, it's been growing the ability to install regular old apps.

48:47 So a lot of people think of app stores like, you know, you get your Apple approved iPhone app that only runs on the phone and can't actually do interesting stuff beyond what the app does.

48:56 Like you can't install a full Python interpreter that can access everything on your system through any of the app stores.

49:02 What's being added to the Microsoft app store, the Windows app store, is the ability to install those.

49:08 So paint.net is actually one of the earlier ones to do this.

49:13 You can install paint.net through the store and it installs really quickly.

49:17 It's properly isolated between users.

49:20 And what we're doing, or at least experimenting with at this point, is adding Python to that store so that when it comes out,

49:29 and the aim is to do this with Python 3.7.2 initially and then keep updating it from there,

49:34 is be able to, rather than going to python.org and downloading the installer and running through all of those steps,

49:41 you'll be able to just go to the Microsoft store and say, get it free.

49:45 Obviously, it's going to be a free app and install it.

49:48 And that will give you Python on your machine, managed by the store, automatic updates.

49:53 It'll do automatic updates.

49:54 Beautiful, yeah.

49:55 It'll do automatic updates.

49:56 I don't know how much trouble that's going to cause yet.

49:58 That's why this is still a little bit experimental.

50:00 I had to make some significant improvements to VN to make it be able to handle that, but I believe it can handle that now.

50:08 Oh, that's great.

50:09 But the really cool thing that comes with store apps that isn't available to regular apps is being able to manage path properly.

50:17 So one of the problems with the path environment variable on Windows, or just one of the differences from POSIX-like systems,

50:25 which I actually talk about in my Python Windows talk, is that on POSIX-like systems, you kind of have,

50:31 here's a directory for my user commands, and any commands I put in there are available to the current user.

50:36 Here's a directory for my machine commands, and any commands in there are available for the whole machine.

50:41 And Windows hasn't had that.

50:42 Windows has, let me add a directory to the path, and then go look in that when you're searching for anything.

50:49 And so you get apps fighting over who's first on the path.

50:53 There's no good way to automatically manage it and say, well, 3.7 should come before 3.6.

50:59 But you can't automatically do that.

51:01 And so that's always been messy, to the point where that's one of the controversial things I did with the new installer,

51:06 was disable setting that up by default, because it just leads to breakage and confusion that we can't manage.

51:11 What the store apps let us do is specify aliases for these commands.

51:18 And those get managed by the operating system.

51:20 There's like a nice GUI for switching which app gets to have that alias.

51:24 And they go in a proper directory, that is, here are my user commands that are available to my user,

51:30 and it's on path automatically.

51:32 That's awesome.

51:32 So this store package, when it comes out, is going to have python.exe on path,

51:37 managed properly so that you'll get that one, you'll get the right one.

51:41 And then when you update the store package, you'll get the latest version because it goes in the same place,

51:47 and the newer version wins over the older version.

51:50 But because it's so cheap and easy to do this, I thought, hey, why not put python3.exe in there?

51:55 Well, since we're here, why not put 3.7.exe in there as well?

52:00 So now you have all of those on path with this package.

52:03 So that python3 command is going to work.

52:05 And then I thought, well, this is still pretty easy.

52:08 Why not keep going?

52:09 Let's put pip in there.

52:10 Let's put pip3 in there.

52:11 Let's put pip3.7 in there.

52:13 Let's put idle in there.

52:15 And so it's very easily got to a point where when you install this package,

52:20 you can go to any command prompt and type idle, and it will start running idle in that directory.

52:25 You can type, when you end up with multiple ones, idle 3.7 will give you the 3.7,

52:30 and eventually idle 3.8 will give you the later version of it.

52:34 It's just there.

52:35 That is such good news.

52:36 Thank you.

52:37 Thank you for doing that.

52:38 That is really, really cool.

52:39 So excited to hear it.

52:41 Today, as we're recording, the first preview one is up.

52:43 And so there's some more kind of private testing going on.

52:46 But my hope is that with the 3.7.2 RC, when that is out, which is probably a week or two ago,

52:53 by the time you're listening to this, it'll be up there publicly.

52:56 So you can just go to the store and type python, and we should be showing up somewhere near the top of the list.

53:01 And you can use that to install Python on your Windows box.

53:06 And the cool thing is that doesn't require admin, but it installs in a way that only installs it once for the whole machine,

53:11 which is really nice.

53:12 So if you have 100 users on a machine, it installs it once and then isolates just the differences per user.

53:20 So it doesn't take up 100 times the disk space or anything crazy like that.

53:24 That is so cool because you see so many tutorials and stuff where people just put the POSIX variant up there, right?

53:32 The type pip3 here or, you know, you do this and it's like, well, that's not exactly what you need.

53:38 So I think this, so you're thinking that this Windows 10 store app of Python 3

53:44 is going to be the way to get Python going forward.

53:47 I hope so.

53:48 There's certainly going to be some cases where it won't be the best option.

53:51 And so the old installers aren't going to go away anytime soon.

53:54 Sure.

53:55 But we'll just have to see how people use it and what people need.

53:58 There's already cases where the existing installers aren't ideal.

54:00 And this is why we have other distros out there.

54:04 This is why WinPython exists, Anaconda exists, because they offer things that, you know, Python.org can't offer.

54:10 So this is another thing to be offered.

54:12 And we can track usage.

54:15 One of the really nice things is we can get all the crash reports.

54:17 So when Python is breaking badly, there's a dashboard for store apps that basically says,

54:23 here's how reliable your app is over time.

54:26 And so we'll start getting those reports for Python, which will be really interesting.

54:29 Yeah, for sure.

54:30 And you also were able to recompile Python with the later tools, like move seven years or five years into the future on the compilers, right?

54:39 As you, as part of your project of taking over all this.

54:41 Yeah, that was, that was actually one of the first things I did was let's move off Visual Studio 2010

54:47 since it's now 2015 or whenever I started doing that.

54:50 Exactly.

54:51 I've had a number of people thank me profusely for that.

54:54 Mostly other people who are trying to build Python itself.

54:57 Yeah, because if you're on Windows and you're a developer, you probably have the latest version of Visual Studio or the Windows Platform SDK or whatever,

55:04 but none of those are likely like a seven or eight year old version of the compiler.

55:09 It gets harder and harder over time to get the old compilers.

55:12 Microsoft supports the latest and try to build in the compatibility that makes it

55:18 viable for people to move forward as soon as possible.

55:21 Around the Visual Studio 2015 point is where we actually made some big dramatic changes to make that easier.

55:27 And so we are at a point now where VS 2015 or 2017 or 2019, which is recently first preview was released,

55:36 will all work.

55:38 and they can all kind of work across each other so that even Python 3.5, which was compiled with the old one,

55:45 can use extensions built with the newest compiler because we've got that.

55:48 I think we finally got the compatibility right.

55:50 We went through about six or seven different ideas on how to make it right.

55:54 And this time I think we've got it in a way that really just helps developers not have to pull all sorts of crazy tricks with their setup to make things work.

56:05 That's also really good news.

56:06 So I think there's so many positive things around here.

56:09 This Python on Windows is okay, actually.

56:11 I mean, it did highlight some of the issues, but also some of the great things.

56:16 So maybe let's close out this conversation around the journey of Python at Microsoft by talking about this Python Summit at Microsoft.

56:24 What's the story on this thing?

56:26 I've only just barely heard of it.

56:27 There's rumors floating around, and we've put up a blog post with a bit of information.

56:32 Being at Microsoft gives me an opportunity to speak to and hear from people at companies that don't normally interact with the community.

56:40 And so through that, we get to hear about problems and solutions that various people have tried for fixing things in Python or just in their Python projects.

56:52 And we start hearing some commonalities across a lot of different places.

56:55 And I'm talking like the Facebooks, Instagrams, through to various big banks who are all heavily invested in Python.

57:04 We start hearing so much in common, but also not actually being aware of everyone else.

57:10 And we started feeling like, hey, maybe we're in the privileged place where we can see all the overlap, and maybe it's on us to do something about it.

57:18 So we basically put out the call to the people that we knew in these places.

57:25 And so this wasn't like a public open call for content.

57:29 This was specific invites to companies we know heavily invested in Python and say, hey, come, let's get in a room together and basically talk through what our struggles have been, what our efforts look like.

57:42 You know, it's like half conference, half counseling session almost.

57:46 Yeah.

57:46 And, you know, do it in a place where we can have kind of an informal NDA going on about, you know, this is not, we're not broadcasting this on the internet.

57:55 So if you want to actually talk about problems that you're having, you don't have to worry that it's going to be a news headline tomorrow.

58:01 And people came and we had a room filled with trillions of dollars worth of Python development, which is something that probably makes absolutely no sense to most of the community because we look around and see the same kind of Python developer all around us at most of the conferences.

58:16 And then in this room, we had the Dropboxes, the Amazons, Facebook, Google show up along with the big banks that are like actually making huge amounts of money using Python to the point where if Python disappeared or if Python fell away, they would be losing like billions of dollars of income or being able to move money around like and investments and stuff.

58:42 So it was just a great opportunity to hear directly from them, hear that and basically find out that, yeah, all of our problems are the same.

58:49 And a lot of them are shared with the community.

58:52 A lot of packaging concerns affect everyone.

58:56 A lot of performance concerns and like the C API for CPython is a hot topic right now, both in the community and in the corporate world.

59:06 And everyone kind of has different ideas about what to do and how to fix it.

59:10 And so it was a lot of people said it was a great time.

59:13 They were very excited that there was a chance to share some of this.

59:18 As I said, it's a little bit private kind of thing.

59:23 So we're not putting out reports on specific things from specific companies.

59:27 But certainly there are those of us like Microsoft who are kind of that channel, right?

59:33 So, you know, we're a little bit of gatekeepers, I guess, which is not ideal.

59:39 But at the same time, if it's the only way for like if the only way for the community to hear what the corporations are worried about is for us to kind of aggregate it and say, we spoke to corporations that won't publicly talk about this stuff.

59:52 And, you know, you'll just have to trust that they said it's a problem to us.

59:56 And we're kind of repeating anonymously, which the interactions between the corporate world and the community of open source are actually really, really complicated.

01:00:05 Yeah, I'm sure they are.

01:00:06 Most of what we've talked about today has been how complicated it can be.

01:00:11 It's still so complicated, still so much uncertainty about how to interact.

01:00:16 There's a lot of investment that these corporations want to make into community projects like Python, like Pip, like NumPy.

01:00:25 And really, there's a huge amount of uncertainty about how to actually do it.

01:00:30 And there's a lot of reasons for that.

01:00:32 And probably most of them are solvable.

01:00:34 But the only way to solve them is by having some amount of open dialogue.

01:00:39 And we're trying to open that up and make it possible for companies to say, hey, this is what we're worried about.

01:00:46 This is our concerns.

01:00:47 This is what we can offer.

01:00:49 And try and link up with the community to make some of that actually happen.

01:00:55 Yeah.

01:00:56 So apart from the corporations, we did have some community people in the room as well.

01:00:59 I think we had seven Python core developers in the room, mostly employees of these companies, but they were there.

01:01:07 You know, we were looking around going, did that guy get up and give an entire talk about using Python and not say any dollar amount under a million dollars?

01:01:16 Like it's like every single, not the only number less than a million was the version number of Python he was talking about.

01:01:21 Yeah.

01:01:22 Yeah.

01:01:22 Did that really happen?

01:01:23 Does Python get used for that?

01:01:25 Yeah, it does.

01:01:25 But how do you turn that like millions and billions of dollars into community contributions is a really difficult problem.

01:01:33 Yeah, it sounds like it is.

01:01:34 And I think if you guys could unlock that somewhat, that would be so amazing because there's so many projects that are underfunded or run by one person in their spare time that are probably critical to the work that these other folks are doing.

01:01:48 If you could somehow get that support to flow the other way or get at least their concerns to help make CPython better, that'd be great.

01:01:55 There's been a few attempts to do this.

01:01:57 There's a number of companies out there that are set up for kind of the purpose of redirecting corporate funding into open source projects.

01:02:06 Anaconda, originally Continuum, is one of those.

01:02:09 Travis Elephant's new venture, QuantSight, is trying to do the same thing.

01:02:14 Tidelift is also following a similar thing.

01:02:16 Yeah, Tidelift is really cool.

01:02:18 It largely comes down to corporations will give out money for exactly one reason, and it's to get a return, which means you either have to convince them that altruism is going to get them some good PR or actually sell them a product.

01:02:31 And Tidelift and Anaconda and QuantSight are all selling the corporations a product and turning that into open source funding, which is something that individual projects have really struggled to figure out.

01:02:43 Yeah, absolutely.

01:02:44 Like the Patreon model or the PayPal donate button, you know.

01:02:48 Oh, yeah.

01:02:49 Good luck.

01:02:50 Good luck.

01:02:50 Yes.

01:02:50 You can't convince, you know, a vice president to, like, to give money to Patreon.

01:02:56 It just, it's not how any of these big companies work.

01:03:00 Yeah, they're just like, well, it doesn't make even any sense to us how this works.

01:03:04 Yeah.

01:03:05 Yeah.

01:03:05 And you have a scale problem as well.

01:03:07 We actually face this with trying to sponsor meetups is finding the person that can approve, you know, a $1,000 sponsorship is actually kind of hard.

01:03:15 Like, because no individual engineer can approve that expense because that's too big.

01:03:19 But what if the only people you can find look at that and say, why are you bothering me with something so small?

01:03:24 Yeah.

01:03:25 Come back when it's six figures or more.

01:03:27 And it's actually this really hard spot to get anything done in a large company when it's too big to go unnoticed but too small to be, like, to deserve a decision maker.

01:03:39 Yeah, it's a weird no man's land in the middle there.

01:03:41 Indeed.

01:03:42 Well, it sounds to me like this summit you guys had is a real positive thing.

01:03:46 So, yeah, keep that up.

01:03:47 And thanks for the report.

01:03:48 Yeah.

01:03:48 Yeah, indeed.

01:03:49 All right, well, I think as much as I have a million more questions for you, I think we have to call it an episode on that.

01:03:55 So, let me ask you the two final questions before you go.

01:03:58 And maybe we can touch on something else as well.

01:04:00 So, if you're going to write some Python code, what editor do you use these days?

01:04:03 Probably not Visual Studio Tools for Python these days, right?

01:04:06 That was my project for a long time.

01:04:07 So, I have a lot of sympathy.

01:04:09 I do still use it.

01:04:10 I wouldn't say it's my primary editor.

01:04:12 Like, Visual Studio Code right now is probably my primary editor.

01:04:15 That was going to be my guess, yeah.

01:04:17 Yeah, well, but the main reason for that is I got up on stage at a conference recently to do demos of it and just failed miserably at a number of things because I had not spent enough time using it.

01:04:27 It's a really cool editor, but it's one of those you kind of have to live in because there's a lot of non – there's not lots of buttons and stuff.

01:04:34 You've got to kind of know how to interact with it, right?

01:04:37 Yeah, and I know Visual Studio inside out, so I can use that comfortably, but I don't feel like I'm learning anything.

01:04:44 So, I'm living in Visual Studio Code these days to learn and get a good feel for it because, you know, it's rightly or wrongly, people expect me to know how it works.

01:04:53 And, you know, it would be helpful for that to actually be true when they ask me questions about it.

01:04:58 So, that is where I'm living right now.

01:05:00 But also the extension for that.

01:05:03 So, the Python extension, when we adopted that, oh, gee, a year, two years ago?

01:05:08 Two years ago, yeah.

01:05:09 It was like a one-person part-time thing.

01:05:12 It now has like 100 million people working on it or something.

01:05:16 I can't even keep track of the team.

01:05:18 Yeah.

01:05:18 We have six people based in Vancouver, most of whom I've never actually met in person.

01:05:23 Like, it's moving along so quickly that it's at that really exciting time where every month when the update comes in, there's new and exciting stuff.

01:05:33 And it feels totally fresh.

01:05:34 And it's just exciting to be using right now.

01:05:36 So, that's also fun.

01:05:37 Awesome.

01:05:37 Yeah, it definitely seems like there's a ton of energy around Visual Studio Code.

01:05:41 And I suspect most people know, but obviously, it runs on all the platforms, right?

01:05:46 It's not just on Windows, even though it shares the Visual Studio name.

01:05:49 Yeah, Windows, Mac, Linux.

01:05:51 Having .NET Core support for all of those platforms is, I feel like that's the limiting factor right now.

01:05:56 Obviously, it's electron-based.

01:05:58 So, anywhere that can run, it'll work.

01:06:00 But some big pieces, especially the Python support, are running in .NET Core.

01:06:04 So, that's about the limit.

01:06:06 That's still a really big limit.

01:06:08 Like, you've got some old versions of Linux distros that we support because it works there.

01:06:14 Yeah.

01:06:15 Where really most people would not dream of using them.

01:06:18 But you can run VS Code there.

01:06:20 Like, it's a tool.

01:06:21 It's got to run where people want to use it.

01:06:24 We can't force you to upgrade your operating system just for the editor.

01:06:27 Yeah, that's great.

01:06:27 All right.

01:06:28 I definitely think there's a ton of energy around VS Code.

01:06:31 And I think that whole Python extension editor side of the story deserves.

01:06:36 Like, sort of mention in the Python's journey at Microsoft, even though it, you know, didn't get it to the very end here.

01:06:41 All right.

01:06:42 And then, notable PyPI package?

01:06:44 Yeah.

01:06:44 So, one that I'm really excited about right now.

01:06:47 The Azure Machine Learning team recently came out with their new product.

01:06:51 And the name is very similar to a lot of things we've had before.

01:06:54 So, it can be tricky to track through exactly what it is.

01:06:57 But Azure Machine Learning Service is basically an entire service for being able to do your machine learning tasks.

01:07:05 Everything from data cleaning to model management and deployment.

01:07:08 In a way that works really nicely balanced between your local editor and running stuff and pushing stuff in the cloud.

01:07:16 So, it's got a whole lot of nice functionality for, hey, run this job in a cluster that is this big.

01:07:22 Let me know when you're done.

01:07:23 Nice.

01:07:24 And run that training job and then store the model somewhere.

01:07:26 You get a good history of all the models that have ever been published.

01:07:30 And that's one of the products that has come out with only support for Python.

01:07:35 Like, if you want to use that, you're using Python.

01:07:37 There's actually no other options right now.

01:07:40 So, I'm excited about that.

01:07:41 But one particular part of that package that's really cool is the Azure ML Data Prep package.

01:07:47 So, this is one part of it.

01:07:48 But it's the part for I have a file on disk or I have a set of files on disk that have some raw data in it.

01:07:55 And I need to preprocess.

01:07:57 I need to clean.

01:07:57 I need to extract data from certain columns and split them into more.

01:08:02 I need to add more columns.

01:08:03 I need to remove rows.

01:08:05 I need to replace missing values.

01:08:06 It's a library for doing that.

01:08:08 It does a couple of really cool things.

01:08:10 So, if anyone's used Flash Fill in Excel, which is this cool feature, you can try to put in a list of people's names.

01:08:17 Just first name, last name, however many names in one column.

01:08:20 And then start splitting them out manually.

01:08:23 You put a few examples of type the first name in the next column, then the last name.

01:08:27 And do that for two or three.

01:08:28 And it will suggest splitting up every single row using those examples and figure out how to do it.

01:08:34 So, that's been in Excel for a while.

01:08:35 It's also in this package.

01:08:37 So, you can take a raw data file that has, you know, maybe it has the date and time in those same column.

01:08:44 And you say, okay, here's a couple of examples of how I want it split out.

01:08:47 Put, you know, the date here and the time in a separate column.

01:08:50 And it will figure out how to do that.

01:08:52 And it can, it will use those examples.

01:08:55 And then you can run it, stream it over a huge, huge file that won't fit in memory.

01:08:59 And it will do it to the whole thing and write out a new file that's been pre-processed.

01:09:04 It's changed together.

01:09:05 Yeah, it's cool.

01:09:06 Yeah, it's really, really neat.

01:09:09 I guess there'll be a link in the show notes to the info on that one.

01:09:11 But that, when we look at how far Microsoft has come in terms of Python usage, and it seems, this feels like a great ending to this show, actually.

01:09:19 Yeah.

01:09:20 This is the new point.

01:09:20 We have some amazing technology here that is just Python.

01:09:25 It's just Python.

01:09:26 Like, it's not even .NET first and then Python.

01:09:29 It's not even .NET second right now.

01:09:31 It's Python only.

01:09:32 It's just Python.

01:09:32 It's Python all the way.

01:09:33 That's beautiful.

01:09:34 And that's so exciting.

01:09:35 And, you know, I can't wait to see what comes next.

01:09:39 Yeah, well, it sounds like there's a ton of momentum going there.

01:09:42 And you've done a lot to get it going in the right direction.

01:09:45 So that's wonderful.

01:09:46 All right.

01:09:46 So people are excited.

01:09:48 Maybe they want to learn more about this history.

01:09:49 Some more of the details.

01:09:50 Things like that.

01:09:51 What do they do?

01:09:52 Final call to action.

01:09:52 I'd say go read the Medium post.

01:09:54 And that's got links scattered throughout to interesting things.

01:09:59 And down the end, there's links to all the current things.

01:10:02 So once you've seen where we came from, where we've gotten to, then I'm trying to make it easy to see what we have on offer today.

01:10:09 Yeah.

01:10:09 Well, a lot of cool stuff.

01:10:11 And thanks for doing this historical journey story.

01:10:14 It's great.

01:10:14 Yeah.

01:10:14 Thanks.

01:10:15 You bet.

01:10:15 Talk to you later.

01:10:16 See ya.

01:10:16 This has been another episode of Talk Python to Me.

01:10:19 Our guest on this episode was Steve Dower.

01:10:22 And it's been brought to you by Linode and us at Talk Python Training.

01:10:26 Linode is your go-to hosting for whatever you're building with Python.

01:10:30 Get four months free at talkpython.fm/Linode.

01:10:34 That's L-I-N-O-D-E.

01:10:37 Want to level up your Python?

01:10:38 If you're just getting started, try my Python Jumpstart by Building 10 Apps course.

01:10:43 Or if you're looking for something more advanced, check out our new async course that digs into all the different types of async programming you can do in Python.

01:10:51 And of course, if you're interested in more than one of these, be sure to check out our Everything Bundle.

01:10:56 It's like a subscription that never expires.

01:10:58 Be sure to subscribe to the show.

01:11:00 Open your favorite podcatcher and search for Python.

01:11:02 We should be right at the top.

01:11:04 You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm.

01:11:13 This is your host, Michael Kennedy.

01:11:15 Thanks so much for listening.

01:11:16 I really appreciate it.

01:11:17 Now get out there and write some Python code.

01:11:19 Thank you.

01:11:19 Thank you.

01:11:39 Thank you.