#486: CSnakes: Embed Python code in .NET Transcript

00:00 If you're a .NET developer or work in a place that has some of those folks,

00:03 wouldn't it be great to fully leverage the entirety of PyPI with its almost 600,000 packages

00:09 inside your .NET code? But how would you do this? Previous efforts have let you write Python syntax,

00:16 but using the full libraries, especially the C and Rust-based ones, has been out of reach

00:22 until Seasnakes. This project by Anthony Shaw and Aaron Powell unlocks some pretty serious

00:27 integration between the two languages. And we have them both on the show today to tell us all about it.

00:33 This is Talk Python to Me, episode 486, recorded Monday, November 4th, 2024.

00:38 Are you ready for your host? Here he is.

00:41 You're listening to Michael Kennedy on Talk Python to Me. Live from Portland, Oregon,

00:46 and this segment was made with Python.

00:48 Welcome to Talk Python to Me, a weekly podcast on Python. This is your host, Michael Kennedy.

00:57 Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using at Talk Python,

01:02 both accounts over at fosstodon.org. And keep up with the show and listen to over nine years of

01:08 episodes at talkpython.fm. If you want to be part of our live episodes, you can find the live streams

01:14 over on YouTube. Subscribe to our YouTube channel over at talkpython.fm/youtube and get notified

01:19 about upcoming shows. This episode is sponsored by Posit Connect from the makers of Shiny. Publish,

01:26 share and deploy all of your data projects that you're creating using Python. Streamlit, Dash,

01:31 Shiny, Bokeh, FastAPI, Flask, Quarto, Reports, Dashboards, and APIs. Posit Connect supports all of

01:38 them. Try Posit Connect for free by going to talkpython.fm/Posit, P-O-S-I-T.

01:45 And this episode is brought to you by Bluehost. Do you need a website fast? Get Bluehost. Their AI

01:50 builds your WordPress site in minutes, and their built-in tools optimize your growth. Don't wait.

01:56 Visit talkpython.fm/Bluehost to get started. Hey, everyone. Before we jump into the conversation,

02:02 I want to talk about infrastructure. Actually, there are a couple of interesting things going on over here

02:09 at Talk Python. And I've written them up on our new Talk Python blog, which I will link to both of these

02:16 articles in the show notes. First, we moved our web hosting over to a brand new hosting center in the

02:24 U.S. run by a German company called Hetzner. And it's super interesting. I wrote this all up. And I

02:30 think people who are considering, like, where should we run our apps? Or if they're unhappy where they are,

02:34 it's worth thinking about. And I just rewrote all of talkpython.fm's code, about 10,000 lines of Python,

02:42 to move to an async web framework. Which one was it? Well, that was part of the journey. And so I did a

02:50 big long write-up on migrating from Pyramid to Quart, which is async Flask, all the other frameworks I

02:57 considered doing that, how I went through the migration, and how it's working. So if either of these two sound

03:03 interesting to you, check out the links to the articles in the show notes. As always, thank you

03:09 so much for listening. Let's jump in the conversation. Hello, Anthony. Hello, Aaron. Welcome to Talk Python.

03:15 Hi, Michael. Hi, thanks for having us. It's great to have you here. Anthony, always, always good to have

03:21 you back on the show. You're one of the most popular guests, and we'll see where it lands, where this puts

03:27 you in the list. But you're right near the top of the guest page out of all of them. I love having you

03:33 on the show. Thank you. And Aaron, welcome. It's great to have you here. So even though you've been

03:38 on a bunch of times, Anthony, maybe a real quick introduction, just who you are for those who have

03:42 not met you yet. Yeah, I'm Anthony Shaw. I work at Microsoft. I'm the lead Python advocate. And I do a

03:49 bunch of open source projects. And I wrote a book called CPython Internals, which is how most people

03:55 know me these days. That's me. Yeah. So we're going to talk about more CPython Internals or maybe putting

04:00 Python inside of .NET. I don't know. It's something's going into the internals of the something else. We'll figure

04:06 out which way that goes. It's a very cool project. Hello, Aaron. Tell people a bit about yourself.

04:10 Sure. Well, I'm definitely the odd one out here. I am someone who is pretty much never written Python in my life.

04:17 Millie, I now know a bit more about it, having worked on this project with Anthony. But I'm a .NET developer

04:21 by trade. I've been doing that for longer than I'll admit on a recorded audio. At least 10 years. Okay, got it.

04:29 Yes, we'll go with at least 10 years. I'm setting a lower bound for us. That's a good one.

04:34 I also work at Microsoft. I mean, a team adjacent to Anthony's doing advocacy around .NET. And yeah, but otherwise, I also write

04:43 JavaScript. So I do everything I can to avoid having to get into the Python space in my time.

04:49 And yet here you are. And here I am. Yes. Amazing. Yeah. So it's a different world, right? Coming from .NET, that's guaranteed static typing, a lot of semicolons, a lot of ceremony around the types. And then Python kind of says, hold on now.

05:03 And white space. Python loves its white space. But then again, I have a very soft spot for F sharp. So I'm not like, I kind of get the white space significance aspect of a Python language. But at the same time, I do miss McCurley brackets.

05:15 Yeah. The trick is, if you use a good editor, you don't really notice it. If you don't, it's pretty bad. So use a good editor. I mean, that's all there is to it. Anthony, what have you been up to? What's new in your world?

05:27 I've been up to this project for a few months now.

05:32 Is this primarily what you're focused on?

05:34 It was for a few months. Yeah. I'm trying to get ready for we've got a massive conference, one of our two big conferences of the year coming up in a couple of weeks. I'm working hard on demos to try and get ready for that. But is that .NET Conf? What is that?

05:46 No, this is for Ignite. It's the partner conference.

05:49 Oh, yeah, yeah. Ignite is so big.

05:51 It's like 15,000 people or something.

05:53 Yes, exactly. I went to a pre pandemic. I think it was 30,000 people.

05:57 Yeah, it was in Florida. And there was a whole podcast row. So Microsoft had me out as one of 10 podcasts or something. I thought, Oh, what is this? I hardly ever heard of this. I've heard of build and some others. And I show up like, this is something else. Where'd all these people come from? It's a big conference. It's crazy.

06:13 Yeah, I'm getting ready for that at the moment. But yeah, I've been working on this project for a few months. And we kind of kicked it off in May. Idea kind of floated around in May at the build conference, which is Microsoft's developer conference.

06:25 After talking to a lot of different developers and people in Microsoft, we were kind of kicking around the idea of if we looked at this again, so like running Python code from .NET, like how would you approach it?

06:38 I kind of had some ideas, put some ideas together and then started working with Aaron on a really early prototype just to see like if it was even possible. And then we kind of got something working, chucked it in the bin, pretty much started again.

06:51 And then redesigned it. And then yeah, we've been sort of fiddling with it ever since.

06:56 From my perspective, I said before, I'm not a Python person. But as someone in the .NET ecosystem, looking at the breadth of libraries that exist for Python, there's a lot of things that I would have loved to have used, particularly doing some of the AI stuff that we've been doing in projects recently.

07:15 And then finally, there's just not as many packages or options in the .NET space. So it's always been that like, "Ah, okay, well, if I want to learn this, the tutorial is in Python or the package it recommends using is in Python." So it was always kind of like, how do we bridge that gap?

07:30 That was kind of one of the early thinkings of just, you know, that's a clear problem space, right? Like a lot of people are learning that, but then going into industry and maybe Python isn't the language that the organization they're working with is using.

07:43 How do we still have that? A fundamental knowledge that they've gained or the like the tooling they want to use available to them?

07:48 Well, just think about me. I'm sure you have thought about how much of an explosion in possibility it is if you can say, and by the way, there's almost 600,000 other libraries, including the biggest community of AI based resources that are now, as far as you're concerned, native to .NET, right?

08:07 Yeah. But from the way you consume it, not the way it runs is what I mean.

08:11 Exactly. Yeah. Like you mentioned there, nearly 600,000 Python packages and you add that to, there's like 400,000 packages on NuGet for .NET. Like you got quite literally a million possibilities.

08:24 Yeah, it's amazing. And just for people listening, primarily Python audience, NuGet is .NET's equivalent of our PyPI.

08:32 Yeah.

08:33 Yep. Yep. Yep. Okay. So Anthony, you had this idea. Somebody said, "Hey, how would you go about doing this?" And then you were sucked in?

08:39 Yeah. It was actually one of the sort of lead designers on .NET asked me at Build. They said, "Oh, we've been looking at how you might integrate Python into .NET for a while. Like if you were to approach it, how would you do it?"

08:52 Mm-hmm.

08:53 And I kind of went through all the projects that have tried this in the past or there are different ways of doing it and pros and cons of each and kind of the pros and cons of a really low level integration and a high level one.

09:05 So like a high level one would be stick a FastAPI on top of your Python app and just call it via HTTP REST.

09:12 And somehow auto-generate some REST endpoints and then make those do their thing.

09:17 Yeah. And you could use it like an open API spec on top of it and auto-generate a client and stuff like that.

09:23 So that's like the high level option. And then like the lowest level option is like, you know, you could get into the bytecode level, which I've explored years ago with Pigeon.

09:33 And yeah, that's a very deep, dark tunnel.

09:36 It's full of terrible edge cases and corners and sharp edges, huh?

09:41 And one of the challenges there is that the like Python's implementation is changing so rapidly right now.

09:47 Like, but just in Python 3.13 and looking at 3.14 now and like all the changes and stuff in there, like it's just continuously changing.

09:56 So it needs to be a happy medium in terms of how we blend the two together.

10:01 Right. There has to be some level of disconnectedness a little bit, right?

10:05 So that Python can do it, have its changes without constantly chasing it, right?

10:10 Yeah, exactly.

10:11 So yeah, kind of what we kind of came up with was we would do the integration at the C level using Python C API.

10:18 There's a way of consuming Python through sort of like an embedded mode.

10:23 Not many projects do this, but it is definitely possible.

10:26 So you can spin up a Python interpreter inside another process, which in our case is a net process.

10:32 Yeah. And I think Blender does that.

10:34 I'm pretty sure that there's a bunch in the film and movie industry that do that for their automation pipelines.

10:41 I don't think it's that great.

10:43 One, I talked to some of the people from the movie and video space, and they, most of them are still running on Python 2 because when they wrote the implementation to integrate the thing, integrate CPython, they're like, well, it's done.

10:56 And like you kind of said, Python moved on and they're like, well, that's a lot of work.

11:00 And then we got to, we'll just leave it on Python 2.

11:03 And that's not, that's not great.

11:04 Yeah, exactly.

11:05 So we looked at this and said, we want to be, we want to support the newest version of Python.

11:09 We want to support the sort of practically old version, which I think you settled around 3.8 was like as old as I was willing to go.

11:18 And I think we maybe push that forward to 3.9 because there were some specific features we need on the .NET side.

11:24 Again, there's a lot of changes happening in .NET.

11:26 Oh, Sharon, maybe you can share a bit more, but like, you know, there's .NET is going through just as many changes.

11:32 So we've got like two moving, constantly moving platforms and we want to make an integration between the two of them.

11:38 So let's think about like, how can we do this in a way which is actually maintainable and we can make it, you know, we can test it and we can make it stable as well.

11:45 Yeah.

11:46 One thing maybe you could speak to as part of this, people might know .NET as this Windows programming library that you use Visual Studio with.

11:55 Right.

11:56 But it's, it's changed a lot since then.

11:58 There's .NET Core, there's open source.

12:00 It runs on Linux.

12:01 There's, there's a lot of different things that have been changed about it.

12:04 Right.

12:04 Yeah.

12:05 And it's a fairly common misconception that we've got is that, that people equate .NET with Windows and like historically that's, that's true.

12:11 Like that's, that's where I got started is developing on Windows.

12:14 And I, I'll be honest, I'm still a Windows developer.

12:17 It's what my primary OS has been.

12:20 I've, I've flirted with macOS and Linux over the years, but Windows is just where I'm more comfortable with.

12:26 But I think it's been a bit over a decade now since the, they essentially relaunched .NET was originally called .NET Core.

12:35 So, so now we kind of have these two streams concurrently.

12:38 You've got .NET framework, which is the one that is what we traditionally thought of as .NET.

12:43 It, it ships kind of like with Windows and stuff like that.

12:46 So it's, it's very much designed around that longer tail support lifecycle and, and, and whatnot.

12:53 It is fairly stable, but also getting minimalistic investment just because it has that kind of need to be stable and consistent year in year out.

13:03 Then we have .NET Core, or now we just kind of refer to it as .NET.

13:08 Just naming things is hard.

13:10 Let's, let's be honest.

13:11 But so we, we have .NET and every year a new release comes out and this is like a, like a major framework, like a major revision to the runtime.

13:20 So .NET 9 is actually coming out next week, at least at the time of recording, it's next week.

13:25 And this is considered a, like a short term support lifecycle.

13:29 So if you're familiar with support lifecycle of a lot of runtimes out there now.

13:34 So like Node popularized this quite a lot now.

13:37 And Python is starting to adopt it a bit more that kind of you have this yearly rolling lifecycle and, and you'll have long-term supports and short-term support.

13:45 So the long-term is every even number.

13:47 So .NET 8 being the most recent long-term support.

13:51 .NET 6 was the previous one, which is actually going out of support next week at the time of recording.

13:56 Yeah, being realistic of where people are and developing their applications.

13:59 Where we couldn't just target Python 3.13 and .NET 9 as the only supported versions, because be honest, that's not where most packages are.

14:08 That's not where most projects are going to be.

14:10 So we have to make, you know, some, some calls there.

14:12 Yeah.

14:13 You do have to have a little bit of, a little bit of slack in there.

14:16 Even for me and some of my web apps and stuff, I had to wait a few weeks to upgrade to 3.13 because some dependency of a dependency had something that had been removed.

14:25 You know, been deprecated for a while and was removed and it didn't work.

14:28 Like, well, try that again in a few weeks and see how it works.

14:30 And eventually they, they caught up, but yeah, it's definitely a challenge.

14:34 This portion of Talk Python to Me is brought to you by Posit, the makers of Shiny, formerly RStudio and especially Shiny for Python.

14:44 Let me ask you a question.

14:46 Are you building awesome things?

14:47 Of course you are.

14:48 You're a developer or data scientist.

14:50 That's what we do.

14:51 And you should check out Posit Connect.

14:53 Posit Connect is a way for you to publish, share and deploy all the data products that you're building using Python.

15:00 People ask me the same question all the time.

15:03 Michael, I have some cool data science project or notebook that I built.

15:07 How do I share it with my users, stakeholders, teammates?

15:10 Do I need to learn FastAPI or Flask or maybe Vue or ReactJS?

15:15 Hold on now.

15:16 Those are cool technologies and I'm sure you'd benefit from them, but maybe stay focused on the data project.

15:21 Let Posit Connect handle that side of things.

15:23 With Posit Connect you can rapidly and securely deploy the things you build in Python.

15:28 Streamlit, Dash, Shiny, Bokeh, FastAPI, Flask, Quarto, Ports, Dashboards and APIs.

15:35 Posit Connect supports all of them.

15:37 And Posit Connect comes with all the bells and whistles to satisfy IT and other enterprise requirements.

15:43 Make deployment the easiest step in your workflow with Posit Connect.

15:47 For a limited time, you can try Posit Connect for free for three months by going to talkpython.fm/posit.

15:53 That's talkpython.fm/posit.

15:56 The link is in your podcast player show notes.

15:59 Thank you to the team at Posit for supporting Talk Python.

16:02 Anthony, when we opened this, you mentioned there had been other attempts, other ways.

16:08 The one that comes to mind for me is IronPython.

16:11 IronPython is a CLR implementation of Python.

16:15 And that's a pretty big difference right there than what you're trying to maybe contrast it with some of the things that people have done previously.

16:23 Yeah, so probably three projects, I think, that have been done previously that kind of sit in the same space, but they have different goals.

16:30 So, I mean, the main goal for our project is that there are loads of .NET developers that have existing applications.

16:38 They are typically quite, it like sits in the enterprise space, a lot of it.

16:42 So they're like quite big, big applications.

16:44 They've got teams working on them and there's a technology in Python that they want to use.

16:49 And that technology is often a package or a library or an example, especially in like data science now, and a lot of the ML and AI libraries that are floating around.

16:59 So a lot of the idea for this and the demand from developers was from, you know, we want to use these libraries or we want to use this particular package in Python.

17:09 And like, how do we import that into our .NET application?

17:12 So like, that's kind of the angle that we've come at this from, which is like, how can we make the best of both worlds?

17:18 Like you've got a big, you're like a fast, mature enterprise application in .NET.

17:23 And it is like a piece of technology that's in Python that you want to use.

17:26 Iron Python is an implementation of Python written in .NET.

17:30 And that was done quite a while ago, I think.

17:33 And yeah, like older versions of .NET, older versions of Python.

17:37 And it's kind of similar to Jython as well.

17:41 Like I think I put it in the same bucket as Jython, which is...

17:44 I would as well. Yeah.

17:45 It would like, and at that time, a lot of people were making different implementations of Python, the language in other frameworks.

17:52 And so, yes, you can write Python syntax and run it in the CLR, but you couldn't do something like import NumPy.

17:58 And that's the difference is that most packages in the ecosystem these days are designed and tested for CPython.

18:05 And this is all, at least they're tested for PyPy.

18:09 But all the other implementations are pretty much a no.

18:12 Like Pandas, NumPy, scikit-learn, like all the data science ML tools.

18:17 Especially the C interop type of stuff.

18:20 Yeah.

18:21 And I've sort of written about this in the past, how, you know, Python's...

18:24 But one of the solutions to Python's performance problem is that you implement extension modules in C, which is a great solution.

18:31 Or you use Cython, which is the same kind of thing, which is a great solution.

18:34 It just means that the ecosystem is more and more tied to CPython as the interpreter of choice.

18:39 So what I didn't want to do is write a whole new interpreter.

18:42 Like that goes back to the compatibility problem and the maintainability issue.

18:47 Like by the time we published it, it would be out of date and we'd just be opening up stuff up for a whole load of work.

18:53 It's not even about, it's not even a matter whether you can do it and you can keep up with the language.

18:58 As we talked about, you know, as Aaron mentioned, there's a million libraries you can use.

19:02 But if you did this, you wouldn't be able to.

19:04 Yeah, I'm Python is one.

19:06 Pigeon, we've covered in a past episode.

19:08 Oh, that's not going to work.

19:09 Is a P-Y-G...

19:12 There we go.

19:13 Is a weird spelling.

19:14 P-Y-J-I-O-N.

19:16 No, that's right.

19:17 I remember now.

19:18 P-Y-J-I-O-N.

19:19 This is a JIT for CPython that happens to use the .NET Core JIT engine because that was one of the most pluggable JITs at the time.

19:31 There are other JITs available.

19:32 It could have been written in ASM JIT.

19:34 It could have been written in LLVM JIT.

19:37 Dino who actually worked on Iron Python.

19:39 So like he knows this space really well.

19:41 Brett Cannon worked on Pigeon originally.

19:43 And then I rewrote it like a few years later.

19:46 That's a different problem.

19:48 That doesn't help you integrate .NET and Python together.

19:51 It just happens to use components from .NET to implement a JIT.

19:55 That project is effectively dead.

19:57 Some of the benefits of it have been rolled into CPython in versions 3.10 and 3.11.

20:03 The things that made it faster.

20:05 Most of those were sort of picked up anyway.

20:07 And then the third one is Python.NET, which is probably the closest to the project that we've started.

20:13 This one has been around for a while.

20:16 This one's been around for a while.

20:17 It's a similar thing, but they mostly focus on the integration the other way around.

20:21 So like how do you call .NET libraries and things from Python?

20:26 You can call Python.

20:27 Oh, right.

20:28 Yeah.

20:29 A lot of the focus is actually the other way around.

20:31 So if there was a popular .NET library that you wanted to use in your Python app, you might Python.NET your way to that.

20:37 And this project has been around for a while.

20:39 So the API is pretty stable.

20:41 And we originally looked at this one actually as a potential solution, but found a number of things that they'd kind of tied themselves to in the design, which are going to go away in .NET 9.

20:50 So it wasn't really an option for us.

20:54 And also like a lot of the way the C API was used was different to how we wanted to do it.

20:58 So yeah, that's kind of the alternatives.

21:01 And so you ended up writing your own because just none of these really match, right?

21:05 The Iron Python was just never going to keep up and it didn't have this package support for data science mostly.

21:12 And then the other one was the other way around.

21:14 And yeah, so well, I guess a good thing to think about is, well, why did you think you could do that?

21:21 Like a lot of people, a lot of smart people put a lot of time into this.

21:24 Why did you think, oh, this is going to be, this is going to work well.

21:27 Do you have a fresh take on it or what were you thinking?

21:29 I really wanted to use some of the modern benefits of both Python and .NET to do this, to design it.

21:37 And one of the biggest changes in actually, as we've been implementing this, we're learning 80% of the challenges are related to typing.

21:44 And Aaron's smiling because he's agreeing with me.

21:48 Like integrating two different ecosystems together like this is about calling functions, methods, whatever.

21:56 But calling things is about calling them with arguments really.

21:59 And it's passing data backwards and forwards, marshaling information and getting things backwards and forwards between different types.

22:05 Like that's if you integrated, I don't know, some function that you wrote in Python into .NET, you're going to be calling it with something like you're going to be sending it some data and expecting some data back.

22:16 Now in Python, there are types.

22:18 It is dynamically typed, but the types do exist.

22:21 Even if you don't put it in your function signature, there's still instant strings and stuff being sent all over the place behind the scenes.

22:28 Yeah, exactly.

22:29 And when we looked at Python.NET, like, you know, you could call a function in from .NET in Python, but you'd have to write a load of code that said like, oh, I think this parameter is probably this type.

22:39 So convert that to this and then when it comes back, then it's probably this one to try that one first.

22:44 And so we sort of said, like, now typing type annotations are more popular in Python.

22:50 What we wanted to do was basically you give it some Python code and you write like a function, which is the one that you want to call and you annotate the function.

22:59 So you say it takes, you know, has these arguments and they are these types and it returns back this type.

23:05 And then what we did is wrote a code generators called a source generator for .NET that would in in your project, it would look at the Python file that you wanted to embed.

23:15 It would see what functions are defined, what their signatures are and what types they take.

23:19 And then it would generate a function that can call that for you with .NET equivalence.

23:24 And then it handles the the marshalling, like the transfer of the types backwards and forwards between .NET and Python.

23:32 And like that's most of the project is really about that particular problem.

23:36 Okay.

23:37 So it's leveraging the C API of Python kind of talk directly to the runtime.

23:42 You can write a function in Python, you know, that takes a string, a list and a tuple and returns back a dictionary.

23:48 Like it's a fairly common thing to do.

23:50 Yeah.

23:51 You would give it type annotations in Python and you drop that Python file into your .NET project and you run build again and you would see a type in .NET with the same name, with a function that is the one you just defined and with arguments which are .NET equivalents to that.

24:07 And you just call the function and it just runs in Python.

24:09 How complicated can this data exchange be?

24:12 Can I come up with my own types and exchange them or do they have to be six or seven core types?

24:17 I think the data exchange is actually one of the really tricky bits, right?

24:21 Here's a list of a million items.

24:23 Why don't you copy them all over to a .NET type, run it or copy it over to a Python list, run the thing and then copy the back to a .NET list with one more item in it or something crazy like that, right?

24:34 Yeah. If you jump to the documentation and then in the reference page, there's a list of the ones that we support at the moment.

24:41 Oh, yeah.

24:42 Then there's a list of the core ones that we support.

24:44 And then anything beyond that.

24:46 So we do generics, for example.

24:48 So if you say it's a list of strings, we will give you back a read only list of string in .NET.

24:54 If it's a dictionary and you type the key and the value, we will give you back a .NET key and value.

24:59 Now in Python, often it could be an any or it could be a union or it could be something complicated.

25:06 So you've got the option to say it could be a dynamically defined type.

25:10 It does still support it.

25:12 And then at runtime, you can look at the type to see what it is and then convert it into something that you need.

25:16 I see a lot of read only on the .NET side.

25:19 And this is the thing receiving the data from Python.

25:22 I'm glad that you picked up on that because that was something that I made a decision when we're doing the .NET service area for this of it.

25:29 So that's the thing that I think that anytime we're receiving data back or when we're sending data to is it should be represented as read only just to indicate that if you're calling a function that's been exposed from Python from .NET and it's returning a collection like a dictionary or a list or anything like that.

25:47 So that's the thing that you're going to do.

26:08 So that's the thing that you're going to do is you're going to do that.

26:13 And then you're going to do that.

26:31 That you're very familiar for .NET developers for doing like creating an array from an existing collection of data.

26:39 And then that becomes obviously a more mutable data structure or like a list, like the read-only list, for example, that then plugs straight into links so that you can do the query syntax or the expression methods syntax to perform operations on those collections of data inside of .NET space.

26:59 Kind of once it's being hydrated back out of Python.

27:02 So I guess you get a lot of these read-only innumerable type of things.

27:05 You could do link on them, L-I-N-Q.

27:07 Yeah.

27:08 I wish we had this in Python.

27:09 I know we have something kind of like it with list compreemptions, but it's not the same.

27:12 One thing that I really wanted to make sure that we were tackling with this and to Anthony's point before of the, that you've got two very different type systems that you're working with.

27:21 And structurally, the way that you write code in Python is going to look different to the way that you write in .NET.

27:27 So we didn't want to feel like the code gen that we do for you is that we've just made this, it's just .NET casing of Python code.

27:37 So I wanted to feel like as a .NET developer, I'm working against a .NET interface.

27:41 So when you work with the types and the modules that you expose out to .NET and we do code gen against, we then generate you a class behind the scenes that is then, we then generate an interface for you.

27:55 And then that's registered into the dependency injection container.

28:00 So you then can just resolve that elsewhere with inside of your application.

28:03 So now as a .NET developer, I'm getting an interface injected into a class that I'm using elsewhere with inside of my code base.

28:12 I haven't had to go, okay, well, no, I'm calling some kind of like an import or I'm doing some kind of resolve or something that might be unfamiliar to me as a .NET developer.

28:24 This is what I'm doing is just, it just feels like I'm writing, or I shouldn't say necessarily .NET developer, I'm running C#.

28:31 And we've optimized this for C#.

28:33 I said earlier on, I have soft swap for F# and it kind of works in F#.

28:39 And then similarly, if you're a .NET developer, we optimize this to the kind of development experience you would expect when you're writing C# code.

28:47 We just so happen to have, you, you haven't written this, some of this C# code.

28:52 We've just automatically generated it for you and injected it into a dependency injection container.

28:57 Amazing.

28:58 So when we're talking about exchanging this data, this is CPython in the same process.

29:03 Is that right, Anthony?

29:04 Yeah.

29:04 How's that work?

29:05 Python's C API, normally if you call, so CPython is the interpreter that you would use to run your Python code.

29:12 It is implemented in C mostly.

29:15 The standard library modules are written, some of them are in Python, some of them are in C.

29:19 You can write modules for Python in C.

29:22 And the way that you would do things like create numbers and dictionaries and lists is you use this API.

29:28 It's a C API.

29:29 And part of that C API is the ability to spin up a Python in C.

29:34 You can spin up a Python interpreter inside another process.

29:38 And that's embedded Python.

29:40 There's different APIs for doing it.

29:42 There's a lot of different ways of approaching it.

29:44 But really what we do is that when you spin up .NET and you use CSnakes, you inject a Python interpreter into the environment.

29:53 It's not like a separate process.

29:55 Like it's not, it hasn't got like a ship command line windows running in the background.

29:59 It's embedded into the process.

30:01 It's like loading the library.

30:02 It's running inside the process.

30:04 And then we have full control over everything in CPython.

30:07 So we can then import modules.

30:10 We can convert things, types backwards and forwards.

30:14 So we're kind of doing a lot of the work that often gets done either in a C extension module or inside the interpreter.

30:20 So if we want to create a Python number, for example, like an integer from a .NET integer, we do that at the C level.

30:27 So we can do that like a really high performance API.

30:30 If we want to look at the size of dictionaries or the types and stuff like that, we do that at the C level.

30:36 So again, it's super high performance.

30:38 And then really importantly, well, the typing stuff is really important.

30:43 The other major challenge with this is, you know, .NET freely uses threading and thread pools and coroutines and stuff like that.

30:52 Python can use threading, but there is a lock.

30:55 So you have to be really, really careful.

30:58 And this is something I don't didn't see like a novel solution from the other projects that are looking in this kind of space about how they would tackle it.

31:05 Or it was kind of based around like older styles of Python and things have changed so much recently.

31:10 So we kind of wanted to make sure that you could run this in a multi threaded environment, whether that's like a UI, for example, like a desktop interface or it's a web app.

31:19 So like one of our demos is like a .NET web app, like API, that each request can run a Python function if it wants to.

31:27 And they're all running in separate threads in a thread pool in .NET.

31:30 And you shouldn't have to worry about the gill.

31:32 Like you shouldn't have to know too much in terms of what it is and how it works.

31:36 So we've done quite a lot of work to make sure that that can happen.

31:41 This portion of Talk Python to Me is brought to you by Bluehost.

31:45 Got ideas, but no idea how to build a website?

31:48 Get Bluehost.

31:49 With their AI design tool, you can quickly generate a high quality, fast loading WordPress site instantly.

31:56 Once you've nailed the look, just hit enter and your site goes live.

31:59 It's really that simple.

32:00 And it doesn't matter whether you're a hobbyist, entrepreneur or just starting your side hustle.

32:05 Bluehost has you covered with built-in marketing and e-commerce tools to help you grow and scale your website for the long haul.

32:12 Since you're listening to my show, you probably know Python, but sometimes it's better to focus on what you're creating rather than a custom built website and add another month till you launch your idea.

32:22 When you upgrade to Bluehost cloud, you get 100% uptime and 24/7 support to ensure your site stays online through heavy traffic.

32:31 Bluehost really makes building your dream website easier than ever.

32:34 So what's stopping you?

32:35 You've already got the vision.

32:36 Make it real.

32:37 Visit talkpython.fm/bluehost right now and get started today.

32:41 And thank you to Bluehost for supporting the show.

32:45 Python has async and await.

32:47 C# has async and await.

32:49 They're similar, but not the same.

32:51 Yeah.

32:52 Can I have an async function in Python that is called by C#?

32:57 No, not yet.

32:58 Just a task.run, async.io.run to completion sort of thing.

33:03 And then keep going.

33:04 Yeah.

33:04 I would love to do this, but there is no API for it on the Python side to manage event loops and embed an event loop into another process or to write a custom event loop.

33:17 It's all kind of built into the standard library or was put into another module.

33:22 There's been a couple of proposals in draft peps, but none of them have been accepted.

33:26 And if they were, then it might be in 3.14 anyway, which we can't backward compatible support.

33:31 Yeah, you're not quite there time wise yet.

33:33 Do you find that async.io event loops and Python are hard to juggle?

33:37 Right?

33:38 Like you have to create one if it doesn't exist, but if it does exist, then it doesn't want you to create a new one.

33:42 But there's not a great way to see if it exists.

33:44 And if you don't have direct access to it, how do you get like, it's just, it always feels to me a little bit out of touch to fully take control over that and work with it.

33:52 And it changes between Python versions.

33:54 Like it gets, it's gotten easier recently.

33:57 Like there's a, there's like a dot run function now.

34:00 And whereas previously you have to like get the event loop and create one and stuff like that.

34:04 But now you can just kind of run the coroutine.

34:06 It's still tricky.

34:07 And the idea of basically writing a custom event loop and then giving it coroutine objects and like polling them and looping them from another runtime.

34:18 I just couldn't figure out the API.

34:20 I'd love to do it.

34:21 If anyone knows, then please ping me.

34:23 Well, UV loop does it.

34:26 Yeah.

34:27 Somehow, but maybe they just swap out some pieces of it.

34:30 Not all of it.

34:31 Yeah. I'm not sure how, but you can set the event loop from UV loop.

34:34 So maybe, I don't know, maybe there's a breadcrumb there to chase.

34:37 All right.

34:38 So I see in the steps, install Python, put your projects in a C# file, use annotations on your Python functions, right?

34:46 That's a really important step that you talked about.

34:48 Install the Csnakes runtime package.

34:51 So maybe from this point on, let's talk a little bit about some of the steps.

34:55 So this Csnakes runtime, this is a NuGet package that you put into your .NET project.

34:59 Is that what I'm reading here?

35:01 Yeah.

35:02 So we essentially have two binaries that we ship.

35:05 One, which is going to do all the code generation for you.

35:08 So from a .NET perspective, you'd be, it wouldn't be uncommon to have multiple different projects with inside of your solution, which is then like one might just be a bunch of services, which are then going to then talk to a database or whatever the case may be.

35:25 So you might have that as just a, like, that would be where you would have the code gen that you want to do.

35:31 So where the Python files are actually getting their C# interop generated for you.

35:35 So you have one project with that.

35:37 And then we have a second binary, which is then the runtime to actually like make that work when your application is running.

35:44 Now we do ship this as a single consolidated package just for ease of use, but that's kind of why we differentiate some of it in terms of like code gen versus runtime because they're different concepts.

35:55 And one is a compilation step versus obviously runtime step.

35:58 Yeah. Okay.

35:59 With our runtime package and the documentation talks a little bit more about this.

36:03 There's just a few steps you've got to do to make sure that the C# compiler is capable of finding the Python files.

36:09 So we actually do stuff with inside of the C# compiler pipeline to generate code that is then added to it.

36:16 So this also means that we can very early on find out if you're going to have compatibility issues.

36:22 So if something is type of the type that we don't understand or the Python code is syntactically obscure, because what I've learned is that there are ways that you can write valid Python code that is very obscure.

36:36 And they showed me some, yes, some things that you can do, but please don't do it.

36:41 And yeah, so you'll actually fail a compilation step rather than fail runtime process.

36:45 And I think like, again, from a from a .NET developers perspective, like you were used to that static typing that compilation step that that belief that the compiler has done a whole bunch of checks for us to give us code that should just work rather than just like that more dynamic or typing system that you get out of Python or like in my experience, like the JavaScript side of things.

37:05 Yeah, that's definitely the expectation.

37:06 the expectation if you're doing C#, C++, if it compiles, at least it's clicked together in a coherent way.

37:13 Right. Whereas, whereas Python, we're pretty sure we don't know for sure, but we're pretty sure.

37:17 So in Python typing, you can lie.

37:20 Right. I could say it takes a list of integers and really what it took was strings, which had numbers as characters.

37:26 Who knows? Right.

37:27 What happens if I lie to the C# code gen thing?

37:30 So it just crashes.

37:31 Don't do it.

37:32 Yeah.

37:34 It does actually just crash when it calls because it has to make the assumption that what you've told it is correct.

37:40 So in .NET, if I've got a tuple of ints and I give it across and it's like, I actually wanted strings.

37:47 No, it's not going to have it.

37:48 It will raise a, yeah, because to get, for example, if it returns a list of strings, but you actually return a list of integers, you know, in order to give you back a list of strings in C#, we first check that it's a list.

38:01 And if it isn't, or if it's not a sequence type, like it could be a, you could have inherited from this to major and custom list, or you could make something which is a sequence.

38:09 Like if it's not that, then we will raise an exception.

38:12 It doesn't just crash.

38:13 I was joking.

38:14 And we've got a lot of testing to make sure there's a whole, like a test environment that's around like lying code and type hints and stuff, which are completely invalid.

38:23 And then for the strings, you know, we need to look at the Python string.

38:26 And then because Python mostly uses UTF-8 and .NET mostly uses UTF-16, you know, we need to convert from one to the other.

38:34 And if it's, and if it's an integer, not a string in the first place, then that's never going to work.

38:38 So we would raise an exception there and say, you taught us it was a string and you lied.

38:42 But you lied exception is thrown.

38:43 Yeah.

38:44 I think it's like a cast, invalid cast exception or something like that.

38:47 Yeah.

38:48 With the description.

38:49 We've tried to raise like the exceptions we raise are .NET exceptions, which are relevant to the concept of the problem that we've covered.

38:57 But at the same time, it actually will give you, imagine if something failed in the Python code as well.

39:02 Like you type annotated that it was like a bunch of strings and we, and I know a bunch of ints and we, we gave you a bunch of strings.

39:09 Well, okay.

39:10 The Python code is going to fail there.

39:12 So where we then surface out the Python exceptions as well as like embedded with inside of the .NET exception.

39:19 So you can, you can actually get down to like the, the lines of code with inside of Python that failed, which I think is kind of.

39:26 The stack trace actually includes C#, C#, C#, Python, Python, Python, potentially.

39:31 Yeah.

39:32 That's pretty awesome.

39:33 In Python, if you didn't raise from it, that will show up as an inner exception in .NET as well.

39:38 So we did a lot of work on exceptions, mostly because we had to like debug this stuff when we're developing it.

39:43 And it was pretty amazing.

39:45 We saw the pain that someone else was going to feel.

39:48 So yeah, you see the, you can see the stack trace, you get like all the frame variables and stuff like that.

39:52 So things that you would probably want from like a PDB, you can see on the .NET side.

39:57 So if you're running the debugger in .NET and it crashes in Python, like it or raises an exception or something, you see the Python exception with all the details.

40:06 And then you see like inner exceptions and stuff like that as well, as well as the locals and the globals and any other frame information.

40:13 Yeah, I was pretty impressed when I saw that, that how integrated, how much I felt from both directions, kind of natural and not just bolted on.

40:21 Yeah.

40:22 Another step is mark these for code for source generation.

40:25 This feels like a Visual Studio 20, whatever year the last release of Visual Studio was.

40:32 Aaron, where are we?

40:33 What's the number?

40:34 22.

40:35 Okay.

40:36 So the reason I bring this up is a lot of people hear Visual Studio, especially in the Python space and think Visual Studio code.

40:41 Does this work in Visual Studio code or is it just a other Visual Studio?

40:45 No, it definitely works in VS Code.

40:47 They sometimes open it in VS Code.

40:49 And obviously you'd likely just installed the C# extension to get a better developer experience on that.

40:54 But what we have to do is because the source generator is actually running as they step with inside of the C# compiler pipeline, we need it to know how to find these files.

41:04 And we kind of toyed with the idea of just to like, just do a directory scan.

41:08 Everything that ends in .py, we're going after.

41:10 That's not particularly efficient.

41:12 And also like if you've just got like modules that are using other modules, like you want to, what's, how do you craft a surface error in Python that you actually want to expose rather than just expose everything?

41:22 So we're inside of the csproj file.

41:26 So this is the file that is the instructions on how to compile a more complex .NET application.

41:33 We can then add nodes to that XML structure to say, like, here's a Python file.

41:39 It's an annotate that for the C# compiler to pick up so that when the source generator hits it, it hits our steps.

41:47 It goes, okay, I'm going to find all of these additional files that you've said I need to be aware of.

41:52 And then I can read them in and it's actually got some optimizations for how like internally it will read that file in rather than us then having to do file IO or anything like that.

42:01 So it just, it's more in physically aware of them and that it will make those available so that it can then generate out the code rather than, rather than like I said, us having to try and guess what Python files you want available.

42:14 Like, you know, they have to be in a special folder or they had to be named a certain way or we just directory scan everything that ended with .py.

42:21 Yeah. You want to be really careful about which ones you're trying to do this to because they require type annotation and you don't want to just expose all the guts of some bit of code.

42:31 You want to maybe talk about the surface area, right?

42:33 As .NET people, you have these protected private internal, protected internal, et cetera, things.

42:38 We don't talk about them in Python.

42:40 I want to be consistent with that, right?

42:41 So Anthony, it says last step to instantiate the Python environment.

42:47 This is, it feels almost like a, an import statement.

42:50 Then you use it namespace style later.

42:53 Do you want to talk about that real quick?

42:54 In order to start like into embed Python into the .NET process, you need to create a Python environment.

43:00 And there's a few things we have to know, like where is Python installed?

43:03 What version is it?

43:04 Do you have a virtual environment?

43:06 Is it just like the global environment?

43:08 Do you have like packages?

43:10 There's like additional things that we need to know that you would have in Python, you would have kind of set those elsewhere.

43:16 Like if there was a virtual environment, you would have activated it.

43:19 You know, if you were running Python, then you would have set which version it was and where it's installed.

43:24 So you've in .NET, you haven't done those steps.

43:28 So we need you to tell us like which version of Python do you want to, do you want to create and where is it?

43:34 And we've created a whole bunch of code to help you find it.

43:37 The UV team actually have been solving a similar problem with like spitting up Python and running Python.

43:44 So I'm kind of curious to see like how they've tackled it.

43:47 And we might even try and look at bringing UV in as one of the mechanisms for finding Python for you.

43:52 Because it's such a complicated.

43:53 Yeah.

43:54 It's something I wanted to talk to you about is you've got to have Python installed and you've got to get the dependencies.

43:59 And yeah, we're not going to install Python for you.

44:02 Yeah.

44:02 But with UV, V and V, give it a Python version.

44:06 It can fetch it.

44:07 It can fetch it and just run it and it's cached.

44:09 And maybe you could embed that as a resource at compile time.

44:12 There's interesting possibilities there.

44:15 Virtual environment support uses either condor or pip as the package manager, depending on what you want.

44:20 And you can say, here's my requirements file, and it will go in and pip install them as well.

44:25 Like it can do all of those things for you because we're kind of assuming that, yeah, you've developing this locally and you're running it locally, but it's going to be deployed somewhere else.

44:34 So you don't want to have to like log into the production servers and like create virtual environments on the command line and stuff like that.

44:41 That's like with hopefully 15, 20 years past people doing that.

44:45 Like, so you should be able to programmatically say this project needs a virtual environment and it needs these packages installed and needs this version of Python.

44:54 And then you can either use the version of Python that comes with your Linux container, for example, or you can pull in one from somewhere else.

45:03 For Windows, you can actually download it from NuGet.

45:06 It's like you can actually add it as a dependency in .NET.

45:09 And that's a really simple solution for Windows development for macOS.

45:12 If you've installed it from Python.org, like using the official installer, you just you just tell it that you just say, use like the macOS Python and give it the version number.

45:22 And then for Linux, it's a lot more complicated because there's no standards.

45:27 So, yeah, we've kind of support every which way of finding it on Linux.

45:31 And then once you've kind of put the Python environment into the process, there should only be one of them.

45:36 Python itself does not want you to ever spin up more than one Python environment in the same like operating process.

45:43 It will have all sorts of problems if you if you do that.

45:45 So it's basically put in as a singleton and then you import a module by creating an instance of it.

45:52 And then on that module, you can then call methods which are the functions in the module.

45:57 So like if you had a module called demo.py, then in your environment, you'd see like a factory for like effectively like import demo and giving back an instance of it.

46:08 And then on that instance, you'd have methods which are the functions.

46:12 And the example you've got on your screen, you know, there's a function called hello world that takes some arguments in a in a file called demo.py.

46:19 So you just say environment.demo, like make a demo instance.

46:22 And then on that instance, you can call dot hello world with the arguments and it gives you back the results.

46:28 So there's no really like in a few lines of code, like four or five, you can spin up a Python environment, import a module run a function.

46:35 So we'd really try to minimize the border plate code by and then following the patterns that are best for both platforms.

46:41 So a lot of places you could go with this is really cool.

46:44 For example, if you if you got a singleton, which is great.

46:47 And that's kind of how Python typically is one environment.

46:49 But with the sub interpreters, you could theoretically create these in parallel.

46:54 But one thing that stands out as maybe a little non expected in this is like you have var module equals env a demo, right where you're creating this.

47:03 But in Python, this thing is called demo.

47:06 Would it be a pattern to maybe name the variable the same as the module name?

47:11 That's probably just more from laziness in the way we've written our docs.

47:14 Yeah.

47:15 But what we're probably lacking from because this is obviously docs and just markdown code is that so when you were so we would get back our IPython environment from our dependency injection container and then things like the demo methods exposed there would then return an i demo.

47:33 I think it is or and that is then the representation of the that demo dot PY file that you've imported.

47:40 So, yeah, probably.

47:42 Yeah.

47:42 So then then all those methods are then exposed on that.

47:45 So, yeah, a module from it like conceptually, if I'm if I'm thinking about this as a dotnet developer looking at it from Python, like, okay, that the module is that Python file.

47:54 And that's what I get back when I say, like, I want an instance of the interface into that file.

48:00 Yeah, and it does honestly it doesn't matter how I feel about this as a Python person.

48:04 It feels about it matters how dotnet people feel about this when they look because that's this is C# code.

48:09 That's their code.

48:10 That's their world.

48:11 It just so happens that C snakes make it makes it really nice possible to talk to the Python world.

48:16 Yeah.

48:17 So one thing I wanted maybe could spend a couple minutes talking about this Anthony is I saw an example where you you showed integrating open telemetry and ASP dotnet and Postgres and all sorts of stuff together, not just a, you know, hello world sort of example, which shows some really cool stuff, but maybe doesn't convince people that they could do cool stuff with it.

48:39 You want to just maybe talk us through some of the moving parts of that to give people a sense of a more realistic example.

48:44 Yeah, so we have a another demo, which is a web app medicine in the project.

48:49 Well, this is kind of what we thought was a more realistic integration example.

48:53 So you've got a web app written in.

48:56 Yeah, well, the front end written in HTML and JavaScript.

48:59 The API is written in dotnet.

49:01 You're using Postgres as the database.

49:04 And you've got like a feature that you want to add to your app where you want to use some Python libraries and some Python code.

49:11 And we were looking at whether everyone loves the weather demo.

49:14 Just real enough.

49:15 Come on.

49:16 It's just real enough.

49:17 The weather demo is like the canonical dotnet demo of when you do file new API projects like here is a weather app.

49:24 So I from a dotnet developer, I feel very much at home.

49:27 Yeah, you're already on board.

49:29 Okay, got it.

49:30 Yeah, we're creating the one that people would expect.

49:32 Now on the dotnet side, we also said, right.

49:34 Yeah, let's be realistic about what people would want to do, because like modern applications, you would write the API in ASP.net and you would, you know, have API endpoints that you would and routes and configure those.

49:47 And you would have a probably a relational database like Postgres or SQL Server or something as the storage environment.

49:54 And then now we support open telemetry in a platform called Aspire, which Aaron can probably explain in a second.

50:01 So open telemetry is a new standard for like tracing and instrumentation and observability generally of applications written in any language.

50:12 And in dotnet you can have.

50:14 So for this app, for example, like, you know, somebody makes a request to the web app and you see the request come in and all the timing and there's any log records in dotnet or if anything crashes or like it's all associated with the same request.

50:26 Now, if we're going to spin up and run a Python function and in Python, you do import logging, you know, logger logging dot info or logging dot error.

50:35 Like, where does that go?

50:36 Because you've just logged something in Python.

50:39 Does it just disappear into the ether?

50:41 Is it like it goes into the terminal output of the container?

50:45 It runs it.

50:46 Who knows?

50:46 Right?

50:47 Yeah, exactly.

50:47 Like it would be kind of useless and not easy to manage.

50:51 And we're coming at this at the angle, like people write big enterprise apps in dotnet.

50:56 So like they would expect that they do logging and it's all in the same system.

51:00 So this demo basically says, OK, one of the API endpoints in on dotnet, it runs a function in Python, which spins up.

51:09 It uses pandas and as a data frame to look at the weather or in it, like calculate some averages and shifting means and stuff like that to give you back like a prediction for what the weather is going to be like this week.

51:20 And we just use Seattle was the data set because it's easy because it's always going to rain.

51:24 And then it's not actually most of the time it doesn't.

51:27 It's not fair.

51:28 So it runs that in Python.

51:30 And in the Python code, we use open telemetry to show the trace of it talking to the database.

51:36 And we we use Python logging and that log information on the trace information all gets put into the same trace.

51:42 So from dotnet from start to finish, you see the request come in on the front end.

51:47 You see the dotnet code executing.

51:49 You see the Python code.

51:50 And then if Python crashes or if the log entries go in, they're all in the same environment.

51:55 So in like an open telemetry dashboard, you see the whole trace across both dotnet and Python or in the same environment.

52:04 And you can just use logging as you would normally.

52:06 And dotnet aspire comes with a like a free open telemetry dashboard that you can run locally.

52:13 It's like a Docker container.

52:15 It's really nice to save you kind of spinning up Jaeger or something.

52:19 Yeah, this is really cool because you've got an ASP.net API.

52:23 But then when it's called, it uses one of these code gen CSnakes Python integrations.

52:28 But a lot of stuff just flows through like the Postgres connection string flows through the open telemetry context flows through the logging kind of goes tangentially off.

52:40 Then it talks to the database in Python and then it returns back.

52:43 And your example is just pure SQL connections and stuff.

52:47 But it's like a PG2.

52:48 But you could use pretty much any ORM or anything, right?

52:52 There's no real constraint there.

52:53 Yeah.

52:54 And it uses a few modules which are tricky like pandas and psycho PG is a anyone who's deployed that to production in containers or in different environments.

53:05 And those is it's hard because like it's like which binary do you use and like importing it is hard.

53:11 And, you know, you need libpq and like it's a bit of a nightmare.

53:16 So it's just demonstrate that they're like, this is this is a real thing you can do.

53:20 And it shows something which is practical in Python and something which is practical in .NET, which is like a proper app with tracing and logging and all the, you know, all those kind of features.

53:30 And on the Python side, we're using pandas, which, you know, and we're using numpy, which is the C extension.

53:36 And we're using psycho PG, which is like depends on a whole bunch of like third party libraries as well.

53:42 And stringing it all together and it just looks and feels seamless.

53:46 Like you see the demo working and you would never know that the, the weather prediction comes from Python or even that it's running in this way.

53:55 Like it just looks like and it's implemented like it's part of the same API.

53:59 It's call a function and one of the libraries and off it goes.

54:02 But yeah, it's all connected together like this.

54:04 All right. I'll let either.

54:05 We're about out of time, but I'll let either of you take this.

54:08 That was a web example.

54:10 What about something like WPF or Maui or any other desktop app possible?

54:17 Yeah. So because I'll talk about Maui because obviously that's the, the more modern UI framework.

54:24 Give a really, really quick definition to listeners who probably think of that as part of Hawaii.

54:29 It's kind of the, the next evolution of Xamarin, which people may be familiar with the way that you do cross-platform UIs.

54:38 I'm definitely not the expert on Maui, but the idea is that, you know, you can have something that is a bit more portable from a application experience,

54:46 if you want to target like a mobile application, like an Android or an iOS or run as a desktop application on Windows or something like that.

54:53 So just like a level of portability and some structure between UI concepts.

54:58 But fundamentally that's just running .NET behind the scenes.

55:01 So there's no reason why we couldn't also drop in a, a bit of a sea snakes in there and have that, yeah, work with some kind of backend service.

55:11 Like you could use this and this is something that Anthony and I have talked about trying to, trying to get a demo of doing like a local model running with,

55:18 like, so you, you pull a model off of hugging face or something like that, just runs it locally to do some kind of AI inference or whatever the case may be for the scenario you want to tackle.

55:29 But that way you don't have to be, you know, calling out to an external service.

55:33 Now, while sure running like a local model, it's doable with .NET.

55:38 It is just way easier to do that with Python.

55:41 And all the examples you would find are going to be written in Python first and foremost.

55:46 So trying to then go, okay, well, so this is doable in Python.

55:49 So what's the C# equivalent?

55:51 How do I find the equivalent .NET libraries on you yet?

55:54 And so on and so forth.

55:56 Well, you know, like with sea snakes, you could expose that model, create some surface areas in just Python APIs that is then going to be consumable by a .NET application.

56:08 So yeah, it's something you can definitely do not 100% sure whether you can do it on mobile.

56:13 I think that's probably stretching it a bit far just because of the way that we have to bootstrap Python with inside of it.

56:19 And you'd probably be limited from a mobile OS runtime to be able to do that kind of bootstrapping.

56:25 But at the very least, like a desktop application, or if you wanted to tackle with like a cross platform UI framework like Avalonia.

56:32 So Avalonia is a cross platform .NET UI framework that means you could build for Linux and Windows and macOS at the same time.

56:41 You could then run sea snakes under the hood of that to then have that desktop experience that is leveraging Python for where Python is going to be better suited to tackle a problem.

56:51 .NET for a different set of problems that's better suited for that just inside this application where you would have no idea as the end user that you're like that that's what's actually happening.

57:00 Very compelling. Very compelling.

57:02 Very compelling.

57:03 All right, Anthony, let's close this out with one final thing.

57:05 I think this is important and we haven't really touched on it yet.

57:07 You guys started the conversation by saying this is really important for data science, right?

57:11 Data science use cases.

57:12 And I made the joke about copying a million items to add one to the list.

57:15 But what about sharing data like NumPy across different, you know, the .NET side and the Python side and so on?

57:24 We have tackled that specifically.

57:26 So there's something called the buffer protocol.

57:28 It's a PEP from a few years ago.

57:31 It's a kind of an API for the internals of something like an ND array in NumPy.

57:39 And the buffer protocol with it, we support that.

57:44 So if your function returns a buffer, which is a type, then we can access the internals of that buffer directly without having to convert the data.

57:54 What this means in reality is that if your function returns a like a NumPy array, like an ND array and the internal values in the ND array are a specific type like a 32 bit integer, you can access them as 32 bit integers in .NET without having to marshal or copy data.

58:13 That's awesome.

58:14 You reach inside of the the ND array, just grab its actual memory and just say, got it. Let's go.

58:20 Got it. Let's go.

58:21 Yeah, that's exactly what we do.

58:22 And so that's a is in .NET.

58:24 It's called a read only span, which is the you can write the data as well.

58:28 So you can do it and you can actually just modify the values inside the NumPy array directly from .NET without having to copy backwards and forwards.

58:36 And we support pretty much every like low level types that NumPy supports.

58:42 And that means you don't have to do a .2 list on the NumPy array because like that is really expensive.

58:49 And also .NET works in a similar low level numeric types that NumPy does.

58:54 You know, it will work with 32 bit ints and 32 or 64 bit floats, for example.

59:00 So like if you're storing a 2D matrices in 32 bit floats in NumPy and then you want to send that to to .NET to I don't know, like pick out some specific values or like do some calculations in .NET, then you can just chuck that back as a buffer.

59:17 And there's information in the docs about how to do this and then you can just access the values directly out of memory.

59:23 So it's super fast.

59:25 And if you even want to like fill a buffer with values from .NET, then you can do that using the writable span.

59:32 That's pretty awesome.

59:33 So, yeah, that really is leaning into the data science benefits here as well, isn't it?

59:37 Nothing ever goes wrong if you're sharing memory across runtimes for different programming languages.

59:42 It's no, I don't see how this could go wrong.

59:44 Yeah, perfectly fine.

59:46 And that's why I pushed to make sure we supported read only span.

59:49 Yeah, right.

59:50 Okay.

59:51 No, I think it's great.

59:52 I think it's great.

59:53 And the UI stuff with Avalonia is something I'm going to look at.

59:57 That sounds really cool to play with.

59:58 So, all right.

59:59 Final thoughts, you guys.

01:00:00 People out there going, oh my gosh, I cannot believe we can finally do this with our .NET app and plug in some of the other things we're already doing in Python.

01:00:08 Should they?

01:00:09 Is it ready?

01:00:10 If somebody were to take this and build a thing, is that a good idea or should they wait?

01:00:13 Yep.

01:00:14 They should download it, give it a go.

01:00:16 The only thing we're possibly changing at the moment is the API based on feedback.

01:00:20 We're trying to keep that as stable as possible right now.

01:00:22 So we're only making changes that are backward compatible.

01:00:25 And yeah, we just want to see more use cases and people can get more feedback.

01:00:30 So yeah, please pull it, grab the samples, spin them up, build something, tell us what you made, and then give us any feedback.

01:00:36 Well, congrats.

01:00:37 It looks really, really nice.

01:00:38 Definitely, definitely looks promising.

01:00:40 And I think it's, it solves a lot of the problems that the other attempts had.

01:00:43 So nice work.

01:00:44 Thanks, Mike.

01:00:44 Thanks.

01:00:46 Yep.

01:00:47 Bye guys.

01:00:52 Be sure to check out what they're offering.

01:00:53 It really helps support the show.

01:00:55 This episode is sponsored by Posit Connect from the makers of Shiny.

01:00:59 Publish, share, and deploy all of your data projects that you're creating using Python.

01:01:04 Streamlit, Dash, Shiny, Bokeh, FastAPI, Flask, Quarto, Reports, Dashboards, and APIs.

01:01:10 Posit Connect supports all of them.

01:01:12 Try Posit Connect for free by going to talkpython.fm/posit, P-O-S-I-T.

01:01:18 And this episode is brought to you by Bluehost.

01:01:22 Do you need a website fast?

01:01:23 Get Bluehost.

01:01:24 Their AI builds your WordPress site in minutes and their built-in tools optimize your growth.

01:01:29 Don't wait.

01:01:30 Visit talkpython.fm/bluehost to get started.

01:01:34 Want to level up your Python?

01:01:36 We have one of the largest catalogs of Python video courses over at Talk Python.

01:01:40 Our content ranges from true beginners to deeply advanced topics like memory and async.

01:01:45 And best of all, there's not a subscription in sight.

01:01:48 Check it out for yourself at training.talkpython.fm.

01:01:51 Be sure to subscribe to the show, open your favorite podcast app, and search for Python.

01:01:55 We should be right at the top.

01:01:57 You can also find the iTunes feed at /itunes, the Google Play feed at /play,

01:02:02 and the direct RSS feed at /rss on talkpython.fm.

01:02:06 We're live streaming most of our recordings these days.

01:02:09 If you want to be part of the show and have your comments featured on the air,

01:02:12 be sure to subscribe to our YouTube channel at talkpython.fm/youtube.

01:02:17 This is your host, Michael Kennedy.

01:02:19 Thanks so much for listening.

01:02:20 I really appreciate it.

01:02:21 Now get out there and write some Python code.

01:02:23 Thank you.

01:02:24 We'll see you next time.