#172: Nuitka: A full Python compiler Transcript

00:00 Quick, name some ways to make your Python code faster.

00:02 Did you think of PyPy, the JIT compiled version of Python?

00:06 Maybe some async and await parallelism?

00:08 How about Cython, where you write in a Python-esque language that compiles to machine instructions?

00:13 Well, I'm here to add a new one to your vocabulary, Nutka.

00:17 Nutka is like Cython in that your Python code is compiled into true machine instructions rather than interpreted.

00:25 But, unlike Cython, you can take standard Python 3 that runs just in regular old Python, CPython,

00:31 without changing the syntax at all, and still compile that to machine instructions.

00:36 And Kai Haien is here to take us on the journey of Nutka, a project he's created and has been overseeing for some time.

00:42 This is Talk Python to Me, episode 172, recorded July 16, 2018.

00:48 Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.

01:08 This is your host, Michael Kennedy. Follow me on Twitter, where I'm @mkennedy.

01:13 Keep up with the show and listen to past episodes at talkpython.fm, and follow the show on Twitter via at Talk Python.

01:19 This episode is brought to you by Cox Automotive and Rollbar.

01:23 Please check out what they're offering during their segments. It really helps support the show.

01:27 Hey, everyone. Thanks for tuning in, like always.

01:30 I have two things I want to share with you real quick before we get to the interview.

01:33 First, I owe you an apology.

01:35 You don't realize it yet, but you will pretty soon.

01:38 When I recorded this, the microphone cable that connects my good microphone, the one I'm talking on now, to my computer, got bumped or something.

01:46 And then the computer just dropped the microphone and fell back to something terrible, like my iPhone headphones or something like that.

01:53 Anyway, the audio is not up to where I would like it.

01:57 It's not terrible. It's somewhat like a lot of my guests, I guess.

02:00 But it could be better, and I really work hard on that, as some of you know.

02:04 And this one, it's not up to my standards.

02:06 So hopefully you enjoy the conversation.

02:08 You can still hear it just fine.

02:09 It's just not as smooth and nice as this microphone.

02:13 So please bear with me.

02:14 It's still a great conversation and absolutely one worth listening to.

02:18 Secondly, I just released a new course, and I'm really excited about it.

02:22 This is one that's been in the works for a really long time.

02:25 It's called Building Data-Driven Web Apps with Pyramid and SQLAlchemy.

02:29 And we cover all sorts of cool stuff around building web apps, but also things like Alembic for migrations and unit testing web apps and deploying to Linux,

02:38 and a whole bunch of things that I think you'll find useful.

02:41 If that course sounds interesting to you, please check it out at talkpython.fm/pyramid.

02:46 Have a look through the course page, watch the course intro video, and see what it's all about.

02:51 Hopefully you love it.

02:52 I really am glad that I got it out there for everyone who's been waiting for it.

02:56 All right.

02:57 Now, with that stuff out of the way, it's time to chat with Kai.

03:00 Kai, welcome to Talk Python.

03:02 Yes, hello, Michael.

03:03 I'm glad to be here.

03:04 Yeah, it's good to have you here.

03:05 We've been trying for a while to set this up.

03:07 We've both been on and off traveling and things like that.

03:10 But it's great to have you here.

03:11 And I'm really looking forward to speaking to you about your project and your efforts to make Python faster.

03:18 Yeah, as you can imagine, I'm very excited.

03:21 Yeah, it looks like you're having some really great success.

03:25 So I'm excited to share that with everyone.

03:27 Let's start, though, with your story.

03:30 How did you get into programming in Python?

03:31 Yeah, I started as a kid.

03:33 I think that was a long time before the Internet.

03:37 So when I was 11, the first thing I ever saw was C64, Commodore 64, my brother, we brought it home.

03:43 And I had like one or two days of learning basic on it.

03:48 And then you went away with it.

03:50 And I didn't have a computer anymore.

03:51 So what I did, I was programming on paper.

03:54 Wow.

03:55 That's a dedication.

03:56 Programs in my mind.

03:57 Yeah.

03:57 That was bizarre.

03:59 It was hard.

04:00 But I suspect that that's actually really good training.

04:04 Because a lot of people, especially with...

04:06 The editors weren't that good, but paper is actually a lot more flexible.

04:13 Yeah, that's probably true, actually.

04:14 But, you know, just a lot of people, I feel like, kind of just poke at the program when they're new.

04:19 You know, they'll write something that doesn't work.

04:22 They're like, well, let me just change it and see if it works now.

04:23 I'll change it a different way.

04:25 Like, you don't actually, you know, so deeply, like, think through the work.

04:29 And if you do it on paper, like, you've got to think it through.

04:32 Yeah, and I didn't learn everything from my brother.

04:35 So I wanted to write something, I think, Hangman, something like this.

04:40 And I came up with that there should be something like a loop.

04:45 But I didn't know what it is.

04:46 And so on the schoolyard, I pestered all the kids who I knew were also programming.

04:51 There were, like, maybe three other guys that I knew of.

04:55 And I pestered them and asked them questions.

04:57 And I learned about the Goto statement.

04:59 These are, like, the early loops.

05:01 The early unstructured wild days of programming before functions and things like that.

05:06 It was, like, you were on an island.

05:08 So the cursor was blinking.

05:10 And you would figure out everything on your own.

05:15 And there were barely people to talk to.

05:16 I also, I grew up in a small village.

05:18 Went to school in a small city, very small city, 6,000 people maybe.

05:22 And, yeah, there was, I think in my village there was no other kid who had a computer that could

05:31 actually do some programming.

05:32 And I was, at least I wasn't aware.

05:36 So I was very isolated.

05:37 Yeah, I think that's one of the bigger differences now that you're connected, right?

05:43 It also made me self.

05:46 I don't know.

05:47 I was autodidactic all my life.

05:49 That's probably stemming from there.

05:53 Because later I learned, like, stack overflow programming.

05:56 You type what you want.

05:58 You copy, paste your code.

06:00 You're done.

06:00 But initially it wasn't like that.

06:03 So I also recall learning a sampler.

06:05 So coming from basic, it had these line numbers.

06:10 And I thought addresses were like line numbers.

06:13 And I wrote down my program.

06:15 And when I wanted to insert a statement, I just took a number between two addresses.

06:21 And, yeah, if you know a sampler, you know that corrupts the program.

06:26 I didn't get it for a long time, what this was all about.

06:30 So, yeah.

06:31 So learning a sampler.

06:32 But eventually I learned the 68K series of processors.

06:37 It was almost like a high-level language.

06:39 The C64, it had no registers.

06:43 You couldn't learn that.

06:45 That was too crazy.

06:46 I couldn't get anything done.

06:49 I think it couldn't multiply.

06:51 It was crazy.

06:53 But this one was actually pretty good.

06:56 And I learned a sampler in C.

06:58 Later I did C++ for some real programs.

07:02 Actually had a day job.

07:04 And met again with my brother, who was studying mathematics and computer science.

07:11 And he's pretty much older.

07:13 He's 13 years older.

07:14 He learned computers from these, what's it called, with the punch cards.

07:19 And he told me, learn Pearl.

07:23 If you know Pearl, you will always be the only guy who can do stuff.

07:28 And he was right.

07:30 So Pearl was really my absolutely first love.

07:34 I had learned a lot of languages until then.

07:37 But Pearl was, like, yeah, amazing.

07:41 And I read the book of Larry Wall, who was a linguist, designing a language.

07:46 And he wrote, or he designed the language in a way that it fits your mind.

07:50 So you would write what you think, however unstructured that is.

07:54 So unlike Python, where you are forced in a certain, yeah, outline of a code, you could write unless at the end of statements.

08:05 You could basically write down as you think, and you would never have to go back.

08:11 You could always just continue writing.

08:13 That was, like, a really productive way of programming.

08:17 Yeah, it's interesting.

08:18 I haven't done much with Pearl, but that sounds quite interesting.

08:21 Yeah, and Python was upcoming.

08:25 But it was very weak in terms of, this is 2002 we're talking.

08:31 It was very weak in terms of supported stuff.

08:35 So with Pearl, you had modules for everything, much like with Python today.

08:40 Nothing can compete with Python anymore.

08:42 But back then, CPAN was like, if you want to do stuff, it's already there.

08:47 So why bother with Python, which cannot even send an email?

08:52 You couldn't get stuff done.

08:54 But it was much more readable.

08:56 And, yeah, 16 years ago, when I switched jobs, we were working on a product which had a lot of share scripts.

09:05 And it was pretty ugly, and we wanted to improve on it.

09:10 And Pearl was, of course, an idea.

09:12 But a better idea was to use Python.

09:14 So I switched to Pearl from that.

09:17 I've since fallen a bit in love with Python, actually, a lot.

09:21 But it's not my first love.

09:23 It's never the same.

09:24 Yeah, yeah.

09:25 So Pearl was the first to enable me to do what people, I think, a lot of people perceive Python to be.

09:32 Right, yeah, that's really cool.

09:33 And for teamwork, it's absolutely much better.

09:36 Yeah, for sure.

09:37 And I think what's really interesting, you brought it up here, often people debate languages on the syntax.

09:44 You know, like, this syntax is better than that syntax.

09:48 And what most of the time rules the day is, yeah, but I have all these libraries that actually let me do stuff when I put the syntax in place.

09:57 Right?

09:58 And that's so important and often sort of forgotten in these language debates.

10:02 Yeah, this is, I think Java also has this rich ecosystem.

10:06 C++ sort of, but not as productive, I think.

10:11 Yeah.

10:11 Yeah, getting things done and being quick.

10:14 That's, for many tasks, it was, with Pearl, I've done things for my J job that were just, yeah, beginning.

10:22 Yeah, great.

10:23 Yeah, great.

10:23 It was like, and Python too.

10:27 And I think now everybody knows this.

10:28 Oh, everybody that's using Python, that's pretty much.

10:31 That is a lot of people today.

10:34 It's pretty amazing, the growth.

10:36 So, speaking of your day job, what are you doing these days?

10:38 Currently, I'm on parrots to leave.

10:40 So, I have a one and a half years kid now.

10:43 And there's this regulation in Germany which forces your employer to give you time off when you have kids.

10:52 And you get paid by the state even for 14 months total, both of us.

10:57 So, my wife took 12 of them.

10:58 I took two of them.

11:00 You get a reduced payment for that.

11:02 And on top of that, I added four months of unpaid parent to leave so I could focus on newt car.

11:11 That's what I currently do.

11:13 I have three times two months to spend on newt car.

11:16 And I'm at the start of a third chunk of that right now.

11:20 In reality, that parent to leave, I mean, you're a parent.

11:25 And, yeah, it's a lot of work to care for a small kid.

11:28 And especially with my wife, she's a doctor.

11:31 And I'm very emancipated because for roughly 10 days per month, she's not here for 24 hours.

11:40 So, actually, I get a fair share of her work there.

11:45 But when I will return, I will be what I've been for 16 years now.

11:51 I will be an engineer for air traffic management software.

11:53 Also did a bit of project management.

11:57 But mostly, I've been an engineer on this time.

12:00 That sounds like software that you have to be really careful with.

12:03 Yes.

12:03 Yes.

12:04 That's actually, we review everything.

12:07 And it's the requirements, the code, tests.

12:13 Everything gets looked with at least four eyes and goes through several phases.

12:20 And it's amazing.

12:22 And I was very lucky that the project that I'm working on, it's very old.

12:27 Now, is it 30 years now?

12:29 I think it's getting 30 years old or is already 30 years old.

12:33 And I was blessed with the opportunity to lead the project to replace the middleware because

12:38 it was closed source software.

12:40 And the idea was to be able to distribute it to parts of the world where we didn't or the

12:48 owner didn't have a license for.

12:49 And so I got to replace the middleware.

12:51 And I got to introduce Python in the safety critical software.

12:55 That was also an amazing experience and a huge success.

12:59 And it's also a reason why a lot of my day job now is Python.

13:03 So that's, I'm very proud of that because obviously that's not easy.

13:09 Java can't go there.

13:11 Yeah, that's cool.

13:12 Because of the virtual machine doing its cleanups whenever it wants to.

13:18 But for Python, we managed.

13:20 Wow, that's really interesting.

13:22 I definitely have heard that about the GC languages.

13:25 The unpredictability of the pauses is just too much sometimes.

13:28 Yeah.

13:29 Interesting.

13:30 Yeah.

13:30 Python is much more predictable with its garbage collection.

13:34 And budgeting is actually, can become a problem.

13:37 Yeah.

13:37 Threading is also something we're not allowed to do.

13:40 And so we are using multiple processes.

13:45 And that fits Python's weaknesses pretty well.

13:49 That's right.

13:50 So you have to do it the way that Python has to do it.

13:52 Perfect.

13:53 We'll use Python.

13:54 Yeah.

13:54 Because that's actually, threading is terrible.

13:58 I recall writing an editor.

14:01 And this editor was also a file system.

14:03 So I would type in my editor and launch the compiler.

14:08 There was no save.

14:09 It wasn't necessary.

14:10 I was just using the file system that was the editor and had a scripting language as well.

14:14 And these are all working on the same data.

14:18 And I learned an important lesson from there.

14:22 You don't do threading.

14:23 It's terrible.

14:24 We're locking.

14:25 And it's such a time killer.

14:27 And yeah, I'm not a fan of threading.

14:31 It sounds like it.

14:32 You know, sometimes, I mean, with modern processors, sometimes it's necessary.

14:37 But it's, yeah, it's maybe not the first thing to jump to.

14:40 Yeah.

14:40 Depending on your problem, you don't have a choice.

14:42 But when it's about safety, I think a process and a thread is not much of a difference, depending

14:49 on how you work with it.

14:50 It's roughly the same.

14:52 It's just more difficult.

14:54 Because with threads, you get access easier, but also careless.

15:00 Yeah, exactly.

15:00 This portion of Talk Python to Me is brought to you by Cox Automotive.

15:06 They're leading the way in cutting-edge, industry-changing technology that is transforming the way

15:11 the world buys, sells, and owns cars.

15:13 And they're looking for software engineers and technical leaders to help them do just that.

15:17 Do you hate being stuck in one tech stack?

15:19 Well, that's not a problem at Cox Automotive.

15:21 Their developers work across multiple tech stacks and platforms.

15:24 They give you the room you need to grow your career.

15:27 Bring your technical skills and coding know-how to Cox Automotive.

15:31 You'll create real-world solutions to today's business problems alongside some of the best

15:36 and brightest minds.

15:37 Are you ready to challenge today and transform tomorrow with Cox Automotive?

15:41 Go to talkpython.fm/cox, C-O-X.

15:45 And check out all the exciting positions they have open right now.

15:50 Speaking of performance, let's talk about your project, Nutka.

15:53 Yeah, so maybe let's just start with a really quick overview of what it is and then how you got started on it.

16:01 The first thing is it's a fully compatible Python compiler.

16:04 So it does everything and it's compatible.

16:08 So the idea is if you run a program with the standard CPython interpreter or you run it with Nutka, it does the same thing.

16:16 So that's basically only a clone of CPython that is capable of producing binaries just faster.

16:26 Yeah, that's really neat.

16:27 And one of the things that I thought was cool about it is you don't have a separate language, right?

16:32 It's not like, say, Cython where you write a slight variation of Python that then can do this, right?

16:37 It's like I could take something I wrote yesterday and just run it through you.

16:41 That's exactly my point is like you have no lock-in.

16:45 You can anytime something is behaving strange, you have these other implementations or the other Python implementations and you can switch to them.

16:55 And it's a drop-in replacement for as long as it works.

17:00 And when it doesn't, you can use something else.

17:03 That's the point.

17:03 And you can, and this is very important, you can drop a million lines of code on it and have it accelerated.

17:12 That's the important point because for small programs, which you fully write on your own, you can accelerate them manually.

17:19 But for libraries from third parties that you have no clue about, you cannot just live with any kind of limitations.

17:28 So a big part of the problem was to get everything working.

17:33 Yeah, to give people a sense.

17:34 Like during your EuroPython talk, you showed compiling Mercurial, which is, is it over a million lines of Python?

17:43 I don't know.

17:43 It's large though.

17:45 It's not a million line, I think, but it's a substantial project.

17:49 I think a million lines and all this.

17:51 Yeah.

17:51 Python is not that bad.

17:53 That's true.

17:54 It's true.

17:54 It should be less.

17:55 But it's a seriously large project.

17:57 It's not just like, oh, look, I can take this and I can write a calculator, a calculator app.

18:03 To make no changes, no changes to Mercurial.

18:06 And actually I was capable of doing this in 2012.

18:09 This was my first demo when Nutka was more like a feasibility study.

18:15 Back then I had, it was more like a templating from Python to C++.

18:21 It was just going to demonstrate that this is achievable.

18:25 And since then it has evolved into an actual real compiler under the hood, still capable of doing the same things and becoming more compatible.

18:35 So yes, people are using this in real life for real programs.

18:41 Obviously, some of the things are not working.

18:44 But I get a lot of positive feedback on a level of compatibility.

18:49 Yeah.

18:49 Yeah.

18:49 I was always thinking that for such a compiler to be acceptable, it needs to run foreign code and guaranteed to behave the same.

19:00 And otherwise, you just cannot trust it.

19:03 So how do you do, I'm probably getting things a little out of order here, but just a moment.

19:08 How do you ensure compatibility?

19:09 Like you run the tests that they run against CPython itself or things like that?

19:14 Mm-hmm.

19:14 I am taking the CPython test suite and, for example, also the Mercurial test suite and run it with compiled and uncompiled.

19:24 And it must pass or fail the same way.

19:26 Actually, the CPython test suite, for me, never really passes all the things.

19:32 And then I run the CPython 3.6 test suite with 3.5.

19:37 And then I get a lot of exceptions.

19:39 So I get extra coverage for error cases and these things.

19:44 So that's one way of doing it.

19:47 Obviously, then users report me back with incompatible behaviors that they still encountered.

19:54 Yeah.

19:54 Maybe something that wasn't covered in the test, right?

19:57 Yeah.

19:58 Sometimes that happens.

20:00 And often they describe the behavior of things in the test pretty well.

20:07 And sometimes I just add another test to see what it is.

20:11 But there's many times there are instances where, for example, running Mercurial tests would misbehave

20:18 and I would trace it down to an incompatible behavior of NewtGuard.

20:23 Yeah.

20:23 This has happened in the past.

20:25 Yeah.

20:25 It's cool that you have these large programs that are open source.

20:28 You can go and you just try them.

20:30 Actually, I would like to go further because right now running the Mercurial test suite is something I do.

20:35 And this is all a lot of CPU cycles.

20:38 And I want to integrate with TOX or these kind of projects and get people to just switch on running their own tests with NewtGuard.

20:46 And see if I pass as well.

20:49 That's something I need to work on to get out and get even more coverage.

20:54 Right.

20:54 If you could get all these other projects with all of their continuous integration automatically running their tests through your stuff,

21:01 that would be great.

21:02 Maybe not automatically, but they have to enable that, I suppose.

21:05 But they shouldn't just say much more.

21:08 If you want to run your tests with PyPy, I think it's as simple as saying do it.

21:13 I'll list PyPy in an enumeration and NewtGuard should also be there.

21:17 But that's just the plan.

21:18 Right now I rely on people doing things manually and reporting back.

21:25 And some people run their test suits with compiled stuff.

21:29 And I'm still pretty early with integration of pytests, running your tests, the compiled code, and then uncompiled tests with pytests.

21:40 That's working with some tricks, but it's not documented yet.

21:45 And it's not complete.

21:46 There's another field where I want to get more exposure.

21:50 There is Distoutils integration is something that people have started through pull requests on GitHub.

21:56 And that's pretty awesome because you don't even have to know how NewtGuard works.

22:03 You just install NewtGuard, enable a flag, and it outputs a wheel, which is a binary for a given platform instead of a no arch wheel.

22:16 And that's ridiculously easy to use then for existing projects.

22:21 Right now it's a manual rerun of what setuppy does that you have to specify on the command line.

22:29 And that's another field of integration.

22:33 So it should also be that simple to use.

22:36 That's also my goal.

22:37 Yeah, that's really great.

22:38 The more you can just make it just another command you give things and you don't have to really understand it.

22:44 That's super cool.

22:46 So maybe we can start looking at what you built by comparing it against other things people might know.

22:52 So people might know Cython.

22:54 Let me see if I can summarize it.

22:56 I think Cython is a lot more successful in terms of adoption because there are people willing to write this hybrid language code

23:06 and get the optimization benefits of manual work.

23:10 And it's useful a lot.

23:12 And I know Stefan Benel.

23:14 Actually, I worked with Cython for a while.

23:16 I was listed as a contributor.

23:19 But I asked them to remove it since initially I wanted to turn Cython into a compatible Python as well.

23:28 But this didn't work out.

23:32 I think it's a pretty good solution for what it does.

23:38 But I think it's bad for the ecosystem for all the reasons that we said.

23:43 The IDE integration, the lock-in, you cannot switch.

23:47 So Cython has a bug.

23:49 There's no way of comparing to anything else.

23:53 Right.

23:53 You're stuck with it.

23:54 You've got to just work around it somehow.

23:56 Yes.

23:57 So I have a lot of respect for Cython.

24:01 I have.

24:01 I'm an even bigger fan of PyPy, which I also contributed a patch to.

24:07 PyPy was our Python specifically.

24:10 We reduced Python for something which I tried out to accelerate actual code of mine long, long time ago.

24:17 And it's such an amazing project.

24:20 And obviously there's very little commonality.

24:24 It's a totally different approach.

24:27 And it's a thought.

24:30 So it's a very active project.

24:32 I think they've been a lot more successful at getting funding.

24:37 So I'm not getting that.

24:40 But I think they got money from the European Union initially.

24:43 And they also got fundraisers and PSF endorsement and all these things.

24:49 And very huge benefit is many of the benefits that Newtka also has.

24:56 It's a drop-in replacement.

24:57 You don't do anything.

24:59 It works when it works.

25:02 I mean, sometimes it doesn't.

25:04 It requires too much memory or can happen.

25:07 I'm not so much using it.

25:10 But I think it's a great alternative or maybe even, I don't know if Newtka does succeed.

25:18 I think it's a great thing.

25:19 But PyPy is here today.

25:22 That's really cool.

25:24 Yes, Cython to me feels a little bit like inline assembler used to be for C, C++.

25:32 It's like I've got most of my code the way I like it.

25:35 But this little part, we've got to make this a lot faster.

25:37 Let's rewrite it sort of here to be a little bit faster.

25:41 And so that's Cython.

25:43 And then PyPy is P-Y-P-Y.

25:46 The JIT compiled sort of CPython runtime.

25:52 Mostly compatible alternative to Python.

25:55 Which sometimes is faster.

25:57 Sometimes it's not.

25:59 Pretty interesting project.

26:00 I think they've made progress with the extension modules.

26:03 But this was a large holding back factor.

26:07 I think they might maybe do now PyQt.

26:11 I'm not so sure.

26:13 Yeah, I think that's right.

26:14 They did, yeah.

26:14 Yeah.

26:15 Yeah.

26:16 But with Newtka, I could do all the extension modules.

26:20 I think we can do some or all now.

26:22 I'm not so sure what's the current state there.

26:24 I don't follow it, actually.

26:26 I've had mixed results with it as well.

26:30 You know, like, I really think the project is awesome.

26:34 I'm glad it exists.

26:34 And I know in some places it makes things a lot faster.

26:37 But I switched out one of my larger websites to run on top of it.

26:41 And it did.

26:42 But it was like two and a half times slower.

26:45 And I thought, well, that's not helping me.

26:47 So I guess I'll just switch back to CPython.

26:50 And I think it was probably the C extensions and the data layer or something.

26:54 I'm not entirely sure.

26:55 But there was something compatibility.

26:57 Yeah.

26:57 If you're on Django or something, that's really something where I need to go is to get Newtka to work out of a box for Django projects, which have their own mess.

27:10 It's for manage.py is doing all sorts of nasty things that you cannot know about at compile time.

27:19 But I think the Django, accelerating Django, even by a bit, will be very welcome to many people.

27:26 And that's also something somebody should go eventually.

27:31 But I'm not focusing on that right now.

27:34 I'm focusing on other optimization.

27:35 So, I mean, for Newtka, I was like, I was observing the landscape for many years.

27:42 And all this time, I was thinking, a compiler for Python, that's totally possible.

27:50 And why is nobody doing it?

27:53 And that was just, I think, a couple of years that I was observing it and wondering.

28:00 And I think around the 2.5 timeframe, I thought, I'm going to make a prototype, make a proof of concept and show that it's possible.

28:10 And that turned out to be pretty much the case because a Python design is very welcoming to be compiled, actually.

28:18 I mean, it's a trivial, it's a non-trivial task.

28:21 It's very hard.

28:22 But you can do it for Python, if anything.

28:25 So, I have this compiled function type.

28:27 And there's the uncompiled function type in Python.

28:31 And all I have to do is for them to behave the same.

28:35 And then you're in.

28:36 So, you can have a compiled function.

28:38 That's basically a lot of how you get to work.

28:42 And then you don't have bytecode, but you have C code behind it.

28:45 Yeah.

28:46 And when the C code compiles to machine instructions, and then you're off to the races.

28:49 Yes.

28:51 And the Python engine really doesn't care all that much.

28:55 Because extension modules are first-class citizens.

28:59 Right.

28:59 So, let's talk about the architecture just a little bit.

29:02 So, the way it works is we take regular straight Python code.

29:06 We feed it to Nutka.

29:08 Nutka translates that into, like, a reduced Python.

29:13 The reduced Python is then translated into C.

29:16 The C is compiled, basically, to a C extension.

29:20 And that runs on top of CPython.

29:23 So, Nutka is not its own special runtime, like, say, PyPy is.

29:27 It's a thing that...

29:29 No.

29:29 No, no.

29:29 It's a thing that...

29:30 No.

29:30 No, no.

29:31 It's a thing that...

29:31 Do you have to install the runtime for it?

29:33 Or does it just create...

29:34 Like, do you have to install CPython, like, 3.5 in order to run it?

29:37 How...

29:38 Has it put together?

29:39 There's two modes.

29:39 There's a standalone mode, where you create something which will be self-contained, and then contains Python runtime.

29:45 And there's also this accelerated mode, and then it just links against a Python installation and loads libPython, like every program that embeds Python needs to do.

29:58 And that's how it goes.

30:02 And I try to avoid the C runtime, the Python runtime as much as possible.

30:07 And that's how I curlerate things.

30:09 So, not having bytecode, and then having knowledge or static optimizations, trying to avoid to go as much into the Python runtime as possible.

30:21 It's basically the acceleration.

30:24 Yeah, but you are still...

30:26 Like, that C code I talked about being generated in the architecture, that still uses, like, the Py object types and stuff like that, right?

30:35 It's as if you're basically...

30:36 Your compiler is writing a C extension from our Python code.

30:40 Yes.

30:40 That's true, except for...

30:42 Did you say our Python?

30:43 The PyPy...

30:45 It's not...

30:45 No, no.

30:45 Our Python.

30:46 Like, the Python that I write, I give to Nuka.

30:48 You write...

30:49 Yes.

30:50 Yeah, yeah, yeah.

30:50 Exactly.

30:50 Yeah, sorry.

30:51 Sounds the same, of course.

30:53 No, our Python would be reduced.

30:55 Yes, exactly.

30:56 It takes your Python and turns it into an extension module, and then a very hacky one.

31:02 It's really nasty.

31:04 I'm really big friend with the dictionary implementation.

31:08 I take advantage of all the internal knowledges that I'm not supposed to use.

31:14 Nice.

31:15 But that's what a compiler is supposed to do, right?

31:17 Yes.

31:19 It's responsibility to do all these nasty things, and yeah.

31:24 And if you order the Python version, the safer it is.

31:27 So 2.7 is not going to change a lot.

31:29 Yeah.

31:30 So we can...

31:31 You can be shorter on that, yeah.

31:36 This portion of Talk Python to Me has been brought to you by Rollbar.

31:39 One of the frustrating things about being a developer is dealing with errors.

31:43 Ugh.

31:43 Relying on users to report errors, digging through log files, trying to debug issues, or getting

31:49 millions of alerts just flooding your inbox and ruining your day.

31:52 With Rollbar's full-stack error monitoring, you get the context, insight, and control you

31:57 need to find and fix bugs faster.

31:59 Adding Rollbar to your Python app is as easy as pip install Rollbar.

32:03 You can start tracking production errors and deployments in eight minutes or less.

32:07 Are you considering self-hosting tools for security or compliance reasons?

32:11 Then you should really check out Rollbar's compliant SaaS option.

32:15 Get advanced security features and meet compliance without the hassle of self-hosting, including

32:20 HIPAA, ISO 27001, Privacy Shield, and more.

32:24 They'd love to give you a demo.

32:26 Give Rollbar a try today.

32:28 Go to talkpython.fm/rollbar and check them out.

32:31 This is Vitar Askin.

32:33 So let's talk about compatibility a little bit.

32:36 So it works on Windows, Linux, and macOS.

32:39 It works on not just that.

32:41 It's because it's creating C code.

32:45 I think it will work on anything that you can work with Python on.

32:50 That's the idea.

32:51 I obviously can only test Windows and Linux.

32:54 It works on Windows and MacOS is working because other people are using it.

32:58 And sometimes I fix something, but I don't have a macOS machine.

33:02 And yeah, you made an interesting point around that as well.

33:06 Like, well, I think iOS and Android probably are also candidates here.

33:10 Yes, Android.

33:11 I think people have done Android.

33:13 Obviously, it has been done.

33:17 And I think I have even done myself.

33:21 There's a cross-platform toolkit for...

33:25 I've done some Android compilations myself.

33:28 It's an ARM-based Linux.

33:30 It has a Python and you can...

33:32 Obviously, the compiling machine might be too slow.

33:35 That's the limitation there.

33:37 But in principle, there's nothing which prevents it from working.

33:41 And I think it has been done occasionally on and off.

33:44 The practicality is currently just not there.

33:48 But I think once the speed-ups are increasing, I would expect that people...

33:56 There's a lot of people currently doing games in Python.

33:58 I met a couple of those.

33:59 It's very interesting to have something like NewtGov for these use cases.

34:05 Right.

34:05 You mentioned the PyPy folks getting the 2D stuff accelerated and working faster.

34:11 And that was pretty cool.

34:12 So...

34:13 But your project is really right in there as well for making these things go faster.

34:18 Yes.

34:19 But it's not happening this year.

34:23 Yeah.

34:23 So lots of OSs.

34:25 Python 2 and Python 3 both?

34:27 Yes.

34:27 3.7 actually.

34:29 At the time you're releasing this, it will be working.

34:34 And 3.2, I just discontinued support for that, mostly because it's impossible to get it running.

34:41 And 3.3.2, 3.7, and 2.6, 2.7, they all work.

34:47 In my design, I have what I call these reformulations.

34:50 I think you can put up a link to these language conversions to make things simpler from my developer menu,

34:58 which is very instructive, and it explains a lot of things.

35:02 Actually, to me, this is a baby Python or this commonality Python internal that I'm using.

35:11 And the with statement, for instance, it's translated into many statements, actually.

35:19 Or a third statement is just if not condition, race, assertion error with my arguments, these kind of things.

35:28 And that means that most of the time, the Python versions do not make as much difference as you would think they do.

35:36 Right.

35:37 You've got some step that goes, oh, I look at a with statement, I'm going to translate that to something else, regardless of which level.

35:44 So there's a huge chunk of code dedicated to reformulating the with statement into a series of try-accept-finally stuff,

35:53 doing lookups of enter and exit, implementing the Python mechanics in explicit terms with temporary variables and so on,

36:04 which then can be optimized at compile time.

36:08 Maybe sometimes we know that it's not going to raise an exception and then we can drop this.

36:14 That's basically the reason why the language versions are not that much of a problem.

36:21 Yeah.

36:22 I suspect that the async stuff was more interesting.

36:25 Oh, this was terrible.

36:26 And especially imagine our coroutines and async jams.

36:30 Like, I've seen this being used.

36:33 I personally never knew why I would use it.

36:37 Well, so I don't use it.

36:40 And because it's like you get a lot of disadvantages of reading without the advantages.

36:45 I'm not so sure why it is good.

36:47 Well, I was just thinking of you, how difficult that must be to implement in this scenario and your comments in the beginning, how you don't really like them.

36:56 I had no idea what it is about.

36:59 It's like I'm trying to find out what it does and use it and the semantics and especially the coroutines, for example,

37:08 in a minor release, 3.5.2, they changed their mind how they work.

37:13 And they introduced a compatibility layer.

37:17 And then in 3.5.2, yeah, it was like a totally new implementation compatible with another one, which is a total mess.

37:26 And the async generators are taking it to the extreme.

37:29 And like, you know, when I started Nutka, there was a yield statement.

37:35 So a generator could yield the execution.

37:39 There was no return value.

37:40 And then this got added.

37:43 Actually, it was through Nutka that I learned that this changed at all.

37:48 So you can imagine that what happened there is it's amazing.

37:54 And they have a full team of people implementing this stuff and throwing out a new release.

37:59 And I get to replicate that.

38:01 That's basically what happens.

38:04 And that's one guy against the whole team.

38:07 And then you try to be faster with your implementation.

38:11 That's actually ridiculous.

38:14 But the last release wasn't so bad.

38:17 It's actually I didn't encounter anything crazy.

38:20 That's cool.

38:21 I'm pretty impressed that you got 3.7 going already because that just came out a couple weeks ago.

38:27 Yeah, yeah.

38:28 Actually, with 3.6, I was reporting bugs.

38:31 So I did with the 3.7.0 some bug report now about dictionaries.

38:35 I was using release candidates and reporting my findings.

38:41 So I have a couple of tests which demonstrate that Nutka is not misoptimizing certain things badly.

38:48 And I found out CPython was so Nutka wasn't, but CPython was.

38:53 And I could turn this into reports.

38:54 And 3.6 was already pretty good at the time it was released.

38:58 Therefore, but for 3.7, they made a huge change to exception handling.

39:03 Exception handling is also absolutely terrible.

39:06 I think in Python 2, it had pretty crazy semantics where a frame had an exception and restored it.

39:14 And it was terrible in Python 3, they changed it again.

39:18 And now in 3.7, they changed it yet again, but mostly on the implementation side.

39:26 So the structures internally, where you store it and so on.

39:30 This changed.

39:31 And this prevented me from, without a lot of changes, to compile with Nutka in 3.7.

39:36 So I'm a bit late, actually.

39:37 But I normally try to be quick with this because the early adopters will pester me about lack of support for the latest, greatest Python release.

39:50 Yeah, it sounds like you're pretty much on top of it.

39:52 That's really good.

39:53 So one thing that I was thinking about as I was looking at this is what basically can be the output of Nutka is it's like a C extension, right?

40:03 A compiled C extension.

40:04 That's one of them.

40:05 Yeah.

40:06 So if I have regular Python code that I just want to run, not compiled or anything, just standard, but there's part of my program that I would like to make faster and optimize with what you're doing, I can put those two things together, right?

40:21 Yes, you can.

40:21 That's absolutely true.

40:23 And actually, one of the things I'm aiming at is making bindings an easy task.

40:31 So my vision of bindings is for you to use C types or something.

40:36 For the middleware, we use C types to make bindings.

40:39 So it's implemented in ADA, but you can think of C code.

40:43 And that's a pretty neat way of doing the bindings.

40:46 And my idea of this for Nutka to optimize that way and not use C types, but to directly C codes and do bindings like this in a comparable fashion.

40:59 And what I also want to achieve is that Nutka makes available the original code.

41:05 So if I use that binding extension then and try to compile a program, I would like for the compilation process to be able to go and inline code from there.

41:19 So I'm not so sure if I was clear or not.

41:22 But the idea was right now there's a barrier.

41:26 So if it's an extension module, it's an extension module.

41:30 I can't look into PyQt at all if it's created with zip.

41:35 And I want to replace this with the ability to make a call into a bound C function in my main program.

41:45 And normally I would use this compiled extension module, compiled with Nutka.

41:50 And if I compile the program, it would be able to inline that extension module's call.

41:55 Right.

41:56 So not just optimizing the Python code you feed it, but actually optimizing the underlying C code that's being used.

42:01 Because ultimately it gets down to like C on C.

42:04 Yeah, generate C code for the compile program again.

42:08 And I think the ecosystem would benefit from having these dual mode codes.

42:15 Like you have this original Python binding and you have the compiled extension.

42:21 You can use the compiled extensions where you want.

42:24 And you can also use the original code if you compile.

42:27 So that's my vision.

42:29 And I think that would be healthy for the ecosystem.

42:33 And that's something that Cython doesn't do.

42:36 And then it would, I think, be much more attractive to do bindings in pure Python.

42:42 Yeah, I think that sounds really great.

42:44 That's awesome.

42:45 That's my vision.

42:47 And I think I will be getting there.

42:49 Yeah, I think you will as well.

42:51 You definitely got some impressive stuff already working.

42:53 Let's talk about some of the optimizations.

42:55 It sounds to me like one of the most important things that you could do in Python is to inline functions.

43:01 Because functions in Python are almost, I want to say, unreasonably slow.

43:05 They're like quite, it's actually quite a high penalty to call a function in Python.

43:12 Just that mechanics of setting up that stack and call and whatever compared to, say, other languages where it's slow.

43:19 It has an effect, but it's not like, you know, super significant.

43:23 Yeah, it did a lot of hacks to accelerate functional method codes, actually.

43:27 And a lot of the acceleration in Utica is coming from being willing to do all kinds of nasty tricks to call functions in a faster way.

43:39 That's a huge overhead and this inlining that will, of course, make all the difference.

43:45 But it requires whole program optimization.

43:48 That's a rather difficult task.

43:51 So as you know, with Python code, it can change everything behind my back without me noticing.

43:56 Right.

43:57 I will always have to do both the things.

44:01 So I will have to check that it's actually what I'm thinking it is.

44:06 Much like the PyPy JIT is having guards, like, is it really the same as last time?

44:12 And is it really what I expect it to be?

44:16 And then I can be super fast or I need to fall back because something crazy happened behind my back.

44:22 Right.

44:22 Like a unit test patch, for example.

44:25 Yes.

44:26 That kind of thing will have to work.

44:28 And then, but that's this duality.

44:31 It's my vision of how things will then eventually work.

44:36 But inlining code will definitely do a lot of things.

44:40 I'm currently working on classes and optimizing the classes and the dictionaries and tracing the values in there.

44:47 And NUTCA and I hope the next release will be capable of statically optimizing most classes into a simple call to type with a static dictionary.

45:01 Of stuff.

45:02 Of stuff.

45:02 That's the first step, obviously, to understanding types fully, because I will be able to precisely know the dictionaries.

45:11 Then I have to see through the meta class mechanics and see that they are actually mostly harmless most of the times.

45:19 And from there, I take it from there.

45:22 This is a, I think it's a huge undertaking.

45:25 It's one of the directions that I want to go.

45:28 So these global program optimizations are one thing.

45:31 And then locally, for cases where I do know that something is an integer, I want to get to using alternative integers and Python integers or just the C type where possible.

45:45 That's the direction I intend to work on the next two months.

45:49 Basically, that is the magic sauce that makes Cython fast is that you explicitly type your stuff and it lets, you know, it just be four bytes on the stack in C rather than all the indirect stuff.

46:03 Like, well, it could be this reference type.

46:06 Like, we don't really know what it is.

46:07 You know, treat it as a full on pie object.

46:10 Yes.

46:10 You're working on this as well.

46:11 Yes.

46:11 That's crazy.

46:12 And this is where I think C level performance is possible.

46:16 And then I have the idea that unlike Cython, I want to make type annotations to that are actually behaving in Python at runtime too.

46:28 So I would like you to declare that the function takes integer arguments and will raise an exception if it isn't integers.

46:37 And then I want to have Newt Cassie through that, take these assertions and generate specialized code from there, much like Cython does, but in a lot more complex way and with the benefit of actually enforcing these things at runtime.

46:54 That's the vision there, which is also a lot of work.

46:59 You would have to write some decorators, but I think other people can do this.

47:04 Just today I asked somebody who offered to volunteer if he could do it.

47:08 That would be great if people joined such efforts because that's all only about CPython, right?

47:15 You can write this without any knowledge of Newt Cassie.

47:19 It's just, I want to have a decorator which says, I have to clear this class and now it's frozen.

47:25 You will not be able to change anything after the fact about this or else you get an exception.

47:30 And that's a hint to the compiler to take advantage of that for static optimization.

47:37 Yeah.

47:37 A little like slots, right?

47:39 Yeah, exactly.

47:40 Like slots.

47:41 But in a more general fashion and enforcing these things.

47:47 And these kinds of decorators, I think, make all sense.

47:52 And yeah, this is another avenue where I think Newtcar will also be beneficial to the ecosystem of making it viable.

48:02 Because what it will do is you slow down your program terribly because it will do all these checks that typically are not useful at runtime if you do with the Python decorators.

48:13 But then when you accelerate some, it's actually better.

48:18 And that's the idea.

48:20 Well, that'll let you claim a much higher performance boost.

48:22 Look, I ran it on her Newtcar.

48:23 It was 20 times faster.

48:25 Just kidding, because it would be slower, right?

48:27 Yeah.

48:28 So I don't know if you're getting this, but Newtcar is a crazy compromise-free project.

48:33 And if it fails, which it probably will, that's why.

48:38 It's all about trying to do the best thing possible.

48:42 Yeah, the compromise-free part is what makes it so challenging, right?

48:46 Yes, it's so challenging, but it's also, yeah, it's rewarding.

48:50 And everything I've achieved challenge-free from when in 2003 to I made these public releases of Newtcar, it was usable.

48:59 It was usable from day one.

49:01 And it has remained usable all this time.

49:05 And it only became ever usable, more usable.

49:09 What it didn't become in, I think, like at least two years, it didn't become faster.

49:14 Because I'm doing these big transformations of internals where, for example, the local dictionary work that I'm doing on classes, class declarations aren't performance critical at all.

49:27 You run that code once.

49:29 It's not in a loop.

49:30 But for scalability, it's hugely important to get this, because if you compile this million line of code and you optimize it better, it's much less code.

49:43 And for these plans, we really need to understand the global picture there.

49:51 So I'm spending a lot of time on stuff that doesn't actually improve performance at all most of the time.

49:58 And I really look forward to finally having fun running benchmarks again, because I really don't enjoy running benchmarks if the numbers are all stacked.

50:09 Yes.

50:10 It's sad.

50:12 It's crazy.

50:12 And it's totally anti what everybody else has done in the field so far.

50:18 So there has been compilers from Google.

50:21 I forgot the name now.

50:23 But they had their own project.

50:25 And the first thing they had is incredibly well-numbered benchmarks.

50:31 And, yeah, but it wasn't doing what it should be doing.

50:35 I think the unladen swallow.

50:38 Yeah, yeah, I think you're right.

50:40 Unladen swallow, I think.

50:41 It just went away.

50:42 And it's not sustainable because you can't use it for anything but benchmarks.

50:47 Right.

50:47 That's basically the idea.

50:50 And I'm doing something which can be used outside of benchmarks.

50:52 But there will always be code which is currently slower and new to curve because I didn't look at it.

50:58 Or actually CPython has a lot of tricks.

51:01 I recall that it took me a long time until I was at the point to emulate their in-place assignment tricks.

51:09 So if you do an in-place assignment to a string, they did some things to avoid allocating a new object unless a reference was held on the outside.

51:20 Which in many instances makes it incredibly, like infinitely faster than Newtco was.

51:27 And so I'm behind in some optimizations even.

51:31 And that's pretty tough.

51:35 And obviously I would like to have also cool benchmarks.

51:39 But what I would like to have is also the ability to say, and you can have it too in your program.

51:46 That's obviously, I don't know.

51:48 And forever it has felt like it's right around the corner that I can do this now.

51:54 And then I discover something else that I need to do first.

51:58 Yeah, that's why I brought up Mercurial.

52:00 Because that's a pretty serious application to process.

52:05 Yeah, and just carrying the full weight.

52:07 So doing all these refactorings.

52:10 And I'm always doing these refactorings in a non-breaking way.

52:13 So I try to not strive too far away from a working state in all the changes that I made.

52:22 So I made gradual changes.

52:23 Like I said, initially it was more like a templating language.

52:28 And I turned it into a single state assignment form compiler gradually.

52:35 One by one, bit by bit.

52:37 And trusted more here and there.

52:41 Right now this value tracing is very reliable.

52:45 But still not used for everything.

52:49 I think right now the major fact that I'm using it is do I need to check if the value is assigned or not.

52:57 And that's mostly what it's being used for.

52:59 And for static optimization.

53:01 And that's working bug free.

53:04 But I'm always expanding this.

53:07 And then I find something which is not yet working.

53:11 That's the crux of it.

53:13 Yeah.

53:13 Well, it sounds like it's getting better and better.

53:15 And there's a lot of things that you could unlock with it.

53:19 So speaking of stuff that's faster, stuff that's slower.

53:23 When should I use this?

53:26 And maybe when shouldn't I use it?

53:28 Oh, you shouldn't use it if you don't care about performance.

53:31 Yeah.

53:32 Compatibility is just like mostly what you care about.

53:35 Yeah.

53:35 Actually, I have this secret pipe dream of being better than PyLint.

53:41 But that's not.

53:42 That's a whole other issue.

53:44 Because my static analysis may uncover things that PyLint wouldn't.

53:49 The program analysis.

53:51 And if you don't care about performance.

53:54 And I think that's roughly 99% of all the people.

53:57 And you should use it if you are into NumPy and these things.

54:04 Although I wasn't having discussions with a university.

54:08 It would take somebody to work on specializing code generation for NumPy.

54:14 So NumPy is a hugely great scientific library where you would just provide very little Python code on your own.

54:22 And for you to do something for each cell, for example, that would be bytecode.

54:27 And it would be massively cool if we could have this little bit compiled.

54:33 Yeah.

54:34 And still run it.

54:36 And I think NumPy would be then practically not entering the Python runtime a lot.

54:45 And that would be something.

54:47 But I didn't quite manage to get some university or something involved.

54:52 So scientific applications and financial applications are also like where you want to do financial stocks, something.

55:02 You want to buy as fast as possible, make your decisions as fast as possible, real-time applications in Python.

55:09 All the algorithmic trading stuff, yeah.

55:10 Some people are crazy enough to do this in Python because of the development turnaround.

55:14 And then they want to throw something easy on it to get it accelerated.

55:20 And on another day, they want to change the algorithm simply.

55:25 And that's the kind of uses.

55:28 I think to most people that do not care that much about acceleration, some people care about deployment.

55:36 And there's a standalone mode, especially on Windows.

55:40 where you do not normally have any Python on an arbitrary user machine and definitely not the version you want to have and definitely not the modules.

55:50 You can run a newtker and use it to create this distribution folder and send that to people's way and have things work.

56:01 That's, I think, what a lot of people use it for.

56:04 I think that it's really excellent.

56:05 And so basically you compile it down to an exe that does not depend on CPython.

56:11 It bundles it up to contain that within itself, right?

56:14 Not on an installation.

56:15 Yeah.

56:17 Yes.

56:17 And yeah, although you could use anaconda python for that too.

56:22 It's, I think, a convenient package.

56:26 I think that also a lot of people are using it to hide their source code from people.

56:32 And that's actually something which I'm, well, that's what compilers allow for.

56:37 It's not my, it's not my daily joy of being part of depriving people of the ability to change source code.

56:46 But yeah, that's people's people own their source code.

56:50 So we can do, they can give it to you or they can't in favor of that.

56:55 But yeah, that's one other mode of operation, which I think is unfortunately the majority of users, I guess.

57:04 And it's also causing a lot of trouble because extension modules often do imports of stuff which they don't tell me about.

57:10 And then they crash when it's not present.

57:13 And it's sometimes very nice.

57:15 Yeah, I can imagine.

57:16 It's tricky.

57:17 So if I'm using external packages off of PyPI, that's no problem, right?

57:23 You can deal with that.

57:24 So if I work with requests, SQLAlchemy, whatever, I can run that through Nuka?

57:28 That's awesome.

57:29 Yes, yes.

57:30 Actually, yeah.

57:32 It's the out-of-the-box experience.

57:34 You just arbitrary code.

57:36 It's supposed to work.

57:38 I think Django is, if it's some of the really dynamic, crazy stuff, would need a plugin to be created to pass it.

57:47 Because I think Django does some dynamic imports of stuff.

57:51 Some people are getting it to work.

57:54 I don't know.

57:55 But that takes some modifications.

57:58 Yeah, sure, sure.

57:59 Interesting.

58:00 So this, yeah, the standalone mode is pretty interesting as well.

58:03 So I think we're getting kind of near the end of our time.

58:07 So one thing I did want to highlight, I think, it sounds like you're looking for contributors.

58:13 And there's a lot of places that people can contribute.

58:16 You talked about writing some standard Python code that then Nuka could use.

58:21 I'm sure if there was a long list of these are all the optimizations we're looking for.

58:25 Could anybody who's good at this kind of stuff and see?

58:28 Could you help here?

58:29 Things like that.

58:31 Are you looking for contributors along those lines?

58:33 Yes, of course I am.

58:34 I actually have to think if I did, I should have a list of issues on GitHub that are marked as help wanted.

58:46 And there's a lot of these things.

58:50 And obviously I could and should create a couple more from these ideas that I just told you about.

58:59 And that's totally something I should do.

59:02 I have relatively bad experience with people joining the project because it's tough to get progress.

59:10 It's also a complex design, not easy to get into.

59:16 It takes a lot of skill and privacy.

59:18 And occasionally I get contributors.

59:23 But I think it's not rewarding enough, I think, for many people, unfortunately.

59:30 Yeah, it takes a lot of effort to see the rewards through, right?

59:32 I'm just thinking that there must be many people who are working on compilers and stuff who just left university and are looking for some kind of project.

59:41 You know, maybe they could jump in, right?

59:44 Yes, I welcome everybody.

59:45 But I have to be honest with you.

59:47 I also don't have time to actually mentor much.

59:52 I can't guide people through doing things and then they lose interest.

59:59 That's happened a couple of times.

01:00:01 So there's a lot of people who want to find out if this is something they would be willing to do.

01:00:07 And yeah, I have a family, I have a day job, and this is my spare time.

01:00:12 Right, absolutely.

01:00:13 It needs a lot of work.

01:00:15 Communication often takes second place after coding myself.

01:00:21 Yeah, that's the challenge of doing this as a part-time project, right?

01:00:25 Yeah, it's totally impossible to keep up with.

01:00:28 So suppose somebody really wants to get into this and yeah, I cannot respond quick enough sometimes.

01:00:37 And that's already frustrating.

01:00:40 Sure.

01:00:41 I guess people could probably start by watching your EuroPython presentations and then go from there.

01:00:46 Yeah, yeah.

01:00:47 Yeah, but also I think what I really would need is somebody with community skills who would be capable of doing a fundraiser for me.

01:00:56 It's a long-term project and a lot of people have short-term needs.

01:01:02 I need to get Newtka to a point where people with short-term needs will feel that investing their time into Newtka is worthwhile.

01:01:13 And then it will be a different story.

01:01:15 Yeah, but coming back to what I would like to have.

01:01:18 I would like to have somebody who organizes some fundraiser and does this so I could get time off my day job.

01:01:28 I work on Newtka and I've invested my own money right now.

01:01:32 Four months unpaid is a lot of effort.

01:01:36 I'm buying hardware and so on.

01:01:38 I'm getting some donations, but it's not enough to buy the hardware that I buy for Newtka.

01:01:43 And that's also something people could do is to send donations my way.

01:01:51 But I think a concerted effort that would gain visibility and with a raising goal and me being able to put a couple of months again into Newtka and pushing really forward, that would be great.

01:02:05 Probably that's something more than I need than actual contributors.

01:02:10 Because I think I need to reach a breaking point where I have something that is attractive enough for people who have actual problems to just add the bits that they are missing.

01:02:22 Like at the point where they have a lot of optimization and then they run into some construct which is not properly optimized and then they can hack it.

01:02:32 And then it is.

01:02:33 That's a thing for them then.

01:02:34 Right now, that's not the case.

01:02:36 Right now, you would have to be somebody with a long term vision and no concrete other problems that require your attention.

01:02:44 So that's it.

01:02:46 I think the couple of months sprint really would make a big difference.

01:02:50 I've seen that make a difference in other projects more than donations or other things as well.

01:02:56 So certainly that sounds like a good thing.

01:02:57 All right.

01:02:58 Kai, before we get out of here, let me ask you the final two questions.

01:03:01 If you're going to write some Python code, what editor do you use?

01:03:05 I'm using Eclipse and PyDef specifically.

01:03:08 I occasionally use VI on the command line just for quickness, but Eclipse and PyDef it is.

01:03:15 Right on.

01:03:16 And then just some notable PyPI packages.

01:03:19 Maybe people haven't heard of them or aren't using them yet.

01:03:23 Obviously, I have a lot of machines running and I'm using Ansible, which somebody introduced me on a Debian conference.

01:03:31 I'm forever grateful for the tutorial I got there.

01:03:34 It's very useful tools like salt, if you know that, Ansible written in Python.

01:03:41 And you just can easily deploy all your machines in a similar fashion.

01:03:47 And that's great.

01:03:49 Nicola is something I'm actually, I forgot my contributor number.

01:03:53 I might be contributor number five or something or static website generation.

01:03:57 That's what people are using a lot these days.

01:04:01 And I really liked the idea and I joined that and I'm using it for my website.

01:04:06 And it's great.

01:04:08 And obviously, BuildBot is also a tool I heavily rely on.

01:04:12 I absolutely despise Jenkins, which I have to use at work.

01:04:16 The BuildBot is great because you get to write your configuration as Python code.

01:04:22 Actually, there's no web interface needed to configure stuff.

01:04:26 You just do your for loops and stuff and everything is consistent, which is great.

01:04:30 And I think something which should get more exposure is pip-env, which combines the virtues of virtual environments and pip install, making it easy to run stuff.

01:04:44 I'm not using that myself a lot, but I think it's very notable.

01:04:49 Yeah, absolutely.

01:04:50 Absolutely.

01:04:51 All right, Kai, thank you for being on the show.

01:04:53 This was a really interesting project that you've been working on a long time.

01:04:57 And it's definitely a cool exploration of this compiled space.

01:05:02 And it's its own take.

01:05:03 And I would like to thank you for hosting me and giving me your opportunity.

01:05:08 And I hope it wasn't rambling too much.

01:05:10 It was some interesting stories.

01:05:12 Thank you.

01:05:12 And yeah, take care.

01:05:14 Okay, bye-bye.

01:05:15 This has been another episode of Talk Python to Me.

01:05:19 Our guest for this episode is Kai Haian, and it's been brought to you by Cox Automotive and Rollbar.

01:05:24 Join Cox Automotive and use your technical skills to transform the way the world buys, sells, and owns cars.

01:05:31 Find an exciting technical position that's right for you at talkpython.fm/cox, C-O-X.

01:05:37 Rollbar takes the pain out of errors.

01:05:40 They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until your users complain, of course.

01:05:48 As Talk Python to Me listeners, track a ridiculous number of errors for free at rollbar.com slash Talk Python to Me.

01:05:56 Want to level up your Python?

01:05:57 If you're just getting started, try my Python jumpstart by building 10 apps or our brand new 100 days of code in Python.

01:06:04 And if you're interested in more than one course, be sure to check out the Everything Bundle.

01:06:08 It's like a subscription that never expires.

01:06:10 Be sure to subscribe to the show.

01:06:12 Open your favorite podcatcher and search for Python.

01:06:14 We should be right at the top.

01:06:16 You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm.

01:06:25 This is your host, Michael Kennedy.

01:06:27 Thanks so much for listening.

01:06:28 I really appreciate it.

01:06:29 Now get out there and write some Python code.

01:06:31 I really appreciate it.