#235: Python in your Browser with Skulpt Transcript

00:00 Michael Kennedy: Do you dream of a day when you can write Python in the browser rather than using JavaScript? This is no pipe dream. There are several ways to write Python that run in the browser already. One of the leaders here is Sculpt. This is not just an experiment. Real, powerful web applications with rich client-side code, Python code, are out in the wild and built with Sculpt today. We dig into it Meredydd Luff and Albert-Jan Nijburg on this episode. This is Talk Python to Me, Episode 235, recorded September 26th, 2019. Welcome to talk Python to me, a weekly podcast on Python. The language, the libraries, the ecosystem and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via @talkpython. This episode is brought to you by Linode and Tidelift. Please check out what they're offering during their segments. It really helps support the show. Welcome to Talk Python to Me.

01:08 Panelists: Hello, great to be here. Hi, thank you.

01:10 Michael Kennedy: Yeah, it's great to have you both and Meredydd, welcome back. We talked about your cool project, Anvil, about two years ago, wasn't it, on the show?

01:18 Panelists: Gosh, yeah, November 2017, feels like a lifetime ago.

01:21 Michael Kennedy: Yeah, in Anvil years.

01:22 Panelists: Oh yes, well start-up time operates, it's like dog years only more intense.

01:25 Michael Kennedy: Yeah, yeah, absolutely. And nice to have you here, Albert. So I kind of wanted to start the conversation here. We're going to talk about Python in a browser with Skulpt. I want to start the conversation with just each of you talking about your involvement in the project, I know you both have done various things to keep it rolling and work on it day to day, so tell us about it. Albert. you want to go first?

01:48 Panelists: I started using Skulpt because of a very bad decision I made a few years ago. Eight years ago I think. We had the idea that we wanted to write client validation and server-side validation in the same language so we'd only have to write them once and our backend at that point was in C#, and the front end wasn't JavaScript and the only thing we could run on both ends was Python and so we ended up with Skulpt.

02:16 Michael Kennedy: That's a funny path, right, I wouldn't think if you combined JavaScript and C# that you end up with Python.

02:22 Panelists: Yeah, I wouldn't advise anybody to do this. It had put me in this position where we found Skulpt in this slightly disheveled state because the original author had sort of left it. And we ran into problems and started fixing them and yikes, it slowly grows.

02:39 Michael Kennedy: Yeah, who was the original author, was that Brad Miller?

02:42 Panelists: No, Scott, forgotten his name.

02:46 Michael Kennedy: Yeah, so Skulpt is not totally new. I was looking through some of the docs and stuff. I certainly saw 2011 here and there and that actually would put the timeframe almost exactly the timeframe that you're talking about Albert.

02:57 Panelists: Yeah, the original author started off, he was really frustrated with JavaScript and he liked Python a lot so he's like, "How hard can it be?"

03:05 Michael Kennedy: Famous last words.

03:07 Panelists: Yeah no, but he's like amazing software engineer because he got such a long way, he got the compiler and the parser and everything. And he got a sizable chunk working, but then abandoned it, there was other stuff to do probably, and we found it in that state and we just took it from there. Ask your next question because I'm going to come in at the point in the story where I actually come in.

03:29 Michael Kennedy: Yeah, I was going to say, so Albert, what's your relationship to the project today? I know that you're doing a lot of overseeing issues and merging PRs and things like that. Are you leading the project or are you working with someone else who is?

03:41 Panelists: The official leader is Brad Miller, who took the ownership over, but two or three in the line who have contributed the most and I feel this thing where if people do issues or pull requests I always feel obliged to act on them so I do. There's no official split of task here.

04:03 Michael Kennedy: Sure, sure, Meredydd, tell us about your involvement.

04:06 Panelists: So I came to Skulpt a little bit later. It would've been, ooh, I guess some time around 2015. So I came to Skulpt via Anvil, which if you haven't heard of it, it's a tool for building full stack web apps with nothing by Python, so Python in the browser, Python in the server, function calls from one to the other, it's all very nice, but obviously to achieve that vision you need to run Python in the web browser as well, so we launched out on a little safari of Python options for running Python in the web browser and I was actually already familiar with Skulpt from some educational work we had done as part of an outreach workshop at the University of Cambridge where my co-founder Ian and I did our PhDs. So it was the closest tool to hand. Actually when we looked around we realized it was in fact the best tool for building a new way to drive web environments with Python because as we'll probably discuss later a lot of the other approaches, they think very JavaScript-y and Skulpt thinks very Python-y and then makes whoever's embedding it in the web page do the work, which as far as I'm concerned, the right way round.

05:14 Michael Kennedy: Yeah, if you want to bring Python to the browser right, it should be as Pythonic as possible and then you make it work, right?

05:20 Panelists: That's the idea, yes. That's certainly Skulpt's idea, and certainly Anvil's idea. So I jumped in then, I sort of announced my arrival with a major pull request turning Skulpt's fundamentally JavaScript-like model, which is to say that it had to execute all at once and then be finished into this asynchronous model where you could write blocking Python code and then it could hand control back to the JavaScript runtime for a little bit, and then when some asynchronous operation had completed like going to the server for something or getting some user entry, it could resume your Python code exactly where it was and from there I've become like Albert, one of the more active contributors on the project and as Albert said, there isn't really division of tasks, we all just jump in wherever it is that we see efforts in need of things, so I do a lot of work on the compiler, I do clean up of bugs that people find in Skulpt, typically while using Anvil in advanced ways and of course...

06:29 Michael Kennedy: You're like, "Wait, that should work in Python, right, but it's not."

06:32 Panelists: Yes, absolutely, like a non-core Python implementation there are going to be diversion bugs where CPython does one thing and we do the other and we need to run in and fix those. And of course most recently Albert and I did the big upgrade of Skulpt from the Python 2.7 grammar to the Python 3.7 grammar, which opens the door to us moving to an actual Python 3 world in Skulpt, getting in under the wire for the end of life for Python 2.

06:59 Michael Kennedy: Yeah, beautiful, heck, you've got months to spare, at least one.

07:04 Panelists: Days, whole days, Michael.

07:06 Michael Kennedy: Whole days, it's so much time.

07:08 Panelists: It feels good though.

07:09 Michael Kennedy: Yeah, it's got to feel good because yeah, it was 2.7 or even 2.6 for quite a while so it's great to see making progress, that's awesome.

07:16 Panelists: Updating the grammar to 2.7 was one of my first big projects in Python and that was only like one step and that took me a summer of coding.

07:26 Michael Kennedy: Just looking at the way that Skulpt creates the JavaScript or the JavaScript stuff is implemented for even something like a print('hello world'), there's like a lot of stuff going on in JavaScript there that it's got to set up the environment, so yeah, I can imagine making those changes took some work.

07:44 Panelists: That question cuts to the core of one of the biggest design questions if for every Python in the browser implementation, which is essentially are you trying to do the minimum possible to allow you to write some Python in a JavaScript world and Transcrypt for example is an example of a project that really does take that approach. It translates Python to JavaScript code that looks very much like the Python code and the price it pays for that is that it's somewhat loose on the Python semantics and the way you interact with the page is certainly not Pythonic at all if you want to do any I/O and Skulpt is right they way at the other end of the spectrum where you say I don't care how much work the JavaScript has to do round here, I am going to deliver a Python-y experience to the user and allow them to be interactive and allow them to think about the outside world as a Python programmer would, as something you can print to as opposed to something where you get some document object and then get some HTML element and then you do some translate DOM manipulation on it and jump out a window.

08:56 Michael Kennedy: Exactly, exactly, you're like, "What, I was trying to avoid all this, now I'm back, right?" At the beginning here, I do want to throw out something real quick for you Meredydd because a lot of these attempts to do this kind of stuff, what I feel like it results in, and tell me if your experience is like this maybe, is it results in like a minimum proof of concept feeling. You know, like yeah, I technically could do NumPy here in this example or I technically could do this Python thing in Rust a little bit, but it's actually, anything real and interesting is kind of beyond it, it sounds like you're describing that maybe with Transcrypt a tiny bit, but the reason I bring this up is your use case where you're using Skulpt with Anvil, those are massive applications doing complete end-to-end Python, much of which I would say 80% of which is in the browser, you're building single page applications with a UI with Python.

09:53 Panelists: That's the point. So if you want to take a step back and ask if you're trying to put Python in the front end, what exactly are you trying to accomplish and sometimes somebody might think, "Oh well, I like Python, I want to hack around with a compiler, that's fine, I'm going to do this as a hobby." But if you want to actually use it then you need to know what it is you're trying to do. And especially Python people tend to be backend or data or sysadmins or systems programmers, we will look at the web and go, "That is ridiculous." The bar that you have to meet to build a full web application is way too high. To build an application on the traditional stack you need to know HTML, and JavaScript, and CSS and Python for the backend and SQL for the database and then React and then Redux and then Webpack and then Bootstrap and then Sass and I've run out of things to count on, but there's more.

10:50 Michael Kennedy: And Nginx and uWSGI and Ansible.

10:53 Panelists: Oh yes, make it stop. Right, and some people look at that and the most acute pain points they can see are to do with the JavaScript ecosystem. They go, "These JavaScript frameworks out of control, if we could only replace JavaScript with a nicer language, we would solve this problem." And actually I think that's missing the point slightly. I'm firmly convinced that the problem with the web is not actually JavaScript, the problem with the web that makes it so difficult to program is that there are so many layers of which JavaScript is just one. So if you take a typical web application, right, your data is going to start as rows in a database, you're going to transform those into objects, say Python objects in your sever-side code, then you're going to transform those into JSON on HTTP endpoints with GET, POST, PUT, DELETE, silly simple set of verbs, and then you've got to transform those into JavaScript objects in the client and then you've got to transform that into HTML DOM and you've got to transform that into pixels with CSS. All those layers come with a big impedance mismatch and you grow all sorts of hairy frameworks, right? ORMs to turn database rows into Python objects are the least of your problems, right? JavaScript frameworks are the biggest, and hairiest of these translation layers, and so that's where people go, "Oh, JavaScript must be the problem." But if you just swap JavaScript out for Python, you haven't actually reduced the number of times you need to translate your data from one representation into another and you haven't reduced the complexity of your application. And I think that's fundamentally what gives you that proof-of-concepty feeling about something like Transcrypt because you build some code with Transcrypt and you go, "Yay, I've written some Python, and it runs in the web browser." But actually to interact with the outside world, that Python has to do things every bit as unpleasant as what your JavaScript would've had to do. And so you haven't actually got yourself to the Pythonic nirvana that you were chasing. And obviously on a large scale this is what we're trying to solve with Anvil because we're trying to give you an environment where you really can just write a Python function on the server and then just write some Python code on the client and just call that function and you know, just makes some text appear on the screen by manipulating a Python object, and that's a very large, full stack conception of it, but even in the small, I think Skulpt has this one right as well. Because Skulpt is widely used outside of Anvil, it's used in educational environments, it's used in a game that you just showed us before this call, what was it, Code Combat?

13:27 Michael Kennedy: Code Combat at codecombat.com.

13:28 Panelists: Until we were prepping for this call I didn't even know it used Skulpt. It's always great to find your code being used in strange places like that, but it's used a lot in educational contexts. If you're teaching someone, you can't afford to let them touch the JavaScript API because they will run away screaming. You've got to give them a Pythonic experience if you're teaching them. And so those are the values with which Skulpt gets imbued from all directions. If you look at Skulpt, Skulpt treats the browser as a platform to run things on and CPython treats C and D operating system as a platform to run Python on. And it's from Skulpt, there's almost no way to touch the environment. It's up to the person writing the extensions in JavaScript.

14:16 Michael Kennedy: The integration layer, right, the person who says, "Import Skulpt, import the Python, take the Python code and bang 'em together to create app."

14:24 Panelists: If you want something to happen in the browser, you will have to write some JavaScript code, and you will have to write that in a way that you can call it from Skulpt and there's no native stuff. We used to have eval, JS eval in Skulpt, but I removed that a while ago because... Thank you! Well, it's actually a security risk because all these tools are running user's code and storing that in databases and it's better that they don't execute JavaScript that they write themselves.

14:54 Michael Kennedy: That's probably true. It's generally true when it's straight JavaScript and if it's embedded within a Python app it's probably also true.

15:01 Panelists: There's a side security rant I could go on all day about this, part of this problem with having all these layers in the web stack is that what you get is programming languages with the source code for other programming languages embedded in them, right? What you typically ship a client is HTML with JavaScript embedded in it which generates actually more HTML, sometimes with JavaScript embedded in it and CSS embedded in the HTML inside the JavaScript and the JavaScript ecosystem is something where you could not remove the eval-like qualities of it because it's baked into the whole system from top to bottom. Again, Skulpt has that luxury because the browser's being treated like a runtime environment, not like something that should be protruding into your living space.

15:45 Michael Kennedy: This portion of Talk Python to Me is brought to you by Linode. Are you looking for hosting that's fast, simple, and incredibly affordable? Well, look past that bookstore and check out Linode at talkpython.fm/linode, that's L-I-N-O-D-E. Plans start at just five dollars a month for a dedicated server with a gig of ram. They have 10 data centers across the globe so no matter where you are or where your users are there's a data center for you. Whether you want to run a Python web app, host a private Git server or just a file server, you'll get native SSDs on all the machines and newly upgraded 200 Gigabit network, 24/7 friendly support even on holidays and a seven day money back guarantee. Need a little help with your infrastructure? They even offer professional services to help you with architecture, migrations and more. If you want a dedicated server for free for the next four months, just visit talkpython.fm/linode. Now before we get too far into this I kind of want to do a survey of the other options. You've mentioned Transcrypt and we've talked about Skulpt and there's some other ones, and I also want to ask you the question, it probably is a slightly skewed audience here, but does Python even belong in the front end, right? Did you see Russell Keith-Magee's, either of you, his talk about black swans at the most recent PyCon 2019?

17:04 Panelists: We skimmed it very briefly before this recording, so not really.

17:08 Michael Kennedy: I'll summarize 'cause you know, a ton of the listeners also won't have heard it. So basically he gave a talk, I think it was the opening keynote at PyCon 2019 in the U.S. And he said, Look Python is great, we love Python, but there are real challenges that could threaten its great rise and super popularity. Which is wonderful for everybody involved in it but what is the most popular type of computer these days? It's a phone and a tablet, does Python run there, no. If I'm a new developer and I want to have some program I create, I want to share it with my friends, how do I package up a Python app and share it, right? If the answer is you pretty much, it's super hard or you can't then maybe the choices is, well maybe I'll go write some other language, right? And one of 'em that he said was, If I need to learn JavaScript anyway to write web applications and I could do it in Node, why would I bother learning Python, if I'm just getting started anyway because I'm going to have to learn JavaScript so let me just learn one and go from there. So he was making the case that the lack of Python in the browser, it could case a serious drain of the enthusiasm in newcomers to the whole language. So what do you guys think, does Python belong in the front end? I mean, obviously you're doing a lot of work to put it there, but where to you see this?

18:28 Panelists: I have a rant, so Albert, if you want to jump in first, you have about five seconds. I, at the moment, can't advise a website to go write their front end in Python unless you're using Anvil. Skulpt won't be your friend because it will make you work really hard to do that.

18:47 Michael Kennedy: Yeah, and this is more of a broader question, not is today the answer, right?

18:52 Panelists: Does it belong there, yeah. I think eventually we will find a way to properly put it there, but we aren't there yet.

19:01 Michael Kennedy: Okay, Meredydd, you had something to say about this you indicated.

19:04 Panelists: I'm afraid, and this is where I have to tread very carefully because I'm on a Python podcast, but I am not a Python partisan. We, I can hear the indrawn breath, we picked up Python for Anvil because it is the language of education, it is the world's first programming language for people who are trying to teach new people to code because it is friendly because it is to comprehend, because it is easy to pick up. And the web, which is our target platform is the very opposite of that in so many ways and we wanted to bring those virtues to the web. And as it happened, Python looked the likeliest tool for us to pick up and try to ram through that barrier. I am not wedded to Python being the choice of technology that produces this. It just seemed like the best shot for exactly what we were doing at the time. I don't think Python has any God-given right to rule the development world. I think it has to stand or fall on its own two feet, on its own merits, and I think we, the community should be chagrined at the idea that a Pythonic experience is not available for web development. That the values that Python is the torchbearer for in the programming language community aren't available on so many of these platforms that are the popular development platforms. Python, the language itself is merely a delivery mechanism and I don't want to get tribal about whether that is the way to deliver the web. As it happens it's the solution we've chosen, but I don't think Python belongs on the web, I think Pythonic values belong on the web.

20:48 Michael Kennedy: Very interesting. And like you said, just replacing the syntax of JavaScript with the syntax of Python might not necessarily make that difference, right? So let's also talk a little bit about the history because I think it'll be interesting. There's some newcomers and some projects that have been there a long time, so Skulpt has been around for a while, at least since 2011, we talked about Transcrypt, what are some of the others? Either of you take it.

21:21 Panelists: At this point I have to plug a blog post written by my colleague, Shawn Taylor-Morgan because he just gave a talk at PyCon U.K. walking through exactly the answer to your question so hopefully that link's going to be appearing in the show notes. But the broad swing of these things is it's a thing people have tried. There are a bunch of different ones. I don't actually know the order of when they came up but there is Skulpt, which we're talking about a lot. There is Transcrypt, which is a Python to JavaScript compiler that produces JavaScript code that looks very much like the Python code so pretty concise, but at the cost of not exactly the right semantics and isn't abstracted at all from the rest of the web platform, so you have to manipulate JavaScript APIs directly with it. It's also because it's compiled to JavaScript, you don't run the Python. In Skulpt, you can use our very limited debugger to actually pause Python at a certain point and look at variables and this debugger lives in the Python runtime on your browser and with Transcrypt there is no, it runs the JavaScript.

22:45 Michael Kennedy: Right, right, so it seems like with Skulpt looking at, it basically creates a little runtime environment, like it creates the modules variables and little switch to run all the code and things like that and it does seem like it's creating a little execution environment for your Python and it just continues to run there.

23:05 Panelists: Brython's exactly the same, I don't want to say that, Transcrypt is reasonably new, and the guy's doing a great job at making learn Python compilable to JavaScript. And Brython in all their projects does a similar job. Yeah, there's a big difference there of course in that Transcrypt is ahead of time compiled to JavaScript, Brython is actually compiled in the browser, so Brython is a project that lets you write script tags, like you'd write in your HTML script tags, script type equals text/JavaScript, you can write script type equals text/Python and write Python code everywhere you've run JavaScript and then there is a library that you load into your browser and it will then search through your document and translate the Python as the page loads. It's a sort of middle ground. It's not dynamic, like you couldn't do an interactive, write some code, hit enter, and run it, sort of thing with Brython like you can with Skulpt. So I'd say Brython, Transcrypt are both sort of aimed at like, "I would just like to do a straight swap out of my JavaScript for my Python in this code." And Skulpt is actually in a different part of the design space, next to projects like PyPy.js and Pyodide where you take a Python runtime and put it into the browser.

24:37 Michael Kennedy: Okay, so there's PyPy.js and I've covered that before on the show quite a while ago. My understanding, it could be outdated, was that project is not really active anymore, is that accurate, do you know?

24:49 Panelists: No, I actually worked on that, it isn't active anymore. Brian from Mozilla started that and I investigated it quite heavily 'cause this gives you the whole of Python, this gives you a full reference implementation of Python in the browser, but compiling PyPy.js is something special. Whoever's tried it, it's awesome. But it turns out it's 12 to 15 megabytes of very, very dense asm.js that outputs it and it takes a few seconds for the browser to understand.

25:26 Michael Kennedy: So the PyPy, P-Y-P-Y comes from the PyPy Project, which is the JIT compiled version, alternative to CPython, right, so it basically as you said, the asm.js, well, let's take that C code and compile that to JavaScript and then just download it, which is pretty large.

25:44 Panelists: It's even more fun than that. So PyPy is Python interpreted as written in Python, which itself feels like a little bit of a contradiction because if you had a Python interpreter written in Python wouldn't you need another interpreter to interpret it and then you'd just be double stacking, but how PyPy's actually done is that it's written in a subset of Python called RPython, which can be translated effectively to machine code and so there is an RPython to native code compiler. It's actually RPython compiles to C and not to native.

26:19 Michael Kennedy: And that compiles to native, yeah.

26:21 Panelists: Oh, that's right, so okay, C and then to thing and then you stick the C code in through in scripting. Yeah, with PyPy.js you can actually compile the JIT compiler, so what you get is you write Python in your browser, and then PyPy.js interprets that and it JITs C code to the asm C code JITer that JITs that, and that JITs to the JavaScript and the JavaScript is also of course a JIT compiler. So it's like this two stage problem every time you press run. It's a magnificent engineering achievement, but it's like getting an elephant to fly, the most impressive thing is that you did it. No, and it's actually faster than CPython. PyPy.js is faster than CPython? Yes. Oh, Lordy, right, okay. Because PyPy.js, if you've ever run PyPy on your machine compared to CPython, PyPy is massively faster, so it's so much faster that even compiled to JavaScript it's a little bit faster than CPython, but the problem is you never notice that performance gain with PyPy.js because it's a 15 megabyte JavaScript download and then your browser takes a synchronous process to read all that JavaScript and interpret it.

27:41 Michael Kennedy: That to me sounds like the weak point of PyPy.js is you've now given the browser this huge amount of source code to go JITing and processing in and of itself. My experience with PyPy has been small stuff is fast, but I tried to get my websites to run under it, and it was actually slower, I don't know, there must have been something going on there. Anyway, that was a little bit older and that's in hibernation.

28:04 Panelists: Well, 'cause Mozilla have moved on to Pyodide, which is a much more sensible implementation of the same thing for the modern web.

28:11 Michael Kennedy: Yeah, tell us about Pyodide, I mean, I covered it actually not long ago, but not everyone's heard that episode.

28:16 Panelists: Right, in which case I'm going to tread very carefully because I will be much less expert than your previous guest, but it's a full CPython implementation with a whole bunch of common data science native libraries that include native code like NumPy and Pandas all compiled to WebAssembly. So it's really impressive to use, it's like part of the Iodide project, which is just kind of like a Jupyter notebook except all executing in your browser, not having to go back to a server, and again, if you load up the sample page and run a few lines of Python code there will be this period of grunting and straining while your computer downloads this blob of WebAssembly that is the compiled CPython in WebAssembly with all these libraries attached, but once it then gets up and running then you have this amazing data manipulation environment right in your browser and you can do things like using NumPy right in your browser and that's the only Python-in-the-browser technology that can pull that trick off.

29:20 Michael Kennedy: It is pretty impressive, and I definitely think in this use case if you're going to bundle up the C code, bundling it to WebAssembly is a way better place to bundle it than to bundle it to JavaScript, which is then turned around and it's like shipping source verses shipping the app, you know, it's just so much easier to run it that way.

29:36 Panelists: Parsing web assembly is so much easier than parsing JavaScript.

29:41 Michael Kennedy: Right, it's meant for machines to read, not for humans even though some JavaScript gets pretty obfuscated, but still. It's not reading characters and parsing it as an abstract syntax tree probably. So, yeah, this is an interesting project. I don't know where it's going to go. I think in order for this type of thing to work, probably there's two things that need to happen. One, we need really good CDN support so that you get the WebAssembly CPython once ever and then it's just from cache or something like that and the other is a smaller CPython. Do you need to ship the whole standard library and all these things, like is there a way to have smaller Python that you can then embed right?

30:25 Panelists: You're not going to run an ecommerce site off that WebAssembly Python interpreter. It's just got too much stuff and latency.

30:31 Michael Kennedy: Yeah, absolutely.

30:31 Panelists: But yeah, that future of a CDN and global Python interpreter that people can rely on being fast because everyone will have loaded it once, that's a very attractive future, but it's a long way away from now. The fact that Mozilla is making this up means that there are definitely people at Mozilla thinking about this very hard. I don't think it's impossible, and very likely even that it will happen. Oh yeah, and we will be sitting there waving pom-poms when it happens.

31:00 Michael Kennedy: Yeah, that'll be cool. To be honest, what I would rather see than a CDN is Mozilla ship the WebAssembly binary with Firefox and have Chrome ship the same thing, Python 3.7, or 3.8 or whatever and just update it. Browsers update frequently, just update that on release.

31:18 Panelists: That would be awesome, but I think that they would have to, they're probably not going to pick winners like that. We as a community would have to demonstrate that that would be grabbed with both hands before we can persuade a big browser platform like that to ship it.

31:32 Michael Kennedy: Of course, they did it with JavaScript, right, so I mean, there was no other choice, but they do ship with JavaScript runtime in there. I think that's more than enough on the alternatives and we'll probably hear from folks, they can put messages in the comments on the show page like, Oh, did you hear about some name, that I can't even imagine, a project that I didn't know about, right, so check that as well. I feel like we've done a pretty good job of covering Skulpt and its positioning in terms of its features and things like that, and sort of compare and contrast. Maybe the next thing that we could talk about is how do I get started, so I'm a web developer, I want to do something with Python, I'm going to choose Skulpt to do it, maybe a couple of things, what are the use cases where that makes sense, and then what are the steps to get started?

32:20 Panelists: So, I'd say by far the best common use case is if you're building an application where you want your user to write Python. So Anvil is an obvious example here, but there's also something called Trinket for whom Albert's done some work, which is a nice education-focused environment where you can write some Python code, do a Turtle, draw some graphics and have that just run. So for them, something like Skulpt was an easy choice because they could edit in the browser, and just run it right there in the browser. Things like Code Combat.

32:52 Michael Kennedy: Yeah, Trinket is really nice, you get here and it gives you a wonderful little editor with auto-complete and it gives you a canvas or whatever.

33:01 Panelists: Ooh, do they have auto-complete now, hurray!

33:03 Michael Kennedy: I'm pretty sure they did, let me double check that before I...

33:06 Panelists: No, no, they have, it's based on Jedi.

33:08 Michael Kennedy: Yeah, yeah, it's all there, it's super nice. It even goes object.thing, and it evens auto-completes that. It's really nice, it's not just auto-completing the characters.

33:16 Panelists: If you want to see a proper auto-complete implementation with Skulpt, you do have to look at Anvil though.

33:23 Michael Kennedy: Yes, of course.

33:24 Panelists: What Trinket does is they send your Python to the server and then let Jedi look at it and Jedi gives you then the list of completions and then we put that up.

33:35 Michael Kennedy: I see, so they remote out the auto-complete.

33:38 Panelists: Yes. Now the problem with that is of course if you're doing high velocity typing you have the latency of you to the server to back and that is going to seriously cramp your style if you're writing code in a hurry. So the Anvil auto-complete is actually built on top of Skulpt. We grabbed the Skulpt parser and compiler and so what we do is if you hit Tab in Anvil we replace your cursor with a random symbol and then we parse your module into an abstract syntax tree and then walk over it in JavaScript in the web browser and then by the time we've walked down through several modules noticing all the variables that are in scope along the way by the time we hit that magic symbol that represents your cursor, we know what's in scope and then we know what we can offer for auto-complete.

34:20 Michael Kennedy: That's super cool.

34:22 Panelists: Having a compiler that is easy to start hacking around with and that exposes primitives like the abstract syntax tree and the parser in the browser is a really great tool if you want to start messing around with compilers and again if you want to start building things to go back to your original question, what would be the best use of Skulpt is really if you're developing something where you want someone to be writing Python as part of using the thing you're building.

34:49 Michael Kennedy: Yeah, so Anvil is a real good example because you're a development environment to run the result of that on your own custom cloud. Trinket is nice because Trinket is a pretty cool editor in the browser, it's got some IDE features and sort of a visual output component. You started to mention Code Combat as well I think.

35:09 Panelists: Yeah, and again that's, I mean, you only showed it to us shortly before this call, but this idea that you write some Python code and then it runs interactively as your characters do things, again, Skulpt was a really good fit for that because writing code into this thing is how you use it.

35:26 Michael Kennedy: Yeah, exactly, the way you play the game is you fill out it says, you're in this dungeon, you need this character to navigate the maze and then build a fence to keep the creatures out or whatever and so you write that out in code with little bits of auto-complete and it just runs that Python to see if you won the game or not or won the level. This portion of Talk Python to Me is brought to you by Tidelift. Tidelift is the first managed open source subscription giving you commercial support and maintenance for the open source dependencies you use to build your applications. And with Tidelift you not only get more dependable software, but you pay the maintainers of the exact packages you're using, which means your software will keep getting better. The Tidelift subscription covers millions of open source projects across Python, JavaScript, Java, PHP, Ruby, .NET, and more. And the subscription includes security updates, licensing, verification, indemnification, maintenance and code improvements and package selection and version guidance, road map input and tooling and cloud integration. The bottom line is you get the capabilities you'd expect and require from commercial software, but now for all the key open source software you depend upon. Just visit talkpython.fm/tidelift to get started today. Yeah, so that's a good example. What else is out there?

36:43 Panelists: The uses of Skulpt prove pretty much that it is very much in this interactive sphere. My original project where I did the client-side validations actually where we run Python in a much more JavaScript-y way, but again Skulpt is not meant for that because it just makes you work way too hard to get stuff done.

37:05 Michael Kennedy: Right, so it's probably not a great use case to say, "I'm doing Angular.js and that's all great, but I actually want these functions instead written in Python because I just can't bear to do them in JavaScript."

37:17 Panelists: Yeah, get yourself Transcrypt for that as quickly as possible, it will be great. Probably. Yeah. Anvil is much more used for things like Anvil and Trinket, Code Combat, there's a bunch of online courses so there's a couple of MOOCs that use Anvil, some of the other contributors are people who run university courses or build online courses and interactive textbooks with Anvil.

37:42 Michael Kennedy: Okay cool, yeah, we'll link to some of those in the show notes. Yeah, so I think Meredydd, you hit it really clearly on the head saying that a use case is if you need the people to create Python as part of using your project. And every one of the examples that we've seen so far is one of those. Maybe somebody else has done something out there with it that you don't do that and it's not obvious because you're not typing Python, but you're just running an app, I don't know, but who knows? So it sounds like the project is pretty good and certainly of all these use cases I would say Anvil is the most holistic and that's the reason I call it out because you're building an entire app ecosystems with this, also Code Combat is a pretty rich app where it does something kind of amazing. But maybe we could talk about the limitations of what people should be aware of in trying to use it.

38:37 Panelists: So the biggest ones we've talked about already, which is that if you're going to be integrating it into a web application where the rest of it is built using HTML and JavaScript and frameworks and so on, you're going to be doing a lot of work to expose those into your Python runtime so it is not good for quick integrations with bits of Python in traditional developments. Obviously none of the packages that require C like NumPy you're going to be able to do in Skulpt because it's in JavaScript. We get the occasional question on the issues like, "I tried to import C strings or C StringIO." and I was like, "Yeah, that's not going to work." So I ported StringIO and I think I, it's a pure Python implementation, and there's like so many stuff that Python's known for, right? "Oh, I want to parse some CGS, or I want to parse CVS, or I want to do this thing." and it's like, no sorry, that's all written in C, not going to happen.

39:37 Michael Kennedy: So is it not going to happen because it's a lot of work and you guys can't get to it or is it not going to happen 'cause it doesn't make sense to happen, right? So if someone's out there and they really want the CSV package, module to be part of it, could they drop in to source/lib and add a .csv or CSV.py or whatever.

39:54 Panelists: Yeah, yeah, definitely. Yeah, 100%, and a lot of this is so many times my own on the issues like, you know, if you really need it, I will help you, explain where stuff fits in and how you should write, you know 'cause they're all modules, there is some help files around as well, but it's just we're four active developers, maybe five, there's just no time for us to spend on stuff like that. And there's also, depends on the motivation, right? If some of those developers are in education and they're not focusing on the NumPy use case and of course with Anvil our answer to that question is always, "Well, that's okay, the server-side stuff is just a function call away, so go do your deep numerical processing on the server, and make one function call and do it." So there's not a lot of pressure when somebody desperately needs a thing it tends to materialize. Way back in early in my contribution to Skulpt, Skulpt acquired a working datetime module because we needed it, so we pulled over the pure Python implementation from PyPy actually.

41:03 Michael Kennedy: Yeah, I was thinking about PyPy. How much of the stuff could just be brought over from that world.

41:08 Panelists: A remarkable amount, but quite often it turns out that it's using, that was a thing with datetime, I ended up fixing bits of the compiler because some of the slightly exotic things that PyPy's implementation of datetime was using. Skulpt is the best example of this 80-20 rule. Due to sheer number of people trying stuff, we've built this 20% that 80% of the people use, but as soon as you use anything from the standard lib or as soon as you let a seriously senior Python engineer touch Skulpt, like line three they'll be like, "Hey, why doesn't this work?" I'm like, "Yes, well, let me tell you about this." Very long story why it doesn't work. From personal experience I'd say that the senior Python engineer problem has got noticeably better over the time that we've been involved with Skulpt, which is probably not coincidental given that that's the demographic Anvil attracts slightly more than Skulpts other users, but yeah, you'll usually find if you're porting any given thing from the standard library, you'll usually find some corner case in the compiler you need to fix up, but you know, for some of us that's our idea of a fun weekend.

42:22 Michael Kennedy: That's right, guess what case I added this weekend. No, that's pretty cool, so it does sound like if people really wanted to, there's a lot of organization and at least willing to accept people adding more Python standard library support if people wanted to write it.

42:40 Panelists: Oh, absolutely, bring it on. More contributors are always welcome.

42:44 Michael Kennedy: Yeah, yeah, so you guys take PRs and all that, huh?

42:46 Panelists: Yes, indeed, you'll most likely find Albert or possibly me commenting on it.

42:50 Michael Kennedy: Yeah super.

42:50 Panelists: Yeah, and I also operate on the thing that as long as it doesn't break the tests, the add functionality, there's a high likely that I will merge it because a lot of times I'm of the opinion that it's better in than out and if it's part of Skulpt, people will start using it and if there are problems with it, we'll find them and fix them later on.

43:10 Michael Kennedy: Sure, so let me ask you a quick question about that. Suppose I want to have the JSON module in there and I think right now the JSON module inside there says something to the effect of this empty module's here to throw an exception if you try to import JSON, or something like that, this empty package. Yeah, that's what it does? And that's actually an improvement, right because now instead of just going, we don't know what JSON is when you try to import it it goes, actually JSON is not implemented, right? Suppose I want to go and write that, do I write Python code, or do I write JavaScript code or do I write combinations thereof, like what skillset do I need to make that happen?

43:51 Panelists: The choice is yours, you can add things to the Skulpt standard library either with a .py file or with a .js file or with a .py file that imports a module from .js, again it's very much like Python with native C modules, it's the same principle. Obviously writing it in JS is an awful lot more verbose, but if you were doing JSON that would probably be the way forward because you could grab all of the built-in JSON support in the browser.

44:19 Michael Kennedy: The first two letters are about JavaScript in JSON, right? Yeah, so that would be the place to do it. Okay, good to know though, go ahead, Albert.

44:25 Panelists: There is actually a JSON implementation for Skulpt, it's just not in the standard lib.

44:30 Michael Kennedy: Ah, okay, I see.

44:32 Panelists: Somebody that did it, I have somebody at Trinket that has it. This is the eternal problem of open source tools and people that develop it, there's never enough time. I have several lists of things I have to do for Skulpt and this is definitely on one of those lists, but I never import JSON in Skulpt, so I never run into this, so I always forget to add it, but we should. Oh, please do, yes, we have users who'll be made happy by that. Okay, I've made a note to bug you after this call.

45:03 Michael Kennedy: Give me that implementation.

45:05 Panelists: Correct.

45:06 Michael Kennedy: Yeah, okay, sounds really interesting. I do have one other question on contributing really quick. How precisely, 100% CPython compatible does something have to be before you're willing to accept it? Could it be some subset, like if I implemented JSON, but it only did some of the things that's in the standard library would that be okay, or does it have to be all or nothing, right, like every possible use case of CPython has to be accepted or what is your thoughts around that?

45:34 Panelists: Albert said it, better in than out because if that thing is accepted with stubs for the things that don't work they're much more likely to get fixed up than if we stood on a molehill of purity and said, "No, not unless it's complete."

45:48 Michael Kennedy: It's like, "Fine, I'm going to go about my life."

45:50 Panelists: There is a caveat to that though. If you're adding to the compiler we will usually say this is not how it works in CPython, so we shouldn't introduce this unless it's, and this is because we have a few of those things in Skulpt and maybe for Anvil it's not such a big problem, but I think for Anvil it is a big problem. So you have a user that uses it, you're maintaining this wrong implementation of CPython forever.

46:20 Michael Kennedy: Yeah, yeah, I was thinking more of like at the module level and not at the language compiler level, sure.

46:26 Panelists: At the grammar level we are very strict and we actually use the grammar definitions from the real CPython, just imported in, so to drive the Skulpt parser. At the compiler level, work very hard to be a strict subset and there may be some things that we don't support, like for example we're in the middle of our Python 3 move and we don't have async yet, you know, it will syntax error after you try and do that. But everything we do support is in the compiler is to the best of our ability a strict subset of CPython.

46:59 Michael Kennedy: All right, perfect, yeah. Let's maybe round this out with two quick topics on Skulpt here, so you started by talking about adding Python 3.7.3 support, which is great. It's one of the major new things, what are the other new things that are maybe worth highlighting real quick?

47:16 Panelists: One of the really interesting things about the Python 3 grammar migration is that Skulpt does not have enough developer resources to fork ourselves into a Python 3 and Python 2 version. And thanks to projects like Anvil and Trinket and all the others, we have a ton of Python 2 code in production, right? Anvil's users have written a bunch of Python 2 code in the browser.

47:43 Michael Kennedy: They've written the only code they could at the time. And it was Python 2, right?

47:46 Panelists: And on the day we upgrade, they are quite reasonably going to wake up and expect their code to work as normal. And so one of the biggest challenges of this Python 3 migration was that we actually did while keeping compatibility with the Python 2 stuff that we already supported and we did this with bits of old grammar imported, so there is a mode where it will accept the print statement rather than just the print function and that kind of thing. And there's switchs coming through the standard library for you know, am I behaving like Python 2 or am I behaving like Python 3 and that's one of the things about this migration that I'm most proud of because it allowed us to move to a place where we can start moving forwards and sending out the standard library and keeping up with the fact that it's 2019 without breaking all the code that's already in production.

48:39 Michael Kennedy: Yeah, that's super. And you've heard that from other projects as well, like NumPy dropping Python 2 support, Django dropping Python 2 support, it's like now we're free to work on going forward and not double-implementing everything.

48:53 Panelists: Yeah, I guess we're still a generation behind that. We're at the point where we want to add Python, we need Python 3 compatibility, we can't let go of our Python 2 users yet. And the pain that they went through is what we will be maintaining parallel implementations of a bunch of the standard library and a whole bunch of flags and the compiler for some time yet and that is pain that was inevitable from the moment the Python 3 non-backwards compatible announcement was made that every project that's supporting this is in some way, shape, or form going to have to go through this caught between two stools moment, but speaking is so much work, so much better than the alternative.

49:31 Michael Kennedy: Yeah, absolutely. I mean it's extra tricky for you because your platform runs their code. It's not like, well, you just leave Python 2 on your server and you just run your Python 2 stuff there, nobody's going to touch your server, but with Anvil, someone's going to touch your server because it's your job to make it run and update it and things like that, right, so this is challenging on your end.

49:53 Panelists: Certainly, we were the most vocal people going for the compatibility approach, but I mean people like Trinket again... Definitely for Trinket was it was not an option to have two Skulpts and load two of them, like one for new Trinket and one for old Trinket that was never an option, so from their perspective it was like, if we're ever going to move to Python 3 it has to be in the same codebase. And that's a consequence of the way that Skulpt is built, right, because the way you interact, you build the environment that your user's Python code can interact with, means that the JavaScript full support that makes that happen has to talk to Skulpt and it has to talk to pieces of Skulpt APIs that will evolve as Skulpt evolves and so if you tried to interact with some Skulpt 2 fork and Skulpt 3 at the same time you would rapidly lose your marbles. So yeah, for all these projects that have, you know, sure there are commercial people expecting the code behind their products chipping lots of money to still work tomorrow morning, but equally with Trinket there are a whole bunch of teachers and kids expecting the thing that worked yesterday to work tomorrow.

51:07 Michael Kennedy: Right, and maybe even textbooks with printed stuff that is super hard to update that says, go here and type this into that website and it'll work.

51:14 Panelists: And it's our responsibility as platform maintainers to ease the passage.

51:17 Michael Kennedy: Yeah, for sure. All right final topic and Meredydd, you and I have spoken about this and some of the stuff here was a little surprising to me or like I didn't realize it or whatever so I think it's interesting, but I'll give you the first word on this. It seems to me like a lot of the energy around some of this stuff, we talked about Pyodide is in the WebAssembly space and certainly and of the asm.js, Emscripten solutions should go down the WebAssembly direction, but what about Skulpt, does it make sense to have a WebAssembly version of Skulpt? Is that something you're thinking about, what's the story there?

51:52 Panelists: I don't think that we would write the compiler in WebAssembly or make a compiler that compiles a compiler in WebAssembly so what might happen is that we might support libraries written in WebAssembly. Or some of the big switch statement that Skulpt produces might be in WebAssembly, but the biggest problems there is that the interop between JavaScript and WebAssembly is from what I understand mostly be a POST message so very...

52:27 Michael Kennedy: Yeah, very slow.

52:27 Panelists: Slow. In Skulpt world, nothing is slow quickly. Skulpt's quite slow by itself. No, it's not the problem that it's slow, the problem is that Skulpt requires that switch statement to be pretty synchronous and so waiting for a function call, you know, you would have to put that in suspension. There's no obvious path for Skulpt to move to WebAssembly although, given enough time it might happen, stuff might happen in that space. But I still see it as external libraries being written in WebAssembly. I think that's exactly right. For as long as you have to use a knife and fork with oven gloves to communicate between WebAssembly and JavaScript then it's going to not make a lot of sense for something like Skulpt to be in WebAssembly, but the idea of native libraries is very appealing, and there are again, Mozilla taking the lead on a lot of this, there are a whole bunch of rumblings about improving the interop story between WebAssembly and JavaScript. And if that arrives further down the line then it could start to make more sense for us.

53:43 Michael Kennedy: It would be an interesting development around the core, hot loop of process each byte code and also it might be a way to bring in the C libraries, right? You could ship that as a WebAssembly instead of a C implementation for example.

53:59 Panelists: Yeah, exactly.

53:59 Michael Kennedy: Okay. Honestly, it was surprising to me when we first spoke, Meredydd, what I was referring to was that the interop was so limited between WebAssembly and JavaScript. I figure, it's probably just passing a pointer or it's all the same, right? Kind of like PyObject pointers or whatever, right? I guess not.

54:16 Panelists: I think WebAssembly's clearly right, that is the opening of the door to browse aside runtimes that are not shackled to the JavaScript way of doing things, by which I don't just mean JavaScript the language, but I mean that whole stack I was ranting about earlier. And there's a whole bunch of people who see the light behind that door and are getting very excited, but the door is not all the way open yet, there's still a lot you have to do and a lot you would have to do to deploy, really any non-JavaScript shaped thing into the browser in a way that's not like a native game engine.

54:53 Michael Kennedy: Sure, right, where it's entirely contained within WebAssembly, right, something like that?

54:58 Panelists: Something like Anvil is like 90% interop, right? The reason you're writing that Python code in the browser is to drive visual elements on the browser and so actually I would be unsurprised even if we put in all the enormous amount of work in the current state of things, it wouldn't necessarily even improve performance, but that stuff is coming, and I think the world is moving in the right direction, we just have to hang on.

55:23 Michael Kennedy: Yeah, that sounds good, well, thanks for that. Okay, I think we've probably used up all the time that we have to talk about Skulpt and this Python in the browser world. Well, let me ask you really quickly here, each of you, the two questions at the end of the show I always ask. Albert, we'll go with you first. If you're going to write some Python code, what editor do you use?

55:41 Panelists: PyCharm.

55:41 Michael Kennedy: PyCharm, right on, Meredydd?

55:43 Panelists: These days of course a lot of the time I use Anvil's built in editor, but PyCharm is still the gold standard, right? When I'm writing my auto-complete or it's their auto-complete or I have my envious eyes on.

55:53 Michael Kennedy: Yeah, how does it work over here, dang that's nice, let's do that. I agree, cool. And then reverse order, notable PyPI package that maybe folks don't know about, but you've ran across, you're like, "This is amazing, I should tell people about Package X."

56:09 Panelists: Postal, like just transforming user input into valid postal address for basically anywhere on Earth. It's one of those things that the task will drive you wild except you can just pip install it and then somebody has solved this whole scary problem for you. It's awesome, it downloads enormous quantities of data, but it's awesome.

56:30 Michael Kennedy: Yeah, Python bindings for libpostal for fast international address parsing and normalization. Oh, sounds great, Albert?

56:36 Panelists: I'm going to falter on this question to be brutally honest because I'm not been a Python developer for as long as I've been a Skulpt developer. I kind of went the wrong way around, I started as a JavaScript developer and then learned Python because of Skulpt.

56:52 Michael Kennedy: Well, I think that's kind of the reality of it, right? You think of core developers as being super Python developers and they are, but a lot of their day to day I think is to write C code, you know to make Python go. And for you, you've got to write JavaScript to make Skulpt go, right?

57:08 Panelists: Well, my day job is actually in Python. I pip install a lot of stuff, but I don't pick packages that often, so I don't have any interesting, new packages to this day.

57:17 Michael Kennedy: I'll throw one out for you that people might not know about, httpx. Have either of you heard of that one?

57:22 Panelists: Ooh, what's that?

57:23 Michael Kennedy: So, you may have heard of something called requests, you've heard of that, right?

57:26 Panelists: Uh-huh.

57:26 Michael Kennedy: Yeah, everyone has. So this is a 100% API compatible thing with requests, but it adds on HTTP2 support, it adds on async and await support, it has some cool intermediate background, parallel work for like doing a whole bunch of requests and stuff. There's a bunch of modern HTTP support on top of the requests API.

57:52 Panelists: Sweet. Cool, yeah.

57:54 Michael Kennedy: That's a good one.

57:55 Panelists: I love the async and await stuff. I can't wait to add that to Skulpt.

58:01 Michael Kennedy: Yeah, that'll be super.

58:02 Panelists: One of the wonderful things about opening the grammar thing now, all we need to do is get that into the compiler. And we have the technology, I am so excited for what happens next.

58:11 Michael Kennedy: That's super cool, and I think async and await pairs super well with JavaScript anyway and it's even coming to the new version of JavaScript if it's not already here.

58:19 Panelists: It is, definitely.

58:21 Michael Kennedy: Sweet, all right, thank you all for being here. I'm going to give you one more chance for a final call to action. People want to get involved with Skulpt, either contribute to it, use it, what do you guys say, what should they do?

58:30 Panelists: Come find us on GitHub, Skulpt, S-K-U-L-P-T.

58:34 Michael Kennedy: All right, super, Albert?

58:35 Panelists: Yeah, I don't have much to add to that. Try out the website, Skulpt.org.

58:40 Michael Kennedy: Super, all right, well, thank you both for being here, it's great to chat with you.

58:43 Panelists: All right, buh-bye. Cheers.

58:44 Michael Kennedy: Yep, bye. This has been another episode of Talk Python to Me. Our guests on this episode were Meredydd Luff and Albert-Jan Nijburg. And it's been brought to you by Linode and Tidelift. Linode is your go-to hosting for whatever you're building with Python. Get four months free at talkpython.fm/linode, that's L-I-O-N-D-E. If you run an open source project, Tidelift wants to help you get paid for keeping it going strong, just visit talkpython.fm/Tidelift, search for your package and get started today. Want to level up your Python? If you're just getting started try my Python Jumpstart by Building 10 Apps Course or if you're looking for something more advanced, check out our new Async course that digs into all the different types of async programming you can do in Python. And of course if you're interested in more than one of these be sure to check out our Everything Bundle. It's like a subscription that never expires. Be sure to subscribe to the show, open your favorite podcatcher and search for Python. We should be right at the top.

59:44 Panelists: You can also find the iTunes feed at /itunes, the Google Play feed at /play and the direct RSS feed at /rss on talkpython.fm. This is your host, Michael Kennedy. Thanks so much for listening, I really appreciate it. Now get out there and write some Python code.