Do you dream of a day when you can write Python in the browser rather than using JavaScript?
This is no pipe dream. There are several ways to write Python that run in the browser already.
One of the leaders here is Sculpt. This is not just an experiment. Real,
powerful web applications with rich client-side code, Python code, are out in the wild and built
with Sculpt today. We dig into it with Meredith Loof and Albert-Jan Nyberg on this episode.
This is Talk Python to Me, episode 235, recorded September 26th, 2019.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem,
and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm,
and follow the show on Twitter via at Talk Python.
This episode is brought to you by Linode and Tidelift. Please check out what they're offering
during their segments. It really helps support the show. Welcome to Talk Python to Me.
Hello, great to be here.
Hi, thank you.
Yeah, it's great to have you both. And Meredith, welcome back. We talked about your cool project,
Anvil, about two years ago, wasn't it, on the show?
Gosh, yeah. November 2017. Feels like a lifetime ago.
Yeah, in Anvil years.
Oh, yes. Well, startup time operates. It's like dog years, only more intense.
Yeah, yeah, absolutely. And nice to have you here, Albert. So I kind of want to start the
conversation here. We're going to talk about Python in the browser with Sculpt. So I want to start the
conversation with just each of you talking about your involvement in the project. I know you both
have done various things to keep it rolling and work on it day to day. So tell us about it. Albert,
you want to go first?
I started using Sculpt because of a very bad decision I made a few years ago. Eight years ago,
I think. We had the idea that we wanted to write cloud validation and server-side validation in the
same language, so we'd only have to write them once. And our backend at that point was in C#,
and the frontend was in JavaScript. And the only thing we could run in both ends was Python.
So we ended up with Sculpt.
That's a funny path, right? I wouldn't think if you combine JavaScript and C# that you end up
with Python.
Yeah. I wouldn't advise anybody to do this. It did put me in this position where we found Sculpt in this
slightly disheveled state because the original author had sort of left it. And we ran into problems,
and I started fixing them. And yeah, it slowly grows.
Yeah. Who was the original author? Was that Brad Miller?
No. Scott. I've forgotten his name.
Yeah. So Sculpt is not totally new. I was looking through some of the docs and stuff. I certainly
saw 2011 here and there. And that actually would put the timeframe almost exactly the timeframe that
you're talking about, Albert.
Yeah. The original author started off as this. He was really frustrated with JavaScript,
and he liked Python a lot. So he's like, how hard can you do it?
Famous last words.
Yeah. No, but he's like an amazing software engineer because he got such a long way. He
got the compiler and the parser and everything. And, you know, he got the sizable chunk working,
but then abandoned it. There was other stuff to do, probably. And we found it in that state,
and we just took it from there.
Great.
Ask your next question because I'm going to come in at the point of the story where I actually come in.
Yeah. I was going to say, so Albert, what's your relationship to the project today? I know that
you're doing a lot of like overseeing issues and merging PRs and things like that. Are you leading
the project or are you working with someone else who is?
The official leader is Brad Miller. Okay. He took the main ownership over, but two or three in the line
of the people I contributed most. And, you know, I feel this thing where if people do issues or pull
requests, I always feel obliged to act on them. So I do. Yeah. There's no official split of task here.
Sure. Sure. Marith, tell us about your involvement.
So I came to Sculpt a little bit later. It would have been, oh, I guess sometime around 2015.
So I came to Sculpt via Anvil, which if you haven't heard of it, it's a tool for building
full stack web apps with nothing but Python. So Python in the browser, Python on the server,
function calls from one to the other. It's all very nice, but obviously to achieve that vision,
you need to run Python in the web browser as well. So we launched out on a little Safari of Python
options for running Python in the web browser. And I was actually already familiar with Sculpt from
some educational work we had done as part of an outreach workshop at the University of Cambridge,
where my co-founder Ian and I did our PhDs. And so it was the closest tool to hand. Actually,
when we looked around, we realized it was in fact, the best tool for building a new way to drive web
environments with Python. Because as we'll probably discuss later, a lot of the other approaches,
they think very JavaScript-y, and Sculpt thinks very Python-y, and then makes whoever's embedding
it in the web page do the work, which is, as far as I'm concerned, the right way around.
Yeah, if you want to bring Python to the browser, right, it should be as Pythonic as possible. And then
then you make it work, right?
That's the idea. Yes. Well, that's certainly Sculpt's idea. And certainly Anvil's idea.
So I jumped in then, I sort of announced my arrival with a major pull request turning Sculpt's
fundamentally JavaScript-like model, which is to say that it had to execute all at once and then be
finished, into this asynchronous model where you could write blocking Python code, and then the
JavaScript runtime, it could sort of hand control back to the JavaScript runtime for a little bit. And then
when some asynchronous operation had completed, like going to the server for something or getting
some user entry, it could resume your Python code exactly where it was. And from there, I've become,
like Albert, one of the more active contributors on the project. And as Albert said, there isn't really
division of tasks. We all just jump in wherever it is that we see efforts in need of things. So I do a lot
of work on the compiler. I do sort of a cleanup of bugs that people find in Sculpt, typically while
using Anvil in advanced ways. And of course, the reason I'm here...
You're like, wait, that should work in Python, right? But it's not.
Yes, absolutely. You know, like any non-core Python implementation, there are going to be
divergence bugs where CPython does one thing and we do the other, and we need to run in and fix those.
And of course, most recently, Albert and I did the big upgrade of Sculpt from the Python 2.7
grammar to the Python 3.7 grammar, which opens the door to us moving to an actual Python 3 world in
Sculpt, getting in under the wire for the end of life for Python 2.
Yeah, beautiful. Yeah. So heck, you've got months to spare, at least one.
Days, whole days, Michael.
Whole days. It's so much time.
It's good, though.
Yeah, it's got to feel good because, yeah, it was 2.7 or even 2.6 for quite a while. So it's great to see
making progress.
Yeah.
That's awesome.
Updating the grammar to 2.7 was one of my first big projects in Python.
And that was only one step, and that took me, I think, a summer of coding.
Just looking at the way that Sculpt creates the JavaScript or the JavaScript stuff is implemented
for even something like a print hello world, there's like a lot of stuff going on in JavaScript
there that it's kind of got to set up the environment. So yeah, I can imagine changing and making those
changes took some work.
That question cuts to the core of one of the biggest design questions for every Python in
the browser implementation, which is essentially, are you trying to do the minimum possible to
allow you to write Python in a JavaScript world? And, you know, Transcript, for example, is an
example of a project that really does take that approach. It translates Python to JavaScript code that
looks very much like the Python code. And the price it pays for that is that it's somewhat loose on the
Python semantics. And the way you interact with the page is certainly not Pythonic at all, if you
want to do any IO.
Right.
And Sculpt is right the way at the other end of the spectrum, where you say, I don't care how much work
the JavaScript has to do around here, I am going to deliver a Python-y experience to the user and
allow them to be interactive and allow them to think about the outside world as a Python
programmer would, you know, as something you can print to, as opposed to something where you get
some document object and then get some HTML element, and then you do some translated DOM manipulation on
it and jump out a window.
Exactly, exactly. Like what? I was trying to avoid all this. Now I'm back, right?
At the beginning here, I do want to throw out something real quick for you, Marith, because
a lot of these attempts to do this kind of stuff, what I feel like it results in, and
you tell me if your experience is like this maybe, is it results in like a minimum proof
of concept feeling, you know, like, oh yeah, so I technically could do NumPy here in this
example, or I technically could do like this Python thing and Rust a little bit, but it's
actually anything real and interesting is kind of like beyond it, right? It kind of sounds like
you're describing that maybe with transcript a tiny bit. But the reason I bring this up
is your use case, we're using Sculpt with Anvil. Those are massive applications doing
complete end-to-end application Python, much of which I would say 80% of which is in the
front, in the browser. You're building single page applications with a UI with Python.
That's the point. So if you want to take a step back and ask if you're trying to put Python
in the front end, what exactly are you trying to accomplish? And sometimes, you know, somebody
might think, oh well, you know, I like Python, I want to hack around with a compiler, that's
fine, you know, I'm going to do this as a hobby. But if you want to actually use it, then you need
to know what it is you're trying to do. And especially Python people tend to be back-end
or data or sysadmins or systems programmers. We will look at the web and go, that is ridiculous.
Like the bar that you have to meet to build a full web application is way too high. To build an
application on the quote-unquote traditional stack, you need to know like HTML and JavaScript and CSS and
Python for the back-end and SQL for the database and then React and then Redux and then Webpack and then
Bootstrap and then SAS and I've run out of fingers to count on, but there's more.
And Nginx and Microwisgee and Ansible and...
Oh god, yes, make it stop. Right. And some people look at that and the most acute pain points they can
see are to do with the JavaScript ecosystem. They go, these JavaScript frameworks are out of control.
If we could only replace JavaScript with a nicer language, we would solve this problem.
And actually, I think that's missing the point slightly. I'm firmly convinced that the problem
with the web is not actually JavaScript. The problem with the web that makes it so difficult
to program is that there are so many layers of which JavaScript is just one. So if you take a typical
web application, right, your data is going to start as rows in a database. You're going to transform
those into objects, say Python objects in your server-side code. Then you're going to transform
those into JSON on HTTP endpoints with get, post, put, delete, silly, simple set of verbs.
And then you're going to transform those into JavaScript objects in the client. And then you're going to transform
that into HTML DOM and you're going to transform that into pixels with CSS. All of those layers come with a
big impedance mismatch and you grow all sorts of hairy frameworks, right?
Orms to turn database rows into Python objects are the least of your problems, right? It's the
JavaScript frameworks that are the biggest and hairiest of these translation layers. And so that's where
people go, oh, JavaScript must be the problem. But if you just swap JavaScript out for Python,
you haven't actually reduced the number of times you need to translate your data from
one representation into another, and you haven't reduced the complexity of your application. And I
think that's fundamentally what gives you that kind of proof of concept feeling about something like
transcript. Because you build some code with transcript and you go, yay, I've written some Python and it
runs in the web browser. But actually, to interact with the outside world, that Python has to do
things every bit as unpleasant as what your JavaScript would have had to do. And so you haven't actually
got yourself to the Pythonic nirvana that you were chasing. And obviously, on a large scale,
this is what we're trying to solve with Anvil, because we're trying to give you an environment
where you really can just write a Python function on the server and then just write some Python code on
the client and just call that function and just make some text appear on the screen by manipulating
a Python object. That's a very large full stack conception of it. But even in the small, I think
Sculpt has this one right as well. Because Sculpt is widely used outside Anvil. It's used in educational
environments. It's used in a game that you just showed us before this call. What was it, Code Combat?
Code Combat at CodeCombat.com.
Until we were prepping for this call, I didn't even know it used Sculpt. It's always great to find your code
being used in strange places like that. But it's used a lot in educational contexts.
If you're teaching someone, you can't afford to let them touch the JavaScript API because they will run
away screaming. You've got to give them a Python experience if you're teaching them.
And so those are the values with which Sculpt gets imbued from all directions.
If you look at Sculpt, Sculpt treats the browser as a platform to run things on it. And CPython treats
C and the operating system as a platform to run Python on. And from Sculpt, there's almost no way to touch
the environment. It's up to the person writing the extensions in JavaScript.
The integration layer, right? The person who says import Sculpt, import the Python, take the
Python code and bang them together to create app.
If you want to have something to happen in the browser, you will have to write some JavaScript
code and you will have to write that in a way that you can call it from Sculpt. And you can't,
there's no native stuff. We used to have eval in JS eval in Sculpt. I removed that a while ago because...
Thank you! That's good.
Well, it's actually a security risk because all these tools are running users' code and
storing that in databases. And it's better that they don't execute JavaScript that they write
themselves. That's probably true. It's generally true when it's straight JavaScript and if it's
embedded within a Python app, it's probably also true. There's a side security rant I could go on all
day about this. Part of this problem with having all these layers in the web stack is that what you get
is programming languages with the source code for others programming languages embedded in them,
right? What you typically ship a client is HTML with JavaScript embedded in it, which generates
actually more HTML sometimes with JavaScript embedded in it. And CSS embedded in that HTML inside the
JavaScript. And the JavaScript ecosystem is something where you could not remove the eval-like qualities
of it because it's baked into the whole system from top to bottom. And again, Sculpt has that luxury
because the browser is being treated like a runtime environment, not like something that should be
protruding into your living space.
This portion of Talk Python to me is brought to you by Linode. Are you looking for hosting that's fast,
simple, and incredibly affordable? Well, look past that bookstore and check out Linode at
talkpython.fm/linode. That's L-I-N-O-D-E. Plans start at just $5 a month for a dedicated server with a gig of RAM.
They have 10 data centers across the globe. So no matter where you are or where your users are,
there's a data center for you. Whether you want to run a Python web app, host a private Git server,
or just a file server, you'll get native SSDs on all the machines, a newly upgraded 200 gigabit network,
24/7 friendly support, even on holidays, and a seven day money back guarantee. Need a little help with
your infrastructure? They even offer professional services to help you with architecture, migrations,
and more. Do you want a dedicated server for free for the next four months? Just visit talkpython.fm/linode.
Now, before we get too far into this, I kind of want to do a survey of the other options. You've
mentioned Transcript and we've talked about Sculpt and there's some other ones. And I also want to ask
you the question, it probably is a slightly skewed audience here, does Python even belong in
the front end. Did you see Russell Keith McGee's, either of you, his talk about black swans at the
most recent PyCon 2019? We skimmed it very briefly before this recording, so not really.
I'll summarize because, you know, a ton of the listeners also won't have heard it. So basically,
he gave a talk, I think it was the opening keynote at PyCon 2019 in the US. And he said, look, Python is
great. We love Python, but there are real challenges that could threaten its great rise and super popularity,
which is wonderful for everybody involved in it. But what is the most popular type of computer these days?
It's a phone and a tablet. Does Python run there? No. If I'm a new developer and I want to have,
you know, some program I create and I want to share it with my friends, how do I package up a Python app
and share it? Right? If the answer is you pretty much, you know, it's super hard or you can't, then
maybe the choice is, well, I'll go write some other language. Right? And one of them that he said was,
if I need to learn JavaScript anyway to write web applications and I could do it in Node, why would
I bother learning Python if I'm just getting started anyway? Because I'm going to have to learn
JavaScript. So let me just learn one and go from there. So he was making the case that the lack of
Python in the browser, it could cause a serious sort of drain of the enthusiasm and newcomers to the
whole language. So what do you guys think? Python belong in the front end? I mean, obviously,
you're doing a lot of work to put it there. But where do you see this?
I have a rant. So, Albert, if you want to jump in first, you have about five seconds.
I, at the moment, can't advise a website to go right there front end in Python, unless you're
using Anvil. Sculpt won't be your friend because it will make you work really hard to do that.
Yeah. And this is more of a broader question, not is like today to answer, right?
Does it belong there? Yeah. I think eventually we will find a way to properly put it there,
but we aren't there yet. Okay. Meredith, you had something to say about this,
you indicated. I'm afraid, and this is where I have to tread very carefully because I'm on a Python
podcast, but I am not a Python partisan. We, I can hear the in-drawn breath, we picked up Python for
Anvil because it is the language of education. It is the world's first programming language for people
who are trying to teach new people to code because it is friendly, because it is easy to comprehend,
because it is easy to pick up. And the web, which is our target platform, is the very opposite of that
in so many ways. And we wanted to bring those virtues to the web. And as it happened, Python looked the
likeliest tool for us to pick up and try to ram through that barrier. I am not wedded to Python
being the choice of technology that produces this. It just seemed like the best shot for exactly what
we were doing at the time. I don't think Python has any God-given right to rule the development world.
I think it has to stand or fall on its own two feet and on its own merits. And I think we as a community
should be chagrined at the idea that a Pythonic experience is not available for web development.
That the values that Python is the torch bearer for in the programming language community aren't
available on so many of these platforms that are the popular development platforms.
Yeah. Python, the language itself is merely a delivery mechanism. And I don't want to get
tribal about whether that is the way to deliver the web. As it happens, it's the loose solution we've
chosen. But I don't think Python belongs on the web. I think Pythonic values belong on the web.
Very interesting. And like you said, just choosing, just replacing the syntax of JavaScript with the
syntax of Python might not necessarily make that difference, right? So let's also talk a little bit
about the history because I think it'll be interesting. There's some newcomers and some
projects that have been there a long time. So Sculpt has been around for a while, at least since 2011.
What are some, we talked about Transcript. What are some of the others? Either of you take it.
At this point, I have to plug a blog post written by my colleague, Sean Taylor Morgan, because he just gave
a talk at PyCon UK walking through exactly the answer to your question. So hopefully that link's going to be
appearing in the show notes. Okay.
But the broad swing of these things is it's a thing people have tried. There are a bunch of different
ones. I don't actually know the order of when they came up, but there is Sculpt, which we're talking about a lot.
There is Transcript, which is a Python to JavaScript compiler that produces JavaScript code that looks
very much like the Python code. So pretty concise, but at the cost of not exactly the right semantics.
It's also because it's compiled to JavaScript. You don't run the Python. In Sculpt,
you can use our very limited debugger to actually pause
Python at a certain point and look at variables. And this debugger lives in the Python runtime on your
browser. And with Transcript, it doesn't run. It runs the JavaScript and you write Python.
Right, right. Yeah. So it seems like with Sculpt, looking at it, it basically creates a
little runtime environment. It creates the modules variables and it
little switch to run all the code and things like that. And it does seem like it's creating
a little execution environment for your Python. And it just continues to run there. Yeah.
Brighton is exactly the same. I don't want to say that. Transcript is, I think,
is reasonably new and the guy's doing a great job. But he's making
modern Python compilable to JavaScript. And Brighton is an older project, but does a similar job.
Yeah, there's a big difference there, of course, in that Transcript is ahead of time compiled to
JavaScript. Brighton is actually compiled in the browser. So Brighton is a project that lets you
write script tags like you write in your HTML script tags, script type equals text, JavaScript. You can
write script type equals text, Python and write Python code everywhere you'd run JavaScript. And then
there is a library that you load into your browser and it will then search through your document and
translate all of, translate the Python as the page loads. So it's, it's kind of this, it's a sort of
middle ground. It's not dynamic, but you couldn't do like an interactive write some code, hit enter and
run it thing with Brighton like you can with, with Sculpt. So I'd say like Brighton Transcript are both
sort of aimed at like, I would just like to do a straight swap out of my JavaScript for my Python
in this code. And Sculpt is actually in a different part of the design space next to projects like
PyPy.js and Pyodide, where you take a Python runtime and put it into the browser.
Okay, so there's PyPy.js. And I've covered that before on the show quite a while ago. My
understanding it could be outdated was that project is not really active anymore. Is that accurate? Do you
know? No, I actually worked on that. It isn't active anymore. Ryan from Mozilla started that and
I investigated it quite heavily because this gives you the whole of Python. This gives you like a
full reference implementation of Python in the browser. But compiling PyPy.js is something special.
I've never tried it, but it's awesome. But it turns out it's like 12 to 15 megabytes of very,
very, very dense ASM.js that outputs it. Yeah. Yeah, it takes a few seconds to put a browser to understand.
So it's the PyPy, P-Y-P-Y comes from the PyPy project, which is the compiled version alternative to
CPython, right? So it basically, as you said, the ASM.js bit is, well, let's take that C code and compile
that to JavaScript and then just download it, which is pretty large.
It's even more fun than that. So PyPy is a Python interpreter that's written in Python,
which itself feels like a little bit of a contradiction. Because if you had a Python
interpreter written in Python, wouldn't you need another interpreter to interpret it? And then you'd
just be like double stacking. But what it's actually, how PyPy is actually done is that it's written in a
subset of Python called RPython, which can be translated effectively to machine code. And so there is an
RPython to native code compiler.
It's actually RPython compiles to C and not to native.
And then that compiles to native. Yeah.
So, okay, C and then to thing. And then you stick the C code in through in script.
But PyPyJS, you can actually run, you can actually compile the JIT compiler. So what you get is you
write Python in your browser, and then PyPyJS interprets that. And it jits C code to the
ASM C code jitter that jits that. And that jits to the JavaScript. And the JavaScript is also,
of course, a JIT environment. Yeah.
It's like this two stage rocket every time you press run.
It's a magnificent engineering achievement, but it's like getting an elephant to fly. The most
impressive thing is that you did it. Yeah.
Yeah. No. And it's actually, it's faster than CPython.
PyPy.js is faster than CPython.
Yes.
Oh, Lordy. Right. Okay. Yes.
Well, that's because PyPyJS, if you've ever run PyPy on your machine compared to CPython,
PyPy is massively faster. So it's so much faster that even compiled to JavaScript, it's a little
bit faster than CPython. But the problem is that you never notice that performance is a performance
gain with PyPyJS because it's a 15 megabyte JavaScript download. And then the browser takes
a synchronous process to read all that JavaScript. That to me sounds like the weak point of PyPyJS is
you've now given the browser this huge amount of source code to go jitting and processing in and of
itself. My experience with PyPy has been like small stuff is fast, but I tried to get my websites to run
under and it was actually slower. I don't know. There must've been something going on there. Anyway,
that was a little bit older and that's in a hibernation. And then-
Well, because Mozilla have moved on to Pyodide, which is a much more sensible implementation. The
same thing for the modern web.
Yeah. Well, tell us about Pyodide. I mean, I covered it actually not long ago,
but not everyone's heard that episode.
Right. Well, in which case I'm going to tread very carefully because I will be much less expert than your
previous guest, but it's a full CPython implementation with a whole bunch of common data science native
libraries that include native code like NumPy and Pandas all compiled to WebAssembly.
So it's really impressive to use. It's like part of the iodide project, which is just kind of like a
Jupyter notebook, except all executing your browser, not having to go back to a server.
And again, if you load up the sample page and run a few lines of Python code,
there will be this period of grunting and straining while your computer downloads this blob of WebAssembly
that is the compiled CPython in WebAssembly with all these libraries attached.
But once it then gets up and running, then you have this amazing data manipulation environment
right in your browser and you can do things like using NumPy right in your browser. And that's the
only Python in the browser technology that can pull that trick off.
It is pretty impressive. And I definitely think in this use case, if you're going to bundle up the C code,
bundling it to WebAssembly is a way better place to bundle it than to bundle it to JavaScript, which is
then turned around and like, it's like shipping source versus shipping the app.
You know, it's just so much easier to run it that way.
But parsing WebAssembly is so much easier than parsing JavaScript.
Right. It's meant for machines to read. Yeah.
Not for humans. No.
No.
Even though some JavaScript gets pretty obfuscated, but still.
It's not reading characters and parsing it as a abstract syntax tree probably.
So yeah, this is an interesting project. I don't know where it's going to go.
I think in order for this type of thing to work, probably there's two things that need to happen.
One, we need really good CDN support so that you get the WebAssembly CPython once ever,
and then it's just from cache or something like that.
The other is a smaller CPython. Do you need to ship the whole standard library and all these things?
Like, is there a way to have smaller Python that you can then embed?
Right.
Certainly, you're not going to run an e-commerce site off that WebAssembly Python interpreter. It's just got too much startup latency.
Yeah, absolutely.
The future of a CDN global Python interpreter that people can rely on being fast because everyone will have loaded it once.
It's a very attractive future, but it's a long way away from now.
The fact that Mozilla is making this stuff means that there are definitely people at Mozilla thinking about this very hard.
So it's definitely, I don't think it's impossible, very likely even, that it will happen.
Oh yeah, and we'll be sitting there waving pom-poms when it happens.
Yeah, that would be cool. To be honest, like what I would rather see than a CDN is Mozilla ship the WebAssembly binary with Firefox and have Chrome ship the same thing, you know, Python 3.7 or 3.8 or whatever, and just update it.
Browsers update frequently. Just update that on release.
That would be awesome, but I think that they would have to, they're probably not going to pick winners like that.
You know, we as a community would have to demonstrate that that would be grabbed with both hands before we can persuade a big browser platform like that to ship it.
Of course, they did it with JavaScript, right?
So, I mean, there was no other choice, but they do ship a JavaScript runtime in there.
I think that's more than enough on the alternatives, and we'll probably hear from folks.
They can put messages in the comments on the show page like, oh, did you hear about some name that I can't even imagine, a project that I didn't know about, right?
So, check that out as well.
I feel like we've done a pretty good job of covering Sculpt and its positioning in terms of its features and things like that and sort of compare and contrast.
Maybe the next thing that we could talk about is how do I get started?
So, I'm a web developer.
I want to do something with Python.
I'm going to choose Sculpt to do it.
Maybe a couple of things.
What are the use cases where that makes sense?
And then what are the steps to get started?
So, I'd say by far the most common use case is if you're building an application where you want your user to write Python.
So, Anvil is an obvious example here.
But there's also something called Trinket for whom Albert's done some work, which is a nice education-focused environment where you can, you know, bring up, write some Python code, you know, do a turtle, draw some graphics, and have that just run.
So, for them, something like Sculpt was an easy choice because they could edit in the browser, just run it right there in the browser.
Things like CodeCompat.
Yeah.
Trinket is really nice.
You get here and it gives you like a wonderful little editor with autocomplete and it gives you a canvas or whatever.
Oh, do they have autocomplete now?
Hurrah!
I'm pretty sure they did.
Let me double-check that before I...
No, no, they have.
It's based on Jedi.
Yeah, yeah, it's all there.
It's super nice.
It even goes object dot thing and it even autocompletes that.
It's really nice.
It's not just autocompleting the characters.
If you want to see a proper autocomplete implementation with Sculpt, you do have to look at Anvil, though.
Yes, of course.
What Trinket does is they send your Python to the server and then let Jedi look at it.
And Jedi gives you then the list of completions and then we set up.
I see.
So they remote out the autocomplete.
Yes.
Okay.
And the problem with that is, of course, if you're doing high-velocity typing, you have the latency of you to the server to back.
And that is going to seriously cramp your style if you're writing code in a hurry.
So the Anvil autocompletor is actually built on top of Sculpt.
We grabbed the Sculpt like parser and compiler.
And so what we do is if you hit tab in Anvil, we replace your cursor with like a random symbol.
And then we parse your module into an abstract syntax tree and then walk over it in JavaScript in the web browser.
And then by the time we've walked down through several modules, you know, noticing all the variables that are in scope along the way,
by the time we hit that magic symbol that represents your cursor, we know what's in scope.
And then we know what we can offer for autocomplete.
That's super cool.
Having a compiler that is easy to start hacking around with and that exposes like primitives, like the abstract syntax tree and like the parser in the browser is a really great tool if you want to start messing around with compilers.
Again, if you want to start building things, to go back to your original question, what would be the best use of Sculpt?
Really, it's if you're developing something where you want someone to be writing Python as part of using the thing you're building.
Yeah, so Anvil is a real good example because you're a development environment to run the result of that on your own custom cloud.
Trinket is nice because Trinket is a pretty cool IDE, not IDE, editor in the browser.
It's got some IDE features and sort of a visual output component.
You had started to mention code combat as well, I think.
Yeah, and again, that's, I mean, you only showed it to us shortly before this call.
But this idea that, you know, you write some Python code and then it runs interactively as your characters do things.
Again, Sculpt was a really good fit for that because writing code into this thing is how you use it.
Yeah, exactly. The way you play the game is you fill out, it says you're in this dungeon, you need this character to navigate the maze and then build a fence to keep the creatures out or whatever.
And so you write that out in code with little bits of autocomplete and it just runs that Python to see if you won the game or not or won the level.
This portion of Talk Python to Me is brought to you by Tidelift.
Tidelift is the first managed open source subscription, giving you commercial support and maintenance for the open source dependencies.
You use to build your applications.
And with Tidelift, you not only get more dependable software, but you pay the maintainers of the exact packages you're using, which means your software will keep getting better.
The Tidelift subscription covers millions of open source projects across Python, JavaScript, Java, PHP, Ruby, .NET, and more.
And the subscription includes security updates, licensing, verification, and indemnification, maintenance and code improvements, package selection and version guidance, roadmap input, and tooling and cloud integration.
The bottom line is you get the capabilities you'd expect and require from commercial software.
Yeah, so that's a good example.
What else is out there?
The uses of Sculpt prove pretty much that it is very much in this interactive sphere.
My original project where I did the client-side validation is actually where we run Python in a much more sort of JavaScript-y way.
But again, Sculpt is not meant for that because it just makes you work way too hard to get stuff done.
Right.
So it's probably not a great use case to say, I'm doing AngularJS and that's all great, but I actually want these functions instead written in Python because I just can't bear to do them in JavaScript.
Yeah.
Get yourself a transcript for that as quickly as possible.
It will be great.
Probably.
Yeah.
Anvil is much more used for things like Anvil and Trinket, Code Combat.
There's a bunch of online courses.
So there's a couple of MOOCs that use Anvil.
Some of the other contributors are people who run university courses or build online courses and interactive textbooks with Anvil.
Okay, cool.
Yeah, we'll link to some of those in the show notes.
Yeah, so I think, Meredith, you hit a really...
It's really clearly on the head saying that the use case is if you need people to create Python as part of using your project.
And every one of the examples that we've seen so far is one of those.
Maybe somebody else has done something out there with it that you don't do that and it's not obvious because you're not typing Python.
You're just running an app.
I don't know, but who knows?
So it sounds like the project is pretty good.
And certainly of all these use cases, I would say Anvil is the most holistic.
And that's the reason I called it out because you're building entire app ecosystems with this.
Also, Code Combat is a pretty rich app where it's something kind of amazing.
But maybe we could talk about the limitations of what people should be aware of in trying to use it.
The biggest ones we've talked about already, which is that, you know, if you're going to be integrating it into a web application where the rest of it is built using HTML and JavaScript and frameworks and so on,
you're going to be doing a lot of work to expose those into your Python runtime.
So it's not good for like, you know, quick integratings of bits of Python into traditional development.
Obviously, none of the packages that require C like NumPy, you're going to be able to do in Sculpt because it's in JavaScript.
We get the occasional question on the issues like, oh, I tried to import C strings or C string IO.
And I was like, yeah, it's not going to work.
Sorry, that's not going to work.
So I ported string IO.
And I think I'm from the pure Python implementation.
But and there's like so many stuff that Python is known for, right?
Oh, I want to parse in CVS or I want to parse CSV or I want to do this thing.
And it's like, no, sorry, that's all written in C.
Not going to happen.
So is it not going to happen because it's a lot of work and you guys can't get to it?
Or is it not going to happen because it doesn't make sense to happen, right?
So if someone's out there and they really want the CSV package module to be part of it, like could they drop in to source slash lib and add a .csv?
Yeah, yeah.
Or csv.py or whatever?
Yeah, 100%.
And a lot of the, this is so many times my answer on the issues.
Like, you know, if you really need it, like I will help you and explain where stuff fits in and how you should write, you know, external modules.
There's some help files around that as well.
But it's just where for active developers, maybe five, there's just no time for us to spend on stuff like that.
And there's also, I mean, depends on their motivation, right?
If, you know, some of those developers are in education and they're not trying, they're not focusing on like the NumPy use case.
And of course, with Anvil, our answer to that question is always, well, that's okay.
The server side stuff is just a function call away.
So go do your deep numerical processing on the server and make one function call and do it.
So, yeah, there's not a lot of pressure.
Yeah.
When somebody desperately needs a thing, it tends to materialize.
Way back in early in my contribution to Sculpt, Sculpt acquired a working Datetime module because we needed it.
And so we pulled over the pure Python implementation from PyPy, actually.
Yeah.
I was thinking about PyPy.
How much of the stuff could just be sort of brought over from that world?
A remarkable amount, but quite often it turns out that it's using, so that was a thing for
with Datetime.
I ended up fixing up bits of the compiler because of the slightly, some of the slightly exotic
things that PyPy's implementation of Datetime was using.
Okay.
Sculpt is the best example of this 80-20 rule.
Due to sort of sheer number of people trying stuff, we've built this 20% that 80% of the
people use.
But as soon as you use anything from the standard labor, as soon as you let a seriously senior
Python engineer touch Sculpt, like line three, they'll be like, hey, why doesn't this work?
We're like, yes.
Well, let me tell you about this.
It's a very long story why it doesn't work.
From personal experience, I'd said that the senior Python engineer problem has got noticeably
better over the time that we've been involved with Sculpt, which is probably not coincidental,
given that that's a demographic and will attract slightly more than Sculpt's other users.
But yeah, you'll usually find in any, if you're porting any given thing from the standard library,
you'll usually find some corner case in the compiler you need to fix up.
But you know, for some of us, that's our idea of a fun weekend.
That's right.
Guess what case I added this weekend.
No, that's pretty cool.
So it does sound like if people really wanted to, there's a lot of organization and at least
willing to accept people adding more Python standard library support if people wanted to write it.
Oh, absolutely.
Bring it on.
More contributors are always welcome.
Yeah, yeah.
So you guys take PRs and all that, huh?
Yes, indeed.
You'll most likely find Albert or possibly me commenting on it.
Yeah, super.
Yeah.
And I also operate on the thing that as long as it doesn't break the tests and add functionality,
there's a high likely that I will merge it.
Because a lot of times I'm of the opinion it's better in than out.
And like if it's part of Sculpt, people will start using it.
And if there are problems with it, we'll find them and fix them later on.
Sure.
So let me ask you a quick question about that.
If I suppose I want to have the JSON module in there.
And I think right now, the JSON module inside there says something to the effect of this empty module is here to throw an exception if you try to import JSON or something like that.
Yeah.
This empty package.
Yeah.
That's what it does.
And that's actually an improvement, right?
Because now instead of just going, we don't know what JSON is when you try to import it, it goes, actually, JSON is not implemented, right?
Yeah.
Suppose I want to go and write that.
Do I write Python code or do I write JavaScript code?
Or do I write combinations thereof?
Like what skill set do I need to make that happen?
The choice is yours.
You can add things to the Sculpt standard library either with a .py file or a .js file or with a .py file that imports a module from .js.
Again, it's very much like Python with native C modules.
It's the same principle.
Obviously, writing it in .js is an awful lot more verbose.
But if you were doing JSON, that would probably be the way forward because you could grab all of the built-in JSON support in the browser.
The first two letters are about JavaScript.
Yeah.
In JSON, right?
Yeah.
So that would be the place to do it.
Okay.
Good to know, though.
Go ahead, Albert.
There is actually a JSON implementation for Sculpt.
It's just not in the standard lib.
Oh, okay.
I see.
There's somebody that did it.
I have somebody at Trinket that has it.
This is the eternal problem of open source tools and people that develop it.
There's never enough time.
This is probably several lists of things that I have to do for Sculpt.
And this is definitely on one of those lists.
But I never import JSON in Sculpt.
Yeah.
So I never run into this.
I always forget to add it, but we should.
Oh, please do.
Yes.
We have users who'd be made happy by that.
Okay.
I've made a note to bug you after this call.
Give me that implementation.
Correct.
Yeah.
Okay.
It sounds really interesting.
I do have one other question.
I'll contribute really quick.
How precisely 100% CPython compatible does something have to be before you're willing to accept it?
Could it be some subset, like if I implemented JSON, but it only did some of the things that's in the standard library?
Would that be okay?
Or does it have to be all or nothing?
Right?
Like every possible use case of CPython has to be accepted?
Or what are your thoughts around that?
I'll have said it better in than out.
Because if that thing is accepted with stubs for the things that don't work, then it's much more likely to get fixed up than if we stood on a molehill of purity and said, no, not unless it's complete.
It's like, fine.
I've got to go up my life.
There is a caveat to that, though.
If you're adding to the compiler, we will usually say this is not how it works in CPython.
So we shouldn't introduce this.
And this is because we have a few of those things in Sculpt.
And maybe for Anvil, it's not such a big problem.
But I know I think for Anvil, it is a big problem.
Oh, yeah.
If you have a user that uses it, you're maintaining this wrong implementation of CPython forever.
Yeah, yeah.
I was thinking more of like at the module level and not at the language compiler level, sure.
At the grammar level, we are very strict.
And we actually use the grammar definitions from the real CPython just imported in to drive the Sculpt parser.
At the compiler level, work very hard to be a strict subset.
And there may be some things we don't support.
But like, for example, we're in the middle of our Python 3 move and we don't have async yet.
You know, it will syntax error out if you try and do that.
But everything we do support in the compiler is, to the best of our ability, a strict subset of CPython.
All right.
Perfect.
Yeah.
Let's maybe round this out with two quick topics on Sculpt here.
So you started by talking about adding Python 3.7.3 support, which is great.
That's one of the major new things.
What are the other new things that are maybe worth highlighting real quick?
One of the really interesting things about the Python 3 grammar migration,
is that Sculpt does not have enough developer resources to fork ourselves into a Python 3 and Python 2 version.
And thanks to projects like Anvil and Trinket and all the others, we have a ton of Python 2 code in production.
Right?
Right.
Anvil's users have written a bunch of Python 2 code in the browser.
They've written the only code they could at the time, and it was Python 2, right?
And on the day we upgrade, they are quite reasonably going to wake up and expect their code to work as normal.
And so one of the biggest challenges of this Python 3 migration was that we actually did it while keeping compatibility with the Python 2 stuff that we already supported.
And we did this with bits of old grammar imported.
So there is a mode where it will accept the print statement rather than just the print function and that kind of thing.
And there's switches coming through the standard library for, you know, am I behaving like Python 2 or am I behaving like Python 3?
And that's one of the things about this migration that I'm most proud of, because it allowed us to move to a place where we can start, you know, moving forwards and filling out the standard library and keeping up with the fact that it's 2019 without breaking all the code that's already in production.
Yeah, that's super. And you've heard that from other projects as well, like NumPy dropping Python 2 support, Django dropping Python 2 support.
It's like now we're free to work on going forward and not double implementing everything.
Yeah, I guess we're still a generation behind that. We're at the point where we want to add Python. We need Python 3 compatibility, but we can't let go of our Python 2 users yet.
Not yet, yeah.
And the pain that they went through is what, you know, we will be maintaining parallel implementations of a bunch of the standard library.
And, you know, a whole bunch of flags in the compiler for some time yet.
And that is pain that was inevitable from the moment the Python 3 non-backwards compatible announcement was made, that every project that's supporting this is in some way, shape or form going to have to go through this caught between two stools moment.
But speaking of so much better than the alternative.
Yeah, absolutely. I mean, it's extra tricky for you because your platform runs their code.
It's not like, well, you just leave Python 2 on your server and you just run your Python 2 stuff there.
Nobody's going to touch your server.
But with Anvil, someone's going to touch your server because it's your job to make it run and update it and things like that.
Right. So this is challenging on your end.
Certainly, we were the most vocal people going for this compatibility approach.
But, I mean, people like Trinket, again, right?
Definitely. For Trinket, it was not an option to have two sculpts and load two of them.
Like one for new Trinket and one for old Trinkets.
That was never an option.
So from their perspective, it was like, if we're ever going to move to Python 3, it has to be in the same code base.
And that's a consequence of the way the sculpt is built, right?
Because the way you interact, you build the environment that your user's Python code can interact with,
means that the JavaScript form support that makes that happen has to talk to sculpt.
And it has to talk to pieces of sculpt APIs that will evolve as sculpt evolves.
And so if you tried to interact with some sculpt two fork and sculpt three at the same time,
you would rapidly lose your marbles.
So, yeah, for all these projects that have, you know,
yes, sure, there are commercial people expecting the code behind their products,
shipping lots of money to still work tomorrow morning.
But equally with Trinket, right, there are a whole bunch of teachers and kids
expecting the thing that worked yesterday to work tomorrow.
Right. And maybe even textbooks with printed stuff that is super hard to update.
Yeah.
That says, go here and type this into that website and it'll work.
And it's our responsibility as platform maintainers to ease the passage.
Yeah, for sure.
All right. Final topic.
And Meredith, you know, I've spoken about this and some of the stuff here is a little surprising to me
or I didn't realize it or whatever.
So I think it's interesting.
But I'll give you the first word on this.
It seems to me like a lot of the energy around some of the stuff we talked about.
Pyodide is in the WebAssembly space.
And certainly any of the ASM.js and Scriptum solutions should go down the WebAssembly direction.
But what about Sculpt?
Does it make sense to have a WebAssembly version of Sculpt?
Is that something you all are thinking about?
What's the story there?
I don't think that we would write the compiler in WebAssembly or make a compiler that compiles a compiler in WebAssembly.
So what might happen is that we might support, you know, libraries that are written in WebAssembly
or some of like the big switch statement that Sculpt produces might be in WebAssembly.
But the biggest problems there is that the interop between JavaScript and WebAssembly is,
from what I understand, mostly via post message.
So very, yeah, very slow, slow.
Not in Sculpt world, nothing is slow quickly.
So Sculpt is quite slow by itself.
No, it's not the problem that it's slow.
The problem is that Sculpt requires that switch statement to be pretty synchronous.
And so waiting for a function call that would, you know, you would have to put that in suspensions.
I don't see there's no obvious path for Sculpt to move to WebAssembly, although given enough time and it might happen,
stuff might happen in that space.
But I sooner see it as external libraries being written in WebAssembly.
Okay.
I think that's exactly right.
For as long as you have to, like, use a knife and fork and oven gloves to communicate between WebAssembly and JavaScript,
then it's going to not make a lot of sense for something like Sculpt to be in WebAssembly.
But the idea of native libraries is very appealing.
And there are, again, Mozilla taking the lead on a lot of this,
there are a whole bunch of rumblings about improving the interop story between WebAssembly and JavaScript.
And if that arrives further down the line, then it could start to make more sense for us.
It would be an interesting development around the core hot loop of, you know, process each bytecode.
And also, it might be a way to bring in the C libraries, right?
You could ship that as a WebAssembly instead of a C implementation, for example.
Yeah, exactly.
Okay.
Yeah.
Honestly, it was surprising to me when we first spoke, right, what I was referring to,
is that the interop was so limited between WebAssembly and JavaScript.
I figure, like, it's probably just passing a pointer.
It's all the same, right?
Kind of like PyObject pointers or whatever, right?
I guess not.
I think the hype about, I think there's, the WebAssembly is clearly, right,
that is the opening of the door to browser-side runtimes that are not shackled to the JavaScript
way of doing things, by which I don't just mean JavaScript the language, but I mean that
whole stack I was ranting about earlier.
And there's a whole bunch of people who see the light behind that door and are getting very
excited.
But the door is not all the way open yet.
There's still a lot you have to do and a lot you would have to do to deploy, really,
any non-JavaScript-shaped thing into the browser in a way that's not like a native game engine.
Sure.
Right.
Where it's entirely contained within WebAssembly, right?
Something like that.
Something like Anvil is, like, 90% interop, right?
The reason you're writing that Python code in the browser is to drive visual elements on
the browser.
And so, actually, I would be unsurprised if, you know, even if we put in all that enormous
amount of work in the current state of things, it wouldn't necessarily even improve performance.
But that stuff is coming, and I think the world is moving in the right direction.
We just have to hang on.
Yeah, sounds good.
All right.
Well, thanks for that.
Okay.
I think we've probably used up all the time that we have to talk about Sculpt and this Python
in the browser world.
Well, let me ask you really quickly here, each of you, the two questions at the end of
the show I always ask.
Albert, we'll go with you first.
If you're going to write some Python code, what editor do you use?
PyCharm.
Right on.
These days, of course, a lot of the time I use Anvil's built-in editor, but PyCharm is
still the gold standard, right?
It's, you know, when I'm writing my autocomputer, it's their autocomputer I have my enemies eyes
on.
How does it work over here?
Dang, that's nice.
All right, let's do that.
I agree.
Cool.
And then reverse order, notable PyPI package that maybe folks don't know about, but you've
ran across.
You're like, this is amazing.
I should tell people about package X.
Right?
Postal.
Like, just transforming user input into valid postal addresses for basically anywhere on
Earth.
It's one of those things that the task that will drive you wild, except you can just pip install
it and then somebody has solved this whole scary problem for you.
It's awesome.
It downloads enormous quantities of data, but it's awesome.
Python bindings for lib postal for fast international address parsing and normalization.
Oh, it sounds great.
Albert?
I'm going to falter on this question, to be brutally honest, because I've not been a Python
developer for as long as I've been a Sculpt developer.
I kind of went the wrong way around.
I started as a JavaScript developer and then learned Python because of Sculpt.
Well, I think that's like kind of the reality of it, right?
Like you think of the core developers as being super Python developers, and they are, but
a lot of their day to day, I think, is to write C code, you know, to make Python go.
And for you, it's you've got to write JavaScript to make Sculpt go, right?
Well, my day job is actually in Python.
I pip install and all this stuff, but I don't pick packages that often.
So I don't have any interesting new packages to display.
I'll throw one out for you that people might not know about.
HTTPX.
Have either of you heard of that one?
Oh, well.
So you may have heard of something called Request.
You've heard of that, right?
Uh-huh.
Yeah, everyone has.
So this is a 100% API compatible thing with Request, but it adds on HTTP2 support.
It adds on async and await support.
It has some cool intermediate like background parallel work for like doing a whole bunch of
like requests and stuff.
There's a bunch of sort of modern HTTP support on top of the Request API.
Sweet.
Yeah, it's pretty cool.
That's a good one.
I love the async and await stuff.
I can't wait to add that to Sculpt.
Yeah.
That'll be super.
Yeah.
One of the wonderful things about opening that grammar thing.
Now, all we need to do is just get that into the compiler and we have the technology.
I am so excited for what happens next.
That's super cool.
And I think async and await pairs super well with JavaScript anyway.
And I think it's even coming to the new version of JavaScript if it's not already here.
Yeah, it is.
Definitely.
Sweet.
All right.
Thank you all for being here.
I'm going to give you one more chance for a final call to action.
People want to get involved with Sculpt.
Either contribute to it, use it.
What do you guys say?
What should they do?
Come find us on GitHub.
Sculpt.
S-K-U-L-P-T.
All right.
Super.
Albert?
Yeah, I don't have much to add to that.
You know, try it on the website, sculpt.org.
Super.
All right.
Well, thank you both for being here.
It's great to chat with you.
All right.
Bye-bye.
This has been another episode of Talk Python to Me.
Our guests on this episode were Meredith Loof and Albert Jan Nyberg.
And it's been brought to you by Linode and Tidelift.
Linode is your go-to hosting for whatever you're building with Python.
Get four months free at talkpython.fm/linode.
That's L-I-N-O-D-E.
If you run an open source project, Tidelift wants to help you get paid for keeping it going strong.
Just visit talkpython.fm/Tidelift, search for your package, and get started today.
Want to level up your Python?
If you're just getting started, try my Python Jumpstart by Building 10 Apps course.
Or if you're looking for something more advanced, check out our new async course that digs into all the different types of async programming you can do in Python.
And of course, if you're interested in more than one of these, be sure to check out our Everything Bundle.
It's like a subscription that never expires.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I'll see you next time.