Jupyter Notebooks and JupyterLab have to be one of the most important parts of Python
when it comes to bringing new users into the Python ecosystem.
And certainly for the day-to-day work of data scientists and scientists in general
who have made some of the biggest discoveries of recent times.
The Jupyter platform has recently gotten a major upgrade with JupyterLab 4 released
and Jupyter Notebook 7 being significantly reworked
to be based on the changes from JupyterLab as well.
We have an excellent panel of guests.
Sylvain Corley, Frederick Collinval, Jeremy Tolup, and Afshindarian
here to tell us about what's new with these and other parts of the Jupyter ecosystem.
This is Talk Python To Me, episode 438, recorded October 16th, 2023.
Welcome to Talk Python To Me, a weekly podcast on Python.
This is your host, Michael Kennedy.
Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython,
both on fosstodon.org.
Keep up with the show and listen to over seven years of past episodes at talkpython.fm.
We've started streaming most of our episodes live on YouTube.
Subscribe to our YouTube channel over at talkpython.fm/youtube
to get notified about upcoming shows and be part of that episode.
This episode is brought to you by Phylum.
Developers and data scientists are the new high-value targets.
That's you.
And yet this group remains severely underprotected.
Phylum detects, reports, and blocks malicious packages.
Get their research and join their community at talkpython.fm/Phylum-research.
And it's brought to you by Python Tutor.
Visualize your Python code step-by-step to understand just what's happening with your code.
Try it for free and anonymously at talkpython.fm/Python-Tutor.
Hey, Darian, Jeremy, Sylvan, and Frederick.
Welcome to all of you to Talk Python to me.
It's a great time to be here.
Thanks for having me.
Yeah, thanks.
Thanks, Michael.
Yeah, you bet.
I'm honored to have so many Jupyter team members, people around here, and all the work you've
done on the different parts of Jupyter Notebook, Jupyter Lab, Jupyter Lite, all the things is
really fantastic.
So I'm looking forward to celebrating the new releases.
They're not quite as new as when we first started talking about this.
It took us a while to put this many people together on for an hour, but yeah, it's still,
it's going to be great, right?
Sounds good.
Yeah, for sure.
So before we get into the topics, maybe just a quick introduction for each of you here, and
we'll go around the Brady Bunch circles in the video.
Darian, we'll start with you.
Sure.
Hi.
So I'm Darian.
I work at Quantstack, along with everyone else on this call.
And I have been working on the Jupyter stack for about eight years now.
And I originally started working on what became Jupyter Lab.
And nowadays, in addition to that aspect of things, I'm also part of the Jupyter Executive
Council, which is one of the governing bodies of the project.
As of this year, we actually changed our governance structure.
And so each year there are elections and we have this new set of governing institutions
that I've been a part of.
That's really fantastic.
You know, there's a lot of projects where there's one maintainer who is primarily the
maintainer.
Then there's projects that get put under an organization, you know, like Flask with pallets.
But there's not very many open source projects like this that have governing bodies.
You know, Python itself comes to mind.
Django.
That's awesome.
I think, you know, Jupyter is such a vibrant ecosystem.
It's an exciting space, for sure.
Yeah, absolutely.
All right, Jeremy, quick background on you.
Yeah, I'm Jeremy.
And I've also been working on the Jupyter stack for a couple of years.
I think something around five or six years.
I started a bit like on my own first, like a spare time thing, you know, contributing to
projects on GitHub before joining Quantstack.
And currently, I'm mostly involved in the Jupyter project.
So, the Jupyter notebook 7, wallet dashboards.
Also, a bit of iPad widgets.
And most recently on the Jupyterite.
Nice.
Hi, everyone.
So, yes, I'm Silva.
I'm the CEO of Quantstack, which I founded in 2016.
I've been working on the Jupyter stack for about 10 years now.
So, yeah, it's been a while.
I think it's almost the anniversary right now.
And in the Jupyter ecosystem, I mostly worked on kernels, widgets, the Jupyter server, and a
tiny bit on the front-end pieces as well.
Excellent.
So, it's a Jupyter space over there at Quantstack.
Top to bottom, right?
That's right.
Awesome.
Hey, Frederick.
Welcome.
Hey, everybody.
So, I'm Frederick.
I'm working at Quantstack, as everybody else.
And I'm working mainly on JupyterLab.
I'm actually the representative to the Software Steering Council of Jupyter for that subproject,
mainly.
And I started contributing, like a bit like Jeremy, five to six years ago, trying to
convince colleagues in my aeronautic company back then to use Python and Jupyter instead
of Excel.
You're fighting the good fight.
That's awesome.
Maybe I could ask you all a quick question, since it might give people an interesting view
into the world of Jupyter.
It's just, you know, there's a lot of reasons people get into programming.
But what attracted you all individually to the Jupyter ecosystem itself?
Darian, how about you?
So, I got lucky.
I liked the idea of working on open source software.
And I'd worked at a few startups that released open source software.
But they were startup projects that were driven primarily by one company.
And some of them had community in the sense that there were lots of users and there'd be a
lot of traffic, say, on forums and stuff.
But not community in the sense of a public-minded group runs this thing and makes decisions.
It was the decision was at my employers.
And nonetheless, I liked it.
And I just sort of accidentally happened on working with the team that was building a new
front end for Jupyter because I liked JavaScript and other people in the Jupyter community did
not.
So, they had to bring from outside the Jupyter community in because Python people at the time
tended to sort of hate JavaScript.
Now, I think that's still the same.
So, I got into Jupyter because I like JavaScript.
But I stayed in Jupyter because I liked a community open source project that was doing something
that I didn't really know you could do, which is to mix your research and your results and
the pros that you write and the output of the analysis that you do all in one document and
share it with people and they can reproduce it and they can run your code and they can tweak it and
they can do all those things without ever going through what I thought were the necessary
prerequisites to writing code.
It turns out none of those were necessary.
You could make it way easier for people.
So, I liked working on that.
I like the community.
I like the people.
And that's why I'm still here.
Yeah, excellent.
It is pretty unique of an idea to say we're going to have kind of a document, we're going
to have results, and we're going to have code, and it's going to be live.
And that's very special.
Jeremy, how did you get into Jupyter land?
Yeah, I started as a user.
I think back in the days when it was still the IPython notebook, like all together, just
Python programming language.
I think that was really this idea of being able to write normal text alongside code and
have this kind of narrative with charts, interactive graphs, and so on, and share that to people.
So, I used it on my own for personal projects, but also at different companies to share reports,
then communicate results more easily.
And after a while, you know, when you start using a tool, you want to make fixes to it
or improve it.
So, that's why, that's how you can go to the side of a contributor side, start submitting
a pull request and getting in touch with the community.
Excellent.
Sylvan, I'm going to come back to you.
Frederick, let's have your story.
So, that's a bit like Jeremy.
As I said, I try to convince people that something better than Excel exists out there.
And that's when I discovered Jupyter.
And what was really interesting was the ability to, like, as Ashton said, like, mixing very
nice and advanced visualization with text and being able to do that interactively so that
in a meeting, you don't, like, have to remember which tab in your Excel is the right result.
It's like, yeah, you just type two lines of Python code and you get a nice graph to display,
like, the points you tried to make from the data.
And that was it.
And as Jeremy then, at some point, she said, hey, I can do some stuff better and I can contribute
because it's open source.
So, I put my finger in that and now all my body is in it.
Awesome.
I think one of the ironies is, and, you know, Darian touched on this actually, that to use
Jupyter often is to write Python, but to contribute Jupyter is often to write JavaScript, TypeScript.
Like, it's, you're kind of taking one for the team, doing the JavaScript that no one else
has to do, right?
And that's a little bit different, isn't it?
I think nowadays with the maturity of TypeScript as something that serious, large projects can
use, it's a little bit better.
But, you know, it wasn't obvious to us as a foregone conclusion that we should use TypeScript
back in 2015.
So, I think over time, that problem has somewhat attenuated, but it's not, we're not in the
clear.
We do definitely have a lot more people who would like to hack Python than would want to
contribute to a gigantic TypeScript plugin-based, you know, front-end application.
Yeah, exactly.
They come to Jupyter because they're like doing Python, they get really excited.
Like, I want to be part of this, I want to help.
Oh, wait, hold on.
It's a bit of an irony.
Or maybe even WebAssembly, as we'll talk about, Jeremy.
All right, Zilvan, back to you.
So, I wanted to change the question a little bit for you and just maybe have you talk about,
like, what are the benefits of having QuantSight, like the whole company, being so in on Jupyter?
Like, how's that affected you all?
Is that for your business?
This is really how it started, actually.
So, I was a Jupyter contributor back in a previous job before I founded QuantStack.
And I really, QuantStack is almost an accidental startup in that I started the company as a means
to continue working on the project being paid by, as I was a core contributor already, being
paid by clients on consulting contracts and whatnot to continue working on this.
And very quickly, we had enough business that I could hire one other person and then another
and then another.
So, we bootstrapped from day one.
So, our success, I would say, is really bound to the success of the project.
And I think it's been a positive feedback loop since the very beginning.
Excellent.
Yeah.
I'm sure it brings a real advantage when people come to you to look and sort of see like,
yeah, this is the company to help me with, you know, Jupyter type of change, maybe adopting
Jupyter or some advanced deployment problem solving.
Yeah.
This, I mean, we're not trying to be the Jupyter company.
There are other groups and we really have a consulting business model and that people can really come
to us for our skills and expertise.
And it turns out that so far, we've been able to contribute back everything that we've done or nearly everything.
In some occasions, we have to do something that's very ad hoc to a client and, you know, custom development that requires using some of their internal APIs or whatever.
But like, I would say that 99% of our work is out there.
So, yeah.
Excellent.
Awesome.
Let's start talking about just a recent event here.
A couple months ago, it was JupyterCon.
I think the videos, no, the videos have been up for two months.
So a little, it's been around for a little while, but multiple of you, I don't know if all of you, I haven't gone through all of the videos here,
but, you know, there's 86 sessions posted for JupyterCon 2023.
There's a ton of good stuff.
Maybe you all could just speak to the talks that you gave there or some of the real interesting things that you want to, people should maybe check out.
Because if you just get into Jupyter, you can learn a lot by watching some of these talks, I think.
Our recommendation, Jupyter Notebook 7 was talk where Jeremy and a few other contributors to the project introduced the new version of Jupyter Notebook.
And usually when you go from version 6 to 7 of a software package, it's, you know, an iteration.
But in this particular case, it's a whole new application with a whole new architecture that has the same experience of the Jupyter Notebook as version 6,
but has all new architecture and can use the same plugins as Lab.
And it's pretty exciting.
So I would recommend checking out that talk.
That was a good one.
That's also something that's main topic of the show is sort of this interplay between JupyterLab and Jupyter Notebook and both of the big new releases,
which is, yeah, it's a big change, isn't it?
Bigger than just the 6 to 7 would indicate.
Yeah.
There are also plenty of other videos.
So there are quite a few, let's say, technical videos or more like product updates, as we call them.
Like each sub-project has a talk where they discuss and present their new features or what they've been to the past few months.
But there are also quite a lot of interesting talks on use cases so that we can learn how people use Jupyter in a while,
like at companies, also in education.
And those are very inspiring because they also give people ideas on what they could do with Jupyter in general.
Well, yeah.
Well, I certainly recommend people check this out.
And then there was a talk that you, Darian, and Martha gave introducing JupyterLab 4, right?
Yes, that's right.
JupyterLab 4 is also the first time we did one of our keynote product launch talks with the desktop version of the application,
just kind of to expose people to this new way to use JupyterLab, which isn't super new, but I think is new to most people.
So we thought that would be a good opportunity, not just to show the new JupyterLab 4 UI and its new features,
but also to show you can download it onto your desktop, double click a notebook, and open it up in JupyterLab.
Yeah.
This portion of Talk Python to Me is brought to you by Phylum, your Python developer security partner.
Developers and data scientists are the new high-value targets.
That's you.
Yet this group remains severely underprotected.
Attackers know that developers control the keys to the kingdom, SSH, GPG, and signing keys,
API tokens, infrastructure secrets, and proprietary source code.
Do you use open-source dependencies in your code?
Of course you do.
We all do.
Vulnerability exploits in open-source code are rare.
But software supply chain attacks via malicious dependencies are unfortunately kind of common now,
and these attacks can compromise developers and organizations.
Phylum detects, reports, and blocks malicious packages,
while other solutions you may have heard of merely look for known vulnerabilities.
Of course, Phylum also reports on vulnerabilities, as well as author engineering and license risks.
But critically, Phylum doesn't wait for a CVE to be published before alerting on bad packages.
Phylum provides integrations to protect developers where they live,
guarding PRs and CI environments,
and with CLI and pre-commit hooks for local development work.
Help you and your organization's Python security game with Phylum.
Get started today.
Subscribe to their newsletter to get Phylum's research sent directly to you,
or even sign up for their service for free.
Visit talkpython.fm/phylum-research,
and enter your email in the subscribe to our research field in the middle of the page,
or create your free account by clicking the sign up button in the nav bar.
Stay safe out there.
Visit talkpython.fm/phylum-research.
That's P-H-Y-L-U-M dash research.
Thank you to Phylum for supporting the show.
Let's set the stage about just what, you know, people, maybe they've heard the word Jupyter,
they've heard the word notebooks, plural, you know,
but they might not know the difference between JupyterLab, Jupyter Notebook.
Maybe you all could like sort of position those two side by side.
What are these two different things?
Are they the same?
Are they different?
Are they synonyms?
Okay.
There is a file format that you will see with the extension .ipynb,
because historically it was the IPython notebook file format, and we've kept that.
And those are called Jupyter Notebooks, commonly.
It's a generic term for that kind of file.
And then there is a browser-based application that millions of people use for rendering those
notebooks and writing them and authoring them.
And that application is called Jupyter Notebook.
And we just released version seven of that.
And there is a data environment that features notebooks, but also has lots of other functionality
that we call the other front end, and that's JupyterLab, right?
And JupyterLab has a notebook renderer in it, among other things, for Jupyter Notebooks.
And that same notebook renderer is what powers the application called Jupyter Notebook.
I think if I keep saying these words, you'll get more confused, so I'm stopping here.
In my mind, I sort of think of Jupyter Notebook as, you know, which is the single file view type
of thing, a little bit, not quite the right analogy, but a little bit like the Vim or the
Emacs user who likes just to have like a single file totally open and just focused on that.
Whereas JupyterLab is more like the IDE style.
I want all the tools and the extras.
I want to be able to get to my, browse my file system, go to the terminal, do all the different
things and have like multiple files open and work between them.
Is that a good mental model or am I not quite there yet?
No, I think that's right.
The Jupyter Notebook is a single document interface.
It is, I mean, even like Vim, it has a command mode and, you know, an edit mode and all those.
So it's not a bad analogy at all.
JupyterLab is just a really flexible environment to do a lot of different activities in your analysis
of any particular data.
It is a much more sort of multi-tool for that.
Okay.
Excellent.
Anyone else want to add more to that, those comparisons or the daring?
Yeah.
I mean, for the genesis of this, of JupyterLab, I mean, the first tool that everybody saw and
the whole world saw in the beginning in 2014 was the notebook, right?
And got a lot of people excited, but there was a, like a very recurrent request for bringing
together in the same page, the text editor, the terminals, the having a console as well,
a regular Python console alongside the notebook in the same, at the same page.
And so this is what we delivered with JupyterLab.
And even though it was a very popular, like common demand from the community, a lot of people
really loved this more rigid full page notebook UI and, you know, kept using the notebook.
So this is where we were about two years ago when we started the discussion about notebook 7 and
modernizing, you know, the classic notebook user interface.
Was there ever a time where people were thinking to have JupyterLab be the next version of Jupyter
notebook and kind of replace it?
And then the people are like, you know what, maybe please don't take that away.
No, it was definitely at some point, our strategy to deprecate the notebook eventually.
And we had a large public conversation about this.
And it just, I think is clear that people like having that choice.
People like having what is just a simple editor for notebooks.
And people also like having the Swiss army knife.
And they're not always the same people, but we want everyone who is using these things
to be able to get both the benefit of the modern architecture and the updates and also the user
experience they want, right?
So if most of the plugins that come out lately are for JupyterLab and you're feeling more and
more left behind, that's not a great experience.
And so deprecating that tool you're using and saying you must use this other one, it just
isn't what our community needed.
So now we, I think, have an architecture that we can sustainably maintain because it's
effectively the same code base.
But we also can give users what they want because there are different contexts.
And it is the case that we would have left some people behind and it's better not to.
You can imagine in different scenarios, different user bases, it might make a lot of sense to
have one or the other, right?
If I'm a data scientist working on complicated projects, maybe JupyterLab is the right place.
But if I'm trying to teach a little bit of computation for a psychology class and I just want them to do
three plots and pull in some data, you don't want to drop non-developers into something that looks kind of
complicated.
like, where do I go?
Here's your page.
You type in these cells, you see the picture, you get the graph for the study we're talking about or something.
Right?
Yeah, that's exactly the idea.
Yeah.
Mm-hmm.
Interesting.
You touched on something interesting there that Jupyter Notebook, the single page or the single view version,
is now based on JupyterLab.
And there's a lot of overlap of these things, right?
Yeah.
I don't know if it was the right time to go more into the details.
Yeah.
Give us a little, it's a bit of a jump ahead, but just give us a sense that people who,
maybe people are listening, go, ah, I'm not going to use JupyterLab.
I'm kind of the notebook type of person, right?
There's a lot of what we're talking about that still directly applies because there's a bunch of similarity or actually reused, right?
So one thing we didn't mention yet about JupyterLab is that, so for end users, it is indeed a more advanced environment, a bit more like an ID, even though it's not really an ID.
But for developers, it's also a framework.
So JupyterLab itself is just a set of plugins and extensions.
They were designed from the very beginning to be reused in, you know, in different ways.
So like, if you actually, if you go to the JupyterLab repo, there is an examples folder.
And in there, there are a few different apps that you can build yourself.
And they are not JupyterLab.
They are like just a single cell on the page or maybe just, yeah, this one, maybe just a console, maybe just a notebook.
And this is made possible because JupyterLab is indeed just a set of extensions.
These examples, they just reuse some of them and they just put them together in a different way.
So we sometimes call them remixes.
And in the end, Notebook 7 is a JupyterLab remix.
It's just a different JupyterLab interface, but focused on a document-centric experience that, you know, users have been liking for many years already.
So, yeah, that's a bit of the second side of JupyterLab.
Excellent.
Yeah.
And there's a bunch of features that we're going to talk about here that come to Jupyter Notebooks in 7.
Yeah, 7 can get the right version for the right product.
Because of what you all have done here, like interactive debugging and real-time collaboration and things like that, yes?
Yeah, all of that was really a byproduct of moving over to the JupyterLab architecture.
And one other key aspect of Notebook 7 over Notebook 6 is that since we use the text editor from JupyterLab 4, which is a newer version of CodeMirror, we also get a lot of benefits in terms of accessibility.
Most of the work on accessibility has been going into the JupyterLab codebase and not into the classic Notebook, which was really based on older versions of, you know, older libraries and, you know, JavaScript from 2010s.
And so we're in a much better shape on this regard with JupyterLab 4 and Notebook 7.
It's a different world with JavaScript these days than it was in 2010.
That's right.
Quite a bit.
Let's start by talking about some of the new features for JupyterLab.
One of the ones is for very large notebooks.
JupyterLab is quite a bit faster, right?
That's excellent.
Who wants to take that one?
I nominate for it.
Yeah, I can go ahead with that one.
Go ahead.
So, yeah, that's one of the complaints of the user is that when you get lots of notebook open or when you get some notebook that are very lengthy, the UI responsiveness can be low.
And so we decided to focus quite some time on that aspect.
And that was made possible by fully upgrading the text editor to the latest version.
So we are using CodeMirror.
It's a awesome third-party library for dealing with text editor.
And we upgraded to the latest one that brings up a very important speedup.
And also we have worked on the styling to try to optimize it because styling can hit your browser pretty hard.
And lately, we have also introduced a new way of rendering notebook, like only rendering what the user is seeing, basically.
That one is still experimental.
So by default, it's off and user can try it.
And we are still working on that one to get it on by default for the next minor version, so for 4.1.
And hopefully, when we reach that milestone, we'll get like a very smooth experience for 99% for a user.
That's the goal.
Yeah, it's tricky because web browsers typically want to just have the whole document, the DOM loaded.
They want you to just work with it.
But if you've got 100 virtual pages, right, that you would go through, you don't want to style all those and do all the widgets and all that if they're not on the screen, right?
So there's this tension.
Yeah, exactly.
And sometimes it can come from tricky part, like just displaying for education, for example, the use, and that's very nice, a lot of equation, nice rendering.
And those are creating so many elements in the web page that sometimes just having not so long notebook with lots of math expression can crush your browser, basically.
That's surprising at first.
All right.
Right.
Okay.
So here's what I'm hearing is like complex math, like calculus, differential equations, those kind of things.
Not just hard for people, but also hard for web browsers.
Math is hard for web browsers.
Got it.
That's pretty interesting.
Darren, you showed in your talk, you showed this off in a video, a side-by-side, JupyterLab 3 versus 4 with a large notebook, and it was much faster.
Did that video have the virtual paging, like the render, the scene culling thing?
Where it doesn't show, it doesn't render the stuff it doesn't see, or is that just the first default version right now?
This is where I come clean to your audience and say, Fred is the person who produced and recorded those videos.
And Fred was originally going to give that talk.
I can tell you that, yes, that's exactly what's going on, but Fred can give you the actual details because he's the one who hit the record button.
Yeah.
Tell us, Fred.
I have a very happy event that prevented me to go to the JupyterCon.
So I was, normally, I was the one that had to do the talk with Marta, but we had to switch last minute and thank again, Afin, for doing that.
But definitely, the video that was displayed was that it's a comparison between the latest 3.6 version and the 4 version with that feature that's off for now by default.
That will bring the best performance for the end user.
Yeah, it was a massive speed up.
So, you know, on one hand, you're like, ah, you're just waiting 10 seconds.
But those kinds of little delays, they just kind of break your thought and your flow.
And you just feel like the tools are getting in the way.
And if it's really, really quick, then the tools kind of vanish to the background and you don't think about them, right?
That's how it should be.
Even when rendering all of the notebook and without enabling the virtual scrolling, we get already a nice speed up already, thanks to the adoption of CodeMirror 6, which is much faster to render and a bunch of other optimizations that we were able to do.
But yeah, I mean, it was the easiest clap in the entire talk was just playing that video and seeing the fast one render and the slow one keep going.
It's a really different user experience, even though, as you say, it's only a few seconds.
It turns out a few seconds is enough to cause the kind of interruption that leaves a negative overall impression in the user's experience.
Absolutely.
This portion of Talk Python to Me is brought to you by Python Tutor.
Are you learning Python or another language like JavaScript, Java, C, or C++?
If so, check out Python Tutor.
This free website lets you write code, run it, and visualize what happens line by line as your code executes.
No more messy print statements or fighting with the debugger to understand what code is doing.
Python Tutor automatically shows you exactly what's going on step by step in an intuitive visual way.
You'll see all the objects as they are represented in Python memory and how they are connected and potentially shared across variables over time.
It's a great free tool to complement what you're learning from books, YouTube videos, and even online courses like the ones right here at Talk Python Training.
In fact, I even used Python Tutor when creating our Python Memory Management and Tips course.
It was excellent for showing just what's happening with references and containers in memory.
Python Tutor is super easy to check out.
Just visit talkpython.fm/python dash tutor and click visualize code.
It comes preloaded with an example and you don't even need an account to use it.
Again, that's talkpython.fm/python dash tutor to visualize your code for free.
The link is in your podcast player show notes.
Thank you to Python Tutor for sponsoring this episode.
This is not a JupyterLab feature exactly, but while we're on this code editor topic, I wanted to ask you all just your thoughts about this.
Now, keep in mind, before I ask this question, my perspective.
I've come from doing a lot of traditional software development with editors like PyCharm and Visual Studio Code and others.
So when I work in Jupyter, Notebook or Lab, it doesn't really matter.
And I'll say some variable dot, it doesn't give me any help unless I press tab to see the autocomplete.
Like, is there an option or discussion around having like more aggressive autocomplete that kind of automatically shows up as you work with it?
Or is the community just like, we don't really want that stuff popping up around.
We want it.
We just want to type.
So the reason it's currently this way is primarily due to the behavior of Jupyter Notebook a long time ago and not subverting user expectations.
But we can often keep that as the default and have some setting to facilitate a different kind of behavior.
With this one, I think there's probably actual appetite for it.
But no one's shown up to do it.
I don't think we would turn down a feature like this hidden behind the setting.
I just don't think anyone has both been bothered enough and also been in the time and place with the resources and wherewithal necessary to do it.
But it doesn't sound, yeah, it sounds totally doable and compatible with both keeping the user expectations intact and offering you the feature you're talking about.
Because, yeah, people have different styles of completion that they're into.
And new things are appearing all the time and creating new expectations like LSP, which changes the language server protocol.
Yeah.
Anyone else have thoughts on that?
I would say that in 4.1, there are things coming to at least give you the ability to more quickly restore history.
Like you type and then it's going to propose you to copy-paste the code that was written previously in some previous cells and stuff like that.
And the same infrastructure will be available for plugins.
And I know some people are working on AI integration in Jupyter and some of them are already working.
And, okay, let's try to do a request when the user is typing to help him complete the code or giving me the hint on how to improve things.
And, yeah, there are a couple of people working on that.
So stay tuned for more.
Yeah, okay.
I really like JupyterLab and working it now.
But if you could just go and these are all the things you can do just as I would, you know, I would make me even more happy.
Darn, you mentioned settings.
And that's actually one of the new features as well as like a UI for settings.
Is that right?
Yeah.
So this is basically a front end that's capable of taking any extensions defined settings and rendering a UI for it.
Because we don't know what the preferences an extension author is going to want are going to look like.
We can't create like a custom UI for it.
Instead, we have to read a schema file that defines such like the extension author who wrote an extension that does Git, right, also has a whole bunch of Booleans that are by default set to true.
So this UI is like, oh, okay, cool.
I'll bring a bunch of checkboxes and check them.
And so it's very flexible.
It can take any extension settings.
And as long as they're described properly according to the API, they will render inside that UI.
And it really beats what we used to have there, which is just a JSON blob editor.
Here you go.
Here's your JSON.
Good luck.
And we tried to make it slightly nicer.
I think we tried to make it support a version of JSON that has inline comments as well.
But like, it's still not that nice.
So I'm very happy to see this as the way you go playing around with all the choices you have.
Because JupyterLap has a lot of settings, but it's really difficult to explore them without something like this.
Especially with all the extensions.
Yeah.
Kind of unbounded set, right?
Exactly.
Yeah.
The settings UI has a search that searches through all the different areas for settings.
It's one of my favorite things on complex apps.
And you're like, I don't know what is in here, but I bet if I start typing, it'll reveal itself if it's in there somewhere, right?
Yeah.
JupyterLap has two places like that.
One is the command palette also lets you search the entire menu system, basically.
And that's become, you know, in other ideas too.
I mean, I think there's a certain point as a user of these tools where you discover something like that.
And then after that point, you don't want to go back.
So I think you're right.
Like I didn't used to always need search in my applications, but now I do, just like you say.
Yeah, exactly.
This really goes into the thing we were mentioning earlier about JupyterLab remixes.
So as you know, JupyterLab is basically only a collection of extensions, right?
And the notebook itself, the command palette, all of these things are extensions to the core app, right?
So remixes are a way to cherry pick some of these core extensions, potentially include third-party extensions into a new thing.
I think that there are not many equivalents to JupyterLab in that regard.
If you try to make a new complex application in the web, like a photo editor, a CAD engine, anything that falls into the category of having to display a lot of information on the screen, doing something that, you know, you spend hours doing every day.
You will want a framework that allows extensibility, theming, a command palette, unified keyboard shortcuts, you know, across the application, a settings editor for your main app and all of your extensions, configurable keyboard shortcuts.
I mentioned that earlier, potentially real-time collaboration.
And these are lots of checkboxes to check if you are starting from scratch.
And so the JupyterLab, basically, application framework allowed us to reuse all of that work for other things that have nothing to do with the notebook, for example.
So earlier this year, we released JupyterCAD, which is a web-based CAD modeler, which is a JupyterLab Remix, really.
It can be used both at the JupyterLab extension where you find everything that you're used to or in a standalone fashion.
And this JupyterCAD is really interesting.
It's a 3D CAD curated design editor that has, that's built on top of the JupyterLab framework, but it's not itself exactly about Jupyter, like programming and so on, right?
And it has really interesting things, like it has real-time collaboration between multiple people.
And all of that stuff comes from just building on top of this, like you said.
That's a good example of something really different, but still possible.
Yeah, so in the way we enable real-time collaboration in JupyterLab, we have to specialize a few things depending on the type of document that you're editing, but really the framework is generic.
And you can start making, enabling real-time collaboration and other types of content than just textual content or whatnot.
And we all got used to real-time collaboration and collaborative editing on text, you know, on the sort of software suite that we use for basic productivity.
But very convinced that this will be the case for more technical computing, not just code, but everything where you have to put together a really complex system.
Like, especially if you are building something that requires a lot of people to coordinate.
Like, you're making a new plane or football stadium, and you have this unified model where everything percolates across many, you know, domain-specific applications from CAD to, you know, electrical systems or whatnot.
And for people to effectively collaborate in these kinds of environments, I think collaborative editing is going to bring a lot if we can implement this vision.
So, really, JupyterCAD is a demonstrator that you can do collaborative editing on other things than just notebooks.
Yeah, JupyterLab may become the foundation for things that we would never expect to be part of programming, like you're talking about.
It's interesting.
Yeah, it's more technical computing than just, you know, interactive computing, I would say.
Yeah.
I was thinking about this book here.
Like, oh, man.
This book is thrown around as, like, by Fred Brooks, basically saying, like, there's no project.
We're adding more and more people.
It's going to speed it up and get it done more on time, right?
It was all about sort of the follies of, like, well, as you add more people,
the complexity of the communication and the overhead of syncing and all of that is kind of going to break everything.
And so, you know, you can't make a baby in, like, one month by having nine women work on it sort of story.
You know, like, but the stuff that you're talking about here and all of these communication tools and all of this interesting async stuff that we're doing these days,
I kind of feel like people should rethink these ideas.
And I think kind of what you're touching on here is, like, it is possible to get deeper, important work done as groups than maybe the mythology that people think about in the software world traditionally has been.
It's a different world than 2023 and on.
I mean, it's kind of a sidebar of it.
It's also important to what this collaborative editing.
Yeah, I like this book.
So for a long time, we have been trying to introduce real-time collaboration into Jupyter.
We've had at least three significant technical attempts at it, right?
It's a hard problem.
It's one we've worked on a long time.
And it's given me a unique opportunity that most people don't have, which is trying to justify to myself over the course of six years why anyone should want real-time collaboration inside a programming environment.
And what I can say is, over that time period, I have spent, I don't know, hundreds of hours inside documents with other people that are Markdown files, because that's how we conduct our meetings, or inside Google Docs, because that's how we wrote our governance, or asynchronously back and forth on GitHub, which is not exactly the same thing, but truly is collaboration.
And so I don't think there's a tension between there's a group of us working on something in a room, and we want technical tools so that we don't all have to be in that room, but we can have that seamlessness and the myth of the mythical, you know, the man month, right?
I actually think those problems are, like on the surface, they look pretty similar, but I actually think they're not the same thing.
It's more like, back when that book was written, groups of people would get inside one room and work collaboratively.
Today, every one of us on this call is in a different location, and some of us are pretty far apart.
We're in different countries, and it's all working out fine.
And Jupyter is bringing tools like that to people, and less tools like, let's add 5,000 people.
Now, what Sylvan was talking about, still, you know, you're not going to have one person opening up the grand document that describes like the whole Coliseum, right?
But there's going to be tons of groups that are working on systems where there's like seven people who need to work on this thing together.
And Google Docs for wiring documents doesn't exist yet, you know, but it could exist.
Yeah, and it could be built on Jupyter Lab.
That's right.
One of the big features is this real-time collaboration.
Let's, you know, let's talk about that for a moment.
I think that's, I think it's a big deal, as I've been hinting that with my sidebar and so on.
Well, in the beginning, I think we need to start small.
Even with, you know, Google Docs, when it came out, I think in 2006, a lot of, many people didn't see the point.
Like, why would I want someone else's cursor in my document?
Like, it sounds dangerous, right?
So if you were to make a survey of what people wanted at the time, nobody would have said that, right?
Yet everybody does that now, right?
And it's probably, people probably have the same feeling when it comes to writing code or doing any sort of technical work.
And I think we'll get past this point where people will get it.
And in the very beginning, this can start being useful for peer programming, doing something together while we're reviewing code over a Zoom meeting.
That sort of thing where, you know, there is a, you know, real-time interaction between people over some kind of medium.
If you look at more technical objects, like, for example, you have two different teams.
Like, someone's making a plane.
There is a team of people that's doing some simulation and see how, you know, the plane goes through the air.
And another person is making physical prototypes of parts of the plane.
And you have this interaction.
You know, normally these two groups would be in two different branches of the company.
And in order to change any kind of spec, you would need to go up and down in the hierarchy by a few, you know, degrees.
Right.
Schedule a meeting for next week, that sort of thing, yeah.
Yeah.
And you change the specs and now everybody has to change or make, you know, changes to their, you know, boundaries.
And if you enable more real-time, you know, feedback on suggestions and whatnot, you may be become a lot more efficient.
So, it's pretty similar to Google Docs, excuse me.
You have to install the Jupyter Collaboration extension for the server, right, in order for this to work?
It's not just pip install Jupyter Lab and you're good to go?
Well, you can pip install Jupyter Lab and pip install Jupyter Collaboration if you need the one-liner.
But the idea was lots of people install for personal use on their own laptops.
And anyone who's going to create an environment where collaboration is happening is doing more than running just one command to install Jupyter anyway.
Yeah, yeah.
Of course.
You know, so, like, as a toy, if the five of us are in the same network, we can launch it that way just to play around with.
But really, any sophisticated user is going to tie in the user system and do a whole bunch of config and all those things.
And so, we're not shipping the code that does that inside core Jupyter Lab for two reasons.
One is that, you know, the main sort of install base doesn't do real-time collaboration.
The other is that we're working on that package much faster than we're working on Jupyter Lab core.
And having them in separate packages allows us to bump versions if we need without carrying that much and to add new features and to just, you know,
it's a smaller subset, it can move way faster.
Yeah, that's a great design philosophy process because there was a conversation like this in core Python and request.
They're like, we have a URL library in Python, but request seems to be the default.
We should move it in and be like, whoa, whoa, whoa, you're going to make us ship request once a year.
We don't want to do that.
This is not, like, this is changing fast and it needs to be more adaptive.
And that sounds like a similar deal here.
Excellent.
Let's hit on some more of the new things.
So we've got one of the things that's new is cell toolbars.
There's like when I'm editing a cell within the little cell, there's a toolbar with some new features.
Anyone want to talk about that?
It's exactly what you said.
One question often asked by extension authors is, okay, I want to do a certain thing to a specific kind of cell, right?
Like I want to somehow pull out this cell and do something extra with it.
Where do I put that button?
Or, you know, it's the answer to that question.
Yeah.
All of the things that you can do were there before.
It's just meant to give users a discoverable way to learn maybe some capabilities they didn't know.
Or maybe even hover over the button and learn a new keyboard shortcut or something.
It's one of those quality of life features.
It's not like revolutionary.
It's more like just making this thing slightly more ergonomic.
Yeah, I was thinking when I saw that, I was thinking of, you know, the biologist who is doing their first semester of a class where they had to use Jupyter notebooks for their work.
And they're like, how do I just seeing those buttons there for the first month of working with it is helpful.
Then, as you point out, you learn the hotkeys.
Like I press B, I don't go over here with the mouse and those kind of things.
But still, I think it's a nice one.
Let's see.
What else did I want to give a shout out to that we haven't done yet?
Oh, collapsing.
Based on markdown headers.
That's kind of interesting that you can organize your document in a way to kind of expose parts or make it more of a report or more of a coding environment and these kind of things.
Who wants to introduce that feature?
The idea was to, especially when you have some document that can be long, like from education that has lots of text and stuff like that.
It helps the student, for example, to be able to focus or to scroll artificially, let's say, to the right place faster.
So that's one of those nice features.
And I think it's something also now we are used to because lots of classical Office text editor have that kind of ability.
So it was a nice feature to add to in JupyterLab.
Yeah, kind of related.
And it's probably going to sound really boring, but it actually is pretty awesome, I think, is the table of contents where it looks like it's showing the table of contents.
But if you say run all cells, it'll actually show you as you go.
It'll sort of like go down the table of contents showing where in like some big doc, if the running cell is off the screen, what it's working on currently, which is actually pretty excellent, I think.
And even soon, you would also report that one cell crashed and you should look at it.
That's coming up in the next version.
Okay, excellent.
What else?
Running kernels in the side panel sounds kind of interesting, like different kernels.
You can go create notebooks based on them and see all of what is at play.
I think that's probably worthwhile as well.
Yeah, so this is more something you would care about if you're running JupyterLab than notebooks.
So it's a sidebar in JupyterLab that just says, in the environment where you're running, here are all the kernels that you as a user can see.
And so you can even have kernels that might not be coming from your Jupyter server at some future date, right?
The idea being, why shouldn't JupyterLab be the place you interact with lots of different resources, wherever they might be?
And for the longest time, we thought of a running kernel session, for want of a better word, as intricately tied to a file, right?
The notebook that it's the kernel for.
But, you know, we've been moving away from that.
I mean, the same way IPython hits a kernel, but isn't really transient.
We have code consoles inside of JupyterLab that aren't really transient.
But you can take a notebooks kernel and open a code console on it so that you could do some scratch work, but like not modify your document.
So there's just having more insight into what's available in your environment is what that's about.
And again, it's not a revolutionary change.
It's just one of these sort of ergonomic features to make it a better multi-tool for being in an environment that gives you lots of different kernels to play with.
Like the sort of use case that we'd like to tackle with this is, for example, there are lots of desktop applications that embed a Python interpreter, like FreeCAD or Blender.
And eventually, if these desktop applications advertise a way to connect to these running interpreters with a Jupyter or kernel API, then they would be listed there.
So the JupyterLab could almost become a scripting front end to like a running Blender project or something like that?
Exactly.
So this has already been done, actually, in Kitware's Slicer 3D project.
So they run a kernel as part of their desktop application.
And so we want to streamline the use case and make it more easier to do.
Yeah, I'm sure the UI for a lot of people's hosted integrated Python is really poor.
You know, there's probably maybe you'll get auto indentation if you're lucky, right?
You know, like it's just there's not a there's not a great story.
So they could just like the other examples we talked about build upon what JupyterLab is already doing.
Just add their part.
Yeah, exactly.
Rather than them having to reinvent everything, they can just reuse Jupyter.
Yeah.
And because of the extensions, you could theoretically have like a Blender extension for JupyterLab, potentially.
Actually, someone has made one.
So for the specific use case, I think it's Jan Hendrick Muller has a Blender extension to render Blender model in line in the notebook.
So, yeah, check it out.
Excellent.
All right, y'all.
We are getting very short on time here.
There's two things I want to touch on really quick before we put a wrap on it.
These are not exactly like brand new things, but they're very exciting.
And one is Jupyter Lite.
And Jeremy, I know you've done a lot with Jupyter Lite.
You want to tell us quick about Jupyter Lite?
Yeah, sure.
The idea behind Jupyter Lite is to have a full Jupyter distribution that runs in a web browser.
So normally when you use Jupyter, you need to install JupyterLab package, Python package, or the notebook Python package on your machine or on a server.
But with Jupyter Lite, you don't need to do this.
So in the end, what you get is just a set of static files.
So it's just a bunch of files sitting on disk.
You just have HTML, CSS, JavaScript, and WebAssembly.
So the main difference compared to a traditional Jupyter installation is the fact that the execution, the environment runs in a browser as well.
So, for example, Python.
But it could also be other languages that are compiled to WebAssembly.
All of this is, again, built on the shoulder of giants.
Yeah, this is awesome.
This is so awesome.
The Jupyter stack is just another way of kind of remixing the whole stack.
So, you know, you take a bit of JupyterLab, a little bit of the Jupyter protocol to interact with the kernels.
And then even the kernels can be reused in some way.
And then you put that together and make something new.
So that's the idea behind Jupyter Lite.
It was also made possible by the work on the Pyodide.
So this is Python compared to WebAssembly.
Since Python is still the most heavily used language in Jupyter, this is kind of what people expect when they use Jupyter and Jupyter Lite to have a Python kernel.
So if I wanted to, say, host some interactive code exploration, I want to have something that people can play with.
Maybe I'm an educator and I've got a thousand students, but I don't want to set up a server and worry about what people might do to it.
But I want to give them flexibility, right?
Like, this is really nice.
You could just put this on a static file server like Netlify.
Yeah, that's definitely a use case.
So GitHub Pages, Arita Docs.
It's very simple.
It's much simpler than having to do sysadmin work to install packages.
And also for the end user, it's also simpler because you don't have to install anything.
Everything is already prepackaged.
It's crazy.
It looks just like JupyterLab, you know, but it's...
Because it is, in the end, just JupyterLab.
Yeah.
And there is also Notebook 7 running as well.
So again, like we were talking about earlier, if some users prefer to have this document-centric interface, then they can also use Notebook 7 instead.
It's excellent.
If you want to give access to some notebook but not run a server, I think there's kind of two options here.
This is one of them.
The other is if I'm going to give a notebook file to somebody and say, here, run this on your computer, there's a lot of assumptions about what happens at that point.
Do you have Python?
Do you have the right Python?
Do you have permissions to install Jupyter and all the other dependencies?
Can you run a server?
Do you know what a terminal is?
Like, that's a long conversation.
And so I think the other thing that's really interesting, I don't know where I pulled it up, but yeah, Jupyter Desktop is also quite interesting as a quick start to getting a Jupyter Python environment on your system, right?
Yeah, that's some work done by Mehmet.
And basically, it bundled an Electron.
So it's a web browser that's packaged with Jupyter Lab.
And it's very powerful because now it can come with its own Python, but you can also use the Python that you have on your computer.
And yeah, that's really a nice way for distributing and easing the installation for the end user, especially on non-friendly operating systems.
Sure.
Or people who are not familiar with it.
Yeah.
That's true.
Like, yeah.
I mean, you've got a Mac.
Well, there is a macOS ARM and 64 version.
There's a Debut and a Fedora.
There's a Windows EXE.
Like, that's...
No, it really solves the problem of I've never used the command line, but I know lots of math, right?
So...
Yes, exactly.
But when I run MATLAB, I just click the MATLAB icon and then I get that shell.
What is...
Why can't I have permissions to install Jupyter?
It says I don't have...
No, I get it.
Like, that whole frustration side could go away, I think.
This is interesting.
But, I mean, this is a harder problem than one might imagine just because of how many packages there are on the ecosystem, what things you might be using.
So, JupyterLab Desktop is pretty clever.
It can also detect if you have Conda or Mamba installed and use those environments.
It can use a running server.
So, it shifts with the newest version of JupyterLab, right?
But it could also connect to a server that's serving a different version of JupyterLab and give you that.
It can be a client to different JupyterLabs is what I mean.
One time, just to see if it would work, I ran Jupyter Notebook and Lab and Desktop with real-time collaboration.
And it turns out you could do that.
So, it really is like first-class JupyterLab and it's very flexible and it tries to integrate with what you've got.
If you've got nothing, it brings what it needs.
If you've got Conda set up, it finds stuff there.
If you've got Mamba set up, it finds stuff there.
So, it basically tries to be exactly what that first user experience you would want is.
You go, you download, you double-click a file.
The only thing that's missing is you have to go to GitHub to get this.
We should probably put it in an easier place for you to find.
Yeah, possibly.
Like, maybe in the various stores.
I don't know.
That's such a hassle.
Well, no, I just mean go to jupyter.org slash download.
I just set a URL.
Sadly, it's not going to work for you.
But maybe we should do that.
Yes, exactly.
That would be a pretty easy thing to do, actually.
Just detect the OS and highlight that or have that little drop down and choose another OS, right?
See all downloads.
I think these two options are going to really make it possible to reach the next 10 million users of Jupyter.
Especially Jupyter Lite in terms of Jupyter Lite in terms of scalability is unmatched at the moment.
There is a deployment of an in-browser Jupyter flavor in use at the moment by the French high school system.
They started with the Paris area and now they enabled all the school districts.
There is over half a million users on this deployment.
200,000 sessions per week.
And it's also one single virtual machine.
And if they were trying to do this with a one Docker image per user session, it wouldn't fly.
They'd need a massive IT team to pull that off.
Exactly.
And France is not home to hyperscaler cloud providers.
So we'd need to host that Azure or Alibaba or Google.
And that would be an enormous cost, right?
And this is incredibly cheap.
There are some limitations to WebAssembly.
And I mean, for the sort of use cases, what a high schooler is going to do in a notebook,
like learning how to write four loops or computing your greatest command in the administrator.
This is more than enough.
Like the complexity of it is lesser than rendering the UI.
So, yeah.
I think you're absolutely right.
That's why I put them both side by side, the Jupyter Lite and the Jupyter Desktop app.
I think they kind of address similar audience, but in a different way.
Check this out, by the way.
This probably won't come through very well.
But this is my iPad.
And I've got Jupyter Lite working just the same on my browser there as I did.
Or it'd be a Chromebook or whatever, right?
Like it really reaches a big audience.
Yeah.
I mean, modern browsers now can all handle it.
It's not the bad old days, at least on that front.
There will be a lot of really exciting things that happen inside browsers that will surprise
anyone who was using this era of the Web.
Like we're about to see crazy things.
We are about to see crazy things.
Well, we already see crazy things.
We'll see more of them.
That's for sure.
All right, y'all.
I think we are out of time, but we are certainly not out of topics.
We could keep talking for a long time, but we'll close it here.
I guess final thoughts, call to action.
People want to check this stuff out.
Maybe they want to upgrade from an older version, either JupyterLab 4 or Jupyter Notebook 7.
What do you say?
Pip install notebook.
Pip install JupyterLab and have fun.
Dash, dash upgrade.
Yes.
Good point.
Okay, so you already got it.
Poor request, welcome.
Awesome.
Yeah.
So it's a huge project with lots of people involved.
Frederick, final thoughts?
That's a very fun project to work on too.
So if people are willing to contribute, just come and contribute to it.
There is plenty of spaces as we have just touched some of them here.
And we are very happy to have new contributors and help people learn the ecosystem to try to contribute.
And there is no small contribution.
Like there is no small contribution.
Everybody is welcome.
Yeah.
Somebody could set up the download page for Jupyter Desktop.
Exactly.
Absolutely.
Jeremy, you get the final, final word.
Yeah.
That was what I was going to say a little bit as well.
It's the Jupyter community is very welcoming.
So you can contribute in many different ways.
It doesn't have to be about technical details or code only.
Even just putting a talk out there.
First of all, going to Jupyter Cloud and sharing your use cases is already very valuable to the community and also to the developers because they can learn about how people use the tools, get a very valuable feedback.
And that would be one call to action.
Don't hesitate to get involved.
And if you think there is room to improve, for sure.
As you were saying before, PRs welcome, as we say often.
And yeah, check out the latest update on JupyterLite because there is going to be exciting new updates to JupyterLab 4 and Notebook 7 coming in a couple of days.
So I think maybe by the time this podcast is released, it will probably be out already.
And also exciting developments when it comes to real-time collaboration in JupyterLite, JupyterLab Desktop, JupyterLab together.
Yeah.
Very exciting.
A lot of things are already great here that we talked about.
These are some big changes.
So thank you all for being on the show and sharing what you and everyone else has been up to.
Thank you.
Thanks very much.
Thank you.
Yep.
Thank you.
Bye.
This has been another episode of Talk Python to Me.
Thank you to our sponsors.
Be sure to check out what they're offering.
It really helps support the show.
Developers and data scientists are the new high-value targets.
That's you.
And yet this group remains severely underprotected.
Phylum detects, reports, and blocks malicious packages.
Get their research and join their community at Talk Python.
Talk Python.fm slash phylum dash research.
Python Tutor.
Visualize your Python code step-by-step to understand just what's happening with your code.
Try it for free and anonymously at talkpython.fm/Python dash tutor.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at Talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show.
Open your favorite podcast app and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play,
and the direct RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air,
be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I'll see you next time.