One of the fastest growing areas in Python is scientific computing.
In scientific computing with Python, there are a few key packages that make it really special.
These include NumPy, SciPy, and the related packages.
But the one that brings it all together, visually, is IPython, now known as Project Jupyter.
And that's the topic of episode 44 of Talk Python to Me.
You'll learn about the big split, plans for the recent $6 million in funding,
Jupyter at CERN and Large Hadron Collider with Min Arkay and Matthias Boutonnier.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm.
And follow the show on Twitter via at Talk Python.
This episode is brought to you by Hired and SnapCI.
Thank them for supporting the show on Twitter via at Hired underscore HQ and at Snap underscore CI.
Hi, folks. No news this week.
I do have a big announcement coming, and I'm really looking forward to sharing it with you all,
but I'm not quite ready to talk about it yet, so stay tuned.
For now, let's get right to the interview with the Project Jupyter core devs,
Min Arkay and Matthias Boutonnier.
Matthias, Min, welcome to the show.
Thanks.
Thanks, Mike, for having us here.
Yeah, I'm really super excited to talk about Python intersected with science in this thing called
IPython, or what's become Project Jupyter.
So that's going to be really great.
And before we get to that, though, let's just talk about how you got involved.
How do you get into programming?
How do you get involved with IPython and all that stuff?
What's your background?
Min, you want to go first?
Sure.
Yeah, so I was an undergrad in physics at Santa Clara University, working with Brian Granger,
one of the founders of the IPython project.
And I was interested in computing and simulation and things, and ended up working on the interactive
parallel computing part of IPython as my undergrad thesis, and started doing my numerical simulation
homework stuff in Python, even though the classes were taught in MATLAB and Octave and things,
and enjoying the scientific Python ecosystem of NumPy and Matplotlib and things.
And that's kind of how I came to the project and scientific Python in general.
Yeah, that's really cool.
And was IPython already a thing when you got started?
Yeah.
Fernando created IPython in 2001, and I was doing my undergrad a few years after that.
And so I joined the project after it had been around for about five years in 2006, and I've
been working on it for the past 10 years, I guess, now.
Yeah, about 10 years.
So how time flies.
Matthias, how about you?
Oh, so I've been to the project much later than Min.
I actually started programming a long time ago and came across one of the huge refactoring
of IPython, MinDead.
I think it finished in summer 2011.
Just after that, they released the Qt console, so the current IPython team at the time.
And the project was much more friendly and in a good shape for beginners Python programmers.
At the time, I was beginning my PhD in biophysics in Paris, and I started contributing to the project.
It was my first big contribution to an open source project.
And I started to spend my night and weekend doing my PhD, improving IPython, which was helping
me really a lot for my PhD.
And I quickly became a core contributor, and I've stayed in the team since then.
Maybe a good place to start talking about this whole project is maybe we can start with the
history.
Originally, this project was called IPython and IPython Notebooks, right?
Yeah, IPython was around for a good 10 years before we got a version of the notebook out, although
we had been working on various versions of notebooks for about five years that most attempts kind
of didn't go anywhere.
So what did it look like?
What did the product look like in the early days?
Yeah, so initially, Fernando created IPython as just a better interactive shell for Python,
so giving you some better tab completions, nice colorful tracebacks, things like that.
Also, Python's a nice, verbose language, but when you're doing interactive stuff, some of
the bash shell syntax is nicer to type when you're doing LS and CD and everything.
So one of the things that Fernando did early on was add this notion of magics for extending
the Python language to give convenient commands for interactively typed things.
like you can type CD in IPython, which you can't type in a regular Python environment.
And then there are magics that are particularly useful for the scientific visualization things
and things like the time at magic for profiling and good map.lib integration for the event loop
and things like that.
I know what the shell looks like today.
You know, you can load it up and it kind of looks like a shell and you type in there,
but you can do things like plot graphs and those will pop up into separate windows and things like that, right?
Yeah, and most of that is provided by tools like MapPotlib,
but often those tools need a little bit of help to make sure that the terminal stays responsive.
And that's one of the things that IPython helps with in terms of what Python calls the input hook
to ensure that the terminal remains responsive while a GUI event loop is also running.
Right, yeah, very cool.
So how did it go from that to the more, I want to think of it as like articles or published style,
these notebooks that you can use to communicate almost like finished work rather than something interactive?
Notebook style interfaces, since a lot of the IPython folks come from a physics background.
So Brian and Fernando, who started the project, were doing their graduate work in physics at Boulder at the same time.
And then I was Brian's physics student.
And notebook environments are pretty common.
There are various commercial and non-commercial products that have kind of notebook processing environments,
especially for math analysis.
So often code is not the best representation of math,
but there are rich, you know, rendered mathematical expressions that are nice.
Brian and Fernando knew that they wanted a notebook type interface fairly early on,
but the tools just weren't there to build it.
And IPython wasn't in a shape to really support it.
So slowly at first, and then it kind of picked up speed.
We added the pieces for putting that together.
But it was kind of in, it was on the horizon for many years before it actually happened.
Sure.
It took a while to build the maturity into it.
What are some of those building blocks that it was waiting on?
Yeah, I think that web technology and web socket were one of the technologies that was missing to actually use a notebook.
If I remember correctly, one of the latest prototypes that we did not release was using AJAX polling.
But the ability to actually push a result to the web front end once, as soon as the kernel gets a result,
is one of the key factors that pushed the notebook forward and allowed us to do the notebook.
Actually, the notebook that you know nowadays, the first prototype, was actually using still draft of web sockets that stayed in draft state a long time.
And so we were really bleeding edge on this technology and adopting a lot of everything, everything in browser,
and everything to just rely on what current browser can do for the notebook.
There is, by the way, we can put that in the note of the podcast, a really nice blog of Fernando that recaps the history of IPython.
And even 150 lines of Python, which is a version of IPython when it was like a few weeks old, which is IPython 0.1,
that we can dig up for people who are interested in trying really early prototype.
Yeah, go back and see the history.
That's a really interesting point, Matthias, because it's easy to think of the web as being this very rich, powerful, capable platform,
because it has been for the last five years or so.
But 10 or even more than 10 years ago, it was not, right?
It was basically just documents on the web, right?
You had a little bit of JavaScript, and that was about it, right?
Yeah, I think so.
I haven't used that much.
I was not developing on the web that much 10 years ago.
I was more a C, C++ person.
You mean maybe it was more developing web at the time?
Yeah, I wrote an early version of the web-based notebook for IPython during the summer of 2006, 2007, I think.
Even then, the tools available really weren't...
It was not a particularly pleasant thing to work with.
I bet it wasn't.
Did you end up in a lot of situations where you're like, oh, this only works in Firefox, and this one only works in IE, and just partly working in a lot of places?
It's frankly still like that.
It still never works in IE.
Yeah.
Yeah, it's hard to love IE, I know.
Well, but I mean, recent versions of Internet Explorer are actually really nice and have good standards implementations and everything, but the reputation of IE6 kind of overshadows.
Yeah, it definitely casts a long shadow.
And, you know, Microsoft, I think just last week, possibly, like very recently, just ended support for all versions of IE other than, I think, in 11 and onward.
Maybe 10 and onward, but certainly knocked out a whole bunch of them.
And, you know, once that kicks in, that's going to be a good day for everyone that has to work on the web.
Microsoft has done a lot of things recently.
Last week also, if I remember correctly, they did release as open source the JavaScript engine that will power the next version of their browser.
So Google has V8, which both power Chrome and Node.js, which is actually one of the technologies that helped the notebook become reality because JavaScript was painfully slow 10 years ago and is now really, really fast thanks to V8.
And so it's really nice to see nowadays Microsoft actually releasing open source software and contributing to the community.
And I hope that in the next few years, Microsoft will lose some fact that everybody is complaining about IE and everything and get actually nice software, not that many security bugs and so on and so forth.
It will be really nice if that comes along because a lot of people run their software and it would be, you know, the world would be a better place if it works really well.
I certainly think they're on the right path.
I think it's pretty interesting.
So one thought I had while you guys were talking about this is how does, what's the cross-platform story or IPython and Jupyter in general?
Does it work kind of equally well on Windows, Linux, OS X or are there places that are more equal than others?
Linux and OS X are a little bit more equal than Windows, but it should work.
It should work everywhere.
And even though all of our developers and everything are working exclusively on Linux and OS X, when we do user surveys and things, we find that roughly half or even slightly more than half of our users are running Windows.
So even though it often doesn't work quite as well or we frequently during the development process will introduce bugs that we don't notice for a while, Windows really is a first-class platform for the kind of local desktop app that happens to use a web browser for UI case of the notebook.
There are certain aspects of installation that are often more challenging on Windows, especially in terms of installing kernels other than the Python one.
So installing multi-language kernels is more challenging on Windows.
And I think that's not necessarily a specific deficiency of Windows.
It's more just the kind of developer maintainers don't tend to use Windows.
So the documentation and education often just don't cover what you need to do for Windows as well.
Right.
If you don't develop and test deploying your packages in the underlying compilers that have to make them go, well, you're more likely to run into problems, right?
Yeah.
I would say also that Trevis CI, so continuous integration, is often on Linux only.
Setting up on Windows is painful.
So we catch up bugs with continuous integration, much often with continuous integration on Linux.
So less prone to bug on Linux.
And the other thing is, I don't always like to say good things about half proprietary tools, but Konda changed a lot of things for the last few years.
It was really painful to install Python on many systems.
And now it's one of the solutions, especially at Software Carpentry Bootcamp, where we ask people to just install Konda and Konda install Jupyter, which now even come vended in it.
And it's almost always works out of the box.
And especially for beginners, it's a really, really nice tool.
Yeah, Konda has really moved the bar for how easy it is to get set up, especially on Windows.
There are lots of different ways to install things on Unix-y platforms that work fairly reliably.
But the binaries provided by Konda and Anaconda are extremely valuable for beginners, especially on Windows, where people don't tend to have a working compiler set up.
And a lot of the scientific packages won't build on people's Windows machines.
So having binaries is extremely important.
And the binaries provided by Konda and Anaconda have been extremely valuable, especially for people getting started in scientific Python.
Yeah, I still think I have scars from the vcvars.bat was not found sort of errors trying to do stuff on Windows.
And we had Travis Oliphant on Show 34, who is behind Konda and Continuum and all that.
And I think it's a really cool thing that those guys are doing, sort of taking that build configuration step and just pre-building it and shipping the binaries, like you say.
That really helps people when they're getting started, I think.
Yeah, it's made a huge difference, especially, as Matthias mentioned, in the workshop, the kind of software carpentry and Python boot camp type environments, which often, you know, just a few years ago, where you spend the first day on installation, basically.
Which is a high price to pay in a two-day workshop.
And now it's often down to an hour.
It's awesome.
It's a super high price to pay.
And it's also super discouraging, right?
People come not because they want to learn how to configure their compiler.
They want to come build something amazing, right?
And they've got to, like, plow through all these nasty configuration edge cases.
And, yeah, very, very cool.
So, before we move farther, you know, just the other day, I was trying to describe IPython as somebody in, like, one or two sentences.
And I didn't do a super job, I think.
Could you guys maybe give me your elevator pitch for what is Jupyter or IPython, which becomes Jupyter?
It's really tough.
Have you seen the Lego movie?
Do you know the song Everything is Awesome?
Yes.
That would be my pitch.
Yeah.
Everything is awesome.
Okay.
Yeah.
So, I would say it's IPython and Jupyter projects together provide tools for interactive computing and reproducible research and software-based communication.
Okay.
It's kind of the high-level gist.
It's fairly different than a lot of what's out there from a programmer's perspective.
So, it does take a little explaining, doesn't it?
Yeah.
So, we have things like an environment in which to do the interactive programming and do the exploratory work.
And then we also have things like the notebook document format, which are for distributing the communication and sharing it with other people.
So, those are kind of the two aspects.
And Fernando likes to say we have tools for the life cycle of a computational idea.
That's a very cool way to put it.
It's a very cool tagline.
I like it.
We're talking about IPython because that's the historical place.
And we're talking about Jupyter because that's the present and the future.
Could you guys maybe talk about how it went from one to the other?
What's the story there?
Yeah.
So, when we started working on building these UIs with rich media displays, the first one of which was the Qt console, the first step of that was separating the front end from what we call the kernel, which is where code runs.
That meant essentially establishing a network protocol for a REPL, basically.
And with that, we have the ability, an expression of, okay, I'm going to send an execute request that has some code for the kernel to evaluate.
And then the kernel sends messages back that are display formats of various types.
So, it can send back PNGs or HTML or text.
We realized, not entirely on purpose, this wasn't what we set out to do, but we realized when we had this protocol that there was nothing Python-specific about it, that any language that understands a REPL can talk this protocol.
And because the UI and the code execution were in different processes, there's no reason that the two need to be in the same language.
Communities like, the first big one was the Julia language community, essentially saw the UI, specifically the notebook UI, and said, you know, we like that, we want to use that, we'd rather not reimplement it.
So, what they implemented was the protocol.
And once they implemented the protocol, they got the UI for free.
The result of that, since we didn't set out to design that, there were a bunch of rough edges where we had assumed Python, but they were kind of incidental, smaller assumptions to work around.
And so, since that started, we've been kind of refining protocols and things to remove Python and IPython assumptions, so that the UI is separate from the language in which execution happens.
Because we don't really, you know, a lot of the benefits of the protocol and the display stuff, there's no reason it should be confined to code executing in Python.
Yeah, that's a really happy coincidence, isn't it?
That's excellent.
Yeah.
This episode is brought to you by Hired.
Hired is a two-sided, curated marketplace that connects the world's knowledge workers to the best opportunities.
Each offer you receive has salary and equity presented right up front, and you can view the offers to accept or reject them before you even talk to the company.
Typically, candidates receive five or more offers within the first week, and there are no obligations, ever.
Sounds awesome, doesn't it?
Well, did I mention the signing bonus?
Everyone who accepts a job from Hired gets a $1,000 signing bonus.
And as Talk Python listeners, it gets way sweeter.
Use the link Hired.com slash Talk Python to me, and Hired will double the signing bonus to $2,000.
Opportunity's knocking.
Visit Hired.com slash Talk Python to me and answer the call.
Matthias, where did Jupyter come from?
It used to be called IPython.
Obviously, that doesn't make sense if you're not using Python.
We've been thinking about renaming part of the project for much longer times than when we actually announced that we will be renaming to Jupyter.
Of course, we were aware that users, especially non-Python users, were confused.
Like, I want to use a notebook with R.
Why should I install IPython?
And you have to understand that many users even don't make the difference between Python and IPython.
And many users also write IPython with lowercase i, and everyone knows that it's uppercase i.
It's not made by Apple.
Come on.
Yeah.
And so, yeah, we were searching for another name to actually something that is easy to Google, that is not already taken, where we can get a domain name.
And that would have a connotation, a scientific connotation.
And we wanted to do thanks to the Astro community that has been using IPython for a long, long time, almost since the beginning.
And I still remember one day, Fernando wrote a mail to the whole team and said, hey, I just found this name.
What do you think?
And it was everybody agreed.
And almost in a couple of days, we decided to grab all the domain name and start working on actually separating the project and everything.
It has been a really tough transition.
People were really, really confused about the renaming.
People are still confused.
But especially for new users, the distinction Jupyter-IPython is really, really useful.
And also, it allowed Jupyter to become something slightly bigger.
That was also in our mind in the back.
Which was like, Jupyter is more a specification that you have a protocol and you have a set of tools.
What is part of Jupyter is much broader and it can allow anybody to basically say, hey, I implement the Jupyter protocol.
And so, it's easier to say, hey, I have a Jupyter Atom plugin.
There are also legal issues around that, that using trademarks that are really close to Python is difficult.
And Jupyter, being a brand new namespace, and you know that namespaces are great, we should use more of them.
Allow people to use that and say that they are multi-language in a much better way than when saying we are compatible with IPython.
Because IPython is also highly connected as a shell.
And Jupyter is more than just a notebook.
So, having Jupyter is much better and we are happy with that.
Yeah, it makes perfect sense.
I'm sure the transition was a little confusing for people who have been doing IPython or they've heard about IPython.
They were going to look into it.
Now it's this other thing.
But there's more than just a couple of languages that are supported, right?
How many are supported?
It depends on when you want to be supported.
We have a wiki page which is still on the IPython repository, which lists, if I remember correctly, 50 or almost 60 languages.
It means that you can have languages that have many kernels.
It means that someone at some point wrote a kernel or a toy kernel that works with IPython.
And if I remember correctly, we have around 60.
60.
That means probably if you have a language you care about, it probably works with Jupyter, right?
Or it's very edge.
Most kernels won't have all the features.
I would say that the one I know works with most of the features are the Python one because we maintain it.
So you can see it as a reference implementation.
There are other Python ones, like toys that are only a few hundred lines to show you how to implement that.
The Julia kernel is a pretty feature complete.
It's actually many of the features that we have in the IPython kernel were actually, by the Julia team, moved into the Julia language itself.
So actually having implemented the protocol, having seen the Nullbook UI, allowed them to make much better abstraction for the Julia language and actually improve performance in some small area.
So that's almost a thing.
The Haskell kernel also have a really good maintainer and have really nice features.
Like if you write some code in Haskell and you can rewrite it in a more compact form, the Haskell kernel will tell you that after running your code.
They say, hey, you can rewrite it this way.
It will be more compact and more readable by someone who does Haskell.
So Ruby had some activity at some point.
I'm not sure now how much activity there is.
And we definitively have people from the R kernel.
The R kernel was created by Thomas Clover, who is now back in the UK, still working with us.
And I've been taken over by some R people who are actively contributing to the R kernel and are also reporting bugs and fixing bugs a lot in IPython itself.
Yeah, and another active kernel author community is the Calico project, which is from the CS department of Bryn Mawr by Doug Blank, where it's kind of a multi-language.
The kernel itself is a multi-language environment.
They can actually switch between different runtimes.
That does some pretty cool stuff.
And they've been very helpful with implementation and protocol testing and things.
Oh, that's cool.
Yeah, and that's kind of related to what I was going to ask you next is if I want to write something in Python and then something in C++ and then something in R, can I do that in like one notebook and have the data work together?
Well, yes.
So there are a couple things to that.
One is we have chosen in the notebook to associate one notebook with one kernel.
So there's one process determining how to interpret the code cells and produce output as a result.
There's another project derived from IPython called Beaker notebook that doesn't do this, that associates each cell with a kernel and then defines a data interchange for moving data around that allows running code and passing data around from JavaScript to R, Python, and like this.
However, a kernel, from Jupyter perspective, a kernel can itself define semantics for running code in other languages.
And IPython, this is where sort of the distinction between IPython and Jupyter comes up.
That as far as Jupyter is concerned, there's one kernel associated with the notebook.
But the IPython kernel can define these things called cell magics that say this is shorthand for actually compiling a block of C++ code with Cython and then running that, or the R magic that actually hands off code to an R interpreter.
As far as Jupyter is concerned, there's only one kernel per notebook, but the kernels themselves can actually provide some of this multi-language functionality.
And IPython does.
Yeah, to extend on what Min said, there is another kernel, one of the Calico kernels, actually, which actually is one kernel that implements many languages, won't go exactly into details.
And there is this nice distinction that a kernel is not always one language, it can be many languages, and in particular the Calico kernel uses triple person syntax to say,
hey kernel, change how you parse the next string.
And so you can actually switch in between three or four languages.
I don't exactly remember.
There is Python, there is Scheme, and something like that.
And it's really interesting.
And you have different ways of actually sharing data between languages.
One of the examples I can try to dig, which is really interesting in multi-language integration,
is the Julia Python magics demonstration.
You can actually move data and actually have a tight integration between Julia and Python.
It's only on Python 2, unfortunately.
We need to update the code for Python 3, but I'm not fluent enough in Julia.
You can define the Fibonacci function recursively in Julia, calling Fibonacci n-1 in Python and Fibonacci n-2 in Julia.
And the Python versions that call Fibonacci n-1 in Julia and Fibonacci n-2 in Python.
And you can ask for a number, and you will actually get a cross-language talk or a stack trace where you have a layer cake of each language,
which is really, really impressive.
You can even create from Julia, you can import Matplotlib from the Python side,
create a figure in Julia with a function which takes sine, for example, from the Julia standard library,
and cosine from the Python standard library on MPI,
and plot that on the Matplotlib figure and get back into Python
and annotates the figure from the Python side without copying the figures.
It means that the two interpreters are actually sharing memory.
So it shows you that you can do some really, really advanced cross-language integration
without having to copy data back and forth.
That sounds really interesting and useful for scientists.
You know, maybe they've got something they've done in R or some other language.
Yeah.
Some little bit of processing, and they're like,
I really just want to plug this in over here, but it's the wrong language, right?
And that sounds like it makes it kind of possible.
Yeah, and it's been, so Julia being a very young language community,
it's been extremely valuable to them to build this bridge,
largely to Python, but also to C and things.
That Julia didn't, in order to come up to speed with what scientific programmers expect,
with things like Matplotlib and stuff, they didn't need to write,
okay, here's the Julia plotting library, just so that people could do anything.
They could start out by saying, well, we'll just use Python libraries.
And because of these really, really slick layers that let Julia talk to Python
and Python talk to Julia in a really native way,
Julia basically gets the entire Python library ecosystem for free,
and then can kind of re-implement as needed.
And as they find more idiomatic Julia ways to do things,
they can start building those libraries.
But they didn't have to start from zero just because it was a new language.
For new languages in general, I think being able to interoperate with other languages
is a really important, really valuable way to start kind of hit the ground running.
That is super interesting because, you know, there's 10, 15 years of super solid data science stuff happening in Python.
And if you can just go, we'll just start from there, rather than from zero, that makes all the difference.
The other cross-language thing that a few people think about is actually the language in the kernel
and the language of the front end, which is JavaScript and HTML.
And the notebook allows you to do like these cross-language bindings really easily, especially with widgets.
And one of the examples I give is you can have interactive 4chan.
You can actually plot something in 4chan and display something in JavaScript
and have a slider that you move and change the result of your 4chan computation.
Wow, that's really awesome.
One of the things I wanted to spend a few minutes talking with you guys about was what you referred to as the big split.
So it used to be that IPython was like one giant GitHub repository, right?
And now you've broken it into many smaller pieces, yeah?
Yeah, I went from one to about a dozen.
That sounds like a lot of work.
Yeah, that was my spring, pretty much.
It took three to six months to kind of get it all split up.
But since we knew we eventually wanted to do something like that,
we had, for the most part, already organized IPython into kind of sub-packages of dedicated functionality.
So it wasn't too crazy difficult to break it up.
But the tricky bits were like the common utilities that we use all over the place and how to deal with that.
Yeah, it's the little interdependencies that don't seem big, but when they're woven in between all the pieces, all of a sudden it gets harder and harder, right?
Yeah.
So it was tricky to execute the big split in a variety of ways.
So we wanted to preserve history, but we also didn't want to duplicate the already large IPython repo so that installing from Git would mean that it now is 12 times as big as getting IPython was.
So we had to do some, on the new repos, some clever Git history rewriting to kind of prune out the history for the files that didn't survive.
That's really interesting.
You talked about that in your blog called The Big Split, which I'll link to in the show notes.
And you had to use some funky commands to sort of make that happen, right?
Be really careful about how you move the history over.
Yeah, and this is what, there are some, I think, in the various version control tools, the fact that you can rewrite history in Git is both really scary and weird and gross, but also really useful sometimes.
But it lets you do things like we did, which is selectively preserve history, which has been nice that you get, you know, you get the history of the notebook work in the notebook repo, even past the creation of the notebook repo.
But you don't get the baggage, right?
Yeah, but we don't get the history of all the rest of IPython.
Yeah, because normally you delete a file out of the repo and it's just, it doesn't show up, but you're still moving it around, right?
To give you an idea, I will just share, I will send you a link.
Someone made a graph visualization of the dependencies into IPython before the Big Split.
And on the same blog post, you have comparison with Django, Twisted, Flask, requests, so that you can get an idea of what the complexity of the entanglement was.
It's on grokcode.com and it's blog post 864.
And I will give you the link so you can put it in the notes.
Oh yeah, thanks.
It's pretty seriously entangled.
There aren't dependency cycles and crazy loops of depending on each other.
It's kind of a tree of dependencies, but there are many nodes on the graph.
Sure, sure.
SnapCI is a continuous delivery tool from ThoughtWorks that lets you reliably test and deploy your code through multi-stage pipelines in the cloud without the hassle of managing hardware.
Automate and visualize your deployments with ease and make pushing to production an effortless item on your to-do list.
Snap also supports Docker and M-browser debugging, and they integrate with AWS and Heroku.
Thanks SnapCI for sponsoring this episode by trying them with no obligation for 30 days by going to snap.ci slash talkpython.
Another thing I wanted to talk about with you guys is this thing called JupyterHub.
What's the story of the JupyterHub?
So JupyterHub came out of the value of Jupyter Notebooks and things in teaching context.
So a lot of people, whether in workshops or classes, are using notebooks to present.
This is the material that we're talking about and the example code and running, you know, doing live demos and things.
You want your students to be able to follow along, and this is one of those cases where, you know, installing scientific Python stack is as much as Conda has made that easier.
It still can be, you know, a significant bar to get over.
So we wanted, and people were building kind of, people were building tools for kind of hacks around IPython at the time to deploy notebooks on behalf of users.
And we wanted to provide kind of an official implementation of hosting notebooks on behalf of a group of users in the context of a research group.
So you've got a machine that a bunch of, you know, half a dozen or so research scientists or students have access to,
or you have a class of 10 or 50 or 100 students.
And you say, all right, I've got these users.
I can install packages for them and then point them at this URL and they can log in and run their notebook.
So basically take away the installation problem by saying, I'm going to control the installation and host the notebooks and everything.
And we wanted to kind of create the simplest, smallest version of that.
And that's JupyterHub.
Okay, nice.
From the technical side, we asked before how, which technology were necessary for the notebook.
And we spoke about WebSockets.
One of the things which is important is we were using really recent technology and WebSockets was really new at the time.
And one of the problems is many proxies or web servers were unable to correctly redirect WebSockets or even to update proxy rules without actually restarting the proxy or server.
And that's one of the requirements we had for the notebook.
If you want to spawn someone notebook without cutting the connection of the other, we had to have a dynamic proxy, which was able to respond to, for example, REST requests, like changes through there without dropping the WebSockets.
And before JupyterHub, only a handful of prototypes of projects were able to do that.
And actually, Min wrote one such an HTTP proxy using Node.js to suit this specific need that no other tools require.
And that's why you actually need JupyterHub to run something.
And JupyterHub needs to be one of the most front-facing software.
I think that no Nginx can do it, too.
You cannot, for example, use Apache or something else.
Or it's really much more difficult to have many notebook servers running.
It's interesting that a lot of the web servers weren't really built for that, right?
Because I guess they came before WebSockets anyway.
And that probably was not a super important criteria for them, right?
Yeah, the web servers were actually surprisingly slow to adopt, to provide WebSocket implementations.
Nginx didn't take too long, but it was quite a while before you could reasonably expect an Apache installation to support WebSockets.
They do now, so there are notebook deployments behind both Apache and Nginx.
But, yeah, we put together the configurable HTTP proxy as this kind of super simple proxy that you can update.
You can update the routing table without relaunching or without losing existing connections or anything.
The more you can make it easier for people to set up these environments, the better.
Because it's not always going to be like some web server admin or a really experienced web developer doing this, right?
It could just be a scientist who just wants this thing for their class, right?
They don't want to deal with Nginx.
Yeah, and that's something we're working on right now.
Because the way JupyterHub is put together, it has two primary extension points.
One is authentication.
So how users log in, and you can kind of drop in any implementation of logging in, whether it's just local authentication with the system, with the password, or using OAuth with GitHub, or your campus sign-on, that kind of stuff.
And then the other is the spawning, how it actually allocates resources for the single-user servers.
But because there are so many choices for how to do that, it actually, one thing we're working on is making kind of more of a turnkey version that you can say,
I want to use this authentication system and this spawning mechanism, and people can just deploy that.
Because there are, from the very simple default behavior of just that works out of the box for, I've just got a shared machine that's on the internet,
and I want to give all the users who already have accounts on that machine access.
That's pretty trivial right now, all the way to a deployment last year that Jess Hamrick did at UC Berkeley
for a couple hundred students in psychology using Docker Swarm and Nginx and a big multi-node deployment for a large number of users
and using Ansible to automate all that deployment.
That sounds awesome.
Is that documented somewhere?
Like, is there an article or something on this?
Yeah.
So she wrote a blog post for the Rackspace developer blog because the hosting for that class was all provided by Rackspace.
So she wrote a blog post covering that, and her Ansible setup is just a repo on GitHub that we can link to.
Okay, awesome.
Yeah, we'll put that in the show notes.
So I can go to Dropbox and get, like, storage as a service.
I can go to Google Apps and get Word Processor as a service.
Can I do that for Jupyter somewhere?
Can I just go and, like, pay $5 a month and get, like, Jupyter?
Yeah, there are a few companies hosting Jupyter Notebooks.
So IBM has their, I believe it's called a workbench, data science workbench.
Continuum Analytics has Wokari, which hosts Notebooks.
I'm trying to think how many others there are.
There's Domino Data Lab.
So there's William Stein with SageMath.
Yeah, SageMath Cloud is probably the primary, the one we're most connected to.
Okay, cool.
That's good to hear.
Yeah, so there are a variety of these hosted notebook things.
Yeah, one thing which is slightly related, I don't know if we might talk to that later.
It's mybinder.org, which has been set up by Jeremy Freeman from Janelia Labs, where basically you set up a GitHub repository, your notebooks, a requirement file, some extra metadata if needed.
And you link to mindbinder.org.
I will give you the link.
And it will actually, just for you, spawn a Docker instance with the requirements and give you a temporary notebook online.
So if you have an article that you want to be reproducible, you can just post it on GitHub.
It's basically like NB viewer, for those who know, but back by a kernel.
And it's paid directly out of Jeremy's pocket.
And huge thanks to him.
If ever someone from Google hears that and likes the project, you would be grateful if you could give some free resource to Jeremy to host that.
Wow, that's really cool.
You know, I was thinking about how Docker might fit into it here.
And that's certainly a really nice use of it.
We have TempNB, which is written by Kyle Kelly at Rackspace, which drives, if you go to try.jupiter.org, you'll get temporary notebook server to kind of try out Jupyter with a few kernels installed.
But it's very locked down.
You can't get your own work in there or do network things.
And so Binder basically combined the NB viewer idea with TempNB to kind of create a workspace where you can actually install packages and get everything running and preload it with notebooks.
So we're kind of we're working on a variety of these kind of applications.
A lot of them are Docker based of deploying notebooks on in various various contexts.
So one thing I wanted to make sure we talk about before we wrap things up is not too long ago, I guess in July, your project got $6 million in funding.
And that's just so awesome.
I mean, I can't think of another open source project that's not deeply tied to a company that got that kind of money.
Yeah, we've been extremely fortunate with the funding over the last few years, thanks to a huge amount of work from Brian Granger and Fernando Perez in chasing that funding down.
And which is what has given me a job for the last few years and Matthias as well.
So, yeah, that's really great.
I mean, what kind of difference does that make for what you guys are doing?
Obviously, a job, but what are the goals for that money?
Really to expand, kind of expand the coverage of the Jupyter project that has a lot to do with education and kind of the document and publication pipeline.
So with NB convert, we can convert notebooks to various formats, but that's a very young project that has a lot of a lot of room for improvement.
And so integrating into publication pipelines and things like that and converting other formats to notebooks in addition to converting what we already do, which is converting notebooks to rendered formats.
And then the other side is in the building web applications with the tools that we have.
So live collaboration, real time collaboration in the notebook is going to be a big part of it.
And then building applications other than the notebook.
So we've got this protocol for execution that we've talked about as kind of the building block.
You can use that without the notebook document.
So you can build a web application that involves running code and producing output, but it may not necessarily be a document editor type application.
It could be something on the other side that's not the notebook app, basically.
O'Reilly built something, basically built a version of this called Phoebe that drives a few demos and some interactive code-based blog posts on the O'Reilly website.
But if you look at how Phoebe works, and Phoebe also itself relies on some TempNB-based Docker deployment stuff.
The way the JavaScript side of the code is organized and also to some degree the server side, we don't make it easy.
So while it's technically true that you can use the kernel execution display stuff without the notebook document piece, we didn't make it easy to do that.
So Phoebe kind of has to hack around a few of the choices we made.
And right now we're in the middle with the help of a lot of volunteer to contribute work from Continuum Analytics and Bloomberg and other folks.
We're working on refactoring the client side of the code into small JavaScript packages, just like we broke up the Python side into the various pieces.
We're breaking up the JavaScript side.
So you can say, I want to install just the client side code that lets me run code and get output.
And with that, then you can build an application that is a web app that maybe you don't even show any code to the user.
You just show sliders and buttons and everything.
But the way your application works actually involves this executing code and getting output with the display protocol.
So you don't have to rewrite all of that logic yourself.
It's kind of similar to the way Julia and Haskell drove us to revise our messaging protocol to be more useful outside of Python.
Things like Phoebe and other applications are driving us to kind of rethink how our JavaScript code works so that you can actually, you can use it kind of pieces at a time in contexts other than the ones that we've already thought about.
Yeah, it sounds like there's going to be a lot of innovation and cool new stuff built on top of this now that you've broken out all the building blocks.
Beyond this just purely technical, we have money for the project and actually write code.
I first want to do thanks.
I mean, the MZ Moore and Sloan Foundation are really people who are nice to work with.
I mean, they just don't ask us for reports.
It's really a back and forth.
And the money is allowing us to do much more things than just being employed.
We, for example, now will be able to host a Jupyter Day where we can actually have talks and invite speakers.
We already had one in November in New York.
And there is, no, it's in November.
Anyway.
And one next month, so in February in Chicago, there is a Jupyter Day.
And without this sound, it would be difficult to actually bootstrap that.
And the other thing is now we are around 20 people across the globe working on the project.
People that are actually on their day job or at Continuum and are at Workspace or IBM, Google, Microsoft are actually allowed during their work day to contribute to the project.
And this helps a lot.
And one of the things that this fund will allow us is to actually gather twice a year, all of us, to actually meet each other.
And once you meet people, you can do much more work than only interacting through the Internet.
And that's something that is really, really great.
And that will change a lot of things, I think, for the project.
I mean, at least last year, I think the NumPy developer had never met all together.
And they got some funding to get them together at SciPy.
And apparently, it was really great to motivate everyone and relaunch the project.
And I'm really looking forward because some people that are working with us have never met in person.
And I would really, in a few weeks, we should meet together thanks to some of these funds.
And that's something I'm really looking forward to.
That's a really interesting point about the community.
There's obvious technical stuff.
But like you said, building the community both from the outside and the inside, that's definitely going to be powerful.
Yeah.
And another aspect of the building and maintaining the community that is facilitated by this funding is being able to,
and one of the things that a lot of open source projects, our own included, really suffer on is documentation.
That especially a fast-moving project, the documentation can really suffer.
Because especially when you're working with volunteers and things, it's a lot more fun to work on code than docs.
Yeah.
Would you rather write a feature, a new feature, or would you rather talk about somebody else's feature that's going to be outdated anyway, right?
Yeah.
And I think it is the responsibility of funded projects like ours to devote some of those resources to maintaining communication with the community and building good docs.
Because it's often easier to find people who can build good docs that you can pay to do that than it is people who can build good docs who are willing to volunteer a bunch of their time.
And so being able to compensate people for contribution to documentation is, I think, an important thing for funded projects to do.
And so we've recently hired a couple people to specifically emphasize improving documentation and things.
And I think that's a really valuable thing that the funding provides for not just us, but for the community.
Good documentation and tutorials and samples.
That really makes it much easier to start using a project.
One point about contribution that has followed me since I started the project, and I felt was great, especially in the Python community, and made me stuck to iPython, is most of the core developers, if you are a beginning Python programmer and you want to contribute to the project,
We will likely spend a few hours with you on GitHub or even on Skype, if you ask, to help you build a feature.
Even if it would take us five minutes, I will personally take a few hours to teach you how to do it if you want to contribute.
So don't be afraid to come and ask and say, I don't know.
I want to build that.
I have the time.
I have the motivation.
It's okay if it takes you four weeks.
It's okay to be wrong.
It's okay to ask.
Unlike some other community who are known for having a BDFL, which is not really nice on the meaning list, we will take time for you to help you, to teach you Git, because we really think that you should feel welcome.
And that's how we had some people that stayed here, because contributing is a project where we help you.
It's not, let me do it and I'm done in five minutes later.
We prefer for people to take four weeks to do something than doing it ourselves.
It's all about teaching people to fish rather than getting them a fish, right?
You can just do so much more if you can get more people brought into the project and comfortable.
And that's really, it's a great message.
Thanks.
So I basically have time for one more question for you guys, and then we'll call it a show.
Min and I were talking about some really cool places using Jupyter before we started recording.
And I just want to ask you each, maybe Matthias first, what's the coolest use or example of people using Jupyter or IPython that you know of?
Peter Norvig posts on NB viewer are some of the most awesome things I've seen.
He is really busy and he had some typos on his notebook and I sent him a patch and he took the time to reply to me.
So he is both someone who wrote awesome notebooks and someone who is really friendly.
And I also like a lot Jake Van Der Plass' blog when he's blogging.
He has a blog engine that publish directly in the book.
So it's Pelican.
And if you can add a plugin and you can directly like wrote your notebook and they're published directly as blogs.
And also my blog, which is Nicholas, is just notebook.
I just push notebook on the GitHub repository.
And Travis CI take care of compiling them to HTML and publish, which makes publishing really easy, I find.
Oh, that's, yeah, those are really cool examples.
Min?
Yeah, so the examples for me are Thomas and I got to visit CERN in December where they're working on some big Jupyter deployments for the scientists working on the LHC and other experiments at CERN.
So that's definitely a really cool thing to see, you know, Jupyter and IPython, as is someone whose degrees are in physics, seeing Jupyter and IPython valuable in that level of physics experiment and theory research has been really cool.
The other things that I think are really cool are where it's been adopted in educational contexts.
So, for instance, Lorena Barba has her AeroPython course teaching aerodynamics with Python is all in notebooks on GitHub.
And then Doug Blank has his computer science courses all with notebooks in a JupyterHub instance for his students.
And Jess Hamrick at Berkeley with her large deployment teaching cognitive computational models for cognition.
And some of these are not programming classes, but they're classes meant to teach scientific principles where programming just happens to be a useful way to illustrate and explore the ideas.
And the fact that notebooks are proving useful in kind of just learning, both learning how to do science for students at the undergraduate and graduate level, and then also for actual scientists doing, you know, doing real cool science.
Both of those, I think, make me feel really good about working on the project.
Yeah, I'm sure those are really motivational.
You wake up, you're like, wow, I'm building software for all of this.
This is fantastic.
On show 29, I had Kyle Cranmer from the Atlas Experiment at Large Hedron Collider on, and he talked about how they were using IPython and stuff like that there as well, which is very cool.
All right, gentlemen, this has been super interesting.
It sounds like IPython has become Jupyter, and it has been such a success, but it seems like it's on the brink of breaking out to be way much more than what it has been.
So, it's an exciting time.
Thanks for having us.
Yeah, thanks for having us.
Yeah, you bet.
Thanks for being on the show.
Talk to you later.
This has been another episode of Talk Python to Me.
Today's guest was Matthias Boussignier and Min RK, and this episode has been sponsored by Hired and SnapCI.
Thank you guys for supporting the show.
Hired wants to help you find your next big thing.
Visit Hired.com slash Talk Python to me to get five or more offers with salary and equity presented right up front and a special listener signing bonus at $2,000.
SnapCI is modern, continuous integration and delivery.
Build, test, and deploy your code directly from GitHub, all in your browser with debugging, Docker, and parallelism included.
Try them for free at snap.ci slash Talk Python.
You can find the links from today's show at talkpython.fm/episode slash show slash 44.
Be sure to subscribe to the podcast.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes and direct RSS feeds in the footer of the website.
Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.
You can hear the entire song on talkpython.fm.
Oh, and don't forget to check out the podcast t-shirt at talkpython.fm/shirt.
Get yours and share your love for Python with the whole world.
This is your host, Michael Kennedy.
As always, I really appreciate you listening to the show.
Smix, take us out of here.
I'll see you next time.
Bye.
.