#268: Analyzing dozens of notebook environments Transcript

00:00 are using interactive notebooks for your data exploration or your day to day programming. What environments do you use? Was it Jupyter and now you've made your way over to JupyterLab. That's a great choice. But did you know there are more environments out there to choose from and to compare? Have you heard of Callisto or iodide? How about cocao or poly note? Those are just the tip of the iceberg. That's why I'm happy to have Sam Lau and Philip gwo. here to share their research comparing and categorizing over 60 notebook environments. This is talk Python to me, Episode 268, recorded June 11 2020.

00:45 Welcome to talk Python to me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter, where I'm at m Kennedy. Keep up with the show and listen to past episodes at talk python.fm and follow the show on Twitter via at talk Python. This episode is brought to you by linode. And century please check out what they're offering during their segments. It really helps support the show, Sam and Philip, welcome to talk Python to me. Pleasure to be here. Same here. Yeah. Nice to have you on the show. Sam. Philip, welcome back. You've been on the show a few times. You were one of the very first guests way back when diving into the CPython source code internals. And then we've talked about a bunch of fun topics, you know, coding and your golden years and maintaining open source projects individually and things like that. It's been a lot of fun to have you on the show. And Nice to have you back. Yeah, it's great to be back. And yeah, that was my first ever podcast with you. That was that was I think almost five years ago. It's good to be back as a podcast veteran now. That's right, you know, the whole deal. And Sam, welcome to the show. Before we get to the main topic, I always like to set the stage for where the guests are coming from. So when somebody says, you know, like, I think you should really do this. Like, they might say that because they're a biologist using Python, not a person running Facebook, or the other way around. Right. So maybe just tell us really quickly about your background and how you got into Python.

02:06 My background in Python came from actually back in high school, I ran into the first wave of like, massive open online courses. And I write and so like, at my school, there wasn't much emphasis on programming much at all. And so when I ran into when I found out this courses, I got really into it, and decided to major in computer science at UC Berkeley, and then ended up doing end up starting this PhD at UCSD with Philip, studying cognitive science.

02:36 Cognitive Science is really interesting. I've talked to a few folks from there. There's a lot of interesting computational CS type problems happening in cognitive science. Yeah. And data science as well. Absolutely. I work for a while at a company that was basically a spin off from a research lab that was a bunch of computer science PhDs and we did all sorts of cool stuff with eye tracking and programming. And man, it's a lot of fun. I really enjoyed my time in that space. Yeah,

03:01 people here are kind of science is so broad, that you have people here using programming for all sorts of tasks, like the people here training deep neural networks, but also people studying fMRI signals and processing those and then working with, like neuro imaging. And so you just get a really diverse, broad range of perspectives and uses for Python.

03:23 Yeah, it's interesting, too, because I feel like there's still a lot of open questions.

03:26 Yeah, there's a lot we don't know. And it feels like the more we know, the more we know about what we don't know.

03:32 Yeah, exactly. Every answer opens up to more questions. That's right. Exactly. All right. Well, we're going to talk today about the research that two of you have done in the space of notebooks, as in computation, notebooks, Jupyter notebooks, that type of thing. And it turns out, actually, there's more than just ipython and Jupyter. Right.

03:53 Quite a few more. Yeah. So in our paper, we talk about a total of 16 different notebook systems that we put together and try to try to draw some patterns from

04:03 Yeah, it's really interesting. I mean, people I think Jupyter ipython, when it was first started, became Jupyter really changed the way people thought about programming. I mean, there are tools like this, maybe you guys can throw out some that felt kind of like that. So when I say that, the one that comes in most of mine is think Mathematica or maple, right? They're really beautifully visual. But then they also had computation. But they also had like output, you know, MATLAB, but I'm not quite as pretty. For whatever reason, more utilitarian, I guess. And when you thought about programming that was, you know, maybe if you're trying to emulate that would be a script. And then it would like output some stuff or would pop up a window. It saved like a JPEG or something. But the notebooks really brought those those two worlds together, say, look, you can write code in a, like a pro programming language in a way that you really would use, like libraries and stuff that are not just baked into the tools, but then they have this kind of interactive world. I think it's really interesting to see all these derivative concepts, right? We've got, we've got Jupyter that had its its way of doing things. But there's challenges there. And there's a lot of variation. Like, were you surprised to find this many different

05:12 systems around? I certainly was, I think I came into this project thinking, Oh, it's probably just Jupyter plus, like maybe three or four other other systems. And I think the deeper we went, the more we were like, Whoa, there's, people are really taking Jupyter and changing it, adding new things to it, making it fit their use case more effectively. And there's a lot of it turns out that there's a lot of different groups studying Jupyter and, and notebook systems in general and trying to figure out how they can make it better for themselves.

05:45 Yeah, I think I just add to that, to what Sam said was that, you know, we started this project. So this is that this paper was co authored by Sam, one of my other PhD students and drosos. One of my undergrad students, Julia Markel, and myself, and we basically just kind of split up the work and just went to find a lot of these notebook systems. And the, the motivation for this was that, you know, in the academic literature in the kind of academic world of studying programming tools, in the past few years, there's been a lot of people studying how people use notebooks. I believe Adam rule was on your podcasts. Michael was Adam. It sounds Yeah, I don't Jupyter the, you know, analyzing the million Jupyter notebooks.

06:25 Yes, that's right. Yeah, that's right. Yeah.

06:27 So there's a bunch of studies around Jupyter notebooks and how people use them for science and stuff. And then people developing new prototypes. At the same time. There's a bunch of industry people, you know, every startup companies, like we're gonna do a notebook, we're gonna do this other notebook. There's another notebook.

06:40 So Google co lab data, lower Azure notebooks, all those? Yeah, yeah.

06:44 So they came like this wild west. And we felt it was around time for this sort of what we call a survey paper or a meta analysis, right? It's like, this paper is not doing original research in that we're not making a new notebook. But we are collecting together dozens of these notebooks from both academic prototypes, and also industry products and trying to like, analyze them and see, you know, how, how does this space fit together? So I got a few of my students together, we kind of, we kind of brainstormed and looked for a lot of notebooks and stuff. And then Sam, who's the first author on this kind of, you know, did a lot of leadership on on this work? So I'll let him talk more about the details. Because I forgot all the details, though. I just Oh, yeah, sure.

07:23 Yeah. And your primary research is in human computer interaction.

07:26 Right. Right. So my primary research area that that my students are in is called HCI, or human computer interaction. And that kind of industry term for this is, you know, people know it as UX design, right user experiences on user interface design, and, you know, HCI, or human interactions, sort of the, the academic term for the research that goes into how do you make better user experiences for, you know, different kinds of users? In this case? It's for scientific programmers and data scientists. Yeah.

07:53 Okay. Now, I think probably, we should start this conversation with a definition, because I kind of opened it saying, well, Jupyter is the prototypical thing of what a notebook is, and it's kind of the natural progression of something like Mathematica. But what would you define a notebook system to be?

08:14 So like, how do other things that are not exactly that fit into this world? In this paper, we define a computation notebook as a system that supports literate programming, where you can blend a text based programming language with a program outputs in a single document.

08:32 Right? And this is not a super new idea, right? This? I mean, Donald Knuth is like in the 80s, or something talked about this idea, right? Yeah. And actually, I think the entire tech program like law tech, the tech that Don Knuth made was a literate program to begin with, he was like the, I would say, like most prominent proponent of literate programming. Okay, so notebooks are these embodiments of this literate programming idea? Basically, I guess I'm hearing that it's, it's the ability to blend together the presentation of like, pros almost, and computation and visualization around that. Yeah, yeah. And it kind of shifted the dynamic of how we view a program, instead of having a program be instructions for a computer. It allowed us to see programs as a kind of like the outputs themselves as a theme that you would show to other people in the end. Yeah. Because traditionally, it would be, here's the graph, exactly. We ran the analysis, and this is the output. Here's the description of the output, maybe a description of the algorithm. And here's the output. And, you know, there's been a big push in the scientific space for reproducibility. And I think also just in the data science space for explore ability, right? Like, I've got data, I don't really know what it is, I need to just get in there and play with it before I turn the algorithms loose on it. And that's kind of what's happening here. Right?

09:50 Yeah, definitely. I would say many of the projects that we looked at in this paper were birthed out of this directly out of like the reproducibility issue where Some person might write a script to write to generate a graph somewhere, but then lose it. And then all you have now is like a graph. In the end, if you had a notebook The, the pitches, if you had a notebook, you wouldn't need to have a separate graph dot jpg, and then a separate program to run it, you just have it in one place. Right?

10:16 Right, right? How often does version control in the science space, or the sort of beginner developer space look like? A bunch of zip files with their, their name to the thing? And then the date or the thing of the date? And then like to

10:30 Oh, yeah, exactly. It's like 05 underscore final final v3, you know,

10:36 exactly final edited. That's not amazing. So having something in here that sort of brings a little more formality and brings, I guess, the paper and the presentation and the script or code that ran it, like into one thing, that's versions continuously, it seems a little bit better.

10:52 Yeah. And actually, there are the basic way to version control a Jupyter notebook is to put it in a version control system like it. But oftentimes, it's not good enough for for actual scientists who have like, the notebook files themselves, but also their data. And also the libraries that the notebook use to generate the outputs. And so there's a group of projects that we surveyed that not only version, a notebook, but also in version, a notebook and a dependencies. Okay.

11:18 Yeah, very nice. One of the challenges however, I think it's ironic, like, it brings stuff together here to make the code and the analysis and the presentation all as one but it actually Jupyter notebooks are not very easy diversion.

11:31 quite hard. Because Yeah,

11:32 yeah, like if they store the last run of the output, and if that is dynamic in the sense that it's like, not every time you run it, you get the same output. That can be mean, even if it changes, right. Even if it is that case, it's still hard. But if it's like changes every time you run it, because it pulls something from the internet that varies or whatever, it's, it's basically always a merge conflict. So there's tools like envy strip out, which are nice, that are like pre commit hooks that will let you do that. And other ones. But still, it's I think it's a these notebooks they come both with, like a new power and accessibility, but also their own challenges.

12:05 Yeah, exactly.

12:06 Yeah. So in your paper, one of the things you will talk about is you said around notebooks, they have these several themes, they have End User Programming, exploratory programming, live programming and literate computing. So what do you mean by that? Like live programming, I understand End User Programming, which is pretty cool, exploratory. Make sense? But what's the live here?

12:28 So yeah, so there are a few kinds of definitions of live programming. One definition comes from kind of the creative space. So there are these people who are graphic artists, or musicians who actually literally, you know, they're streaming on Twitch, or they're doing these, you know, pre COVID, I guess they're doing these live performances on stage where they're using some kind of graphical software to, you know, Maximus p or whatever, to put together live music performances, or graphical art, another form of live programming that I think what we mean in this paper is more like, like a programming environment that updates live, right. So as you're typing the code, it's actually just constantly running it, and generating output. So some of the notebook systems we've looked at, allows you to kind of do this live programming where as you start typing, it might autocomplete, it might start generating graphs for you, just to kind of cut down this edit, run loop and such, right, right. So this I'm going to edit and when it's time to hit Run, I run or you know, maybe in some languages, there's a compile, and then a run. And there's a big delay, would something like, like some of the features in the editors like pi charm and probably VS Code? I don't know what plugin you need for it, but where it's like continuously running unit tests, just as you type base, yeah. liveliness, right, the kind of, you know, whatever. And even like, you know, even the squiggly lines, right, the IntelliSense and, you know, linting, and those things, you know, anything that kind of cut down on the, the friction of having to write a bunch of code, because especially for beginners, right, so for beginners, and for people who might be scientists and people who may not be as programming experience, you know, one thing you often see beginners do they write a lot of code, or they write a lot of code, and they stand they had run, they're like, Why doesn't it work? Right? Whereas more experienced programmers kind of know to write a line, check it, print something out, make sure it works, then write the next line, write the next

14:10 section and such. Yeah, it's easy to do a bunch of work and then have it especially when you're new, and have it not come out the way you like, and you're like, well, now it's broken. Oh, no, what am I gonna do, right? It's small bites, of course, small steps. This portion of talk Python to me is brought to you by linode. Whether you're working on a personal project or managing your enterprises infrastructure, linode has the pricing support and scale that you need to take your project to the next level, with 11 data centers worldwide, including their newest data center in Sydney, Australia, enterprise grade hardware, s3 compatible storage, and the next generation network. linode delivers the performance that you expect at a price that you don't get started on the node today with a $20 credit and you get access to native SSD storage, a 40 gigabit network industry leading processors They're revamped Cloud Manager cloud.linode.com root access to your server along with their newest API and a Python COI. Just visit talkpython.fm/ linode when creating a new linode account, and you'll automatically get $20 credit for your next project. Oh, and one last thing they're hiring go to linode.com slash careers to find out more, let them know that we sent you. When you had live here, and you talked about the live sort of interactive, almost onstage type of stuff, I was thinking of Fox dot and the live like musical composition with Python have either you guys seen that? No, I haven't seen this. I have not

15:39 called Fox.

15:40 I'll put a link of a presentation. So there's this thing called Fox dot, which is Python based people can go up there. And they basically create little like symphonies like, oh, how to talk to them in with Python. And it's super interesting. I was gonna ask your thoughts on that as a sort of HCI person. But yeah, yeah, I'll get it next time.

15:58 Yeah, I grabbed. Yeah, I'm just looking at the page. Now. Yeah, that's a whole other world. I mean, it's an awesome Well, I mean, this whole world of just live performance, and you know, live streaming and stuff is fascinating. Yeah, but maybe the next twitch stars are going to be data scientists, right? Who were, you know, live streaming their, their exploratory analysis, there are some data scientists I've seen who do these live streams, or do these, you know, recorded sessions, like, let me just play with this data or do this kaggle thing, and just watch me play with it, which is, which is really cool. That's really I

16:26 think you're right, I think it's super interesting. I feel like a lot of algorithms and libraries, and just software products, I guess you'd call them. It could be very stale, you know, much like math or science, like, here's the final result, here's the algorithm or the formula. But that doesn't give you an appreciation of like seeing the How to die like Bumble around and like discover and think and try and explore and then come up with an answer. I think that'd be really interesting for data science and like kaggle competitions and stuff.

16:55 Yeah. And I think that's segue so that our notebook work, too, because I feel like a lot of these notebooks maybe Sam would talk more on this later is that like, a lot of these sorts of interfaces could support this sort of lightness and this sort of iteration and keeping different versions around and seeing how you, you know, Bumble, your way toward a final result.

17:11 Yeah, maybe the next big particle physics discovery, Nobel Prize, I'll be live streamed. All right. So let's get to the the analysis. So you said there were a bunch of different environments that you studied. And you broke them into three categories, the academic world, the product world and the RNG world. Maybe give us a sense of what's in each one. I don't know if we want to go through all of them, but read off maybe like five from each of those that you feel like is representative of that area, just to give people a sense of what's all out there.

17:42 Yeah, so probably start with the industry world, because that world is probably the most familiar to us, and maybe to listeners as well. So in the product world, we kind of have Jupyter notebook. And we also have like Google collab, which is Google's version of Jupyter notebook, as well as Azure notebooks, which is Microsoft's kind of cloud offering of Jupyter notebooks. We also put down in this category, data bricks notebooks, as well as observable notebooks, which is kind of like this up and coming notebook system for JavaScript. Okay. Oh, right,

18:15 right. Yeah. There's this move to try to make a JavaScript equivalent, basically, of what Python has, right? I mean, it's pretty natural to run JavaScript. But the trick is to find the libraries, right, exactly. Yeah, it may be integers. Yeah. No integers in JavaScript can make that a little hard, but still, very cool. All right.

18:37 And then next category is academic notebook projects. And these projects typically come in the form of like papers, so people will prototype some interaction, and then submit a paper and human computer interaction conference. And these papers typically have like, they take a base notebook system and extend it in one particular way. For example, the Kalista system takes Jupyter notebook and adds in like, some live collaboration tools like a chat window, as well as I think being able to see other people's cursors and where they are on the screen a little bit like a Google Doc

19:14 ification of Jupyter.

19:15 Yeah, yeah, that's what they're going for. We have tools like Rex, which is a tool specifically for data cleaning. So the idea there is that you would, you could display data frame in your notebook. And then, by example, show the show Python, in a sense, like what you want the data to look like after data cleaning, and you can try to infer the Python code that would generate that data, change the data in the way that you want. That's pretty cool. And so it uses a bit of machine learning or something.

19:45 Yeah, use a what's called program synthesis. Yeah, that actually was a project led by Ian who is my other PhD student who is a co author on this paper that he did this at a Microsoft Research internship. So they have this some program synthesis technology, they have this you know, engine that basically synthesizes pieces of code. And they apply this technology to notebooks. So what you can actually do is you can you know, in a data table, you can say, here's some examples of the way I want to clean the data. And then oh, infer, I don't think it actually uses machine learning because it's not database. It's kind of rules base. But right, right basically infers a little snippet of like, here's the Python dot split dot strip dot v, or whatever regular expression that gets you that transformation. So yeah, go ahead, Sam, continue with the academic ones. Yeah,

20:28 there's a project called verdammt, which, not only Well, it's a kind of, it's like versioning plus plus. So verge is a notebook, not a notebook file itself. But no versions, each cell individually and even like, individual Python tokens within each cell. So you kind of have like these experimental, but also like you look at and you're like, well, it could be it could be useful in a real system. Yeah.

20:53 Yeah, I could see that for sure. You maybe drop a marker like, Hey, I'm gonna go explore this might want to like a save point type of thing. Maybe you want to get back here. Exactly. And then right, it'll just sort of that's like, behind the scenes. We're going to create a git branch and start doing stuff for you. Right? Like, that'd be really nice for people who are, they don't know how to like deal with Git branches or want to deal with it and stuff like that.

21:14 Yeah, that's exactly the use case that they're going for. It's for this very exploratory early stage like data exploration. Yeah.

21:22 Yeah. Okay, cool. The one in the product space that seems to be closest to that. It's gigantism. I had those guys on the show. And there's a little bit of like, auto versioning collaboration stuff in that world.

21:34 Yeah. Yeah. With the gentleman this paper, anything, we did mark it down as being relatively similar to that.

21:39 Yeah. Cool. All right. The third category was experimental and r&d.

21:43 Yeah. So experiments, r&d is, in a sense, very much like the academic projects. And that they are, they're trying to test out new interactions and new features that might not like don't have immediate, like, industry use cases, or maybe like business propositions. And so we have tools like carbide is an interesting one. carbide actually allows you to create widgets for your Python variables in line with your Python code, and then allows you to like use sliders to change variables on the spot. And okay, it's like way out there. It's really interesting, but you can actually infer is like, what values are Python variables should take on to produce an output. So if you use a widget to say like, Oh, I want y to be like 32, then if y depends on x, then it also it'll also update x at the same time. So it does like this weird, it doesn't really like intriguing, like program synthesis or like program like inference, like behind the scenes to get your program to look right, right to look at the dependencies and figure out what so we run. Yeah, yeah, that's really cool. It sounds a little bit like the interactive widgets and Jupyter that you can have, you know, you can sort of put some sliders and adjust some things, but I don't believe the Jupyter one has a lot of pendency type of stuff on down the line, right? It's just like this one sells, you slide the widget recomputes Yeah, yeah, like carbide is like those Jupyter widgets. But super next level there, we just can change the code itself. Okay, that's pretty wild. Yeah, we have a few industry projects here as well. We have projects from like Netflix and stripe, where they take notebooks, but use notebooks as like, kind of like recurring jobs, instead of just one off like exploratory scripts, where they use notebooks as these jobs. And so they can read, the notebook might automatically rerun, like once a day to perform some data operation, to kind of like broadening the scope of what Novus can be used for in a large scale setting.

23:40 Yeah, you know, the stuff that Netflix was doing with like paper mill was super interesting how they're using it to automate a bunch of stuff and sort of schedule a bunch of things to work, but also, the fact that if it fails, the notebook has the output of the failure. So basically, you can, you know, save the notebook in that state and just go look at it and see what went wrong, rather than there's a log message that says something went wrong, like, there's the whole sequence with the history of the notebook output, which is pretty awesome.

24:09 Yeah, I think the paper mill stuff, and we'd have a blog post about it, but we link to all of this in the paper. So if you link to our paper, we have all the URLs. I mean, I thought that was interesting. I mean, we didn't really capture this in the, in our space, because, you know, most of it is for data scientists, but the paper mill projects interesting because it's using notebooks for like DevOps and for, you know, production deployment work, right. It's like generally using the notebooks as a kind of a DevOps II sort of tool rather than direct data science tools. I thought that was a really interesting and out of the box use case, it's like instead of running cron jobs and seeing like gigantic log files on your terminal or tech grepping through text files, you can just you know, reconstruct a notebook and actually use a notebook to debug your production software, which is cool.

24:50 Yeah. And the essence of that, I think was one of the things you do with paper mill, if I have it, remembered correctly, is the notebooks can take inputs and they can generate output there. Like sort of become functions in the traditional blackbox sense of it. Yeah. Yeah, that's pretty neat. Another one I see that you have in there is indie Dev. Indie dev is a really interesting project. I just, we talked about that Brian nock, and I did a couple episodes ago on Python bytes. And I was like, Oh, this might be kind of interesting. I looked at it. Wow, this is something really quite special. He does a lot of neat things over

25:23 there. Right? So if I'm remembering correctly, MB dev allows you to take a notebook and like deploy it as a Python library.

25:32 Yes, exactly. Yeah, you can basically

25:34 turn it into a package, right. And it can have the test like built into it. And yeah, it's it's, again, a really interesting example of, like, notebooks are kind of being reused, and all sorts of ways that we would see traditional Python scripts being reused. So in this case, instead of writing a set of Python files to become a Python package, you could write a notebook instead. And that could be like your package. Yeah, I'll

25:56 give you the quick rundown for people who are listening might be interested just off the like their little highlights. So you can export the function of a notebook to a library, they have a COI commands you can use to interact with envy Dev, export to HTML builds documentation for your library. Sync brings the Python so you can edit the Python code and then bring it back into your library or you can put the tests in there and run them in ci, there's just a button that also has this like envy strip out concept as well for checking stuff in. So yeah, that's I mean, there's a bunch of exploration and innovation happening here.

26:29 Yeah, it's really interesting. One other interesting project in the experimental section is like iodide. iodide is others like C for experimental. Yeah. But they, they compiled Python and some of its data science libraries into webassembly. And then, because of that, you can run you can write Python in the browser. And it doesn't require a separate Python process to run that. Instead, it runs completely within the browser, because your browser is interpreting the webassembly version of like, the Python interpreter. And like, I think you have, like NumPy, and like, not positive support in there. So yeah,

27:05 they've re compiled many of the core, what would be C based Python libraries as well, right? Like you said, like NumPy, and maybe matplotlib, I can't remember. But some of those types of libraries, right, it's really

27:17 took a while for me to wrap my mind around it. But I was really amazed when I first saw that I think

27:21 webassembly has some real interesting possibilities to bring a lot of this stuff together. You know, it's one of the things that's interesting about some of the stuff you were covering in these different projects here is just what it means to run the notebook. Right? Sometimes that means a Docker image is spun up and configured exactly for you in some environment, or even like with gigantism on your local machine, with Google colab. There's like a way to say run this on a GPU, by the way, and just get a GPU to run it on and iodide runs in the browser through webassembly. There's just there's a bunch of options here that are pretty cool.

27:56 Yeah, kind of to Michael's point, it's like, you know, we've just informally kind of starting this project, we've kind of heard about all these, it's felt like the Wild West, right? Like, oh, you can run the notebooks locally, you can run it some JavaScript, observable ones, run the browser, and like, you know, iodide pie, or die compiles to weather satellite runs the browser, and then you need some other ones in the cloud. And then some of them have this live editing, some don't. So that's why kind of motivation for putting together this paper was, you know, how did we just categorize all this in the most kind of concise way we can, and I mean, talk about more detail that we have one category about runtime environment as we categorize. Here's the four types of main types of runtime environments people have another thing, so they're kind of main contributions paper was really, people have all these intuitions they've heard about these notebooks doing all these different things. Let's just try to plop them all together. And you know, map in a sense are like patterns, everything lie on this, this wild, this high dimensional space.

28:48 Yeah, is highly dimensional. Okay, talk about the dimensions in a minute. It's more than three. This portion of talk Python, to me is brought to you by century. How would you like to remove a little stress from your life? Do you worry that users may be having difficulties or are encountering errors with your app right now? Would you even know it until they send that support email? How much better would it be to have the error details immediately sent to you, including the call stack and values of local variables as well as the active user stored in the report? With century This is not only possible, it's simple and free. In fact, we use century on all the talk Python web properties, we've actually fixed a bug triggered by our user and had the upgrade ready to roll out as we got the support email. That was a great email to write back. We saw your error and have already rolled out the fix. Imagine their surprise, surprise and delight your users today, create your free account at talk Python, FM slash century and track up to 5000 errors a month across multiple projects for free. If you use the code talk Python, oh and word. It's good for two free months of centuries team plan which will give you up to 20 times as many monthly events instead. Other features, so create that free account today. One thing I do want to talk about maybe first is just like some of the challenges of notebooks. And then the Paradox of Choice. So, first of all, I think this comes from the paper, but also some of the feelings I had. So some of you talked about some of the challenges, Sam being stale data, out of order, execution, abundance of code, and then the inability for composition, things that papermill, for example, are trying to solve. You want to speak to like some of the challenges you saw on the space? Well, one of the first challenges that a user of Jupyter notebooks encounters is typically the out of order execution problem, where, because you can run cells in any order you want. what might happen is you run some cells, and then you delete some cells, and then you write more cells, and you run those in some order way. And what happens is, well, one problem is that sometimes your variables just get changed, and you don't know where that change came from. Because your code might be gone. Right? You might even delete the cell that define them. But yeah, yes, the kernel still running. So they're still in memory. For now. It's gonna be okay. Exactly, exactly. It

31:07 happens. Really, really often, in my experience, when we teach students how to use Jupyter notebooks for the first time, they hit some random keyboard shortcut, and the cell is gone. And let's see the code running fine. But then when we, when they try to turn into code, everything like kind of goes up in flames. Yeah,

31:23 well, but it's so much the power of Jupyter as well, right? Like, there's some reports and stuff I have for my business behind the scenes, where there's like, a 32nd, bunch of reading data, correlating data, and then you want to graph it and slice it and analyze it. And the ability to just go rerun the cell that sorts it this way and shows that aspect. It runs instantly, right? It just goes and goes and goes in. It's so powerful to not have to rerun that code to be able to like change your analysis. Yeah, you're exploring it right.

31:56 Yeah, exactly. That was the selling point of Jupyter. That's what I've got everyone hooked on Jupyter to begin with. Yeah, like that. Do the expensive part.

32:03 Exactly. Yeah. But it's like the world's worst go to and then some, you know,

32:09 sure. It's like the human powered go to right. Like you're like yourself, like running the saws and go to like, statements yourself.

32:16 Exactly. Like with a real go to it. It might be really hard to understand. But you can literally go through their code.

32:22 Yeah, at least deterministic.

32:24 Yeah, it's at least deterministic. Right. But yeah, it's not even with notebooks. So that's a challenge. Yeah, that's the biggest challenge. Yeah, it's this sort of out of order stuff. But

32:32 it's so hard to say like, well, let's just not do that. Because it's also the superpower. Right? Right. One of the dimensions in the space. I don't know if we're getting there quite yet. But one dimension that specifically addresses this, because a number of projects that we see from, from academia and from industry try to, like, address this specific, in order out of order execution issue. projects, like paper mode, which we briefly talked about before, addressing a different way, by trying to just like, make sure that whenever a notebook is run, it only runs from top to bottom like computer would rather than how cimi my Okay, my execute it. So this different angles on how to get around out of order execution thing, but as you mentioned, it's like, it's like the main benefit, and also the main weakness of these notebook systems.

33:16 Yeah. Did you guys study streamlet? stream? Yeah, it. So streamlet has a really interesting way of solving that, in that they basically use func tools, LRU cache, they know exactly that. But they more or less, the concept is the same. They've got these different functions that more or less act like cells. And when you rerun it, if you give it the same input, it uses the cached version. But if the inputs happen to change, then it'll give you a different output. So it's, it sort of keeps that, you know, don't recompute the stuff I already know,

33:46 aspect. Right? Right. So streaming has an interesting execution model that I think we also noticed and some other systems where it looks to you as a user, that whenever you make a change your your notebook gets to be read from top to bottom every time. But then behind the scenes, it does some smart caching and computing to to avoid the work. So you're gonna get the benefit of running a script every time without the drawback of having to like wait for you like, wait a long time for some expensive operation to compete.

34:16 Yeah, yeah. Interesting. This is a big challenge for all these things together. There's, I mentioned the Paradox of Choice, which is a fantastic book if people haven't read it, but I feel like one of the challenges that things you all may experience is if you look at all these different things, you're like, oh, Rex does this amazing thing and Callisto does that and streamlet does this other thing. And then Jupyter has this amazing aspect. Not any single one of these has the all the good things you want. But any choice down one path will make you feel like well I don't get the machine learning like auto cleanup in this one and I don't get the you know, the cursors like, do you feel like there's a bit of a fear of missing are feeling of missing out as you like have to go down. You got to use it. You can't use them all.

35:01 Oh, yeah, for sure. Nowadays when I use Jupyter, I mean Jupyter, the system I still use. But now that I know about all these other systems, oftentimes when I use Jupyter now I'm like, dang, if only I had, you know, nexternal If only I had like, databricks right now.

35:15 Yeah. But then you'd want JupyterLab back for something else, right? I mean, it's like, there's a lot of different things going on here. It's a lot of flowers blooming, I would say, I guess.

35:23 Right. Right. So we talked a little bit the end of the paper and the discussion part about, you know, what, if you did this kitchen sink, right, whatever, you just threw everything together where someone made a notebook that has everything. And, you know, one, it's an enormous software engineering challenge. I know, nobody would want to take on that challenge of maintaining 500 different sorts of features. And the other one is that it just made make the interface really complicated, right? It's like, yeah, if you can choose to imagine, like, you can choose different kinds of execution orders, or choose a reactive versus a in order what and you choose all this stuff, like, it would just be really, really hard to, for people to set up. So you know, some of these notebooks systems, you know, say Jupyter or some other ones. I think they've been successful, because they've, I would say, honestly, straightforward, but they their feature set is limited, right? That you know, right, right. It's, even though Jupyter is out of order issue, it's like, Okay, what do you see is what you get, as long as you execute in that order.

36:16 Yeah, and some of the real power is you can have semi structured, reusable, tiny bits in, say, cells, when you're just a biologist or an economist, and you don't have to become a computer scientist person to like, learn Haskell to take advantage of this thing. Like it's possible. Notably, yeah, it's incredibly accessible to just jump in and write a few lines of imperative code, and then get some really awesome output. And so, yeah, I feel like if it became too advanced, right, you're like, Well, do you want the reactive model, the asynchronous model, you're like, I don't know what this means. I just want a graph. Right?

36:53 Yeah. Because I mean, one of the challenges would say observable, right? I mean, observable has this really elegant, reactive thing where the stuff auto updates, and they have a lot of examples right on their website, but I still think it's gonna be hard for regular people, regular programmers to pick up because it looks kind of like JavaScript, but then when you write it, it's like, it has all these, like little extensions to JavaScript that you need to wrap your head around. So you know, it's the age old thing of, you know, more powerful programming languages, environments are just going to be higher barrier to learn, right? So it's very hard in practice. Yeah.

37:24 So let's talk about the dimensions. Sam, you've put together 10 different dimensions on how you sort of evaluate these things. And you know, a spectrum along them, for example, like data sources, you have local files, cloud storage, large data, streaming data, I want to run us through this.

37:41 Sure. We started with like the highest level breakdown, we organized the design space dimensions here around the steps in the workflow of a data scientist. So as a data scientist, we imagine that you might start a project by importing your data, and then writing editing your code and your prose around that around the code. And then you would want to run that code. And finally, you want to publish a notebook in some way. So it's like the input, editing and output steps for a data scientist. And those like the workflow steps and the 10 dimensions that we pulled out, more data sources you mentioned, and things like versioning, and collaboration, as well as execution models, and execution, liveliness. And for publishing a notebook we talked about, like, you might want notebook as a static HTML page, or maybe as a as a software package. As we alluded to earlier, there's a lot to consider here, this, just maybe grab a couple and to give people a sense, then they can go look at the paper, you've got a big full page, graph chart type of thing. Yep, chart, I guess. For example, I mentioned data sources, right? We've got local files. And you see, okay, this is access local file system. And you always give an example of a system that implements that, like so our studio, for example. Right, right. What else is on that access there? Yeah, so we have local files, which means that the system or basically only like, natively supports opening files that are stored locally on the same computer. Some notebook systems also allow you to read in cloud files it follows on on their servers as though they're local. This is what Google collab does, it allows you to reading Google files from Google Drive as if they're stored locally. alongside a notebook. We have some systems that have some special handling for large data sets, like the SS that don't fit in a computer's memory, like data, bricks uses spark to handle that, right. And we also have some systems that support streaming data. So like, you might hook up your notebook to some like web socket on the internet. And as new data comes in, your notebook will automatically rerun it update to reflect the latest original versions of this data. Oh,

39:53 like if you want to set up a dashboard, and as things behind the scenes change, like the notebook is always on. up to date with the state of the world polling almost, but better.

40:03 Yeah, exactly. So you can kind of see from the way he described that, that every single like dimension here has some use case. And the use case is kind of motivated by some real world. Like, somebody sat down and was like, man, I don't like reading my notebook deltin my dashboard every time I just wish I had some way to like, have on nicely, like streaming the data and update itself.

40:23 Yeah, yeah,

40:24 for sure. So let's go just through two more, and then I'll let people go check out the details for the rest. So how about execution order?

40:31 Yeah, so execution order goes directly to the in order out of order problem that we were talking about earlier, where that Jupyter notebooks allow users to run cells in any order they choose. So I can jump around cells freely. Other systems like observable require a certain order of cells. So what observable does is, takes a cell looks at all the dependencies like the variable dependencies of that cell, and enforces a notebook to run it in the topical, logically sorted order of cells, right, depending on like, what cells to run should depend on other cells. We also have systems extremely, which force like this in order execution model that we discussed earlier, where it looks to the user, like the cell, like the notebook always runs from top to bottom, every time. So those are the three, three main variations we found. Ordering execution. Yeah.

41:23 Interesting. Let's go to the very last part, the end of this life cycle, if you will, and talk about notebook outputs.

41:29 Yeah. So notebook outputs, we found the most common use case, I think, reflected across all of our systems was taking a notebook and publishing it as a static report, so to speak. So it's essentially like taking a notebook and converting it to HTML, and emailing the HTML file or putting it up as a webpage somewhere. The next step up would be like a dashboard, where you don't have a we don't show viewers the code that you might, you might require to produce two charts or graphs, you just show you the charts or graphs like themselves, right. Okay. And then finally, within that space, we have like, what we call software artifacts, which encapsulate the idea of you can use notebooks as cron jobs, or you can use notebooks as software packages.

42:13 Right, right, like indie dev creating something you could put on pipe.

42:16 Yeah, yeah, exactly. Exactly.

42:18 Yeah. Yeah. I really think it's cool that you broke it down like this. But then each one has an example type of things. You're like, Okay, well, how does say gigantism? Do it? Or how does streamlet do this thing, or observable and so on? And yeah, I think actually seeing all this stuff together might inspire features start cross pollinating. Hopefully, not every single feature, right. That would be too much. But

42:43 the kitchen sink, powerful. Kitchen Sink notebook. Yeah.

42:49 Yeah. I mean, maybe kitchen sink is available on pipe? Yeah. Like you could just

42:53 install kitchen say, Yeah.

42:55 Because you have

42:57 half an hour.

42:59 kitchen sink. wait five minutes.

43:01 Awesome. All right. Well, this is a really interesting view, I guess, you know, maybe let's kind of close out the conversation around this stuff by asking, like both each weigh in on this. What surprised you like as you went through this, what did you expect? And what was different than that? What surprised you as you went through this? I mean, first thing that surprised me is how many there were but

43:21 what else? For me? I think it was this tension between we hear about the problems of notebooks quite often. And yeah, we, from this analysis, we don't often see those desires reflected in actual products. So the in order out of order execution model, for example, the vast majority of the systems, we looked at only allowed any order execution where the user defines execution. Collaboration between users and notebook authors is relatively common, like in a company, you rarely find someone working on notebook all by them loans by their loans himself. But at the same time, we see very few systems that support like the sort of real time collaboration that Google Docs has made popular.

44:00 Right. And it seems so, so easily on the cusp of that because it's already in the browser. Often it's already hosted. Right, right.

44:09 Right, exactly. So it kind of reveals, I think, as a whole, the analysis reveals that there's still like room for improvement for a lot of these products.

44:18 Yeah. Yeah. And I guess, I don't know, if it was a surprise as much as you know, something that, you know, we mentioned within the paper was, you know, on the, you know, Sam kind of mentioned the product side of, you know, practically speaking things are pretty tied to Jupyter like things because people are used to that. And a lot of products are Jupyter extensions, or Jupyter hosted on Azure, Jupyter hosted on AWS Jupyter hosted on Google Cloud. And so my kind of focus is thinking about the academic side, think about the research papers that people have been prototyping for this. And, you know, I don't have as a surprise, but it's something that was kind of a call to action we had at the end of the papers that, you know, I felt personally that the academic work was not sufficiently weird, maybe right. That's all you want. You kind of you You know, one of the roles of academic research and prototypes is you want to push the bounds and try stuff that's very different, right? So imagine experimental programming language or experimental toolkits, know, the point I need to be practical. The point is to stretch the limits of what we know. So, because I think notebook research is still pretty early, it's all these systems in academia have been done probably within the last four, four or five years. It makes sense at the first wave is fairly kind of, you know, making the extensions that people expect, right version a in order execution, collaboration and stuff. So I think that what would excite me more to see in the coming years is this, just to see the academic projects or the r&d projects becoming more weird, right? So one of the things that we pitch at the end is just thinking about multi device, different kinds of off desktop devices, right? We already were computing with phones, and smartwatches, and augmented reality and displays, but we're still just hunched over in front of our laptop, typing in just literally just typing in a text box in the web browser, right? Like, that's all the notebooks are, they're just little text boxes and web browsers. So I think we can really expand our imagination thinking, you know, way beyond that would excite me a lot more.

46:06 Yeah. Very interesting. Yeah, I'm into echo that point will be I drew out the notebook systems was like this data science workflow. But the reason why we drew out the data science workflow is because I think every single system that we've evaluated here was was designed for data scientists, for the most part. And so with data scientists as the audience of choice, I think they're necessarily like, it comes with like a limitation on what we can imagine notebook be used for. So in the paper, we talked about broadening notebook use cases for other types of users, like, perhaps instructors, for perhaps like maybe like, artists, let's say, and each of those audiences has probably an entirely different set of desires and things they want to accomplish with programming. And I think that comes like a whole nother set of trade offs, and, and perhaps design dimensions for notebook systems. That's a good point. Yeah, like DJs musicians, orchestras,

47:03 book, like, what kind of world would you build if you were trying to, you know, coordinate that kind of stuff? Philip, you brought up our phones, which are pretty incredible these days. Did you guys look at carnet? I mean, it's not really something that needs separate analysis, because it's just basically Jupyter, but Carnot car in ETS, which is a open source thing, on least a free thing? I'm pretty sure it's open source for Jupyter running natively on iOS. Oh, interesting.

47:34 Huh? Yeah, it's

47:35 pretty new. I just learned about it a few months ago. Still. Yeah,

47:37 that's on GitHub. yeah, you should after analysis, if we go cool.

47:44 It is about I mean, it's not really fundamentally different. But it's interesting that it has NumPy. It's just on the platform. Yeah. And a lot of those libraries, like natively executed on iOS not like a interface to some cloud thing. But like, legitimately there, you know,

47:58 yeah. I mean, already, people are talking about, you know, on our mobile phones, right, that, you know, for machine learning, right? They you download these pre trained models, so then you can just do inference on your phone without going to the cloud, right? So both for privacy, right? If you want to do image recognition, both for privacy and for bandwidth, you can do all this computation on your phone locally. So you could imagine, you know, what would it mean? If you could do data science out in the world, right? You have your smartwatch, you have your phone, you could just you know, do your analysis, your Google Glass or whatever? It is this kind of sci fi future? Yeah, I guess like the thought that that we have as kind of human computer action or HCI researchers is, you know, what would the future of computing look like right, right now? It still looks like people hunched over on their laptops, typing and text boxes, right? Like, you want to go beyond that somehow still waiting to just plug

48:43 in interface, right? Yeah, neck. Right. No more typing.

48:47 For JSON to read data. You can you can Jupyter in your eyes. Exactly. Sign me up. Awesome.

48:56 You're maybe not the first. But early adopter. Let's put me in that category. All right, you guys, this was super interesting. It definitely opened my eyes to some things that are going on out there that I had no idea about. So super cool. Research. Thanks.

49:09 It was really fun for us to talk about it.

49:11 Yeah, for sure. Now, before we get out of here, I'm going to ask you the two questions that go at the end of the show all the time. And I was Philip. I'll pick on you Since you're a veteran. If you're going to write some Python code, what editor do you use? I still use vim because I haven't learned anything new. So I still use vim with whatever defaults. Right on. All right, and then notable pi package.

49:32 Something that you've used is interesting. I'll do a plug for Sam's actually. So Sam, for his master's thesis at Berkeley, he made this package called MB interact, which is related to Jupyter notebooks. It's called MB interact, and you can pip install it. So I'll let Sam talk more about that. That's a lead into Sam's

49:50 Nice. Yeah. Cool. All right, Sam, if you're gonna write some Python code editor,

49:53 yeah, so I use VS Code, but I've hopped from like, I pop from sublime to Emacs to vim, and then not to VS Code. So Non happy with me happy VS Code user. Alright, cool.

50:03 Do you do any bindings like you'd like setup like vim bindings or Emacs bindings, and there's something

50:07 I said I'm inviting. But if the VS Code team is listening, there are some problems I have with the vim bindings in VS Code that I love about.

50:15 This is a call out section where we start the beef. They call out section at the end of the episode. All right, it's, it's the

50:22 underhanded compliment. Not exactly what it's like, I love it. But

50:26 yeah, now VS Code is a good one. All right, you want to tell us about NB interact? Sure me interact. It lets you take a Jupyter notebook with some widgets. And normally those widgets don't work outside of the notebook environment. Like if you convert the notebook to a webpage to HTML page, by default, it will just will just break because you don't have a Python service running underneath the notebook. But the interact allows it kind of like takes your HTML page and hooks it up to the binder service from the Jupyter team. So that you can have a you can have like this HTML page with interactive widgets, but doesn't require you to have notebook notebooks. Everybody locally, so you can kind of like share those things. You can share your interactive notebooks more readily. That's the idea. I see. You don't have to set up a proper Jupyter server with all the execution and stuff. Yeah, you're gonna send someone a link to a webpage, and then they can run your interactive widgets. That's good. Okay. Yeah. Excellent. That's my personal use. I like the TQ dm package. It gives you like a progress bar. And when you run when you run, yeah, it's really handy.

51:24 Yeah, that's really cool. There's a couple of interesting progress bars that you can use, but it's those little touches, you know, color progress bars, maybe a few emojis, it just makes it feel so much nicer to work in. c li up. Yeah. Our next paper is analyzing 60 progress bar libraries.

51:41 Yes. The 16 bar libraries and

51:45 they use emojis.

51:46 Yeah, perfect. Perfect. All right. Well, you guys, this is really fun. People want to know more about your research your paper, what do they do? How did they find it? I'll put a link in the show notes. But what else?

51:56 Yeah, they can just go to the links. Michael, I sent you a link in the chat about NB interact. And then we sent you the paper links. And such we can put both of our home pages on there. And such Sam any other plugs?

52:08 Yeah, I think you remind me that we haven't put up the paper on I don't know if you have the paper online anywhere. Yeah, I have it on my page. Yeah. Oh, you do? Okay. So we can maybe link to the paper if it goes pitch.

52:19 Yep. Sounds good. Well, I'll definitely do that. Or you guys, thank you so much for being on the show. It's been really interesting. And yeah, there's a lot of a lot of notebook exploration and flowers blooming. So thanks for putting it all together into like, one analysis here.

52:34 Yeah, really big. Thanks to you, Michael, for having us on. And it's really, really fun to talk about all the flowers, all the fields of flowers.

52:42 Yvette Good work. Likewise, thank you so much, Michael, have a good day do later.

52:46 This has been another episode of talk Python. To me. Our guests on this episode were Sam Lau and Philip gwo. And it's been brought to you by linode. And century. Start your next Python project on the nodes state of the art cloud service. Just visit talkpython.fm/ linode li n od E, you'll automatically get a $20 credit when you create a new account. Take some stress out of your life get notified immediately about errors in your web applications with century. Just visit talk Python FM slash century and get started for free. Want to level up your Python if you're just getting started, try my Python jumpstart by building 10 apps course. Or if you're looking for something more advanced, check out our new async course the digs into all the different types of async programming you can do in Python. And of course, if you're interested in more than one of these, be sure to check out our everything bundle. It's like a subscription that never expires. Be sure to subscribe to the show, open your favorite pod catcher and search for Python we should be right at the top. You can also find the iTunes feed at /itunes. The Google Play feed is /play in the direct RSS feed net /rss on talk python.fm. This is your host Michael Kennedy. Thanks so much for listening. I really appreciate it. Get out there and write some Python code