WEBVTT

00:00:00.120 --> 00:00:05.860
Jupyter is an amazing environment for exploring data and generating executable reports with Python.

00:00:05.860 --> 00:00:12.400
But there are many external tools, extensions, and libraries to make it so much better and to make you more productive.

00:00:12.400 --> 00:00:15.120
On this episode, we're going to cover a ton of them.

00:00:15.120 --> 00:00:19.100
We have Marcus Shanta, the maintainer of the awesome Jupyter list on the show,

00:00:19.100 --> 00:00:21.620
and we'll highlight a bunch of Jupyter gems.

00:00:21.620 --> 00:00:27.160
This is Talk Python To Me, episode 394, recorded December 1st, 2022.

00:00:27.160 --> 00:00:43.780
Welcome to Talk Python To Me, a weekly podcast on Python.

00:00:43.780 --> 00:00:45.520
This is your host, Michael Kennedy.

00:00:45.520 --> 00:00:53.000
Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython, both on fosstodon.org.

00:00:53.000 --> 00:00:55.600
Be careful with impersonating accounts on other instances.

00:00:55.600 --> 00:00:56.580
There are many.

00:00:57.080 --> 00:01:01.620
Keep up with the show and listen to over seven years of past episodes at talkpython.fm.

00:01:01.620 --> 00:01:05.660
We've started streaming most of our episodes live on YouTube.

00:01:05.660 --> 00:01:13.200
Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.

00:01:13.200 --> 00:01:17.820
This episode is sponsored by the AWS Insiders podcast.

00:01:17.820 --> 00:01:19.880
AWS is changing fast.

00:01:19.880 --> 00:01:24.720
Listen in to keep up over at talkpython.fm/AWS Insiders.

00:01:25.280 --> 00:01:27.180
And it's brought to you by Sentry.

00:01:27.180 --> 00:01:29.140
Don't let those errors go unnoticed.

00:01:29.140 --> 00:01:30.040
Use Sentry.

00:01:30.040 --> 00:01:33.100
Get started at talkpython.fm/sentry.

00:01:33.100 --> 00:01:44.760
Transcripts for this episode are sponsored by Assembly AI, the API platform for state-of-the-art AI models that automatically transcribe and understand audio data at a large scale.

00:01:44.820 --> 00:01:48.600
To learn more, visit talkpython.fm/assemblyai.

00:01:48.600 --> 00:01:50.840
Marcus, welcome to Talk Python To Me.

00:01:50.840 --> 00:01:51.400
Hi, Michael.

00:01:51.400 --> 00:01:52.280
Thanks for having me.

00:01:52.280 --> 00:02:05.460
This is going to be a very broad, not necessarily super deep episode, but we're going to talk about a ton of cool little extensions and widgets and libraries that you can plug into your Jupyter work and make it awesome.

00:02:06.140 --> 00:02:12.460
I really like talking about these topics where it's like, oh, that's only a 10-minute commitment to see if it's going to help me out or not, right?

00:02:12.460 --> 00:02:14.760
And not like some huge framework you got to learn.

00:02:14.760 --> 00:02:18.640
And there's going to be a lot for people to take away if they do anything at all with notebooks, I think.

00:02:18.800 --> 00:02:23.220
Yeah, and I think one of the nice things about notebooks is they're very easy to use.

00:02:23.220 --> 00:02:32.040
I think the barrier to entry to you just being able to do something tangible with them is very low, much more so than maybe a lot of other things.

00:02:32.040 --> 00:02:45.360
Yeah, they're very, very welcoming environments, especially if someone set them up for you, if it's one of the hosted ones, or if you're not in there trying to configure a kernel with a special virtual environment, like, how do I get to the right installs?

00:02:45.360 --> 00:02:46.320
And why is this thing not there?

00:02:46.320 --> 00:02:49.880
Or a whole super-nature hub running on Kubernetes.

00:02:49.880 --> 00:02:50.780
Yes, exactly.

00:02:50.780 --> 00:02:55.160
But once you're kind of beyond that shell and many people have that set up for them, then it's ready to roll.

00:02:55.160 --> 00:02:58.080
All right, so going to be tons of fun to talk about these things.

00:02:58.080 --> 00:03:00.120
But before we get to them, let's start with your story.

00:03:00.120 --> 00:03:01.580
How did you get into programming in Python?

00:03:01.580 --> 00:03:03.480
How did I get into programming?

00:03:03.480 --> 00:03:10.600
I think it sort of started when my parents decided to buy a computer when I was eight years old, back in the 90s.

00:03:11.200 --> 00:03:16.600
And then sort of at 12, I started to realize, oh, you can program these things.

00:03:16.600 --> 00:03:23.500
And some well-intentioned adult, like, bought me a book on C and, you know, like 12-year-olds and pointers.

00:03:23.500 --> 00:03:24.500
It doesn't mix.

00:03:24.500 --> 00:03:26.660
C, the easy parts.

00:03:26.660 --> 00:03:27.760
Yeah, exactly.

00:03:27.760 --> 00:03:28.380
Just void, star, star.

00:03:28.380 --> 00:03:29.020
Come on, kids.

00:03:29.020 --> 00:03:29.560
Get with it.

00:03:29.560 --> 00:03:29.940
Yeah.

00:03:29.940 --> 00:03:30.260
Yeah.

00:03:30.640 --> 00:03:35.160
That was just, like, conceptual a bit too much for a 12-year-old.

00:03:35.160 --> 00:03:40.500
But then luckily, like, once I got into high school, they actually taught us Boral and Delphi.

00:03:40.500 --> 00:03:42.680
I don't know if she ever came across that.

00:03:42.680 --> 00:03:48.400
It was a very safe, easy-to-use, like, GUI-driven way to get into programming.

00:03:48.400 --> 00:03:53.960
And you could do a lot of things even that, like, a 14-year-old would find exciting, right?

00:03:54.120 --> 00:04:00.300
Just, like, C-style, hello world, and I can create numbers in a loop, but actually, like, fun GUI-like stuff.

00:04:00.300 --> 00:04:06.560
You know, I never used that, but I used Visual Basic and some Windows Forms and other stuff.

00:04:06.560 --> 00:04:13.580
And I really think, even today, that style development, it's kind of withered and it's too bad, right?

00:04:13.580 --> 00:04:16.240
It used to be so amazing.

00:04:16.240 --> 00:04:19.000
You would just go over there, you know, button, text box.

00:04:19.000 --> 00:04:20.320
I click on this thing.

00:04:20.320 --> 00:04:21.440
Here's what it calls.

00:04:21.440 --> 00:04:23.380
The interaction was so simple.

00:04:23.500 --> 00:04:28.560
And now you're doing, you know, maybe you're doing the web, and it's like, I'm going to do Webpack.

00:04:28.560 --> 00:04:34.480
And then, you know, bring in all these things, and I'm doing TypeScript, and then it's all CSS and HTML.

00:04:34.480 --> 00:04:39.320
And I'm not bemoaning that style of development, but it's like, that's kind of replaced this.

00:04:39.320 --> 00:04:43.060
Just give me a visual of what it looks like, and let me build something simple.

00:04:43.060 --> 00:04:44.560
It has been replaced with...

00:04:44.560 --> 00:04:50.520
I feel like it was a very good pedagogical tool to teach programming and GUI development.

00:04:50.520 --> 00:04:56.880
Like, I remember we did find things like rectangles on the screen that just chased each other around, right?

00:04:56.880 --> 00:04:57.420
Yes.

00:04:57.420 --> 00:04:59.620
That was a fun way into programming.

00:04:59.620 --> 00:05:00.060
Yeah.

00:05:00.200 --> 00:05:03.620
And I think there's some bits like that still around, isn't it?

00:05:03.620 --> 00:05:06.160
What's the, like, programming learning language?

00:05:06.160 --> 00:05:07.320
Something with an animal?

00:05:07.320 --> 00:05:08.360
Is it small talk?

00:05:08.360 --> 00:05:10.020
Or the thing with a turtle?

00:05:10.020 --> 00:05:10.720
Yeah, yeah.

00:05:10.720 --> 00:05:12.660
The turtle where you can do some drawing.

00:05:12.660 --> 00:05:13.200
There's that.

00:05:13.200 --> 00:05:13.780
Yeah, yeah.

00:05:13.880 --> 00:05:19.860
There's Anvil, actually, which is a Python sort of front-end and back-end web framework that has this.

00:05:19.860 --> 00:05:21.280
It's really, really similar.

00:05:21.280 --> 00:05:23.600
My daughter used that to play with it a lot.

00:05:23.600 --> 00:05:29.540
But yeah, I would love to see like a Visual Basic Pi or something, you know?

00:05:29.540 --> 00:05:30.980
Something like that would be so amazing.

00:05:30.980 --> 00:05:32.400
Maybe it exists, yeah.

00:05:32.720 --> 00:05:37.300
But that's where I really started developing or programming on my own.

00:05:37.300 --> 00:05:38.680
Then I went to university.

00:05:38.680 --> 00:05:41.000
That was in the early 2000s.

00:05:41.000 --> 00:05:48.680
And they taught us mostly Java, but also sort of a weird mix of like logic-oriented programming, functional programming.

00:05:49.020 --> 00:05:55.580
And I really only ever got in touch with Python when I did my master's in the States during some courses.

00:05:55.580 --> 00:06:00.320
And then professionally, I learned proper Python when I was working in London.

00:06:00.320 --> 00:06:05.500
And that was a company, Man Group, with a lot of really good Python developers.

00:06:05.500 --> 00:06:08.380
And there I picked up like proper Python.

00:06:08.380 --> 00:06:08.820
Yeah.

00:06:08.820 --> 00:06:10.660
And stuck with it ever since.

00:06:10.660 --> 00:06:13.680
If I have a choice, I will stick to it.

00:06:13.680 --> 00:06:17.900
It's interesting that we come from, you know, my background was not that different than yours in some ways.

00:06:17.900 --> 00:06:20.560
I did like C++ in the early days.

00:06:20.560 --> 00:06:29.600
And you think as you advance in your career and you get more technically capable and stuff, you would think, well, you're going to be doing more intense stuff and more like deep dive.

00:06:29.600 --> 00:06:31.040
Like now you're writing like kernel drivers.

00:06:31.040 --> 00:06:34.600
Like it's for me at least, and it sounds like you a little, quite the opposite.

00:06:34.600 --> 00:06:39.160
Like more towards some of these higher level languages that let you build really amazing things.

00:06:39.160 --> 00:06:44.360
But you're not juggling like pointers to pointers and like all this crazy aspects, right?

00:06:44.360 --> 00:06:50.620
I think the first thing, like your expectation that you're describing is maybe a function of sort of technical expertise.

00:06:50.620 --> 00:06:55.940
In the second development, what actually what you end up doing, right, is a function of age, right?

00:06:55.940 --> 00:06:59.780
That you like, as you become older, you focus more on the high levels things.

00:06:59.780 --> 00:07:01.020
You just want to get stuff done.

00:07:01.020 --> 00:07:02.400
You don't care about showing off.

00:07:02.400 --> 00:07:02.680
Exactly.

00:07:02.680 --> 00:07:03.800
You can make this compile, right?

00:07:03.960 --> 00:07:05.920
What's the shortest way for me to be?

00:07:05.920 --> 00:07:07.680
And very often that's Python.

00:07:07.680 --> 00:07:08.600
Absolutely it is.

00:07:08.600 --> 00:07:08.880
All right.

00:07:08.880 --> 00:07:10.040
So some real-time follow-ups.

00:07:10.040 --> 00:07:11.360
The audience has helped us out here.

00:07:11.360 --> 00:07:15.440
Don says, Delphi is a wonderful way back into Pascal from the college days.

00:07:15.440 --> 00:07:15.720
Indeed.

00:07:15.720 --> 00:07:25.000
And some of those languages, Logo was one of these visual building ones, as well as the one with the cat is Scratch, which, what an amazing name.

00:07:25.000 --> 00:07:26.160
I haven't really used Scratch, but.

00:07:26.160 --> 00:07:30.960
I think the cats or animals in general seem to be good metaphors for teaching programming.

00:07:30.960 --> 00:07:32.080
Yes, they do.

00:07:32.080 --> 00:07:39.000
And the O'Reilly publishers, the publisher O'Reilly, has built their entire book series on unique animals and programming.

00:07:39.000 --> 00:07:39.660
All right.

00:07:39.660 --> 00:07:40.600
Well, how about now?

00:07:40.600 --> 00:07:41.460
What are you doing these days?

00:07:41.460 --> 00:07:45.280
Right now, I am a founding partner at Blue Balance Capital.

00:07:45.280 --> 00:07:51.860
We are a small, independent, alternative asset management firm based in Vienna, Austria.

00:07:51.860 --> 00:07:55.800
I started this company three years ago with my partners.

00:07:55.800 --> 00:07:59.960
Before that, I worked at an asset management firm in London.

00:07:59.960 --> 00:08:05.060
Yet before that, I worked at Goldman Sachs as a quantitative analyst.

00:08:05.060 --> 00:08:14.320
And so basically, during my professional career, I was always more a sort of data analyst rather than a systems developer.

00:08:14.820 --> 00:08:20.840
So everything or most of the things I use Python for are more from the perspective of here's a piece of data.

00:08:20.840 --> 00:08:31.060
And I want to sort of analyze that data set to better understand, like, I don't know, a company or an economic trend or an industry or a particular trade.

00:08:31.540 --> 00:08:40.420
So I was always interested in Python in the context of, like, some data set and some insight that you can glean from analyzing that data set.

00:08:40.420 --> 00:08:55.260
How much of that was taking a block of historical data, like, here's the last quarters and, you know, reports or whatever, versus trying to make predictions, you know, like real-time trading or other types of real-time information?

00:08:55.700 --> 00:08:59.780
So I think sort of the starting point is almost always some kind of like time series.

00:08:59.780 --> 00:09:07.140
And then the asset management firm, Mancobura, they were actually developing automated trading systems.

00:09:07.140 --> 00:09:18.100
So it was like very much the first thing you described, where you have an input that is a time series, you apply some transformation, and then a computer actually produces trades on the back of that.

00:09:18.540 --> 00:09:33.280
Whereas at other times, more like what I do now is you think about it, you get some insight from the data, but then there's also some other real-world considerations or exogenous factors that you think about as a human person and then make your decision.

00:09:33.280 --> 00:09:44.460
So I think for me, it was the whole gamut of sort of the product is a direct trade that the machine puts on or trade that you put on or maybe just some advice that you give to a company.

00:09:44.460 --> 00:09:53.180
Right, right. We're thinking that tech, generally, that the tech indexes are going down over the next three months versus buy today and sell tomorrow, yeah?

00:09:53.180 --> 00:09:53.840
Or buy now, sell tomorrow.

00:09:53.840 --> 00:09:56.260
Predictions are hard, especially about the future.

00:09:56.260 --> 00:10:01.520
Yeah, well, especially now with COVID and wars and like...

00:10:01.520 --> 00:10:07.080
I have been in finance for more than 10 years now, but I've never seen anything like the last two years.

00:10:07.080 --> 00:10:07.940
Yeah, that's nuts.

00:10:07.940 --> 00:10:11.480
I think much more happened in the last two years than in the 10 years before.

00:10:11.480 --> 00:10:13.920
It is living through some history, isn't it?

00:10:13.920 --> 00:10:14.180
Yeah.

00:10:14.180 --> 00:10:16.800
One more question on this background side of things.

00:10:16.800 --> 00:10:22.380
You did a lot of work for these other companies like Goldman, and now you're founding a smaller company.

00:10:22.380 --> 00:10:30.980
How do you feel that your programming and data science background has suited you to be more of leading this new company?

00:10:30.980 --> 00:10:39.140
I think I have a great sort of education from both Goldman as well as Man Group.

00:10:39.140 --> 00:10:44.660
They're both very technically capable organizations with a lot of very smart people.

00:10:44.660 --> 00:10:54.800
And as I was starting out together with my partners as a very small team, one of the things that gave me comfort is that I have this stack of things that I know work together.

00:10:54.800 --> 00:10:59.520
I understand how they work together and how to sort of apply them to do useful things.

00:10:59.520 --> 00:11:07.140
So having that sort of under your tool belt or ready to go and not having to figure out sort of how they go together.

00:11:07.140 --> 00:11:11.360
But just on day one, here's an AWS instance or EC2 instance.

00:11:11.360 --> 00:11:15.360
Spin it up, get a notebook running and produce some nice charts.

00:11:15.360 --> 00:11:19.460
That was something very valuable for me to just be able to do very quickly.

00:11:19.460 --> 00:11:19.760
Cool.

00:11:19.760 --> 00:11:21.140
That's what I would think.

00:11:21.140 --> 00:11:32.840
And I feel even if you have these skills, even if you're not the one writing, if you hire a team or you find a consultant, being able to speak with them and understand like, yeah, no, no, I have a recommendation.

00:11:32.840 --> 00:11:38.580
And I think actually this, you know, like, let me tell you, that tool is good, but this one is better and it would fit because X, Y.

00:11:38.580 --> 00:11:40.340
I think just that's super valuable.

00:11:40.340 --> 00:11:42.680
So I just wanted to kind of check in with you on that.

00:11:42.680 --> 00:11:51.260
Yeah, like I tried and test a stack of tools and every once in a while you get to branch off and try some new shiny and see if it works, see if it sticks.

00:11:51.260 --> 00:11:55.040
But having this trusted thing that you know and how it works is valuable.

00:11:55.040 --> 00:11:55.380
Yes.

00:11:55.380 --> 00:11:59.940
Like if you're doing JavaScript, you want to have a really trusted tool that's been around for at least a month.

00:11:59.940 --> 00:12:00.260
Yeah.

00:12:00.260 --> 00:12:01.800
I mean, just kidding.

00:12:01.800 --> 00:12:04.540
Luckily, we don't have it quite that bad in Python.

00:12:04.540 --> 00:12:05.880
It's a lot more stability there.

00:12:05.880 --> 00:12:06.180
Okay.

00:12:06.180 --> 00:12:11.660
Well, let's talk about this list, which is why we got together and I invited you here.

00:12:11.920 --> 00:12:14.020
Is this awesome Jupyter list.

00:12:14.020 --> 00:12:14.740
You know, what is it?

00:12:14.740 --> 00:12:15.280
Where did it come from?

00:12:15.280 --> 00:12:20.480
I mean, most people are familiar with awesome lists, but maybe what's the philosophy of yours?

00:12:20.840 --> 00:12:21.060
Yes.

00:12:21.060 --> 00:12:30.320
So awesome lists, they're just like curated lists with resources that are useful or pertinent to some particular topic.

00:12:30.320 --> 00:12:32.560
And then in this case, it's Jupyter notebooks.

00:12:33.120 --> 00:12:40.060
So this is really a list of things that I started with 20 entries and then just put it up on GitHub.

00:12:40.060 --> 00:12:47.980
And then over time, just more people added to that list of things they find useful and have some relationship with Jupyter.

00:12:48.480 --> 00:12:55.240
And I think that up to this point, I think more than 100 people have collaborated on other things that they find useful to this list.

00:12:55.240 --> 00:13:01.240
So it's just like living, breathing thing of whatever people find useful that has a relationship to Jupyter.

00:13:01.380 --> 00:13:08.080
I still find these very valuable, even though a good part of my job is like to track what is new, what's interesting, what's trending.

00:13:08.080 --> 00:13:10.540
Still, I find so many things that are new here.

00:13:10.760 --> 00:13:15.280
And when I first got to Python, I was like, wow, look, it just keeps going.

00:13:15.280 --> 00:13:17.380
I just thought there was three web frameworks.

00:13:17.380 --> 00:13:19.760
There's one way to talk to a database.

00:13:19.760 --> 00:13:20.720
Look how many there are.

00:13:20.720 --> 00:13:21.480
It's so amazing.

00:13:21.480 --> 00:13:22.480
And it's always delightful.

00:13:22.480 --> 00:13:28.180
You probably remember like back in the old days of the internet, you had directories, right?

00:13:28.180 --> 00:13:29.100
I mean, that was Yahoo.

00:13:29.100 --> 00:13:31.220
That was the first search engines, right?

00:13:31.220 --> 00:13:31.540
Yes.

00:13:31.540 --> 00:13:33.460
You had these like catalogs of things.

00:13:33.460 --> 00:13:36.060
And here's a website that is about cats.

00:13:36.060 --> 00:13:37.500
And here's one about dogs.

00:13:37.700 --> 00:13:48.820
And in some ways, it feels like in this day and age, we have come back to like you actually have a person who is keeping some kind of like directory of things that are useful or pertinent to a particular topic.

00:13:48.820 --> 00:13:50.080
It's kind of funny that way.

00:13:50.080 --> 00:13:58.180
And what's quite interesting for me is sort of one of the benefits that I have from doing this is that I see like what other people find useful.

00:13:58.180 --> 00:14:02.540
And so for myself, I just know, hey, these are the things that people are using.

00:14:02.780 --> 00:14:09.420
And so I've got a pretty good radar of the whole Jupyter and Notebook ecosystem just because I'm sort of curating this thing.

00:14:09.420 --> 00:14:09.780
Yeah.

00:14:09.780 --> 00:14:15.040
You probably have had people recommend things and you're like, no idea what that is, but that looks awesome.

00:14:15.040 --> 00:14:16.540
So it belongs on awesome Jupyter.

00:14:16.540 --> 00:14:18.760
I try to be very inclusionist to this.

00:14:19.520 --> 00:14:25.560
So when people include things on this list, more often than not, I exclude them.

00:14:25.560 --> 00:14:33.680
And even in the cases where I'm like, I don't see myself using that, but I'm sure there's some category of people who might find that useful.

00:14:33.680 --> 00:14:35.320
And it just goes on the list.

00:14:35.320 --> 00:14:35.600
Yeah.

00:14:35.760 --> 00:14:42.000
You don't want to over-index on your specific use of Jupyter and your vertical, right?

00:14:42.000 --> 00:14:44.880
Because we've got astronomers who are using this stuff.

00:14:44.880 --> 00:14:45.980
We've got economists.

00:14:45.980 --> 00:14:47.560
We've got biologists.

00:14:47.560 --> 00:14:49.220
We've got students, right?

00:14:49.220 --> 00:14:50.500
It's publishers.

00:14:50.500 --> 00:14:51.420
All sorts of folks.

00:14:51.420 --> 00:14:51.640
Yeah.

00:14:51.640 --> 00:14:52.000
Yeah.

00:14:52.000 --> 00:14:52.380
Yeah.

00:14:52.380 --> 00:14:57.300
People who care more about, I don't know, like keeping, turning their computation on a cluster.

00:14:57.300 --> 00:14:59.120
Other people are more into visualization.

00:14:59.120 --> 00:15:00.940
You name it, you have it.

00:15:00.940 --> 00:15:01.360
Yeah.

00:15:01.360 --> 00:15:01.640
Right.

00:15:01.720 --> 00:15:05.200
Like ML folks might have one concern over others.

00:15:05.200 --> 00:15:18.720
A lot of people who use it in education, that's actually one of the sections in there is a whole section dedicated to education, people teaching courses using notebooks and what the best tools around that is for it.

00:15:18.720 --> 00:15:19.580
Like, I don't know.

00:15:19.580 --> 00:15:24.740
It may be even grading homework assignments that you distribute to people in the form of notebooks.

00:15:24.740 --> 00:15:25.000
Yeah.

00:15:25.000 --> 00:15:29.820
I definitely want to highlight that because my, I haven't talked about it very much, but I was a graduate TA.

00:15:30.180 --> 00:15:40.920
So, boy, I graded a lot of, a lot of calculus, a lot of linear algebra and various other applied calculating type things like MATLAB type of stuff and automating.

00:15:40.920 --> 00:15:42.040
Oh, it would have been good.

00:15:42.040 --> 00:15:42.380
Okay.

00:15:42.380 --> 00:15:43.100
So.

00:15:43.100 --> 00:15:45.440
You just replace yourself with a regression test.

00:15:45.440 --> 00:15:47.060
Exactly.

00:15:47.060 --> 00:15:51.360
Submit your calculus test to the continuous integration.

00:15:51.360 --> 00:15:52.160
We'll see how you did.

00:15:52.160 --> 00:16:00.540
This portion of Talk Python is brought to you by the AWS Insiders podcast.

00:16:00.540 --> 00:16:08.480
When was the last time you ordered a physical server to host your functions as a service, your latest API, or your most recent web app?

00:16:08.480 --> 00:16:10.160
I remember the last time I did.

00:16:10.160 --> 00:16:12.180
That was around the year 2001.

00:16:12.180 --> 00:16:14.360
And yes, it was quite the odyssey.

00:16:14.360 --> 00:16:16.540
Of course, we don't do that anymore.

00:16:16.540 --> 00:16:22.000
We run our code in the cloud with near instant provisioning and unparalleled data centers.

00:16:22.000 --> 00:16:25.620
And the most popular cloud provider is AWS.

00:16:25.620 --> 00:16:32.920
But for all the ways that AWS has made our lives easier, it has also opened a massive box of choices.

00:16:32.920 --> 00:16:34.780
Should you choose platform as a service?

00:16:34.780 --> 00:16:36.840
Or maybe it's still VMs with IaaS.

00:16:36.840 --> 00:16:38.240
What about your database?

00:16:38.240 --> 00:16:41.720
Maybe you should choose a managed service like RDS with Postgres.

00:16:42.040 --> 00:16:43.820
Or is DynamoDB better?

00:16:43.820 --> 00:16:44.640
Maybe Aurora?

00:16:44.640 --> 00:16:45.420
No, wait.

00:16:45.420 --> 00:16:48.160
I hear good things about Amazon DocumentDB too.

00:16:48.160 --> 00:16:51.440
And that's where the AWS Insider podcast comes in.

00:16:51.440 --> 00:16:56.700
This podcast helps technology leaders stay ahead of Amazon's constant pace of change and innovation.

00:16:56.700 --> 00:17:04.460
Some relevant recent episodes include Storage Wars Database Edition, Microservices or Macro Disaster,

00:17:04.460 --> 00:17:08.600
and Exploring Computer Vision at the Edge with AWS Panorama.

00:17:08.800 --> 00:17:13.500
They bring on guests to debate the options and the episodes are vibrant and fun.

00:17:13.500 --> 00:17:20.740
So if you want to have fun and make sense of AWS, head on over to talkpython.fm/AWS Insiders.

00:17:20.740 --> 00:17:25.360
Yes, I know you probably already have a podcast player and you can just search for it there.

00:17:25.360 --> 00:17:28.940
But please use the link so that they know you came from us.

00:17:28.940 --> 00:17:33.800
Thank you to the AWS Insider podcast for keeping this podcast going strong.

00:17:36.480 --> 00:17:40.400
So there's a couple of sections that I'm not sure I really want to dive into because I think,

00:17:40.400 --> 00:17:43.880
I don't know, they're not exactly the notebook ones.

00:17:43.880 --> 00:17:47.660
But one is this collaboration education stuff.

00:17:47.660 --> 00:17:49.080
So maybe we could start there.

00:17:49.080 --> 00:17:55.380
And let me just set the stage by saying if a little while ago I met with Sam Lau and talked with him.

00:17:55.540 --> 00:18:02.960
He and Philip Guo, they did a research project where they studied 60 different network environments,

00:18:02.960 --> 00:18:07.840
not just Jupyter, but like Google CoLab and 58 others.

00:18:07.840 --> 00:18:14.440
And so just kind of putting it out there, like you might think just Jupyter versus JupyterLab is the discussion,

00:18:14.440 --> 00:18:18.980
but there's a whole lot of different places where you can do notebooks, right?

00:18:19.360 --> 00:18:23.180
Yeah, and some of them, like you can run on your own machine.

00:18:23.180 --> 00:18:26.840
That's sort of what I have in this runtime slash environment section.

00:18:26.840 --> 00:18:28.860
Those things tend to go in there.

00:18:28.860 --> 00:18:34.720
And there's a separate category of, I call them in the list, hosted notebook solutions.

00:18:34.720 --> 00:18:39.180
Those are things that you don't really run on your own machine, but they run somewhere in the cloud.

00:18:39.180 --> 00:18:44.140
So I think basically that is one way you can break them down into categories.

00:18:44.140 --> 00:18:47.720
It's just do you run them yourself or do they run in the cloud somewhere else?

00:18:47.720 --> 00:18:53.020
One thing that I didn't see on the list, but maybe would be kind of its own special.

00:18:53.020 --> 00:18:54.160
Open a pull request.

00:18:54.160 --> 00:18:54.920
Yeah, right.

00:18:54.920 --> 00:18:55.340
Here we go.

00:18:55.340 --> 00:18:58.540
Is the JupyterLab desktop app.

00:18:58.540 --> 00:18:59.840
Have you seen this?

00:18:59.840 --> 00:19:01.100
Not sure I've seen that.

00:19:01.100 --> 00:19:01.480
Yeah.

00:19:01.480 --> 00:19:08.800
So what it is, is it's an ElectronJS app that bundles the runtime environment to be JupyterLab.

00:19:08.800 --> 00:19:11.740
And it comes with its own Python and everything.

00:19:11.740 --> 00:19:16.360
So it's a thing you can hand to somebody that runs locally that lets them do notebook stuff

00:19:16.360 --> 00:19:20.080
without them having to have Python installed and set up the environments.

00:19:20.080 --> 00:19:24.120
And it just kind of has a little wizard to get it started, which is, I'm not sure I would

00:19:24.120 --> 00:19:25.920
use it personally, but it's pretty interesting.

00:19:25.920 --> 00:19:30.080
It sounds like a very low barrier to entry notebook environment.

00:19:30.080 --> 00:19:30.440
Yeah.

00:19:30.440 --> 00:19:34.220
I think it could be good for, you know, like in a school environment where you're like,

00:19:34.220 --> 00:19:36.260
all right, kids, just take this and run it.

00:19:36.320 --> 00:19:42.940
I don't want to have to debug why you can't install Python 3.10, but you need 3.7, you know, whatever.

00:19:42.940 --> 00:19:47.660
Lots of different people have different kinds of Jupyter or notebook setups.

00:19:47.660 --> 00:19:54.300
Mine personally tends to be, I've got to, I actually run mine in the cloud because I find it convenient

00:19:54.300 --> 00:19:57.740
to be able to access it from different machines.

00:19:57.740 --> 00:20:02.680
So like I access it from work and then I can access it from my notebook.

00:20:02.680 --> 00:20:07.700
Even like when I'm at a friend's place or something, all I need is a browser to access it.

00:20:07.700 --> 00:20:11.640
And I can just like continue where I left off on the other machine.

00:20:11.640 --> 00:20:12.740
Other people prefer.

00:20:12.740 --> 00:20:15.060
You can probably be closer to the data, right?

00:20:15.060 --> 00:20:15.580
Exactly.

00:20:15.580 --> 00:20:16.080
Right.

00:20:16.200 --> 00:20:16.380
Yeah.

00:20:16.380 --> 00:20:22.560
And you've got a lot faster pipe and you're not that dependent on what your own network

00:20:22.560 --> 00:20:24.580
situation looks like wherever you are.

00:20:24.580 --> 00:20:24.980
Yeah.

00:20:24.980 --> 00:20:25.240
Yeah.

00:20:25.240 --> 00:20:29.520
You're just shipping the answer, not the gig of data required.

00:20:29.520 --> 00:20:29.880
Exactly.

00:20:29.880 --> 00:20:30.240
I get the answer.

00:20:30.240 --> 00:20:30.840
Exactly.

00:20:30.840 --> 00:20:31.260
Right.

00:20:31.260 --> 00:20:36.000
And you've got a pretty easy machine on the other end that can deal with all the calculations.

00:20:36.000 --> 00:20:36.580
Yeah.

00:20:36.580 --> 00:20:37.960
So I find that pretty cool.

00:20:37.960 --> 00:20:41.660
At some point I overdid it and even dockerized the whole thing.

00:20:42.140 --> 00:20:47.660
And then I felt like that was getting more in the way of it than being helpful.

00:20:47.660 --> 00:20:48.140
Yeah.

00:20:48.140 --> 00:20:50.080
You're like, I gave myself a DevOps job.

00:20:50.080 --> 00:20:51.100
Why did I do that?

00:20:51.100 --> 00:20:52.120
Yeah, exactly.

00:20:52.120 --> 00:20:54.240
I only maintain one of these installations.

00:20:54.240 --> 00:20:55.620
Why do I dockerize them?

00:20:55.620 --> 00:20:56.540
Yeah.

00:20:56.540 --> 00:20:57.780
That's actually a really good point.

00:20:57.780 --> 00:20:59.600
I have the same philosophy on web apps.

00:20:59.600 --> 00:21:03.060
It's like, well, if there's just going to be one of them and it's just me, how much flexibility

00:21:03.060 --> 00:21:04.060
does this thing really need?

00:21:04.060 --> 00:21:04.320
Okay.

00:21:04.320 --> 00:21:04.460
Yeah.

00:21:04.460 --> 00:21:11.380
So there's a whole section on these with honestly many places I haven't heard of and ways

00:21:11.380 --> 00:21:11.880
to run it.

00:21:11.880 --> 00:21:16.820
But let's talk about two things in this collaboration education section.

00:21:16.820 --> 00:21:19.560
Three actually, but two are kind of in my mind, put them together.

00:21:19.560 --> 00:21:21.720
One is NB grader, like real quick.

00:21:21.720 --> 00:21:23.600
Like this is a pretty cool project.

00:21:23.600 --> 00:21:24.480
Tell people about this.

00:21:24.480 --> 00:21:26.860
This is pretty much what I described before in the abstract.

00:21:26.860 --> 00:21:27.580
Right.

00:21:27.580 --> 00:21:33.360
If you are a person in education and you teach a course and you want to want your students

00:21:33.360 --> 00:21:38.960
to do a particular assignment and then they send in their submissions, you don't want to

00:21:38.960 --> 00:21:40.620
hand grade them one by one.

00:21:40.620 --> 00:21:45.320
What you can do is formalize basically what you want the answers to look like in a form

00:21:45.320 --> 00:21:46.260
of regression tests.

00:21:46.260 --> 00:21:49.220
And that is basically what NB grader is.

00:21:49.400 --> 00:21:53.980
So like you get one notebook and you define what you want the answers to look like.

00:21:53.980 --> 00:21:56.340
And then it just does the rest of it for you.

00:21:56.340 --> 00:21:57.160
That's pretty interesting.

00:21:57.160 --> 00:22:01.220
I've never used it myself because I'm not working in academia.

00:22:01.220 --> 00:22:01.720
Yeah.

00:22:01.720 --> 00:22:04.700
Like the value prop is obviously obvious with that one.

00:22:04.780 --> 00:22:06.260
Well, I think there's two values here.

00:22:06.260 --> 00:22:10.340
Obviously the less effort on the instructor.

00:22:10.340 --> 00:22:13.420
There's also a little bit of more fairness.

00:22:13.420 --> 00:22:14.000
Yeah, sure.

00:22:14.000 --> 00:22:15.320
There's an interesting angle.

00:22:15.320 --> 00:22:17.860
Like I'm sure that this is true for grading.

00:22:17.860 --> 00:22:22.620
You know, is it morning and you're rested and patient or is it late and you're in a rush

00:22:22.620 --> 00:22:23.580
and you're frustrated?

00:22:23.580 --> 00:22:27.880
I don't know which affects which in terms of how the grades go, but it's got to have an

00:22:27.880 --> 00:22:28.480
effect, right?

00:22:28.480 --> 00:22:34.460
I was talking to some folks who did machine learning for discovering planets on Talk Python and they

00:22:34.460 --> 00:22:41.480
said after the afternoon coffee and cake or cookies or whatever it was they had at this

00:22:41.480 --> 00:22:45.160
university, more exoplanets to be discovered than in the morning.

00:22:45.160 --> 00:22:46.640
Okay.

00:22:46.640 --> 00:22:50.960
I was always told to like call people after lunch.

00:22:50.960 --> 00:22:54.940
That's when they are usually most contained and most open to.

00:22:54.940 --> 00:22:55.480
Exactly.

00:22:55.480 --> 00:22:57.420
And so there's probably a thing about grading.

00:22:57.420 --> 00:23:00.580
So the fact that this doesn't care, it doesn't get coffee.

00:23:00.580 --> 00:23:01.980
It gets electrons.

00:23:01.980 --> 00:23:02.980
That's good.

00:23:02.980 --> 00:23:07.900
And I think there's also like a social science paper on like jury verdicts and sort of the

00:23:07.900 --> 00:23:10.860
harsh use of jury verdicts over time of date.

00:23:10.860 --> 00:23:11.240
Right.

00:23:11.240 --> 00:23:14.060
That's a little bit, that's a little bit harsh to think about, isn't it?

00:23:14.060 --> 00:23:17.660
Like I got an extra year in prison because they were grumpy, right?

00:23:17.660 --> 00:23:19.720
That's not how justice should work.

00:23:19.720 --> 00:23:19.940
Yeah.

00:23:19.940 --> 00:23:21.460
They didn't have their coffee yet.

00:23:21.620 --> 00:23:26.140
So the other angle that I think is interesting with this is if you're a student, you get

00:23:26.140 --> 00:23:29.360
to know whether or not you passed that question.

00:23:29.360 --> 00:23:29.760
Right.

00:23:29.760 --> 00:23:33.620
A lot of times when you're doing complicated things, it's like, I think this is right.

00:23:33.620 --> 00:23:39.180
But if it's not just really straightforward, like a calculus, here's what the formula derivative

00:23:39.180 --> 00:23:39.580
is.

00:23:39.580 --> 00:23:39.760
Right.

00:23:39.820 --> 00:23:41.200
But it's a slightly more nuanced.

00:23:41.200 --> 00:23:42.660
It's hard to know what the right answer is.

00:23:42.660 --> 00:23:43.900
And so here you're like, well, the test passed.

00:23:43.900 --> 00:23:44.700
So we're good to go.

00:23:44.700 --> 00:23:45.220
I like that.

00:23:45.440 --> 00:23:50.120
I had a course like that at university once where you could do multiple submissions and

00:23:50.120 --> 00:23:53.280
the system would tell you like how many points you scored.

00:23:53.280 --> 00:23:58.480
And it actually sort of was very motivating to sort of keep going until you score a perfect

00:23:58.480 --> 00:23:58.960
answer.

00:23:58.960 --> 00:24:02.540
I think having something like that in a course would be super cool.

00:24:02.540 --> 00:24:03.120
I totally agree.

00:24:03.120 --> 00:24:07.260
Before we move on real quick in the audience, David says, I use NB Grader for my teaching.

00:24:07.260 --> 00:24:08.340
It's super helpful.

00:24:08.340 --> 00:24:12.720
NB Grader identifies wrong answers and then you can go in and assign partial credit.

00:24:12.720 --> 00:24:13.320
Yeah.

00:24:13.320 --> 00:24:13.800
I love it.

00:24:13.800 --> 00:24:14.720
That's actually really neat.

00:24:14.720 --> 00:24:15.320
Really neat.

00:24:15.400 --> 00:24:15.600
Okay.

00:24:15.600 --> 00:24:22.660
The other one that's more of a educational demonstration or exploration is NB Tutor here.

00:24:22.660 --> 00:24:25.980
So NB Tutor lets you go in.

00:24:25.980 --> 00:24:28.640
Will you tell people about it if you're familiar with this one?

00:24:28.640 --> 00:24:31.400
No, I haven't used it lately, but it looks like we're dancing.

00:24:31.400 --> 00:24:34.320
I mentioned Philip Guo and Sam Lau.

00:24:34.320 --> 00:24:40.420
They did Python Tutor, which lets you go and write some Python code.

00:24:40.420 --> 00:24:45.100
And it shows you basically how it executes and how variables are related with pointers and

00:24:45.100 --> 00:24:45.340
stuff.

00:24:45.340 --> 00:24:47.500
And this is inspired by that.

00:24:47.500 --> 00:24:52.900
So what it lets you, not this one, it lets you basically run a, what is it?

00:24:52.900 --> 00:24:54.120
A magic command with a percent?

00:24:54.120 --> 00:24:55.200
Cell magic.

00:24:55.200 --> 00:24:55.920
Cell magic.

00:24:55.920 --> 00:24:57.820
You run some cell magic to turn it on.

00:24:57.920 --> 00:24:59.720
And then to the right of the cell.

00:24:59.720 --> 00:25:02.520
It starts showing the pointers and how things are relating.

00:25:02.520 --> 00:25:07.340
So if you're trying to understand computer science and things, I think this would be cool for teaching.

00:25:07.340 --> 00:25:14.520
And you have your code and all you have to stick on to get the visualization is this one short cell magic and you get the rest for free.

00:25:14.520 --> 00:25:15.320
That's pretty cool.

00:25:15.320 --> 00:25:16.400
Yeah, it's really cool.

00:25:16.400 --> 00:25:18.920
They give credit right here to OnlinePythonTutor.

00:25:18.920 --> 00:25:20.180
There's some other ones.

00:25:20.440 --> 00:25:24.780
The Jupyter Drive one to integrate Google Drive looks pretty neat.

00:25:24.780 --> 00:25:27.640
I think it's a little bit expired when I opened it.

00:25:27.640 --> 00:25:31.360
It is definitely more of the experimental flavor.

00:25:31.360 --> 00:25:31.780
Yeah.

00:25:31.900 --> 00:25:39.200
And I imagine sort of whoever develops this is also on the mercy of Google Drive keeping their API stable.

00:25:39.200 --> 00:25:39.980
Absolutely.

00:25:39.980 --> 00:25:46.120
I think that is one way or it's in general, it's a non-trivial question figuring out how to best store your notebooks.

00:25:46.120 --> 00:25:49.900
I mean, if you're just one person, you can probably stick them into Google Drive.

00:25:49.900 --> 00:25:50.180
Yeah.

00:25:50.180 --> 00:26:00.560
But as soon as you have more than a handful of people working on the same set of notebooks, you probably want a solution that is a bit more sophisticated than that.

00:26:00.560 --> 00:26:01.320
I totally agree.

00:26:01.540 --> 00:26:06.100
One of those solutions might be a proper Git story, right?

00:26:06.100 --> 00:26:08.120
And some of the tools we'll talk about are going to cover that, right?

00:26:08.120 --> 00:26:08.920
Yes, exactly.

00:26:08.920 --> 00:26:16.340
The other one could be a collaborative, like a Google CoLab or some other, one of these other environments where it's like Google Docs.

00:26:16.340 --> 00:26:27.020
There's hosted environments that sort of have that as a built-in or you basically, the other way is you roll your own and make it Git-based and both have their advantages and disadvantages.

00:26:27.580 --> 00:26:32.840
I think with Git, you always know a little bit better what you have and what it does.

00:26:33.040 --> 00:26:44.040
Whereas with the other one, that might come with some other fringe benefits like being able to comment on it or having versions of the notebook very nicely integrated with your GUI.

00:26:44.040 --> 00:26:44.400
Yeah.

00:26:44.400 --> 00:26:46.980
And it's like whatever you prepare for sure.

00:26:46.980 --> 00:26:47.980
Like both work.

00:26:48.040 --> 00:26:48.220
Yeah.

00:26:48.220 --> 00:26:52.820
The online ones often have like infrastructure that comes with them too, right?

00:26:52.820 --> 00:26:57.260
When the ability to press go and run it on a GPU if you're willing to pay or whatever.

00:26:57.260 --> 00:26:57.540
Yeah.

00:26:57.540 --> 00:26:57.720
Cool.

00:26:57.720 --> 00:26:58.040
Okay.

00:26:58.440 --> 00:27:03.400
Let's, I think that probably is the interesting ones that jumped out at me from there.

00:27:03.400 --> 00:27:06.840
And then next one is visualization.

00:27:06.840 --> 00:27:10.540
I mean, this is at the heart of the value of notebooks in the first place.

00:27:10.540 --> 00:27:13.120
So Altair, tell us about that one.

00:27:13.120 --> 00:27:17.200
I'm biased, even though I certainly, I'm not a developer of Altair.

00:27:17.380 --> 00:27:21.120
I think what that team has developed is pretty amazing.

00:27:21.120 --> 00:27:27.700
I use it for most of the things, for most of my visualization needs or almost exclusively.

00:27:27.700 --> 00:27:37.080
What's neat about Altair is that it is declarative and it is built on top of a technology, on top of another package, which is called Vega.

00:27:37.080 --> 00:27:42.940
And Vega is a platform agnostic visualization framework.

00:27:43.520 --> 00:27:52.780
So basically what you have to do is if you want to have a chart like that, you just write some JSON declaration of basically, here's your data set.

00:27:52.780 --> 00:27:54.660
This is the URL to the data set.

00:27:54.660 --> 00:27:56.080
It's a tabular format.

00:27:56.080 --> 00:28:00.160
The variable that I want on the X axis is called Foo.

00:28:00.160 --> 00:28:03.480
The variable that I want on the Y axis is called Bar.

00:28:03.480 --> 00:28:05.000
And I want a scatter plot.

00:28:05.000 --> 00:28:08.240
And please make origin in this example here.

00:28:08.240 --> 00:28:10.560
Please make that the color of the dots.

00:28:10.700 --> 00:28:13.360
And you just specify that in a declarative format.

00:28:13.360 --> 00:28:21.640
And then you can, what that allows you to do is you can create this declaration from Python.

00:28:21.640 --> 00:28:27.800
But it might just as well be JavaScript or even like a handwritten JSON, right?

00:28:27.800 --> 00:28:28.100
Right.

00:28:28.100 --> 00:28:32.660
There's some kind of JSON data definition that goes to Vega and that drives the picture.

00:28:32.660 --> 00:28:37.300
So basically Altair generates that data set that goes down to the next layer, right?

00:28:37.440 --> 00:28:37.800
Exactly.

00:28:37.800 --> 00:28:41.760
So like Altair is the Python binding on top of Vega.

00:28:41.760 --> 00:28:48.320
And I think sort of declarative systems, most of the time they have a higher level of abstraction.

00:28:48.320 --> 00:28:50.580
They have more concise notation.

00:28:51.240 --> 00:29:01.160
And the way I always explain this to people when they ask about it is Vega and Altair is to visualization what SQL is to data query, right?

00:29:01.160 --> 00:29:04.960
Your SQL query, you can execute that from within Python.

00:29:04.960 --> 00:29:07.020
You can execute it from within a GUI.

00:29:07.020 --> 00:29:12.080
It's sort of a language agnostic specification of what data you want to query.

00:29:12.440 --> 00:29:16.160
And this is basically the same thing for visualizations.

00:29:16.160 --> 00:29:16.520
Yeah.

00:29:16.520 --> 00:29:17.360
It looks really great.

00:29:17.360 --> 00:29:26.500
There's a beautiful picture of a scatterplot with a legend and multiple colors kind of pulling out some nuance in the data.

00:29:26.500 --> 00:29:29.580
And it's like, I don't know how many lines of code you put that in.

00:29:29.580 --> 00:29:32.100
Maybe four if you didn't multi-line one of them.

00:29:32.100 --> 00:29:33.080
I mean, it's really.

00:29:33.080 --> 00:29:37.120
You can probably golf it together in four lines if you're.

00:29:37.120 --> 00:29:38.840
Without semicolons.

00:29:38.960 --> 00:29:41.220
So if you did semicolons, you could do it one, but that would be wrong.

00:29:41.220 --> 00:29:46.240
But like four reasonable lines, you could do this beautiful picture here, including the important.

00:29:46.240 --> 00:29:47.840
And that already gives you a reason.

00:29:47.840 --> 00:29:49.640
They're quite impressive visualization.

00:29:49.640 --> 00:29:52.300
I think it has some nice defaults.

00:29:52.300 --> 00:29:56.760
Like it knows how to nicely space the labels of the axes and stuff like that.

00:29:56.760 --> 00:30:02.400
And then what it still allows you to do is build pretty complex visualizations too.

00:30:02.400 --> 00:30:07.420
So there is this one example where you basically have a scatterplot on top.

00:30:07.520 --> 00:30:14.920
And then on the bottom, you have something like a histogram and you can select the range in the histogram at the bottom.

00:30:14.920 --> 00:30:16.620
And there you go.

00:30:16.620 --> 00:30:22.940
And then you get this beautiful interactive animation and you don't actually have to write any imperative code.

00:30:22.940 --> 00:30:28.700
You just specify what you want and Altair and Vega kind of like do the rest for you.

00:30:28.700 --> 00:30:37.600
So I write some JavaScript, but years ago, I used to see these like really nice and beautiful animations that were built of D3.

00:30:37.600 --> 00:30:40.240
And I'm like, I want to do cool stuff like that.

00:30:40.240 --> 00:30:41.180
That's really pretty.

00:30:41.180 --> 00:30:42.680
But I don't know any JavaScript.

00:30:42.680 --> 00:30:47.660
And I feel like this is like leveling the playing field a little bit more.

00:30:47.760 --> 00:30:51.100
And it allows you to do similar things from within Python.

00:30:51.100 --> 00:30:51.800
It's really nice.

00:30:51.800 --> 00:31:01.320
One of the things that I think is a little ironic is for the people who create these tools, like the people who created Altair, they have to write so much JavaScript.

00:31:01.320 --> 00:31:03.280
And not that much Python, right?

00:31:03.280 --> 00:31:07.640
Because they're building these interactive, beautiful experiences on the front end for us.

00:31:07.780 --> 00:31:17.520
We get to write the Python and there's like a lot of that complex JavaScript is encapsulated into these tools that we don't have to think about, but we get to use, which is great.

00:31:17.520 --> 00:31:20.060
It's a dirty job, but somebody's got to do it.

00:31:20.060 --> 00:31:24.340
And I have a lot of appreciation what those folks are doing for the rest of us.

00:31:24.340 --> 00:31:24.960
I do too.

00:31:24.960 --> 00:31:26.240
All right.

00:31:26.240 --> 00:31:27.980
So Altair is number one, right?

00:31:27.980 --> 00:31:28.720
On the visualization.

00:31:28.720 --> 00:31:32.700
I mean, it doesn't hurt that it starts with A, but also maybe one of the best ones.

00:31:32.700 --> 00:31:36.040
Some of the other shout out.

00:31:36.040 --> 00:31:40.720
In all fairness, like these things are sort of usually alphabetized.

00:31:40.720 --> 00:31:44.500
These every once in a while, like things in the wrong place.

00:31:44.500 --> 00:31:49.200
And by now I even built myself a linter that keeps the lists nicely alphabetized.

00:31:49.200 --> 00:31:50.720
I'm sure that makes a lot of sense.

00:31:50.720 --> 00:31:51.040
Yeah.

00:31:51.040 --> 00:31:53.240
So Bokeh is one that's out there.

00:31:53.240 --> 00:31:54.860
That's pretty well known.

00:31:54.860 --> 00:31:55.580
People use a lot.

00:31:55.580 --> 00:31:55.880
Yeah.

00:31:55.880 --> 00:31:56.420
Yeah.

00:31:56.420 --> 00:31:57.040
Yeah.

00:31:57.040 --> 00:31:58.580
There's a lot of them, right?

00:31:58.580 --> 00:31:58.940
There are.

00:31:58.940 --> 00:32:04.960
And I think when you talk about visualization, the other very popular ones are probably MatBlootlib,

00:32:05.080 --> 00:32:09.840
which probably was one of the first plotting engines for our backends for Python.

00:32:10.040 --> 00:32:13.720
And then Seaborn, which kind of like builds on top of that.

00:32:13.720 --> 00:32:14.220
Yeah, absolutely.

00:32:14.220 --> 00:32:15.020
Seaborn is nice.

00:32:15.020 --> 00:32:19.920
One that I've seen just recently on notebooks is TQDM.

00:32:19.920 --> 00:32:22.880
I've always used this from CLI applications.

00:32:23.480 --> 00:32:29.940
And TQDM is a way to just take a for loop and whatever you're going to loop over, you just

00:32:29.940 --> 00:32:37.200
put that in TQDM, bracket that thing, and it becomes this live animated progress bar, which

00:32:37.200 --> 00:32:37.860
is really neat.

00:32:37.860 --> 00:32:41.920
But I've only thought of this as a terminal CLI type of thing.

00:32:41.920 --> 00:32:43.600
But it works in notebooks too, I just learned, right?

00:32:43.600 --> 00:32:44.660
Yes, it does.

00:32:45.120 --> 00:32:49.320
TQDM is one of those, like, does exactly what it says on the team kind of things.

00:32:49.320 --> 00:32:51.700
It does one thing and it does it very, very well.

00:32:51.700 --> 00:32:53.140
Yeah, it's not an incredible output.

00:32:53.140 --> 00:32:54.820
But at the same time, it's like, you know what?

00:32:54.820 --> 00:32:59.580
I want to have a little bit of feedback for the users or for myself.

00:32:59.580 --> 00:33:05.940
And you're like, OK, it's just literally wrap your iterator in TQDM and it's good to go.

00:33:06.080 --> 00:33:07.880
It's a very natural thing to want.

00:33:07.880 --> 00:33:13.060
Just imagine you've got like some long running computation over a loop, right?

00:33:13.060 --> 00:33:16.000
And you just don't want to stare at a blank screen for two minutes.

00:33:16.000 --> 00:33:16.420
Yes.

00:33:16.420 --> 00:33:19.480
You can kind of see how maybe the idea developed from there.

00:33:19.480 --> 00:33:21.220
I've got a lot of those type of things.

00:33:21.220 --> 00:33:27.040
This portion of Talk Python To Me is brought to you by Sentry.

00:33:27.040 --> 00:33:29.920
How would you like to remove a little stress from your life?

00:33:29.920 --> 00:33:35.600
Do you worry that users may be encountering errors, slowdowns, or crashes with your app right

00:33:35.600 --> 00:33:35.900
now?

00:33:35.900 --> 00:33:38.960
Would you even know it until they sent you that support email?

00:33:38.960 --> 00:33:43.520
How much better would it be to have the error or performance details immediately sent to

00:33:43.520 --> 00:33:48.520
you, including the call stack and values of local variables and the active user recorded

00:33:48.520 --> 00:33:49.360
in the report?

00:33:49.360 --> 00:33:52.780
With Sentry, this is not only possible, it's simple.

00:33:52.780 --> 00:33:56.340
In fact, we use Sentry on all the Talk Python web properties.

00:33:56.340 --> 00:34:01.880
We've actually fixed a bug triggered by a user and had the upgrade ready to roll out as we

00:34:01.880 --> 00:34:02.900
got the support email.

00:34:02.900 --> 00:34:04.880
That was a great email to write back.

00:34:05.120 --> 00:34:08.280
Hey, we already saw your error and have already rolled out the fix.

00:34:08.280 --> 00:34:09.700
Imagine their surprise.

00:34:09.700 --> 00:34:11.900
Surprise and delight your users.

00:34:11.900 --> 00:34:15.960
Create your Sentry account at talkpython.fm/sentry.

00:34:15.960 --> 00:34:21.440
And if you sign up with the code talkpython, all one word, it's good for two free months

00:34:21.440 --> 00:34:26.720
of Sentry's business plan, which will give you up to 20 times as many monthly events as

00:34:26.720 --> 00:34:27.640
well as other features.

00:34:27.640 --> 00:34:32.040
Create better software, delight your users, and support the podcast.

00:34:32.040 --> 00:34:37.020
Visit talkpython.fm/sentry and use the coupon code talkpython.

00:34:39.820 --> 00:34:46.660
Brian out in the audience says, HV plot, hollow views, bokeh, and panel are all awesome and

00:34:46.660 --> 00:34:47.460
tightly interconnected.

00:34:47.460 --> 00:34:48.040
Yeah.

00:34:48.040 --> 00:34:49.240
And those are really nice.

00:34:49.240 --> 00:34:49.740
All right.

00:34:49.740 --> 00:34:51.940
So next one is the publishing.

00:34:51.940 --> 00:34:55.420
This might be also a little bit at the very heart of notebooks.

00:34:55.420 --> 00:35:01.100
The original idea of the notebook was, I want to have some explanation and then some executable

00:35:01.100 --> 00:35:02.760
code and then some visualization.

00:35:03.100 --> 00:35:08.320
Almost like I want to tell the story of a research project or something like that.

00:35:08.320 --> 00:35:08.580
Right.

00:35:08.580 --> 00:35:11.340
And so this section, it's right there, isn't it?

00:35:11.340 --> 00:35:12.140
Yeah, exactly.

00:35:12.140 --> 00:35:17.780
I think it's sort of less clear cut what this category is as compared to some of the others.

00:35:17.780 --> 00:35:24.040
But it is basically anything in sort of that space of how do you run it?

00:35:24.040 --> 00:35:26.940
How do you tell a story with a notebook?

00:35:26.940 --> 00:35:30.780
How do you point out little things inside those notebooks?

00:35:30.780 --> 00:35:37.300
One of the entries in there that I find quite interesting and useful, and it's also pretty

00:35:37.300 --> 00:35:40.020
awesome from a technical perspective, is binder.

00:35:40.420 --> 00:35:47.420
So what binder allows you to do is you can basically take any GitHub or even GitLab or

00:35:47.420 --> 00:35:52.380
other hosted Git solution URL and put it in there.

00:35:52.380 --> 00:36:00.820
And then what binder does is it builds a Docker image that has all the dependencies of those notebooks

00:36:00.820 --> 00:36:07.360
and builds that image, finds an executable node somewhere in the cloud in their infrastructure,

00:36:07.360 --> 00:36:12.480
and then points you to a Jupyter instance that has that notebook running.

00:36:12.480 --> 00:36:16.580
So what it allows you to do is you see a notebook of GitHub and you're like,

00:36:16.580 --> 00:36:18.980
geez, what if I want to poke around with this thing?

00:36:18.980 --> 00:36:24.500
You just go on binder, put in the URL, and you can play around with a notebook interactively.

00:36:24.500 --> 00:36:25.320
It's really cool.

00:36:25.320 --> 00:36:30.980
So if people have seen the launch binder, a little tag or whatever you call that on like

00:36:30.980 --> 00:36:35.820
a GitHub repo or somewhere else, I guess it could even be in an article that then just

00:36:35.820 --> 00:36:37.100
points back over to one of these.

00:36:37.100 --> 00:36:40.980
If you click it, just as you said, it's going to create an executable environment with the

00:36:40.980 --> 00:36:45.360
right dependencies and let you run your code there, which is kind of impressive that that's

00:36:45.360 --> 00:36:49.460
available to the world openly, publicly without authentication, right?

00:36:49.460 --> 00:36:51.900
It's an incredible engineering feat, right?

00:36:51.900 --> 00:36:52.220
Yeah.

00:36:52.220 --> 00:36:58.780
Just all the considerations of like finding a node that has like sufficient sort of resources

00:36:58.780 --> 00:37:00.580
available to be able to do that.

00:37:01.080 --> 00:37:01.560
Assembling.

00:37:01.560 --> 00:37:06.160
Basically, you don't know what people are going to throw at you in those repos, right?

00:37:06.160 --> 00:37:11.440
Doing a pretty good job at sort of making a whole lot of notebooks executable.

00:37:11.440 --> 00:37:16.360
I had Carol Willing, among others on the show recently to talk about Mastodon.

00:37:16.360 --> 00:37:18.600
And we talked about like the federated storing.

00:37:18.600 --> 00:37:23.380
You know, there's a bunch of people who are creating servers and allowing others to use it,

00:37:23.380 --> 00:37:26.660
sort of volunteering to add a little bit of resources.

00:37:26.660 --> 00:37:32.300
And she said, you know, a lot of what she sees over there actually was there's some parallels

00:37:32.300 --> 00:37:37.920
over in the binder space about how certain universities and other places are like, we'll set up the

00:37:37.920 --> 00:37:42.980
ability to run some of these binders to, you know, add a little bit of compute and resource

00:37:42.980 --> 00:37:43.600
to the world.

00:37:43.600 --> 00:37:45.540
And yeah, it's similar.

00:37:45.540 --> 00:37:46.860
It's pretty amazing.

00:37:46.860 --> 00:37:51.860
Like from what I can see, like what goes on behind the scenes, there is some of this,

00:37:51.860 --> 00:37:56.380
I think is the execution is run on Google hardware.

00:37:56.380 --> 00:38:01.240
So like, and then beyond Google, they have three other hardware providers.

00:38:01.240 --> 00:38:06.480
So not only do they manage to sort of make all those code notebooks executable, but they

00:38:06.480 --> 00:38:11.620
even run them on four different sets of infrastructure, which is pretty amazing.

00:38:11.620 --> 00:38:12.180
Very cool.

00:38:12.180 --> 00:38:12.760
Okay.

00:38:12.760 --> 00:38:14.300
So a couple of others.

00:38:14.300 --> 00:38:18.780
Another one here that really jumps out at me is Jupyter Book over.

00:38:19.480 --> 00:38:20.460
Let me pull up there.

00:38:20.460 --> 00:38:26.380
So build beautiful publication quality books and documents from computational content.

00:38:26.380 --> 00:38:27.660
Now, really nice, huh?

00:38:27.660 --> 00:38:33.360
I think there's a couple of sort of projects that try and do similar things, which is basically

00:38:33.360 --> 00:38:40.180
you create a set of notebooks and then you either get a webpage or you get a book.

00:38:40.740 --> 00:38:46.940
And two of the things that these are useful are, well, one, you're trying to write a book

00:38:46.940 --> 00:38:50.780
about some subject matter, a machine learning book or something like that.

00:38:50.780 --> 00:38:56.480
And the other, the other case where it's really useful is documentation, right?

00:38:56.480 --> 00:39:01.640
If you are a developer or a maintainer of a software package and you want to document your

00:39:01.640 --> 00:39:05.140
API, something like that can be very useful.

00:39:05.140 --> 00:39:11.540
So what this gives you is not only the ability to write documentation, but also to include

00:39:11.540 --> 00:39:13.420
code in that documentation.

00:39:13.420 --> 00:39:18.900
And then in some cases, if you have the binder link, you can even set it up so that you've

00:39:18.900 --> 00:39:24.660
got a piece of code in there and people by clicking a tab next to it can even try out

00:39:24.660 --> 00:39:28.960
what that code does, fiddle with it a little bit, and then see what that does.

00:39:28.960 --> 00:39:31.600
That's it allows you to do some pretty cool stuff.

00:39:31.600 --> 00:39:33.580
That's a pretty interesting way to bring it back around.

00:39:33.580 --> 00:39:38.060
Like we've taken this computational thing, got it going, turned it into a static book that

00:39:38.060 --> 00:39:38.340
you have.

00:39:38.340 --> 00:39:40.600
But if you click this button, you can go back and kind of.

00:39:40.600 --> 00:39:45.740
What we just talked about also sounds very much like NBDIV, which is actually a project

00:39:45.740 --> 00:39:52.740
that I use in, has a very similar flavor where it's specifically geared towards people who

00:39:52.740 --> 00:39:54.400
write software packages.

00:39:54.600 --> 00:40:01.120
And the idea there is that you take your code and you define your classes inside of

00:40:01.120 --> 00:40:01.880
a notebook.

00:40:01.880 --> 00:40:09.700
And then people can, so you can both actually have your code lives inside that notebook.

00:40:09.700 --> 00:40:12.820
You can also define your tests in that notebook.

00:40:12.820 --> 00:40:18.520
And then some added benefits that you get from that is you can run your tests from a notebook.

00:40:18.740 --> 00:40:23.860
And these things don't live in seven different places, like your code base and then your test

00:40:23.860 --> 00:40:26.020
base and then your documentation repo.

00:40:26.020 --> 00:40:29.520
But they all live together in one space.

00:40:29.520 --> 00:40:34.400
And if you make a change that influences or has an impact on all three of them, you don't

00:40:34.400 --> 00:40:38.760
need to do it in three places, but you can just do it in the notebook where it's all together.

00:40:38.760 --> 00:40:43.560
Of all the things that plug into Jupyter, I think I'm most impressed with NBDIV.

00:40:43.560 --> 00:40:44.520
It's pretty nuts.

00:40:44.780 --> 00:40:44.940
Yeah.

00:40:44.940 --> 00:40:47.800
I think this is Donald Newt, right?

00:40:47.800 --> 00:40:50.660
He called this a literate programming environment.

00:40:50.660 --> 00:40:56.640
And I feel like this is the kind of stuff he envisaged where he back in like 83, when he

00:40:56.640 --> 00:40:59.400
wrote his book around about literate programming, right?

00:40:59.400 --> 00:41:01.380
This is really what he had in mind.

00:41:01.380 --> 00:41:02.460
It took a while to get there.

00:41:02.460 --> 00:41:02.760
Yeah.

00:41:02.760 --> 00:41:04.920
The tools he was working with, they weren't like these.

00:41:04.920 --> 00:41:05.220
Yeah.

00:41:05.220 --> 00:41:05.400
Yeah.

00:41:05.400 --> 00:41:09.960
So with NBDIV, you can have your documentation, you can publish, you can take your notebook

00:41:09.960 --> 00:41:15.840
and export it or convert it into or build it into a Python package or a Conda package.

00:41:15.840 --> 00:41:18.240
You can publish it to PyPI and Conda.

00:41:18.240 --> 00:41:20.980
You can have tests, you can have continuous integration.

00:41:20.980 --> 00:41:26.280
Then also you can, if you've got complicated code that you need to integrate in other ways,

00:41:26.280 --> 00:41:29.040
you can sync it to Python files and then back.

00:41:29.240 --> 00:41:31.740
I think that two-way integration is pretty cool.

00:41:31.740 --> 00:41:31.920
Yeah.

00:41:31.920 --> 00:41:36.260
For this, I really need to get this out of the notebook into Python directly, but then

00:41:36.260 --> 00:41:39.620
don't just carve it off, like keep them sort of connected, right?

00:41:39.620 --> 00:41:40.100
Exactly.

00:41:40.100 --> 00:41:45.640
That's one of the key points there is that NBDIV allows you to do all of these things

00:41:45.640 --> 00:41:47.260
in the same place, right?

00:41:47.260 --> 00:41:52.560
Sure, you can do all of these things separately and a lot of people do them separately, but then

00:41:52.560 --> 00:41:58.300
having them all in one place is just so much easier when you say make an API change.

00:41:58.520 --> 00:41:58.600
Yeah.

00:41:58.600 --> 00:42:02.900
Another one which wasn't really even shouted out there in that highlights that they have

00:42:02.900 --> 00:42:08.680
is NBclean, which if you're doing, and we talked about the two possible ways for collaboration,

00:42:08.680 --> 00:42:14.060
if you're doing the Git way, you know, these notebooks, their files contain the output,

00:42:14.060 --> 00:42:16.840
which could completely vary from run to run.

00:42:16.840 --> 00:42:20.320
So like every time you save it or rerun it, it's a merge conflict, right?

00:42:20.320 --> 00:42:24.000
And so this will strip out that kind of information to avoid merge conflicts.

00:42:24.000 --> 00:42:26.820
So it can be a real good way to sort of prepare it.

00:42:27.020 --> 00:42:29.060
I suspect that could be a Git pre-commit hook.

00:42:29.060 --> 00:42:30.120
I'm not sure of it, probably.

00:42:30.120 --> 00:42:31.500
That's one way to do it.

00:42:31.500 --> 00:42:37.620
And sort of, I think sort of the fact that the notebook format contains the cell output

00:42:37.620 --> 00:42:39.400
and also some many information.

00:42:39.400 --> 00:42:42.180
It's a bit of a blessing and a curse at the same time.

00:42:42.180 --> 00:42:47.280
On one hand, it sort of, when you have a notebook file, you can load it up and you immediately

00:42:47.280 --> 00:42:51.680
see the output without having to run the cells, right?

00:42:51.680 --> 00:42:55.880
So imagine someone sends you a notebook file and you cannot run it on your machine.

00:42:55.880 --> 00:43:00.380
You still get the benefit of seeing what their output was when they generated it.

00:43:00.460 --> 00:43:01.540
So it's nice for that.

00:43:01.540 --> 00:43:07.100
It's also nice for the fact that if you put it up on GitHub, GitHub can render this notebook

00:43:07.100 --> 00:43:10.460
file with the output in a very nice and sensible way.

00:43:10.460 --> 00:43:11.980
You immediately see what the output was.

00:43:11.980 --> 00:43:18.940
But the disadvantage of doing that is that whenever you rerun it, sort of the contents of that file

00:43:18.940 --> 00:43:22.180
change and it doesn't produce clean diffs.

00:43:22.240 --> 00:43:27.520
For example, if you just change one line and then that line produces a block, your diff might

00:43:27.520 --> 00:43:29.500
be a couple of kilobytes long, right?

00:43:29.500 --> 00:43:30.680
If you don't want that.

00:43:30.680 --> 00:43:37.300
And one of the tools that sort of deals with that is something we had on the list, ChupyText.

00:43:37.300 --> 00:43:45.360
So ChupyText is basically, so ChupyText deals with this problem by basically giving you paired

00:43:45.360 --> 00:43:45.880
notebooks.

00:43:46.600 --> 00:43:51.860
So you have an IPNB file and you tell your Jupyter IDE.

00:43:51.860 --> 00:43:59.460
In addition to that IPNB file, I also want a markdown file or a PY file that contains just

00:43:59.460 --> 00:44:00.280
the input cells.

00:44:00.280 --> 00:44:06.340
And then what you can do is if you want to version your notebook is just check in that

00:44:06.340 --> 00:44:12.600
clean or stripped PY file or markdown file and rather ignore the IPNB.

00:44:12.600 --> 00:44:16.120
So that is what we actually do in our company.

00:44:16.340 --> 00:44:21.660
We have a couple of notebooks that serve as reports and we edit them collaboratively.

00:44:21.660 --> 00:44:27.480
And the way we version them is by stripping out the output via ChupyText.

00:44:27.480 --> 00:44:29.840
And so you really get nice, clean diffs.

00:44:29.840 --> 00:44:33.460
And when someone makes a change, you can tell what they changed in a reasonable way.

00:44:33.460 --> 00:44:34.220
That's super valuable.

00:44:34.220 --> 00:44:37.520
Do you want to go back to the MB dev for just one second?

00:44:37.520 --> 00:44:41.360
Because there's a big, long list here that you would get if you just ran MB dev help.

00:44:41.800 --> 00:44:46.580
And so many of these are like, as a standalone command, they'd be like, oh my gosh, what

00:44:46.580 --> 00:44:47.240
an amazing tool.

00:44:47.240 --> 00:44:53.560
Like another one I just saw is MB dev changelog, which will create a changelog.md file just from

00:44:53.560 --> 00:44:55.680
like your closed and labeled GitHub issues.

00:44:56.060 --> 00:44:58.060
And, you know, like that's a cool feature on its own.

00:44:58.060 --> 00:45:02.400
I could see installing that and never even using a notebook and just running that on my

00:45:02.400 --> 00:45:06.600
Git repo, which is, you know, so there's a bunch of, bunch of neat things here.

00:45:06.600 --> 00:45:12.240
You look at the list and what MB dev does for more than five minutes and it makes you want

00:45:12.240 --> 00:45:13.620
to build a software package.

00:45:13.880 --> 00:45:15.380
Yes, it does.

00:45:15.380 --> 00:45:17.840
Which you could do with MB dev, by the way.

00:45:17.840 --> 00:45:19.260
So that's, that's kind of meta.

00:45:19.260 --> 00:45:19.680
Yeah.

00:45:19.680 --> 00:45:21.740
Another one around there is NB convert.

00:45:21.740 --> 00:45:26.420
NB convert is, I mean, it's, it's almost there in the name, right?

00:45:26.420 --> 00:45:33.420
Basically one thing it gives you is a command line command where you can do NB convert for

00:45:33.420 --> 00:45:38.280
dot IP and B and say, like, I want to convert this notebook to HTML.

00:45:38.280 --> 00:45:43.260
So you can script the conversion of notebook on a command line level.

00:45:43.260 --> 00:45:49.740
And then beyond that, what you can do with NB convert is it's also accessible as a Python package.

00:45:49.740 --> 00:45:55.220
So you can control that notebook conversion from within your Python code.

00:45:55.220 --> 00:46:02.300
And that is useful for a number of things when you really want fine grained control over how you

00:46:02.300 --> 00:46:05.000
execute or convert your notebook.

00:46:05.000 --> 00:46:10.360
So one of the things that I have built with that is basically a way to convert notebooks

00:46:10.360 --> 00:46:12.520
without the input cells.

00:46:12.520 --> 00:46:18.760
So if you want to build a report out of a notebook and you have a non-technical audience and you

00:46:18.760 --> 00:46:23.020
want to get the code out of their way and you really want to just show them the output,

00:46:23.020 --> 00:46:25.460
that is something you can build with NB convert.

00:46:25.460 --> 00:46:26.520
That's really cool.

00:46:26.520 --> 00:46:28.480
I hadn't realized it did that.

00:46:28.480 --> 00:46:30.380
So here's what I want to show you.

00:46:30.380 --> 00:46:35.240
And if you actually want to see the code, click on this binder version or view it statically

00:46:35.240 --> 00:46:35.740
on GitHub.

00:46:35.740 --> 00:46:40.860
But for most people, they just want to see here's the description and here's the figures.

00:46:40.860 --> 00:46:43.320
It's still potentially accessible at least.

00:46:43.320 --> 00:46:43.820
Totally.

00:46:43.820 --> 00:46:49.720
And I think notebooks are actually quite a nice way to design a report, right?

00:46:49.760 --> 00:46:51.900
Because what you have is building blocks.

00:46:51.900 --> 00:46:58.960
And my report is I want the total number of orders for the last week and I want to chart.

00:46:58.960 --> 00:47:03.880
And then I want like the number of orders by zip code or whatever.

00:47:03.880 --> 00:47:10.960
And you build that in a notebook and then you just like NB convert it with the input stripped

00:47:10.960 --> 00:47:16.900
out and you get an HTML file that you can serve on a web server and have something like a dashboard

00:47:16.900 --> 00:47:17.700
for your team.

00:47:17.700 --> 00:47:18.240
Very good.

00:47:18.240 --> 00:47:20.640
So it's super cool for building things like that.

00:47:20.640 --> 00:47:21.020
Yeah.

00:47:21.020 --> 00:47:21.840
Yeah.

00:47:21.840 --> 00:47:22.320
Very neat.

00:47:22.320 --> 00:47:24.760
One more in this section before we move on.

00:47:24.760 --> 00:47:28.460
There's a bunch, but I think Paper Mill is pretty unique.

00:47:28.460 --> 00:47:30.260
You want to tell people about Paper Mill?

00:47:30.260 --> 00:47:31.180
I haven't.

00:47:31.180 --> 00:47:32.720
To be honest, I haven't used it that much.

00:47:32.820 --> 00:47:40.160
I haven't either, but I did read this Netflix Paper Mill article about what they were doing.

00:47:40.160 --> 00:47:44.000
And they were basically using...

00:47:44.000 --> 00:47:44.780
Ah, Kyle Kelly.

00:47:44.780 --> 00:47:45.140
Yeah.

00:47:45.140 --> 00:47:45.420
Yeah.

00:47:45.420 --> 00:47:51.000
They were doing some work to take notebooks and use those as like building blocks for managing

00:47:51.000 --> 00:47:51.820
their infrastructure.

00:47:51.820 --> 00:47:58.180
So there were a lot of interesting benefits and Paper Mill will let you basically turn the

00:47:58.180 --> 00:48:01.460
variables at the top of the notebook into inputs.

00:48:01.460 --> 00:48:04.180
And then the variables at the end of the notebook as outputs.

00:48:04.180 --> 00:48:05.600
And you could treat it like a function.

00:48:05.600 --> 00:48:12.160
So they're chaining these together for all sorts of crazy DevOps-y type things, I believe.

00:48:12.160 --> 00:48:18.300
It basically sounds like something I once rolled my own of where basically what I just described

00:48:18.300 --> 00:48:23.460
that I'm doing with NB Convert, I once built a version where sort of you could in the first

00:48:23.460 --> 00:48:26.720
cell have a variable and serve that like a function input.

00:48:26.720 --> 00:48:31.380
And it sounds like they probably did a much better job of sort of imagining you have a

00:48:31.380 --> 00:48:36.620
dashboard and then you've got one for the US and you want one for international orders,

00:48:36.620 --> 00:48:36.980
right?

00:48:36.980 --> 00:48:39.700
Or for Canadian or what have you country, right?

00:48:39.700 --> 00:48:45.880
You can basically do that out of the same notebook file, but just say, hey, these are two separate

00:48:45.880 --> 00:48:46.360
versions.

00:48:46.360 --> 00:48:49.840
One for the US, one for Canada, one for Mexico, whatever.

00:48:49.920 --> 00:48:50.360
Exactly.

00:48:50.360 --> 00:48:55.140
One of the benefits that they said they were getting was if something goes wrong, like if

00:48:55.140 --> 00:49:01.620
a step crashes, because as you described the notebook, as it executes, it stores the exact

00:49:01.620 --> 00:49:02.840
output and the inputs.

00:49:02.840 --> 00:49:06.520
It's a snapshot in time of what happened when it went wrong.

00:49:06.520 --> 00:49:09.940
So instead of just having a log message of it went wrong and here was the input, it's like,

00:49:09.940 --> 00:49:13.300
well, here are all the steps and you can see the variables and the output coming.

00:49:13.300 --> 00:49:14.340
And then here's the crash.

00:49:14.340 --> 00:49:16.520
You're like, oh, look, these are the three inputs.

00:49:16.520 --> 00:49:17.740
And here's how this one got.

00:49:17.740 --> 00:49:20.040
And actually gives you nice diagnostics.

00:49:20.040 --> 00:49:20.340
Yeah.

00:49:20.340 --> 00:49:22.260
It's like a report of what went wrong.

00:49:22.260 --> 00:49:27.060
It's almost like a, you know, you have these parametrized tests, right?

00:49:27.060 --> 00:49:28.000
It is.

00:49:28.000 --> 00:49:28.340
Yes.

00:49:28.340 --> 00:49:34.220
Parametrized tests with some diagnostic output about what went wrong during that execution.

00:49:34.220 --> 00:49:35.060
That's pretty cool.

00:49:35.060 --> 00:49:35.740
It's unique.

00:49:35.740 --> 00:49:38.180
I think worth a shout out, I suppose, there.

00:49:38.180 --> 00:49:40.020
How about the version control side?

00:49:40.020 --> 00:49:41.620
Anything you want to give a shout out to there?

00:49:41.620 --> 00:49:46.860
That is a section that is a bit more sort of well-defined and clear what goes in there.

00:49:46.860 --> 00:49:53.860
It's basically all sorts of tools for diffing, merging, code reviewing, changes in notebooks.

00:49:53.860 --> 00:50:00.980
NB Dime is probably one of the more well-known packages in that category that looks very well-built.

00:50:00.980 --> 00:50:01.420
Right.

00:50:01.420 --> 00:50:03.020
The diff and merge tools.

00:50:03.020 --> 00:50:03.340
Yeah.

00:50:03.340 --> 00:50:14.280
So imagine you have an image, right, in an IPMB file, you would usually get like a horrendously large diff if one of the things in that image changes.

00:50:14.280 --> 00:50:25.340
And it just displays that that diff not has 50 lines that change, but just like a neat, here is something that changed one line and doesn't mess up your whole diff.

00:50:25.340 --> 00:50:25.600
Right.

00:50:25.600 --> 00:50:30.620
This plot change, not these 700 lines of plot definition change.

00:50:30.620 --> 00:50:30.860
Yeah.

00:50:30.860 --> 00:50:31.180
Yeah.

00:50:31.180 --> 00:50:31.600
Yeah.

00:50:31.600 --> 00:50:32.400
Very cool.

00:50:32.720 --> 00:50:34.260
And merging as well, which is neat.

00:50:34.260 --> 00:50:34.520
Yeah.

00:50:34.520 --> 00:50:34.860
All right.

00:50:34.860 --> 00:50:37.280
We're getting a little short on time.

00:50:37.280 --> 00:50:37.660
Let's see.

00:50:37.660 --> 00:50:44.160
I want to, I know you want to give a shout out to Deep Note, both because you like them, but also they're a sponsor of the...

00:50:44.160 --> 00:50:44.880
They support the list.

00:50:44.880 --> 00:50:45.180
Yes.

00:50:45.180 --> 00:50:45.980
They support the list.

00:50:45.980 --> 00:50:46.140
Yeah.

00:50:46.140 --> 00:50:47.280
So maybe tell us about them.

00:50:47.280 --> 00:50:50.780
Deep Note is basically one of these hosted notebook solutions.

00:50:50.780 --> 00:50:52.380
We already mentioned Binder.

00:50:52.380 --> 00:50:54.060
We already mentioned Colab.

00:50:54.060 --> 00:51:00.280
And they have built one that is a bit more centered around collaboration of teams.

00:51:00.280 --> 00:51:07.060
You also get an execution environment with different sets of hardware from them.

00:51:07.060 --> 00:51:09.820
And they really emphasize collaboration.

00:51:09.820 --> 00:51:15.300
So you can comment and have discussions about individual cells in your notebook.

00:51:15.300 --> 00:51:27.320
You can very nicely see what changes other people did to your notebook and all of that in a very well done, very dense GUI that makes a lot of things very easy for you.

00:51:27.320 --> 00:51:37.980
If you're not already using one of these environments and you have a team that is maybe also not particularly technically focused on where it runs, how it runs.

00:51:37.980 --> 00:51:39.600
This is a very turnkey solution.

00:51:39.600 --> 00:51:43.660
Less DevOps, more Google Docs type of style.

00:51:43.660 --> 00:51:46.000
They solve a lot of those problems for you.

00:51:46.000 --> 00:51:46.860
It looks really neat.

00:51:47.040 --> 00:51:49.260
The collaboration seems very, very nice.

00:51:49.260 --> 00:51:52.880
I think that's a pretty unique thing to do in a polished way.

00:51:52.880 --> 00:51:57.080
Yeah, and they also have nice integrations for a lot of data sources.

00:51:57.080 --> 00:52:05.140
So you can directly query SQL from their GUI and pipe SQL results into a data frame, right?

00:52:05.140 --> 00:52:12.140
So you have one cell where you write your SQL and that goes directly into the data frame that you can then visualize in the next cell.

00:52:12.600 --> 00:52:18.440
So the ergonomics of Deep Note are probably better than many other notebook solutions.

00:52:18.440 --> 00:52:19.080
That's pretty neat.

00:52:19.080 --> 00:52:28.860
And so like you said, it's worth pointing out they're a sponsor of your list, not of our show, but they are a commercial thing, but they are a commercial venture, but they do have a free version.

00:52:28.860 --> 00:52:38.040
There's something for me where I feel like it's, I kind of like to support companies that are purpose built, right?

00:52:38.080 --> 00:52:50.860
Like Deep Note is built for running notebooks and collaboration, whereas a lot of these big tech things, it's like, well, I know Facebook's a social media company, but I could also do this other thing that runs on, or I could run this on Google.

00:52:50.860 --> 00:52:59.540
And there's something about like, okay, there's a company whose only job is to do this versus like, I'm not sure I could ever get support.

00:52:59.540 --> 00:53:03.540
Like if I had a problem with my Gmail, I don't know that I could ever get help, ever, right?

00:53:03.540 --> 00:53:09.580
Whereas like, you know, if I went to an email company and got email from them, they would help me with email because it's their thing, right?

00:53:09.580 --> 00:53:17.900
I think where it shows is that they spent a lot of time thinking about how people use notebooks and what they want to do with those notebooks.

00:53:17.900 --> 00:53:27.200
They think a bit beyond just sort of the Jupyter technical, but more the ergonomics of how you actually use a notebook within a team.

00:53:27.200 --> 00:53:36.640
And I think sort of, you can definitely tell that they know how to use notebooks and have thought about all the ways in which people use notebooks and how to make that better.

00:53:36.640 --> 00:53:36.980
Nice.

00:53:36.980 --> 00:53:40.800
Well, I've never used them, but if I have the need, I'll check it out.

00:53:40.800 --> 00:53:41.160
Sounds good.

00:53:41.160 --> 00:53:41.480
All right.

00:53:42.280 --> 00:53:43.140
What else?

00:53:43.140 --> 00:53:45.440
Maybe we've got time for one more.

00:53:45.440 --> 00:53:48.360
Is there one we haven't talked about yet that you're like, oh, we really got to cover this one?

00:53:48.360 --> 00:53:50.080
Let's do cell magics, right?

00:53:50.080 --> 00:53:50.420
Okay.

00:53:50.420 --> 00:53:54.000
If you maybe just Google for IPython cell magics.

00:53:54.000 --> 00:53:54.860
I'll kaggy for it.

00:53:54.860 --> 00:53:55.360
Yeah.

00:53:55.360 --> 00:54:03.340
They are not strictly a Jupyter thing, but I think they're a not so well-known thing that is super neat.

00:54:04.020 --> 00:54:13.700
And they are, since Jupyter itself started in, from, was born out of IPython, these are things that are also built into IPython.

00:54:13.700 --> 00:54:16.580
So one of them is, for example, a percent debug.

00:54:17.020 --> 00:54:28.760
So when you have a notebook open and you get an exception and your code gives you an error message, you can then, in a cell directly below that, type Terset debug.

00:54:28.760 --> 00:54:36.160
And that will open up a debugger session that takes you directly to the point where your code fail.

00:54:36.160 --> 00:54:36.700
Right?

00:54:36.700 --> 00:54:44.940
So if you write something like, I don't know, I accessed true, the array true at an index position of five, right?

00:54:44.940 --> 00:54:46.600
And that's where your code fails.

00:54:46.600 --> 00:54:54.420
You can do percent debug and look at, well, at that point in the execution, what was actually stored in the list full?

00:54:54.420 --> 00:54:54.740
Right.

00:54:54.740 --> 00:54:57.980
And did it actually have five elements or did I run out of bounds?

00:54:57.980 --> 00:55:07.340
So, like, whenever I get an error during an execution of a notebook and it's not obvious, I like to use percent debug to find out what's going wrong with my code.

00:55:07.340 --> 00:55:08.060
Very nice.

00:55:08.060 --> 00:55:13.600
Another one that is very useful is percent time, which is, have you used that?

00:55:13.600 --> 00:55:19.220
No, but I often want to answer this question of, like, is this getting a little bit faster, a little bit slower?

00:55:19.220 --> 00:55:26.560
And I don't really want to go into a profiler and I don't want to go to, like, write the date time or timestamp code to, like, print it out myself.

00:55:26.560 --> 00:55:27.720
It's just, like, yeah.

00:55:27.720 --> 00:55:34.460
I use it a lot in those situations where, like, I've got, I've got way A to do something and way B to do something, right?

00:55:34.460 --> 00:55:42.460
And then I wonder, is this actually faster or is this actually performance-wise worse than doing it the other way?

00:55:42.460 --> 00:55:47.320
And this gives you just a very quick and dirty answer to, like, orders of magnitude.

00:55:47.320 --> 00:55:50.200
Is this the same level or should I be doing things differently?

00:55:50.200 --> 00:55:51.500
It's very useful for that.

00:55:51.500 --> 00:55:51.720
Yeah.

00:55:51.720 --> 00:55:54.860
Percent time and some function call and see what it takes.

00:55:54.860 --> 00:55:55.420
It's brilliant.

00:55:55.420 --> 00:56:01.980
And then another one I want to mention is just exclamation point and then the shell command.

00:56:01.980 --> 00:56:02.380
Yeah.

00:56:02.380 --> 00:56:07.420
It's very useful for, I think it's also available as %sx.

00:56:07.420 --> 00:56:08.380
Yeah.

00:56:08.500 --> 00:56:09.500
So it's basically a shorthand.

00:56:09.500 --> 00:56:17.060
So you can do exclamation point LS to get to a directory listing just to see what is in the directory.

00:56:17.060 --> 00:56:22.940
Or you can do, like, I don't know, who am I to see what user you're running something as.

00:56:22.940 --> 00:56:27.500
Or you can do ping a machine to see if it's up directly from your notebook.

00:56:27.500 --> 00:56:35.240
So this is a quick and dirty way of running command line commands without having to leave your notebook.

00:56:35.240 --> 00:56:35.800
It's really cool.

00:56:35.800 --> 00:56:36.040
It's really cool.

00:56:36.040 --> 00:56:41.120
You know, if you wouldn't just interact with the file system or you see what files are available to me or all these things, right?

00:56:41.120 --> 00:56:41.320
Yeah.

00:56:41.320 --> 00:56:41.480
Yeah.

00:56:41.480 --> 00:56:43.340
You wouldn't know because this hasn't published yet.

00:56:43.340 --> 00:56:51.260
But in the sequence of when these shows come out, I just talked to, or you're ruined from data science at the command line.

00:56:51.260 --> 00:56:52.460
Wait, have you seen this book?

00:56:52.460 --> 00:56:52.720
No.

00:56:52.720 --> 00:56:52.940
Yeah.

00:56:52.940 --> 00:56:59.940
So it's got a bunch of interesting things that you can do on the command line for, like, querying data or running things in parallel.

00:56:59.940 --> 00:57:06.180
And a whole bunch of these, these sort of ideas of, like, how do I do really cool stuff with the shell?

00:57:06.180 --> 00:57:10.540
Just use your bang command and they become integrated into your notebook, right?

00:57:10.540 --> 00:57:12.160
Which is, I think that's super cool.

00:57:12.400 --> 00:57:21.020
You can even take it to do things like you do an LS and then your Python session gets past the contents of that directory as a list.

00:57:21.020 --> 00:57:25.600
And you can then use it as a variable and assign that list of variable.

00:57:25.600 --> 00:57:33.440
So it's like, that is a really, like, easy, but also, like, very shoddy way of listing the directory contents very quickly.

00:57:33.440 --> 00:57:33.700
Yeah.

00:57:33.700 --> 00:57:35.460
I mean, you should probably use pass a little bit or something.

00:57:35.460 --> 00:57:36.080
We shouldn't even mention that.

00:57:36.080 --> 00:57:37.420
We should not encourage that.

00:57:37.420 --> 00:57:42.120
But if there's, like, really interesting stuff that's happening on the shell you want to use, like, for example,

00:57:42.120 --> 00:57:44.340
there's this symbol that says total lines equals.

00:57:44.340 --> 00:57:45.300
That's a brilliant one.

00:57:45.300 --> 00:57:45.620
Yeah.

00:57:45.620 --> 00:57:48.400
Which is a Jupyter level command.

00:57:48.400 --> 00:57:53.440
And this says bang arrow to redirect input of some text file word count dash L.

00:57:53.440 --> 00:57:54.700
And that tells you how many lines are there.

00:57:54.700 --> 00:57:59.820
Like, yeah, maybe you want to make this all Python so it doesn't, it's not platform dependent, right?

00:57:59.820 --> 00:58:00.100
Sure.

00:58:00.100 --> 00:58:02.440
If you want to package this up, right?

00:58:02.440 --> 00:58:03.380
In that package.

00:58:03.500 --> 00:58:08.640
But if you only end up doing it once and you just want to know how many lines there are, that's a way to do it.

00:58:08.640 --> 00:58:09.000
Right.

00:58:09.000 --> 00:58:11.780
You're in, like, an exploratory situation, right?

00:58:11.780 --> 00:58:13.660
And you just generated this file and now you want to know.

00:58:13.660 --> 00:58:14.180
I don't know.

00:58:14.180 --> 00:58:16.240
So there's, that's a really powerful one.

00:58:16.240 --> 00:58:17.240
I'm glad you brought that up.

00:58:17.320 --> 00:58:25.240
And then just the last one from there is, because it definitely saved me in a couple of very hairy situations, is percent history.

00:58:25.480 --> 00:58:35.340
So that basically gives you a history of the 20, 10, or however many last commands you executed in a Python session.

00:58:35.340 --> 00:58:41.700
Why it's sometimes useful is just imagine you accidentally delete a cell in a notebook.

00:58:41.700 --> 00:58:44.300
I mean, you can always undo that, right?

00:58:44.300 --> 00:58:52.540
But every once in a while, your undo history is so horribly messed up that you somehow sort of lost that cell forever.

00:58:52.540 --> 00:58:55.620
And there was, like, a lot of finely tuned code in that.

00:58:55.620 --> 00:59:01.760
So you can go back to that with percent history and get your deleted cells from beyond the grave.

00:59:01.760 --> 00:59:02.080
Yeah.

00:59:02.080 --> 00:59:05.040
And you can do percent history dash in five.

00:59:05.040 --> 00:59:06.900
Just show me the last five changes.

00:59:06.900 --> 00:59:07.320
Yeah.

00:59:07.320 --> 00:59:08.780
Yeah, it's really nice.

00:59:08.780 --> 00:59:09.340
Cool.

00:59:09.340 --> 00:59:10.120
All right, Marcus.

00:59:10.120 --> 00:59:15.480
I think we might be out of time, but, you know, what a cool project this awesome list is.

00:59:15.480 --> 00:59:17.860
We've got a lot of people listening in.

00:59:17.860 --> 00:59:25.360
I'm sure there's a lot more awesome packages, tools, resources that could and should go on this list.

00:59:25.360 --> 00:59:38.700
So if whoever is watching or listening to this has a package that they feel should be mentioned there, it's just as easy as doing a pull request and adding your favorite tool or resource to that list.

00:59:38.940 --> 00:59:41.060
We're very inclusionist on the list, right?

00:59:41.060 --> 00:59:43.560
And we like to include people's suggestions.

00:59:43.560 --> 00:59:44.460
Yeah, fantastic.

00:59:44.460 --> 00:59:50.500
There's a couple that I could see showing up there, like a few comments in the live stream that those could be PR.

00:59:50.500 --> 00:59:51.760
So have at it.

00:59:51.760 --> 00:59:52.260
That'd be awesome.

00:59:52.260 --> 00:59:53.080
Yeah.

00:59:53.080 --> 00:59:53.460
Awesome.

00:59:53.560 --> 00:59:53.680
All right.

00:59:53.680 --> 00:59:55.900
Final two questions before I let you out of here, though.

00:59:55.900 --> 00:59:59.180
If you're going to write some Python code, what editor do you use these days?

00:59:59.180 --> 00:59:59.680
Jupyter.

00:59:59.680 --> 01:00:00.480
Of course.

01:00:00.480 --> 01:00:01.460
I overuse it.

01:00:01.460 --> 01:00:02.560
Beautiful.

01:00:02.560 --> 01:00:08.720
And then notable PyPI or Conda package or something out there or even a Jupyter plugin.

01:00:08.720 --> 01:00:13.780
I know basically this entire show has been one after another, but something you want to give a shout out to.

01:00:13.780 --> 01:00:16.840
If I can just give one a shout out, it's Altair.

01:00:16.840 --> 01:00:17.120
Yeah.

01:00:17.120 --> 01:00:18.760
You should be doing the plotting in Altair.

01:00:18.760 --> 01:00:19.480
Altair right on.

01:00:19.480 --> 01:00:20.720
Yeah, it's quite nice.

01:00:20.720 --> 01:00:21.300
All right.

01:00:21.300 --> 01:00:22.860
Well, thank you so much for being here.

01:00:22.860 --> 01:00:24.100
It's been great to have you on the show.

01:00:24.100 --> 01:00:25.060
Thank you for having me.

01:00:25.060 --> 01:00:25.720
Yeah, you bet.

01:00:25.720 --> 01:00:26.180
Bye.

01:00:26.180 --> 01:00:26.760
Have a good one.

01:00:26.760 --> 01:00:26.980
Bye.

01:00:26.980 --> 01:00:30.960
This has been another episode of Talk Python To Me.

01:00:30.960 --> 01:00:32.780
Thank you to our sponsors.

01:00:32.780 --> 01:00:34.380
Be sure to check out what they're offering.

01:00:34.380 --> 01:00:35.820
It really helps support the show.

01:00:35.820 --> 01:00:43.440
AWS is the lead cloud for developers, but with over 250 services, it's an overwhelming set of choices.

01:00:43.440 --> 01:00:47.020
That's where the AWS Insiders podcast comes in.

01:00:47.020 --> 01:00:50.620
Their job is to help you make sense of all those AWS options.

01:00:51.220 --> 01:00:55.360
Listen to an episode at talkpython.fm/AWS Insiders.

01:00:55.360 --> 01:00:57.800
Take some stress out of your life.

01:00:57.800 --> 01:01:03.580
Get notified immediately about errors and performance issues in your web or mobile applications with Sentry.

01:01:03.580 --> 01:01:08.580
Just visit talkpython.fm/sentry and get started for free.

01:01:08.580 --> 01:01:12.180
And be sure to use the promo code talkpython, all one word.

01:01:12.180 --> 01:01:13.800
Want to level up your Python?

01:01:13.800 --> 01:01:17.840
We have one of the largest catalogs of Python video courses over at Talk Python.

01:01:17.840 --> 01:01:23.020
Our content ranges from true beginners to deeply advanced topics like memory and async.

01:01:23.020 --> 01:01:25.700
And best of all, there's not a subscription in sight.

01:01:25.700 --> 01:01:28.600
Check it out for yourself at training.talkpython.fm.

01:01:28.720 --> 01:01:30.500
Be sure to subscribe to the show.

01:01:30.500 --> 01:01:33.280
Open your favorite podcast app and search for Python.

01:01:33.280 --> 01:01:34.580
We should be right at the top.

01:01:34.580 --> 01:01:43.960
You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the direct RSS feed at /rss on talkpython.fm.

01:01:44.800 --> 01:01:47.380
We're live streaming most of our recordings these days.

01:01:47.380 --> 01:01:55.220
If you want to be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at talkpython.fm/youtube.

01:01:55.220 --> 01:01:57.060
This is your host, Michael Kennedy.

01:01:57.060 --> 01:01:58.360
Thanks so much for listening.

01:01:58.360 --> 01:01:59.520
I really appreciate it.

01:01:59.520 --> 01:02:01.420
Now get out there and write some Python code.

01:02:01.420 --> 01:02:22.180
I'll see you next time.