CoCalc: A fully colloborative notebook development environment

00:00 Everyone in the Python space is familiar with notebooks these days.

00:03 One of the original notebook environments was SageMath.

00:06 Created by William Stein and collaborators, it began as an open-source, Python-based computational environment focused on mathematicians.

00:13 It has since grown into a full-blown company and has become a proper collaborative environment

00:18 for things like Jupyter Notebooks, Linux-backed Bash shells, and much more.

00:23 Think Google Docs, but across all these facets of development in your browser.

00:27 We welcome back William Stein to give us an update on his journey from professor to entrepreneur

00:32 building CoCalc along the way.

00:34 This is Talk Python to Me, episode 273, recorded July 7th, 2020.

00:40 Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.

00:59 This is your host, Michael Kennedy.

01:01 Follow me on Twitter where I'm @mkennedy.

01:03 Keep up with the show and listen to past episodes at talkpython.fm.

01:07 And follow the show on Twitter via at Talk Python.

01:10 This episode is brought to you by Linode and Talk Python Training.

01:13 Be sure to check out what the offers are for both of these segments.

01:17 It really helps support the show.

01:19 Speaking of Talk Python Training, have you been thinking about taking one of our courses?

01:23 We're participating in the latest Humble Bundle deal for Python developers along with a bunch of other great educators and tool developers.

01:31 Until July 22nd, you can get $1,400 worth of Python goodies, including three of our popular courses, for just $25.

01:41 Yeah, Humble Bundles are crazy.

01:43 That's $25 for our three courses and all those other things combined.

01:48 Just visit talkpython.fm/humble2020.

01:51 That's talkpython.fm/humble2020 altogether before July 22nd to take advantage of this offer.

01:58 Now, let's get to that interview.

02:00 William, welcome back to Talk Python to Me.

02:03 Thanks for having me back.

02:04 It's really great to have you back.

02:05 Your story is one that I've thought about a couple times over the years because it's really inspiring.

02:11 And it's quite interesting how you created SageMath and you went on this journey.

02:16 And it sounds like the journey is still going strong and you've come a long ways.

02:20 Yeah, it's been a very intense and exciting couple of years.

02:25 I started SageMath way back in 2004, worked on it an enormous amount until about 2013.

02:32 And for the last few years, I've been working on CoCalc, which is a web application whose goal partly is to make it really easy to use Sage without having to install anything, but also a lot of other open source software.

02:45 Right, absolutely.

02:46 So kind of notebooks before notebooks were cool, huh?

02:49 Yeah, so in Sage, we needed some sort of notebook interface.

02:52 So we wrote something called the Sage notebook back in 2006 through 2008, which back then, it looked a lot like it was the first sort of web-based computational notebook.

03:02 And we had to make a lot of interesting design decisions at the time.

03:07 And we put a lot of work into it over many years.

03:09 And then eventually, the whole Sage project, we switched to using Jupyter, using their file format and using Jupyter notebooks with a Sage kernel, rather than having to maintain our own completely separate notebook just for Sage.

03:22 Yeah, I guess before we get too far into it, maybe just tell folks what Sage is.

03:26 It has been almost four years since you were on episode 59 talking about Sage math and what it was.

03:32 So a lot's happened since then.

03:33 Yeah, so Sage is a big piece of software that's open source and free.

03:39 And the goal is to provide an alternative to Mathematica and Maple and Magma and various mathematical software systems.

03:46 But technically, what it is, is basically a Python library for mathematics.

03:52 And main things that distinguishes it from, say, SimPy is that we really, really focus on advanced research mathematics applications.

04:01 So things in pure mathematics like number theory, topology, differential geometry, etc.

04:07 So it gets used in physics and mathematical research.

04:11 But it also has a lot of functionality related to undergraduate mathematics teaching.

04:15 Oh, nice.

04:15 Yeah, that's a very interesting angle that we're going to talk about.

04:18 And your background's in number theory, right?

04:19 Yes.

04:20 Yeah.

04:20 I did my PhD in number theory in 2020 years ago at UC Berkeley.

04:25 Number theory.

04:26 So interesting.

04:27 That's one of the things I didn't study that much when I was working on my math PhD, which, to be clear, I didn't finish.

04:32 I dropped out halfway through and went and did computer stuff, which is fine.

04:36 But yeah, I was more on the analysis side.

04:38 But very, very cool.

04:39 Number theory is quite an interesting topic.

04:41 Now, I think, you know, let's just get straight into a little bit of the history.

04:46 So back in 2006, you, some of your grad students, you had some other students helping out sort of along the way, right, to build out SageMath.

04:55 At the time, you were a professor at University of Washington in the math department, right?

05:00 Yeah.

05:00 And then you built this up to be used in the classroom and to be used in the research, also with this undergraduate focus that you talked about.

05:08 Yeah?

05:09 Yes, that's right.

05:10 Probably 20 different students at University of Washington worked on Sage over the years and then hundreds and hundreds of people around the world.

05:18 And I taught frequently a class to about 20 to 50 undergraduates really about how to use software in mathematics.

05:28 So I guess nowadays it might be called a data science class.

05:31 Yeah, exactly.

05:32 Right.

05:33 It's sort of like, remember back when engineering, like you would go get an electrical engineering degree to actually do computer science or like programming, right?

05:42 Like you, there wasn't really a fit.

05:44 So like it would kind of go there.

05:45 Interesting.

05:45 Okay.

05:46 The course covered a bunch of topics like how to use R, how to use Pandas, which is a Python library, how to use LaTeX.

05:53 So we'd spend a week or two on just how to use LaTeX.

05:55 And I mean, it was really kind of a whirlwind and it was difficult to coordinate all this software with the students to get them to install something on their computer.

06:03 It was just, I mean, not really an option.

06:05 Right.

06:06 And you have all these different platforms, right?

06:08 Yeah.

06:08 And I liked having a part of class every meeting where I would give them something to work on and then walk around and help them and just see what was happening live.

06:16 So it was, you know, more interactive.

06:17 And so I ended up writing, well, first using the Sage notebook and then later use writing CoCalc specifically to make teaching that class more effective.

06:27 So they could do LaTeX, they could use R, they could use Python libraries, they could use Sage, everything in the course of the class.

06:33 And moreover, I did it in a way that was collaborative.

06:36 So absolutely all the functionality of CoCalc is collaborative.

06:39 So you have multiple people simultaneously using a Linux terminal, editing a Python script, editing a Jupyter notebook, editing a LaTeX document, whatever.

06:47 That's sort of built, baked in from the ground up in CoCalc.

06:50 Yeah.

06:51 So CoCalc is, I guess the way I see it is like Sage math as a service plus a bunch more.

06:57 Is that like a fair representation of it?

07:00 Yeah.

07:00 Okay.

07:00 We'll talk about what those things are, but there's a lot of things going on there.

07:05 And this collaborative side, I think is super interesting.

07:08 I mean, we're just starting to see that in a lot of other areas.

07:11 For example, you know, with VS Code having like the live share stuff, right?

07:15 You know, Jupyter Notebook still doesn't have, that I'm aware of, this rich, like Google Doc level of back and forth that you can actually have, right?

07:24 There's some online systems that have it.

07:27 I think maybe Datalore maybe has it, but it synchronizes a sentence or a cell, right?

07:35 It's not like you see it going.

07:37 And what's really interesting about the collaborative stuff, which is why I think this is really valuable.

07:41 So I'm diving into it a lot here right at the beginning is you can even do like a collaborative bash shell and stuff, right?

07:48 Yes.

07:48 There's more than just, I can work on the notebook and then there's other stuff.

07:51 It's like all that is collaborative.

07:52 Every single thing is collaborative.

07:54 So you can have a collaborative bash shell with a little chat on the side where you can chat back and forth to each other.

07:59 And it's nicely integrated.

08:00 You see the other person as they're typing.

08:02 And we actually, it's kind of funny, we were looking at different terms to, you know, kind of do SEO for.

08:08 So just for fun, we played around with Linux online and online terminal.

08:12 And strangely, we're the top, we've been the top hit for both of those search terms for quite a while now.

08:18 And we're getting a lot of users for learning Linux online.

08:22 They use CoCalc, they use the terminal and they learn Linux.

08:25 Right, right, right.

08:26 Sure.

08:27 Because you get basically a synchronized interactive bash shell.

08:30 Exactly.

08:30 Oh, I hadn't thought about that.

08:31 But of course, that makes a lot of sense.

08:33 And there's also a code editor.

08:34 So you can edit a .sh file and then run it in the bash shell.

08:38 Okay, very cool.

08:39 You know, you mentioned the chat.

08:40 And one of the things on the website that I thought was cool.

08:43 So you've got cocalc.com, pretty straightforward.

08:47 That's awesome.

08:47 One of the things you mentioned, which I recently ran into in a programming variant, is you have math friendly chat.

08:56 Tell us what math friendly chat is.

08:58 Okay, so first, anytime you write, so the chat is marked down, but you can use dollar signs.

09:03 And it will typeset using LaTeX, whatever's between dollar signs.

09:07 So in that sense, it's kind of like you're writing in LaTeX.

09:09 So that basically makes it math friendly.

09:11 One other thing that I think makes it math friendly is you can look at any message in the past and double click on it and edit it in order to fix something that's wrong.

09:20 Mathematicians like everything to be exactly right always.

09:22 So whenever you see something that's wrong, you really want to be able to fix it.

09:25 Exactly.

09:26 You can edit other people's chats, not just your own.

09:28 So if somebody else writes a chat and you see a typo or something that's wrong, you can make an edit to their chat.

09:34 And it's recorded.

09:35 So there's a history of the changes.

09:36 And you can see, you can click a button to see what's the history of edits to a particular chat cell.

09:40 Wow.

09:41 That's really, really interesting.

09:42 Yeah.

09:43 You know, I slight diversion.

09:45 I think that that would fix so many problems with social media like Twitter, for example.

09:51 They say, you know, we're not going to let you edit chats because we want to have an authoritative view.

09:56 We don't want people to twist what it means.

09:58 Like, why isn't there just a history?

09:59 Like, show original.

10:00 You know what I mean?

10:01 Yeah.

10:02 It's so bad.

10:03 I just recently tweeted and tried to amplify this message as much as I can.

10:06 And I'm like, oh, no, there's a bad grammatical error in that.

10:09 But I can't fix it.

10:10 I just got to live with it.

10:11 Right.

10:11 And so I think that's a super cool feature.

10:13 The scenario where I ran into the equivalent of this is I was in a Zoom meeting.

10:18 I have office hours for my online courses.

10:20 And I wanted to share some bit of code.

10:22 I'm like, this little fragment of Python is what you need to fix it.

10:25 And I paste it into Zoom.

10:26 No indentation.

10:28 Right.

10:29 Not even indentation.

10:30 The person's got to go through and like rebuild it.

10:32 And it's obviously not a fixed width font and all those things, which I guess doesn't matter if you take away the indentation.

10:38 But still, just having those little touches, I think, is actually really nice.

10:43 And the co in co-calc stands for collaborative.

10:45 Am I guessing that right?

10:46 Yes, that's right.

10:47 Yeah.

10:47 So really focusing on that rich exchange, that sort of interactive bits there seems valuable.

10:52 Yeah.

10:52 For a long time, it was called SageMath Cloud.

10:54 And that really made you think it was just a cloud-based way to run SageMath.

10:59 And we ended up really, really supporting pretty much everything in the open source math-related ecosystem equally.

11:06 And so focusing just on Sage was really kind of not really what we were about.

11:11 And we also put so much effort into collaboration.

11:14 So we ended up coming up with a name that really focused on collaboration and just general calculation, not just Sage.

11:20 Yeah.

11:20 It's a great name.

11:21 I really like it.

11:22 I think it's the right move.

11:23 So one thing I do want to touch on, I think, is probably interesting for a lot of people out there listening.

11:29 We've got a lot of academics in the audience.

11:32 You started out as a professor at University of Washington in the math department, which is a very coveted and challenging position to get.

11:40 You worked on this project, and I'm sure it was not easy, but you decided to leave the professor position, leave the department, and then focus exactly on this.

11:48 Could you talk just a little bit about that for folks out there who may be wondering how it went or in similar situations?

11:54 I went to graduate school and was terrified would I ever be able to get a job at all in academia as a postdoc and didn't expect to.

12:03 But then amazingly, I did.

12:05 And I had a job as a postdoc for a while, and then I got a job at UC San Diego.

12:11 And surprisingly, I got hired with tenure out of my postdoc, which was not at all what I expected to have happen.

12:17 And so the job at University of Washington was fantastic.

12:20 It's a great department.

12:21 The chair of the department is a very active Sage developer.

12:25 So it's a very supportive environment.

12:28 I have no issues with the department at all.

12:31 I didn't leave for reasons like that.

12:32 But I just really, really felt it was the right time to focus full time on making CoCalc better.

12:39 And, you know, we have a critical mass of users, so it's worth it to do so.

12:43 We have three full time employees of the company, including myself.

12:46 So, you know, we can support.

12:48 There are enough users to at least support significant full time development.

12:51 But, yeah, it was a terrifying choice to leave a tenure job, take a salary cut, significant salary cut to work on CoCalc all the time.

13:02 So I really had to be very, very confident that this is going to work.

13:05 I know exactly how you feel.

13:09 It's quite a stressful thing to have a really good job in hand and say, I have this theory that this thing is going to work out amazing.

13:17 Let me, you know, with bills to pay and responsibilities, let me just jump in and try the thing.

13:23 And also my wife is faculty at University of Washington.

13:26 So we had solved the two body problem.

13:28 Oh, my gosh.

13:30 Yeah, that's really hard to solve a second time.

13:32 And I'm also very aware of how hard it is to get an academic position or to go back because half the time I was at University of Washington, there was very, very little hiring due to the 2008 financial crisis.

13:43 Yeah.

13:44 And it was around that time.

13:45 So now looking back, now that you're on the other side of that fence, how do you feel about it?

13:50 I really, really like my current job a lot working on CoCalc.

13:53 I love it.

13:54 You get pulled in a lot of directions as a professor or as an academic, right?

13:57 It's hard to focus on any one thing.

14:00 You have the, you know, the Monday, Wednesday, Friday, you end up with a full schedule, basically.

14:03 So I would basically Monday, Wednesday, Friday, all about teaching and then Tuesday and Thursday, all about meeting with students from, you know, 9 a.m. to 6 p.m.

14:10 I would just schedule in a, you know, an hour per student or per pair of students and just fill up my days.

14:16 And so we just get an incredibly end up with a very rigid schedule.

14:20 So instead of waking up and thinking, OK, what is the best possible thing that I could spend my time on today?

14:25 It was sort of already decided what I would be doing for the next, you know, 10 weeks.

14:29 You've got that 30-minute gap in there where maybe you could squeeze it in for the day and the rest is already booked, right?

14:34 I do like the freedom a lot of being able to iteratively decide on what's the best thing to work on.

14:39 And I feel like I still have a positive impact on education with CoCalc because a lot of educators use it in their classes.

14:46 And I'm still contributing to the education of students just in a slightly less direct way.

14:52 Maybe more so in some senses, right?

14:55 Hopefully.

14:55 Yeah.

14:56 At least in terms of numbers and maybe, you know, average out the impact, right?

15:00 Like what I was doing before is I was doing developer training and I would go and work for a week with 20 people or something like that.

15:07 And that was, I had a big impact with those 20 people.

15:09 But, you know, I had to leave that world to go do the podcast and do the online courses, which, you know, you work with thousands of people.

15:17 It's a really interesting trade-off to make.

15:19 It's very difficult to scale teaching in certain contexts.

15:23 Like as a university professor, there's various limits to how much you can scale up due to it.

15:29 Yeah.

15:29 Sort of being all owned by the university system.

15:31 Yeah.

15:32 We're both running into similar things.

15:33 So, yeah, it's nice that CoCal can scale up a lot more in your online courses as well than something in the context.

15:39 It's basically a matter of awareness rather than, you know, making more of you or whatever.

15:47 This portion of Talk Python to Me is brought to you by Linode.

15:50 Whether you're working on a personal project or managing your enterprise's infrastructure, Linode has the pricing, support, and scale that you need to take your project to the next level.

15:59 With 11 data centers worldwide, including their newest data center in Sydney, Australia, enterprise-grade hardware, S3-compatible storage, and the next-generation network, Linode delivers the performance that you expect at a price that you don't.

16:14 Get started on Linode today with a $20 credit and you get access to native SSD storage, a 40-gigabit network, industry-leading processors, their revamped cloud manager, cloud.linode.com, root access to your server, along with their newest API, and a Python CLI.

16:30 Just visit talkpython.fm/Linode when creating a new Linode account and you'll automatically get $20 credit for your next project.

16:38 Oh, and one last thing.

16:39 They're hiring.

16:40 Go to linode.com slash careers to find out more.

16:44 Let them know that we sent you.

16:44 Before we move off this whole academic side of things, let me just ask you really quick.

16:51 With all this COVID stuff going on and everything around universities and online education and Zoom education and whatnot, what's going to happen to universities?

17:00 Are they going to get a massive transformation?

17:03 Is it going to be hard for them to fill up their courses?

17:06 Do you think it's just going to go back to the way it was?

17:08 What's your view of the next five years of universities quickly?

17:12 Wow.

17:12 I really wish I had a good answer to that question.

17:15 I really want to know the answer to that because it has a big impact on how I should develop co-calc and my wife's job and many other things.

17:23 There are people who you'll see a lot of comments online by people who kind of think they know the answer to that question and have a pretty dire predictions.

17:32 They often are not seeing the whole picture of what a university is because a lot of research.

17:38 I mean, a really important part of the university is the research aspect of it.

17:42 Right.

17:42 Also, the social aspects of people meeting.

17:44 And it is definitely possible to have an online course, as long as it isn't too large, where I think you get a lot of the value of an in-person course.

17:52 If you conduct it over Zoom in real time, where you don't just record the lectures and play them back, you actually have real-time discussion between the students.

18:01 So I think the traditional value of the university can still be propagated via web-based mechanisms, via the web.

18:08 So I think that doesn't go away with COVID.

18:11 But yeah, I really, I think things aren't going to change dramatically, but evolution in a more of an evolutionary way.

18:19 So there'll be some changes, but it's not going to all come crumbling down.

18:23 Yeah, and that probably speaks good things for co-calc, which we'll get into in just a second.

18:27 Or just like the real value of the universities is the professors and the students could have a meeting.

18:32 And you still have that.

18:34 And the professors still exist, and the students still want to talk with them and learn a lot from them.

18:39 And I guess the research side of things, I hadn't really considered how much money grants bring into the university, right?

18:45 Like if you get a million-dollar grant, half a million or something in that scale goes to the university just straight out of pocket, right?

18:52 And so I think the whole research side of the university is still going to have an interesting story there.

18:58 We can have a massive-

18:59 Supporting them, yeah.

19:00 Yeah, it brings in, like at University of Washington, it's well over a billion dollars of their budget every year.

19:04 It's just research money that comes in.

19:06 How interesting.

19:07 All right, so let's dive into some of the features of co-calc.

19:10 I mean, we've been bouncing around a couple of them.

19:13 And maybe with this conversation, like let's start with talking about teaching a course.

19:17 So one of the things that you focused on with co-calc and its origins were helping teach a scientific or mathematical class, either in a classroom or potentially remotely these days, which I don't know how much you envisioned that before, but now it's pretty easy to do.

19:32 When I would teach classes, I would usually put all the content for the class on GitHub or on my website and then tell the students, hey, here's the website for the course.

19:40 Grab the content, download it, upload it, you know, whatever.

19:43 But as soon as other people started using co-calc and I made it available more generally, instructors would just write, I need a functionality in co-calc that does the following.

19:52 It takes a directory of files and assignment, pushes it out to all the students, lets them work on it in a way where I can see what they're doing.

19:58 And then at a certain point, I can collect the assignment, grade it and return it to them.

20:03 And I want that to all be integrated in the system.

20:05 So we implemented that and then in 2014 and then iterated on it and polished that kind of functionality over the years.

20:14 So now co-calc has its own integrated course management system where you, you know, paste in a list of email addresses of all of your students.

20:21 They get invited and they have their own little private workspace, a Docker container where they work on content for the course.

20:27 And they periodically receive assignments.

20:30 The assignments just appear in their project and they work on it.

20:35 And they don't have to do anything to submit the result.

20:37 It's just when it's due, it gets collected.

20:39 So that functionality is integrated into co-calc.

20:43 And it's done in a way where while the students are working, you can see their cursor moving.

20:47 You can like, basically you can look over their shoulder and see how they're doing.

20:51 See if they're running into errors.

20:52 See if things are working well.

20:53 Right.

20:54 Get a report of when they were working.

20:56 If you wonder how their notebook got to a particular state, we have this time travel feature.

21:00 It's kind of like a track changes.

21:01 You click a button and you get a slider and it shows you.

21:04 Exactly what the notebook was at about every three seconds as it's being edited actively.

21:10 That's pretty fine grained.

21:11 Yeah.

21:11 It's very, very fine grained.

21:13 And the students also have access to it.

21:14 And so I often notice students when I would walk around in a physical classroom with them,

21:19 they'd be working.

21:20 They'd say, I had this figured out five minutes ago and I messed it up.

21:24 And it's really hard to just undo, you know, back five minutes in time in a Jupyter notebook,

21:28 especially, which doesn't even have a global undo.

21:30 So with time travel, they just have a separate panel right next to the notebook, which is every version of the notebook every three seconds.

21:37 And so they can just look and see exactly what they had five minutes ago, copy, paste it back over, and they're back in business.

21:43 So it's a kind of reproducibility in a really practical sense.

21:46 It's super interesting.

21:47 I did notice that that jumped out at me, that the time travel story was pretty cool.

21:52 I didn't realize it was every three seconds.

21:54 That's great.

21:54 And I suspect you're already exchanging that data with the server for the collaborative side.

22:01 So just save it, right?

22:02 How this came about was I implemented an algorithm in 2014 for real-time collaboration called Differential Sync, which just passes around lots of messages.

22:11 It's this really weird, complicated algorithm, and it doesn't record any history of what happened.

22:16 And then an undergraduate, Jonathan Lee, who was working on CoCal as a student project, he added in some sort of time travel history feature because I think maybe some like online hackpad type program had it.

22:28 I looked at what he was doing, and I realized that you could basically rewrite our whole sync algorithm or come up with a different sync algorithm that automatically recorded the history.

22:37 And that was sort of the basis for the real-time sync as well.

22:40 So I threw out the old algorithm we were using, wrote a new one, and it made the thing he was implementing a lot easier to implement.

22:46 And then I started using it just in coding.

22:48 Like I'm editing a Python file or a TypeScript file or something, and I realized, hey, five minutes ago I had some code that I wanted, and I don't want to undo to get it.

22:57 It was not put in Git yet.

22:58 It's gone.

22:59 But with time travel, I just have it there.

23:01 So now whenever I'm coding, it's like a critical, I don't know, tool for me.

23:05 I can't imagine writing code without having this slider that gives me all past states in my code.

23:09 Well, it's a little bit like writing a document in Google Docs or Sheets or something like that, right?

23:14 Yeah.

23:15 I don't envision that I ever have to save or anything.

23:18 It doesn't quite have that fine-grained slider feel to it.

23:21 But I was watching a show yesterday.

23:24 It was like a sitcom type thing.

23:25 And the joke was these college or high school age kids were working on some – it was actually advanced math – on their laptop.

23:33 And there was a power surge, and their laptop died.

23:36 And they lost all their work.

23:37 I'm just like, what do you mean you lost all your work, right?

23:39 Like, this is such an outdated show from – I think it was really recent, actually.

23:43 But still, like, the concept of my work is not being saved is kind of wild.

23:47 With CoCalc, also with development, I've always assumed that for a while we had a lot of bugs, and you'd have to refresh your browser a lot because, you know, memory leaks or it just crashes.

23:56 So I've always designed kind of assuming that you're going to have to refresh your browser frequently or your network connection is really flaky.

24:03 And so a second or two after you type something, it goes to the server, and that's it.

24:08 That's all this – like, after that happens, you can refresh your browser, and you'll be exactly where you were.

24:12 And so that's just the assumption with the system that your browser is going to be crashing all the time.

24:16 Your network's terrible.

24:17 It's not true nowadays, but – It's a little bit like the chaos engineering from Netflix, right?

24:23 Like, let's just throw monkeys into the – you know, wrenches into the mix or whatever analogy you want and say, let's design assuming that it's going to be wrecked.

24:32 That's also a fundamental design constraint with collaboration, I think.

24:36 With CoCalc, you make a project which is a Docker container, you use Jupyter Notebooks in it, and you invite collaborators to work on it.

24:43 And the collaborators could easily, in a few keystrokes, cause all kinds of damage.

24:46 And so you want to be able to trust adding collaborators.

24:49 And so with CoCalc, as they edit your files, all the changes are saved.

24:54 And if you want to revert back to before they messed it up, you just do that, and you won't have to worry about anything being lost.

24:59 Also, the files on disk are snapshotted every 20 minutes, like the complete state of all the files.

25:04 So if they accidentally delete all the files, you can – I mean, you might lose a few minutes, but you're not going to lose very much.

25:09 And there's no way for collaborators that you add to delete the history of editing.

25:14 So it's really immutable.

25:15 It just is a moving forward immutable thing.

25:17 The only way to delete it is to create a support request and ask us to delete it.

25:21 At least that person.

25:22 That's great.

25:22 In a teaching context, that's really good.

25:25 Look, if you want to have like a scientific paper you published where you have no history of it, copy, paste.

25:30 Here's the new version, right?

25:32 I mean, same thing with Google Docs.

25:33 Exactly.

25:33 If I work on something and I share the Google Doc with someone else, you know, say it's like a business proposal or something,

25:40 I assume that what I've typed, they're going to potentially go and look at.

25:44 So either export it as a PDF or just, you know, just know when you go in that that's how it's going to be and it's just – that's fine, right?

25:51 Yes.

25:51 So let's round out this teaching side of things, I guess, really quick.

25:54 So probably in the fall, I don't know what the world is going to look like.

25:58 Fingers crossed, it's vaccines.

25:59 Everything's well, but, you know, that's not necessarily the likely outcome.

26:02 So there's probably a lot of teachers, both middle, high, college, and even grad school level teachers thinking,

26:10 right, well, I'm going to have to do this online.

26:12 One option is I get a whiteboard and I write on it with a pen, you know, like a digital pen to teach something.

26:19 It sounds like this might be a better option for certain types of classes, at least as a side-by-side with a video call or something.

26:26 What do you say to those people, like, how could they use it?

26:30 What role would it fill there?

26:31 If you ever had like a computer lab part-tier class where you wanted all the students to work on a Jupyter notebook or something like that,

26:39 like, CoCalc basically just puts that in the cloud in a way where you really can walk around and look at students' work and, you know, jump in and start editing and see when they're active.

26:48 So it solves that.

26:50 And it makes it a lot easier to have homework assignments that involve Jupyter.

26:53 Could I have like 20 students and just open 20 tabs and just, you know, command alt arrow between my students?

26:58 Yeah.

26:59 And you also, whenever they have an issue, they can type a chat in the side and you'll see a notification and you can click on that notification and you'll open exactly their document.

27:07 But yeah, absolutely.

27:08 You could just command, you can just tab through all the students.

27:11 Or if you had a really big monitor, watch them all at once.

27:13 Exactly.

27:15 Yeah.

27:15 With that 55-inch 4K TV to good use.

27:18 We are constantly being contacted by instructors who are basically being forced to move classes online and looking into options.

27:26 They find CoCalc.

27:27 They think it might work, maybe not.

27:30 They ask us a bunch of questions.

27:31 The answers are all yes.

27:33 And then they're amazed and they start using it.

27:36 So people often assume, I think, some of the things that we can do well, probably nothing can do them yet.

27:41 But we can do them now.

27:42 Very cool.

27:43 Now, I was going to save this question for a little bit later, but let me jump it ahead because it might be more relevant here.

27:49 What does this cost?

27:51 Is it, I know there's a free tier and you can go over to CoCalc.com and do stuff for free.

27:55 But is this teaching a course free or what's the story there?

27:59 You can use CoCalc for free in a couple of ways.

28:02 But if you really want to teach a course and have the students have a good experience, etc., then it's basically, I mean, there's a range of different costs.

28:13 But basically, think $15 per student.

28:15 That's roughly, on average, what it comes out to.

28:17 There's a fee the students can pay directly or you can have your university pay us or something else.

28:24 Yeah.

28:24 I mean, it seems like maybe in a high school scenario, you might have the school district pay.

28:28 But in a college scenario, man, students are used to just paying and paying.

28:33 I remember when I went to my first, I first started out in community college and I paid my tuition and it was like $400.

28:40 Like imagine that, right?

28:41 For a full 15 hours course.

28:43 And then I went to the bookstore and it was like $600.

28:46 I'm like, how can the books cost more than the entire tuition of all the courses?

28:50 Like this doesn't make any, like $200 for a chemistry book or something like that.

28:54 So I think $15 to take a math course is probably a drop in the bucket.

28:59 It's probably easy enough.

29:00 If there's a textbook, it's about the sales tax on the textbook.

29:03 It's about the cost of the sales tax on the textbook.

29:06 Yeah.

29:06 It's interesting when, you know, from a starting a business side of things, some universities, it's the culture that all the students pay directly.

29:13 And other universities, that would be, it's considered unethical and crazy to have the students pay directly.

29:18 It's kind of like random which one it is.

29:20 Like University of Washington, the students pay.

29:22 That's just how it is.

29:23 Whereas various universities and other places, it would be considered just crazy to do that.

29:28 Right, right.

29:29 How interesting.

29:29 Well, if you're talking Mathematica or MATLAB or Maple, like student pay is like a lot of money.

29:35 Yeah.

29:36 Whereas there's the base rate, but oh, I also need the wavelet decomposition toolbox and I need this other toolbox.

29:42 And now all of a sudden it is legitimately really expensive.

29:44 All right, cool.

29:45 So the teaching side, I think it's really interesting.

29:48 And maybe let's talk about some of the other things that it does.

29:50 So you talked about the time machine thing, and that's a form of version control.

29:56 It's a little less structured than I'm saving a point in time actively.

30:00 Is there a way to integrate this with a GitHub like a thing or external source control?

30:05 Can I export it?

30:06 How does that happen?

30:07 A CoCalc project is literally just a Docker container running Ubuntu, currently 1804, but it seemed to be 2004.

30:14 And you can use the terminal and any Git commands in the terminal.

30:18 That's what people frequently do.

30:20 So there's like in the Docker, the shared Docker collaborative space, there's a Sage workbook or something to that effect, plus the other files.

30:29 And I can just get add those things, get in it and then get push and get add and so on.

30:33 Yeah, you just have a home directory.

30:35 It's a file system.

30:36 The default free quota is three gigabytes per project.

30:39 You can create as many projects as you want.

30:41 And within a project, you can say, you know, get clone, get pull, get push, get add, and just use all the Git commands, which usually is pretty easy.

30:50 Because anything you could possibly imagine wanting to do with Git, you just Google it and you find it pretty quickly.

30:55 And there's like 10 examples.

30:56 Yeah, for sure.

30:58 For sure.

30:59 But there's no graphical integration yet.

31:00 It's all like you use the terminal.

31:02 Sure.

31:02 What about file exchange?

31:04 Is there like a draggy droppy in the tool or do I have to use SCP or anything like that?

31:10 There's a pretty nice, I guess, file explorer application inside of CoCalc.

31:14 And it has drag and drop functionality in that you can just drag a file or directory from your desktop.

31:20 onto the file listing and it just gets uploaded automatically.

31:23 And next to any file, there's a download button.

31:25 So you can download files back.

31:26 You can also set up an SSH key, just like in GitHub, either for an individual project or across all projects you're using.

31:33 And once you upload an SSH key, your public key, then you can SSH to any of your projects.

31:38 And then you could use SCOPPY or R-Sync or some graphical tool on your desktop to sync files back and forth.

31:45 Yeah.

31:46 It's really just like a remote Linux server, basically.

31:48 Yeah.

31:48 Cool.

31:49 So you say Docker.

31:50 A lot of people hear Docker, they think transient.

31:52 Is there like a persistent map volume or something like that where it's kind of permanent?

31:59 We have a really cool thing.

32:00 What we do is we have this, it feels like you're running a Docker container that's about where the image is about 300 gigabytes.

32:07 Because we install every Python, like 9,500 Python packages, like 10 different versions of Sage and a whole bunch of versions of Anaconda.

32:16 We install everything and we install everything and the Docker container.

32:26 When the Docker container when the Docker container spins up.

32:27 And we also plug a home directory into that Docker container.

32:31 And so you have a fairly lightweight little Docker container that via remote file systems has a huge amount of useful information inside of it.

32:39 Yeah.

32:39 That's cool.

32:40 It's important because we want to be able to start the thing fairly quickly.

32:43 Yeah.

32:44 Yeah.

32:45 Every little bit of computational resource, you want to keep that, right?

32:48 Yeah.

32:49 Yeah.

32:49 Speaking of which, where are you running this?

32:51 When I go to CoCal, can I calculate?

32:54 Where do I calculate?

32:55 It's on Google Cloud Platform and Google Compute Engine.

32:57 And physically, it's on the East Coast currently.

33:00 That seems like one of the best choices, right?

33:03 Yeah.

33:03 That's where my servers are.

33:05 They're New York City, actually, right now.

33:07 But you've got to pick somewhere.

33:10 It's good for the US.

33:11 It's good for Europe.

33:12 It's less good for other folks.

33:13 But if you've got to pinpoint somewhere in the globe, that's probably, as terms of user, at least from what I experience, that's a good place.

33:20 We have a lot of users in Europe.

33:22 And so it makes sense time-wise.

33:24 We do plan to expand to other data centers when we have more users.

33:28 And also, one of our main developers, Harold Schilly, lives in Austria.

33:33 And so he wants it to be at least not ridiculously far from him.

33:36 Yeah.

33:37 Yeah.

33:37 No, that makes perfect sense.

33:38 I agree.

33:38 I think it's a good spot.

33:39 You talked about running graphical software.

33:42 I guess let me go back and touch on one other thing.

33:44 You talked about all these packages.

33:46 So it sounds like if I can SSH in or I can go to the Bash shell through cocalc.com's live interactive bits, I can pip install whatever I want when I get in there?

33:56 Yes.

33:56 Yeah, you can pip through your pip install.

33:58 And it just installs it into tilde slash dot local.

34:01 And yeah, so you can install any packages.

34:04 You might find that almost anything you want to install has already been installed because our Python environments have basically everything anybody's ever requested.

34:12 since 2014.

34:13 Is it standard Python?

34:16 Is it Anaconda?

34:18 What do I get when I drop it?

34:20 Yes, in the sense that there's a Python 2 system-wide, a Python 3 system-wide.

34:24 There's several versions of Anaconda.

34:26 There's each version of Sage has its own Python install inside of it.

34:30 So there are many, many different Pythons available.

34:32 If you look at the list of Jupyter kernels that we have there, it takes up the whole page.

34:36 Interesting.

34:37 So speaking of Jupyter and Jupyter kernels, it sounds like SageMath is slightly different than Jupyter itself, Jupyter Notebooks.

34:46 But also that there's some compatibility between, like, could I take a Jupyter notebook, IPYNB, whatever the extension is, and drop it up there and work on it?

34:55 In CoCalc, you can use Jupyter Notebooks.

34:57 We have a reimplementation of the Jupyter stack, but it uses standard Jupyter kernels.

35:02 And it's designed to be as compatible as we can possibly make it with, you know, the official upstream Jupyter project.

35:08 And that's a big difference, I think, from some of the other Jupyter-ish implementations, like CoLab and Kaggle.

35:15 Like, we have all the same menu options as Jupyter Classic.

35:19 And we really, I put, you know, basically my to-do list when implementing the new version of Jupyter for CoCalc was to implement every single feature that Jupyter Classic has.

35:29 Maybe we're at 99%, but our goal is to be 100% compatible.

35:34 That's cool.

35:35 So it sounds like if you have students and you're talking to them about, here's, they know Jupyter or they're looking at Jupyter documentation, it'll pretty much just work.

35:43 Yeah.

35:44 Not only that, in CoCalc, we have our built-in Jupyter client, but we also have a button that you can click, which runs a Jupyter Classic server from the project.

35:52 And then it's, you're looking at exactly the same.

35:55 It literally is.

35:56 Yeah.

35:57 It is Jupyter Classic.

35:58 And we have another button that runs Jupyter Lab.

35:59 So you can fully just use Jupyter Lab or Jupyter Classic to interact with your project instead of CoCalc's view of the project.

36:06 Do you probably lose the collaboration at that point?

36:08 Yeah.

36:09 Because those don't have collaboration.

36:10 So you lose the collaboration, but at least you can, you get 100% compatibility with those programs.

36:16 Right, right.

36:16 That's a good escape hatch.

36:18 One thing that I thought was pretty interesting that I didn't necessarily expect was I saw on the documentation, X11.

36:25 Yes.

36:26 That's like a remote windowing system for a full-on Linux applications that are Windows-based.

36:32 So you can do that in here as well?

36:34 Yes.

36:35 This is kind of weird, but there's a couple of HTML5-based X11 servers out there like Apache, Guacamole, and Xpra.

36:43 And people, every once in a while, people requested that we add something like that to CoCalc.

36:49 And it was always like, you know, they can do 99% of what they need to teach this class, but there's this 1% where they want to use an old Java application with a graphical interface for a few days in the class to look at some data or, you know, something like that.

37:01 And so I decided to finally try to figure out how to do it and ended up writing an Xpra client.

37:06 So I took, so Xpra, Xpra is some X11 server that runs under Linux, and it has an HTML5 client that they have.

37:16 And I took their code, and I basically rewrote it.

37:18 In a way, basically like paint the screen on an HTML canvas or something like that.

37:22 It sends you a bunch of data and then use an HTML5 canvas to display that data, and you handle clicks and so on.

37:28 So I just rewrote their client to fit into CoCalc collaboratively.

37:32 Like you can have multiple people looking at the same X11 application, and when somebody moves their cursor, you'll see the cursor move around.

37:38 So that's actually collaborative.

37:40 That sounds really cool, actually.

37:41 The gotcha is just, you know, it's over the network, so it's slow.

37:45 There can be latency, which is really annoying for a graphical application.

37:49 Yeah, but it exists.

37:50 It's better than nothing.

37:51 There are a few, like if you want to play around with Tickle TK to do some old school Python graphical user interface development for like, you know, an assignment in a class, you can totally do that in CoCalc.

38:01 You can fire up Idle, which is the official Python, you know, graphical interface that nobody uses, but it's official.

38:07 Yeah.

38:08 And you can type in some commands to pop up graphics and they'll work.

38:11 Yeah, it's pretty neat.

38:12 Or you can run GIMP or Inkscape or something.

38:14 Yeah.

38:15 So I don't think it gets used a lot, but again, it was really aiming at the very edge case that would be something that stops somebody from teaching a course or otherwise using CoCalc.

38:25 Right. Or they have this one GUI interface that they did for this research project and they want to move the research project there.

38:32 Yeah.

38:32 And if they're going to do that, they need it to work or something, right?

38:35 Yeah.

38:36 Yeah.

38:36 Very interesting.

38:37 All right.

38:37 Well, what else is notable?

38:38 You know, something that jumps out at me is you kind of have like MATLAB support in a bit.

38:43 Yeah, Octave.

38:44 In a sense, right?

38:45 Through Octave.

38:45 Tell people about that.

38:46 The Octave Jupyter kernel, we, you know, make sure that graphics work nicely.

38:50 And yeah, so you can use Octave.

38:52 Octave is very similar to MATLAB, at least for the basic language.

38:55 And yeah, Jupyter.

38:56 And in the graphical interface we just talked about, there's also an Octave client, which, you know, shows Octave graphics.

39:02 And kind of looks like MATLAB.

39:04 I don't know much about Octave.

39:05 I haven't, I used to do some MATLAB stuff, but I never do think of Octave.

39:09 How close are those two things?

39:10 Very close, but there's deep functionality for certain special domains that aren't implemented in Octave, but are available in MATLAB.

39:18 So like, there's various toolboxes and so on that aren't there.

39:21 But I think the language is nearly compatible.

39:24 Cool.

39:24 So that's a pretty good option.

39:25 And again, online as part of this thing?

39:28 Yeah.

39:29 And other things to note about CoCalc, you can run CoCalc yourself.

39:32 There's a Docker image that we regularly update.

39:35 And so you just type a command in Docker and you're running your own CoCalc server on your laptop or on a remote server.

39:40 It's like a mini version of CoCalc with all the same functionality that you run locally or anywhere you want.

39:46 Yeah.

39:46 Very cool.

39:47 Okay.

39:47 Nice.

39:48 LaTeX.

39:49 We already talked about LaTeX in chat.

39:51 And for those who don't know, LaTeX is a markup language, kind of like HTML, but for visually accurate mathematical representation.

40:00 So if you want to do a sum from N equals one to a thousand, it has the little N equals one at the bottom and a thousand.

40:06 Like it's the representation that you would write in, you know, proper math class.

40:10 Right.

40:11 So it's kind of a markup for that.

40:12 And you, you all have some pretty good support for that there as well.

40:15 Yeah.

40:15 So it's what everybody who writes papers and math and physics and probably some other areas in academia will use for sure, because it looks very, very professional.

40:24 It has excellent support for cross-referencing between different sections and splitting your document up into different files and so on.

40:32 So we fully support that.

40:34 Also, there's a package called, I think, PyTech, like use package, P-Y, and then you go backslash P-Y, and then right in your LaTeX document, you can type a Python, a bit of Python code, and it gets evaluated whenever the document gets updated.

40:48 So that's pretty cool.

40:50 And there's something similar for Sage called SageTech, which lets you embed plots and arbitrary Sage code in your document, which automatically get updated when the document gets compiled.

40:59 And we make that very easy to use inside of CoCalc.

41:02 So you don't have to install anything or mess with anything.

41:05 It just works.

41:05 Yeah.

41:06 And one of the things I thought was pretty cool there was right on the home screen, you've got a representation of some nice, it's not that fancy, but it is, you know, proper LaTeX representation of, you know, fractions and square roots and stuff.

41:21 So you say, show LaTeX.

41:24 So you've got a solver that solves an equation.

41:28 And then you say, just show LaTeX of the solution.

41:31 And boom, there it is.

41:32 That's pretty cool.

41:32 If you just say LaTeX of the solution, then it shows you the code that you would put into a LaTeX document or a markdown file to get that beautiful formula.

41:41 Cool.

41:42 So it shows you that like the LaTeX markup, the dollar or whatever.

41:44 Yeah, exactly.

41:45 Yeah.

41:45 Okay.

41:46 That's basically a feature of Sage and other systems.

41:49 Like SimPy has something similar.

41:51 So you build up some complicated answer and then you just say, give me the LaTeX so that I don't have to try to convert this ASCII looking thing into something nice.

42:02 You'll make a mistake or a typo.

42:03 Yeah.

42:04 You definitely don't want that.

42:06 Yeah.

42:06 So another thing that stood out to me was that you have database support for things like Postgres and whatnot.

42:12 You know, again, it's just Ubuntu Linux.

42:15 So we have a lot of things pre-installed that people requested.

42:17 And it's just sitting there and you can use it from using Python or other languages.

42:22 Yeah, sure.

42:23 So maybe you analyze some data, throw it in there, put some indexes on it and let you ask a bunch of questions quick and fast.

42:29 And the nice thing is it's all in one place, like all in the same site, all in the same container.

42:34 You can work with some Python code.

42:36 You can use it in a Jupyter notebook.

42:38 You can play around with the data from a database and then write a paper in LaTeX all in the same place with collaborators.

42:44 Yeah, that's really cool.

42:45 And it's all saved there.

42:46 Very nice.

42:47 So there's a bunch of other features that we're not talking about, but these all seem really, really cool.

42:51 And the collaborative bits are really neat.

42:53 Like there's our support and other stuff.

42:55 I guess maybe we could talk a little bit, getting shorter on time here, I guess, but talk a little bit about some of the internals.

43:01 You already mentioned that it's basically your own re-implementation of the Jupyter stack because the way Jupyter worked, it didn't really support real-time collaboration, right?

43:11 Yeah.

43:11 Basically, Jupyter has a front-end client that receives a bunch of messages from the kernel.

43:17 They just kind of get proxied from the back end to the front-end client.

43:20 And the front-end client decides what to do with all that information.

43:22 So a huge amount of processing goes on the front-end.

43:26 And with CoCalc, that kind of model really doesn't make sense for us.

43:29 So instead what happens is back on our servers, we process all the messages from the Jupyter kernel and then figure out how that should make the document evolve.

43:38 And then we just synchronize the document between the front-end and the back-end.

43:43 And it's a very different architecture than Jupyter currently has.

43:47 And so it made real-time sync difficult.

43:49 I did implement real-time sync directly on top of Jupyter Classic for a couple of years.

43:54 But it mostly worked.

43:56 But every once in a while, things would go wrong.

43:58 And we kept getting complaints from users.

43:59 And I really wanted that last 1% to work.

44:02 And I kept trying to figure out how to do it.

44:05 And I couldn't find any way to do it besides just rewriting the whole thing, which took months and months of effort.

44:09 It's one of those things that doesn't seem so...

44:11 It's a tough decision, right?

44:12 You're like, oh, no, we're going to have to do this.

44:14 Here we go.

44:14 You have to underestimate the difficulty in order to start.

44:17 Otherwise, you wouldn't do it.

44:20 Exactly.

44:20 I bet we could pull this off two months later.

44:23 You're like, boy, this was more than I bargained for.

44:25 Exactly.

44:26 Now you're committed, right?

44:27 Yep.

44:27 Yep.

44:28 Yeah.

44:28 And we continue to be committed.

44:31 So we have to implement every extension as well.

44:33 We have to re-implement all the extensions.

44:35 So we prioritize that based on what is most in demand.

44:40 So people want the table of contents extension to Jupyter.

44:42 So we have to re-implement that so we don't get anything for free.

44:45 Do you have telemetry and stuff like that that tells you what's going on?

44:48 Or is it sort of support requests coming in?

44:50 We have a lot of telemetry because of this time travel.

44:53 We're recording everything that's happening and so on.

44:55 But we basically ignore it for the most part and make our development decisions and our direction based almost entirely on support requests from users.

45:04 We make it very easy for users to ask us questions.

45:06 Just click a button help in the top of the screen, type something, and then we get the support request.

45:11 And so that really drives our development.

45:13 It's just what people are wanting and people are willing to share.

45:17 We also have a Discord chat room where people often tell us what they want.

45:21 So there's probably a lot of Python going on here.

45:24 But there's also probably a lot of JavaScript being basically a very fancy single page application.

45:31 Exactly.

45:31 Right.

45:32 Yeah.

45:32 It's lots and lots of TypeScript code.

45:35 Both the front end and the back end is mostly JavaScript.

45:39 We use Python mostly for managing the complicated back end stuff, like getting projects to run and moving data around.

45:46 And there's just a lot of tasks that periodically happen on a Kubernetes cluster and monitoring tasks.

45:53 But CodeCalc is mostly a TypeScript application.

45:55 Because a lot of it runs in the web browser and that's just the canonical language these days.

46:01 It sadly is.

46:02 I wish that the story could be a little bit different.

46:05 I mean, on one hand, I lament it for Python.

46:08 On the other, what other environment is there only one language?

46:12 Yeah.

46:12 It's pretty amazing how JavaScript has been dominant.

46:16 Other things haven't really picked up in the browser.

46:18 Yeah.

46:19 It's really surprising.

46:20 Things like Dart and other things just haven't caught on.

46:22 Yeah.

46:22 And a lot of the foundations we might have with WebAssembly and so on.

46:26 I guess I'll take us on a slight diversion.

46:28 I really wish that there was a way to have more runtimes running as WebAssembly in the browsers.

46:35 Right.

46:36 I mean, on one hand, you say, well, it's C, so you compile it to WebAssembly and then you can just link it and download it.

46:41 But no app is going to say it's acceptable to have a 10 megabyte download to get my page to show.

46:46 Right.

46:47 Yeah.

46:47 I mean, maybe if it's something like CodeCalc where you're there for a while, but like not in the general sense.

46:53 But, you know, if you had a set of canonical, like here's the Java runtime, here's the .NET, here's the CPython, here's the.

46:58 And those just got shipped as part of Firefox, Chrome, Edge, and that you didn't have to download them.

47:04 All of a sudden, you know, we would be free from JavaScript.

47:08 But that's a world I see in the future.

47:10 It doesn't sound like Python on the front end has a lot of value to you guys at this moment.

47:16 We don't get any requests for that from our users ever.

47:19 I don't know if that's because there aren't lots.

47:22 I'm not saying that there isn't a huge demand for it.

47:24 It's just that I think maybe other sites like Replete and just other things already solved that problem.

47:31 So people don't come to us because they don't need us for that, to solve that problem.

47:34 You already have a way for them to execute Python.

47:37 Yeah.

47:37 Right.

47:38 It happens back there on the server in a totally fine way.

47:41 It would really be just for you guys so that you could say we'd rather write Python if that were the case.

47:46 I don't even know if that necessarily makes sense.

47:48 Right.

47:48 Because you probably have nice front end frameworks.

47:50 Yeah.

47:50 We just type it.

47:51 We use TypeScript and React.js.

47:53 Exactly.

47:54 TypeScript's pretty good and provides a lot of value and it kind of fits into the browser ecosystem.

47:59 So there's that.

48:00 I mean, Python's somehow developed as a language that's super good for data science and scripting

48:06 and a lot of types of applications.

48:09 But I think TypeScript and JavaScript have developed to be really good for asynchronous applications

48:14 where you're often responding to a flow of events and where everything is not as non-blocking as

48:20 possible.

48:21 And most code, individual code runs very quickly and is responding to things.

48:24 So it's, they kind of tend to be solving different problems well or differently.

48:29 Right.

48:29 Well, that's what you get when you write a language that has no mechanism for synchronous calls.

48:34 Yeah.

48:34 Right.

48:35 It has to be callback or async and await, depending on how modern it is.

48:39 Interesting.

48:40 All right.

48:40 Well, it sounds like a pretty cool project.

48:43 I mean, it's, I love the collaborative side of this thing.

48:47 And when I heard CoCalc and I thought about it, I envisioned the notebook thing to be very

48:53 collaborative, but I didn't expect, you know, collaborative GUIs, collaborative Linux terminals,

48:59 collaborative, all these things.

49:00 I think this is, you know, at a higher level of collaboration than I realized, which is,

49:04 you know, a real testament to what you guys built.

49:05 Cool.

49:06 Yeah.

49:06 All right, William, before I let you out here though, I've got to ask you the two questions

49:09 and I'm looking forward to the answer to this one.

49:12 First of all, if you're going to write some code, what editor do you use?

49:15 So CoCalc itself, when I started writing it, I also bought one of the original Chromebooks

49:19 from Google.

49:20 I can't remember what it was called, but it was...

49:22 One of the nice ones.

49:23 Yeah.

49:23 But it was really running Chrome OS, which was very limited back in 2013, 2014.

49:28 And I decided to write CoCalc entirely from within CoCalc.

49:32 And, you know, I bootstrapped it a little bit to get going, but then after that,

49:36 all the development since then has been from within CoCalc.

49:38 So really the editor under the hood is Codemirror.

49:42 I've developed, you know, added a ton of extensions to Codemirror and functionality to...

49:46 For example, if you're editing a page of code, you can split it horizontally,

49:49 vertically as many times as you want and look at lots of different points in the document.

49:53 There's code folding, syntax highlighting.

49:55 The time travel feature just...

49:58 I kind of consider that a basic need for editing code now.

50:01 I find it impossible to imagine editing code without that.

50:05 So the one thing that is really lacking from CoCalc for being a full IDE is VS Code-style

50:11 language server protocol support, where you get extensive information about typing and other

50:17 sort of static analysis of your code.

50:18 And there's no reason that I can't add it.

50:21 It's just hard and I haven't done it yet.

50:22 And I'm working on that.

50:23 For sure.

50:24 But before CoCalc, I used Emacs for a very, you know, for like 20 years.

50:28 Well, you always think about, you know, when as a language, a proper language, when it can

50:33 build itself, you know, when it can compile its own runtimes and when it's creating itself.

50:39 And this is an interesting thing for the editor as well, right?

50:42 When can you create this tool with this tool itself?

50:46 And I think that's a pretty awesome thing.

50:49 It's got to help make it better, right?

50:51 Because it's one thing to say, well, students get to type in here and they're a little annoyed

50:55 with that thing, the way it works, but whatever.

50:57 Yeah.

50:57 Versus I got to live in this thing day to day.

51:00 No, we're fixing that problem all of a sudden, right?

51:03 Is there some of that?

51:03 Oh, yeah.

51:04 Yeah.

51:04 Like we've had a million problems over the years and they hit me all day long every

51:09 day.

51:09 So it definitely helps me prioritize them.

51:12 Not taking this paper cut anymore.

51:14 Exactly.

51:15 Tomorrow I'm fixing this or whatever.

51:16 All right.

51:17 And then a notable PyPI package.

51:19 So there is a package for SageMath.

51:21 You can do pip install SageMath, but it's a kind of silly package because all it does is

51:26 check that you have Sage installed somewhere on your computer.

51:29 It doesn't actually install anything except this sort of like tiny little five lines of code.

51:34 code.

51:34 And in the future, though, I hope someday, maybe in five years, when you do pip install

51:40 SageMath, it will actually install Sage into whatever Python environment you're using.

51:45 So that's a major challenge because Sage is like a million lines of code.

51:50 That's new code written for Sage plus a bunch of dependencies, which are themselves, you know,

51:55 hundreds of thousands of lines of C code and assembly and other things.

51:58 But, you know, that's if I had a million dollars to spend on Sage development, that would be what

52:03 I would want to do.

52:04 I would want to break up Sage into smaller pieces.

52:05 Take like the really cool Python functionality we've written for Sage, break it out of Sage

52:11 and make it lots of Python packages.

52:12 So my favorite PyPI package is the many non-existent packages that exist only in my head, which together

52:19 someday would make pip install SageMath install SageMath.

52:23 There you go.

52:24 That's like calling your shots, right?

52:26 Yeah.

52:26 We're going to do this.

52:27 Here it comes.

52:28 Awesome.

52:28 It's from 2030.

52:30 Exactly.

52:30 Put it on the calendar.

52:32 All right.

52:32 Final call to action.

52:33 People want to get started with CoCalc.

52:36 Maybe try it in their classroom or try it in one of their research projects.

52:39 What do they do?

52:39 Just type CoCalc.com into your browser and you'll get to the website and there's lots

52:44 of information right on the website about how to use it.

52:46 You can also be type share.coCalc.com.

52:50 You can browse through like tens of thousands of documents people have publicly shared from

52:55 CoCalc.

52:56 So they're like Jupyter notebooks, markdown files, all kinds of different things.

53:01 And when you're looking at any one of those documents, there's a big green button at the

53:05 top, which is called open and run or run now.

53:07 And when you click it, it will start up CoCalc and interactively run that thing.

53:12 And you don't have to make an account.

53:13 You don't have to do anything at all.

53:15 Just sit back for a few seconds and then you'll be using CoCalc.

53:18 And it's completely anonymous and free to do that.

53:21 Yeah.

53:21 Very, very cool.

53:22 All right.

53:22 Well, great work on this project.

53:24 It's so nice to check in with you.

53:26 You know, we last spoke in 2016 and I think at the time you were still in your professor

53:31 role, but working on SageMath.

53:33 So what a long journey you've been on and well done.

53:36 Thank you very much for having me on the podcast.

53:37 Yeah, you bet.

53:38 Bye.

53:38 Bye.

53:39 This has been another episode of Talk Python to Me.

53:42 Our guest on this episode was William Stein, and it's been brought to you by Linode and us

53:47 over at Talk Python Training.

53:49 Start your next Python project on Linode's state-of-the-art cloud service.

53:54 Just visit talkpython.fm/Linode.

53:56 L-I-N-O-D-E.

53:58 You'll automatically get a $20 credit when you create a new account.

54:01 Want to level up your Python?

54:03 If you're just getting started, try my Python Jumpstart by Building 10 Apps course.

54:08 Or if you're looking for something more advanced, check out our new async course that digs into

54:13 all the different types of async programming you can do in Python.

54:16 And of course, if you're interested in more than one of these, be sure to check out our

54:20 Everything Bundle.

54:21 It's like a subscription that never expires.

54:23 Be sure to subscribe to the show.

54:25 Open your favorite podcatcher and search for Python.

54:27 We should be right at the top.

54:29 You can also find the iTunes feed at /itunes, the Google Play feed at /play,

54:33 and the direct RSS feed at /rss on talkpython.fm.

54:38 This is your host, Michael Kennedy.

54:40 Thanks so much for listening.

54:41 I really appreciate it.

54:42 Now get out there and write some Python code.

54:44 I'll see you next time.

54:45 I'll see you next time.

54:46 Bye.

54:46 Bye.

54:47 Bye.

54:47 Bye.

54:49 Bye.

54:50 Bye.

54:51 Bye.

54:52 Bye.

54:53 Bye.

54:54 Bye.

54:55 Bye.

54:56 Bye.

54:57 Bye.

54:58 Bye.

54:59 Bye.

55:00 Bye.

55:01 you you you Thank you.