Quarto: Open-source technical publishing

00:00 In this episode, I'm joined by JJ Allaire, founder and executive chairman at Posit,

00:03 and Carlos Scheidegger, a software engineer at Posit, to explore Quarto, an open source tool

00:10 revolutionizing technical publishing. We discuss how Quarto empowers users to seamlessly transform

00:15 Jupyter notebooks into polished reports, dashboards, e-books, websites, and more.

00:20 JJ shares his journey from creating RStudio to developing Quarto as a versatile multi-language

00:25 tool while Carlos delves into its roots in reproducibility and the challenges of academic

00:31 publishing. Don't miss this deep dive into the tool that's shaping the future of data-driven

00:36 storytelling. This is Talk Python to Me, episode 494, recorded December 20th, 2024.

00:42 Are you ready for your host, here he is!

00:46 You're listening to Michael Kennedy on Talk Python to Me. Live from Portland, Oregon,

00:51 and this segment was made with Python.

00:53 Welcome to Talk Python to Me, a weekly podcast on Python. This is your host, Michael Kennedy.

01:01 Follow me on Mastodon, where I'm @mkennedy, and follow the podcast using @talkpython,

01:06 both accounts over at fosstodon.org, and keep up with the show and listen to over nine years

01:12 of episodes at talkpython.fm. If you want to be part of our live episodes, you can find the live

01:18 streams over on YouTube. Subscribe to our YouTube channel over at talkpython.fm/youtube and

01:23 get notified about upcoming shows.

01:26 JJ, Carlos, welcome to Talk Python to Me. Great to have you two here.

01:29 Thanks. Great to be here.

01:30 Thank you. It's great.

01:31 It's really fun to have you both here. It's the week of Posit for me here at Talk Python,

01:36 just coincidentally having two groups from Posit on at basically almost the same time, right?

01:44 Last recording, last episode was Great Tables, which made me have a much deeper understanding

01:49 appreciation for what you can do at Tables. I thought, well, what is there more than kind

01:54 of an Excel look to it, you know? Formatting, maybe?

01:56 Yeah. That's what most people think, and then they see like four examples and like, oh,

02:00 yeah, that's right. I see.

02:02 Yeah, exactly. But Note Tables, well, there might be a table in the presentation today,

02:08 but that's not the topic. It's more about leveling up your notebook game, right?

02:13 Yeah, I think that's the most sort of straightforward win for Python users is sort of the ability to go

02:22 from a Jupyter notebook to a website, to an academic article, to a report with parametrization,

02:30 to an e-book, sort of growing from a notebook to the many different ways in which you want to share

02:34 it with people.

02:36 I mean, put maybe just a little bit more broadly, you know, we data scientists create all kinds of

02:42 very, very valuable things with their code. And this can be, as we said, you know, tables,

02:48 it can be visualizations, it can be the results of models. And then how do I project that out into the

02:55 world? How do I make it valuable for other people? And a lot of times that means turning it into some kind

03:00 of production output, you know, a PDF, a report, a dashboard, you know, a website. And so I'd say

03:07 that's, you know, in the large, what Quarto is about.

03:10 Yeah, I think it's a super cool tech. I've been hearing a lot of people recommend it lately, which

03:16 is always a good sign, always encouraging. Before we dive into that whole side of things, though,

03:21 let's hear a little bit about yourself. You know, who are you, quick introduction, and how'd you get to

03:26 working on Quarto and data science and all that? Sure. My name is JJ Allaire, and I have been a kind

03:33 of software tool builder for, I guess it's almost 30 years now, which dates me quite a bit. But I've

03:40 worked on lots of different... That catches up on you. It's really like, what? Yeah. And I've worked on,

03:46 you know, web servers and development tools and authoring tools and programming languages and lots

03:52 of different things. But about, I guess, coming up on 14, almost 15 years ago, 14, 15 years ago,

03:59 I decided that I wanted to focus on open source software. And I had early in my career, prior to

04:06 becoming a software engineer, I was very into quantitative analysis for economics and political

04:10 science. And I used a bunch of the tools that were available at the time. And so I, when I found out

04:15 about, this is actually even before, like, PyData was a thing, before Pandas was a thing, I was like,

04:21 oh, there's open source statistical computing. R, amazing. I can just work on that. So that's kind

04:27 of what I did. I set out to improve the, I'd say, the computing tools around R, and I worked on RStudio.

04:34 And then that has, I won't go into all the gory details of that. But really, that led into a lot of

04:40 these use cases, because a lot of the people using R were trying to figure out how to share and publish

04:45 their work, whether they be writing scientific articles or inside a company presenting things.

04:51 And so I worked on a thing called R Markdown, which was like, you know, merging R and Markdown

04:57 for publishing things. And then that more recently led to, wow, like, it's great that we did this thing

05:04 for R, but that's like 10% of the universe. So can we generalize this? And that was like the

05:11 Genesis of Quarto, like, let's generalize this idea and make it truly multi-engine. And it really is.

05:17 It works with R, it works with Jupyter, it works with, you know, the Julia, is it the Plato? No,

05:25 no, the Julia. There's a Julia engine. There's two Julia engines, actually, that it works with

05:30 observable. So the idea was like, let's take this core idea and make it work everywhere where there's

05:35 interesting computations happening. So anyway, that's a little money.

05:38 Yeah, amazing. And if you capture or have tools for the Python space and for the JavaScript space

05:46 and the R space, that's a good coverage of the data science world.

05:50 It seems to be, yeah. Yeah. That was our idea. Yeah. Awesome. Carlos, how about you?

05:54 Thanks. My name is Carlos Scheidegger. I'm an engineer here at Posit. I've been working in

06:00 on Quarto since I joined about two and a half years ago. My background is in computer science. I'm a

06:07 recovering academic, like I tell folks. I came to the US in 2005 for grad school. And I was working on a

06:16 completely different area. I was working in sort of like geometry processing, computer graphics.

06:20 But as part of my work in research, I started realizing that a lot of the experiments we were

06:25 publishing in papers had sort of a depth of like parameter settings and sort of choices and things

06:31 that like really dominated the output result. And it was really hard to go on a paper and see what was

06:36 happening there and sort of be able to figure out like what were the choices that someone made when

06:41 they published the work? How do I compare my work to theirs? And so I switched and I, as part of my

06:45 graduate work, I built a system in Python for sort of reproducible computational analyses and sort of,

06:52 how do you share these work and sort of, you know, as you're sharing like a PDF, you'd get like all of

06:56 the parameters and all of the things that came with them. So this was back in, you know, 2008, sort of

07:01 ancient history at this point. I then sort of joined AT&T research at that point and they have a really

07:08 strong R team there and sort of started doing some related work there. And that's sort of how I crossed

07:13 paths with JJ as they were founding our studio back a long time ago. And we more or less kept in touch.

07:20 Yeah.

07:20 So I eventually joined academia and kept interested in sort of this work on how do we make sure that the

07:26 experiments that we have are as easily shareable as the artifacts that we end up all reading, like the PDFs,

07:32 websites, but if you need to go back to figure out that what did actually happen there, we want to make

07:37 that as easy as possible. And so the idea of Quarto resonated really strongly with me. And so as I was

07:42 realizing that there's a lot of academia that I didn't, wasn't a great fit for me personally,

07:50 we can have a separate conversation about that. I reached out to JJ and he talked about Quarto and said,

07:56 this sounds like a great project and we really hit it off and I've been working on it since,

08:01 since we started. So that's how I ended up here.

08:04 That's cool.

08:04 That's right.

08:05 I think there's a lot of overlap on the data science side between our people, Python people,

08:10 and our tools and Python tools.

08:11 There's a lot more overlap than I think folks realize from the outside. And when you sort of notice

08:18 that like the people working on these tools end up sort of picking the tool that's best for the job

08:22 and Python is fantastic. Like I've been writing it for 20 years and sort of you find people trying

08:27 to find similar solutions to problems and sort of some things are more easily expressible or easily

08:33 to reach for in one language or another. But I think in 2024, the story is that more often than we realize

08:40 like teams and even people are like, they're polyglots, right? So they will speak like they will write

08:45 enough Python, they will write enough R, but they want something that sort of works together as well as you

08:50 can. And so that's something that we've focused on for two very, very explicitly. So you can have a

08:55 website in which, you know, half of your team is working in R and half of your team is working on

08:59 Jupyter notebooks, and they both can work on the environments that are most observable.

09:05 Yeah.

09:05 Or observable.

09:06 So directly in HTML and JavaScript.

09:09 So that's really part of what we're trying to do is sort of make it as natural as possible for people to keep

09:16 continuing their work, but then be able to publish it and sort of not forget like all of the computational

09:21 things that came with it and make it easily accessible.

09:23 I think also another demographic, another group in this is the people who are receiving the research,

09:30 receiving the stuff. They might not be either of those people. They don't want to see a bunch of code

09:35 and then a DF.head and then a plot. And then they want to report looking thing.

09:41 That's right. Yeah. And there's, we've definitely done a lot because it really does vary. I would say

09:46 the preponderance of people are in that category. I would say maybe 80, 85% of the people are in that

09:51 category, but then you do get the, the odd person who says, well, wait a minute, you know, where,

09:55 where did this come from? So we have a bunch of things, you know, like for publishing notebooks,

09:59 they'll let you, you know, interactively show the code or not show all the codes. Don't,

10:04 you know, show the source code. You know, we have a feature where like you see a plot in a,

10:08 you see a plot in a website and it just, it's just as a plot. There's nothing. And there's a

10:13 little text here at the bottom that says source notebook. You click it. Now, suddenly you're in

10:17 a notebook that you're in a lot. So ways to make it available, make the computations available without

10:22 hitting people over the head. That's cool. It's almost the equivalent of right-click view source

10:27 of the early web. Right, right, right. Exactly. But that's right. Yeah. That's very much.

10:31 But really to your point, I mean, that's why when I say production output, it's like the quality of

10:38 output that people are used to getting, you know, out of like office, or when people make

10:42 professional PDF reports, that's kind of what people are looking for at some level. Sometimes

10:47 people just want, Hey, just give me the information. I don't need it to be, but a lot of times they want

10:50 the production quality. And so we do spend quite a bit of time on making that possible.

10:54 Sure. And I would say the other thing that it's seems like this solves a bunch is the,

10:59 I have a notebook I created myself. How do I share it with the world? Yeah. Yeah. Right. A lot

11:04 of data scientists are like, there's interesting solutions out there, but there there's a lot of

11:08 uncertainty on what to do. You know, do I need to learn how to do Django or flask? Probably not.

11:14 Right. Right. Do you lean on something like streamlet where you kind of program to their model,

11:19 but it's cool and interactive, right? What do you do? You export to a PDF and just, or HTML, just here's

11:26 the notebook, but static, right? None of those are streamlets and pretty interesting option, but it's,

11:30 it's a very focused sort of tool, right? It's not a publication tool for sure. It's more interactive

11:36 tool. For sure. Yeah. And I think that one of the really interesting things that happens when you

11:41 sort of engage with the folks that are actually trying to produce this, like in, in a company or,

11:47 you know, in an academic department or like in a government agency, those, those stories you just

11:52 said, like, do I want a PDF? Do I want an HTML? They are all true, but for different stakeholders.

11:57 Yeah. Yeah. Right. So, you know, if I'm a data scientist in a company and I want the CTO to see

12:02 what I'm doing, the CTO is not going to have time to go pour over the sort of gigantic report that has

12:07 all the details. They want something that has up-to-date data that looks in a sort of expected,

12:12 like might have the company branding and so on. And ideally something that's like email to them,

12:16 like on a daily basis. Right. So we want the same notebook that you've used to sort of create your

12:23 analysis to go as easily from that into something that can be like a scheduled send with CI or with

12:29 one perhaps positive commercial offerings, or, you know, you can easily publish it as HTML,

12:34 through GitHub pages, through Netlify or through your own provider or...

12:38 Yeah. I see. So you get basically a static site that you can...

12:41 That's right. ... publish as you see fit, right?

12:43 Yeah. Yeah. Okay.

12:43 That's right. So at the very basic, right, Quarto sort of jumped the gun a little bit. Quarto is a

12:48 command line tool that you can use and you can take your notebook, right, as an input, like your

12:53 IPINV file and just say Quarto render, and then we will produce an HTML file. So you're sharing your

13:01 screen now. So if you go and get started on Quarto.org's webpage, which is where we are right now,

13:06 you sort of, you see there's a tutorial for computations on the left sidebar. And then there's

13:12 sort of a number of different pathways that you can go through there. And one of them is Jupyter,

13:15 which is actually the one that you're seeing there by default. And so in here, we're highlighting that

13:20 you can take like your Jupyter notebook or your Jupyter lab input. Once it's saved to a file, you can just use

13:25 Quarto directly to produce an HTML. So you can say Quarto render HTML, you can say Quarto preview,

13:31 and that will generate sort of like a local hosted website, but you can say Quarto publish.

13:36 And then we have publishing sort of targets to Hugging Face, to our own Quarto pub free service,

13:43 to GitHub pages, to a number of other places. So at a very high level, you can take your notebook as it

13:49 is, and it can produce a really nice looking HTML page that's available.

13:52 This episode is sponsored by us here at Talk Python. Of course, you can support Talk Python by checking out

13:59 our courses at talkpython.fm/courses and recommending them to your colleagues and team.

14:05 But this time around, I want to just encourage you to connect with us on a few other channels.

14:10 You're listening to the podcast, and that's really great. But did you know that we also have a blog?

14:15 It has a couple of really interesting articles I recently posted. Jump over to talkpython.fm/blog

14:22 and subscribe to the RSS feed there. And we also have a great mailing list where in addition to

14:28 announcements about Talk Python, I also share some tools and tips that I learned recently.

14:32 So if you're a fan of email, just visit talkpython.fm and click newsletter to sign up. That's it.

14:37 As always, thanks for listening.

14:39 And one of the really fun recent things that happened is Hamel Hussain, MB Dev guy, sort of

14:48 fantastic author. So he's working for Answers AI now, and they just published a service where you can

14:54 literally go on Answers AI and point to a GitHub repository that has an IPINB page, and they will

15:01 dynamically sort of on the backend, find that GitHub page, create the Quarto page from your notebook,

15:07 and then just publish it. Right? So like sort of zero click, if you have a public repository on

15:11 GitHub and you just want an HTML page to share with someone else, you literally do not have to do

15:16 anything. You can just go there, get the output. And so that's a really sort of like zero click way to do

15:22 that. Yeah. Go ahead. I was going to say it's called NBSanity. You can find it at NBSanity.com.

15:30 In the show notes also, I think there should be a link to Hamel's blog post announcing it.

15:36 And he goes into more details because one of the cool things is that you can actually use,

15:40 even if you're not running Quarto, you can put a bunch of the Quarto options in there. You can say,

15:44 oh, I want to show source code or not show source code. It makes source code expandable. Or,

15:49 and you, if you just put those in your notebook, when they render it, then they actually use those.

15:54 So it's a, it's sort of a way to use Quarto to publish your notebooks without even

15:57 installing Quarto. So it's, it's pretty cool.

16:00 Yeah. It basically, if you can just figure out the URL to put together, it's, it'll just pull it

16:06 together for you.

16:07 Yeah. And I think, I think that is like the concept of Quarto, which is okay. I have Python code or I have

16:14 a notebook and I want to project it. And I want to do that in a flexible way. And that's, you know,

16:19 that sort of web is like a default case. But I think the thing that people should also understand is that

16:24 we really do go quite a ways beyond that with like making PDFs, making Word documents, you know,

16:29 stitching things together in a website, you know, making presentations, you know? So it's,

16:35 there's a lot there. One of the curses of it is actually, it can be a little hard to explain.

16:40 It's like, what is Quarto? Oh, publish anywhere, anything, any computation, any language.

16:44 That doesn't really concretely connect with anything I'm thinking that I want to do, you know? But,

16:51 but if you dig into it, it does, it does, I think, connect pretty well to the things that people

16:56 want to do. But I think there's, there's interesting value to be gained from those kinds of decisions.

17:01 Michael, I know you had recently Charlie Marsh on the podcast and UV and rough and all these tools,

17:06 right? And sort of UV has taken a few of these moves that like, you know, make perfect sense,

17:11 which is as you start sort of like consolidating these tools, there's a lot of really interesting

17:15 use cases that arise naturally from the way you can compose them, right? So yes, we have render,

17:20 we have publish, we have preview, and it takes a little bit of time for you to explain. But in the

17:24 end, you get the system that sort of like grows with you. And like, we hope it will be the tool

17:29 that people reach for when they have sort of like data science, technical content production needs.

17:34 And so we hope to- Instead of trying to, like yourself, trying to piece together, say,

17:38 I'm going to try to convert my notebook to markdown and then use Pandoc on this and run this tool

17:44 against it. If you guys own that whole workflow, then there's new opportunities. Is that what you're saying?

17:48 That's right. That's correct. That's right. Yep. Yep.

17:50 Yep. So as an example, folks have, you know, we talked about Hamel and BDEV. Another web page that

17:58 folks might have known or sort of website and company is Fast AI, right? So Jeremy Howard's

18:02 Yeah. Sort of initiative. Fast AI is a Quarto website, sort of, you know, like it just,

18:08 yeah, so just go on Fast AI right now. If you open the source. Yeah. So fast.ai is the, yeah, got it.

18:17 Yeah. Right there. Yeah. So this is actually a Quarto website. So the blog here is, but it's all of

18:22 these apps. So if you see there, it's published with Quarto 1635 or something in there. And these are all

18:29 directly from notebooks. So the entries in here on the blog entries, they're all IPINB files. And then,

18:36 Quarto, among the other things we can do is sort of collate all of the different sort of blog

18:42 entries you have, make listings so you can filter them. So this is very useful for a blog. It's useful

18:46 for folks that have sort of like an ongoing set of reports that like need to be sorted chronologically.

18:51 If you have an academic website and you want to list your articles, that kind of stuff comes up all the

18:56 time. But the input for all of those are IPINB files, right? So Jeremy Howard: Yeah, that's right.

19:02 So this will work entirely in IPINB. And so from there, we generate the HTML. We sort of extract the

19:08 headers and create the listings all sort of automatically. So this is like an entirely

19:12 Quarto website.

19:14 Quarto, I guess it makes sense being a static website that it's pretty well suited for this.

19:19 A lot of these tools that I talk to people about, they're like, this is a great thing for internal

19:25 publishing of our site, or it's a great thing if you were going to interact with it, but it's kind of

19:30 heavyweight and it doesn't really support a lot of design. And it's not a, you would never build your

19:35 public website with this is like what I hear from a lot of these tools, right? But it's,

19:39 this is pretty excellent.

19:40 Jeremy Howard: That's you definitely can.

19:41 Yeah, we leaned, yeah, we leaned pretty hard into, into, you know, allowing a lot of CSS customization.

19:51 So we use Bootstrap and we have a whole theming system that's really based on Bootstrap themes,

19:56 but you can use SCSS directly to create your own themes. So, you know, it's, it's, you can just go

20:01 like, oh, make me a nice website. I don't want to think about it. People do that. But then people

20:04 who really want to like party on, I want to make this look just so can absolutely do it.

20:10 So yeah, again, we might, because I was going to say, we might miss a little bit of the details

20:15 for folks who are only listening, but another example that you might want to search just to

20:20 show sort of, you know, the range where you can go from that. It's also another quarter website.

20:24 So if you just search for real world data science, this is an initiative from the Royal

20:28 Statistical Society. They have a website, which yep, right. That one. So they have a number of

20:35 different writers that write sort of like columns and content pieces and ideas. This is also a quarter

20:40 website. So I believe this one is based on markdown. So they are using instead of IPA and BS inputs,

20:46 they use our markdown, but the same sort of thing. And the only reason I'm, I'm bringing this up is that

20:53 you can see how very different it looks, right? So it has categories and things like that. But if you want to

20:58 take sort of, you know, the time to do these and sort of have all the like the banner images and sort of like that

21:03 kind of thing, you can absolutely do it. And any result, you get to publish through a number of different

21:10 places that you might want to. Cool. Yeah. For people listening, it kind of has that landing page look with the

21:15 hero section with big pictures, right? You know, it doesn't feel very notebooky at all. That's for sure.

21:20 Another use case out in the chat is Kevin O'Malley says, "I teach a summer school on using Quarto for

21:27 academic publishing in the university I work in. It's a great tool, but we spend a lot of time thinking

21:31 about how to make documentation better. Interesting." Yeah. Documentation is a, I mean, part of,

21:36 we, I think Quarto's documentation is very good. Like I honestly say this without, I think it's, you know,

21:44 I don't want to say best in class because folks have actually realized how important documentation

21:49 is and there's a lot of really good documentation out there, but I think we're up there. I think our

21:53 documentation is very good. It doesn't mean that it couldn't be better. And I think part of the issue

21:59 with Quarto is that diffusion of entry points that we talked about earlier, right? Do you want to make a

22:05 single PDF? Do you want a website? Do you want dashboard? Yeah. And all of those become different

22:11 places. And are you coming from a Julia background? Are you coming from a Python background? Are you

22:14 coming from an R background? So we tried to create these like different entry points on the get started

22:19 webpage and our guides are customized for VS Code experience for, Jupyter experience and so on.

22:25 But we a hundred percent agree that documentation remains one of the hardest parts for us.

22:30 If you click on the guide, on the guide tab at the top, you'll see that, I know everyone

22:38 can't see this, but like, it's just such a vast number of things and it is easy to feel lost.

22:44 We try to organize it, but it's like, what does this not do? You know? So, so a little bit of a

22:50 choose your own adventure. It's a little bit of that, which is not always what people are looking for.

22:53 They might be looking for like, no, tell me, I want to get this to happen. Just give me step one through

22:58 four. And, I think we could probably stand to do a little more of that.

23:02 Yeah. But here, this trees, this is kind of a decision tree, you know, like,

23:05 what are you trying to do authoring? Are you trying to do diagrams? Okay. Are you using Python? Okay.

23:10 You go down this path and then we'll help you.

23:12 Yeah. Yeah. Yeah. Yeah. We, one, one thing I want to touch on just, I know a lot of people have

23:16 not experienced this. I know you two have, I have, and it's mind blowing. This is somewhat built upon

23:24 Pandoc, right? Yep. Yep. Yes. We can. And Pandoc is ridiculous, right?

23:29 It's ridiculous. Yeah. It is like the most, the biggest workhorse that no one, software that

23:34 no one knows about, you know, it is, it is an incredible piece of software. Yeah. Yeah. If you're

23:39 like, well, I have a, a LaTeX document and I want it as a media wiki markup format. Could I possibly,

23:48 yeah, you could just convert those? Yeah. If you, if people are willing to at the pandoc.org website,

23:52 yeah. On the right, there's a little gray. Yeah. I don't know what you would call it,

23:57 but it's a gray. It looks like if you drew the diagram for rock, paper, scissors, but 500 ways,

24:02 it shows all the connections. Exactly. All the connections.

24:05 All the connections. How do I get, yeah. The, basically the given one format,

24:10 what other formats can they go to? And literally it's illegible because it converts,

24:14 converts so many things to others. And that's a really interesting foundation for you guys, right?

24:18 Well, I want to talk about this a little bit because I think that the, the really big idea

24:22 behind pandoc is the, the, a consequence of the big idea is that you can do all this conversion.

24:27 But the big idea behind pandoc, if you, if you're familiar, like if in programming languages,

24:32 when you're parsing and writing interpreters, you have this abstract, abstract syntax tree,

24:37 right? Or IL, there's a sort of a abstract representation of the program. and then you can compute

24:43 on it. And I think what pandoc has done is they've created essentially an abstract representation of

24:48 what document. and so we think of documents as just like, oh, we just spray a bunch of content

24:53 into a file and it's just a big blob. pandoc to use documents as something you can compute on.

24:58 And so all the things that might be in the document, you know, tables and footnotes and figures and

25:03 bullet lists and, you know, all these things are in an abstract model. And so what, what happens is it

25:09 takes that, you know, whatever that is, word doc or markdown file, and it brings it into its abstract

25:14 model. And now you can actually compute on the document. And that's really actually the heart

25:18 of how Quarto is able to do, you know, Quarto is built on pandoc. And that's how we're able to do

25:22 almost everything is that we're actually not just dealing with text and markup. We're dealing with

25:27 this sort of abstract model of a document, and then we can do all these powerful things with it. So I

25:32 don't know if you want to expound on that. You probably could expound way too long on that,

25:35 Carlos, if you, if you chose to, but, but I think the shortest, important bit is yeah. So

25:43 Pandoc operates on this abstract syntax tree. It's a document that has paragraphs. Paragraphs can have

25:49 spans with, you know, strong text and their text and things like that. And we court, you can think of

25:55 Quarto as a very, very big orchestrator of Pandoc and sort of like configuration,

26:01 orchestrator or like choreographer for Pandoc. So Quarto itself is a command line application. We ship,

26:08 Pandoc with it. So like our bundles all include Pandoc with it, but fundamentally we are a TypeScript

26:14 application that, sort of puts itself in front of Pandoc and then after it, right? So, you know,

26:20 all of the complicated things you might want to do to generate multiple websites, to extract bits of the

26:26 documents, to know the titles, to create your blog posts and your entries, right? So Quarto gets in front of it,

26:31 does all of that orchestration work, then calls Pandoc a number of times and then calls sort of

26:36 some post processors. And the way this integrates with engines and Jupyter and so on is our, what we

26:43 call engines in Quarto are the things that turn, your document that has executable code with the

26:48 document that has the results of the execution, right? And so all that, that needs to happen for

26:54 an engine in Quarto to exist is that it takes Markdown or Jupyter notebooks as input, and then it produces

26:59 Jupyter notebooks as output or the Markdown annotated with those, results. And then we just sort of

27:06 process them and send to Pandoc, right? So really Pandoc is, is at the center of what we can do with

27:11 Quarto. And you can think of, Quarto as just sort of sitting around it and sort of expanding the scope

27:16 of the things you can do with Pandoc.

27:18 We've done a few more things. Like if you, to use Pandoc typically, you know, there's, you know,

27:23 160 command line options and you just kind of figure out how to, you know, it's tremendously powerful.

27:29 So we've tried to, I would say, organize that experience a little bit. so it's like, oh,

27:33 I just say I want a PDF and it's in YAML and I do a couple options and it does the right. So I think,

27:38 you know, you can think of Pandoc properly as like this sort of engine that you can do anything

27:43 with. And we try to give you like the happy path to a bunch of things that you probably want to do.

27:48 Yeah. I think that's hugely valuable because while Pandoc is great, it's also

27:52 super complicated. And a lot of times, if you want to combine different documents, you know, maybe I'm

27:58 actually working on a project that has a bunch of Markdown files, but they really need to be one

28:03 Markdown file on a certain order. And then that thing gets processed. But then as it gets transformed,

28:08 there needs to be changes to it. And right. You know, I'm, I can program, I can write that code to

28:12 do it, but there's a ton of common use cases, like take this notebook and publish it on the web.

28:17 Yeah. Right. It could just be built into your tool. Right.

28:20 So one of the thing that, I'm sorry, go ahead, you're going to go ahead.

28:23 I was going to say, even the thing you're talking about, I need to paste together a bunch of files

28:28 and turn it into a single file or turn it into a book or turn it into a, you know, those are things that we can,

28:32 we facilitate those sorts of things, things as well, you know, like this, this is really a book

28:37 actually. And it's got 20 chapters and each one is in its own file. And when it's a website,

28:43 I just want it to be like a website that lets you navigate my book. When it's a Word doc,

28:46 I want them all concatenated together into a Word doc, you know, et cetera.

28:50 Right. And then the final destinations and EPUB and a PDF or something.

28:53 Yeah. EPUB. Yeah. Or I went, yeah. EPUB or PDF. Sure. Yeah. Make the EPUB,

28:57 you know, create the EPUB archive just so, just so. And yeah. So sorry, Carlos.

29:02 I was just going to say that to Michael's use case of sort of having a number of markdown documents

29:08 and so on. I wouldn't presume to say you should use Quarto. Maybe it's a good tool for you. But one of the

29:14 things that we do believe in very strongly is sort of making sort of, it's a principle that we try to

29:18 abide by. It's sort of making hard things easy, but never at the expense of making very hard things

29:24 impossible. So, you know, we will provide you the happy path. We sort of, you know, the standard

29:29 YAML options, we have validation for them. We have completion, we have sort of integrated documentation

29:34 if you're in VS Code, for example. But if you need to actually extend things, we give you a number of

29:40 escape patches and entry points. And so if you're in a Quarto project, for example, we have a fairly

29:45 complete system of like pre-render scripts and post-render scripts, or we'll give you the set of things that we

29:51 found on your project. You can run TypeScript code against it. You can run Python code against it. Tell

29:55 us what you've done to pre-process our project or to post-process our project. So if you need to start by

30:00 collating markdown documents or by, you know, going to some database and pulling the documents that you

30:05 want to render on a webpage, all of those things are enabled by this pretty extensive extension system,

30:12 for lack of a better term. Sort of, we try to put extension entry points in as many places as we can,

30:17 so that folks are not stranded when they go from a simple project to inevitably something that's more

30:21 complicated. And so we try to make Quarto grow with them as your project tends to inevitably grow as well.

30:28 Yeah, that sounds really great. When I'm sure there's some pre-built extensions, but I also imagine

30:34 that we could write our own.

30:35 Is that in JavaScript? Is that in any language?

30:39 Okay.

30:40 Is that shelling out to...

30:40 Well, let me tell you. So a couple of the things are pretty open. So what Carlos said about pre-render,

30:52 post-render, sure, any language, that can be just the shell. Okay. There's a thing called Pandoc filters,

30:58 which is essentially take the AST in and transform the AST. Those also can be written in any language,

31:04 filters. So for example, there are two different libraries to write them in Python, but people write

31:09 them in all kinds of languages. That's available. But I would say the official mechanism for extending

31:15 that has more affordances, more APIs, more flexibility that we have is based on Lua, because Pandoc has an

31:21 embedded Lua interpreter in it. So as it's running, they give you access to the runtime and you can do

31:28 all sorts of really flexible things inside Pandoc as you're running. So Lua is, if people aren't familiar

31:36 with it, was originally created actually for embedded game engines. So the idea is we need very, very fast,

31:45 very fast execution, very high level language. It looks not dissimilar from Python, I would say,

31:54 but it runs very fast. So for example, like the Cloudflare kind of, what is it called? Envoy for

32:05 writing super high performance, like REST proxies uses Lua. So like, okay, this code is going to run

32:12 on every HTTP request that happens. We're not writing that in any language that isn't just

32:16 going to be screaming fast. So I think the reason why Pandoc used Lua was they said, well, we can very

32:22 easily embed it and it's going to be fast because when you're processing documents with a lot of nodes

32:27 in them, you really want to be using a language that's very fast at runtime. So it is Lua. And if

32:32 you go to our website, there's a whole section on extensions and we do quite a bit too. Let's see.

32:38 Yeah. Go to where's extensions, Carlos. It's got to be.

32:41 Yeah. If you just search for extensions, you should be able to find good documentation and sort of,

32:47 yeah, you can start from. Yeah. Yeah. Yeah. Yeah. That's right.

32:51 There's a bunch of ones that people, yeah, these are ones people are, have already done. So they've

32:55 sort of added features. Yeah. Oh, you guys support the concept of short codes.

32:59 Yes, we have.

33:00 Yes, we have short, you can write your own short codes. Yeah, exactly. Yeah. So you, you can write that,

33:05 the two kind of workhorse extensions are short codes and filters. And so short codes are like

33:10 content injectors, filters are content transformers. And so you can see examples of,

33:15 you know, different, different short codes and filters that people have.

33:19 Yeah. So an example for people who maybe are not familiar with this idea is pretty common in static

33:24 websites. Yeah. Static site generators. For example, Hugo has the concept of short codes as well. Yeah. And

33:30 it's like, I want to write a markdown, but I also want to embed a YouTube video.

33:35 Exactly.

33:35 Exactly.

33:35 Exactly.

33:35 And that is like an iframe and all sorts of things.

33:39 Exactly.

33:39 Yeah.

33:40 The video short code we support natively.

33:42 Yeah. So we have a video short code that's, that's built in. We wrote a video short code built

33:47 in because for this, it's so, it's so common, but you know, people make these specialized ones. One of

33:51 my favorite ones was somebody who was like a short code and you like give it some kind of,

33:54 I don't know what that was. It was like some kind of like chemistry or biology, you know, code. And

34:00 it creates this like interactive molecule visual visualizer, you know, right in there.

34:04 Yeah. Like a 3d, whatever the language for 3d stuff in the browser is that came out.

34:09 Yeah. And it just, it's wild. And you're just a short code and give it one little code and it just

34:13 does it. So, yeah. So these are, and these are, these are written in Lua, but, and part of that is

34:19 also because we want, again, this idea of, of being pan box supports Lua, but we also want

34:24 people to run these extensions with zero dependencies. So like, if I'm a, you know,

34:28 I write an extension, I don't know if you've got R or Python or like, I don't know what you're, what

34:33 you're, you know, and I don't want to have to satisfy a bunch of dependencies. So we want to have like

34:38 fast dependency free and well and agree with Pandoc. So that's why we ended up with Lua. And we, we like it.

34:43 and, it's served us well. We've done, we did some stuff in our quarter extension. We do a bunch

34:48 of like, we actually add a bunch of like static typing annotations for Lua so that it's like, we get

34:54 really nice auto-complete and linting and stuff. So. Yeah. Cool. You know, JJ, five years ago, if you said,

35:01 I need to write an extension in Lua. Yeah. I would have to learn Lua. Maybe not well, but I would have to

35:07 like, I could probably write that. I could probably write it in Python or, or R and then just tell chat

35:13 TPT. You definitely could. Yeah. Yes. Yep. So. What do you guys think about this? This crazy LLM stuff

35:20 that's taking over programming? I think, I think it's pretty interesting. I think it's, I would say

35:25 that I was kind of a little wary of it until I saw, I'd say probably Claude 3.5 and O.1 are a huge,

35:34 a huge step change in capability. and so I was kind of waiting for that. but seeing that,

35:40 I would say they're already now quite effective at accelerating many parts of development.

35:46 I think part of what we have to do is figure out the, the human, the sort of, what is the right,

35:51 I think for, for many, most types of software for the foreseeable future, it's going to be human in

35:58 the loop. I believe. And so what does the interface for human in the loop? What does it look like?

36:02 Is it, you know, it can go all the way from do a thing that's complicated. Let me see the diff and

36:07 I'll approve it too. I just want to talk about it and then I'll do it too. Let's talk about it and you

36:12 can do some things, you know, and I'll tell you if they look good or not. And I think that's a really

36:17 interesting area of exploration for, for the tool building community for the next few years.

36:21 It's, I agree. It's super interesting. I, I, I don't have it built stuff for me, but there's a lot

36:25 of times I'll be like, I could look this up, but I'd, I'd like to see an example from it and then I'll,

36:30 I'll take it and make it mine. Or I have a thing, explain it to me. Like, what does this regular

36:34 expression do? Or what does this curl command do again? I can't be bothered to study them.

36:38 I have a funny anecdote about this. Cause I, cause I do the same thing. Like, let me look this up.

36:43 So I was like, I was writing a VS Code, I was working on a VS Code extension and I was basically

36:47 trying to save some state, you know, per workspace state. And I'm like, I know there's like, there's

36:52 a one liner for this actually VS Code. They have this built in. So I said, I just want to do some

36:56 per workspace state. Like, how do I do that? And it goes, oh, great. And it goes like, I'll make

37:00 workspace state manager as it produces like 180 lines of code, you know, and it's got all this kind of

37:05 events. It's got all this stuff. And I'm like, isn't, I was like, isn't there like a one-liner?

37:09 Like, oh, totally. There is a one-liner like here. It's right here.

37:14 You do have to, you know, when you have it, try to build stuff for you, you have to kind of go,

37:18 don't lead the witness too much, you know? Yeah. Yeah. Yeah. For sure. I will just add that

37:26 I think people who try to just say, I'm going to take this thing and just have it build the thing

37:30 and then build the next thing. And if you take it too far, you're going to end up with a pile.

37:34 You're dead in the water. You cannot understand. You cannot maintain. It doesn't quite do the thing.

37:39 That's exactly right. Yeah. Yeah. Carlos.

37:41 Carlos, you spend a lot of time universities teaching people.

37:47 This is... Yeah. It's interesting. I left, I think, just before this became really,

37:54 really bad. My spouse actually teaches Intro to Python. And so, you know, she's sort of much closer

38:00 to the thick of the bad parts of like sort of exposing like this really access to LLMs for folks

38:05 who don't otherwise understand what's happening. And so I'm of two minds of this. One is I think there's

38:12 something fundamentally complicated for us to handle, us as in like society and humans, which is it is much

38:20 harder to tell that something might be wrong than to generate the output, right?

38:24 Yeah. Sort of it's really complicated, right? Like it's actually just genuinely hard to do so.

38:29 And it's even harder when you don't have the expertise. And so it's absolutely the case that

38:33 there's a number of folks that are using ChatGPT to generate code, to generate sort of solutions to

38:38 homework is like the bad case that happens all the time. Right? Yeah.

38:41 Yeah. But even if you sort of step that aside and say like there's no grade in play or anything

38:45 like that, the risk is that folks really don't have the ability to tell when something is going bad or

38:51 wrong. And I think that's a complicated thing to do. The counterpoint that I think is quite interesting

38:56 is that we have computers right in front of us, right? Like, you know, if ChatGPT gives me like a

39:01 snippet of Python code, one thing I can do with it is I can run it myself, right? Like, you know, I can go and

39:06 check it. Right? And so I think the places where I find it to be extremely valuable are like when it

39:11 tells me something that I can very easily verify whether or not it's the case, right? Sort of like

39:16 I say, okay, can I tangentially, I'm trying to teach myself Rust because it's 2024 and like, that's what

39:21 everyone else is doing. And I found it to be sort of exceedingly valuable to just sort of like, you know,

39:25 ask for something, how do you do that in Rust? And it gives me something and I just, you know, sort of

39:29 use the code and then like, I go look at what it does and sort of, but I'm actually the one running the

39:34 Rust code, right? Or something else is there's, there's a like third party that can verify the

39:38 output in like a meaningful sense. And that combination to me seems really powerful, sort of

39:44 like the ability to sort of, you know, generate these potential things with my ability to quickly

39:48 check whether they are true or not, that seems transformative to me. Totally agree. Yeah.

39:52 And so, so that's where I think it's most exciting work that's going to come out is how do we combine

39:58 like these things? And one fascinating thing to me is that if you think about like PyLens, Pyright or

40:04 Pydantic, these sort of like typing efforts for Python that exist in many other languages, something

40:09 like an LLM gets to benefit a lot from it on both sides, because the annotation is information that it

40:15 can use to sort of figure out what it actually needs to tell you. But you can also use the annotations

40:20 because the annotations means that there are fewer valid programs out there, right? That's what a type system

40:25 does. It disallows some programs to exist. And so if it gives you a wrong result, it's much more

40:29 obvious that that's the case. Right. You might see a type error.

40:32 That's exactly right. That's right. Yeah. And so that to me is where I think all of this stuff is

40:37 happening. If you, so you mentioned you're a mathematician by training, Michael. So Terence

40:43 Tao is a fields medalist, sort of very famous number theorist guy. He's been talking a lot about his

40:48 efforts to use lean, which is an automated theorem prover. So people are sort of trying to formalize

40:53 mathematics through like, you know, automated theorem prover programming language that has

40:57 like formal semantics and so on and using LLMs for sort of coming up with ideas for like the proof

41:03 strategies and so on. And there, if it doesn't, if the theorem is proven wrong, it will tell you

41:08 right away. Yeah. So that's the kind of thing that's very, very powerful to me.

41:12 Yeah. Yeah. I'll often, I'll often for something that's a little complicated, you know, I'll, I'll

41:21 have it generated. I'll look at it. Look, as Carla said, I'll look it over, I'll run it. And then I'll

41:25 say, I'll, I'll say like, now write me a ton of tests, you know, and then like, did you test for

41:30 this? Did you test for that? Did you? And then suddenly there's like 80 tests and that helps it's,

41:34 that's a little bit, it's not, that's not a guarantee, but it helps with that closed domain kind of thing.

41:40 Right. Not a proof checker, but, but you know, it's a proxy.

41:44 Yeah. I didn't mean necessarily take us this deep down the LLM path, but it's amazing. It's

41:47 super interesting. It's easy to go there. It's easy to go there. Yeah.

41:50 It's affecting all of us so much. And I think it's going to bifurcate the, the software developer.

41:56 Yeah. World on one hand, I think folks who are like, you guys are saying, I can check it. I'll take it.

42:01 Then I'll run with it. It's only going to make it better, faster, right? You want to do extension.

42:06 You don't know Lua. No problem. It doesn't matter. Yeah.

42:08 Right. We'll make that work fine. On the other hand, I think, you know, to your wife's position,

42:14 Carlos, I think there's a really serious danger that LLMs will stunt the growth of students.

42:20 Yeah. Yeah.

42:21 You already see it happening.

42:22 It's already happening.

42:22 Yeah. It's already happening.

42:23 This is already happening.

42:24 Yeah.

42:25 Yeah.

42:25 And it's, it's a really, I think it's, it's a shame because in computer science specifically,

42:31 the, the fix is easy, right? The computer is right in front of you. You can test that, right? Like that

42:37 kind of perspective that like, you know, you have a computer in front of you that can run code and tell

42:41 you what it's doing. There's nothing stopping you from doing that. That perspective is so important,

42:45 right? That's why open source is so amazing. You can like go into the Python code and like download it and

42:50 try different things. There's no one stopping you from doing that. So for computer science, I do

42:55 think that we have an easier way out where I think this is terrifying is where folks use LLMs as a

43:01 search engine with having no idea that this thing invents the fact that I don't know, like George W. Bush

43:08 has issued a part in the history of the app, right? That kind of stuff comes up, right? People publish stuff

43:12 on the news without minimizing this. It's already here and it's not just students, it's everyone.

43:17 Yeah. Yeah.

43:18 It's a slightly broader point. And then we can get off the LLM train here, but a slightly broader

43:22 point that typically to get the most out of these things in a lot of domains, and this could be law

43:27 or medicine programming, you really need like substantial human expertise to oversee them.

43:33 And that person probably has 10 years of experience. And so great. Okay. So those people are going to

43:40 become more and more valuable. I know how to be the human in the loop because I, you know, but how do we

43:45 develop the new, you know, the, the, the 10 years of experience? How do I ever get the 10 years of

43:49 experience if I'm constantly letting the LLM do things and I'm never learning? So that's the crisis.

43:55 I would say. It's an absolute, it's going to be a real problem in a few years. How do you,

43:59 how do you make the gap from new out of school to I have the 10 years experience? That is going to be

44:04 a huge problem. It is a huge problem. Yeah.

44:07 But do you know, William Gibson's quote about like, the future is already here. It's just not

44:10 even as distributed. That problem already exists. It doesn't need LLMs for that, right? This problem

44:15 of checking expertise, right? To tie back to the portal conversation, right? Like a number of there's

44:22 what got me started in the reproducibility crisis is this inability for us to tell whether some output

44:26 of an academic paper is correct or not. Right? Like it's really much harder to check. That is the same

44:31 problem that now absolutely everyone is going to run into on every place. Right? And where I think

44:36 Porto has something to say about it and like this entire project of computational reproducibility and

44:41 documents that have the code that generated them is that you have the ability to check, right? So, you

44:46 know, our manuscript support that JJ alluded to early, right? When you create a website, right? You have

44:53 the entire website and there's a link to the side that's like, here's the code that generated it. You can

44:56 run it on binder, right? You can run it in code spaces. Yeah. And so you can check when you look

45:01 right here. That's in the upper right. Yeah. This thing right here. That's right. Yeah. So that

45:04 shows the code statically, right? So there's a separate mode in which you not only have like the

45:09 source code that you can see it, but you actually have the full IPNB like available for you to run it

45:14 on binder and like check for yourself that it's actually producing the results that you expect.

45:18 Right? So you can go from the pros that is hard to tell whether it's true to the code that actually runs.

45:24 So you can at least see what the person is using to make the claims that they did. And I think that

45:29 kind of like integrated metadata that like sort of corroborates the statements you're making is only

45:35 going to become more important as we go forward rather than less. Yeah. Yeah. I agree. That harkens

45:40 back to your reproducible science initiatives, right? That's exactly right. Yeah.

45:45 So let's talk about using Quarto. Where can I run it? So it's, it looks like it's supported at least on

45:51 the major OS's Mac, Windows, Linux. Yeah. Right. That's right. It's a, it's a command line tool

45:57 that you can install separately. You can also just pip install it. oh, interesting. Yeah. Yeah.

46:02 I mean, just that's a lot of people do that. and you can pip install older versions and,

46:06 you know, versions from GitHub and all that sort of thing. in terms of, I would say, you can

46:12 just use it with any editor with a command line tool. I do think it benefits from, from some tooling,

46:18 like some auto complete and some amount of just like, Hey, preview this thing, show it to me.

46:23 and so there is, there is pretty good tooling across there. So there's like a Jupyter extension.

46:28 There's a VS Code extension that does quite a bit of stuff. there's a Neo Vim extension.

46:34 there's a support integrated support in our studio. So, that's important. If you go to our, like,

46:39 if you go to our, like, get started thing, you can see, we sort of, let's see,

46:45 I think you have to scroll up. you go to get started. It'll kind of, if you scroll down,

46:51 we kind of say to you like, okay, what are, what, you know, what's your preferred tooling

46:56 environment? And then we kind of set you up in that environment with the right

46:59 extensions or other things. And so there's always use a text editor, but you know, I think the tooling

47:05 is helpful. Yeah. So one of the things I was thinking about when I was asking that,

47:08 two of the things really that I think are rough edges that you probably solve that people don't

47:13 necessarily know they're rough edges yet till they try it is one hand doc has a lot of dependencies

47:19 dependencies that are not obviously dependencies. Yeah. Like for example, I was trying to get,

47:25 some file out of pandoc and it was saying this LaTeX thing is not installed on your Mac.

47:29 Yeah.

47:30 I'm like, why is this here? Why is this happening?

47:32 That's right. We definitely, yes, yes, yes. We, we have, we have definitely solved that. So on,

47:37 so it depends on the, now it depends on the scenario. So for LaTeX, we don't embed a LaTeX distribution. However, if we notice you don't have LaTeX, we tell you just

47:46 type quarto install tool, you know, it's a, what is our LaTeX distribution?

47:52 Tiny tech.

47:53 Tiny tech. Install tiny tech. And then that's actually a tech distribution that we maintain

47:58 that is like very small, has different form factors and isn't like five gigabytes to start

48:03 with and does auto installation of packages. So for that, we kind of say, yeah, you need,

48:07 you need LaTeX. We're going to make it super easy. So it's a one liner. You can get LaTeX.

48:11 In other cases, like, types is another, many people may or may not have heard of. It's a,

48:16 it's sort of a competitor to LaTeX. It's another PDF, among other things, PDF engines.

48:22 and it's re it's a phenomenal tool. It's, you know, creates incredible output. It has a much to me,

48:30 like more sort of natural and scalable, programmable interface than LaTeX does. We just embed

48:36 the, the, the types runtime in Quarto. Cause it's not very, it's not, there's not much to it.

48:42 You know, it's not, it's not huge. It's not, it's not like that's cool.

48:45 If you haven't used types and you use PDFs, you generate PDFs as part of your day job. and you've

48:53 heard of LaTeX and you have the reaction that every single person that needed to use LaTeX is,

48:58 it's like fondness for the pain it caused and the quality of the documents it generates.

49:03 Types-

49:04 It generates incredible, it doesn't get better.

49:06 Types gets to what I would say is 90% of the quality that LaTeX does. There are some

49:11 micro typography tricks that they haven't quite incorporated yet, but they do a number of other

49:16 things. And it, so Types is genuinely amazing. I could not recommend it more strongly. It's a little

49:23 weird to type as Typst.app. It's another open source product. They have an offering that is sort of like

49:31 a collaborative editor where you do those things, but you can use it as a command line. T-Y-P-S-T.

49:36 Types-t. Types without an I. Types-t. That's right. Yeah. Okay. Yeah, got it. All right. Wow.

49:44 Yeah, we can add that to the show notes. It's hard to, it's hard to phonetically tell me how

49:49 to type it and search it. But it's amazing. Once you know, it's going to hit, this is not going to

49:53 come up. It's very searchable. Okay. I'll put the show notes. It's a fantastic, fantastic tool.

49:57 Yeah. So yeah, Types is incredible. Yeah.

49:58 Pandoc supports outputting from Markdown to Types because of course they do. And so we can do that as

50:04 well. And it's phenomenal. It's incredibly, like I hope in five years we will not have to reach for LaTeX.

50:13 It's early, right? It's sort of existed for a couple of years, but I am genuinely, the team is extremely

50:19 impressive. It will not surprise anyone if I say it's a Rust application because again, of course it is.

50:26 But amazingly technical work. Compilation is impossibly fast. To me, the difference between

50:34 using LaTeX and Types is the difference between using Mercurio and Git. If you ever remember the

50:39 first experience when you said Git something and you just got the result back and you thought

50:44 something went wrong, it could not possibly be this fast. That's the same impression you have from Types.

50:49 It generates this amazing PDF in like 0.2 seconds. And you just look at your computer and like,

50:55 I cannot believe this just happened. It's really nice. So we support that.

50:59 We've made a big investment in Types and so we bundle the runtime and we're bullish on it.

51:06 All right. I'm going to have to learn more about this. This is very exciting.

51:09 So yeah. Okay. So that was the one thing I want to talk about was the rough edges of tools like

51:15 Pandoc and others. The other I want to talk about with you all is continuous integration. So you could

51:22 set up say an automation in GitHub where somebody pushes a new version of a notebook, it automatically.

51:28 Yeah. That's right.

51:29 Yeah. So yeah, we have the integration. We got GitHub actions integration. What's the story?

51:34 Yeah. So if you go to, if you go in the guide here, just, I know folks listening won't see this,

51:38 but we'll just talk through, go to publishing. You can see, we sort of have this idea of, yeah,

51:44 I guess you can go to publishing basics there and it'll give you an overview of everything.

51:48 So we basically have this idea. We have a Quarto publish command that can kind of publish to lots

51:54 of different destinations. So it can publish to Netlify, GitHub pages, you know, some posit things.

52:01 We have our own little hosting service, hugging face spaces. So the idea is we know how to talk to all

52:06 those. And so you can just say Quarto publish and it'll remember, oh, you published to this URL before you

52:10 want to update it. So that's like the workhorse for publishing. And then all of that works in,

52:16 you can see at the bottom, there's a note publishing with CI and all of this can work

52:20 by basically like put your token in CI, you know, you know, export the token in the environment,

52:26 call publish. And so lots and lots of people use CI to update things with Quarto. And they may be,

52:34 again, they may be publishing to get a pages, maybe Netlify, maybe hugging face, maybe, maybe another

52:38 service. Yeah. You want to give a shout out to Quarto pub? Yes. Yeah. The Quarto pub is just a free

52:44 service. I don't really know much about it. What is this?

52:45 It's literally just a free service that you can use to publish static, static documents and

52:51 websites. And that really all we wanted to do here, it doesn't support all the fancy features

52:56 like authentication and custom domains and all that our goal. And there are other services that do that.

53:01 but our goal was to have a free service that makes it dead simple to publish your thing.

53:06 You get your own, you know, your own like sub domain and you can just publish stuff. So we

53:10 wanted to be just like, I can definitely publish my thing to the web. and so that's kind of what,

53:14 what, what it's for. and then if you want to do things like, for example, if you're using

53:18 get a pages, well now I can, you know, I can like have it authenticate. So only people who have access

53:22 to the repo have access to the pages, or, you know, if I'm using Netlify, I can do all manner of

53:27 fancy things. So it's, it's designed to be simple, easy, and free, to kind of get your stuff out to

53:32 the web. So I use that. Like when I'm, if I publish, I go give a presentation and I have a slide deck and

53:36 it's written in Porto. I just put it on Porto, you know? Yeah. Beautiful. So I suppose one of the

53:41 other publishing destinations other is just a pile of HTML, CSS. That's right. Exactly. Yep. Put it

53:48 wherever. Yeah. It's just going to make Porto is just going to make a directory for a website. It'll

53:52 be underscore site, you know, for a PDF, it'll be a PDF and whatever.

53:56 Put down some engine X or caddy and off. Exactly. Do whatever. Yeah, that's right. That's right.

54:01 One thing I will notice a note about the CI just before we switch subjects is, we have a

54:06 repository that I think we linked to, somewhere where we have a number of examples of using GitHub

54:11 actions. So it's a GitHub/Quartodev/Quarto actions that we maintain. And we document

54:17 with a number of different use cases that folks might want to do if they need to like grab dependencies,

54:22 right? So your CI often where the sharp edges are that you're going to have to install R, you're going

54:27 to have to install Python and so on. And so we have sort of, you know, a number of actions that will

54:31 talk Quarto will render it and it will publish it. So from there, you can just take it where it is,

54:37 drop some of these into your own repository, and then you're off to the races. So this is another

54:41 place where we're trying to like reduce the sharp edges to like actually get people to see the stuff

54:46 you've built. Yeah. Yeah. Cool. I'll link to it in the show notes so people can find it and use it.

54:50 All right. we have time maybe we, if we lightening rounded a little bit, we got time for,

54:56 for two, we can do it. We can do it here. So, one, this is open source under what license?

55:04 MIT, which may basically means that commercially you can do whatever. So

55:10 Quarto is sort of unencumbered in that sense. That's right. Right. Yeah. That's right. Exactly

55:15 right. Yeah.

55:15 What's the business model? How does this tie back to Posit and why is this?

55:19 So Posit has a, a pro so product Posit has a product called Posit Connect that is essentially

55:25 like an, I would call it an enterprise publishing platform for what we call data products. And so

55:31 Posit Connect does a lot of things. It publishes shiny apps. It does stream load apps. It does dash

55:36 apps. It does flask apps. It does apps. And it also does, content. So it does websites or documents.

55:43 And then it has, it adds some things like, oh, you can schedule, the content. So like, you know,

55:48 run this, update this once a day, update this website once a day, or whenever this thing,

55:53 updates, you know, send an email to these people. So it's, that's the idea. so Posit Connect is,

55:58 is, a commercial product that we sell. And so I think, from a business model standpoint,

56:03 if people are successful with Quarto, as obviously we make it very easy to publish it to

56:08 everywhere. We're not trying to privilege, you know, or say, oh, oh, you know, it's a Roach Motel.

56:14 You have to publish with Posit Connect, but we're, we're, you know, we do, we do try to have,

56:18 yeah, yeah, we do. We do try to have a value added product. It focuses, I'd say, honestly,

56:23 more on the like internal publishing, you know, than the, than the public, publishing.

56:29 Yeah. That's cool. And just for, for, for disclosure, people haven't listened to other episodes,

56:32 you guys have sponsored the show before for Posit Connect. And I want to say, thank you.

56:36 Okay. Okay. Yeah. Awesome. I really appreciate it. But, but this is not sponsored, right? This is just,

56:40 no project. So, okay. So that was, yeah, that was one of the ones I wanted to talk about. Just like,

56:48 how open is it? It feels a little bit like the Visual Studio code model. Microsoft makes this.

56:53 It would be awesome if there were some great integrations to Azure and encourage people.

56:57 Oh, right, right, right, right. Yeah. I mean, right. People love it for whatever,

57:02 but if it's, if there's the right use case, then, Hey, it's a perfect tool.

57:05 Yeah. So a little bit of that, we, we've sort of like, Hey, we've tooled this up for internal,

57:10 you know, enterprise usage, as much as we can. we, you know, I'd say we've had,

57:15 we've had lots of contributions. I think our biggest external contribution was,

57:20 would you say Carlos, the folks from Julia added the, Julia engine, which is a pretty,

57:25 pretty, pretty substantial piece of work. and so we, we very much like to collaborate and take

57:30 pull requests and yeah.

57:32 The yours are welcome, huh?

57:33 For sure.

57:34 The yours are very, are very welcome. Yeah.

57:36 And currently Quarto is, I think, you know, a, a large code base that is not super approachable.

57:42 I think that's a fair assessment that we would agree with. We are spending a lot of time in 2025,

57:47 actually figuring out ways to sort of let the community in and sort of change the ways that

57:51 people can write extensions and sort of make it easier for people to build on it. I will say that

57:55 in contrast to the way that Microsoft does like VS Code versus Codeium, right? So we don't have to split

58:00 products, right? So we have just Quarto, but it is true that inside Quarto, there are sort of things that

58:06 only make sense, for, so there's a published to like RStudio connect, for example, right? So

58:11 there are things that integrates with RStudio that are inside the Quarto code base that are there

58:16 because of RStudio. but that's an entirely fair trade-off that people can make, right? They don't

58:21 have to use those tools if they're, that's right.

58:23 That's right.

58:23 Not at all. Not at all.

58:24 Yeah.

58:24 Yeah.

58:25 But you want to serve your customers as best you can. That's right.

58:28 Yeah.

58:28 Yeah.

58:28 Okay. I don't know who wants to take this one, but people know node.js.

58:33 And one of the great ironies of a lot of these data science tools is a lot of the people working

58:37 on them, for example, Jupyter themselves, it's kind of like, we'll take one for the team and we'll

58:42 write all the JavaScript to make it interactive. So you all can stick in your data science language

58:46 and not worry about it.

58:47 Yeah.

58:47 That's right.

58:47 It sounds a little bit like that here. And so you guys are using Deno, Deno Deno.

58:52 Yeah.

58:52 Yeah.

58:52 How, one, how do you say that to, to like what it's, it's like an alternative to node.js,

58:58 right?

58:58 That's right.

58:58 What's the deal with this internal story?

59:00 Yeah. I don't know if it's pronounced Deno. I just usually pronounce Deno. But it is a,

59:06 I think Deno is right, right?

59:06 Yeah. So it is a, so it has V8 inside it, just like node does. It's actually founded by the same,

59:13 by Ryan Dahl, who is the founder of node and he's now CEO and founder of Deno, the company behind it.

59:18 So the main thing with Deno is sort of, it really wants to be TypeScript first. So you can run

59:25 JavaScript, but Deno has sort of like a really built-in type checker that uses sort of like,

59:30 you know, the TypeScript type checker, but it's, it's really quite built in and it has a really

59:35 advanced sort of capability system for folks that need to bundle things. You can run Deno and say,

59:41 run Deno, but do not allow it to go on the net. Right? So run Deno, but do not allow

59:45 - Oh interesting. - allow to like import other things. Run Deno, but allow only access to the

59:52 file system or do not allow access to the file system or do not allow. So they have a really

59:56 fine grain capability system that is quite nice. And for us, so that was, so one of the things that we

01:00:03 wanted was something that sort of let us just do that and quick and sort of, it's easily also

01:00:09 embeddable. So Deno is like a single binary that you get to ship and you get V8 with it. And so

01:00:14 we just bundle Deno internally. And so it's just a pleasant like modern alternative to,

01:00:20 to node where most of the standard library and all of those are TypeScript first. And so you have the

01:00:27 really nice type annotations. The standard library is all in TypeScript and you get access to that. And so

01:00:32 that's what drew us to it. - This is cool.

01:00:34 Yeah. I didn't know, I mean, I'd heard of it, but I hadn't really paid much attention.

01:00:39 Wow. It's popular a hundred thousand GitHub stars and it's less than two years old or something like that.

01:00:43 It's not bad. - Yeah.

01:00:44 Yeah. It's quite popular. They have a number of interesting things. So the company is built behind

01:00:49 and trying to use this Deno deploy, which is sort of like this, you know, serverless where you ship

01:00:55 - Edge computing. - TypeScript, edge computing. And so, you know, you can have like a very small piece

01:00:58 of TypeScript instead of having node that runs on the edge. And so that kind of, that's their business

01:01:02 model for us. It's just a modern alternative to node with all the lessons that they are.

01:01:08 Awesome. Right time.

01:01:08 Yeah.

01:01:09 Yeah.

01:01:09 That's cool.

01:01:10 All right, guys, I think we're out of time here as much as I want to talk more about it.

01:01:14 Absolutely.

01:01:14 But let's leave it with a little bit of a call to action. People are interested with Quarto.

01:01:20 Yep.

01:01:20 They want to get started. What do you all tell them?

01:01:22 I would go, I would go to the Quarto.org website and I would literally just, there's a get started

01:01:26 button and it basically will let you pick your tool, whatever VS Code or Jupyter, or just

01:01:31 NeoVim. And then it'll take you through a tutorial that's step-by-step that shows you the basic

01:01:37 mechanics of the tool. And then it shows you how to, how it works with computations, how you can embed

01:01:43 code and run the code and control how the code is run and all that. So I think if you go through that

01:01:48 tutorial, it'll probably take you, you know, a half an hour or something like that. You will have a very

01:01:53 strong, you'll be up and running and you'll have a, you'll be in your favorite tool and you'll have a very

01:01:57 strong idea of what the thing can do and how it's useful. And then from there, I'd kind of just go

01:02:01 back to the homepage and sort of browse through and say, okay, what can I do with this? So I can make

01:02:05 dashboards, I can make presentations, I can make reports, I can make websites. And I think it'll be

01:02:10 easier to connect concretely with those things once you've gone through the tutorial.

01:02:14 The other thing I want to point out is we have a pretty active discussion forum on GitHub. We don't

01:02:20 have a discord. We'll probably fix that in 2025. But we do keep a very close eye on both issues.

01:02:27 And on discussions and sort of, you know, we are, we try very hard to be responsive to everyone and

01:02:32 sort of, you know, if people want to share their work or ask questions, sort of take, you know, the

01:02:38 the quarter sort of documentation and then figure out like, you know, okay, what are the things? How

01:02:41 do I put them together? And they have questions. We are like, we keep a really, really close eye on that.

01:02:45 And we try to be very responsive. So I would encourage people to just hop on, on GitHub discussion,

01:02:50 ask questions, and we'll be there to help you.

01:02:52 Yeah, excellent. It's nice to have discussions because a lot of times people will come and

01:02:57 if there's no discussions, they'll file an issue, which is really a question. And you're like,

01:03:00 well, when do I close this? Because I kind of answered it, but you don't necessarily know you

01:03:05 answered it and they won't close it because it's there anyway.

01:03:07 That's right.

01:03:08 Yeah. So awesome. Well, congratulations, both of you on such a cool project. And I know it's

01:03:13 doing good things for a lot of people.

01:03:15 So I want to quickly shout out just the folks. And so, you know, there's a large number of group

01:03:20 folks like inside Posit that both help, like, you know, sort of besides JJ and Charles who contributed

01:03:25 early, Christoph Dervie, Gordon Woodhull, Charlotte Wickham, Mene, Mikael is a huge help on our issues

01:03:31 as well. So it takes, it takes a village and I just want to shout out those folks too.

01:03:36 Yeah.

01:03:36 Yes.

01:03:37 Cool. That's excellent. All right. Bye guys. Thanks for being here.

01:03:40 Thank you. Take care.

01:03:41 Thank you.

01:03:41 This has been another episode of Talk Python to Me. Thank you to our sponsors. Be sure to check

01:03:48 out what they're offering. It really helps support the show. Want to level up your Python? We have

01:03:52 one of the largest catalogs of Python video courses over at Talk Python. Our content ranges from true

01:03:57 beginners to deeply advanced topics like memory and async. And best of all, there's not a subscription

01:04:03 in sight. Check it out for yourself at training.talkpython.fm. Be sure to subscribe to the show,

01:04:08 open your favorite podcast app and search for Python. We should be right at the top. You can

01:04:13 also find the iTunes feed at /itunes, the Google Play feed at /play and the direct RSS feed at

01:04:19 /rss on talkpython.fm. We're live streaming most of our recordings these days. If you want to be

01:04:25 part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel

01:04:30 at talkpython.fm/youtube. This is your host, Michael Kennedy. Thanks so much for listening. I really

01:04:36 appreciate it. Now get out there and write some Python code.