#354: Sphinx, MyST, and Python Docs in 2022 Transcript
00:00 When you think about the power of Python, the clean language or powerful standard library may come to mind.
00:04 You might certainly point to the external packages as well.
00:07 But what about the relative ease of picking up new libraries or even parts of the standard library?
00:12 Documentation plays an important role there.
00:15 And the tools in the Python space for building solid documentation and even publishing articles and books involving live code are huge assets.
00:24 In this episode, we have Paul Everett, Radian Gedem, Chris Holdgraff, and Chris Saywell to update us on Sphinx, Mist Parser, Executable Books, Jupyter Book, Sphinx Themes, and much more.
00:35 This is Sock Python under me, episode 354, recorded January 19th, 2022.
00:40 Welcome to Talk Python under me, a weekly podcast on Python.
00:57 This is your host, Michael Kennedy.
00:59 Follow me on Twitter where I'm @mkennedy and keep up with the show and listen to past episodes at talkpython.fm.
01:04 And follow the show on Twitter via at Talk Python.
01:08 We've started streaming most of our episodes live on YouTube.
01:12 Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.
01:19 This episode is brought to you by SignalWire and Tonic.ai.
01:23 Please check out what they're offering during their segments.
01:26 It really helps support the show.
01:27 Transcripts for this and all of our episodes are brought to you by Assembly.ai.
01:32 Do you need a great automatic speech-to-text API?
01:34 Get human-level accuracy in just a few lines of code.
01:37 Visit talkpython.fm/Assembly.ai.
01:40 Pradyan, Paul, Chris H., and Chris S., welcome to Talk Python to Me.
01:46 It's great to have you all here.
01:47 Good to be here.
01:47 Thanks for doing this, Michael.
01:48 Nice to be here.
01:49 Thank you.
01:50 Yeah, absolutely.
01:50 Happy to be doing it.
01:52 I think talking about documentation and all the static site stuff and book generation stuff that we're going to talk about is going to be super fun.
02:01 You all are at the center of various fulcrums of that throughout the ecosystem.
02:07 So it's great to have you here.
02:09 Great to be talking about that stuff.
02:10 So let's just start really quick with a quick introduction and then what you're up to these days.
02:16 Paul, you're a regular here, so you want to go first?
02:19 Sure.
02:20 I'm Paul Everett.
02:21 I'm a JetBrains developer advocate.
02:22 Big fan of the show.
02:24 Big fan.
02:25 Total fanboy of the other three people here.
02:29 I've got all kinds of stories about where I was when I found out about Myst and the parser and desperately wishing I could join Pradyen and all the things that he's doing.
02:40 And I will add big, big fan of Carol Willing.
02:44 And when I grew up, I want to be Carol Willing.
02:46 And you're still at JetBrains?
02:49 Still at JetBrains.
02:50 Right on.
02:50 Keeping the PyCharm flowing.
02:52 Right on.
02:53 Pradyen, how about you?
02:54 I work at Bloomberg, although not what I'm going to talk about today is related to work.
02:59 Software engineer there.
03:00 Work on the bathroom infrastructure team.
03:02 I am a maintainer on PIP.
03:05 I have written a Sphinx team for two, depending on how you count the second one.
03:10 And I'm involved in a bunch of sort of various efforts around Sphinx at this point.
03:17 I've sort of made myself comfortable in these spaces and sort of made my way into the various discussion forums, I guess.
03:25 Yeah.
03:25 Fantastic.
03:26 With me, I guess, in this topic.
03:28 Awesome.
03:29 Well, welcome.
03:29 You seem to be the emergency maintainer of many open source projects, like Live Reload.
03:34 Yeah.
03:36 I have a mild issue of not knowing when to stop.
03:40 That's good.
03:41 All right.
03:41 Chris H.
03:42 Yeah.
03:42 So my name is Chris Holgraf.
03:44 I'm the director of the International Interactive Computing Collaboration, or 2i2c, which is a nonprofit that runs cloud infrastructure for interactive computing and the Jupyter ecosystem and the sort of surrounding ecosystem alongside of it.
03:57 I'm a little bit of a lot of research and the Jupyter ecosystem.
04:02 I'm a little bit of a lot of the Jupyter ecosystem.
04:03 I've worked a lot on Jupyter Hub, which serves multiple Jupyter sessions via some kind of centralized cloud infrastructure.
04:08 I also work a lot on the Binder project, which focuses on more scientific reproducibility and shareable computing environments in a kind of cloud agnostic and pretty flexible manner.
04:20 Yeah, that's fantastic.
04:20 For people who don't know about Binder, that's basically if you are at a GitHub repo or some published notebook that's not interactive, you can click that and it'll fire up a little environment where you can go and explore the notebook for real, right?
04:33 Yeah, exactly.
04:33 And so a lot of these things are kind of surrounding this general topic of like scientific communication, scientific reproducibility, bringing those kind of like data workflows and facilitating them with software development, which is related to a lot of the stuff we're talking about today as well.
04:48 Yeah, fantastic.
04:48 You also have worked on the MIST project, right?
04:51 Yes, yeah.
04:51 And one of the projects that I'm focusing on right now, along with Chris S and a few other collaborators, is called the Executable Books Project.
04:58 And this is basically an attempt at improving the state of open source kind of community-driven tools in the Python ecosystem around scientific communication and building a lot on top of the Sphinx ecosystem, because there's a lot of good material there to work with and a lot of improvements that can be made that'll benefit the broader Python community as well.
05:18 Yeah, yeah, very cool.
05:19 Looking forward to talking about Jupyter Book as well.
05:21 That'll be fun.
05:22 Chris Siewel, you're back.
05:24 Tell us about yourself.
05:25 Yeah, so I work in the Executable Books.
05:28 I guess essentially my kind of focus is on making the best tools for scientists to make open reproducible science.
05:37 So I work 50% of the time here in Switzerland at EPFL on a package called ADA, which is a Python workflow engine for running simulations, orchestrating simulations.
05:50 And then I work the other 50% of the time in executable books on Mist and Jupyter Book.
05:55 Yeah, that's really fantastic.
05:56 Is any of your work in Switzerland, does it have to do with CERN or other projects?
06:01 No, not to do with CERN.
06:02 It's an EPFL in a group on materials discovery.
06:07 Yeah, fantastic.
06:08 Let's start by talking about Sphinx.
06:12 So when I think about Sphinx, I think about Python documentation.
06:17 So who wants to sort of set the stage for Sphinx?
06:21 I mean, I guess it does say the Python documentation generator, but it does more than that as well if you wanted to.
06:27 So guys, what is Sphinx?
06:28 When do I use this?
06:30 What's its value?
06:30 It's been around for a long time.
06:31 I know that much.
06:32 Let's do a round table.
06:33 I'd love to hear everyone's elevator pitch.
06:36 Yeah, exactly.
06:37 What's the elevator pitch for Sphinx?
06:38 I can start with sort of how I got into Sphinx and then maybe how I use Sphinx now, which is a little bit different from how I initially used it.
06:45 I think that a lot of people, so Sphinx has been around for a really long time and it's a really powerful tool for documentation.
06:50 And a lot of people use it alongside of their software projects that they work on.
06:55 So my first introduction to Sphinx was via, it was actually a neuroscience analytics package called MNE Python.
07:01 And we needed documentation to describe, you know, the APIs and the functions and the classes and things like use cases and examples.
07:08 And so one of the really useful things about Sphinx is that it has an inherent extensibility and a lot of flexibility so that you can both generate like narrative documentation with it, but also include programmatically generated documentation, API documentation, and that kind of thing.
07:23 So it lets you kind of like more seamlessly merge together code documentation or documentation embedded within code and your more kind of traditional narrative examples, tutorials, that kind of a thing.
07:35 I see. Almost like a wiki plus an API documentation generator in one.
07:41 Yeah. Plus that's been around for a long time and with an inherent extensibility.
07:45 So a lot of different sub communities have kind of built out their own community specific documentation that builds on top of the kind of basic Sphinx building blocks.
07:54 Sure. I bet the scientific community has got a lot of specializations.
07:58 They're like, we need to be able to express stuff like this, right?
08:00 Well, so that kind of gets me to the second part of what I was going to describe, which is I think over time, and this is one of the inspirations behind Jupyter book to some degree, is the realization that like technical and API documentation is also really useful.
08:16 Well, I should say the things you need to build for really good technical and API documentation are really useful for other kinds of use cases as well.
08:23 And so I saw some other groups in the scientific ecosystem.
08:27 There's a really interesting one called Simpeg, which I think is just S-I-M-P-E-G dot X-Y-Z.
08:32 I'm pretty sure that's what it is.
08:33 And they basically built out a whole geospatial analytics tutorial and sort of documentation resource just built on Sphinx.
08:41 And so for me, seeing that, it kind of unlocked an aha moment in my head to realize that you could also use the same documentation engine for documenting lots of other things, not just software packages and things like that.
08:54 Technical documentation is quite generic and overlaps quite a lot with like scientific documentation and scholarly documentation.
09:02 And so that's the space that we've been exploring over the last couple of months and years.
09:06 Yeah, fantastic.
09:07 And a lot of the outputs are super flexible.
09:08 It outputs HTML, which you might host on somewhere like Read the Docs or Netlify, but also LaTeX, which is really important for publications and EPUB for eBooks and whatnot.
09:20 So pretty cool.
09:20 I think Sphinx is documentation generated written in Python, primarily intended for technical documentation with the ability to sort of intertwine narrative documentation with auto-generated documentation that picks up things from your code.
09:36 Right.
09:36 Yeah.
09:37 And this whole thing is combined with the ability to have a variety of output formats, as well as a variety of extension points within the tooling to extend basically every aspect of building that documentation.
09:54 Interesting.
09:54 Yeah.
09:55 So for example, testing the code snippets that you might have in your documentation.
09:58 Yes.
09:59 Or doc strings or something like that, maybe.
10:01 Yep.
10:02 So that is doc test and that's baked into Sphinx and Python's documentation, which happens to also be written in Sphinx.
10:09 Okay.
10:10 Interesting.
10:10 By Sphinx.
10:11 So there's a lot of capability and power hidden underneath the shell of Sphinx.
10:17 And as Chris was mentioning, there's really extensive customizability here that you can then take and specialize it to your specific use case.
10:27 And that's sort of both the power and, you know, there's a con to that, which is, hey, you got to maintain this and you got to keep this functioning and stuff.
10:37 But yeah, it's a really powerful documentation generation tool that is perhaps a little too powerful for its own good.
10:44 Very good.
10:47 Very good.
10:48 Yeah.
10:48 So docs.python.org is generated by Sphinx, huh?
10:52 Yes.
10:52 Fantastic.
10:53 All right, Chris.
10:54 Yeah.
10:54 I think the other point that I wanted to touch in is one of the brilliant points there on the Sphinx site is the cross-referencing capability is just second to none.
11:05 You know, you can reference inner page, you can reference cross-page within your own documentation.
11:10 But also, and one of the things that's really helped to build on Sphinx is the Read the Docs community and the work they're doing so that you can reference any other site that's built on Read the Docs as well in a really nice way.
11:25 Yeah, that's something that really surprised me when I learned about Sphinx is, you know, give a quick shout out to Paul.
11:32 Paul wrote a course that we hosted over on Talk Python, generating static sites with Sphinx and Markdown, which is really, really cool.
11:39 And one of the things that surprised me is when I think about creating multiple pages.
11:44 So, for example, if I'm on GitHub and I want to have this part of some read me here, point to some other read me or some other Markdown, I just go in there, I type, you know, here's the relative path over to that thing.
11:55 And I come up with the text that goes there.
11:57 Sphinx allows you to have kind of an index into all the sub elements of the page, not just the pages, but like parts of the pages, you know, headers and whatnot.
12:07 And you can link to them by name, right?
12:11 And so if for some reason you change the title of a header, your A tag text that you are linking to will change, which is pretty awesome, right?
12:18 Exactly.
12:19 Yeah.
12:19 And also, enter Sphinx, right?
12:23 Which is a way to expose those endpoints, not those endpoints, those index points across domain or across project, right?
12:33 So I could reference to something on Python in my documentation, specifically by a reference point, not by URL, which is pretty excellent.
12:41 And I think that's a big part of why Sphinx is as adopted as it is, by the fact that interSphinx exists and the fact that you can very easily cross-reference bits in other parts of the ecosystem, as long as that part of the ecosystem is in Sphinx.
12:58 Yeah.
12:58 As the interSphinx file generators, right?
13:00 Like in theory, other ecosystems could, or other tooling could generate it as well.
13:05 But yeah, I think that is quite capable if the fact that it's in restructured text has some fun interactions with all the linking syntaxes that exist, but it's genuinely very powerful, as I said.
13:21 Yeah.
13:21 So let's talk about that for a little bit, because when I thought about Sphinx originally, I always thought about restructured text.
13:29 And restructured text is kind of funky.
13:32 I don't know.
13:32 I just have a hard time getting used to it.
13:34 But all those formats and whatnot, it takes some getting used to, right?
13:38 You've got, let me see if I can find an example we can pull up here.
13:41 You know, things like dot, dot, image, colon, colon, some other thing.
13:45 And then, you know, sort of almost like a YAML style.
13:47 And to me, when I'm writing, I feel like, wow, I would just much rather write Markdown and just blaze through this and write it nice and clean.
13:56 And I'm willing to give up a little formatting or something to allow me to live in a simpler world that doesn't require so much stuff.
14:04 So traditionally, restructured text has been the way of the documentation through Sphinx.
14:10 It's also been like the way that you put your information on PyPI and describe it there, right?
14:17 But the PyPI.org version moved over to at least support Markdown.
14:21 And through some of the work you all are doing, Sphinx now has an integration layer with doing everything in Markdown as well, right?
14:29 You want to talk about that maybe?
14:30 Yeah, I think so.
14:31 As Preza mentioned, Sphinx is just incredibly powerful.
14:36 But it's trying to harness that power and make it not overwhelming to non-technical users.
14:45 For people who want it, it's brilliant and you can do everything under the sun.
14:49 But trying to sell it to the masses, as it were, and trying to make it as easy to use as possible for simple use cases.
14:58 But then having that extensibility there is, I guess, kind of what we've been looking at in executive books.
15:05 And with this, with the Markdown, most everyone now knows about Markdown, Common Mark, how to write.
15:13 Right.
15:13 We've got GitHub and Stack Overflow have basically forced the software development community to understand it, right?
15:19 Exactly.
15:21 Or whether people love it or hate it, it's there and you know it.
15:25 So being able to just copy and paste things from GitHub or Stack Overflow or just write something that's quite intuitive to write.
15:35 That's what we're really trying to hide some of the intricacies of Sphinx.
15:41 Make it more user-friendly on the front.
15:44 This portion of Talk Python to me is brought to you by SignalWire.
15:49 Let's kick this off with a question.
15:51 Do you need to add multi-party video calls to your website or app?
15:54 I'm talking about live video conference rooms that host 500 active participants, run in the browser and work within your existing stack, and even support 1080p without devouring the bandwidth and CPU on your users' devices.
16:07 SignalWire offers the APIs, the SDKs, and edge networks around the world for building the realest of real-time voice and video communication apps with less than 50 milliseconds of latency.
16:17 Their core products use WebSockets to deliver 300% lower latency than APIs built on REST, making them ideal for apps where every millisecond of responsiveness makes a difference.
16:28 Now, you may wonder how they get 500 active participants in a browser-based app.
16:32 Most current approaches use a limited but more economical approach called SFU, or Selective Forwarding Units, which leaves the work of mixing and decoding all those video and audio streams of every participant to each user's device.
16:44 Browser-based apps built on SFU struggle to support more than 20 interactive participants.
16:50 So SignalWire mixes all the video and audio feeds on the server and distributes a single unified stream back to every participant.
16:57 So you can build things like live streaming fitness studios where instructors demonstrate every move from multiple angles,
17:03 or even live shopping apps that highlight the charisma of the presenter and the charisma of the products they're pitching at the same time.
17:10 SignalWire comes from the team behind FreeSwitch, the open-source telecom infrastructure toolkit used by Amazon, Zoom, and tens of thousands of more to build mass-scale telecom products.
17:20 So sign up for your free account at talkpython.fm/signalwire, and be sure to mention Talk Python to me to receive an extra 5,000 video minutes.
17:29 That's talkpython.fm/signalwire, and mention Talk Python to me for all those credits.
17:35 What do you guys got to add to this restructured text markdown duality here?
17:40 I think it's an interesting duality.
17:42 And I missed my...
17:45 I don't know how to say this.
17:46 I find this...
17:48 When I first discovered this, I was really fascinated because I was like, hey, this looks cool.
17:53 This looks like a really good...
17:56 I found it especially amusing to see how mist...
18:00 Let's pick one phrasing.
18:01 Yeah.
18:02 I guess.
18:03 I really liked how mist ends up reconciling the complexity or power, depending on how you look at it.
18:09 Yeah.
18:09 Sphinx, and sort of fitting that into markdown and not...
18:14 And having it still look like it belongs there, right?
18:17 Because there's a whole bunch of, well, directives, roles, or whatever you want to call them,
18:22 that, well, you can use to manipulate the text and include things and have those extensible points.
18:29 Right.
18:29 Because markdown is not nearly sophisticated enough to handle things like interest Sphinx and
18:35 these other types of constructs that's in Sphinx, right?
18:38 Yes.
18:38 So, like, one example of this would be the ability to have the cards that you're showing
18:44 on the screen at the moment, right?
18:45 Like, the closest thing you could do for that in markdown would probably be embed a bunch
18:51 of HTML.
18:52 Yeah.
18:52 Yeah.
18:53 That or tables when you get basically no styling there.
18:55 Like, it's not cards.
18:57 It's all jammed together in a table.
18:59 But yeah.
18:59 And even when you have the ability to have inline sort of markup of markup that's in the
19:07 same line as the paragraph that you're writing in, you have a very limited set of those, right?
19:11 Like bold italics, underlines, maybe strikes if the platform you're using supports it and
19:18 stuff like that.
19:19 But with MIS or restructured text or whatever, you have a lot more capability there.
19:24 Yeah.
19:24 You in part to Sphinx, you in part to DockerDills, which is what implements the restructured text.
19:29 On this particular point of the power expressing itself into markdown, I've enjoyed watching Chris
19:37 and Chris over the years iterate through this.
19:40 Roll some things out.
19:42 Realize, wait, you know, it's better in tooling if we do a triple colon instead of a triple
19:48 back tick because the body of the directive can then be rendered by some inline viewer or
19:55 something.
19:55 So I'd be interested in hearing Chris and Chris talk a little bit about your like voyage of
20:00 discovery for, you know, what's the gestalt of markdown?
20:04 Yeah, it's interesting.
20:05 So obviously with, well, with, with Jupyter's book and things we're very much focused on
20:11 it, you know, in, in these, in Jupyter's notebooks, you have the markdown set, everything, Chris
20:16 and markdown.
20:16 And that historically has a single kind of renderer that renders things.
20:22 So trying to harness the simplicity of markdown, they're trying to make it look nice as you're
20:30 writing with that.
20:31 It doesn't look like a complete mess within these markdown cells.
20:35 Whilst still having the capabilities of like a restructured text has been an interesting
20:41 challenge.
20:42 So obviously we had this, we wanted to make the mark, the missed format kind of degradable
20:51 to common mark so that it can be passed normally as with a normal markdown parser, although it
20:57 wouldn't know what to do with things like notes and all of the other, these other directives
21:02 and roles that you, that you can have within the Sphinx.
21:05 But at the same time, yeah, have all the capabilities.
21:08 So, it's a hard balancing act, I'd say.
21:12 I'm trying to just say, here's all of these roles and directives.
21:16 You don't want to spoil the simplicity of markdown though, right?
21:19 Yeah, exactly.
21:20 It's having the readability of source text, essentially, weighing that against the flexibility of actually
21:27 making these lovely HTML pages.
21:30 Right, exactly.
21:31 Chris H., you want to add something to that?
21:32 Yeah, I mean, I think that the interesting thing about markdown and probably something that we should clarify for some of the people here is that there is no one markdown.
21:40 Yes.
21:41 That surprised me, actually.
21:42 I didn't realize that.
21:43 Yeah.
21:43 I mean, there's like, you know markdown when you see it, but there is no like canonical markdown.
21:48 The closest thing to a standard that exists is common markdown, which is like an attempt at standardization across a few different communities.
21:55 But over the years, what we've seen, whether it's, you know, Stack Overflow or GitHub or whatever, basically, people want to extend common mark in a couple of ways.
22:03 So they added a new couple of syntax features to support and they have then created a new flavor of markdown.
22:10 So people often talk about markdown in terms of flavors, like what functionality do you want to turn on?
22:14 What functionality do you want to turn off?
22:16 And there are a couple of nice like renderers and parsers.
22:19 And for example, the JavaScript world that will let you, you know, enable extensions in markdown to turn on and off different kinds of syntax.
22:26 And so what's interesting about mist is that it's kind of intentionally trying to create like a privileged extension point, either for block level writing or for inline writing.
22:37 In part with the goal that you don't have to create a new flavor of markdown if you want to extend markdown's functionality in a new direction.
22:44 And I think it's going to create an interesting question of, okay, you've created a new extension point,
22:50 but you want to have some kind of reliability or knowledge that your document is going to behave the same in one implementation of a mist renderer or a mist parser versus another implementation.
23:02 So there's an extra sort of interesting standards question of, okay, if a directive is infinitely extensible, right?
23:09 It's basically like a Python function, but in markdown.
23:12 Is there some like minimal subset of directives that you want to support as a part of core mist so that if it's a mist parser for Sphinx or a mist parser for, you know, an HTML website or for some other documentation engine, you know that they'll behave similarly.
23:27 That is a big challenge.
23:28 A quick question from the audience.
23:30 Ryan asked, does GitHub use their own version or do they use common mark?
23:34 They use GitHub flavored markdown.
23:37 GF.
23:38 Which is a superset of common mark?
23:42 Yes.
23:42 It is indeed.
23:43 Yes.
23:44 It's got, it now has, I mean, I think if you Google it, I guess, GitHub flavored markdown, it has its own specification, which is built on top of the common mark specification.
23:54 The title is different as well.
23:58 Yeah.
23:59 Yeah.
24:00 And it adds a few extensions, a few slightly different things like tables.
24:05 Right.
24:05 Exactly.
24:06 Funnily enough, it's not in common mark, which is quite a basic thing, but that is not in common mark.
24:12 That's only GitHub flavored mark.
24:13 Yeah.
24:14 Yeah.
24:14 Yeah.
24:14 Interesting.
24:15 It was avoiding the politics of what it got to get it to where it got.
24:19 There was a decent amount of, oh, this has to be maximally compliant.
24:24 Like this has to be the thing that works everywhere.
24:27 Yeah.
24:28 The minimum functionality, the maximum reach sort of story, right?
24:31 Yeah.
24:32 And they sort of went, okay, what words basically everywhere?
24:36 And that's sort of what ended up becoming some mark.
24:39 So any effort to like, oh, how about we extend this has sort of not gone very well so far.
24:46 Sure.
24:47 So I do want to give a quick shout out to this app I came across recently called TypeHora for writing markdown.
24:53 And one of the things I think is interesting, the reason I bring it up is it has the standard markdown stuff, but it also has like inline bits for like mathematics and other stuff like diagrams and whatnot.
25:05 And this is the kind of stuff you're talking about, Chris, right, where you want to take the core, but there's ways to extend it in Myst, right?
25:13 Yeah.
25:14 I mean, that's the beauty of Python, right?
25:16 Or a lot of languages where you can define functions is that you can show a lot more creativity and add extra functionality in a very intentional, structured way, right?
25:25 So that it's easier for you to replicate other people's work and build on top of it and things like that.
25:30 And so in some ways, that's how I think about directives and roles is it's like bringing functions, but into a realm where you're dealing with like human written text or code that is like interpreted.
25:40 But inside of, you know, these little directives and things like that, which just opens up a lot of extra room for creativity and trying out different kinds of things.
25:48 I guess because we're going to mention this a bunch in sort of Sphinx glossary or terminology, directives are essentially a block of text that you're all saying.
25:59 It also has this characteristic associated with it.
26:02 Like, hey, present within like a notebook or whatever, whereas a role would probably be something in line.
26:09 Hey, make this bold.
26:10 Hey, this is actually a link to another thing and stuff like that.
26:14 In case folks who are listening aren't familiar, this would probably be helpful.
26:19 Although I wish we had said this sooner.
26:23 And if you're coming from the front end world, I'm a kind of a refugee from Gatsby in which Markdown is almost like a database format, a lot of structure.
26:34 But when it comes to Markdown and extension points and things like that, oh my God, the monkey business they jump through to try and get information from the document to the extension point or whatever.
26:52 And they all have to invent their own little mechanism for packing stuff into the space after the code fence invocation.
27:01 It's just clearly obvious that they need something like what Myst has done, which is a consistent syntax that just hands things over to restructured text directives.
27:13 I think what's interesting about, you know, Spanx and DocuTools is that intermediate document model that exists, which I think is something that differentiates it from a lot of other like Markdown parsers and renders that are out there.
27:26 I think a lot of the like a lot of static website generators, they're effectively going from blobs of Markdown mapped onto blobs of HTML as a sort of like heuristic, like it's a very programmatic output kind of a thing.
27:39 And actually the original version of Jupyterbook, maybe like two and a half years old now, it was a wrapper around Jekyll, the Jekyll website generator.
27:47 But because Jekyll was fundamentally just doing, you know, blobs of Markdown to blobs of HTML, there wasn't that intermediate rich document representation where you can do things like resolve cross-references and collect, you know, bibliographic entries and collect, you know, equation and figure labels so that you can refer to them elsewhere.
28:08 And once you add that extra model in, it gives you a lot of extra, I think that's where a lot of that extra like power and complexity comes from in the Sphinx ecosystem.
28:18 And so in some ways Markdown, Miss Markdown is just like a, it's almost like a front end on a user side.
28:24 It's just giving you, at least in my opinion, a more user-friendly entry point into the Sphinx ecosystem.
28:31 Now, most times you just want to write straight Markdown, but every now and then you need something like one of these references over to another part of the site or you need more control.
28:40 And so Miss has this ability to say, kind of like run some inline restructured text here, right?
28:46 That as well as the ability to just hook into the thing that restructured text would, right?
28:53 So that will be the directive.
28:54 So you can embed restructured text inline, but that's usually not what you need to do or want to do.
29:01 Usually you're just able to directly use the thing that you wanted to use.
29:05 So yeah, there you go.
29:06 Okay.
29:06 So you would go and hook in and write some Python code to process one of these directives.
29:11 That's pretty excellent.
29:12 Yeah.
29:12 And a bunch of these directives exist already in, well, Sphinx essentially.
29:17 And you can just reuse them as you would in restructured text in Mist, except with a slightly different syntax because you're operating in a different model of language, you know?
29:29 Sure.
29:30 Yeah, exactly.
29:30 So the terminology they use is interpreted text.
29:33 So it's just a block of text and here's the name of the class function that is going to interpret this.
29:41 And here's maybe some options to help you interpret it.
29:46 And then you can do what you want to do.
29:47 Nice.
29:48 So does Mist come with a bunch of these extensions already that you can use for maybe scientific graphs or things like that that you can pull in?
29:58 Or where do I find more of these if I don't want to write them myself?
30:01 So one of the goals of the Jupyter book project is trying to bring in functionality from the Jupyter ecosystem around interactive computational document models like IPyND, Jupyter notebook files.
30:13 And also kernels that can run arbitrary, you know, usually data centric code for visualizations and analysis and things like that, where those code will generate outputs like, you know, PNG images or HTML, you know, interactive visualizations or, you know, tables with statistical analyses in them and that sort of stuff.
30:32 And so one of the goals of the executable books is also to build sort of entry points for the Jupyter ecosystem into Sphinx and into Mist Markdown.
30:44 So you can kind of get the complexity of the PyData ecosystem or the Jupyter ecosystem, but with the ability to embed that into, you know, a documentation narrative structure as well.
30:58 So I think that that's where a lot of the scientific use cases come from is like using scientific code that gets executed alongside of your documentation build in a programmatic fashion.
31:08 And where those outputs of the code are then inserted into your document in a way that from a reader's perspective, looks like it's just part of the, you know, narrative flow of everything else that was there.
31:19 This portion of Talk By The Nome is brought to you by Tonic.ai.
31:25 Creating quality test data for developers is a complex, never-ending chore that eats in the valuable engineering resources.
31:32 Random data doesn't do it.
31:34 And production data is not safe or legal for developers to use.
31:38 What if you could mimic your entire production database to create a realistic data set with zero sensitive data?
31:45 Tonic.ai does exactly that.
31:48 With Tonic, you can generate fake data that looks, acts, and behaves like production data because it's made from production data.
31:56 Using their universal data connectors and a flexible API, Tonic integrates seamlessly into your existing pipelines
32:03 and allows you to shape and size your data to scale, realism, and degree of privacy that you need.
32:08 Their platform offers advanced subsetting, secure de-identification, and ML-driven data synthesis
32:15 to create targeted test data for all your pre-production environments.
32:20 Your newly mimicked data sets are safe to share with developers, QA, data scientists, and, heck, even distributed teams around the world.
32:28 Shorten development cycles, eliminate the need for cumbersome data pipeline work, and mathematically guarantee the privacy of your data with Tonic AI.
32:37 Check out their service right now at talkpython.fm/tonic, or just click the link in your podcast player's show notes.
32:44 Be sure to use our link, talkpython.fm/tonic, so they know you heard about them from us.
32:50 Maybe tell people a bit about the Jupyterbook project itself.
32:56 It's an open source project for building beautiful publication quality books and documentation.
33:01 As you said, from sort of taking the code in the notebook and generating the output, like some of the graphs and whatnot, in a live way.
33:08 But it sounds fascinating.
33:10 Maybe when would I use it?
33:12 Tell people a bit about that.
33:13 Yeah, I mean, I think of the simplest, I mean, from a technical standpoint, since we're kind of riffing off of Sphinx, Jupyterbook is a distribution of Sphinx.
33:20 It's basically a collection of pre-configured Sphinx extensions, some of which are developed by the executable books team and community, others of which have been developed by the broader Sphinx community and that are just reused and contributed upstream to by people in executive books.
33:37 And those extensions have been created by the other way.
33:43 So things like Sphinx Contrib Bib Tech is a bibliography and citation extension in Sphinx.
33:51 And that's activated automatically with Jupyterbook.
33:54 Right on.
33:54 Because of course you're going to need that.
33:56 And you want something like Evernote that's going to pull from some source that's always like I always quote this article or something, right?
34:02 Yeah.
34:02 I mean, just it's a use case that's not built into core Sphinx, like having references and citations and bibliographies.
34:07 And so it pulls that workflow into Sphinx via an extension.
34:11 So Jupyterbook is kind of a collection of these extensions and then wrapped in a command line interface and a configuration structure that's a little bit more user-friendly.
34:20 I think that's one of the things about Sphinx is that at least historically it has tended to be both developed by and catered to the developer community, which is a little bit different from the scientific community.
34:31 A lot of scientists know just enough code to be dangerous with, myself included, and are often not as familiar with traditional software development workflows.
34:40 And so things like, you know, Jupyterbook is configurable with a YAML file rather than with, in Sphinx, the default configuration is with a Confident.
34:48 Oh, interesting.
34:49 It's like a Python file directly.
34:50 So a little quality of life improvements to make it a little bit easier for people to get started with this more opinionated distribution of Sphinx.
34:58 Yeah, it sounds like a fan.
34:59 Is what Jupyterbook is called out.
35:00 Oh, yeah.
35:00 What a fascinating resource for people.
35:02 And I meant EndNote, not Evernote.
35:04 Sorry.
35:04 That's the one you use for references.
35:07 There's some good questions in the audience I want to ask you all.
35:09 But before we do, I want to ask about MIS just a little bit more.
35:13 So MIS looks really interesting and it looks like it allows me to do much more, many more powerful things with Markdown than just straight Markdown.
35:22 Right now for when I'm in my code, like say on the website, it might have to take some Markdown content and turn it into a page or something.
35:30 I'm just using something like Markdown 2 or one of the arbitrary just Markdown parsers.
35:36 Does this make sense as something to run live in your application rather than a publication generation story?
35:45 Like would it make sense to replace just using that one function from that library and then allow me to write like directives that do more for the site, for example?
35:53 Not exactly sure what you're asking.
35:54 I think what Michael wants is the layer under MIS.
35:57 There is a parser inspired by, I guess, the JavaScript parser written by Chris and Chris.
36:05 It was actually the way I found MIS when I was looking at MIS tune and Mistletoe.
36:10 And so, yes, there's a wonderful parser that lets you hand it strings and you get back Markdown.
36:17 Yeah.
36:17 You hand it Markdown and you get HTML, you mean?
36:19 Sorry.
36:20 Yeah.
36:20 You hand it strings and Markdown and you get HTML.
36:22 Sorry.
36:22 Yes.
36:22 Yeah.
36:23 That's what I want.
36:23 Yeah.
36:24 Yeah.
36:24 Exactly.
36:24 What I want is a nice way to have richer Markdown, like in an application, in production, not just as, you know, something that I run, like a build process, I guess.
36:35 So there's a bunch of, I guess, parsers from MIS.
36:39 There's an implementation in Python.
36:41 There's an implementation in JavaScript.
36:42 If.
36:43 Yep.
36:44 Work in progress.
36:45 So I take that as a yes.
36:47 There's multiple implementations of that and you can integrate that into an application if you wanted to.
36:52 So I think this is where we're working on this kind of MIS specification of what would be core MIS as in what can we parse, not just in Sphinx, but with other kinds of parsers.
37:05 And then you say embedded into applications.
37:07 So one of the things we've been working on say is this JavaScript application.
37:13 And one of the nice things, firstly, about how we parse, how we use MIS with Sphinx in the Python world is that it uses Markdown at Py, which is a Python implementation of Markdown it, which is a JavaScript.
37:27 That is a JavaScript package passing Markdown.
37:32 So we use on both sides in Python and in JavaScript, we can use exactly the same, essentially exactly the same code, passing exactly the same.
37:40 So then we're working on this Markdown at Dr.
37:42 Teels in JavaScript to see if we can bring.
37:46 There's much of the functionality, probably not all the functionality of Sphinx in also the JavaScript world, where obviously that's what you need if you want to be able to do anything kind of live without a backend.
37:59 Right.
37:59 Runtime.
38:00 Yeah.
38:00 Okay.
38:01 Exciting to hear you all are working on it.
38:02 All right.
38:03 Let's bring this maybe back to the documentation side.
38:05 Right now, the audience asks, could we describe the whole process that a MIS document goes through before it becomes an HTML document?
38:14 Like what happens to my inputs to get either the HTML or the EPUB or whatever I get?
38:20 Essentially, so you have your Markdown text file.
38:23 Markdown, we then parse it, as I just mentioned, with Markdown at Py, which turns it into a bunch of syntax tokens.
38:32 And then we take them tokens and we convert them into the Docutils-ask syntax tree, which is what Sphinx works with.
38:43 And then as we're converting that into the Docutils-ask, Sphinx works with, we're running all of these directives and these roles, this interpreted text against all of the functions, the extensions that have been loaded within Sphinx.
39:01 And we end up with this nice syntax tree, Python thing, nodes, and then we say to Sphinx, there you go.
39:11 Take it away.
39:13 Right.
39:13 Traditionally, Sphinx has gone through some sort of restructured text process to generate that.
39:18 And you're like, we're going to generate that and give it to you in a different way.
39:20 Now just do what you do to generate your documentation, right?
39:23 Exactly.
39:23 So once you've passed it through this MISTS, or you've done restructured text, you end up with exactly the same thing.
39:31 This syntax tree with nodes.
39:34 And then Sphinx can go through that and that's kind of agnostic to any kind of output format.
39:39 Yeah.
39:39 That's just, here's a paragraph and then within that, here's some text and here's a bit of bold and a bit of italical.
39:47 And looping this back around to what we mentioned at the start of like the power of Sphinx, right?
39:53 Of the extensibility of it is that the markup format that you're writing with is decoupled from anything else that you do with it.
40:02 Not the extensions, not the directives that you use, none of that.
40:07 They're all sort of a separate step from that.
40:10 And that getting that separation happens through the intermediate documentary or doc tree, which if anyone's worked with Sphinx has probably seen mentions of in the build directory and stuff.
40:22 So yeah, doc tree sort of acts as that separator.
40:25 Oh, fantastic.
40:26 And you go from your input text to your doc tree, then to your output for formats, HTML, latex, et cetera.
40:35 And on the way, you mess around with all the doc tree to make sure you've got a piece of room to document references and all this kind of thing.
40:44 Well known fact, if you go to SourceForge and look at DocUtils, you'll see that they, from the very beginning, have anticipated restructured text not being the only format.
40:56 Restructured text is just one.
40:58 It happens to be the default parser, but it wasn't intended to be the only parser.
41:02 Yeah.
41:03 Wow.
41:03 A lot of what we've been discussing is things that Sphinx does.
41:07 Sphinx, for thought of the most part, is really a good rapport around a lot of what DocUtils provides.
41:16 It's much more friendly package, in my opinion, to interact with.
41:22 And generally, in some senses, Jupyter Book sort of wraps Sphinx into a nicer package to use.
41:30 Sphinx does that for DocuTils, right?
41:33 And it builds upon it to give you additional functionality.
41:37 It gives you additional points to hook into in the build process that you would still have with DocuTils.
41:43 It's clearly not doing anything in addition to, like, in the build process, in addition to what DocuTils would.
41:51 It just gives you a better framework to do that.
41:53 And a lot of what we've been talking about, apply that intermediate format and stuff.
41:57 Those are all concepts coming up from DocuTils into Sphinx.
42:01 Yeah, very interesting.
42:02 Alvaro out there asks, can we use the AST to translate between markup languages?
42:07 Something like Pandoc, which is a pretty neat thing.
42:09 It sounds like that might already work.
42:11 You've got the different output formats and stuff already, right?
42:13 Yeah, so an interesting example of this actually is there's a little helper tool that Chris wrote as a part of the Executable Books project called RST to Mist.
42:21 And essentially, it's a converter if you have a bunch of documentation written in restructured text,
42:27 and you want to automatically convert it into Mist Markdown.
42:31 Because Mist Markdown and restructured text have the same, like, fundamental vocabulary,
42:34 they just have different syntaxes that map onto that, you know, DocuTils doc tree,
42:38 you can go from one to the other relatively easily.
42:42 And that's what the RST to Mist package does, is it, you know, it parses RST into these abstract tokens,
42:48 and then it can render those tokens as Mist Markdown rather than restructured text.
42:54 And that's because of that sort of intermediate document format that's there.
42:57 That's very fascinating.
42:58 It's like this huge library of rules, basically, of how do you go from these abstract tokens into like a million different output formats.
43:09 And that sort of speaks to the community of the Pandoc world that's been around for quite a long time and is doing a lot of really awesome work there, too.
43:15 Yeah, indeed.
43:16 All right, let's bring it back to Sphinx a little bit.
43:18 I know you all wanted to give a quick shout out to Juan about some of the tutorials that he created, right?
43:24 Absolutely.
43:25 So one of the things that Sphinx, as we've been talking about, has, it has a documentation site, right?
43:30 And it is a documentation generator, but it didn't have a tutorial to get you started.
43:36 So that's one of the things that Juan worked on recently.
43:39 And I think a bunch of that work is actually piggybacking off of Read the Docs getting funding from EZI,
43:48 Trans-Igobab Initiative, as an essential open source software.
43:52 I think it's under the E-RSS program, but I could be wrong on that.
43:56 And yeah, he worked on a tutorial for Sphinx that, well, introduces people to Sphinx, essentially,
44:03 and, well, tells you what your first steps to document your project using Sphinx would be.
44:08 Yeah, very cool.
44:08 It sounds certainly useful to have, right?
44:12 Like the API documentation is not enough to make it feel really great.
44:16 All right, so the next thing I want to talk a bit about with Sphinx is sort of the look and feel side of things, right?
44:25 Can I really quickly make a plug, actually?
44:27 Yeah, yeah, absolutely. Go for it.
44:29 I think that one of the other reasons that, like Juan, for example, has been contributing some of the improvements to the Sphinx documentation is that, like, Sphinx is pretty old.
44:37 In the computer science world and technology world, it's like an ancient technology, you know?
44:42 It's like eight years old or something like that.
44:44 And I think that one of the...
44:45 More than that.
44:45 One of the...
44:46 Dokudones.
44:46 Yeah, that's true. That's like the early, early days of Python.
44:50 So anyway, my point is that I think that the documentation about Sphinx has been around for a long time.
44:55 But I think that the community's understanding of, like, what makes for good documentation has evolved quite a lot in the last, like, 15 years.
45:04 There's just more expectations around different kinds of documentation that you expect to find embedded in one place.
45:10 I put a link in the YouTube comments for a really interesting framework that's been gaining some traction lately called the Dietaxis framework.
45:17 But this idea that you sort of clearly separate out, like, tutorials and how-to examples and reference documentation and explanations.
45:24 That's just one example.
45:25 But I think that, like, there's...
45:27 The community has sort of evolved and made more complex its own idea of what makes for good documentation.
45:33 In some ways, faster than a lot of these Python packages that have been around for, like, decades, in some cases, have kept up with that pace.
45:41 And so I think there's a lot of low-hanging fruit to improve a lot of these aspects by making contributions to Sphinx stocks and other pieces in the ecosystem.
45:48 Yeah, absolutely.
45:49 If I can make an addition to your addition.
45:50 Do.
45:51 One of the things I think of Sphinx as is I think of it as a miracle.
45:54 It's an underappreciated miracle.
45:57 They crank out bug fix updates with long lists of features and bugs fixed and all over and over for year after year.
46:08 They don't get to go do Greenfield development.
46:11 They're still stuck in Python 2.3 land or something like that.
46:15 They've got this main template, which was written back when Mark Andreessen was still in grad school.
46:20 It's just heroic what they do.
46:23 Yeah, it's amazing when software continues to live like that, right?
46:26 It's foundational software through the ecosystem.
46:29 And yeah, there's very little Greenfield development there.
46:34 And there's very little sort of exciting work that sort of, it's all complicated problems with lots of complexity to deal.
46:41 Both terms of compatibility as well.
46:44 Oh, not having enough visibility into how your users are using things to know for sure if a change will break them.
46:52 So basically everything is a breaking change.
46:55 Let's operate with that.
46:56 And the constraint that brings with it.
46:59 So yeah.
47:00 That's a big constraint for sure.
47:01 Cool.
47:02 All right.
47:02 So let's talk about look and feel.
47:04 So there's the whole idea of Sphinx themes, right?
47:09 Yeah.
47:10 How, and maintained by some people we know here?
47:14 Yes, that is.
47:15 Yes.
47:15 That is.
47:15 That is what I use in the image.
47:16 Rajan, you want to give a quick shout out to the themes and tell us a bit about it?
47:21 Sure.
47:22 So one of the things that Sphinx has, as we've mentioned multiple times, is a bunch of variety of outputs, right?
47:28 And even within those, even within the HTML output format, you have the ability to change how your output stylized, right?
47:37 What theme you end up using.
47:38 This is roughly analogous to, I guess, Jekyll themes or Hugo themes, in that you give it a bunch of templates, maybe a bit of logic, depending on what you're doing.
47:47 And yeah, it rendered.
47:49 Sphinx has that.
47:51 And what's probably on screen at the moment, as we've talked about this, is SphinxThemes.org, which is one of the sites I helped update to be more pretty and more up-to-date and sort of a more curated set of, well, themes for Sphinx that you can and maybe should be using when you move away from the defaults, essentially.
48:14 As Chris mentioned, Sphinx is fairly old.
48:18 And when you look at the themes that shift with it, they're better than aesthetic with them.
48:24 They don't look like they were built last week by someone who has been doing this since not too long.
48:33 So, yeah.
48:34 If you ever go to the Wayback Machine and you look at something that is, you know, a popular website from the early days.
48:42 Yeah, I think it's as a Polish.
48:44 I don't know.
48:45 Yeah.
48:46 It's an amazing experience to just like pull up Google or Yahoo.
48:50 I mean, to look at something like this and say, yeah, that's one of the biggest companies in the world.
48:55 Kind of reframe your opinion.
49:01 But having these themes, I think, is an important aspect.
49:04 I do think having them look really good is something that, you know, is these are starting to come along, right?
49:11 Like I think the furrow theme and the book theme look really nice.
49:14 There's the PyData theme.
49:16 If you've done Read the Docs, right?
49:17 Read the Docs is one of the themes that people I'm sure are familiar with.
49:20 I think for a very long time, the only major good theme or one of the two major good themes was the Read the Docs theme.
49:29 The other one, Alabaster, which is not the default.
49:33 Alabaster is not the default.
49:34 And it's been fairly recent that these new themes have sort of come in and gained major adoption in sort of the timeline of Sphinx, I would say.
49:45 And I think that's a good thing because, well, I'm personally motivated to do a lot of this, right?
49:52 I have, like, one of the themes you mentioned is one that I wrote from scratch.
49:56 And it's been an interesting, fun experience.
49:59 And I can see why there's not a lot of these.
50:02 It's tricky, huh?
50:05 I really would like to give you credit for this.
50:07 I think Furo is the tipping point, the exemplar that made people rethink what to expect from Sphinx themes.
50:16 If I was to guess the two reasons people are switching to MakeDocs, number one is Markdown.
50:21 So, Chris and Chris, thank you for that.
50:23 And number two is it just looks a lot better.
50:26 Part of the reason Markdown or MKDocs looks really good is because there's one theme that's really good there, MKDocs material.
50:35 Yeah, the material, yeah.
50:36 Right.
50:36 That's all it takes.
50:37 And I have not.
50:39 And I've spent a decent amount of time looking as part of research for all of this work that I'm doing.
50:44 Is there any other major theme there?
50:47 And there isn't.
50:48 There's just one.
50:49 And pretty much most of the ecology there just revolves around that.
50:54 That isn't to say bad thing.
50:56 Certainly not.
50:57 And that isn't to say MKDocs is just that.
51:00 It's not.
51:00 Right.
51:01 But a decent amount of the overlap there is.
51:04 Well, the overlap there is huge.
51:06 It is what I would go for.
51:07 And kudos to Squid Funk, whose name is not coming to my head at the moment for the amount of work he's put into that.
51:16 Yeah, absolutely.
51:16 You and Chris are working on the new chapter of this idea, right?
51:21 Which Chris?
51:22 Chris H.
51:22 I believe the two of you are kind of thinking about what if we didn't have to live with the old contract?
51:28 I think the way you're referring to is some of the infrastructure improvements around developing the Sphinx ecosystem.
51:34 Is that what you're...
51:35 No, I was thinking of...
51:36 I thought there was a theme itself you were working on that kind of threw out the basic theme, which is the predecessor of all themes.
51:46 And I thought that you had a repo where there was a new theme you were working on.
51:50 There is.
51:50 Almost like an abstract theme or something.
51:52 Yeah.
51:53 So the way Sphinx's theme ecosystem works is there's a basic theme.
51:58 And what that basic theme has is like the JavaScript for Sphinx's search and all kinds of like fundamental things for Sphinx.
52:07 And a very, very, very bare bones HTML structure, essentially.
52:12 There's, I don't think, any CSS even.
52:14 Right?
52:15 So it's just a very bare bones page to just serve as a basic skeleton for you to build upon.
52:21 Right?
52:21 And I think through the work that Chris and I have been doing on themes and among...
52:29 There's more people.
52:30 And sort of the name escapes me at the moment.
52:34 And one of the things we found out was, oh, there's like a lot of common work across these themes.
52:41 That we put into a layer on top of that.
52:44 And those things there.
52:46 So that it reduces the amount of duplicated work.
52:48 It still gives us the bits of flexibility that we want in the individual themes to make opinionated choices, design choices, and whatnot.
52:56 But there's a decent chunk of, oh, we all will do the same thing.
53:00 Like breadcrumbs have this HTML structure.
53:05 Doing things like that will reduce the amount of duplicated effort and lower the barrier for entry, essentially, in your writing.
53:12 Which is sort of what I'm personally motivated in at the moment, having written one.
53:17 It's like, oh, this thing goes to my right sort of brain areas.
53:23 Because I, as I've mentioned, I'm a maintainer on PIB.
53:26 Developer workflows are a thing that I happen to be interested in.
53:30 And I would say have a decent amount of experience dealing with.
53:35 So I was like, hey, this looks like a great place to put in a bunch of my JavaScript, HTML, web tech experience, combined with a bunch of Python packaging and user workflows around that.
53:47 And sort of put in energy there.
53:49 So that led Sphinx team builder, which is like...
53:53 Fantastic.
53:53 Yeah.
53:54 Yeah, it's been excellent.
53:55 I think it's kind of just bringing in that maybe what Sphinx has lacked in the past is that kind of expertise in web design, really.
54:02 I mean, there's a lot of, you know, excellent people working, as you say, within Sphinx and DocuTools and things on the kind of back end and how all that works and all the Python code.
54:13 Possibly there's been less so in the past of going, right, let's actually make all this good work and actually show it off and have these just lovely themes that actually show what you can do.
54:26 It was super fascinating for me because sort of when I jumped in, I was like, oh, recreationally, right?
54:31 And to be honest, my motivation jumping into the Sphinx ecosystem was precisely this of like, I want it to look better.
54:39 Like, it doesn't look good.
54:41 I would like it to look better.
54:42 And funnily enough, none of this would have happened if pip hadn't gotten a grant where we had a bunch of computer experience experts sit with us and our users and sort of have that channel of feedback through them as well as their expertise.
54:58 And just having them state multiple times like, hey, PIP's documentation is not that great to sort of navigate and stuff.
55:05 Like, there's content.
55:06 They don't know how to get there.
55:07 And sort of in those conversations, I was like, yeah, I don't like this site.
55:13 Now that you've made me look at it a bunch of times.
55:15 Let me put something on the screen for you all just to put side by side here.
55:19 And just think about this.
55:21 And you all can pull this up really easily who are listening.
55:23 Just think about this from the perspective of someone who's choosing a programming language or new to programming and deciding, is Python the space for me?
55:32 You know, if on one hand, we've got, you know, just the docs.python.org and the other, you know, something like Tailwind, where you look at it and it's just like, it just feels, you know, so fresh and welcoming.
55:48 Whereas the other, this is not to take away from the hard and important work of writing the documentation, but the way it feels when you land there, I think, is in desperate need of the work that you do.
55:58 Yeah, but Michael, have you seen what it looks like in a man page?
56:01 I know.
56:02 It probably is looking the same, right?
56:05 Those three man pager users are really happy.
56:08 Pretty close.
56:10 And that isn't to say those users aren't about to be clear, right?
56:14 Yeah.
56:14 I mean, it's kind of like moving from MySpace to Instagram.
56:19 It is.
56:20 It is.
56:20 And like I said, I'm not bashing on it.
56:23 I'm just saying this is the face that that part of Python is presenting to the world to say, we want you here.
56:30 Come be with us.
56:31 Right.
56:31 And there's a different face that other places are putting on that I think probably will appeal a lot more to people who don't know better.
56:39 You know, they're getting to write in JavaScript.
56:40 And this is really an aspect that I would sort of think, I would like to put my energy into this and make these improvements.
56:47 And I'm by far not the only one.
56:50 There's, in fact, now a documentation working group being formed in the CPython core development community.
56:59 The folks who develop the language around improving this.
57:03 And there's like a public issue driver that hopefully going to sort of start ramping up soon because I would like to be involved with that.
57:11 And I'm aware that Paul is as well.
57:13 And yeah, this is by no means news to the folks involved.
57:17 They're aware of this.
57:18 And I think the first issue itself in this sort of dogs community issue tracker that they have is moving to a more modern documentation theme.
57:28 As it turns out, there aren't many in Sphinx.
57:31 So that's sort of been another thing I've picked up and gone.
57:35 Yeah.
57:35 So maybe one more shout out to the Sphinx theme builder that you put together, right?
57:39 Yeah.
57:39 Sphinx theme builder comes in nicely into this.
57:41 So there's a bunch of Sphinx themes today, right?
57:44 There's Furo.
57:45 There's Sphinx book theme.
57:47 There's PyData Sphinx theme.
57:48 There's Alabaster.
57:50 There's a bazillion of those.
57:51 The first three that I mentioned, right, are the folks on this discussion are involved with those.
57:58 And one of the things that came out of having worked on all of these was it would be nicer to simplify these workflows a bit, make the interaction with JavaScript easier, reduce the amount of bootstrapping required and stuff like that.
58:12 And that essentially ended up in Sphinx theme builder where it sort of streamlines the JavaScript and Python build processes and sort of makes the development workflow easier.
58:22 Yeah.
58:22 It even comes with a development server that does like refresh the browser on change, right?
58:27 Yes.
58:27 And I think that's also really nice to have in Sphinx documentation authoring as well.
58:32 And Paul sort of mentioned this earlier of like me taking up maintainership of too many things.
58:38 So I went, I like this.
58:41 And the repository had maintainers needed on it.
58:45 So I just went and opened an issue and said, hi, add me.
58:49 And I'm more basically a defactor maintainer on like live reload as well as Sphinx auto build, which is nice.
58:57 If you don't use Sphinx auto build and write a lot of Sphinx docs, start using it because it's great, I think.
59:02 Yeah.
59:03 Awesome.
59:03 I love those auto reload aspects.
59:05 Yeah.
59:06 All right, you guys, we are just about out of time.
59:09 Maybe I'll open it up for if anyone else wants to just give a shout out to something or mention something while we're all here together.
59:15 What have we not covered that we need to quickly talk about?
59:18 The main thing that I would just say is to reiterate that I think that what a lot of these like the theme conversations and improving developer workflows around the Sphinx ecosystem, one of the reasons that they're so successful is because I think that there's a lot of low-hanging fruit in the Sphinx ecosystem to basically like signal boost all of this work that's been done over the last, you know, 14 years.
59:41 And DocuTales and Sphinx and building this documentation engine and this whole community of people writing extensions for it.
59:47 In a lot of ways, I think that there's a lot of potential energy there that hasn't been unlocked yet, in part because of some of the things around developer friction or themes that don't look kind of like modern and nice and web develop-y.
01:00:00 And I think that there's still a lot more low-hanging fruit that can be accomplished with, for example, improving documentation about Sphinx itself or the extensions or whatever.
01:00:10 And so I think that all of the success of these projects is largely possible only because they're kind of piggybacking on top of this really well-established community where a lot of work has already been done.
01:00:20 And a lot of kudos go out to that broader community.
01:00:22 For sure.
01:00:23 And one other thing that I'm sort of hoping to see coming out of this is as sort of these efforts around making it easier to present your thing nicer, making it easier to write your docs and all of these happen.
01:00:37 So, you know, because when you have something that looks nicer, that's all of that, it also results in a better quality of documentation where because you're able to go, oh, actually, I can see the structure of my site more clearly.
01:00:57 I think I would like to restructure it slightly to make it clearer what the flow is.
01:01:02 And, oh, I'm missing this bit of content in this section, but I have it there.
01:01:06 Maybe I should just edit it.
01:01:07 And when these things become more obvious to you through either clearer markup or clearer site design or whatever, it will lead to better documentation.
01:01:17 And also a lot more people would be willing to write those.
01:01:20 Because it's more like, oh, why am I writing this?
01:01:23 Yeah.
01:01:24 And also moving away from Sphinx just being seen as a developer documentation into Ghibli books, trying to do, make it available for scientists and things to go, oh, wait a minute.
01:01:36 I can use this relatively easily and I can share my work or I can write tutorials, science, share my research and things like stuff like that.
01:01:47 That part's really interesting to me.
01:01:49 And it's why I did the course with Michael on static websites, not static documentation, is to think bigger than just docs.
01:01:57 I have an interest in knowledge bases.
01:01:59 I'm a developer advocate.
01:02:00 We create artifacts that are rich and interconnected and richly linked.
01:02:05 Sound like something?
01:02:06 Sphinx has inside of it this engine.
01:02:09 Sphinx and DocuTools has this engine inside, which nothing else has.
01:02:13 The very first contact you have with the mist in the course that I did is just the humble link.
01:02:21 But when you do it in mist, it will tell you if the thing on the other side isn't there.
01:02:26 It'll extract the title and inline it on your side.
01:02:30 That is magic to all these other systems.
01:02:33 And it's knowledge-based kinds of things that are valuable.
01:02:37 And I think that we could tell the story of Sphinx in a bigger way beyond documentation and start doing all of the things that people on technical teams want to do for storytelling.
01:02:52 I feel like I want to interject and mention that, hey, you know Sphinx can do blogs?
01:02:57 Yes, there's the Ablog extension.
01:02:59 Right on.
01:03:00 And Chris H, at least, has his blog on his personal website, as far as I'm aware.
01:03:08 Look inside the code of Ablog, which I've followed for years, it shows you tapping into the equivalent of front matter and walking through all the doc trees and looking for structure and doing back references.
01:03:21 People don't expect to be able to do that in Markdown.
01:03:25 It's a document database.
01:03:27 That's what you should think of, Sphinx says.
01:03:29 And I think there's a lot there of unlocking.
01:03:33 I really do want to reiterate what Chris said of there's a huge body of excellent work that has been done over more than a decade.
01:03:44 I mentioned older than me, it's like four years younger or something like that.
01:03:48 I'm four years older than me.
01:03:51 Okay, whatever.
01:03:51 I was wrong there.
01:03:52 But yeah, there's a huge body of work there.
01:03:56 And there's just, it sort of needs the little touch of, hey, paint.
01:04:02 Get it to look like it belongs in the sort of modern web.
01:04:07 But it also needed Mist.
01:04:09 It needed Mist to come along and express that power in a human-oriented way.
01:04:14 And there are so many things in Mist that are mind-blowing.
01:04:17 People haven't scratched the surface with Mist yet.
01:04:21 I actually, to that point, one other quick plug that I would give is, I know we've been talking about Sphinx in this conversation.
01:04:28 But to reiterate, I think that the goal of Mist is to be a tool or implementation agnostic specification for Markdown.
01:04:36 And I think that there is a lot of exciting possibilities if we can sort of find the right standards to apply at the Mist, you know, directives level or roles level or whatever.
01:04:48 And to see if you could get a flavor of Markdown that is flexible enough that it can be reused across like a few different kinds of applications.
01:04:56 Maybe some are pure web page, maybe some that are full-blown documentation, maybe Pandoc, you know, whatever.
01:05:01 And so I invite people who, if they can think of use cases or tools that they're working on or ecosystems that they're working in,
01:05:08 that would benefit from something like a sort of tool agnostic flavor of Markdown that has natural block and inline level extension points,
01:05:19 then reach out and we would love to chat because I think it would be really cool to see Mist being applied in other kinds of contexts as well.
01:05:25 Yeah, it would just help it grow, get stronger.
01:05:27 Yeah, I guess on a similar sort of call to action sense, I think if there's anyone around in listening who has sort of JavaScript and web design jobs
01:05:37 and sort of finds all of this interesting and wants to sort of contribute to making Sphinx look prettier, like reach out to honestly any of us on this call.
01:05:47 And yeah, we'd really like to have more people on board, not least because it's good to have fun collaborators to collaborate with, as Chris and Chris have been,
01:05:57 but also because I think the ecosystem could use a bunch.
01:06:01 I think Sphinx itself and even Dark Itals, like the maintainership story on those and the work that those maintainers do is,
01:06:09 well, the maintainership story is, let's leave it at interesting.
01:06:14 And the work that the maintainers do is really amazing for the amount of resources that they have.
01:06:19 Yeah, fantastic.
01:06:19 Good call to action.
01:06:20 Let's leave it there.
01:06:21 Let me ask you all one of the two final questions I typically ask before we get out of here.
01:06:26 Regine, I'll start with you.
01:06:29 If you're going to write some Python code, what editor are you using these days?
01:06:32 I am using Visual Studio Code.
01:06:34 Okay.
01:06:35 Used to use Sublime Text, but I've sort of switched over.
01:06:38 Yeah, I feel like that's the natural transition for Sublime users that seems to be able to advertise.
01:06:42 Good it is.
01:06:44 I'm wondering which one you might be using.
01:06:45 No, I love PyCharm as well.
01:06:50 The best, baby.
01:06:50 The best.
01:06:51 Absolutely.
01:06:52 Right on.
01:06:53 Chris Siwa?
01:06:54 Mainly Visual Studio Code.
01:06:57 Is there, I suspect, probably some notebooks in there as well.
01:07:00 There's probably some notebooks in there as well.
01:07:03 Like some Jupyter.
01:07:04 Partly Jupyter Lab, but mainly Visual Studio Code.
01:07:08 And there's also this extension for Visual Studio Code.
01:07:11 And we're working on the extension for Jupyter Lab as well.
01:07:15 Oh, fantastic.
01:07:16 All right, cool.
01:07:17 Chris?
01:07:17 H?
01:07:18 I kind of split 50-50 VS Code and Jupyter Lab.
01:07:21 Basically, if I'm doing development, software development, then I do VS Code.
01:07:25 And if I do data exploration, interactive computing, that kind of workflow, then I use Jupyter Lab.
01:07:29 Over on the notebooks.
01:07:31 Right on.
01:07:31 All right.
01:07:32 Well, thank you all for being here.
01:07:33 It's been really fun.
01:07:34 And thank you for all the hard work on bringing all this stuff up to 2020.
01:07:39 Plus.
01:07:40 Wait, wait.
01:07:41 You have two questions.
01:07:42 You only ask one.
01:07:43 I know.
01:07:44 All right.
01:07:44 Really quick then.
01:07:45 Notable PyPI package.
01:07:46 And so I don't want to take too much of your time.
01:07:48 Okay.
01:07:49 I guess for me, it'll be Pursuit PyPair, PPB.
01:07:52 It's a really cool educational game engine that's way better than what I had when I got started
01:07:58 with gaming.
01:07:59 Yeah.
01:07:59 I just finished doing the Python Bytes podcast with Brian before we started this one.
01:08:04 And ESA, the European Space Agency, just put two Raspberry Pis on the International Space Station
01:08:11 for kids and students to program against.
01:08:14 I'm like, that's way better than the turtle I got to drive around when I was in school.
01:08:17 All right.
01:08:19 Paul, really quick.
01:08:20 What's a package or library you want to just give a quick shout out to?
01:08:22 I have a fascination with dependency injection in a human oriented way.
01:08:28 Antidote.
01:08:29 Antidote.
01:08:29 Okay.
01:08:30 Very interesting.
01:08:31 That's new to me.
01:08:31 Chris S.
01:08:33 Got a library you want to give a quick shout out to?
01:08:35 So is that Chris S?
01:08:36 Yes, it was Chris S.
01:08:37 It was.
01:08:38 I'm Pipex.
01:08:40 I hadn't used yet until recently.
01:08:44 And yeah, I love you.
01:08:47 Yeah.
01:08:47 I'm all for that.
01:08:50 People talk about how it's hard to distribute Python applications and little utilities by packaging
01:08:55 them up because, well, you got to download the scripts and then set up the environment.
01:08:58 That is the homebrew of the Python world.
01:09:00 If you want something, I can go to the terminal and just type a command and it runs.
01:09:05 Well, if you Pipex install it, it's going to just be there and it's fantastic.
01:09:09 I love it.
01:09:09 People should use it more for that use case.
01:09:12 Chris H.
01:09:13 I feel like in the name of improving UI, UX and window dressing on technology that some
01:09:19 people think is outdated.
01:09:20 I would shout out to Rich.
01:09:22 Oh, yeah.
01:09:22 Sort of like UI components, visualizations, whatever.
01:09:25 I would love to find a way to get Rich into Jupyterbook or even Sphinx or something like that because
01:09:31 I think it makes for a really nice user experience.
01:09:33 The only problem is that the maintainer of Rich is much better at Wordle than I am.
01:09:37 And so he's like consistently beating me by two or three tries.
01:09:41 Every day.
01:09:42 Fantastic.
01:09:45 Let me wrap it up a bit because this seems relevant.
01:09:49 One of the things I've been doing through work as well as personal time is improving error
01:09:53 messages in Fit and Rich has played a decent thing.
01:09:56 Oh, yeah.
01:09:57 Fantastic.
01:09:57 So it's pretty likely that in the coming release or two, you'll see better error messages,
01:10:02 partly thanks to Will's work on Rich.
01:10:05 Yeah, absolutely.
01:10:05 He's got the whole traceback enhancements and everything.
01:10:08 All right.
01:10:08 Thank you all for being here.
01:10:09 It's been great.
01:10:10 Chat with you later.
01:10:11 Thanks, Michael.
01:10:12 This has been another episode of Talk Python to Me.
01:10:15 Thank you to our sponsors.
01:10:17 Be sure to check out what they're offering.
01:10:19 It really helps support the show.
01:10:20 Add high-performance, multi-party video calls to any app or website with SignalWire.
01:10:25 Visit talkpython.fm/SignalWire and mention that you came from Talk Python to Me
01:10:31 to get started and grab those free credits.
01:10:33 Tonic.ai creates quality test data that does not contain personally identifiable information.
01:10:39 Your generated data sets are safe to share with developers, UA, and data scientists.
01:10:44 Most importantly, they behave like production because they're made from production data.
01:10:49 Check them out at talkpython.fm/Tonic.
01:10:53 Want to level up your Python?
01:10:55 We have one of the largest catalogs of Python video courses over at Talk Python.
01:10:59 Our content ranges from true beginners to deeply advanced topics like memory and async.
01:11:04 And best of all, there's not a subscription in sight.
01:11:07 Check it out for yourself at training.talkpython.fm.
01:11:10 Be sure to subscribe to the show, open your favorite podcast app, and search for Python.
01:11:14 We should be right at the top.
01:11:16 You can also find the iTunes feed at /itunes, the Google Play feed at /play,
01:11:21 and the direct RSS feed at /rss on talkpython.fm.
01:11:25 We're live streaming most of our recordings these days.
01:11:28 If you want to be part of the show and have your comments featured on the air,
01:11:32 be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
01:11:36 This is your host, Michael Kennedy.
01:11:38 Thanks so much for listening.
01:11:39 I really appreciate it.
01:11:41 Now get out there and write some Python code.
01:11:43 I'll see you next time.