#380: 7 lessons from building a modern TUI framework Transcript
00:00 Terminals seem like the very lowest common denominator for software platforms.
00:04 They have to work over SSH, they only show text, you can't do much with them.
00:09 Or can you?
00:10 Will McGoogan and team have been building Textual based on Rich, which looks more like an animated web app than a terminal app.
00:18 And he has learned a bunch of lessons trying to maximize terminal-based apps.
00:22 He's here to share his seven lessons he's learned while building a modern TUI,
00:27 that is Text User Interface Framework.
00:30 This is Talk Python to Me, episode 380, recorded September 5th, 2022.
00:35 Welcome to Talk Python to Me, a weekly podcast on Python.
00:51 This is your host, Michael Kennedy.
00:53 Follow me on Twitter where I'm @mkennedy, and keep up with the show and listen to past episodes at talkpython.fm.
00:59 And follow the show on Twitter via at Talk Python.
01:02 We've started streaming most of our episodes live on YouTube.
01:06 Subscribe to our YouTube channel over at talkpython.fm/youtube to get notified about upcoming shows and be part of that episode.
01:14 This episode is sponsored by Microsoft for Startups Founders Hub.
01:18 Check them out at talkpython.fm/founders hub to get early support for your startup.
01:24 And it's brought to you by Sentry.
01:26 Don't let those errors go unnoticed.
01:28 Use Sentry.
01:29 Get started at talkpython.fm/Sentry.
01:33 Transcripts for this and all of our episodes are brought to you by Assembly AI.
01:36 Do you need a great automatic speech-to-text API?
01:39 Get human-level accuracy in just a few lines of code.
01:41 Visit talkpython.fm/Assembly AI.
01:44 Will, welcome back to Talk Python to Me.
01:47 Thank you.
01:48 It's good to be here.
01:48 Yeah, it's fantastic to have you here.
01:50 Really looking forward to talking about the progress you've made on Rich and Textual and your company,
01:56 which is pretty fantastic.
01:58 The show is not specifically about that.
02:00 It's more about all these fantastic lessons that you've learned while building it.
02:04 But, you know, of course, we'll get a chance to talk about it and give some updates too, I'm sure.
02:08 Cool.
02:09 Sounds good to me.
02:10 Yeah.
02:10 Well, normally I ask folks how they got into programming and people, if they want to hear that story,
02:17 they can go back and check out your episode 336 on Talk Python way back, way back last year, about a year ago.
02:25 It's a lifetime ago.
02:26 So much has happened.
02:28 In terms of what's happened with your projects and stuff, it really is kind of a lifetime ago.
02:32 So what I want to ask you instead is, you know, what have you been up to the last year?
02:36 Give us an update.
02:37 Yeah.
02:37 Okay.
02:38 So I founded Textualize.io, a new startup.
02:42 I've hired developers and we're very busy working on Textual and it's come on really well.
02:49 It's amazing.
02:50 There's more than I thought it would do, to be honest with you.
02:52 Oh, Textualize.io.
02:56 Yeah.
02:56 Textualize.
02:57 There we go.
02:58 The other one's for sale if you want it.
03:00 I get that all the time.
03:04 Sorry.
03:04 Yeah.
03:04 So you've been working on Textualize and you actually got some investments and you're hiring
03:09 and amazing.
03:10 I'm so happy for you.
03:13 Thank you.
03:14 Yeah.
03:14 When I first heard that, I'm like, well, what is the business model?
03:17 What is the business model here?
03:18 Like, what are you all working towards?
03:20 Like really trying to just leverage the terminal, right?
03:23 Even more.
03:23 Yeah, that's right.
03:24 So that's a very reasonable question that everyone asks.
03:27 So the first part is Textual, which will be open source and distributed just like any other
03:33 open source project.
03:34 But we'll add on this commercial service where you can take those Textual apps and then you
03:41 can put them in the web.
03:42 And then when they're on the web, we can charge companies and organizations a monthly fee for
03:49 various services such as accounts and things and maybe, you know, payment portals and things.
03:54 But there will be a very generous free tier for hobbyists and for open source projects that
04:00 want to do the same.
04:01 Fantastic.
04:01 So we have SAS for software as a service.
04:05 We have PaaS, P-A-A-S for platform as a service.
04:09 And the whole style of apps from Textual and to a lesser degree rich often go under the
04:17 terminology of TUI, a text user interface, right?
04:20 So TUI as a service, TAS, T-A-S.
04:24 T-A-S.
04:24 Yeah.
04:25 Are you creating a new as a service?
04:27 Yeah, I kind of like it.
04:28 TAS, you get, you know, in tech, we love our acronyms.
04:32 So if I don't invent at least one, I'll be disappointed.
04:35 That's right.
04:36 That's right.
04:36 And if they can have like, you know, multiple cases, right, like a capital T-A-S, then I
04:42 think it's going to be great.
04:43 Yeah.
04:43 It's an even, it's one of the better types of acronyms.
04:45 Yeah.
04:46 Yeah.
04:47 Yeah.
04:47 Fantastic.
04:48 Now, I think this is great.
04:50 And, you know, there's one of the things I'm really fascinated about and for a long time
04:54 I've been trying to pay attention to and highlight is how do people go from open source projects
05:00 to business?
05:01 And when I first started the podcast and really started to think about these things, it seemed
05:06 like there was not that many great answers.
05:08 It felt like a lot of, well, here's my PayPal donation link or something.
05:13 And it just seemed like, well, okay, that might feel good as a thanks, but you cannot make
05:18 that your job to say, buy me a coffee most of the time.
05:21 Yeah.
05:21 But there is, there's a lot of progress lately, isn't there?
05:24 Yeah.
05:25 Yeah.
05:25 Yeah.
05:25 Yeah.
05:25 Some GitHub sponsors and similar programs.
05:28 You can get sponsorship and it can be enough to live on.
05:34 It's not easy.
05:35 The type of projects to get sponsored are the ones which are super critical to businesses.
05:40 And in that case, companies don't mind donating $300 a month.
05:46 And if you get enough of those, you could and three live on it.
05:49 And some people certainly do.
05:50 But I don't think it's practical for most people.
05:54 Even if your open source project is widely used and popular, it might not bring in enough
06:01 sponsorship to live on, unfortunately.
06:03 Yeah, unfortunately.
06:05 And so I do think GitHub sponsors really is kind of that done right.
06:11 You know, it's recurring.
06:13 It's automatic.
06:14 Yeah.
06:15 It has a social signal benefit, right?
06:19 Like you can see who is sponsoring which projects.
06:22 And so if you're an organization, right, like you could say, oh, look, our company sponsors
06:26 that project or whatever.
06:28 Yeah.
06:28 It's very well done.
06:29 Actually, I've got no problems with it.
06:32 It's done.
06:33 You can, like I said, you can contribute every month or you can contribute a one-off and you
06:37 can get your name mentioned on the project.
06:39 And the author of the project can offer various perks.
06:43 So I think it's nicely done.
06:46 The other one that I see really, this might be the most popular one, is take an open source
06:52 thing and then take away, help alleviate or completely solve the operational side of things.
06:59 Right.
07:00 You know, for example, we have MongoDB and then they have Atlas, which lets you push a
07:05 button and manage your cluster automatically inside your own AWS or Azure account.
07:10 We've got Streamlit, which just got acquired, has a lot of similarities to what you're doing.
07:15 You know, it's like it's got an open source version.
07:17 You can create these dashboards.
07:19 It's really cool.
07:20 But then how do you put them on the internet?
07:22 How do you maintain them?
07:23 Right.
07:24 And then, well, guess what?
07:25 There's a paid tier that just runs it in our cloud for you.
07:28 Right.
07:29 So it seems like a really great, really great path to proceed down for what you're doing here.
07:33 Yeah, that's right.
07:34 They've got a term for it.
07:36 It's called OpenCore.
07:38 So you use the open source part to drive adoption.
07:43 And then you can use that to sell some related services, which will just make life easier for
07:48 businesses and things because there's a long history of businesses making use of open source
07:53 projects and making a lot of money out of it.
07:57 But they haven't really put that money in the business.
08:00 So if the people that are building and maintaining these open source projects can also build side
08:06 businesses around the open source project, then everyone benefits because your code gets
08:11 maintained for the indefinite future and people can make a living from it.
08:16 Yeah.
08:17 If you would otherwise have to hire somebody to manage a Kubernetes cluster or whatever,
08:22 and instead you can pay $20 a month, that's a real good deal.
08:26 Exactly.
08:26 And you don't want to have to solve all these problems individually for everyone that uses it.
08:31 It's actually far better for someone to solve it once and for all, or at least reduce the
08:37 footprint for maintenance.
08:39 Yeah.
08:40 Who else better to figure out how to put it in the cloud than people who are creating it,
08:44 right?
08:44 Yeah, exactly.
08:45 Yeah.
08:45 Yeah, for sure.
08:46 Okay.
08:46 So fantastic.
08:48 I'm looking forward to TUI as a service to come on strong here.
08:52 That's great.
08:52 So maybe before we jump into the lessons learned, just tell people about what textual is and
08:59 then textual is built upon rich.
09:02 So give us a quick rundown on take whichever order you think is better to go first, rich
09:08 or textual.
09:09 I think rich.
09:10 I'll go in chronological order.
09:12 So I started rich.
09:14 Gosh, it must be like three years now.
09:17 And the idea was to be able to write colored text, the terminal in a nice, elegant way, and
09:23 also be able to build on that with larger components.
09:27 So we've got things like tables.
09:29 We've got progress bars.
09:31 We've got log messages, panels, all sorts of things.
09:34 And they're all using the same core rendering technology, which basically takes your objects
09:40 and then turns it into ANSI codes and text.
09:43 And that got fairly popular, shall we say?
09:46 I was surprised.
09:47 I would say.
09:48 I would say so too.
09:50 Yeah.
09:50 I mean, it's in pip now, which still blows me away because that means that virtually every
09:56 Python developer is running my code, which also scares me just a little bit.
10:01 This is nuts, right?
10:02 Because I mean, if you have Python, you have pip.
10:04 So you have rich.
10:06 Is it a package with pip as a dependency or is it vendored in?
10:10 It's vendored because you've got a chicken and egg problem with pip.
10:14 Right.
10:14 Because pip is how you get the things, right?
10:16 Exactly.
10:17 Exactly.
10:17 So everything is vendored.
10:18 pip is just one project with lots of vendored projects inside of it.
10:23 Rich, for people who haven't seen it in action, it's not just like Colorama or something,
10:28 which I'm a big fan of Colorama.
10:30 But that's just about how do I make this line this color or whatever.
10:34 But you're talking about like tables with auto, you know, auto ellipsing and all sorts of
10:40 really, really powerful content, right?
10:43 Yeah.
10:43 And some things which you might take for granted in the browser, like text wrapping, that wasn't
10:48 easy to do prior to Rich.
10:50 It's more complex than you might think.
10:53 Things like emoji and Chinese characters, those take up two cells in the terminal.
10:59 If you use the built-in text wrap module, that won't account for that.
11:03 And your text wrapping will be misaligned.
11:06 So I wanted to solve those kind of problems.
11:09 I just want it to be effortless.
11:10 You know, I just, you just say, print this text and it'll wrap it for you, which generally
11:15 makes it a lot more readable.
11:16 Yeah, absolutely.
11:17 There's a lot, there's a lot going on there.
11:19 I would say so.
11:20 And I don't know how many projects are using Rich now, but it's, there constantly seems
11:26 to be like, and now this has Rich support.
11:28 So it has much better, you know, output or it's more understandable or whatever, right?
11:33 A lot of cool things happening because of it.
11:36 I'm delighted every time I read one of those.
11:38 I do try to make it quite easy to drop things in.
11:42 For instance, if you're printing out Python data structures, you can use a Rich method and
11:48 you'll get pretty printing and colorizing built in.
11:52 And things like exception handling, we've got very pretty exceptions that show you snippets
11:57 of code and can, you know, highlight the line where the error occurred.
12:01 And you can add those with just a few lines.
12:04 So I think that's, that's kind of pushing adoption is the fact that there is very low barrier to
12:08 entry.
12:09 Yeah.
12:09 And it's just beautiful, right?
12:11 It's easy to make beautiful UIs.
12:12 And if you make it easy for people to make nice looking things, they, they want to use it.
12:17 I mean, it's not as used as much these days, but think of when Bootstrap came
12:22 on the scene, you know, eight years ago or 10 years, whatever it was 10 years ago, every,
12:26 everything started to look like Bootstrap because you could just apply this magic and like,
12:30 oh, everything looks like professional, but we're not professional.
12:32 And I feel like Rich is a little bit like that for two weeks.
12:35 I think so.
12:37 Cause it was, it was very difficult to add pretty formatting prior to Rich.
12:41 I mean, there were libraries that existed, but they didn't integrate very well.
12:45 Rich kind of combines all that functionality together.
12:49 so it's, it's very easy to, to add a pretty content and, it's not just pretty for pretty
12:55 sake, you know, pretty can also mean more readable.
12:58 you know, developers, we get presented with pages and pages of content that we learned
13:03 to decipher and pick out meaning.
13:05 but we can do that much quicker if there's been some forethought into, into how it's presented
13:11 and rendered in the terminal.
13:12 And Rich does give you that capability.
13:16 This portion of talk Python to me is brought to you by Microsoft for startups founders hub.
13:21 Starting a business is hard by some estimates.
13:24 Over 90% of startups will go out of business in just their first year.
13:28 With that in mind, Microsoft for startups set out to understand what startups need to be successful
13:34 and to create a digital platform to help them overcome those challenges.
13:37 Microsoft for startups founders hub was born.
13:40 Founders hub provides all founders at any stage with free resources to solve their startup
13:46 challenges.
13:47 The platform provides technology benefits, access to expert guidance and skilled resources, mentorship
13:53 and networking connections, and much more.
13:55 Unlike others in the industry, Microsoft for startups founders hub doesn't require startups
14:01 to be investor backed or third party validated to participate.
14:05 Founders hub is truly open to all.
14:08 So what do you get if you join them, you speed up your development with free access to GitHub
14:12 and Microsoft cloud computing resources and the ability to unlock more credits over time.
14:17 Help your startup innovate.
14:19 Founders hub is partnering with innovative companies like open AI, a global leader in AI research and
14:24 development to provide exclusive benefits and discounts through Microsoft for startups founders
14:29 hub.
14:30 Becoming a founder is no longer about who you know, you'll have access to their mentorship network,
14:35 giving you a pool of hundreds of mentors across a range of disciplines and areas like idea
14:40 validation, fundraising management and coaching sales and marketing, as well as specific technical
14:45 stress points.
14:46 You'll be able to book a one-on-one meeting with the mentors, many of whom are former founders
14:51 themselves.
14:51 Make your idea a reality today with a critical support you'll get from founders hub to join the
14:57 program.
14:58 Just visit talkpython.fm/founders hub, all one word, no links in your show notes.
15:03 Thank you to Microsoft for supporting the show.
15:05 Jamie on the audience says, "I love Rich.
15:09 Rich traceback is so helpful." Maybe just quickly touch on the traceback.
15:13 And then also, Demetrius is asking about, is this for Jupyter?
15:17 Is it only a thing for Linux terminals?
15:19 And maybe sort of touch on those two things and we'll talk textual.
15:22 Tracebacks, yeah, you can install a traceback handler and it will capture unhandled tracebacks
15:29 and present them nicely in the terminal.
15:32 What was the other question?
15:33 Sorry.
15:34 Where does this apply?
15:35 Like, can I use this on Windows?
15:36 Can I use it on Mac?
15:37 Is it for Jupyter?
15:38 What is this?
15:39 Pretty much everywhere.
15:40 So you can use it on all the major platforms, Linux, Mac and Windows.
15:45 And yes, it does work in Jupyter.
15:48 So it will convert the ANSI codes to HTML automatically.
15:53 That's actually the most impressive thing to me.
15:56 That must be a lot of work to maintain all these different output destinations and whatnot.
16:01 The terminal is not too bad.
16:03 So macOS and Linux are frankly the same as far as I'm concerned.
16:08 Windows has a number of tweaks and hacks because there's two windows basically.
16:15 Newer Windows has virtual terminal sequences which make it just like the Linux and Mac
16:20 terminal and that's terrific.
16:22 But we also support the older style terminal which is quite limited.
16:27 And the review like...
16:28 Right, right.
16:29 CMD.exe is not nearly as powerful as the new Windows terminal.
16:32 No, it's not.
16:34 So we have to make some compromises and some sacrifices.
16:38 It works.
16:39 It's usable.
16:40 It won't look quite as pretty.
16:43 But there's a lot of people using that out there.
16:44 So it's important to maintain.
16:46 Yeah, you can't really drop it.
16:47 Yeah.
16:48 No, you can't drop it.
16:49 Okay.
16:50 And then the other thing that's more directly related to your article and to your business,
16:56 although obviously one's a building block.
16:58 So same as Textual.
17:00 Yeah.
17:01 So Rich was for writing kind of static content into your kind of terminal, your scroll back
17:06 buffer.
17:07 What Textual does is it uses Rich, but it completely takes over your terminal to create an application.
17:13 You don't see a command prompt, but Textual will handle key presses and mouse movements and
17:19 it'll render quite sophisticated applications, which look more like web apps than the previous
17:25 generation of Tuis.
17:26 Yeah.
17:27 But under the hood, it's using the same code that renders tables in Rich.
17:32 It's kind of a, I call it a rendering engine.
17:34 It's designed to take sort of abstract data and then turn it into what I call segments, which
17:40 are basically a piece of text with a style.
17:42 It's pretty amazing.
17:43 You know, if you look at the UI, it's got, it really does feel a bit like a web app.
17:48 It has say like a title bar across the top and you know, this isn't just like text you
17:53 printed out on the top of the output.
17:54 No.
17:54 It's stuck to the top, like a nav bar, I guess is the right word.
17:58 You have these different widgets you can put in line, like code highlighting.
18:02 You've got a text widgets with like scrolling within scrolling.
18:07 You've got animated sidebars that pop out all these cool different aspects, right?
18:12 Even a hotkey support for like D to toggle dark and light mode if you want.
18:16 Yeah.
18:17 Yeah.
18:17 We've even got data tables now, which are pretty cool.
18:20 They look kind of like little Excel windows type of spreadsheets.
18:23 Oh, fantastic.
18:24 Yeah.
18:24 So it's, I think my background is as a web developer, basically.
18:29 So I want to make something which anyone who knows how to use a web, which is pretty much
18:35 everyone, they would be comfortable using one of these applications.
18:38 You could just sit them down in front of it and they'll know exactly what to do.
18:42 They'll know that this is a menu.
18:43 They can click that.
18:44 Here's a scroll bar.
18:45 I don't know how to use a scroll bar.
18:46 It'll work with a mouse wheel or two fingered scroll on the trackpad.
18:51 It'll just be very, very familiar.
18:53 But at the same time, I also want to keep it keyboard focused because one of the benefits
18:58 of TUI is it doesn't interrupt your flow.
19:02 You know, you can be at the terminal typing commands, enter into a TUI and you could work
19:08 with that and use the keyboard and they can drop back straight into the command line.
19:11 So it's kind of this marriage of the command line and more sophisticated applications which work a lot like desktop and web.
19:21 previously when people wanted to create apps like this, well, one, it was very difficult to do.
19:25 But if they wanted to come somewhat close to these types of things, they would use a library called Curses or something like that, right?
19:33 Yeah.
19:34 And Curses has been around for a long time, I think decades.
19:38 I can't remember off the top of my head, but I think in the 90s it came around.
19:42 And, you know, people have done some very cool things with it and have been like attempts to improve upon it.
19:49 But I think still it does take, you have to be very well motivated to create an application with Curses because you're going to have to deal with some quite archaic issues.
19:59 Yeah.
20:00 I haven't done a ton with it, but it feels to me like the equivalent of saying like textual versus Curses would be like you could either use something like Pygame where you can give some sprites and they can move around the screen.
20:13 Or you could fire up OpenGL or DirectX or Metal and you could start rendering pixels on your own.
20:20 Like you could accomplish the same UI, but one is tremendously difficult.
20:25 The other one kind of gives you much higher level building blocks to accomplish the same thing.
20:30 Yeah, exactly.
20:31 So textual is more abstract.
20:33 You don't have to plot individual characters.
20:35 You just say put a text box in this part of the screen and a button in this part of the screen and, you know, textual handles the rest.
20:43 So it's less archaic aspects of terminals that go back decades that you have to think about.
20:51 Let the framework handle it and handle the differences, right?
20:55 Yeah, exactly.
20:56 That's what computing should be.
20:57 It should make your life easier.
20:59 You shouldn't have to think about, you know, decades.
21:02 You know, you hear in science people talking about, you know, standing on the shoulders of giants, of like building on Einstein and whatever.
21:10 There's nothing else that stands on the shoulders of previous work like software, right?
21:15 It's just like layers and layers and layers.
21:18 We don't have to think about that anymore.
21:20 We now have a different set of concepts to think in.
21:23 Now let's go build, you know?
21:25 Yeah, yeah, which is fine for the most part, as long as you don't ever have to look behind the curtain.
21:30 You know, it's the same with any piece of computing.
21:33 It presents a nice, clean interface, but there's a lot of effort to get there.
21:38 It's like an iceberg.
21:39 You only see the very tip of it, but two-thirds are, like, below the water level.
21:43 Basically, if it becomes a leaky abstraction, right?
21:48 And then you've got to deal with the underlying layer, but not exactly anyway.
21:52 Exactly.
21:52 All right, last question, and then we'll move off.
21:55 Talk about the lessons.
21:57 Luca asks, will Textual have a polyfill for blink, like text blinking in terminals, which
22:04 don't support it, or, you know?
22:06 But basically, I guess more interesting, does it have polyfills and, like, other ways to
22:10 make stuff happen that's not naturally supported?
22:12 You know, it could quite easily.
22:14 You could set up a timer called set interval, which would toggle something to show the blink
22:20 or hide the blink.
22:22 I don't think I want to support in Textual because I don't want Textual apps to be blinking.
22:25 It's a terrible user interface for the most part.
22:28 The exception would be a cursor.
22:31 Sometimes you want a cursor to flash so you can see where it is.
22:34 I think I might leave that one up to developer to implement, and it probably is, like, a two-line
22:40 job.
22:40 I can see a future where some movie has, like, a fake hacker UI implemented in Textual, and they
22:49 probably have a blink thing about, like, when something's going to blow up going on.
22:52 All right, let's move on.
22:54 Let's go through your lessons.
22:55 So this is what you've been building and what you are building.
22:59 And obviously you've worked on Rich for a long time.
23:02 You've worked, you and your colleagues at the company, for a year now.
23:06 So tell us about the lessons.
23:08 Let's go through these.
23:09 Yeah.
23:09 Okay.
23:10 So terminals are fast.
23:12 This might surprise people.
23:14 And I can understand why.
23:15 It surprised me.
23:16 Yeah.
23:16 Because when you use a terminal, you type a few characters, and the characters appear
23:19 on screen.
23:20 You hit return.
23:21 You know, half a second later, you'll get a response with some text.
23:25 And you don't think of terminals as being fast.
23:27 But nowadays, they're built on the same technology that runs video games.
23:31 So it uses the GPU to update the terminal with new characters, which have a background and
23:38 a foreground.
23:39 And it turns out that if you can write updates at 60 frames a second from the Python code,
23:45 the terminal will happily display it.
23:48 I was surprised at how smooth it was.
23:51 We had 60 frames per second animation of something moving across the screen, and it looked silky smooth.
23:57 And that was updating the entire screen, like every frame.
24:00 Wow.
24:01 I'm really impressed with the fact that so many of these terminals are hardware accelerated,
24:07 like GPU accelerated.
24:08 And I'm just poking around.
24:09 What am I running here right now?
24:12 I'm running iTerm.
24:12 And it has all these settings for GPU render redraws, and basically the maximum frame rate.
24:19 And so things like this.
24:22 It's just, it's, you would never expect to go find hardware acceleration features inside
24:29 the terminals.
24:29 But they have them.
24:30 They do, yeah.
24:31 And it's quite strange, because not much software takes advantage of that.
24:36 I mean, your day-to-day work at the terminal, it doesn't need to be fast.
24:40 It really doesn't.
24:41 As long as it can add new data, you know, within a second, you're probably quite happy.
24:46 So these terminals have been getting faster and faster and faster.
24:50 Some of them, like I use Kitty and something called Westerm.
24:55 And those are incredibly fast.
24:56 They're really optimized at getting things on the screen.
25:00 As in this thing?
25:01 Most software.
25:02 Yes.
25:02 Kitty's excellent.
25:03 It's super fast.
25:04 But most software doesn't need it.
25:06 You know, it's like you use Vim or HTOP.
25:11 HTOP updates once a second.
25:12 So it's like the developers of these terminals are making it faster and faster and faster.
25:17 But there's very little software that makes use of it.
25:21 Except for like sort of hobbyist demos where they do like video to text things.
25:27 So you can see like your face that's made up with ASCII characters, which is fun, right?
25:32 But you don't think that's not really productivity software.
25:34 So I was very pleased that when I started working on Textual that it wasn't a bottleneck.
25:39 You know, if I could write things fast enough to terminal, the terminal would happily accept it and render it.
25:46 That opens the possibility for so many things if you can get high frame rates.
25:51 You know, and the iTerm default was limit frames to 60 frames a second.
25:56 60 frames a second, that's plenty fine for really smooth animations.
26:00 Yeah.
26:01 And, you know, I think it's probably what my LCD monitor is stuck at, right?
26:05 So that's as fast as you're going to see it basically anyway.
26:08 That's it.
26:09 And it can't, there's not much use in it being any faster because in order to see greater than 60 frames a second,
26:15 you have to have something that's moving greater than 60 frames a second.
26:20 And generally, you don't want things flying about your terminal, unless maybe you're making a video game.
26:29 We use animation quite sparingly for nice kind of things like when a panel slides in, sidebar pops in.
26:37 It's quite smooth.
26:39 We can also use it for fades.
26:41 So we can set the opacity of a block of text and have it fade away and fade in again.
26:46 And that's sometimes quite useful if you want to draw the user's attention to the fact that you've added a new item.
26:53 Rather than having it appear immediately, it'll fade in.
26:57 So we can use animation in those places.
27:00 But again, it doesn't need to be anywhere near 60 frames a second.
27:05 No, it doesn't have to.
27:06 But I do agree that little bits of animation are super important for highlighting things that people need to pay attention to without much effort.
27:15 And it doesn't have to be so bright colored or right in the way, right?
27:20 If you just have a little thing slide down that says, you know, this job has finished while the other work is happening.
27:26 That's really nice.
27:27 And you have a video here on this article, the seven things article that we'll link to, obviously.
27:33 And it shows you running this thing called BasicPi.
27:36 Is this included with Textual or is it?
27:38 Yeah, that's one of our sandbox apps.
27:41 So whenever we're testing something, we just put it in Basic.py.
27:45 So that shows off like a number of the features we've got.
27:48 I think that video is quite old.
27:49 We've got some more cool stuff.
27:50 I'm sure you do.
27:52 Yeah.
27:53 But one of the things that's really nice is it shows a lot of the animations.
27:57 And one of the animations is changing the theme.
28:00 Another one is to pop out the side navigation.
28:03 And those are really nice, right?
28:05 The fact that that happens at a frame rate fast enough that it looks completely smooth means it doesn't just look like some terminal app where a thing clicks in and then it clicks out.
28:14 It actually feels really, you know, really nice and polished.
28:18 Yeah, we've kind of identified there's like a sliding scale of animation.
28:23 At one end, you've got like the scroll bars.
28:26 It might not be obvious, but those animate.
28:28 So if you click below it, it will scroll smoothly downwards.
28:32 And also it will filter when you click and drag.
28:34 It will filter sort of frames between the motion of the mouse.
28:39 And it makes the scrolling look a lot more smooth.
28:41 That's not obvious, but that is a use of animation.
28:43 And at that end of the scale, I think most people would agree that's good.
28:48 That's a bonus.
28:49 At the other end of the scale, we've got things which are a bit gratuitous, like the sidebar.
28:54 I like it, but some people might think, I don't want to be distracted by this animation.
28:58 I just want it to appear instantly.
29:00 I want it to be more, feel more snappy rather than animated.
29:04 So I think what we're going to do is we're going to have like a switch where you can decide as a user what kind of animation you like.
29:10 If you just want the scroll bars, you can set it to be quite low.
29:15 And that would make the sidebar pop in instantly and disappear instantly.
29:18 But if you love the animation, you can just whack it up to the maximum.
29:23 And then it will use animation wherever it can for fading and sliding, etc.
29:27 For example, with a scroll bar, that's not actually the terminal scrolling it.
29:31 That's textual redrawing the screen and reprocessing it.
29:35 If it's not smooth, if it just goes clunk, clunk, clunk, it's super hard to track when I scroll down.
29:42 Where do I continue from?
29:44 And that kind of stuff, right?
29:46 It's not just there to make it feel good or feel smooth.
29:49 It really has an important effect.
29:52 Yeah, if you press the page down, if it was to jump down instantly, you wouldn't be able to find your place in the text again.
30:00 Your eye would just have to focus in to find where you're reading.
30:04 But if it scrolls in over, say, 300 milliseconds, your eye tends to follow where you were reading.
30:11 So you move your eye, you follow the animation, and then you're sort of reading again from the top.
30:16 And that's actually beneficial.
30:18 It's not just eye candy.
30:19 It's not just gratuitous use of a feature, you know?
30:23 Yeah, absolutely.
30:24 It's very, very helpful.
30:25 So you talked about all of this happening without flickering in the terminal.
30:30 We already discussed the GPU accelerated, hardware accelerated aspects.
30:34 But you said also that's not necessarily enough.
30:37 So there's a couple of tricks that you highlight here.
30:40 The protocol of terminals came about over many decades.
30:44 And it wasn't designed to avoid flickering.
30:47 I think when people built the protocol...
30:49 They never imagined what you would be trying to do with it.
30:51 No, they didn't think someone was going to be animating it at 60 frames a second.
30:56 So there's no...
30:58 Well, there's very little help to reduce flicker.
31:01 So there's a few things that you can do.
31:03 One I've discovered is it's better to overwrite content without clearing it.
31:09 So if you want to animate, say, a piece of text, make it move, you might think, I'll clear this text, and I'll draw on top of it.
31:16 But what that does is it'll introduce a potential for a frame where you see where it's cleared.
31:22 And then you see the frame where it's updated, and that'll cause flicker.
31:25 So if you just overwrite the content without clearing it, then that reduces the chance of flickering.
31:31 Yeah, I've also seen sometimes some of these progress bars or the little tables updating.
31:37 They seem to write new whole sections.
31:40 And it's like filling your history with every frame.
31:44 Your terminal history is full of every frame and stuff like that.
31:48 If you start to resize it, and there's all this weird stuff that happens if you're not actually overwriting it.
31:53 Textual goes into what's called application mode.
31:55 This is a feature of terminals.
31:58 So when you're in application mode, you don't have a scroll back buffer.
32:01 The scroll back buffer has that problem.
32:04 It'll tend to fill up with garbage if you're trying to animate things.
32:08 And it's also quicker because the terminal doesn't have to worry about appending data and moving it, etc.
32:14 So Textual goes into application mode.
32:18 And it's generally much faster and able to reduce that flickering.
32:22 Trick two?
32:22 Trick two.
32:23 One of the problems with the protocol is that it's not aware of flickering.
32:29 It doesn't know when to paint the screen.
32:31 So you're sending it data.
32:32 It needs to know when to update the screen.
32:35 If you write lots of small pieces of data, say you're going to update a character over here, you're going to update a progress bar here.
32:42 It might update a few of them and then paint the screen and then update some others and paint the screen.
32:47 And it's that brief moment where you've got half an update, which is what causes flicker.
32:53 So what I've discovered is if you batch all your updates into one write, so you just write standard output the ones rather than several writes, that will reduce the potential flicker quite a bit.
33:05 Interesting.
33:06 Of course it would.
33:07 And I've never really thought about it.
33:09 When you're doing regular pixel-based graphics programming, OpenGL, DirectX, those types of things, they often set up what's called a double-buffered mode, where you actually draw the screen on a hidden piece and then you swap that to be in one Vsync to be the screen.
33:25 So you don't see the pieces streaming in as it goes.
33:29 And this is the same equivalent thing for terminals, huh?
33:32 It's like double-buffered mode.
33:33 You write to it like a whole thing and then you make it the screen.
33:36 It's a very similar concept.
33:38 The terminal doesn't give you that.
33:39 It doesn't have an invisible buffer that you can take your time to draw to.
33:43 So you have to implement that yourself.
33:45 But yeah, essentially, we're implementing a double-buffer.
33:47 And the third one has to do with synchronized output.
33:50 Doing the one write thing, that's very helpful.
33:53 But there's also a fairly recent addition to the protocol, which helps with this.
33:58 Basically, you write an escape code which says, I'm beginning an update.
34:01 And then you write your data.
34:03 And then you write another escape code which says, I've finished the update.
34:07 But between those two escape codes, the terminal won't update.
34:10 It'll only update at the very last moment.
34:12 So that's kind of accomplishing the same thing as the double-buffer.
34:16 Yeah, similar.
34:17 So we do both because not all terminals support these new escape codes.
34:23 So by doing both, we can ensure that it's flicker-free on newer terminals and older terminals.
34:29 Yeah, fantastic.
34:30 Speaking of different terminals, Kim in the audience asks, is application mode that you talked about earlier with the buffer a universal across all the terminals?
34:39 It is, yeah.
34:40 It's been around for a long time.
34:42 If you've used HTOP or anything like it or a full-screen terminal app, that'll use application mode.
34:50 This portion of Talk Byte Under Me is brought to you by Sentry.
34:53 You know Sentry as a longtime sponsor of this podcast.
34:56 They offer great error monitoring software that I've told you about many times.
34:59 It's even software that we use on our own web apps.
35:02 But this time, I want to tell you about a fun conference they have coming up.
35:07 Deploying new code can be a lot like making a really great sandwich, taking a bite, and then having all the contents fall out.
35:13 Exciting, chaotic, and maddening.
35:16 If you know the feeling, then Dex by Sentry might be for you.
35:20 This is a free conference by developers for developers, where you sort through the madness and look for ways to improve your workflow productivity.
35:28 Join Sentry for this event in San Francisco or virtually on September 28th and discover new ways to make your life a little easier.
35:36 Save your seat now at talkpython.fm/Dex.
35:40 That's talkpython.fm/D-E-X.
35:43 The link is in your show notes.
35:45 Thank you to Sentry for supporting Talk Python to me.
35:48 Final question while we're on this performance thing.
35:52 When will we see Doom implemented?
35:54 People keep asking that.
35:57 I don't know.
35:58 It might happen.
35:59 I think it has to happen.
36:00 I mean, Doom runs virtually everywhere.
36:03 It does.
36:04 Maybe the first Doom.
36:06 Maybe not Doom Eternal.
36:07 Yeah, yeah.
36:07 We're thinking the original Doom that really was amazing.
36:11 Right after Castle Wolfenstein.
36:12 Very good stuff.
36:13 Yeah, I love that.
36:14 All right.
36:14 So that was all some really interesting stuff about terminals.
36:18 But we also have some other recommendations and discoveries that are pretty awesome, even if you're doing other types of programming.
36:25 And one of them is Dict Views.
36:27 Apparently, I used Dict Views a lot, but I didn't really know they had a special name.
36:31 Tell us what are Dict Views and then why are they useful?
36:34 So Dict Views are the object returned from a dictionary when you call items or keys.
36:41 And I think almost all Python developers have used this, but we tend to use them to iterate over to get the keys and the items.
36:48 What you might not know is these are special objects.
36:51 They're not just simple iterables.
36:52 And they act a lot like sets, which kind of makes sense.
36:56 Because if you've got a dictionary, the keys are all unique.
36:59 So you can't have a repeating key.
37:01 So if you've got a key view, it's set-like, which means you can do set-like operations on it.
37:07 Right.
37:08 You know, ask the quick question, like, is this thing I'm about to add in here or not?
37:12 Something like that, right?
37:13 Yeah.
37:14 Yeah.
37:14 And you can take two different, you can take a set and a key view and combine them and see the intersection and see what keys are common in both or neither.
37:25 And those type of operations turned out to be very useful for what I was using it for.
37:31 And to be honest, I think this might be the first time I've ever used that in anger.
37:35 It's one of the things that you read about in the notes for the latest Python.
37:39 And you think, I can't imagine I'll ever use that.
37:42 And I didn't for years, but then I had a problem where in Textual, when we do a new update via CSS, we've got two different data structures.
37:51 One with the old positions of things and then one with the new positions of things.
37:55 What we want to do is compare those two data structures and find the things which have moved or have changed size.
38:02 And I started writing code for this and it wasn't straightforward code.
38:05 It was a couple of hours in.
38:07 I had written a lot of code and I think, that's a bit too slow.
38:09 It's a bit too awkward.
38:11 And only then did I remember the fact that you could use these key views as set operations.
38:17 And the code turned out to be almost a one-liner.
38:20 And, you know, I was so happy about this.
38:23 I forgot the fact I'd spent two hours writing code that I had just deleted.
38:28 Let me give people an example here who are listening.
38:32 So if you check out the blog post, there's a nice code example.
38:34 And, you know, the UI of Textual is made up of all these widgets.
38:38 They've got names like header, footer and sidebar.
38:41 And they have boundaries and regions, rectangles.
38:44 And as the UI changes, you might only want to redraw the delta, not the whole thing for performance reason, right?
38:50 And so given the old view and the new view, the question is, well, what widgets have changed?
38:56 And so you just say render map dot items, carrot or symmetric difference with the other newer frame dot items.
39:04 And that tells you these are the pieces that have to be redrawn, right?
39:08 Yeah, that gets exactly the information needed.
39:09 That tells me which things are new, which things have been removed, and then which things have changed position.
39:15 And those three types of things are the things that I need to redraw on the screen.
39:20 And, you know, it's a one-liner.
39:22 And I get that information for free because it is very fast.
39:26 It runs at the C level and produces just information that I needed.
39:30 Honestly, this surprises me that this is possible.
39:34 I would totally expect that with dot keys because keys have to be set like.
39:38 But items, you could have the same item assigned to different keys all over the place and stuff.
39:43 So it's a little interesting here.
39:44 The item is, you know, obviously the key plus the value.
39:48 So together, they're unique.
39:50 The values are not unique.
39:51 Oh, I see.
39:52 That's right.
39:52 Because that's a tuple of the key.
39:54 I see.
39:55 Got it.
39:55 Okay.
39:55 So the keys are guaranteed to be unique.
39:58 That makes sense.
39:58 That's therefore, yeah, whatever you add to it.
40:01 It's not going to make it less unique by adding stuff.
40:03 More possible differences.
40:05 All right.
40:06 So not totally related to this, but I got to ask it because it's just so meta.
40:10 Andrew out in the audience asked, can you embed a terminal within a textual app?
40:14 I can't.
40:15 I've been asked this a few times.
40:16 I'm just, I'm wondering what people want to use this for.
40:18 But in theory, it should be possible.
40:21 It means I'd have to write a software layer which interpreted all the escape codes and then
40:26 translated that to like a region of the textual app.
40:30 But in theory, yes, it's possible.
40:32 Will we do it?
40:33 Maybe.
40:34 One day.
40:34 It's not the highest on the priority, but maybe.
40:38 Yeah, maybe.
40:39 Okay.
40:39 Very interesting.
40:40 Is there something like a textual.contrib?
40:43 You know, these extras that people put in, like there's Django.contrib and there's other
40:48 contribs.
40:49 Yeah.
40:49 Not currently, but I'm really hoping that the community takes it and starts building things.
40:54 So you could search PyPy for textual underscore and get lots of widget libraries and various
41:00 add-ons.
41:00 Is there like a plugin or extension aspect?
41:05 Anything official like that?
41:07 Yes.
41:07 So the widgets are designed to be built that way.
41:10 Widgets are like independent portions of the screen that can handle events and things and
41:16 updates.
41:16 And those are bundled up into separate libraries, third-party libraries.
41:21 So that would be the easiest way to implement something.
41:23 You could implement anything you wish, like a full IDE if you wanted to and just import
41:29 it as a Python library.
41:30 Sure.
41:31 Related to using dictvues for speed, it's really hard to beat caching for speed, isn't
41:37 it?
41:37 Yeah.
41:38 Caching is awesome and it's one of the things which allows textual to be fast.
41:43 So LRU cache, I think a lot of people have used it, but maybe not appreciated how fast
41:48 it is.
41:49 Caching and generating the key is done at the C level.
41:51 So it's super fast and you can use it for quite small functions.
41:55 We've got a lot of calculations which are pretty quick.
41:59 You know, they're well under milliseconds, like fractions of a millisecond.
42:03 But we do them a lot of times.
42:05 So we might do them like 10,000 times.
42:07 But if we introduce the LRU cache, the time it takes to do those function calls becomes time
42:14 it takes to essentially do a dictionary lookup.
42:16 Then they become very fast.
42:18 So that can, that led to quite surprising speed ups.
42:22 Yeah.
42:22 So you have an example here that you talk about where you're given a couple of regions
42:29 regions and you do the intersection of those two, right?
42:32 A region being like a rectangle like thing.
42:35 Yeah.
42:36 So you've got to figure out, well, which is the top left most and like all those, there's
42:40 a bunch of comparisons to find the overlapping rectangle if it exists.
42:45 Right.
42:46 And then, yeah.
42:47 It's not very complicated.
42:48 It's not very slow.
42:49 It's just doing sort of arithmetic and it's working on local variables.
42:52 That's generally the fastest kind of code you can expect from Python.
42:56 You also, you create a region object which has to reserve some memory.
43:01 But if you put the LRU cache on it, that becomes a dick lookup.
43:05 So it becomes very fast.
43:06 And the type of calculations we're doing, we often do the same ones many, many times.
43:13 So we'll use the same two values to find intersection.
43:15 Like that the first time you update the screen, you calculate some intersections.
43:20 And the second time, maybe one or two items has moved.
43:22 But most of the same calculations are done again.
43:26 So LRU cache ensures that you don't do those calculations again.
43:30 You just pick them out of a dictionary.
43:31 And yeah, big wins.
43:33 And it's so easy to just copy that one line and everything works.
43:37 Yeah.
43:37 Also, it's kind of related to what we're going to get to here.
43:40 But this kind of code is fantastic because in terms of caching, because it doesn't really
43:46 depend on something that could change behind the scenes.
43:49 It doesn't have hold of like weird pointers that other, you know, could change in other ways.
43:55 And so it's very, very deterministic.
43:58 It's going to give you the same answer every time.
44:00 And, you know, it's not like the cache is going to get weirdly stale, right?
44:05 Yeah, exactly.
44:06 So it's an immutable object.
44:08 I think region is actually a named tuple.
44:11 And you get all those benefits.
44:13 There's no side effects.
44:14 You can write these functions that have got an input and an output.
44:17 And it doesn't depend on any other state.
44:20 And when you have that kind of function, I think they call it a pure function.
44:24 It's caching works beautifully.
44:26 There's no hidden surprises.
44:28 And also, I think, makes your code more easy to reason about.
44:32 You can trace it, you know, just manually.
44:35 You can tell what's going on.
44:37 Just by looking at one function, you can see the full story.
44:41 Sure.
44:42 Yeah.
44:43 Immutable objects.
44:44 If your code can use immutable objects, I think you should favor it.
44:48 It doesn't always make sense.
44:49 But immutable is definitely best.
44:51 Yeah.
44:52 Definitely easy to reason about.
44:53 Kim in the audience says, presumably, the memory cost of caching many frequently called functions isn't a big issue on a reasonable machine.
45:02 And maybe it's worth pointing out the max size parameter you passed to the LRU cache.
45:06 That's right.
45:07 So you set a maximum size.
45:09 And if you add more items than that, it'll throw out the oldest one.
45:13 So you can define the set, the maximum size that you will record.
45:18 And it depends on how you use it.
45:20 If you tend to have like a common set of calculations, which you're doing repeatedly, and those will kind of be in the cache most of the time.
45:30 And then you might have some calculations which happen infrequently, you know, combinations of input and output.
45:35 And then those get recalculated occasionally.
45:38 But it's still a big win.
45:40 And for 4,096 items for that, the regions name tuple is quite small.
45:46 So that keeps the memory usage to something reasonable.
45:50 I did want to highlight this project called Async Cache.
45:55 Because I think, I haven't checked lately, but I think the LRU cache is only for synchronous functions.
46:02 I think so, yeah.
46:03 Yeah, I think so too.
46:04 You know, maybe it's been upgraded.
46:06 I haven't checked.
46:07 But if it's still true, there's this project called Async Cache, which has nearly the same UI, but applies to Async functions.
46:18 So basically, the decorator has to return a function, which is first checks the cache and then calls the function.
46:24 For the Async version, that has to be an Async function returned out of the decorator.
46:29 Hence the difference, right?
46:31 And so this Async Cache has something really similar where you can have an LRU cache where you set the max size.
46:37 Or you can also have a time to live, like I only want these results to stick around for 60 seconds.
46:43 It also has a bunch of other interesting features that you can bring in, like works on ORM objects, request objects, a bunch of other things.
46:51 It has sort of custom support for custom types that are one of the things that this needs here.
46:58 I think it's hashable.
47:00 I think you have to have hashable arguments for LRU cache.
47:04 Yeah, that strikes me as something very useful.
47:06 If you've got a calculation which does some of weights and you want to cache it, then yeah, that looks like a fantastic project.
47:13 Not super popular.
47:14 People can check it out, but it looks pretty useful.
47:16 Okay, we kind of touched on this already, so maybe we won't go too much into it, but just one of the actual lessons you said is immutability is definitely a good thing.
47:26 So we can get this from tuples, named tuples, which are like better tuples.
47:31 You know, you can address them by a variable type or frozen data classes.
47:36 I tend to think immutable first.
47:39 So I'll prefer to make my objects immutable.
47:42 And only when I think that's going to become a hindrance do I make them mutable.
47:46 And that's, I think, I started doing that a few years ago, and I think that's benefited my code.
47:51 So I'd recommend mutable objects.
47:53 Yeah.
47:54 And then if you're out there doing Pydantic, which many of us are these days, it has faux immutability, which is kind of like it.
48:02 It takes a shot at making things immutable.
48:05 And you can just say allow mutation false as the config for your Pydantic model, which is pretty fantastic.
48:12 Yeah.
48:12 I think Python doesn't support true immutability.
48:14 Immutability.
48:16 If you really wanted to, you could do some horrible hack to change an object, which should be immutable.
48:22 But, you know, like Python's philosophy is we're all adults here, right?
48:25 So don't do crazy things like that.
48:28 And even for languages that have like the word const and stuff, it still doesn't necessarily mean what you think it means.
48:35 There's a little bit of Monty Python going on.
48:38 No, Princess Bride sort of thing going on there where, okay, so this object says it can't be mutable and it has pointers to other objects.
48:46 And so sure, you can't reassign those pointers, but that thing points to something which points to something, which internally you call a function which changes.
48:52 You know what I mean?
48:53 Like to get true immutability is super hard and super restrictive.
48:58 Super hard, yeah.
48:59 Yeah, yeah.
48:59 So Python isn't too bad in that respect.
49:02 It's definitely safer than C, C++, et cetera.
49:06 Yes, that's for sure.
49:07 Okay, immutability is good.
49:09 So is Unicode art.
49:12 What the heck is this?
49:13 In the textual code, there's quite a lot of functions like this, which are kind of geometric.
49:18 So it might divide something into two.
49:20 Yeah, it's always doing visual things, right?
49:23 Quite a lot of visual things.
49:24 So here we've got an example.
49:25 It's a function which takes a region and splits it into four by making two, or making a horizontal and a vertical cut.
49:32 And that's quite hard to express in English succinctly.
49:36 I mean, you can do it, but it'll take you like a big paragraph.
49:39 But if you create something like this, this kind of Unicode art using various box characters, you can draw a diagram to show what you're doing.
49:50 And if you're coming to that code later, that just makes it really obvious at a glance what it's doing.
49:57 Yeah, it's fantastic.
49:58 I'm a big fan of that kind of Unicode art.
50:01 And I use an application called Monodraw, which is macOS only.
50:05 But there are similar applications for other platforms.
50:08 And yeah, I use it wherever I can.
50:11 Also because it's kind of fun.
50:13 It is cool.
50:15 A powerful ASCII art editor designed for the Mac.
50:18 It costs money, $10.
50:21 I love that it has educational pricing, but it only costs $10.
50:24 That's kind of cool.
50:28 But yeah, it's pretty neat.
50:30 I guess you have little arrows and boxes.
50:34 And just like you might draw with PowerPoint or Keynote or something like that is kind of your, let me put together a visual graphic.
50:42 But then output is ASCII art.
50:44 Yeah.
50:44 And it's kind of like a vector graphic type thing.
50:47 So if you draw a box, you can click on the box and drag it.
50:50 It's not like a bitmap.
50:52 Oh, that's fantastic.
50:53 Yeah, it's quite powerful.
50:55 I hope the authors made a fortune out of it, frankly, because I love it.
51:00 So what is the output like?
51:01 You output to clipboard?
51:02 Yeah, you can cut and paste it.
51:05 You output to clipboard.
51:06 You can write it to a text file.
51:08 I usually cut and paste it directly into Python code.
51:11 Yeah, of course.
51:12 So that's really cool.
51:13 It also made me think of something that's not exactly the same purpose, but Balsamic.
51:18 I don't know if you're familiar with Balsamic, which is really great for developing UIs.
51:22 I don't know if they have a gallery or something.
51:24 They need some more graphics on this thing here.
51:27 Yeah, I've used Balsamic for a while.
51:29 It's been around for a long time, hasn't it?
51:31 It's been around forever.
51:32 Yeah, but it basically lets you create UIs that look almost like as if they were created
51:37 using ASCII art.
51:39 Not quite, but it's an interesting goal.
51:42 I did also want to highlight the most useful piece of ASCII art.
51:47 And to be clear, all of this discussion is what goes in the comments, right?
51:51 What goes in the code comments or the doc strings?
51:53 The one for, what is it, for the object allocator in Python.
51:58 So if you look in obmalloc.c in CPython, there's all this fairly intense looking code for allocating
52:08 memory when it's particular CPython objects, like a py object or a py long or whatever, right?
52:16 If you scroll down to the section here around line 777, it has a paragraph.
52:22 It's an object allocator for Python.
52:24 Here's an introduction to the layers of Python's memory architecture, etc.
52:28 And instead of having an essay in here, it has this incredible graphic that is like vertically aligned and shows you,
52:36 here's where we allocate things like integers and lists.
52:39 And here's Python core non-object memory allocation.
52:43 Here's how it relates down to the OS and to actual hardware.
52:46 What do you think of this?
52:47 That's pretty awesome.
52:48 A lot of respect to the author because I'm fairly sure they didn't use monodraw.
52:52 I think they did that by hand with typing the characters and spaces and everything.
52:57 Yeah, it's pretty awesome.
52:59 Yeah, I think this is really fantastic.
53:00 I was trying to piece together, you know, I did a course on Python memory and I was trying to piece together like, well, how do I visually represent how it is described that the memory works?
53:10 And I'm like, well, let me go look in the source code.
53:12 And I was looking at this like, there's an actual picture in here.
53:15 This is amazing.
53:16 I can't believe it.
53:17 So, so neat.
53:18 Yeah.
53:18 So anyway, if people need a concrete example of the type of stuff you're talking about, here's one that's pretty close to home.
53:24 We all use it every day and we didn't know.
53:26 Fractions.
53:27 I was always amazed at the fact that 0.1 plus 0.1 plus 0.1 is not equal to 0.3.
53:32 I learned that in the school and it was, but for some reason it's not.
53:35 What do you think?
53:36 Yeah, so this is a problem that goes, this is, it's not just Python.
53:39 It's pretty much any language which uses floats and doubles, which is like almost all of them.
53:44 And it trips up beginners and experienced developers.
53:48 It tripped up me when I was working on Textual.
53:50 We have lots of code which will divide the screen into various proportions.
53:54 So you might have a third and then a two thirds and you might divide the height into sevenths.
53:59 And what I found was that when I used floats, I got a lot of rounding error.
54:03 Not, actually not a lot of rounding error, but occasional rounding error, which means it wouldn't draw a line or it wouldn't draw a column.
54:09 Because this kind of issue where the rounding error is compounded and it's been rounded down to the nearest character.
54:17 Right.
54:17 Because you've got to, you've got to talk in like little block pieces anyway.
54:20 And if it just misses it, right?
54:22 If it just misses it by 0.0000001, then, then you'll be a whole character out.
54:28 And I did come up with like various ways of solving this where I used integers rather than floats and kept track of the error, which I think is the standard approach of doing it.
54:38 But it was quite tricky code and I get it wrong too frequently.
54:42 But then I remembered fractions.
54:44 So fractions is, they behave just like numbers, but they start at life as a numerator and a denominator.
54:51 So now fraction 1, 2 equals a half.
54:54 And the great thing about fractions in the standard library is they don't suffer from that rounding error.
54:59 So if you have, you know, three, one tenths, it'll add up to three tenths exactly with no rounding error.
55:06 And it makes that kind of code, which I do a lot of in textual, to be much simpler.
55:11 So I was very happy when I re-remembered it because I must have known years ago and thought, why do I need fractions?
55:18 Yeah, exactly.
55:19 Yeah.
55:20 This is not elementary school.
55:22 I don't need this.
55:23 Come on.
55:23 We could just say 0.1.
55:25 Yeah.
55:26 So you learn that you think, well, that's probably useful to mathematicians or something or someone else, not to me.
55:32 But if you work long enough, you'll eventually find a problem where that is actually the perfect solution for it.
55:38 I didn't know about these either.
55:40 I see comments in the audience as well, like, fractions, what is this madness?
55:43 So you're telling us that you can only work in rational numbers, not, you know, irrational numbers like pi and E.
55:51 We can't have columns that are E-wide.
55:54 No, no, you can't.
55:56 Or pi high.
55:58 It's pi by E.
56:01 No, it's cut off.
56:02 Darn it.
56:03 Yeah.
56:03 The cool thing is that fractions, you can drop in replacements to float.
56:08 So if you've written some code which was previously using floats and then you pass in fractions, everything will work as normal, except you'll not get the rounding error.
56:17 It's really kind of beautiful.
56:18 Okay.
56:19 So you're saying the fraction library supports things like division, multiplication, additions, and basically would duck typing behave the same?
56:27 Yeah.
56:27 It's duck typing as a rational number.
56:29 So anywhere we use a float, fractional work.
56:32 Okay.
56:33 I learned about that.
56:34 Amazing.
56:35 Let's see.
56:35 Emojis are hard.
56:37 You talked a little bit about things that take up different sizes and Unicode and whatnot.
56:41 Yeah.
56:42 Yeah.
56:42 So when I started Textalize and I got my first employee, I thought this is the problem that I want to tackle because it has been bugging me for two whole years.
56:53 And the problem is basically this, that some characters will take up two cells in the terminal, they'll double wide, and some characters will take up one cell.
57:01 And if you want to do any kind of formatting, say to draw a line or a panel around it, you need to know exactly how many cells a piece of text will take up.
57:10 And it sounds like a simple problem.
57:12 All you need to do is know how many cells a character will take up.
57:16 So you might have a function which takes a character and returns either one or two.
57:20 And in essence, that's what Rich does.
57:23 But at some characters, you can't know because they will render differently on different terminals.
57:30 They'll render differently on iTerm and Windows and Linux.
57:34 And it gets even more complicated.
57:36 You can get multiple characters, so multiple code points in the characters.
57:41 So you might have something like a flag.
57:44 And a flag has a two-letter character code and another character code which tells you this is a flag.
57:49 So if you iterate over that, you get three code points.
57:52 So you have to know, first of all, how everything works together.
57:56 And there's quite a lot of those type of characters.
57:58 I can't imagine how tricky it is to be done inside Unicode.
58:01 Yeah.
58:02 Yeah.
58:02 It's crazy.
58:05 It's very, very complex.
58:06 And, you know, we thought, well, we'll just do it.
58:09 We'll just apply some engineering effort and just do it.
58:12 But we discovered that it was impossible to know because you can't tell how the terminal is going to render these Unicode.
58:18 It might do it correctly.
58:19 It might actually render a single character.
58:21 If the terminal is not aware of multi-code point emojis, then you might get three characters.
58:27 They might render as nothing or they might render as three double width characters.
58:33 They might not even render properly.
58:35 So they might overlap following character.
58:37 Some number of boxes.
58:39 Yeah.
58:40 It's just every terminal did it differently.
58:42 Not only the terminal, it was platform dependent.
58:45 So if the terminal used the Unicode database on the operating system, then you'd get different results if it shipped its own copy of the Unicode database.
58:54 And it just turned out it was impossible to know how many characters.
58:58 So what do you do?
59:00 Well, there's a subset up to about Unicode 9 where things seem to be mostly sane, most terminal support.
59:07 So if you use those, those are fine.
59:09 But after that, it becomes unreliable.
59:12 If you have flags and multi-code point characters, then it might not work.
59:18 It might cause the alignment of tables and panels to be out.
59:23 You know, the end character might be shifted out by one.
59:25 And it just comes up so frequently.
59:27 And I would solve it if I could.
59:30 But as far as I can tell, there is no reasonable solution which will work across all platforms.
59:36 Too many unknowns, right?
59:39 Too many unknowns, yeah.
59:40 It's a crazy thing, but it kind of makes sense because these Unicode characters came out in the last few years.
59:46 And operating systems and terminals haven't caught up and they haven't agreed on how to render them.
59:52 So it's a frustrating situation.
59:55 I'm sure that it is.
59:56 Yeah, I mean, if you look at like some of the nerd fonts, for example, when I, you know, at nerdfonts.com, you know, you look at some of these, what is possible.
01:00:06 I think here, you know, you have these like colored arrows.
01:00:10 And do they have, I don't know, if you go over to Oh My Posh, which I don't really have time to talk about.
01:00:16 But if you look at the themes that are in here, the different themes that you can pick and just some of the effects, like a character that looks like a git branch with an arrow in it.
01:00:27 Like, how is that?
01:00:29 How are you supposed to decide how big that is?
01:00:31 Or, you know, how is some of this stuff accomplished?
01:00:34 It's just...
01:00:34 I don't know how they represent those characters.
01:00:37 I wonder if they reuse existing characters.
01:00:41 When I saw that, I'm like, okay, this is...
01:00:43 I don't know how this is possible, right?
01:00:45 Rounded edges and all sorts of stuff.
01:00:47 So I'm like trying to find one where it's like a sparkly fade from one character to the next.
01:00:52 It's these things over in the OhMyPosh.dev themes.
01:00:56 If I thought about your job to figure out, like, what is that supposed to look like?
01:01:00 I don't know.
01:01:00 I would just give up.
01:01:01 Because these are...
01:01:03 I mean, they're really impressive and really useful.
01:01:05 But this one, for example, the cert theme, where it's got like little dots that fade from one to the next.
01:01:11 I just...
01:01:12 I just don't know.
01:01:12 Amazing, but...
01:01:14 Those look pretty cool.
01:01:14 How are you supposed to know, right?
01:01:16 Right.
01:01:16 So those are your lessons.
01:01:19 Very, very cool stuff.
01:01:21 I really appreciate you coming on to talk about the seven lessons.
01:01:24 You know, terminals are fast.
01:01:26 Dictive views are amazing.
01:01:27 LRU cache is fast.
01:01:29 Immutable is good.
01:01:30 Unicode R is good.
01:01:31 Fractions are good.
01:01:32 Emoji is bad.
01:01:33 Does that summarize it?
01:01:36 That about...
01:01:37 Yeah, in a nutshell.
01:01:38 Oh, let's wrap it up with one more thing real quick here.
01:01:41 Let me see if I can find it.
01:01:43 This one?
01:01:44 There we go.
01:01:44 So one of the...
01:01:46 Sort of bring it full circle back around.
01:01:48 One of the cool things to make terminals nicer that you've talked about recently is rich-cli.
01:01:55 You want to close out the show?
01:01:56 Just give us...
01:01:57 Tell us what rich-cli is.
01:01:58 Sure.
01:01:58 Okay.
01:01:59 So as you know, rich is a library.
01:02:00 And rich-cli is a CLI interface for that library.
01:02:04 So most of what rich can do is exposed by rich-cli.
01:02:09 So you can cat most file formats, and it'll be nicely syntax highlighted.
01:02:13 You'll have line numbers and guidelines and all sorts of things.
01:02:17 And you can do things like panels.
01:02:20 You can format text.
01:02:22 What else?
01:02:23 There's a whole bunch of other features.
01:02:25 Yeah.
01:02:26 So for example, if I had a JSON document on the terminal, I could type...
01:02:31 I could open it in some terminal-based editor, you know, think SSH somewhere.
01:02:35 Or I could just type more or cat the name of it, and it would print out just plain text of whatever's on the inside.
01:02:42 Or now I could type rich, the file name, and I get, you know, highlighted, colorized, formatted content for like CSV and JSON and all those kinds of things, right?
01:02:53 Yeah, yeah.
01:02:53 So the rich, the JSON example is quite good because that will decode the JSON.
01:02:58 So if you've got like compressed JSON with no white space, it makes it impossible to read.
01:03:03 Rich will decode it, and it'll also format it.
01:03:07 I see.
01:03:08 Like pretty prints it.
01:03:09 Pretty prints it, exactly.
01:03:10 Yeah.
01:03:10 And it'll do markdown.
01:03:12 It'll do a reasonable job of rendering markdown.
01:03:15 And it'll take CSV files, and it'll turn those into nice rich tables.
01:03:21 And if your output is quite large, you can add hyphen, hyphen, pager.
01:03:24 And that'll put in a nice pager where you can scroll up and down with scroll bars and do page up, page down, et cetera.
01:03:30 So it's kind of like a toolbox for fancy, rich formatting of all sorts of different data types.
01:03:36 Definitely a cool project.
01:03:38 And I know you're concerned about emojis, but Paul in the audience says, fortunately, Doom does not require emojis.
01:03:44 So it's still on the table.
01:03:45 I suppose not, yeah.
01:03:49 That's right.
01:03:50 All right.
01:03:51 Well, fantastic lessons.
01:03:53 Thanks for sharing all of your experience.
01:03:55 Final two questions before you get out of here.
01:03:57 If you're going to write some Python code, work on rich, what editor are you using these days?
01:04:01 It's not so meta that you're using Textual to write Textual yet, is it?
01:04:04 No.
01:04:05 Maybe one day.
01:04:06 But no, I use VS Code.
01:04:07 And I quite like it.
01:04:09 I'm comfortable with it.
01:04:10 My colleague uses, what's the editor by JetBrains?
01:04:14 PyCharm.
01:04:14 PyCharm.
01:04:16 And he's very proficient at PyCharm.
01:04:17 And to be honest with you, I am jealous of some of the features of PyCharm.
01:04:20 It does some really cool things.
01:04:22 So I'm kind of tempted to try PyCharm.
01:04:25 Looking over the shoulder.
01:04:26 Awesome.
01:04:27 Yeah.
01:04:27 Very cool.
01:04:28 And then notable PyPI package.
01:04:30 I mean, we've touched on some good ones that start or end with Rich.
01:04:34 But anything else you run across that you're like, oh, this is fantastic.
01:04:37 People should check this out.
01:04:38 Oh, gosh.
01:04:39 There's so many.
01:04:40 I'm drawing a blank.
01:04:42 I should have heard one in advance.
01:04:44 How about one that you use that makes Rich work well or something?
01:04:48 Well, there's Prompt Toolkit.
01:04:51 So I owe Prompt Toolkit a big debt of gratitude.
01:04:55 Because when I was figuring out the textual stuff, I looked at the Prompt Toolkit source
01:04:59 code, which is a great thing about open source that you can look at other people's code.
01:05:03 And it is very good.
01:05:04 It helped me understand things.
01:05:06 And it still does things which Textual doesn't do yet.
01:05:09 So I think Prompt Toolkit is an excellent project.
01:05:11 And if you haven't used it, you should definitely check it out.
01:05:13 Yeah.
01:05:13 Prompt Toolkit's interesting.
01:05:14 And you'll be typing along.
01:05:15 And all of a sudden, there's like a combo drop-down box, like a select right in the middle
01:05:21 of your terminal.
01:05:22 And then you just carry on.
01:05:23 Yeah.
01:05:23 It makes things like, I think it's used in IPython.
01:05:25 It makes that much nicer and much more friendly.
01:05:28 Absolutely.
01:05:29 All right.
01:05:30 Well, final call to action.
01:05:32 People want to do stuff with Rich Textual, maybe take some of these lessons and run with
01:05:37 them.
01:05:37 What do you tell them, Will?
01:05:38 Yeah.
01:05:38 Check out the website.
01:05:40 Check out my Twitter profile.
01:05:42 And if you have any questions, feel free to ping them over to me.
01:05:44 I'm always happy to talk to people about Python-related things.
01:05:48 Yeah.
01:05:48 Fantastic.
01:05:49 And of course, I'll link to the article with your list of topics there so people can check
01:05:54 that out.
01:05:54 Thank you so much for being here.
01:05:55 It's been great to catch up with you.
01:05:57 Yeah.
01:05:57 Thank you.
01:05:57 You bet.
01:05:58 Take care.
01:05:58 Bye.
01:05:58 Bye-bye.
01:05:58 This has been another episode of Talk Python to Me.
01:06:02 Thank you to our sponsors.
01:06:04 Be sure to check out what they're offering.
01:06:06 It really helps support the show.
01:06:07 Starting a business is hard.
01:06:10 Microsoft for Startups, Founders Hub provides all founders at any stage with free resources
01:06:15 and connections to solve startup challenges.
01:06:18 Apply for free today at talkpython.fm/founders hub.
01:06:22 Join Sentry at their conference, Dex, Sort the Madness, the conference for every developer
01:06:28 to join as they investigate the movement and trends for better and more reliable developer
01:06:32 experiences.
01:06:33 Save your seat now at talkpython.fm/Dex.
01:06:37 Want to level up your Python?
01:06:39 We have one of the largest catalogs of Python video courses over at Talk Python.
01:06:42 Our content ranges from true beginners to deeply advanced topics like memory and async.
01:06:48 And best of all, there's not a subscription in sight.
01:06:50 Check it out for yourself at training.talkpython.fm.
01:06:53 Be sure to subscribe to the show.
01:06:55 Open your favorite podcast app and search for Python.
01:06:58 We should be right at the top.
01:06:59 You can also find the iTunes feed at /itunes, the Google Play feed at /play,
01:07:04 and the direct RSS feed at /rss on talkpython.fm.
01:07:09 We're live streaming most of our recordings these days.
01:07:12 If you want to be part of the show and have your comments featured on the air,
01:07:15 be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
01:07:20 This is your host, Michael Kennedy.
01:07:22 Thanks so much for listening.
01:07:23 I really appreciate it.
01:07:24 Now get out there and write some Python code.
01:07:26 I really appreciate it.
01:07:47 Thank you.