WEBVTT 00:00:00.020 --> 00:00:05.200 Digital humanities sounds niche until you realize that it can mean a searchable archive of U.S. 00:00:05.420 --> 00:00:11.580 amendment proposals, Irish folklore, or pigment science in ancient art. Today I'm talking with 00:00:11.720 --> 00:00:17.780 David Flood from Harvard's DARTH team about an unglamorous problem. What happens when the grant 00:00:18.060 --> 00:00:24.180 ends? But the website can't. His answer? Static sites, client-side search, and sneaky Python. 00:00:24.600 --> 00:00:30.680 Let's dive in. This is Talk Python To Me, episode 538, recorded January 22nd, 2026. 00:00:48.540 --> 00:00:52.880 Welcome to Talk Python To Me, the number one Python podcast for developers and data scientists. 00:00:53.180 --> 00:00:54.780 This is your host, Michael Kennedy. 00:00:55.130 --> 00:00:58.720 I'm a PSF fellow who's been coding for over 25 years. 00:00:59.360 --> 00:01:00.420 Let's connect on social media. 00:01:00.800 --> 00:01:03.880 You'll find me and Talk Python on Mastodon, BlueSky, and X. 00:01:04.239 --> 00:01:06.060 The social links are all in your show notes. 00:01:06.820 --> 00:01:10.340 You can find over 10 years of past episodes at talkpython.fm. 00:01:10.520 --> 00:01:13.720 And if you want to be part of the show, you can join our recording live streams. 00:01:14.080 --> 00:01:14.540 That's right. 00:01:14.730 --> 00:01:18.000 We live stream the raw uncut version of each episode on YouTube. 00:01:18.580 --> 00:01:23.020 Just visit talkpython.fm/youtube to see the schedule of upcoming events. 00:01:23.180 --> 00:01:26.880 Be sure to subscribe there and press the bell so you'll get notified anytime we're recording. 00:01:27.820 --> 00:01:29.480 This episode is brought to you by Sentry. 00:01:29.800 --> 00:01:31.040 Don't let those errors go unnoticed. 00:01:31.230 --> 00:01:32.840 Use Sentry like we do here at Talk Python. 00:01:33.340 --> 00:01:36.200 Sign up at talkpython.fm/sentry. 00:01:37.040 --> 00:01:42.320 And it's brought to you by CommandBook, a native macOS app that I built that gives long-running 00:01:42.520 --> 00:01:43.980 terminal commands a permanent home. 00:01:44.360 --> 00:01:46.360 No more juggling six terminal tabs every morning. 00:01:46.820 --> 00:01:51.180 Carefully craft a command once, run it forever with auto-restart, URL detection, and a full 00:01:51.300 --> 00:01:51.620 CLI. 00:01:51.960 --> 00:01:55.100 Download it for free at talkpython.fm/command book app. 00:01:56.040 --> 00:01:59.200 Hello, David. Welcome to Talk Python To Me. Amazing to have you here. 00:01:59.760 --> 00:02:03.500 I'm glad to be here. Talk Python has been part of my story up to this point. 00:02:03.760 --> 00:02:09.220 Has it? Okay. Well, you are about to write the next chapter in the story. So that's pretty excellent. 00:02:10.020 --> 00:02:14.800 I have a sense of what's coming. We planned out what we're going to talk about and that sort of thing. 00:02:15.200 --> 00:02:20.420 And I'm really excited about this topic. So it's going to be a good one. 00:02:21.060 --> 00:02:34.860 Honestly, I think one of the real powers of the Python community and the reason the language has such staying power is there's such a diversity of use cases, technology, like technology standpoints, right? 00:02:34.870 --> 00:02:44.740 Like I build software for this group or I build these types of apps and it's not just, you know, like Ruby on Rails, which, you know, it's been very popular, but it's, it's for websites, right? 00:02:44.770 --> 00:02:45.280 You know what I mean? 00:02:45.920 --> 00:02:46.780 Yeah, absolutely. 00:02:47.280 --> 00:02:57.260 I mean, web development has dominated my use of it, but my entry into it, which I suppose I'll mention in a moment, was through all those little tools. 00:02:57.660 --> 00:02:58.320 Let's hear it. 00:02:58.930 --> 00:03:00.440 Who are you, David Flood? 00:03:00.490 --> 00:03:03.760 Tell us, introduce yourself real quick and tell us about how you got into it. 00:03:04.480 --> 00:03:09.140 So my background is in music and the humanities. 00:03:09.630 --> 00:03:14.900 I mean, in 2019, I didn't know what Python was or the name of any programming language. 00:03:16.180 --> 00:03:22.280 and I've been doing textual criticism, which is, you know, there's lots of criticisms in the academy. 00:03:22.860 --> 00:03:26.520 This is the one where if you have lots and lots of versions of the same text, 00:03:27.100 --> 00:03:33.280 you are comparing them to work out what the initial text was and like how it changed over time. 00:03:33.760 --> 00:03:35.160 Okay, give us an example. 00:03:35.700 --> 00:03:39.800 Okay, so one of the famous examples, hope I can remember it off the top of my head, 00:03:40.220 --> 00:03:41.600 is from Shakespeare. 00:03:42.460 --> 00:03:44.500 We're all familiar with the line to be or not to be. 00:03:45.020 --> 00:03:52.520 is the question. That is the question. Well, there's a variant of it. One of the early copies 00:03:53.820 --> 00:03:58.900 written by Shakespeare himself has... Somebody's going to be able to type into the chat exactly 00:03:58.960 --> 00:04:03.780 what it is. They'll know this anecdote. But it's something more like, "To be or not to be, I." 00:04:04.080 --> 00:04:09.940 That's the question. And so, which one is the original one? Why did he change it? That's kind 00:04:09.900 --> 00:04:14.900 of one example i work mainly in the in the new testament which is especially complicated because 00:04:15.360 --> 00:04:23.300 no other corpus from ancient history has as many copies of the same text as that corpus does so it's 00:04:23.340 --> 00:04:29.320 quite um quite quite complicated and our techniques have have grown grown because of that and perhaps 00:04:29.560 --> 00:04:37.639 become more advanced than now i mean that many variations over that huge span of time over 00:04:37.660 --> 00:04:42.740 different groups with different, maybe not intentions, but certainly colored by different 00:04:43.060 --> 00:04:47.620 worldviews and philosophies and so on. And yeah, I see the trouble. 00:04:47.920 --> 00:04:53.940 No, yeah. And they were people of the book. So copying it is something that happened a lot. And 00:04:54.160 --> 00:05:01.260 they copied the monks, like the medieval monks copied everything. They copied our Greek classics. 00:05:01.900 --> 00:05:06.840 So that's what I was interested in. And because of the wealth of data that we have, 00:05:07.200 --> 00:05:10.700 Computer tools are more and more important in that field. 00:05:11.020 --> 00:05:17.200 So when I started my PhD in 2019, I knew that I wanted to use some of these cutting-edge tools. 00:05:17.660 --> 00:05:19.260 Some of them may be surprising. 00:05:19.860 --> 00:05:24.100 For example, we've been using phylogenetic software. 00:05:24.480 --> 00:05:35.440 This is software that evolutionary biologists are using or computational biologists are using to track, for example, how COVID strains mutate over time. 00:05:35.680 --> 00:05:36.420 Oh, interesting. 00:05:36.440 --> 00:05:39.800 What they're comparing are the DNA letters. 00:05:40.320 --> 00:05:43.740 And so you have the sequence of letters and you're comparing how those change over time. 00:05:44.000 --> 00:05:48.160 Well, you can swap in textual variants for DNA letters. 00:05:48.600 --> 00:05:55.380 And now we can track how texts change over time and group them into families, things like that. 00:05:56.120 --> 00:05:59.580 It's like a time series, but of words or letters or something. 00:05:59.720 --> 00:06:06.400 Yeah, I mean, yeah, there's lots of important algorithms for comparing 00:06:06.400 --> 00:06:11.720 sequences of things. And so if we can just swap in Greek words and Greek text instead, 00:06:12.480 --> 00:06:16.420 then we can maybe apply it to textual criticism. So I was pretty interested in those things. That 00:06:16.600 --> 00:06:21.380 wasn't actually the method that brought me into it, but something like that, kind of computer 00:06:21.620 --> 00:06:27.760 intensive tools. What I learned is that these tools weren't actually available to me. They 00:06:27.940 --> 00:06:36.380 weren't desktop applications. And for the most part, they weren't public web applications. They 00:06:36.400 --> 00:06:38.000 PyPI or something like that, right? 00:06:38.420 --> 00:06:39.060 Yeah, exactly. 00:06:39.360 --> 00:06:39.460 Exactly. 00:06:39.540 --> 00:06:40.040 Or Java. 00:06:41.180 --> 00:06:43.380 And I needed to glue them together. 00:06:43.780 --> 00:06:49.660 So the long story short on that is during the first year of my PhD, I was picking up Python, 00:06:50.000 --> 00:06:51.880 watching YouTube videos while I was doing the dishes. 00:06:52.800 --> 00:06:57.220 And then the pandemic hit while I was living in Edinburgh in Scotland, probably not far 00:06:57.460 --> 00:06:58.160 from Will McCoogan. 00:06:59.220 --> 00:07:06.360 And so the pandemic gave me the excuse to spend even a few more hours each day picking up these 00:07:06.380 --> 00:07:12.900 new, these new technical skills. And so I did it, I was able to use these advanced tools in my in my 00:07:13.100 --> 00:07:17.440 work. But what was really important to me was sharing, like making that available to my colleagues, 00:07:18.120 --> 00:07:23.860 is I had to I had to move from writing these like bad top to bottom Python scripts into things that 00:07:23.860 --> 00:07:29.500 could be reused by other people. And that led me into the web, because the web is where that's how 00:07:29.500 --> 00:07:35.740 I can share with anybody. It's really wild how much the web is kind of the last bastion of 00:07:36.640 --> 00:07:42.480 app freedom. It's so bizarre because, you know, I've many times told the stories of the insane 00:07:42.900 --> 00:07:48.500 battles of just getting our apps that just playback video of content that's already on the web 00:07:48.860 --> 00:07:54.740 into the app store. I mean, weeks of fighting about the weirdest, most nonsensical things with 00:07:54.860 --> 00:08:01.819 both Google and Apple. But we also now have the Mac platform and the Windows platform very 00:08:01.840 --> 00:08:07.780 aggressively looking for digital code certificates and all sorts of signing and other kinds of proof 00:08:07.920 --> 00:08:12.840 like it you can't even just send somebody an executable anymore it won't run it's it's crazy 00:08:13.120 --> 00:08:18.940 it's it's down to like okay put it on the web i guess that's right i i i played the game of 00:08:19.080 --> 00:08:24.540 distributing desktop apps that's how i did it that's why i initially distributed things um 00:08:25.140 --> 00:08:30.860 and at this point i just require people to install python and then install my desktop app from pypi 00:08:30.880 --> 00:08:33.400 because it's too hard otherwise for me. 00:08:33.820 --> 00:08:36.479 I mean, I could pay for the code signing from Apple 00:08:36.890 --> 00:08:37.599 and do all of that, 00:08:37.740 --> 00:08:40.320 but it's just, it's too much work for the time that I have. 00:08:40.500 --> 00:08:42.140 Yeah, I'm about to do another round of it. 00:08:42.200 --> 00:08:42.979 I'm working on an app 00:08:44.060 --> 00:08:45.680 and my developer account is still active. 00:08:45.880 --> 00:08:47.680 So we might have a fresh round of fun. 00:08:47.820 --> 00:08:49.260 Hopefully it goes through this time. 00:08:50.320 --> 00:08:52.160 Anyway, I do think it's such a challenge. 00:08:52.380 --> 00:08:53.520 And are you leveraging? 00:08:53.940 --> 00:08:55.180 I don't know if the timing was right. 00:08:55.300 --> 00:08:56.300 Like maybe this was too early, 00:08:56.780 --> 00:08:59.740 but these days, are you leveraging things like uvx 00:09:00.060 --> 00:09:03.640 to run, or are you just pip install this thing and then run it? 00:09:04.100 --> 00:09:08.200 Yeah, I haven't updated the readme in a while, so I think it just asks for pip. 00:09:08.740 --> 00:09:14.220 But certainly, if somebody asked me today, I would say, yeah, just install this with uv. 00:09:14.920 --> 00:09:16.260 Because then they don't even need Python. 00:09:16.700 --> 00:09:17.100 Exactly. 00:09:17.420 --> 00:09:17.860 And that's brilliant. 00:09:18.440 --> 00:09:22.900 And that's a really, it is another barrier reduced in distributing these applications, 00:09:23.160 --> 00:09:23.220 right? 00:09:23.300 --> 00:09:28.600 Like, if you can get uv installed on a machine, then you don't even have to say install, just 00:09:28.560 --> 00:09:32.960 The way you run it is uvx my thing and it's all transparent to you, right? 00:09:33.020 --> 00:09:33.520 Which is beautiful. 00:09:33.900 --> 00:09:34.880 So what was it like? 00:09:35.100 --> 00:09:35.320 Yeah. 00:09:35.790 --> 00:09:42.340 So what was it like coming from what sounds like a not super screen focus, super 00:09:43.020 --> 00:09:47.300 techie aspect and having to dive into this world and someday you're probably 00:09:47.420 --> 00:09:49.900 like, how is it that I'm publishing stuff to PyPI? 00:09:49.990 --> 00:09:50.740 What has happened to me? 00:09:51.300 --> 00:09:51.720 Yeah. 00:09:51.970 --> 00:09:56.259 well, yeah, I remember when I, when I first signed up for GitHub, because 00:09:56.320 --> 00:10:02.160 you know, whatever YouTube tutorial I was working through at the time, you know, said that I needed 00:10:02.160 --> 00:10:08.720 to do that. You know, I think it all started making a lot of sense. I didn't have any technical 00:10:08.980 --> 00:10:16.880 background, but the world kind of open source software, it just kind of made sense. It felt 00:10:17.020 --> 00:10:23.820 like it fit really well into my academic, you know, circle. I think a lot of the attitudes are 00:10:23.840 --> 00:10:27.980 similar. I agree. I think they are actually. And I think that's, I think that's a pretty neat thing. 00:10:28.480 --> 00:10:34.240 Yeah. Very cool. All right. Well, let's talk about what you're doing with digital humanities. 00:10:34.760 --> 00:10:40.400 You're actually at a really interesting project or organization, I guess, that does many projects, 00:10:40.600 --> 00:10:45.020 right? Yeah. Yeah. So fast, fast forwarding, I did, I finished my PhD in the humanities. 00:10:45.200 --> 00:10:50.780 Sorry. I had so much fun. No, that's fine. That's fine. I had so much fun writing like these tools 00:10:50.860 --> 00:10:54.120 and then just solving the distribution problem 00:10:54.450 --> 00:10:55.760 to share them with other scholars. 00:10:56.740 --> 00:11:00.120 That was so fun that I was open to this kind of opportunity 00:11:00.770 --> 00:11:01.980 where now I'm doing this full time. 00:11:02.570 --> 00:11:04.200 And so, yes, so I'm on the, 00:11:04.500 --> 00:11:06.720 we call it affectionately Darth, 00:11:07.440 --> 00:11:10.240 which is digital arts and humanities at Harvard. 00:11:11.200 --> 00:11:14.160 There has to be a lot of Star Wars memes and references, 00:11:14.310 --> 00:11:14.680 I'm sure. 00:11:14.980 --> 00:11:16.400 If you can pull up a 404, 00:11:16.770 --> 00:11:19.060 I think there will be a Darth Vader reference. 00:11:19.470 --> 00:11:20.660 Seriously, I'm here for it. 00:11:22.360 --> 00:11:25.680 Yes, page not found. I find your lack of nav disturbing. 00:11:27.660 --> 00:11:33.020 You know what? I think that is beautiful. And I really, I really think that people should embrace 00:11:33.580 --> 00:11:40.860 the 404, the fun 404 page, you know, more, right? There should really be something going on that 00:11:40.900 --> 00:11:45.560 like makes it, you know, something hasn't worked out, but you can just, you can make people laugh. 00:11:46.280 --> 00:11:47.340 Yeah. I appreciate that. 00:11:48.560 --> 00:11:50.360 I've heard people push back against it. 00:11:50.490 --> 00:11:57.900 Like if you're on a, if you're on like your medical website and you're maybe about to get bad news and then you get like a picture of a kitten. 00:12:00.160 --> 00:12:01.880 Dr. Kitten doesn't know where your results went. 00:12:02.020 --> 00:12:02.760 Like I get that. 00:12:02.800 --> 00:12:03.300 That's not funny. 00:12:04.060 --> 00:12:05.780 But I mean, most things are not that serious. 00:12:06.560 --> 00:12:06.780 Yeah. 00:12:07.540 --> 00:12:07.920 Mostly. 00:12:08.780 --> 00:12:08.920 Okay. 00:12:09.180 --> 00:12:11.740 So what kind of things does Darth do? 00:12:12.080 --> 00:12:16.680 You've described this as kind of a web or tech agency within Harvard. 00:12:17.280 --> 00:12:18.200 Yeah, it is very much. 00:12:18.420 --> 00:12:21.740 So, you know, Harvard has a gigantic IT group. 00:12:21.890 --> 00:12:28.080 I don't know how many hundreds of people work, but more than 500 people in IT. 00:12:28.840 --> 00:12:33.000 We are a small team and we operate very much like a small agency. 00:12:33.540 --> 00:12:41.500 So usually what happens is a faculty member has a funded research project that's going to last for an amount of time. 00:12:42.210 --> 00:12:44.640 And then we consult with them to build it. 00:12:44.880 --> 00:12:53.160 And most of the time, I kind of think of these as I kind of have these different categories of these kinds of projects that I think of. 00:12:54.070 --> 00:12:56.060 I lost in my notes what I call them. 00:12:56.170 --> 00:12:57.240 But they are there. 00:12:57.390 --> 00:13:01.160 You have like a one is like a virtual research environment. 00:13:01.450 --> 00:13:07.400 So the focus is this is this is a platform that we're building for the research to be done on. 00:13:07.720 --> 00:13:17.000 Like the reason the research should be done in like a web app would be because you have access to visualization, to Postgres, to Pandas. 00:13:17.170 --> 00:13:23.420 So we can kind of build up this platform to do the actual research on and some of the data entry. 00:13:23.700 --> 00:13:26.060 So like a full on research application. 00:13:26.660 --> 00:13:27.040 Yeah, exactly. 00:13:27.580 --> 00:13:36.040 I guess you can also kind of see your work through the different stages of research projects and academic research and so on. 00:13:36.220 --> 00:13:41.900 And we'll get to maybe end of life in a sense further down in the conversation. 00:13:42.470 --> 00:13:48.680 But so this would be we have a grant or we just work here and we're going to work on some form of research. 00:13:49.210 --> 00:13:49.960 What do you give them? 00:13:50.480 --> 00:13:58.540 Right. And I think that's a super interesting challenge because one of the real common answers would be Jupyter, Jupyter Lab, Marimo, whatever. 00:13:59.130 --> 00:14:05.380 But that's still pretty code heavy for people who are possibly philosophers or something, you know. 00:14:05.800 --> 00:14:12.820 Oh, exactly. That's why in digital humanities, I won't even, maybe I won't even attempt to define 00:14:13.710 --> 00:14:19.820 it in any narrow sense, because I'll get in trouble with somebody. But you have two groups 00:14:20.370 --> 00:14:26.580 that are interfacing with each other. And one is digital humanities as a field, like as a subfield, 00:14:26.800 --> 00:14:31.320 all of its own. And these are people who have humanities domain, like knowledge, 00:14:31.860 --> 00:14:36.680 and technical skills, and they're bringing them together. And in a lot of cases, the audience for 00:14:36.840 --> 00:14:42.480 that kind of work is other people working in the digital humanities. But far more common, 00:14:42.780 --> 00:14:49.220 and this is what we work with, is people who have humanities domain expertise, and they want to 00:14:49.600 --> 00:14:55.560 publish or do research or share with other people who have that same humanities domain expertise, 00:14:55.640 --> 00:14:59.640 and they are now interested in adding a technical component to it. 00:14:59.960 --> 00:15:02.000 How can we supercharge what they have? 00:15:03.500 --> 00:15:06.120 This portion of Talk Python is brought to you by Sentry. 00:15:06.580 --> 00:15:09.680 I've been using Sentry personally on almost every application 00:15:10.060 --> 00:15:12.480 and API that I've built for Talk Python and beyond 00:15:13.260 --> 00:15:14.260 over the last few years. 00:15:14.580 --> 00:15:17.460 They're a core building block for keeping my infrastructure solid. 00:15:18.060 --> 00:15:19.360 They should be for yours as well. 00:15:19.640 --> 00:15:20.020 Here's why. 00:15:20.680 --> 00:15:22.100 Sentry doesn't just catch errors. 00:15:22.200 --> 00:15:24.900 It catches all the stuff that makes your app feel broken, 00:15:25.280 --> 00:15:27.560 the random slowdown, the freeze you can't reproduce, 00:15:28.260 --> 00:15:30.620 that bug that only shows up once real users hit it. 00:15:30.960 --> 00:15:31.820 And when something goes wrong, 00:15:32.180 --> 00:15:34.500 Sentry gives you the whole chain of events in one place. 00:15:34.720 --> 00:15:37.700 Errors, traces, replays, logs, dots connected. 00:15:38.080 --> 00:15:39.900 You can see what's led to the issue 00:15:40.040 --> 00:15:41.880 without digging through five different dashboards. 00:15:42.700 --> 00:15:44.720 SEER, Sentry's AI debugging agent, 00:15:45.200 --> 00:15:47.180 builds on this data, taking the full context, 00:15:47.840 --> 00:15:49.820 explaining why the issue happened, 00:15:50.400 --> 00:15:52.780 pointing to the code responsible, drafts a fix, 00:15:52.880 --> 00:15:55.840 and even flags if your PR is about to introduce a new problem. 00:15:56.680 --> 00:15:57.720 The workflow stays simple. 00:15:58.160 --> 00:15:59.900 Something breaks, Sentry alerts you, 00:16:00.080 --> 00:16:01.880 the dashboard shows you the full context, 00:16:02.220 --> 00:16:05.360 Seer helps you fix it and catch new issues before they ship. 00:16:06.080 --> 00:16:08.920 It's totally reasonable to go from an error occurred 00:16:09.080 --> 00:16:11.140 to fixed in production in just 10 minutes. 00:16:12.200 --> 00:16:14.960 I truly appreciate the support that Sentry has given me 00:16:15.060 --> 00:16:17.520 to help solve my bugs and issues in my apps, 00:16:18.160 --> 00:16:20.740 especially those tricky ones that only appear in production. 00:16:21.100 --> 00:16:22.580 I know you will too if you try them out. 00:16:22.880 --> 00:16:24.520 So get started today with Sentry. 00:16:24.700 --> 00:16:29.720 Just visit talkpython.fm/sentry and get $100 in Sentry credits. 00:16:30.240 --> 00:16:30.960 Please use that link. 00:16:31.060 --> 00:16:32.320 It's in your podcast player show notes. 00:16:32.420 --> 00:16:37.760 If you're signing up some other way, you can use our code talkpython26, all one word, 00:16:38.340 --> 00:16:40.900 talkpython26, to get $100 in credits. 00:16:41.680 --> 00:16:43.360 Thank you to Sentry for supporting the show. 00:16:44.500 --> 00:16:49.319 Maybe just take a moment and speak to, maybe, I don't know if this venue will actually speak 00:16:49.340 --> 00:16:54.200 directly to anybody who I was imagining here, but people who work with folks, what would you tell 00:16:54.340 --> 00:16:58.720 somebody who works with a group who have some technical skill, who could create some of these 00:16:58.880 --> 00:17:02.280 things that we're going to talk about, but the people who they've created for don't necessarily 00:17:02.540 --> 00:17:09.420 think they need it or know that they need it. I've gone often on rants about how programming is a 00:17:09.680 --> 00:17:15.260 superpower, not a replacement for your job, right? Yeah. That's a problem for a lot of people, 00:17:15.360 --> 00:17:20.500 especially because you might use some new computer tools to supercharge your research. 00:17:20.980 --> 00:17:25.740 But the article that you publish or the research output of that, the audience, they may not 00:17:25.860 --> 00:17:27.660 be interested in hearing about that at all. 00:17:28.040 --> 00:17:32.760 And so for most people who are working in this space, the tools, you have to use them 00:17:33.000 --> 00:17:37.720 in such a way that you can talk about the research output without talking about the 00:17:37.860 --> 00:17:38.000 tool. 00:17:38.260 --> 00:17:42.979 And we have other venues to talk about the tools themselves, like the Journal for Open 00:17:43.000 --> 00:17:48.660 source software and you can kind of get some of it out there. But that is a, that's the significant 00:17:48.880 --> 00:17:53.020 challenge is convincing people that it, that it could be useful and then convincing the audience 00:17:53.230 --> 00:17:57.800 that they should be interested in kind of the methods behind how some of the new research comes 00:17:57.800 --> 00:18:02.940 up. Also, I think I'm a big believer that presenting stuff in the right order is really, 00:18:03.130 --> 00:18:07.620 really important. If you present your research and it's beautiful and powerful and oh, look, 00:18:07.760 --> 00:18:12.500 we've also, by the way, covered a hundred times more data than any prior research. Surprise, 00:18:12.760 --> 00:18:13.520 I wonder how I did that. 00:18:14.160 --> 00:18:15.400 And then people are like, this is amazing. 00:18:16.580 --> 00:18:19.580 Then after you kind of hook them with the inspiration and what's possible, 00:18:19.680 --> 00:18:21.480 then you're like, let me tell you about the tool. 00:18:21.600 --> 00:18:22.860 And all of a sudden you're like, that's a cool tool, right? 00:18:22.920 --> 00:18:26.100 This is not just like geekery, like programmer, you know, 00:18:26.440 --> 00:18:28.120 Charlie Brown speak, wah, wah, wah, wah, wah. 00:18:28.300 --> 00:18:29.660 You know, it's like, no, I'm listening. 00:18:29.880 --> 00:18:30.460 Tell me now. 00:18:30.820 --> 00:18:31.260 Yeah, exactly. 00:18:31.620 --> 00:18:34.960 I mean, one of the things I think that really opens people's eyes 00:18:35.300 --> 00:18:37.720 is a really powerful search interface. 00:18:38.260 --> 00:18:39.740 You have all of this research data. 00:18:40.120 --> 00:18:45.020 just put it behind Elasticsearch with some really good filtering on it. And all of a sudden you have 00:18:45.180 --> 00:18:50.740 fast, rapid access to the data in a way you never had before. Like you were never scrolling through 00:18:51.140 --> 00:18:55.160 the Excel spreadsheets and finding exactly what you wanted, like you were with this new search 00:18:55.400 --> 00:19:00.360 interface. And that by itself is like so simple. We're so used to that in web development that 00:19:00.480 --> 00:19:05.099 like everything needs to have a fantastic search now. But so many people have their data locked 00:19:05.120 --> 00:19:07.500 behind, you know, a terrible search interface. 00:19:07.960 --> 00:19:10.400 Yeah, just a few things to sort of expose that. 00:19:10.500 --> 00:19:14.880 So this, give us a sense of what these data exploration web apps might look like. 00:19:14.940 --> 00:19:20.060 These are probably kind of mostly stuck to the inside, kind of internal to the research 00:19:20.540 --> 00:19:22.820 lab research team groups and so on. 00:19:22.960 --> 00:19:24.720 These are probably not that public facing, right? 00:19:24.980 --> 00:19:28.580 Almost everything we work on does end up having a public facing component. 00:19:28.940 --> 00:19:33.960 So maybe the research itself is done, locked behind a user login. 00:19:34.300 --> 00:19:35.440 That's just for the researchers. 00:19:36.290 --> 00:19:38.880 But then they expose that research to the public, 00:19:39.520 --> 00:19:41.080 usually with a good search interface 00:19:41.640 --> 00:19:44.840 and different pages for exploring their data 00:19:45.020 --> 00:19:47.200 and visualizations and things like that. 00:19:47.380 --> 00:19:49.360 So yeah, everything we do ends up becoming 00:19:49.850 --> 00:19:52.560 a production public web app in the end. 00:19:52.760 --> 00:19:54.740 And then another one of your categories, 00:19:54.830 --> 00:19:57.000 you put it was virtual research environments 00:19:57.260 --> 00:19:59.740 like data entry, publishing, authoring, collaboration. 00:20:00.050 --> 00:20:00.540 Tell us about that. 00:20:01.280 --> 00:20:03.139 Yeah, so a good example of this maybe 00:20:03.160 --> 00:20:08.820 is one of the projects that... Well, actually, the best example of it is the project I worked on 00:20:08.930 --> 00:20:16.380 during my PhD. It's called Apatosaurus. The short story behind the name is that it sounds like 00:20:16.540 --> 00:20:24.280 apparatus. In textual criticism, when you are displaying and visualizing variant readings to 00:20:24.710 --> 00:20:31.959 a base text, that form of visualizing it is a critical apparatus. A critical apparatus is a 00:20:32.040 --> 00:20:37.500 a pretty boring website name, but Apatosaurus dinosaurs might make textual criticism sound fun. 00:20:37.720 --> 00:20:43.180 Yeah, I do love dinosaurs. No, that's really cool. So this, this comes out as a web app. And I know 00:20:43.180 --> 00:20:46.160 you also have some, you talked about some desktop apps as well. 00:20:46.640 --> 00:20:50.460 Yep. Yep. That's right. So, yeah. So, so there's this people, people upload their, 00:20:50.550 --> 00:20:54.960 their collation to this and then they can visualize it. And like there, there's a public 00:20:55.440 --> 00:21:00.320 component of this as well, but really the backend is editing, editing a collation, 00:21:00.500 --> 00:21:03.160 and adding notes to all of the different readings and stuff. 00:21:03.560 --> 00:21:07.060 So I could show what the backend looks like, 00:21:07.280 --> 00:21:08.200 but we can also move on. 00:21:08.440 --> 00:21:11.420 - Let's move on just because most people 00:21:11.620 --> 00:21:14.540 will not totally hear, but just give us a sense of like, 00:21:14.880 --> 00:21:18.740 like what do people, what do you create for people 00:21:18.900 --> 00:21:21.660 so that they're like, yeah, I can use this app, right? 00:21:21.760 --> 00:21:23.440 Like give us a sense of some of the features, 00:21:23.660 --> 00:21:24.960 I guess is what I'm getting to. 00:21:25.260 --> 00:21:29.319 - Yeah, so another good example is we have a project 00:21:29.360 --> 00:21:32.380 at Harvard called Mapping Color in History. 00:21:33.150 --> 00:21:36.980 And this is a collaboration with a lab. 00:21:37.150 --> 00:21:38.900 This lab brings in pieces of artwork 00:21:39.480 --> 00:21:42.560 and they do spectral analysis on the pigments 00:21:42.590 --> 00:21:45.160 so they can identify what was used 00:21:45.160 --> 00:21:48.360 to make a particular color of this red 00:21:48.550 --> 00:21:50.860 or what was made to make this color of blue. 00:21:51.420 --> 00:21:53.640 And then the idea is tracking 00:21:54.000 --> 00:21:56.880 how did people make those pigments over time, 00:21:57.280 --> 00:22:01.720 over time and specifically in Asian art. 00:22:02.260 --> 00:22:04.380 Is this the Dharmra, Puna, Puna? 00:22:05.440 --> 00:22:07.960 No, this is mapping color in history. 00:22:08.030 --> 00:22:09.640 I don't think it's up here. 00:22:09.770 --> 00:22:10.280 Sorry about that. 00:22:10.420 --> 00:22:10.680 Somewhere. 00:22:10.940 --> 00:22:11.300 That's all right. 00:22:11.330 --> 00:22:11.840 I'll find it. 00:22:12.030 --> 00:22:12.440 Keep talking. 00:22:13.680 --> 00:22:13.840 Okay. 00:22:14.050 --> 00:22:16.300 So the front end is great. 00:22:16.510 --> 00:22:18.040 You know, like the public end, 00:22:18.180 --> 00:22:21.000 this is people can explore by pigments 00:22:21.280 --> 00:22:24.440 and then see the images that contain those pigments. 00:22:24.560 --> 00:22:30.680 Now in the back end, what the researchers will be able to do is correlate exactly which 00:22:30.960 --> 00:22:34.260 point of a painting the analysis was done on. 00:22:34.490 --> 00:22:38.640 So they have this deep zoom image viewer where they'll zoom in and they'll select the point 00:22:39.390 --> 00:22:40.280 where that was taken from. 00:22:41.090 --> 00:22:47.640 So how else would you do that other than a digital interface to indicate on an image of 00:22:47.950 --> 00:22:52.060 a painting where that spectral analysis was performed? 00:22:52.380 --> 00:22:55.020 Sounds almost like astronomy in a weird way. 00:22:55.050 --> 00:22:55.300 Oh, yeah. 00:22:55.840 --> 00:23:04.580 We zoomed into here and we took a different spectrum of the painting and we realized that it's actually identical to this, you know, something crazy like that, right? 00:23:04.900 --> 00:23:06.140 Yeah, yeah, yeah, that's right. 00:23:06.200 --> 00:23:08.560 Yeah, so it's essentially a pigments, like a pigments database. 00:23:10.100 --> 00:23:17.940 So the third category of these digital humanities projects that you put down was like data extraction, transformation. 00:23:19.260 --> 00:23:29.540 In data science, they often say, you know, 80% of the work is the data wrangling, which is like cleaning, organization, just getting it so you could possibly start asking questions about it. 00:23:29.820 --> 00:23:30.960 I'm sure you all do a lot of that. 00:23:31.180 --> 00:23:31.440 Absolutely. 00:23:32.560 --> 00:23:40.880 So often, the very beginning of a project might be an Excel sheet or several spreadsheets. 00:23:41.800 --> 00:23:46.020 And the first task is to ingest these into, you know, a proper database. 00:23:46.640 --> 00:23:48.600 Not so much MongoDB for us. 00:23:48.760 --> 00:23:49.840 It's going into Postgres. 00:23:50.340 --> 00:23:51.420 We're Django Shop. 00:23:51.680 --> 00:23:52.480 We're Django Shop. 00:23:52.630 --> 00:23:53.760 So it's going into Postgres. 00:23:55.090 --> 00:24:06.560 And yeah, no, that is probably the number one challenge of the early stage is figuring out what the right data model is, what the right relationships are to model the data. 00:24:07.300 --> 00:24:17.020 Doing that work is advantageous to everybody because, you know, it helps both the researchers who brought the data to think about it in a more organized way. 00:24:17.410 --> 00:24:18.540 I mean, they've been trying to do that. 00:24:18.720 --> 00:24:19.680 And they have the spreadsheets. 00:24:20.160 --> 00:24:27.800 But now we're modeling out the data so that we can add it to database tables and then to use later. 00:24:27.880 --> 00:24:29.340 So that works out well for everybody. 00:24:30.000 --> 00:24:30.720 And yeah, absolutely. 00:24:31.100 --> 00:24:45.760 Cleaning the data, getting dates, working with fuzzy dates, being able to parse July of 2020 or summer of 2020 and handling kind of all of those cases so that we do get dates in the end. 00:24:45.780 --> 00:24:55.980 One of the crazy stories from data parsing history is one of the, I can't remember exactly what it was, you talked about biology tools or genetics tools earlier. 00:24:56.100 --> 00:25:03.780 One of the groups that names genes had to change the name of a gene because it kept getting parsed by Excel into a date. 00:25:04.880 --> 00:25:05.520 Yeah, I remember that. 00:25:05.590 --> 00:25:06.200 I remember that. 00:25:06.320 --> 00:25:06.600 That's right. 00:25:07.260 --> 00:25:07.580 Yes. 00:25:08.100 --> 00:25:10.580 So these are the weird edge cases I'm sure you run into. 00:25:11.940 --> 00:25:13.120 Like it's not even supposed to be a date. 00:25:13.220 --> 00:25:13.940 Why is this a date? 00:25:13.990 --> 00:25:14.780 I don't know. 00:25:14.940 --> 00:25:16.240 Why is it helping out here? 00:25:16.920 --> 00:25:17.820 The code keeps crashing. 00:25:18.000 --> 00:25:20.480 Like pandas parsed it as a date and it's not or whatever. 00:25:21.220 --> 00:25:21.540 Absolutely. 00:25:21.980 --> 00:25:22.060 Yeah. 00:25:22.140 --> 00:25:22.300 Yeah. 00:25:22.340 --> 00:25:27.320 So yeah, usually lots of test suites around that ingest process until we've got it. 00:25:27.640 --> 00:25:32.220 Now, once we've got it in, usually the research is ongoing and then we're able to provide 00:25:32.420 --> 00:25:38.140 them now a new cleaned interface to do the additional data entry as the project is going. 00:25:38.420 --> 00:25:39.780 And that's usually a win-win for everybody. 00:25:40.180 --> 00:25:40.340 Sure. 00:25:40.620 --> 00:25:45.780 And so this sort of ETL ingestion side of everything is it's like, don't worry, 00:25:46.420 --> 00:25:47.460 Darth has got it for you. 00:25:47.760 --> 00:25:51.180 And then we'll provide you like a database connection to start working. 00:25:51.480 --> 00:25:54.700 Or do you give them the tools and then they kind of iterate on them? 00:25:54.940 --> 00:26:00.220 And how much is this you and how much is this you providing like CLI tools and stuff 00:26:00.460 --> 00:26:01.540 or notebooks over to people? 00:26:03.560 --> 00:26:08.520 I'd say most of the people that we're working with are aware of the technical tools, 00:26:08.640 --> 00:26:10.380