WEBVTT 00:00:00.020 --> 00:00:03.060 How many files does your query read before it reads any data? 00:00:03.500 --> 00:00:07.360 On some data lakes, you go through JSON and metadata files first, 00:00:07.720 --> 00:00:10.000 just to learn which Parquet files actually matter. 00:00:10.700 --> 00:00:13.180 Duck Lake asks one SQL question instead. 00:00:13.860 --> 00:00:15.300 The metadata lives in a real database. 00:00:15.700 --> 00:00:17.460 The data stays in plain Parquet. 00:00:17.980 --> 00:00:19.060 That's the entire format. 00:00:19.660 --> 00:00:22.680 Pedro Holanda joined DuckDB in 2018, 00:00:23.320 --> 00:00:25.700 and it was still a research prototype at CWI. 00:00:26.080 --> 00:00:27.820 He's the lead Duck Lake developer. 00:00:28.320 --> 00:00:32.900 And Guillermo Sanchez-Doinas works on DuckLake and the new Quack protocol. 00:00:33.580 --> 00:00:38.900 With Quack as the catalog, DuckLake handles 200 transactions per second under heavy contention. 00:00:39.320 --> 00:00:41.600 No other open table format even comes close. 00:00:42.320 --> 00:00:48.580 This is Talk Python To Me, episode 562, recorded August 31st, 2026. 00:01:06.560 --> 00:01:10.880 Welcome to Talk Python To Me, the number one Python podcast for developers and data scientists. 00:01:11.400 --> 00:01:16.720 This is your host, Michael Kennedy. I'm a PSF fellow who's been coding for over 25 years. 00:01:17.440 --> 00:01:18.460 Let's connect on social media. 00:01:18.540 --> 00:01:21.940 You'll find me and Talk Python on Mastodon, Bluesky, and X. 00:01:22.240 --> 00:01:24.060 The social links are all in your show notes. 00:01:24.820 --> 00:01:28.340 You can find over 10 years of past episodes at talkpython.fm. 00:01:28.520 --> 00:01:31.720 And if you want to be part of the show, you can join our recording live streams. 00:01:32.040 --> 00:01:32.540 That's right. 00:01:32.730 --> 00:01:36.000 We live stream the raw uncut version of each episode on YouTube. 00:01:36.600 --> 00:01:41.020 Just visit talkpython.fm/youtube to see the schedule of upcoming events. 00:01:41.210 --> 00:01:44.880 Be sure to subscribe there and press the bell so you'll get notified anytime we're recording. 00:01:45.880 --> 00:01:50.900 This episode is brought to you by Six Feet Up, the Python and AI experts who solve hard 00:01:51.240 --> 00:01:51.660 software problems. 00:01:52.420 --> 00:01:57.380 Whether it's scaling an application, deriving insights from data, or getting results from 00:01:57.520 --> 00:02:00.740 AI, Six Feet Up helps you move forward faster. 00:02:01.440 --> 00:02:02.880 See what's possible with Six Feet Up. 00:02:03.180 --> 00:02:06.000 Visit talkpython.fm/sixfeetup. 00:02:06.600 --> 00:02:08.179 And it's brought to you by us. 00:02:08.720 --> 00:02:11.900 Talk Python and Python Bytes both now have MCP servers. 00:02:12.480 --> 00:02:16.880 Point your AI at 10 plus years of Python episodes, transcripts, and show notes. 00:02:17.300 --> 00:02:17.440 Free. 00:02:17.840 --> 00:02:21.620 Click MCP in the nav at talkpython.fm and at Python Bytes. 00:02:22.500 --> 00:02:23.780 Guys, welcome to the show. 00:02:24.260 --> 00:02:25.260 Awesome to have you here. 00:02:25.700 --> 00:02:26.660 Yeah, awesome to be here. 00:02:27.040 --> 00:02:27.900 Absolutely a pleasure. 00:02:28.760 --> 00:02:29.500 It is a pleasure. 00:02:30.340 --> 00:02:31.480 I'm a big fan of databases. 00:02:31.880 --> 00:02:38.880 I think databases unlock so much potential for software and also data science, especially, 00:02:39.060 --> 00:02:41.000 you know, talking DuckDB side of things. 00:02:41.240 --> 00:02:46.920 But once you really get good with databases, you can just answer questions so well, so quickly. 00:02:47.220 --> 00:02:50.880 And honestly, it kind of makes the other side of programming easy. 00:02:51.140 --> 00:02:58.740 So I'm interested to see what you all are doing with Duck Lake, do some refreshers on DuckDB and all of these things. 00:02:59.200 --> 00:02:59.720 Let's roll. 00:03:00.240 --> 00:03:00.880 Let's do it. 00:03:01.400 --> 00:03:01.920 Let's roll. 00:03:02.100 --> 00:03:02.320 All right. 00:03:02.720 --> 00:03:05.020 Well, Pedro, let's have you kick us off. 00:03:05.160 --> 00:03:09.500 Before we get into that, just to give everyone a bit about your background, introduce yourself. 00:03:09.920 --> 00:03:10.320 All right. 00:03:10.720 --> 00:03:12.800 So, well, my name is Pedro. 00:03:13.940 --> 00:03:17.060 I came to the Netherlands, I think, about nine years ago 00:03:17.190 --> 00:03:21.400 to do my PhD in the Database Architectures Group at CWI. 00:03:22.100 --> 00:03:24.600 This is also where I met both Hannes and Mark, 00:03:24.930 --> 00:03:27.540 who are the co-creators of DuckDB. 00:03:27.920 --> 00:03:30.580 I actually used to live together with Mark, 00:03:30.850 --> 00:03:33.840 and Hannes was also my co-supervisor, 00:03:34.120 --> 00:03:36.780 so the connection has been there pre-DuckDB, in a way. 00:03:38.260 --> 00:03:45.920 And yeah, I did my PhD at CWI and towards the end of my PhD, I had already finished it. 00:03:45.980 --> 00:03:47.260 The book was already being printed. 00:03:47.400 --> 00:03:49.800 I still have six or nine months to go. 00:03:49.900 --> 00:03:54.000 So I just came to market and was like, hey, you guys have this cool research prototype. 00:03:54.600 --> 00:03:55.740 Can I do something in it? 00:03:56.140 --> 00:04:01.660 And basically start nerding out about it and be having tons of fun since then. 00:04:03.120 --> 00:04:12.280 Funny enough, one of the first things I did in DuckDB, I think this is 2018, maybe, like literally the first month that I was in was the CSV reader. 00:04:12.620 --> 00:04:15.080 And I came back to it a few years later. 00:04:15.360 --> 00:04:18.560 And I'm actually currently working on something that's CSV reader as well. 00:04:18.579 --> 00:04:20.480 So it's my true passion, I guess. 00:04:21.260 --> 00:04:23.480 You were born to work on CSV. 00:04:24.860 --> 00:04:25.320 It seems. 00:04:26.080 --> 00:04:28.920 It's also known as the unsexiest database problem. 00:04:29.020 --> 00:04:30.260 But what can I say? 00:04:31.300 --> 00:04:40.900 Well, I think it's a subset of the unsexiest data science problem, but also the biggest is the data wrangling, plant cleaning, all that kind of stuff. 00:04:41.020 --> 00:04:49.120 And CSV is a fairly raw format that doesn't actually communicate a lot of stuff about what types it actually intends that column to be and so on. 00:04:49.420 --> 00:04:55.680 The challenge, of course, not only being efficient about it, but also it's a pretty crazy lens. 00:04:55.980 --> 00:04:58.180 I call it like the Wild West of data formats. 00:04:58.780 --> 00:05:01.940 So being able to actually read these things, it's challenging. 00:05:02.380 --> 00:05:07.300 And I guess, well, also the main reason I've been invited to here 00:05:07.400 --> 00:05:10.840 is because I've also been working in Duck Lake for the past year. 00:05:11.710 --> 00:05:12.380 Oh, yeah, that. 00:05:13.480 --> 00:05:16.740 I'll dare to say that I'm currently the main Duck Lake developer 00:05:17.440 --> 00:05:19.520 from our team as well for the past year. 00:05:19.740 --> 00:05:21.680 So I might know a little bit about it. 00:05:22.280 --> 00:05:24.080 Yeah, that is actually why I invited you. 00:05:24.080 --> 00:05:26.120 I mean, CSPs are interesting, don't get me wrong. 00:05:27.740 --> 00:05:33.240 I have a plan of having CSV files replacing parquet in Duck Lake, and then that's a big comeback, you know? 00:05:33.960 --> 00:05:34.380 Oh, yeah. 00:05:34.600 --> 00:05:38.640 Just zipped up compressed CSV. 00:05:39.760 --> 00:05:40.720 Just give it a new extension. 00:05:40.980 --> 00:05:41.580 Nobody will know. 00:05:41.920 --> 00:05:42.440 Yeah, exactly. 00:05:42.740 --> 00:05:42.920 Exactly. 00:05:43.940 --> 00:05:48.480 So it sounds like you were there from early, early days with DuckDB. 00:05:48.910 --> 00:05:52.100 Did it come out of the university there, or was it a project after? 00:05:54.160 --> 00:05:56.540 So it came out from CWI. 00:05:56.680 --> 00:05:57.960 CWA is not actually a university. 00:05:58.260 --> 00:06:00.780 It's, in fact, a research center. 00:06:01.200 --> 00:06:02.600 So this is also kind of a fun thing. 00:06:02.720 --> 00:06:03.820 If you're doing a PhD there, 00:06:04.140 --> 00:06:06.740 you need to also be affiliated to a university to get a diploma. 00:06:07.380 --> 00:06:08.760 But you don't really step on the university. 00:06:08.860 --> 00:06:10.340 You just stay at the research center. 00:06:10.740 --> 00:06:15.200 But the story goes that we had a bunch of projects 00:06:15.460 --> 00:06:17.080 with different companies as well during our PhD. 00:06:17.260 --> 00:06:18.640 So I was working with Honda. 00:06:19.200 --> 00:06:20.720 Mark was working with Dottis too. 00:06:20.940 --> 00:06:22.660 And these were more like data science-y projects. 00:06:23.220 --> 00:06:25.440 And one thing that was very clear to us, 00:06:25.740 --> 00:06:28.740 And Hundes also had a lot of influence in the R community. 00:06:29.060 --> 00:06:31.060 And I think that one thing that was clear to everyone 00:06:31.240 --> 00:06:33.800 was that everyone was trying to run away 00:06:34.080 --> 00:06:35.200 from database systems, right? 00:06:35.340 --> 00:06:38.060 Like you would see people using data wrangling tools, 00:06:38.340 --> 00:06:40.020 data frame tools, things like that, 00:06:40.240 --> 00:06:42.040 but no one wanted to use a Postgres 00:06:42.140 --> 00:06:44.540 or things like similar to that. 00:06:44.860 --> 00:06:47.040 So I think like Mark and Hundes noticed that gap, 00:06:47.300 --> 00:06:47.720 noticed that like, 00:06:47.840 --> 00:06:49.780 why are people not using database systems 00:06:50.040 --> 00:06:51.720 and started talking to these data scientists 00:06:52.000 --> 00:06:54.279 and quickly realized there was a lot of frustration 00:06:54.700 --> 00:06:56.320 in setting things up, running queries. 00:06:56.650 --> 00:06:59.700 Like I remember that during my PhD, 00:07:00.000 --> 00:07:01.620 anything that I wanted to do with like MonitDB 00:07:01.800 --> 00:07:03.660 or Postgres would take me just like a day 00:07:03.900 --> 00:07:05.820 to build things from the sources 00:07:06.100 --> 00:07:07.320 and set up a database system. 00:07:07.660 --> 00:07:09.400 And we might take it for granted nowadays 00:07:09.740 --> 00:07:12.900 that you can just like bootstuckDB up in half a second 00:07:13.160 --> 00:07:15.560 and in one line you are querying something. 00:07:15.840 --> 00:07:19.140 But yeah, like 10 years ago or nine years ago, 00:07:19.300 --> 00:07:20.680 this was very much not the reality. 00:07:21.020 --> 00:07:22.159 So they realized the gap 00:07:22.320 --> 00:07:25.820 and started doing this research database system, 00:07:25.960 --> 00:07:28.840 let's say, more of like, yeah, towards academia. 00:07:29.880 --> 00:07:31.500 And I think at some point they had the realization 00:07:31.840 --> 00:07:33.520 that a lot of people were taking it quite seriously. 00:07:34.100 --> 00:07:36.980 And a lot of people were already trying it. 00:07:38.380 --> 00:07:39.860 This is also the funny thing. 00:07:40.040 --> 00:07:41.200 People try things in production, 00:07:41.370 --> 00:07:43.160 even though it's not yet to release. 00:07:44.140 --> 00:07:45.120 Same thing with DuckLake. 00:07:45.600 --> 00:07:47.240 We had people use it in production 00:07:47.460 --> 00:07:48.560 before DuckLake 1.0. 00:07:48.720 --> 00:07:50.720 And I was always like, cool, cool, cool. 00:07:51.700 --> 00:07:52.460 What could go wrong? 00:07:53.900 --> 00:07:54.960 So they basically realized... 00:07:54.960 --> 00:07:56.500 Yeah, but when you see things that are exciting, 00:07:56.720 --> 00:07:58.780 you're like, this is going to solve our problem, right? 00:07:59.080 --> 00:08:00.400 It's a good sign you're onto something. 00:08:01.020 --> 00:08:01.700 No, no, absolutely. 00:08:02.040 --> 00:08:05.700 I think they were super excited about it 00:08:06.540 --> 00:08:07.540 when having this realization. 00:08:08.820 --> 00:08:10.120 And same thing for us with Duck Lake. 00:08:10.180 --> 00:08:13.660 It was like, okay, people are taking this seriously enough 00:08:13.780 --> 00:08:15.840 that they're starting to bet their company on it. 00:08:15.980 --> 00:08:17.580 So that's cool, definitely. 00:08:18.180 --> 00:08:19.620 Always a little bit scary, but cool. 00:08:20.880 --> 00:08:23.140 And yeah, from that on, I think it just took off. 00:08:23.880 --> 00:08:27.860 Initially, I think it was just the three of us, like Mark and Hannes as the creators. 00:08:28.060 --> 00:08:30.040 And I was kind of like the first person to be hired. 00:08:30.580 --> 00:08:32.599 But then it quickly expanded. 00:08:32.960 --> 00:08:35.479 I think in the first year, we're like six people. 00:08:35.659 --> 00:08:37.700 We're now, I think, 35 maybe. 00:08:39.060 --> 00:08:40.940 So yeah, it has been quite a journey. 00:08:41.919 --> 00:08:42.460 Very exciting. 00:08:42.520 --> 00:08:44.180 It's cool to see it gaining traction. 00:08:44.560 --> 00:08:49.840 Now, quickly here, you also worked on progressive indexes. 00:08:49.980 --> 00:08:51.340 Is this for your dissertation? 00:08:51.880 --> 00:08:55.780 If I'm being honest, my plan was to actually work on CSV files during my P2D, 00:08:56.020 --> 00:08:58.320 but I was told that was a solved problem. 00:08:59.939 --> 00:09:03.040 So I went something else, which was progressive indexes. 00:09:04.280 --> 00:09:08.320 The basic gist of progressive indexes is that you can create an index while 00:09:08.600 --> 00:09:13.020 curing the data instead of creating up front and having the downtime and whatnot. 00:09:14.260 --> 00:09:19.140 But it's actually quite difficult to implement it in practice in the database system. 00:09:19.500 --> 00:09:23.380 So in my thesis, I even have a chapter called The Elephants in the Room, where I basically 00:09:23.640 --> 00:09:24.940 diss all my work. 00:09:25.320 --> 00:09:30.520 Now, I point out why it's complicated to implement in production and things that I would personally 00:09:30.760 --> 00:09:33.500 look into if I wanted to still continue in that direction. 00:09:34.720 --> 00:09:36.040 But it was also quite a fun journey. 00:09:36.320 --> 00:09:41.420 And then you have the image of the website, and there's my little cover. 00:09:41.790 --> 00:09:45.620 And my little cover is like a tree eating all these database systems and growing from 00:09:45.820 --> 00:09:47.120 a little tree to a bigger tree. 00:09:48.620 --> 00:09:49.520 It is, isn't it? 00:09:49.760 --> 00:09:50.180 Very cool. 00:09:51.240 --> 00:09:52.360 Guillermo, how about you? 00:09:52.820 --> 00:09:53.360 Welcome to the show. 00:09:53.720 --> 00:09:53.960 Yes. 00:09:54.220 --> 00:09:54.680 Thanks, Michael. 00:09:55.140 --> 00:09:56.860 My background is a little bit different from Pedro. 00:09:57.060 --> 00:10:02.140 I actually was working as a software and data engineer for quite a while before I joined 00:10:02.300 --> 00:10:02.560 Activity. 00:10:03.780 --> 00:10:09.120 And actually, I come from the experience of running some of these beefy systems. 00:10:09.620 --> 00:10:15.460 I think Pedro mentioned the Monet TV or Postgres, but also the BigQuery, Statorix, Snowflakes, 00:10:15.620 --> 00:10:18.840 which is like, let's say, a second generation kind of system 00:10:18.970 --> 00:10:22.600 where the experience is quite good, but the cost is quite expensive. 00:10:23.320 --> 00:10:29.240 And, I mean, one of the things actually that prompted me to join MDD 00:10:29.320 --> 00:10:31.200 was that Tag Lake was released, actually. 00:10:31.680 --> 00:10:38.760 And I listened to this podcast that announced Tag Lake with Hannes and Mark. 00:10:39.220 --> 00:10:45.580 And, yeah, just a lot of it really resonated with what I thought was maybe 00:10:45.600 --> 00:10:49.000 slightly wrong from other OpenTable formats. 00:10:50.200 --> 00:10:51.760 I just really like the simplicity of it. 00:10:51.800 --> 00:10:53.920 I was a fan of DuckDB for quite a while now 00:10:54.060 --> 00:10:57.620 because if you're interested in reducing the cost of your stack, 00:10:57.780 --> 00:10:59.880 it was probably the best tool out there by far. 00:11:00.800 --> 00:11:03.980 So then I actually wrote an email to Hannes and Mark directly, 00:11:04.460 --> 00:11:06.260 saying like, hey, I want to work with you guys. 00:11:07.960 --> 00:11:10.540 Yeah, I initially joined as a DevRel, actually, 00:11:10.840 --> 00:11:14.480 and then got ready to work in DuckLake together with Pedro. 00:11:14.840 --> 00:11:20.940 But then as time evolved, I became like this kind of like hybrid role where I was a bit of like doing product management things. 00:11:21.090 --> 00:11:26.320 And also I contribute to Daclex sometimes and also to Quack, which is our new client server protocol. 00:11:26.800 --> 00:11:28.300 So we're not only embedded anymore. 00:11:29.920 --> 00:11:33.680 But yeah, so that's kind of the history of how I got to Daclabs. 00:11:34.020 --> 00:11:38.600 Yeah. If someone reaches out to you and says, I heard you talk about this and I believe in your mission. 00:11:38.840 --> 00:11:40.440 Yeah, that's a pretty big endorsement. 00:11:40.580 --> 00:11:42.460 So I can see why they're like, yeah, you have to come work for us. 00:11:42.780 --> 00:11:43.500 Yeah, for sure. 00:11:43.660 --> 00:11:46.560 I mean, I literally told them, like, I'm quitting my job. 00:11:47.010 --> 00:11:48.480 I want to work with you, you know? 00:11:48.520 --> 00:11:50.160 I think it's actually pretty cool what you're doing. 00:11:51.160 --> 00:11:52.360 Hadoop, it's done to me. 00:11:52.620 --> 00:11:53.200 Spark, never. 00:11:54.360 --> 00:11:58.200 No, I was already kind of, I think, done with those technologies. 00:11:58.490 --> 00:12:04.000 Because the things that, for example, if you have Databricks as a platform, right? 00:12:04.180 --> 00:12:09.560 Like, it's such a large thing that, for sure, they give you, like, a lot of, like, nice services. 00:12:09.830 --> 00:12:11.080 Like, you don't have to worry about things. 00:12:11.170 --> 00:12:11.980 And it's serverless. 00:12:12.260 --> 00:12:13.580 but it's also so expensive. 00:12:13.880 --> 00:12:14.800 And I was thinking like, 00:12:14.960 --> 00:12:17.740 there's so much better things out there 00:12:17.800 --> 00:12:18.480 that you can do, 00:12:18.740 --> 00:12:19.960 that you can run yourself, right? 00:12:20.120 --> 00:12:21.240 Like if you're interested in this, 00:12:21.260 --> 00:12:23.580 this is like DacLig and DacDB feel like 00:12:23.760 --> 00:12:27.120 foundational things that you can run any software on. 00:12:27.340 --> 00:12:29.120 And I think this got me really excited. 00:12:29.320 --> 00:12:32.140 You know, in software and deploying systems 00:12:32.640 --> 00:12:32.980 and database, 00:12:33.620 --> 00:12:35.840 there's these things that just layer and layer 00:12:35.940 --> 00:12:37.540 until they just get so complicated. 00:12:38.040 --> 00:12:39.800 And then there's always this alternative, 00:12:39.960 --> 00:12:41.500 like, well, what if we just did a simple thing? 00:12:41.840 --> 00:12:45.200 You know, if we could just make it not so complicated, would it still be working? 00:12:45.250 --> 00:12:47.180 If it did, how amazing would that be? 00:12:47.540 --> 00:12:52.600 So when you say cost, are we talking like operational costs, licensing costs? 00:12:52.720 --> 00:13:00.240 You know, do I need a large cluster of machines so there's always a quorum to vote on different things for durability? 00:13:00.550 --> 00:13:01.720 Or what do you mean by cost? 00:13:02.380 --> 00:13:14.100 I mean, cost is actually, let's say, the service cost that, for example, platforms like Databricks or Snowflake, you know, charge you to use infrastructure that they run, right? 00:13:14.270 --> 00:13:18.420 Because obviously Snowflake and Databricks, they run on AWS or Google Cloud, depending. 00:13:18.510 --> 00:13:20.320 You can even choose your deployment type. 00:13:20.620 --> 00:13:24.480 But obviously, because they provide the service, but they still have to pay for the infrastructure. 00:13:24.650 --> 00:13:27.080 The service is basically something on top of that infrastructure. 00:13:27.580 --> 00:13:29.680 And they decide how much it is, basically. 00:13:30.160 --> 00:13:40.500 And because they're pretty much the bigger players in the market together with existing cloud vendors like GCP, AWS and Azure, they charge quite steep. 00:13:41.040 --> 00:13:44.080 And I mean, of course, like the service is nice. 00:13:44.270 --> 00:13:45.980 Like you don't have to worry about things, right? 00:13:46.070 --> 00:13:48.020 You don't have to worry about downtime and all these things. 00:13:48.300 --> 00:13:51.880 And for sure, they handle the replication and everything that you need. 00:13:52.100 --> 00:14:00.620 But still, though, yeah, if you want control over your stack, this is like a very high level of abstraction that they are offering. 00:14:01.760 --> 00:14:02.060 Yeah. 00:14:02.160 --> 00:14:08.680 You also mentioned open table formats, and that's going to become relevant to our Duck Lake side of things. 00:14:09.310 --> 00:14:11.460 More generally, what are open table formats? 00:14:11.860 --> 00:14:12.020 Yeah. 00:14:12.140 --> 00:14:15.220 So, I mean, they used to be called open table formats. 00:14:15.380 --> 00:14:23.760 And then I think people are leaning more towards this lake house format name, which I'm not sure why. 00:14:23.960 --> 00:14:28.300 I think it's actually maybe because of data rigs, because I think they started calling it the lake house. 00:14:28.340 --> 00:14:30.520 And then everybody's like, yeah, the lake house. 00:14:31.280 --> 00:14:36.820 But yeah, OpenTable formats are basically, I think Pedro correct me if I'm wrong, 00:14:36.880 --> 00:14:44.880 but it's basically just some metadata on some parquet files that you can query in a transactional manner. 00:14:45.320 --> 00:14:50.220 And you can do operations also, atomic operations on this data as well. 00:14:50.510 --> 00:14:54.360 In the most basic case, it's just basically a pointer, 00:14:55.390 --> 00:14:58.620 sorry, a metadata file with a pointer to a list of files 00:14:59.140 --> 00:15:03.200 that all of those files form part of a table. 00:15:03.880 --> 00:15:06.040 And you can also have things like a schema to it, 00:15:06.190 --> 00:15:09.060 like what data types do you expect from this table? 00:15:09.380 --> 00:15:11.860 And you can add more complex things like statistics, 00:15:11.980 --> 00:15:15.760 like, you know, which files contain which data. 00:15:16.280 --> 00:15:18.480 And it can get obviously a lot more complex than that, 00:15:18.640 --> 00:15:20.260 but that's sort of the basics, I think. 00:15:20.460 --> 00:15:23.280 Yeah, and I think like the interesting thing 00:15:23.460 --> 00:15:25.360 of the OpenTable formats is, of course, 00:15:26.280 --> 00:15:27.480 the openness part of it, right? 00:15:27.660 --> 00:15:28.920 So they have a specification, 00:15:29.640 --> 00:15:31.540 the file formats where your data is stored, 00:15:31.740 --> 00:15:32.740 they also have a specification. 00:15:33.280 --> 00:15:35.040 So by just reading the specification, 00:15:35.460 --> 00:15:37.900 anyone should be able to implement a reader 00:15:38.100 --> 00:15:39.680 and writer to that format, right? 00:15:39.780 --> 00:15:43.620 So the whole idea of this, or maybe the beauty of this, 00:15:43.740 --> 00:15:46.160 is that it should be able to solve the problem 00:15:46.360 --> 00:15:49.760 where you're locked in into a certain system, right? 00:15:49.920 --> 00:15:51.380 So you suddenly don't need, 00:15:51.700 --> 00:15:54.560 like if you start off your database from Oracle, for example, 00:15:54.960 --> 00:15:55.840 you're kind of locked in. 00:15:56.120 --> 00:15:58.860 Like doing an Oracle migration is known to be painful, 00:15:59.090 --> 00:16:01.040 to be costly, so you're going to be using, 00:16:01.420 --> 00:16:04.220 you're going to be giving Larry Edson a second island in Hawaii 00:16:04.930 --> 00:16:07.140 in the next few years, that's fine. 00:16:07.620 --> 00:16:11.720 But then if you actually use an open table formats, you kind of solve that problem, right? 00:16:11.740 --> 00:16:15.420 It's like you can literally switch from different engines and different providers. 00:16:15.640 --> 00:16:24.780 And if you're tired of Databricks and now you want to use NoFlake or now you want to use your own solution based on DuckDB, you can theoretically quickly jump from these, right? 00:16:25.140 --> 00:16:37.260 And it's also super scalable because if you're just dealing with file formats or like a catalog as well, but your data is in parquet files, you can easily distribute your reads on it. 00:16:38.040 --> 00:16:41.560 So there's quite some cleverness behind it. 00:16:43.340 --> 00:16:46.120 This portion of Talk Python To Me is brought to you by Six Feet Up. 00:16:46.500 --> 00:16:47.600 Let me ask you a question. 00:16:48.040 --> 00:16:48.860 What's stopping you? 00:16:49.480 --> 00:16:53.860 Maybe it's an application that won't scale or an AI initiative that just isn't delivering. 00:16:54.320 --> 00:16:56.080 That's where Six Feet Up comes in. 00:16:56.660 --> 00:17:01.360 With deep expertise in Python and AI, they solve hard software problems, 00:17:02.080 --> 00:17:04.780 modernize platforms, and get teams to market faster. 00:17:05.220 --> 00:17:07.480 These folks have been doing Python since version one. 00:17:07.870 --> 00:17:10.740 They know the frameworks and ecosystems like the back of their hands. 00:17:11.300 --> 00:17:13.819 Six Feet Up's impact speaks for itself. 00:17:14.439 --> 00:17:18.240 Automated healthcare pipelines for hospitals, helping NASA explore Pluto, 00:17:18.920 --> 00:17:20.560 building severe weather prediction tools, 00:17:20.819 --> 00:17:24.260 and applying AI to connect farmers with vital crop data. 00:17:24.760 --> 00:17:26.600 When the stakes are high and the problems are hard, 00:17:26.920 --> 00:17:28.700 Six Feet Up is the partner that delivers. 00:17:29.320 --> 00:17:30.820 See what's possible with Six Feet Up. 00:17:31.020 --> 00:17:33.580 Visit talkpython.fm/sixfeetup. 00:17:34.020 --> 00:17:36.920 The link is on the episode page and in your podcast player's show notes. 00:17:37.300 --> 00:17:39.060 Thanks to Six Feet Up for sponsoring the show. 00:17:40.220 --> 00:17:43.340 The examples I've seen are like a whole bunch of Parquet files 00:17:44.120 --> 00:17:46.940 on S3 or Azure Blob Storage or something like that. 00:17:47.170 --> 00:17:50.540 And then some other file may be also stored there. 00:17:50.840 --> 00:17:53.540 that you can say, well, let's read the metadata file 00:17:53.560 --> 00:17:54.860 that describes what files are here. 00:17:55.220 --> 00:17:57.080 But I mean, your dream could be back. 00:17:57.100 --> 00:17:59.960 It could be zipped up CSV files stored up there 00:18:00.060 --> 00:18:01.000 with some metadata, right? 00:18:02.419 --> 00:18:03.940 So of course in Duck Lake, 00:18:04.460 --> 00:18:07.600 so maybe one quick thing is that Iceberg, 00:18:07.840 --> 00:18:10.800 which is, I guess, the other main open table formats, 00:18:10.980 --> 00:18:12.540 it uses files all over, right? 00:18:12.680 --> 00:18:15.040 So you have your files in the metadata 00:18:15.060 --> 00:18:17.880 and then your files that actually store the data. 00:18:18.440 --> 00:18:20.500 In Duck Lake, we'll talk a bit more about this. 00:18:20.580 --> 00:18:23.560 but the metadata is actually a database system 00:18:23.840 --> 00:18:25.260 and your files are still parquet files. 00:18:25.640 --> 00:18:28.600 So there's nothing stopping us from a technical point of view 00:18:28.820 --> 00:18:30.840 to actually replacing these files with CSV files. 00:18:31.920 --> 00:18:34.520 From the Duck Lake spec, that's actually already allowed 00:18:34.720 --> 00:18:37.500 because we can store in the catalog that these are CSV files 00:18:38.000 --> 00:18:41.060 and we could theoretically bring the stats necessary. 00:18:41.460 --> 00:18:43.500 So it is an achievable dream. 00:18:44.460 --> 00:18:47.740 Exactly. The dream of the CSVs are still alive. 00:18:48.120 --> 00:18:48.760 All right, really quickly. 00:18:49.040 --> 00:18:53.940 I don't know I'm on the CSV kick, but what's the largest CSV you've ever parsed? 00:18:54.260 --> 00:18:54.600 Oof. 00:18:55.300 --> 00:19:00.740 Personally, I think I've reached close to a terabytes on benchmarks. 00:19:02.200 --> 00:19:07.520 But I've seen people, like, you know, people, they, there were many times in CSV land there 00:19:07.640 --> 00:19:10.700 was like, yeah, this sounds like a rather reasonable limitation. 00:19:11.460 --> 00:19:12.720 No one's ever going to do something. 00:19:13.620 --> 00:19:17.879 I think I had a limitation, for example, that no one would ever have a line in a CSV file 00:19:17.900 --> 00:19:19.500 that was over 32 megabytes. 00:19:19.870 --> 00:19:22.480 And that was kind of nice for me to be able to do parallelism. 00:19:22.710 --> 00:19:24.260 And then, of course, some guy, 00:19:24.900 --> 00:19:28.280 hey, this CSV is throwing an error for me now, 00:19:28.480 --> 00:19:30.100 saying that my line is over 32 megabytes. 00:19:30.150 --> 00:19:31.460 I'm like, Jesus Christ, man. 00:19:32.240 --> 00:19:33.320 What are they storing in there? 00:19:33.320 --> 00:19:35.880 Like a book per line? 00:19:36.720 --> 00:19:36.960 Yeah. 00:19:37.580 --> 00:19:39.780 So if I've done a terabyte, 00:19:40.000 --> 00:19:42.600 I think some people have gone a bit crazier. 00:19:43.380 --> 00:19:45.080 Let's jump into our topic here a bit. 00:19:45.360 --> 00:19:50.160 And I want to work our way, sort of do a little bit of talking about DuckDB. 00:19:50.440 --> 00:19:54.240 And you talked about Quack. 00:19:54.640 --> 00:19:57.880 In the early days, DuckDB was an embedded database. 00:19:58.540 --> 00:20:01.020 You know, the most common one of these is SQLite. 00:20:01.240 --> 00:20:03.400 And Postgres is all the rage, obviously. 00:20:04.180 --> 00:20:04.700 Others as well. 00:20:04.700 --> 00:20:08.920 But Postgres certainly seems to be taking over a lot of the database side of things. 00:20:09.540 --> 00:20:13.320 But I think there's still a bit of a resurgence of embedded databases. 00:20:13.700 --> 00:20:17.100 And I think DuckDB is an example of that interest, right? 00:20:17.400 --> 00:20:21.040 So maybe just you guys give me your thoughts on SQLite 00:20:21.250 --> 00:20:23.220 and some of these other embedded database ideas. 00:20:23.820 --> 00:20:30.020 I just wanted to mention that I think Pedro can give more research background on this, 00:20:30.140 --> 00:20:31.340 but I think from a user perspective, 00:20:31.870 --> 00:20:35.620 I think the idea of having your database within your application system 00:20:35.860 --> 00:20:37.260 was always very attractive to me, right? 00:20:37.520 --> 00:20:38.900 I think Pedro mentioned this at the beginning. 00:20:39.340 --> 00:20:43.260 It's very nice to not have to provision a Postgres database. 00:20:43.880 --> 00:20:47.840 or any other type of database on a separate server to run your application. 00:20:48.420 --> 00:20:50.360 And I think SQL lights off this wonderfully. 00:20:50.940 --> 00:20:53.020 And of course, when DuckDB came by, 00:20:53.100 --> 00:20:57.180 it was also with the idea to solve this from an analytical database perspective. 00:20:57.460 --> 00:20:59.460 But of course, DuckDB can also do transactions, right? 00:20:59.780 --> 00:21:01.760 Pedro can talk about it if he wants. 00:21:01.960 --> 00:21:03.520 But this is also very nice, right? 00:21:03.600 --> 00:21:05.300 But you can still run transactions in DuckDB. 00:21:05.640 --> 00:21:08.760 But overall, I would just be very happy with the idea 00:21:09.140 --> 00:21:12.620 of being able to embed your database system wherever your application is. 00:21:12.740 --> 00:21:15.420 because I think there's a lot of use cases where this is just good enough. 00:21:15.860 --> 00:21:20.220 And actually, one of the coolest use cases I think that DuckDB brings to the table 00:21:20.560 --> 00:21:22.960 is the fact that you can use a Wasm client in the browser 00:21:23.460 --> 00:21:26.660