LanceDB is a developer-friendly open-source database for AI.
It's used by well-known companies such as Midjourney and Character.ai.
On this episode, we have Chang-Chi, the CEO and co-founder of LanceDB,
on to give us a look at the concept of multimodal data
and how you can use LanceDB in your own Python apps.
This is Talk Python to Me, episode 488, recorded November 26, 2024.
Are you ready for your host, here he is!
You're listening to Michael Kennedy on Talk Python to Me.
Live from Portland, Oregon, and this segment was made with Python.
Welcome to Talk Python to Me, a weekly podcast on Python.
This is your host, Michael Kennedy.
Follow me on Mastodon, where I'm @mkennedy,
and follow the podcast using @talkpython,
both accounts over at fosstodon.org,
and keep up with the show and listen to over nine years of episodes at talkpython.fm.
If you want to be part of our live episodes, you can find the live streams over on YouTube.
Subscribe to our YouTube channel over at talkpython.fm/youtube and get notified about upcoming shows.
This episode is brought to you by Sentry.
Don't let those errors go unnoticed.
Use Sentry like we do here at Talk Python.
Sign up at talkpython.fm/sentry.
And this episode is brought to you by Bluehost.
Do you need a website fast?
Get Bluehost.
Their AI builds your WordPress site in minutes,
and their built-in tools optimize your growth.
Don't wait.
Visit talkpython.fm/bluehost to get started.
Hey, Chang.
Welcome to Talk Python to me.
Hey, how are you?
Excited to be here.
I'm excited to have you here.
We're going to talk about LanceDB, multidimensional data, all sorts of fun,
multimodal data, all sorts of fun things.
So really cool database, kind of in the same categories, SQLite, DuckDB.
People want a quick mental model, but different kind of data, right?
Yeah, absolutely.
So I think we specialize in multimodal data or AI data.
And what we think about is included in that is, you know,
the tabular data that we have from previous generations of data engineering, data science,
but also, you know, embedding vectors, images, videos, PDFs,
basically anything that don't really fit neatly into like data frames or Excel sheets.
Yeah.
Less tabular data.
That's right.
If it's not good tabular, then you need something else probably.
Yep.
Awesome.
Which is pretty interesting since I was one of the earliest co-contributors to Pandas.
So I spent years of my life working on data frames and big data.
And now I'm working on-
You've given up on squares and rectangles?
Oh my gosh.
Well, never, but there's room to build something more here.
Yeah, I hear you.
And also, also I agree.
So it's going to be super fun to dive into all of that stuff together.
Before we do though, let's hear your story.
How do you get into programming?
I know this is written in Rust, but also has some programming APIs.
How much Python are you doing these days versus other languages?
Yeah.
The core of Lance format and Lance CB are in Rust.
And then there are Python APIs and wrappers around that, as well as TypeScript.
And the team spends a significant amount of time thinking about what the Python user experience
looks like and how best to combine the performance that you get from Rust and the ease of use
you get from Python.
And then also making it more as extensible as possible so that even though it's a Python library,
if you want to add features, for example, you don't have to learn Rust just to be a contributor
to Lance or Lance DB.
You know, I think Python has benefited a lot from the various Rust-based projects out there,
right?
Obviously, we've got UV and Ruff.
We've got Pydantic, a bunch of other things that make it faster.
But there has been some pushback that if everything gets done in Rust, then it's hard for Python
people to contribute.
I don't necessarily subscribe to that because previously the answer was when things are too slow,
we write them in C.
Maybe it's just traditional Python people also know C, but traditional Python people don't also know Rust.
I think that's honestly the crux of the issue or the crux of the pushback.
So I really like to hear that, you know, the extensibility points and you've thought through
how Python people can extend it and contribute and stuff.
Yeah, I've spent a lot of time writing like Cython and C to make Python code go faster back in the day.
I think you're right.
A lot of the points are the same for Rust.
And actually, Rust is a lot safer, actually.
So when we originally started Lance in the very beginning in C++,
and we spent months writing C++ code for a new format.
And I think it was the Christmas break of 2022 that that's what I call our Rust pill moment was when we decided to switch over.
And I think we ended up, we took about three weeks and we rewrote roughly four or five months of C++ code there.
Overall, I think we actually ended up getting better performance for the most part.
And the biggest thing was just us having the confidence to move forward very quickly without that, like, in the back of our mind, like, where's the next SEGFAULT coming from?
Yeah.
Oh, absolutely.
Or the next security vulnerability.
Exactly.
Because you did an S print F, not at one of the safe variants or whatever.
It's been a long time since I've done C++.
But tell me about that process.
I don't know how many people are working on the project at this time and how much experience you all had with Rust.
Why did you make that choice?
And how did it go?
Oh, yeah.
I'm a Rust neophyte.
Like, before that Christmas of 2022, I've never written a line of Rust.
Most of our team are also new to Rust.
Or they're new when they join LanceDB.
And I think what helps is that most of them were also already proficient in C++, right?
So the sort of core thinking around algorithms and design is all the same, performance and all that.
And the Rust language makes it just easier to learn.
You're more productive.
And it's just safer.
And I think one of the surprising benefits that we saw that I don't see people talking about a lot,
and maybe that's just us not being very good at CMake, is that productivity piece.
When we were writing lots of C++ code, we spent tons of time just wrestling with CMake to make the builds work.
And once we moved to Rust, Cargo just made that so easy.
Basically, we spent zero time wrestling with that.
Wow.
That's pretty wild.
So you all were pretty new to Rust.
How much do you think that LLMs, ChatGPT, Mistral, Llama, all these things, the availability to just go,
hey, chat thing, how do you do this in Rust?
Here it is in C++.
Did you all find that beneficial?
Did you all do this?
We use a lot of coding aid.
So from Copilot to Cursor, Continue, Zed.
We try out lots of different tools as well.
I think certainly the quality of the tooling matters, but at the core of it, there's that model performance.
And so the models today performs pretty well on Python and TypeScript, but on Rust, much less so.
Oh, interesting.
Yeah.
And so we always joke about, you know, God grant me the confidence of ChatGPT coming up with random Rust syntax.
Yeah.
I've had that as well.
I asked for some, how to do this in Python, because I was feeling lazy or something.
I just needed it for an example real quick.
And it said, you use this.
It looked completely plausible.
It was converting time zones and date times.
And just the time zone part of it was driving me nuts.
I'm like, all right, Chat, let's do this.
And it just made up stuff.
So, oh, you're going to use this function or this property based on this.
No, that doesn't exist.
I'm like, that doesn't exist.
Try again.
Oh, I'm so sorry.
Let me try another one.
Yeah.
What's interesting is I think that like in Python, if you're looking at really well-known libraries out there,
it generally does much better.
But even for long-known libraries that don't have as much training corpus or attention focus on it,
it tends to hallucinate a lot.
Even for something like Apache Arrow, like PyArrow kind of API, it's the standard for in-memory data.
But the ChatGPT still makes up APIs for that.
And if you're looking at in terms of like the effect on the developer community,
so there are lots of times where I've seen either like Hacker News posts or comments on Hacker News posts
where there's a choice between two similar libraries.
And then I think the commenters are like, well, I'm choosing A because ChatGPT or Copilot generates better completions for that one.
How interesting.
I've never thought about that.
But yeah.
Wow.
You know, the Python, the PSF and JetBrains Python developer survey that came out,
one of the really interesting stats is 50% of the people who filled out that survey said they've been doing professional software development for less than two years.
Oh, wow.
That is surprising.
Yeah.
I mean, 50% of all Python developers, less than two years.
And I think there's got to be a strong pull for, yeah, I could go research that.
Let me just ask Chat first and see what it says.
Or chat, ask Copilot or whatever tools they're using.
That shouldn't be surprising.
And also what's interesting, I think I've noticed kind of the same thing with AI versus, you know, previous generations of machine learning.
In previous generations, it was all Python.
With AI, there's a huge TypeScript community that has formed around AI.
And then also just up in previous generations of machine learning, you had to have years of training to learn math and stats and data analytics and data science before you're a productive machine learning engineer.
And I think today it's with AI, it's definitely not the case that you could quickly through some experimentation and self-learning to become a fairly proficient AI engineer, which is also a new term that's been coined in the last two years.
It's going to be really interesting to see how this affects the industry.
On one hand, it's going to supercharge people getting going faster, make them help them become unstuck if they're stuck on a problem.
But it could also end up hollowing out deep knowledge maybe in the long term.
I'm not entirely sure.
We'll see.
You know, that is very interesting.
I do think that what we're seeing is there's a lot of focus around sort of model capabilities and then maybe like RAG and, you know, how to marry context and knowledge bases to the model through, you know, vector search and vector databases.
But what we're seeing a lot is more traditional and large enterprises are now starting to adopt AI in wholesale.
And the way they're thinking about it is, you know, let's build an excellence center around AI internally and make AI applications easily accessible to the rest of the company across many different groups.
So you don't have to worry too much about AI infrastructure where the data store and how that interfaces with a model.
And maybe just think about, you know, more at the business level and the user level, how you want these things to work.
So I think that deep knowledge around AI statistics, machine learning will probably start to coalesce around these centers of innovation within these large enterprises.
And of course, within, you know, AI native startups, they, you know, they all have to have much deeper knowledge in advance of the rest of the market to stay ahead of the curve.
It's interesting.
And, you know, I started to see these bots.
We'll get to the database in just a second.
I started to see these bots as kind of almost replacing search.
You know, I find myself, maybe I'll just skip search.
Maybe I'll just ask this thing because I got a good shot of a good answer straight away rather than.
Right.
And that kind of ties into this enterprise thing.
Like, you know, searching Google is a little bit tricky sometimes.
I use cat or whatever you're using.
It's tricky because there's SEO tricks going on.
There's ads.
There's a lot of stuff where you're like, is this really relevant?
And I think that's probably even worse than trying to search within your enterprise.
Like, do we have documents on this or whatever?
It's probably just like, here's a 175 email email thread.
Like, no, no, thanks.
This isn't going to help.
Let's talk LanceDB and multimodal data.
And I guess maybe that's the place before we get into the details of Lance.
What is multimodal data anyway?
Yeah, absolutely.
So like I was saying earlier, I think with AI data, you know, vectors and embedding vectors is just scratching the surface.
One of the most powerful things about AI is that it makes it a lot easier to interact with the multimodal or unstructured data that we have.
And so, you know, if you're thinking about, you know, images, videos, audio forms, 3D point clouds, there's a lot that's happening in, you know, generation in things like autonomous vehicles.
But even for traditional enterprises, right, you have just just oodles of like PDF documents or slide decks.
And there's lots of use cases for a tool that can help those users extract insights and then possibly train models on top of those on top of that kind of data.
And so if you look at data by by volume, if you look at your average like TPCH data table, it's like what, like 150 bytes or something like that per row.
And then embeddings, you know, if you just look at the previous generation of open AI, it's like, you know, 25 times that.
And then if you look at images and videos, you're quickly getting to these tables where even if you have the same number of rows, the data is just huge.
Right. It's not username, address and a few other things are still simple.
Yeah.
When we talk to a lot of our customers who are trying to build this new what they call unstructured data lake, the expectation, they are already coming in with the expectation.
Okay, this in terms of data volume, this is going to this is going to trounce the existing previous generations of data lakes.
We had big data before, but now we've got big data in another level, right?
Yeah, this is even bigger.
And it's not only that each row is bigger, but, you know, previous generations of tooling, you have, you know, like users or humans manually generating one data point at a time.
And now, you know, AI is generating its own training data at thousands of tokens per second, right?
You know, when you're when you're doing completions, that completion itself becomes training data.
And so the volume at which the data is increasing is also growing at very, very rapidly.
Yeah, if anyone wants to get a sense for just how intensely the world is focused on this, they just reopened Three Mile Island, the nuclear power plant, purely to plug in a single Azure data center for AI.
So, yeah, I think we're putting a lot of energy into generating data.
What would you say?
Well, I'm glad this data center, even if something goes wrong, the consequences will be much less dire.
Yes.
Honestly, I think nuclear is it's worth considering if if you rather than coal or whatever.
But still, that's a whole different discussion.
We don't need to go down that hole right now.
Maybe maybe later at the end.
Who knows?
You can turn this show into talk politics instead.
No, no, no, no.
Please.
No.
It's too early.
This portion of Talk Python To Me is brought to you by Sentry.
Code breaks.
It's a fact of life.
With Sentry, you can fix it faster.
As I've told you all before, we use Sentry on many of our apps and APIs here at Talk Python.
I recently used Sentry to help me track down one of the weirdest bugs I've run into in a long time.
Here's what happened.
When signing up for our mailing list, it would crash under a non-common execution pass, like
situations where someone was already subscribed or entered an invalid email address or something
like this.
The bizarre part was that our logging of that unusual condition itself was crashing.
How is it possible for our log to crash?
It's basically a glorified print statement.
Well, Sentry to the rescue.
I'm looking at the crash report right now, and I see way more information than you'd expect
to find in any log statement.
And because it's production, debuggers are out of the question.
I see the traceback, of course, but also the browser version, client OS, server OS, server
OS version, whether it's production or Q&A, the email and name of the person signing up.
That's the person who actually experienced the crash.
Dictionaries of data on the call stack and so much more.
What was the problem?
I initialized the logger with the string info for the level rather than the enumeration.info,
which was an integer-based enum.
So the logging statement would crash, saying that I could not use less than or equal to between
strings and ints.
Crazy town.
But with Sentry, I captured it, fixed it, and I even helped the user who experienced that crash.
Don't fly blind.
Fix code faster with Sentry.
Create your Sentry account now at talkpython.fm/sentry.
And if you sign up with the code TALKPYTHON, all capital, no spaces, it's good for two free
months of Sentry's business plan, which will give you up to 20 times as many monthly events
as well as other features.
Let's talk about something, maybe dissect, kind of like what the H2 or the one-sentence elevator pitch here for LanceDB is.
So I'll read it out and we'll take it apart and you make sense of it for all of us, okay?
LanceDB is a developer-friendly, open-source database for AI.
From hyper-scalable vector search and advanced retrieval for RAG to streaming data,
interactively exploring large-scale AI sets.
Best foundation for your AI application.
Yeah, so open-source for starters.
Go over here to GitHub, you can see just under 10,000 stars, which is, you know, congratulations.
That's excellent.
Actually, across the different bits, but yeah, so really cool.
And it's under the Apache 2 license.
That's pretty open for people to do most things what they want, right?
Maybe not build a closed-source business on top of, but other than that, right?
The way we're thinking about it is there's a base layer of how do you work with AI?
And we've made Lance format, which is open-source and it's a columnar format that's optimized for AI.
And on top of that, we're building the different workloads that allow our users and customers to search, retrieve,
run analytical queries, run training workloads across the board for all of their AI needs across the enterprise.
So the repo that you're looking at, which is Lance CB open-source, that's one of the tools that we've built on top of the Lance format.
And the idea is when you, super common workload is you want to start experimenting with building RAG or agents with memory,
semantic search, AI-enabled semantic search, like image, text image and video search and things like that.
Lance CB is sort of the easiest way for you to get started and prototype.
So like you said in the beginning, the mental model would be something like equivalent to like a SQLite or DuckDB, right?
So the data is just a file.
There's no service to manage.
There's nothing to sort of connect to.
It's totally open.
So you can look at it from with other tools that you have.
And then the actual database runs every day.
I love this idea.
I believe, I think things like SQLite and Lance CB, I think this, I just have a file and it, there's a really smart engine on top of that file.
Instead of a complex server architecture to run and manage and security and firewalls and all of that, migrations, all these things.
It's really pretty positive, I think, to just say, you can just have a file.
When your app is running, it's part of your app.
And when your app shuts down, it saves the data and it shuts down with it, right?
A couple of interesting things about the, you know, it's just a file is we took that a little bit further than that.
One is I think that we wanted the file to be open and so that you can inspect it, you can work with it using your other tools, right?
So when you're building that AI application, one is if you, you have more than just the, that search and the vector lookup.
There's, you know, you might want to run SQL to look at metadata.
You might have tensors that, that will upload to PyTorch training workloads and things like that.
And then we wanted that layer.
So we wanted that file to be accessible by all your familiar tools, right?
From like pandas and polars, you know, new data frame engines like DAF, for example, and even distributed engines like Spark and Ray.
That was one big thing that we focused on with, with Lance Format is making it plug in into the ecosystem.
For folks who are coming with a Python background and a data background, working with AI would feel familiar.
Right. A lot of people know how to do SQL to ask questions about their data, even if they're not programmers.
Maybe a business analyst is like, well, I can write a query, maybe connect that result to some graph or whatever, right?
This was the primary reason why we chose Apache Arrow as the standard interface and the standard type system.
It's easy to plug in all these other tools on top.
So good question.
Now the audience out there from Carol is, how's the integration with Narwhals?
I am not familiar enough with Narwhals.
So Narwhals is a layer that will let you work with multiple data frame libraries.
So if you wanted to work with say pandas and polars, you can use this and it will adapt depending on what the original data source is.
It'll adapt everything to a subset of polars commands.
So maybe the question is, how does it work with polars or pandas, right?
So maybe if it does with one or the other, then there might be a way to connect it.
The core interface input output is arrow.
And then with LanceDB, there's on the output end, you can convert results to pandas data frames or polars data frames.
And then on the input end to, I think natively, we can take pandas data frames as batches of input, but we convert that to arrow tables.
And so for others, you know, it's like polars, for example, it's really easy to convert polars to like an arrow table.
So a lot of that, even if it's not already automatic, it's like dot two arrow one command away.
Yeah, Carol added, given it works with arrow, it probably just works.
Yeah.
When we talk about a file, we don't just mean your local file system, but also object store.
So if you throw the data set on S3, you can just query it directly.
Now, you know, you'll hit additional latency because object store.
But it works.
So a lot of our users see that as a bridge into production so that, you know, they use LanceDB open source to do the prototyping MVP.
And then, you know, they, if their use cases fit before they need like a distributed system, it's easy for them to just say, hey, the embedded library lives in my application code.
And then the I have a shared data layer in S3 or maybe an EFS or something like that.
And you can run distributed queries very easily still without having to manage your own additional systems.
Okay.
Just use really scalable storage layer like S3 or probably anything that has an S3 compatible API, you know, like DigitalOcean.
Yeah.
Anything that's like a POSIX compliant interface or an S3 compliant API.
So even MinIO or something like that, right?
Like one of your, yeah.
Yeah.
I think MinIO published their own integration with LanceDB as an example as well.
Oh, did they?
Yeah.
That's cool.
Maybe tell people real quick what MinIO is.
It sounds like you're familiar with it or I can.
Yeah, go ahead.
So MinIO is a self-hosted high-end sort of pretty complete version of what you would think of as S3.
But if you don't want to use S3, you want to host your own, on your own infrastructure with your own storage layer and things like that.
But you still want to talk S3 APIs and security and all that stuff.
Yeah, that's what MinIO is.
It's just a file really, really far.
And I think the results end up speaking for itself.
We're seeing really good results.
And I think it really matches with the way that a lot of our users think about existing tooling and how to add AI to their existing business and applications.
We might be getting just a little bit ahead of ourselves, but we'll backtrack and talk APIs and how it works and stuff.
But what's the, you know, if it's just a file, what you said is obviously like a pretty broad statement by that, the way you all have implemented it.
But what's the go to production story?
Absolutely.
If you're using SQLite and that's your middle model, you're like, well, you probably should just switch to Postgres.
Right.
But it doesn't sound like that's your answer.
It sounds like you've thought it through on how to run it at larger scale.
In terms of large scale production, there's a number of different workloads for both online and offline.
So this is our commercial offering.
I'm just calling it LanceDB Enterprise.
And essentially, it's one distributed system that gives you a huge scale, low latency, high throughput, a system that can be backed by the same Lans data.
It's just a file in S3.
So the trick is to create that distributed system in a way that allows you to enjoy the cost efficiency of ObjectStore and the scalability of it while being very, very fast and very performant.
And so that's the production story for our customers who are putting AI applications into production where they need high throughput or they need very large index sizes or just to manage lots of vectors and create a small portion of it without the same sort of budget constraints as like you would with OpenSearch or other tools that doesn't give you that compute storage separation.
So it says it has GPU support for building vector indexes, which is pretty awesome.
So databases, indexes are, they're like the magic speed dust you can sprinkle on them, obviously.
What's the story with indexing and this GPU support?
With traditional database indices, it's not very computationally intensive.
Unfortunately for vector indices and N indices, they're quite so.
So I think with Rust, you can make CPU based indexing for vector data pretty efficient.
So if you're looking at, you know, hundreds of thousands or even up to like maybe, you know, 30 or 50 million vectors, CPU indexing is pretty good and can get pretty fast.
But for our customers that have 15 billion vectors in one table that they need to index one index, that's going to take days.
All right.
So with GPU support, you know, we cut that down by more than like 15, 20 X.
So then it becomes like a, something that they can do repeatedly and have a acceptable feedback loop in terms of, you know, maybe adding new data, refreshing that, that index.
What is also what I love about the composable data ecosystem.
And we talk about that, but there's also an extension, which I'd like to add is, is like that composable AI data ecosystem.
So we talked a little bit about the benefits of like having arrow, which is just, it means if we make the input output APIs arrow, it just works with the rest of the ecosystem.
This is one of the points where we get additional benefits, where with GPU support, it's easy through the arrow interface to actually talk to PyTorch.
So we can use a lot of the GPU tooling in PyTorch to build a lot of our accelerated vector indexing tool tools.
So that way it saves us lots of time messing with CUDA interfaces and, and, and things like that, that are not quite mature in the, the Rust ecosystem yet.
It's something you don't have to build.
It just gets better on its own and you just get to upgrade it.
Right.
Exactly.
That's part of the magic of package managers and package repositories and stuff.
It's really cool.
So what is the workflow?
Like, take me through the journey that somebody might go on.
They have a bunch of data, they want to put it into the database and they want to put the resulting thing into production so that they could use it behind an API or something that they're implementing for their app or something like that.
Is there a big training block of time that you do?
And then you move the resulting data somewhere?
Or do you just, you put it in production and start adding to it over and just keep adding to it over time?
What's the workflow?
With the Lansi B Enterprise, like the system that's running, you can just keep adding data to it.
The indexing and all of that is automatic once you've configured it properly, which is just, you know, here's the schema and like create indices on these columns, right?
Much like what you would do with a, like a Postgres table.
Once you have that set up and as you add data, the indexing is automatic.
And then for small amounts of data, our users typically will say like have Python dictionaries, JSON, or, you know, Pandas data frames.
They can add those to the database through the API for really large scale data.
So if you're working with terabytes or even petabytes of data, you don't want to be shoving that through the API like a thousand rows at a time.
So this is where the open data layer comes in.
So if you have a large data set and you have, whether it's Spark or Ray, you can use those large distributed systems to write data directly to S3 in the, in Lansi open source format.
And then the system actually picks it up from object store and takes care of the indexing and compaction and all of that.
That's sort of like when we're talking about adding data to it, it's, it's, it's both.
This portion of talk Python to me is brought to you by Bluehost.
Got ideas, but no idea how to build a website.
Get Bluehost with their AI design tool.
You can quickly generate a high quality, fast loading WordPress site instantly.
Once you've nailed the look, just hit enter and your site goes live.
It's really that simple.
And it doesn't matter whether you're a hobbyist entrepreneur or just starting your side hustle.
Bluehost has you covered with built-in marketing and e-commerce tools to help you grow and scale your website.
For the long haul.
Since you're listening to my show, you probably know Python, but sometimes it's better to focus on what you're creating rather than a custom built website and add another month till you launch your idea.
When you upgrade to Bluehost cloud, you get a hundred percent of time and 24 seven support to ensure your site stays online through heavy traffic.
Bluehost really makes building your dream website easier than ever.
So what's stopping you?
You've already got the vision.
Make it real.
Visit talkpython.fm/bluehost right now and get started today.
And thank you to Bluehost for supporting the show.
So I'm talking to you on a Mac mini M2 Pro with maxed out RAM and four terabytes of space or something.
Can I do interesting stuff on my computer?
Is it too small?
It's not too small at all.
One of the things about LAN format is it's all disk, right?
So most of the magic that we talked about with LanceDB, at least open source, lives in that open source format layer.
And that format is three things in one.
One, there's the columnar format, the file format, and then there's a table format, and then there's the indexing.
So the indexing is what speeds up the NN vector lookups.
Typically, we've made the index disk base so that if you're looking up a few partitions in the index, it's only loading those partitions.
And so the RAM requirement is actually quite small.
And so on your systems, probably the more limiting factor is going to be like the disk size rather than how much memory you have.
A couple of terabytes.
I should be able to put some data on that.
Yeah, exactly.
A couple of free terabytes, that is.
All right.
Let's talk about the architecture a little bit, maybe a look inside.
Like give us a, you've talked about some of the pieces and formats, but give us a sense here.
This is sort of an architecture that includes more than just Lance, but it's more about, you know, how our customers are thinking about building their own multimodal data lake.
So we call this like vector data lake or unstructured data lake, and there's no standardized terminology yet, but this is what I was mentioning before.
Nice.
Yeah, I'll just add for the listeners, I'll put a link to a diagram that talks about this a little bit.
People can check out if they want.
So the idea is from these companies perspective, they now have, on the one hand, they have lots of data that they can now take advantage of because of new AI models and applications.
And on the other hand is there's lots of business units across that enterprise that wants to experiment with AI or add AI applications, whether it's agents or, you know, internal tooling or customer success, you know, video and image search.
And lots of different use cases where there's a lot of like, hey, can you sprinkle some AI magic on my existing business and maybe look for that 10x in terms of ROI.
You see, we have all this SharePoint data.
Can you help me out?
That's literally a conversation.
I'm sure that it is because SharePoint is bad and you don't want anything to do with it, but people keep putting stuff into it.
I don't know.
It's not just about the AI capabilities, but like, how do you build this base layer and make it a lot easier to access for the rest of the folks in the company?
And most of these companies aren't starting from scratch.
They already have made significant investments into the data science and analytical tooling, training for their data scientists, analysts.
And they also made significant investments into their existing data lake for large scale data processing, right?
So you don't want to make them have to throw all that away.
And instead, that open data layer is super important to plug into the rest of their existing ecosystem.
And with Lance, unlike Parquet or JSON or WebDataSet, all of their multi-bono data can live in one place so that they can do search and retrieval on the vectors.
They can run SQL.
They can do training and data processing workloads all sort of in one engine and one piece of data.
So it just makes things a lot simpler.
It allows us to add a lot more performance optimizations.
And then it saves, because of the size issue, it also saves these enterprises a lot in terms of cost so that they don't have to keep making different copies of different parts of the data for different parts of that workload.
Right.
This thing needs that format of the data.
That thing needs that format of the data.
Right.
Exactly.
Two examples that really stuck out to me, speaking with one user recently, and they're doing EDA in DuckDB.
So they have lots of metadata.
They're doing EDA in DuckDB.
Quick acronym, please.
What's EDA?
Exploratory data analysis.
Okay, got it.
They're running SQL in DuckDB, but then they need to fetch, you know, there's like 10 columns of images and like 10 columns of multi-vectors.
And so when they, at the end of where they finish running that join and filter, it takes just ages to fetch that data.
So whether, because of the lack of random access support across the board.
So to be, then you have to think about other tooling, either distributed engine or somehow just load all of your data in memory.
And that's also not tenable.
Whereas with Lance format, because you can do random access very quickly, you can actually just quickly move on to the next step in the same Python application.
And because it works with DuckDB, you can get the row IDs out, fetch the rows, and then quickly move on to downstream processing, whether it's for like train a UMAP or, you know, something like that.
So that is just one small example where, you know, Lance makes it so that new workloads just works with existing tooling.
Yeah, that interop layer is really nice.
And another example is around, and again, this is like, you know, the size changes everything where we spoke with folks that are like, you know, we manage a bunch of image and video data in, let's say, like web data set.
And we want to iterate on our model and to create new features means we have to download terabytes of web data set, which are basically tarballs.
Open up the tarballs, like compute the feature from that data, put it back in the tarball and then upload it.
So to access one small piece and write a small piece, you often are downloading or transferring like 100x that because the large multimodal data.
Whereas with something like Lance, instead of using web data set, the table format features of Lance allows you to just keep that data in place, just add a new column, which writes a new file.
And then you can, the readers can just see an updated version of that with new schema and you can move forward, you know, not having to pay that transfer cost.
And it's not just the infrastructure cost, but also in the time that your engineers are spending on that, managing that process and just running across maybe petabytes of that data.
That's a crazy amount of data.
Yeah.
Let's talk through what it, what it's like to write a little bit of code, you know, connecting to the database, doing some queries or adding some data.
You guys have a quick start on here.
Maybe you could just talk us through real quickly on how to get started in Python with it.
Overall, the goal for the quick start is you have a package that you install in seconds.
And then depending on how quickly you're running through the quick start that, you know, between like a minute to five minutes, you've got a quote unquote vector database running, embedded vector database running,
where you can start building applications on top, right?
So the first step is very familiar.
You just run pip install Lance DB and that pulls in, you know, Lance format and installs our Rust packages.
And then you can start connecting to it, right?
So the default is you just give it a local file path and you say Lance DB dot connect.
And then that gives you a database connection, which there's two flavors of that, a synchronous for a client and an asynchronous client.
I love it.
So the async is basically just connect underscore async.
And roughly the interfaces are roughly equivalent.
There's some disparities which we're addressing now, but the main difference is you just sprinkle some await in places.
The async and await keywords make concurrent programming so much easier than threads and callbacks and all that business.
For sure.
Yeah.
So the connect bit is if people are familiar with SQLite, you just here's a URL to a file.
That's pretty much you give it that.
And then you have what's your naming convention on the async API?
Are all the functions ending in underscore async?
Or do you create like an async client?
Or how do you differentiate?
That's a good question.
Yeah.
So we create an async client.
The method names are the same between the two.
It's really just that initial connect call that's different.
I see.
But that's kind of the factory method for the client, either sync or async that comes back.
And then you just, you know what you got.
In the second example, we see that in action.
First step is let's create a table and let's add some simple data to it.
Let's say we have two fields, item and a price, right?
And then an embedding vector here for the example, we're just going to use two digits for that vector.
In practice, of course, it's going to be like 1,500 or, you know, 3,000 or something like that.
That's hard to print on your code.
Right, right.
Yeah.
So here we just have, you know, essentially digits of pi here.
And then you can see whether you have the async connection or the sync connection.
Both you call create table.
You add it.
You give it the table name.
And then optionally, you can give it data to initialize it.
So if you have data initially, the schema is inferred from the data that you provide it when you call create table.
If you go to the next example, you can also initialize an empty table that has with just a schema.
So here in this quick start, you can have an arrow schema.
Basically, this is the same schema as the data before, but you create an empty table, and then you can add data to it later.
I think I should update this.
But I think what's more convenient is that we've, as you see in the box below, you can actually, we've added pedantic support.
Beautiful.
There's a translation layer between pedantic and arrow schemas.
And so I think for a lot of our Python users, it's much easier to think in terms of pedantic objects as the data model rather than manually dealing with the PyArrow API to create a schema.
We saw lots of issues where users are like, well, how do I create a fixed size list?
What is a fixed size list?
And why does my vector have to be that?
Should I do float 32 or 64?
There's lots of stuff that is just much easier to think of in terms of Python types and Python objects.
I love the Python, the pydantic integration there.
That's super cool.
Well, the data layer used for my course's website and the podcast website and stuff is all based on Beanie and Mongo, which is async.
Basically, you're writing async queries against pydantic models, which is, it's a real, got the validation, but you're not writing directly to the database and just random dictionaries.
And who knows if it stays consistent.
So speaking of which, you have a schema here.
Is this, how hard, how strictly is this enforced, right?
Is this Postgres level or is this MongoDB level?
Like, probably should be this.
Right.
This is fairly strictly enforced.
So think, think like arrow tables and, you know, writing to writing arrow tables.
There's a little bit of like give around like nullability.
Because a lot of times if you're providing data, the types can be inferred, which is, it's easier to do casting.
And then if you're inferring like nullable versus nullable, that can get you in trouble when you're inserting data.
So for example, if you have a non-nullable column that you've declared in schema, but new data coming in, sometimes that translation layer into arrow just automatically turns it into a nullable, even though you didn't give it any nulls in the data.
And then when you insert it, it'll produce an error.
Do you respect things like optional typing in Pydantic models to control nullability, like optional float versus float being not nullable?
Yep.
So we do a bunch of that as well.
So in terms of pedantic integration, you know, it's another Rust project.
I know we've loved working with it for a long time.
If any listeners are out there that's interested in just messing with like pedantic and arrow, so that translation layer, we'd love to get some help as well.
So for some of the more like complicated nested types out there, so where that's like lists of lists or lists of fixed with lists and dictionaries and that kind of thing, the translation layer is incomplete.
But we know a couple of other companies and tools in the ecosystem who've also built their own kind of translation layer.
So it would be really interesting, I think, if there was a community member who can lead, say like, let's create a, let's get a bunch of projects together and let's create a standardized translation layer.
Not everyone just doing their own copy, right?
Right, right.
And then maybe like either pedantic or maybe arrow, one of the two projects can own that translation layer.
I think that would be really great for the, for the ecosystem.
That sounds like a fantastic idea.
And let's maybe wrap up our little code sample here with talking about querying the database or doing a search because it's fun.
I know it is fun to put data into a database and define schemas, but the actual purpose is to ask questions, right?
That's right.
We wanted to make it so that it's really familiar for people who've worked with databases and data frame engines.
So the main workload for Lansi B open source is that search API.
So when you, you can say table.search, you can pass in the vector, the query vector, and then you can call .limit to say how many results we want.
And then a .2 underscore blah, blah, blah, blah to determine what format you want the results back.
So you have two pandas here in the, in the example, but you can convert it to polars or you can just get it back as a, as a list.
Awesome.
Yeah, that's really cool.
I love it.
Oh, and also a couple of other things.
So here you can also do, if you have a Pydantic model that you use as the table schema, you can also say to pedantic and pass in the model and it'll automatically return a list of pedantic objects back, back to you.
So this is particularly useful for like multi-step, like agent workflows where you want that structured data.
It's easy to connect it to the rest of your.
Right.
You would just maybe want to take your pedantic and say, turn that into JSON and hand it over to the next agent or something like that.
If we sort of move forward in that example a little bit.
Yeah.
In addition to setting up the schema, there's also the embedding API that's really interesting so that when you create the schema.
So here's an example where we can create the schema using pedantic.
So in this block that I've declared the class words, which is a Lance model.
And the Lance model is just a pedantic base model that knows how to convert itself to, to arrow.
And then I have two fields in here, text and vector.
And what we have in Lance CB is an embedding registry where you can hear, I declared a function of embedding function called func.
And basically I call get registry dot get open AI dot create.
And I give it the name of the embedding here.
We're using ADA two, but new models have been released since.
And I can use the pedantic annotations to say, hey, the text field is the source field for that function.
So it's text string equals func dot source field.
And then the function itself knows how many dimensions it is.
So I don't need to think about like how many dimensions this, the vector, the embedding has.
Once I've declared the schema, I can call a familiar dot create table workflow.
And then I can call table dot add to add data to it.
And here, because I've declared the embedding function in the schema, I don't actually have to generate the embeddings myself.
So in this example, I'm only passing in the text field as input and Lance CB.
Oh, that's really cool.
Yeah, just takes care of calling the open AI API on your behalf and then adding the vectors before, you know, adding the whole batch to the table.
So open AI was was our first one, but there's dozens of compatible embedding models.
So pretty much anything you can pull off hugging face.
And then there's lots of other vendors like cohere, for example, if you were running open source like olama, there's also a integration for that.
Nice. You can use some of the on machine ones potentially as well.
Exactly. Yeah.
So a lot of the the ones you can pull off hugging face can just run locally and it exposes the options.
So if you do have for your Mac mini or or even your MacBook laptop, there are options.
There are lots of hugging face models where you can specify NPS to actually make it run a little bit faster.
Oh, nice. That uses the neural processing units or something instead of CPU or GPU.
OK. Yeah.
OK, that's cool. I didn't know that.
Very nice.
And it looks like I did want to ask you sort of what's the integration with things like open AI or Gemini or other things.
It looks like a lot of Lance DB is kind of for you building your app that is self-contained.
But also, you know, here's integration with some of the open AI API stuff.
What's how much do you depend on using other external AI systems versus just your own?
Right. So I think typically the embedding model is you either bring your own run locally or you call a third party API.
And then the actual completion that's outside of the scope of Lance DB.
So the prompt engineering and the calling the completion model once you have the context retrieved, that's not part of the Lance DB API.
That said, so we integrate with, let's say, like a Langchain or a Llama Index.
So if you're comfortable with that layer of sort of AI orchestration or RAG orchestration, you can use their APIs and plug in Lance DB into that.
So that that takes care of a lot of that, the other parts of the workflow.
And then if you're in the, let's say, like AWS ecosystem and you're familiar with, you know, Bedrock and things like that, there are a few AWS folks who've built a complete serverless RAG stack where they're using the Bedrock APIs for embedding creation and then for completion.
But then they have Lance data that's sitting on S3 and they're running Lance DB open source in AWS Lambda function so that you can essentially have, you don't have to manage any servers and just be calling serverless functions to build your RAG application.
Sounds super cool.
All right, we're getting short, short on time.
Let's wrap it up with a couple of things here.
First of all, we have the open source Lance DB.
You talked about the enterprise stuff as well.
Let's just talk business on a little really quick.
I think it's interesting.
Always interesting to see companies that have maybe open core or some kind of open source foundation and how you guys are making this work for both contributing to open source, but also eating.
What's the story here?
I think it's always a very complicated topic.
I think the way that we think about it is your journey building AI.
When you're just starting out and you're doing that prototype and that MVP, a lot of times you don't even know the value of the thing that you want to build or whether it works.
So why would you want to go through the hassle of managing complicated infrastructure or pay a third party vendor to just have some small amounts of data, right?
So we wanted to make Lance DB open source super easy for you to just get started and also just bridge you into production in that early stage.
The cloud and enterprise offerings are essentially for when you go into production and you want to have a super scalable and highly performant vector database where you still don't have to worry about the infrastructure, but you have lots more challenging systems needs.
Lance DB enterprises, Lance DB cloud is it's a hosted serverless offering.
So for, you know, small teams that have production needs, but maybe don't have, you know, billions of vectors and, or, or, you know, really challenging security requirements and things like that.
And Lance DB enterprises, we really have, okay, I need like a thousand to 10,000 queries per second.
I have just a ridiculous amount of data to catch myself before I curse and, on-prem as well.
Maybe.
Yeah.
I need my data to live on on-prem.
And so the enterprise comes in, in two different packages.
One is a managed offering where you can bring your own bucket and then we just run the compute for you and give you a private link or it's, it can be fully, we call it BYOC.
So it runs within the customer account.
So nothing leaves the premises except for basic telemetry data.
Awesome.
All right.
And then, you know, quickly close it out with, I guess, what's next?
Like where are things going here?
I see Lance DB cloud is in private beta early access mode.
You know, what are you, where are y'all going?
There's a couple of really exciting, first on the open source, you know, our vision for, for Lance is to make it the,
standard for working with AI, right?
So I think we already have lots of folks who are depending on Lance with image data, video data, large scale training, and we'll continue to, to make that better.
And then add additional encoding.
So for folks that not just have those, but, you know, compressing text data and the metadata that they have to make it as efficient as possible.
And I think there we've, you know, between Lance and Lance DB, I think we just broke through 2.2 million monthly downloads.
So we're really excited about that and, you know, that community uptake as, as well.
And, you know, we're getting lots of community collaborators and contributors, and we're looking to grow that community.
So on the open source side, it's, you know, better APIs, more automation around the data management, like compaction, and then just better encodings for the non-large blob types, right?
So we're looking to get smaller strings, smaller strings, numerical data, and things like that.
And then for Lance DB Enterprise, a lot of times, if you have to look at the whole search engine, right?
So right now, our customers still have to do the chunking and embedding themselves, but we're basically looking to make it much easier for those types of workloads, not just in terms of product activity, but to also save them costs as well in terms of embeddings.
So a lot of times what we find is our customers will update a table with a new version that's like 80% the same.
So if we have the right APIs and manage the embedding calls for them in the, we can essentially save 80% of that cost in terms of embedding APIs without complicated sort of like query cache or embedding cache that adds more complexity.
And then just deepening the, so more complete search engine for AI.
And then on the offline side, complete features at scale around building that training, pre-processing, and exploratory data analysis workflow for those types of customers.
Well, it looks like a really cool product and set of services and nice work.
Thank you.
Yeah, you bet.
And thanks for being here.
You know, final parting thoughts.
People want to get started with Lance DB.
What do you tell them?
Oh, come to our discord.
I don't know if I have the link, but give it to me and I'll put it in the show notes for people later.
Yeah, sounds good.
So come to our discord.
We're all in there and it's pretty active and we respond fairly quickly.
So there's not a lot of noise.
So it's mostly sort of practical topics, debugging and talking about new features, maybe having a little bit of fun with new examples and things like that.
Awesome.
All right.
Well, thanks for being here.
See you later.
Thank you.
This has been another episode of Talk Python to Me.
Thank you to our sponsors.
Be sure to check out what they're offering.
It really helps support the show.
Take some stress out of your life.
Get notified immediately about errors and performance issues in your web or mobile applications with Sentry.
Just visit talkpython.com.
Just visit talkpython.fm/sentry and get started for free.
And be sure to use the promo code talkpython, all one word.
And this episode is brought to you by Bluehost.
Do you need a website fast?
Get Bluehost.
Their AI builds your WordPress site in minutes and their built-in tools optimize your growth.
Don't wait.
Visit talkpython.fm/bluehost to get started.
Want to level up your Python?
We have one of the largest catalogs of Python video courses over at Talk Python.
Our content ranges from true beginners to deeply advanced topics like memory and async.
And best of all, there's not a subscription in sight.
Check it out for yourself at training.talkpython.fm.
Be sure to subscribe to the show.
Open your favorite podcast app and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, the Google Play feed at /play,
and the direct RSS feed at /rss on talkpython.fm.
We're live streaming most of our recordings these days.
If you want to be part of the show and have your comments featured on the air,
be sure to subscribe to our YouTube channel at talkpython.fm/youtube.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
We'll see you next time.