Michael Kennedy: Your cloud SSD is sitting there, bored, and it would like a job.
Michael Kennedy: Today, we're putting into work with DiscCache, a simple, practical cache built on SQLite
Michael Kennedy: that can speed things up without spinning up Redis or other extra servers.
Michael Kennedy: Once you start to see what it can do, a universe of possibilities opens up.
Michael Kennedy: We're joined by Vincent Warmerdom to dive into DiscCache.
Michael Kennedy: This is Talk Python To Me, episode 534, recorded December 19th, 2025.
Talk Python To Me, yeah, we ready to roll.
Upgrading the code, no fear of getting old
Async in the air, new frameworks in sight
Geeky rap on deck, Quart crew
It's time to unite
We started in Pyramid, cruising old school lanes
Had that stable base, yeah sir
Michael Kennedy: Welcome to Talk Python To Me, the number one Python podcast for developers and data scientists.
Michael Kennedy: This is your host, Michael Kennedy.
Michael Kennedy: I'm a PSF fellow who's been coding for over 25 years.
Michael Kennedy: Let's connect on social media.
Michael Kennedy: You'll find me and Talk Python on Mastodon, Bluesky, and X.
Michael Kennedy: The social links are all in your show notes.
Michael Kennedy: You can find over 10 years of past episodes at talkpython.fm.
Michael Kennedy: And if you want to be part of the show, you can join our recording live streams.
Michael Kennedy: That's right.
Michael Kennedy: We live stream the raw uncut version of each episode on YouTube.
Michael Kennedy: Just visit talkpython.fm/youtube to see the schedule of upcoming events.
Michael Kennedy: Be sure to subscribe there and press the bell so you'll get notified anytime we're recording.
Michael Kennedy: Vincent, hello.
Michael Kennedy: Michael, Michael, we're back.
Michael Kennedy: Awesome.
Michael Kennedy: Awesome to be back with you.
Michael Kennedy: Yeah, this is almost the sequel to the last time you were on the show.
Michael Kennedy: So it's going to be fun.
Vincent Warmerdam: Yeah, so sequel in this case, not the query language,
Vincent Warmerdam: like an actual sequel of events.
Vincent Warmerdam: Yes.
Vincent Warmerdam: Yeah, you can correct me if I'm wrong,
Vincent Warmerdam: but I think what happened is you had me on a podcast a while ago
Vincent Warmerdam: to talk about a course that I made,
Vincent Warmerdam: and a big chunk of the course that we were very enthusiastic about
Vincent Warmerdam: was about this tool called DiscCache.
Vincent Warmerdam: And then we kind of came to the conclusion,
Vincent Warmerdam: well, we had to cap it off.
Vincent Warmerdam: Maybe it's fun to do an episode on just DiscCache.
Vincent Warmerdam: since we're both pretty huge fans of it.
Vincent Warmerdam: I think that's how we got here.
Michael Kennedy: I think that is how we got here as well.
Michael Kennedy: And we're going to dive into this.
Michael Kennedy: Honestly, it's a pretty simple library called Disc Cache,
Michael Kennedy: but what it unlocks is really, really sweet.
Michael Kennedy: And I'm going to talk about a lot of different angles.
Michael Kennedy: And now, even though it's just been not that long since you were on the show,
Michael Kennedy: maybe just give us a quick intro of who you are.
Vincent Warmerdam: Hi, my name is Vincent.
Vincent Warmerdam: I've done a bunch of data machine learning stuff, mainly in the past.
Vincent Warmerdam: That's sort of what a lot of people know me from.
Vincent Warmerdam: These days, though, I work for a company called Marimo.
Vincent Warmerdam: You might have heard from us.
Vincent Warmerdam: We make very modern Python notebooks.
Vincent Warmerdam: We took some lessons from Jupyter,
Vincent Warmerdam: and we take a new spin of it.
Vincent Warmerdam: So that's my day to day.
Vincent Warmerdam: But I still like to write notebooks
Vincent Warmerdam: and do kind of fun little benchmarks and also stuff
Vincent Warmerdam: with LLMs.
Vincent Warmerdam: And I've just noticed that for a lot of that work, boy,
Vincent Warmerdam: disk cache is amazing.
Vincent Warmerdam: And I also use it for web stuff.
Vincent Warmerdam: And I think that's also what your use case
Vincent Warmerdam: is a little bit more of.
Vincent Warmerdam: But yeah, in notebook land, you also
Vincent Warmerdam: like to have a very good caching mechanism
Vincent Warmerdam: And on the Mremo side of things, we are also working on different caching mechanisms, which I might talk about in a bit.
Vincent Warmerdam: But just for me, the bread and butter, the thing I've used for years at this point is disk cache whenever it comes to that territory.
Michael Kennedy: Yeah, it's funny.
Michael Kennedy: This was recommended to me for Python Bytes as a news item over there quite a while ago, like years ago.
Michael Kennedy: And I'm like, oh, that's pretty interesting.
Michael Kennedy: And then I saw you using it in the LLM Building Blocks course, and it just unlocked for me.
Michael Kennedy: Like, oh, my.
Michael Kennedy: Oh, this is something else.
Michael Kennedy: And so since then, I've been doing a bunch with it, and I'm a big fan.
Michael Kennedy: I've been on this, like trying to avoid complexity, but still getting really cool responses, performance, et cetera, out of your apps.
Michael Kennedy: And I think this is a really nice way to add multi-process, super fast caching to your app without involving more servers and more stuff that's got to get connected and keep running and so on.
Michael Kennedy: But before we get into the details of that, maybe let's just talk about caching in general.
Michael Kennedy: Like what types of caching is there?
Michael Kennedy: You know, I sort of give a little precursor there.
Michael Kennedy: But yeah, dive into it.
Vincent Warmerdam: So like in the course, the main example
Vincent Warmerdam: I remember talking about was the one--
Vincent Warmerdam: you've got this LLM, and you want to do some benchmarks.
Vincent Warmerdam: And it might be the case that, I don't know,
Vincent Warmerdam: using an LLM for, let's say, classification,
Vincent Warmerdam: like some text goes in, we got to know whether or not
Vincent Warmerdam: it's about a certain topic, yes, no, or something like that.
Vincent Warmerdam: Then it would be really great if, suppose, the same text came
Vincent Warmerdam: by for whatever reason, that we don't run the query on the LLM
Vincent Warmerdam: Again, it's like wasted compute, wasted money.
Vincent Warmerdam: So it'd be kind of nice if the same text goes in that we then say,
Vincent Warmerdam: oh, we know what the answer to that thing is already.
Vincent Warmerdam: We cached it, so here you can go back.
Vincent Warmerdam: And that's the case when you're dealing with heavy compute ML systems.
Vincent Warmerdam: But there's a similar situation that you might have, I guess,
Vincent Warmerdam: with expensive SQL queries,
Vincent Warmerdam: or you want to reduce the load on a database somewhere.
Vincent Warmerdam: Then having some sort of a caching layer that's able to say,
Vincent Warmerdam: oh, you're querying for something, but I already know what it is.
Vincent Warmerdam: Boom, we can send it back.
Vincent Warmerdam: I think the classical thing you would do in Python
Vincent Warmerdam: is you have this decorator in functools, I think, right?
Vincent Warmerdam: The LRU_cache.
Michael Kennedy: Yeah, exactly.
Michael Kennedy: Yeah.
Vincent Warmerdam: That's a hell of a world to that.
Vincent Warmerdam: But the downside of that thing is that it's all in memory.
Vincent Warmerdam: So if you were to reboot your Python process,
Vincent Warmerdam: you lose all that caching.
Vincent Warmerdam: So that's why people historically, I think,
Vincent Warmerdam: resorted to--
Vincent Warmerdam: I think Redis, I think, is the most well-known caching tool.
Vincent Warmerdam: It's the one I've always used.
Vincent Warmerdam: There's Memcache, I think.
Vincent Warmerdam: There's other tools.
Vincent Warmerdam: You could use Postgres for some of this stuff as well.
Vincent Warmerdam: But recently, especially because disks are just getting quicker
Vincent Warmerdam: and quicker, people have been looking at SQLite
Vincent Warmerdam: for this sort of a thing as well.
Vincent Warmerdam: So that's, I think, the quickest summary
Vincent Warmerdam: and also sort of the entryway to how I got started with disk cache.
Michael Kennedy: Yeah, and so for this example that you highlight in the LLM
Michael Kennedy: Building Blocks course, it's not a conversation.
Michael Kennedy: It's like a one-shot situation, right?
Michael Kennedy: You come up-- you say, I have some code or some documents,
Michael Kennedy: and I have almost like an API.
Michael Kennedy: I'm going to send that off to the LLM and ask it, tell me X, Y, and Z about it.
Michael Kennedy: And sure, it's got some kind of temperature and it won't always give an exactly the same answer,
Michael Kennedy: but you're willing to, you know, you're willing to accept an answer.
Michael Kennedy: And at that point, like why ask it again and again and again, which it might take seconds,
Michael Kennedy: it might cost money.
Michael Kennedy: Whereas if you just remember through caching somehow, you remember it, it's like, boom, instant.
Vincent Warmerdam: Yeah, and it tends to come up a lot in when you're doing benchmarks, for example.
Vincent Warmerdam: So you have this for loop, you want to go over your entire data set, try all these different approaches.
Vincent Warmerdam: And if you've got a new approach, then you want that to run, of course.
Vincent Warmerdam: But if you accidentally trigger an old approach, then you don't want to incur the cost of like going through all those different LLMs.
Vincent Warmerdam: I should say, like, even if you just forget about LLMs, let's just say machine learning in general.
Vincent Warmerdam: Let's say there's some sort of image classification thing you're using in the cloud.
Vincent Warmerdam: There also, you would say, like, file name goes in.
Vincent Warmerdam: that's an image and if the same file name goes in we don't want the expensive compute cost to happen
Vincent Warmerdam: either so it's definitely more general than llms but llms do feel like it's the zeitgeisty thing to
Michael Kennedy: worry about yeah i think for two reasons one because they're just the topic du jour and two
Michael Kennedy: because they're they're i think a part of computing that most people experience that is way slower than
Vincent Warmerdam: they're used to yeah well and especially if you're you know if i suppose that you you have a an
Vincent Warmerdam: an attic somewhere and you're a dad and you want to do home lab stuff and you're playing with all
Vincent Warmerdam: these open source LLM models, then you also learn that, yeah, they're fun to play with, but they also
Vincent Warmerdam: take a lot of time to compute things. So then immediately you get the motivation to do it the
Michael Kennedy: right way. Yeah, I built a couple of little utilities that talk to a local LLM. I think it's
Michael Kennedy: the OpenAI OpenWeights one, that 20 billion parameter one I have running on my Mac Mini,
Michael Kennedy: and it's pretty good, a little bit slow, but, you know, it's fine for what it's being used for. And
Michael Kennedy: put-- use your disk cache technique on it.
Michael Kennedy: And if I ask it the same question again, it's like, boom.
Michael Kennedy: You don't need to wait 10 seconds.
Michael Kennedy: Here's the answer.
Michael Kennedy: Yeah.
Vincent Warmerdam: So that-- and I guess like-- but I guess from your perspective,
Vincent Warmerdam: I think your main entry point to this domain
Vincent Warmerdam: was a little bit more from the web dev perspective, right?
Vincent Warmerdam: Like that's-- and I suppose you're using it a lot
Vincent Warmerdam: for preventing expensive queries to go to Postgres,
Vincent Warmerdam: or I don't exactly know your backend.
Michael Kennedy: You know how-- you won't believe how optimized my website is.
Michael Kennedy: There's not a single query that goes to Postgres,
Michael Kennedy: because they go to MongoDB.
Michael Kennedy: I'm just kidding.
Vincent Warmerdam: There you go.
Michael Kennedy: No, but your point is totally valid.
Michael Kennedy: Go into the database, right?
Michael Kennedy: Now, I don't actually cache that many requests.
Michael Kennedy: I don't avoid that many requests going to the database.
Michael Kennedy: They're really quite quick, and so I'm OK with that.
Michael Kennedy: But when you think about a feature-rich database,
Michael Kennedy: feature-rich web app, there's just tons of these little edge
Michael Kennedy: cases you're like, oh, got to do that thing.
Michael Kennedy: And it's not a big deal, but we've
Michael Kennedy: got to do it 500 times in a request.
Michael Kennedy: Then it is kind of a thing.
Michael Kennedy: So let me give you an example.
Michael Kennedy: I'll give you some examples.
Michael Kennedy: So for example, the good portions of the show notes
Michael Kennedy: on talkpython.fm are in Markdown.
Michael Kennedy: I don't want to show people Markdown.
Michael Kennedy: I want to show them HTML, right?
Michael Kennedy: So when a request comes in, it'll
Michael Kennedy: say any fragment of HTML that needs
Michael Kennedy: to be turned into Markdown instead of just going, oh,
Michael Kennedy: let me process that.
Michael Kennedy: It just goes, all right, what is the hash of this
Michael Kennedy: or some other indicator of the content?
Michael Kennedy: And then I've already computed that and stored it in disk cache.
Michael Kennedy: So here's the HTML result.
Michael Kennedy: Another example is there's a little YouTube icon on each page.
Michael Kennedy: And that's actually in the show notes, but then the website parses the YouTube ID out
Michael Kennedy: and then embeds it with an, like, there's a bunch of stuff going on there to keep YouTube
Michael Kennedy: out of spying on my visitors.
Michael Kennedy: But stuff happens, YouTube ID is used.
Michael Kennedy: That could be parsed every time.
Michael Kennedy: Or I can just say this episode has this YouTube ID.
Michael Kennedy: That information goes into a cache, right?
Michael Kennedy: And because it's a disk cache sort of scenario,
Michael Kennedy: like a file-based one, not an LRU cache.
Michael Kennedy: It doesn't change the memory footprint
Michael Kennedy: and it's shared across processes.
Michael Kennedy: So in like the web world,
Michael Kennedy: it's really common to have a web garden
Michael Kennedy: where you've got like two or four processes
Michael Kennedy: all being like round robin to
Michael Kennedy: from some web server manager thing, right?
Michael Kennedy: If you don't somehow out of process that,
Michael Kennedy: either Redis or SQLite or database or something,
Michael Kennedy: then all of those things are recreating that, right?
Michael Kennedy: They can't reuse that, right?
Michael Kennedy: So there's a lot of interesting components there.
Vincent Warmerdam: And I suppose your web deployment,
Vincent Warmerdam: you have like a big VM, I suppose,
Vincent Warmerdam: and then there's like multiple Docker containers running,
Vincent Warmerdam: but they do all have access to the same volume,
Vincent Warmerdam: and that's how you access SQLite.
Michael Kennedy: Bingo, yeah, exactly, exactly.
Michael Kennedy: And how am I doing?
Michael Kennedy: Yeah, so what I have done is in the Docker Compose file,
Michael Kennedy: I have an external,
Michael Kennedy: This is also important for Docker.
Michael Kennedy: So I have an external folder on a big hard drive in the big VM that says, here's where
Michael Kennedy: all the caches go.
Michael Kennedy: And then depending on which app, it'll pick like a sub directory it can go look at or
Michael Kennedy: whatever that it's using.
Michael Kennedy: And so that way, even if I do a complete rebuild of the Docker image, it still retains
Michael Kennedy: its cache from version to version and all that kind of business.
Michael Kennedy: You could do that with a persistent VM as well, volume as well.
Michael Kennedy: But I've just decided--
Michael Kennedy: you can go and inspect it a little easier
Michael Kennedy: and see how big the cache is and stuff like that.
Vincent Warmerdam: OK, so we're going to get into the weeds of how
Vincent Warmerdam: disk cache works exactly.
Vincent Warmerdam: But I'm triggered here because it sounds like you've done
Vincent Warmerdam: something clever there.
Vincent Warmerdam: Because what you can do in disk cache
Vincent Warmerdam: is you can say, look, here's a file that's SQLite.
Vincent Warmerdam: And then it behaves like a dictionary,
Vincent Warmerdam: but it's persisted on disk.
Vincent Warmerdam: But what I just heard you say is that you've got multiple caches.
Vincent Warmerdam: So am I right to hear that, oh, for some things that
Vincent Warmerdam: need to be cached, let's say the YouTube things,
Vincent Warmerdam: that's a separate file.
Vincent Warmerdam: And then all the markdown stuff, that's also
Vincent Warmerdam: a separate file, and therefore if connections need to be made to either,
Vincent Warmerdam: it's also kind of nicely split.
Vincent Warmerdam: Is that also the design there?
Michael Kennedy: Yeah, that is.
Michael Kennedy: And actually, before, like, we're going to dive into all the details of how it works,
Michael Kennedy: but I'll just go, I'm just to give people a little glimpse.
Michael Kennedy: I'll go ahead and show, I've got this whole admin back in here.
Michael Kennedy: And I've got different caches for different purposes.
Michael Kennedy: Because they're just SQLite files, you can either say, give me the same one,
Michael Kennedy: or you can say, this one is named something else,
Michael Kennedy: and it has a different file name or different folder or whatever.
Michael Kennedy: Right, so I've got one that stores things like that YouTube ID I talked about
Michael Kennedy: any markdown, any fragment of markdown anywhere in the web app that it needs to say that needs
Michael Kennedy: to go to HTML, like just.
Vincent Warmerdam: Yeah, and it's like 8,000 items in that thing.
Michael Kennedy: Yeah.
Michael Kennedy: In this one, there's 8,970 items, which is nine megs, right?
Michael Kennedy: I mean, it's not huge, but it's not too bad.
Michael Kennedy: And you can actually even see where it thinks it lives, but that's not really where it lives
Michael Kennedy: because there's, you know, the volume redirects and stuff.
Michael Kennedy: But I've also got stuff for directly about the episodes that it needs to pull back.
Michael Kennedy: And then I do a lot of HTTP caching.
Michael Kennedy: And one of the things that I think is really wrong with web development is people say,
Michael Kennedy: well, that's like a stale image or that's a stale CSS file or JavaScript, you know,
Michael Kennedy: all that kind of stuff.
Michael Kennedy: So if you just do like super minor tricks and just put some kind of hash ID on the end
Michael Kennedy: of your content, it will, and you teach your CDN or whatever, that that's a different file
Michael Kennedy: if it varies by query string, then you never, ever have to worry about stale content ever.
Michael Kennedy: right but computing that can be expensive especially for remote stuff like if it's it's on a different
Michael Kennedy: it's like a s3 thing but you still want to have it do that so i have a special cache for that and
Michael Kennedy: that takes that's like pretty complicated to build up because it's got to do like almost 700 web
Michael Kennedy: requests to figure out what those are but once they're done it's blazing fast you don't have to
Michael Kennedy: do it again right unless it changes then it doesn't change much and so on so there's that's the way
Vincent Warmerdam: that i'm sort of using and appreciating disk cache yeah it works well in your setup because you've
Vincent Warmerdam: gone for the VM route. I mean, if you go
Vincent Warmerdam: for something like Fly.io or maybe even
Vincent Warmerdam: DigitalOcean has like a really, I think
Vincent Warmerdam: it's a nice like app service, but that
Vincent Warmerdam: all revolves around Docker containers that like
Vincent Warmerdam: spin up horizontally. And I
Vincent Warmerdam: don't think those containers can
Vincent Warmerdam: be configured in such a way they share the volume.
Vincent Warmerdam: So in that sense, you could still use
Vincent Warmerdam: disk cache, but then
Vincent Warmerdam: each individual instance of the
Vincent Warmerdam: Docker container would have its own cache, which still could
Vincent Warmerdam: work out.
Vincent Warmerdam: Not going to be as well
Vincent Warmerdam: well functional. It's going to be better with your setup, though.
Michael Kennedy: Yeah, absolutely. I agree, though. You could still do it. Or you could go, I'll take the
Michael Kennedy: zen of what Vincent and Michael are saying today, and I'll apply that to Postgres, or
Michael Kennedy: I'll apply that to whatever data. You could pull this off in a database.
Vincent Warmerdam: You would just have to do more work. Yeah. I mean, I've had a couple of, I think
Vincent Warmerdam: it was like a Django conference talk I saw a while ago. They were also raving about
Vincent Warmerdam: disk cache. But the merits of disk cache do depend a little bit on your
Vincent Warmerdam: deployment, though.
Vincent Warmerdam: That is, I think, one observation.
Vincent Warmerdam: Like in your setup, I can definitely imagine it.
Vincent Warmerdam: Interesting.
Vincent Warmerdam: Yeah.
Michael Kennedy: Yeah.
Michael Kennedy: Well, I don't even think we properly introduced this thing
Michael Kennedy: yet, so.
Vincent Warmerdam: But let's maybe go there.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: Let's start there.
Michael Kennedy: Let's start there.
Vincent Warmerdam: It's time.
Vincent Warmerdam: OK.
Vincent Warmerdam: It's time.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: I guess the simplest way I usually describe it,
Vincent Warmerdam: it really behaves like a dictionary,
Vincent Warmerdam: except you persist a disk and under the hood
Vincent Warmerdam: is using SQLite.
Vincent Warmerdam: I think that's the-- it doesn't cover everything,
Vincent Warmerdam: but you get quite close, if that's the way it is.
Michael Kennedy: I think there might be--
Michael Kennedy: you know, I keep harping on this on the show,
Michael Kennedy: but there are so many people that are new to Python
Michael Kennedy: and programming these days.
Michael Kennedy: Many, many of them, almost half of them.
Michael Kennedy: I think it's worth pointing out, just like, what is SQLite?
Michael Kennedy: Like, why is it different than any other database?
Michael Kennedy: Like, why have I been using the word database or SQLite
Michael Kennedy: when SQLite is a database, right?
Michael Kennedy: That's weird.
Vincent Warmerdam: - So, I never really took a good database, of course.
Vincent Warmerdam: I might be ruining the formalism of it.
Vincent Warmerdam: But the main, like, for me at least,
Vincent Warmerdam: the way I like to think about it is Postgres,
Vincent Warmerdam: that's a thing I can run on a VM,
Vincent Warmerdam: and then other Docker containers can connect to it
Vincent Warmerdam: because it's running out of process.
Vincent Warmerdam: There's some other process that has the database somewhere,
Vincent Warmerdam: and I can connect to it.
Vincent Warmerdam: And I think the main thing that makes SQLite different
Vincent Warmerdam: is that, no, you got to run it on the same machine,
Vincent Warmerdam: on the same process where your program is running.
Vincent Warmerdam: And that's, I think, the main--
Vincent Warmerdam: and there's all sorts of little details,
Vincent Warmerdam: like how the data structures are used internally,
Vincent Warmerdam: and SQLite doesn't have a lot of types.
Vincent Warmerdam: There's lots of other differences.
Vincent Warmerdam: I think that's the main one.
Vincent Warmerdam: Unless, Michael, I forgot something.
Michael Kennedy: MICHAEL LUTH: Yeah, no, I think it's--
Michael Kennedy: and it's--
Michael Kennedy: operationally, it's a separate thing
Michael Kennedy: run. It has to have both, it has to be secure because if your data gets exposed, like-
Vincent Warmerdam: For Postgres, is it not for SQL? Yes, it's running somewhere. People can SSH in if you're
Vincent Warmerdam: not careful. You've got to be mindful of passwords and all that stuff. That's totally true.
Michael Kennedy: Right. And it can go down. Like it could just become unavailable because you've screwed up
Michael Kennedy: something or whatever, right? It's a thing you have to manage in the complexity of running your app
Michael Kennedy: when it's like, well, it used to just be one thing I could run in a Docker container. Well,
Michael Kennedy: now I got different servers, they got to coordinate and there's firewalls and there's like, it's just,
Michael Kennedy: it just takes it so much higher in terms of complexity that like SQLite is a file.
Michael Kennedy: Yes.
Vincent Warmerdam: I mean, I do want to maybe defend Postgres a little bit there.
Vincent Warmerdam: Cause one thing that's like really nice and convenient in terms of like CICD and deployments
Vincent Warmerdam: and all that, oh, suppose you want to scale horizontally and there's like Docker containers
Vincent Warmerdam: running on the left and there's this one Postgres thing running on the right.
Vincent Warmerdam: I mean, you can just turn on and off all those Docker containers as you see fit.
Vincent Warmerdam: they're just going to connect to the Postgres instance.
Vincent Warmerdam: And I've done this trick for Calm Code a bunch of times
Vincent Warmerdam: where I just switch cloud providers,
Vincent Warmerdam: because Postgres is running there,
Vincent Warmerdam: and I can just move the Docker containers
Vincent Warmerdam: to another cloud provider, and it all works fine.
Vincent Warmerdam: No migration necessary.
Vincent Warmerdam: With SQLite, that aspect is a little bit more tricky.
Vincent Warmerdam: You have to be a bit more mindful.
Vincent Warmerdam: Although, I should mention, might be worth a Google.
Vincent Warmerdam: There's actually this one new cloud provider
Vincent Warmerdam: that's very much Python-focused.
Vincent Warmerdam: It's called Plash, P-L-A dot S-H, I think.
Vincent Warmerdam: Oh, this is new to me.
Michael Kennedy: Yeah, so I think--
Michael Kennedy: Wow, OK.
Michael Kennedy: Look at this.
Michael Kennedy: From.py to.com in seconds.
Vincent Warmerdam: Yeah, it's the Answer AI, Jeremy Howard and friends.
Vincent Warmerdam: I don't know to what extent this is super production ready.
Vincent Warmerdam: And SQLite, you've got to be mindful of the production aspect
Vincent Warmerdam: for some reasons as well.
Vincent Warmerdam: But one thing that is kind of cool about them
Vincent Warmerdam: is they give you a persistent SQLite as a database
Vincent Warmerdam: and a pipeline process that can just kind of attach to it.
Vincent Warmerdam: And they just-- in their mind, that's
Vincent Warmerdam: the simplest way that a cloud provider should be.
Vincent Warmerdam: take a very opinionated approach.
Vincent Warmerdam: So yeah, if you're interested in maybe running this
Vincent Warmerdam: as a web service, migrations are a little bit tricky
Vincent Warmerdam: in that realm, because you do have
Vincent Warmerdam: to download the entire data set due to migration
Vincent Warmerdam: and upload it again, I think, if I recall correctly.
Michael Kennedy: And for some apps, that's no big deal.
Michael Kennedy: Others, that's a mega deal.
Michael Kennedy: Depends how big that data is.
Vincent Warmerdam: So I'm not suggesting this is going
Vincent Warmerdam: to be for everything and everyone,
Vincent Warmerdam: but I do think it's cool, which is why I figured I'd mention it.
Michael Kennedy: Oh, it's new to me.
Michael Kennedy: I'm going to follow up with a lightstream.io.
Michael Kennedy: Have you seen this?
Vincent Warmerdam: Yeah, that is also really neat.
Vincent Warmerdam: So basically, what if you want to back up your SQLite?
Vincent Warmerdam: Like, how could you do that?
Vincent Warmerdam: Oh, it might be nice to do that with S3.
Vincent Warmerdam: And I think it's like the guy who made the thing works at Fly.io.
Vincent Warmerdam: He's doing a bunch of low-level stuff.
Vincent Warmerdam: One thing about that open source package
Vincent Warmerdam: is also really interesting, by the way,
Vincent Warmerdam: is I think he refuses PRs from the outside.
Vincent Warmerdam: He just wants to have no distractions whatsoever.
Vincent Warmerdam: He has a very interesting way of developing software.
Vincent Warmerdam: You can submit issues, of course.
Vincent Warmerdam: I think if you scroll down, there used to be a notice that
Vincent Warmerdam: basically said, hey, this is a--
Vincent Warmerdam: I'm not running this--
Vincent Warmerdam: Yeah.
Vincent Warmerdam: There you go.
Vincent Warmerdam: We welcome-- yeah, contribution guide.
Vincent Warmerdam: We welcome bug reports.
Vincent Warmerdam: Yeah, this is a way where you can basically
Vincent Warmerdam: stream updates to S3.
Vincent Warmerdam: And the main observation there is S3 is actually really cheap
Vincent Warmerdam: if all you do is push stuff into it.
Vincent Warmerdam: If you never pull it out, usually getting it out
Vincent Warmerdam: is the expensive bit of S3.
Vincent Warmerdam: So this is like pennies on the dollar
Vincent Warmerdam: for really decent backup.
Vincent Warmerdam: And you can also send it to multiple--
Vincent Warmerdam: you can send it to Amazon and also to DigitalOcean,
Vincent Warmerdam: if you like.
Michael Kennedy: Yeah.
Michael Kennedy: Yeah, because these days, S3 is really
Michael Kennedy: a synonym for blob storage on almost any hosting platform.
Michael Kennedy: Like, it used to be S3 might go to literally S3 at AWS.
Michael Kennedy: But now it's like, or DigitalOcean object spaces,
Michael Kennedy: or to you name it.
Michael Kennedy: They've all adopted the API, kind of like OpenAI's API.
Vincent Warmerdam: Yeah, I will say it's a little bit awkward that you have to--
Vincent Warmerdam: like, sometimes you go to a cloud provider,
Vincent Warmerdam: and they say, you have to download a SDK
Vincent Warmerdam: from a competing cloud provider, and then you
Vincent Warmerdam: can connect to our cloud bucket.
Michael Kennedy: I know.
Michael Kennedy: And it's usually Bodo 3.
Michael Kennedy: And Bodo 3 is--
Michael Kennedy: if you want to cry because you're using a library,
Michael Kennedy: like, Bodo 3 has a good chance of being the first one
Michael Kennedy: to make you do it.
Michael Kennedy: It is so bad for me.
Michael Kennedy: It's so not custom--
Michael Kennedy: It's not built with craft and love.
Michael Kennedy: It's like auto-generated where you pass these--
Michael Kennedy: like, you pass this kind of dictionary,
Michael Kennedy: and then the other argument takes a separate dictionary
Michael Kennedy: that relates back-- it's just like,
Michael Kennedy: could you give me a real API here?
Vincent Warmerdam: IAN MCKAYAN: I mean, the one thing
Vincent Warmerdam: I can appreciate about Bodo that I do think is honest to mention
Vincent Warmerdam: is they do try to just maintain it.
Vincent Warmerdam: The backward compatibility of that thing
Vincent Warmerdam: also means it can't move in any direction as well.
Vincent Warmerdam: And I can't-- there is this meme where Google kills all
Vincent Warmerdam: of its products way too early, and Amazon's meme
Vincent Warmerdam: that they kill them way too late, sometimes never.
Vincent Warmerdam: Right?
Vincent Warmerdam: So in that sense, I can appreciate that they just
Vincent Warmerdam: try to keep Bodo just not necessarily as user friendly,
Vincent Warmerdam: but they do keep it super stable.
Vincent Warmerdam: Like, I get there's a balance there.
Michael Kennedy: Yeah.
Michael Kennedy: I feel like we still haven't really introduced this cache.
Michael Kennedy: We've kind of set the stage.
Vincent Warmerdam: Anyway, but yeah, SQLite, super cool.
Vincent Warmerdam: How does it work under the hood?
Vincent Warmerdam: Well, it's really just like a Python dictionary.
Vincent Warmerdam: So you can say something like, hey, make a new cache.
Vincent Warmerdam: And then you can do things like cache, square brackets,
Vincent Warmerdam: string name, equals, and then whatever
Vincent Warmerdam: Python object you like can go in. And Python has this serialization method called a pickle.
Vincent Warmerdam: Serialization just means, well, you can persist it to disk in some way, and then you can sort of
Vincent Warmerdam: get it back into memory again. And that's what disk cache just uses under the hood. So in theory,
Vincent Warmerdam: any Python object that you can think of can go into disk cache. The only sort of thing to be
Vincent Warmerdam: mindful of is if you have like Python version, if NumPy version 1 in Python 3.6, and you're going
Vincent Warmerdam: to inject a whole lot of that into this cache.
Vincent Warmerdam: Don't expect those objects to serialize nicely back
Vincent Warmerdam: if you're using Python 3.12 and NumPy version 2 or something.
Michael Kennedy: Right, because pickle is almost an in-memory representation
Michael Kennedy: of the thing.
Michael Kennedy: And that may have evolved over time.
Michael Kennedy: That's also a true statement about your own classes,
Michael Kennedy: potentially.
Vincent Warmerdam: Yeah, so if you're dealing with multiple Python versions
Vincent Warmerdam: and multiple versions of different packages,
Vincent Warmerdam: there's a little bit of a danger zone to be aware of there.
Vincent Warmerdam: That said, for most of the stuff that I do,
Vincent Warmerdam: that's basically a non-issue.
Vincent Warmerdam: But I do get this nice little object that can just store stuff into SQLite and can get it out.
Vincent Warmerdam: And it's very general.
Vincent Warmerdam: It's going to try to be clever about it.
Vincent Warmerdam: Like if you give it an int, it's going to actually store it as an int and not use the pickle format.
Vincent Warmerdam: So there's a couple of clever things that it can do.
Vincent Warmerdam: And it's also really like a Python dictionary.
Vincent Warmerdam: So you can do the square bracket thing.
Vincent Warmerdam: You can also do the delete and then cache square bracket thing to delete a key from the cache.
Vincent Warmerdam: Just like a Python dictionary, you have the get method.
Vincent Warmerdam: So you can say dot get key.
Vincent Warmerdam: And if it's missing, you can pass a default value.
Vincent Warmerdam: So it's very much like a dictionary.
Vincent Warmerdam: I think Bob's your uncle on that one.
Vincent Warmerdam: Unless, Michael, I've forgotten something.
Vincent Warmerdam: But I think that's the simplest way to do it.
Michael Kennedy: Yeah, pretty much.
Michael Kennedy: Yeah, I think so.
Michael Kennedy: The difference being it's not in memory.
Michael Kennedy: It's stored to a file.
Michael Kennedy: It happens-- it's not always a SQLite file.
Michael Kennedy: But often, it is a SQLite file as its core foundation
Michael Kennedy: that it's stored to.
Michael Kennedy: So it gives you process restart ability,
Michael Kennedy: where it still remembers the stuff you cached.
Michael Kennedy: It's not like LRU cache.
Michael Kennedy: We got to redo it every single time.
Michael Kennedy: And I think, I don't know where it is in the docs here,
Michael Kennedy: but the thread safety bit of it and the cross-process safety
Michael Kennedy: is really nice about, is it persistent?
Michael Kennedy: You've got this whole table here, things like, is it persistent?
Michael Kennedy: Yes.
Michael Kennedy: Is it thread safe?
Michael Kennedy: Yes.
Michael Kennedy: Is it process safe?
Michael Kennedy: Yes.
Michael Kennedy: Compared against other things people might choose.
Michael Kennedy: And that, honestly, I think that is the other half of the magic.
Vincent Warmerdam: Yeah, so especially for your web stuff, I would say that that's the thing you really want.
Vincent Warmerdam: And some of that, of course, is just SQLite itself.
Vincent Warmerdam: Historically, one reason why people always used to say, like, use Postgres, not SQLite,
Vincent Warmerdam: has to do with precisely this concurrency stuff.
Vincent Warmerdam: My impression is that SQLite is really good at reading, but writing can be slow if multiple processes do it.
Vincent Warmerdam: Some of that, I think, is related to the disk as well.
Vincent Warmerdam: I don't know to what extent that has changed.
Vincent Warmerdam: But historically, at least, whenever I was doing Django, hanging out at Django events,
Vincent Warmerdam: People are always saying, like, just use Postgres
Vincent Warmerdam: because it's better for the web thing.
Vincent Warmerdam: But it is safe, the SQLite.
Vincent Warmerdam: It might become slower, but it is thread safe if it's--
Vincent Warmerdam: MARK MANDEL: Right.
Michael Kennedy: There's actually-- they've thought a lot about in this thing
Michael Kennedy: about transactions, concurrency, and basically dealing with that.
Michael Kennedy: But it is ultimately, for the most part,
Michael Kennedy: still SQLite underneath.
Michael Kennedy: But the thing with a cache is if you're writing it more
Michael Kennedy: than you're reading it, you probably shouldn't have a cache.
Michael Kennedy: Yeah.
Michael Kennedy: I mean, like...
Vincent Warmerdam: That beats the purpose.
Michael Kennedy: Exactly.
Michael Kennedy: Like, you get no value if you're recreating it.
Michael Kennedy: You're only probably just doing overhead and wasting memory or disk space.
Michael Kennedy: So it's inherently a situation where it's going to be pretty read-heavy,
Michael Kennedy: and SQLite is good at read-heavy scenarios.
Vincent Warmerdam: And maybe it's also fair to say, like, the LRU cache that you get with basic Python,
Vincent Warmerdam: so also to maybe explain that one,
Vincent Warmerdam: so the LRU cache is a little bit different because you decorate a function with it,
Vincent Warmerdam: and then given the same inputs, one output goes out,
Vincent Warmerdam: you can kind of keep track of a dictionary that's in memory.
Vincent Warmerdam: If you don't have a lot of stuff to keep in the back of your mind,
Vincent Warmerdam: then maybe you don't have to write to disk, right?
Vincent Warmerdam: So there's also maybe a reason to just stick to caching mechanisms
Vincent Warmerdam: that use Python memory, because I also think,
Vincent Warmerdam: I would imagine it to be quicker too.
Vincent Warmerdam: But maybe that's also...
Vincent Warmerdam: Probably, yes.
Vincent Warmerdam: That should be quicker.
Vincent Warmerdam: It's just that if you're capped at memory,
Vincent Warmerdam: then you might want to spill to disk,
Vincent Warmerdam: and then disk cache becomes interesting too.
Michael Kennedy: Right, for example, you have literally zero serialization, deserialization.
Michael Kennedy: What you put in LRU cache is the pointer to the object
Michael Kennedy: that you're caching, right?
Michael Kennedy: If you've got a class or a list that's part of the LRU cache.
Vincent Warmerdam: The one thing that is good to mention
Vincent Warmerdam: is also a really nice feature of disk cache
Vincent Warmerdam: is just like LRU cache has a decorator,
Vincent Warmerdam: so you can decorate a function, disk cache also has that.
Vincent Warmerdam: And it works kind of interestingly, too.
Vincent Warmerdam: So when you decorate the function,
Vincent Warmerdam: you do have to be a little bit careful if you use that.
Vincent Warmerdam: But then disk cache will--
Vincent Warmerdam: I think it will hash the function name and the inputs
Vincent Warmerdam: that you pass.
Vincent Warmerdam: I don't know if it also hashes the contents of the function.
Vincent Warmerdam: Like if you change the function itself,
Vincent Warmerdam: I don't know if this cache will actually put that
Vincent Warmerdam: in a different slots, if that makes sense.
Michael Kennedy: Yeah, you can say @cache_memoize,
Michael Kennedy: which is the design pattern speak for just remember this.
Michael Kennedy: Yeah.
Michael Kennedy: It takes the arguments.
Vincent Warmerdam: And then it has like the Fibonacci sequence,
Vincent Warmerdam: which is the classic example, of course.
Vincent Warmerdam: And like there are some extra things you can set there as well.
Vincent Warmerdam: So you can say things like, hey, I think you're able to--
Vincent Warmerdam: yeah, you're able to set the expiry.
Vincent Warmerdam: So you can say things like, I want to cache this,
Vincent Warmerdam: but only for the next five minutes or so,
Vincent Warmerdam: which can make a lot of sense if you're doing a front page
Vincent Warmerdam: kind of a thing.
Vincent Warmerdam: So like the Reddit front page or something
Vincent Warmerdam: like that that updates, but not every second.
Vincent Warmerdam: It probably updates once every five minutes
Vincent Warmerdam: or something like that.
Vincent Warmerdam: And then you do want to have something that's cached,
Vincent Warmerdam: but then after that, you want the cache
Vincent Warmerdam: to maybe basically just reset.
Vincent Warmerdam: And that is something you can also
Vincent Warmerdam: control with a few parameters here.
Michael Kennedy: Right, that's interesting.
Michael Kennedy: There's a couple good use cases that come to mind for me.
Michael Kennedy: Like one, if I put this on the function that
Michael Kennedy: generated the RSS feed for Talk Python,
Michael Kennedy: I could just say every one minute
Michael Kennedy: and then it might be a little bit expensive to compute
Michael Kennedy: because it's got a pars, you know, 535 episodes or whatever.
Michael Kennedy: But then for one minute, all the subsequent requests,
Michael Kennedy: just here's the answer, here's the answer.
Michael Kennedy: And then without me managing anything,
Michael Kennedy: it will just automatically the next minute refresh itself
Michael Kennedy: by the nature of how it works, right?
Vincent Warmerdam: How much traffic do you get on that endpoint?
Vincent Warmerdam: Just roughly, like you're asking.
Michael Kennedy: One terabyte of RSS a month.
Michael Kennedy: Okay, gotcha.
Vincent Warmerdam: Okay, but there you go.
Vincent Warmerdam: Like then just doing that like once a minute instead of like many times a minute will be a huge cookie.
Michael Kennedy: I would say it's probably more than one request a second.
Michael Kennedy: And the file size, the response size of the RSS feed is over a meg.
Michael Kennedy: And so it's a non-trivial amount of asking, you know.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: And then like, and how do you fix that with a whole bunch of infrastructure?
Vincent Warmerdam: No, with a decorator.
Vincent Warmerdam: Like that feels...
Vincent Warmerdam: Exactly.
Michael Kennedy: Exactly.
Michael Kennedy: You pretty much summed up all the reasons why I'm so excited about this,
Michael Kennedy: because it's like you could do all of this complex stuff
Michael Kennedy: or just like you could just literally
Michael Kennedy: in such a simple way,
Michael Kennedy: just not recompute it as often.
Michael Kennedy: Yeah.
Michael Kennedy: Here's the danger.
Michael Kennedy: What if there's a race condition?
Michael Kennedy: And oh my goodness, two of them sneak in.
Michael Kennedy: You know what I mean?
Michael Kennedy: Like, okay, so you've done a little bit extra work
Michael Kennedy: and you throw it away.
Vincent Warmerdam: Who cares?
Vincent Warmerdam: I want to use Redis now.
Vincent Warmerdam: And Redis is cool,
Vincent Warmerdam: but I've never used it before.
Vincent Warmerdam: Better buy a book.
Vincent Warmerdam: Okay, no.
Vincent Warmerdam: With this cache, if you,
Vincent Warmerdam: I mean, I'm sure it won't solve everything,
Vincent Warmerdam: but this, I make a bit of a joke by saying,
Vincent Warmerdam: just use a decorator.
Vincent Warmerdam: But it's honestly that feeling that this library really
Vincent Warmerdam: does give you.
Vincent Warmerdam: You can just use it as a decorator,
Vincent Warmerdam: which has a lot of great use cases.
Vincent Warmerdam: You can just use it as a dictionary.
Vincent Warmerdam: So it still feels like you're writing Python.
Vincent Warmerdam: It's just Python with one concern less.
Vincent Warmerdam: And that is the magic.
Michael Kennedy: MARK MANDEL: And it takes on so many of the cool aspects
Michael Kennedy: of these high-end servers like Postgres or Redis or Valkey.
Michael Kennedy: Valkey is sort of the shiny new Redis, right?
Vincent Warmerdam: I would actually love to do a Redis benchmark.
Vincent Warmerdam: I haven't done that yet.
Vincent Warmerdam: But one thing I do wonder with disks are getting so much faster.
Michael Kennedy: Yes.
Vincent Warmerdam: Right?
Vincent Warmerdam: And so you can actually at some point wonder like how much faster is Redis really going to be
Vincent Warmerdam: and how much money are you willing to spend on it?
Vincent Warmerdam: Because if your cache is huge and it allows to go in memory in Redis,
Vincent Warmerdam: it could be wrong, but Redis is fully in memory, I think, right?
Michael Kennedy: I believe so.
Michael Kennedy: There is a database aspect.
Michael Kennedy: Redis is weird because it could be so many.
Michael Kennedy: Redis is cool.
Michael Kennedy: It can do a lot.
Michael Kennedy: But they do have, they actually have benchmarks here on.
Vincent Warmerdam: Ah, there you go.
Michael Kennedy: Compared against memcached and Redis.
Michael Kennedy: And it has the get speed and the write speed.
Michael Kennedy: And this is smaller is better.
Michael Kennedy: Yeah, look at this.
Michael Kennedy: Disc cache beats Redis.
Vincent Warmerdam: And I imagine that's because of the network hop or something.
Michael Kennedy: Yeah, exactly.
Michael Kennedy: I bet it's the network, the network connection.
Michael Kennedy: So if you're running it on the same machine,
Vincent Warmerdam: you would have a different number there.
Vincent Warmerdam: Might be good to maybe caveat that.
Michael Kennedy: Yeah, it might be.
Vincent Warmerdam: I mean, but also that I think that I don't know when they ran this benchmark,
Vincent Warmerdam: but I just checked on PyPI.
Vincent Warmerdam: This project started in 2016.
Vincent Warmerdam: So it might-
Michael Kennedy: I bet this is 2016 data right here.
Michael Kennedy: If I know how these docs go.
Vincent Warmerdam: Yeah, so it could also be that those are old disks
Vincent Warmerdam: comparing old memories, right?
Michael Kennedy: So then this is one of those weird benchmarks.
Vincent Warmerdam: You got to really run them every six months or so
Vincent Warmerdam: for them to remain relevant.
Michael Kennedy: Yeah, yeah.
Michael Kennedy: I mean, the new NVM, VVM, whatever, disks,
Vincent Warmerdam: SSD disks are so fast.
Vincent Warmerdam: And also they're not memory.
Vincent Warmerdam: Memory is expensive nowadays.
Vincent Warmerdam: Yes, it is.
Vincent Warmerdam: People want to build data centers with them, I've heard.
Vincent Warmerdam: Yeah, yeah.
Michael Kennedy: And on the cloud there, this is a totally,
Michael Kennedy: this is another really interesting aspect to discuss.
Michael Kennedy: Probably more of a web dev side of things.
Michael Kennedy: But if you do LRU caches or even to a bigger degree,
Michael Kennedy: I run a whole separate server,
Michael Kennedy: even if it is just a Docker container
Michael Kennedy: that holds a bunch of this stuff in memory,
Michael Kennedy: that's going to take more memory on your VM
Michael Kennedy: or your cloud deployment or whatever.
Michael Kennedy: And if you just say, well,
Michael Kennedy: I have this 160 gig hard drive. That's an NVVM high speed drive. Like maybe I could just put a
Michael Kennedy: bunch of stuff there and you could really thin down your deployments, not just because it's not in
Michael Kennedy: memory in a cache somewhere, but if you're not having any form of cache, you might be able to
Michael Kennedy: dramatically lower how much compute you need and avoid them. Right. Like there's layers of how this
Vincent Warmerdam: could like shave off. And again, it's one of those things of like, oh, I just, can I pay for disk
Vincent Warmerdam: instead? Oh, that's a whole lot cheaper. What else do I got to do? You just got to write a tech
Michael Kennedy: creator yeah i think i pay five dollars for uh i think i remember exactly but i pay something like
Michael Kennedy: five dollars for 400 gigs of disc there you go and do you know how much 400 gigs of ram will cost
Vincent Warmerdam: on the cloud um well i mean more there goes the college tuition but the exactly sorry kids yeah
Vincent Warmerdam: no but like it's um and again like i vividly remember when i started college people were
Vincent Warmerdam: always saying oh keep it in memory because it's way faster than disc but i i think we got to let
Vincent Warmerdam: a lot of that stuff just go.
Michael Kennedy: Interesting idea. Yeah, I agree, though.
Michael Kennedy: I think you're right. But anyway,
Vincent Warmerdam: so far, we've mainly
Vincent Warmerdam: been discussing the mechanics of it, but
Vincent Warmerdam: there's some bills and whistles I think we should maybe also mention.
Vincent Warmerdam: The expiry definitely is one of them.
Vincent Warmerdam: There's also first in, first out
Vincent Warmerdam: kinds of things that you can do.
Vincent Warmerdam: So maybe if we could go back to that.
Michael Kennedy: Yeah, there's a bunch of features, actually,
Michael Kennedy: there.
Michael Kennedy: The expiry is interesting already
Michael Kennedy: because
Michael Kennedy: if you do regular, say, LRU caching
Michael Kennedy: like that, you have a natural.
Michael Kennedy: It's going to go away when the process
Michael Kennedy: restarts, and that's going to happen eventually.
Michael Kennedy: Even in a web app, you ship a new
Michael Kennedy: version, you've got to restart the thing or something.
Michael Kennedy: But when it goes to the disk,
Michael Kennedy: it starts to pile up, right? That's why I have this little admin
Michael Kennedy: page that has, like, how big is this
Michael Kennedy: and a button to clear it.
Vincent Warmerdam: In fairness, that can blow up
Vincent Warmerdam: to a lot of Palooza as well, if you're not careful.
Michael Kennedy: It is a concern. Yeah, it is definitely a concern.
Michael Kennedy: And so I think a big way to fix it,
Michael Kennedy: like the expiry, we already talked about why it's
Michael Kennedy: interesting for stale data.
Michael Kennedy: You want it to just like auto refresh,
Michael Kennedy: but it's also just a safeguard of maybe we'll just recompute this once a month.
Michael Kennedy: It's really quick and easy.
Michael Kennedy: Yeah.
Michael Kennedy: Maybe just don't let it linger forever.
Vincent Warmerdam: I think what you can also do though, if I'm not mistaken,
Vincent Warmerdam: is I think you can also set like a max key size.
Vincent Warmerdam: So you can say this cache, this particular disk cache can only have 10,000 keys in it
Vincent Warmerdam: and use a first in first out kind of a principle.
Michael Kennedy: Right, or last accessed or number of times.
Michael Kennedy: There's a bunch of metrics there actually for how that works.
Michael Kennedy: Yeah, it's pretty interesting.
Vincent Warmerdam: I've never had to fiddle around with them too much,
Vincent Warmerdam: but it's one of those things where even if I don't need it right now,
Vincent Warmerdam: it is just a relief to see that the feature is there
Vincent Warmerdam: in case you might need it later.
Michael Kennedy: Yeah, yeah, for sure.
Michael Kennedy: Let me see if I can find out where that is.
Michael Kennedy: I don't know, keep bouncing around the same spot.
Michael Kennedy: I've got it, let's just talk through them.
Michael Kennedy: So I think the tags and expiries are pretty interesting,
Michael Kennedy: but then there's also, I think, something that surprised me a little bit
Michael Kennedy: is these different kinds of caches.
Michael Kennedy: So there's like a fan out cache.
Michael Kennedy: Have you looked at these?
Michael Kennedy: These are interesting.
Vincent Warmerdam: I remember reading about, I've never used them, but I do remember reading them.
Michael Kennedy: So let me give you the quick rundown and then you'll understand instantly.
Michael Kennedy: It's super quick.
Michael Kennedy: So it uses sharding, which is a database term, right?
Michael Kennedy: So sharding is like, if I've got a billion records and it's a challenge to put them all
Michael Kennedy: in the same database entry, database table or server, I could actually have 10 servers
Michael Kennedy: and decide, well, okay, what we're going to do is if it's a number of the ID of the user,
Michael Kennedy: if the first number is one, then they go into this database.
Michael Kennedy: If it's two, then they go in that, right?
Michael Kennedy: So 2005 goes into the second database and so on.
Michael Kennedy: So it does that as well.
Michael Kennedy: This is one of the things it does to try to avoid the issues of multiple writers, I believe.
Michael Kennedy: So you're less likely to write to the same database.
Michael Kennedy: So it doesn't have to lock as hard.
Vincent Warmerdam: It's kind of what you do where you say, oh, I've got this for the YouTube link.
Vincent Warmerdam: And I've got this for the HTML markdown.
Vincent Warmerdam: Except those are like different chunks because of their use case.
Vincent Warmerdam: But you can also imagine, well, I've got this long list of users.
Vincent Warmerdam: but I still want to benefit from having multiple SQLite instances.
Vincent Warmerdam: And I suppose that's when you use this, right?
Michael Kennedy: Yeah, I think that is why.
Michael Kennedy: So it says, it's built on top of cache, cache fanout, automatically shards.
Michael Kennedy: Automatically, you just said how many you want, it figures out what that means.
Michael Kennedy: And it says, while readers and writers don't block each other,
Michael Kennedy: writers block other writers.
Michael Kennedy: Therefore, a shard for every concurrent writer suggests it.
Michael Kennedy: This will depend on your scenario, default is eight.
Michael Kennedy: So that's pretty cool.
Vincent Warmerdam: Yeah, okay.
Vincent Warmerdam: And presumably internally does something like hashing to figure out how to like
Michael Kennedy: send it around the shards.
Michael Kennedy: Right.
Michael Kennedy: The keys themselves have to be hashable anyway, probably.
Michael Kennedy: So just hashes the key and then shards on like the first couple letters or whatever.
Michael Kennedy: Yeah, this is cool.
Michael Kennedy: So it avoids the concurrency crashes.
Michael Kennedy: The one difference for me, the reason I didn't choose fanout cache is because I want to be
Michael Kennedy: able to say, I want to clear all the YouTube IDs, but I want to keep the really expensive
Michael Kennedy: to compute hashes.
Michael Kennedy: I want to be able to clear stuff if I really have to by category.
Michael Kennedy: And I guess you could also do that with tags, but I'm just not that advanced.
Vincent Warmerdam: Well, and also keep it simple, right?
Vincent Warmerdam: And again, it's one of those things where it's, oh, it's nice to know that this is in here,
Vincent Warmerdam: even if you don't use it directly.
Vincent Warmerdam: I do agree.
Vincent Warmerdam: This is a really nice feature.
Michael Kennedy: It is.
Michael Kennedy: And I probably will never in my life use it, but it's really cool that it's like, you know,
Michael Kennedy: it's one of those things about a library that when you're thinking about picking it, it's
Michael Kennedy: like, okay, the core feature is great, but if I outgrow it, what is my next step?
Michael Kennedy: Do I have to completely switch to something really different
Michael Kennedy: like Redis or Valkey?
Michael Kennedy: Or do I just change the class I'm using?
Vincent Warmerdam: I had that with this.
Vincent Warmerdam: So I actually had that feeling a while ago.
Vincent Warmerdam: We're going to get to my example I think a bit later.
Vincent Warmerdam: But I was using the Memoize decorator
Vincent Warmerdam: to decorate a function to properly cache that.
Vincent Warmerdam: But the one issue I had is an input to that function
Vincent Warmerdam: was a progress bar.
Vincent Warmerdam: It's kind of a remote specific thing.
Vincent Warmerdam: I wanted this one progress bar to update
Vincent Warmerdam: from inside of this one function.
Vincent Warmerdam: But the downside was that every time
Vincent Warmerdam: I rerun the notebook, I make a new progress bar object.
Vincent Warmerdam: Oh, and that means that the input
Vincent Warmerdam: has a different object going in.
Vincent Warmerdam: MARK MANDEL: So you never actually hit the cache.
Vincent Warmerdam: JOHN MUELLER: So, oh my god, I'm never hitting the cache.
Vincent Warmerdam: This is horrible.
Vincent Warmerdam: Then it turns out the Memoize decorator
Vincent Warmerdam: also allows you to ignore a couple of the inputs
Vincent Warmerdam: of the function.
Vincent Warmerdam: So you can also--
Vincent Warmerdam: MARK MANDEL: Oh, interesting.
Michael Kennedy: Like ignore by keyword or something, keyword argument.
Vincent Warmerdam: JOHN MUELLER: Precisely.
Vincent Warmerdam: And there's a bunch of use cases for it, and this was one.
Vincent Warmerdam: And you can just imagine my relief
Vincent Warmerdam: after writing the entire notebook to then look at the docs
Vincent Warmerdam: and go, oh, sweet.
Michael Kennedy: Yeah, that's super sweet.
Michael Kennedy: Yeah.
Michael Kennedy: Okay.
Michael Kennedy: I think there's right below this.
Michael Kennedy: Yeah, there's a couple.
Michael Kennedy: We can just go down this list here.
Michael Kennedy: There's some cool.
Michael Kennedy: Oh, Django.
Michael Kennedy: So they have a legit Django cache.
Vincent Warmerdam: Yeah, yeah, yeah.
Michael Kennedy: Straight in.
Vincent Warmerdam: Sweet.
Vincent Warmerdam: Yeah, I recall a little bit.
Vincent Warmerdam: It was like a huge Django following for this thing.
Michael Kennedy: Yeah, and I think this is a part why.
Michael Kennedy: And when I first saw it, the reason it got sent to me is somebody's like,
Michael Kennedy: oh, this disk cache, Django cache is a really cool thing to just drop into Django.
Michael Kennedy: And I'm like, that's cool.
Michael Kennedy: I'm not using Django, but I admire it.
Michael Kennedy: That's why I didn't really look into it until I saw your use case outside of Django.
Michael Kennedy: I'm like, oh, okay, I understand how much this can do.
Michael Kennedy: So the Django dish cache says it uses the fanout cache,
Michael Kennedy: which we just discussed with the sharding,
Michael Kennedy: to provide a Django-compatible cache interface.
Michael Kennedy: And you just do that in your settings file,
Michael Kennedy: and you just say the back end is diskcache.django cache,
Michael Kennedy: and you give it a location.
Michael Kennedy: Boom, off it goes, right?
Michael Kennedy: So really, really nice.
Michael Kennedy: Cool.
Michael Kennedy: Yeah, and it sounds like you've done more Django than me.
Michael Kennedy: How's this sit with you?
Vincent Warmerdam: I mean, to be very clear, I do think Django is really nice and really mature.
Vincent Warmerdam: I do sometimes have a bit of a love-hate relationship with it because Django can go really, really deep.
Vincent Warmerdam: And some of that configuration stuff definitely can be a little bit in your face.
Vincent Warmerdam: So the main thing I just want to observe is doing everything manually inside of Django can be very time-consuming.
Vincent Warmerdam: So it's definitely nice to know that someone took the effort to make a proper Django plugin in the way that Django wants it.
Vincent Warmerdam: That's definitely the thing to appreciate here.
Vincent Warmerdam: I've never really used this in a Django app, to be honest.
Michael Kennedy: Yeah.
Michael Kennedy: You know, it has a lot of nice settings here.
Michael Kennedy: Like you can set the number of shards, the timeout, so in case there's a write or read
Michael Kennedy: contention, it can deal with that or it can at least let you know you're failing.
Michael Kennedy: It even has a size limit.
Vincent Warmerdam: Does it say how you can configure what to cache and whatnot?
Vincent Warmerdam: Or is like the-- I've never really used caches in Django in general.
Vincent Warmerdam: So I don't know if there's a general cache feature in Django itself that it will just
Vincent Warmerdam: plug into or if--
Michael Kennedy: I think there is a general cache feature in Django.
Michael Kennedy: The Django people are like screaming silently.
Michael Kennedy: Yes.
Michael Kennedy: I apologize.
Michael Kennedy: I know, I know.
Michael Kennedy: But I'm pretty sure it's just like a built-in Django cache functionality.
Michael Kennedy: Exactly.
Michael Kennedy: Yeah, okay.
Michael Kennedy: It just routes into this thing.
Vincent Warmerdam: Exactly.
Vincent Warmerdam: So instead of configuring Redis, you just feed it this and you're good.
Vincent Warmerdam: That's the idea.
Michael Kennedy: Yes, exactly.
Michael Kennedy: Exactly.
Michael Kennedy: So there's more.
Michael Kennedy: The next one, I have to rage against the machine.
Michael Kennedy: I'm sure it's the way, but DQ, pronounced deck.
Vincent Warmerdam: Yeah.
Michael Kennedy: So I don't know.
Michael Kennedy: For me, I still say DQ.
Michael Kennedy: I don't say deck.
Michael Kennedy: like it's spelled D-E-Q-U-E.
Michael Kennedy: And I know a lot of computer science people just call that deck,
Michael Kennedy: but this cache dot DQ or deck, however you want to use it.
Michael Kennedy: It provides, there's,
Michael Kennedy: there's a couple of higher order data structures that like operate on what we
Michael Kennedy: talked about so far, but to give you data structure behavior, right?
Michael Kennedy: Like what we talked about so far is sort of dictionary,
Michael Kennedy: but not list or order or any of that.
Michael Kennedy: But this, you can actually do go and add a thing.
Michael Kennedy: How do we add one?
Michael Kennedy: Anyway, you can say like pop left, pop left.
Michael Kennedy: over and over to get, I guess you just add it.
Vincent Warmerdam: It's kind of like a queue.
Michael Kennedy: Yeah, like a queue, exactly.
Michael Kennedy: With the goal of taking stuff out of it instead of into it.
Michael Kennedy: But you don't normally think of a cache as doing that.
Michael Kennedy: But it'd be a cool way actually to fan out work across processes.
Vincent Warmerdam: I was about to say, that's a really good, I think,
Vincent Warmerdam: I mean, there's people that have made, I forget the name,
Vincent Warmerdam: the Python queuing system, salary.
Vincent Warmerdam: So that one is also built in such a way that you can say like,
Vincent Warmerdam: oh, where do you have the list of jobs that still need doing?
Vincent Warmerdam: And I think also Redis is used--
Vincent Warmerdam: MARK MANDEL: Yeah, right, Redis and Qum.
Michael Kennedy: Yeah, exactly.
Michael Kennedy: Or Rabbit and Qum as well.
Vincent Warmerdam: FRANCESC CAMPOY: Yeah, exactly.
Vincent Warmerdam: But you can configure SQLite if you want to, though,
Vincent Warmerdam: if I recall with those.
Vincent Warmerdam: It's just that in this particular case,
Vincent Warmerdam: if you don't want to use salary, you can still kind of roll
Vincent Warmerdam: your own by using this cache as well.
Vincent Warmerdam: I'm assuming it uses the same pickle tricks,
Vincent Warmerdam: so you can do general Python things
Vincent Warmerdam: and if the process breaks for whatever reason,
Vincent Warmerdam: you still have the jobs that need doing.
Michael Kennedy: MARK MANDEL: Yeah, we're still going
Michael Kennedy: to have to talk about this serialization thing,
Michael Kennedy: These pickles.
Michael Kennedy: Yes.
Michael Kennedy: Not yet.
Michael Kennedy: Let's go through this.
Michael Kennedy: Let's go through this first.
Michael Kennedy: Let's first.
Michael Kennedy: Before we get distracted, because it's a deep.
Michael Kennedy: So DEC, I guess we'll go DEC.
Michael Kennedy: DEC provides an efficient and safe means for cross-thread, cross-process communication.
Michael Kennedy: Like, you would never think you would get that out of a cache, really.
Vincent Warmerdam: But it's...
Michael Kennedy: You would do that in SQLite.
Michael Kennedy: Yeah, exactly.
Michael Kennedy: But you would do work to do that, right?
Michael Kennedy: You would do like transactions.
Michael Kennedy: And you would do sorting.
Michael Kennedy: You would figure out, well, what if there's contention?
Michael Kennedy: I mean, the fact that it's just kind of a pop, it's pretty nice.
Vincent Warmerdam: Yeah, that's definitely nice.
Michael Kennedy: No, that's definitely true.
Vincent Warmerdam: Although one thing that makes it easy in this case, though, is again, it is all running in like one process.
Vincent Warmerdam: So it's not like we've got SQLite running in one place and there's 16 Docker containers that can randomly interact with it.
Vincent Warmerdam: No, that can, though.
Vincent Warmerdam: Is that the case?
Michael Kennedy: Because disk cache itself is already cross-process safe.
Michael Kennedy: Like, that's why I was so excited about it.
Vincent Warmerdam: But it has to be on the same machine, though.
Vincent Warmerdam: Like, that's what I do think.
Michael Kennedy: Yes, it's got to at least be accessible.
Michael Kennedy: RushRite, that is true.
Michael Kennedy: because technically there's nothing that says
Michael Kennedy: you can't put the file anywhere.
Michael Kennedy: I think there's mega performance issues
Michael Kennedy: and locking issues on,
Michael Kennedy: it says basically don't use it on network drives.
Vincent Warmerdam: Yeah, so that's the thing.
Vincent Warmerdam: Some of this is like, okay, you can do the locking.
Vincent Warmerdam: You can do all those things.
Vincent Warmerdam: You can do it well,
Vincent Warmerdam: but the practicality of the network overhead
Vincent Warmerdam: is something that usually causes a lot of confuffle,
Michael Kennedy: at least in my experience.
Michael Kennedy: Yeah.
Michael Kennedy: Okay, another one is disk cache index,
Michael Kennedy: which creates a mutable mapping and ordered dictionary.
Michael Kennedy: So if you kind of really want to lay into the dictionary side,
Vincent Warmerdam: Yeah, you can do that.
Michael Kennedy: That one's transactions as well.
Michael Kennedy: So you can actually, it has like sort of in-place updates
Michael Kennedy: and other things you can do.
Michael Kennedy: So you can say, I want to make sure that I'm going to get
Michael Kennedy: two different things out of the cache.
Michael Kennedy: And I want to make sure that they're not changed
Michael Kennedy: while I'm doing that, right?
Michael Kennedy: Just like you would with threading or something.
Michael Kennedy: Yeah, so nothing can happen in between.
Vincent Warmerdam: So they both have to come out at the same time.
Vincent Warmerdam: So the two values that I get, they were, they were,
Vincent Warmerdam: they both existed at the same time in the cache
Vincent Warmerdam: at the point in time that I was retrieving it.
Michael Kennedy: Yeah, yeah, exactly.
Michael Kennedy: So just with cache.transact and you just go to town on it.
Michael Kennedy: That's pretty straightforward, right?
Michael Kennedy: Yep.
Michael Kennedy: Are there any more in here?
Michael Kennedy: There's a bunch of recipes for like barriers and throttling and probably semaphore-like stuff,
Michael Kennedy: but I don't really want to talk about.
Michael Kennedy: But you touched on these eviction policies.
Michael Kennedy: Here's where I was looking for.
Michael Kennedy: There's these different ones here that are kind of cool.
Michael Kennedy: Whoops.
Michael Kennedy: I didn't go away.
Vincent Warmerdam: Yeah, so you can set a maximum to the cache.
Vincent Warmerdam: I think you do that by number of items typically in it.
Vincent Warmerdam: It could also be the case.
Michael Kennedy: Size or something, yeah.
Vincent Warmerdam: Yeah, or like total disk size,
Vincent Warmerdam: maybe we should double check.
Michael Kennedy: The default for the disk size is one gig.
Michael Kennedy: Yeah, there you go.
Michael Kennedy: So there's already a built-in one, yeah.
Michael Kennedy: Which might catch people off guard.
Michael Kennedy: Much of the stuff is cash, but not always.
Michael Kennedy: I don't understand.
Vincent Warmerdam: Yeah, you've got to be a little bit mindful of that,
Vincent Warmerdam: I suppose.
Vincent Warmerdam: But it's the same default,
Vincent Warmerdam: not to have it go to infinity.
Vincent Warmerdam: Agreed.
Vincent Warmerdam: Yeah, I guess small screen on my side, but like, yeah, last recently.
Michael Kennedy: I'll read them out.
Michael Kennedy: I'll read them out for you.
Michael Kennedy: Yeah.
Michael Kennedy: So we got recent last recently stored as a default, every cache item
Michael Kennedy: records the time it was stored in the cache and that adds an index to that
Michael Kennedy: field.
Michael Kennedy: So it's nice and fast, which is cool.
Michael Kennedy: We have, this is, there's some other ones that are more nuanced, like least
Michael Kennedy: recently used, not in terms of time, but we've got one that was accessed a hundred
Michael Kennedy: times and one that was accessed two times.
Michael Kennedy: Even if the one is accessed two times was just accessed,
Michael Kennedy: that one's getting kicked out because it's not as useful.
Michael Kennedy: I don't know.
Michael Kennedy: That's a pretty neat feature.
Michael Kennedy: And then the one people would expect is, I don't know,
Michael Kennedy: maybe at least recently used.
Michael Kennedy: How are you?
Vincent Warmerdam: Yeah.
Vincent Warmerdam: Yeah, exactly.
Vincent Warmerdam: And there's also pruning mechanisms, if I'm not mistaken.
Vincent Warmerdam: So there's all sorts of fun.
Vincent Warmerdam: You can argue there are bells and whistles until you need them.
Vincent Warmerdam: And one thing I have always found is every item that I see here,
Vincent Warmerdam: you might not need it right now,
Vincent Warmerdam: but for every item you see, you do plausibly go,
Vincent Warmerdam: oh, but that might be useful later down the line somewhere.
Vincent Warmerdam: Like the transaction thing where you retrieve two things at the same time.
Vincent Warmerdam: I don't really have a use case for it, but I can imagine it one might,
Vincent Warmerdam: where the consistency really matters.
Michael Kennedy: Yeah, I can.
Michael Kennedy: I could see using the fan out cache.
Vincent Warmerdam: Yeah, definitely.
Michael Kennedy: But probably not the transaction.
Michael Kennedy: But I'm already talking to MongoDB, which doesn't have transactions effectively.
Michael Kennedy: So not really.
Michael Kennedy: What about performance?
Michael Kennedy: Should we talk about your graphs?
Michael Kennedy: You brought pictures.
Vincent Warmerdam: Yes.
Vincent Warmerdam: So that might be, so when you told me like, hey, let's do an episode on disk cache,
Vincent Warmerdam: and I kind of told myself, okay, then I need to do some homework.
Vincent Warmerdam: Like, I actually have to use it for something real.
Vincent Warmerdam: It's a bit complex.
Vincent Warmerdam: So what we're going to try and do is we're looking at a chart right now,
Vincent Warmerdam: and I'm going to explain to Michael what it does,
Vincent Warmerdam: and I'm going to try to explain it in such a way such that if you're not watching
Vincent Warmerdam: but listening, that you're also going to be fairly interested in what you're seeing.
Michael Kennedy: And I'll link to the chart, of course, people can.
Vincent Warmerdam: Yeah, so this is all running on GitHub pages,
Vincent Warmerdam: and the charts that you see here definitely needed a bit of disk cache
Vincent Warmerdam: to make it less painful.
Vincent Warmerdam: So what I've done is I've downloaded a Git repository.
Vincent Warmerdam: What you're looking at right now
Vincent Warmerdam: is the Git repository for Marimo.
Vincent Warmerdam: And then I just take a point in time
Vincent Warmerdam: and I say, okay, let's just see all the lines of code.
Vincent Warmerdam: And then I take another point in time.
Vincent Warmerdam: And then I basically just do kind of a Git blame
Vincent Warmerdam: to see if the line got changed in between.
Vincent Warmerdam: So what you're looking at here
Vincent Warmerdam: is kind of a chart over time
Vincent Warmerdam: where it's basically like a bar chart,
Vincent Warmerdam: but it changes colors as time moves forward.
Vincent Warmerdam: and the shape that you see is that things that happened early on,
Vincent Warmerdam: well, there's a nice thick slab,
Vincent Warmerdam: but it gets a little bit thinner and thinner as time moves forward
Vincent Warmerdam: because some of those lines of code got replaced.
Vincent Warmerdam: But in the case of Marimo, you can see that, you know,
Vincent Warmerdam: most of the lines of code actually stay around for a long time
Michael Kennedy: and it's kind of like a smooth sedimentary layer every time we move forward.
Michael Kennedy: It's compressing a little over time.
Michael Kennedy: Like there's the weight of the project has sort of compressed it.
Michael Kennedy: So, yeah, it's pretty interesting.
Michael Kennedy: So, okay, so that's pretty cool.
Vincent Warmerdam: But you can also go to Django.
Michael Kennedy: So there's a director on top.
Michael Kennedy: MARK MANDEL: So if I go to Django.
Michael Kennedy: Oh, you can put this cache in here.
Michael Kennedy: This is really different.
Michael Kennedy: FRANCESC CAMPOY: Yes.
Vincent Warmerdam: MARK MANDEL: What is this telling us?
Vincent Warmerdam: FRANCESC CAMPOY: So what you can see here
Vincent Warmerdam: is that at some point in time, there's
Vincent Warmerdam: a huge shift in the sediment.
Vincent Warmerdam: There's a lot of light sand and a lot of the dark sand goes away.
Vincent Warmerdam: There's also a button that allows you to show the version number.
Vincent Warmerdam: So I've--
Vincent Warmerdam: yep, there you go.
Michael Kennedy: MARK MANDEL: There we go, yeah.
Vincent Warmerdam: FRANCESC CAMPOY: So you can see that right before a new version,
Vincent Warmerdam: a bunch of changes got introduced or right after.
Vincent Warmerdam: It's usually around the version number
Vincent Warmerdam: that you can see that shift.
Michael Kennedy: Right, once the feature freeze is lifted,
Michael Kennedy: some stuff comes in, PRs come in maybe or something.
Michael Kennedy: Yes.
Vincent Warmerdam: And one other thing that's actually kind of fun,
Vincent Warmerdam: if you go--
Vincent Warmerdam: there's this project called Psychot LEGO
Vincent Warmerdam: that you can also go ahead and select.
Vincent Warmerdam: And folks--
Michael Kennedy: I've heard a pretty cool guy maintains that, yeah.
Vincent Warmerdam: Well, so the funny thing is you can see that there's
Vincent Warmerdam: a massive shift there at some point.
Michael Kennedy: OK.
Vincent Warmerdam: That's when we got the new maintainer.
Michael Kennedy: Are you on the purple or the green side?
Vincent Warmerdam: So there's this dark blue sediment that sort of goes down massively at that point.
Vincent Warmerdam: But yeah, no, so but in this case, like the first thing he did is redid all the docs. So we went
Vincent Warmerdam: from Sphinx to make docs. And that's like a huge, if you look at the lines of code that changed as
Vincent Warmerdam: a result that, you know, that's quite a lot. But if we now start talking about how you make a chart
Vincent Warmerdam: like this, you got to imagine like, I take the start of the GitHub history, I take the end of the
Vincent Warmerdam: GitHub history, I sample like 100 points in between, and then for every line in every file,
Vincent Warmerdam: I do a git blame.
Michael Kennedy: MARK MANDEL: I think Django is something like 300,000 lines
Michael Kennedy: of code.
Michael Kennedy: I mean, that's a lot of--
Michael Kennedy: A lot of--
Vincent Warmerdam: MARK MANDEL: So that thing took two hours and 15 minutes
Vincent Warmerdam: on my M4 Mac.
Vincent Warmerdam: And if you go there, you can actually select it.
Vincent Warmerdam: That one-- that was a chunky boy, is what I'll say.
Vincent Warmerdam: MARK MANDEL: Yeah, 550,000 lines.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: MARK MANDEL: There you go.
Vincent Warmerdam: But you can see that there's one version change, I think,
Vincent Warmerdam: where they made a bunch of changes.
Vincent Warmerdam: And it could be that--
Vincent Warmerdam: MARK MANDEL: Yeah.
Vincent Warmerdam: That might have been, again, because I checked the docs
Vincent Warmerdam: as well on Markdown files, it might have been a big docs
Vincent Warmerdam: change.
Vincent Warmerdam: But hopefully by just looking at this,
Vincent Warmerdam: you can go like, oh yeah, this is probably a notebook somewhere.
Vincent Warmerdam: And there's a huge for loop that does threading
Vincent Warmerdam: and tries to do as much in parallel as possible.
Vincent Warmerdam: And there's a progress bar.
Vincent Warmerdam: Right?
Vincent Warmerdam: Yeah.
Vincent Warmerdam: And we don't--
Michael Kennedy: Now I see why you had this problem with the caching.
Vincent Warmerdam: That's right.
Vincent Warmerdam: But yeah, but here's also where the threading came in.
Vincent Warmerdam: Because the moment you say this point in time, now do all the files, do the git blame, that's definitely something that can happen in parallel.
Vincent Warmerdam: But then for every file, for every point in time, you do want to have something in the cache that says, okay, if I have to restart this notebook for whatever reason, that number is just known.
Vincent Warmerdam: Don't check it again.
Michael Kennedy: Yeah.
Michael Kennedy: Yeah, super interesting.
Michael Kennedy: Okay.
Michael Kennedy: I wonder if this 4.0 in 2022, is that might be when they switched to async?
Michael Kennedy: They started supporting async.
Michael Kennedy: It could be docs as well.
Michael Kennedy: I'm not sure.
Vincent Warmerdam: Well, yeah, so that's kind of the hard thing of some of these charts.
Vincent Warmerdam: Like, I could expand these charts by saying things like,
Vincent Warmerdam: okay, only the Python files, et cetera.
Vincent Warmerdam: But, like, the way that this is hosted,
Vincent Warmerdam: this is, like, really using the disk as a cache
Vincent Warmerdam: because all these charts are Altair charts,
Vincent Warmerdam: and you can save them to disk,
Vincent Warmerdam: and then you can easily upload them to GitHub pages.
Vincent Warmerdam: So I do everything in disk cache to make sure that if I,
Vincent Warmerdam: for whatever reason, the notebook fails,
Vincent Warmerdam: I don't have to sort of do anything fancy to get it back up.
Vincent Warmerdam: But then once it's time to actually put it on the site,
Vincent Warmerdam: I could use disk cache to show the charts,
Vincent Warmerdam: but then I would need a server.
Vincent Warmerdam: So actually using disk to actually just serve some files
Vincent Warmerdam: is also just a pretty fine and good idea.
Vincent Warmerdam: There are some things on the Marimo side
Vincent Warmerdam: where we are also hoping to maybe give better caching tools
Vincent Warmerdam: to the library itself.
Vincent Warmerdam: It's just that when I was doing this,
Vincent Warmerdam: I actually found a bug in our caching layer,
Vincent Warmerdam: so then I switched back to disk cache.
Michael Kennedy: You know what?
Michael Kennedy: Look, that's valuable.
Michael Kennedy: That's maybe not the way you will find, but it's valuable.
Vincent Warmerdam: It's-- oh, so one thing you learn is that caching is actually
Vincent Warmerdam: hard to get right.
Michael Kennedy: Oh, it is.
Vincent Warmerdam: It is.
Michael Kennedy: Very hard.
Vincent Warmerdam: It's on par with naming things.
Michael Kennedy: It is one of the two things that goes wrong--
Michael Kennedy: naming things, cache invalidation, and off by one errors.
Vincent Warmerdam: Yes, exactly.
Michael Kennedy: It's the middle one.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: Dad jokes are amazing.
Vincent Warmerdam: Anyway, so one thing about this repo, by the way,
Vincent Warmerdam: this is all my--
Vincent Warmerdam: we're going to add a link to the show notes.
Vincent Warmerdam: There is a notebook, so if you feel like adding your own project
Vincent Warmerdam: that you want to just add, feel free to spend your compute
Vincent Warmerdam: resources two and a half hours to add a popular project.
Vincent Warmerdam: I would love to have that.
Vincent Warmerdam: One thing I think will be cool with these sorts of charts
Vincent Warmerdam: is to see what will change when LLMs kind of get into the mix.
Vincent Warmerdam: Do we see more code shifts happen if more LLMs get used
Vincent Warmerdam: for these libraries?
Michael Kennedy: MARK MANDEL: Very interesting.
Vincent Warmerdam: I don't know if more code--
Michael Kennedy: old code will get changed, but they are verbose code writers,
Michael Kennedy: those things.
Vincent Warmerdam: Yes.
Vincent Warmerdam: So this, assuming we can do this over time
Vincent Warmerdam: and we're going to start tracking this,
Vincent Warmerdam: I'm calling this code archaeology.
Vincent Warmerdam: I do think it will be an interesting chart.
Vincent Warmerdam: As is, I think it's already quite interesting to see differences
Vincent Warmerdam: between different projects.
Vincent Warmerdam: I think if you go to sentence transformers,
Vincent Warmerdam: you can also see when the project got moved from an academic lab
Vincent Warmerdam: to hugging face.
Vincent Warmerdam: So there are interesting things you can see with these charts.
Vincent Warmerdam: But you are going through every file, every line,
Michael Kennedy: get blame 100 times.
Vincent Warmerdam: Yeah, a lot.
Michael Kennedy: You got to do 100 times per line as well.
Vincent Warmerdam: Well, so the start of the project with a Git repository,
Vincent Warmerdam: and then to make a chart like this,
Vincent Warmerdam: you got a sample over the entire timeline.
Vincent Warmerdam: And it is a bit cheeky because sometimes you can go like,
Vincent Warmerdam: OK, but there's a character that changed because of a linter.
Vincent Warmerdam: And then is that really a change?
Vincent Warmerdam: Does it really matter?
Michael Kennedy: It's whoever decided to run the format on the thing or whatever.
Vincent Warmerdam: We're looking at the Django chart.
Vincent Warmerdam: It could also just be that black just got an update
Vincent Warmerdam: or something like that, right?
Vincent Warmerdam: Yeah, exactly.
Vincent Warmerdam: It's also possible.
Michael Kennedy: It's very possible.
Vincent Warmerdam: It's unlikely, but it's not impossible, let me say.
Vincent Warmerdam: But yeah, anyway.
Vincent Warmerdam: Yeah, this was one of the benchmarks
Vincent Warmerdam: that I did with disk cache that I thought was pretty amusing
Vincent Warmerdam: and pretty interesting.
Vincent Warmerdam: But there's this one other feature
Vincent Warmerdam: that I think we should also talk about, which
Vincent Warmerdam: is that if you want to, disk cache actually lets you
Vincent Warmerdam: do the serialization yourself.
Vincent Warmerdam: So normally, what it would do is it would say,
Vincent Warmerdam: like, OK, let's do the pickle thing.
Vincent Warmerdam: And it's a bit clever, right?
Vincent Warmerdam: So if the thing you're storing is like an integer,
Vincent Warmerdam: doesn't go through the whole pickle thing and just stores it as an integer there's these native types
Michael Kennedy: that sqlite has and then you know it's able to do something clever but right as soon as it becomes
Michael Kennedy: like a custom class or a list of weird things then yeah then then it's i personally don't like it
Michael Kennedy: pickling i would i would prefer that it makes me do something it's i think it's weird well so the
Vincent Warmerdam: thing is you can write your own disk class and then what you can do is you can pass set disk class
Vincent Warmerdam: onto uh the disk cache itself and i'm just kind of wondering like when might it make sense to do
Vincent Warmerdam: this sort of a thing and if you go to the docs there's actually like a really good example just
Michael Kennedy: right there which is jason yeah i i lost your example if i'll get it back if you type disk
Vincent Warmerdam: you'll find it so jason has this interesting thing the it is text if you think about it but
Vincent Warmerdam: it's text that has a bit of structure and that means that there are these compression libraries
Vincent Warmerdam: you can actually run on them and especially if you have like a pattern that repeats itself let's say
Vincent Warmerdam: a list of users or something like that and there's always the user key and there's always maybe the
Vincent Warmerdam: the email key and those things just repeat themselves all over the place,
Vincent Warmerdam: then there is an opportunity to,
Vincent Warmerdam: there's this library called zlib where you can just take that string,
Vincent Warmerdam: you can compress it and then that compressed representation can go into disk cache instead.
Vincent Warmerdam: Yeah, I figured that sounds like a lot of fun.
Vincent Warmerdam: You can just grab the implementation there.
Vincent Warmerdam: I have this notebooks repository where I have LLMs just write fun little notebooks.
Vincent Warmerdam: I always check the results obviously just to be clear on that.
Vincent Warmerdam: But one thing that was I think pretty cool to see,
Vincent Warmerdam: if you just do the normal data type and you pickle it, then you get a certain size. And if you just
Vincent Warmerdam: have a very short, normal Python dictionary basic thing, then it's negligible. You shouldn't use this
Vincent Warmerdam: JSON trick. But the moment you get text heavy, there's just a lot of text that you're inputting
Vincent Warmerdam: there and there's some repetition of characters. Or if you really do something that's highly
Vincent Warmerdam: compressible, it is not unheard of to get like 80%, 90% savings
Vincent Warmerdam: on your disk space, basically.
Vincent Warmerdam: Now, there is a little bit of overhead
Vincent Warmerdam: because you were doing the compression and decompression.
Vincent Warmerdam: But if you're doing text-heavy stuff,
Vincent Warmerdam: this is something that can actually save you a whole bunch.
Vincent Warmerdam: And I can imagine for LLMs, this would also be a win.
Michael Kennedy: OK.
Michael Kennedy: So this JSON disk, not only does it serialize to and from JSON,
Michael Kennedy: which I think is safer.
Michael Kennedy: It can be a pain if you had date times.
Vincent Warmerdam: You've got to do something about that.
Vincent Warmerdam: or JSON or something like that.
Vincent Warmerdam: You can.
Michael Kennedy: Yeah, yeah, I think.
Michael Kennedy: But then it's using Zlib here.
Michael Kennedy: You know, I just actually did something like this
Michael Kennedy: for just something in my database.
Michael Kennedy: It had nothing to do with caching.
Michael Kennedy: But these records are holding tons of text
Michael Kennedy: for something sort of tangential to the podcast.
Michael Kennedy: And I'm like, I don't really want to put 100K of text
Michael Kennedy: that I'm not going to query against or search
Michael Kennedy: into the database.
Michael Kennedy: So I used Python's XZ implementation.
Michael Kennedy: And like you said.
Vincent Warmerdam: There's a bunch of compression algorithms you could use.
Michael Kennedy: It was way fast.
Michael Kennedy: So I just store it as bytes now, and it's like a tenth the size.
Michael Kennedy: It's great.
Michael Kennedy: So I guess this is the same, but for the cache back end, right?
Vincent Warmerdam: Yeah.
Vincent Warmerdam: And I think-- well, you can see, I think Zlib
Vincent Warmerdam: is being used internally.
Michael Kennedy: Yeah.
Michael Kennedy: I mean, it's not XZ, but it's the same idea.
Vincent Warmerdam: Yeah, exactly.
Vincent Warmerdam: And there's always new compression algorithms.
Vincent Warmerdam: Like, feel free to check whatever makes sense.
Vincent Warmerdam: But the fact that you have one very cool example
Vincent Warmerdam: to add on the docs, because you can just copy and paste it.
Vincent Warmerdam: A lot of people benefit from it.
Vincent Warmerdam: But why stop here?
Michael Kennedy: Because this is the company you can do for JSON.
Vincent Warmerdam: What else can you do?
Michael Kennedy: Before we move on, though, if I were writing this,
Michael Kennedy: I would recommend using microJSON or ORJSON
Michael Kennedy: or whatever, some of the more high performance versions
Michael Kennedy: right there.
Vincent Warmerdam: Yeah, and I think ORJSON--
Vincent Warmerdam: I mean, performance is cool.
Vincent Warmerdam: The reason I use ORJSON a lot more
Vincent Warmerdam: has to do with the types that it supports.
Vincent Warmerdam: So it can accept the NumPy arrays, for example,
Vincent Warmerdam: and just listifies it.
Vincent Warmerdam: And I think it has a few things with dates as well.
Vincent Warmerdam: It just has a slightly better support for a few things.
Michael Kennedy: OK, good to know.
Michael Kennedy: All right, where are we going?
Michael Kennedy: What's next?
Vincent Warmerdam: Numpy arrays.
Vincent Warmerdam: OK.
Vincent Warmerdam: So a lot of people like to do things with embeddings nowadays.
Vincent Warmerdam: So like text thing goes in, some sort of array thing comes out.
Vincent Warmerdam: And then hopefully, if two texts are similar,
Vincent Warmerdam: then the arrays are also similar.
Vincent Warmerdam: So you can do all sorts of fun little lookups.
Vincent Warmerdam: And I do a fair share of doing things with embeddings.
Vincent Warmerdam: And embeddings are also not notoriously expensive
Vincent Warmerdam: to calculate, but still pretty expensive to calculate.
Vincent Warmerdam: OK, but you can write your full Python thing in there.
Vincent Warmerdam: So if you compare storing NumPy as bytes compared to that
Vincent Warmerdam: to a pickle, it's actually even.
Vincent Warmerdam: There's very little to gain there.
Vincent Warmerdam: But one thing you could do is you could say, well,
Vincent Warmerdam: let's maybe bring it down to float 16.
Vincent Warmerdam: That's a thing you can do.
Vincent Warmerdam: You can sort of say, before we save it,
Vincent Warmerdam: we actually make it just a little bit less accurate
Vincent Warmerdam: on the numeric part of it.
Vincent Warmerdam: But that'll save us a whole bunch of disk space.
Michael Kennedy: So that's already kind of old.
Michael Kennedy: Well, you need it to be super precise
Michael Kennedy: when it's involved in calculations.
Michael Kennedy: But then in the end, if you're not going to report the numbers
Michael Kennedy: to great decimal places, maybe going down is good, yeah.
Vincent Warmerdam: Yeah, it depends on the use case.
Vincent Warmerdam: But typically, you could argue maybe a 1% difference
Vincent Warmerdam: in similarity if we have 100x savings on disk.
Vincent Warmerdam: That'll be kind of a win.
Vincent Warmerdam: So one thing I was sort of focusing on is just this--
Vincent Warmerdam: you can do things like, OK, come up
Vincent Warmerdam: with your own little weird data structure where you say,
Vincent Warmerdam: OK, let's pretend we're going to quantize the whole thing.
Vincent Warmerdam: So we're going to calculate the quantiles of the float values that it can take.
Vincent Warmerdam: And we're going to take basically 256 buckets.
Vincent Warmerdam: We're going to store the scale.
Vincent Warmerdam: We're going to store the mean.
Vincent Warmerdam: And then we're going to store in what bucket the number was in.
Vincent Warmerdam: And you can turn that into a string representation.
Vincent Warmerdam: These things are pretty fun to write.
Vincent Warmerdam: Nice.
Vincent Warmerdam: And yeah, and then you scroll down into your big notebook and then you find this.
Vincent Warmerdam: There you go.
Vincent Warmerdam: That's a retrieval time.
Vincent Warmerdam: I think I got like a 4x improvement in terms of like disk space being saved.
Vincent Warmerdam: It was like a 1% similarity score that I had to give up for doing things like this.
Vincent Warmerdam: Mileage can vary, of course, but like, again, these are fun things to sort of start playing with
Vincent Warmerdam: because you have access to the way that you write that down.
Vincent Warmerdam: So that was also like a fun little exercise to do.
Michael Kennedy: Yeah.
Michael Kennedy: Could you save NumPy arrays by just putting up, just converting it to bytes?
Michael Kennedy: There's probably some efficient.
Michael Kennedy: You know what?
Michael Kennedy: What about a parquet file?
Michael Kennedy: Like in an in-memory Parquet file, then you just say,
Michael Kennedy: here's the value in bytes.
Vincent Warmerdam: So I tried the bytes thing and compared it to the pickle thing,
Vincent Warmerdam: and that was basically the same size.
Michael Kennedy: OK.
Vincent Warmerdam: That barely led to anything.
Vincent Warmerdam: About the Parquet one, I mean--
Vincent Warmerdam: You do get compression.
Vincent Warmerdam: Well, yeah, but I could be wrong on this one.
Vincent Warmerdam: But I think Parquet is optimized to be a disk representation.
Vincent Warmerdam: And then once you want to have it in memory,
Vincent Warmerdam: it becomes an arrow representation.
Vincent Warmerdam: I see.
Vincent Warmerdam: Yeah, probably.
Vincent Warmerdam: So in that sense, what I would do is, OK,
Vincent Warmerdam: if you have something in Arrow, you use this cache
Vincent Warmerdam: to make sure it's written as parquet.
Vincent Warmerdam: But then you have to be-- you kind of have
Michael Kennedy: to know what you're doing if you're going to make parquet
Vincent Warmerdam: files.
Vincent Warmerdam: And also, the benefit of a parquet file
Vincent Warmerdam: is that you have one huge table.
Vincent Warmerdam: Because then--
Vincent Warmerdam: Right, you can scan it, yeah.
Vincent Warmerdam: Yeah, it's a columnar format.
Vincent Warmerdam: So then if I were a column, boy, would I
Vincent Warmerdam: want to have all the rows in me.
Vincent Warmerdam: So in that sense, what you--
Vincent Warmerdam: yeah, so in that sense, what I would do instead
Vincent Warmerdam: is if you, for whatever reason, you just have a lot of data,
Vincent Warmerdam: It's still kind of a cache, but it makes more sense to store all of it in like a huge parquet file.
Vincent Warmerdam: In parquet, you can store a partition.
Vincent Warmerdam: So you can say this one column that's partitioned, a date would be like a very typical thing to partition on.
Vincent Warmerdam: And then if you point polars to like parquet, but you say, I only want to have this date,
Vincent Warmerdam: it can sort of do the forward scan and only pick the rows that you're interested in.
Vincent Warmerdam: And I would imagine that that would beat anything we might do with this cache, especially if the table is big.
Michael Kennedy: So you don't always want to cache stuff.
Michael Kennedy: Like I said, I actually don't avoid hitting the Mongo database
Michael Kennedy: a lot for my projects because it's like the response time
Michael Kennedy: is quick enough and might as well.
Michael Kennedy: I want to take two little avenues here.
Michael Kennedy: But the first one is, what about DuckDB?
Michael Kennedy: I know at least on the data science side and the analytics
Michael Kennedy: side, DuckDB is really popular, really well respected,
Michael Kennedy: really fast.
Michael Kennedy: Maybe you don't even cache it.
Michael Kennedy: Maybe you just use DuckDB as a thing.
Vincent Warmerdam: How do you feel about that?
Vincent Warmerdam: I mean, DuckDB does solve a very different problem
Vincent Warmerdam: than SQLite or Postgres in a way.
Vincent Warmerdam: So I don't believe-- to name one thing,
Vincent Warmerdam: I believe DuckDB does assume that everything under the hood
Vincent Warmerdam: is immutable.
Vincent Warmerdam: So it will never be ASCID compliant,
Vincent Warmerdam: because it doesn't necessarily have to be.
Vincent Warmerdam: You can still insert rows, if I'm not mistaken.
Vincent Warmerdam: But the use case is just assumed
Vincent Warmerdam: to be analytical in general.
Vincent Warmerdam: That like--
Vincent Warmerdam: I see.
Vincent Warmerdam: --it's really designed to sort of fit that use case.
Vincent Warmerdam: You can insert rows, though.
Michael Kennedy: So like--
Michael Kennedy: I mean, you might be caching data science things
Michael Kennedy: that you're only computing once.
Michael Kennedy: Like, for example, your charts.
Michael Kennedy: Once you've computed that, it's not going to change because it's historical.
Vincent Warmerdam: I mean, I might want to rerun it a month later or something like that.
Vincent Warmerdam: That's something I might want to do.
Vincent Warmerdam: And in that particular case, it would be cool if the sampling is the same
Vincent Warmerdam: and I just want to add one sample at the end that all those samples I had before,
Vincent Warmerdam: that those are in a cache somewhere.
Michael Kennedy: Or maybe you want faster, better resolution.
Michael Kennedy: Instead of going 100, you're going to go to 1,000 points,
Michael Kennedy: but you could do 10% less because those are done, right?
Vincent Warmerdam: So stuff like that.
Vincent Warmerdam: But then what you would never do with a cache is do a group buy
Vincent Warmerdam: and then a mean, for example.
Vincent Warmerdam: It's like--
Michael Kennedy: It's not-- it's outgrown its use at that point.
Michael Kennedy: That's for sure.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: And if it were a part of it, then the docs would say so.
Vincent Warmerdam: But like-- no, so in my mind, DuckDB really just
Vincent Warmerdam: solves a different problem, similar to like general SQLite
Vincent Warmerdam: also solves a different problem than disk cache.
Vincent Warmerdam: And also Postgres is also solving a slightly different problem.
Michael Kennedy: Sure.
Vincent Warmerdam: All right, fair.
Michael Kennedy: Like the other angle--
Michael Kennedy: Yeah?
Michael Kennedy: No, go ahead.
Michael Kennedy: Finish your thoughts.
Vincent Warmerdam: Well, I also really love Postgres, I got to say.
Vincent Warmerdam: Like the thing I really like about it
Vincent Warmerdam: is that it is like boring, but in a good way, software
Vincent Warmerdam: where like I have a Postgres thing running there
Vincent Warmerdam: and whatever SSH thing I need, I can just swap the cloud
Vincent Warmerdam: provider and it'll just go ahead and still
Vincent Warmerdam: run without me having to move the data
Vincent Warmerdam: or do any migration or anything like that.
Vincent Warmerdam: That is also just like a nice feeling,
Vincent Warmerdam: but it solves a different problem.
Michael Kennedy: Yeah.
Michael Kennedy: These would very likely be used together, not instead of--
Michael Kennedy: I mean, Postgres can be used instead of disk cache,
Michael Kennedy: but disk cache, definitely not instead of Postgres.
Michael Kennedy: So the other one, the other angle I wanted to riff on, have you riff on just a little bit is
Michael Kennedy: think people, especially people who are maybe new to this idea of caching, they can end up thinking,
Michael Kennedy: okay, I'm going to store stuff. We talked a lot about like, I get something back from the database.
Michael Kennedy: I could store that in a cache. So I don't have to query it again or whatever. And those are
Michael Kennedy: certainly good use cases. But I think a lot of times an even better use case is if you're going
Michael Kennedy: to get 20 rows back from a database, do some Python and construct them along with a little
Michael Kennedy: other information into some object.
Michael Kennedy: And then that's really what you want to work with.
Michael Kennedy: Store that constructed thing in the cache.
Michael Kennedy: You know what I mean?
Michael Kennedy: Like, as far as you can go down the compute layer, like, don't just stop like, well, it
Michael Kennedy: comes back from the database, so we cache it.
Michael Kennedy: Like, if there's a way to say it, like, there's a bunch of work after that, think about how
Michael Kennedy: you might cache at that level.
Vincent Warmerdam: Okay, I'm going to pitch you a dream then.
Vincent Warmerdam: Imagine you have a Python notebook and, oh, you're running a cell, you're running a cell,
Vincent Warmerdam: you're running a cell, and halfway the kernel dies for whatever weird reason.
Michael Kennedy: Right.
Vincent Warmerdam: it'd be nice if I could just reboot the notebook and it would just pick it up again and move further.
Vincent Warmerdam: Because again, I'm picking something out of a database and I'm doing something little with it
Vincent Warmerdam: and processing, processing, processing. But wouldn't it be nice if maybe every cell had a caching mechanism?
Vincent Warmerdam: If only you had some influence. If only we're a company that did this kind of stuff.
Vincent Warmerdam: You can imagine these are things that we are thinking about. And again, what I'm about to
Vincent Warmerdam: suggest is definitely a dream. This is not something that works right now. Don't pin me on this.
Vincent Warmerdam: like we're thinking out loud here.
Vincent Warmerdam: You can also imagine this being super useful
Vincent Warmerdam: where an entire team can share the cache.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: Right?
Vincent Warmerdam: So if your colleague already calculated something,
Vincent Warmerdam: you don't have to recalculate it again.
Vincent Warmerdam: There's all sorts of use cases like that as well.
Vincent Warmerdam: But there may be, there are these moments
Vincent Warmerdam: when you want to have very tight manual control
Vincent Warmerdam: over what goes into the cache.
Vincent Warmerdam: That makes a lot of sense and it's great.
Vincent Warmerdam: But there are also moments
Vincent Warmerdam: when you just really don't want to think about it at all.
Vincent Warmerdam: And you just want everything to be cached.
Michael Kennedy: Could I just use this thing as a checkpoint?
Michael Kennedy: I can autosave as my code runs.
Michael Kennedy: Yeah, that's cool.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: And again, doing this right is hard,
Vincent Warmerdam: because there's all sorts of weird Python types.
Vincent Warmerdam: And I mentioned the progress bar thing.
Michael Kennedy: And there's all sorts of things that we've
Vincent Warmerdam: got to be mindful of here.
Vincent Warmerdam: But if you're thinking about really,
Vincent Warmerdam: how would you use this in data science
Vincent Warmerdam: when you fetch a little bit of data and deal with it,
Vincent Warmerdam: to me, it is starting to feel more natural
Vincent Warmerdam: than thinking about cells in a notebook,
Vincent Warmerdam: maybe cache on that level.
Michael Kennedy: Yeah, that's pretty interesting.
Michael Kennedy: just sort of cascade them along as a hash of the hashes
Michael Kennedy: of the prior cells or--
Vincent Warmerdam: Well, and this is where things become tricky, of course,
Vincent Warmerdam: because then, OK, I've got this one cell,
Vincent Warmerdam: and I change one function in it.
Vincent Warmerdam: Oh, if you're going to cache on the entire cell,
Vincent Warmerdam: oh, everything has to rerun.
Vincent Warmerdam: And then, oh, if you're not careful,
Vincent Warmerdam: your cache is going to be huge.
Vincent Warmerdam: So like, OK, how do you do this in a user-friendly way?
Vincent Warmerdam: There's all sorts of--
Vincent Warmerdam: it sounds easier than it is the one thing I want to say.
Michael Kennedy: Yeah, well, I think you've got a better chance
Michael Kennedy: with Marima than with Jupyter, at least,
Michael Kennedy: because you have a dependency graph.
Michael Kennedy: So you can at least say, if this one is invalid,
Michael Kennedy: that means the following three are also invalid,
Michael Kennedy: being sort of propagated there.
Vincent Warmerdam: Totally.
Vincent Warmerdam: But I try to focus a little bit more on the user experience
Vincent Warmerdam: side of things.
Vincent Warmerdam: And one thing I've really learned from the notebook
Vincent Warmerdam: with the progress bar is just there
Vincent Warmerdam: were moments when I felt like, oh, I just
Vincent Warmerdam: want this entire thing to be automated.
Vincent Warmerdam: Don't make me think about this.
Vincent Warmerdam: And then there were moments where I thought, oh, it's really
Vincent Warmerdam: nice to have tight manual control.
Vincent Warmerdam: How do I provide you with both?
Vincent Warmerdam: Yeah.
Vincent Warmerdam: That's quite tricky.
Vincent Warmerdam: But it is a dream, so that's something to keep in mind.
Michael Kennedy: Yeah, maybe someday there'll be a turn on cash flow checkbox
Michael Kennedy: something.
Vincent Warmerdam: Yeah, or well, at least till then,
Vincent Warmerdam: I do think having something that works on disk instead of memory
Vincent Warmerdam: these days is also just a boon.
Michael Kennedy: Right.
Michael Kennedy: So this works in data science notebooks.
Michael Kennedy: It works in web apps.
Michael Kennedy: It works in little TUIs.
Michael Kennedy: It doesn't care.
Vincent Warmerdam: It works with LLMs.
Vincent Warmerdam: And if you have a kind of--
Vincent Warmerdam: actually, similar to your set of,
Vincent Warmerdam: If you have multiple processes with your web app running on one VM,
Vincent Warmerdam: if you have one big VM that you share with your colleagues,
Vincent Warmerdam: you can also just share the cache, actually.
Michael Kennedy: Yeah, that's true.
Michael Kennedy: So that's the reason you can't just point the same file, yeah.
Vincent Warmerdam: Yeah, and especially if you're doing like big experiments
Vincent Warmerdam: like grid search results or stuff like that,
Vincent Warmerdam: that you really don't want to recalculate the big compute thing.
Vincent Warmerdam: That's actually not too unreasonable.
Michael Kennedy: Yeah, that's cool.
Michael Kennedy: Yeah, if you have a shared Jupyter server.
Vincent Warmerdam: Yeah, and a bunch of universities have that, right?
Michael Kennedy: Yeah, exactly.
Michael Kennedy: I don't want to go down this path because we're basically out of time.
Michael Kennedy: I respect your time.
Michael Kennedy: However, I do think there's a whole interesting conversation to be had about like, how do you
Michael Kennedy: choose the right key for your, what goes into the cache?
Michael Kennedy: Because you can end up with staleness really easy if something changes.
Michael Kennedy: But if you incorporate the right stuff, you might never run into stale data problems because,
Michael Kennedy: you know, like for example, I talked about the YouTube ID.
Michael Kennedy: Basically the cache key is something like episode, episode data, episode YouTube thing,
Michael Kennedy: colon YouTube, colon the hash of the show notes, right?
Michael Kennedy: Something like that, where there's no way
Michael Kennedy: that the show notes are gonna change
Michael Kennedy: and I'll get the old data because guess what?
Michael Kennedy: It's constructed out of the source, right?
Michael Kennedy: Things like that.
Michael Kennedy: There's probably a lot, especially like in your notebook side.
Michael Kennedy: There's a lot to consider there, I think.
Vincent Warmerdam: - Yeah, I mean, so I remember this one thing
Vincent Warmerdam: with the course where I still wanted it to be cached,
Vincent Warmerdam: but I wanted to have like text goes in
Vincent Warmerdam: And then five responses from the LLM go out.
Vincent Warmerdam: And the way you solve that is you just add another key.
Vincent Warmerdam: But you have to be mindful of the cache key.
Vincent Warmerdam: And you can-- oh, and you can use tuples, by the way.
Vincent Warmerdam: That's also something you can totally use as a cache key.
Michael Kennedy: Right, right.
Vincent Warmerdam: So that was easy to fix.
Vincent Warmerdam: It's just that you have to be mindful.
Michael Kennedy: Yeah, that's kind of-- I want to give a quick shout out to that.
Michael Kennedy: I want to leave--
Michael Kennedy: I don't want to leave on a sour note.
Michael Kennedy: But I think it's necessary to give this shout out,
Michael Kennedy: or this like call this out, rather, is the way I should say it,
Michael Kennedy: is I think this project is awesome.
Michael Kennedy: You think it's awesome.
Michael Kennedy: Honestly, I think it doesn't need really very much.
Michael Kennedy: But if you look at last updated,
Michael Kennedy: if you look at the last updated date,
Michael Kennedy: it's really, it hasn't got a lot of attention
Michael Kennedy: in the last six months or something like that.
Vincent Warmerdam: Yeah, and if I look at PyPI,
Vincent Warmerdam: the last release was 2023,
Vincent Warmerdam: which, yeah, a year and a half ago.
Michael Kennedy: Yeah, and it's okay to,
Michael Kennedy: I would like to say that it's okay for things to be done.
Michael Kennedy: It doesn't have to, things don't have to change,
Michael Kennedy: but there's also a decent amount of,
Michael Kennedy: like conversations on the issues and they haven't, you know,
Michael Kennedy: like a couple of days ago, actually someone asked about this, but you know,
Michael Kennedy: the last change, I believe the guy Grant who works on it,
Michael Kennedy: started working at open AI about the time changes stopped going on.
Michael Kennedy: I'm not entirely sure. I feel like I could be confused with another project.
Michael Kennedy: So Grant, if that's not true, I apologize, but I think that it is.
Michael Kennedy: Pretty sure that is. Do we have LinkedIn?
Vincent Warmerdam: I mean, I am comfortable stating is--
Vincent Warmerdam: let me put it this way.
Vincent Warmerdam: Yes, my colleague might make a different caching mechanism.
Vincent Warmerdam: And yes, I might use that at some point.
Vincent Warmerdam: But at least for where I'm at right now,
Vincent Warmerdam: this cache needs to break vividly in front of my face
Vincent Warmerdam: for me to consider not using it.
Vincent Warmerdam: Because it does feel like it's done in a really good way.
Vincent Warmerdam: The main thing that needs to happen, I think,
Vincent Warmerdam: functionally to make sure this doesn't get deprecated too
Vincent Warmerdam: badly is just you got to update the Python version.
Vincent Warmerdam: When a new Python version comes out,
Vincent Warmerdam: you got to update PyPI to confirm,
Vincent Warmerdam: like, OK, we do support this Python version.
Vincent Warmerdam: But I mean, most of the--
Vincent Warmerdam: if you look at the area that needs to be covered,
Vincent Warmerdam: a lot of that has been covered by SQLite.
Vincent Warmerdam: And that thing is definitely still being maintained.
Michael Kennedy: It's getting mega maintained.
Michael Kennedy: That's right.
Michael Kennedy: So I also don't see the problem.
Michael Kennedy: I'm not going to not use it.
Michael Kennedy: I just want to put it out there on the radar for people
Michael Kennedy: for whom my co-look is to go, oh, Michael and Vincent
Michael Kennedy: were so psyched, and I started to use this.
Michael Kennedy: And now I'm really disappointed because of whatever.
Michael Kennedy: I saw this.
Vincent Warmerdam: I mean, the only real doom scenario I can come up with
Vincent Warmerdam: is if SQLite made like a breaking change.
Vincent Warmerdam: That's the only thing I can kind of come up with.
Michael Kennedy: But the odds of that seem very low.
Michael Kennedy: Yeah, and it's on GitHub.
Michael Kennedy: You can fork it.
Michael Kennedy: I fork it.
Michael Kennedy: Exactly, exactly.
Michael Kennedy: So, no, I definitely am still super excited about it.
Michael Kennedy: I just want to make sure that we put that out there.
Michael Kennedy: I'd intended to talk about it sooner in the conversation,
Michael Kennedy: but you know what?
Michael Kennedy: We were just so excited.
Vincent Warmerdam: Yeah, no.
Vincent Warmerdam: This is definitely in my top five favorite Python libraries
Vincent Warmerdam: outside of the standard lib.
Vincent Warmerdam: Awesome.
Michael Kennedy: Yeah, I've really, really have gotten awesome results out of it as well.
Michael Kennedy: So remember the way we opened the show and you talked about this,
Michael Kennedy: like when we talked about the LLM building block stuff
Michael Kennedy: on the previous time you were on the show,
Michael Kennedy: it was like, oh, we better not go too deep on this,
Michael Kennedy: even though we're both so excited because it's going to derail the show.
Michael Kennedy: We're now one hour and 15 minutes into it.
Michael Kennedy: We kind of cut ourselves off.
Michael Kennedy: I think that was accurate.
Vincent Warmerdam: Yeah, I mean, you get two dads making dad jokes
Vincent Warmerdam: and riffing on tools they both like.
Vincent Warmerdam: It's bound to exceed a barbecue.
Michael Kennedy: Yes, I know.
Michael Kennedy: I wonder what would happen if sometime we just removed the time limit,
Michael Kennedy: just got real comfortable and just riffed on something.
Michael Kennedy: It could be hours.
Michael Kennedy: It would be fun.
Michael Kennedy: But maybe not today.
Michael Kennedy: Two-hour live streams exist, Michael.
Michael Kennedy: I know.
Michael Kennedy: I've listened to some podcasts at over three hours.
Michael Kennedy: I'm like, how is this still going?
Michael Kennedy: But you know what?
Michael Kennedy: Yeah.
Michael Kennedy: It's all good.
Vincent Warmerdam: But yeah, this is a good point in time.
Michael Kennedy: We're both excited.
Michael Kennedy: That's the summary.
Michael Kennedy: That is the summary.
Michael Kennedy: And I think I'm going to let you have the final word on this topic here.
Michael Kennedy: like maybe speak to people just about caching in general
Michael Kennedy: and disk cache in particular as we close it out?
Vincent Warmerdam: I mean, I guess the main thing that I learned
Vincent Warmerdam: with the whole caching thing in the last couple of years,
Vincent Warmerdam: I always thought it was kind of like a web thing.
Vincent Warmerdam: Like, oh, you know, front page of Reddit,
Vincent Warmerdam: that thing has to be cached.
Vincent Warmerdam: That's the way you think about it.
Vincent Warmerdam: Yeah, of course.
Vincent Warmerdam: And thinking about it too much that way
Vincent Warmerdam: totally blocked me from considering, like, oh,
Vincent Warmerdam: but if you do stuff in notebooks and data science land,
Vincent Warmerdam: then you need this as well.
Vincent Warmerdam: And I think there's actually a little emerging discovery
Vincent Warmerdam: phenomenon happening where people that do things
Vincent Warmerdam: LLM's at some point go like, oh, I need a cache.
Vincent Warmerdam: And then, oh.
Vincent Warmerdam: So that's the main thing I suppose I want to say.
Vincent Warmerdam: Like, even if you're doing more data stuff,
Vincent Warmerdam: like give this disk cache thing a try.
Vincent Warmerdam: It's just good.
Michael Kennedy: Yeah, it's so easy to adopt and try out.
Michael Kennedy: Like, you can throw it in there.
Michael Kennedy: Just add a decorator.
Michael Kennedy: Exactly, see what you get.
Michael Kennedy: See what you get.
Michael Kennedy: All right, Vincent, welcome.
Michael Kennedy: Oh, thank you for coming back.
Michael Kennedy: I really appreciate it.
Michael Kennedy: Thanks for having me.
Michael Kennedy: Always good to talk to you.
Michael Kennedy: Yeah.
Vincent Warmerdam: And yeah, see you next time when we, again,
Vincent Warmerdam: find out there's a cool Python library.
Michael Kennedy: Yeah, that's going to be the three-hour episode.
Michael Kennedy: Watch out, y'all.
Michael Kennedy: Have a good one.
Michael Kennedy: Later.
Michael Kennedy: This has been another episode of Talk Python To Me.
Michael Kennedy: Thank you to our sponsors.
Michael Kennedy: Be sure to check out what they're offering.
Michael Kennedy: It really helps support the show.
Michael Kennedy: If you or your team needs to learn Python,
Michael Kennedy: we have over 270 hours of beginner and advanced courses
Michael Kennedy: on topics ranging from complete beginners to async code,
Michael Kennedy: Flask, Django, HTMX, and even LLMs.
Michael Kennedy: Best of all, there's no subscription in sight.
Michael Kennedy: browse the catalog at talkpython.fm.
Michael Kennedy: And if you're not already subscribed to the show
Michael Kennedy: on your favorite podcast player,
Michael Kennedy: what are you waiting for?
Michael Kennedy: Just search for Python in your podcast player.
Michael Kennedy: We should be right at the top.
Michael Kennedy: If you enjoyed that geeky rap song,
Michael Kennedy: you can download the full track.
Michael Kennedy: The link is actually in your podcast blur show notes.
Michael Kennedy: This is your host, Michael Kennedy.
Michael Kennedy: Thank you so much for listening.
Michael Kennedy: I really appreciate it.
Michael Kennedy: I'll see you next time.
I'm out.