Michael Kennedy: In this episode, I'm talking with Vincent Wormerdam about treating LLMs as just another API in your
Michael Kennedy: Python app with clear boundaries, small focused endpoints, and good monitoring. We'll dig into
Michael Kennedy: patterns for wrapping these calls, caching and inspecting responses, and deciding where an LLM
Michael Kennedy: API actually earns its keep in your architecture. This is Talk Python To Me, episode 528,
Michael Kennedy: recorded October 23rd, 2025.
Michael Kennedy: Welcome to Talk Python To Me, the number one podcast for Python developers and data scientists.
Michael Kennedy: This is your host, Michael Kennedy. I'm a PSF fellow who's been coding for over 25 years.
Michael Kennedy: Let's connect on social media.
Michael Kennedy: You'll find me and Talk Python on Mastodon, Bluesky, and X.
Michael Kennedy: The social links are all in the show notes.
Michael Kennedy: You can find over 10 years of past episodes at talkpython.fm.
Michael Kennedy: And if you want to be part of the show, you can join our recording live streams.
Michael Kennedy: That's right.
Michael Kennedy: We live stream the raw, uncut version of each episode on YouTube.
Michael Kennedy: Just visit talkpython.fm/youtube to see the schedule of upcoming events.
Michael Kennedy: And be sure to subscribe and press the bell so you'll get notified anytime we're recording.
Michael Kennedy: This episode is brought to you by Sentry.
Michael Kennedy: Don't let those errors go unnoticed.
Michael Kennedy: Use Sentry like we do here at Talk Python.
Michael Kennedy: Sign up at talkpython.fm/sentry.
Michael Kennedy: And it's brought to you by NordStellar.
Michael Kennedy: NordStellar is a threat exposure management platform
Michael Kennedy: from the Nord security family,
Michael Kennedy: the folks behind NordVPN
Michael Kennedy: that combines dark web intelligence,
Michael Kennedy: session hijacking prevention,
Michael Kennedy: brand and domain abuse detection,
Michael Kennedy: and external attack surface management.
Michael Kennedy: Learn more and get started keeping your team safe
Michael Kennedy: at talkpython.fm/nordstellar. Hey, welcome to Talk Python To Me. Hi. Welcome back. Pleasure
Vincent Warmerdam: to be here. Yeah, it's been like, this is our second Talk Python interview, I think. We've
Vincent Warmerdam: interacted before also on socials and all that, but like, yeah, second time. Yay. Happy to be here.
Michael Kennedy: Yeah. Yeah. It's very exciting to have you back. I, you know, I'm trying to remember, I did an
Michael Kennedy: blog post on
Michael Kennedy: the most popular episodes
Michael Kennedy: of last year.
Michael Kennedy: I think, if I do recall correctly...
Vincent Warmerdam: Yeah, the
Vincent Warmerdam: number one last year was our
Vincent Warmerdam: interaction, I think, on Spade.
Michael Kennedy: That's not terrible, is it?
Michael Kennedy: Yeah, it was a bragging right for a day.
Michael Kennedy: Yeah, it's pretty cool. So your episode
Michael Kennedy: and double bragging
Michael Kennedy: rights as your episode was released
Michael Kennedy: in, I don't know when that was,
Michael Kennedy: something like October or something, almost exactly
Michael Kennedy: a year ago to the day. And still it was the number one most downloaded for the whole year. So
Michael Kennedy: fantastic. And I think that might be an indication of where we're going to another very exciting
Vincent Warmerdam: topic, you know? Yeah. Like ML and AI only went up, right? So, so.
Michael Kennedy: You know what? I would love to get your thoughts on this. I, I'm seeing out on the internet and I
Michael Kennedy: want to put this up as kind of a, to counter this, this thought or whatever. But, but I'm seeing a
Michael Kennedy: lot of people get really frustrated that AI is a thing. You know, OpenAI just released their Atlas
Michael Kennedy: browser, which I'm not super excited about, but it's like their version of what an AI browser
Michael Kennedy: wrapping Chromium looks like, you know what I mean? And it's fine. But the comments in Ars Technica,
Michael Kennedy: which are usually pretty balanced, are just like people losing it over the fact that it has AI.
Michael Kennedy: Are people tired of this stuff? Or why do you think that's the reaction, I guess, is really what I
Vincent Warmerdam: want to ask you i mean i guess i have like a couple of perspectives but like i do like i think i could
Vincent Warmerdam: say like we are all tired that like every tech product out there is trying its best to find a
Vincent Warmerdam: way to put ai in the product like oh any any gmail any calendar app let's put ai in there and it
Vincent Warmerdam: gets a bit tiring like i wouldn't be too surprised if like fire hoses and refrigerators also come with
Michael Kennedy: ai features these days like it feels a bit nuts i know it's like what what is my refrigerating ai for
Vincent Warmerdam: Yeah, or like soccer ball of whatever object, like it feels like people are overdoing the AI thing. So okay, that can be like legitimate source of frustration. But then it also becomes a little bit personal, because I can also imagine if you put like, your heart and soul into like being a good engineer, and like you want to take the internet serious, like you consider it something is somewhat sacred, because it helped you so much in your career, then I definitely can also imagine like, Oh, my God, please don't turn that into a slop. Like I built a career on that. So that might be the source of all these things. I guess like my main way of dealing with it,
Vincent Warmerdam: is more along the lines of like, okay, like, this genie is not going to go back into the
Vincent Warmerdam: ball in a way. But if we're able to do more things with these LLMs, if they're able to
Vincent Warmerdam: automate more boilerplate and that sort of thing, then it seems that then I can try and
Vincent Warmerdam: get better at natural intelligence as well, instead of this artificial intelligence. So
Vincent Warmerdam: like, okay, I can vibe code all these things now. So the bottleneck is just do I have good
Vincent Warmerdam: ideas? And maybe I should invest in good ideas. And oh, maybe I can actually learn about
Vincent Warmerdam: JavaScript, but then I should not focus on the syntax, but it shouldn't focus maybe on
Vincent Warmerdam: the browser and like how does CSS work and oh I should actually maybe do a little bit of stuff
Vincent Warmerdam: with flashcards just so I know the syntax just enough so I can review okay like I try to be very
Michael Kennedy: conscious about it that way that's kind of my approach more and I it's easy to get really lazy
Michael Kennedy: right and just push the button and say do the next thing and not use it as an opportunity like oh it
Michael Kennedy: did something I didn't know let me have a conversation and try to understand the uh I
Vincent Warmerdam: didn't know I think the key word here is you want to be deliberate about it like I think if you can
Vincent Warmerdam: sort of say like okay i'm deliberate about the way i use this stuff and i'm also learning as i'm doing
Vincent Warmerdam: this like one thing i really like to do is i just give give like some sort of bonkers front-end task
Vincent Warmerdam: to the llm that i have no idea how i would solve it and then study the output like that's the thing
Vincent Warmerdam: i actually do once in a while um but yeah i mean i do get it that like people have mixed emotions
Vincent Warmerdam: about it and that's totally cool and fine it's just that for me in my situation this is how i
Michael Kennedy: deal with it yeah i think that's fair i think also i i definitely have long since been tired of like
Michael Kennedy: every email app having to rewrite this with AI
Michael Kennedy: and it usually destroys the email.
Michael Kennedy: You know, like removes all the formatting.
Michael Kennedy: And then you just, you know,
Michael Kennedy: there's always the cartoon of like,
Michael Kennedy: I wrote an email as bullet points.
Michael Kennedy: I asked AI to expand it.
Michael Kennedy: I sent it to you.
Michael Kennedy: You're like, what a long email.
Michael Kennedy: I'm gonna ask AI to summarize this.
Michael Kennedy: We should have just sent the bullet points.
Vincent Warmerdam: - Yeah, I mean, I do fear the amount of slop though.
Vincent Warmerdam: Like I do, it almost feels like every good tech YouTuber,
Vincent Warmerdam: there aren't a lot of good tech YouTubers anymore
Vincent Warmerdam: they're all doing AI stuff instead of doing like actual really great programming. There's still a
Vincent Warmerdam: few. Anthony writes code is like a really good one still. Like he's awesome. Definitely follow
Vincent Warmerdam: him if you want to learn Python. He does cool stuff. But yeah, the main thing, I guess from my
Vincent Warmerdam: like selfish perspective, YouTube used to be better because nowadays it's all about the AI and you've
Vincent Warmerdam: always got this thumbnail of a guy pulling his hair out like, oh my God, this changes everything.
Vincent Warmerdam: Yeah. I would love to see a video that's maybe not that.
Michael Kennedy: Have you heard that there's an AI bubble that's going to burst?
Vincent Warmerdam: um yeah well we'll see exactly how far i think so actually i mean we can speculate like i i can
Vincent Warmerdam: argue that it's been over invested i can also argue there's still so much potential the one
Vincent Warmerdam: thing i will say though um we have clod kinds of tools and like we're all looking forward to like
Vincent Warmerdam: the vibe coding oh it's never been easier and that sort of thing but i would have expected an explosion
Vincent Warmerdam: of awesome apps to be going along with it and it doesn't necessarily feel like we have much better
Vincent Warmerdam: software even though we supposedly have way better tools so something about that is making me a little
Vincent Warmerdam: a bit suspicious, but it might also just be that the apps that are being built that I'm not the
Vincent Warmerdam: target audience for, because I can imagine that. Because let's say you have something awesome for
Vincent Warmerdam: dentists, they can still be an awesome crud app that you could build with Claude, but I'm not a
Michael Kennedy: dentist. So, you know, wow, there's a lot of new cool dentistry management apps. You're right.
Vincent Warmerdam: If a dentist can outprogram, right? Like I do believe in the story that, oh, as a dentist,
Vincent Warmerdam: you might be best equipped to write an app for dentists, right? And if they're now more empowered
Vincent Warmerdam: to maybe do some stuff with code.
Vincent Warmerdam: I mean, there's a story there
Vincent Warmerdam: that every single niche profession
Vincent Warmerdam: will have someone who can do enough clod
Vincent Warmerdam: to make the app for that profession.
Michael Kennedy: Yeah.
Vincent Warmerdam: Well, time will tell.
Michael Kennedy: It used to be that you have to have a programming skill
Michael Kennedy: and a little bit of, let's say, dentistry experience
Michael Kennedy: to build the right sort of specialized vertical.
Michael Kennedy: And now I think it maybe is reversed.
Michael Kennedy: You need a lot of dentistry
Michael Kennedy: and a little bit of coding skill to go along with an AI.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: The character sheet used to be 80 points here, 20 points there.
Vincent Warmerdam: and now it's 80 points there and 20 points here.
Michael Kennedy: Yes, exactly.
Michael Kennedy: That's exactly what I was thinking.
Michael Kennedy: Like, I think maybe that's actually shifted
Michael Kennedy: until you said that.
Michael Kennedy: I've never really thought about it,
Michael Kennedy: but it just may be.
Michael Kennedy: I do want to point out for people listening,
Michael Kennedy: we're not going to really talk too much
Michael Kennedy: about like using agentic tools to write code for us.
Michael Kennedy: Like we have been speculating
Michael Kennedy: about this theoretical dentist.
Michael Kennedy: But what instead we're going to do
Michael Kennedy: is we're going to talk to you, Vincent,
Michael Kennedy: about how can I use an LLM
Michael Kennedy: like a library or an API to add functionality, features, behaviors to an existing data science
Michael Kennedy: tool or to an existing web app or whatever it is we're building, right?
Vincent Warmerdam: Yes.
Vincent Warmerdam: So also people might probably know it, but I made a course for Talk Python, of course,
Vincent Warmerdam: and we're going to talk about some of those details.
Vincent Warmerdam: But the idea for that course was not necessarily like, how can I use LLMs to build me software?
Vincent Warmerdam: It's more, oh, how can I build software that also uses LLMs under the hood?
Vincent Warmerdam: Like if I have an app that makes a summary, how can I make sure the summary is reliable?
Vincent Warmerdam: And basically get the ball rolling on those building blocks.
Vincent Warmerdam: That's what that course is about.
Vincent Warmerdam: That's also probably going to be the main bones of this topic as opposed to a dentist
Vincent Warmerdam: Bob and his ambitions to make a new tech startup.
Michael Kennedy: Exactly.
Michael Kennedy: We're going to grow new teeth if we just could get the software right.
Michael Kennedy: Now, yeah.
Michael Kennedy: So how do you do that?
Michael Kennedy: And we're going to talk about that before.
Michael Kennedy: I just want to give you a chance before we really dive too far into the details of how
Michael Kennedy: we make that work is just, you know, give people a sense of what you've been up to lately.
Michael Kennedy: You've done a lot of cool stuff with CalmCode.io.
Michael Kennedy: I can see your YouTube game is growing strong here.
Michael Kennedy: And you've been doing a lot of data science at Marimo.
Michael Kennedy: Yeah, what's...
Vincent Warmerdam: Right.
Vincent Warmerdam: Yeah, so we haven't spoken in a year, so maybe we should catch up.
Vincent Warmerdam: So long story short, CalmCode is still a project that I maintain.
Vincent Warmerdam: It's basically 99% free educational content on Python.
Vincent Warmerdam: that thing is just going to maintain itself.
Vincent Warmerdam: Super happy to keep maintaining it.
Vincent Warmerdam: The main thing I'm doing with that nowadays
Vincent Warmerdam: is just every two months I switch cloud providers
Vincent Warmerdam: just because I can,
Vincent Warmerdam: and then I can sort of see what it's like on the other side.
Vincent Warmerdam: So that's the thing that I do.
Vincent Warmerdam: From Calm Cult, though,
Vincent Warmerdam: I figured I might also start a YouTube channel,
Vincent Warmerdam: and that ended up being a YouTube channel
Vincent Warmerdam: where I still talk about Python stuff,
Vincent Warmerdam: but I mainly talk about these fancy ergonomic keyboards
Vincent Warmerdam: because I had a couple of RSI issues.
Vincent Warmerdam: In a year's time,
Vincent Warmerdam: I went from 100 subscribers to 5,000 something,
Vincent Warmerdam: so this thing kind of took off.
Vincent Warmerdam: That's awesome.
Vincent Warmerdam: I actually have sponsors now.
Vincent Warmerdam: So I actually have a couple of companies in Asia sending me their custom boards for me to review.
Vincent Warmerdam: So that's been a really fun journey.
Vincent Warmerdam: But since last time we spoke, I also switched employers.
Vincent Warmerdam: So people might have heard of Jupyter before.
Vincent Warmerdam: That's an environment in Python where you can do interactive things.
Vincent Warmerdam: And I work now for a company that makes Maremo, which is an alternative, a very likable one.
Vincent Warmerdam: It does work completely differently.
Vincent Warmerdam: One of the main things that attracts people to it is the fact that these kinds of notebooks are Python files under the hood.
Vincent Warmerdam: They're also a little bit more interactive.
Vincent Warmerdam: You can do more rapid prototyping with UI.
Vincent Warmerdam: You can blend Python with it very nicely.
Vincent Warmerdam: So I will say like all of my rapid prototyping, especially with LLMs, I do that in Marimo
Vincent Warmerdam: nowadays just so you can blend Python with UIs very easily.
Vincent Warmerdam: And there's demos on the site that people can go ahead and check out.
Vincent Warmerdam: But that's also the second YouTube channel I made this year.
Vincent Warmerdam: I do a lot of stuff for Marimo.
Vincent Warmerdam: I'm a little bit more on the growth side of things than the hardcore engineering side of things.
Vincent Warmerdam: I still make a lot of plugins for Marimo, of course, but that's a little bit more of what I do nowadays.
Vincent Warmerdam: Making sure that people learn about Marimo.
Vincent Warmerdam: You can do things in a notebook now that you couldn't think of before.
Vincent Warmerdam: I write my command line apps in a notebook nowadays because it's actually not just because I can, but because it's convenient too.
Vincent Warmerdam: So we'll talk a little bit about that later when we talk about LLMs, but that's effectively the short story of what I've been doing last year.
Michael Kennedy: Nice. All right. I am a fan of Marimo.
Michael Kennedy: Happy to hear it.
Michael Kennedy: Yeah, yeah, yeah. I did a data science course this year, just enough Python and software engineering for data scientists. Like, you know, what could you bring from the software engineering side to like sort of make your, your data engineering, data science stuff a little more reliable. But I was right on the fence of should I, should I use this? Because it's, I think it's clearly better. But at the same time, I also want to use what people are using. So I didn't quite go for it. But I just, the UI is fantastic.
Michael Kennedy: the reproducibility, reliability of it,
Michael Kennedy: where it uses the abstract syntax tree
Michael Kennedy: or concrete, whatever,
Michael Kennedy: to understand relationships across cells
Michael Kennedy: so they don't get out of sync, potentially.
Michael Kennedy: Yeah, I think it's really nice.
Michael Kennedy: I had Akshay on the show,
Michael Kennedy: who you work with also,
Michael Kennedy: to talk about it.
Vincent Warmerdam: The one thing in Jupyter that's,
Vincent Warmerdam: I mean, let me start by saying,
Vincent Warmerdam: Jupyter is still a really cool project.
Vincent Warmerdam: Like if I think back of the last decade,
Vincent Warmerdam: like all the good that that project
Vincent Warmerdam: has brought to my career,
Vincent Warmerdam: not just directly as a user,
Vincent Warmerdam: but also indirectly,
Vincent Warmerdam: all the algorithms that got invented
Vincent Warmerdam: simply because we had a good enough
Vincent Warmerdam: interactive environment suddenly, which we never
Vincent Warmerdam: had before, it's done wonders.
Vincent Warmerdam: So we should definitely be thankful.
Michael Kennedy: Yeah, I'm not bashing on it either.
Vincent Warmerdam: Yeah, but I do always want to
Vincent Warmerdam: make sure that I give credit where credit is due, because
Vincent Warmerdam: the project had a lot of impact.
Vincent Warmerdam: But there is this really annoying thing with Jupyter, though.
Vincent Warmerdam: Besides, you can be a bit grumpy
Vincent Warmerdam: about the file format, sure, but the one thing that's
Vincent Warmerdam: very annoying is you can write down
Vincent Warmerdam: X is equal to 10, and then
Vincent Warmerdam: have all sorts of cells below it that depend on X,
Vincent Warmerdam: You could then delete the cell and the notebook will not complain to you about it.
Vincent Warmerdam: Even though if anyone else tries to rerun the notebook, X is gone, it won't run, and your notebook is broken and you can't share the thing anymore.
Vincent Warmerdam: And nowadays, fast forward like many years later, and we've got stuff like uv now.
Vincent Warmerdam: So we can add metadata to a Python file to add dependencies.
Vincent Warmerdam: And oh, wait, because Marimo is a Python file, we can add dependencies to the notebook that makes it super reproducible.
Vincent Warmerdam: There's just all this stuff that you can rethink now simply because we have a good notebook format that is still a Python file.
Vincent Warmerdam: That's really the killer feature here.
Vincent Warmerdam: We can talk more about it if you like, but I can talk for ages about this topic, just warning you.
Vincent Warmerdam: One final thing, maybe also for the software engineering side of things, because you didn't mention it.
Vincent Warmerdam: Just to give an example of something we added recently.
Vincent Warmerdam: If you have a cell in Marimo and there's a function in there that starts with test underscore, we will automatically assume it's a pytest.
Vincent Warmerdam: So you can actually add unit tests to your notebook as well.
Vincent Warmerdam: And then if you say python notebook.py,
Vincent Warmerdam: then pytest will just run all the tests for you,
Vincent Warmerdam: even though you can also run the tests in line in your CI CD as well.
Vincent Warmerdam: So there's all sorts of these Python specific things that we can add,
Vincent Warmerdam: again, because it's just a Python file.
Michael Kennedy: Yeah, it's sort of the second mover advantage or nth mover advantage
Michael Kennedy: where n is greater than one, where you see, okay, that was awesome.
Michael Kennedy: Maybe this was a little bit of a rough edge.
Michael Kennedy: And what would we do to work around that or smooth it out, right?
Vincent Warmerdam: Well, and also we're also lucky that we're born in the age of uv.
Vincent Warmerdam: I got to say like a lot of like quality of life improvements to a notebook.
Vincent Warmerdam: A lot of that is also due to the fact that uv is around.
Vincent Warmerdam: That's definitely a huge help as well.
Michael Kennedy: This portion of Talk Python To Me is brought to you by Sentry's Seer.
Michael Kennedy: I'm excited to share a new tool from Sentry, Seer.
Michael Kennedy: Seer is your AI driven pair programmer that finds,
Michael Kennedy: diagnoses and fixes code issues in your Python app faster than ever.
Michael Kennedy: If you're already using Sentry, you are already using Sentry, right?
Michael Kennedy: Then using Seer is as simple as enabling a feature on your already existing project.
Michael Kennedy: Seer taps into all the rich context Sentry has about an error.
Michael Kennedy: Stack traces, logs, commit history, performance data, essentially everything.
Michael Kennedy: Then it employs its agentic AI code capabilities to figure out what is wrong.
Michael Kennedy: It's like having a senior developer pair programming with you on bug fixes.
Michael Kennedy: SEER then proposes a solution, generating a patch for your code and even opening a GitHub pull request.
Michael Kennedy: This leaves the developers in charge because it's up to them to actually approve the PR.
Michael Kennedy: But it can reduce the time from error detection to fix dramatically.
Michael Kennedy: Developers who've tried it found it can fix errors in one shot that would have taken them hours to debug.
Michael Kennedy: SEER boasts a 94.5% accuracy in identifying root causes.
Michael Kennedy: SEER also prioritizes actionable issues with an actionability score, so you know what to fix first.
Michael Kennedy: This transforms Sentry errors into actionable fixes, turning a pile of error reports into an ordered to-do list.
Michael Kennedy: If you could use an always-on-call AI agent to help track down errors and propose fixes before you even have time to read the notification, check out Sentry's SEER.
Michael Kennedy: Just visit talkpython.fm/SEER, S-E-E-R.
Michael Kennedy: The link is in your podcast player show notes.
Michael Kennedy: Be sure to use our code, Talk Python.
Michael Kennedy: One word, all caps.
Michael Kennedy: Thank you, Dysentry, for supporting Talk Pythonemy.
Michael Kennedy: I saw on, gosh, where was it?
Michael Kennedy: Speaking of Reddit, I saw on slash R,
Michael Kennedy: learn Python or slash R Python.
Michael Kennedy: One of the slash R's with a Python substring.
Michael Kennedy: Someone asks, what Python package manager would you use now?
Michael Kennedy: And it's just like, how many times can you say UV?
Michael Kennedy: The feedback comments.
Michael Kennedy: I mean, it was someone new who wasn't unsure, right?
Michael Kennedy: They've seen all the different ones.
Michael Kennedy: And yeah.
Vincent Warmerdam: I mean, the coolest comparison I've seen, I think it was a tweet someone posted.
Vincent Warmerdam: But like, definitely, like, suppose you're in the data field right now.
Vincent Warmerdam: Like, what are the tools 10 years ago?
Vincent Warmerdam: What are the tools now?
Vincent Warmerdam: It definitely does feel like, okay, before we had pip, now we have uv.
Vincent Warmerdam: Before we had Pandas, now we have Polars.
Vincent Warmerdam: Before we had Matplotlib, now we have Altair.
Vincent Warmerdam: And before we had Jupyter, now we've got Marino.
Vincent Warmerdam: You can kind of see a generational shift, not just on the notebook side of things,
Vincent Warmerdam: but like on the package manager, on the data frame library,
Vincent Warmerdam: we're all learning from the previous generation
Vincent Warmerdam: and it kind of feels like.
Michael Kennedy: Yeah, absolutely.
Michael Kennedy: It's an amazing time.
Michael Kennedy: Every single day in Python is just more exciting
Michael Kennedy: than the day before.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: Although I should also mention, especially with the Polars
Vincent Warmerdam: thing, there's a fair bit of rust too.
Vincent Warmerdam: That also makes a difference.
Michael Kennedy: Yeah, yeah, of course.
Michael Kennedy: So before Flyby, I mean, you talked a bit about your YouTube
Michael Kennedy: channel, you review the economic keyboards.
Michael Kennedy: you have for those people who are just listening, not watching your background is super nice.
Michael Kennedy: Your bona fides are solid here, but you've got a bunch of different ones. And I personally also,
Michael Kennedy: I don't know, you might even just kick me out of the club if I should do that.
Michael Kennedy: That's the Microsoft Sculpt, right?
Michael Kennedy: Microsoft Sculpt ergonomic. And I like it so much because I can take this little bottom thing off
Michael Kennedy: and it's like razor thin if I could rotate and reverse and jam it in my backpack and take it
Michael Kennedy: with me. But I've also had RSI issues for 25 years or something. And it was really bad. And I switched
Michael Kennedy: I used to type them on like laptops and stuff.
Michael Kennedy: And I switched to just something like this and I could type 10 hours a day and
Michael Kennedy: there's no problem. Whereas before, if I,
Michael Kennedy: if I were to force to type full on, on a laptop,
Michael Kennedy: I would probably be like unable to work in a week. If not a week.
Vincent Warmerdam: So we had an offsite in San Francisco. So I was there like two weeks ago,
Vincent Warmerdam: a lot of fun, but like, I'm not going to bring like,
Vincent Warmerdam: airport security is going to look at this and wonder what,
Vincent Warmerdam: like what kind of weird alien device this is. Right. So I figured, okay,
Vincent Warmerdam: I'll leave those at home and just bring my normal laptop. And after a week,
Vincent Warmerdam: I'm feeling it in my wrist again.
Vincent Warmerdam: It's not good.
Vincent Warmerdam: Yeah, it's something that takes serious.
Vincent Warmerdam: Although I will also say like
Vincent Warmerdam: because the kid was sick and the wife was sick
Vincent Warmerdam: so I wasn't able to do a whole lot of exercise at home.
Vincent Warmerdam: Also do exercise.
Vincent Warmerdam: That also totally helps.
Vincent Warmerdam: If I had to like,
Vincent Warmerdam: sure, buy an ergonomic keyboard if you like.
Vincent Warmerdam: Programmatic keyboards, they're great.
Vincent Warmerdam: But if you're going to do that as an excuse
Vincent Warmerdam: not to do exercise, you're doing it wrong.
Vincent Warmerdam: That's the one thing I do want to add to that.
Michael Kennedy: There's lots of stretches you can do.
Michael Kennedy: Taking breaks matters.
Michael Kennedy: Exercise matters.
Michael Kennedy: But I think those keyboards are an essential feature.
Michael Kennedy: They do totally help.
Vincent Warmerdam: And also, sometimes it's not so much a healing measure.
Vincent Warmerdam: It's more of a preventative measure.
Michael Kennedy: Yes, 100%.
Michael Kennedy: And I would like to put this message out just to everyone listening who is young, absolutely indestructible.
Michael Kennedy: Please.
Michael Kennedy: I know they're a bit of a pain in the butt with the weird curviness.
Michael Kennedy: It is so worth it to just say, I've never had any RSI issues.
Michael Kennedy: I just use this weird keyboard and people make fun of me,
Michael Kennedy: but I don't care.
Michael Kennedy: You know, that is a better thing than like, I'm working on it.
Michael Kennedy: Yeah.
Michael Kennedy: Oh, my.
Michael Kennedy: It hurts.
Michael Kennedy: You know?
Vincent Warmerdam: Well, another thing is also, especially now
Vincent Warmerdam: we've got stuff like Claude, right?
Vincent Warmerdam: I just want-- if I can point out like one thing on that front.
Vincent Warmerdam: So these ergonomic keyboards, you can often
Vincent Warmerdam: program them as you see fit.
Vincent Warmerdam: So you can say, like, if I hit this key, it's K.
Vincent Warmerdam: But if I hold it down, it becomes command,
Vincent Warmerdam: or whatever you like.
Vincent Warmerdam: But it also means that you can map macros or shortcuts.
Vincent Warmerdam: So whenever I hit this button, an app called MacWhisperer boots
Vincent Warmerdam: That's going to record my voice.
Michael Kennedy: I love Mac Whisper.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: And then there's alternative variants for it.
Vincent Warmerdam: I'm sure you've got stuff for Linux as well.
Vincent Warmerdam: But the main thing it does is just really good speech to text.
Vincent Warmerdam: And then whenever I'm done talking,
Vincent Warmerdam: it's just immediately going to paste whatever I'm in.
Vincent Warmerdam: So, oh, that's actually very nice,
Vincent Warmerdam: because now the keyboard shortcut is just me holding my thumb down
Vincent Warmerdam: instead of some sort of weird claw I got to eject to my keyboard.
Vincent Warmerdam: And suddenly, it doesn't necessarily become a convenience thing.
Vincent Warmerdam: It also becomes kind of a power user thing.
Vincent Warmerdam: Like you can really customize your computer experience
Vincent Warmerdam: if you can really map everything to your keyboard
Vincent Warmerdam: just the way that you like.
Vincent Warmerdam: It's like having Vim, but for all the apps out there.
Michael Kennedy: Okay, that's very neat.
Michael Kennedy: Yeah, I've remapped Caps Lock to my Activate Mac Whisper.
Michael Kennedy: So if I hit Caps Lock and hold this down,
Michael Kennedy: then I can dictate.
Michael Kennedy: And I know computers have had dictation for a long time,
Michael Kennedy: but it's been really bad, right?
Michael Kennedy: The built-in dictation to macOS or Windows isn't great,
Michael Kennedy: especially when you try to talk acronyms like,
Michael Kennedy: hey, do PyPI, and it's like...
Michael Kennedy: Yeah, Mac Whisper.
Vincent Warmerdam: No, but Mac Whisper, it's not like all the way there,
Vincent Warmerdam: but I will say it surprised me in quality a few times, definitely.
Michael Kennedy: Yeah, yeah.
Michael Kennedy: And that app, you can go in and actually set replacements.
Michael Kennedy: Like if you ever think you're going to write this, write that instead.
Michael Kennedy: So I've done that with like Talk Python
Michael Kennedy: because it always just turns it into a camel case combined.
Michael Kennedy: I'm like, no, those are two separate words.
Michael Kennedy: You know, whatever.
Michael Kennedy: You can sort of customize it a bit.
Vincent Warmerdam: Yeah, and I think my favorite one,
Vincent Warmerdam: I think Mac Whisper actually gets this one wrong still,
Vincent Warmerdam: But whenever I write scikit-learn, which is a very popular data science package, it always
Vincent Warmerdam: translates it to psychologists learn.
Vincent Warmerdam: Which, you know.
Michael Kennedy: I never want you to ever write this series of words.
Michael Kennedy: Because if I have to, I'll just type it all in time.
Michael Kennedy: It's not my common thing.
Vincent Warmerdam: It's one of these moments where actually I type scikit-learn often enough where it's
Vincent Warmerdam: like almost becoming the issue.
Vincent Warmerdam: So I'm like at the verge of adding these rules manually
Vincent Warmerdam: for all these weird open source packages
Vincent Warmerdam: that I interact with.
Vincent Warmerdam: But yeah, it's take ergonomics serious people.
Vincent Warmerdam: That's the one thing I wanna say.
Vincent Warmerdam: You don't always have to buy a super expensive keyboard.
Vincent Warmerdam: The, if you wanna explore like programmatic keyboards
Vincent Warmerdam: because you can customize things, that's an excellent reason.
Vincent Warmerdam: But like take a break and do exercises and just,
Vincent Warmerdam: you know, be healthy.
Vincent Warmerdam: That's the, that's the, that's how you win a marathon.
Michael Kennedy: - Yes, that's for sure.
Michael Kennedy: Now this is not the Vincent's keyboard review marathon,
Michael Kennedy: But let's wrap it up with, if you could take any keyboard
Michael Kennedy: hanging around on your wall there, which one would you use?
Vincent Warmerdam: - Well, so there's four pairs of these,
Vincent Warmerdam: so it's probably this one.
Vincent Warmerdam: So there's a couple of other boards that are great too.
Vincent Warmerdam: This board is not the board I would recommend to everyone,
Vincent Warmerdam: but if you have serious RSI issues,
Vincent Warmerdam: I do think the GloVe 80 is your best bet.
Vincent Warmerdam: It's simple, in terms of the shape,
Vincent Warmerdam: it is probably the most ergonomic shape for most hand sizes.
Michael Kennedy: - Yeah, okay, awesome.
Michael Kennedy: All right, well, let's switch over to talking about programming.
Michael Kennedy: Yes.
Michael Kennedy: LLMs with LLMs, not programming LLMs.
Michael Kennedy: And I guess, like you already called out, you wrote this course called
Michael Kennedy: LLM building blocks for Python.
Michael Kennedy: Super fun course, really neat, it's pretty short and concise.
Michael Kennedy: And it really talks about how can you reliably add some kind of LLM into your code.
Michael Kennedy: I guess what you're talking about in this course really applies regardless of
Michael Kennedy: whether it's a self-hosted one or it's open AI or Anthropic, right?
Michael Kennedy: You can have, there's some,
Michael Kennedy: some choices you can make on which LLM to use, right?
Vincent Warmerdam: So the, like the main idea I had with the course was like,
Vincent Warmerdam: an LLM is a building block at some point,
Vincent Warmerdam: but it's very unlike a normal building block when you're dealing with code,
Vincent Warmerdam: because normally with code,
Vincent Warmerdam: you put something into a function and like one thing comes out.
Vincent Warmerdam: But in this particular case, you put a thing into a function,
Vincent Warmerdam: you have no idea upfront what's going to come out.
Vincent Warmerdam: And not only that, but you put the same thing in twice
Vincent Warmerdam: and something else might come out as well.
Vincent Warmerdam: So that means that you're going to want to think about this tool
Vincent Warmerdam: a bit more defensively.
Vincent Warmerdam: It's a weird building block.
Vincent Warmerdam: It's like you have to put a moat around it
Vincent Warmerdam: because otherwise the building block is going to do stuff
Vincent Warmerdam: you don't want it to do, almost.
Vincent Warmerdam: And some of that is syntax.
Vincent Warmerdam: Some of that is how do you think about these Lego bricks.
Vincent Warmerdam: And some of it is also just what is good methodology in general
Vincent Warmerdam: to statistically test if the thing is doing roughly what you want it to do.
Vincent Warmerdam: Nice.
Vincent Warmerdam: That's the gist of it.
Michael Kennedy: Yeah, cool.
Michael Kennedy: Yeah, I pulled out a couple of ideas, concepts, and tools
Michael Kennedy: that you talked about throughout the course.
Michael Kennedy: And you don't have to have taken the course
Michael Kennedy: to have these things be useful.
Michael Kennedy: I just thought it might be fun to riff on some of the things
Michael Kennedy: you touched on here.
Michael Kennedy: The main thing I think will be fun is--
Vincent Warmerdam: it's been half a year, I think.
Vincent Warmerdam: It will be fun how much of it is still intact.
Vincent Warmerdam: I think most of it still definitely is.
Vincent Warmerdam: But it might be fun to see if we can find anything
Vincent Warmerdam: that might be dated, just to see if the world has moved on
Vincent Warmerdam: quickly.
Vincent Warmerdam: I think there's only one thing, but I'm just kind of curious.
Vincent Warmerdam: Let's see.
Michael Kennedy: Yeah, all right.
Michael Kennedy: Well, let's keep our radar up for that.
Michael Kennedy: It's definitely something that's more changing quicker
Michael Kennedy: and has a higher likelihood of being dated.
Michael Kennedy: But I think it holds up pretty well.
Vincent Warmerdam: Yeah, okay.
Vincent Warmerdam: One of the things I remember emphasizing is you want to do some stuff like caching.
Vincent Warmerdam: So let's say you've got a function and you use an LLM for it.
Vincent Warmerdam: And let's keep it simple.
Vincent Warmerdam: Let's say we're just making summaries.
Vincent Warmerdam: So talk Python episode paragraph goes in, single sentence is supposed to come out.
Vincent Warmerdam: Something like that.
Vincent Warmerdam: Okay, well, you might have a loop and you're going to do maybe one pass, try one LLM with
Vincent Warmerdam: one type of setting, try another LLM with different type of settings to generate all
Vincent Warmerdam: this data.
Vincent Warmerdam: It would be a shame that you're going to use an LLM, which is kind of an expensive compute
Vincent Warmerdam: thing, if you put the same input in by accident and then you incur the cost twice.
Vincent Warmerdam: That would really stink.
Vincent Warmerdam: So one of the things you always want to do is think a little bit about caching.
Vincent Warmerdam: And there's a Python library called disk cache that I've always loved to use.
Vincent Warmerdam: And I highly recommend people have a look at it.
Vincent Warmerdam: I think Michael, you've also used it in one of your courses before.
Vincent Warmerdam: The trick is we have to talk about this.
Vincent Warmerdam: It's so good.
Vincent Warmerdam: It is so good.
Michael Kennedy: It is SQLite and is so good.
Michael Kennedy: It is even better than SQLite.
Michael Kennedy: It is unbelievably good.
Michael Kennedy: And I have you to think I knew about it, but yeah, whatever.
Michael Kennedy: And then after I saw you use it, I'm like, genius.
Vincent Warmerdam: It is genius.
Vincent Warmerdam: Yes.
Vincent Warmerdam: No, so it's like having the Eliru disk cache.
Vincent Warmerdam: But it's also on disk.
Vincent Warmerdam: So if you were to restart Python, you still have everything in cache basically.
Vincent Warmerdam: And it's a SQLite database, so you can always inspect all the stuff that's in there.
Vincent Warmerdam: If you wanted to, you can also do fancy things like add time to live to every single object.
Vincent Warmerdam: And this is something you could do in a Docker container for a web app.
Vincent Warmerdam: But the main thing that's always nice when you're dealing with LLMs is you always want
Vincent Warmerdam: to be able to say in hindsight, like, okay, how did this LLM compare to that one?
Vincent Warmerdam: You want to compare outputs.
Vincent Warmerdam: then just writing a little bit of a decorator on top of a function
Vincent Warmerdam: is the way to put it in SQLite, and you're just done with that concern.
Vincent Warmerdam: That is just amazing.
Vincent Warmerdam: And we're using this cache directly.
Vincent Warmerdam: If you're using LLM by Simon Willison from the command line,
Vincent Warmerdam: there's also a mechanism there so that you can get it into SQLite
Vincent Warmerdam: if you wanted to.
Vincent Warmerdam: So that's also a feature you could consider.
Michael Kennedy: But--
Michael Kennedy: MARK MANDEL: Of course, he's going to put something in SQLite.
Michael Kennedy: Like, he can write a library that doesn't put something in SQLite,
Michael Kennedy: given his data set project.
Michael Kennedy: MARK MANDEL: It's Simon Willison.
Vincent Warmerdam: He'll put SQLite in SQLite if it's a Sunday.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: But if you haven't used Discash before,
Vincent Warmerdam: it definitely feels like one of these libraries
Vincent Warmerdam: that because I have it in my back pocket,
Vincent Warmerdam: it just feels like I can tackle more problems.
Vincent Warmerdam: That's the short story of it.
Vincent Warmerdam: And again, the course uses it in a very sort of basic fashion,
Vincent Warmerdam: but knowing that everything you do in an LLM
Vincent Warmerdam: only needs to happen once,
Vincent Warmerdam: if you're interested in using it once,
Vincent Warmerdam: that just saves you so much money.
Michael Kennedy: Yeah, and it's so useful.
Michael Kennedy: So one of the things you did in the course is you said,
Michael Kennedy: All right, the key for the value that we're going to store is going to be the model, the model settings and the prompt as a tuple, right?
Michael Kennedy: Something along those lines.
Michael Kennedy: And then you use that as the key.
Michael Kennedy: So if any of those variations change, does the model change?
Michael Kennedy: Do the settings change?
Michael Kennedy: Or like anything, that's a totally different request.
Michael Kennedy: And then you just store the response.
Michael Kennedy: And then boom, if you ask the exact same question of the same model with the same prompt with the same settings, why do you need to go and wait 10 seconds and burn money and environmental badness?
Michael Kennedy: when you could literally within a microsecond
Michael Kennedy: get the answer back.
Vincent Warmerdam: Yeah, and there's also a fancy thing.
Vincent Warmerdam: It's a trick you can do on top.
Vincent Warmerdam: So sometimes you want to say,
Vincent Warmerdam: well, there's a statistical thing also happening.
Vincent Warmerdam: So sometimes I want to have one input
Vincent Warmerdam: and actually store maybe 10 outputs
Vincent Warmerdam: so I can look at all these outputs,
Vincent Warmerdam: maybe annotate that later.
Vincent Warmerdam: And the way you solve that
Vincent Warmerdam: is you just add an integer to the tuple, basically.
Vincent Warmerdam: And then you're also able to store many outputs
Vincent Warmerdam: if you really like.
Michael Kennedy: The sky's the limit.
Michael Kennedy: The fourth one, the eighth one, whatever.
Michael Kennedy: Yeah, that's flexible.
Michael Kennedy: Yeah.
Michael Kennedy: And it does a whole bunch of neat, neat things that are really, really wild.
Michael Kennedy: Like, so it looks like just, Hey, I put this value into it.
Michael Kennedy: Right.
Michael Kennedy: And it stores it.
Michael Kennedy: It's really powerful because across application executions.
Michael Kennedy: So like maybe if you're caching that response in your notebook and what are
Michael Kennedy: you doing some testing, if you come back later, you've start the notebook back up
Michael Kennedy: or restart the kernel or whatever, like it's not like an LRU cache.
Michael Kennedy: It remembers cause it's stored somewhere in like temporary storage in a local
Michael Kennedy: SQLite file, which is amazing.
Michael Kennedy: It also has interesting ideas.
Michael Kennedy: I'm not sure really where they are,
Michael Kennedy: but it has different kinds of caches as well.
Michael Kennedy: So maybe you're storing a ton of stuff in there.
Michael Kennedy: And so it'll do basically built-in sharding.
Michael Kennedy: Oh, yeah.
Michael Kennedy: Multiple SQLite files.
Michael Kennedy: And it's really, really good.
Michael Kennedy: This is a deep library.
Michael Kennedy: This is not just, oh, yeah.
Michael Kennedy: It's like LRU, but to desk.
Vincent Warmerdam: So the cool thing about that library is it really does go deep.
Vincent Warmerdam: But if you really just want to use it as a dictionary, you can.
Vincent Warmerdam: That's the thing that I really love about it.
Vincent Warmerdam: But for all intents and purposes, you just treat it as a dictionary, and you're good.
Vincent Warmerdam: Or use it as a decorator on a function, and again, you're good.
Vincent Warmerdam: So again, it's one of those little hacks where, oh, if you just know about the library,
Vincent Warmerdam: you just become more productive at typical Python stuff.
Vincent Warmerdam: Yeah.
Michael Kennedy: I'll give you a place where I'm using it, actually.
Michael Kennedy: I use it on some LLM stuff, like where I'm programming against an LLM, for exactly the
Michael Kennedy: reason you did.
Michael Kennedy: Because if you have the same inputs, don't ask the question again.
Michael Kennedy: You just hear the answer.
Michael Kennedy: It's real, real fast.
Michael Kennedy: But if you go over to like Talk Python, let's see here.
Michael Kennedy: Go over to the guests, for example, right?
Michael Kennedy: So there's a bunch of guests.
Michael Kennedy: And here we have Vincent.
Michael Kennedy: Not that Vincent.
Michael Kennedy: There you are, that Vincent.
Michael Kennedy: All these Vincents.
Michael Kennedy: They're all over the...
Michael Kennedy: I have like 560 guests or something.
Michael Kennedy: There's a lot.
Michael Kennedy: But in here, you'll notice that if you go into the view of the source on this thing,
Michael Kennedy: like all over the place, anytime there's a picture,
Michael Kennedy: it'll have this little cache-busting ID on the end.
Michael Kennedy: And that's fine when it's served locally
Michael Kennedy: because you can just look at the file
Michael Kennedy: and go just the file still with the same cache at startup.
Michael Kennedy: But if it comes from like an S3 blob storage,
Michael Kennedy: you know, and the app restarts,
Michael Kennedy: how do I know what that is?
Michael Kennedy: Like it has to go,
Michael Kennedy: it would have to re-download the entire content of the blob.
Michael Kennedy: So check this out.
Michael Kennedy: Yeah.
Michael Kennedy: So I just added.
Michael Kennedy: Yeah, yeah, yeah.
Vincent Warmerdam: So this feels like there's something between the proper CDN
Vincent Warmerdam: and then getting it from S3.
Vincent Warmerdam: Like there are these moments
Vincent Warmerdam: when you want to have something that's kind of in between.
Michael Kennedy: And then disk cache could actually be a good solution for it.
Michael Kennedy: Exactly.
Michael Kennedy: So what the site does is it has a disk cache.
Michael Kennedy: And anytime it says, hey, I want to refer to a resource that's external,
Michael Kennedy: it'll download it once, compute the hash,
Michael Kennedy: and then store it in the disk cache unless you change something behind it.
Michael Kennedy: And so it's automatically using this, and it makes everything like,
Michael Kennedy: there's never stale resources, and it's instantly fast,
Michael Kennedy: even if they're served out of something remote like S3 equivalent.
Vincent Warmerdam: And do you also do a time to live?
Vincent Warmerdam: Is it also the time to live is also like every day it's allowed to refresh once or something like that, I suppose?
Michael Kennedy: Yeah, for the S3 stuff, I don't because I've set up all the admin functions that if I ever change one through the admin, it deletes it out of the cache.
Michael Kennedy: Gotcha.
Michael Kennedy: So it's like it's internally consistent.
Michael Kennedy: But for like other things, like if it parses something out of the, say, the description, which is set in the dictionary, that stuff, it's just got a time to live of like a day or something.
Michael Kennedy: And it's got like, there's a bunch of those.
Michael Kennedy: I'm using all these different places.
Michael Kennedy: And wow, it's so good.
Michael Kennedy: I just wanted to say thank you, because I
Michael Kennedy: knew we were going to talk about it today,
Michael Kennedy: because this is so good.
Vincent Warmerdam: Yeah, I know.
Vincent Warmerdam: If it was part of the standard library,
Vincent Warmerdam: I would honestly not be surprised.
Vincent Warmerdam: That's also the story with that.
Vincent Warmerdam: But yeah, SQLite is great.
Vincent Warmerdam: DiskCache is great.
Vincent Warmerdam: It feels like a dictionary, but it gives you so much more.
Vincent Warmerdam: It's great.
Vincent Warmerdam: That's the only thing I can say about it.
Michael Kennedy: So people are probably like, wait,
Michael Kennedy: I thought we were talking about LLMs.
Michael Kennedy: Yeah, we are.
Michael Kennedy: I think this is one of the interesting things,
Michael Kennedy: It's because there's all these interesting tools and it's not even about using agentic AI.
Michael Kennedy: And it's not like there's really cool libraries and tools that you can just apply to this LLM problem, but also apply everywhere else.
Michael Kennedy: Right.
Michael Kennedy: It's one of those.
Vincent Warmerdam: So it's one thing I have found with like LLMs in general if you're building software with it.
Vincent Warmerdam: On the one end, I think it can be very helpful if you're like a proper like senior engineer kind of a person because then you know about things like, oh, I want a cache.
Vincent Warmerdam: And what's the pragmatic cache?
Vincent Warmerdam: And you can also pick Redis, by the way.
Vincent Warmerdam: If you would have used Redis for this,
Vincent Warmerdam: that could have also worked.
Vincent Warmerdam: Just that I think--
Michael Kennedy: - Sure, it would have been fine.
Vincent Warmerdam: - Yeah, it would have been fine.
Michael Kennedy: - But you know what I like is there's no servers.
Michael Kennedy: I don't have to deal with the servers.
Michael Kennedy: Like, it's just, it's a file.
Vincent Warmerdam: - It's very true.
Michael Kennedy: - And it just keeps it simpler.
Vincent Warmerdam: - It's totally true.
Vincent Warmerdam: But the point I wanted to get out here,
Vincent Warmerdam: like your previous experience as a proper engineer
Vincent Warmerdam: will still help you write good LLM software.
Vincent Warmerdam: However, from a similar perspective,
Vincent Warmerdam: I also think that we do have this generation
Vincent Warmerdam: of like data scientists,
Vincent Warmerdam: and maybe data engineer, data analyst kind of person,
Vincent Warmerdam: like thinking analytically being quite critical of the output of an algorithm.
Vincent Warmerdam: That's also like a good bone to have in this day and age, I would say.
Vincent Warmerdam: Because if I've built a couple of recommenders back in my day, like it's been a decade now,
Vincent Warmerdam: but like one of the things you do learn when you're building a recommender is that you're stuck with this problem of,
Vincent Warmerdam: hey, I gave this user this recommendation and they clicked it.
Vincent Warmerdam: Would they have clicked this other thing if I would have recommended it to them?
Vincent Warmerdam: And suddenly you're dealing with a system that's like really stochastic and hard to predict.
Vincent Warmerdam: and you have to be kind of strict about the way you test and compare these different algorithms.
Vincent Warmerdam: And you want to think twice about the way you A-B test these things.
Vincent Warmerdam: And, oh, actually, just like disk cache is useful as a tool,
Vincent Warmerdam: having a little bit of methodology statistically in your mind will also help you
Vincent Warmerdam: because comparing LLMs, a lot of it is doing evaluations, being kind of strict about that.
Vincent Warmerdam: And that's also what I try to do in the course.
Vincent Warmerdam: I try to just show you that if you're really strict about evaluations,
Vincent Warmerdam: then you can also learn that for some problems, you're still better off using scikit-learn
Vincent Warmerdam: because you just evaluate it and then you learn that like the number is better on the side could
Michael Kennedy: learn side of things yeah that's when you feel like oh maybe i did it wrong where you paid
Michael Kennedy: a bunch of money to run expensive slow lms and you're like i would just use well so it's funny
Vincent Warmerdam: you say that so i'm i've actually been talking to a bunch of people that do lms at companies here in
Vincent Warmerdam: the netherlands and you know you you go to a pi data you go to a conference and you give them just
Vincent Warmerdam: enough beer so they're honest in that kind of a situation the nda curtain opens just a little or
Vincent Warmerdam: or whatever.
Vincent Warmerdam: Plus, more deniability is the name of the game in this one, yes.
Vincent Warmerdam: But then the stories you hear, which I did find somewhat encouraging,
Vincent Warmerdam: is they do all kind of go, well, there's budget now to do AI stuff,
Vincent Warmerdam: which means that we try out all the different LLMs,
Vincent Warmerdam: and we really can invest in evals and that sort of thing.
Vincent Warmerdam: And funnily enough, we also put some base models in there,
Vincent Warmerdam: like as a standard benchmark that we should beat.
Vincent Warmerdam: And I've heard a story a bunch of times now
Vincent Warmerdam: that because of the hype around LLMs and AI,
Vincent Warmerdam: after it was implemented, after they did all the benchmarks,
Vincent Warmerdam: It turns out that AI is the reason that scikit-learn is now in production in a bunch of places.
Michael Kennedy: And it's also the same thing with like spaCy.
Vincent Warmerdam: Because what a lot of people do learn is that, hey, if the spaCy model or like the lightweight model, so to say,
Vincent Warmerdam: is like somewhat equivalent to the LLM model after you give it like enough training data,
Vincent Warmerdam: which you do want to have either anyway, because you need to need that for evaluations.
Vincent Warmerdam: Well, typically those models are more lightweight and they will always produce the same output.
Vincent Warmerdam: Same thing goes in, same prediction will always come out.
Vincent Warmerdam: And you can really fence it off, have it on like a normal VM.
Vincent Warmerdam: That's also totally fine.
Vincent Warmerdam: Oh, and you know, another benefit, it's super lightweight to run.
Vincent Warmerdam: You just need a Lambda function and you're good,
Vincent Warmerdam: as opposed to like a GPU or like a huge cloud bill or something like that.
Vincent Warmerdam: So some of the stories that I'm hearing do suggest that,
Vincent Warmerdam: okay, these LLMs are also helping out like standard data science work, if it were,
Vincent Warmerdam: if only because management now really does want to be serious about the investment
Vincent Warmerdam: and really wants to do the experiments.
Michael Kennedy: portion of Talk Python To Me is brought to you by NordStellar. NordStellar is a threat exposure
Michael Kennedy: management platform from the Nord security family, the folks behind NordVPN that combines dark web
Michael Kennedy: intelligence, session hijacking prevention, brand abuse detection, and external attack service
Michael Kennedy: management. Keeping your team and your company secure is a daunting challenge. That's why you
Michael Kennedy: need NordStellar on your side. It's a comprehensive set of services, monitoring, and alerts to limit
Michael Kennedy: your exposure to breaches and attacks and act instantly if something does happen. Here's how
Michael Kennedy: it works. Nordstellar detects compromised employee and consumer credentials. It detects stolen
Michael Kennedy: authentication cookies found in InfoStealer logs and dark web sources and flags compromised devices,
Michael Kennedy: reducing MFA bypass ATOs without extra code in your app. Nordstellar scans the dark web for
Michael Kennedy: cyber threats targeting your company. It monitors forums, markets, ransomware blogs, and over 25,000
Michael Kennedy: cybercrime telegram channels with alerting and searchable contexts you can route to Slack or your
Michael Kennedy: IRR tool. Nordstellar adds brand and domain protection. It detects cyber squats and lookalikes
Michael Kennedy: via visual, content similarity, and search transparency logs, plus broader brand abuse
Michael Kennedy: takedowns across the web, social, and app stores to cut the phishing risk for your users.
Michael Kennedy: They don't just alert you about impersonation, they file and manage the removals.
Michael Kennedy: Finally, Nordstellar is developer-friendly.
Michael Kennedy: It's available as a platform and an API.
Michael Kennedy: No agents to install.
Michael Kennedy: If security is important to you and your organization, check out Nordstellar.
Michael Kennedy: Visit talkpython.fm/nordstellar.
Michael Kennedy: The link is in your podcast player's show notes and on the episode page.
Michael Kennedy: Please use our link, talkpython.fm/nordstellar,
Michael Kennedy: so that they know that you heard about their service from us.
Michael Kennedy: And you know what time of year it is.
Michael Kennedy: It's late fall.
Michael Kennedy: That means Black Friday is in play as well.
Michael Kennedy: So the folks at Nord Stellar gave us a coupon, BlackFriday20.
Michael Kennedy: That's BlackFriday, all one word, all caps, 20, two, zero.
Michael Kennedy: That grants you 20% off.
Michael Kennedy: So if you're going to sign up for them soon, go ahead and use BlackFriday20 as a code,
Michael Kennedy: and you might as well save 20%.
Michael Kennedy: It's good until December 10th, 2025.
Michael Kennedy: Thank you to the whole Nord security team for supporting Talk Python To Me.
Michael Kennedy: You could say, we're going to use your mandate is to add AI.
Michael Kennedy: And then you go user, well, we got to see how well it's working.
Michael Kennedy: So we tested it.
Michael Kennedy: And then we have to compare it to something of a benchmark.
Michael Kennedy: Exactly.
Michael Kennedy: The benchmark is fast and effectively free and about as good.
Michael Kennedy: So we're good.
Michael Kennedy: We're just going to do that.
Michael Kennedy: You know, I think it's fine.
Vincent Warmerdam: Some organizations need like mandate from above in order to get something done.
Vincent Warmerdam: And this LLM craze, if nothing else, does seem to have caused the mandate from above.
Vincent Warmerdam: I'm sure it is.
Vincent Warmerdam: So that's something I would keep in the back of your mind as well, dear listener.
Vincent Warmerdam: Like sometimes that mandate can also give you permission to do other things.
Michael Kennedy: Yeah.
Michael Kennedy: Not working for a large enterprise type company for quite a while.
Michael Kennedy: I imagine I'm pretty blind to how that is transforming directives from the top.
Michael Kennedy: But I'm sure bosses are like, I'm using ChatGPT and it's so much better than our software.
Michael Kennedy: What are we going to do about that?
Vincent Warmerdam: you know yeah it's uh well i mean i'm in the developer tooling space so i still talk to big
Vincent Warmerdam: companies and small companies as well you do notice that big companies they work differently
Vincent Warmerdam: than small companies that is certainly true um for better or worse like there's also good there's
Vincent Warmerdam: also good reasons why bigger companies like if i'm a bank like uh i'm pretty sure it's a good idea to
Vincent Warmerdam: also have some rules that i have to abide by which yeah i'm just thinking more of like how disconnected
Michael Kennedy: is the higher level management from the product and like how much do they just like dictate a thing
Michael Kennedy: Okay, so let's talk about some of the tools. One of the things you primarily focused on was using
Vincent Warmerdam: Simon Willison's LLM library. Tell us about this thing. So the funny thing about that library is
Vincent Warmerdam: that it's actually kind of more meant as a command line utility. I think that's kind of the entry
Vincent Warmerdam: point that he made. But then he also just made sure there was some sort of a Python API around it.
Vincent Warmerdam: And after looking at that library, and also after playing around with all these other libraries,
Vincent Warmerdam: and I'm about to say this, but this is a compliment.
Vincent Warmerdam: I just found the LLM library by Simon Willison
Vincent Warmerdam: by far to be the most boring.
Vincent Warmerdam: And I mean that really in a good way, just unsurprising,
Vincent Warmerdam: only does a few things.
Vincent Warmerdam: The few things that it does is just in a very predictable way.
Vincent Warmerdam: And especially if you're doing rapid prototyping,
Vincent Warmerdam: what I felt was just kind of nice is that
Vincent Warmerdam: it does feel like you have building blocks
Vincent Warmerdam: that allow you to build very quickly,
Vincent Warmerdam: but it doesn't necessarily feel like you're dealing with abstractions
Vincent Warmerdam: that can sort of wall you in at some point.
Vincent Warmerdam: So for a hello world getting started,
Vincent Warmerdam: just do some things in a very Pythonic way,
Vincent Warmerdam: this boring library really did the trick.
Vincent Warmerdam: And I'm also happy to report it's still a library that I use.
Vincent Warmerdam: It's still definitely a library in my tool, though.
Vincent Warmerdam: Whenever I want to do something real quick,
Vincent Warmerdam: that is definitely a tool that I refer to.
Vincent Warmerdam: And one of the things--
Michael Kennedy: Is it kind of an abstraction across the different LLMs
Michael Kennedy: and that kind of stuff?
Michael Kennedy: If I want to talk to Anthropic versus OpenAI, that doesn't matter?
Vincent Warmerdam: So the way that this is built is also, I think, with good taste.
Vincent Warmerdam: So what you don't want to do as a library is say, like, I'm going to basically support every library under the sun.
Vincent Warmerdam: Because as a single maintainer in the case of Simon Willis, you're just going to drown in all these different providers.
Vincent Warmerdam: So what he went for instead was a plugin ecosystem.
Vincent Warmerdam: Now, the downside of a plugin ecosystem is that then you defer the responsibility of maintaining a plugin for a specific source to another maintainer.
Vincent Warmerdam: But you might get someone who works at Mistral to make the Mistral plugin.
Vincent Warmerdam: And you might get someone who works at Open Router to make the Open Router plugin, etc.
Vincent Warmerdam: So you do distribute the workload in kind of a nice way.
Vincent Warmerdam: And all the bigger models are definitely supported.
Vincent Warmerdam: So the Anthropic and the OpenAI ones,
Vincent Warmerdam: those are just always in there.
Vincent Warmerdam: But you will also definitely find some of the more exotic ones
Vincent Warmerdam: that they will also just have a plug in themselves.
Vincent Warmerdam: And one thing that also helps under the hood
Vincent Warmerdam: is that OpenAI has a standard under the hood.
Vincent Warmerdam: Their SDK has become a bit of a standard across industry.
Vincent Warmerdam: So you can also reuse the OpenAI stuff.
Vincent Warmerdam: It would not surprise me at all that if you were just
Vincent Warmerdam: to change a URL in the setting, that you can also
Vincent Warmerdam: connect to Mistral via the OpenAI objects.
Vincent Warmerdam: I would have to double check, but that wouldn't surprise me.
Michael Kennedy: Yeah, that is a very common way of just like,
Michael Kennedy: you know what, we're all going to just adopt Open.
Michael Kennedy: It's a little like S3.
Michael Kennedy: Like when I was saying S3 earlier,
Michael Kennedy: I was actually talking to DigitalOcean.
Michael Kennedy: But it doesn't matter.
Michael Kennedy: I'm just still using Boto3 to talk to it.
Vincent Warmerdam: Yeah, it does feel weird like, oh, you want to use DigitalOcean?
Vincent Warmerdam: you have to download a SDK from a competing cloud provider and then you can.
Vincent Warmerdam: Exactly. It's so weird.
Vincent Warmerdam: But I mean, the thing I do find that it's just a little bit funny here is like technically,
Vincent Warmerdam: this thing is meant to be used from the command line. It just happens that it's written in Python
Vincent Warmerdam: and it just happens that also it has a decent Python API. And that's the thing I actually end up
Vincent Warmerdam: using more than stuff from the command line. That's because I do a lot of things in notebooks. So the
Vincent Warmerdam: stuff that I tend to use is a little bit more in the Python side.
Michael Kennedy: Sure, of course. Same here.
Michael Kennedy: Yeah, I don't think I would use it very much on the command line.
Michael Kennedy: But, you know, I can see using it as a step in a series of things happening on the command line,
Michael Kennedy: like some sort of orchestration, like X, Y, ask the LLM, Z, you know?
Vincent Warmerdam: Well, the main thing I actually use it for from the command line is you can do a git diff
Vincent Warmerdam: on whatever you're working on and then sort of pipe that through to the LLM to make the commit message.
Vincent Warmerdam: Like there are like these little moments like that where you do want to have a little automation,
Vincent Warmerdam: maybe in CI and like being able to do this from a command line is definitely useful it's just not
Vincent Warmerdam: the thing i use it for most or like if i use it that way it's a thing i automate once and then
Vincent Warmerdam: never really look at it but the interaction really for me is more in the notebook oh yeah that's a
Michael Kennedy: very cool librarian i definitely need for um like one thing that might be fun to point out because
Vincent Warmerdam: that's something i built i think uh around the same time like if you were to type into google
Vincent Warmerdam: uh smart funk and then my github alias koning a k-o-a-n-i-n-g um because it's a little
Michael Kennedy: okay you already had it open good man i'm really good with google actually i use a start page these
Vincent Warmerdam: days but i'm sure i could find it there oh okay here you're well you have an ergonomic keyboard
Vincent Warmerdam: so there you go quick typing um already yeah no so like a thing i was able to build on top of uh
Vincent Warmerdam: the thing that simon willison made and it's something that appears in the course as well
Vincent Warmerdam: that. Yeah, there you go. So basically what this function does is you can add a decorator.
Vincent Warmerdam: You just have a normal Python function. The inputs, you just put types on it. So you can say like,
Vincent Warmerdam: this input is a string, this input is an integer, or what have you. You then add a decorator that
Vincent Warmerdam: says what backend you want to use. So GPD4 or anything that Simon Willis and supports. And then
Vincent Warmerdam: the doc string is the prompt you give to the LLM. So you can do something like, hey, I have a function
Vincent Warmerdam: that accepts a term called paragraph.
Vincent Warmerdam: That's a string.
Vincent Warmerdam: And then the doc string of the function says,
Vincent Warmerdam: summarize this.
Vincent Warmerdam: And then lo and behold, you now have a Python function
Vincent Warmerdam: that can do summaries.
Michael Kennedy: I see.
Michael Kennedy: And the doc string can be effectively a Jinja template.
Michael Kennedy: Yes.
Vincent Warmerdam: If you wanted to, it could be a Jinja template.
Vincent Warmerdam: Alternatively, what you can also do
Vincent Warmerdam: is you can also say, well, what the function returns
Vincent Warmerdam: is the prompt.
Vincent Warmerdam: That's also something you can do.
Michael Kennedy: Got it.
Vincent Warmerdam: But this is just a very quick hack for some of the stuff
Vincent Warmerdam: that I want to do real quick.
Vincent Warmerdam: I don't necessarily recommend other people to use this,
Vincent Warmerdam: but this is something you can build on top of the LLM library
Vincent Warmerdam: from Simon Willison.
Vincent Warmerdam: So if you want to make your own syntactic sugar,
Vincent Warmerdam: your own abstractions, it's definitely something
Vincent Warmerdam: that you can also do.
Vincent Warmerdam: And also, again, the name of the game here
Vincent Warmerdam: is quick iteration.
Vincent Warmerdam: So also feel free to think of your own tools now and again.
Vincent Warmerdam: But you want the base layer to be as boring as possible.
Vincent Warmerdam: And Simon Willison's library, again, does that in a good way.
Michael Kennedy: Right, right.
Michael Kennedy: Just don't make me think about the details.
Michael Kennedy: Yes.
Michael Kennedy: Let me swap out the model providers, which is actually
Michael Kennedy: doing quite a bit, but not conceptually.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: So again, one thing I do think at this point in time,
Vincent Warmerdam: if you've not played around with this software stuff already,
Vincent Warmerdam: I do recommend maybe doing that sooner rather than later.
Vincent Warmerdam: And I think Simon Willison's approach,
Vincent Warmerdam: if you're already a Python person,
Vincent Warmerdam: it's probably the simplest way to get started.
Vincent Warmerdam: But I do want to warn people in the sense of these are just tools.
Vincent Warmerdam: Tools are useful.
Vincent Warmerdam: Tools are great.
Vincent Warmerdam: But at some point, it's not so much about the tools that you use,
Vincent Warmerdam: but more about how you think about the tools,
Vincent Warmerdam: Like the way you interact with these tools,
Vincent Warmerdam: is it not some, like you can use a hammer
Vincent Warmerdam: in many different ways,
Vincent Warmerdam: but the way you think about a hammer
Vincent Warmerdam: is like also equally important.
Michael Kennedy: - Yeah, 100%.
Michael Kennedy: So speaking of thinking about how this goes,
Michael Kennedy: one of the challenges is,
Michael Kennedy: so let's just take this example here.
Michael Kennedy: It says, the example on your smartphone,
Michael Kennedy: GitHub readme, it says generate summary,
Michael Kennedy: take some text, it says generate a summary,
Michael Kennedy: the following text, colon text.
Michael Kennedy: - Yes, text.
Michael Kennedy: - Super straightforward.
Michael Kennedy: But what if you, what if you said, change the prompt a little bit said, give me all of the
Michael Kennedy: Python libraries discussed in the following markdown content.
Michael Kennedy: Right.
Michael Kennedy: And do you want a novella back, like a novel back when, when really what you want is like,
Michael Kennedy: I want a list of strings and URLs or whatever.
Michael Kennedy: Like how do I get programmability instead of just chatability, I guess.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: So the thing that Simon Willis's library does allow you to do, is you are able to
Vincent Warmerdam: say, well, I have a prompt over here, but the output that I'm supposed to get out, well, that has to be
Vincent Warmerdam: a JSON object of the following type. And then you can use Pydantic. So you can do something like, well,
Vincent Warmerdam: I expect a list of strings to come out over here in the case that you're trying to detect Python
Vincent Warmerdam: libraries or Pokemon names or what have you. Or you can say, well, there's more like a classification.
Vincent Warmerdam: I believe you can pass a literal with like only these values and that's an also constraint. And
Vincent Warmerdam: the reason that you can do that is some of these LLMs are actually trained on specific tasks. One of
Vincent Warmerdam: these tasks could be tool calling. Another one of these tasks that are trained for these days is
Vincent Warmerdam: that you have to say the right word here, something, something output, structured output,
Vincent Warmerdam: so that the LLM can sort of give a guarantee that if you declare that you get a list of strings out,
Vincent Warmerdam: that you actually do get a list of strings out. And that's something that these things are
Vincent Warmerdam: typically trained for. So that's something you can do. If you're using open source models,
Vincent Warmerdam: definitely check them out. They're cool. But quality does vary, is what I will say.
Vincent Warmerdam: So that's something they always keep in the back of your mind.
Vincent Warmerdam: And also the more complex you make your PyDonic objects,
Vincent Warmerdam: like if you make like a nested thing of a nested thing of a nested thing,
Vincent Warmerdam: at some point the LLM is going to have a bit of trouble with that.
Vincent Warmerdam: Even though PyDonic is of course great,
Vincent Warmerdam: but like you make it harder for the LLM to do the right thing at some point.
Vincent Warmerdam: But that's also a thing that definitely you want to use
Vincent Warmerdam: if you're doing stuff with software.
Vincent Warmerdam: Like the really cool thing about PyDonic is you can say,
Vincent Warmerdam: I expect something that looks like this to come out
Vincent Warmerdam: and you can force an LLM to also actually do that.
Vincent Warmerdam: Like, you will get the right types that come out.
Vincent Warmerdam: The contents might still be maybe a bit spooky.
Vincent Warmerdam: But at least you get the right types out,
Vincent Warmerdam: which makes it a lot more convenient to write software.
Michael Kennedy: You can say, I can loop over the things that sent back,
Michael Kennedy: and you're not into vague text parsing land.
Vincent Warmerdam: Well, the main thing you get is--
Vincent Warmerdam: I remember when the first OpenAI models sort of started coming
Vincent Warmerdam: out, I worked back at Explosion.
Vincent Warmerdam: We made this tool called spaCy there.
Vincent Warmerdam: And one of the things that we were really bummed out about
Vincent Warmerdam: was LLMs can only really produce text.
Vincent Warmerdam: So this structural, like if you want to detect a substring in a text, for example,
Vincent Warmerdam: which is the thing that spaCy is really good at,
Vincent Warmerdam: then you really want to guarantee, well, I'm going to select a substring that actually did appear in the text.
Vincent Warmerdam: I want to know exactly where it starts and exactly where it ends, and that's the span that I want to select.
Vincent Warmerdam: So normally in NLP, text comes in and structured information comes out.
Vincent Warmerdam: But here come these LLMs and text comes in and text comes out.
Vincent Warmerdam: And then you've got to figure out a way to guarantee that structured output actually comes out.
Vincent Warmerdam: So that's the thing that the LN providers also acknowledge.
Vincent Warmerdam: So that's something that they're actually training these models for.
Vincent Warmerdam: But I think it's a lot better these days,
Vincent Warmerdam: but it was definitely a huge part of the problem
Vincent Warmerdam: like three to four years ago when these things just happened.
Vincent Warmerdam: It was super annoying.
Michael Kennedy: So if you use Pydantic, you define a Pydantic model
Michael Kennedy: that drives from Pydantic.basedModel.
Michael Kennedy: You put the types, type information, fields, et cetera.
Michael Kennedy: And then what does it do?
Michael Kennedy: Does it say, kind of use the schema generation
Michael Kennedy: that Pydantic already has?
Michael Kennedy: Like people are familiar with FastAPI.
Michael Kennedy: You set the response model, it'll do open API spec,
Michael Kennedy: which I believe just comes more or less from Pydantic, right?
Vincent Warmerdam: I could be.
Vincent Warmerdam: I think it's a JSON standard in the end.
Vincent Warmerdam: Like, you know how you can generate JSON schemas as well?
Vincent Warmerdam: I think it's that spec that is using under the hood
Vincent Warmerdam: that is then also giving off to OpenAI or what have you.
Vincent Warmerdam: So I think in the end, it's a JSON spec,
Vincent Warmerdam: but I could be wrong.
Michael Kennedy: But it's driven by the Pydantic model.
Michael Kennedy: And then that's given as part of the prompt.
Michael Kennedy: Like, here's my question.
Michael Kennedy: your answer will look like this.
Michael Kennedy: And it uses the Pydanic, something to that effect.
Vincent Warmerdam: Yeah, there's one small caveat there
Vincent Warmerdam: in the sense that you can also add custom types
Vincent Warmerdam: with Pydanic.
Vincent Warmerdam: So you can do something like,
Vincent Warmerdam: well, this is a non-negative integer, for example.
Vincent Warmerdam: And you can definitely imagine that,
Vincent Warmerdam: especially with these validations
Vincent Warmerdam: that you can write in Python,
Vincent Warmerdam: not everything that you can come up with there
Vincent Warmerdam: is going to be supported by this JSON spec.
Vincent Warmerdam: So you are going to get some of these weird edge cases
Vincent Warmerdam: maybe where Pydanic can do more
Vincent Warmerdam: than what the JSON spec can do on your behalf.
Vincent Warmerdam: There is a really cool library, though,
Vincent Warmerdam: called Instructor that actually--
Vincent Warmerdam: Yes, I was just thinking about that.
Vincent Warmerdam: So what they do is actually kind of cute.
Vincent Warmerdam: The impression I have is that they acknowledge
Vincent Warmerdam: that PyDanic can actually do a bit more than what the JSON
Vincent Warmerdam: spec can do, especially because the validator that you write
Vincent Warmerdam: can really be a custom whatever thing you care about.
Vincent Warmerdam: So what they do as a trick is they say, well,
Vincent Warmerdam: if the LLM gives me something back that doesn't get validated
Vincent Warmerdam: by PyDanic, we take the error message,
Vincent Warmerdam: together with the thing that we got back,
Vincent Warmerdam: make a new prompt to the LLM and just ask it again.
Vincent Warmerdam: And see, maybe now with the hint,
Vincent Warmerdam: it is able to give us the proper structure back.
Vincent Warmerdam: That's sort of the term.
Vincent Warmerdam: It sounds silly,
Michael Kennedy: but anyone who's done agentic AI and stuff,
Michael Kennedy: like, you know, Claude or whatever,
Michael Kennedy: that's a lot of the magic where you say,
Michael Kennedy: it tried this.
Michael Kennedy: Oh, I see.
Michael Kennedy: I actually forgot an import.
Michael Kennedy: Oh, I see.
Michael Kennedy: I passed the wrong command.
Michael Kennedy: Let me try again a different way.
Michael Kennedy: Oh, now it's working.
Michael Kennedy: And, you know,
Michael Kennedy: I think there's a lot of potential with that.
Vincent Warmerdam: So, well, definitely true.
Vincent Warmerdam: The one thing I think is safe to say at this point,
Vincent Warmerdam: although another researcher, right?
Vincent Warmerdam: So don't pin me down on this.
Vincent Warmerdam: That library did originate from an early era of LLMs
Vincent Warmerdam: when the structured stuff wasn't really
Vincent Warmerdam: being trained for yet.
Vincent Warmerdam: And it is my impression that LLMs have gotten a fair bit better
Vincent Warmerdam: at this structured task at this point.
Vincent Warmerdam: So you could argue that maybe that library is solving
Vincent Warmerdam: a solved problem with the technique that they've got.
Vincent Warmerdam: But I still don't want to discount it,
Vincent Warmerdam: because you inevitably, at some point,
Vincent Warmerdam: you might want to go for a very lightweight model
Vincent Warmerdam: that you can run locally that isn't fine-tuned
Vincent Warmerdam: structured output task. And in that situation, this technique can actually do a lot of good.
Vincent Warmerdam: Like you might be prompting the LLM for like three times, but you might be able to get away
Vincent Warmerdam: with that because the model is just so much lighter because it doesn't have to do all those tasks that
Michael Kennedy: those big LLMs are trained for. Right. Especially if you're running on your machine or you're
Michael Kennedy: self-hosting the model, you're not like, oh, it's going to cost us so much money. It's just,
Michael Kennedy: it'll take a little longer, but it's fine. Yeah. So like there's a lot of support for
Vincent Warmerdam: instructor. There's also Olama support for
Vincent Warmerdam: the LM library from Simon Willison,
Vincent Warmerdam: by the way. But yeah,
Vincent Warmerdam: you can definitely imagine this is still going
Vincent Warmerdam: to be relevant with the Olama stuff going forward.
Vincent Warmerdam: And also, it's a reasonable
Vincent Warmerdam: trick if you think about it.
Vincent Warmerdam: The main thing that we do is we just add a retry
Vincent Warmerdam: mechanic to the structured outputs by using
Vincent Warmerdam: the PyDonic validator, effectively.
Michael Kennedy: I mean, think about what you probably do, certainly what
Michael Kennedy: I do when I'm working with ChatGPT or something.
Michael Kennedy: It gives me a wrong answer like, no, this is not what I
Michael Kennedy: wanted. This is what I wanted. It's like, oh, okay.
Michael Kennedy: I see.
Michael Kennedy: And it's kind of automating that, more or less.
Vincent Warmerdam: It's certainly an attempt, yeah.
Vincent Warmerdam: In my experience, I have also seen moments
Vincent Warmerdam: when it sends the same error back five times,
Vincent Warmerdam: and then it just kind of fails.
Vincent Warmerdam: So there is also a limit to the retry mechanic.
Michael Kennedy: Sure, sure.
Michael Kennedy: It's a little bit like tenacity or stamina.
Michael Kennedy: Exactly.
Michael Kennedy: APIs or whatever.
Michael Kennedy: It uses tenacity under the hood, if I'm not mistaken.
Michael Kennedy: So there's definitely that mechanism in there.
Michael Kennedy: Nice.
Michael Kennedy: I just-- didn't work.
Michael Kennedy: Try it again.
Michael Kennedy: It's like turn it off and turn it on again.
Michael Kennedy: See if it works now?
Michael Kennedy: Yep, pretty much.
Michael Kennedy: Television.
Michael Kennedy: Yeah, a little more smart than that, though,
Michael Kennedy: because it does validate and return the error.
Michael Kennedy: So super cool.
Michael Kennedy: Yep.
Michael Kennedy: Okay.
Michael Kennedy: Have you worked with, I guess, I don't know,
Michael Kennedy: higher order programming models?
Michael Kennedy: So, you know, I saw Samuel Colvin yesterday
Michael Kennedy: talking about Pydantic AI.
Michael Kennedy: I saw on X, I saw Sidney Ruckel talking about
Michael Kennedy: some big release, like a 1.0 release
Michael Kennedy: or something at LangChain.
Michael Kennedy: What are your thoughts on some of these
Michael Kennedy: higher order orchestration libraries?
Vincent Warmerdam: I mean, the main thing for me, at least,
Vincent Warmerdam: this was a demo I did do with the Pydanic AI one.
Vincent Warmerdam: I do think types are quite powerful when you think of LLMs.
Vincent Warmerdam: There's a live stream you can check on the Marimo YouTube
Vincent Warmerdam: if you're keen to see a full demo.
Vincent Warmerdam: But let me just sketch you a problem that
Vincent Warmerdam: used to be really hard.
Vincent Warmerdam: Six years ago, we used to work at this company called Raza.
Vincent Warmerdam: We work on chatbot software.
Vincent Warmerdam: And a thing that was really hard is you--
Vincent Warmerdam: Let's say you're a chat bot, you can order a pizza.
Vincent Warmerdam: So you can do things like, yep, that's the one.
Vincent Warmerdam: It's like also one of the PyDonic AI engineers,
Vincent Warmerdam: and he's also the creator of Starlet, by the way.
Vincent Warmerdam: Cool project, Mariemost built on top of Starlet.
Vincent Warmerdam: Main maintainer of that project.
Vincent Warmerdam: He deserves to get more high fives from the community,
Vincent Warmerdam: so do you want to point it out?
Vincent Warmerdam: Well, let's say you're working on a pizza bot,
Vincent Warmerdam: and okay, someone is ordering a pizza.
Vincent Warmerdam: Suppose you go to the bot and you say,
Vincent Warmerdam: "Hey, I want to order a pepperoni pizza."
Vincent Warmerdam: But you can imagine there's a pizza type,
Vincent Warmerdam: and a pizza type says, "What's the kind of pizza?
Vincent Warmerdam: What's the size of the pizza?
Vincent Warmerdam: And do you want to have extra toppings?
Vincent Warmerdam: And that has to be a list of ingredients, let's say.
Vincent Warmerdam: But if I tell you, hey, you want to have a pepperoni pizza,
Vincent Warmerdam: but I don't know the size yet, well, then I
Vincent Warmerdam: need the LLM to ask a good follow-up question.
Vincent Warmerdam: OK, how are you going to build that logic?
Vincent Warmerdam: Because that can get quite gnarly quite quickly,
Vincent Warmerdam: especially if, in this case, the pizza is simple.
Vincent Warmerdam: But you can come up with a complicated object,
Vincent Warmerdam: like a complicated form almost.
Vincent Warmerdam: And then--
Michael Kennedy: Right, right.
Michael Kennedy: I only need to ask this question if they've answered that one
Michael Kennedy: in that way.
Michael Kennedy: Right, it could get really tricky, yeah.
Vincent Warmerdam: Well, and then the question is,
Vincent Warmerdam: how can you actually get the LLM to do this for you?
Vincent Warmerdam: And there's a trick that you can do with a type,
Vincent Warmerdam: because what you can say is,
Vincent Warmerdam: well, the response from the LLM has to be of type pizza,
Vincent Warmerdam: or bar string.
Vincent Warmerdam: And then if the prompt says something along the lines of,
Vincent Warmerdam: well, parse the output of the user such that it is a pizza,
Vincent Warmerdam: but then if this validation error occurs,
Vincent Warmerdam: oh, how do you express that logic?
Vincent Warmerdam: Well, you can express it with a type.
Vincent Warmerdam: You just say, this is a string,
Vincent Warmerdam: or it's the type that I'm actually interested in the pizza.
Vincent Warmerdam: And then the LLM has to figure out,
Vincent Warmerdam: is the response either going to be the string where I'm asking the user for missing information
Vincent Warmerdam: or is it going to be the pizza object and if it's a pizza object then the code will actually talk
Vincent Warmerdam: to the back end so you do get to rethink how typing works because we have llms at our disposal now you
Vincent Warmerdam: can actually have the llm handle some of that logic as long as your type is like strict about
Vincent Warmerdam: it or like well defined i should say um it's it's a bit of a weird advanced feature of types and llms
Vincent Warmerdam: but i can definitely imagine uh some of the pydantic ai stuff does have does use this trick in order
Vincent Warmerdam: to perform some of the higher order stuff and maybe you don't need a full framework as long as you are
Vincent Warmerdam: really um like deliberate about your types is what i that that's kind of the mentality i try to
Vincent Warmerdam: have right now like let's interesting let's not worry about like the big framework just yet if we
Vincent Warmerdam: can just focus on the right types and make sure that all that stuff is kind of sane then you also
Michael Kennedy: keep the extractions at bay which is i think also convenient especially early on it sounds a little
Michael Kennedy: bit like an analogy you might be able to draw with databases like you could have like a really strong
Michael Kennedy: structured Postgres or relational database where the database thing, aka the AI, is in charge of it.
Michael Kennedy: Or you could be using an ODM or ORM where the code will only accept responses, right? So like,
Michael Kennedy: for example, you can get away talking to a document database if you have strongly typed
Michael Kennedy: models. If you're just passing dictionaries, you're going to end up in a bad place real quick
Michael Kennedy: because something's not going to match, right? So it's kind of like the Pydanic thing is being a
Michael Kennedy: check for if it's not quite perfect. I don't know. I feel like there's an analogy there.
Vincent Warmerdam: There's a bit of an analogy there. The main thing with the LLM is like, oh, how much of the logic do
Vincent Warmerdam: you want to control yourself as a guarantee? And how much would you like the LLM to do it? And
Vincent Warmerdam: oh, that's always kind of a struggle. Like what is the right glue for that? And it turns out that
Vincent Warmerdam: Pydantic is in a very funny, unique position to say, how about we just use types? And that actually
Michael Kennedy: It just kind of works.
Michael Kennedy: How ironic is that that that's such a popular thing coming out of a effectively untyped language?
Vincent Warmerdam: Yeah.
Michael Kennedy: Maybe that's why it's so popular.
Michael Kennedy: I think it partly is.
Vincent Warmerdam: I mean, it's my impression that there's a lot of this stuff also happening on the JavaScript
Vincent Warmerdam: side of things, if I'm honest.
Vincent Warmerdam: Now, for the JavaScript side of things, the argument is, of course, oh, you can also build
Vincent Warmerdam: the web UIs a bit easier and the user experience and that whole story.
Vincent Warmerdam: But they also do a lot of this validation stuff.
Vincent Warmerdam: I forget the library, but there's also--
Vincent Warmerdam: I'm learning a little bit of JavaScript now.
Vincent Warmerdam: there's actually pydantic things happening in JavaScript land as well.
Michael Kennedy: Yeah, it's probably jidantic.
Vincent Warmerdam: Just-dantic.
Vincent Warmerdam: Just-dantic, yeah, j-s-dantic.
Vincent Warmerdam: Or TypeScript or whatever.
Vincent Warmerdam: But there's also like ORM libraries for JavaScript that are like type E,
Vincent Warmerdam: but not like super TypeScript just yet.
Michael Kennedy: Yeah, there's a lot of interesting analogies from the JavaScript space for us.
Michael Kennedy: You know, like the whole TypeScript team just rewrote everything in Go,
Michael Kennedy: whereas we're seeing a lot of stuff in Python being modified.
Michael Kennedy: Rust. Rust. Yeah, exactly. So pretty interesting. But let's, let's close this out a little bit with
Michael Kennedy: like running some of our own models. Okay. So clearly there's Anthropic, there's OpenAI,
Michael Kennedy: there's, you know, all the big, huge foundational models, but some of these things we're talking
Michael Kennedy: about, it's like this identity schema validation type of thing. And so on, you know, maybe we could,
Michael Kennedy: as you said, get away with running a smaller model, maybe small enough that we could run it
Michael Kennedy: ourselves on our servers or on my Mac mini here m2 pro 32 gigs ram i have the um the open ai
Michael Kennedy: 20 billion parameter model running as like my my default llm for any my code that i write that
Vincent Warmerdam: needs to talk to an llm so um yeah so we actually have a video on the maroon channel that one but
Vincent Warmerdam: go check that out later but basically i've recorded that link we'll put in the show notes yeah no so
Vincent Warmerdam: I kind of find myself exploring all these different open source LLMs like once every two months or so.
Vincent Warmerdam: And then Marimo has a couple of interesting integrations that makes it really easy to either hook into Olama or,
Vincent Warmerdam: and that's like the only thing I might change on the course that we made,
Vincent Warmerdam: there's now also the service called Open Router.
Vincent Warmerdam: Like I don't know if you've ever seen that one.
Michael Kennedy: Yes, I've heard of it.
Michael Kennedy: Because I was looking at Klein.bot.
Michael Kennedy: Are you familiar with Klein.bot?
Vincent Warmerdam: Oh, yeah.
Michael Kennedy: Klein is cool.
Michael Kennedy: Klein is really cool.
Vincent Warmerdam: I've been using that for a bit.
Vincent Warmerdam: Tell me what Klein is real quick.
Vincent Warmerdam: So Klein is basically Copilot for VS Code,
Vincent Warmerdam: but the one thing that they do
Vincent Warmerdam: is they really let you do any model,
Vincent Warmerdam: like just plug it in.
Vincent Warmerdam: And they are really strict about plan mode
Vincent Warmerdam: versus like run mode, if it makes sense.
Vincent Warmerdam: So you really got to,
Vincent Warmerdam: and one, the UI is kind of cool.
Vincent Warmerdam: So like every time that you're like planning
Vincent Warmerdam: or doing whatever,
Vincent Warmerdam: you kind of see the progress bar of your context.
Vincent Warmerdam: And then you also see how much money it costs.
Vincent Warmerdam: So you're like always aware of the fact that like,
Vincent Warmerdam: okay, this thing is costing me bucks now.
Vincent Warmerdam: And it's just a refreshing take on GitHub Copilot.
Vincent Warmerdam: And I will say, you can't do sessions in parallel,
Vincent Warmerdam: because it is, I think, at the moment, still stuck to the VS Code
Vincent Warmerdam: extension ecosystem.
Vincent Warmerdam: Could be wrong with that.
Michael Kennedy: They just introduced a CLI.
Vincent Warmerdam: Ah, there you go.
Michael Kennedy: There you go.
Michael Kennedy: I bet you can do it with that.
Vincent Warmerdam: Yeah, so that's also--
Vincent Warmerdam: the main thing that drew me in was the whole, oh,
Vincent Warmerdam: they're very strict about plan mode versus Go mode,
Vincent Warmerdam: which I think is definitely super useful.
Michael Kennedy: And on top of that, you have some nice UI.
Michael Kennedy: Yeah, it's also real similar to Cursor and others.
Michael Kennedy: And I think it's a big selling point.
Michael Kennedy: open source and they don't charge for inference so you just put in your api key and it just just
Michael Kennedy: passes through but you're right they do have like the as it's thinking it's got a little tick tick
Michael Kennedy: tick price of this opera and it's like oh it's up to three cents five you know it's just interesting
Vincent Warmerdam: to watch it go you also it's kind of like the fibonacci sequence you kind of realize that it's
Vincent Warmerdam: additive so it's like oh i've spent like half a dollar now oh and the next problem is going to be
Vincent Warmerdam: half a dollar plus extra the next one's going to be like that extra exactly it makes you manage your
Michael Kennedy: your context size real well.
Michael Kennedy: So anyway, OK, that's a bit of a diversion.
Michael Kennedy: But I think this is a fun tool to shout out.
Vincent Warmerdam: No, I agree.
Michael Kennedy: So the reason I brought this up is I
Michael Kennedy: saw they were number one on Open Router.
Michael Kennedy: I'm like, wait, what's Open Router?
Vincent Warmerdam: Yeah, so Open Router is basically you have one API key,
Vincent Warmerdam: and then they will route to any LLM that you like.
Vincent Warmerdam: And one thing that's really cool about it
Vincent Warmerdam: is that if there's a new open source model
Vincent Warmerdam: that you see all at Twitter and YouTube rave about,
Vincent Warmerdam: it'll be on Open Router.
Vincent Warmerdam: And then you can just give it a spin.
Vincent Warmerdam: If you have, let's say, stuff in your cache,
Vincent Warmerdam: and you've got a couple of functions that you use for evaluation,
Vincent Warmerdam: and you've got a setup for that,
Vincent Warmerdam: then it's just changing a single string to give a new LLM model a try.
Vincent Warmerdam: And they will make sure everything is nicely routed.
Vincent Warmerdam: And that's like the whole service also, basically.
Vincent Warmerdam: I think they add like a 5% fee hike or something like that,
Vincent Warmerdam: but they've got all sorts of GPU providers competing for the lowest price.
Vincent Warmerdam: So they do all the routing to all the LLM models,
Vincent Warmerdam: but also the open source ones.
Vincent Warmerdam: And that's the main convenient bit.
Vincent Warmerdam: Because I might not have-- so it's partially that I don't have the biggest GPU for the biggest models,
Vincent Warmerdam: but it's really convenient to just do kind of a for loop.
Vincent Warmerdam: Like, okay, take the 7B model from Qwenn, the 14B model, and just loop over all of them,
Vincent Warmerdam: and see if there's like a sweet spot as far as like quality is concerned and also model size.
Vincent Warmerdam: Like those kinds of experiments become a whole lot simpler if you just have one API that has all the models.
Vincent Warmerdam: And OpenRouter kind of gets you there.
Michael Kennedy: Yeah, okay, very neat.
Vincent Warmerdam: And it's also why Klein likes using it.
Vincent Warmerdam: So they have a direct connection with Klein.
Vincent Warmerdam: Just in the user interface, you can say,
Vincent Warmerdam: well, I want to have this model from Open Router.
Vincent Warmerdam: Just click, click, click, and it's all sort of ready to go.
Vincent Warmerdam: But that's their service.
Michael Kennedy: OK, very neat.
Michael Kennedy: So maybe for people who haven't run their own models,
Michael Kennedy: there's a couple of options.
Michael Kennedy: Sounds like you recommend Olama.
Vincent Warmerdam: I mean, that's the one that I have used.
Vincent Warmerdam: You've used LM Studio, I think?
Michael Kennedy: I use LM Studio because LM Studio
Michael Kennedy: has this UI for discovering and setting them up
Michael Kennedy: and playing and configuring them.
Michael Kennedy: But then you can just say, run in dev mode,
Michael Kennedy: and it just runs, and here we go,
Michael Kennedy: an open AI compatible--
Michael Kennedy: - There you go. - API endpoint
Michael Kennedy: on my local network.
Michael Kennedy: And so I can just point anything that can do ChatGPT,
Michael Kennedy: I can point it at this, whatever's running there.
Michael Kennedy: And I was running the GPT OSS 20 billion one
Michael Kennedy: last time I tried.
Michael Kennedy: - There you go.
Vincent Warmerdam: Okay.
Vincent Warmerdam: So Ollama also has like a very similar thing.
Vincent Warmerdam: My impression is that LM Studio in the end
Vincent Warmerdam: is like pretty similar,
Vincent Warmerdam: just has a slightly different UI.
Vincent Warmerdam: It's a bit more of an app, I suppose.
Vincent Warmerdam: Like Ollama is definitely more of a command line utility.
Vincent Warmerdam: Does the job though, I would say,
Vincent Warmerdam: but it doesn't have that much to offer in the UI department.
Michael Kennedy: Yeah, I think if I were ever to try to set it up on my server,
Michael Kennedy: I'd probably just do Olama, maybe inside a Docker or something.
Michael Kennedy: I don't know, something like that.
Vincent Warmerdam: Yeah, not true.
Vincent Warmerdam: The one sort of mysterious thing with a lot of these services,
Vincent Warmerdam: like how do they make money?
Vincent Warmerdam: And I think I read somewhere that Olama is going to start doing
Vincent Warmerdam: open router kind of stuff as well,
Vincent Warmerdam: so it wouldn't surprise me if they ended up going in that direction.
Vincent Warmerdam: But I mean, there's all sorts of vendors,
Vincent Warmerdam: and there's another thing I'm also seeing a lot of nowadays
Vincent Warmerdam: use like big models that generate a data set such that they can take a super tiny model and fine
Vincent Warmerdam: tune on the data that was generated by the big model and that way like a future maybe that we
Vincent Warmerdam: might have is that you have a dutch model that's really good for legal maybe we'll have like a
Vincent Warmerdam: super bespoke lm for that at some point or like a um like an lm that's like really good at css but
Vincent Warmerdam: only css like that's also a thing that we might have and then those models can be really tiny and
Vincent Warmerdam: run on your laptop and who knows this is the age we live in um but i certainly can foresee a world
Michael Kennedy: like that where it's like okay i'm gonna work on a project instead of spinning up a general world
Michael Kennedy: knowledge model i'm gonna just let it on demand go okay i have a css question well what framework
Michael Kennedy: are you using i'm using tailwind okay we're gonna spend up the tailwind llm model and we need to ask
Michael Kennedy: it a question yeah we're gonna shut that you know what i mean like really focus but then have some
Michael Kennedy: kind of orchestrating thing that says now i gotta talk to this now let me let me work with that one
Vincent Warmerdam: but that could be great yeah and it might also be a solution to the whole oh the time cut off is
Vincent Warmerdam: January to 2025.
Vincent Warmerdam: And like everything happened after we have no knowledge about.
Vincent Warmerdam: Like it could also be a way for us to have a system where we just retrain a thing weekly
Vincent Warmerdam: and it's just a diff on top or something like that.
Vincent Warmerdam: Right.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: It could also happen.
Michael Kennedy: Yeah.
Michael Kennedy: And if it were small enough, then it would be okay.
Vincent Warmerdam: Yeah.
Michael Kennedy: It would be kind of like SETI at home.
Michael Kennedy: All of our stuff would just be like kind of have the fan going.
Vincent Warmerdam: The, I mean, it's a bit of a science fiction thing, but you can imagine at some point every
Vincent Warmerdam: home will have solar panels and then maybe every home will also just have a GPU and maybe
Vincent Warmerdam: we won't have huge-ass server farms.
Vincent Warmerdam: It's just that we have a network, and everyone
Vincent Warmerdam: has a GPU, and if my neighbor suddenly
Vincent Warmerdam: wants to do math, then he can borrow my GPU.
Vincent Warmerdam: I don't know. We'll see.
Vincent Warmerdam: It kind of sounds cool, doesn't it?
Vincent Warmerdam: Yeah. If people like
Vincent Warmerdam: distributed systems,
Michael Kennedy: might as well go for it.
Michael Kennedy: If you want to add to the
Michael Kennedy: unpredictability of
Michael Kennedy: the LLM, mix in a distributed
Michael Kennedy: grid computing that you don't know about too much either,
Michael Kennedy: and then you'll get some real...
Vincent Warmerdam: I do want to give one quick shot
Vincent Warmerdam: on this one front, though, because I do think
Vincent Warmerdam: it's an idea that's worth pursuing.
Vincent Warmerdam: So in Holland, we've got this one cloud provider called
Vincent Warmerdam: LeafCloud, and I think they're still around.
Vincent Warmerdam: But what they do is they install these server farms,
Vincent Warmerdam: like what we're discussing.
Michael Kennedy: They have a GPU rack at the bottom of an apartment building.
Vincent Warmerdam: And any excess heat, which is one of the main cost drivers,
Vincent Warmerdam: that's actually used to preheat the water for the rest
Vincent Warmerdam: of the apartment.
Vincent Warmerdam: Yeah.
Vincent Warmerdam: So on paper, at least, you do have something that's not only
Vincent Warmerdam: carbon negative, but also more cost effective because the person, like the waste product of the
Vincent Warmerdam: of the compute is now a thing that the heat that someone is willing to pay for as well. So
Vincent Warmerdam: on paper, you can also be cheaper if you play your cards right.
Michael Kennedy: I ran across this and I was really inspired by it. I ended up not using LeafCloud for anything,
Michael Kennedy: but it's a cool idea.
Vincent Warmerdam: So the one downside of that approach is you do have to be close to a data center because
Vincent Warmerdam: do you really want to store your private data on disk in an apartment building or in a secure
Vincent Warmerdam: facility. So what they do is they just have glass fiber cables. Like the disc is in the facility,
Vincent Warmerdam: but the memory and the GPU and the computer is in the apartment building. That's sort of the way
Michael Kennedy: it's like a mounted volume over fiber, sort of like a network volume over fiber. Sure. I think
Michael Kennedy: that's pretty common. A lot of data center setups, it just happens to be spanning an apartment
Vincent Warmerdam: complex, which is unusual. Oh, it's also only for certain types of compute loads, right? You're
Vincent Warmerdam: probably not going to do your web app fire that apartment building. It's probably not going to be
Vincent Warmerdam: productive but if you're doing like big simulation big like deep learning things like that actually
Michael Kennedy: everyone everyone knows you can do machine training machine learning training and it's all good for
Michael Kennedy: serverless you just send the stuff out it just goes finds an apartment complex runs it serverless and
Michael Kennedy: it comes back right now just kidding so yeah it's a pretty neat idea yeah there's um we'll see what
Vincent Warmerdam: the future brings like when i did data science the joke i always made is like as a data scientist i
Vincent Warmerdam: I know for sure that the future is really hard to predict.
Vincent Warmerdam: Maybe the past is easier to predict,
Vincent Warmerdam: and that's what we do in machine learning.
Vincent Warmerdam: We try to predict the past, and yeah,
Vincent Warmerdam: we'll see what these LLMs do.
Vincent Warmerdam: But until then, again, also the repeating lesson from the course,
Vincent Warmerdam: there's some good habits, but in the end,
Vincent Warmerdam: it's not so much the LLM, it's more the stuff around
Vincent Warmerdam: and how you think about it.
Vincent Warmerdam: And also, small shout-out to Marimo.
Vincent Warmerdam: I do think having some UI in the blend as well,
Vincent Warmerdam: such as you can say, oh, there's a text box here,
Vincent Warmerdam: another text box there.
Vincent Warmerdam: you can kind of play around making your own little LLM app. Like the thing that I typically do is I
Vincent Warmerdam: hook up Gemini to my YouTube videos to make like summaries for my blog. I'd like try to find little
Vincent Warmerdam: excuses to like improve your life by automating something with these LLM tools. And that's a nice
Michael Kennedy: way to get started. Yeah, there's really, I think we all probably have little opportunities to automate
Michael Kennedy: some low stakes thing, but that we spend a lot of time with or hassle or whatever and let an LLM
Michael Kennedy: have at it and write a little Python to make that happen, right?
Vincent Warmerdam: Yeah, and also expose yourself to your own slop as well.
Vincent Warmerdam: The nice thing about building your own apps is also because you then suddenly start to realize when it's also still good to be a human.
Vincent Warmerdam: Like, I'm not going to have an LLM automate messages to my wife and my kid.
Vincent Warmerdam: That's just not going to happen because I've seen what these LLMs can output.
Vincent Warmerdam: Like, that's going to remain on the human side of the spectrum.
Vincent Warmerdam: Exactly.
Vincent Warmerdam: But making a summary for my blog on a video, I mean, that feels in the ballpark.
Michael Kennedy: 100%.
Michael Kennedy: All right.
Michael Kennedy: Final word for people interested in this topic they want to build.
Michael Kennedy: I want to plug in LLMs as part of that building block.
Michael Kennedy: What do you say?
Vincent Warmerdam: Open Router is nice and flexible.
Vincent Warmerdam: LLM by Simon Willison is great.
Vincent Warmerdam: Disk cache is a thing that you do want to disk cache.
Vincent Warmerdam: Maybe a final thing that I do maybe want to...
Vincent Warmerdam: It's also fine if you just want to learn Python as well.
Vincent Warmerdam: There's some of this talk of people saying,
Vincent Warmerdam: oh, we never have to...
Vincent Warmerdam: Oh, actually, people from Peru are tuning in.
Vincent Warmerdam: Peru is amazing.
Vincent Warmerdam: from seeing people sort of replying the conversation.
Vincent Warmerdam: Yeah, absolutely.
Vincent Warmerdam: Oh, sweet.
Vincent Warmerdam: Best food of my life, Peru.
Vincent Warmerdam: Also, if you're looking for great excuses to learn Python,
Vincent Warmerdam: there's plenty.
Vincent Warmerdam: If you're looking for great excuses to go to Peru, food.
Vincent Warmerdam: But also, I think in the end,
Vincent Warmerdam: you are going to be better at using LLMs
Vincent Warmerdam: if you can still do things yourself.
Vincent Warmerdam: One thing I've started noticing is that
Vincent Warmerdam: it's a bit embarrassing, but like,
Vincent Warmerdam: oh, what's the CSS property of that thing?
Vincent Warmerdam: I used to know that.
Vincent Warmerdam: But nowadays, I just, oh,
Vincent Warmerdam: am I seriously going to ask this, the ChatGPT?
Vincent Warmerdam: This should be in my memory for Pete's sakes.
Vincent Warmerdam: Yes.
Vincent Warmerdam: And sometimes it's like, I'm going
Vincent Warmerdam: to have a whole conversation with ChatGPT
Vincent Warmerdam: so I can change four characters in a style sheet.
Vincent Warmerdam: What is this nonsense?
Vincent Warmerdam: So part of me is also thinking like, definitely
Vincent Warmerdam: expose yourself to LLMs.
Vincent Warmerdam: And if that inspires you, that's great.
Vincent Warmerdam: But also try to not overly rely on it either.
Vincent Warmerdam: It's like I'm building flashcard apps for myself.
Vincent Warmerdam: So I'm still kind of in the loop, if that makes sense.
Michael Kennedy: So you've got to be a guardian of your own mind.
Michael Kennedy: And yes, so I can see it so easily.
Michael Kennedy: that you just become a machine that just says next okay yes continue and then in six months you're
Michael Kennedy: like gosh i can't program anything i just got to ask the chat and it's it's a problem yeah it's
Vincent Warmerdam: it's something well and also um and that kind of gets back to the point i made earlier um you want
Vincent Warmerdam: to be very mindful that if you if you have better tools you should also have better ideas and if
Vincent Warmerdam: that's not the case then something is wrong because that's then you get into the self-learned
Vincent Warmerdam: helplessness territory. And that's the one thing I do want to kind of guard against.
Vincent Warmerdam: Andre Carpathy gave a great interview a little while ago where he mentioned the
Vincent Warmerdam: worst outcome of humanity. Did you see the movie WALL-E? Remember seeing that?
Vincent Warmerdam: Oh my gosh. That movie really made me sad.
Michael Kennedy: It's a good movie though. It's a fantastic movie, but as an adult,
Michael Kennedy: it's got a hard undertone to it. Oh my gosh.
Vincent Warmerdam: And especially now, because for people who never watched the movie, there's a scene in the movie where you see what humanity
Vincent Warmerdam: comes to and it's basically every everything has to be convenient so people are overweight
Vincent Warmerdam: they're in these like bikes that move them around they're always able to watch television drink a
Vincent Warmerdam: milkshake and eat a pizza like every convenience is sort of taken care of but you also just see
Michael Kennedy: that they're not happy and they can't even walk out like they can't they're like just they're so uh
Michael Kennedy: just dependent on the thing it's bad exactly so that's a sweet show it's a very sweet show i
Vincent Warmerdam: recommend it yeah it's it's a sweet movie but it's like if at some point you notice like hey
Vincent Warmerdam: learning feels hard, then maybe that's fine because maybe it's just like exercising.
Vincent Warmerdam: It's okay if that's a bit painful.
Vincent Warmerdam: In fact, if it's not painful, then maybe just participating in entertainment instead as well.
Vincent Warmerdam: So that's the thing.
Vincent Warmerdam: These are thoughts that I'm sort of battling with.
Vincent Warmerdam: I'm also sort of trying to be really critical of like, okay, I work at Marimo now.
Vincent Warmerdam: I've accessed all these LLMs and stuff.
Vincent Warmerdam: And can I build more awesome software that I wouldn't have before?
Vincent Warmerdam: One thing I made a while ago is I added gamepad support to Marimo Notebooks.
Vincent Warmerdam: So you can interact with Python with the gamepad and stuff.
Vincent Warmerdam: So this is just a experiment.
Vincent Warmerdam: But I would be really disappointed if I'm the only person making packages that I've never made before that are definitely trying to reach new heights.
Vincent Warmerdam: It does feel a bit lonely because it does feel like more people should do it.
Vincent Warmerdam: That's like the one sort of hint I want to give to people.
Vincent Warmerdam: Try to inspire yourself a bit more and do more inspirational stuff so more cool things happen on my timeline.
Vincent Warmerdam: There's less AI sloth.
Vincent Warmerdam: There's really cool software being built, people learning and all.
Michael Kennedy: Hear, hear.
Michael Kennedy: I appreciate that.
Michael Kennedy: All right.
Michael Kennedy: Well, Vincent, thanks for being back,
Michael Kennedy: sharing all these techniques.
Michael Kennedy: And yeah, I hope people go out there
Michael Kennedy: and build using AIs, not just with AIs.
Michael Kennedy: Definitely.
Michael Kennedy: Y'all have a good one.
Michael Kennedy: Yeah, see you later.
Michael Kennedy: Bye.
Michael Kennedy: This has been another episode of Talk Python To Me.
Michael Kennedy: Thank you to our sponsors.
Michael Kennedy: Be sure to check out what they're offering.
Michael Kennedy: It really helps support the show.
Michael Kennedy: Take some stress out of your life.
Michael Kennedy: Get notified immediately about errors
Michael Kennedy: and performance issues in your web
Michael Kennedy: or mobile applications with Sentry.
Michael Kennedy: Just visit talkpython.fm/sentry
Michael Kennedy: and get started for free.
Michael Kennedy: And be sure to use the promo code Talk Python, all one word.
Michael Kennedy: And it's brought to you by Nordstellar.
Michael Kennedy: Nordstellar is a threat exposure management platform
Michael Kennedy: from the Nord security family,
Michael Kennedy: the folks behind NordVPN
Michael Kennedy: that combines dark web intelligence,
Michael Kennedy: session hijacking prevention,
Michael Kennedy: brand and domain abuse detection,
Michael Kennedy: and external attack surface management.
Michael Kennedy: Learn more and get started keeping your team safe
Michael Kennedy: at talkpython.fm/Nordstellar.
Michael Kennedy: If you or your team needs to learn Python, we have over 270 hours of beginner and advanced courses on topics ranging from complete beginners to async code, Flask, Django, HTML, and even LLMs.
Michael Kennedy: Best of all, there's not a subscription in sight. Browse the catalog at talkpython.fm.
Michael Kennedy: Be sure to subscribe to the show. Open your favorite podcast player app. Search for Python. We should be right at the top.
Michael Kennedy: If you enjoy the geeky rap theme song, you can download the full track.
Michael Kennedy: The link is your podcast player show notes.
Michael Kennedy: This is your host, Michael Kennedy.
Michael Kennedy: Thank you so much for listening.
Michael Kennedy: I really appreciate it.
Michael Kennedy: Now get out there and write some Python code.
Michael Kennedy: Talk Python To Me, and we ready to roll
Vincent Warmerdam: Upgrading the code, no fear of getting old
Vincent Warmerdam: We tapped into that modern vibe, overcame each storm
Vincent Warmerdam: Talk Python To Me, I-Sync is the norm
Vincent Warmerdam: you