WEBVTT 00:00:00.001 --> 00:00:03.080 Have you ever wondered why certain data points stand out so dramatically? 00:00:03.650 --> 00:00:07.640 They might hold the key to everything from fraud detection to groundbreaking discoveries. 00:00:08.200 --> 00:00:12.800 This week on Talk Python To Me, we dive into the world of outlier detection with Brett Kennedy. 00:00:13.540 --> 00:00:20.680 You'll learn how outliers can signal errors, highlight novel insights, or even reveal hidden patterns lurking in the data you thought you understood. 00:00:21.440 --> 00:00:30.700 We'll explore fresh research developments, practical use cases, and how outlier detection compares to other core data science tasks like prediction and clustering. 00:00:31.160 --> 00:00:35.560 If you're ready to spot those game-changing anomalies in your own projects, stay tuned. 00:00:36.280 --> 00:00:42.200 This is "Talk Python To Me," episode 497, recorded January 21st, 2025. 00:00:43.860 --> 00:00:45.640 - Are you ready for your host, please? 00:00:46.280 --> 00:00:52.740 - You're listening to Michael Kennedy on "Talk Python To Me." Live from Portland, Oregon, and this segment was made with Python. 00:00:55.960 --> 00:01:09.440 - Welcome to "Talk Python To Me," a weekly podcast on Python. This is your host, Michael Kennedy. Follow me on mastodon where I'm @mkennedy and follow the podcast using @talkpython, both accounts over at fosstodon.org. 00:01:09.860 --> 00:01:18.980 And keep up with the show and listen to over nine years of episodes at talkpython.fm. If you want to be part of our live episodes, you can find the live streams over on YouTube. 00:01:19.400 --> 00:01:29.580 Subscribe to our YouTube channel over at talkpython.fm/youtube and get notified about upcoming shows. This by Posit Connect from the makers of Shiny. 00:01:30.000 --> 00:01:34.120 Publish, share and deploy all of your data projects that you're creating using Python. 00:01:34.619 --> 00:01:40.760 Streamlit, Dash, Shiny, Bokeh, FastAPI, Flask, Quarto, Reports, Dashboards and APIs. 00:01:41.580 --> 00:01:43.180 Posit Connect supports all of them. 00:01:43.440 --> 00:01:48.800 Try Posit Connect for free by going to talkpython.fm/posit, P-O-S-I-T. 00:01:49.600 --> 00:01:51.600 Brett, welcome to Talk Python To Me. 00:01:51.780 --> 00:01:53.280 - Well, thank you very much for having me. 00:01:53.420 --> 00:01:54.400 - It's awesome to have you here. 00:01:54.920 --> 00:01:56.240 A pair of Kennedys on the show. 00:01:56.660 --> 00:01:58.200 - First time ever, that's pretty awesome. 00:01:58.420 --> 00:01:59.860 - Yep, well it should be much more common. 00:02:00.780 --> 00:02:03.380 - Indeed, well thank you for coming on the show. 00:02:03.540 --> 00:02:08.920 I'm really interested to talk some data science and mathy type things with Python. 00:02:09.520 --> 00:02:16.160 Python is being used for data science and analysis and just general science and mathematics so much these days. 00:02:16.440 --> 00:02:26.980 I wonder if people have maybe a perception out there of thinking, well Python is kind of like, yeah people use that for the web and stuff and true, but these days it's data science, machine learning, those things. 00:02:27.220 --> 00:02:28.960 Other languages are used a fair amount. 00:02:29.400 --> 00:02:31.400 Statistical analysis are still used a lot. 00:02:32.240 --> 00:02:46.580 And for high performance, machine learning, people will look at other languages, C++ is used a lot and like Mojo and things are coming, coming through now that are, but still a very, very large percent is done in Python. 00:02:47.340 --> 00:02:53.480 Like all the work we're seeing with large language models, for example, like that's pretty much all, all Python. 00:02:53.860 --> 00:02:57.580 And a lot of standard traditional machine learning is done in 00:02:57.580 --> 00:02:59.060 a 00:02:59.060 --> 00:02:59.980 very large portion of 00:02:59.980 --> 00:03:00.700 it's done in Python. 00:03:00.820 --> 00:03:00.940 Yeah. 00:03:01.160 --> 00:03:01.280 Yeah. 00:03:01.720 --> 00:03:03.440 And outlier detection as well. 00:03:03.600 --> 00:03:03.700 Yeah. 00:03:03.780 --> 00:03:18.040 I think Python is probably, I mean, arguably, but I think probably the richest set of tools for outlier detection of any language, again, R has quite a good suite and, and you get good, good tools in Java and in some other places. 00:03:18.180 --> 00:03:29.960 But Python is probably where you find the strongest, just the sheer number of And also it covers a lot of the deep learning techniques and some kind of approaches to outlier detection 00:03:30.440 --> 00:03:32.380 aren't really as well covered in other languages. 00:03:32.700 --> 00:03:33.980 - Well, we're gonna dive into it. 00:03:34.060 --> 00:03:37.120 You've written a whole book on it and a bunch of cool techniques you pull down. 00:03:37.200 --> 00:03:41.560 We're talking about some libraries and things like that, as well as some other examples. 00:03:42.040 --> 00:03:46.840 But before we get into that, let's just hear a quick introduction about who you are and what you do these days. 00:03:47.120 --> 00:03:48.800 - Yes, well, my name is Brett Kennedy. 00:03:49.180 --> 00:03:53.100 So I've worked in software in one capacity or another for about 30 years now. 00:03:53.400 --> 00:03:58.200 I did my master's degree quite a long time ago at the University of Toronto. 00:03:58.700 --> 00:04:01.900 And that was actually, well, I was around '98, '99. 00:04:02.120 --> 00:04:05.100 So during one of the times when AI was being a bit of a winter. 00:04:05.400 --> 00:04:07.480 Yeah, that was a very, very good experience going there. 00:04:08.140 --> 00:04:14.060 And yeah, I've worked for one company doing software that was used for financial auditing. 00:04:14.260 --> 00:04:17.480 So and that's where I really started to get into data science. 00:04:17.579 --> 00:04:19.299 And that was about 15 years ago. 00:04:19.640 --> 00:04:29.220 of what is a little bit not as mainstream as it is today. I mean, there's certainly a lot of other people doing it, but it wasn't the gigantic industry that it is now. 00:04:29.380 --> 00:04:30.040 >> Yeah, there was definitely 00:04:30.040 --> 00:04:50.920 some inflection points or places where it's really taken some steps, significant steps up in the Python data science and ML space. Probably one is the Pandas NumPy era around 2006, something around 2012. I don't know really what caused And certainly in the machine learning, AI is a whole nother spike of growth there. 00:04:50.930 --> 00:04:54.560 So you were doing, it was kind of before mainstream data science, right? 00:04:54.700 --> 00:05:05.160 It is starting to become mainstream, but like when working with text, for example, there's certain, there were spaCy and word2vec and glove and things like that. 00:05:05.280 --> 00:05:11.680 But this is long before GPT, but scikit-learn existed and yeah. 00:05:11.860 --> 00:05:17.740 So we're actually using a lot of the same tools that you would, you would use now, but they're a little bit earlier versions of them. 00:05:17.820 --> 00:05:18.600 They're, yeah, sure. 00:05:18.780 --> 00:05:18.940 Yeah. 00:05:19.090 --> 00:05:22.220 So things have, have, have progressed certainly a lot. 00:05:22.560 --> 00:05:27.060 Though some things just, just work and have, have been, been fairly stable. 00:05:27.090 --> 00:05:27.980 In that space. 00:05:28.080 --> 00:05:30.020 I imagine a lot of the work 00:05:30.020 --> 00:05:34.640 that's been done on the faster CPython initiative is probably paying benefits. 00:05:34.820 --> 00:05:35.020 Yeah. 00:05:35.030 --> 00:05:44.780 I mean, one of the things about Python is it, it is, does have a reputation as being not, not the fastest language you can use, you know, old versions of Python were certainly slower. 00:05:44.840 --> 00:05:46.120 It's, it is getting faster. 00:05:46.380 --> 00:05:46.520 Yeah. 00:05:46.620 --> 00:05:56.100 C++ is probably about as performant as you can get in it, but you know, a lot of like scikit-learn for example, is, uses Cython so you get C code. 00:05:56.560 --> 00:05:56.900 Characteristics. 00:05:56.980 --> 00:05:57.080 Yeah. 00:05:57.240 --> 00:05:57.500 Yeah. 00:05:57.600 --> 00:05:59.500 It also goes back to what are you measuring? 00:05:59.900 --> 00:06:01.260 What is your metric for fast? 00:06:01.380 --> 00:06:01.560 Yeah. 00:06:01.960 --> 00:06:02.820 What are your expectations? 00:06:03.120 --> 00:06:10.860 And yeah, I could make a really fast thing in two weeks in C++ or I could make a somewhat slower thing, but get you the answer in half a day in Python. 00:06:11.560 --> 00:06:14.020 It might run five times longer, but you'll have the answer today. 00:06:14.050 --> 00:06:14.500 You know what I mean? 00:06:14.520 --> 00:06:15.320 like that kind of thing. 00:06:15.580 --> 00:06:16.380 There's really a trade off. 00:06:16.440 --> 00:06:26.980 And I think, I mean, it's kind of the, well, it's, it's really the approach we've taken in data sciences to really privilege the data scientists time over the computation time. 00:06:27.040 --> 00:06:27.180 Yeah. 00:06:27.400 --> 00:06:27.540 Yeah. 00:06:27.800 --> 00:06:31.280 And when it really does become a problem, there's a lot of stuff written in C or Rust or whatever. 00:06:32.160 --> 00:06:33.400 Polars, pandas, et cetera. 00:06:33.620 --> 00:06:33.900 Yeah. 00:06:34.040 --> 00:06:35.660 That, you know, Polars is a big breakthrough 00:06:35.660 --> 00:06:36.000 too. 00:06:36.060 --> 00:06:36.240 Yeah. 00:06:36.280 --> 00:06:37.060 Are you a fan of Polars? 00:06:37.120 --> 00:06:37.860 They've been using any? 00:06:38.060 --> 00:06:39.880 I use it when, when there's a need. 00:06:39.980 --> 00:06:43.200 I mean, I'm still a big fan of pandas and I still use it quite a lot. 00:06:43.440 --> 00:06:47.620 But when there's a performance issue, yeah, Polars is one of the things I'll turn to. 00:06:47.670 --> 00:06:50.740 >> Yeah, excellent. So we're going to talk about outlier detection. 00:06:51.460 --> 00:06:58.680 I think maybe a good way to introduce this topic is to maybe understand the problem space just a tiny bit. 00:06:59.040 --> 00:07:08.200 Because sometimes you might think, well, if the mean is really weird or the median is really weird or something like that, then we might have an outlier and we can go check it out. 00:07:08.300 --> 00:07:10.040 But it's really a lot more subtle than that. 00:07:10.080 --> 00:07:21.800 And I want to maybe go back to a guest I had, was that six months ago or something like that, Stephanie Molin, and she has this really cool project called Datamorph, which I'll link to on GitHub, and we talked about it then. 00:07:21.920 --> 00:07:37.980 But if you pull up her GitHub repo, it's an animated GIF, I believe, of a whole bunch of different, very clear concrete shapes, like a star, literally a panda, not pandas, but the animal, and continuous animation, a bunch of data points that go from one to the other. 00:07:38.360 --> 00:07:39.900 And it shows the mean over time. 00:07:40.200 --> 00:07:40.740 It's the same. 00:07:40.840 --> 00:07:42.680 The standard deviation is the same. 00:07:42.680 --> 00:07:47.160 The correlation, the same effectively do like four to the thousandth sort of thing. 00:07:47.580 --> 00:07:55.020 And it just shows you this is not as simple as finding stuff that's just off the that's linear interpolation line or something like that, right? 00:07:55.140 --> 00:07:55.280 Yeah. 00:07:55.360 --> 00:07:56.060 It's a good example. 00:07:56.200 --> 00:08:03.500 We're using a single or a handful of metrics to evaluate a data set can be a bit oversimplistic. 00:08:04.160 --> 00:08:06.180 You can miss a lot of nuance when you do that. 00:08:06.740 --> 00:08:19.540 Yeah, and you do definitely get into the same sort of issues when you're looking at outliers in a lot of areas of data science where you can kind of be misled by looking at metrics that maybe are relevant, but aren't just don't tell the tell the full story. 00:08:19.820 --> 00:08:22.000 It's one view, but it maybe it's not. 00:08:22.280 --> 00:08:22.940 It doesn't for sure. 00:08:22.980 --> 00:08:24.120 Like you said, doesn't tell the whole story. 00:08:24.480 --> 00:08:30.200 So let's talk about what is this outlier detection stuff and really why 00:08:30.200 --> 00:08:33.800 do people care what kind of problems are being solved with it and so on. 00:08:34.260 --> 00:08:50.240 When people start to learn machine learning, one of the early things you'll learn is there's supervised machine learning and there's unsupervised machine learning. And when you get into unsupervised machine learning, the two bigger areas of it are probably clustering and outlier detection. 00:08:50.620 --> 00:09:06.420 And sometimes dimensionality reduction is kind of considered unsupervised as well. It's kind of like there's a whole host of reasons you might want to do clustering. It's a little but like that is that it's just one of those things that turns up time and again, when you're doing machine learning. 00:09:06.740 --> 00:09:14.520 Outlier detection can be done for, for example, if you have a data set and you just want to say, you just want to try and understand the data. 00:09:15.100 --> 00:09:19.780 One hand you can say, well, if you want to understand the data, there's really kind of two main parts of that. 00:09:19.860 --> 00:09:25.320 There's what are trying to figure out what are the general patterns in the data and what are the exceptions to those? 00:09:25.700 --> 00:09:33.340 And once you have both of those, you say, well, here's the patterns and Here's the extent to which those patterns hold true in the data, which is the outliers. 00:09:34.220 --> 00:09:37.040 That kind of gives you a good sense of what the data is about. 00:09:37.580 --> 00:09:41.560 Often when you're checking for data quality, outlier tests are very useful. 00:09:42.000 --> 00:09:51.680 It gives you, like if you have say a table of whatever it is, credit card transactions or whatever, you can do this for image data or audio data or data of really any type. 00:09:51.740 --> 00:09:54.600 But let's take a simple example of tabular data. 00:09:55.030 --> 00:10:00.600 You might have a spreadsheet with a few million rows of credit card transactions or SQL table. 00:10:00.960 --> 00:10:11.960 If you want to check it for errors before you go on to do other sort of analysis out of the data mining or building predictive models or the like, one thing we often do is try and see, well, what kind of data quality am I dealing with here? 00:10:12.060 --> 00:10:21.240 And there's a number of ways to do that, but one of the steps you might often perform is say, we'll do outlier detection and see what are the strangest rows in my data. 00:10:21.380 --> 00:10:35.660 And if you look at those, you see the strangest credit card transactions in there, Or if you could be a little more involved in analysis, you might look at the most unusual string of transactions for a given user or a given store or something like that. 00:10:36.140 --> 00:10:45.240 And if you say, well, okay, these are the weirdest ones and they're only this weird, then we can have some sort of sense of what sort of range of anomalousness might be in your data. 00:10:45.620 --> 00:10:49.040 Yes, it's used, alert detection is used a lot as well in security. 00:10:49.600 --> 00:10:52.780 We're gonna be used like checking for fraud, checking for errors. 00:10:53.160 --> 00:10:56.140 I mean, it's used in scientific analysis a lot. 00:10:56.700 --> 00:11:04.020 It's very common for scientific projects for just to collect data through sensors or hand collecting data or whatever. 00:11:04.300 --> 00:11:06.680 In the case of astronomy, it's through telescopes. 00:11:07.160 --> 00:11:19.660 Astronomy is a good example where we collect such phenomenal volumes of data that if you want to find anything interesting in there, we used to be able to find interesting phenomena in astronomy data just by manually going through it. 00:11:21.500 --> 00:11:52.980 example given the book is that's how pulsars were discovered. The low-hanging fruit's gone, probably. To really find something really, really interesting, given that we might collect petabytes of data per night, really the most effective way to do it is just go through an anomaly, run an anomaly detection on the data and say, "What are the strangest things in this set of data?" And those are odds are there was going to be the most interesting there that maybe things we've never seen before or haven't seen a whole lot. At least some examples. 00:11:53.260 --> 00:11:53.380 - Yeah. 00:11:53.420 --> 00:11:56.140 - Testing for online video games as well as... 00:11:56.160 --> 00:12:00.040 - When I talked to Sarah Eisen about imaging a black, the black hole, the first 00:12:00.040 --> 00:12:20.380 black hole image with Python, she talked about shipping, actually shipping hard drives around to sync up the data because there was so much data that it was faster to put it onto a plane and send it from all the different locations around the world. Like it literally couldn't keep up the internet. And so box them up into these big crates and ship them around. 00:12:21.080 --> 00:12:24.640 When you're talking that much data, you gotta get a plane involved, it's pretty wild. 00:12:24.780 --> 00:12:29.960 - We weren't doing anomaly detection at the time, but yeah, one of the companies I've worked with, just dealing with financial data, we 00:12:29.960 --> 00:12:32.980 actually collected so much, it's actually faster to mail it. 00:12:33.120 --> 00:12:34.820 - Really, you ended up doing that as well? 00:12:34.900 --> 00:12:35.620 - At times, yeah. 00:12:35.840 --> 00:12:38.960 - Yeah, how much of a hard drive would you put-- - Oh, gosh. 00:12:39.080 --> 00:12:39.620 - In the mail? 00:12:39.640 --> 00:12:43.380 I mean, it wouldn't be just a terabyte, it would have to be a pretty big hard drive, or set of hard drives, right? 00:12:43.480 --> 00:12:47.280 - Well, part of the explanation of why we're doing this was this was about 15 years ago. 00:12:47.580 --> 00:12:50.080 - I see, a terabyte sounds like a whole lot. 00:12:50.340 --> 00:12:57.260 - Yeah, I think the internet we were paying for at the time was, you know, having sort of speeds we have with the internet now, the costs were prohibitive back 00:12:57.260 --> 00:12:57.460 then. 00:12:57.560 --> 00:13:05.860 - I have gigabit up and down at home for 75 bucks, which you could have sent it over that probably, but it's not quite back when the internet had a sound. 00:13:06.130 --> 00:13:07.860 Do you remember when the internet had a sound? 00:13:08.200 --> 00:13:11.020 You would connect to it and you would be like, okay, that's a good connection. 00:13:11.120 --> 00:13:13.020 We're gonna have some stuff happening today. 00:13:13.700 --> 00:13:18.740 That's early 90s when the internet stopped having a sound, mid 90s, it stopped having a sound. 00:13:18.860 --> 00:13:18.960 >> 00:13:18.960 --> 00:13:23.580 That's about when I first started getting a little, just as it's being opened up. 00:13:24.779 --> 00:13:27.820 Remember the early days, it's basically university students that were on there. 00:13:27.840 --> 00:13:30.260 >> What I would do when I was in high school is, 00:13:31.419 --> 00:13:41.340 my brother and I, we would call the local university and they had a library dial-up for dial-up modems, and then you get on to Telnet and go for an Archie. 00:13:42.080 --> 00:13:47.660 This is before the World Wide Web existed, before Mosaic, it was, for people that know I'm talking about. 00:13:47.660 --> 00:13:48.060 >> TP. 00:13:48.240 --> 00:13:48.900 >> Yeah, exactly. 00:13:49.440 --> 00:13:50.520 >> Telnet, yes. 00:13:50.700 --> 00:13:56.000 >> Unencrypted FTP. By the way, we had so much trust. We were so naive. 00:13:56.140 --> 00:13:56.560 >> Yeah. 00:13:56.780 --> 00:13:59.840 >> But yeah, when you would connect, it would go beep, 00:14:00.060 --> 00:14:00.140 beep, 00:14:00.240 --> 00:14:00.780 beep, beep, 00:14:01.300 --> 00:14:01.620 [NOISE] 00:14:01.840 --> 00:14:08.080 and depending on the sound, you would get to where you would go, "Oh, that means I connect to this bit rate or that bit rate," right? It's so wild. 00:14:08.120 --> 00:14:14.200 >> Yeah, those are >> I'm nostalgic for it and I want nothing to do with it. I'm so happy we are where we are. >> Yeah, there's a certain charm to it. 00:14:14.200 --> 00:14:16.220 But if you went back to it, you would hate it 00:14:16.220 --> 00:14:17.780 because 00:14:17.780 --> 00:14:19.440 you would watch the images load. 00:14:19.680 --> 00:14:22.280 You would almost see the progressive scan lines come in. 00:14:22.340 --> 00:14:24.120 I mean, it was really, it wasn't ideal. 00:14:24.500 --> 00:14:28.440 No, I had a roommate once who used to download images. 00:14:28.540 --> 00:14:35.740 These are just JPEGs, the same sort of files we have today, but it would take literally overnight to download a 00:14:35.740 --> 00:14:36.220 JPEG. 00:14:36.360 --> 00:14:39.440 They were interesting times, but we've certainly moved on. 00:14:39.540 --> 00:14:40.680 So that's all good. 00:14:41.800 --> 00:14:45.020 This portion of Talk Python To Me is brought to you by the folks at Posit. 00:14:45.540 --> 00:14:54.000 Posit has made a huge investment in the Python community lately. Known originally for RStudio, they've been building out a suite of tools and services for Team Python. 00:14:54.940 --> 00:14:59.900 Today, I want to tell you about a new way to share your data science assets, Posit Connect Cloud. 00:15:00.820 --> 00:15:22.140 Posit Connect Cloud is an online platform that simplifies the deployment of data applications and documents. It might be the simplest way to share your Python content. Here's how it works in three easy steps. One, push your Python code to a public or private GitHub repo. Two, tell Posit Connect Cloud which repo contains your source code. Three, click deploy. That's it. 00:15:22.580 --> 00:15:33.480 Posit Connect Cloud will clone your code, build your asset, and host it online at a URL for you to share. Best of all, Posit Connect Cloud will update your app as you push code changes to GitHub. 00:15:34.160 --> 00:15:46.600 If you've dreamed of Git-based continuous deployment for your projects, Posit Connect Cloud is here to deliver. Any GitHub user can create a free Posit Connect Cloud account. You don't even need a special trial to see if it's a good fit. 00:15:47.020 --> 00:16:03.940 So if you need a fast, lightweight way to share your data science content, try Posit Connect Cloud. And as we've talked about before, if you need these features, but on-prem, check out Posit Connect. Visit talkpython.fm/connect-cloud, See if it's a good fit. 00:16:04.440 --> 00:16:07.340 That's talkpython.fm/connect-cloud. 00:16:07.510 --> 00:16:09.560 The link is in your podcast players show notes. 00:16:10.120 --> 00:16:12.160 Thank you to Posit for supporting Talk Python To Me. 00:16:13.920 --> 00:16:22.020 You talked about some examples where you're looking for outliers for fraud, like credit card, and you talked about mailing around the financial data for those kinds of things. 00:16:22.360 --> 00:16:29.080 It seems to me that outliers might also reveal new science or new information that you didn't expect. 00:16:29.150 --> 00:16:41.400 Like it's an outlier because it doesn't match our model, but something is happening over here. And I think you pointed out astronomy, right? Like that part is really bizarrely bright. Why is that so bright? Is that a, you know, oh, that's a quasar. That's what that is. 00:16:41.520 --> 00:16:44.760 - Brightness you're not expecting or a shape of a galaxy you're not expecting, that sort of thing. 00:16:44.800 --> 00:16:50.040 - That nebula turns out to be a galaxy. We thought there was only one galaxy. Now there's many, right? That kind of thing. 00:16:50.140 --> 00:16:59.220 - It's definitely a case, I think science is definitely a good example where outliers is something you want to find. Yeah, it's like biological data. It's used a lot too. 00:16:59.400 --> 00:17:11.579 Well, I guess a lot of areas, the CERN particle accelerator at CERN uses outlier detection a lot just to find interesting things and to check that things are within certain bounds of what they expect or not. 00:17:12.280 --> 00:17:38.840 But yeah, I mean, when you're dealing with certain domains, like one example that's kind of maybe the other end of the spectrum would be when you have, say you're monitoring industrial systems, like an assembly line or something like this, that's an environment where you really want something, we want the whole system to work in a specific way. And any kind of veering from that is probably a problem or indications of a potential problem down the road. 00:17:39.000 --> 00:17:48.660 Right. You've got some IoT thing or you've got a more industrial type of robot and all of a sudden it's drilling holes outside of where you used to have it drill holes. You probably want it to stop. 00:17:48.700 --> 00:17:48.800 It's 00:17:48.800 --> 00:17:59.820 behavior is unusual or just the sensors picking up temperatures that are unusual or vibrations or sound levels that are higher than normal or oscillating faster than normal or something like that. 00:17:59.960 --> 00:18:00.520 Yeah, yeah, yeah. 00:18:00.620 --> 00:18:05.140 Quite often, yeah, an outlier can be something interesting or it can be a problem. 00:18:05.640 --> 00:18:06.840 Yeah, it really depends on the context. 00:18:06.920 --> 00:18:18.260 But in most cases, there's something that's, for one reason or other, it's common for them to be worth investigating for one, yeah, to expand your knowledge of the system or just to find fared out problems. 00:18:18.480 --> 00:18:19.300 - Sure, what's the 00:18:19.300 --> 00:18:22.920 scale of data that outlier detection makes sense for? 00:18:23.270 --> 00:18:26.120 If I've got 20 measurements, does it make sense to ask this question? 00:18:26.440 --> 00:18:30.080 If I've got 200 billion, is that too much to ask the question for? 00:18:30.220 --> 00:18:31.500 Like, what's the story there? 00:18:31.680 --> 00:18:32.340 - That's a good question. 00:18:32.460 --> 00:18:40.940 I mean, when you have a very, I mean, it's like a lot of areas that could be, I guess the same thing with clustering or predictive models or generative models, anything along those lines. 00:18:40.950 --> 00:18:42.680 You have too little data or you can have too much. 00:18:42.740 --> 00:18:48.740 And there is kind of a sweet spot where the results are statistically stable, but it doesn't take you forever to find 00:18:48.740 --> 00:18:48.900 them. 00:18:49.060 --> 00:18:51.360 - They're statistically stable, but they can be computed. 00:18:51.580 --> 00:18:53.920 - Yeah, it's not a lot of problems. 00:18:54.360 --> 00:19:02.560 Like if you have, say you're checking a large company and you're just checking your accounting data for anomalies, there can be a lot of value to it. 00:19:02.560 --> 00:19:05.780 And you can probably, on a lot of cases, you can measure it in dollar value. 00:19:06.060 --> 00:19:20.060 If you find, well, if you find fraud, like you're a high level fraud where your CFO is misrepresenting the company or something, - Or low level fraud where your staff are ripping you off or just errors or inefficiencies or not 00:19:20.060 --> 00:19:20.380 follow up. 00:19:20.400 --> 00:19:22.900 - Contractors or what, subcontractors or something, yeah. 00:19:23.020 --> 00:19:29.640 - Yeah, they can often, when contractors are working on you some way, they'll be doing kind of a concert with maybe some of your staff. 00:19:29.980 --> 00:19:32.200 There's could be some collusion or things like that. 00:19:32.320 --> 00:19:36.920 So looking at your staff behavior, it could be worth a lot of money and there could be a lot of benefit in it. 00:19:37.060 --> 00:19:40.000 But if it takes you 600 hours to run a 00:19:40.000 --> 00:19:41.640 process 00:19:41.720 --> 00:19:44.840 and you find one thing, It may or may not be cost-effective. 00:19:45.160 --> 00:19:47.280 Well, that example wouldn't be cost-effective. 00:19:47.720 --> 00:19:50.700 Yeah, so there's kind of a balancing act you want to follow. 00:19:50.960 --> 00:19:56.520 If you have an extremely small number of points, you can say that one is anomalous, given the others. 00:19:56.580 --> 00:20:04.600 If you have a bunch of, say, you're going through your, maybe just looking at, just thinking of the example of staff and errors at a company. 00:20:04.740 --> 00:20:07.740 Say you have, on staff, you have a bunch of people employed. 00:20:07.940 --> 00:20:12.400 They're purchasers, they're just employed to buy supplies from your suppliers. 00:20:13.120 --> 00:20:19.000 And some, maybe you have 10 of them and most of them make an average of about four purchases a day. 00:20:19.110 --> 00:20:22.760 And you have one that every Friday makes 200 purchases a day. 00:20:24.400 --> 00:20:34.720 So you only have like 10 points there, but one of them is so off the scale that you can say with some confidence, this is not necessarily a problem, but it is anomalous and it is worth understanding. 00:20:35.200 --> 00:20:41.400 Either there's some errors or fraud or inefficiency or something, or the other people are just, They're the anomaly. 00:20:41.980 --> 00:20:43.040 They're not doing their-- 00:20:43.360 --> 00:20:44.140 They're not doing their job. 00:20:44.620 --> 00:20:45.440 Yeah, they're not buying enough. 00:20:45.580 --> 00:20:46.440 You're definitely right, too. 00:20:46.550 --> 00:20:50.440 You can have the other problem where you just have so much data, it's hard to go through. 00:20:50.580 --> 00:20:52.760 Are there techniques that people use for, like-- 00:20:52.920 --> 00:20:59.760 use, for example, Dask or QPi, like the CUDA stuff for GPUs? 00:20:59.760 --> 00:21:03.480 Are there sort of like really high-end compute stuff that people are using for these problems? 00:21:03.640 --> 00:21:07.500 Yeah, and that's-- especially if you're using a deep learning-- well, I guess only. 00:21:08.360 --> 00:21:14.440 But if you're using something that has a lot of matrix operations, a GPU will speed that up. 00:21:14.800 --> 00:21:19.100 And Dask will often work out, especially have memory limitations as well. 00:21:19.860 --> 00:21:22.700 Outlier detection can be done in parallel, which can speed things up. 00:21:23.180 --> 00:21:29.200 Outlier detection works a little bit like predictive models in that there's a train step, and then there's a inference step. 00:21:29.480 --> 00:21:29.620 OK. 00:21:30.040 --> 00:21:33.420 You train it what's normal, and then you ask it if things seem normal or not. 00:21:33.860 --> 00:21:33.980 OK. 00:21:34.120 --> 00:21:34.580 Exactly. 00:21:34.700 --> 00:22:16.280 So normally what you do with outlier detection of data. So, well, if you take the example of an industrial process, so say you monitor your, let's say in this example, you have sensors that are collecting a lot of data, so they're collecting temperature and sound and humidity and so on, information. And, you know, say you have a process that's run for weeks without any problem. So during that time, you know that everything was running properly. So you can take that data and give it to an outlier detection to train on, and then that builds up a sense of what's normal. In this case, it's not only statistically normal, but it's also kind of like a platonic norm. It's desirable as well. That's your inference step. 00:22:16.340 --> 00:22:46.680 And a lot of times, that's only done once a month or every few months or something. And so, it's not terrible if it takes a little bit of time to train. In other contexts, in like a real-time situation, you can have streaming environments where you're just constantly reading in data and it's your sense of what's the new normal. You might be retraining every few minutes or a few hours in which case, yeah, the training time becomes a lot more relevant. It can be done offline, but then it's your inference time and that can be really relevant. 00:22:47.450 --> 00:22:59.680 There are certain ways to keep down your training time as well. Sometimes if you have like months of sensor readings, you might not need to train on all of them. You can just take a sample of them or a certain time slice or something like that. 00:22:59.680 --> 00:22:59.800 And 00:22:59.800 --> 00:23:03.040 that's good enough to establish a decent sense of what's normal. 00:23:03.260 --> 00:23:08.720 Right. It seems to me that the training could be done on a subset and that would probably be OK. 00:23:09.080 --> 00:23:13.260 But I feel like the detection might need to run nearly on everything. 00:23:13.680 --> 00:23:16.780 Right. Because it might be that that one in a million sort of thing. 00:23:16.860 --> 00:23:23.320 Right. Astronomically, you're looking for that one little bright spot or that one little dip in brightness for a transit of a planet or something. 00:23:23.600 --> 00:23:40.860 or I've had my credit card put on a block until I unblocked it because I bought something in a vending machine for a dollar with a credit card. And they're like, well, a lot of people who steal cards, they'll take them to vending machines to try them because it's a safe place with no one there to see if the stolen card still works. And then they'll go do bad stuff. 00:23:41.040 --> 00:23:58.540 Well, it's hard to say what they're doing behind the scenes. Why they flagged it, it could have been because they do run outlier detection routines. They could have been done that or they could have been using like a rule-based system like you're describing where If it's in a certain location for a certain dollar value at a certain time of day and so on, they might have just flagged it because of that. 00:23:58.550 --> 00:23:59.320 >> It was very shady. 00:23:59.420 --> 00:23:59.800 I was at the 00:23:59.800 --> 00:24:00.240 university. 00:24:00.410 --> 00:24:09.920 >> Yeah, it's definitely the case that most of the time when you're running outlier detection, yeah, you want to check all of your data because otherwise it kind of – the question is, are there any anomalies in my data? 00:24:10.140 --> 00:24:19.300 There are exceptions to that where you just want to check – you just kind of want to spot check your data and just want to make sure it's largely free of outliers or that there's not – I mean, depending on how you define them. 00:24:19.880 --> 00:24:46.580 example, a sensor readings, like you always expect a certain number of point anomalies, like a certain point in time where something spikes up and it's not it is an anomaly, but it's not a big deal. So in some cases you might just want to make sure that your number of point anomalies is normal. So you don't have an unusual number of unusual events in that case or cases like that or if you're just doing data quality checks, for example, you might just want to make sure your data is large. 00:24:46.720 --> 00:24:53.680 How do you know that you don't get outliers in your If you just feed it like a stream of data, you're like, well, this is what's normal. 00:24:53.840 --> 00:24:57.640 Actually, there were outliers, but you weren't ready to detect them yet, and now it thinks they're normal. 00:24:57.740 --> 00:24:58.780 Most of the time 00:24:58.780 --> 00:25:04.180 when you train, you do assume that there's a certain number of outliers in your data. 00:25:05.000 --> 00:25:09.280 Yeah, so what you're training on is often a statistical norm. 00:25:09.660 --> 00:25:10.620 It's what's normative. 00:25:11.299 --> 00:25:13.780 It's not necessarily what's ideal and clean. 00:25:14.120 --> 00:25:16.220 It's very hard to get data that's really clean. 00:25:16.740 --> 00:25:24.060 And if you do, it often biases the data in one way or the other, which kind of undermines outlier detection sometimes. 00:25:24.300 --> 00:25:25.700 It's almost like overtraining, maybe. 00:25:25.860 --> 00:25:31.640 Yeah, similar idea to that in that it learns very specific patterns as being normal. 00:25:31.740 --> 00:25:41.300 Actually, what's fairly common in outlier detection is to train and to inference on the same data with the idea that there's outliers in there anyway. 00:25:42.240 --> 00:25:47.040 So in some cases, like, well, go back to the example of a, let's say a credit card company. 00:25:47.360 --> 00:25:52.260 So you have, you want to check all your transactions for the last day. 00:25:52.740 --> 00:25:54.240 So you have a hundred gazillion. 00:25:54.550 --> 00:25:55.540 You can train on that. 00:25:55.700 --> 00:25:58.220 And then you can also run inference on that. 00:25:58.460 --> 00:26:05.200 So you can say, relative to today's transactions, which are the unusual ones, unusual relative to the other ones. 00:26:05.450 --> 00:26:08.660 So it's something you wouldn't do in prediction problems, but you actually can. 00:26:09.200 --> 00:26:10.840 It's fairly common to do that note letter detection. 00:26:11.340 --> 00:26:25.020 just with the idea that as long as most of the data is typical, most algorithms will be able to find the records that are the most extreme or the most unusual in one way or another relative to the other ones in the data. 00:26:25.080 --> 00:26:25.860 Yeah, that's very interesting. 00:26:25.960 --> 00:26:26.960 You can train on 00:26:26.960 --> 00:26:30.540 the data and then ask it to find the outliers in that same data. 00:26:31.100 --> 00:26:32.179 That seems very counterintuitive. 00:26:32.900 --> 00:26:33.000 Yeah. 00:26:33.200 --> 00:26:42.260 It is a little counterintuitive, but like, well, I guess the simple example I gave earlier where you have 10 purchasers for your company. 00:26:42.800 --> 00:26:45.700 You can use those 10 to establish a sense of what's normal. 00:26:46.060 --> 00:26:48.440 And it would say, well, about four purchases a day is normal. 00:26:49.000 --> 00:26:53.080 And this one person that has 200 every Friday, that's unusual. 00:26:53.580 --> 00:26:55.840 So it's actually doing, you can do that in the same data. 00:26:56.340 --> 00:27:05.840 Now having said that, what you do do in outlier detection is you sometimes do that a little bit iteratively where you say, start with the original data and find the most anomalous records in there. 00:27:06.100 --> 00:27:12.780 And then you might inspect them manually and say, "Okay, these are things, yeah, that we wouldn't normally expect to happen. 00:27:13.160 --> 00:27:16.660 So we can remove those." Or there are things you say, "They're extreme, but 00:27:16.660 --> 00:27:18.620 there's 00:27:18.620 --> 00:27:19.340 nothing wrong with them. 00:27:19.400 --> 00:27:20.360 They could happen again." 00:27:20.420 --> 00:27:23.620 Or maybe they're a lightning flash when you're looking through a telescope. 00:27:24.560 --> 00:27:25.460 It's really bright. 00:27:25.480 --> 00:27:34.140 Well, you get a lot of, like, when you're dealing with anything like that, you get a lot of data artifacts where it's not that the phenomena that happened that you're monitoring was unusual. 00:27:34.540 --> 00:27:39.220 the equipment you use to monitor it had some sort of power surge or something like that. 00:27:39.540 --> 00:27:52.100 And that's a data artifact, you might want to remove it. So sometimes you just, you kind of iteratively make your data cleaner and cleaner and gets kind of more pristine sense of what's normal. But you can overfit doing that, you can go too far doing that. 00:27:52.240 --> 00:27:54.920 Yeah, would you maybe find, sort of do a, like you say, a 00:27:54.920 --> 00:28:04.260 loop a couple times, say, train it and then ask it what's abnormal and figure out, well, these things are truly not part of the things we want to consider. So we'll take them out and then train it again. 00:28:04.660 --> 00:28:07.120 excluding those and kind of like iterate on it that way? 00:28:07.260 --> 00:28:11.200 - It's not imperative, but that would be a fairly, fairly standard way to do things, yeah. 00:28:11.440 --> 00:28:16.440 - We haven't properly introduced your book yet, Outlier Detection in Python. 00:28:16.760 --> 00:28:17.540 - Yes. - From Manning. 00:28:17.640 --> 00:28:24.560 And I want to give you a chance to give a shout out about it and then I want to talk to you about a couple of things that you've covered in your book, of course. 00:28:24.960 --> 00:28:29.900 So I know that we have some discount code for the book that Manning has given us. 00:28:29.940 --> 00:28:31.840 I'll put that in the show notes for people if they're interested. 00:28:32.440 --> 00:28:36.900 Well, yeah, tell us, tell us about your book of it and why do you write it and what people can learn from it? 00:28:36.940 --> 00:28:41.860 One job I worked at a few years ago, I just spent an enormous amount of time. 00:28:42.539 --> 00:28:46.120 I was managing a research team and there's about 10 of us on the team. 00:28:46.180 --> 00:28:52.380 And we looked into a lot of areas of machine learning too, but I, outlier detection was the biggest one for us. 00:28:52.980 --> 00:28:55.140 And so we were doing a lot of work around that. 00:28:55.580 --> 00:29:46.380 For, well, this is the company, as I mentioned, that was doing financial auditing. So we're building tools that would allow financial auditors to, when performing an audit, examine their client's data and just in a fairly easy way, see what's unusual in their data. The idea that, for example, if you're looking through their financial transactions, like if you're an accounting firm and you're auditing spatula world, this example I used to always give for the word elf fans, I guess, is they would have a set of sales and a set of purchases and set of payroll transactions and so on. So you can go, what auditors would often do is just find the most anomalous of those with the idea that those are the ones that are among the most important to investigate, to see if there's anything off with them that could suggest an error or something like that. 00:29:47.140 --> 00:29:55.040 So we spent a lot of time doing that. We were working with regulators around the world, not just audit firms, but also regulators and bodies like that. 00:29:55.560 --> 00:30:01.360 And we spent a lot of time looking at how can we justify the outliers that we're finding? 00:30:01.860 --> 00:30:07.140 Because if we run some of these, most of these algorithms are a little bit like black boxes, blocks algorithms. 00:30:07.290 --> 00:30:13.180 So like if you're doing a predictive model and you send something through an XGBoost model or a neural net or something, it gives you a prediction. 00:30:13.320 --> 00:30:15.480 And it's hard to say why it made that prediction. 00:30:16.100 --> 00:30:32.720 So a lot of outlier detection algorithms are like that, but it's kind of an issue sometimes because in outlier detection, probably more than with prediction or classification or something, or clustering, I should say, it's important to have an explanation of why something is unusual. 00:30:33.300 --> 00:30:49.160 Like if you're flagging something as maybe being potentially fraudulent, or if you're doing security and you're flagging some web activity as being problematic, or a person in some sort of security context is possibly being some sort of security risk. 00:30:49.290 --> 00:30:54.800 You gotta know why in order to investigate effectively and quickly. 00:30:55.480 --> 00:31:13.940 So we spent a lot of time looking at how to make Outlier Detection 1 interpretable and also just kind of justify the results that we're finding if we assess the tables of data and come back with a summary of them saying, these are the most unusual records in this data. 00:31:14.160 --> 00:31:17.960 But it's actually a hard problem to try and explain why that was the case. 00:31:18.420 --> 00:31:24.500 So I ended up just reading hundreds of, I think like literally hundreds of papers on the subject. 00:31:24.680 --> 00:31:28.980 And we did a lot of original research and it ended up building up quite a lot of expertise. 00:31:29.240 --> 00:31:34.140 But also during that time, it was just a really fascinating, interesting problem. 00:31:34.440 --> 00:31:53.600 Realized outlier detection, it's just a really, it just kind of grabs you because yeah, It's just kind of something fascinating, but you're trying to figure out if you have a collection of images or a collection of sound files or a collection of, like one job I had, we were working with network data, like social networks in that case, connection kind of graph models. 00:31:54.440 --> 00:31:56.980 What are the most unusual records in there? 00:31:57.680 --> 00:31:58.660 This is really challenging. 00:31:59.180 --> 00:32:03.180 There's no definitive answer, but there's certainly good answers. 00:32:03.320 --> 00:32:13.920 And there's ways to do it efficiently and interpretably and accurately, But it's difficult, but it's just really rewarding when you kind of get something, a hard problem 00:32:13.920 --> 00:32:14.860 like that working well. 00:32:15.020 --> 00:32:22.560 - Yeah, I imagine it's the kind of thing that if you have those answers, you kind of become real good friends with the CEO or the decision makers. 00:32:22.800 --> 00:32:27.940 They're like, this person can look at it and answer the questions that we can't even know to ask. 00:32:28.100 --> 00:32:28.640 - Yeah, exactly. 00:32:28.760 --> 00:32:31.940 It does open up different ways of thinking about things. 00:32:32.280 --> 00:32:34.380 And yeah, it's just, yeah. 00:32:34.840 --> 00:32:37.080 It has a lot of benefits when you're looking at data. 00:32:37.260 --> 00:32:38.000 - I imagine you could 00:32:38.000 --> 00:32:38.400 probably 00:32:38.400 --> 00:32:39.000 also 00:32:39.000 --> 00:32:44.180 make really important decisions for where to put your energy in your company, right? 00:32:44.500 --> 00:32:46.600 Like you had spatula world as an example. 00:32:46.740 --> 00:32:56.920 Like if some product all of a sudden seems like it's doing really well and that's an outlier, like you should be able to answer, this is actually gaining a lot of traction, we should manufacture a bunch. 00:32:57.060 --> 00:33:08.360 Or no, that's just some weird glitch and that's not gonna be, that's not predictive of, should we order 100,000 green spatulas or is that just something weird that happened some restaurant come and bought all the ones at a store. 00:33:08.450 --> 00:33:08.580 So 00:33:08.580 --> 00:33:13.580 it's really hard to, but important to be able to quantify these things and just give some context of 00:33:13.580 --> 00:33:14.360 you 00:33:14.360 --> 00:33:16.680 could end up in a bad place if you make an investment. 00:33:16.880 --> 00:33:17.080 Yeah. 00:33:17.340 --> 00:33:17.460 Yeah. 00:33:17.580 --> 00:33:19.560 So it's the curse of, of half understanding 00:33:19.560 --> 00:33:21.120 of 00:33:21.120 --> 00:33:21.900 a little bit of knowledge. 00:33:21.900 --> 00:33:22.280 I guess. 00:33:22.560 --> 00:33:23.540 I've got a great idea. 00:33:24.020 --> 00:33:24.260 Yeah. 00:33:24.420 --> 00:33:24.680 All right. 00:33:24.720 --> 00:33:27.800 Let's talk about some of the tools that you highlighted in your book. 00:33:28.140 --> 00:33:31.360 Like the PI OD, Python outlier detection. 00:33:31.840 --> 00:33:31.900 Yeah. 00:33:31.920 --> 00:33:32.720 Let's talk about this one. 00:33:33.140 --> 00:33:34.080 What was the story of this thing? 00:33:34.080 --> 00:33:39.240 It's used quite a bit, used by 4,000 people and 8.8 thousand GitHub stars. 00:33:39.360 --> 00:33:40.780 That's pretty, pretty well used. 00:33:40.930 --> 00:33:41.480 It is. 00:33:41.620 --> 00:33:41.800 Yeah. 00:33:41.920 --> 00:33:45.820 It's really the, it's the go-to one for outlier detection in Python. 00:33:46.000 --> 00:33:46.180 Okay. 00:33:46.300 --> 00:33:49.520 Anyone's doing any kind of machine learning work in, in Python. 00:33:49.730 --> 00:33:52.160 I mean, the one tool there they'll know is scikit-learn. 00:33:53.040 --> 00:33:56.940 And scikit-learn comes with four outlier, well, it comes to four outlier detectors. 00:33:57.440 --> 00:34:03.280 And also some tools that make it relatively easy to do some other types of outlier detection, like as a KDE tool. 00:34:03.280 --> 00:34:18.960 we'll see kernel density estimation, which makes it fairly easy to find outliers to just points in space that have low density as well clustering method called Gaussian mixture models and GM, which makes it fairly easy to do outlier detection using that as well. 00:34:19.440 --> 00:34:25.560 But PIIOD, yeah, so PIIOD has those, has basically everything in scikit-learn, wraps it in its own interface. 00:34:26.120 --> 00:34:31.879 And I think a couple dozen more, and it also has a number of deep learning based ones. 00:34:32.000 --> 00:34:36.659 So for tabular data, it's really where you want to go first, probably. 00:34:37.419 --> 00:34:42.300 So yeah, in terms of deep learning, it has well autoencoders, variational autoencoders, GANs. 00:34:42.899 --> 00:34:46.460 There are some limitations with PYAD, and I do get into that in the book. 00:34:46.860 --> 00:34:57.020 It assumes you're dealing with strictly numeric data, for example, which if you're dealing with tabular data, realistically, you have mixed data, categorical columns, date columns. 00:34:57.600 --> 00:34:59.200 So PYAD does not handle that. 00:34:59.780 --> 00:35:01.560 And it's strictly for tabular data. 00:35:01.640 --> 00:35:07.800 So if you're working with time series data or image data or audio data, that sort of thing, there are other tools involved for that. 00:35:08.060 --> 00:35:11.600 But yeah, for tabular data, it's pretty much the most used one. 00:35:11.760 --> 00:35:17.000 - Okay, got the NLP 80 bench for NLP anomaly detection. 00:35:17.080 --> 00:35:19.160 So I'm guessing this is in text. 00:35:19.620 --> 00:35:28.280 I'm not sure, you'll have to tell us what kind of stuff you detect with that, but I wanna give it some articles or maybe give it social media posts or something along those lines, then ask it. 00:35:28.480 --> 00:35:29.940 What kind of questions could you ask it about 00:35:29.940 --> 00:35:30.140 that? 00:35:30.320 --> 00:35:34.700 NLP is kind of a, it's a dicey area without layer detection. 00:35:35.260 --> 00:35:36.880 And I shouldn't, dicey is too strong a word. 00:35:37.220 --> 00:35:38.160 It's a harder area. 00:35:38.320 --> 00:35:59.680 It's certainly a harder area to work with because like if you have tables of data again, like say accounting data, for example, if a bunch of transactions, it's difficult to say which are the most unusual transactions in your data, but it's kind of, like if you see something that's very unusual, you can say, okay, that transaction is unusual. 00:36:00.040 --> 00:36:02.380 With text, that is harder to say. 00:36:02.940 --> 00:36:09.200 I mean, if it's a completely different language or something like that, or the tone, or the writing style is completely 00:36:09.200 --> 00:36:09.840 different. 00:36:10.040 --> 00:36:17.300 - And you have curse words, and people might think it's angry, but in fact, it's just sort of a coarse way of expressing excitement, right? 00:36:17.520 --> 00:36:19.180 Like, oh, these people are really angry. 00:36:19.260 --> 00:36:23.300 Like, no, they're actually really happy, but they don't really express it super properly. 00:36:23.680 --> 00:36:30.480 - Right, but if the rest of the corpus did not have any expletives, then in a sense that's anomalous in that way. 00:36:31.240 --> 00:36:35.540 Yeah, with text, there's just so many ways that text can be just a little bit different. 00:36:35.860 --> 00:36:41.800 I would consider it one of the more challenging areas of machine learning of outlier detection, yeah. 00:36:41.880 --> 00:36:42.160 - I'm sure, 00:36:42.240 --> 00:36:45.700 we'll talk about LLMs as well and see how that changes things or not in a little bit. 00:36:45.780 --> 00:36:47.860 But the other one you mentioned is scikit-learn. 00:36:48.020 --> 00:36:53.180 Like this is kind of the granddaddy of a lot of the data science type of stuff. 00:36:53.220 --> 00:36:55.800 And they've got a whole section on novelty and outlier detection. 00:36:56.280 --> 00:37:03.720 maybe I'm headed up on the screen for a little bit, but give people a sense of kind of problems you can ask with this one or problems you solve questions you can ask. 00:37:03.780 --> 00:37:12.100 It's a little bit like PIOD, it's less so, but it doesn't have like 30 or so detectors, it has four, but includes two of them. 00:37:12.560 --> 00:37:14.040 So again, it's just for tabular data. 00:37:14.080 --> 00:37:19.800 And again, it's just numeric tabular data, but it has two of the go-to algorithms. 00:37:19.880 --> 00:37:21.760 They're called one's called isolation forest. 00:37:21.860 --> 00:37:23.860 And the other is called local outlier factor. 00:37:24.220 --> 00:37:31.140 I think most of the time with outlier detection, if you're doing tabular data, if you're only gonna use two algorithms, those are the two. 00:37:31.840 --> 00:37:45.520 Outlier detection is a little bit different than say prediction, because if you're building a predictive model, say on tabular data, most of the time you would use, say an XGBoost model or a CatBoost model or a random forest or something like that. 00:37:45.630 --> 00:37:46.840 You would tend to use one model. 00:37:47.340 --> 00:37:52.400 Sometimes if you really wanna squeeze all the accuracy you can out of it, you'd create an ensemble. 00:37:52.740 --> 00:38:08.240 Actually, internally is an ensemble of, it's a boosted ensemble of decision trees, but normally you wouldn't, sometimes you do, but probably more often we don't create an ensemble of a KNN and an SVM and a neural net and a random forest. 00:38:08.520 --> 00:38:11.200 In outlier detection, that's a lot more common to do that. 00:38:11.320 --> 00:38:18.100 And the reason is that each outlier detection algorithm, it can be a little bit limited in one way or another. 00:38:18.320 --> 00:38:25.220 They're not in an extreme way, but they each look for different types of outliers and they each look for outliers in different ways. 00:38:25.880 --> 00:38:33.320 And it's just, there's a lot more advantage to using multiple detection algorithms with outlier detection than you would with prediction. 00:38:33.610 --> 00:38:35.540 And you can do it in just really simple ways. 00:38:35.570 --> 00:38:46.540 If you say, scikit-learn has four outlier detectors, it has isolation forest, it has local outlier factor, has one called one class SVM and has one called elliptic envelope. 00:38:47.160 --> 00:38:59.780 So if I run the four of those on my data, and let's say they're all four of them are appropriate and work well. Any records that are scored highly by all four of those detectors, you can say, well, these records are really 00:38:59.780 --> 00:39:00.480 weird. 00:39:00.990 --> 00:39:02.120 You can say, yeah. 00:39:02.360 --> 00:39:08.320 Yeah, you can let it say, we've checked many different ways and we've got the same answer that it's an outlier in these different aspects. 00:39:08.430 --> 00:39:09.660 And so it's really likely. 00:39:09.820 --> 00:39:14.880 It's really likely. It's not necessarily a problem, but it is statistically unusual. 00:39:15.200 --> 00:39:28.400 And yeah, and I think a lot of time, if you're just, if you don't have a lot of intense outlier detection work to do, you just have a small data set and you just wanna quickly get a sense of what are the most anomalous records in it. 00:39:28.700 --> 00:39:33.720 Just using scikit-learn with the assumption that you, you're using scikit-learn anyways for a lot of your work. 00:39:34.140 --> 00:39:37.820 Just using that is probably gonna be, yeah, sufficient for a lot of cases. 00:39:38.100 --> 00:39:53.280 If you want to do a little bit more thorough job using more detectors again, and just using detectors that can be a little bit more, PIOD has ones that are a little bit more performative, has some that are a little more interpretable. 00:39:53.720 --> 00:39:58.100 It could be a good decision just to use PIOD and use a bunch of the detectors there. 00:39:58.440 --> 00:40:02.440 One of the nice things about PIOD is it has the same API signature for all of its detectors. 00:40:02.920 --> 00:40:11.400 So if you write code, and it's usually only like four or five lines, it's really kind of boilerplate easy, other than the hyperparameters for them. 00:40:11.700 --> 00:40:17.320 It's really quite easy to just kind of swap one out and swap in another and just try a whole bunch of them. 00:40:17.920 --> 00:40:25.360 One thing I do get into the book is that there's a lot of algorithms beyond even what's in PIOD that can be very useful to use. 00:40:25.640 --> 00:40:32.520 The ones in PIOD, they're good and they're sufficient quite often, but they're limited into how interpretable they are. 00:40:32.880 --> 00:40:45.480 So again, if you're in a situation where you're saying, well, this looks like a security threat or this looks like failing equipment this looks like a novel specimen. Say from astronomy, this looks like a novel transit. 00:40:45.860 --> 00:41:01.940 This is a transit that's not normal. You want to know why. So a lot of the detectors from PIOD don't provide that necessarily to the degree you would wish. So I do suggest a bunch of others that could be useful for that purpose and that support categorical data 00:41:01.940 --> 00:41:02.720 as well. 00:41:02.840 --> 00:41:07.860 That's non-numeric stuff like male, female or high income, low income or whatever. 00:41:08.180 --> 00:41:08.400 - Exactly. 00:41:08.840 --> 00:41:10.280 - We talked about time series a little bit. 00:41:10.280 --> 00:41:16.500 You said that time series are pretty challenging and I guess it depends on how rapidly the time series data is coming in as well. 00:41:16.640 --> 00:41:22.700 - Yeah, that's actually one of the things about time series data is that it can be streaming and coming in extremely fast. 00:41:23.100 --> 00:41:30.080 And yeah, so just like I was saying, like very often we use ensembles of outlier detectors to try and analyze the data. 00:41:30.320 --> 00:41:32.200 By their nature, ensembles can be a little bit slower. 00:41:32.280 --> 00:41:33.120 Now you can run them in parallel. 00:41:33.460 --> 00:41:43.400 - Yeah, right, 'cause it's basically, you take three or four or a thousand decision trees them all what their answers are and then you kind of like decide what the, it's like a voting process or something almost, right? 00:41:43.500 --> 00:42:30.620 And that would be exactly how you would do it in a predictive model. So with an outlier detection model, well there's probably the closest thing to what you just described with outlier detection is there's an algorithm called isolation forest, which is a forest of isolation trees. So it's kind of similar ideas. It's probably close to, a little bit close to random forests, probably a little closer to extra trees for anyone who's familiar with predictive models. That's isolation forest. So it's kind of the outlier detection equivalent of extra trees model. And like that is that it's composed of a whole bunch of trees and depending on the implementation of it, not scikit-learns, but depending on the implementation of it, you can run all those trees in parallel, which can give you your answers quite fast if you have sufficient hardware. 00:42:30.990 --> 00:42:57.860 What can also be done with outlier detection too is, which can actually, depending on your environment, work a little faster, can actually be faster to run your detectors in sequence, believe it or not, because what you can do then is, depending on how you define your outliers, but say you say, I'm only concerned with things that are flagged by the isolation forest and the local outlier fracture model and the KNN model and the autoencoder model. 00:42:58.160 --> 00:43:07.900 You can run them in, so anything that's filtered out by the first model doesn't need to be sent to the second model and then, or the third model or the fourth model, so you can run it, you can kind of put the fastest one out first and then the. 00:43:09.240 --> 00:43:09.380 Yeah. 00:43:09.480 --> 00:43:09.880 That's interesting. 00:43:10.240 --> 00:43:26.360 Almost like thinking about it, like an if statement or a while loop or something in programming where, where the ands short circuit, so like you want to do the simplest test first and then the next one, the next one, and you're only going to evaluate it until you hit a false and then you stop, be a positive in this case or something. 00:43:26.540 --> 00:43:26.680 Yeah. 00:43:26.800 --> 00:43:27.680 That's a good way to put it. 00:43:27.760 --> 00:43:30.520 Yeah, if you do it in sequence, that's, that'll definitely speed it up. 00:43:30.630 --> 00:43:34.480 Rather than taking all the data, run it in parallel across all the cores. 00:43:34.660 --> 00:43:35.900 You're like, well, look how awesome it is. 00:43:35.960 --> 00:43:40.260 It's using all the cores, but if you're like, no, we just run less of it. 00:43:40.340 --> 00:43:41.460 So we don't have to do it so much. 00:43:41.540 --> 00:43:42.720 There will still be benefit to that. 00:43:42.730 --> 00:43:53.740 If you want to know specifically how anomalous things are, which you might, like, if you're not looking for to flag the individual anomalies, you're trying, you're more interested in collecting statistics about the volume of anomalies. 00:43:53.840 --> 00:44:00.060 Like you're, it's more data monitoring project or something as opposed to picking out the individual anomalies. 00:44:00.520 --> 00:44:10.980 You may still want to do that, but if you just want to pick out the really anomalous records, then yes, they might be the ones that are flagged by all of your, or almost all of your detectors, say. 00:44:11.060 --> 00:44:11.140 - Right, 00:44:11.240 --> 00:44:13.160 well, you don't even have to do, you could even do it partially. 00:44:13.600 --> 00:44:18.440 If it's not flagged by three out of five, then we're not gonna pass it further down the line, right? 00:44:18.500 --> 00:44:24.860 Like you could still do some sort of intermediate, more squishy analysis like that, but you could still short circuit it some. 00:44:25.000 --> 00:44:26.520 - That would be a good balance, I think, yeah. 00:44:27.260 --> 00:44:32.080 I mean, it's like anything, there's gonna be false positives and false negatives, but you 00:44:32.080 --> 00:44:33.540 want to get a good balance of that. 00:44:33.800 --> 00:44:44.020 - One of the other projects that you called out is Profit, time series anomaly detection with Profit, and it's based on the Facebook's open source library, Profit. 00:44:44.380 --> 00:44:55.900 This is Profit anomaly detection versus just straight Profit, which is a tool for producing high quality forecasts for time series data that has multiple seasonality with linear and nonlinear growth. 00:44:56.220 --> 00:44:59.960 I imagine that Facebook probably has a lot of data in its time series. 00:45:00.280 --> 00:45:02.740 - If anyone does, yes, they would have phenomenal amounts. 00:45:02.740 --> 00:45:05.300 And there's probably all sorts of things they wanna forecast. 00:45:05.730 --> 00:45:07.920 - So tell people about this project and how they might use it. 00:45:08.080 --> 00:45:11.840 - There's a number of ways to do outlier detection with time series data. 00:45:12.140 --> 00:45:17.960 It's a little bit like I was saying with tabular data, you can look at your data in all kinds of different ways and they might all be legitimate. 00:45:18.010 --> 00:45:19.840 They just find different types of outliers. 00:45:20.260 --> 00:45:29.800 A really, really simple case, which can be useful, despite its simplicity is if you just have just a timeline over, over time of points. 00:45:31.040 --> 00:45:36.360 So say you have a, well, say you're a social media BMF that has a half of humanity signed up. 00:45:36.640 --> 00:45:40.960 You might want to look at your server usage or ad revenues. 00:45:41.160 --> 00:45:43.580 And there's, there's all, just all kinds of things that you're, you're predicting. 00:45:43.860 --> 00:45:45.520 One thing that things will be going viral. 00:45:45.640 --> 00:45:56.580 And so maybe we should boost that in the algorithm and show it to more people, which could be very good or not so good, but anyway, it still works with the incentives and the goals of, "Hey, we want more engagement on the platform," right? 00:45:56.680 --> 00:46:06.420 I guess one would be simple example, maybe weather data or going outside of a social media company where you just tracked temperature over time and there's certain seasonality to that. 00:46:06.760 --> 00:46:09.600 What is that plot there? That is... Hard to say what that is. 00:46:09.740 --> 00:46:11.800 Their example is, let's see if they call it out. 00:46:12.200 --> 00:46:12.780 Oh, that 00:46:12.780 --> 00:46:13.640 could be weather data 00:46:13.640 --> 00:46:14.920 or it could be stock. 00:46:15.120 --> 00:46:18.860 They don't really say. Have they named their... All their data is named DF 00:46:18.860 --> 00:46:20.060 and 00:46:20.060 --> 00:46:20.420 stuff. 00:46:20.700 --> 00:46:20.780 - They 00:46:20.780 --> 00:46:25.740 do, the way profit works is that there's only, you work with tables data and they only have two columns. 00:46:25.900 --> 00:46:28.800 There's the time column and there's the value column. 00:46:29.100 --> 00:46:29.480 And 00:46:29.480 --> 00:46:30.840 you can't expand much. 00:46:31.060 --> 00:46:33.600 - The legend says anomaly actual predicted base. 00:46:33.700 --> 00:46:39.460 There's no information here that tells you what data this is but it still gives you a, it's still a cool picture to show you how it works. 00:46:39.520 --> 00:46:43.760 It's like, here's the prior block of data like last week or something. 00:46:43.930 --> 00:46:45.460 And now here's the next week. 00:46:46.060 --> 00:46:49.300 And are we gonna predict if this is an anomaly or not? 00:46:49.380 --> 00:46:49.700 That 00:46:49.700 --> 00:47:13.980 is kind of the nature of time series data is it doesn't really matter if it's weather data or stock data or bird migration data. As long as there's certain patterns to it that can be found, you can work with that kind of equivalently. What I think they're showing there is something you often do in time series data is you have a certain pattern that's existed in time. So there's a certain downward trend in that and there's some seasonality. 00:47:14.960 --> 00:47:19.700 So maybe those might be days of the week or yeah, those look like days of the week. 00:47:20.200 --> 00:47:23.520 So maybe things are busier on a weekend or something like that. 00:47:24.840 --> 00:47:36.380 So given the patterns that have occurred in the past where you know, look at the time of the day and the day of the week, the month, the year and the like, you can make projections into the future forecasts. 00:47:36.920 --> 00:47:38.600 And then you look at your actual data. 00:47:39.080 --> 00:47:46.760 If your forecast is generally correct, but you have some points that are way off from your projection, that's a form of anomaly. 00:47:47.160 --> 00:47:49.560 So that's something that's often done for outlier detection. 00:47:49.720 --> 00:47:57.960 It's just saying, given the history of what we've seen, we would have really expected a value of 10 here, but we see a value of 38. 00:47:58.900 --> 00:48:00.480 So there's something unusual there. 00:48:00.590 --> 00:48:05.660 It just doesn't follow the normal seasonality and normal trends that we've seen. 00:48:05.730 --> 00:48:06.880 >> Yeah. It probably can't necessarily 00:48:06.880 --> 00:48:07.680 give 00:48:07.680 --> 00:48:10.920 you too much insight as to why, but you should go look at it, right? 00:48:11.000 --> 00:48:11.540 'Cause it's just 00:48:11.540 --> 00:48:12.240 the 00:48:12.240 --> 00:48:14.220 quantity and time, right? 00:48:14.300 --> 00:48:16.320 It's pretty sparse data. 00:48:16.480 --> 00:48:33.000 - You might be able to see why you made the prediction you did because it's just based on how you, there's a lot of ways to do, profit works a certain way and there's a lot of ways to do time series forecasting, but generally you're just looking at the regular patterns, the general trend and maybe just like lag features and things like that. 00:48:33.190 --> 00:48:37.840 So you can figure out why you made the prediction you did, but yeah, why it actually had the actual value that it did. 00:48:38.560 --> 00:48:40.500 Yeah, you'll probably have to go and investigate. 00:48:41.120 --> 00:48:41.220 Sure. 00:48:41.340 --> 00:48:45.420 Maybe pull up some, some different tools and different algorithms now that you can focus in on that. 00:48:45.620 --> 00:48:48.520 All right, Brett, let's talk about one more thing while we're here. 00:48:49.160 --> 00:48:52.240 What's the story with LLMs and outlier detection? 00:48:52.420 --> 00:48:57.120 And I think there's two angles I'd like to talk about this with you pretty, pretty quickly, just briefly. 00:48:57.540 --> 00:49:04.000 One, can I just throw a ton of data at ChatGPT and I'm not talking the ChatGPT-4 model. 00:49:04.200 --> 00:49:09.720 I'm, I'm talking like, Oh, one reason the higher end model that can take could hold all the data potentially. 00:49:09.980 --> 00:49:11.020 And it's a little bit more thorough. 00:49:11.250 --> 00:49:12.520 You can ask it, find me the outliers. 00:49:12.940 --> 00:49:15.840 Like, is that a world, a thing that you might, might work? 00:49:15.870 --> 00:49:21.420 And the other is, could I give it some data and say, describe the data and say, okay, here's the situation. 00:49:21.720 --> 00:49:22.540 Here's what this data means. 00:49:23.200 --> 00:49:28.440 Recommend me algorithms and tools and libraries and, and even techniques to analyze it. 00:49:28.800 --> 00:49:30.620 Like, does chat make sense for either of 00:49:30.620 --> 00:49:30.800 these? 00:49:30.910 --> 00:49:32.240 I mean, it's going to be like a lot of things. 00:49:32.270 --> 00:49:38.860 If you have an, if you have an expertise in these, sort of areas, you're you're gonna be better off just to take advantage of that. 00:49:38.960 --> 00:49:47.720 But like a lot of areas of data science, if you're working into an area that you're, you don't happen to have spent years thinking about, these, yeah, these tools can be a good place to get you started. 00:49:48.300 --> 00:49:49.420 I've worked with them a bit. 00:49:49.840 --> 00:49:54.660 And what I've found is, though I have to admit, I haven't used O1 yet. 00:49:55.100 --> 00:49:56.880 I used tools that were a little bit older. 00:49:57.140 --> 00:50:04.560 And what I was finding is it was suggesting outliers, but in a really perfectly legitimate, but a limited way. 00:50:04.780 --> 00:50:16.180 So it would suggest doing like a Z score test on your numeric columns or checking interquartile ranges and things like that, which are good ways to find outliers. 00:50:16.700 --> 00:50:21.020 And I think it may be suggested using an isolation forest and a couple of things. 00:50:21.220 --> 00:50:25.600 So I think it went over the basics reasonably well. 00:50:26.320 --> 00:50:33.940 I kept trying to do a lot of prompt engineering to try and get it to push it towards more sophisticated analysis of the data. 00:50:34.060 --> 00:50:35.420 and that wasn't happening. 00:50:35.740 --> 00:50:40.920 Though, who knows, maybe if not this version, maybe the next one, a version of the LMS. 00:50:41.500 --> 00:50:44.440 I think we're heading to that point where it can start to do that. 00:50:45.160 --> 00:50:49.220 And again, yeah, same with trying to summarize the data and find the patterns in there. 00:50:49.540 --> 00:50:51.780 It's able to do fairly well. 00:50:52.600 --> 00:51:00.420 It's not at the level of like what a real, certainly not at the level of someone with domain expertise that really understands the data is gonna be able to do. 00:51:00.780 --> 00:51:01.900 But it can provide 00:51:01.900 --> 00:51:03.720 some rough summarization of the data. 00:51:03.820 --> 00:51:08.180 maybe point you in the right direction to find some tools you don't know maybe and you could go use them. 00:51:08.280 --> 00:51:12.380 I'm kind of like leaning towards saying it's not going to be doing the best. 00:51:12.720 --> 00:51:19.940 But having said that, they work a lot better for when I have tested them for outlier detection than a year ago or something like that. 00:51:20.140 --> 00:51:23.040 So it's heading in the direction where they might be able to do something quite useful. 00:51:23.300 --> 00:51:28.580 Yeah, I've started using the O1 model and it's a lot slower, but you just got to change your mindset. 00:51:28.740 --> 00:51:29.920 It's not a chat conversation. 00:51:30.060 --> 00:51:35.420 and it's like I've given an intern a job and I expect it to come back with me a little later. 00:51:35.860 --> 00:51:37.100 It's getting better and better. 00:51:37.280 --> 00:51:38.440 It's a crazy time. 00:51:38.540 --> 00:51:39.120 - It's remarkable. 00:51:39.680 --> 00:51:40.920 Yeah, mixed feelings. 00:51:41.300 --> 00:51:41.660 - I know. 00:51:42.480 --> 00:51:46.980 Let's close this whole thing out with maybe just a final call to action. 00:51:47.040 --> 00:51:48.720 People are interested in this topic. 00:51:48.840 --> 00:51:49.720 They wanna do more. 00:51:50.180 --> 00:51:52.500 Maybe another shout out to your book if you want. 00:51:52.820 --> 00:51:54.880 And yeah, people are interested. 00:51:54.940 --> 00:51:56.220 They wanna do more, go deeper. 00:51:56.320 --> 00:51:56.740 What do you tell 'em? 00:51:56.940 --> 00:51:58.280 - I would recommend the book, of course. 00:51:58.620 --> 00:51:59.940 I really, it's on 00:51:59.940 --> 00:52:00.200 Manning. 00:52:00.620 --> 00:52:07.920 I really wanted to write, when I was looking to write with Manning, it's just a publishing company I always thought highly of. 00:52:08.260 --> 00:52:18.220 Yeah, if you just go to manning.com and look for outlier detection, and I think, well, the affiliate link, that'll give you 45% off too, which is a nice link. 00:52:18.530 --> 00:52:21.640 I think, I mean, the book has gotten quite good feedback. 00:52:22.320 --> 00:52:30.180 One thing I was really happy with is some of the people that are architects of the most important algorithms in outlier detection. 00:52:30.380 --> 00:52:35.340 So things like isolation forest, local outlier factor, extended isolation forest. 00:52:35.700 --> 00:52:37.980 They gave some really nice feedback on the book. 00:52:38.520 --> 00:52:40.760 It's gotten quite good feedback so far. 00:52:41.220 --> 00:52:45.540 And I think it's probably as comprehensive a book as you would need. 00:52:45.640 --> 00:52:46.640 - I'll say it's quite comprehensive. 00:52:47.040 --> 00:52:47.800 I was going through it. 00:52:48.480 --> 00:52:49.980 - It covers what you would, yeah. 00:52:50.360 --> 00:52:56.020 If you're not doing time series analysis, for example, you can probably skip the chapter on time series analysis, but, or deep learning. 00:52:56.400 --> 00:53:00.900 But most of it, I think, is actually-- there's a lot of gotchas with outlier detection. 00:53:01.200 --> 00:53:04.320 I mean, they're not too hard to get your head around, but you do have to think about them. 00:53:04.800 --> 00:53:08.640 And so it does cover pretty much anything you would need to know. 00:53:08.720 --> 00:53:13.240 I think for almost any case, you wouldn't need to look for too many other resources after 00:53:13.240 --> 00:53:13.680 reading this. 00:53:13.900 --> 00:53:13.980 Excellent. 00:53:14.220 --> 00:53:17.620 And PyOD is also another good resource maybe to get started with, you think? 00:53:17.720 --> 00:53:20.960 Depending on your situation, either just scikit-learn or PyOD. 00:53:21.260 --> 00:53:26.600 And then if you really need to go into deep learning, I would go in deep OD or even just pyod. 00:53:26.860 --> 00:53:29.900 And for time series, there's quite a number of libraries 00:53:29.900 --> 00:53:30.760 you can use as well. 00:53:30.920 --> 00:53:31.140 - Okay, 00:53:31.480 --> 00:53:31.720 excellent. 00:53:32.180 --> 00:53:35.780 Well, thank you so much for being on the show and sharing your work. 00:53:35.810 --> 00:53:36.960 - Well, thank you very much for having me. 00:53:37.120 --> 00:53:37.780 Yeah, that was very good. 00:53:37.940 --> 00:53:38.460 - Yeah, you bet. 00:53:38.900 --> 00:53:39.080 Bye now. 00:53:39.960 --> 00:53:44.340 This has been another episode of "Talk Python To Me." Thank you to our sponsors. 00:53:44.760 --> 00:53:46.060 Be sure to check out what they're offering. 00:53:46.120 --> 00:53:47.400 It really helps support the show. 00:53:48.140 --> 00:53:51.860 This episode is sponsored by Posit Connect from the makers of Shiny. 00:53:52.340 --> 00:53:56.340 publish, share, and deploy all of your data projects that you're creating using Python. 00:53:56.820 --> 00:54:02.980 Streamlit, Dash, Shiny, Bokeh, FastAPI, Flask, Quarto, Reports, Dashboards, and APIs. 00:54:03.800 --> 00:54:05.400 Posit Connect supports all of them. 00:54:05.660 --> 00:54:11.020 Try Posit Connect for free by going to talkpython.fm/posit, P-O-S-I-T. 00:54:11.760 --> 00:54:12.640 Want to level up your Python? 00:54:13.080 --> 00:54:16.820 We have one of the largest catalogs of Python video courses over at Talk Python. 00:54:17.110 --> 00:54:21.880 Our content ranges from true beginners to deeply advanced topics like memory and async. 00:54:22.300 --> 00:54:24.560 And best of all, there's not a subscription in sight. 00:54:24.960 --> 00:54:27.440 Check it out for yourself at training.talkpython.fm. 00:54:28.180 --> 00:54:32.440 Be sure to subscribe to the show, open your favorite podcast app, and search for Python. 00:54:32.710 --> 00:54:33.640 We should be right at the top. 00:54:34.070 --> 00:54:43.000 You can also find the iTunes feed at /itunes, the Google Play feed at /play, and the Direct RSS feed at /rss on talkpython.fm. 00:54:43.380 --> 00:54:46.000 We're live streaming most of our recordings these days. 00:54:46.230 --> 00:54:53.920 If you wanna be part of the show and have your comments featured on the air, be sure to subscribe to our YouTube channel at talkpython.fm/youtube. 00:54:54.760 --> 00:54:55.860 This is your host, Michael Kennedy. 00:54:56.220 --> 00:54:57.120 Thanks so much for listening. 00:54:57.260 --> 00:54:58.280 I really appreciate it. 00:54:58.600 --> 00:55:00.220 Now get out there and write some Python code.