#261: Monitoring and auditing machine learning Transcript

00:00 Traditionally, when we've depended upon software to make a decision with real world implications that software was deterministic, it had some inputs, a few if statements, and we could point to the exact line of code where the decision was made, and the same inputs would lead to the same decisions. Nowadays, with the rise of machine learning and neural networks, this is much more blurry. How did the model decide has the model inputs drifted apart? So the decisions are outside what it was designed for? These are just some of the questions discussed with our guest Andrew Clark on this episode 261 of talk Python to me, recorded April 17 2020.

00:47 Welcome to talk Python to me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter, where I'm at m Kennedy. Keep up with the show and listen to past episodes at talk python.fm and follow the show on Twitter via at talk Python. This episode is sponsored by linode. And Ruben Lerner. Please check out what they're both offering during their segments. It really helps support the show. Andrew Clark, welcome to talk

01:13 Python to me. I'm Michael, glad to be here. It's always a pleasure, listen to your podcast, and really excited to get to talk to you today.

01:19 Well, thank you so much. And I'm excited to talk to you as well. It's gonna be really fun to talk about machine learning. And I think of all of the programming type of stuff out there, more than most machine learning is truly a black box. For a lot of people. The decisions that it makes is sort of unclear. It's sort of statistical, I don't know, to me, it feels a little bit like quantum mechanics over like, Newtonian physics or whatever. Like, I know, there's like rules that guided, but it's kind of just it does its thing, and you can't really know for sure. But at the same time, the world is beginning to rely more and more on machine learning, right?

02:00 Yes, definitely. And there's there's definitely different levels of the complexity of it, and how much is deterministic and how much of it really is quantum mechanics.

02:09 Nice. All right. Now, before we get into that, though, of course, let's start with your story. How did you get into programming and into Python?

02:14 Well, definitely I, my undergrad, I was actually an accounting major. And then about halfway through, I realized accounting is a great career. But it's really boring for me. So started trying to just get into programming and stuff. Big Data was like the craze when I was in college, everybody's talking about big data, and Hadoop and all that kind of kind of thing. So I really started wanting to learn more about it and work through Code Academy and learn Python the hard way. And some of those, those other books that are around, try a bunch of different things to see what worked with for me and learn Python the hard way. So it really seemed to stick. And then really started just trying to find a little side projects of little things to do to actually build the programming. So like I tried doing the top down, like, here's a theory more computer science II approach didn't really work for me some of the bottom up of like, I'm trying to solve a problem, I want to move this file to this location. And I want to like orient this image in my Excel file these like different things like that, to help learning how to program Do you

03:04 think that program is often taught in like in reverse in the wrong way? Right? It's like very high level, and then we're gonna work our way down and down. And like, here's the concepts and then here's the theory. And instead of just like, here's a couple problems, and a few things you need to know, and a whole bunch of stuff that we're not gonna even tell you exist about, right? Like, you don't even know what a heap is, I don't care, we're gonna just make this work until you like, eventually care what like, you know, memory allocation is right, at least I've found, especially in the academic world, I'd felt sort of flipped on its head.

03:34 I definitely think so a lot of careers are built that way. It's like you need to have some sort of a concrete of why does this matter? Why do I care? And like being able to do get your hands dirty, and then go into the theory. So you really want to do both in tandem. But really starting with the hands on? He's really helped me with that approach. I know some people can do at the top down theory way, but I definitely think computer science could have a lot better programmers faster if they did it from the bottom up.

03:56 Yeah, I agree in computer science, or programming in general, is, is nice in that it has that possibility, right. Like it's harder, you know, we joked about quantum mechanics, it's harder to just like, well, let's get your hands dirty with some quantum mechanics. And then we'll talk about like, wave particle duality, whatever I get just right, it just doesn't lend itself to that kind of presentation or that kind of, like analysis, I don't think but programming does, right. You could say we're gonna take this little API, I'm gonna build a graph. And how cool is that? Now let's figure out what the heck we

04:26 did. Exactly, exactly. So that's one of the beauties about programming, especially Python, with its ease of starting up and just getting something running on on your computer, you can really start dealing with problems you wouldn't be able to do in some of these fields until he had a postdoc, you can't even start talking about quantum.

04:41 Exactly. So back to your story, or you were talking about like this was the way that you found work for you to get.

04:48 Yeah, and it really worked well with the different internships I was doing at the time. So I was working or in financial audit. And during that time was really when auditors were starting to think about using data and using more than just normally in auditing, you do random sampling. So you'll look, I look at a bunch of transactions from like accounts payable for like, these are bills that company paid. And being able to find that there's a lot of times there's duplicates, some, some people are paying invoices twice and things like that. So being able to use programming to start solving just audit problems to speed up the process from my company and is making my life easier, honestly. It's really how I started getting very excited about Python besides just like very small little utils on my computer. So I like seeing what can we really do with this? And how can we make solve business problems with it? So I really came out of that pragmatic way. That's cool.

05:32 Did you find that your coworkers, other people doing trying to solve the same problems, we're trying to solve it with Excel, and you just had like this power that they didn't have?

05:41 Definitely, definitely Excel and some like specific analytics tools, like there's a couple out there for auditors that like CaseWare idea and ACL and things and they're just not as easy to use, they Trump Excel, but they have a massive learning curve themselves. So it's like being able to do something like Python, I can hit both camps of different types of otters, either Excel or or ACL auditors, were able to just run circles around them with with Testing, testing.

06:06 That's pretty cool. Well, that's a really interesting and practical way to get into it. I've recently realized or come to learn that so many people seem to be coming into Python and into especially the data science side of Python from economics.

06:20 Yeah, there's a big move these days from people traditionally in economics would be using like matplotlib. And no, sorry, not pythons on the brain here, MATLAB and moving in to do more quantitative analysis. And now really, Python has kind of taken over in economics. And there's like this a ton of people that are coming over to the data science from those types of modeling for econometrics and stuff. So there's definitely a big surge of, of economists turn data scientists these days. Yeah. Cool. What are you doing day to day now? Like, he started out doing auditing and then getting into Python that way? Now what? Well, it's been it's been a really crazy ride. But I'm now doing a startup called monitor that works on machine learning assurance, like how do we provide assurance around machine learning models. So I'm the co founder and CTO within so my day to day right now. And we're currently in the midst of TechStars. Boston is very much just like meeting with different investors meeting with their startup founders meeting with client potential clients and running different parts of business. So right wearing a lot of hats right now. And then I run all the r&d type side and figuring out like, what are the things we want to build? And then I can start working with the engineering team to execute, do you

07:24 find that you're doing all these things that sort of a tech background, wholly unprepared you for?

07:30 There's definitely some of that.

07:33 In my accounting degree is definitely coming in handy. Yeah, a lot of the business side things, but there's definitely a lot of moving pieces. And it's more than just strictly like being good Python programmers. So it's a little more than artifacting. On that side.

07:45 Well, I do feel like a lot of people. And this is like, my former self included is like, if you build it, they will come all we need is like really good tech, and like a compelling project or technology or product. And then we'll figure out the rest of the stuff. We'll figure out the the accounting, the marketing, the user growth, like all that stuff. It is like no, no, like, the product and the programming is table stakes. all this other stuff is what it actually

08:12 takes to make it work. Right? Definitely, definitely, you'd have innovative tech, but that's only a component of it. Like, that's like one fourth of the problem. And you know, for I think a lot of myself, and I think a lot of first time founder startup individuals as well, you come into that thinking like, hey, tech is going to be king, when there's actually so many other moving pieces like here, like you mentioning, so it's an eye opening experience, but I really respect founders more.

08:37 Yeah, same. Well, I do find it really interesting. And at me almost disheartening when you realize, like, I have all this amazing programming skill. And I know I can build this if I had two months, and I just really focused we can build this thing. And you realize, like, actually, that's just a small part of like, having it built like this is where it starts. This is not like basically, it's ready to roll. So I don't know, it's, it's something I've been thinking about over the last few years. And it's, it's not the way I think a lot of people perceive it. starting something like this, if you have like, we're gonna get a cool programming team together want to build this thing, it'd be great. It's like, Okay, well,

09:12 that gets you started. But then water, right, exactly. And the other problem you have with a startup is like you can't even do, okay, give me two months ago can build it. Like with all the other competing business things and stuff. You can't even get that straight two months of just building because you have all the investor obligations and things. So it's definitely a lot of things to juggle. So you get really good at time management and priority juggling.

09:32 But it's it's exciting, right? It's a lot of fun.

09:34 Yeah. Oh, it's it's fantastic. And being part of TechStars is really wonderful. It's, I wouldn't trade it for anything. It's just definitely eye opening and a great learning experience.

09:42 Yeah, I've had some people on the show a while back who had gone through TechStars. But maybe folks haven't listened to that show or whatever. So I probably the most popular well known. Something like that is probably Y Combinator. I don't know that it's exactly the same thing, but it tells me About what techstar is a little bit just so they have a sense of what you're working on or what you're doing.

10:06 So TechStars is an accelerator. There's like accelerators and incubators, I didn't really know there's much of a difference. But being part of TechStars, the accelerators are really very mentor heavy, like the first couple weeks of TechStars, you have a ton of different mentors that you meet with, like five a day or more in trying to like find, like, Who's going to help guide the vision of your company? How you get the right people around the table, like what's your company vision, your mission statement? How are you going to do fundraising? What's your marketing strategy? What's your go to market? How do you get your first few customers? How do you keep those customers? It's very focused on the execution and like, how do you make a successful business that will last? So it's a lot of very business II things for a tech run, sort of startup company thing? It's very, very focused on the non tech aspects, the business aspects, the how do we differentiate ourselves from the crowd and things? So there's seminars, mentor meetings, there's just lots of moving pieces with that. But it's, it's a very, very good thing. And it helps in about three months to keep take your company from like a cool idea. And you have a product to really being able to execute your vision and, and have a better staying power. Yeah, well,

11:06 it sounds like it's a lot of stuff to sort of build that structure in support of what we were just talking about, if like what you don't get if you know how to write code and create a product, right?

11:15 It's exactly right, because like tech stars 10 companies are accepted out of over 2000 applicants. So it's like, you already have a product, you already have good tech to make it across the bar. They're like, Okay, everybody knows what they're doing how to build tech, now, we got to teach you guys how to actually run a company. So yeah, it's a very, it's a very good way of approaching it. It's not like you'll sit in a room with a bunch of groupthink and programmers making it more geeky, you think like, okay, you might even need to scale back the complexity of this a little bit. So we can actually market this to the user. It's very, very good program for small companies go through

11:48 this portion of talk Python to me is brought to you by linode. Whether you're working on a personal project or managing your enterprises infrastructure, linode has the pricing support and scale that you need to take your project to the next level, with 11 data centers worldwide, including their newest data center in Sydney, Australia, enterprise grade hardware, s3 compatible storage, and the next generation network linode delivers the performance that you expect at a price that you don't get started on the node today with a $20 credit and you get access to native SSD storage, a 40 gigabit network industry leading processors, their revamped Cloud Manager at Cloud linode.com root access to your server along with their newest API and a Python COI just visit talk python.fm slash linode. When creating a new linode account, you'll automatically get $20 credit for your next project. Oh, and one last thing they're hiring go to lynda.com slash careers to find out more, let him know that we sent you. Well, I want to talk a little bit more about that at the end and how it's probably changed as the world has changed with COVID-19 and all that. But let's talk about machine learning and stuff like that first. Yeah, definitely. We talked a little bit about machine learning. And I joke that like, at least to my very limited experience, it feels a little bit quantum mechanic see in that same sense. But let's tell people, I guess what is machine learning? Obviously, we have a bit of a sense, but like, let's try to make it a little bit more concrete for people who maybe it's still like a buzzword. And what makes up machine learning, I guess,

13:24 definitely. So I'll start by saying there's a, there's a lot of terminology when people kind of interchange together like AI and machine learning, deep learning. So if you think of three big circles, AI is just a broad range of trying to make machines to copy him human tasks. So AI doesn't have to exactly be modeling. It can also be rule based systems and things that try and mimic human intelligence. So it's like a broad fuel of trying to do something that copies human behavior, right

13:50 computers, making decisions,

13:52 kind of, right, exactly, trying to make decisions like a human. So there's a lot of different components in that. And it's been a field that people been talking about for a long time, and you've had movies about it for a long time. So it's just a broad field, there's very many components a part of it because you have some things like expert systems and, and some sort of rule based engines that actually do a pretty good job of mimicking that when there's not actually any modeling. So they have a limited use, because like self driving cars, if you think about that, you can't build enough rules. To specify every time a car should turn, there's too much going on, you have to use something more complex than that, by itself doesn't mean modeling. Machine learning is types of statistical models that learn from data to make decisions. So that is just the broad field of modeling. That is what most people think about when you think about machine learning. And it's just the modeling aspect. And we can go into the classes of models you have like class supervise, which is basically you can have a model look at a bunch of data that has a decision. So like if you look at a bunch of pictures of cake, and it says it will be cake and then you look at another picture that's a bunch of raw ingredients. It'll say it's not cake, you can train an algorithm to look at those two pictures and make the correct decision. Optimizing a function, right? Where you have you already know the answer.

15:04 And you kind of tell us like a kid. That's right. No, that's wrong, right? Oh, good job that was right, you know, and so on. Yeah. And it takes that feedback sort of iteratively. And like, evolves itself.

15:14 Exactly. And it's extremely, extremely dumb. So you have to just have a lot of training data that is balanced, and meaning you have the same amount of pi, not Pi type scenarios. So it knows how to learn from that and have very catered datasets that don't generalize very well. On the far other extreme side of things, you have completely unsupervised learning, which is basically, let's throw some data at this algorithm and see what patterns emerge to groups happen. So it's very good to use like for very early on exploratory analysis, of like, I have a bunch of data, I have no idea what to do with this, let's see if there's anything interesting here. But in between the supervised and unsupervised, you'll have like supervised learning or semi supervised semi supervised learning and reinforcement learning, which basically what they do is, it's a combination of those two, where you have not quite explicit labels on everything. But it's not completely just sending an algorithm out in the unknown to figure out what's happening. And that's where you've seen like, a lot of AlphaGo, and a lot of really cool problems have been done with reinforcement learning and semi supervised learning. And there's a lot going on in that space as well.

16:15 Yeah, well, I think that's a pretty good overview. And the unsupervised side, that's probably the realm of where people start to think, you know, computers are just learning on their own. And what if they get too smart, and they just, you know, take over or whatever. But that's kind of, I would say, that's probably the, almost the science fictiony side of things. But it's also real, obviously, you are doing it, but that's probably what people think of when they think of AI is like, there's data, the computer looked at it, and then it, understood it and learned it and found things that humans maybe wouldn't have known about, definitely

16:50 some of the continuous learning stuff, which is a part of a subset of that. Yeah. And then the very minor part of deep learning was basically as a subset of machine learning, which is looking at trying to copy how your brain synapses work. And it's just a type of machine learning where you think of a convolution neural networks and things like that. So there's like the three big tiers and the different parts that fit in between those. Yeah,

17:09 and when I was talking about quantum mechanics, analogies, I was thinking of deep learning.

17:13 Exactly, exactly. And like deep learning is extremely valuable. A lot of the really cool research problems, you're seeing people develop open AI, and all these people are using that it for doing reinforcement learning with deep learning and things like that is not what a lot of companies are using today, in regulated industries, or even in most of most of America, you know, a lot of companies are using more traditional supervised and just even getting past linear regression, in a lot of instances, the deep learning stuff that science fictiony people are using, but it's not very widely deployed yet. Because there's a lot of problems with the deployments. Yeah. And probably even just knowing with certainty that it's right. And that is reproducible,

17:49 completely. Yeah. Yeah. I think that the, you know, it's such a buzzword, machine learning and AI that, right, if a lot of people seem to think, well, even if just an if statement is in there, like well, the computer decided, so that's AI, it's like, that's not really what that means. But that's okay. The computer decided, yes, you're right. Because we're having statements for a super long time, I was thinking back to something in the British Parliament where they were upset that there was some airline, which is bad that some airline was looking at when people would book tickets, if the people had the same last name, they would put them in different seats away from each other, and then charge them a fee to put them back next to each other. And they were saying that there's this AI system that is causing this, there are a view that this is not AI, that's just an if statement, if the names are the same exact part and offer them to upgrade to get it back together. Exactly. Yeah. So I think it's interesting to think about how companies can apply machine learning, and maybe the hesitant hesitancy to do so. Right? Like if, if I'm applying for a mortgage, and the company is using, you know, some deep learning algorithm to decide whether I'm a good fit to provide a mortgage to right. Some states, the United States have like walkaway laws, right? You just decided, like, I don't want to pay any more. And you can have the house even if it's like in crappy condition and not worth what it was right. So it's a big reason. So they want to get that answer, right. Yeah, exactly. But at the same time, I'm sure there's regulations and stuff saying you can't just randomly

19:28 assign or not assign or have some super biased algorithm, just making this decision and say, well, it's not our fault. The computer did this, you know, unjust thing or whatever. Right? Exactly. And that's what a lot of insurance companies for instance, are struggling with right now is there are laws and lesson like fairness and and you can't do gender discrimination or those sorts of different protected classes discrimination, but there's not a very hierarchical full regulation around machine learning right now. So a lot of companies are kind of in the dark on like regular don't really like machine learning. They don't understand it. They don't like the non deterministic aspects or like it. They like rule sets, right? So if this happens, we'll do this, they are very comfortable with having those sorts of rules. But having this kind of nebulous modeling example,

20:14 it's easy to do a code audit and say, Okay, this is the part where they tested, like their income to credit card debt ratio, and it was above the threshold, you offset. And so you said no, and so it's because they have too much debt. And in fact, I can set a breakpoint. And I can step down and look at the code, like making those decisions and comparing those values. And that here's what it does. Right. And as far as I know, like breakpoints and deep learning don't really mean a lot.

20:47 Yeah, for because a lot of times, if you use like a support vector machine algorithm, you won't have a linear decision boundary, meaning that sometimes if you have income over this amount, and this other thing will give you a loan, and other times we won't, it depends on what some of the other inputs are. So that's where regulators get very nervous, because you can't have that strict rule based system.

21:05 Yeah, but at the same time, this is an amazing technology that could make, you know, make those decisions more accurate. And it could be better, right? It could actually understand like, yeah, okay, so you do have a lot of credit card debt, but you're doing this other thing that really shows that you are a good borrower, right? their code would have completely ignored, but the machine learning is like no, no, that actually those type of people are good. You want to loan to them, right? You want to lend to them? Exactly. Because it could be good to have this right. It's not just a negative, it's just if you just don't know, that's a part of the challenge.

21:39 Exactly. Because you could have be able to do more targeted rates for consumers. And then they can have lower rates the customer, the the insurance company will have lower default weights, basically everybody wins. The problem is for for executives and regulators is how do you know that this, how do you have assurance and trust the system is doing what you want to do, and you're not going to end up on the front page to like the apple credit card thing a few months ago with Oh, you were really discriminatory or you miss messed this up. So the problem is, we're being able to provide that assurance and trust over algorithms to allow them to be deployed. And that's kind of where the industry is struggling with right now. Because no one really knows quite how to have that with these deterministic qualities, like how do you get that assurance around the model doing what supposed to be doing? So that's where there's a lot of work right now on the different machine learning, monitoring you could doing? Do you need a third party to monitor those sorts of things. So you can provide that trust and risk mitigation to be able to deploy the algorithms that are beneficial for both the company and the consumer?

22:32 Sure. And you know, we're talking about America, because we both live here and whatnot. But I think actually, Europe probably has even stricter rules around accountability and traceability definitely in for my, my cursory experience of both,

22:47 they definitely in Europe actually doing a better job of first regulating it right now, and also providing guidance around the regulations. So GDPR has several, as the general data privacy act in in Europe has some regulations around like, if a consumers decision has been made fully automated, such as the example you just gave you, the consumer needs to know when they can request the logic on how that decision was made. So there's a lot of user consumer protections around machine learning algorithm use in Europe. And there's also a lot more guidance, Europe is now a little behind on they're creating a white paper on guidance around how would you make sure the album's are doing what they should be doing to help the companies but Britain, the UK has Information Commissioner's Office office has released two different white paper drafts on how you do explainable AI, how do you make sure you have the assurance around your algorithms and actually doing some prescriptive recommendations on how to do that correctly. So like Europe is really ahead of the us right now on first regulating AI, and then also helping companies and consumers understand it and find ways so they can still use it. But us hasn't really addressed those yet.

23:50 Yeah, yeah. And it was exactly that situation that you were talking about. If a computer has made a completely automated decision, the consumer can request. Why. Right, exactly. And that was like thinking around this entire conversation like that is the the core thing that I think is at the crux of the problem here that made me like interested in this because the deep learning stuff, and the ability to get a computer to understand the nuances that we don't even know exist out there and make better decisions. I think that's, that's really powerful. And as a company, you would definitely want to employ that if it's required that you say, Well, it's because your credit to debt ratio was too far out of bounds. Like, how are you going to get a deep learning, like, forest sort of thing can go on and on and go, Well, it said this, it's like, well, the weights of the nodes were this and so it said no, that's not satisfying or even meaningful, right? Like, I don't even know how, how did we get there? Maybe we get maybe, ultimately people would be willing to accept like, I don't even know maybe there's something like okay, it it's made these decisions. And if we had these false positives, and we had these false negatives, like, almost like an A B testing or clinical trial type of thing, like, we're gonna let 10% of the people through on our old analysis anyway. And then compared to how the machine learning did on both, like the positive and negative, I don't know, but like, how do you see that there's a possible way to get that answer.

25:21 Yeah, there's a couple of approaches, one of the main ones has been a lot of my research the past couple years, which is you can provide assurance around the implementations because like, a lot of the points that you just mentioned, we don't have that ability with a human usually to understand why did a loan officer get that loan either. Like, there's that type of understanding some people are asking from algorithms doesn't really exist with humans, either. If you ask a human Two weeks later, why you made that exact decision, they're not going to say the same thing that they were thinking at that time. So you want to provide a trust and an audit trail and and transparency around an algorithm basically give it a history and show that it's been making reliable decisions, and it's operating within the acceptable bounds for the inputs and the outputs. So being able to provide this holistic business understanding and process understanding is very huge. It's very, it's not really as much of a tech problem as it is a business problem and a process problem, but also be able to provide the ability of this is what the algorithm saw when it made this decision. So for even a deep learning algorithm, say you are taking in FICO score, and you're taking an age and you're taking in zip code, and things like that, to make this decision. Some of those are protected classes, but you're taking in information about a consumer to understand you don't need to see like what neural network layer said something it's like based on these features, because your your FICO score is above 600. And because your zip code was in this high income area, we're going to approve this loan or not approve this loan, so that you can see with some research anchors is a great library that was open sourced a few years ago. It's there's Python implementation and Selden alibi has a library that's really, really good that has a production guide implementation can help you see what did the algorithm see when it made this decision? So you can start addressing that inside the whole bigger process of providing business and process understanding? I see. So there's all these different facets or features, yes, that come in. And you can say, you can't exactly say, you know, here's how the flow went through the code. And here's the statement, but you can say it keyed off of these three things that keyed off the fact that you have three cars leased, and you have this credit cards, or whatever, right? Like I said, those are the things that, like sort of helped it to sign a decision on. Exactly, exactly. So when you put that in tandem with understanding the whole process, providing the ability to go back and verify, you can start getting more assurance around the implementation and start getting comfortable with it. Because that's the other problem a lot of times with these algorithms is like it makes a decision. And as a company, like how in the world if a customer asks, Why did you make this decision about me six months ago, how are they going to go back and get the exact log file, make sure you have the exact model version, be able to rerun that specific model version, and see all these types of information information. It's not strictly just a tech problem of Let me see which neural network layer was activated and which neuron like it's, it's more process based and, and there's a lot a lot of components to it. So that's where a lot of times we see right now in the data science community, people trying to solve this problem by building better explainability by understanding that neural network better. But that's not really addressing the consumers ask on like, why did this make decision happen? How do I know that? You just didn't arbitrarily make this? How do we know what's not biased? Those sorts of more fundamental issues? Aren't something you can just address by better neural network explainability tool? Yeah.

28:29 Do you think there's somewhere out in the future the possibility of like a meta network, in the sense that like, there's a neural network that looks at how the neural network is working, and then tries to explain how it works? Like, use AI to like, get it, get it to sort of answer what the other AI did,

28:46 there's definitely some things like that currently, in research, there's a whole bunch of different good explainability libraries out there, and people that are addressing specifically those types of problems. And that's monitor is doing that as well, except for more the business process side and the basic understanding about the model, it's really to get the consumers comfortable, it's going to be understanding and be able to prove why something was done, which is more than just specific deep learning interpretability. Because a lot of times these loan models and stuff are still like we're moving from a loan officer to trying to do a very basic machine learning model. Like they haven't even gotten to complexity of deep learning. So it's like incremental improvements,

29:21 right? You talked about the origins being in the Big Data era. And all that, is there some sort of statistical way that people might become you think comfortable with things like deep learning making these decisions like so for example, you've got all of the records of the mortgages in the US, if you were able to take a gear algorithm and run it across those as if it was trying to make the decision. And then you have the actual data, kind of like the the supervised learning, you talked about going through and saying, okay, we're gonna apply it to all of this. Maybe we don't share that data back with the But they give us the model, we run it through all of it. And it's within bounds of what we've deemed to be fair to the community or to the country.

30:09 Definitely. So that's, those are some definitely tests we can do. And ideally, if you're building a loan out algorithm, you're going to want to see what those historical statistics are to make sure that our our model is doing, say classification percentages to be in line with what's expected for the general population. So for instance, if you're doing an algorithm, if you have basically this, at certain cycle bands, by ko score bands, you're going to have x amount of acceptance. So if you start seeing your model starts really not accepting people that are in the certain range, like we can definitely start raising a flag those concept drift going on. So there's definitely a test you can do around there, there's a fisher exact test, they'll let you check, say, if you have age, you don't really want to be using age as a as a indication or gender, for instance, but in some instances, you have to, you can run a test to see if it's ever statistically significant, that that one variable is negatively influencing the outcome, there are definitely ways you can make sure that the algorithm based on the example you said with the loans, this is the amount of normally acceptance, non acceptance that we've had over the past X amount of years in America for loans. And that's kind of what we think is, is okay, there are definitely tests you can do such as the Fisher exact test, it's a statistical test and others that you can do around making sure an algorithm isn't biased, and it's going within that percentage that you're wanting it to do for acceptance. So there's a lot of tests that companies can do there, some that we're implementing some that other people are implementing, and a lot of things that people can do, but really, for the public to really start accepting that machine learning is okay, I think there really needs to be some sort of more government regulation, it's at least even like rubber stamping that this is okay. And we've looked at this algorithm algorithm, there needs to be like third party assurance. And third party audits of algorithms are not even you don't even have to share your code, if it's like it's a proprietary thing. But just the understanding that this is doing what it should be doing. And it's not discriminatory, me very important. Yeah, people to be able to trust AI systems, I think it's gonna be super important. And when I would throw that idea out there, what I was thinking of, was the X PRIZE stuff done around health care, and breast cancer, where instead of getting all the data, you submitted a Docker image with your algorithm

32:12 that was then run and it couldn't communicate out, it was run against the data, and then you got just the trained model out of it. And then you could basically go from there. So like, there was this sort of arbitrage of your model meets the data, but you don't ever see the data and no one else sees your model kind of thing.

32:30 Definitely, it just somebody needs to facilitate that. That's a trusted party. So at some sort of government regulation to enable that kind of thing. But those sorts of processes will definitely start allowing people to trust the system and allow state regulators and things to be able to be signing off on systems being comfortable to let the public enjoy better insurance policies. All right, well, we need to have some sort of assurance to get there.

32:54 This episode of talk Python, to me is brought to you by me Rubin learner and weekly Python exercise, you want to do more with less code or just write more idiomatic Python, it won't happen on its own or even from a course, practice is the only way. Now, in its fourth year, weekly Python exercise makes you a more fluent developer, between pi test tests, our private forum, and live office hours, your Python will improve one week at a time, developers of all levels rave about weekly Python exercise, get free samples and see our schedule at talk python.fm slash exercise.

33:31 So you talked a little bit about some of the Python libraries that are out there to help people understand how these models are making decisions and understand a little bit better. Some other things that are probably we could talk about some of the other things you're doing and you know, some things that might matters, like you talked about the company, that if I asked them six months later, why did you decide this thing, right, they probably still have all of my form fields, I filled out like my credit history is x, my average income is whatever, I've been in the job for this long. But how would they go back and run that against the same code? Right? So maybe what they would do is they would say, Well, let me just ask the system again, what would the answer be now and why? But they may have completely rolled out a new version of code, right? It might not even do the same thing. And I'm sure they don't have like retroactive versions of like the entire infrastructure of the insurance company around to go back and run it. Exactly. Maybe they do, but probably not in a lot of companies like the ability to almost a version control, but like production, right? So knowing how the model changes over time, like what else do you guys need to look at or keep track of to be able to hit like, give that kind of, you know, why did this happen? Answer

34:51 definitely. And that's the key part is summarizing all information and being able to replay that exact algorithm with the decision. So to be able to do that you really have to have Exact feature inputs, like what exactly did the user say the exact model version will need to know the select the model object file, the pickle file, something like that the exact production Python code, then you have to have the actual version of the library used on in the same environment. So there's a lot of things going on there and the amount of logging and that you have to have in place there and have it very easily to be accessible and non tampered with and be able to recreate that environment. That's a hard technical problem. A lot of companies don't have, in addition to just having like, though, how to understand that the exact some sort of interpretability for the decision, and then the metrics and monitoring around it. That's a big ask. And that's where a lot of companies are struggling when they start hitting these regulatory audits.

35:39 Sure. Well, I can imagine, it's really tricky to say, Alright, well, when our app started back, then we went back and we looked in the logs, and it said, This version of this version, this version of the library, but maybe somebody forgot the log, oh, yeah, there's a dependency library that actually matters. And it was that version, or we were running on this version of Python. And its implementation of, you know, floating points slightly, slightly changed. And it had some effect, or I, you know, I don't know, right, like, there's just all these things. And it seems like there's probably some analogies here, to the reproducibility, challenges and solutions that science even has

36:17 exactly, it's, it's along the exact same lines. But it's even more exacerbated and more difficult to solve. Because if you think about science, it's like trying to reproduce the results of one paper, you know, if it's a biological experiment, it might be a little harder to recreate those conditions. But in computer science, like you should be able to save the seeds in a Docker file with everything and rerun your algorithm results, right. But people have been having a hard time to do that. Because the amount of process and forethought into how you're going to be packaging things, how you're going to be setting things up a lot to deal with. But like reproducibility is it's a major crisis in science right now, too. But it's really hitting the corporate environment really hard to be able to provide that kind of ability around the algorithms to be able to answer questions, because if like, you want to implement these things, people are going to want to have audits, and they're gonna want to understand why something was done. So exactly what's happened in the scientific community, except even on a larger scale.

37:04 Now, two things you talk about when you look at monitors website talking about what you all do there. One is counterfactuals. And the other is detecting when model and feature drift occur. You want to address those problems a little

37:20 Yeah, definitely. So counterfactuals is basically what we've just described is the performance of an exact transaction. So we record all those things, the versioning of all the files. So when you go back six months from now, six years from now, and go and select on a specific transaction, and if it's a tabular transaction, we can actually hit a button, and we will pull all that stuff up in a Docker container and rerun it. So that's what we're calling counterfactuals is the ability to go back and re perform a transaction and then perform what if analysis, say like, one of the variables said your income was $200,000. And if you want to change it to $150,000, you can go do that and rerun the transaction as well off of that old version. So allows you to sensitivity analysis, if a consumer is like, well, what if my income was slightly different, but also we perform for audit tracing that it's doing exactly what it said it was going to do? Sure.

38:09 Okay. And then model drift?

38:11 Yeah, model Jeff is exactly like what we were talking about a few minutes ago, with the use, you have loans, they normally 60% of loans are rejected, 40, or accepted. And that's the kind of the average for this risk class, model drift will allow to see and onto our platform when your model has started to drift out of those bounds of you say I'm okay, if the model is between 75% classification and 50% classification of you don't get along and projection. And if we start slipping to 80 80% of loans that are being rejected, we're gonna throw you alerts and say, Hey, your model is drifted Something is wrong, busy.

38:42 Yeah. So it's not necessarily detecting that the model is making some variation in its predictions, but it's saying, You set these bounds. And if it's outside of these bounds, like something is wrong,

38:55 let us know. Exactly. And that will be saying that the algorithm is making kind of a drift and when what it's supposed to be predicting, same with features, as a lot of times, there's a great paper a couple years ago by Google called the high debt or high credit card debt or something in machine learning implementations. And the name is not quite right there. But it's very popular paper on technical debt, where it's basically a lot of times when you have an algorithm, a lot of the code around it is where your issues are going to happen. So like if you have a model that's used to having features between one and 10, and you start having a drift with Amina five, and you start having a drift up, that can be affecting the models in the models, outputs, but you can't really detect it until it's too late. So what monitor does is we allow you to look at the feature drift, and see, like, if I'm expecting this feature to between one to two standard deviations of five, a mean of five, and you start getting higher numbers out of there, we'll know like, hey, there's feature drift, which means your model is not in performing the same environment that it was built for. So we'll be able to know that, hey, you need to go look at this. The situation your models built for has changed because a lot of times when these models start misbehaving, it's not because the model code has changed quote unquote Because the environment that it was built for has changed and no longer in that same environment. I see. And that's one of the real challenges of the whole ml story, right is, the model is good at answering the question the model was intended to answer. But it, it may be completely inappropriate for something else, right? Like self driving cars were green, in Arizona, but you put them in the snow, and they can't see the road anymore,

40:24 because they don't know to look at snowy roads or whatever. Right?

40:27 Exactly. Because algorithms are extremely, extremely dumb. Like people are trying to invent them better with this transfer learning and some of the semi supervised learning and things. But when it comes down to the root of it, there is no thinking going on here doesn't matter how accurate it is at detecting cancer in radiology images, there is no thinking it's a dumb algorithm, it's made for a specific set of circumstances, it can be fantastic as long as those circumstances change. But that's where the key problem happens in production is those circumstances no longer hold true to your model sets performing badly, your model is doing the exact thing that was thing was trained to do just the inputs are different.

40:59 I see. So one of the things that might be important is to keep track of the input range, and document that in the early days, and then keep checking in production that it still holds true.

41:13 Exactly. Okay, then that's when we can start testing for bias and things like that, as well. Seeing with like, with one specific variables that we've mentioned, like gender variable, if that starts being a key influencer for decision, we also know something is up there. So you can start doing proactive bias monitoring interesting. So maybe you don't actually want to take gender into account, but it's somewhere in the algorithm or somewhere in the data. And you can test whether or not it actually seems to be influence it like you can detect bias, in a sense, exactly. You choose whatever feature like if you're building an algorithm, you should try never have gender or something like that in there. But occasionally, such if you think of cancer screening, you're going to have to include that because that's a key component, I would argue you should never include gender if you're doing a credit card application. But if you're doing something more fundamental, like cancer screening or something, you still need to be able to have those sorts of things or like, if it's a image recognition algorithm, you're gonna have to include race, just because it may made different skin tones may affect the results of the algorithm doesn't mean we're trying to be any sort of bias. But that's why you want to have these controls, make sure bias doesn't occur, we've had a lot of Radiology implementations that won't work as well, and so on certain, certain individuals. So like, there's all these things that are machine learning can improve everybody's lives, we just need to have the right safeguards in place. Because almost every single company is deploying machine learning. None of them are trying to be discriminatory. It's that's not the purpose. They just there's these things that will happen and they're not aware of. So it's just making sure you have a controlled environment to make sure that doesn't happen, or being able to catch it when it does happen. So you can fix it. Yeah,

42:43 that's pretty awesome. What else should people be thinking about that they either need to be doing logging, trying to use libraries to help with in, especially in production with their models.

42:57 Definitely like that the structure of how you do your models is very important. I'm a huge advocate of using micro services and Docker containers and trying to do, especially for these complicated deployments make as many of a service as possible. So it's just like, sometimes you want to have your algorithm is strictly in a container. And it will interact with your logic in a different container. Because when sometimes you have everything combined into one area, is when you can start having like that, that technical debt and things build up. And it's very hard to figure out what's broken, and where is it broken? So being able to keep things as separate as possible in complex deployments really helps to figure out the root cause.

43:32 Yeah, that's interesting. Because if you have it kind of mixed in your app, you probably deployed 10 versions of your app, did any of them actually affect the model? Or was that just like to change, you know, some other aspect of an API or some aspect of the website or something like that, that, you know, is just what we change how users log in, they can now log in with Google, but that didn't affect the machine learning model that we're deploying. But if you have it as a separate API endpoint, that's its own thing, then you know,

44:01 exactly. It's the same thing from like sciences, how do you get rid of the confounding variables and do as much of a control test as possible. So the more you can have those things that you know, what's changing, you'll know what to fix when something breaks. So having those sorts of architectures and really coming in with document, document, document and auditing, because in the next auditor, it wasn't documented, it doesn't exist. Not really true, but that's how auditors look at things. But that's extremely important to do when you're working in regulated industries, or you're working in areas when you're building these models. You need to document all of your assumptions, where's your data coming from all of this, the playing business II things that, frankly, data scientists hate to do? I hate to it as well. But you need to have that kind of thing for being able to, to show other people, different stakeholders and also, frankly, even cover yourself if something comes back later. It's like you here's why we did something and helping you remember if you look at this code six months ago, so yeah, just the documentation, having these sorts of like planned way that your billing algorithm instead of just agile lying, like agile is great, but you have to have an overarching plan. land for some of these things instead of just in the pit until it hits production? Sure. And well, it also, it sounds like a little bit of the guidance is also like, just good computer science. Right? Exactly. It's a very big problem in this space is data scientists are not normally good software engineers. So a lot of the great software engineering practices haven't really translated into machine learning code, there's becoming a big trend towards out machine learning engineers and things. So we're definitely trending in the right direction. But still, there's a lot of models that get deployed, that don't have the engineering rigor that is common in the Python community. Like there's certain ways we do things with ci CD with unit tests and stuff. And a lot of machine learning code doesn't have those things. Sometimes you can't really in a non deterministic outcome, it's hard to unit test, but like there's a lot of the processes of good engineering that we can apply to help make everybody's lives easier and better AI deployments. Sure, I was thinking about that as well in testing. And it just seems like that's one of the areas that is tricky to test. Because it's not like, well, if I set the user to none, and then I asked to access this page, I want to make sure it does a redirect over to the login page, if I set the user to be logged in, and I asked them the same thing. I want them to go to their account page. And so it's super easy to write that test users none called the test check the outcome users this call the test. What about testing these ml models, but it's definitely a challenge. And it's developing art. And it's it's more of a science at the moment and more of an art than a science. But one of the ways you could do it is kind of like how we talk about ranges is you'll say like to unit test the machine learning algorithm like hey, we're expecting a rain between this and this we're accepting these are acceptable ranges. And you just use it there, you won't know if 100% is working, because you'd have to look it over time to see if there's drift. But you'll be able to say like, hey, if it's a regression problem, meaning I have an output between one and 10 and supposed to be a five, if I'm hitting an eight or two, that means something's probably not quite right, you can put some sort of ranges for you unit test. So it can't be quite the deterministic, like, I know exactly this test is failing or not, but at least can give you some assurance and some heads up

47:02 for you're sure, sure. So maybe it feels a lot like testing the scientific things as well, right? You can't say if I call the you know, estimate, whatever, right. And I give it these inputs, all floating point numbers, I can't say it's going to be equal to two, or it's going to be false. It's probably some number that comes out. And you're willing to allow a slight variation because like the algorithm might evolve. And it might actually be more accurate. If it's a little you know, if it's slightly better, but it's got to basically be within a hundredth of this other number, right? So you got got to do like range, like it's in this range. And it's not too far out. I guess it's probably similar. What do you think about like a hypothesis, or other sort of like automatic property based testing, where you could say, give me some integers here in this range, and give me one of these values and sort of give me a whole bunch of examples. And let's test those,

47:57 that's definitely a very good sensitivity analysis is a very good way to do that. So like, ideally, you'd want to do that kind of testing. So it's better, you can't really unit test as well, like we talked about that. But sensitivity analysis, testing is fantastic to do, which is like here are the different scenarios here, the different users, we would get, and run a bunch of them with slight variations through your model and see if it performs as it should. So that's definitely a very good way to test a model. And you should never deploy a model without doing that. The other test that you can kind of do is you can't really unit test machine learning code to accurately but you can unit test data. So there's a couple good libraries out there that will help you unit test data right now, I think marbles might be one in Python. And I think there's great expectations that you see like is the schema, right? So that's really huge, because like, this is supposed to be an int? Or is this supposed to be a float or like a float between these numbers and things so you can really check the input data to? So there are some tests, you can do a little more accurately around your data, in addition to the sensitivity analysis,

48:52 yeah, great expectations. It's cool. It's got things yeah, expect column values to be not null, or expect column values to be unique, or expect them to be you know, between such and such values. That's pretty cool.

49:05 Yeah, so it's very cool library. Great to do that for the testing.

49:08 Yeah, I suppose if you have some sort of, you talked about unexpected situations as like, it's you know, my analogy was built for drive, Sunny roads, not snowy roads, an automatic car driving or auto, automated driving. But similarly, it's probably not built to take an all or none. spot where it expected a number. Right What you're gonna do with that, right,

49:31 and that's, it's great to do the looking for the using the tele example Black Swans, like it's good to test your model for when something bad is gonna happen. Like even just to do the correct. Try accepts type statements around it for like, here's something crazy, that should never happen. But let's see what happens anyway, so we can handle that. So we don't become the next news story with this company really screwed up type scenario. So it's good to do those types of tests that are just like we hopefully never have this but what happens if we can build the types of exceptions in law around those sorts of crazy scenarios, sounds good.

50:03 Sounds like a lot of things that people can go figure out, you know, find some of these libraries, or to add some of these techniques. I mean, you guys over a monitor are doing this, you're not quite out of TechStars yet. You know, people can check you out. What else would you recommend that folks maybe look into to make this practical in their environments,

50:23 so definitely those those two great expectations is great for like unit testing your data. alibi is a library that has anchors implementation, I was talking about anchors that explainability library, it basically gives you if statements online via a transaction was done. It's kind of a 2.0 from line, which is a very popular data explanation. Those are some good ones. shap is a great library as well, that gives you It's game theory based for giving you values of what's what how your algorithms interpreting a decision. That's cool. Yeah. So it's a very, very cool, cool library. And then it's just really starting to apply. So that's all of the like, explainability type scenarios, pi metrics, has a very cool library that does some bias detection. That's something that's really cool to check out. quantis is a bias and fairness audit toolkit that is really cool. And that's out of the University of Chicago. So those are some good ways to get started on how to provide more assurance around your models. And then just really, you want to be doing Docker, you want to be doing best practices on your engineering. Yeah, is also a good place to start.

51:18 Yeah, absolutely. You know, something that came to mind while you're talking, there's this thing called missing, no, missing in O, which is a Python library for visualizing missing numbers and data. And so you can give it interesting, I think, Panda data frames or something like that. And it gives you these graphs with like, continuous lines, if it's everything's good, or like little marks where there's like, bad data stuff. It's it's a pretty cool library for like, try and just like throw your data set at it and visually, quickly identify if something's busted.

51:49 Well, that's very cool. I've not heard that. But I'm gonna go check that out. Yeah, absolutely. Absolutely. Okay. One final machine learning question, I guess, keep it positive. Like, what are the opportunities for this? I, you talked about how many companies are kind of at the rudimentary stage of this modeling and predictability. And there's obviously a lot more way to go with the challenges that the whole episode basically has been about. But what do you see is the big opportunities, you get this, you get these key components, right, you're going to be able to start having companies providing cheaper insurance, better medical screening, I mean, self driving cars, these are all the types of things that are really going to help society, humanity when we get right. There's just a few things like we talked about to get ironed out. But really, like the future is very bright machine learning can do some great things and like together as a community, we will be able to make that happen.

52:40 Yeah. Awesome. And, you know, so you talked also at the beginning about your TechStars experience, and you guys are going through and launching monitor through TechStars. And that's awesome. But I suspect that a bit of a change has happened like TechStars, to me, these incubators, and accelerators all about like coming together. It's almost like a hackathon. But for a company, you come together with all these people. And it's just like this intense period where you're all together and working with mentors and co founders and creating. And we've all been told to not do that. So how's that work?

53:15 Well, TechStars did a very good job of switching to online. So still all the same things happen. They even started setting up some like watercooler sessions and stuff for people that kind of just chat informally. And so they moved to online to zoom very, very early on and did a very smooth transition. So that was very good. They did a great job of handling it. We're having the virtual Demo Day in two weeks, into April, which is it's not ideal, but we're gonna also have a live one in September as well, I believe. So they're, they're really going above and beyond to help with the transition for companies and doing a bunch of extra, like, extra classes around fundraising during COVID, and things like that. So TechStars has done a great job transitioning, and they had all the technology in place to make it a pretty smooth transition. So you know, it's not the same, like the quality is still very much there. And we're still getting a lot out of it.

53:59 Yeah, that's good. I'm glad that it's still going. I mean, it'll be rough. How did you get into it? You talked about, I don't remember the exact numbers, but it sounded like 100 to one applicant to acceptance ratio, or something like that. Like if people out there listening, they're like, I would love to get through something like tech stars, maybe ideally in a year when it's back to it's in person one, but even if even not right, I'm sure it'd be super helpful. Was that experience? Like how did you decide to get get going? And how did you get in there?

54:26 So we did some networking around with from the Boston area, kind of like meeting some of the founders in the area, different VC firms, kind of like early socializing, like who we are what we're about. So kind of got our name around. So starting, like the socialization with the there's TechStars in different cities, kind of like getting kind of the community of founders and startups kind of get yourself known. And then it's the application process. You just need to really rock that. And then this has been part of the community and doing a good application have a compelling story, but definitely the pre networking before the application to kind of like start showing who you are Do as many meetups as you can like, get your name out there. So someone's heard of you when they when the application comes across, it's definitely helpful.

55:06 Yeah, I'm sure they Oh, I've heard of them. Actually, we're doing something pretty cool. I talked to exactly. That kind of stuff goes really far farther than it seems like it should necessarily just like on the surface. But yeah, absolutely.

55:17 There are definitely companies that got into this year's tech stars program that had the word completely cold. So it's not like it's just based on the merit, we happen to be in the area and have the ability to network with people before but honestly, like, if you're in tech stars, or non tech stars, you should be doing what we were doing anyway, socializing your company. So like, they are very, in very sure they're fair company, several of our companies are female only lead. So it's like they're doing a great job of being a fair and inclusive place that's trying to be as non biased as possible with acceptance.

55:46 Yeah, that's excellent. I suspect that machine learning in particular, and data science type of companies in general are probably pretty hot commodities at the moment. And above average chance of getting into these type of things, just looking in from the outside. So even a lot of listeners who particularly care about this episode, maybe they've got a good chance,

56:09 definitely let me know. I'd love to hear your idea as well. Awesome. Feel free to reach out. I'm still connected in the space. So let me know. But definitely innovative AI ml. Companies are definitely big right now. It's just making sure you're you have something fully baked in or solving a real business problem and not just a tech problem. Yeah.

56:25 Very cool. All right. Well, I think we're just about out of time there, Andrew. So I'll hit you with the final two questions. Sounds good. Yeah. If you're gonna write some Python code these days, or editor Steve,

56:35 is the famous question. When I'm writing real Python code. I'm using Visual Studio when I'm doing exploratory data analysis stuff, I'm can't be Jupiter. Yeah, those are my to go twos. I used to be an atom with vim for my real coding stuff. But the newer versions of Visual Studio are just so good. I can't pass it up anymore, even though I like to being a rebel and doing open, completely open source, like gym type stuff. But

57:02 and you know, visuals to your code have such similar origins, right? They're both electron atom came from GitHub, Microsoft now, sort of overseeing GitHub. I like there's, yeah, there's a lot of like Zen between Visual Studio code and Atom these days. So it's not, it's not that wild, right. It's not that yeah, that's cool. And then you already mentioned some really good Python packages and libraries out there. But other notable ones, maybe were thrown out there.

57:28 Yeah, on network x. I really love network x, or doing graph theory and things like that. And it's a very good connection with like pandas and the Python dictionaries to be able to have different nodes and edges and things. So if you're doing any type of network based work, which I've done some previous companies and PhD program, it's a very good library to have. My favorite Machine Learning Library has to be sky cat learn. It's just so easy to use intuitive. The documentation. Like I've never seen a Python library with such good documentation. And so many good examples. Yeah. And then for alibi, I really like as well for like that anchors implementation I was talking about. So those are probably my three at the moment. But the changes weekly.

58:07 Yeah, those are great, awesome. And then final call to action. People want to put their machine learning models into production, or they have them and they want them to be better, maybe what can they do in general? And then also, how do they learn about what you guys are up to

58:20 definitely monitor.ai comm check us out. If you haven't, if you're a regulated industry, and you're wanting to play machine learning model, we offer services to help you do that. And also the Machine Learning Record platform that will allow you to have that audit trail and the counterfactuals. And all the things we talked about in this episode to allow you get past the the auditors and regulators and being able to move those models production. Yeah.

58:41 Cool. All right. Well, it's been really fun to talk about machine learning and watching how it goes verifying what it does. And you know, I think these like you said, these are the things that are going to need to be in place before it can really sort of serve the greater public of the greater companies out there that need to make use of it. Right. So great.

59:02 Thanks. Thank you so much. been great talking to you. Yeah, you bet. Bye.

59:05 Bye. This has been another episode of talk Python. To me. Our guest on this episode was Andrew Clark, and it's been brought to you by linode and ruven. Learners weekly Python exercise. Start your next Python project on the nodes state of the art cloud service. Just visit talk python.fm slash linode li n o d, you'll automatically get a $20 credit when you create a new account. learn Python using deliberate practice every week with Ruben learners weekly Python exercise just visit talk python.fm slash exercise one to level up your Python. If you're just getting started, try my Python jumpstart by building 10 apps course or if you're looking for something more advanced, check out our new async course the digs into all the different types of async programming you can do in Python. And of course, if you're interested in more than one of these, be sure to check out our everything bundle. It's like a subscription that never expires, be sure to subscribe to the show, open your favorite pod catcher and search for Python. We should be right at the top. You can also find the iTunes feed at slash iTunes. The Google Play feed is slash play in the direct RSS feed net slash RSS on talk python.fm. This is your host, Michael Kennedy. Thanks so much for listening. I really appreciate it. Don't get out there and right