Data Science from Scratch

00:00 You likely know that Python is one of the fastest growing languages for data science.

00:03 This is a discipline that combines the scientific inquiry of hypotheses and tests,

00:07 the mathematical intuition of probability and statistics, the AI foundations of machine learning,

00:12 affluency in big data processing, and the Python language itself. That's a very broad set of skills

00:18 you'll need to be a good data scientist, and yet each one is deep and often hard to understand.

00:23 That's why I'm excited to speak with Joel Gruse, a data scientist from Seattle. He wrote a book to

00:28 help us all understand what's actually happening when we employ libraries such as scikit-learn or

00:32 numpy. It's called data science from scratch, and that's the topic of this week's episode.

00:36 This is Talk Python to me, episode number 56, recorded April 19th, 2016.

00:56 Welcome to Talk Python to me, a weekly podcast on Python, the language, the libraries, the

01:11 ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm at

01:16 mkennedy. Keep up with the show and listen to past episodes at talkpython.fm, and follow the show on

01:22 Twitter via at talkpython. This episode is brought to you by SnapCI and Hired. Thank them for supporting

01:28 the show on Twitter via at snap underscore CI and at Hired underscore HQ. Hey, everyone. It's great to be

01:36 back with you. I have something to share before we get to the main part of the show. I had a chance to be on the

01:40 Partially Derivative podcast this week, where we did a short segment on programming tips for data scientists,

01:46 relevant to this episode, wouldn't you say? I talked about using generator methods and the yield

01:51 keyword to dramatically improve performance while processing lots of data in a pipeline.

01:55 If you want to hear more about that kind of stuff, check out the link to the Partially Derivative episode

01:59 in the show notes. Now, let's talk to Joel. Joel, welcome to the show. Thanks. I'm glad to be here.

02:04 Yeah, I'm really looking forward to doing data science, but from scratch today.

02:09 I know. That's my book. I'm the right person to be here for that topic.

02:12 Fantastic. So we're going to dig into data science. And I think your idea of

02:16 taking it from the fundamentals and building something that's not so complicated or so well

02:22 polished and optimized that you can really understand what you're doing is great. But before we get into

02:26 that, let's just sort of start from the beginning. How did you get into programming in Python?

02:31 So if you go back about 10-ish years or so, I was in grad school for economics. I took a class called

02:39 probability modeling, which was a lot of simulating Markov chains and doing Monte Carlo simulations and

02:45 things like that. And the class was actually taught in MATLAB. School had a site license for MATLAB that

02:50 only worked on campus. And so that meant that if I went home and worked at home, which I like to do,

02:55 I couldn't use MATLAB. And so as an alternative, I found that I could use Python and NumPy to basically

03:02 do all the MATLAB-y things without that site license. So that's actually why I started being

03:08 into Python. And then I just kind of liked it and stuck with it.

03:10 Yeah, I can definitely see how you would like working in Python more than MATLAB. I've

03:14 spent my fair share of time in the .im files and it's all right, but it's not Python.

03:20 You were studying economics and that's kind of how you got interested in data science,

03:25 trying to answer these big questions. The subject of economics or some other way?

03:30 I followed sort of a convoluted path. My background was math and economics.

03:34 And I actually started off doing quantitative finance, options, pricing, mathematics of financial

03:39 risk. I worked at a hedge fund. And then when the hedge fund went out of business, I just kind of

03:43 lucked into this job at a startup called Faircast, where I was doing a lot of kind of BI work,

03:50 writing SQL queries, building dashboards. And over the years, I just sort of moved more and more in a

03:58 data science kind of direction and eventually became a data scientist and software developer.

04:02 And certainly the math and economics training was helpful for that. But it was

04:06 it was a career that I sort of ended up in by accident rather than through a deliberate plan.

04:11 That's a little bit like my my career path as well. Studied math and just sort of had to learn the

04:17 programming and stumbled into it. That's cool. What do you do day to day now?

04:21 So I just started a new job. I work at a nonprofit. It's called the Allen Institute for Artificial

04:27 Intelligence. It was founded by Paul Allen, who was one of the Microsoft founders. And we're

04:33 basically doing kind of fundamental AI research. I can't tell you that much about what I do. So

04:38 it's really my second week and I'm still kind of just learning what goes on. I'm learning my way

04:43 around. But I work on a team called Aristo, which is basically building AI to take science quizzes and

04:51 to understand science. Oh, very interesting. Yeah, that sounds like a fascinating place to work.

04:56 It's really neat. Yeah. A lot of smart people, interesting problems.

05:00 Would you say it's different than working in a hedge fund or is it surprisingly similar?

05:03 No, it's totally different. I mean, it's similar in that there's a lot of smart people,

05:07 but it's not really similar in any other way. Yeah. The goals are not necessarily the same,

05:11 are they? The goals, the incentives, the sort of day-to-day stress levels, it's all different.

05:16 Yeah. Yeah. Sounds great. So let's talk about your book a little bit. It's called Data Science

05:21 from Scratch. And I think that's a really cool way to approach it. We have all these super polished

05:28 data science tools. You know, we have them in Python, we have them in R and other languages as well.

05:34 Why from scratch, rather than just grabbing one of these libraries or a set of these libraries and

05:38 talking about how to use them? So a couple of reasons. One is, as I mentioned,

05:42 I come from a math background and the math way of doing things is you can't really use something

05:49 until you can prove it. I once had a teacher who came in and he looked at the syllabus for the

05:58 previous semester and he said, oh, good, you proved this theorem. That means I can use it in this class.

06:03 And so there's this real rigor around, you can't use things unless you understand them.

06:06 And that approach always kind of resonated with me. And so that, to a large degree,

06:11 that's the approach I took in the book. At the same time, you also have all these really powerful,

06:17 really easy to use libraries like Psych and Learn, where you can go in and basically copy and paste,

06:24 you know, five lines of code. And you've built a decision tree or you've built a regression model.

06:29 And it's very easy to not know what you're doing. And you can pick up books and they'll tell you what

06:34 commands to type. But you can also type those commands and again, not know what you're doing.

06:39 And so I thought the book I would like, the book that I would have found valuable would be,

06:44 here's what these models are actually doing behind the scenes. And so then when it's time you apply

06:50 them, you kind of understand the principles, you understand where they go wrong, where they go right,

06:54 what they're good for, and what they're actually doing.

06:56 Yeah, I think that's, that's really interesting. And that is very much the mathematical way of,

07:01 if you can't prove it, you can't use it, sort of a way of thinking, which is not that common

07:07 in programming, you know, if there's an API, hey, just grab it and use it. But I think it's really

07:12 important in data science, because you have all these different disciplines and backgrounds coming

07:18 together to make it work, right? You're not just a pure programmer, and you're not, you know, a

07:24 mathematician or some sort of domain expert, right? You kind of have to blend these together.

07:28 Yep. And the other thing that I think ended up working pretty nicely about this approach was,

07:34 there's a lot of really mathy books about data science and machine learning, that's like,

07:38 here's an equation, here's an equation, here's an equation. But what I ended up doing was a little bit

07:43 of math. But then when it came time to, here's how something works, do it in working Python code.

07:49 So that it's rigorous in terms of here's all the steps laid out. But it's also, that's the code you

07:54 can run and sort of follow along.

07:57 Yeah, absolutely. Why Python and not something like R or some other language?

08:02 So the short answer is that, for whatever reason, R is not sympathetic with the way my brain works.

08:08 And whenever I try to sit down and start writing things in R, I just find that it doesn't work the way

08:13 I expect it to. The joke is that R is a language designed by statisticians for statisticians.

08:18 There's some truth to that, some unfairness to it. But for whatever reason, Python is much

08:24 friendlier to the way that my brain works and the way that I solve problems. Actually, the first draft

08:29 of my book was a lot harsher against R. And then some of the earlier reviewers didn't like that.

08:35 They really don't like that. So I kind of revised it to be more gentle.

08:39 Yeah, I think one of the things that's nice about Python is if you invest to learn data science

08:44 through Python, or the Python data science tools, rather, and then you want to build sort of a more

08:50 working application, you don't have to, you know, convert that to some other language, right? You're

08:55 already in Python. It's sort of a full stack thing you keep working with, right?

08:59 That's one important aspect. I think another important aspect is that from my perspective,

09:03 Python code, if written well, is super readable. And so you don't necessarily have to be a Python

09:11 expert to take a well written piece of Python code and understand what it's doing. Whereas other

09:16 languages can be a lot more impenetrable. And so I find it a nice teaching language from that

09:21 perspective as well.

09:22 Yeah, I would say, you know, most of the universities in the US came to that same conclusion,

09:27 right? With Python being the most common CompSci 101 course these days, as opposed to like Java or

09:33 Scheme, which I took.

09:34 Yeah, I mean, when I took, I also did Scheme when I took CompSci 101 way back in the day. And I actually

09:40 thought that was a really nice language for learning computer science. But I can see why Python would be

09:46 a better language for, you know, learning programming as opposed to computer science.

09:50 So when you start off your book, you actually have a few sections that are just about,

09:55 I'm going to teach you just enough Python to get started so that you can understand the data

10:02 science tools and what we're doing. What kind of stuff do you put in there? Like, what do you

10:06 consider fundamental for doing data science? And what can be sort of learned later?

10:10 The most important things I put in were obviously functions, you want to do any kind of

10:16 analytic work or Python functions. The other thing I really tried to emphasize,

10:20 was writing kind of clean Pythonic code. So I also did a lot of list comprehensions,

10:27 as well as understanding all the basic data structures. So lots of dicks and sets and some

10:34 of the less common ones like default dicks and counters, I also make quite heavy use of.

10:40 So basically, anything you would need to understand, here's how to write out an algorithm in Python,

10:46 here's how to use the correct data structures in Python, I kind of baked into that initial

10:52 introduction. Whereas more advanced things like really specialized data structures like queues,

10:58 I just kind of introduced as needed later in the book, some of the more esoteric math functions,

11:03 I just kind of introduced when they came up.

11:05 I think obviously, nicer dictionaries, things like default dict and so on,

11:09 played a really important role in the subsequent pieces where you're trying to make concise,

11:15 clean statements. I thought your use of sort of list comprehensions and generator expressions,

11:23 along with the other functional pieces like zip and counter and so on, were really,

11:30 really interesting. You write some very concise code without being unreadable. I thought that was

11:35 nice.

11:35 That's one of the things I aim for. And as much time as you spend sort of revising the text of the book,

11:41 I also spent probably just as much time revising the code of the book. I went back over it many,

11:46 many times thinking, can I make this simpler? Can I make it cleaner? Can I make it easier to

11:50 understand? And I try and do that. I'm kind of a stickler for code elegance. It's kind of a

11:57 personal fault, maybe.

11:58 But also, you come out with very polished codes. I found it very easy to follow. And I thought the

12:04 focus on this functional style programming was really neat. So one of the things you talk about

12:10 in the beginning is about visualizing data, because you have all these numbers. And obviously,

12:15 to understand them, putting a picture to it is very nice. And what is that? Matt Plotlib?

12:19 Yes. In the book, I only use Matt Plotlib. I was just having a conversation with some of my data

12:25 science buddies about a week ago, where they were, as always, complaining that the data visualization

12:31 story in Python is still not very good.

12:33 Yeah. So you said that Matt Plotlib was starting to show its age in your book. What did you mean by

12:38 that?

12:39 When I first started learning Python, what I said, more than 10 years ago, as a Matlab replacement,

12:44 like Matt Plotlib was around back then, and it had the same features and the same interface, and it

12:51 produced the same not particularly attractive plots. So it hasn't kind of evolved as the rest of the

12:57 language has evolved, say. And there's been some projects like Seaborn that try and put some prettiness

13:03 on top of it. And there's been some other attempts to kind of bring the R-style GG plot into Python.

13:09 But from my perspective, none of them has really like won the mindshare and become the solution.

13:14 So everyone kind of does something different.

13:16 Yeah. You also talked about Boku, which is from the Continuum guys. I had them on show 34. Can you

13:23 maybe talk about that really quickly? Just what is it? What's it used for? That sort of thing.

13:28 It is another visualization library. And I have to be honest with you, I haven't really

13:32 checked it out since I was writing the book like more than a year ago. So I seem to remember that it

13:38 seemed to have some facility for building in interactive plots and things like that.

13:42 It's kind of D3, fancy graphics in the browser type stuff, right?

13:47 I believe so. But I haven't spent much time playing with it.

13:49 Yeah.

13:49 I haven't been doing a lot of, probably doing actually more D3 than Python visualization recently.

13:55 Right, right. Tell everyone what D3 is. I don't think we've talked too much about that on the show.

13:59 Oh, sure. So D3 is actually a JavaScript library for building interactive data visualizations.

14:06 It brings sort of an interesting model where you bind your data to DOM elements, which are quite often

14:12 SVG elements. And so what that makes it easy to do is to have a data set. And when you add new data to it,

14:21 your plot updates, and it allows a lot of really interesting interactivity. If you check out the

14:26 D3 website, they have a gallery of all sorts of amazing visualizations where you look at and think,

14:32 how on earth did they do that?

14:33 Yeah. Whenever I go to the D3 website, I'm like, wow, I want to use this. I don't have a use for it,

14:39 but it's fantastic. How can I just build stuff that looks like this?

14:42 Well, I mean, so I have a joke, which is that good data scientists copy from the D3 gallery and great

14:48 data scientists steal from the D3 gallery. And they give you the code for all of them. So I would say

14:53 probably close to 100% of the D3 visualizations I've ever built have been something I found in the D3

15:00 gallery and kind of tweaked until it fit my data.

15:02 Yeah. Yeah. Nice. So the next section you talked about was like the fundamentals of the math and

15:08 science that you need to know in order to be a data scientist. And I like the way that you put it.

15:14 You said these are like the cartoon versions of big mathematical ideas. How much math do you need to

15:19 know to call yourself a data scientist? Like if I studied just programming, I'm not a data scientist.

15:25 So I studied just math. I'm not a data scientist. Like what's the story there?

15:28 Data science is a funny thing in that there are as many different jobs called data science as there

15:34 are data scientists. So there are people who will call themselves data scientists and they write SQL

15:39 queries all day. There are people who call themselves data scientists and they do cutting edge machine

15:44 learning research all day. There are people who do data scientists who convince people to click it at.

15:49 Like you could know almost no math and still get a job where you call yourself a data scientist.

15:53 You could know very little programming and get a job where you call yourself a data scientist.

15:58 It's a field that's still kind of figuring itself out. And there's just such a breadth of the different

16:03 roles that are all calling themselves data science. In an ideal world, you would know, you know,

16:08 linear algebra, you would know probability, you would know statistics. But among people who are data

16:14 scientists, some people know that stuff really well. Some people know that stuff not so well.

16:17 Yeah, I can imagine. It's really about you've got a lot of data or some specific type of data and you

16:23 want to answer questions about it or even discover the questions you could ask that nobody's asked, right?

16:29 SnapCI is a continuous delivery tool from ThoughtWorks that lets you reliably test and deploy your code

16:48 through multi-stage pipelines in the cloud without the hassle of managing hardware. Automate and visualize

16:55 your deployments with ease and make pushing to production an effortless item on your to-do list.

16:59 Snap also supports Docker and M browser debugging, and they integrate with AWS and Heroku.

17:05 Thanks SnapCI for sponsoring this episode by trying them with no obligation for 30 days by going to

17:12 snap.com/talkpython Sometimes it's actually the opposite, which is, I have a question I want to ask. Where can I find some

17:26 data that will allow me to answer that question? So, you know, sometimes you start with the data and

17:31 go to the questions. Sometimes you start with the questions and then you got to find the data.

17:34 Right. Okay. Very interesting. So that actually brought you to your next section that you talked

17:39 about in your book, which was getting data. And so what are the common ways that you talk about there?

17:45 Nowadays, there's a lot of people who just like post interesting data sets. So if you go to like

17:50 Kaggle, which is a site that does data science competitions, every one of their competitions has

17:55 a data set, which might be interesting in ways that are not related to the competition. The government websites

18:01 publish all sorts of data, some of which is potentially interesting. If you like weather,

18:05 they publish weather data. If you like economics, they publish economic data. There's a lot of

18:10 Python libraries for scraping websites. So even if a data set is not available, you can always go out and

18:15 try and scrape it and collect it yourself and clean it. My go-to source is always Twitter. I always build

18:21 things using Twitter data. So Twitter and all the other sites will have APIs where you can just make

18:27 restful calls or even use libraries. Python has some Twitter libraries that abstracts all that away. And you can,

18:33 you know, collect tweets on a given topic or collect tweets from certain users and

18:37 collect your own tweets and do analysis on those.

18:41 Yeah. And you had a section about that in the book. What was the package you were using for that?

18:45 In Python, the package I usually use is called Twython. There's a bunch of them. That's the one that I

18:50 got to work the easiest, but I think highly of it.

18:53 I don't think I believe any other ones.

18:56 Yeah. Yeah. Okay, cool. Yeah. I think Twitter is fairly special among the social networks.

19:01 To me, Twitter is the social network of ideas more than it is of friends or family or whatever. So

19:08 yeah, pretty cool. I love to get data from Twitter as well.

19:11 Yeah. It's just that they won't give you the fire hose, but for most things, they'll give you

19:15 more tweets than you can handle anyway. So if you want to find out what people are saying on a certain

19:19 topic or just what people are talking about in general, it's an awesome source for that.

19:24 Yeah. I saw some study or heard about some study a few years ago where people were studying, they were

19:30 trying to do sentiment, mood analysis on Twitter, and then trying to tie that back to the stock market

19:38 and predict how people were feeling on Twitter to near-term changes like in the next hour in the

19:45 stock market, which I thought was a pretty interesting project.

19:48 So ideas you can come up with using that data are pretty much endless. Every month, I think I'm done

19:53 with it and then I think of something else to do.

19:54 Nice. So then you kind of get into the topics that I think of as traditional data science,

20:01 machine learning, neural networks, network analysis, that kind of stuff. And I thought maybe we could go

20:07 through each section and you could tell me what it is that you need to sort of fundamentally

20:13 understand what are the from scratch basic pieces of each part of this data science and then maybe

20:20 some examples of problems you might solve with it.

20:22 Sure.

20:22 Yeah. So the first one you talk about is machine learning. And when you said machine learning

20:27 and later neural networks from scratch, I'm like, wow, that's a pretty big thing to take on from

20:33 scratch. What's the story there?

20:34 Machine learning at a high level is just learning some kind of model from data rather than sitting

20:41 down and writing out the model yourself by hand. And so if you have a small amount of data, then

20:47 you know, coming up with an algorithm that's going to learn a model from it is actually a pretty

20:51 reasonable thing to do if you go with a simple model and don't add too many bells and whistles.

20:56 It's only when you want to, you know, start producing recommendations at Netflix scale

21:01 or when you want to start, you know, building something to recognize speech patterns from audio files

21:06 that your beautiful handcrafted Python is probably not going to be up to the task.

21:10 Yeah. So I suppose the types of things you solve with machine learning is fairly unbounded. There's

21:19 a lot of problems that machine learning answers. What are some of your favorite examples?

21:22 Everything out there is machine learning these days. But if you want to talk about like projects that

21:29 I've worked on for fun, one time I built a classifier to predict or to identify hacker news articles that I would be interested in or not

21:37 interested in. So it would take kind of the feed of new hacker news stories and come up with a score between zero and one about how interested, you know, it thought I would be based on some initial seed values that I leave.

21:49 How did you teach it?

21:58 How did you teach it?

22:06 I was for a few other kind of idiosyncratic features I threw in, like, you know, is it an ask hacker news? Is it a show hacker news? Does it have a dollar

22:15 amount in there? Because there are a lot of kind of bad stories that's like, here's how to make $5,000, I think. So that was a good negative signal.

22:23 It's sort of a spammy signal.

22:24 Yeah. So, and then it turned out I wrote a blog post about it and someone posted the story to hacker news and then

22:31 the hacker news community got very angry that someone would think that not every story there was worth

22:36 reading. Because of course every story there is worth reading. So why would I want to filter some of them out?

22:39 They accused me of wanting to live in a bubble.

22:41 And yeah, they can be fairly critical on there. But that's funny. So the big question is, did your system

22:49 like your article, your blog post?

22:51 That's a good question. I don't know that I actually checked. I should go back and try and find that.

22:55 It'd be funny if it recommended it to you. Or not. Either way, it would be funny to know.

22:59 Well, you know, it's funny speaking of this. So, you know, I have an Android phone and so

23:03 Google now will sometimes recommend me articles of things I'll be interested in.

23:07 And occasionally it will recommend to me like my own blog posts, which I guess means doing a good job.

23:13 You know, one of the things I did with machine learning was I have a five year old daughter and I do her clothes shopping.

23:19 And I noticed that little boys clothes were very interesting and little girls clothes tend to be

23:23 kind of like really boring. The boys clothes have like dinosaurs and rockets and robots and

23:28 girls clothes have like hearts and flowers. And so I built a machine learning model to take an image of a

23:33 children's t-shirt and predict whether it was a boy shirt or a girl shirt.

23:38 Awesome. Yeah. And I agree with the classification there as well. I have three daughters and we still

23:44 bought a fair number of boy clothes for them.

23:46 Yeah, I did the same.

23:47 Nice. So the next topic that you covered was nearest neighbors. And I mean, conceptually,

23:53 I kind of know what nearest neighbors are pretty easily, but it's a computation hard problem,

23:58 especially in higher dimensions. And so what kind of stuff do you do? What kind of problems do you solve with

24:03 this?

24:03 One thing you can do is when you don't have like a great parametric model for what's going on.

24:08 So for instance, one place where I've seen this applied is when I have some kind of like

24:12 time series type signal, and I want to know if it represents some kind of bad anomaly. And it might be,

24:19 you know, like thousands of points in some weird shape. And I don't have a great way to classify it.

24:22 But one thing you can do is if you have a bunch of labeled signals, you know, some are good or some

24:28 are bad, you just say, I'm just going to take my set of reference signals and figure out which are

24:33 the ones it's closest to. And that way, I don't need to necessarily have a model of what's a good

24:38 signal or what's a bad signal. I can just say, I have some labeled data, that's enough. And so I

24:43 think that's kind of the situation where it can be useful.

24:46 Yeah, nice. When you're still trying to explore things, and maybe you don't, you don't have a model

24:51 you're trying to fit it to, you're just trying to understand it, right?

24:53 Yeah. Or think about where like, you have some weird kind of shapes, or weird kind of patterns

24:59 that you can't really put math behind. What's a good pattern? What's a bad pattern? But you have

25:04 some labels, then you could use nearest neighbors to basically classify without having to have a

25:10 mathematical model behind it.

25:11 Okay, yeah, very cool. So then you get to another topic called Bayesian Bayesian analysis, which the context I know this is around kind of determining spam and

25:21 filtering and stuff like that. But what's, what's the story of Bayesian analysis?

25:26 So it's named after Bayes' rule, which is just a theorem in statistics, having to do with ways of

25:33 reversing conditional probabilities. So at a high level, if you have some knowledge about

25:38 what is the probability of seeing certain features in email, given that an email is spam, you can use that

25:45 data to produce estimates of what is the probability that an email is spam, given that you see those

25:50 features. And so typically, if you have a lot of, you know, spam and non-spam, you can make estimates

25:56 of, okay, given that an email is spam, it's likely that I'll see Viagra, and it's likely that I'll see

26:02 Gary Twick. And then basically just a classifier that turns that around and allows you to now look for

26:07 these features. And in a math-manipede rigorous way, come up with the probability, okay, here's the

26:11 probability that this is spam. Okay. Yeah. Very interesting. What other types of problems?

26:15 Spam is certainly easy to understand, but you know, hopefully that's solved by Gmail and other people

26:22 for us, right? I think it was Paul Graham who wrote a pretty influential article about this approach,

26:27 probably about 15 years ago now. And so I think a lot of the tools that, I don't know if it's what

26:33 Gmail uses, but that a lot of these BAM assassin and some of the other mail providers used are still

26:37 based on this principle of naive base. But basically the, anything where you have kind of big clumps of

26:44 say text, and you want to classify it into one or more classes, like two or more classes, and you're

26:52 willing to make this really big kind of technical assumption that the words in it are kind of independent

26:58 of each other. So you're treating them kind of like a bag of words rather than as a sequence of words,

27:02 then that's where you would use this kind of model. Okay. Yeah, cool. So maybe the opposite of that,

27:08 where you treat words sort of as, as having more meaning is in natural language processing, which is

27:14 another thing you talk about, right? Natural language processing is, is a huge field. Like people,

27:18 there are textbooks about it. So in fact, there are Python books about it. So cram it into a chapter of a

27:25 book is really just kind of, here's a really high level overview. And here's, you know, a couple of

27:31 examples to give you a flavor. Natural language processing is, it's actually relevant to this sort of

27:37 problems that I'm getting into in artificial intelligence. It gets used a lot by all the voice recognition in

27:43 your phone, and a lot of these new, basically reading texts, and having a computer extract information from it. So it's a

27:52 pretty important, pretty, pretty rich field. Yeah. And there's some pretty decent Python libraries for

27:57 doing that, right? Yeah. So there's a, so the big one is called NLTK, the natural language toolkit,

28:04 I think it stands for. And so it has its own book just about natural language processing in that library.

28:10 And I think that's actually free on the web. But yeah, that would be a great place to start it. If you

28:15 want to come to understand more deeply, there's also a nice Coursera course on natural language processing.

28:20 It took several years back that I liked. Oh, okay. Yeah, excellent. So let's see. Another topic you

28:25 covered were decision trees. And what's the story of these? A decision tree is kind of what it sounds

28:31 like. Yeah. You know, intuitively, you might have a model of making decisions based on a tree. So I have,

28:39 I want to know whether to buy a certain car or not. So, you know, I might start asking a question. Okay,

28:44 is it an American car or a Japanese car? American car. Okay. Go to the next question. Does it have

28:49 four doors or two doors? It has two doors. Okay. Is it gas or diesel? Diesel. And then you just have a

28:55 sequence of questions to ask and you classify it based on those questions. Okay. It's diesel. That

29:00 means, yes, I want to buy it. And so conceptually, you know, a tree like that is not that complicated.

29:06 Interesting part is given some set of data, how do I build such a tree? And there, there's a variety of

29:13 algorithms, but the one that we talk about in the book, it is a pretty simple one, which is basically

29:18 around, okay, I have a bunch of data. It has a bunch of attributes. I can split my data on each

29:24 attribute. So I could say, okay, at this stage, I want to look at two door versus four door. I want to

29:28 look at Japanese versus American, or I want to look at gas versus diesel and which of those choices is

29:35 going to allow me to kind of really separate the good buys from the bad buys the most. So if gas versus

29:43 diesel totally splits where diesel is the buys and gas is the don't buys, that's a great thing to choose.

29:50 If gas versus diesel splits where, you know, on gas, I want to buy half and not by half and on diesel,

29:56 diesel and by half and up by half. That's not a good split to choose because it doesn't really

29:59 help me at all. And so the mathematics here are just around kind of making this precise with this

30:05 entropy and building these trees from the data. Okay. Yeah, that's very cool. The way where you

30:11 build them knowing the data intimately, you know, that, that makes a lot of sense. And you don't really

30:17 necessarily need data science to do that, right? That's just helping, you know, asking a few questions

30:23 and making a decision. But the reverse, I think is pretty interesting.

30:27 They build some of these expert systems in this kind of way. I think they apply them sometimes in like,

30:31 medical diagnosis and sometimes they do better than doctors.

30:35 Wow. Okay. Sometimes just let the data talk, huh? Rather than intuition.

30:40 This episode is brought to you by hired hired is a two sided curated marketplace that connects the

30:56 world's knowledge workers to the best opportunities. Each offer you receive has salary and equity presented

31:01 right up front, and you can view the offers to accept or reject them before you even talk to the company.

31:07 Typically candidates receive five or more offers within the first week, and there are no obligations

31:11 ever. Sounds awesome, doesn't it? Well, did I mention the signing bonus? Everyone who accepts a job

31:16 from hired gets a thousand dollars signing bonus. And as talk Python listeners, it gets way sweeter.

31:22 Use the link hired.com/talkpythontome and hired will double the signing bonus to $2,000.

31:29 Is knocking. Visit hire.com/talkpythontome and answer the call.

31:32 Somewhat related to that maybe is, you talked about neural networks as well.

31:42 Yeah. So neural networks are a super hot topic right now, because I'm sure you've probably heard

31:47 that all the buzz about, or about deep learning, right? And deep learning tends to have neural networks

31:54 at the root of it. So neural networks are basically a way of building up kind of layers and layers of

32:01 representations. And the deep learning, this book doesn't really go into deep learning because it's,

32:06 the single neural network is hard enough to do by hand or from scratch. But basically it's a way to

32:12 kind of build a classifier that works similar to how a toy model of a brain might work. So you have

32:20 artificial neurons, and each neuron has a bunch of inputs that go into it with weights. If the weighted

32:26 sum of the inputs exceeds some certain threshold, the neuron fires. And if it doesn't exceed that

32:30 threshold, the neuron doesn't fire. So you present an input, which could be like a, an image. So basically

32:36 a bunch of zeros and ones, and that causes some neurons to fire. And that propagates through this

32:42 network. And in the end it will spit out, you know, I think you should be an image of a cat,

32:46 or I think you should be an image of a dog. And you train it by showing that a lot of labeled images

32:52 and adjusting the weights based on how you got them wrong.

32:55 Yeah. Interesting. So you basically just say, these are the inputs and the decisions that you feed it

33:02 known data. You say, you know, like, here's a cat, here's a cat, that's a dog. It's just a cat or a dog,

33:07 right? And then you ask that question. I was gonna say, well, one of the problems that I sort

33:11 of remember from neural networks is that they, as you try to get them more accurate,

33:15 they get over-trained to just only do little bits of stuff. And so how does things like decision

33:21 forests or these things and this deep learning, how does it deal with that?

33:26 So random forests are basically taking multiple decision trees and combining them. So there's this

33:33 kind of general principle where a lot of times rather than building one model, it's like super predictive.

33:40 I take a lot of much less predictive models and just kind of let them vote on things or average

33:47 the results or whatever. So you're right that it's really a decision tree. And I included like all

33:52 hundred features in my data set that there's a good chance I'm going to like overfit and

33:57 overlearn my training data and not generalize outside of that. So one thing that happens with decision

34:04 trees is people often don't use the bare decision trees. They use the random forest where they'll build

34:10 a bunch of smaller decision trees, each of which is really restricted to a small subset of features

34:16 so that each one is individually less powerful. But then when you combine them, they do well in the aggregate

34:22 and they don't have necessarily the same overfitting problem that single decision tree with lots of features would.

34:28 So in neural networks, especially in deep learning, they're not exactly the same techniques, but other

34:34 techniques where you will zero out some of your weight sometimes and train without them to make

34:40 sure that they don't learn too much. And there's a lot of other techniques that get used in order to

34:44 make sure you're not overfitting. There's something that, you know, data scientists and machine learning

34:47 people worry about a lot.

34:48 It's a really hot field, like you said right now. That's awesome. So one more section I'd like to touch

34:54 on is you talked about this thing called recommender systems.

34:58 So mobile, they're just what they sound like. Anyone who's used the internet has a pretty good experience

35:02 where you go to Netflix and it says movies for Joel and it's trying to predict what it thinks I'll like.

35:09 Or you go to Amazon and it will recommend items for me. And you can go a little further where you have

35:16 all these startups like Stitch Fix is a very hot data science startup where you tell what kind of clothes

35:22 you like and they'll send you a box of clothes every month that they think you'll like a lot.

35:28 And so generating these kind of recommendations is a pretty popular task within data science. A lot of data scientists

35:36 work on these kind of problems. There's a lot of data scientists work in this job. So they have to sell you stuff

35:41 and they always want to sell you stuff that they think you'll like.

35:44 You convert better if you figure out what people actually might want, right?

35:47 Yeah, I mean, well, if Amazon sends me an email, "Hey, Joel, you know, these five things are on sale

35:54 and they're five things that I really want to buy," then that's much better for them and potentially

35:58 for me than if they send me a random email that's like, "These are the five most popular things on

36:03 Amazon today." Yeah, it definitely is better. I really like going to my Kindle and looking at

36:08 the recommended things based on what I've been reading. But the Netflix recommendations, I know they

36:14 do really great work, but it just doesn't work for me because my seven-year-old daughter watches

36:19 Strawberry Shortcake and other random things. I get a lot of kid shows recommended to me.

36:25 They have a "Who's watching" button that you can click and say "A kid is watching."

36:30 I know, but my daughter won't use it. She'll just randomly pick one.

36:33 Yeah, so you know, it's funny. I use that button pretty well, but my Netflix recommendations are also

36:40 not that good. But I think it's just because I don't like anything on there. So no matter what they're

36:45 telling me, I'm not going to like it. Yeah, I hear you. I feel like you covered a pretty

36:49 wide swath of data science from scratch. Some of the topics were really accessible. Some of them

36:57 required more math and more thinking, but it was all a really nice presentation. What do you feel like

37:04 you left out? There aren't any topics that, if anything, I kept adding stuff while I was writing it.

37:11 So I don't feel like there were any huge topics that I necessarily left out. But in terms of coverage,

37:17 probably my biggest regret is that I used Python 2 instead of Python 3.

37:20 Yeah. Okay. And I saw on your GitHub repo, you had something about Python 3 in there. Is that right?

37:27 Yes. So pretty much as soon as the book came out, one, I got a lot of emails from people saying,

37:32 "Hey, why do you not use Python 3?" And then I also got a number of emails saying,

37:37 "I would like to use Python 3. Will the code work?" And so I wrote them back and I said,

37:41 "Yeah, you know, I don't feel like the code shouldn't work. Give it a try. I bet it works

37:46 with probably a few changes, add some parentheses, print statements and so on." And then eventually one

37:51 guy wrote me back and he said, "It doesn't work." I said, "Okay." So I sat down and I said, "I'm going to

37:57 convert the code to Python 3." And it took me about, I'd say four to five hours. And that's with me

38:04 knowing the code intimately. So it would have taken someone who hadn't written the code in the first

38:09 place a lot longer than that. And then I felt kind of guilty that I've been telling all these people that

38:12 it was so easy to do when it wasn't.

38:14 Just spend a week. It'll be fine.

38:16 Yeah. But so yeah, I have the Python 3 versions of the code up on the GitHub. I sort of regret not

38:23 having just done it that way in the first place.

38:25 Yeah, sure. So at the end, you talked about data science, not from scratch, and you pointed out

38:31 a lot of the libraries you might actually use, like NumPy and so on. Do you want to talk about that

38:37 really quick? Like what's the real data science versus the from scratch data science comparison?

38:43 Yeah. So I would say that NumPy is pretty fundamental. That's basically the linear algebra

38:51 library for Python. So it provides you matrices, matrix algebra, high performance arrays, things

38:58 like that, you don't get just built in. And you might not use it directly, but a lot of the other

39:05 libraries are really built on top of it. So kind of the most broadly accessible machine learning

39:11 library for Python is called Scikit Learn. And it has really nice documentation and really nice tutorials

39:16 and a fairly standard API for building machine learning models. Anything you want to build a

39:21 regression model or a random forest model or any kind of classifier, that's probably the place to go.

39:27 There's also Pandas, which is the data frame library, which is good if you're working with tabular data. So

39:34 not necessarily the machine learning side of data science, but more of the, I have kind of a

39:39 spreadsheet data set and now I want to clean it and aggregate it and pivot it and look for kind of data

39:48 analysis type insights.

39:49 Yeah. If you're exploring, like you said, tabular data and you kind of kind of load it up and clean it,

39:56 Pandas seems really fantastic for that.

39:57 It's a really nice library. And then kind of the new kid on the block is a TensorFlow, which is

40:03 Google's deep learning library. And it's only was released, you know, a few months ago and it's gone

40:09 through. It's not 1.0 yet, but it seems like people are sort of converging around it as that's how they're

40:15 going to do deep learning in Python. Now there are other sort of previous libraries that some people

40:20 used and still use, but TensorFlow seems to be gaining a lot of mindshare.

40:24 Okay. Yeah, that's really cool. And I've definitely seen TensorFlow talked about a lot in this context,

40:30 but it's just a library. You can download that and run it locally. It's not like a cloud type thing,

40:35 right?

40:35 Yes. So currently it's a local library and I'm not sure if they've come out with a version that you can

40:42 kind of do your own cloud, like on various AWS instances. But, but I know they're definitely going

40:47 in that direction because a lot of this stuff, these deep learning models take a really long time to

40:52 train. And so you want to go to use them for anything serious. You want to distribute them and

40:56 throw them in high-powered machinery and not just run them on your laptop.

40:59 Yeah, of course. If you have tons of data, maybe it's better to get a bunch of machines

41:03 for an hour. So what do you think about these cloud learning or data science platforms? I'm thinking like

41:09 Azure machine learning, or, you know, I just had SigOpt on the show a few shows ago. I'm not really sure

41:15 what else is out there in terms of like go out to the cloud and grab some data science stuff.

41:20 What do you think about that?

41:21 I haven't spent much time looking at any of those. I think they can add value in terms of either one,

41:31 if you don't have the data science or machine learning expertise in house to do whatever you need to do.

41:37 Or two, if you have some kind of model that you've built, but you need help either putting it into

41:43 production or operationalizing it somehow, that they can build a pretty good role there. But my sense is

41:50 that most people doing data science are aligned more on running the libraries and running the models

41:56 themselves. But that could be just my bias sample of the people who I talk to.

42:00 Yeah, of course, of course. Cool. So another thing that you said you're into is taking some of the

42:07 ideas from Haskell and thinking about how those might manifest in Python. What do you went up to there?

42:12 Haskell, if you're not familiar with it, is kind of the purest of the pure functionally typed

42:19 languages with strong types and lazy evaluation and things like that. And so once I spent some

42:28 time in that world, I spent a lot of time thinking about how can I bring some of these concepts back

42:33 into Python. And in Python 3, lazy evaluation plays a much bigger role in the sense that like range

42:42 is a generator instead of a list and all the map and filter and things like that also are generators

42:48 instead of lists. But I started getting into the iter tools library, which starts giving you tools for

42:55 generating basically infinite sequences and just trying to see how far I could go using

43:01 pure functions and infinite sequences and avoiding mutable variables and other things that you try not

43:07 to do when you're working in a Haskell-like language.

43:10 Yeah. And how did you feel like it came out in the end? Do you feel like you were able to bring a lot

43:14 of those ideas over?

43:15 I was, and I ended up producing code that was really neat and really impenetrable. The lazy

43:23 infinite sequences stuff, that was more almost academic in terms of like, yes, I managed to do it. Like,

43:29 this is mathematically interesting and it works well, but it's not readable at all.

43:34 So imagine you wanted to represent, let's say a binary tree in Python. Kind of the two,

43:41 I would say obvious approaches would be one to make some kind of like class where it has a,

43:47 you know, a value element and a left element and a right element. And then you also might just use

43:53 a dictionary to represent it where it had those keys. In a language with algebraic data types,

43:57 like Haskell, you would just basically represent that kind of tree as a product type where it just

44:04 has like three elements. And so I said, you know, what if I just represented a tree as a tupple with

44:09 three elements where first element is the left subtree, the second element is the value,

44:13 third element is the right subtree. And similarly, if you want to do like linked lists in Python,

44:17 which you probably don't want to do, but if you did want to do linked lists in Python, you just treat

44:21 them as a tupple. First element is the element and the second element is the tail linked list. And so,

44:28 I actually found that I was able to write some pretty nice code using those kinds of ideas.

44:32 And I did some coding interviews that way. I'm not sure the interviewers appreciated it.

44:38 What is this guy talking about? Yeah, cool. So what question given your work at the AI Institute and

44:47 your background in data science and so on, I wanted to ask is the last year or so, there's been a lot of

44:54 news items and people coming out saying that artificial intelligence is a danger to humanity.

45:01 What's your thought? Is like AI something we should be super excited about? Or is it something we should

45:07 be maybe cautious about? I would say I probably fall in the middle. I mean, I'm excited about it because

45:12 this is my job, but I don't go around encouraging everyone else that they have to be excited about it

45:18 because I don't know that that's necessarily warranted. But at the same token, I don't spend a lot of time

45:23 worrying about how dangerous it is. I think we're pretty far off from the time when we have to worry

45:29 about that. And I do have some friends who think we should worry about it now before it's too late. But

45:35 I think there's a lot more important things to worry about in the world when I read the news.

45:39 Yeah, I kind of agree with you on that. And I think there's two, certainly two ways to look at that. On

45:47 one hand, if you think about things like self-driving cars, let's just take that as an example. Like,

45:53 I believe one of the biggest job categories for men in the United States is some form of driving,

46:02 like driving a truck, driving a taxi, those types of things, delivery vehicles and so on. And if self-driving

46:08 cars were to like remove all that, like that would have large social effects. But I think, you know,

46:13 that's not so much the way that people, at least recently in the news, were talking about it. It's

46:19 more like Terminator style, right? And so that I, I'm not too worried about this personally. Who knows?

46:26 Yeah. I mean, I personally would pay a lot of money for a self-driving car because I,

46:30 I don't like driving that much. And I'd much rather be able to read while I'm going somewhere.

46:35 Yeah. Driving is fun until you get stuck on i5 for half an hour inching along. Then you know,

46:40 I don't like driving anymore. Exactly.

46:42 Awesome. Well, we're kind of coming up near the end of the show. Let me ask you just a

46:46 few closing questions. I always ask on my guests, if you're going to write some Python code,

46:50 what editor do you open up? So these days I tend to use Adam for pretty much everything,

46:55 memory leaks and all. Nice. Yeah. That's from a GitHub, right?

46:58 Yeah. It's pretty similar to sublime, but it doesn't nag you for money. So yeah, it's,

47:03 it's pretty nice. And I think it's Adam.io. They have a really cool little video about how

47:08 it's the editor of the future. It's nice.

47:10 Yeah. I don't know if I go that far, but it's the editor of the present at least.

47:14 Yeah. It's like a George Jetson sort of like a promo video. It's, it's pretty funny.

47:18 It is pretty nice. Isn't it? Does it have good Python support?

47:21 You know, I'm never, I'm not someone who leans on like ID functionality a lot. So if good Python

47:28 support counts as syntax highlighting, yes, but that's all I tend to use it for. So yeah.

47:32 Yeah. Yeah. Okay. Cool. And if you look at on the Python package index, there's,

47:38 you know, 75 plus thousand packages and we all have experience with, you know, different parts

47:46 of it and there's things that we love and would recommend, like, what is your favorite one you

47:49 might recommend that people maybe don't know about?

47:51 So the one I recommend that people don't necessarily know about is called Beautiful Soup. It's,

47:57 basically a HTML parsing library. And so if you start scraping data from webpages, you're going to get

48:04 a big mess of ugly HTML. That's probably not even well formed. most of the time,

48:10 most people don't bother to well form their HTML.

48:12 Are you telling me that I can't just like load that up as an XML document or something like this?

48:17 No, I'm just kidding. Of course it's, it's terrible trying to work directly on the web,

48:21 right? And Beautiful Soup is really, I really like it as well.

48:25 It's really nice. I mean, you have to spend a little bit of time getting used to

48:28 it's API and interface and everything, but it's super handy for getting data out of webpages and

48:34 doing anything where you have to do a bunch of scraping.

48:37 And you cover that in your book, right? In the getting data, I use Beautiful Soup a little bit and show people how to use it.

48:42 Yeah, I cover it a little bit. It's a nice addition to the data scientist toolkit.

48:46 Nice. So Joel, how do people find your book? Amazon? Just a Google search for data science

48:52 from scratch? Yeah, it's on Amazon and you can buy it from O'Reilly.com. But yeah,

48:57 if you Google data science from scratch, you'll find it.

48:59 Cool. And I'll put a link to the GitHub repo where you have all the code examples and so on as well.

49:04 It's been really fun to talk about data science. And I think you have a really interesting way of teaching

49:10 people to appreciate the tools that we're all fairly familiar with by showing you how to build it from

49:16 scratch. So thanks for that.

49:19 My pleasure.

49:21 This has been another episode of Talk Python to Me. Today's guest was Joel Gruse and this episode

49:25 has been sponsored by SnapCI and Hired. Thank you guys for supporting the show. SnapCI is modern,

49:30 continuous integration and delivery. Build, test and deploy your code directly from GitHub,

49:34 all in your browser with debugging, Docker and parallelism included. Try them for free at snap.ci

49:39 slash talk.com. Hired wants to help you find your next big thing. Visit Hired.com slash talkpython to me to get five or

49:45 more offers with salary and equity presented right up front and a special listener signing bonus of $2,000.

49:50 Are you or a colleague trying to learn Python? Have you tried books or videos that left you bored by just

49:56 covering the topics point by point? Check out my new online course, Python Jumpstart by building 10 apps at

50:01 talkpython.fm/course to experience a more engaging way to learn Python. You can find links

50:07 from this show at talkpython.fm/episodes slash show slash 56. Be sure to subscribe to the show. Open

50:14 your favorite podcatcher and search for Python. We should be right at the top. You can also find the

50:18 iTunes feed at /itunes, the Google Play feed at /play and the direct RSS feed at /rss on

50:25 talkpython.fm. Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.

50:31 You can hear his entire song at talkpython.fm/music. This is your host, Michael Kennedy.

50:36 Thank you so much for listening. Smix, take us out of here.

50:39 Stating with my voice. There's no norm that I can feel within. Haven't been sleeping. I've been using lots of rest. I'll pass the mic back to who rocked it best.

50:49 I'm first developers.

50:51 I'm first developers.

50:58 Developers, developers, developers, developers.

51:01 .

51:01 Thank you.

51:01 Thank you.