It's the end of the year, and many of you are probably kicking back and taking it easy without a TPS report to be seen.
So we'll keep this fun and lighthearted this week.
We're running down the top 10 data science stories of 2015 on episode 40 of Talk Python to Me with guest Jonathan Morgan, recorded December 13th, 2015.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the library, the language, the library, the language.
The ecosystem and the personalities.
This is your host, Michael Kennedy.
Follow me on Twitter where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via at Talk Python.
This episode is brought to you by Hired and DigitalOcean.
Thank them for supporting the show via Twitter where they're at Hired underscore HQ and at DigitalOcean.
Hey, everyone.
This episode's a little unique.
I've partnered with a great data science podcast called Partially Derivative, and we're doing a joint show multicast of both podcasts.
If you like this sort of thing, be sure to check out partiallyderivative.com and subscribe to their show.
Also, I wanted to let you know I'm not releasing a show next week.
I'm on vacation.
So I'm going to take next week off, do a little resting and relaxation, hang out with the family, and be ready to get back and do a ton of awesome shows for you in 2016.
Now, let's get on to this co-hosted episode I did with Jonathan Morgan.
Hey, Jonathan.
Welcome to the show.
Hey, Mike.
Thanks so much for having me.
Yeah, I'm really excited to do this joint Talk Python partially derivative podcast about the end of the year and data science and all that awesome stuff.
Yeah, I'm super excited, too.
I think when our powers combine, it is the ultimate resource for Python data, Python stuff in the universe.
Absolutely.
Yeah, it's going to be great.
It's going to be great.
So for those of you who don't know me, my name is Michael Kennedy.
I'm the host of Talk Python2Me, a sort of developer-focused Python podcast.
And this week, I'm teaming up with the partially derivative guys.
And for those of you who don't know me, I'm Jonathan Morgan, one of the co-hosts of the partially derivative podcast, a podcast about data science, kind of, but also sort of about screwing around and drinking beer.
Yeah, that's really the most common combination of any two things, I would say.
It's probably drinking beer and data science, right?
All of the best data science is done in a two-beer buzz.
It's the secret nobody tells you.
That's right.
You only learn it in grad school.
Yeah, exactly.
I kind of like to hear about your company.
The last I heard is you guys were starting a data science company, and that's about all I heard.
It's called Popply, right?
Yeah, that's right.
So it's called Popply, and my two co-hosts of partially derivative, Chris Albin and Vidya Spandana, started this data science company.
And so basically, we were realizing that there's a whole bunch of data science that's super hard.
And everybody, I think, is familiar with this kind of artificial intelligence and complicated machine learning.
But then there's a lot of data science that's actually pretty straightforward.
It kind of boils down to inference and making charts out of data that you just have sitting around.
So you have a better everyday idea about what's happening.
And it turns out that that's super hard.
Like, actually, even for pretty technical people, it's super hard.
You know, like, I meet a lot of developers who are like, they'll ask me a question like, so I've got like, like, thousands of rows of data in my database.
And they're all like, at a time.
But then how do I like, look at it, you know, like, over time?
And it's like, oh, that's right.
Like, I mean, super technically competent people who are still like, I just don't really understand how the data thing works.
And that first kind of turning that first corner was really important to us.
We're like, okay, cool.
We could actually empower a lot of people to do data stuff if we could make that first step automatic.
Like, if we could just go from some raw data to charts to give you an idea about what's happening inside your data, we should definitely do that for people.
And so that seems easy.
It took us a little bit longer than we thought.
All those details, they keep sneaking in there.
Exactly.
But yeah, so, but we've released a product.
It's in private beta.
So talk Python to me, listeners.
You should definitely be part of the private beta.
I'm sure there'll be some contact information.
It's poppily.com.
You can go request an invite or just email me or at me on Twitter or something, and we'll get you in the private beta.
Yeah, and we're releasing publicly early next year.
So it's super fun.
We're having a really good time.
That's great.
You guys are actually using Python quite a bit there, right?
Oh, yeah.
Up and down the stack.
It's all Python.
So some of the data people out there might know that there's another couple languages that do some data science or that people use to do data science.
One is called R.
People use a language called Scala.
None of them live up to the awesome power and flexibility of Python, which is why we use it in almost everything we do.
From the web app that people interact with.
The web app that people interact with when they're actually using the system.
All of that's in Python.
It's actually a Django app.
I hadn't coded in Django for a while.
It was super fun.
And then the back end uses SciPy and the whole SciPy stack for some of the machine learning and data processing stuff that we're doing on the back end.
So we are a Python shop all the way.
That sounds really fun to be putting all that together.
I'm sure you guys are liking it.
Yeah.
It's actually the coolest thing about Python from my perspective is that you can do kind of complicated scientific computing and stats and then plug it right into the web app that you'd already built because the language is so flexible.
So it's been fun.
Yeah.
Very cool.
So I'll put a link in the show notes, but definitely if you guys, if that sounds interesting, check out poppily.com.
Is that right?
Yeah.
That's it.
Poppily.com.
We thought about poppily.io, but poppily.io, it was just weird.
It was too much.
Get one of those Libyan domains.
Those are always good for the startups.
The LOIs.
Yeah.
Poppily.
Would have been the best.
Yeah.
Missed opportunity.
Well, you can always change the name if you really have to.
All right.
So you want to talk about this year?
I mean, this show's going to come out on Talk Python on the 29th, and I suspect around the same time on Partially Derivative.
It's perfect, like right at the end of the year, to talk about sort of what has happened in Python world intersected with data science, I guess.
Yeah, absolutely.
I mean, it's been a big year.
A big year for Python and a big year for data science.
The first pick, maybe not the most important, is probably most relevant to people while they're listening to this show.
Like, if it comes out on the 29th, you know, you got some vacation.
Maybe you'll pick it up around the 31st or the 1st.
That's typically when we make our New Year's resolutions, right?
It is.
And the first story is all about how you're pretty much going to fail at this.
Don't get your hopes up, right?
I know.
The numbers are.
The probability is that you're not going to stick with that New Year's resolution, which is funny because I think it's something that we all know.
But this is actually from last year.
Mona Chalabi, who's not at 538 anymore.
She's doing data journalism for The Guardian.
But she was at 538 at the time.
And this was actually kind of like a larger story of the last year that I think kind of data journalism also really came into the, you know, kind of came into the public mindset.
And she did a really interesting piece where she broke down the stats of how often people fail at their New Year's resolutions and like how long they keep them.
And I guess it's something like 70 odd percent fail within the first two weeks.
I'm going to change my life.
I promise this year will be different.
Oh, maybe.
But it's a Tuesday.
And, you know, my friends are going out or whatever, right?
Yeah, yeah, totally.
There's a lot of, I like, I really like the aspiration of New Year's resolutions.
And I'm just not going to think about the cold, hard reality of their eventual failure until later this month.
Yeah, exactly.
You're going to wait at least two weeks.
Exactly.
Yeah.
So some of the most common ones, well, the most common by quite a measure was lose weight.
And closely related to that was exercise more.
And then the third one really, in terms of popularity, really puts a sort of a challenge on being able to determine whether or not you've achieved it, which is to be a better person.
How do you analytically answer that, right?
That's true.
It's tough.
I mean, I guess you probably could say there's a little bit of wiggle room.
If 70% of people failed at that resolution, that's actually really worrying.
I need to be just an incrementally better person.
They're like, no, I tried, but I'm still a jerk.
No, I yelled at the neighbor again and kicked over.
Yeah, exactly.
Nice.
Oh, well.
So you and I are both big fans of podcasts.
We listen to a bunch and obviously we produce some that we're very passionate about.
And the next one, the next item is actually about the most popular podcast of all time, something called Serial.
Yeah, in fact, I think it was the only thing in 2015 that was more popular than Talk Python to me and data science.
The only thing.
Everything else, you know, sort of pales in comparison to these two giants of media dominance.
But of course, there is Serial.
Yeah, Serial is, if you guys haven't heard of it, it's a podcast that goes back and looks at a person accused, maybe convicted of murder.
I can't remember.
No, he was convicted.
Convicted.
Yeah, that's what I thought.
And goes through this high school guy and sort of rehashes, reevaluates the evidence, redoes the interviews.
And it's like an investigative journalism look, but through the eyes of a podcast rather than maybe through like Time magazine or whatever.
And it was downloaded something like five million times a week.
It completely broke all the records of all time.
Yeah, it was pretty amazing.
I mean, and by the way, spoiler alert, the dude is totally...
Totally guilty.
I'm just...
I mean, this is not really a spoiler because that's not how the show ends.
But this is going to be cool.
This is going to be really divisive for your listeners because half, I think, are going to write angry emails and the other half are going to be like, totally.
So that's where I stand.
I feel like I just...
I feel the responsibility to get that off my chest.
No, I can't.
I hash it.
Well, this second item, also at FiveThirtyEight.com, sort of a journey of some data sciences folks to actually go through and try to apply data science to this journalism to answer statistically or, you know, sort of through data science.
Is he guilty or not?
Yeah.
And this was actually interesting because it's not really something that can be quantified.
In fact, that was really a big theme in the show was that all of the information that we had at the time and that we have now to assess whether or not this man was guilty of the crime that they, you know, they think that he committed this murder is really suspect.
It's really not – it's hard to say conclusively what did and didn't happen because it relied so much on, you know, personal accounts of the events of the day.
That said, there was some information that they could point to that definitely happened.
And the big argument that everybody who believes that he's guilty was making was that none of these – none of the events in particular, like, made it certain that he was guilty.
But when looked at in the aggregate, then that was really unlikely.
That was sort of the intuition that everybody had.
But the interesting part was that the – so, 538 interviewed a couple people who are data scientists and they went through kind of a Bayesian process for, you know, assessing the likelihood that each of the events could happen in concert.
Like, whether or not he basically just had, like, super bad luck.
And it actually, you know, when you look at this as something called a multiple testing problem, that's a way that you can test the hypothesis in lots of different ways.
And looking at it through that lens makes it seem a little bit more probable than you might first assume that all of these different things could happen to him.
So, basically, like, he asked the victim of the crime for a ride.
He lent his car and his cell phone to somebody else who was also accused of the murder.
And then his phone was in a location that was in, like, you know, within, like, a small distance from where the body was found.
And then his cell phone records seemed to corroborate with a bunch of other, like, of the prosecution's testimony about how he totally – so, basically, like, there was, like, four or five things that made it, like, dude, that's impossible.
Like, if all those things are true, you were definitely guilty, even though none of them as an individual piece of information is super damning.
But it turns out, according to these data scientists, that, well, you know, maybe we should give this a second look.
It's actually not that unlikely that he could have had that much bad luck.
So, we'll see.
Yeah, that's – I think it's really interesting to take, you know, a hard science like data science that's working with numbers and apply it to something soft like interviews and likelihood that someone's telling the truth and these kinds of things.
So, I think even in that regard alone, it's really interesting.
Yeah, totally.
I mean, ultimately, it was pretty subjective.
But it was – I mean, and it's hard because this was – these are, like, real people's lives that we're talking about.
It was a true story.
You know, it's not like a murder mystery.
But it was really hard not to get into it and take sides and think about it like, you know, like a whodunit.
So, you know, apologies to all involved that we're talking about this in such an insensitive way.
Yeah, absolutely.
It's definitely a harsh reality that something bad happened to somebody.
They're having a second season.
I haven't listened to it yet.
What's the story of the second season?
Do you know?
Just came out.
No.
Yeah.
No, I don't know.
I think they're probably, you know, trying to keep up with the times because, you know, if you don't – as we know, you got to keep putting out content or people get bored.
So, Serial 2 – but I guess all I know is that it's probably – I think it's not about the same guy.
So, if you're sick of hearing about this guy's story, then you're in luck because it's about – I'm assuming another unsolved murder.
Yeah, it's got to be.
It's got to be.
All right.
Moving on to the next item is something very near and dear to the Talk Python listeners' hearts, I'm sure.
And that's Jupyter and iPython and iPython Notebooks.
Yeah.
Yeah, absolutely.
And they've had a huge year.
Yeah.
So, I think – yeah.
And so, I don't – I'm assuming a lot of your audience will be pretty familiar with iPython and Jupyter.
Although, that said, I guess there's like two camps of Python developers, I feel like.
Like sort of web developers and software engineers and then kind of data and stats folks.
Did you come at Python from a computer science perspective or did you have to find some language to do your specialty and kind of grow into programming?
I suspect that that second category is very well familiar with the iPython stuff.
So, maybe I should just tell everyone.
iPython Notebooks are these sort of interactive documents.
You can load them up as web pages and you can write a little bit of Python code and then you can actually execute them real time right there.
Like so, you could pull some data from a database and then do some sort of science on it and a graph pops up.
And then you write a little bit more code and another bit of data pops up.
And these things are sort of live research documents.
Very powerful.
And this has been sort of generalized out of the Python world through this Jupyter project to apply to many different programming languages.
So, this Jupyter project is an open source project run by Fernando Perez and some other guys.
I'm actually working on having them on the show shortly.
So, a couple of people have asked if they can be on the show.
And yes, we'll have somebody from iPython and Jupyter soon.
But these guys are an open source project that has just received $6 million in funding.
Yeah, which is kind of amazing.
I mean, it's a massive amount of funding for a project like this that's effectively a developer tool.
But it's so useful.
I think anybody – I was a little bit skeptical of it at first because I came at Python first from more of a computer science and software engineering background.
And it was later that I got into data stuff.
But to have something that you can basically record all of your actions – because when you're doing data projects, so often you're like, you need to explain – you get to a number or you get to a chart or you get to a discovery or whatever.
And then you need to communicate why that matters to somebody else.
And it only really matters if you can give the context.
So, well, like first I took this slice of the data and then I manipulated it in this way.
And then I extracted these features.
And then once I had those features, I decided to clean some of them up by doing X, Y, and Z.
And then once I'd done all of those things, obviously, if you look at this chart, it's really meaningful.
But without those steps up to that, it's like, that's awesome.
All I can see is like three bars.
To be able to play that back for somebody and capture it is really awesome.
And it's nice too when you're not a great coder, which I'm not, and you have to step through and go, wait, why didn't that work?
Let me run it again.
Why didn't that work?
Let me run it again.
Yeah, yeah.
They're very cool.
If I think of writing sort of science-based academic papers, it seems crazy to not do them as something like this, where rather than just saying, oh, I did something on my own.
Here's a chart.
Believe it, right?
Here is the actual code.
And here is the data.
And you can just have the whole thing and run it if you like.
Right?
That's amazing.
Yeah, totally.
To be able to reproduce research by literally running the code that led to the discoveries that informed the paper is huge.
I mean, Chris, my co-founder and the co-host on Partially Derivative, went and got a PhD.
And he talked about that all the time.
Often in postgraduate work, you get assigned a task to reproduce the research in somebody else's paper.
And he was always stunned at how difficult that was.
Because even when you had access to the raw data, trying to work with it in such a way that produced the same results was just, you know, it's just really rough.
And so being able to capture that process where like every little decision that you made to manipulate the data in a particular way.
And when I say manipulate in this context, I mean like a legitimate transformation of the data, not like a shady manipulation of the data.
Not like witness tampering type manipulate, but like we're going to make some assumptions about the underlying physics or statistics and then get a better answer.
Yeah.
Yeah, totally.
Well, and because so much of this kind of work is like it's a little bit about it.
There's some it takes some creativity.
It takes a little bit of intuition, especially if you're working with like natural language and you're trying to extract some you're trying to like make sense of unstructured data.
You know, you make a lot of little choices on the way and those do impact the results that you see.
And that's why being able to reproduce it is so important and for everybody to understand the assumptions that you made.
And so I think I mean, I can only assume that was part of why they received this like massive bucket of funding.
It's a super popular product.
So, yeah, it's going to definitely be a really important foundation of science, period.
But if you just think of any open source project, like what other open source project do you know that's not got a company behind it that somebody gave six million dollars like this?
This is a big deal.
This episode is brought to you by Hired.
Hired is a two sided, curated marketplace that connects the world's knowledge workers to the best opportunities.
Each offer you receive has salary and equity presented right up front and you can view the offers to accept or reject them before you even talk to the company.
Typically, candidates receive five or more offers in just the first week and there are no obligations ever.
Sounds pretty awesome, doesn't it?
Well, did I mention there's a signing bonus?
Everyone who accepts a job from Hired gets a $2,000 signing bonus.
And as Talk Python listeners, it gets way sweeter.
Use the link Hired.com slash Talk Python to me and Hired will double the signing bonus to $4,000.
Opportunity's knocking.
Visit Hired.com slash Talk Python to me and answer the call.
Okay, so number four on our list is artificial intelligence.
There have been some cool shows about artificial intelligence like X Machina and that actually happened to feature a little Python code in the show as well, which is cool.
But people are freaking out about it.
Yeah, this is this is kind of a new thing this year.
I don't know.
Maybe it's because machine learning and AI are becoming more a part of like mainstream conversations.
Or like you said, there was an entire movie made about it where they referenced the Turing test specifically, like in a major Hollywood movie that blew my mind.
Because that's pretty nerdy stuff in normal circumstances, right?
It's super cool.
But yeah, and now all of a sudden you have these super prominent leaders of the technology community coming out like for and against, like speculatively against the idea of AI because, you know, the matrix or whatever.
So I don't know.
It was really interesting to see that debate happening in public.
I don't know if you I don't know if you track that very much.
I mean, like Bill Gates was warning us and Stephen Hawking warning us about artificial intelligence.
And then, of course, other people on the other side giving really nuanced defenses of the way that artificial intelligence helps humans by making better decisions.
Or it was it's super fascinating conversation.
Yeah, it's very fascinating.
The person that came out that sort of in my mind carried the most weight, honestly, was Elon Musk.
Yeah.
Well, I mean, everything that Elon Musk is always when Elon Musk speaks, we listen.
That's right.
Because, I mean, Bill Gates, I'm actually a fan of his.
I think he did some really cool stuff.
But I kind of feel like he's, you know, has a certain worldview that's sort of already here.
And Stephen Hawking has some amazing views of the universe.
But at the same time, I'm not entirely sure how practical his actual interaction with AI and programming is.
But Elon Musk says, I'm going to build an electric car that's like amazing.
He built it.
I'm going to build things that go to space.
And somehow he does that.
I mean, he could actually build AIs if he wanted, if anybody can, I would say.
I think that that's the only way that he's really going to be a Bond villain, because at the moment he's got he's got like super cool, you know, international technology companies that are doing amazing things.
Like you said, going to space.
But it's it's not quite supervillain status yet.
And I think if he builds a robot that like seamlessly works its way into society and obviously initially for good.
But, you know, it gets out of control.
That's where that's where the plot of the movie really starts to get thick.
I'm excited about this.
Yes, absolutely.
The AIs had to be created to man the supercharging stations up and down the West Coast.
And then it just went all wrong.
They got in the cars and spread.
Exactly.
They decided that they deserved more than the menial tasks that they've been assigned.
Yeah.
So we'll link to a really cool article from this project that was sort of getting respected leaders in the AI space to sign sort of a pledge to proceed with caution.
But one of the follow up articles that we'll also list that I really liked was this thing about Mario as in like Super Mario Brothers.
That was really cool.
Yeah.
Yeah, it's pretty funny.
There were some researchers that like basically made Mario sentient.
I mean, maybe not quite.
That's a stretch.
But they empowered Mario, the character, with his own intelligence and then like would let him loose in the original game to see how well he could defeat the Goombas and all of the other dangers of Mario characters.
It was super fun.
It's like on the flip side.
There's these like really super accessible, fun artificial intelligence projects that aren't necessarily a threat to, you know, humanity.
Yeah.
Maybe it's a threat to our high scores on Mario Brothers, but not maybe humanity in large.
So, you know, those guys are, they're German.
They're in Tubingen at the university there, which is actually like 35 minutes away from me right now.
That's cool.
They made a video.
Yeah.
They made a video and we'll link to it.
It's on Mashable.com.
And they do all sorts of cool stuff.
There's a bunch of different intersections.
It's not just like one part of AI, but there's a lot of sort of understanding the world, understanding language, learning.
And so they would do things like they would ask Mario, like they can speak to him in English and he would answer in English.
It sounded a whole lot like war games.
It's like, you know, that really sort of choppy text to speech, that war games computer.
So it's kind of funny to hear Super Mario Brothers speak that way.
But they would say things like, jump, if you jump on Goomba, he will die.
And then they say, now jump on Goomba.
And of course the character dies.
And they say, what do you know about Goomba?
He says, if I jump on him, he will certainly die.
And then later they reset his mind and they tell him to go over and jump on Goomba.
They don't tell anything.
And then they later, first they ask him, what do you know about him?
He says, I don't know anything about him.
Jump on him.
The guy dies.
He says, now what do you know about him?
He goes, I know that he may die if I jump on him.
And it was really interesting.
He has all like different emotions and knowledge.
It's cool.
Check it out.
Yeah, it is super cool.
Actually, that's probably the best part about the project is like Mario's sort of existential,
like his existential ennui.
Yeah, absolutely.
So another thing that happened this year had to do with the New England Patriots.
And for those of you who maybe don't follow American football super close, this was sort of a big deal.
The New England Patriots, I don't really care one way or the other, but they'd kind of been seen as a team that, let's put it nicely, takes as much advantage of the situation as they can by, you know, maybe doing things they shouldn't.
That was so diplomatic.
And it had come to a head where around the Super Bowl time, they had actually been accused of deflating the footballs for their team.
And for a while, I didn't know what that meant.
Like, okay, well, maybe it hurts a little less to catch it.
I don't know.
But one of the things that deflated footballs let you do is hold on to them much tighter.
They're not slippery.
And so you won't fumble the ball and make some of these game losing mistakes or, you know, mistakes.
If you can, like, change the physics so it's not a problem.
Well, that's easier to solve than being better at football.
So the story is actually some data science folks came and looked at that.
And I don't know.
What's your opinion after looking at all the charts and graphs they built?
Well, I think it boils down to, in a simple way, that the New England Patriots were, like, a massive outlier.
And so you're right on that if you deflate the football a little bit, apparently, it makes it easier to hold on to.
And so for those of you who don't watch American football, it's a lot like rugby, if you're familiar with rugby.
Or more or less, it's like you have a ball that you're holding on to and you're running through a large group of very strong men who are all trying to take that ball from you.
And so you often lose it because it's a large group of really strong big men trying to take it.
And pretty much anything that you have in that scenario is not going to be yours for very long.
But the – and the trend overall in the league, the NFL, the league in which the Patriots play, was that the teams were having – were more successful, like, holding on to the football.
They had more plays that they could run.
So the ratio between the number of attempts that they made and the number of times they lost the ball was going up.
So you could run more plays and fumble less in general.
That said, the Patriots improved their ratio sort of exponentially more than all of the other teams in the league.
And so when you see this one outlier way at the top right of a graph, it tends to go like, well, that's not right.
Something doesn't make sense.
Something's different about that one data point.
And so, of course, that sparked some – it sparked a lot of speculation.
Like, why is it that the Patriots fumble so infrequently?
And, you know, where there's smoke, there's fire.
And in this case, at least.
And they found that the Patriots were deflating footballs just a little bit all of the time.
And by doing that, they were able to maintain better control of it because it was easier to grip the ball because there was less air inside.
So it was less buoyant.
Which – buoyant is probably not the right word to use in this context.
But whatever.
Whatever.
They could hold on to the ball.
And so that was the whole thing.
But basically, it was like the way that that was determined was through, you know, just doing some relatively simple data science.
Relatively simple analysis.
I mean, it's not that simple in the aggregate that the data – gathering the data was complex, connecting the dots, understanding the consequences of the things that we were learning.
But by and large, it was – you know, if you actually look at the analysis that detected that outlier that I just described, it's pretty straightforward.
So anyway, it's kind of cool.
And it was a super fun story to follow that ultimately was followed at such depth because, oh my gosh, sports in America, that it got a little tedious.
But at the beginning, it was cool.
It was like a wonderful little scandal.
Yeah, it was an interesting scandal.
And I think it really shows the power of data science because these sports guys, they can go back and forth.
They're talk radio and da-da-da-da-da.
No.
Look at that graph.
There's something going on here.
That's it.
The question is what is going on.
It's more likely something sort of sneaky is going on rather than they're just dramatically better than even second place, right?
So very interesting use of data science there.
I like that one.
So speaking of –
Yeah, it was super fun.
Yeah.
So speaking of U.S. things, the United States now has a person called the chief data scientist.
That's pretty awesome.
Yeah.
It's kind of amazing, right?
Like we kind of went from like data science like, whoa, okay, that's something like a bunch of geeks do.
And now there's like somebody who has a title like over the whole domain of the entire United States doing data science.
They're chief.
They're the chief data scientist.
Yeah.
I mean that's like almost like a politician has got data science.
This is crazy.
No, I think it's a really positive move.
I mean you think of places that have lots of data.
Sports.
We were just talking about that.
They've got a lot of data.
CERN.
At the large Hadron Collider, those guys generate a lot of data.
But the United States, we have so many different things that we track about people.
And the answers to those questions really matter, right?
We make policy based on those numbers.
So having somebody in charge of doing that right makes a lot of sense.
Yeah.
And actually, you know what's super cool about the stuff that DJ Patil, he's the guy who is the chief data scientist, the first one.
The stuff that he's focused on is really awesome.
Like a big initiative of his is that he wants to open up a lot of the data that the U.S. government collects.
So to keep the government agencies more transparent than they've been before and just to advocate for the open data movement in general and get the general public interacting with the data that these agencies produce in a way, almost like as a means of civic engagement.
It's really awesome.
And so this whole idea that like you're the sort of the general public's relationship with government can be more modern, can be more technology driven, can be more part of the 21st century.
It's really cool.
So the open data projects in particular have been really fascinating.
And then I know healthcare has been a big focus of DJ's office.
They've been really looking at, again, how to encourage the public to improve the way that the healthcare system works by investigating the data that's being made available.
And by making the data that we do produce easier to transport, easier to access, easier to sort of combine and investigate.
It's really I think it's it's making data accessible and kind of top of mind for an entire generation of kind of, you know, early career Americans.
It's it's it's really fascinating.
Yeah, that's cool.
And, you know, maybe the United States government themselves has access to things that you wouldn't otherwise share.
Right.
Like you mentioned health data.
I think one of the problems with analyzing health data in the large is people don't want to just give away every single thing about themselves for good reason.
And so it's hard to to talk broadly about that.
Right.
But maybe there's extra data in there somewhere that they can, you know, help make people healthy or something.
That's cool.
Yeah, totally.
And it's in general, I think part of this whole idea of like quantified social science, I think for the longest time, science, social science that's done about, you know, the behavior of populations has been kind of anecdotal.
A lot of, you know, a lot of people doing qualitative research where they're, you know, using their intuition and using their, you know, their knowledge to connect the dots and tell a good story.
And I'm sorry, that probably sounds super like diminutive of a lot of social science research.
And I only kind of mean it to be.
But there's a move towards saying, hey, you know, if you can't back up your statements with data, even if you have to, you know, be creative about how that data is collected or be creative about how that data is interpreted.
I mean, that's fine.
But if you're not backing it up with any data, then it doesn't really mean anything.
And I think that that's, it's very encouraging for people of my persuasion.
Yeah, I totally agree.
The other thing that's cool, you talked about the open data project.
I mean, when you live in a democracy, you would expect that the data, your government, you're supposed to be sort of the boss of the government.
You should be able to have access to those things.
And so that's a really positive trend.
And also on that note, I want to recommend a video by Catherine Devlin.
She did the keynote at Pi Ohio this year, and she works for some government agency, sort of a, let's see if I remember it right.
She works for a group of programmers within the U.S. government that are basically an open source wing of this programming group.
And so they'll go into other places and say, we will help you with this project, but only if we get open source the results.
And they're trying to spread open source within the U.S. government as well.
So let's tie together nicely.
Yeah, absolutely.
There's everybody kind of involved with the CTO's office.
Megan Smith is the current CTO of the U.S.
And almost all the initiatives they're doing are, it's like a totally different way of interacting with government than I think ever existed before.
So if you get a chance, definitely check out what the CTO's office is doing.
There's a lot of cool ways to get involved.
There's a lot of cool projects that they do.
And there's a lot of interesting open source projects like you just mentioned.
So, and a lot of cool data sets.
And which, and by the way, some of those data sets are really great for learning if you're just kind of, you know, cutting your teeth on data science.
There's a lot of really interesting things about your community and your state and the country in general that might be more fun than working with, I don't know, a data set about advertising clicks.
Or whatever.
This episode is brought to you by DigitalOcean.
DigitalOcean offers simple cloud infrastructure built for developers.
Over half a million developers deployed to DigitalOcean because it's easy to get started, flexible for scale, and just plain awesome.
In fact, DigitalOcean provides key infrastructure for delivering Talk Python episodes every day.
When you, or your podcast client, download an episode, it comes straight out of a custom Flask app built on DigitalOcean, and it's been bulletproof.
On release days, the measured bandwidth on my single $10 a month server jumps to over 900 megabit per second for sustained periods.
And there's no trouble.
That's because they provide great servers on great hardware at a great price.
Head on over to DigitalOcean.com today and use the promo code TALKPYTHON, all caps, no spaces, to get started with a $10 credit.
Let's move on to number seven.
It's going to be late December.
Winter's coming, probably.
Yeah, maybe.
It's hard to know.
What's the story of this one?
So there's a handful of things on here that really I was attracted to because I just love this idea that data science and just kind of data analysis in general is becoming so much more mainstream.
And so this post was about the Game of Thrones.
And you may know, this is actually a shame that my co-host Chris isn't on the show right now because Chris produced a data set from A Song of Ice and Fire, the books, not necessarily the HBO TV show, Game of Thrones.
Obviously, the two are related, although I hear the TV shows kind of going off the rails and moving away from the books.
Anyway, so there's a lot of dying in this TV show, Game of Thrones, and the books, A Song of Ice and Fire.
And my co-host made a data set of all of the battles where he tallied the number of deaths.
And obviously, there's some estimation in here because the books don't go into detail all of the time.
But so actually, hold on.
Let me take a step back.
So what I should probably tell everybody is if you're not familiar with Game of Thrones and the books, the series of books are called A Song of Ice and Fire.
If you're not familiar with them, it's basically kind of like medieval fantasy type stuff.
So there's, you know, dragons and there's houses and, you know, the houses fight with each other over land and there's different families that are at war with one another.
Kind of the whole thing, right?
And so, of course, because they fight all the time, there's a lot of dying.
And if you were of a more data statsy persuasion, you might want to know, like, quantitatively, which house is the best.
Like, who wins the most battles?
Who kills the most soldiers?
Who has the biggest army?
And these are the sorts of questions that Chris's data set answers.
So you should definitely go find that.
I think it's just Game of Thrones battles on GitHub.
If you Google for it, I bet you'll find it.
But there's other work, other very interesting work that's being done about the likelihood that you'll die if you're a character in the books or the TV shows.
And so I just loved it, right?
Because, I mean, it's a fictional world.
It's made up.
The likelihood that you'll die is whether or not the author decides that you should die.
Like, that's really what's going on here.
But based on the sort of constructs of the world that the author created, this data scientist went through and he used – there's a whole Wikipedia clone called a wiki of ice and fire.
Haha, snort.
That catalogs and documents the deaths of every major character, like every major death in the book.
And so he went through and basically calculated the likelihood that any given character will survive based on their characteristics.
So are they from a particular family?
Are they highborn?
Are they lowborn?
Are they a man?
Are they a woman?
How old are they, right?
All of the things that you might say are like a feature of a character.
And then once you understand the features of a character, you can calculate the likelihood that they'll die.
Anyway, it's fascinating.
I'm sure, you know, we'll link to the blog post so you can go check it out.
But it pans out kind of like you'd expect.
You have a much higher chance of dying if you're not of the noble class.
I think men die more often than women.
And so on and so on.
You should go check it out.
But really, the reason to bring it up is just to say doing data analysis about characters in a fictional universe, I think is we're at some kind of peak.
We're at like peak data science awesomeness when that's a possibility.
Yeah, that's really cool.
And you could also run this algorithm on your favorite character and know whether or not you should get attached to them, right?
Yeah.
I mean, I think to be fair with these stories, the answer is probably always no.
That's right.
Yeah.
I mean, readers of the books will know.
I'm not, I'm not, this is not a spoiler or anything.
I'm not giving anything away.
If you like a character and you hope for them, if you have any hope that the character will survive, that's it.
I think that's probably the best indicator that they're going to die.
It's the kiss of death.
Yeah.
So speaking of dying, we're all getting older.
And Microsoft has decided that machines can know really well how old we are.
So if you go to how-old.net, Microsoft is using machine learning to guess how old you are.
I know.
Isn't that, how old did it get?
Do you want to say?
You don't have to reveal if you don't want to.
I don't mind.
So I've always kind of looked a little younger than I actually am.
People are always like, what?
You have kids?
What?
So my whole life has been kind of like this.
Like it was a problem in high school because it's not cool to look younger than everybody else when there's only a four-year stretch.
But when I get older, it looks better.
So I uploaded a picture of myself, the one I have on my main website.
It's the one you probably see on Skype right now.
And it actually said I was 47.
I'm like, what?
I'm only 42.
This is crazy.
But I have my glasses on, right?
And so I'm like, I'll try one without my glasses.
Uploaded it.
It said I'm 42.
They hit it straight, right on exact.
Whoa.
What does that say about the algorithm, though?
That you got aged by five years just because of your glasses?
Yeah.
I think the glasses, well, they may trend upwards.
I would wreck, if you don't want to be old, take your glasses off.
That's all I got to say.
How accurate was it for you?
Yeah.
Was it in the ballpark?
I was similarly offended, actually.
I uploaded a photo of myself and it guessed that I was in my early 40s, which I'm not.
Which is fine.
Early 40s are great.
I'm happy for you.
But you shouldn't be in them when you're not.
I don't.
Well, when you're in your, like, I'm in my early 30s and I was like, wait a second.
Wait a second.
Okay.
So I thought the same thing.
I was like, maybe there's something about my appearance in the photo that is, you know, cause for concern.
Right?
I was like, okay.
Did it like search for ties or like a collar?
Maybe throw on like a hoodie?
Exactly.
Like put on a hoodie.
I found a photo of myself without a beard.
I don't, this is like super vain.
I was like, wait, I can't live with this.
So I found a photo, an old photo of myself without a beard.
I put it into the system and it guessed that I was like 37, which is also older than I am.
It's better though.
And so I've had to accept.
It's moving in the right direction, but I've had to accept that either, either.
It's just like everybody got offended because like the, if, like if you were Microsoft and you were training this algorithm, like wouldn't you want it to skew young?
Like, let's be honest.
Yeah.
Okay.
But let's say, okay, so it skews old.
Or, or I'm just old looking.
Like maybe I'm aged beyond my years.
I have a kid that, you know, this, the kids do this to you.
I bet, I bet it would have guessed that I was like 25 until the day she was born.
And then it was like 40.
One week later, you aged like 10 years like that.
Yeah, exactly.
Exactly.
So number nine on our list here is actually a little bit of a sad story.
There was this really cool study about how, how much a reasonable argument with some, not argument as fight, but like a logical argument or a discussion around a political position may or may not change someone's opinion.
And so this, this study was done trying to change people's opinion, open, open them up being more favorable towards same sex marriage.
And they found in this study that if they went around and they had people canvas the, the location, go and knock on the door and talk to people after talking to them, they could actually make them more open to this idea.
Which this flew in the face of a lot of political science, which is if you argue with somebody from an opposite perspective on something political, they typically dig in and like, ha, I'm way against you, man.
Right.
It like hardens them against your argument rather than brings them over.
So this was a big deal until it was a fraud.
Yeah.
Yeah, exactly.
And, and, and I think everybody was, I mean, it was one of those things where it was like, okay, because I think part of the, one of the big parts of the study was that, or, you know, purportedly.
Turned out not to be true.
Was that if the person who was having a discussion with you about your views on same sex marriage was themselves gay, then that would increase the likelihood that you would, your opinion would change.
So in a good way, like, or toward, you would be more favorable of same sex marriage if you were talking to a gay person.
Basically saying that, like, if you, this idea that the reason that most of us might hold a prejudice are simply because we're not exposed to people of a certain group.
And as soon as we are, our prejudices start to melt away, which is kind of a nice idea.
Right.
And so everybody was really excited about this survey.
But it, well, I mean, like we're talking about, it turns out that it was just made up.
And what's also interesting is the way in which some additional researchers discovered the flaw.
They were just, as often happens, they were going to do an extension of the survey.
They were going to build on the research that had already been published, which is pretty common.
But then when they looked into it, they discovered that the responses in the survey were pretty normal, like pretty consistent, which is actually like they were too consistent.
Right.
The pattern was too clean.
And what they actually found was that there were like irregularities in the data and the irregularities that they found in the data looked like the sort of thing that a human would do if they were trying to be random.
Like, isn't that amazing?
Like, so like actual randomness is hard to produce and like human beings just aren't good at producing things randomly because that's just not what we do.
Like we go like, oh, I picked a three last time.
So I have to pick an eight this time.
And I've already picked an eight.
So randomness says that I wouldn't pick an eight two times in a row.
I should pick a 12.
And that's actually not how randomness works.
So it's like that's super intentional.
Like you're layering your own bias on top of it.
And so what they saw was like very uniform noise in the data set.
And they were like, wait a second.
This actually looks like you just made it up.
And it's true.
Like I guess he took the results from a previous study and then tried to apply them to this one.
Like and it basically applied the previous study's results in a new context and then sprinkled some noise on top in a way that he felt would look random.
And these other researchers basically had to say this like huge study, this like seminal study that had been published in, you know, I forget which, you know, nature maybe or whatever.
Like one of them.
And it was covered in a ton of like high profile news outlets too, right?
New York Times and places like that.
So it got a lot of traction until it crashed.
Yeah.
Yeah, absolutely.
And so, I mean, on so many levels, it's like it's really – I mean, on the one hand, you look at this and you go, oh, what a horrible fraud this person perpetuated on, you know, the public.
And on the other, you go, well, I guess, you know, the process kind of works.
I mean, it shouldn't have been published in the first place.
Peer review should have caught that.
But ultimately, the academic community discovered the fraud and out of it, which I think was good.
But it was – so it's interesting in two ways.
It's just interesting how research, the kind of the mechanics of the research industry work.
And then on the other hand, it was interesting because it was this really – the thing that – this kind of novel idea about randomness that's not normally part of mainstream discussion that everybody had to start talking about in order to explain why the story worked out the way that it did.
Yeah.
How did they catch him making fake random?
I don't understand.
What does that mean?
Yeah.
That's really cool.
Yeah, yeah, exactly.
What does that mean?
Like, yeah, it was cool.
That's great.
And on this note, you know, I was seeing this story in a really positive light.
Like, hey, maybe we could sit down and talk to each other and we could, like, help evolve each other's opinions one way or another.
And we could kind of come to an understanding.
But it turns out that, like, no, probably we can't.
There's a really interesting book that I think is related to, like, data science.
It just has an insane amount of data analysis in it called The Big Sort, how the clustering of like-minded America is tearing us apart.
And it really just goes to, like, 20, 40 years of data of, like, people's opinions and working together.
And it sort of would not support the study that we can just talk about things and agree more.
But check it out.
That's interesting.
I'll have to go check out the book.
Yeah, it's an amazing book.
One of my favorites.
So, number 10, which I'm going to nominate to be my favorite of this year, is that Python is at an all-time high as a programming language.
So, the TOB index is one of the more respected sort of how popular is my technology and the technology fight indexes.
And Python is now number four of all programming languages.
And it jumped from eighth to fourth in one year.
So, it's one of the very few with, like, a double up arrow.
This thing is super growing fast.
So, for a language that was created in the late 80s, came out early 90s, and it's been around for a long time, this massive jump, you know, I think it's interesting to ask, like, where is it coming from?
It's coming from academics somewhat.
Like, this Python is now the most popular language for first-year college students studying computer science.
But I think it has a lot to do with data science as well.
Yeah, I mean, it's hard to see how it couldn't be, actually.
Because I think Python's always been great as kind of along the same lines as PHP, which I can only assume for all the software engineers who listen to this podcast because they like Python so much, I've pretty much just been sacrilegious.
So, don't get me wrong, you guys.
I hate PHP.
Those guys suck.
But also, Python is a pretty good general-purpose programming language, as is PHP, for building web applications and all of the, you know, all of the sorts of products that we've seen being released on the internet in the past, whatever, 10 years.
This huge boom in web-based products.
And Python is great for that.
It's great for writing software that gets released on the internet.
But at the same time, it's also overtaken another statistical programming language called R and become, I would say, the de facto language for doing data science and analysis.
And which is really cool because now it's a powerhouse.
Now you can do two things in the same language that used to be totally separate from one another.
So, we talked about before, there's statistical computing, which is basically like, I already know how to do the stats, but I need to script it.
And what language will help me do that, you know, almost like an Excel power user.
And then that goes all the way down through kind of, you know, complicated machine learning and artificial intelligence and neural networks and all that stuff.
And so, the fact that you can couple that with how do I respond to an HTTP request and hit a database and return some content, like, those two worlds used to be different.
But now we can build smarter and smarter applications that are seamlessly integrated with one another thanks to Python.
So, I mean, it's no surprise to me.
It's awesome.
Yeah, I think there must be a huge boost coming from that direction.
And generally, people are really jumping into it.
But, you know, thanks to data science for making our language look even more popular and awesome like it should be, right?
I know.
The only languages I think above it on the list are, like, really lame ones that nobody wants to write in.
They're probably, like, forced to by their boss.
Like, Java.
I mean, come on.
Yeah.
Let's be real.
Oddly, yeah.
A little tear just formed in my eye.
No, but seriously, like, Java literally is number one.
And C is up there as well, I believe.
It's pretty interesting.
But it's, you know, C and Java are fighting for first place.
Then it's C++.
And then it's Python.
Right?
So, that's beautiful.
Yeah.
I mean, honestly, kind of any dynamic language being as popular as that.
I think there are a lot of old school software engineers who are crying in their coffee at the release of this report.
You know, like all of that.
That's a toy language that will never be useful for anything in production.
And, yeah, now here we are.
Python's running some pretty massive, massive products.
So, it's very cool.
As I'm sure all your listeners know, they listen to your podcast every week.
Yeah, I'm sure a lot of them are building some pretty amazing stuff out there.
But, yeah.
I think this is great news for everybody who wants to get into Python.
It seems like the job possibilities, job prospects in regular programming as well as data science are just going up.
So, if I were betting on my career, I would consider Python one of the top choices.
Yeah, absolutely.
And it makes it so much easier to start to explore new concepts.
So, when you want to start, when you're ready to start bringing in kind of machine learning or any kind of statistical, you know, data processing into your applications, it's complicated.
But it's a lot less complicated when you're already familiar with the language that you're writing in.
So, I think it really gives you a leg up when you're trying to make that transition from one to the other.
Or vice versa.
If you've been, you know, for the partially derivative listeners, if you've been sort of scripting a lot in Python to try out, to do some research or to work on some analysis, and you want to start building applications to make the things that you're, make the products that you're building accessible to the world, there's a whole ecosystem waiting for you.
And the water is warm, my friends.
The water is warm and shallow.
Wade right in.
Yeah, exactly.
So, yeah, it's fun.
And it's just, it's a fun language to program in.
Let's be honest.
Yeah, absolutely.
Love it.
So, Jonathan, that's our top 10 list for the big news in data science this year.
So, I got to ask you, what is your resolution you're going to break this year?
You're going to give up on in two weeks.
Have you already decided what you're going to not hold up?
I already decided what I'm not going to do.
That's right.
Let's see.
The other people on my engineering team would probably appreciate it if my New Year's resolution was to write more unit tests.
Or better comment my code.
And I'm almost certain that neither of those things will happen.
So, let's call, I'll make, I'll do a dual resolution this year.
Unit tests and documentation.
That's a pretty safe one to throw out there, I suspect.
I think I'm going to make mine to actually put proper comments on my get check-ins.
Yeah.
Those could be some improvements.
Yeah.
Kind of get in a hurry and they're not so good sometimes.
Yeah.
Totally.
Like fixing stupid bug.
Yeah.
Like that's, I think, that's probably not as helpful as it could be to my fellow programmers.
It does express your emotional feel about the code in the check-in, but it doesn't really help them figure out what it meant.
Yeah.
That's fair.
That's a good resolution.
But you don't think you'll stick to it?
I mean, it's a hard habit to break.
I'm going to stick to it until I get to a super big hurry and there's some production bug to fix.
And I'm going to like probably skip it.
Not that I think that's a good idea.
I'm not recommending it.
I'm just telling you, these are the resolutions you make and you break.
Yeah.
I love it.
Like this is the recommendation.
So all of you out there who are learning Python and software programming, engineering for the first time, here's what to do.
Talk a big game and then ultimately just write a bunch of spaghetti code that nobody can read.
That's how the pros do it, you guys.
That's how the pros do it.
On that lovely note, I think we should probably call it a show.
Jonathan, this has been really fun.
Thanks for teaming up to put together an end of the year show for us.
Absolutely.
This has been a blast.
Thank you so much.
Thank you so much for suggesting it and having me on the show.
This has been a blast.
You bet.
Thanks.
Bye.
This has been another episode of Talk Python to Me.
It's a joint episode with Partially Derivative and Jonathan Morgan.
And it has been sponsored by Hired and DigitalOcean.
Thank you guys for supporting the show.
Hired wants to help you find your next big thing.
Visit Hired.com slash Talk Python to me to get five or more offers with salary and equity
presented right up front and a special listener signing bonus of $4,000.
DigitalOcean is amazing hosting blended with simplicity and crazy affordability.
Create an account and within 60 seconds, you can have a Linux server with a 30 gig SSD at
your command.
Seriously, I do this all the time.
Remember the discount code.
It's Talk Python, all caps, no spaces.
You can find the links from today's show at talkpython.fm/episode slash show slash
40.
And be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes and direct RSS feeds in the footer of the website.
Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.
You can hear the entire song on talkpython.fm.
Just look for music in the nav bar.
This is your host, Michael Kennedy.
I really appreciate you taking the time to listen and share this with your friends.
Smix, take us out of here.
I'm out of here.
Bye.
you