#145: 2017 Python Year in Review Transcript
00:00 Michael Kennedy: It's been an amazing year for Python. We've seen its meteoric growth continue to become the most popular major programming language. We've seen significant grants and funding come in for open source, and this just might be the year that the Python 2 or Python 3 question was finally settled. Join Brian Okken, my co-host from our Python Bytes podcast, and me, as we look back at many of the major milestones for Python in 2017. This is Talk Python to Me, Episode 145 recorded December 21st, 2017. Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy, follow me on Twitter where I'm @mkennedy, keep up with the show and listen to past episodes at talkpython.fm, and follow the show on Twitter via @talkpython. This episode is sponsored by Linode and Rollbar. Please check out what they're offering during their segments. It really helps support the show. Hey everyone, before we get to the interview, I want to share a quick update about our Python courses with you. Do you work on a software team that needs training and could really use the chance to level up their Python? Maybe your entire company is looking to become more proficient. We have special offers that make our courses here at Talk Python the best option for everyone you work with. Our courses don't require an ongoing subscription like so many corporate training options do, and they're roughly priced about the same as a book. We're here to help you succeed. Send us a note at email@example.com to start a conversation. And here's a bit of a timely announcement: we are giving away a PyCon ticket to one lucky individual. Every year PyCon in the US sells out. If you don't have a ticket, go buy a ticket, or you could become a friend of the show. Many of you already are, and thank you for that. If you're not, just go to talkpython.fm/friends, enter your e-mail address, and you'll be in the running. On January 8th we're going to do a draw, using a little random.choice, and figure out who the lucky winner is who wins a free ticket to PyCon. So be sure to become a friend of the show if you want to be a part of that. Now, let's get to the interview. Brian, welcome back to Talk Python.
02:26 Brian Okken: Thanks for having me back for this special episode.
02:28 Michael Kennedy: I always love these year-in-review shows, and it's really excellent that you could be here to do it together with me. You were on previously a couple years ago to talk about the testing column, how testing pyramid was broken and we need more testing column, right?
02:44 Brian Okken: I guess. I can't even remember what we talked about, so that makes sense.
02:47 Michael Kennedy: It's been a long time, but I do remember the testing column is now a thing after that episode. That was cool. People may know you from our work together on Python Bytes, our other podcast, they may know you from your Test and Code podcast, and they may know you from your book, right?
03:04 Brian Okken: Yeah, so this last year got the book Python Testing with pytest, with Pragmatic Publishers, published this last year, so yeah.
03:13 Michael Kennedy: Yeah, awesome. That's definitely one of the things we'll review as we go through here. We're going to get together today and talk about what we think were the major accomplishments, major pieces of news, not necessarily the major projects, say, in terms of projects you'll find on GitHub or something, but more like the trends in Python in 2017. I think that's going to be a lot of fun. Do you want to kick us off with what may be one of the biggest trends out there?
03:44 Brian Okken: Yeah, one of the biggest trends that we saw in 2017 is a dramatic shift from Python 2, or legacy Python, to Python 3, or modern Python.
03:54 Michael Kennedy: I love the term legacy Python and modern Python, or just Python and legacy Python. I think it's actually making a difference. And Matthias, who originally started that trend, well done man; that is awesome. I think this is finally the year where the question seems silly to ask. It's been a few years where, like, yeah, you can do Python 3, you can do Python 2, how is this progressing? It should really be going quicker, but to me it feels like if somebody says, "I'm going to start a new project with Python 2." you kind of smack them and go, "What? What are you doing man? Come on." It seems like that's the feeling in the industry these days, which is really good, right?
04:30 Brian Okken: I definitely think so, and I think there's been a switch, hopefully there's been a switch in the education, like in colleges and stuff, people using Python 3 instead.
04:39 Michael Kennedy: There's a couple milestones around this that we wanted to highlight. One of them is this article by ActiveState, the guys at ActiveState, called Python 3 vs. Python 2: It's Different This Time. They really lay out some of the things, some of the facts around this, for example, 95% of the top 360, don't know why they chose that number, but 360 packages listed on py3readiness.org, 95% of those are Python 3 ready. Traditionally, the blockage has been this sort of catch-22 around packages not supporting Python 3, right?
05:13 Brian Okken: Yeah, a couple years ago that was definitely the case of people trying to switch and they found out they had some key package that hadn't been migrated yet, but for the most part that's behind us.
05:24 Michael Kennedy: Yeah, absolutely. So Django played a big role this year in sort of putting the nail in the Python 2, or legacy Python coffin, right?
05:34 Brian Okken: Right. I mean, even just this month, Django 2.0 came out and it does not support Python 2, it's 3.4 and above, and with a little side note that 3.4's only going to be supported for another year-ish, so move up to modern Python.
05:50 Michael Kennedy: Yeah, that's awesome.
05:52 Brian Okken: 3.5 I guess, but I think if you're going to move up, may as well just move up to 3.6.
05:56 Michael Kennedy: Right, yeah. The only reason I think to not move to 3.6 is if you're working on like a Linux distribution, and it doesn't support it, or it's a complete pain to, like a non-standard, but even so, 3.5 already has async in a way, it has type hints, it has a lot of the magic features that you really care about, which is awesome. So Django, the current shipping version of Django does not support legacy Python. That's awesome, right?
06:21 Brian Okken: Yeah, that's awesome, and I think that Django, when it used to default to Python 2.7, I think that was part of what was keeping people behind.
06:29 Michael Kennedy: Even previous to this, I actually don't know the timing on when this happened, it's a little while ago, but they switched the default instructions from their tutorials, their getting started tutorials, from saying use Python 2 to saying use Python 3, and that made a significant statistical difference in just the consumption of Python 3 libraries on PyPI, just from the documentation switch, which was pretty powerful.
06:55 Brian Okken: Let's talk about some of the... There's a few other switches. Heroku switched their default to 3.6 recently.
07:01 Michael Kennedy: Yeah, that's awesome. Its was like 2.7.13 or something, so now if you go to Heroku and say, "Give me a Python app," you get Python 3 unless you say, "No, no; I mean legacy Python. "Excuse me."
07:10 Brian Okken: NumPy has laid out their plans for dropping Python 2. Looks like you've got until 2020.
07:16 Michael Kennedy: Yeah, the NumPy folks, they're not moving super-quick, but the fact that they say, "We are dropping Python 2 support." should put it on the radar of everybody in the data science world. Same thing; start moving. I think their motivation is very similar to the Django folks in that they've had extra work and extra burden. It's harder for people to come and contribute to the project because they have to contribute two versions sometimes: a Python 2 and a Python 3 version of everything, and they're like, "Why do we have all this code?" So they're like, "Not only do we think it's time to move, but we think we will have a better chance of adding new features and getting new contributors if we drop this Python 2 thing." so that's pretty cool.
07:57 Brian Okken: This was laid out at the last 2017 PyCon, was a talk with Instagram switching to Python 3, and how they did it on one branch, and that was amazing.
08:08 Michael Kennedy: That was insane, right?
08:09 Brian Okken: You're right. I think that was a great talk in that it showed everybody else, man, if these guys can do it with a very complex code structure, you can too.
08:18 Michael Kennedy: My biggest takeaway from that was, oh my gosh, they did that on the main branch. They made that change without branching, and they got it working in production and everything, which is pretty awesome. And the whole way in which they did that was cool, but my main takeaway was, whoa, if they can do it, pretty much anyone can. There may be a few companies, let's say Bank of America. They've got an enormous Python 2 code base, but unless you are like at that scale, this is sort of a case study in how you could make this happen for you.
08:49 Brian Okken: What's interesting is that we started the year still talking about trying to get people to move on to Python 3, and at the end of the year we're not really talking about that so much anymore. We're talking about some of the new, exciting projects that are Python 3 only, like API Star, and all the type hint libraries and stuff going on.
09:09 Michael Kennedy: That's really cool. API Star is by Tom Christie, same guy that did Django REST framework. He's like, "I want to re-envision a REST framework, but with Python 3 features." Really interesting use cases of things that are only available in Python 3. And then we have, you said we've got PyAnnotate, Mypy, and MonkeyType, which are all around this type hinting, type annotation thing, and it's available only in Python 3.
09:36 Brian Okken: I started the year not thinking type hints were a good idea, but I think they are now. Everybody else has convinced me.
09:42 Michael Kennedy: I try to use them judiciously, but I do really like them. I think on this section, on this topic, our first one, this move to Python 3, we should leave people with the Python 3 Death Clock, or the Python 2 Death Clock--
09:54 Brian Okken: What, pythonclock.org?
09:56 Michael Kennedy: Yeah, pythonclock.org, and so we have about two years, three months, and 14 days until Python 2 drops even security fixes, and so if you have a manager, or your team's like, "I don't know if we should really do this switch. We'll just stick with Python 2. We already all know it," put this up on a big screen somewhere and just let it count down. It's a little scary now. I think in about two years it'll be real scary.
10:20 Brian Okken: A similar timer shows up on some of the Python libraries' documentation, like Requests has that at the top, I think.
10:26 Michael Kennedy: Yeah, that's awesome. All right, so--
10:29 Brian Okken: Let's move on.
10:30 Michael Kennedy: Speaking of Requests, you'll find Requests on PyPI, right?
10:33 Brian Okken: Yeah, along with a whole bunch of other things.
10:35 Michael Kennedy: A whole bunch of other things. What if you misspell Requests?
10:39 Brian Okken: That would have been a problem for a while, yeah.
10:41 Michael Kennedy: Absolutely, so one of the things that happened this year was that hackers subverted PyPI, and they got 10 malicious libraries installed, and the malicious code is not in the library itself, they literally took the open source code and just put it into their library, but when you pip install things, it has the setup.py that runs to configure everything, and that can do anything, and so they put malicious code in there.
11:05 Brian Okken: It was a weird thing. It was kind of bizarre. What surprised me is that they had it running so that the actual package, the real code was still running, it was just the setup was all messed up.
11:18 Michael Kennedy: That's right, so you would not be able to determine that you had installed one of these virus packages by it not working, 'cause it was literally the same code. You imported, it ran, it just happened to be really, really bad. The way they did this was by typo-squatting, basically. If you type Requests, but you misspelled it somehow, they might create a package that is this virus-injected version by copying Requests and uploading it to PyPI under misspellings of these various things.
11:49 Brian Okken: Yeah, it's just wacky, but the community kind of got around this, and got it dealt with pretty quickly, which was pretty nice. A couple things they add: they new pip won't let you do misspellings of some of the common packages.
12:04 Michael Kennedy: You're not allowed to create standard library names, like they used urllib or something like that. That was not actually a package, but you would reasonably import it, so they could trick you into pip installing it.
12:17 Brian Okken: Anything that's been seen has been taken down, all of these malicious libraries are gone now, and then, well hopefully, and then somebody went around and squatted on a whole bunch of names to make sure that it wasn't taken by somebody else.
12:31 Michael Kennedy: Yeah, so they were like, "All right, we're going to just put a message or some sort of importer install warning here and just say, "No, you can't have these." There's also some utility that would generate variations, or misspellings, of your package so you could squat on your own if you run a popular package as well, which is pretty nice. I don't know, on one hand this is really disturbing, on the other, if you look at how many times those things were downloaded, they're in the 10s or 100s, not millions, so that's one thing.
13:02 Brian Okken: One of the good things about this, I think, that came out is... I mean, some people's first reaction might be, "Why didn't the team that runs PyPI, why didn't they proactively fix this or know about it?" And it highlighted the fact that all these people are volunteers, and maybe we need to do something about that.
13:21 Michael Kennedy: Well yeah, and PyPI itself is utterly critical for making the Python ecosystem work. Any time you set up a new machine, you set up a new project, you do an automated build, there's probably some sort of pip install -r requirements.txt process that happens, and that flows through PyPI. The amount of data that flows through there is super-incredible, and it turns out that there's just a handful of people, as in one or two, who work on this at all, and like you said, they're volunteers. One of the big pieces of news is that Mozilla came along to fix this, right?
13:55 Brian Okken: Right, so the next thing we were going to talk about was the huge grant, the huge, in open source terms, $170,000 grant from Mozilla, to help with the sustainability of PyPI, which is just awesome.
14:12 Michael Kennedy: Yeah, that is really, really great. I'm so glad to hear that they're doing this. On one hand, it's not a huge amount of money, but it really does put the foundation in place.
14:22 Brian Okken: A lot of this is going towards moving over to Warehouse, I think, so that'll be good. I think that'll help with a lot more people being able to contribute to the maintenance and update.
14:35 Michael Kennedy: That's really interesting. One of the problems with PyPI, the website, is that it's super-old and crusty. It's not like shiny Flask, it's not cool new Django 2.0, it's hand-rolled, custom couple of files, thousands of lines of funky code, and it's great that it works, but it really needs to be modernized, and so one of the things that this includes is moving over to what you already mentioned, Warehouse. Warehouse is the new implementation, and what you'll find at PyPI.org, not the PyPI.python.org/PyPI, why that's there twice, who knows, but the new one, and that's actually written in Pyramid, it's well-factored code with unit tests.
15:21 Brian Okken: I think that move is a great move and I'm excited to put the old version behind us as soon as we can. It still works great, but I understand when people complain about having to maintain some legacy code that's crusty.
15:33 Michael Kennedy: Yeah, and the problem was, people would want to come and submit a feature for it, and they would look at it and go, "Actually, nah, this is a complete mess. I can't work on this. I'm out of here." And so we have this sort of dual problem. One, that there's people who are working on this critical thing only in this volunteer mode, more or less, and then there are people who want to contribute to it, but its current state repels them, technically speaking.
16:01 Brian Okken: Exactly.
16:02 Michael Kennedy: So it's like a double negative.
16:04 Brian Okken: Those... some of this funding to try to convert over, then maybe we can get swarms of open source people to help out.
16:11 Michael Kennedy: Yeah, that'd be awesome. I'm looking forward to that little pre-release, pre-production warning at PyPI.org to go away. Thanks for Mozilla helping on that. And just related, really, really quickly, you talked about major open source projects getting funding. NumPy got a whopping grant for $645,000 to improve it over the next few years. That's awesome open source support right there.
16:36 Brian Okken: That's a lot of money, but that's a drop in the bucket for the people that are really leaning on data science a lot.
16:43 Michael Kennedy: Yeah, it definitely drives a lot. What I would just like to see is Django get that, Pyramid get that, Request get that. You know, so many of these popular open source packages, that are these third-party packages, are so critical to so many companies making billions of dollars on it, they could throw in a little bit of support.
17:03 Brian Okken: It does seem interesting that... I mean, Requests is used, for instance, as an example. There's so many like that, though, that are used by so many people, and they still have to have day jobs.
17:14 Michael Kennedy: That's right. Let's move on to something slightly political. I don't want to make this overly politicized, but I think there is something we really should touch on a little bit here, and that is, I can see part of the sides of the whole Democratic side of the US government, the Republican arguments some of the times, but this anti-fact, anti-science, anti-climate-science bent of the current administration, it's a real danger to a lot of what we really need to focus on. I mean, climate change, in my opinion, is the biggest challenge we are facing currently as humans, period, and it's been in danger.
17:56 Brian Okken: It doesn't even matter what side of the fence you're on. Taking data down is... Data on climate change should be... It's just data, it's not something we should argue about.
18:06 Michael Kennedy: Right. Facts should be facts, period. You can argue about the interpretation, but facts are facts, and they're really important for us to make the right decisions. One of the huge challenges is, there was a suspicion that when Donald Trump was going to become president, that many of these government-controlled, super-important data sources were going to disappear, like NOAA, the oceanic institute, EPA, National Renewable Energy Laboratory, all these different places, theoretically some of this data could be vanished.
18:41 Brian Okken: I don't know what the current state is, though, of some of that data, but some of it is, at the very least, harder to find.
18:47 Michael Kennedy: Yeah, and you hear about things like there being forbidden words at the EPA, like you can't use the word climate change, for example. How scary is that? So there were these data activists who decided, we're going to save this, right?
19:02 Brian Okken: Yeah, so they went off, and it started right at, what, right at January 20th, the inauguration, started copying as much data as possible, and storing them in European servers. That's a huge effort, but that's a pretty cool thing.
19:20 Michael Kennedy: Yeah, it is. There were 60 scientists and programmers who gathered at the Department of Information Studies in UCLA. There was a bunch more in Toronto, Philadelphia, Chicago, all over the world, and I'm sure Request, and Beautiful Soup, and a lot of that, was in full force, going around just sucking all of this data off these websites. You might say, "Okay, these people are paranoid." whatever, but in fact, right at noon, the day Trump was sworn in, a lot of the climate change pages on whitehouse.gov just disappeared. So those were grabbed just in time, I think.
19:54 Brian Okken: That's just weird, to take stuff down right away.
19:56 Michael Kennedy: Quite an interesting data hacktivist sort of story. I mean, they weren't actually hacking things, but they were definitely out there using code, and probably a lot of Python, to try to rescue the data, which is quite interesting.
20:07 Brian Okken: There definitely, probably, are people using Python, but also, one of the things that I'm glad we highlighted this story is that there is not a... You can't separate your interaction with the governments and the world, and the fact that you're a programmer, and in this story we see that programmers are able to save things that politicians might want to get rid of, so it's good.
20:31 Michael Kennedy: Was really heartened to see this. Quite an interesting story. This portion of Talk Python to Me is brought to you by Linode. Are you looking for bulletproof hosting that's fast, simple, and incredibly affordable? Look past that bookstore and check out Linode at talkpython.fm/linode, L-I-N-O-D-E. Plans start at just $5 a month for a dedicated server with a gig of RAM. They have 10 data centers across the globe, so no matter where you are, there's a data center near you. Whether you want to run your Python web app, host a private Git server, or a file server, you'll get native SSDs on all machines, a newly-upgraded 200-gigabit network, and 24/7 friendly support, even on holidays, and a 7-day money-back guarantee. Want a dedicated server for free for the next four months? Use the coupon code Python17 at talkpython.fm/linode. All right, so another thing that I feel like was really significant this year was the whole async and await, asyncio thing in Python.
21:29 Brian Okken: We saw a lot of new web frameworks come in in 2017. At least we heard about them.
21:34 Michael Kennedy: I don't know how many of them are production ready, but they're definitely on the way, which is pretty cool. One of the challenges for any of these web frameworks to be async-enabled... Maybe it's worth mentioning why that might be good. If, especially because you have the GIL, it's hard to add parallelism to your Python code in simple ways. But usually, what do web servers do? They call other web services, microservice type things, or external services like payment gateways, they talk to databases, and then they do a little bit of stuff, maybe they talk to the file system, and then they get back to you, but a lot of it is this kind of orchestration bit, really, and all of those can be pushed into the asyncio loop. If you use Postgres, there's a async version for Postgres, async library for consuming it, so you're going to wait, like a query, which is really awesome, but if your web framework doesn't support it, like Flask doesn't support it, Pyramid doesn't support it, Django doesn't support it, you can't use that, it's useless. You cannot use that in your website, so you really need a framework that will, all the way down through it, allow you to sort of release these awaitable coroutines.
22:48 Brian Okken: I'm probably talking through my hat on this one, since you're more the web developer--
22:52 Michael Kennedy: But we did see three interesting ones come along, and probably more. I guess I should probably throw aiohttp in there as well, but the three that come to mind, for me, are Sanic, which is a new web framework entirely built on alternate implementation. I think uvloop is what Sanic's based on. You can go and write code against your web framework, and use all of this async and await stuff. There's Japronto, which is really interesting, and Japronto, I think, caused a lot of a stir because they originally published an announcement on Medium, and they have a really, pretty impressive graph right on their GitHub page, saying, "We're getting millions of requests per second on a single server." They show it next to Go and Node, and it's like, I don't know, 20 times faster. I mean, it's a lot, lot faster than the other things. So on one hand they caused a big stir about people going, "Well, you're doing your benchmarking wrong, for this reason, or that reason, or whatever." Japronto's pretty interesting. The thing that's unfortunate is it's kind of stalled out in the last 10 months. It came on really strong, and there's just not a lot happening in the commit history.
24:09 Brian Okken: Interesting.
24:10 Michael Kennedy: Yeah, but Sanic, on the other hand, it's going pretty strong. A new one, that we recently covered on Python Bytes, is called Cort. Cort takes the Flask API and makes it async and await-able, so you could take the code that you know and make it work, basically, in this async world that you want.
24:29 Brian Okken: This one does seem to be fairly active as well.
24:32 Michael Kennedy: I'm going to have the guy who has created it on this show, on Talk Python, in a few weeks as well, so we'll dig more into it, but basically it's a Python asyncio web micro-framework with exactly the same API as Flask, but you can say your views can be async def method name, not just def method name, so it's really, really cool.
24:54 Brian Okken: Hats off to him for coming up with just taking the Flask API and reusing that. Why come up with something new?
25:01 Michael Kennedy: That's right. That's smart. I also saw Armin Ronacher talking about this kind of stuff, who's behind Flask, talking about some of these ideas and stuff around Flask on Twitter. I don't know if there's anything officially happening directly in Flask, but I wouldn't be surprised.
25:16 Brian Okken: That'd be a good thing to throw money at, hey, everybody throwing money at projects.
25:20 Michael Kennedy: Yeah, exactly. Instead of switching to Go or something, let's just make our web frameworks actually do what Python can do. One of the things that is at the core of all of our Python programs is CPython. If people want to contribute to it, how and where that's happened has kind of been a journey.
25:38 Brian Okken: I didn't realize the history. I knew that in 2017 it moved over to GitHub, but it's moved around a little bit.
25:45 Michael Kennedy: It has. It was somewhere where I actually don't even know really where it was pre-2006. In 2006 it was on Subversion, then they moved it to Mercurial. Back then, Mercurial, Git, they were kind of vying for action, but I don't know, Mercurial makes a lot of sense, it's written in Python, so Python should be hosted on Python, it's kind of meta, but in 2017 it really is the modern workflow for social programming in open sources is on GitHub these days. The big news is Python, CPython code, moved to GitHub.
26:19 Brian Okken: I'm not going to make a judgment call as to whether all of the activity on GitHub is a good thing or a bad thing: it just is. You can't ignore that most open source developers are using GitHub, so it makes sense to put that there too.
26:32 Michael Kennedy: Absolutely. It's like saying, "I want to go and do stuff on social networks, but I don't want to be on Facebook." It's like, well, that's just where a lot of it is. I'm sorry. Yeah, so this was actually done by, sort of motivated by Brett Cannon a little bit, and he said one of his goals was he would like the ability to review an external contribution, from submission to commit, all on a tablet, while at a beach, on Wi-Fi, and he can do that with GitHub, but he couldn't do that with Mercurial and the other things. So that's pretty cool. It was also motivated by people submitting patches, but the whole flow of reviewing it, the pull request style and stuff there, so it just wasn't really... The requests would sit idle for like a year. You'd make some change to CPython, and people wouldn't even pay it attention, right?
27:19 Brian Okken: No, that's terrible.
27:19 Michael Kennedy: That'll take away your motivation for sure, right?
27:21 Brian Okken: Yeah, and whatever we can do to get Brett on the beach more often.
27:25 Michael Kennedy: Yeah, that's right. So well done, well done everyone involved in that. The next one I want to talk about has to do with Python becoming a very, very popular language, and this is not an achievement, per se, of 2017, but this is where it certainly has hit some milestones, and the world has started to pay a lot of attention to it.
27:47 Brian Okken: The growth of Python... I mean, it's been growing fast for a long time, but having it overtake some of the other languages in certain surveys, happened this year, which is great.
28:31 Brian Okken: Yeah, it's great. Hopefully that's not because it's the most confusing language.
28:35 Michael Kennedy: What I thought was really interesting is, you can go read the article, The Incredible Growth of Python, on Stack Overflow, and if you look at the comments, and the comments, at least at the time when it came out, were turned on, so people were commenting at the bottom. It read a lot like people were sort of disappointed that their favorite language didn't make it: "Well, Ruby's used so much in Silicon Valley. Why is Ruby not here?" Or, "I think Python must be so complicated. That's why it's got so many visits. You should just use C++: something simple that doesn't need explaining." and you're like, "Wait a minute, no. There's no way that C++ doesn't need more explaining than Python and its language syntax." There's a lot of things that could contribute to this. I think, partly, PyPI might actually be part of it. Like if you want to talk about Requests, that's got a Python tag. You want to talk about Django, that's got a Python tag. You want to talk about SQLAlchemy, that's got a Python tag. All these different things around Python, that people use and love, also contribute to this fact here.
29:36 Brian Okken: Yeah, and we're seeing so many different groups, like UsePython, that weren't before. There's a lot of people not using R anymore for data science, machine learning, switching to Python. We've got technical people at universities. A lot of people that used to use visual programming in the science and engineering fields are using free Python tools now.
29:58 Michael Kennedy: Absolutely.
29:59 Brian Okken: It's just great.
31:04 Brian Okken: I don't think things switch too quickly, either. One of the things that came up, I think it was in this article, is the growth in high-income countries in Python is faster than a lot of other countries, the non-high-income countries, I don't even know what that means, but tend to follow by a year or two, so we should see even more growth worldwide in Python in the upcoming years, I think.
31:30 Michael Kennedy: I think it's awesome. On one hand I feel like people probably say, "Oh well, but programming's not a popularity contest." So what if it's more people use this versus that. A lot of people use PHP, and PHP is crappy. Well, on one hand PHP has a downward graph. One of my rules of thumb is, when you look at the trajectory of a technology that you want to bet your career on, that graph should have a positive derivative: it should be pointing upwards, preferably increasingly so. A second derivative being positive also good, but you should not necessarily bet your career on downward graphs.
32:12 Brian Okken: And also, if you think about what hangups people have when they start Python, it's not things like... Like C, you got to come up with memory management and all that stuff that's really hard to deal with. There's a lot of hard-to-deal-with stuff in C++. I spend a lot of my time in C++, so I kind of love it too, but the big hangup for Python is just the spaces thing, and that's easy to get over, and then you're on to making great things.
32:37 Michael Kennedy: Yeah, you're definitely on to making great things. As soon as you pip install a couple of amazing things you're like, "Oh wait: spaces? They're cool. I can deal with this." and then, after a week, you're like, "Wait, why do I have to put curly braces? What are all these symbols? This is so annoying."
32:50 Brian Okken: Yeah.
32:50 Michael Kennedy: Yeah, pretty funny, pretty funny. I think this is definitely interesting. People out there thinking about where they should focus for their career: definitely have a look at this article and draw your own conclusions. But I think it's really a generally positive thing for everybody involved in Python in a lot of different ways.
33:08 Brian Okken: Yeah, especially us.
33:09 Michael Kennedy: Yes, especially us. People love to listen to us talk about Python news. Awesome. One of the things I feel like has been the trend, really since before the Web, since the Internet came out, early William Gibson days, has been this real fascination with technology. Technology, it's almost assumed that it's going to make lives better. We can connect more, and look, if we connect on Facebook more we can share our views, and they might be different, and then we'll come to a greater understanding, and there'll be less fighting, and less disagreement and whatnot, and that's actually seemed to play out for quite a while, but I'm starting to feel like there's a trend to saying, maybe not so much anymore.
33:53 Brian Okken: Yeah, I guess I'm not quite sure where you're going with this.
33:55 Michael Kennedy: Well, there's a couple of really interesting articles, and sort of trends coming out around how technology is not necessarily, by default, a positive thing for human society, for society. There's a couple of articles. One is called Tech People Think They're Good Guys: They Aren't. That was in Wired, came out recently. There was one in the MIT Technology Review called The Dark Secret at the Heart of AI. And then the last one is The Real Threat of Artificial Intelligence on the New York Times. That's a pretty big turn from where we have been over the last 20 years.
34:34 Brian Okken: That's true, but I think in new technologies and programming, people don't really understand what it is, and so they're often scared of the unknown.
34:42 Michael Kennedy: Yeah, that's for use. I think the Tech People, They Think They're the Good Guys: They Aren't, this article in Wired, they talked a lot about how people in Silicon Valley, they really focused on Silicon Valley, but also in tech in general, they need to be aware that they're losing this view where, well, these startups, and these tech startups, they're good by default unless they prove otherwise. They kind of have to prove that they're doing the right thing. We used to talk about how big data was going to do a lot of good things for people, also the dangers of it, but Netflix recently had some kind of social media post that was like, "Hey, we just noticed that there are 10 people that watched this." I don't know, it was some Christmas show or something. "We just noticed there's these 10 people that watched the same Christmas show for 18 days in a row: how sad." or something like that. That's crazy, right?
35:36 Brian Okken: Wow.
35:37 Michael Kennedy: There's a lot of wondering, okay, well, why would you watch the same show 18 days in a row? Maybe it's just on in some store, and they always want to show this Christmas show. Maybe you fell asleep 17 times, you just want to finish it , like, "This time I'm going to finish it. I'm getting a coffee. I'm going to watch it." I can't even remember what the show was, but it wasn't something amazing. I feel like there's this interesting trend of people being more skeptical. We have Apple and the encryption war sort of thing going on, with Congress and the FBI at the beginning of the year. There's a lot of stuff there.
36:09 Brian Okken: Being cautious is a good thing, like people adding things like Google Home and whatever Amazon's thing is called--
36:19 Michael Kennedy: They're Echoes and various things--
36:20 Brian Okken: Yeah, into their house, and in other forms of presence in your life that didn't used to be there. We do kind of have to trust these companies to not do anything evil with all of that information, and yeah, maybe it's good to be a little skeptical, and have some people check it out, making sure that there's nothing weird going on.
36:40 Michael Kennedy: That's for sure. I do feel like the default behavior, or default assumption, was, "Well, it's Google: they'll probably be fine with my e-mail, or, it's Amazon, what are they going to do? I'm just buying a book from them." and so on. It's kind of not so much that way. I have an Amazon Dot, which is one of the Alexa-type things, and it's sitting here. I made it quiet so it wouldn't disrupt the podcast, and the way you do that is you press this mute button on it, but the color that it changes to when it's not listening is really interesting. It changes to a bright red ring, like there's something wrong and dangerous about it when it's not listening.
37:22 Brian Okken: Oh, interesting.
37:23 Michael Kennedy: You would think it would turn, I don't know, green, or white, or... Like, everything's safe, say your secrets, whatever you want, we're not listening. But no, it turns bright red when it's not listening.
37:34 Brian Okken: Interesting.
37:35 Michael Kennedy: Yeah, pretty--
37:35 Brian Okken: What does it look like when it is listening?
37:37 Michael Kennedy: It's normally dark, but then if it's listening to you it gets a blue ring that kind of whirls around. It's a pretty cool device. I really like it, but I only have it in my office, not the rest of the house. There's two other things around this that I also wanted to talk about, and those were, I guess they're the same subject, but from two perspectives, and that's sort of this artificial intelligence machine learning. So let me set the stage for you. Last year there was this experimental vehicle created by researchers at Nvidia, the graphics people, and it was an autonomous car, but it wasn't like what Google, or Tesla, or GM is doing, where there's some sort of conditional programming for it. They didn't provide it a single instruction on how to do what it did. Instead, what they did is they taught it how to drive by having it watch and observe what humans do to cars while humans drive.
38:32 Brian Okken: Hmm, yeah, that's pretty interesting.
38:33 Michael Kennedy: It's pretty interesting, but then it leads... This type of thing is not just done with cars, although it's pretty illustrative with cars, it's also done with mortgages, with cancer diagnoses, various other things, police activity, so if this car crashes in unexpected ways, if it's going down the street and then it just swerves over and crashes into a tree, how do you know why it did that? You can't ask it. It's just, well, it learned pretty well, and most of the time it drives right, and then sometimes it doesn't. So there's some really interesting ramifications of living with machines that are integral to our lives, that can't be separated from human society once we depend upon it. Like, at one point we didn't need the Internet, but imagine if we took it away now, the economies would crumble. There's actually a proposal for a law in the EU saying that you need the ability to have the computer tell you why it made a decision about you. So if you are denied for a mortgage loan, why? Things like that. So these full-on deep learning AIs, that just learn through observation, they can't tell you why they did a thing. They're not looking at themselves, knowing why.
39:49 Brian Okken: Possibly not, but I think that systems can be made such that they are interrogatable.
39:54 Michael Kennedy: Yeah, I hope so, but I think it's a really interesting trend that we're seeing more and more of this AI stuff appear, up and down our interactions, where we know less and less about it.
40:04 Brian Okken: A couple of interesting things about that is, now I don't have the reference for it, but there was some talk about, how soon should be have driverless cars on the streets? One view is that we should make sure that they're very, very safe, and another view is that they just need to be safer than people. If you lower the death rate, even if there still is a non-zero death rate, it's still better than what we have now, and that's interesting.
40:33 Michael Kennedy: I think that's a super-interesting trade-off, and from a high-level ethics perspective, I think you say, soon as it is safer than human drivers, if by deploying this technology we save lives, it seems like the right thing to do, but if one of the behaviors of that life-saving AI is, "Well, there was an oncoming car, so I swerved and drove off a cliff to avoid hitting it, and killed my own driver instead of, I saw there were five people in the oncoming car, for example, so I saved four lives, or maybe even more, by driving this one person off." The news coverage and the human shock of being in control, or subject to one of these types of systems, it's going to be really interesting to see how that plays out. I still, from an ethics perspective, I think soon as it's ready to be safer.
41:21 Brian Okken: But I remember a lot of these questions, like the five versus one, these came up with, I remember these in philosophy in college, and I never really thought it would actually apply to technology right now.
41:34 Michael Kennedy: That's a really interesting point, 'cause neither did I. I remember, well, if there's a burning building, and there's a baby inside, and there's these people over here, and you could only save either the baby or the three older people, which is the right... You know, these weird trade-offs. You're like, "Well, nobody's ever going to do that. Nobody's going to make those conscious decisions." They'll sit there and go... At best it's like you made a split reaction in an emergency, and you did what you did, but when you're building these systems, you're like encoding it, basically.
42:03 Brian Okken: Yeah, it's like, "No, this isn't a rhetorical question. We really need to know: which one should we pick?"
42:08 Michael Kennedy: That's going to take some serious adjustments, I think.
42:11 Brian Okken: Yeah. Since I don't know a lot about machine learning and stuff, when I had Katharine Jarmul on Test and Code, I asked her this, is it even possible to build systems that are interrogatable? And her answer went over my head, of course, but she was confident that if those laws were in place, that it's going to be possible, so that's a good thing.
42:31 Michael Kennedy: Yeah, that's interesting. I think, if you design for it, you can probably, at least get some kind of... you probably can design for it, but we'll probably have to do that as an explicit requirement, right?
42:43 Brian Okken: Yeah
42:44 Michael Kennedy: 'Cause otherwise you just make it work, and you're happy, right?
42:46 Brian Okken: Yeah, and the interesting thing about that is, I think, it will benefit otherwise too, the reasons why artificial intelligence system came up with a conclusion might surprise us, and it might teach us more about how we can better make decisions even without the AI.
43:02 Michael Kennedy: When Google beat the world champion in Go, the top Go player started to study the non-intuitive solutions that the AI was applying to the game to beat this guy, and they started becoming better, and learning new techniques. It's pretty interesting.
43:20 Brian Okken: And that'll hopefully happen in detecting problems in CAT scans and things like that.
43:26 Michael Kennedy: Right, reliability in airplane motors, so they don't crash and stuff. Awesome. I definitely think it's both a very positive and kind of, we're going to go back to philosophy class as a society as well. Pretty interesting. This portion of Talk Python to Me has been brought to you by Rollbar. One of the frustrating things about being a developer is dealing with errors. Ugh, relying on users to report errors, digging through log files, trying to debug issues, or getting millions of alerts just flooding your inbox and ruining your day. With Rollbar's full stack error monitoring, you get the context, insight, and control you need to find and fix bugs faster. Adding Rollbar to your Python app is as easy as pip install rollbar. You can start tracking production errors in deployments in eight minutes or less. Are you considering self-hosting tools for security or compliance reasons? Then you should really check out Rollbar's Compliant SaaS option. Get advanced security features and meet compliance without the hassle of self-hosting, including HIPAA, ISO 27001, Privacy Shield, and more. They'd love to give you a demo. Give Rollbar a try today. Go to talkpython.fm/rollbar and check them out. Another big piece of news that we got was that Python is starting to appear inside of some major applications.
44:43 Brian Okken: Yeah.
44:44 Michael Kennedy: Surprising ones, I would say, actually. They definitely surprised me.
44:47 Brian Okken: Yeah, we may have jumped the gun on the Excel and Python, but hopefully that'll happen.
44:51 Michael Kennedy: Yeah, if we can encourage Microsoft to do that. I think it's definitely one of the highest-voted things, but it's not a thing yet. Let's start with something that is out right now in 2017, and that is SQL Server, Microsoft's database server, which is ubiquitously installed inside these large corporations. It's like inside these large Microsoft Windows enterprises, SQL Server is just the de facto database. What's interesting is now Python 3.5 ships inside SQL Server so you can do in-process machine learning against your data.
45:30 Brian Okken: Oh, that's great. Yeah, cool.
45:32 Michael Kennedy: Yeah, so if you got 100 gigabytes of data in your database, instead of trying to suck that across the network, or move it around, you just write a stored procedure that is in Python, and you can use all the Anaconda libraries, TensorFlow, all that kind of stuff in-process, which is really going to be pretty awesome.
45:51 Brian Okken: Yeah, that's great.
45:52 Michael Kennedy: So that is a concrete thing that has happened. A thing that may happen, has got a lot of news in the last couple weeks, is Excel.
46:00 Brian Okken: I'm not sure who this came from, but I think it was part of the, what is it, people vote on new features or something?
46:06 Michael Kennedy: On UserVoice they have a, like, what new features do you want in Excel, and this is, I think, the most popular one.
46:11 Brian Okken: And then they sent out kind of a survey to find out why people want it. We pushed it on Python Bytes to try to get more people to answer this survey.
46:19 Michael Kennedy: I personally don't care a whole lot about SQL Server or Excel, but especially with Excel, if... Currently what you use to program Excel, to automate it, is this thing called VBA, which is a crappy... Even for Visual Basic, it's like a crappy, cut-down version of that. It's like a bad programming language. So there's this push to get Python to be the new macro programming language of Excel, inside Excel, which would be really awesome because within companies, Excel is probably the biggest database, in quotes, that companies have, and so many people work inside of Excel, and are sort of Excel experts, and if we could get all of those people somewhat interested in Python, that would be just positive across the board, I think.
47:09 Brian Okken: Yeah, definitely. Be great.
47:11 Michael Kennedy: Yep. So everyone out there, go upvote this thing on UserVoice. We'll put the link in the show notes.
47:16 Brian Okken: It might increase the use of Excel, which might not be a good--
47:19 Michael Kennedy: Yeah, that's true. Maybe it would be negative. Try to keep that down. The last one for the year has to do with Google, right?
47:27 Brian Okken: Google implements Python on Go, is that what we have here?
47:31 Michael Kennedy: That is what we have. The people at YouTube created this thing called Grumpy, which, I don't know about naming open source projects, but if you want people to love it and to be attracted to it, calling it Grumpy might not be the best name, but it's, instead of having Python implemented in C, as it is in CPython, it's that underlying implementation written in Go, with some of that... Go's sort of claim to fame is, like, asynchronous from the scratch, parallel from the very core of it, so if you want to do the parallel, asyncio-type stuff we were talking about, then having Go as the foundation makes that incredibly easy and powerful.
48:13 Brian Okken: Any idea where this stage is? Is it still something that's going along? Are people contributing to it?
48:18 Michael Kennedy: That's a pretty interesting question. I'm not entirely sure what's going on with this.
48:24 Brian Okken: I actually think it's a great name, 'cause people Grumpy about having to use Go can use Python instead.
48:29 Michael Kennedy: It's not super-active. The last checkin was November 21st, so that's about a month ago, but this also could just be, how much is Google pushing this out. I think what this was, was really a research project for, what would it look like if we implemented the foundations of Python, instead of with C, we'll do it with Go. The reason is, this is from the YouTube team, and they process... Well, YouTube is running the front-end parts on Python 2.7, I believe, and they are processing over a million requests per second, and so they have, let's say unique parallels and requirements. So they're like, "Okay, well what could we do if we wrote this in Go?" and it's pretty interesting, but it's, like you sort of pointed out, it stalled out a little bit in the last four or five months, so I'm not sure if it's going to go somewhere, if it's being worked on internally and then they're going to release it, or... I don't know.
49:32 Brian Okken: Maybe somebody from YouTube can contact us and tell us what's going on?
49:34 Michael Kennedy: I did have them on the podcast. I had a guy named Dylan Trotter on the podcast, who was a major player in this at YouTube, and talked about it, but that was back on Episode 95, so the news is not super-fresh.
49:50 Brian Okken: One of the things you brought up here in the notes is that we often think about Python as just CPython, but this Grumpy is another version that's running on Go, but there's a whole bunch of others that--
50:02 Michael Kennedy: There's a crazy bunch of others, for people who don't know.
50:04 Brian Okken: Yeah. So you have another article called How Big is the Python Family?
50:08 Michael Kennedy: Yeah, this is from the CheckIO guys, who have the gamification of Python learning, and they wrote this article called How Big is the Python Family? We already have CPython, Jython, IronPython, for one variation of Python on .NET, then there's Python for .NET, there's Cython, there's PyPy, there's MicroPython, and now we have Grumpy, and there's also some other with Java that I can't remember the name of, but it wasn't in the article.
50:35 Brian Okken: I think that's cool that people are tackling it for different things. I'm really excited to see MicroPython keep growing.
50:41 Michael Kennedy: Yeah, I'm actually really excited about MicroPython. Those of you who don't know, MicroPython runs on microcontrollers, so like $5 chips, and basically your Python program is the running operating system on there, and you can just do whatever you want. You can do insane things like you can actually take a lambda expression and hook it directly to a hardware interrupt on the microcontroller.
51:03 Brian Okken: Yeah, that's just awesome.
51:04 Michael Kennedy: It's such a cool little project. Nice. All right, well that's our year in review. I know we missed a bunch of things, and if you feel like something's important, go to talkpython.fm/145, and put a entry in the show notes, but before we kind of round out what we want to talk about, there's a couple of honorable mentions left there, Brian.
51:24 Brian Okken: I think it was notable that you, both the podcasts Python Bytes and Talk Python, hit every week of the year in 2017, which is pretty awesome.
51:34 Michael Kennedy: Thank you. Yeah, that's actually quite awesome, and thanks for your help with that, because I went and like tooled around Europe and the Middle East for like four weeks, and you and some other folks kept it rolling, so awesome.
51:46 Brian Okken: Oh that's right. We had some guest hosts. That was fun. And Test and Code had seven episodes. I'm going to try to get an eighth out before the end of the year. That's not as impressive, but still going.
51:57 Michael Kennedy: No, but you did 52 weeks of Python Bytes as well, and that was awesome. Those were very fun, and the ones that you did have on Test and Code were really good. The Chaos Monkey and some of the other ones, really, really nice.
52:10 Brian Okken: All right, thanks. One of the most, like, the most recent one with testing in machine learning and data science, with Katharine, was really popular, which is great.
52:20 Michael Kennedy: That's cool. She does a great job.
52:21 Brian Okken: The other thing, we didn't bring them up because they weren't huge stories to talk about, but I think there's a trend to more and more people embracing virtual environments, with things like pipenv coming along, and I think that's a good thing. People should be using more of those, so yay.
52:38 Michael Kennedy: Yay, yeah, absolutely. We have pipenv, we got tox, and there's some other ones as well. I think fade was one that would do interesting... There's a lot of libraries that are really trying to make virtual environments just seamless, and it's really nice.
52:53 Brian Okken: I have been using PyCharm a lot, and it has picked up its game with using local virtual environments too, and that's a good thing.
53:01 Michael Kennedy: I love that. Yeah, you create a new project, it says, "All right, new virtual environment, in a well-known folder, within this project. Unless you want to say otherwise, this is what we're doing." I really think that's the right move.
53:11 Brian Okken: Yeah, it's great.
53:12 Michael Kennedy: Awesome. Another big thing, of which you hinted at at the beginning, is you shipped your book, which was quite the long labor of work, but you have your Python Testing with Pytest out, and that's been going strong, right?
53:24 Brian Okken: It's been phenomenal, and people have been sending me pictures on Twitter, of them and their book all over the world. I've got one picture from France, and one in... all over, a couple in Africa. It's pretty cool to see all over the world. It's great.
53:40 Michael Kennedy: Yeah, it's been great to sort of sit on the sidelines and watch the reception of your book. It's on Pragmatic. You can get it there in paperback, or digital, and you can also get it on Amazon, right?
53:49 Brian Okken: Yeah, definitely.
53:50 Michael Kennedy: Awesome. Okay, so it's time to ask you the two questions that are always at the end of the show.
53:56 Brian Okken: All right.
53:56 Michael Kennedy: All right, so what is your... If you're going to write some Python code, let's say it that way, what editor do you open up?
54:01 Brian Okken: Now, it's PyCharm. When you interviewed me a couple of years ago it was Sublime, but I have fully switched to PyCharm.
54:09 Michael Kennedy: Interesting. That's my favorite as well. Everyone knows I go on and on about it, but tell me why. What was the motivation?
54:15 Brian Okken: The integration with pytest, of course, is pretty darn good, and they're working with me to try to get it even more integrated, which I'm pretty impressed with the development team to be responsive to some of my requests.
54:27 Michael Kennedy: Yeah, it's awesome.
54:27 Brian Okken: It's great.
54:28 Michael Kennedy: They are very responsive.
54:28 Brian Okken: The other thing is I write a lot of Markdown, and the Markdown plugins make writing Markdown in PyCharm just very, very nice, so it's kind of side-by-side viewer, and it's cool. I like it.
54:43 Michael Kennedy: Yeah, nice. All right, and notable PyPI package?
54:45 Brian Okken: Well, pytest--
54:47 Michael Kennedy: Awesome.
54:47 Brian Okken: but I want to add that when I'm using pytest, lately I'm always adding the pytest HTML plug-in from Mozilla, to make reports. It's pretty cool.
54:59 Michael Kennedy: Okay, nice to get better HTML-friendly, pretty output, and not just text log.
55:11 Michael Kennedy: Yeah, we started the show talking about, with our first item, talking about the Python 3 and the type annotation stuff with PyAnnotate, Mypy, MonkeyType, and all that. Pytest has some PyAnnotate integrations as well, right?
55:25 Brian Okken: Oh right, yeah. I forgot about that. That's pretty cool.
55:27 Michael Kennedy: Yeah, I can't remember. Was that pyannotate-pytest, or pytest-pyannotate, or pytestannotate? I don't know. I could put the link in the show notes, but yeah, it's pretty cool.
55:35 Brian Okken: All right.
55:35 Michael Kennedy: All right. All right, well I think it's been quite the year in the Python space for 2017, so any final thoughts on what you've observed?
55:46 Brian Okken: I just want to thank everybody for their support of both this podcast and all the Python podcasts out there, and I want to mention to people that if you like this sort of banter of a couple of people talking about what's going on in the Python world, you should also check out Python Bytes if you haven't already.
56:04 Michael Kennedy: I also want to say thanks. That's a great point. People out there who listen, and give us feedback, and really enjoy what we're doing, that's why we can do this, so thank you everyone, and definitely visit pythonbytes.fm and subscribe over there, because we do this kind of stuff more or less weekly, but in shorter form over on Python Bytes.
56:22 Brian Okken: The number of people that showed up to say hi to us at PyCon, we had a booth there, that was an incredible turnout. Lot of people, lot of listeners coming up and talking with us. It was great.
56:33 Michael Kennedy: Yeah, that was really, really awesome, and we're going to do it this year in Cleveland.
56:36 Brian Okken: Yeah, definitely.
56:37 Michael Kennedy: Brian and I have a booth lined up, along with some other folks, and so be sure to stop by in 2018. And get your PyCon ticket, because it's going to sell out very soon.
56:46 Brian Okken: Definitely. I got mine.
56:48 Michael Kennedy: I got mine as well. All right, Brian. Thank you so much for getting together and sharing this look back at 2017 with me.
56:53 Brian Okken: Yeah, thank you.
56:54 Michael Kennedy: You bet, 'bye. This has been another episode of Talk Python to Me. Today's guest was Brian Okken, and this episode has been brought to you by Linode and Rollbar. Linode is bulletproof hosting for whatever you're building with Python. Get your four months free at talkpython.fm/linode. Just use the code Python17. Rollbar takes the pain out of errors. They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed, until your users complain, of course. As Talk Python to Me listeners, track a ridiculous number of errors for free at rollbar.com/talkpythontome. Are you or a colleague trying to learn Python? Have you tried books and videos that just left you bored by covering topics point by point? Well, check out my online course Python Jumpstart by Building 10 Apps at talkpython.fm/course to experience a more-engaging way to learn Python, and if you're looking for something a little more advanced, try my Write Pythonic Code course at talkpython.fm/pythonic. Be sure to subscribe to the show. Open your favorite podcatcher and search for Python. We should be right at the top. You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm. This is your host, Michael Kennedy. Thanks so much for listening. I really appreciate it. Now get out there and write some Python code.