Monitor performance issues & errors in your code
00:00 Michael Kennedy: Python is said to be a language that comes with batteries included. That has many meanings depending on the level you're focusing on. The lowest, it's a very rich and expressive language. Most commonly, this phrase means Python has a very powerful and comprehensive standard library. Itertools or ElementTree, anyone? But more holistically, most of the batteries are external, interchangeable ones, from PyPI. Right now, on PyPI.io, there are over 97,000 packages listed there. That's fantastic. However, it does lead to a discoverability problem. What options do you have for admin backends on web apps, how about cryptography, or raw socket programming and so on. This week, Matt Makai is back with me. Remember him talking about deployments on episode 26 last year? This time, we're talking about Awesome Python, a website and open source project attempting to categorize and sort of rank these options. This is Talk Python to Me, episode 96, recorded January 16th, 2016. Welcome to Talk Python to Me, a weekly podcast on Python: the language, the libraries, the ecosystem, and the personalities. This is your host, Michael Kennedy. Follow me on Twitter, where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter, via @talkpython. This episode has been sponsored by AnacondaCON from Continuum Analytics and by Hired. Thank you both for supporting the podcast. Hey, Matt, welcome back to Talk Python.
01:53 Matt Makai: Thanks, Michael, it's good to be back.
01:55 Michael Kennedy: It's great to have you back. You just pointed out a little bit ago that it's been 70 episodes since you were on episode 26, talked about deployments with Python. I can't believe it's been so long.
02:06 Matt Makai: It's hard to believe. It's been, like, a year and a half, almost two years.
02:09 Michael Kennedy: Yeah, yeah, that was a great episode, it was really popular, and you know, the world has changed a lot since then. So a lot of cool things we're going to be talking about today. We're going to focus on stuff that we found, mostly you found, that's awesome in the Python community. But before we get into those, let's, you know, 70 episodes ago, I'm not sure everybody can remember back 70 episodes. What do you do day to day?
02:33 Matt Makai: Sure, so I'm a developer evangelist at Twilio. And Twilio makes it easy for developers to add communications, so if you want to send a text message or you want to make a phone call with Python, or any programming language, or add video chat to an application, you can do that with Twilio. What I do as a developer evangelist, I'm first off a software developer. I write a lot of code. Most of my code is open source, so it'll be examples for how to use Twilio, or even just how to create something as a developer. It doesn't have to use Twilio. And then I travel and speak at tech conferences, all on software development, no marketing fluff, just really being a software developer. I get to write code, explain code, and teach other people, which is just absolutely awesome.
03:20 Michael Kennedy: Yeah, that's great, I've always thought that these developer evangelist jobs were just fantastic, right, just like, it's part research and play, it's part telling people about stuff that's cool, and you have a lot of freedom to kind of learn and explore, right?
03:32 Matt Makai: Yeah. I mean, the big thing for me is just getting the time to create projects. Like, when I have an idea, I can just go sit down, write some code, implement it, and then show it to other people. It's really, it's one of my favorite parts about it.
03:45 Michael Kennedy: How much does that feedback tweak or inform the APIs that Twilio uses?
03:50 Matt Makai: Oh, a lot.
03:50 Michael Kennedy: From the evangelists.
03:52 Matt Makai: Yeah, I mean, one of the best parts about the gig is I get to use all of our APIs before they come out, so before anybody gets their hands on them. Myself and some of my colleagues are testing out the APIs, giving feedback to the API teams, and that helps to improve the results that ultimately get into the hands of developers.
04:10 Michael Kennedy: Oh, that's great. Now, if I go out and search for something on Python, there's a very good chance that something that you are responsible for, that's not directly part of Twilio, will be, like, the number one Google result for my answer. You do Full Stack Python, right?
04:23 Matt Makai: Yeah, so I'm the author of Full Stack Python, which is a open source project that started over four years ago, and the whole idea behind Full Stack Python was, I'm a Python developer and I had, this is before I joined Twilio, I had a bunch of junior developers that were joining my team, and they kept sending me emails, hey, what's a WSGI server, what is that, and I kept sending all these emails, and I was like, this would probably be relevant to much more than just our team. So I started putting together a project with pages that had explanations on every single Python topic, and it's just expanded out from there. I pretty much work on it every day and it's read by over 80,000 developers a month at this point.
05:02 Michael Kennedy: That's fantastic. And you have a book that goes with it, as well, right?
05:03 Matt Makai: Yeah, that's what I was on for episode 26, back 70 episodes ago. It's about, it's a step by step guide to deploying Python applications. So if you've never done it before, you've built your first web application, this is every single step to get a virtual private server, and to stand everything up that you need: DNS, the WSGI server, first proxy. All of those things that trip developers up, and certainly tripped me up several years ago when I was trying to deploy my own applications.
05:31 Michael Kennedy: Yeah, it feels to me like that kind of information, those kinds of things, are the things that you kind of, it's a lot of work and a lot of research to get it all, all the pieces combined just right, so your web apps work great, but then you write it down, you put it in a script, and you forget about it. So having a book as reference to learn it the first time and keep it around is really cool.
05:50 Matt Makai: Yeah, exactly. And as we talked about on that episode, first, I teach how to do it manually, and then we automate it with Ansible. So it's a pretty good way to kind of understand the nitty gritty and then automate it, and you know what, if you want to forget about it at that point and just use the scripts, you can, but at least you learned it the first time.
06:05 Michael Kennedy: Yeah, absolutely. You'll understand it when you come back, right. You'll know what you're doing. If you need to come back and tweak the automation. That's cool. Now we're going to talk about Awesome Python, the project, the GitHub project, but before we get into that, how about we do a quick update on our Python for Entrepreneurs course that you and I are writing together?
06:21 Matt Makai: Yeah, let's do it!
06:21 Michael Kennedy: Yeah, let's do it. So that course is almost done. I have to say, this one is, how shall I put it, longer than I expected. Taking a little longer to write than I expected, but I haven't even told you, I've finished almost all the chapters I have to record, they're off to editors. I have basically one and a half chapters left to record, and I'll be done, and that just leaves a few more to go, and that class will be all wrapped up.
06:46 Matt Makai: Yeah, and for me, it's just finishing up some of the git, the second git chapter, and then we're going to get into deployments and SEO and everything that goes along with actually running the application that you build, so I'm really excited, I'm so excited to get this thing out to people, I know people have been watching it in early access mode, which you get from just buying, pre-ordering the course, but I'm excited for it to be, like, really wrapped up and done.
07:11 Michael Kennedy: Yeah, I really hope this turns out to be a great resource for the Python community, around through this whole life cycle of idea to web app to deployment to business. It'll be fun. Okay, so we'll come back and let people know when that's actually finished, but it'll be soon, I promise, those of you who are out there waiting. So let's talk about some stuff that's awesome.
07:30 Matt Makai: Sure, so the Awesome Python repository has the great slogan, "Life is short, you need Python." And it is one of many awesome ists. So the Awesome Lists popped up on GitHub where people just started throwing together lists of tools and resources that they were using, and so this repository came up in the Python community, and so this is at GitHub vinta/awesome-python. It was created by Vinta Chen, about a little over two years ago, and this is kind of the main one for the Python ecosystem that goes over all of the different open source projects and categorizes them, whether they're a task queue or web framework, has some sort of basic categorization for each of the projects.
08:16 Michael Kennedy: Yeah, I think this is really handy when you don't know what it is you need for something. And so I don't know how many projects are listed here, I probably could have gone through GitHub source code and written some script or something to figure it out. But it seems to me like it's close to like 500, 400 projects, and they're all categorized, like you said. Here's the admin backends, here's authentication, here's caching, and maybe under caching, there's, like, 10 choices on how you can do caching and plug that into your apps. So if you don't know what you need or what even a good choice for caching is in Python. You're just like, "Hey, I know I need caching, but that's as much as I know", you know, drop into Awesome-Python.com and then you can go to the caching section and pick.
09:03 Matt Makai: So I think for me, as a fairly senior developer, what I think these lists are most useful for is if you've already been programming in one language, say you're a Java developer, I actually came from Java and then started in Python several years ago, and I didn't know what the equivalent to a web application server was, well, that's a WSGI server. And with the Awesome Lists, I can see at a glance, like, what should I be using for caching in Python? So if you were already familiar with all the conceptual ideas, like a task queue, like web framework, then you can see the implementations in a different programming language. So you could go to the Awesome Ruby repository and see the equivalence to the Python projects that are out there, in case you just happen to want to learn Ruby, or you're on the job and you have to suddenly write some Ruby code.
09:52 Michael Kennedy: Yeah, I think, exactly, if you know, I need caching, and you want to say, these are the five things I should probably research, and you could compare them pretty quickly. Because you open up the GitHub page, you can tell, does this have five stars or 5,000 stars, right, how active is it, what are the deployments, things like that, it's really good for that. If you totally don't even know what you need to look for, you know, maybe something like Full Stack Python makes more sense, right? You want to talk about how you see this differently?
10:19 Matt Makai: Yeah, so there's a bit of difference between an Awesome List and what I'm trying to create with Full Stack Python, which is, the Awesome List is really a laundry list of all of the open source code that is out there that might be applicable to your project, but again, going back to the task queues, for instance, if you're a junior developer or you're in a boot camp or you're just starting to program, and you don't know what a task queue is, well, seeing a list of implementations really is not going to help you, it's not going to, Celery, RQ, and these different implementations, you don't know what they're going to do for you. So that's why I have, like, a task queue page that explains the conceptual idea, and then explains the differences between the implementations. So I think what's missing from an Awesome List is the context, but if you're a more senior developer, you've already been working in one ecosystem and you're trying to switch over to another one, or you're just trying to see what other implementations are out there compared to the implementations you've been using, I mean, it's great for that. It just kind of gets past the verbiage and gives you all of the implementations that you need.
11:21 Michael Kennedy: Yeah, I find, having been a programmer for a long time, I find these lists really, really helpful for discovering things that, you know, maybe are not in the top of Google, but actually turn out to be just the thing that I need. When you're totally new, maybe it's not the best place to start, but still, even if you're new, go check out the project, you'll find it interesting, I think. So you and I chose some things out of this Awesome Python list that we thought would be fun to highlight as part of the show.
11:50 Matt Makai: Yeah. My criteria for this, so we picked 10 libraries at random that are listed in the Awesome Python, and by the way, Talk Python to Me is not one of the 10 we chose, but it is listed on Awesome Python, as it should be.
12:02 Michael Kennedy: And so is Full Stack Python.
12:04 Matt Makai: And so the criteria that we used were projects that were not completely obvious to the community, but that are almost everywhere, or that have a ton of potential to really impact the Python ecosystem. So for example, we didn't choose Django, because Django is talked about widely and used widely in the Python ecosystem, but we did pick a CMS that is powered by Django under the cover. So this CMS runs on top of Django and we'll get to that in a few minutes. So that's sort of the criteria we used to pick 10 projects at random that we wanted to highlight because they're important to the Python ecosystem or they have potential to become really important in the future.
12:46 Michael Kennedy: I think that's really interesting, and really great, because it's, while we could do a show where we just go and say, let's just go find the 10 most downloaded packages on PyPI, that would actually be pretty interesting, but I think here we're going to hopefully expose people to things, they're like, "Oh, I didn't even know that existed, that's going to be awesome, I'm gonna go check it out." So let's see if we can deliver on that, huh?
13:06 Matt Makai: Exactly, all right, you want to dive in?
13:09 Michael Kennedy: Yeah, let's get started. So the first thing that I'm really happy to see that's on our list is something that helps us move from legacy Python to Python or some people say from Python 2 to Python 3.
13:19 Matt Makai: Sure, yeah, so this is the library six, so six is a Python 2 to 3 compatibility library, and this is one that you're not necessarily going to build an entire application around, but it is incredibly useful for developers who have built a Python 2 library and now they want to support both Python 2 and 3. So you can think about it as a bridge, six is a bridge from Python 2 to Python 3, and allows you to support both ecosystems at once. So it's really providing utility functions to smooth over some of the differences between the Python versions. And the goal is really to have that Python code that you're writing for your own library be compatible with both Python 2 and Python 3.
14:07 Michael Kennedy: Right, so where there's some kind of difference, you know, like, raw input versus input, or bytes versus str, things like that. You can use this library and it will know which version of Python it's running on and it will switch between them, right?
14:22 Matt Makai: Yep, exactly, so if you're using Python 2.6 or above, Python 2.6, 2.7, and then you want to support Python 3, for example, the syntax compatibility, so we've got print functions that have changed, we have the way that Unicode is handled, all those things need to be either smoothed over in your own code or you can just use six as a dependency, and it's only one Python file, which is awesome, so you can just copy this into your project and then this helps you to smooth over those differences that would otherwise come up because you're trying to support both 2 and 3.
15:00 Michael Kennedy: Right, and hopefully you can do this without having, if Python 3, this code, if Python 2, that code, but right, just leveraging this library, and presumably, somewhere deep inside of it, it's doing those types of things for you. So you mentioned some of the areas that you can help with. What else does it cover, where might I use this?
15:18 Matt Makai: For example, if you have unit test assertions in your code or you're renaming modules and attributes, so there's actually a huge laundry list of areas that you may not be that intuitive if you're a Python developer, that if you're building a library for others to use, it just crops up as you're looking to port from 2 to 3, and that's really where this would come in.
15:42 Michael Kennedy: Yeah, okay, cool. I remembered there's, like, a module, something like copy_reg in Python 2 and it's copyreg without the underscore in Python 3, or vice versa, you know, things like that, right, like, why are these so just barely different, this is crazy. Anyway, okay, so yeah, if you've got some Python 2 code that you want to bring forward or maybe you can even use it to help somewhat bring a Python 3 library to support Python 2 if you're writing some kind of package, right?
16:10 Matt Makai: Yeah, yeah, and the reason why I picked this one to highlight is it's really everywhere. Many of the really popular, most commonly used Python libraries rely on six library in order to smooth over the differences in Python 2 and 3, and so it's sort of the guy behind the scenes who's helping everybody out, but no one really talks about as, like, the main person, it's the guy behind the curtain.
16:34 Michael Kennedy: Yeah, okay, awesome, so check it out, six. And it's named six because two times three. Yeah.
16:38 Matt Makai: Yeah. There you go, nice.
16:40 Michael Kennedy: Let me take just a moment and tell you about an offer from one of our sponsors, Continuum Analytics. They're hosting AnacondaCON 2017. It's the first conference for open data science leaders around the world. This is the definitive gathering place for the Anaconda crew, whether you're a new or a long-standing member of the data science community, focused on business or technology, AnacondaCON will help you conquer your biggest data science challenges. Over 20 industry experts will discuss the open data science movement and share their best practices for leveraging Anaconda. You'll hear how they're discovering cures for rare genetic diseases, performing city planning, and refining tax policy analysis with Python. Team Anaconda wants to help empower your whole team so they're offering a two-for-one ticket price at AnacondaCON 2017 from now through January 31st, 2017. Register today at talkpython.fm/acon to take advantage of this excellent offer. That's talkpython.fm/acon. Thank Continuum for supporting Talk Python, they help make the show possible. Now back to the interview with Matt. There are a lot of data scientists and actual straight scientists that listen to this show and they might be happy to hear about your number two item here.
17:58 Matt Makai: Yeah, so I picked bokeh, or you can also pronounce it bouquet. In the official FAQ for the project, it says you can pronounce it either way. I'm not sure if there's a more right version, but we'll call it Bouquet for now. And so what this allows you to do is write Python code and create data visualizations, and so I have tried to create data visualizations in many different JavaScript libraries before. The most popular right now it seems like is D3.js, and I love those libraries, but when I'm in Python land, sometimes I just, I can't think about switching over into the JavaScript world, and so bokeh allows you to just write the Python code and then it will build the visualizations for you. Now, it's not relying on D3.js, but it draws visualizations that are very similar to that, and so you're not just getting a cut-rate data visualization. This is actually building something that is, you can include it in your web application and it is nice and professional.
18:59 Michael Kennedy: Yeah, this is a really, really cool library. And if you're already doing some kind of scientific computation or numerical computation in Python. You don't want to, like, drop out and drop a bunch of data into a JavaScript file or to a HTML file and then write a bunch of JavaScript, D3 to actually turn that back into something visual. You can basically do all the visualizations in Python and then just plug it in. And what's really cool is, it's not just generating, like, a picture, say, with Matplotlib. So with Matplotlib, you could say, here, I'd like to have this picture, boom, save it to an image and then put the image on a page, but these things, these bokeh graphs and visualizations that go up there, they're totally interactive, right?
19:42 Matt Makai: Yeah, exactly, so it's not just, as you said, it's not just an image that is generated. It is the actual visualization that is fully interactive as well, and so you can control how much interactivity and what data and what axes you want the user to be able to explore, but you can do that all in Python itself. And I actually, I found out about this project a couple of years ago at PyCon 2015, Sarah Bird gave an awesome talk where, you can actually go find it on YouTube if you search, "Sarah Bird PyCon 2015", and she drew an entire map with just bokeh. And I was just like, whoa. You just wrote, she live-coded some Python code, and I was like, whoa, you've got a map on there with all your data? That would have taken me a week in D3.js, I was just so impressed with it.
20:27 Michael Kennedy: Yeah, it's really, really cool. Okay, so another thing that's really popular in Python these days, it seems, is static site generators, right?
20:38 Matt Makai: Yeah, that, well, and Pelican, which is a static site generator, which is not the project we picked, but Pelican, I use every single day to create Full Stack Python, so I picked a separate one, and this one is either pronounced make-docs, or MkDocs. I actually just gave a talk on static site generators at San Francisco Python, and I tweeted at Tom Christie, who's the creator of not only Django REST Framework, but this project as well, MkDocs. And I said, hey, I want to check myself before I wreck myself, how do I pronounce this project, and he said, well, you know, I say mick-docs, with a Scottish accent,
21:15 Michael Kennedy: Mick-docs!
21:17 Matt Makai: Okay, I'm not--
21:17 Michael Kennedy: just checked the mick-docs.
21:20 Matt Makai: I don't really have a Scottish accent, so okay, sure. And then he said, but you know, I don't really know, let me talk to one of the other core committers, and he said, m-k-docs, that's how he pronounced it, but they don't really have a particular opinion about it. So you know, whatever is easiest for you. If you've got a great Scottish accent, I say go for that route, but I certainly do not. So what MkDocs is is a static site generator really for project documentation.
21:46 Michael Kennedy: Let me ask you a quick question or a clarification just for everyone listening, because many people would know but not everyone may know: what's a static site?
21:53 Matt Makai: Yeah, so a static site is kind of taking you back to the origins of the web. It's just HTML flat files, HTML and it has CSS and usually JavaScript, but there's no database backend, there's no dynamic content on the backend. So it's really just files that can be served up by a web server and the reason why it's called a generator, so that would be the static site, the generator part of it is you write your content either in Markdown or restructured text or some sort of markup format, and then you have templates. So for Pelican, for example, Pelican has Jinja templates that you can use, so Jinja's a template engine, and MkDocs has its own, actually MkDocs makes it even easier. You don't even have to write the templates if you don't want to, you just write the content. And what that does is, it combines the content that you've written with some sort of template engine and then produces HTML files, or some sort of output, it could be XML, whatever you want, but most commonly, you're producing a static site which would be an HTML.
22:53 Michael Kennedy: Okay, yeah, that's really cool, and there's a lot of benefits to that. It's super easy to scale a site like that, right, you don't have to worry about the databases and concurrenct execution, you're just serving static files, and it's also easy to put that onto, like, a CDN sort of thing, like fast light or something, right?
23:11 Matt Makai: Yeah, so the way that I deploy Full Stack Python is, it's just a bunch of HTML files that get generated with some CSS and JavaScript, and then I just have a content delivery network, so I have the files hosted, sometimes I use GitHub pages, I've also switched it over to Amazon S3. In front of that is a CDN, and I use CloudFlare, and what that does is it just serves up all those files, there's no database calls or anything like that, it's just really fast, immediately serving up those files to whatever web browser is requesting them.
23:43 Michael Kennedy: Nice, and it serves them up, it moves them around the globe and serves them up from some nearby place--
23:47 Matt Makai: Exactly.
23:47 Michael Kennedy: Which is pretty awesome, so people on the other side of the Earth don't suffer because you decided to put your server in New York or whatever.
23:55 Matt Makai: Globally distributed, exactly.
23:56 Michael Kennedy: Yeah, that's awesome. Okay, so what's cool about MkDocs? Why is it a great one or how does it work?
24:03 Matt Makai: I think the thing about static sites is they've traditionally been made around blogs, but project documentation is one of the best examples that I think, if you want to use the word use-cases, that I've seen. So the idea behind MkDocs is, it's already being used to power a really popular project, a Python project, which is Django REST Framework, and that allows MkDocs to have a real project using it, and then they open sourced it so that anybody can use it. And the idea is, generate beautiful project documentation. Now, for many developers, documentation is sort of not the first thing on their mind, but in the Python community, we have, I think we have really great documentation for most of our projects. It's kind of something that is a badge of pride. And so that's where, I think, using like MkDocs and just writing your content, you don't have to worry about the styling if yo don't want to, it's just going to create a beautiful set of project documentation for you.
24:59 Michael Kennedy: Yeah, that's excellent. And I think, you know, you could even check in the docs into your GitHub repository, and if somebody clones your repo, they can just go, like, right there and just have offline docs as static files, it's really cool. Very nice, okay, MkDocs. You know, speaking of Markdown, I think having Markdown wherever you accept user input is really nice in your web apps. You know, Stack Overflow definitely popularized this. Markdown comes from John Gruber, long ago, I don't know, like 2008 or something like this. It's not a brand new idea, but certainly using Markdown to accept input from users is much safer than accepting pure, you know, rich-formatted HTML, right?
25:39 Matt Makai: Yeah, so the thing about Markdown is, it's a certain format, and there's, well, for better or worse, without getting too into the weeds, there are different variations on that format, but there is sort of a standard Markdown format, and there's a Python project that parses that out. So then you can work with it in your Python code. And so the Python Markdown project is kind of everywhere. Whenever you're talking about a content management system or a static site generator, like MkDocs, or Pelican, which I use, really, when you do pip-install Markdown, what happens is that a Markdown parser is installed and allows whatever libraries you're depending on to then parse out that Markdown and work with it in their own code.
26:20 Michael Kennedy: Yeah, that's really cool. So, for example, like, Python Bytes, the new podcast that I started, so pythonbytes.fm, on there, there's a backend CMS type thing that I created for it, and there's a place where you put the show notes, and on Talk Python, that was just an HTML fragment, but every now and then, I would mess up that HTML fragment and it would screw up the site or, you know, I just have to format HTML in the browser and write it by hand and it's not so fun. So when I did Python Bytes, I'm like, that's it, this part is going to be input as Markdown, output as, you know, HTML. And so I use something similar to Python Markdown for that, which is really cool, I actually used this thing called Python-Markdown2, I don't think they're related, these two things. It's nice because it has extensions. You can add on, like, variations, like I want to add, like, GitHub table support type things and stuff like that.
27:14 Matt Makai: Yeah, again, this is just another one of those libraries that's everywhere, and whether they're using Python Markdown, Python-Markdown2, or there's many other parsers that are out there. They're kind of the guy, the gal behind the scenes is doing a lot of the heavy lifting and allows other projects to take advantage of that work that's done.
27:29 Michael Kennedy: Nice. So while we're on the topic of, hey, I want to put some stuff into some kind of backend on my site and have it show up, one thing you can do is write a site from scratch and fill in a few pieces, like show notes and episode names, or you could grab a whole CMS, right?
27:45 Matt Makai: Yeah, so Django can often be used as a CMS, which is a content management system.
27:51 Michael Kennedy: I would say, like, WordPress is probably the most popular of these, right, if you look at overall deployment, so maybe Squarespace, but plug it into Django is really sweet, so you found a really nice one.
28:02 Matt Makai: What happens is if you start building a site on Django, and particularly if people that are not developers are using it, you often find that you run into limitations, and that's not, I wouldn't say that this is necessarily something that Django has to fix. This is where a content management system comes in. So let me give you a specific example. If you have people that are not developers, that are punching their content into this content management system, like Django, the Django admin panel, for example, and then you want to roll back, you say, oh shoot, I deleted a page, you know, let's roll back, well, Django doesn't have that type of capability and that's where content management systems are on top of that. So the one that we picked here is Wagtail, and Wagtail is actually a newer content management system than some of the other ones like Mezzanine or Django CMS, but Wagtail is, I've been really impressed with Wagtail, I've used it a bit, both as a non-developer user to punch content in and a little bit on the backend, as well, and it's been really powerful so far.
29:05 Michael Kennedy: Yeah, that's really great. And they've got a great example, they have a demo video for it, as well, for Wagtail, and it was used at the Royal College of Art, it's in the UK somewhere, and it basically powers that website with thousands of webpages and many, many users, and all sorts of really cool work flows. It's actually quite visual and pretty. One of the things that comes to mind when I think of CMSs is like they seem synonymous with slow and clunky to me.
29:36 Matt Makai: There's a trade-off with CMSs, which is that they're building all this extra functionality on top of an existing web framework, and they need to be used by people that are not developers. So you don't want to get into a situation where the CMS is preventing you from doing things with your content that you want. So there's a trade-off, sometimes CMSs are too heavy handed, sometimes they're just really confusing to use, but I've found that Wagtail, which we're actually using at Twilio, is actually one of our favorite CMSs, and we did a large evaluation with a lot of different programming languages, and this is the one that came on top for us. It's not necessarily the best for everybody, but it's the one that works really well for us.
30:15 Michael Kennedy: I can say, the Royal College of Art website is quite zippy, even though I'm like nine hours time zone away from it, so any sort of slowness would be ping time. So I feel like this is a CMS that doesn't have that feeling, right. It's really nice, and it's awesome, because you're using it at Twilio, too.
30:32 Matt Makai: Well, the thing about Wagtail is that they've done a great job with making it really polished, and I think that's one of the reasons why we were so impressed with it. And so this is what's powering the Twilio docs. We actually look at our documentation, the documentation for the Twilio APIs, as a platform itself, because many different teams around Twilio are using the docs. They're writing docs, they're updating the docs when things need to be fixed, and so we needed some sort of powerful platform that allowed many different teams to interact with it. So after our evaluation, we used Wagtail as our CMS for the Twilio docs.
31:05 Michael Kennedy: Yeah, it's really cool. I'm looking forward to finding a place to use it, because it looks like it's fun to use, the backend looks real friendly and modern, not like some crummy sort of app.
31:14 Matt Makai: It's nice, it's a really nice and polished project, really. I think a little bit newer and up and coming as a CMS.
31:19 Michael Kennedy: Awesome. And it's cool that it was on the Awesome List. Okay, so another thing that I think is really interesting is design patterns, I've always loved design patterns, you know, I read, I remember reading The Gang of Four book about C++ and small talk design patterns, like, way long ago, and then reading it again, so I can actually understand it, like a few years later. But one of the things that I think is interesting is if you take those patterns and you apply them to Python, sometimes they look very different, and sometimes they look very similar. Like, for example, the decorator pattern. You know, that has one form in C++, but it's almost a language feature in Python.
31:58 Matt Makai: Yeah, that's where, on the Awesome List, is the Python Patterns repository. So there's a link to Python Patterns, and this is the implementations of various patterns. Whether it's like behavioral patterns or structural patterns, different conceptual ideas and they're implemented in Python code. Which allows someone, for example, when I was going from Java and I had constantly used the singleton pattern or the factory pattern, to then see what that looks like in Python.
32:27 Michael Kennedy: I think it's really great for people who are coming from another language. They already have these ideas in their mind, right. They can drop in really quick and you just pull up, like, you know, 20 lines of code, and you're like, oh, this is how you do it in Python. I never would have thought to do it this way, this is so much better than what I was trying to do a minute ago. And they have some interesting ones in here. They have the standard gang of four ones, like state, strategy, iterator, visitor, and so on, but then they also have some things like the borg design pattern, and the blackboard design pattern, so I actually learned a few things, and I don't really know yet where I'd use the borg, but I'm pretty sure I like the name of it.
33:04 Matt Makai: This repository is great for learning. I would caution anyone that is coming from another programming language to suddenly drop different patterns into their code when those patterns may not be idiomatic to the way that you would write Python code. Because Python code has its own way of, a different way that you would write Python code, for example. Making everything explicit rather than implicit like you would in the Ruby ecosystem. So I think these are awesome for just learning about how different patterns could be implemented, but I would caution people to make sure that this is actually how you would write Python code as opposed to just dropping them into your code without thinking about them.
33:44 Michael Kennedy: Yeah, I kind of have, like, an Einstein type of philosophy about design patterns. Like, design patterns are awesome, and you should use them in your code, but just enough, right. Things should be as simple as they can but not simpler, right? Like, you should have patterns that solve problems but I feel like people, when they learn design patterns, at first, they're like, oh, I'm going to put patterns everywhere, I can't wait to turn everything into a state pattern, and then the visitors go in here, like, oh no, oh no, that's going to be unmaintainable, right? And so patterns, I don't know, to me, they're like salt, right, a little bit here and there.
34:16 Matt Makai: Yeah, and if you find yourself with a specific problem in your code, knowing what those patterns are and how they can be implemented in your programming language can be really powerful.
34:24 Michael Kennedy: Yeah, absolutely. You know, another thing that I like to lean onto some kind of framework for is authentication. Especially things like oAuth. Those are super, super complicated to figure out yourself, but luckily, on our list here, we have some oAuth stuff, right?
34:40 Matt Makai: Yeah, so the project we picked here is authomatic, and the tagline here is, simple but powerful framework authentication and authorization client. I think there's a spectrum with authentication libraries. On one hand, you've got sort of these maybe heavy handed, they tell you exactly how to implement authentication models, so, for example, oAuth, let's say you want to have Twitter-based login or a GitHub-based login. Some of the authentication authorization frameworks will tell you, hey, in these three steps, add this to your code, and then you'll have an oAuth login. But on the other hand, these frameworks have to work with, or these libraries have to work with various frameworks, so the way that you would implement something in Django versus Flask versus Pyramid versus TurboGears can be different, and that's where I think taking a look at all the different libraries can be helpful. So authomatic seemed like a really great one because it had examples with different frameworks, which I really liked.
35:41 Michael Kennedy: Yeah, you can definitely go in there and it's like, these are the few things you drop in for Django, here's the few things you do for Flask, here's what you do for Pyramid, even one for Google App Engine, and I really found that some of the other ones that seemed comprehensive had almost no examples, and you're like, but there's all these moving parts! How do I put this together in this particular web framework, like, why is there no example here? Yeah, I looked at a number of them and maybe we should put a little word of caution in, that this hasn't been updated for a little while, but that's kind of true with some of the other options out there as well, so they're kind of, seem more steady state, or they're in some sort of flux, or something. So look through the whole list. Here's just one example to sort of point out, like, there's a bunch of great things about oauth and social auth and things like that.
36:23 Matt Makai: Yep, and this makes it much easier for you to implement them rather than having to rewrite everything in your own code.
36:29 Michael Kennedy: Yeah, for sure. This portion of Talk Python to Me is brought to you by Hired. Hired is the platform for top Python developer jobs. Create your profile and instantly get access to 3,500 companies who will work to compete with you. Take it from one of Hired's users who recently got a job and said, "I had my first offer on Thursday. After going live on Monday, I ended up getting eight offers in total. I've worked with recruiters in the past, but they've always been pretty hit-and-miss. I tried LinkedIn but I found Hired to be the best." "I really like knowing the salary up front. Privacy was also a huge seller for me." Sounds awesome, doesn't it? Well, wait until you hear about the signing bonus. Everyone who accepts a job from Hired gets $1,000 signing bonus, and as Talk Python listeners, it gets way sweeter. Use the link hired.com/talkpythontome and Hired will double the signing bonus to $2,000. Opportunity is knocking. Visit hired.com/talkpythontome and answer the door. You talked about static sites earlier and we mentioned the deployment and the performance story and all that, and that's great, but sometimes, you just need a data-driven web app, right. Like for example, it's super hard to go to, like, a static site and add voting, or comments, or things like this, right? And so maybe the thing that you're building, you really do need to have some kind of data driven thing here, but if it's not fast enough, what are you going to do?
38:05 Matt Makai: Well, you've got to have caching, and so caching is one of these really hard problems in computer science, so taking advantage of a library that allows you to implement caching can be really helpful. And I know this one you picked, Beaker, why don't you tell us a little bit about Beaker?
38:20 Michael Kennedy: Yeah, yeah, Beaker looks really awesome. So Beaker is a way in which you can cache binary representations, pickled versions of Python object graphs. So you just take the thing you want to save and you just stash it in Beaker. And it can go into all different places, right. It can go into databases. It can go into memcached. It can go into just be stuck in memory until your process reboots, it can go straight to the file, and it also has a SQLAlchemy plugin, so you point SQLAlchemy at the database that you care about, right, because SQLAlchemy supports all sorts of different databases, and then it just talks to SQLAlchemy, so pretty much any type of database you want to use as a backend for your cache. If you don't want to put it in memory, you can do it here. So there's all sorts of cool stuff, like you can take it and plug it in as middleware in WSGI servers. So it automatically has little plugins for Pylons, Pyramids, TurboGears, things like this. Yeah, I think this could be really helpful for people who need that one or two slow pages to go much faster.
39:26 Matt Makai: Yeah, and if you can't build your site as a static site, certainly you're probably going to investigate caching at some point in order to improve the performance. So Beaker seems like a really capable one to use.
39:36 Michael Kennedy: Yeah, yeah, it definitely does. So some of the features they list are, like, it's fast, obviously, that's a requirement for a cache, right, but it has multiple readers, reader-writer lock performance. So it avoids duplicate creation but allows for high parallelism for reading. All the backends, like I said, you can extend it and write your own backend if you want to plug into TinyDB or, you know, whatever. All that kind of stuff is really cool.
40:02 Matt Makai: Yeah, sounds like a good one to check out. So another one that you picked, which I think is really smart, because people always say, hey, don't roll your own crypto, and you don't want to write your own password storage hashing library, unless you're just trying to learn how it's done, but you don't necessarily want to write something for production, but you picked passlib, which you've used in the past. Why don't you tell us about passlib?
40:25 Michael Kennedy: Yeah, I use passlib all over the place. I use it on the training website, for example, for accounts there. I don't want people to end up on, you know, some sort of news story. "Oh, a development server with production data was compromised and here's, you know, a quarter million accounts, and too bad they used poor, like, an old, outdated hashing algorithm or something like that, and it turns out that everybody's going to have to reset all their passwords on the internet everywhere." So passlib, I hope, will help people avoid that fate. The idea with passlib is, it takes all the best practices around password storage and verification and validation and puts them into two functions: encrypt and verify.
41:10 Matt Makai: That's easy.
41:12 Michael Kennedy: If you can't remember to call encrypt and verify, you know, you probably have no business writing authentication code, right? But the idea is they provide a whole bunch of different caching algorithms, or sorry, hashing algorithms, and they only support ones that have no known vulnerabilities, are public, have public reference implementation, and can be made to be computationally expensive, you know, have a lot of scrutiny, and things like that, so argon2, bcrypt, SHA 512 crypt, things like that. So you're already starting from, thank goodness they didn't use MD5 for the hashing, right? Which is one of the problems. The other problem is, with the advent of graphics, the GPU processing, people can use the GPUs to do, advent of computational GPU processing, not for graphics, they can use that, the power of those graphics cards, to just try so many variations on these hashes because the graphics cards can do them really quickly. And so you want to make the stored passwords computationally difficult to guess at, so instead of just hashing with a strong algorithm, passlib will take it and it'll hash it, like, 200,000 times, so it'll take the input, stuff it back around, hash that thing, and then fold it over 200,000 times, and it also adds random salt to each thing it stores, and it of course folds that all through. So it's just super hard to guess what the possibility is. But it's easy for you, you get back some kind of password hash, you call verify with the plain text password and either it matches or it doesn't.
42:47 Matt Makai: Yeah, well, it seems like it provides a really good trade-off between, developers want to spend the least amount of time working on security, so they have more time to work on features, and it seems like this is making a good trade-off where it's, hey, this is really simple, here's what you do, call these two functions and we'll take care of this bit for you, so that seems like, as far as a library, that's probably some of the best you can ask for.
43:09 Michael Kennedy: Yeah, it's really cool, and you can actually evolve over time. So for example, part of the thing that gets stored in the database has the number of rounds, or number of folding times, iterations you've done, and so as computers get faster, you can actually crank that number up over time without breaking the old passwords, and then when the user comes back, you just regenerate with a newer, more computationally expensive variant.
43:33 Matt Makai: Nice.
43:34 Michael Kennedy: Yep, also supports two-factor authentication, but I know nothing about this.
43:39 Matt Makai: You could use Authy for that or implement with SMS via Twilio. That's my little Twilio commercial for this episode.
43:46 Michael Kennedy: Oh, awesome, very cool. Okay, so another thing that's cool, well, something that's just awesome about Python, the whole ecosystem, is really pip and the deployment packaging story, right, that I can just go grab all these awesome packages, and you know, 95,000 others.
44:03 Matt Makai: Yeah, so the fact that we have PyPI as the central repository for all of our dependencies and it's really easy to upload your own libraries to this central repository, as a Python developer, it makes things so much easier for me. When I was a Java developer and I used to use Maven, Maven was incredibly complicated. They have these massive XML files. It worked, but it was just, it was really complicated, and then I came over to Python, and you've got a requirements.txt file. And requirements.txt, when I read it, I thought, this is so beautiful, I literally thought it was beautiful. There, you have the name of the library that you depend upon and then you optionally have the version number that you want to use for that library. Now, I say optionally, but really, it should be required. Before I joined Twilio, I did a lot of Python consulting, and it was my nightmare when I would get a project, and either would not have a requirements.txt file, you know, this is something that had been running in production for a few years and they said, yeah, we threw all this stuff together, you know, we used Django, and I thought, okay, Django, that's great, which version? And they're like we don't know. And so, so we've got, we luckily have now some tools that help out with figuring out what those dependencies are, so pip-tools is a set of tools not only to keep your pinned dependencies fresh but also to kind of infer what some of the requirements are that you may have. What's great about this is, if you don't necessarily know what the exact version numbers are of all of your dependencies, you can use some of the pip-tools in order to infer some of those things. So for example, like Django, which version of Django, is it 1.4, is it 1.10, this will help you to figure that out.
45:53 Michael Kennedy: Yeah, absolutely. And pip-tools will actually generate the full dependency graph for you. So one thing you might say in your requirements file is, like, I require Flask. Maybe you even say, I require Flask 0.10.1, or whatever it happens to be, right, but Flask itself requires itsdangerous and Jinja2 and MarkupSafe and these types of things, so you can go to pip-tools and you can say, run pip-compile, and it will actually generate a requirements.txt that has the sort of closure of all of your dependencies. The dependencies, and their dependencies, and so on, so you can see and pin every single dependency you have. And it also adds a little comment, like, these dependencies were added because of Flask, this one was added because of Jinja which was added because of Flask, things like that.
46:41 Matt Makai: Yeah, definitely and just incredibly handy. If you get dropped into a project and you have no idea what the exact version numbers and requirements and dependencies of dependencies are.
46:53 Michael Kennedy: Yeah, absolutely, and the thing that I use, actually, in production for this kind of stuff, it's not the same, but I use pyup.io, I met those guys through the show a while ago and I'm like, this is such a cool project. Pyup.io, and you basically use, it's like an online system, you point it at your GitHub repo, public or private, and it will watch this file, and if there's changes or releases or especially security releases, it will send you a pull request, saying there's a security vulnerability and it's dangerous, or something, here's the updated requirements.txt to fix it in your app.
47:25 Matt Makai: Wow, that is incredible. That's super useful.
47:28 Michael Kennedy: Yeah, it's the notification. I mean, these things are changing all the time, but it's like, the push notification to you that your app now has, like, an important update to its dependency, which I thought was really cool. Awesome, okay, well, those were a great bunch of libraries. Hopefully you guys learned a lot about them. Matt, it was fun to explore these with you. I didn't know about all of them before we started talking.
47:50 Matt Makai: Well, I don't think anybody knows all of the Python projects that are out there. You wouldn't get any coding done if you knew what all the projects were. So that's what I think is great about the Awesome Lists, is you get to see at a glance just different libraries that you may never have known about.
48:07 Michael Kennedy: Yeah, I totally agree, and that's why I think it's so interesting to bring it up for people. Because you can drop in there and really quickly learn about all these tools that you didn't know about a few minutes ago, very cool. All right, so as always, before I let you get out of here, let me ask you the two questions.
48:22 Matt Makai: Okay.
48:23 Michael Kennedy: So I don't even actually remember your answer from a year and a half ago. So when you write some Python code, or any code, really, what editor do you use?
48:30 Matt Makai: So I use Vim. That hasn't changed. It's been Vim for a fairly long amount of time. Now, to be fair, I use Vim for Python code, most things I do. I use Xcode for my Swift Code, and I use IntelliJ, the Ultimate IntelliJ Edition for Java code, so I've been doing some more Java coding lately, so I kind of bounce back and forth, but I almost always use the Vim mode in whatever editor that I'm in.
48:57 Michael Kennedy: Yeah, that's probably a requirement for you, right. That it has a Vim mode.
48:59 Matt Makai: Yes.
49:01 Michael Kennedy: Nice, all right, and we kind of touched on a bunch here, but you want to pull out a favorite PyPI package?
49:07 Matt Makai: Oh man, oh, I've got to give a shoutout to Pelican. I just, as a static site generator, I think when you use a tool, and I may have said this on episode 26, but when you use a tool for six, seven years, and you still love it, I think that's a really good, really good sign. So Pelican is that tool for me, and it's a great static site generator that powers Full Stack Python.
49:32 Michael Kennedy: Yeah, that's awesome. I think that is a really good sign. If you love it after you've lived with it for a while, then it must be something special. Very cool. So I feel kind of like as a final call to action, I noticed there were some projects not listed there that I thought were awesome. You look in various, some of the categories, you're like, oh, why is this not listed here? So maybe if you're out there and you own a package or you're really passionate about it and you don't see it there, you think it's awesome, maybe go send those guys a message and let them know they should add your package. All right, well, Matt, it's been fun to talk to you. Thanks for sharing these packages with everyone.
50:09 Matt Makai: Yeah, thanks, Michael. It's been a great time.