#97: Flask, Django style with Flask-Diamond Transcript

00:00 Michael Kennedy: There's a whole spectrum of Python web frameworks, on one end we have the micro-frameworks, like Bottle, Flask and, to a lesser degree, Pyramid. On the other, things like Django or even CMSes like Wagtail, which are built on top of Django, way at the far end of the spectrum. While this is often positioned as an either or choice between these frameworks, this week you'll meet someone who's doing a lot to build some middle ground in the spectrum. Ian Dennis Miller, creator of Flask-Diamond, an extension of Flask which brings the good things from Django to Flask's simple and small API. This is Talk Python To Me, episode 97. Recorded January 16th, 2017. Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the ecosystem and the personalities. This is your host Michael Kennedy. Follow me on Twitter where I'm @mkennedy. Keep up with the show and listen to past episodes at Talkpython.fm and follow the show on Twitter via @talkpython. This episode is brought to you by Rollbar and Hired. Thank them both for supporting the show. Check them out @rollbar and @hired_hq on Twitter and tell them thank you. Ian, welcome to the show.

01:39 Ian Dennis Miller: Thanks for inviting me.

01:40 Michael Kennedy: Yeah, I'm really excited. I often feel like we have to make this trade-off between micro-frameworks when we're choosing a web framework, like Flask, to some degree Pyramid, Bottle, things like that, or Django, which has all these cool building blocks. And I think you're kinda here to dispel the fact that that's the choice to some degree, right?

01:57 Ian Dennis Miller: I hope so. I think there's room for something that exists somewhere in between. Or maybe they can look like one thing sometimes and look like the other thing other times.

02:08 Michael Kennedy: Yeah, absolutely. So we're gonna talk about your project Flask-Diamond, which brings some Django like features to Flask. But before we get into those, let's get your story. Where'd you get into programming?

02:20 Ian Dennis Miller: Yeah, many moons ago in, let's see, I guess sixth, seventh grade, something like that. I made some kind of transition from using the computer basically as a gaming device and I started experimenting with this weird program I found lurking in the DOS proms called GW-BASIC, which was a basic interpreter that just shipped with DOS and I actually started off by entering random words into basic. Because I didn't really have a book for it and the internet didn't exist at the time, at least not the way that it exists now.

03:04 Michael Kennedy: You didn't want to like Gopher for--

03:06 Ian Dennis Miller: Yeah, right. Now, Gopher, so I mean, you know, as a sixth grader, all that stuff was way beyond my reach. But I guess kids just, our time is worth so little. We sit there and we can just pound away at a keyboard and eventually randomly things started working and it took years actually of doing this. But I learned how to program in Basic and then as I grew up, I was able to take courses at school so that actually accelerated the process dramatically.

03:38 Michael Kennedy: That's really cool. It's amazing that you actually were persistent to do that. That's really quite incredible.

03:44 Ian Dennis Miller: Uh yeah.

03:45 Michael Kennedy: You must have thought like when you got to a programming class, you're like, you mean they give you all the stuff you can type, you just put it together, this is like, nobody's, I can do this, this is easy. Yeah, that's great. Yeah, so you, you studied in university? Or where, when you went to school, what level was that that you started taking programming courses?

04:40 Ian Dennis Miller: I'm talking here about later in high school. The AP or the Advanced Placement track at the time, was in a language called Pascal, which by now has kind of you know, fallen away into history. But in the course of taking this AP test, I learned this dead language and the following year they switched it over to C++ so I retook the course again, still in high school and learned C++ in the process and then yeah, in university I actually went a kind of different direction and studied cognitive science. But it still has a foundation in a knowledge of certain computer science principles, but that's a pretty different beast.

05:27 Michael Kennedy: It is, it's pretty different. Although, my experience with cognitive scientists is a lot of them have some reasonable programming skills to do their modeling and whatnot with.

05:36 Ian Dennis Miller: Yeah, that's been my experience too.

05:38 Michael Kennedy: Yeah, cool, so Basic, Pascal, C++, how'd you get to Python?

05:43 Ian Dennis Miller: I took a very long pit stop in the Perl universe. After university, I got pretty involved with building web applications and CGI. This was, you know, the early 2000s, CGI was the way to do it and Apache had a strong connection to the PHP community and to the Perl community and I dug really deep with Perl for years, but found that, this was Perl 5.8 thereabouts and it really didn't have a good object model. It didn't have inheritance and a lot of the things that I came to like from statically typed languages such as C and Java. And I found that I wasn't able to scale up Perl web applications as large as I wanted to. But eventually the chaos just took over and the bit rot and I watched a number of software projects die in my hands. I loved them so much, but I just couldn't speak to them anymore because my own Perl was just so backwards and anyway, I went looking for something that would be an interpreted language, but also provide strong concepts about object orientedness and inheritance and Python was perfect and it also had the momentum. The community of the Python, all the extensions and everything, then the decent documentation made it pretty easy to get started, so I was really attracted because I just needed to make bigger applications and Python just perfectly filled that space.

07:26 Michael Kennedy: Right. Just better structure, better packaging, better ability to separate concerns, all sorts of stuff, right?

07:32 Ian Dennis Miller: Exactly, yeah. And I mean, you could do it in Perl. There was just an awful lot of ritual, literally ritual involved in that stuff. And they just thought through it the right way in Python. I love it.

07:46 Michael Kennedy: Yeah, I think that's really important because a lot of people when they're comparing languages they're like well, I can do this in this language or I can do that in that language or whatever. And it's not always about whether it's possible to be done, it's also how much is that part of the idioms of the language? How hard, do you have to fight the thing? Or do you, there's a saying that I like, falling into the pit of success, right? So if you design the language and the APIs right, if you just kind of follow the breadcrumbs, you end up in the right place, versus the wrong place and I think there's a lot of that in the Python, the falling into the pit of success story. So that's really nice.

08:22 Ian Dennis Miller: Yeah, well and you know, it's kinda like we, you have infinite power with Perl and it's the kind of thing where you need to stop and ask yourself if you should, you know? Like, in the first place, do you embrace that and the people who do become actual wizards and power maybe, well, I don't know. Perl is really powerful, let's just say that.

08:46 Michael Kennedy: Yeah, absolutely.

08:47 Ian Dennis Miller: And I've actually, yes, just following good Python practices like PEP 8, for example, or the Google style guides are great ways to accidentally write better code. I mean, not accidentally, but just due to habit create code that is so much more valuable.

09:07 Michael Kennedy: Yeah, yeah, totally agree. Totally agree. So how'd you get interested in the combination of Django and Flask? Like, you were obviously doing CGI and Apache stuff with Perl before, so you're already in the web space.

09:20 Ian Dennis Miller: Definitely and I learned Django first because the documentation is so fantastic. They have covered the novice, like the tutorials for beginners. They have covered all the depth that you would ever need in terms of API documentation. It was just such a great example of how to build a really vibrant community that is able to get up to speed and then live with it for a long time. So I learned Django, I really like it, I've thought an awful lot about the Model-View-Controller pattern because that generalizes across a whole bunch of different frameworks. But what I came to, the friction point that I found was the typical object relational mapper. The Django way of doing it is part of the core library and I was really attracted to SQLAlchemy, just as a good single concern library that just solved the problems with SQL for all time. So I didn't want to sacrifice the clarity of using ORM like SQLAlchemy, just so that I could have a better web application experience. I just didn't want my object relational mapper to be so tightly tied to my web stack and so Flask really provided a model that separated those concerns a lot more cleanly. And when I say Flask, I'm also just kind of lumping in the Flask SQLAlchemy extension, which bridges it in a very natural way. So for me, the ability to deal with the database alone, just in terms of SQLAlchemy and set aside the web concerns for a time, meant that it would be a powerful enough way of expressing some really strong statements about the data. I have a lot to say about the data. But as I got deeper into scientific computing, that's where my concerns really lay. I couldn't imagine using Django to perform that kind of work.

11:34 Michael Kennedy: I definitely don't want to give the impression you can't do these things with Django.

11:38 Ian Dennis Miller: Of course.

11:38 Michael Kennedy: Because I know that you, it's very, you very much can customize it and set it up just the way you want. But I think it also goes a little bit back to that pit of success right. It doesn't, that's not the first thing that it encourages. It doesn't say bring your own ORM and use it however you want, bring your own ECP client and do whatever you want with it, right? Like it's more about these are the building blocks and you build with them. Whereas with Flask or Pyramid it's like, you go through and you pick the type of thing you want to work with, the pieces you want to use across the whole, you know, 96,000 packages and then you put them together however you feel like it.

12:14 Ian Dennis Miller: Exactly.

12:15 Michael Kennedy: That doesn't work for everyone. But it sounds like it works for you and that's also the way that I like to work. So I think it's an interesting option.

12:21 Ian Dennis Miller: Well, you know, it did and it didn't work for me. I actually started running into trouble with taking Flask and then having the freedom, having the power, again, sort of thinking about Perl and wizardry, the ability the glue together absolutely anything that you can imagine, can lead you into chaotic projects. Places where you do things in an idiosyncratic way and suddenly you can't really collaborate about your project any more. Because you've done things, because Flask let you. I mean, I'm not blaming Flask, but it just requires such discipline to use it in a way that's predicable enough that any teammate could actually meet you half way. Read your source code, extend it. I'm being a little bit hyperbolic here, because I know that there are people out there that are being very successful using Flask in a team context. But I found that I was getting burned by some of the power and I wanted to regularize some of the ways that I was using Flask and make some of those things repeatable. If only because I didn't want to have to start from scratch every single time. As freeing as that is, I just didn't want to do that anymore.

13:37 Michael Kennedy: Yeah, I do find a lot of times, if you've been working in one company or one environment for a long time, you do feel like you're doing the same thing over and over and over again. One company I worked at, I went so far as to build like a new app generator. And it would spit out a new app with logging already set up, we'd have like exception handling services, kind of things like Rollbar before they existed. All that stuff was integrated and you pressed the button and then you added, you know, where you went from there.

14:08 Ian Dennis Miller: Exactly.

14:09 Michael Kennedy: That is really handy, as long as you can tweak and do whatever you want, I suppose.

14:13 Ian Dennis Miller: Well so yeah, that's a whole other can of worms there. Because I kind of think that scaffolds, so there's a whole ecosystem now of Flask scaffolds. Because the idea of an app generator is actually really attractive. A lot of people have gained efficiencies by automating some of that stuff. And I went down that path. I started using a Python library called Mr. Bob that just takes a template, a Jinja type template and spits out a file structure, like a directory hierarchy with auto-generated code and what I wanted there then was the ability to keep the app generator alive. Such that I could then fold back things that I learned in the process of the previous app into the scaffold so that the scaffold evolved along with my other projects. And so actually that's a large part of Flask-Diamond, is that it is an app generator. It will scaffold an application maybe kind of like what you were just describing, you know. It has logging already embedded, well anyway.

15:22 Michael Kennedy: Yeah, so tell us about your project. I mean, I think we've sort of found our way to your project in this whole conversation. So the thing that you created is called Flask-Diamond. What is that?

15:32 Ian Dennis Miller: So Flask-Diamond is, I see it is consisting of a couple of important components. So at a high level, Flask-Diamond is a batteries included Python Flask framework. So the batteries included part means it will do something without you really doing anything at all. So if you pip install Flask-Diamond and then you scaffold an app, the app that comes out already works. And everything's in it, so it does have logging enabled and it does have all these features. And some of these things I might describe a little bit more later, but the batteries included means the Flask extensions are already there and they've already been hooked up in a way that basically makes sense for a typical case. This is the kind of thing that's impossible to anticipate everybody's individual product considerations. So you want to get kind of a happy medium between providing something that actually does something kind of neat, without any configuration and that doesn't ideally have too much bloat in terms of getting in the way of you configuring it for however your project is going to deviate from whatever the defaults are, whatever the opinions are.

16:50 Michael Kennedy: Right, the more opinionated you make it and the more features you add to it, kind of ironically almost, I suspect, means the fewer people who are interested and enjoy it, right?

16:59 Ian Dennis Miller: Yes, I think so.

17:01 Michael Kennedy: Because if you're like oh, it needs logging, most people are like yeah, logging is good. Oh, it also needs like caching, it also needs message queues and all the sudden it's like whoa, I don't need all this stuff, I'll just... You know, you start to push people out, right? As you like overdo it, so you know, how did you find that balance?

17:15 Ian Dennis Miller: I don't know that I have just yet. So I'll issue that disclaimer. But I will say that for my own uses I do think that it meets a certain balance by being, I'm calling it radically decomposable. So what I mean by that is, it is actually safe to strip away pretty much everything that comes sort of bundled by default with Flask-Diamond and actually create a command line app that doesn't even use Flask to spin up a web sever and a database model. I've actually used the whole Flask-Diamond setup to create really simple little scripts, but I've also used that basic pattern of a command line driver with a full fledged SQLAlchemy database lurking behind it. And again, I'll use Flask-Diamond to do that, because it merely simplifies the interfaces and the process and the extensions and importing the database models and setting up the namespaces in a predictable way. It does all of that for free, such that I actually don't care if Flask happens to be a WSGI environment, you know, an HTTP server. I'm actually using it as just this really general purpose application framework. And I even consider the web part to be decomposable and extractable from the whole rest of the application platform. So what I'm saying is I don't even always write web apps with Flask any more. There's a way of using the Flask application context to script an app that does not actually need to be running in conjunction with a WSGI server. It's an incredibly powerful pattern and it's another thing that I would never do with a thing like Django that is, it's very heavy and when you start up a Django process, it's not that easy to slice out different pieces of the core. But with Flask, pretty much nothing is included in the core to begin with.

19:24 Michael Kennedy: Yeah, even Flask itself is made up of a number of building blocks.

19:29 Ian Dennis Miller: Exactly.

19:30 Michael Kennedy: That are put together, like itsdangerous and werkzeug and those sorts of things, right?

19:33 Ian Dennis Miller: That's right. You know, you mention itsdangerous and one of the things that is kind of nice about folding in something like itsdangerous using something like Flask-Diamond is that by default it's just folded in in a way that is going to work. So the pathway there is through Flask-Security and Flask-Security will sanitize all your inputs with itsdangerous and perform all of the cross site scripting protections and so you know, it's there, you know it's there and that your input will be sanitized. And frankly, if you don't want it, you don't have to have it. You can also get rid of all of that.

20:30 Michael Kennedy: This portion of Talk Python to Me has been brought to you by Rollbar. One of the frustrating things about being a developer is dealing with errors. Relying on users to report errors, digging through log files trying to debug issues, or a million alerts just flooding your inbox and ruining your day. With Rollbar's full stack error monitoring, you'll get the context, insights and control that you need to find and fix bugs faster. It's easy to install. You can start tracking production errors and deployments in eight minutes or even less. Rollbar works with all the major languages and frameworks, including the Python ones, such as Django, Flask, pyramid, as well as Ruby, JavaScript, Node, iOS and Android. You can integrate Rollbar into your existing workflow, send error alerts to Slack or HipChat or even automatically create issues in JIRA, Pivotal Tracker and a whole bunch more. Rollbar has put together a special offer for Talk Python To Me listeners. Visit rollbar.com/talkpythontome, sign up and get the Bootstrap Plan free for 90 days. That's 300,000 errors tracked all for free. But hey, just between you and me, I really hope you don't encounter that many errors. Loved by developers at awesome companies like Heroku, Twilio, Kayak, Instacart, Zendesk, Twitch and more. Give Rollbar a try today. Go to rollbar.com/talkpythontome. Before we get to the nitty gritty details of it, tell me why and when did you create this. How old is this project? What got you to say alright, I'm making this thing.

22:00 Ian Dennis Miller: Totally, so this has been going on for years by this point and it's the kind of thing that didn't happen intentionally. No one ever sits down and says I'm gonna write a framework today, but I needed it. I found that I just needed something to fill a missing spot in my research pipeline basically. So I was doing a lot of work with online memes and their propagation through social networks and so I was building a couple of web apps and I guess this is 2011, 2012. I was using a framework called WebPIE, which Reddit was using and at the time, the principal author was a guy named Aaron Swartz, who died in early 2013. And the WebPIE project died with him, as far as I can tell. So I needed something.

22:56 Michael Kennedy: Yeah, that whole story is super tragic, yeah. It's unfortunate, yeah, it's very sad.

23:01 Ian Dennis Miller: Yeah, it's interesting to consider some of just the rippling costs, the things that, a huge tragedy actually. And so when you're trying to do some kind of academic work that's going to be reproducible, be repeatable, the code needs to be sharable, it needs to be portable, you need to be able to give it to other people and they need to be able to run it and it also needs to be able to run ideally five, 10, 20 years in the future. And so a web framework that's not going to be evolving beyond 2012 or so is no good. So I experimented with Bottle and I liked it and just, I discovered Flask. Kind of I think because of the play on words between Bottle and Flask and in summer 2013, I started really making some of these ideas into a repeatable template. So I was building, at this time, an application to model Twitter. I was scraping a ton of Twitter data and so I had all these different types of objects. There were accounts and there were Tweets and there were links between them and all the data model that you need to do to scrape Twitter. What I really needed was a way for other peers, collaborators, to be able to look at this data without needing to understand SQL themselves, without needing to understand Python. Ideally, without even me needing to be there. And it was through Flask that I discovered this whole ecosystem then of things like Flask-Admin, and Flask-Security and the whole rest of it that I started assembling into this kind of research data support infrastructure. So all my data then I could put on a server somewhere in a cloud and just give a login to an advisor and they would be able to look at it. So the need was really, I just had so much data and I wanted to talk about it. I wanted to collaborate with it. And like I said earlier, I didn't want to be tied to the Django ORM. I just wanted to use SQLAlchemy.

25:17 Michael Kennedy: Cool, I think that's a really forward looking way of doing it. Here I'm doing this research, I'm gonna put it on the web, you can just execute it, slice it, dice it, whatever you want. This predates Docker, but it seems to me like Docker's being used by a to of scientists for that purpose as well. Like, I'm gonna package up the execution of my analysis into a Docker image and then you could go do what you want, so it's like the next level or something.

25:42 Ian Dennis Miller: Yes, absolutely. A kind of anecdotal story is that back in the day I was aware of a guy in this one department, who needed, for timing purposes he needed these 286s. Because that was the only thing that would execute his code at the right timing for a certain kind of reaction time task that we was recording. So he actually stockpiled these 286s, and I mean, by the way, this was happening in like 2008. So someone with a big pile of 286s, all for the purpose of getting the software to run exactly right decades after it was originally written. It's a serious problem.

26:20 Michael Kennedy: It really is. You take the 286, 16 megahertz and don't press the turbo button or it won't work right.

26:25 Ian Dennis Miller: Yeah, exactly. That'll screw up all your data. Yeah, but no, I mean it's a real problem and reproducibility is a big, it's a hot topic in science right now. Especially in the field of social psychology which is kind of my home base right now. These issues of sharing data between labs and getting the same results even when different researchers do it, it's actually getting a lot of cool papers written right now. So anyway, yeah, it's really important for data to be kind of social.

26:55 Michael Kennedy: Sure, I don't think that we as an industry or users or programmers across the board we're not used to thinking of execution as a thing that has to be kept for long terms of period. We think about how are you gonna archive data, you know, how are you gonna make that interoperable so you can read it in the future, but not like how am I gonna be able to run this algorithm that nobody's gotta rewrite, or you know, or something like this. This is an interesting problem.

27:23 Ian Dennis Miller: It's not something that I, that affected my day to day live until relatively recently, but it gives me new respect for mainframe type stuff. Like the whole IBM System 360 stack that goes back 50 years. And what they've done to virtualize hardware and enable code, ancient code to still execute today. I mean, I think about just the horrors that the banks must be facing on a regular basis. So yeah, you know, let's be sensitive to those forward looking issues and, I don't know, maybe mitigate that a little bit, we'll need the solution, but anyway.

28:03 Michael Kennedy: Yeah, it's a small deposit.

28:05 Ian Dennis Miller: Exactly.

28:06 Michael Kennedy: I think you're probably right that like emulation and virtualization and you know, throw containers in there, those will like alleviate that to some degree, but still, it's gonna be tough.

28:14 Ian Dennis Miller: I think so.

28:15 Michael Kennedy: There's a bunch of cool features of Flask-Diamond, let's dig into those for a bit.

28:18 Ian Dennis Miller: So what I found was, you know, when we were talking about not wanting to repeat the same steps every single time you start a new application, I found there were a couple of features that pretty much every application I needed to build, it required those features. And I've already talked about some of these things that I just needed in the process of just normal, like I say, social data. Collaborative access to my data. And so this account management is a pretty obvious feature. It is provided by Flask-Admin in association with a number of other really helpful extensions. So I've already mentioned Flask-Security and one of the things that I actually liked a lot about Django is the way that you get this GUI management interface kinda for free. Whatever your object model is, Django makes it so easy to get in there, see it in your web browser, create test data, and so that's something that Flask-Admin provides pretty easily and so I really wanted to bake that in. So administrative access just works. And then databases are especially important for me. I've used a whole bunch of different SQL back ends, but I've also used MongoDB very successfully with Flask. This is another instance where just because Flask-Diamond has a little opinion about databases and the basic entity relationship pattern for describing object models, despite that, it's really not very hard to switch that out for Mongo. And when I say not very hard, I mean five lines of code. There's no extremely deep epistomological commitment to using a SQL database just because the default is a SQL database.

30:12 Michael Kennedy: That's cool. And so you have, is SQLAlchemy part of that story?

30:15 Ian Dennis Miller: In a couple of different ways, actually. So SQLAlchemy is not the only object relational mapper, but it has documentation that is spiritual. I think Alchemy is a great term for what SQLAlchemy does. They go to crazy places in terms of what they are doing to layer the SQLAlchemy core on top of these relatively stupid databases. And I'm just so impressed with what SQLAlchemy has done.

30:47 Michael Kennedy: Yeah, I really like it too. I use it a lot.

30:50 Ian Dennis Miller: Yeah. But you know, there's other ones like Peewee that are really popular and a lot of people like it. In terms of being opinionated, you know, in terms of offering these little opinions for optimizations, the reason that I'm going with SQLAlchemy is because sometimes, sometimes there's just a really crazy thing that you just wanna do. And they've already thought about it. I haven't found something that I can't do yet with SQLAlchemy and if there is something that maybe you can't express with SQLAlchemy, you can always just drop to a database shell and do a direct line postgres, or whatever. So that's another thing that I never wanted to get too far away from was the ability to just physically interact with the database and not lose some connection back to the original web application. And like I say, again, the alchemy is magic.

31:46 Michael Kennedy: Nice.

31:47 Ian Dennis Miller: What they have done makes it possible.

31:49 Michael Kennedy: Yeah, it is really great. I think, it never ceases to amaze me that you really can keep on doing that. I can also model that in SQLAlchemy, right?

31:56 Ian Dennis Miller: Basically I think it's good enough for real research.

31:59 Michael Kennedy: Yeah, I totally agree.

32:00 Ian Dennis Miller: I guess that's the story.

32:02 Michael Kennedy: Yeah, and there's a lot of interesting places using it in production. I don't to say the wrong one off the top of my head, but some really major apps that are very cool, major companies using it. You also talked about pointing Flask-Diamond at existing databases and having, presumably, the SQLAlchemy classes, the models generated from existing databases. Tell us about that, because that sounds like saving a ton of work.

32:28 Ian Dennis Miller: This is actually just some kind of experimentation that I've done on the side with some of my buddies. And we have done it successfully. SQLAlchemy ships with, it's a Python script that'll scrape your SQL schema and auto generate model code for you. So I don't want to give you the wrong idea that this is already baked into Flask-Diamond. Simply that we have done it in a Flask-Diamond app, it worked pretty much and it did save us time in this one instance. So I mean, again, this is kind of like, sort of spooky, pseudo-magical stuff here. And I'm saying like yes, yes, this is possible. But I will only speak of it, I won't actually show you. But you'll just have to take my word for it, but it's possible.

33:14 Michael Kennedy: Yeah, I've done that before as well. I think there was a third party package, it didn't come directly from SQLAlchemy.

33:20 Ian Dennis Miller: That sounds right, maybe the Contrid.

33:22 Michael Kennedy: Yeah, it might have been Contrid or something that was like hey, this works with SQLAlchemy and it'll like read your schema and it'll actually generate your classes and maybe the classes have stupid names and you need to go edit them a little bit, but still, like, when you're looking at a big, daunting database and you're like, aw, I gotta get started with this? The ability to auto-generate or pre-generate like a starting place is really cool.

33:44 Ian Dennis Miller: Yeah, like I say, it worked most of the way.

33:47 Michael Kennedy: Which saves you most of the work.

33:49 Ian Dennis Miller: That's right.

33:51 Michael Kennedy: That's awesome. You said it also has, one of the features is Model CRUD. Create, read, update, delete?

33:57 Ian Dennis Miller: Yeah, that's right, create, read, update, delete is a really useful pattern that I've encountered all over the place and all the more so with RESTful http Interfaces where the basic http verbs more or less map onto create, read, update and delete and so I just found that I was doing this constantly. Flask-Admin and also Django provide a CRUD like experience through the GUI interface. And so I wanted to persist that through just the general object model and so this is actually implemented as a mixin that you can include with any SQLAlchemy object that you create. And it simply extends your model with create, read, update and delete for free and it extends it a little bit more with the ability to rapidly load and dump object from JSON or you can extend it further for something like CSV. So the idea is to take some of the nice stuff from the GUI CRUD and enable that pythonically. It's just a convenience thing, but I use it constantly. It's really the only way that I'm interacting with objects anymore is through this CRUD interface. Huge time saver for me at least.

35:11 Michael Kennedy: Yeah, that sounds cool. Another thing a lot of apps need is email.

35:16 Ian Dennis Miller: Definitely. It's a necessary evil. Email is kind of an out of channel or out of band way of communicating things like account changes and actually Flask-Security provides a number of features for email account confirmation if you're going to incorporate some kind of second factor authentication into your platform. So email is a necessary thing, it's provided by Flask-Mail. Again, needs to be integrated and once again, Flask-Diamond gives you an integrated email solution, for free, batteries included. However in this case, this is the kind of thing that not everybody needs. And so I've gone the extra step and I've actually commented out the one line that would enable email by default and so if you find that you actually want it in your app, you are going to have to uncomment that line. But then, you know, email is just there and it's free. And it works.

36:19 Michael Kennedy: Yeah, that's really great. And testing? One of the things I feel like is more possible, things like Flask, the micro-frameworks, is because they're just less, they're easier to test.

36:30 Ian Dennis Miller: Yeah, it's possible to get in there and test some really low level things. Like the generation of signals. Or like I mentioned the application context earlier. The ability to create an application completely separate from any kind of web server or any kind of WSGI serving infrastructure is so powerful for testing. And so Flask testing makes some of these things really easy, like creating a debugging server and creating a phony http test client for local testing of http interfaces. So of course that's stitched in, but also, a simple testing framework is included with the scaffold to provide some examples of how you might use Nose Test, which is one of the Python testing suites. Other examples are pytest or tox. And you know, once again, you can actually use another testing platform part from Nose, relatively easily, so I actually use tox on a couple of projects. But the scaffold includes a pattern for building a test harness using those tests and integrating it with the command line. Making it pretty easy to run single tests, to run all of the tests, to automatically run tests when a file changes, certain things that become really more of like the devops side of things. Once again, a basic workable solution comes for free out of the box with Flask-Diamond. And once again, it's the kind of thing that if you don't like it, you can just delete it and it's gone, because it's really not baked in very deeply. It's just hooked in enough that it'll work, that the mk file has test command and it'll do the right thing unless you don't want it to.

38:26 Michael Kennedy: Right, there's probably like a test.py or something like that hanging around somewhere.

38:31 Ian Dennis Miller: Well, it's a test directory, but anyway.

38:33 Michael Kennedy: Right, OK, sure. You could delete the test directory and you're done with the test if you don't want them or whatever, yeah. This portion of Talk Python To Me is brought to you by Hired. Hired is the platform for top Python developer jobs. Create your profile and instantly get access to 3500 companies who will work to compete with you. Take it from one of Hired's users, who recently got a job and said, "I had my first offer on Thursday, after going live on Monday and I ended up getting eight offers in total. I've worked with recruiters in the past, "but they've always been pretty hit and miss. I tried LinkedIn, but I found Hired to be the best. I really like knowing the salary up front. Privacy was also a huge seller for me." Sounds awesome doesn't it? Well wait until you hear about the singing bonus. Everyone who accepts a job from Hired gets 1,000 dollars signing bonus. And as Talk Python listeners, it gets way sweeter. Use the link hired.com/talkpythontome and Hired will double the signing bonus to 2,000 dollars. Opportunity's knocking, visit Hired.com/talkpythontome and answer the door. You also talked about deployment. Like, how's the deployment different or enhanced with Flask-Diamond.

39:51 Ian Dennis Miller: I came to really like a Python package called Fabric that extracted away some of the weirdness of SSH and rsync and simplified some of the management of remote tasks that occur on your testing server or on your production server. And so I started baking this into the scaffold again. So what comes for free is a little FAB file that, actually I want to take a step back. For deployment, you can do it in a couple of different patterns. There's a way in which you could push your code to a central Git server or something like that and have all of your testing machines automatically pull from that server or have your production machines pull from that server. And so that's kind of a passive deployment. And then there's also an active deployment where you physically push the code to the machine that you're actually going to be running it on. You know, you don't relay it through some other intermediary server. And so I wanted to support both styles, actually, of deployment and so that's supported with Fabric and with a command line interface. Because this is the kind of thing that every command is a one off and you just want to rapidly be able to SSH to a deployment server or push the code directly to a deployment server and these things happen constantly. I just wanted again, a simple solution that came for free, out of the box.

41:20 Michael Kennedy: Yeah, that's cool. So nice that that's built in there. So another thing that when I was creeping around your website that I saw, you were talking about facets. What are facets in this project?

41:31 Ian Dennis Miller: In a round about way, I've kind of been describing what these facets are. A facet is kind of a feature. So you know, a facet is kind of like a face of a surface or as a play on words, a diamond is faceted with many faces. And so each of these features is actually a facet. So there's an account facet, there's an administration facet, there's a databases facet, there's an email facet and so on. And the basic idea is that together, these facets are the default types of considerations, the typical considerations that many of these web applications just require. So far as many of them need accounts and administration, well, those are facets. At the Python level, each facet is really like a proxy for the extensions that are emoted to actually provided that facet. So I mentioned that Flask-Admin and Flask-Security have an interplay where Admin uses Security for login purposes and the facets just ensure that those get, that all the objects are created in the correct order and that Flask-Security knows about Flask-Admin and what this means, again, pythonically, is that you can either inherit and override, or completely comment out any of these facets. And really control whether or not your application has that facet at all.

43:05 Michael Kennedy: Right, or replace it with something different.

43:07 Ian Dennis Miller: Exactly.

43:08 Michael Kennedy: Yeah OK.

43:08 Ian Dennis Miller: So it creates like a predictable hook. Where if you're gonna use a different database, then that's where you do it. You know, you don't just randomly decide when you're sitting down to create the application that, you know, I'm gonna do by database whatever. This is predictable place where all of the database stuff happens, you just go to the database facet and if you're gonna use Mongo, then you do it there. You just inherit and override what I wrote and boom, that's the right place or the opinionated place, certainly not the right place, but that's the agreed upon place.

43:45 Michael Kennedy: Sure, I see. So it's kind of a convention, the facets sort of bundled up like your convention on top of like, Flask-Security or some part of the API.

43:55 Ian Dennis Miller: Exactly, convention is a good word for it. It's creating some conventions.

43:58 Michael Kennedy: OK, conventions are good, that's cool. So let's assume that people out there are basically familiar with how you write Flask apps and how you write Django apps. Can you talk just a little bit about like, what does it look like, what's the API look like, does it look a lot like Flask or look like Django, what do I, what are the feel I get when I write code in this, on top of this.

44:19 Ian Dennis Miller: Totally. I would expect it to be really familiar for Flask users because what comes out is actually a Flask application. The process starts off a lot like Django. You know, I was saying that Django has done such a great job of creating tutorials and making it easy for new users to get started. So just like Django, Flask-Diamond begins with a scaffold and it creates a directory structure that would be really familiar to somebody who uses Django. So there's a manage.py. And it does basically what you would expect. You can do manage.pyserver and then it'll, you know, just run your server. Or manage.pydb and it gives you access to all of your database stuff. So in that sense, it's reminiscent of Django. And incidentally all of that manage.py stuff is implemented with Flask-Script, another really great extension. But after you scaffold your application, it really comes to resemble Flask. The basic pattern uses the Flask create app idiom, where somewhere in your project code, instead of just directly instantiating all of the Flask objects, you wrap that in a function called create app and that really simplifies testing and it does all these things. And that's kind of the Flask way to do it. And all the extensions have an init app style of object creation, so Flask-Admin, it has a slightly opinionated way of setting it up. So once you have your application scaffold, it follows all the Flask conventions. You'll find create app is inside your init file and you'll find that each extension is created using the typical Flask init app idiom. I would expect that everything else should just look pretty much like another Flask scaffold.

46:20 Michael Kennedy: OK.

46:21 Ian Dennis Miller: That you might find online.

46:22 Michael Kennedy: Yeah, cool, so maybe not like a new Flask app, but like a more mature one you've dropped into or something.

46:26 Ian Dennis Miller: Yeah.

46:28 Michael Kennedy: You talked about the Django tutorials and documentation and they are really important, they make a big difference. Actually when the Django tutorials switched from just by default recommending Python 3 over Python 2, that made a significantly measurable difference in the Python 3 usage on PyPI.

46:47 Ian Dennis Miller: Yeah.

46:48 Michael Kennedy: You know, that I think speaks pretty strongly how powerful the Django documentation is. But yours is really nice as well. And you have a tutorial called Planets Tutorial. Which I thought was cool. And it has a video as well as, like a screencast video as well as just the steps, which I really like that.

47:06 Ian Dennis Miller: Yeah, thanks. Like I say, I'm really inspired by Django and what they have done. And one of the things that I've put a lot of time into over the years, because this is several years in the making, is creating some kind of, a compelling story or a tutorial that kinda made sense at some level and so I used to use an example about chess and you know, there's a board and there's pieces and there's players and for some reason it didn't, I don't know, I guess I didn't care at a certain point. Like it wasn't interesting implementing chess. And I had a breakthrough a couple of months ago when I realized that whatever tutorial I write about it really needs to ship with working code. The two need to play off of each other. So over the course of a couple of weeks, these ideas kinda came together and the planets tutorial actually ships as another code scaffold, there's only two that ship with Flask-Diamond. The one will create an application for you and the other will set up this planets tutorial. So it's working code that comes in the package and it maps onto the documentation. So the idea is it should be a consistent story. Part of why that's important is to make it possible. So let's assume that you're doing some team collaboration, some development that involves a couple of people. And let's assume that they know Python and maybe even they know Flask already. The idea is you should be able to point them at something and make it pretty easy for people to become initiated or get up to speed such that they can collaborate. So the planets tutorial is really just, there's planets and there's moons or there's satellites and it walks through creating Earth and Mars and Venus and just adding a couple of moons around those planets. And at that level, it's actually pretty pedestrian. They're some simple models. But I just kinda liked the story a little bit more than I don't know, chess.

49:10 Michael Kennedy: It sounds intriguing, I want to go through a planets tutorial. That sounds cool.

49:15 Ian Dennis Miller: Yeah, well hey, it's an 11 minute video, so.

49:18 Michael Kennedy: Nice. How about, how are you using this? You said you're using this in a few interesting ways in production today, right?

49:25 Ian Dennis Miller: Yeah, so I am actually using it in production. And I gave a few examples of some research projects where I wanted advisors or peers to have access to this stuff but I've done a couple of other projects more recently that are actually live and out there and even potentially getting some decent traffic. So pplapi.com, so that's pplapi.com, is something I've been working on for a couple of years. It's part of my dissertation work. It's a virtual database of the human population. So for everyone who is alive, approximately, in 2014, that was about 7.2 billion people. There's an entry in this database for everybody. Now, to be clear, it's synthetic data that's not actually scraped from Facebook or something like that. I have a bunch of algorithms based on demographics that make guesses about where everybody is and how old they are and all that stuff. And you know, the guesses are pretty good. But the idea is they're not real people just yet. But it's billions, I mean, it's a really big database.

50:36 Michael Kennedy: That is a big database.

50:37 Ian Dennis Miller: Yeah, something like seven terabytes. And I was really happy that this got featured on Hacker News and Product Hunt last week. And the website did not fall over. I was sustaining 150 simultaneous connections and it held up on a virtual machine in the DigitalOcean cloud. So that was actually a pretty cool validation of the ability of one of these applications to sustain moderate traffic and it's also a demonstration of the ability to manage a pretty sizable database using SQLAlchemy and exposing that over the web and coping with some kind of scale. Coping with like I say, a little, like a moderate amount of traffic. So I'm pretty happy about that pplapi.com. And but another thing that I'm doing that kind of relates to me as an open source developer is that I've been looking at GitHub and they have a ton of data available through their API. And there are actually a bunch of academics out there who are scraping GitHub. There's a lot of interesting research you can do with that data, so I have created an analysis called GH Impact. This relates to the number of projects that have a certain number of stars or more. So if you have 10 projects with at least ten stars, then you have a GH Impact score of 10. And if you have 100 projects, each with 100 stars or more, then you have a score of 100. And only companies like Google and Microsoft and Mozilla are able to get these scores that are really high, to have 100 projects and every single one of those projects is well liked on GitHub. And typically, even prolific developers have fewer projects and they get fewer stars. Or if they have one popular project, then they probably don't have a second popular project. So this is kind of a social analysis of the impact that people are able to achieve through platform like GitHub. Once again, this is live on the internet. And once again, I did all the data modeling with Flask-Diamond and in this case, one of the ways that that was really useful was the ability to extract small amounts of data from this 150 gigabyte database through a REST API. And so that enabled me to use a statistical package called R to not have to load the whole 150 gigabyte database. I was able to just sip little tiny portions of data through the REST interface that I built. In that manner, I was able to fit my analysis in RAM. Because R is just not powerful enough to deal with an enormous database of many gigabytes. And so for certain analyses, a tool like Flask-Diamond fills a really crucial data translation role of adapting huge data down to relatively underpowered statistical workstations.

53:50 Michael Kennedy: Yeah, that's cool. So is it kind of like, did you kind of like sample or poll the GitHub population and then compare a name you put in there against that?

53:59 Ian Dennis Miller: No, I got all of it. With like, certain caveats. I scraped as much as these other researchers over in the Netherlands have scraped. Their project is called GHTorrent. The torrent thing I think is irrelevant now. I'm pretty sure it's just a distributed scraping project. As much as they were able to scrape, which is, I think, certainly more than 90% of GitHub.

54:21 Michael Kennedy: Yeah, that should be statistically relevant, yeah.

54:23 Ian Dennis Miller: Yeah, yeah, I got all of it. So no, I didn't sample any of it. I just extracted specific columns and subsetted the data so that I was able to fit only what I cared about into RAM and ignore the rest.

54:36 Michael Kennedy: Nice, so I had to type my name into it, because it's the only one I knew off the top of my head. I have a GitHub Impact score of eight and a 99 percentile.

54:43 Ian Dennis Miller: Yeah, yeah.

54:44 Michael Kennedy: That's cool.

54:45 Ian Dennis Miller: It ceilings out pretty rapidly. And you know, and so you know, your score might be a little bit higher than that, because like I say, maybe they haven't scraped the whole thing just yet. But yeah, what that means is you have at least eight projects with each eight stars or more. In all likelihood, some of those projects have way more than eight stars. But the way that it draws the line is I guess there are not nine projects each with at least nine stars.

55:10 Michael Kennedy: Yeah, that sounds right to me actually. It's pretty cool, it's a cool measurement.

55:14 Ian Dennis Miller: Yeah, thanks. They like it in academia.

55:17 Michael Kennedy: Very nice. So the last point one that you want to talk about in production is GThnk.

55:22 Ian Dennis Miller: This one's a weird one.

55:23 Michael Kennedy: Without the I. GThnk.com.

55:27 Ian Dennis Miller: Yeah, so I do pronounce it g-think, so you got that right. It's not actually hosted remotely. This is one that you download and then you run it locally and on OS 10, GThnk installs as a launch service. So it's just kind of always running in the background. And it's a note taking application of sorts. It's not in the cloud. It's not hosted remotely, which is I believe a security feature. Like I really don't trust a hosted cloud service to store my personal notes. I would much rather just have a program running on my own computer. That's what GThnk is. It's like a self hosted Evernote or something along those lines.

56:12 Michael Kennedy: OK, cool.

56:13 Ian Dennis Miller: It demonstrates that Flask-Diamond can be packaged in a couple of different ways. Not only hosted online.

56:18 Michael Kennedy: That is really cool, to have it as a desktop app. OK, nice, that's quite the interesting twist. So tell me the story of Python 2 versus Python 3 with your project.

56:28 Ian Dennis Miller: Yeah, this was a bit of a struggle, actually. Because the ecosystem was a real barrier. In so far as all the packages that Flask-Diamond relies upon, they all need to have made the switch to Python 3 and by now they have. But also, you know, there was a little bit of stuff inside Flask-Diamond itself that needed to be updated. I eventually made the leap within the last year. But I think I found that Travis, actually, and automated testing, is a real asset with Python 2 and Python 3 compatibility. So the ability to automatically run the test suite with different Python interpreters has saved me a lot of headaches. Because I'm catching a lot of bugs that, well, less now the transition has been made, but it caught an awful lot of bugs moving from 2 to 3. But I find now that I make all my new projects using Python 3 and personally I'm really enjoying what's happened with Python 3 and I do kind of, I check in on the arguments whenever it comes up. You know, why isn't Python 3 whatever. I like to sort of torture myself reading that stuff online. Because I was held back for quite a while on Python 2.7. Now that I have made the switch, there's no going back. I really do like.

57:51 Michael Kennedy: Yeah, I'm with you. There's a whole bunch of cool stuff. Especially around concurrency and async stuff and just a lot of polish of the language.

57:57 Ian Dennis Miller: Yeah, strings and bytes in unicode. It's just so much better.

58:01 Michael Kennedy: I feel like we have, as a community, have sort of turned a corner. A year ago, a few years it wasn't entirely clear that Python 3 was like, of course you should doing it, people were still like no, I don't know about this. I feel like most people, even though maybe they have a project that's not quite converted or they don't foresee themselves able to move whatever project to Python 3, like, the recognition that it is the way to go I think is definitely growing. So I think it's a pretty positive move.

58:30 Ian Dennis Miller: Yeah, I think and finally, the critical mass is there.

58:33 Michael Kennedy: Yeah, yeah that's it.

58:34 Ian Dennis Miller: And it's the way forward, clearly.

58:36 Michael Kennedy: Absolutely, absolutely. Alright, well your project looks really cool. I feel like, you know, we've got Flask at one end with the micro-frameworks, we've got Django at the other end as more sort of full featured building blocks, I talked with Julia Elman and Mark Lavin on Episode 88 about a lightweight Django and bringing, you know, sort of unrolling some of the stuff out of Django to make more lightweight, to bend it however you want. And this is kind of taking some of the niceties out of Django and putting them into Flask and you're kind of like filing out the spectrum.

59:08 Ian Dennis Miller: I hope so, yeah.

59:09 Michael Kennedy: I think it's a cool contribution.

59:11 Ian Dennis Miller: Yeah, thanks.

59:12 Michael Kennedy: Yeah, let's see, before we go. Let me ask you the two final questions. If you're gonna write some Python code, what editor do you use?

59:18 Ian Dennis Miller: It's Sublime Text 3 these days. It's Python all the way down. I love Emacs, I love the Emacs philosophy. And I love Lisp, but the idea of redoing Emacs with Python as its heart and is brain is too cool. So it's Sublime Text.

59:39 Michael Kennedy: Yeah, that's pretty cool. Have you seen the Anaconda plugin for it?

59:45 Ian Dennis Miller: I have not actually.

59:46 Michael Kennedy: It's not the same as the Anaconda distribution. I think these things are unrelated. But it's like an extension for Sublime that brings all sorts of Python features, apparently. I just ran across it recently. But I haven't really explored it.

59:58 Ian Dennis Miller: I would have just discounted it as packaging of the other Ana, well that's awesome actually, I gonna check it out.

01:00:05 Michael Kennedy: I'm quite sure that it's a different sort of thing.

01:00:08 Ian Dennis Miller: Cool, cool.

01:00:08 Michael Kennedy: Yeah, so. I'll put a link. Yeah, so here it says, I open it up and it says Anaconda, the Sublime Text 3 Python IDE and it gives you all sorts of functionality. So it's not the same thing as the Anaconda, it's a different thing, check it out.

01:00:25 Ian Dennis Miller: I'm giving that a look, yeah.

01:00:27 Michael Kennedy: Awesome, OK, and then there's over 96,000 PyPI packages. I can't wait till it gets to 100,000, there's gotta be some kind of celebration. But you know, why don't you tell people about some that you know that you think are cool, they should know about, check out.

01:00:40 Ian Dennis Miller: Alright, so, I've already said so many good things about Flask and SQLAlchemy. Together they have changed my life. I think probably for the better. So everybody needs to go and look at those two. But I also just wanna, there's a special place in my heart for Beautiful Soup. It's a throwback to an earlier time. But when Beautiful Soup came along and made it easy for me to parse even really broken and terrible HTML and actually I was able to start doing useful stuff with web scraping and I began sort of collecting some data that I wasn't able to access before. I think there's a way that Beautiful Soup might have improved my life more than anything else in PyPI.

01:01:30 Michael Kennedy: That's really cool.

01:01:31 Ian Dennis Miller: It's a special place in my heart for that one.

01:01:33 Michael Kennedy: Yeah, you talked about the alchemy. The alchemy nature of SQLAlchemy. Beautiful Soup is kind of magical as well in that it like--

01:01:40 Ian Dennis Miller: Yeah, yeah.

01:01:41 Michael Kennedy: Everything becomes an API, even if it's not supposed to be.

01:01:44 Ian Dennis Miller: Right.

01:01:47 Michael Kennedy: Awesome and it's surprisingly easy to use. So if you guys need to do web scraping, check it out. It's definitely cool, I can recommend it as well. Alright, Ian, how about a final call to action? You looking for contributors to this project? People to try it out?

01:01:58 Ian Dennis Miller: Yeah, definitely checkout the website, flask-diamond.org, that'll link you through to GitHub. I'm looking for people to battle test this on Windows, because that's harder for me to do and ideally give me feedback. Or even better would actually be some edits to the documentation. Because I want to break down the barriers to get some new users initiated into this Flask ecosystem.

01:02:26 Michael Kennedy: Yeah, that sounds great. You know, people are always asking me like, how they can contribute to open source. Something that just came to mind is maybe they could come write a tutorial and you could have like a whole bunch of tutorials.

01:02:35 Ian Dennis Miller: Exactly.

01:02:36 Michael Kennedy: That would be a fun way to, like a fun and easy way to get started, right?

01:02:38 Ian Dennis Miller: I would love that. And you know, make a screencast video and man, you'll be my friend for life.

01:02:46 Michael Kennedy: That's awesome. Alright, well this is a really cool project and I think it fills an interesting space in the spectrum of the web and Python. So thanks for building it and thanks for taking the time to share the story.

01:02:56 Ian Dennis Miller: Thanks, it was really my pleasure.