#99: Morepath: Super Powered Python Web Framework Transcript
01:28 Martijn Faassen: Hello.
01:29 Michael Kennedy: I'm excited to have you here. I really love web development in Python, and I think web development in Python is actually one of the really early places that Python became super relevant. And we've seen it spread to other places really well like data science is the most recent of these, but at the heart, I think a lot of Python developers are web developers. When people ask me, "Hey, what do you do?" and I don't want a long conversation, I just want a quick answer like, "I write web apps." "I'm a developer." Something like this, so I love web apps and I think it's really cool even though there are many Python web apps around that you've created a new one relatively recently, within the last couple years. So, I'm excited to talk to everyone about Morepath. Before we do, though, let's hear your story. How'd you get into Python and programming?
02:14 Martijn Faassen: Well, programming I've been doing since I was a child trying to create games in BASIC on various computers. First, at my father's office because we had no computer at home, and then later on, on home computers.
02:30 Michael Kennedy: What was that first computer?
02:32 Martijn Faassen: The very first computer that I remember using, I think that's actually the very first computer I used was called a Triumph-Adler, which is, I don't know what kind of brand that is. It was a CP/M machine. And then after that, moved to early PCs. Olivetti M24s are the ones I recall also being very early. And my home computer was an MSX2, which was Z80-based and more popular in Japan and Europe than the United States.
03:04 Michael Kennedy: Okay, and you started originally just poking around on these things. Did you try to write some games?
03:09 Martijn Faassen: Yes, yes, like I imagine many programmers tried to write some games, very simple things.
03:14 Michael Kennedy: I think it's interesting, like, people really want to get into it and write games, and games are some of the hardest types of software to write.
03:20 Martijn Faassen: Yeah, it used to be easier, I think, back then, because the, even mobile phone games are more complicated than the games that were commercial back then. But even, there are a lot of sort of details like real-time requirements and things like that that you don't really have so much in other contexts.
03:36 Michael Kennedy: Yeah, absolutely. So, where'd you go from there? Playing around with it at home, did you study computer science in university or where?
03:42 Martijn Faassen: Yeah, I did something more artificial intelligence oriented, but that's a very long time ago. But I had some courses, various languages, Lisp, Prolog. I taught myself C. I had a Pascal course. I did some work with Delphi for a while back in the '90s. And then I was programming in C++. I was writing this artificial life simulation. And then I heard about scripting languages, read an article in some magazine at some point that listed Perl and Tcl and Python, maybe one or two others. And I basically just wanted to write some tools that could help me manage my C++ codebase, and I looked at Perl because that was sort of the obvious candidate at the time and it was on my computer. And messed around with it for a while, and I could see how that would be a useful language, like, it had made things like data pipes really easy, but it never quite clicked in my brain. Then I remember printing out the Python tutorial from python.org, and I read it one evening before sleeping, and the next day I was a Python programmer.
04:53 Michael Kennedy: That's awesome. I'm sure, it's one day, right?
04:56 Martijn Faassen: Yeah.
04:57 Michael Kennedy: I've done this as well, but it's been very, very long that I printed, like, I remember printing like 100 pages of source code in C++ trying to understand some application, and printed it and took it with me because I was going away somewhere for the summer and I didn't have a computer with me the whole summer. The world has changed so much, right?
05:15 Martijn Faassen: Yes, yes. This was just without a computer for during the evening, but I just had printed out the tutorial. And yeah, then a few weeks later I actually managed to get a job writing Python code in 1998, which was kind of--
05:27 Michael Kennedy: That's awesome.
05:29 Martijn Faassen: Kind of surprised myself, but I was at a, well, it was actually the veterinary department of a Dutch university in Utrecht, and they were looking for somebody who would clean up some script that had fallen apart that was written in some combination of DOS batch files and dBase code. They wanted me to rewrite it, and I said, "Can I use Python?" And they said, "What's that?" And I said, "Well, it's a programming language." And they said, "Is it readable?" I said, "Yeah, yes, it's very readable," which is really what they were looking for. And then I found myself having a job writing Python code, which then quickly sort of moved into the web as well. I started, that was '98 that Zope was released as an open source product back in '98 as well, so I started playing around with that and then moved into web development from there.
06:24 Michael Kennedy: That's cool, yeah. That first week or two of the job where you're just like, "What am I doing here? I can't believe they just hired me, I just barely know this."
06:32 Martijn Faassen: Well, I mean, I guess I had just been playing with Python for like a few weeks I guess, but it just clicked in my brain so much that I thought, okay, this is the right tool for the job. And it was sort of more or less a scripting job anyway, so that fit. And I remember I was reading like Complete Python, the Python use group, and people were saying, "Oh, I wish I had a Python job. I'm stuck here programming in this other language." That it looks really unrealistic to get a real Python job, you know, back then those were really rare. And I was like, "Hey, I actually have a Python job." And I, I didn't even particularly look for one. I just had one, so I was lucky, yeah.
07:11 Michael Kennedy: Yeah, that's really lucky. I still hear that from people today that, you know, they're working in some other technology, but they play with Python at home and they're looking to get there, so that's really cool.
07:21 Martijn Faassen: Yeah, it's everywhere though now. I mean, Python, it's really a different world. I remember, Python, I had to explain to everybody what Python was. I mean, even other software developers. These days, I don't have to do that anymore. It's really everywhere, yeah.
07:58 Martijn Faassen: Uh-huh, yes, yes. And there is just lots of commercial projects that use Python in one and other way that don't even show up in GitHub as well.
08:07 Michael Kennedy: Yeah, absolutely. So, you talked about Zope and some of the, thinking of the successors a little bit like Pyramid and whatnot. It seems to me like those were kind of inspirations in your new web project, 'cause you referenced them somewhat positively in your documentation and things like that.
08:24 Martijn Faassen: Sure, yeah, no, I started with Zope in '98 and I was very involved with it until, I think even, yeah, up to about 2010. I was even a chairman of the Zope Foundation for a few years. And Zope was really, it was really a killer app for Python in a way because there are a lot of the, as you mentioned, web development was an early thing for Python. Though, I think data science was there very early as well. But the conferences, like the early EuroPythons and the International Python Conference that I went to even before then, a lot of the people there were there because of Zope. That's how they were attracted to the conference and to Python, so it drew in a lot of people, similarly to, I imagine, Django does now. Zope was a web framework before we even had a word for that because web framework only arose as a concept later so it was called a web application server, I believe.
09:26 Michael Kennedy: And these frameworks were extracted out of like, let's build this web app, oh, and actually, there's some kind of framework we could almost use here, right? In that sense.
09:32 Martijn Faassen: Zope was actually, yeah. It was a very strange beast, and in part, it's because there was no normal beast out there at the time because everybody was exploring and in part, because it was enormously creative. What it had was a web-based user interface, which doesn't sound very amazing now because lots of things do, but back then, that's kind of unusual for web application development framework. And through that, you would add templates and later on, you could add little pieces of Python code. And then it sort of slowly morphed into doing more and more sort of larger-scale Python projects using Zope's plugin architecture. And fairly quickly, we were dissatisfied with Zope's plugin architecture, and then this whole Zope 3 project emerged, which was sort of intending to rewrite Zope and make it all better, and create the future paradise, and while we create a lot of interesting technology and we learned a lot, and I actually built some real applications with Zope 3. It never really took off because we sort of overshot in the other direction, as you tend to get.
10:42 Michael Kennedy: Yeah, you rarely make one rash decision. It's usually, you swing back the other way, right? It's just, I see that all the time in technology that people have some kind of monolithic thing, and they're like, oh, we're gonna fix this. We're gonna make it extensible. You can enable, disable, et cetera. Then it just becomes like, well, wait, this is no longer easy to use. I liked it 'cause it was easy, and now it's like, there's a hundred choices and it's hard to fix when it's broken and all that kind of stuff.
11:05 Martijn Faassen: Yeah, the early Zope was never really easy if you were gonna build a larger Python-based project with it, but if you just wanted to write something quick sort of using the web user interface, then yeah, it was very easy. And yeah, I actually I spent time in making the Zope 3 technology available in Zope 2, so I created this little glue framework that I called Five, which is what inspired the Python name for Six. The name Five was already taken, so they multiplied two and three.
11:36 Michael Kennedy: Yeah, oh, that's cool. I didn't know you were behind that, yeah.
11:39 Martijn Faassen: So I'm not behind Six, but two plus three was already taken, so they had to go to Six instead of Five.
11:45 Michael Kennedy: Nice, yeah, you already had the package name, so that's solves that debate.
11:48 Martijn Faassen: Yes, that technology is still in use inside of the Plone CMS to this very day. And then what I did in 2006 is actually started together with a bunch of other people, but sort of I kicked off that project, the Grok web framework, which was based on the Zope 3 technology, but tried to put a friendlier face on it, also inspired by sort of the new wave of web frameworks that had been coming then like--
12:12 Michael Kennedy: Right, that's like Flask and Django time. Yeah, Ruby.
12:15 Martijn Faassen: Flask was not around yet, but yeah, Ruby on Rails and Django were new at the time. So, we tried to do something like that with Grok. We had some sort of moderate success because it sort of benefited also from the larger Zope community. People did start using it. It was a lot easier to start a new web framework project back in 2006 than it is in 2013.
12:38 Michael Kennedy: Yeah, the expectations were way lower of what a web framework had to be, right? It didn't have to be as polished. And the web itself was simpler, right?
12:47 Martijn Faassen: I think the web itself was simpler, but I think the main thing that I think has changed a lot is documentation. Back then you could start a new web framework and have, you know, some documentation but not really exhaustive documentation or very good documentation, and still get some traction. These days, that's a lot harder I think. The web was simpler, but the basics of AJAX, I remember I implemented those early on in Grok as well, so that hasn't changed all that much in some ways. But of course, client-side development was still very different. There wasn't as much of it going around back then.
13:25 Michael Kennedy: Yeah, it was much, much smaller scale. I mean, maybe jQuery was around then? I don't even remember the time, maybe not. I think it was, right?
13:31 Martijn Faassen: Yeah, I remember. I think I used MochiKit for a while, but yeah. I don't know whether jQuery was around then.
13:37 Michael Kennedy: Yeah, but definitely not AngularJS and a lot of those types of things, right?
13:41 Martijn Faassen: No, I did a framework there as well. I just keep creating stuff--
13:46 Michael Kennedy: You do create a lot of frameworks, I know.
15:01 Michael Kennedy: Oh, that's cool. It was absolutely the time. Yeah, it was definitely the time when there was many, a lot of evolution and creation around that kind of stuff. There still is, but a lot, it was like every month you'd hear about something that's gonna change the world as you know it, right?
15:31 Michael Kennedy: Definitely doing it before it was cool.
15:33 Martijn Faassen: Yes.
15:34 Michael Kennedy: Very interesting. So, let's talk about your project Morepath. One of the things, I watched a nice talk from you at EuroPython in 2014, and I'll be sure to link to the video, and one of the things you started out talking about was why would one go and create a new web framework when Flask, Django, Pyramid, and lots more exist already?
15:56 Martijn Faassen: Yes, and that's basically because I can't help myself. It's just a creative urge. I mean, of course, there are reasons why I created one. I had a customer at the time who wanted something that could do REST APIs really well and was small enough to plug into a larger application, so I could've used an existing framework for it, but I said, you know, what if I just take some ideas I already have? And I had some pieces of code lying about, and I just put it together and I'll create a little framework and that's where Morepath came from. And it's still a relatively small framework. It hasn't grown that much.
16:33 Michael Kennedy: Yeah, you describe it as a micro framework and it seems certainly like it fits well in there. It seems like somewhere between Bottle and Flask, maybe.
16:42 Martijn Faassen: I cheat a little bit because Morepath, the core itself is just a couple of thousand lines of code. It's built on top of this predicate dispatch library called Reg, which I also crated, which is just 500 lines of code, I think. But then I spun off another library for Morepath I call Dectate, which is a configuration of Python, sort of decorator configuration library that Morepath uses to command its directives, and that's also some, I don't know, hundreds of lines of code as well, so you can still be micro if you depend on a few libraries that you're also maintaining. But nonetheless, it's still pretty small. I think it's about the size of Flask or so. Depends a little bit on what you count, if you count the template language implementation or you count the request-response implementation, which is WebOb in case of Morepath. And you know, that's, how do you add up thee things? How many lines of code is a micro framework? But yes, it's small, yes.
17:40 Michael Kennedy: Yeah, in my mind, micro framework is more about the exposed feature set and how much I have to learn to work with it more than whether it's 1,000 or 2,000 lines or whatever.
17:51 Martijn Faassen: That's true. Yeah, that's true.
17:52 Michael Kennedy: Yeah, and since I feel like your framework is something you can pick up pretty quickly and you can kind of understand a good portion of it pretty quickly. So, let's talk about the goals first. What were the goals of this project?
18:04 Martijn Faassen: So, one of the things that I've been doing with Grok is I started using, so I have to go back to Zope again. Zope and also Grok use this database called the ZODB, the Zope Object Database, which is a pretty nice object store for Python. You can just put in any almost arbitrary Python object, hook it up to another Python object just as an attribute, and as long as there's some path to a root object, then everything gets persisted automatically for you. That makes it really flexible 'cause you can just basically store any kind of Python object to represent any model that you'd like.
18:45 Michael Kennedy: That sounds like a kind of early concept of what document databases became.
18:50 Martijn Faassen: Yeah, I mean, it's very tied to Python, which is both its strength and its weakness because, you know, if you change your class, then suddenly the instances need to be updated. But it used to be very hard to explain because when you said database in the late 90s, early 2000s, people just assumed you were talk about relational database with SQL on it, and there was nothing really else that people were familiar with. And then NoSQL happened and then it became a little easier to explain.
19:16 Michael Kennedy: It's like a hipster NoSQL database 'cause it was NoSQL before there was NoSQL also. Interesting.
19:21 Martijn Faassen: Yes, yes. I mean, that makes some things very easy, but I needed to integrate a relational database with Grok, and one of the nice features about the Zope Object Database is that you can very, well, okay, we have to go back to the Zope publisher. Zope has this sort of publisher that's based on traversal, so you just traverse a Python object. So, if you have a path, a/b/c, to resolve that to an object and then call a method on that object to represent it as HTML or JSON or whatever, it would basically just do a lot of getattr and getitem into the object tree, so you would just get attribute a from the root, and then you would get item b from the item that you just got, and then you ask for getattr or getitem from that object, and you get another object, and then you call its represent me as HTML method on it. So that's how Zope worked, and the nice feature of that is that it also allows you to construct URLs really well because you can retrace the path that you took. So, if you have an object like a document, and you want its URL, you can construct it automatically. So, I like that feature of object database and traversal. So then with Grok, I wanted to integrate a relational database and I was using SQLAlchemy, but I wanted to retain that feature. And then first I sort of tried to coax SQLAlchemy to become a traversable tree, and while SQLAlchemy, being very flexible, made that possible, it was not really very, it didn't really fit the models very well. And to have to adjust the models just so you can actually traverse to it with URL paths didn't really work very well. So I had this idea, okay, I'll do it another way. I'll create a routing framework, but one the property that you route not to a view, but to the model just like I was used to from the ZODB, and also have the property that you can take instance of a class that is in a routing system and ask for a link to it, and then it will just construct it automatically from the routing information that it has.
21:34 Michael Kennedy: Right, so you could do something like if you had, say, like a customer, let's say, your routing URL might look like, something like /customers/7.
21:45 Martijn Faassen: Sure.
21:46 Michael Kennedy: It might know that, okay, well that 7 actually corresponds to the customer ID, and then you have decorators that will indicate, well, here's the function that you call to actually do that query and represent it however it's gonna be represented, right?
22:00 Martijn Faassen: Yes, yes, that's how it works in Morepath. In Grok, it still had a slightly different sort of, there were no decorators, but otherwise, yeah. That's more or less the principle. So, to resolve something, you first look up the model instance. You first do your query or you construct some kind of Python instance. You return this from a function that's connected to the route, and then after that, you look up a view function for it and it can do this based on the class or what you just found, and information in the request and things like that. So it can find this view function, and then the view function gets the request and also the instance that you just returned from the path function, and then you do your rendering in there. And that has the nice property that you can link to any class that you expose to that routing system.
22:45 Michael Kennedy: Do you register documents or type entities and say these are the entities that you might be able to go to?
22:52 Martijn Faassen: I just, or whatever models my views need, so in some cases those are backed by relational database like a customer, and in some case I need to create a model that represents, for instance, a collection, which is not really backed directly by a relational database, but you can implement it in terms of queries as well, and then you just attach views to that collection class. So yeah, with Grok I figured this out and then with Morepath I sort of took that idea and cleaned it up and made that sort of one of the central principles of the web framework. And I think besides the easy linking, which was for the primary motivation, I think it also really helps structuring code. With other web framework, often you see some kind of, get this object from the database, or if you can't get it, do a 404 error kind of convenience function. And if you don't use that convenient function, you have to watch for, you know, a failure to find the object. And if you do so, you have to remember to raise a 404 error yourself. With Morepath, 'cause it has more knowledge about being able to find the model or not, it's not hidden inside of a view function like you would have in Flask or Django.
22:52 Michael Kennedy: Right, and if you screw that up in, let's say Flask, you probably get like a, a None type does not have whatever property you're looking for error, right?
22:52 Martijn Faassen: Yeah, you'll get a 500. Yeah, you're getting a 500 error.
22:52 Michael Kennedy: And that's surfaces as, you know, cannot process request, 500, which always gives people confidence in your application.
22:52 Martijn Faassen: Yes, yes. It turns out that if you write, they have to write that boilerplate code anyway in your application, it almost doesn't cost anything anymore to just write a separate function to separate the two stages. And you get other features from that as well. You can do permission checks also 'cause you have an idea of the instance that is being represented, so you can say, okay well, customers are only accessible, views for customers are only accessible if you have a certain permission. And you can actually make the concept of customer play a part in your authorization system as well because you constrain yourself to really route to a model instance first and then to a view you get from that constraint, the framework and benefit and build on top of it.
22:52 Michael Kennedy: Let me take just a moment and tell you about Metis, a sponsor of this episode. A data scientist's responsibilities can range far and wide. How can you be sure you have the necessary skills and training to enter the field or keep up with emerging technologies? Metis, a data science training company based in New York City, San Francisco, Chicago, and Seattle provides full-time, immersive boot camps, evening part-time professional development courses, online resources, and corporate programs all with a goal of training you to become a data scientist and help you stay on top of new and necessary skills. Metis has a long list of professional development courses starting soon in any of these four cities taught by industry leaders. Interested in honing your skills? Then these courses are for you. How do they work? Courses on topics like data visualization, deep learning with tensor flow, machine learning, and statistical foundations run two nights a week for six weeks during the evening hours that fit into your busy schedule. These courses are laser-focused on relevant topics and skills that are sure to enhance your career. They have a special offer for you at thisismetis.com/talkpython. Be sure to check out what they're offering. It helps support the show. Sounds to me like you focused very heavily on RESTful type of ideas, right? Like the routing framework assumes that there's some kind of resource or model at the end, and you can use like get to do default queries on it and things like that.
22:52 Martijn Faassen: Yeah, one of the basic ideas behind Morepath when I started developing it is to be good at doing REST. It turn out that actually, you know, a framework that's good at doing REST actually is also pretty good at doing traditional server-side web applications as well that do generate HTML. Those things are not in conflict with each other at all, but yeah, the initial goal was to do REST well and actually template language implementation took, I think, a year and a half or so until I finally got around to doing that.
22:52 Martijn Faassen: Yeah, because I had been developing, even with Obviel, I've been developing that way so I had some ideas what I wanted my framework to be able to do, also from the client-side perspective. And one of the things that I always have been is I tried to actually do real REST. These days, REST, the term, has become so eroded that it basically just means, yeah, it means endpoints that the client just needs to know about and it returns some kind of data format, but it doesn't involve linking or anything like that. So, I always actually try to do end work with real REST, where your JSON data structures links to other structures, so sort of the other end points, and that you can sort of enter your application with one good endpoint, and then find all the links, and follow them, and perhaps also embed objects if you want to get...
22:52 Michael Kennedy: To me, REST has become a synonym for HTTP-based services, usually that return JSON, but I think the two things that are often neglected that you are hinting at is, one is content-type negotiation and the other is the dynamic linking and discovery, right? You request an original resource and it tells you how you can
22:52 Martijn Faassen: Yes.
22:52 Michael Kennedy: continue to explore the service. Morepath has good support for content-type negotiation, right?
22:52 Martijn Faassen: Well, actually, that's a thing I have cared less about, so it's actually, we do have a really powerful system, which allows you to build that should you want to, but I haven't seen a lot of people actually try. So, the idea that if you request HTML, you'll get HTML from the same URL as where if request JSON, then you'll get JSON or perhaps you have even multiple versions of your API that are negotiated that way. In practice, I haven't seen a lot myself yet. That doesn't mean very much.
22:52 Michael Kennedy: Sure, but you can go to a view and say, you know, "@App.json", "@App.html", and things like that, right? Like theoretically.
22:52 Martijn Faassen: Yes. What you can do is you can name multiple views and then you can do, it's all negotiation, but you can just say okay, I want a different version of this, or a different view entirely, actually because views are looked up by the class of the model, but also by the request method and also by the name of the view, which is the last part of the path. If it's there, you can use an at sign. And you can also look up views if they have a body. You can actually interpret somehow the post body if it's JSON, and then switch to a different view if you post something else, just have it dispatch to a different function. And that whole system is extensible, so you could just write a new directive and add more criteria for your views in your own application. It's actually possible to have, in the same runtime, to have one application that uses more criteria than another one, and they will just live next to each other 'cause of Morepath configuration system. So, you can add stuff that looks at the accept header. That's pretty easy to implement in a basic case, but if you want to do full accept header support, it tends to get a little bit hairy 'cause there are a lot of different possibilities there.
22:52 Michael Kennedy: I find you're right that that's a very rarely used feature of services. Usually people say, "Go to this URL, do a get, and you're gonna get JSON." Or something like that, you know?
22:52 Martijn Faassen: Yeah, yeah, because the information is encoded in URL, right? You can just see it, and if you suddenly have to worry about, I mean, yeah, content types that you do the request with, it becomes a lot harder to get an idea in your head of what's going on.
22:52 Michael Kennedy: Yeah, yeah, for sure. So, one of the things that you talked about that I thought was cool, and you had some nice hand-drawn pictures--
22:52 Martijn Faassen: Nice, yes.
22:52 Michael Kennedy: In your presentation, yeah. And one was, like, a web framework embodied as a person pulling their dress shirt open, and there's like a Morepath Superman-type suit underneath and you talk about the super powers of Morepath. Do you wanna touch on those a little?
22:52 Martijn Faassen: Right, so some of them I've touched on already, but one of the super powers I would say is the dispatch system, so the view dispatch system, but it's also used in other areas, like this authorization system uses it, for instance, allows you to write Python functions that dispatch on not just one argument like you have normally in Python with a method that dispatches on self, which you can look at it that way, so the class of the self argument of a method with the same name determines which actual method gets called. You can also dispatch on other arguments as well, and not just its class, but also other attributes, so those arguments, and that allows the view system to dispatch on the class with the model. The request methods get composed of the request. The view name, which is also stored in the request, the last part of the path, but also the accept header or the type of the JSON body that you submit, or whatever else you come up with.
22:52 Michael Kennedy: Yeah, I can tell linking is super important in this whole, routing and linking is really a key part of what you're trying to focus on.
22:52 Martijn Faassen: Right, but yeah, this is the dispatch system.
22:52 Michael Kennedy: Okay.
22:52 Martijn Faassen: That's not the linking or the routing system. It allows you to write views that can be really general, work for any subclass over a particular base class, or be really specific for specific class as well. And you can dispatch a lot of different factors. So, that's the Reg bit.
22:52 Michael Kennedy: Yeah, there's almost like a inheritance thing that you can do with that stuff, right?
22:52 Martijn Faassen: Yes, yes.
22:52 Michael Kennedy: You can kind of override a view in a specialized class.
22:52 Martijn Faassen: Yes.
22:52 Michael Kennedy: Okay, gotcha.
22:52 Martijn Faassen: Yes, yes, so it's like basically, if you just look at the model bit of how you register views, it's very much like you define methods but you don't define them inside of the class, you define them just as functions outside of the class, and your decorator says where they belong, but basically that's just one factor of what it does the lookup with. And authorization also works that way. It's like, do I have this particular permission for this particular model instance? And it also knows about inheritance, so you can be very specific about, well, this subclass needs to have a different permission rule than everything else. And that's Reg, that there is the, well, the automatic link generation I already talked about, which I think is really neat, and most frameworks don't really do it. I mean, they do it sort of half way, where you say okay, well you need to give all your routes a name, you need to come up with a name and remember that, and then when you want to generate a link from your Python code or your template or whatever you need to use the special API or do the special thing, and then you need to give it all the variables that are needed to construct the link again. So, you need to know still what those things are, whether this idea or whatever the link construction is based on--
22:52 Michael Kennedy: Yeah, and it usually appears in multiple places. There's usually some configuration thing where you define the route for the URL dispatch and then there's the actual template where you put the thing and you have to know, okay, I have to replace these two parts of the URL with this part of data, and then in your actual view, you've got to pull that data out of the route data that was passed and it's at least in three places.
22:52 Martijn Faassen: Yeah, yeah. So you have to do that, and if you have multiple places where you're linked to the same thing, you have to do it sort of in multiple places. And that also makes the code less generic because suddenly it's dependent on the particular structure of what you're linking to, and if you were to change that, then you would have to change all the templates that link to it.
22:52 Michael Kennedy: Yeah, everything breaks. So, what I do on my web apps is on my model, I will use some kind of rich class and then convert it to a dictionary at sort of render time, but I'll put a method like here's the function or the property that gets me the URL with all the pieces filled in. And so I'll go in template and say, you know, dot purchase URL or whatever.
22:52 Martijn Faassen: Yeah, exactly, yeah.
22:52 Michael Kennedy: You've kind of baked this concept directly into Morepath in a sense, right?
22:52 Martijn Faassen: Yes, yes. So it's that concept, but then you don't have to burden your classes, your model classes, with information about the web, which is basically, I mean, it's a reasonable compromise. I see Django, you can also do it, but Django does convention there as well. It's still why would my model class know about URLs. That's the whole point about routing framework is to not have to worry about that kind of stuff in the inner model classes.
22:52 Michael Kennedy: Exactly. You look at the standard MVC design pattern, there's an arrow from down to the model, but there's no arrows back from the model, right? Which is like these, basically what I'm proposing, right? That it knows how to get back to these things.
22:52 Martijn Faassen: Yeah, and Morepath knows how to do that, but the information is stored outside of the model classes themselves.
22:52 Michael Kennedy: Nice.
22:52 Martijn Faassen: In the configuration system. But that allows generic code as well if you make, you have a list of objects, so you just want to create a list of links to it. And those objects might be in completely different places in your website. I mean, if you were to solve that with, well, you can solve it if you, again, burden your model with this get URL method, then you can do it, but without that you would have to use a lot of special cases in your template or whatever to create the right link. So, Morepath does that for you. It gets the information that it needs from the models. By default, there's a convention that the name that you use in the route is the name of the attribute, but you can override that to reconstruct it. So that's really powerful. And then there's the configuration system so Morepath has an actual sort of thought through configuration system as an entity by itself, which should not be that unusual for web frameworks, but it actually is. And what I mean with configuration system is the system that lets you express things like, okay, well this route exists, and this view exists, and I have this permission to access that view, or I want to install middleware that sits between the request and the handler and the response and sort of can interfere in there. Or I have a template rendering engine that I want to plug in. All those things I consider to be configuration, and that's really my Zope history speaking because in Zope 3, that was one of the big innovations. We created this XML-based configuration language called ZCML, where you would express all of this in XML files, which sounded really cool at the time, but one of the things that I did--
22:52 Michael Kennedy: You just put this namespace here on this XML element that, oh, yeah. Just kidding.
22:52 Martijn Faassen: Oh yeah, well, you know, you would just have a little XML language that was namespaced, of course, where you would say, okay this, while do that routes, but you would say this view has this permission, and if you, though all kinds of things that could be, can say like, configure you would have adapters and things like that, but that's all pretty low level stuff. But also services like the service that gets an email is here, so if the application asks for it, just look at the configuration, basically, for the service that gets an email. That kind of path that you tend to see a lot in larger applications that you can customize in different context like a CMS, for instance.
22:52 Michael Kennedy: This portion of Talk Python To Me is brought to you by Hired. Hired is the platform for top Python developer jobs. Create your profile and instantly get access to 3,500 companies who will work to compete with you. Take it from one of Hired's users, who recently got a job and said, "I had my first offer on Thursday after going live on Monday, and ended up getting eight offers in total. I've worked with recruiters in the past, but they've always been pretty hit and miss. I tried LinkedIn, but I found Hired to be the best. I really like knowing the salary upfront." "Privacy was also a huge seller for me." Sounds awesome, doesn't it? Well, wait until you hear about the signing bonus. Everyone who accepts a job from Hired gets a thousand dollar signing bonus, and as Talk Python listeners, it gets way sweeter. Use the link hired.com/talkpythontome and Hired will double the signing bonus to $2,000. Opportunity's knocking. Visit hired.com/talkpythontome and answer the door.
22:52 Martijn Faassen: What I did with Grok is already try to merge that back into Python so you can express it, well, we didn't really have class decorators at the time, and function decorators were very new, so we did it in another way, but you can express it inside of Python code. And then in Morepath, I express it with decorators. And all those decorators are attached to classes, application classes, subclasses of the Morepath application that allows you to the many classes you can inherit so you can say, if you want to inherit all the configuration of a particular base class, and it overrides specific bits of configuration like one view or add a piece of middleware or whatever, you could just do that with subclassing, and then you add your configuration to the subclass, and then your base class still has the original configuration, so if you were run that behind a WSGI server, it would still behave as it did originally, but your subclass has a new, if you instantiate that, you get a WSGI app, and that will have the new behavior. And that's really powerful. What's also powerful is that you actually compose those little applications together into a larger one, so Morepath allows you to mount the applications onto other applications, which all could have their complete different configuration. It could even override some very basic behavior of Morepath and run in the same runtime.
22:52 Michael Kennedy: Sure, so give me some examples of like what two or more applications might look like, how do they work together, what's a practical story there?
22:52 Martijn Faassen: Well, so for instance, if you wanted to develop something like a GitHub where you have projects, projects could have issue trackers, and you also have a wiki perhaps in a project. Now, those issue trackers and wikis could be implemented as separate applications that don't know anything about projects or each other. So, you could configure them with their own routes, and views, and things like that. And then you can mount them together onto the project application should you want to use the wiki and the issue tracker there, but if you wanted to mount them somewhere else in a completely different project, you wanted to use the wiki, you could mount it there. And in the mounting, the act of mounting, you define how to translate from the outer application's set of concepts to the, for instance, find the ID of the wiki app or whatever, to find the wiki in the database. So they're really isolated from each other, but you can combine them with composition, and then, you know, if you have a customer who likes the wiki and it's all perfect, but just wants to tweak just this one little thing in their deployment, then you could subclass the wiki, and change it in that one little way. Hopefully, your wiki will let you do that. That's still up to the design of the application, but yeah, that kind of feature is something that I was used to from Zope 3, and Pyramid is the other Python web framework that has this kind of capability, thought it does not have the mounting system. I don't believe Pyramid allows you to run multiple configurations in the same application, but I might be wrong.
22:52 Michael Kennedy: Sure, you might have to just merge it.
22:52 Martijn Faassen: At least it has this notion of configuration override extension, that kind of thing. Morepath tries to let you do that while also being really simple to use, so you just, you know, you have application classes and you can subclass them.
22:52 Michael Kennedy: Sure, that's really cool. So instead of having one big monolithic app, you could write a bunch of small apps and then sort of glue 'em together.
22:52 Martijn Faassen: Yes, yes.
22:52 Michael Kennedy: Okay, that's really, yeah, that's really cool. You have a couple of ways to get started. Like, if I wanna get started with Morepath, what are the few steps? Obviously, pip install morepath, then what?
22:52 Martijn Faassen: Well, there is a--
22:52 Michael Kennedy: Do we have scaffolding? Yeah, we have a cookie-cutter template out there now, it's in the documentation. So you could use that and then you get set up with it, so a lightly bigger Morepath application. I've been resistant to doing something like this for quite a while because I find that if you do a lot of scaffolding, it sort of becomes tempting to put a lot of, generate a lot of code basically when your project starts. And then I always think, okay, why is that not a library? Why is there not configuration? Why is there so much stuff that gets generated just to get an application going? Yeah, I think that Django sort of has that, right?
22:52 Martijn Faassen: Yes.
22:52 Michael Kennedy: People think that Django is huge, but you can actually create a one-file, simple Django app. It's just nobody does it, so it feels larger I think, in some sense. I think also Django is larger in other senses, but I believe--
22:52 Martijn Faassen: No, yeah, it is larger, but it has its ORM, and a template language, and a lot of other things.
22:52 Michael Kennedy: You know, and I think Pyramid suffers from this as well in that it feels really big 'cause when you create it, there's so many pieces. I don't necessarily mind that, but I think that's a message that it does send, right?
22:52 Martijn Faassen: It's a different philosophy, I think. Both of them have their benefits. Pyramid, I think, really tries to keep a lot of options open and to be sort of a little toolkit that you can use in very different ways, with a ZODB, with a relational database, with routing, with traversal. It has all those different options, and it tries to be pretty neutral about what the way you pick is, and that's, I mean, that's useful, but with Morepath I tried to be a little bit more opinionated. I mean, it's still a micro framework. You could still do a lot of different things. But I do say, okay, there's routing and you route the models. There's no traversal, there's no routing to views. You could plug in the ODB database. It's completely database agnostic, but it tries to be a bit more opinionated about some bits in hopes that that also makes it more approachable so that, you know, basic application is not so much heavy lifting.
22:52 Michael Kennedy: Sure, well, I think opinionated has been a success of Ruby on Rails. It's been a success of Django to some degree. So, it's not necessarily bad.
22:52 Martijn Faassen: No, no, no opinionated is not bad, but it's sort of a tricky balance between be a micro framework and be flexible, but still being opinionated. But you know, if you're gonna be a framework, you have to have some opinions, because otherwise no point to it at all. You're a framework, you're supposed to guide people in one direction or another. And Pyramid does have opinions. It has opinions about the configuration system, about these, it uses the Zope Component Architecture to put things together, and those things are pretty fundamental to its use.
22:52 Michael Kennedy: Yeah, absolutely. So, there's a place in Morepath where I can plug in some kind of middleware? Like, what are the options for that? There are many things I can choose from and wire together.
22:52 Martijn Faassen: There's a tween system and I just took that term and the concepts from Pyramid. I stole a lot of ideas from Pyramid and Chris McDonough because he did, he went there first, right? He was coming from Zope and then he thought, okay, I'm gonna do it better, and then he was very successful doing that. And then I thought, okay well, let's try as well, see where I end up.
22:52 Michael Kennedy: Sure. So tell people about what tweens are in case they don't know.
22:52 Martijn Faassen: It's a slightly higher level abstraction of what a WSGI middleware is. A WSGI middleware is a little bit too low level because WSGI's very low level, but a tween is a function that takes a request object and calls some kind of handler, which is your application or another tween that's sitting below, that returns a response object. And because you have this function that you plug in, you can do stuff to the request just before it goes into the handler, or you can do stuff to the response just before it comes out of the handler, and that allows you to do all kinds of neat stuff like--
22:52 Michael Kennedy: I see.
22:52 Martijn Faassen: Commit transactions or do stuff to the HTML that comes out of the system or whatever.
22:52 Michael Kennedy: Right, like I think rollbar is a tween in Pyramid, and you can install it, and any time there's a 500 error, it'll capture all these details and send that off, and all you gotta do is pyramid.include rollbar middleware name, it's like pyramid.contrib.rollbar or some variation of that name.
22:52 Martijn Faassen: Yeah, Django also has its own middleware system, which is very similar. With Morepath, it plays into the configuration system. Though, of course, only the outer application, mounted applications, they all share the same middleware of the outer application. But with inheritance, yeah, it allows you to basically make available a Morepath application that has some middleware installed, and then if you subclass that and that middleware is there as well for you, and so you don't have to know very much to use middleware. And you can also configure middleware to go on top of that or just under it. And then Morepath takes care of doing the sorting so that everything happens in the way you specified if it's possible at all, of course.
22:52 Michael Kennedy: Okay, sure, of course. Yeah, yeah, very nice. What kind of templates are supported? Like do you have Chameleon? Do you have Jinja2?
22:52 Martijn Faassen: Yes, Chameleon, Jinja2, and Mako, I believe.
22:52 Michael Kennedy: Yeah, okay, those seem to be--
22:52 Martijn Faassen: A bunch of plugins. They're not out of the box. You just need to install the template dependency, more.chameleon, or more.jinja2, et cetera, and then you just subclass from a base class that those packages expose. And then you have that template language available, so then when you use that extension, those particular extensions for like .pt for Chameleon, and .jinja2, I believe, for Jinja 2, then it will just render using that template language. You could even use both in the same app if you wanted to.
22:52 Michael Kennedy: Yeah, and one of the other types that you have is just straight up JSON, right? You can just put in a decorator, say--
22:52 Martijn Faassen: Yes.
22:52 Michael Kennedy: This thing renders as JSON. It just converts directly.
22:52 Martijn Faassen: Yeah, then you just return a Python structure that can be, yeah, serialized to JSON.
22:52 Michael Kennedy: Okay, excellent. What's the deployment story? This is just a straight up WSGI appliation, so I put it on micro WSGI class backed by NGINX, or Gunicorn, or whatever I want, right?
22:52 Martijn Faassen: Yes, yes. If you take the application class and you instantiate it, then you get a WSGI application, and you could just slip it in to whatever you like.
22:52 Michael Kennedy: Okay, excellent, excellent. How about Python 3 versus Python 2? What's the status of the various versions?
22:52 Martijn Faassen: Well, we support both. We have for some years now. For Morepath, that's fairly simple. For Reg, there's a few very dark corners of Python that we have to be aware of. But yeah, both are supported.
22:52 Michael Kennedy: Okay yeah, excellent. It's good to see lots of projects supporting Python 3 these days. Nice to here that one does.
22:52 Martijn Faassen: Yeah, it's easier for libraries than for applications, so if you have a test suite, that's pretty comprehensive. It's pretty doable.
22:52 Michael Kennedy: Okay, excellent. So, you're also involved in some other projects and we're getting kind of into the show, so maybe we just really quickly say what they are and then one or two sentence about 'em. Don't wanna go deep on it, but just to let people know the other projects that you've worked on. So we talked about Zope and Grok already and Obviel, Morepath. But you've also done some stuff with lxml, right?
22:52 Martijn Faassen: Yeah, I created the lxml years ago. Yeah, I'd been messing around with XML in Python. XML was way cool back in 2000.
22:52 Michael Kennedy: I remember.
22:52 Martijn Faassen: So I messed around with Python and XML, and tried to write my own DOM implementation a few times, and created sort of a start of an XPath implementation at one point. It was all very interesting and I learned a lot, but none of them became anything very useful. And then, at one point I had already worked with the libxml2 library, which is this really fast, very featureful XML library libxslt on top of that, written in C, and they had Python bindings, but they really sucked because it was like programming C in Python. So if you initialized something wrong, then you get a segfault in Python.
22:52 Michael Kennedy: Oh no.
22:52 Martijn Faassen: And if you forget to free something, you would just get a memory leak in Python. And those things are not something you want from a Python library, so I decided okay, let's be minimally ambitious, and build on top of this libxml2 library 'cause it's really fast. And I tried create a Python API, and also not invent a Pythonic API for XML because Fredrik Lundh already had done that. He created ElementTree, and there wasn't any C element tree at the time, just the ElementTree. So what I did is okay, let's create an ElementTree API with extensions for XSLT, and XPath, and all those things on top of libxml2, so that's what I did. That actually is quite tricky because of trying to do garbage collect C objects that can be in tree, in tree shape to each other, and things that can be separate from each other, but maybe, there's all kinds of interesting details there. And I use a technology Pyrex at the time, but it's now being renamed to Cython, so I was a very early adopter of Pyrex 'cause I didn't want to write
22:52 Michael Kennedy: Oh, that's cool.
22:52 Martijn Faassen: And then, one of the best things in my open source career sort of happened. So, I worked on lxml for maybe a year or so, and I got it working and it was doing garbage collection, and there was an ElementTree API, and it was all working pretty nicely, and then this German guy, Stefan Behnel, he started sending me lots and lots of patches, and all kinds of enhancements and features and things like that, and basically I was busy. I couldn't really keep track of all the patches, and then I made the conscious decision at one point, I remember it being the end of the year at one point, and I mailed him and I said, you know, now you're co-maintainer of the project. You can do whatever you like. So I gave him that power and responsibility, and that was a very good move because he's been maintaining lxml since then, which is like, I don't know, more than 10 years now, and added lots of features and improvements and things to it, and he also, he send it down the stack because he ran into Pyrex 'cause I was using that, and he wanted to improve that as well. And in the end, he and people from the data sciences angle, they forked Pyrex and they basically created Cython because, you know, he was descending to that as well, and I sort of always wondered if he keeps descending, you know, eventually he'll be doing micro code architecture for CPUs or whatever, but he stopped there as far as I know. He's only doing Cython, but he's been a very capable maintainer so I always say that I put lxml in the open source bank, and I'm getting interest on it because people have added features to it since then and I still use, I was using lxml just this week in a project. I still run into it and I just use it as a library now, but I did create it originally.
22:52 Michael Kennedy: It's probably a good feeling, though, to run across other people's projects and go, "Oh, there it is, yeah."
22:52 Martijn Faassen: It's a very good feeling. It's also very useful to say to potential, 'cause I'm a freelancer, so it's very useful to tell Python people that you created lxml because then, "Oh yeah, we use that."
22:52 Michael Kennedy: Most people know, yeah, yeah. "Oh, prove to me that you actually know what you're doing." "Yeah, that library you're using?"
22:52 Martijn Faassen: Yeah.
22:52 Michael Kennedy: "I wrote that already, so."
22:52 Martijn Faassen: Yeah, yeah.
22:52 Michael Kennedy: Oh, all right then.
22:52 Martijn Faassen: That's pretty neat. Plus, yeah, I've been drawing interest on the, I mean, usually we have an open source project, you don't really draw interest on it. This is an exception, but yeah, there's been features to lxml, like HTML parsing and all that stuff that I've used in the past that were just pure interest payments on my initial investment.
22:52 Michael Kennedy: Oh, absolutely, cool. You also did something called Fantastic and BowerStatic. What are those two?
22:52 Martijn Faassen: Yeah, Fanstatic actually. It's very hard to pronounce.
22:52 Michael Kennedy: Oh, Fanstatic, yeah, yeah. I gotta read more carefully.
22:52 Michael Kennedy: Okay, wow, very nice.
22:52 Michael Kennedy: Bower is one of those, right?
22:52 Michael Kennedy: Those are both really nice touches. All right, so I think I'll have to leave it there for Morepath. We've definitely got into a lot of detail, but I know there's a lot more we can touch on. All right, Martijn, before I let you out of here, how about your favorite editor? When you write some Python code, what do you open up?
22:52 Martijn Faassen: I use Emacs. I sometimes try out other editors, and then I go back to Emacs. I realize now what it is because I'm not really an Emacs power user. At least, that's what I say, but I've been using Emacs so long that I--
22:52 Michael Kennedy: You probably are.
22:52 Martijn Faassen: I, you know, learn more and more features over the years, but nonetheless, I willingly call myself an Emacs power user, but you know, if you use it for 10 years, you learn a few features every year you do know a bunch of features in the end. But the thing I really like about Emacs is that it just knows how my Python code is supposed to be indented, and if I press Tab it tends to do the right thing. Also, in other languages it indents stuff, and I'm so used to that that I tried Atom a while ago and it just didn't do indentation right for my tastes. I could press Tab, I could install an extension and edit. It would let me press Tab and edit, it would just indent stuff wrong. And then I went back to Emacs again. So yeah, we'll see.
22:52 Michael Kennedy: All right, cool. There's increasingly more options around the editors that are good, I think. That's cool.
22:52 Martijn Faassen: Yes, yes.
22:52 Michael Kennedy: And favorite PyPI package? We've got 96,000, almost 97,000.
22:52 Martijn Faassen: I have trouble deciding. I mean, I like lxml, of course, because I, yeah, I mentioned already I'm drawing interest on that one, so maybe I'll just stick to that.
01:00:02 Michael Kennedy: All right.
01:00:02 Martijn Faassen: I use so many, and there are so many very nice ones, I just can't really think of any one in particular that stands out.
01:00:10 Michael Kennedy: Sure, it's like, which of one your children do you love the most, huh?
01:00:13 Martijn Faassen: Well yeah, it's not even my own children. It's other people's children, who do I love the most?
01:00:18 Michael Kennedy: That's true, that's true.
01:00:19 Martijn Faassen: No, there are a lot of nice packages there, and let's see what I, I mean, I used click recently, I like that.
01:00:24 Michael Kennedy: Yeah, that is nice.
01:00:25 Martijn Faassen: It's a nice little library. What else have I used recently? You remember how I like, I've been messed around with Numeric and pandas recently, but I was playing around with developing a data-driven game engine because you know, why use Numeric and panda for what they're really for? So I was trying to blast data into OpenGL with pyglet. pyglet was also a very nice library. See, there I remember packages now. So, I blast OpenGL data into pyglet coming out of, oh, sorry, NumPy I mean, of course, not Numeric. So out of NumPy arrays, blast them into OpenGL to try to reduce overhead.
01:01:06 Michael Kennedy: Okay, that's a pretty interesting use case.
01:01:08 Martijn Faassen: Yeah.
01:01:09 Michael Kennedy: All right, awesome. Well, very interesting answers. Okay, so you got a final call for action. How do people get started with Morepath? What can they do? What do you need? Do you need contributors? What's on your radar that you'd love to have?
01:01:22 Martijn Faassen: Well, contributors are always great. We have a few core contributors now, which I'm very happy about. And I notice that if I do one major outreach thing like when I spoke at EuroPython in 2014 for like hundreds of people, one guy started using Morepath and he became a core contributor. So that's just a few core contributors, already such a wealth in an open source project. So those things I like very much. We have a documentation website, morepath.readthedocs.io. There's also a link to a Discord chat, so it just opens in your web browser and you can talk to Morepath developers, and a lot of people use Morepath. Create an issue on the issue tracker. That's also a way to reach us. And yeah, I started thinking a little bit about, I was looking at Django a bit recently, so I start doing a little bit about trying to create something like the Django admin UI, but then for Morepath using client-side technology somehow and perhaps using, I've never used it before, but I looked at the PonyORM and that looks like a nice ORM, so maybe I'll try that one day.
01:02:32 Michael Kennedy: Yeah, PonyORM is really nice. I had those guys on the show about 10 weeks ago and it's really cool.
01:02:38 Martijn Faassen: Yes.
01:02:39 Michael Kennedy: That's excellent. Everyone get out there, check out Morepath. And Martijn, thank you for being on the show. It was great to talk with you.
01:02:45 Martijn Faassen: Thank you very much. I thank you for letting me talk.