#125: Django REST framework and a new API star is born Transcript

00:00 Michael Kennedy: APIs were once the new and enabling thing in technology. Today they're table steaks. And getting them right is important. Today we'll talk about one of the most popular and mature frameworks in Django REST framework. You'll meet the creator, Tom Christie and talk about the framework, API design, and even his successful take on funding open source projects. But Tom is not done here. He's also creating the next generation API framework that fully embraces Python 3's features called API Star. It's all about frameworks on Talk Python to Me, episode 125 recorded June 8, 2017. Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem and the personalities. This is your host Michael Kennedy. Follow me on Twitter where I'm @mkennedy. Keep up with the show and listen to past episodes at talkpython.fm, and follow the show on Twitter via @talkpython. This episode is brought to you by Linode and Rollbar. That's right, welcome to Linode who has joined Talk Python to Me as a major sponsor. Be sure to check out what both of them are offering during their segments. It really helps support the show. Tom, welcome to Talk Python.

01:32 Tom Christie: Hi!

01:33 Michael Kennedy: Hey, it's great to see you again and great to have you on the show finally. I've been admiring Django REST framework for a long time and really a fan of your new work that we're going to get into, so I'm happy to have you here to talk about it.

01:44 Tom Christie: Cheers, Mike good to see you again.

01:45 Michael Kennedy: Yeah, yeah.

01:45 Tom Christie: How you doing, anyway?

01:46 Michael Kennedy: I'm doing super well, super well. So I want to hear like how you got into programming and Python and all that, but how was PyCon? We were both there the last time I saw you. Even though right now we're like 6,000 miles apart we were in the same room just a little while ago. How'd you enjoy it?

02:02 Tom Christie: It was hectic. I don't know...

02:07 Michael Kennedy: You had the booth right?

02:08 Tom Christie: Yeah, we had a booth. So I haven't done PyCon before mostly because with it being over in the States in the past it's been hard for me to justify taking the time and the cost of the transatlantic flight and so on. Since I've been, you know, I guess we'll talk about this later, but since I've doing open source for a living, all of a sudden, "Okay I really better step up and get across there." And we, I kind of realized slightly later on in the day, "Oh wait a minute, they have this thing where they offer booths to open source projects. We better do that. That sounds like a good idea." And, yeah, just about managed to organize getting one of those with a lot of help from Anna who I also work with. And I didn't know what to expect, but it was just non-stop, basically.

03:01 Michael Kennedy: That's really good, right? People are dropping in to talk to you about your project.

03:04 Tom Christie: Yeah and there's so much kind of positive reinforcement coming from people at the stage where it was just non-stop talking for about three days together with the poster session as well. So I have no idea how many, it felt like hundreds, but ...

03:21 Michael Kennedy: Yeah, that's really cool. We had a booth as well for all the podcasters that we had gotten like grouped together and had the five podcasts all put together in the one booth. And it was a real similar experience. Like the positive vibes and the feedback was amazing. Some of the other guys who hadn't been there before, they were like, "Wow, this kind of makes it so real with all these people." Did you have that similar experience like, you know there's a number of stars on GitHub and you get a message and stuff. But like when hundreds of people come up to you and talk to how you've like enabled something, did that strike you pretty strongly?

03:53 Tom Christie: I kind of had this odd, this odd mixed feelings about, not mixed feelings exactly, but just this strange sensation because being at the booth and you've got, like you say, all these people coming up. You know saying thanks for the work that you do on the projects and all of these kind of good vibes, but then at the same time I'm standing there at the booth and I'm looking over at all the big, real professional companies that, yeah, that know what they're doing and thinking, "Hmm, that's where I'd really like us to get to," you know, so? - Yeah, yeah, sure.

04:27 Michael Kennedy: Yeah, yeah, sure.

04:28 Tom Christie: That was interesting. But what was also nice as well was having having a good excuse to only talk to people at the conference. The only time that I managed to get upstairs into the talk spaces was for the talk that I gave. And the rest of all of the talk session time I was just talking to different people and I really enjoyed that. I got a lot out of that.

04:53 Michael Kennedy: Yeah, I felt the same way. I only went to the keynotes and the open spaces and other than that I really just was at the booth talking to people. It was really, really amazing and you know, thanks to everyone who stopped by my booth and I'm sure you feel the same way. It was quite cool. By the end of the second day I had basically lost my voice from talking.

05:12 Tom Christie: Yeah. I should also say that we got a bunch of help from other volunteers who offered to step up and give us some of their time on the booth, which was awesome as well.

05:24 Michael Kennedy: Yeah, that's cool.

05:25 Tom Christie: Thank you to them.

05:25 Michael Kennedy: All right. So yeah, one positive experience from PyCon. And people who are out their listening, be sure to get your PyCon tickets early, because as always it sells out and then I get messages like, "Hey can you get me a ticket it's sold out?" Like nope, no secret tickets. All right, so. Let's get into your story first before we talk about your RESTful things.

05:47 Tom Christie: Okay.

05:47 Michael Kennedy: How did you get into programming in Python?

05:48 Tom Christie: Oye, okay, well let's start with how'd I get into programming I suppose. ZX Spectrum 128. When I was a little kid I did a little bit of programming from magazines and stuff, you know when it was in the days when they came with a tape on the cover. Back then. And my dad bought me a ZX Spectrum. And I did a little bit of noodling around then, just silly little games, stuff. Well, not even that really. At that point that was just copying stuff out of magazines or trying to figure out little bits and pieces. And then I kind of dropped it for awhile. And later on when I was a teenager I started getting into QBasic and programming games together with a friend. And we had like a little thing going where we did that. And that got me into it enough to decide to do it for a university course. And then I've had a big old windy road since then. So, I started out working in speech recognition, stayed in that field for several years. I then moved into networking and worked with a content distribution network who did a peer to peer content distribution network.

07:12 Michael Kennedy: Peer to peer, huh? That's pretty cool.

07:13 Tom Christie: Yeah. Yeah well that, what they started out with initially was providing BitTorrent caches to ISPs, so the ISPs wouldn't have to pay for the upstream costs to the Internet and could instead just cache the really popular streams.

07:32 Michael Kennedy: Yeah, yeah. Do you feel like a

07:34 Tom Christie: Anyway...

07:36 Michael Kennedy: BitTorrent was ruined by people using it too much for piracy? Like BitTorrent I think would have been a really fantastic for all sorts of web content delivery, video streaming, so many things, but you know, since it got so associated with pirating MP3s and videos it's kind of a black mark.

07:59 Tom Christie: I don't know I think if it had been the right solution technically then it would have won out on its own merits anyway. You know the legal side of it I don't think was really the main influencer of that necessarily.

08:15 Michael Kennedy: Right, okay.

08:17 Tom Christie: Well and really I guess the reason why I say that is because I was there busy working at a company which was using BitTorrent and was trying to provide a here's how you do a CDN using . I don't know, It's too long ago for me to give you a coherent answer. I'm going to just stop now.

08:41 Michael Kennedy: All right, that's good. All right, so you did that for awhile. Was that in Python or something else?

08:45 Tom Christie: So that, see I can't remember at what point things stepped into Python. It was either when I was working in speech recognition and at some point all of the, the sorts of the build tools that you need for doing things like building the language models, right? The sorts of things that we were doing when I started there was writing shell scripts and perl scripts. And I can't quite remember whether I moved into Python then 'cause it's been long enough now, or whether it was when I then later started doing networking. But whenever it was all of a sudden it was like, "Oh wow! This is so, you know..."

09:30 Michael Kennedy: This is different.

09:31 Tom Christie: Having been used to working in C and C++ and either Perl or Bash for scripting, it was, yeah, such a breath of fresh air. Well, it's quite surprising almost to be here after all this time and still be so infused by the language and not feel like that's really changed.

09:56 Michael Kennedy: Yeah, that's really an interesting point. I've kind of marveled at that myself as well. Even if you just look at like some of the growth curves, popularity curves of Python and the ecosystem it's been around for over 25 years now, but it's still growing and growing more quickly now than it did 10 years ago or when it was brand new. It's really interesting.

10:19 Tom Christie: Right and we're just coming up to potentially a new kind of slice of stuff that it can break into as well. You know now that Python's readoption is finally really starting to hit that tipping point and the Asyncio primitives that are in there, all of a sudden there's this new slice of stuff that Python could be really, really good at that it historically hasn't particularly been.

10:47 Michael Kennedy: Yeah, it's definitely an exciting time, yeah for sure. All right so thanks for the story. What do you do day to day? You said you're a professional open source developer. How awesome is that?

10:59 Tom Christie: Yes, that's quite a rare, a rare thing isn't it really?

11:03 Michael Kennedy: I love to share these stories, yeah.

11:07 Tom Christie: You know been really lucky with the way things have worked out just with timing and so on. So yes, I as of about 12 months ago I now work on open source for my living full time. So let's approach it this way. So it started on REST framework about five years ago and all of that was as a side project as it is with most people. When we came up to the point where I was thinking about all the things we'd need to do for REST framework 2, I figured that I was going to need some full time on it. And in order to do that we launched the Kickstarter project for it. And that was really successful. And I managed to get...

11:54 Michael Kennedy: What year was that, do you remember?

11:55 Tom Christie: That must have been either 2014 or 2015. 2015 about...

11:59 Michael Kennedy: Okay, so pretty recently, yeah.

12:01 Tom Christie: Yeah, not so long ago. It probably, it was probably started in 2014 and I was probably working on it up until 2015.

12:09 Michael Kennedy: How'd the Kickstarter go by the way?

12:11 Tom Christie: Oh, it was awesome. We got about 35,000 pounds, which I think was about $50,000. The way that I organized it I was with a very small web consultancy who are very, very into supporting open source, and the person who's the CTO of the company runs the local Python meet up group and so on. And we figured out a way that I would be able to take a slice of time out of my working day in exchange for us doing the Kickstarter through the company. And I was able to work on the project for I don't know maybe about nine months or so as a result of the Kickstarter, with most of my time being dedicated to, or about 50% time. So that was great and was much more successful than my highest estimate would have been for it.

13:08 Michael Kennedy: What goal were you asking for?

13:09 Tom Christie: I guess our starting point was probably something like 5,000 pounds and maybe, maybe I was hoping for 10,000 and way passed all of that.

13:20 Michael Kennedy: Cool, congrats.

13:20 Tom Christie: Yeah it was great, yeah. And then more recently when I was thinking about the next chunk of functionality that I wanted to deliver, I was trying to figure out how to do it and launching another Kickstarter was one of the options, but trying to find a way that we could do it that would remain sustainable rather than just being a big one off thing.

13:46 Michael Kennedy: Yeah, Kickstarters are nice, but they're very much one off, like here's a boost, now what?

13:51 Tom Christie: Yeah, yeah, yeah.

13:51 Michael Kennedy: You can't just keep doing Kickstarters.

13:54 Tom Christie: Yeah, and it has to be, you have to have a really nicely wrapped up bit of functionality. It's difficult to kind of sell it as triage on the project or open ended bits of work. And it's a lot of work as well. You can't hit go until you're completely confident that you've really nailed the pitch that you want to make, which can make it quite difficult because even if you've got something you think, "We really need to do a Kickstarter for this," until you feel that you're at the point that, "Okay, I've got it, we've got it just right, let's do this," you keep on wanting to hold off, hold off and try and, anyway. But this time round, I started thinking about launching an ongoing funding model instead. We were really fortunate because it tied in at the same time with when Mozilla launched their MOS Awards, so the Mozilla Open Source Awards where they give awards to projects that they're using and also help them develop further functionality.

15:07 Michael Kennedy: And that comes with some sort of funding right?

15:09 Tom Christie: It does, it does indeed. So we put forward an application for the project and when we were told that we'd been successful, and I can't remember exactly how much it was right now. It was probably about $35,000 I think that we requested. When we found out that we'd been successful that was the point at which I said, "Okay, well I better give this a shot now." I told my company that I was going to go and try working on open source full time and we launched the collaborative funding model is what I tend to call that.

15:48 Michael Kennedy: That's cool, how long ago was that? How long have you been doing it in this path?

15:52 Tom Christie: So yeah, just over a year. We literally just passed over the year milestone now. Yeah, so...

15:59 Michael Kennedy: Are you enjoying it? Has it been a fun year?

16:02 Tom Christie: It's great. It's not, you know, being entirely self driven and being in such an atypical business model and quite amorphous set of constraints, you know, that can be odd at times. But I try to spend most of my working time out of co-working space rather than just working from home, so that helps give things structure. And it's been amazing how much you can push things forwards when businesses actually have that relationship of trust to invest a little bit of their money in you in order to give you that space to do that, so yeah.

16:51 Michael Kennedy: Yeah, I feel like there should be way more of this. There are businesses, there are billion dollar businesses, whose primary product is built on layer upon layer of these super important open source things and they contribute nothing to it.

17:04 Tom Christie: Right.

17:05 Michael Kennedy: And at best they'll like do a PR back to it, but even maybe not even that, right? They would probably get in that benefit financially just by supporting the people to tweak it just a little bit to suit their needs better.

17:21 Tom Christie: Right, so yeah and the responsibility of people like myself who are fortunate enough to have managed to find themselves in this position is to prove the business model. So there's two sides to it. One is the marketing angle of it, which is important, but in a way it's something that I'm less interested in. The thing that I really want to demonstrate is when you take a collaborative approach to investing in infrastructure, if it's shared infrastructure then the rewards that you get for your investment are so out sized to the rewards that you can get by solely investing only within your own company, because you've got all of these other companies that are also pitching in at the same time. So the way that I tend to talk to potential sponsors is I say, "Hey I want to work for your business full time, for $50 a month, or $100 a month, or $150 a month."

18:25 Michael Kennedy: And they're like amazing, how do we do this?

18:27 Tom Christie: Whatever figure I tell them, you know, how is that not a reasonable business proposition? Okay, I'm not working on exactly day to day what they tell me, but I am working on trying to support their businesses, you know. It's not...

18:42 Michael Kennedy: Yeah, absolutely, absolutely.

18:43 Tom Christie: Pie in the sky stuff.

18:45 Michael Kennedy: I'm not coming to your meetings, but I'm going to be working on your business.

18:49 Tom Christie: Yeah.

18:51 Michael Kennedy: Yeah I think it's a super, super straight forward story. And I think you're right. There's people sort of blazing that trail and how well it works in the future is going to depend on how well I think the first few successful ones blaze that trail, right?

19:09 Tom Christie: Yeah, yeah. We need to build stuff that makes those companies more successful, and we also need to sing and dance about the companies that are supporting open source financially. And do things like try to get the community into a mindset of when you're looking for your next job, look to companies that are supporting open source financially as your first port of call and try to drive things that way around as well.

19:39 Michael Kennedy: Yeah, I feel like it's something of an awareness thing as well. It's a pretty straight forward sell to go to pick a random company, Fidelity Investments or whatever and say, Look if you guys paid like $200 or $1,000 a month to these five open source projects to keep them going, and everyone did this, like you would have a much more stable platform, your scalability would be better, and you would be able to you know trade half a microsecond, milisecond faster, whatever right? That's a really simple story to tell to the business owners, but it's, I don't know, it's, I'm not sure how much it's happening yet.

20:17 Tom Christie: It's step by step, right? We need to be able to have really great examples where we can point to whether it's frameworks or platforms or projects that have really succeeded because they have been financially successful and say, "Look, this is what can happen when you do support these projects."

20:36 Michael Kennedy: This is what the people who supported it and anyone who's using it actually got out of this experience. And we can do it over here, and we can do it here, and we can do it here.

20:43 Tom Christie: And also to do things like look retrospectively and say, "How much did that cost each business?" When you go and you look at all these bits of, well anyway.

20:52 Michael Kennedy: Yeah no, I think it's I think it's really great. We should probably talk about your projects in particular as well.

20:57 Tom Christie: Yeah, I know we've been...

20:58 Michael Kennedy: I really... But I think that this is super interesting to people. It's very powerful and for everyone who's doing it there's probably 1,000 people who wish they could. And so like just the awareness of us talking about it I think serves a good purpose.

21:14 Tom Christie: I think it's going to change. I'm really optimistic. I see things have been moving forwards, and I feel like we're on the start of a wave. I mean that's easy for me to say because I'm fortunate enough to actually be in that position, but I do feel like it's starting to change.

21:33 Michael Kennedy: Sure I definitely think it's something that 15 years ago would have been really, really unexpected, and now it seems pretty normal. It's just something that's got to get moving.

21:43 Tom Christie: Yeah.

21:45 Michael Kennedy: This portion of Talk Python to Me is brought to you buy Linode. Are you looking for bulletproof hosting that is fast, simple and incredibly affordable? Look past that bookstore and check out Linode at Talkpython.fm/linode. That's l-i-n-o-d-e. Plans start at just $5 a month for a dedicated server with a gig of RAM. They have 10 datacenters across the globe, so no matter where you are there's a data center near you. Whether you want to run your Python web app, host a private Git server, or even a file server, you'll get native SSDs on all the machines, a 40 gigabit network, 24/7 friendly support, even on holidays, and a seven day money back guarantee. Want a dedicated server for free for the next four months? Use the coupon code python17 at talkpython.fm/linode. You've got a couple of projects that are amazing. So let's take them I guess from the beginning. So the first one which we've mostly been talking about is Django REST framework. This is the thing you started back in 2010 or so?

22:47 Tom Christie: Yeah.

22:48 Michael Kennedy: Yeah all right, so. You know everyone knows what Django is. They probably know what REST is. What's Django REST framework? What is this project?

22:57 Tom Christie: Django REST framework is unsurprisingly an API framework for Django. And it tries to fill in all of the gaps of where Django was historically designed as a web framework, what are all the extra bits and...

23:16 Michael Kennedy: Right, here's HTML going to browsers. You wanted to make it talk to machines not people, right?

23:21 Tom Christie: What are all the various bits and pieces that you would want to provide additionally if you wanted to use Django primarily as an API framework? So for instance, approaching authentication differently providing things like throttling and permissioning in a way that it's really easy to use the in different slices of your API in different ways. Handling things like HTTP content negotiation and all sorts of bits and pieces, providing generic classes of views that give you default functionality for the CRUD case. So if you've got an API where you want to provide a set of different URLs and methods on those URLs that are used for creating records, retrieving records, updating and deleting records giving you all of those sorts of bits of tools at the bottom.

24:20 Michael Kennedy: So maybe you have like /API/users and then you can do a get to get all the users at get /id to get a particular user, a post, to create a user, things like that?

24:30 Tom Christie: Yes. And also dealing with the if you've got some data that you want to present to the user, or rather if you've got some objects that you want to present to the user, what do you want the representation of those objects to look like? If we're dealing with relationships between the objects are you presenting those as the plain primary keys or should they be presented as hyperlinks? All of those sorts of that. Like do you want to return big nested sets of data where you've got all sorts of interlinks objects all being returned in bulk, or do you want flat set data structures? All of those sorts of things.

25:12 Michael Kennedy: Okay cool, so if I'm working with Django ORM or even SQLAlchemy or MongoDB, something like this, and I'm getting sort of ORM objects back does it handle like the serialization of those types of things?

25:25 Tom Christie: So it works, being Django based, it works primarily with the Django ORM. There's stuff that you could do to integrate it with other backends as well, but...

25:38 Michael Kennedy: The question is like suppose I'm in my web app that I do a query and I get like a list of users back from Django ORM, can I just return that list and have it magically JSONified, or XMLified, or whatever?

25:50 Tom Christie: Yes and no. So yes, you can, no, you probably don't want to.

25:54 Michael Kennedy: I shouldn't do that. You can, but don't do it.

25:56 Tom Christie: Exactly. So, you can ask it to give you the default representation and just say, "Return all of the fields on this model." So, give it a model serializer, tell it which model it's pointed at, and say, "And I just want all of the fields on that," and that will give you a default representation that depending on which class you use just uses either primary keys or hyperlinks for relationships that that model happens to have. What you would want to do more typically is be a little bit more precise about the representations, so declare the fields that you want returned explicitly and independently of the model itself. So, list them all out, so yeah.

26:44 Michael Kennedy: Yeah, and going the other way maybe is even more important. Like if I post a JSON body to a method, is there serialization on that side, or do I like read it?

26:53 Tom Christie: There are two different aspects to this. So, there's the taking the ORM object and turning that into plain Python data types that are ready to be serialized that are ready to be rendered, that we call the serialization going between the plain Python data types and ORM objects. Then there's the parsing or rendering, which is on the parsing side taking request data that's coming in and turning that into Python data types, so json.loads, and on the other side the rendering, which is taking the data types and turning those into the response body, so json.dump typically.

27:40 Michael Kennedy: Yeah, so it looks like you have a lot of control over all those different things.

27:43 Tom Christie: Yeah, so you can for instance you control the serialization style which is what information do you want in the representation, but separately to that you can also control the renderings of that. So, you could for instance have both a JSON and a YAML output for the same database model.

28:04 Michael Kennedy: Sure. Did I notice somewhere in your docs that you can basically use like a fake or a false extension to control that? So, like /user/1.json versus /user/1.xml and things like that?

28:21 Tom Christie: Yeah, so there are different ways that you can control which rendering you get. By default, REST framework will use content negotiation where the client sets an accept header in the request, which informs the server which response type it would like, which content type it would like in the response, or you can have URLs that specifically are for one particular type of rendering using suffix, so yes, .json or .yaml.

29:00 Michael Kennedy: Yeah, yeah. So, the most popular API that I know that does that is the Basecamp API. They do like thing.json. Even though it's entirely not needed, that's just like their style, right? And some people like that, some people don't.

29:13 Tom Christie: Yeah, but the thing that actually drove all of that side of the rendering in the first place anyway was the Browsable API. So, when you go to a URL in REST framework, if you make that request with curl or a command line client, you will typically get back depending on exactly what you configured, but you'll typically get back plain JSON. When you go to it in the browser and the browser sends an accept header that says, "I would prefer HTML," then we're able to return an HTML rendering of the view with the data in line on the page, but also a whole bunch of extra information around the outside and controls to allow the user to interact with that API endpoint.

30:01 Michael Kennedy: That's awesome. So, it's like a self documenting API out of the box?

30:06 Tom Christie: Yes, exactly.

30:07 Michael Kennedy: That's really cool. Yeah, that's one of the things that I think often gets forgotten on APIs or gets out of date, right? Like either there's no ... Basically, almost no API. Like, "Here's an example," and it's incomplete or whatever, or, "Here's the documentation, but our API has evolved and it's no longer..."

30:24 Tom Christie: Yeah, well I'm still surprised that there aren't more frameworks that are starting to provide this. The Browsable API stuff, that was the entire reason that REST framework came into existence in the first place because I was thinking about, Why are APIs, why can't we interact with them using the web browser? This is a bit weird. And that was the entire driver for, Okay, well how would you do that? Oh, you'd do it like this. Oh, wait a minute, this is a framework. Oh, if I think this is a good idea then I better make it a good framework.

31:06 Michael Kennedy: My prototype is almost a thing. Yeah, that's awesome.

31:09 Tom Christie: Yeah, yeah, so.

31:11 Michael Kennedy: Yeah, I feel like that's always been a weak point, or traditionally it's been a weak point of HTTP frameworks or APIs is that they're really easy to interact with, but it's hard to see the documentation unless somebody has taken the time to do that separate. So, can you give us just like a quick sense, not too much in code, but just like the conceptual steps of like how would I create a project and like set it up and add an API method using Django REST framework?

31:40 Tom Christie: It slightly depends on whether you are building a project that is primarily a CRUD style API or not, because REST framework gives you an awful lot of tools for building those sorts of APIs with a minimal amount of code if you are, and that's the most common use case really.

32:01 Michael Kennedy: Do you point that out like your ORM objects and say, "Here's an ORM object. Give it various operations."

32:07 Tom Christie: The way round that I would approach a project with REST framework I would start by taking REST framework entirely out of the picture and not think about what tool is this API being built with anyway, and instead think about the basic, the fundamentals of the API. So, which endpoints do I want to expose? What do I want the representations of those endpoints to look like? How do I want relationships between different objects to look? And try to get that side of the design figured out in my head first independently. Then, you've also got the data models, as well, and thinking about the storage side of things, which is very closely tied to the representation but isn't necessarily the same thing exactly. So, once you've got the start of those things in your head, then it's a case of blocking out the Django models, blocking out corresponding serializers for those models, and I typically... So, REST framework gives you a lot of different levels of tooling that you can work at. If you want, you can work at a very high level of abstraction where we provide things called ViewSets that bundle up a whole lot of different interactions into a single class and provide, mostly automatically provide the rooting of URLs onto all of those points of interaction. And I tend to not work at that level of abstraction. I would usually go down and work at a the level of per URL write either a function based or a class based view for that URL and write out the methods of each of those individually. And again, there's still another different couple of layers of abstraction that you can be working at there, because you can either be using the built in functionality of give me the automatic create an object function, or you can go and write that code yourself. When I'm building an API with REST framework I tend to prefer the explicit approach. So, if it happens that a listening view or an update view or a delete view is exactly already provided by the framework in the right way then I'll use that, but if it's not, rather than trying to overwrite it at particular hook points within those class based views, I'll just overwrite the get, or the post, or the delete method entirely and write that out explicitly, because that really keeps everything surfaced up to the top and I find makes it is quite a maintanable approach to building the API.

35:17 Michael Kennedy: That's cool, I really like that you approach it from the outside, like how should this look to the people consuming this API, forget the implementation, and then let's make that happen with Django REST framework, right? Because really in the end that's what's going to determine whether people are happy with the API you've delivered them, right?

35:37 Tom Christie: Yes, yeah, absolutely, yeah.

35:38 Michael Kennedy: It's not like, "Well, you do it this way," because it was easier. It was automatic in Django REST framework, but it should have been this other way, right?

35:46 Tom Christie: I mean, it started off maybe when the API frameworks and the landscape were younger. Maybe it was a bit different then, because there's also the aspect of it's useful if the frameworks guide you towards what set of design choices should you be making there.

36:06 Michael Kennedy: Yeah, but in 2017 we have more experience building APIs, more examples of good APIs than long ago. So, you want to tell us about some of the popular deployments, like who is using it, how they are?

36:21 Tom Christie: Gosh yeah, I should probably open up things like the sponsor layer to remind myself there. So, let's do it. No, I'm not going to do that. I'm just going to ... So Mozilla. Mozilla are obviously a really big name. Eventbrite were a big sponsor of the Kickstarter. I can't remember how much they're using it at the moment, but I think that they are. I know that Red Hat are using it at least within a couple of places. Heroku uses it for something or other. And well, we've got about 60 corporate sponsors, so stacks and stacks of different consultancies and product businesses of all types really. It's always interesting when you get a new sign up finding out what the business is, because there's so many different surprising angles that it's being used from.

37:19 Michael Kennedy: Yeah, can you think of one you're like, "Whoa, you're doing what with my framework?"

37:26 Tom Christie: Not off the top of my head, but that's probably like post PyCon haze, because they all went in, and then yeah, all at once if I may.

37:39 Michael Kennedy: Yeah, excellent. Alright, so it sounds like a really successful project. This is the one that basically let you through the Kickstarter and the collaborative funding and these things lets you become a professional open source developer. You basically created a business around your framework, and congratulations to you on that. That is super, super cool. I think it's so neat. I'm happy to be able to shine a light on it.

38:02 Tom Christie: Thanks, yeah it is, it's wonderful.

38:04 Michael Kennedy: Yeah. So, there's another thing that I came across that you just started working on as well that's very exciting. I don't know how it ties into the funding stuff, but I certainly think it's cool, is called API Star. Tell people what API Star is.

38:20 Tom Christie: This is a new API framework that I started working on. REST framework is still my my main gig I guess, but there are a couple of different things that I think are important to explore at the moment, and API Star is my step towards doing some of that. It's an API framework that's not tied into Django or the Django ORM and has a different approach to the fundamental interface that you use for building your code base. So, rather than having views, which are functions that take a request and return a response, instead you have views where you declare on the view function using type annotations, the sorts of data and the sorts of components that that function is going to take. And you also declare what type of data it's going to return, as well. And once you take that different approach there's a lot of interesting things that you can start to do on top of that. So to start with because you've got a plain data interface, rather than take a request and return a response, you can layer on different types of network interface or different ways of interacting with that function on top of it. So for example, as well as accepting HTTP requests, you could root a command line command into your view, or you can potentially look at being able to build views that accept either HTTP requests or web socket connections, and that provide a real time subscription endpoint. There are ways you can start to think about doing that, or even start to think about can we build views that we can share, that we can kind of share functionality being exposed to GraphQL endpoints and regular HTTP calls. Now, not all of those things are done, but I'm starting to think about how would we be able to do those. As well as having this different style of interface, something else it provides is a type system that corresponds to the same capabilities as JSON's schema. So, we have a type system that you can use to declare what your data structures look like both for the input and for the output for these functions. And once you've got that you've suddenly got something really powerful where you're able to for instance build API documentation based on those type annotations that is guaranteed to always be in sync with your code base. Not only can you build this API documentation, you can also build interactive API documentation because you've got enough information in those types that you can build HTML form controls with appropriate controls given the shape of the data input. You have this proxy all of a sudden that you have a really nice one to one mapping between your function declarations and an API schema language such as Swagger or RAML. So, you can have a really clear way of generating Swagger or RAML from your type declarations. If you wanted you could even go the other way and say, "Okay, I've got my Swagger blueprint. Build me this stubbed out set of functions that I'm going to want for my project." You've even got enough information that once you write out these declarations even before you implemented anything you could start running, or you will be able to start running, a mock API. So, you've got all the response types. You know what the inputs look like and whether they're valid or they're not valid. So, what you can do is have your team build up all the function declarations and present to your front end team a mock API that's already ready to start then build the front end again, while your back end team starts to fill in all the blanks of, "Oh, well we better actually implement the behavior here." So, there's all these...

43:06 Michael Kennedy: That's really cool.

43:07 Tom Christie: Really interesting things you can do. Oh, and also, what else? Dynamic client libraries that are able to interact with your functions. They have enough information being exposed in the schemas that you're able to generate that you can present an interface on the client side that corresponds to the same on the serve side. So, all sorts of cool stuff once you've got into it.

43:32 Michael Kennedy: Yeah, that's actually been one of the weak points of HTTP APIs over say SOAP services, right? SOAP services are dreadful, but one of the things that they were good at is they had this WSDL thing that you could then generate basically clients that know all the methods, all the types that are exchanged there. I'm thinking of Suds in Python, or obviously like Java or .NET in their tooling. But here you're kind of describing enough information, this is the stuff that goes in, this is the stuff that goes out, that you could actually bring that back in a sense.

44:03 Tom Christie: Yes, what this is it's a way of ... It's still a RESTful approach. It's just a way of providing a better description to what the interface is, and that also has an impact on what source of APIs you build, actually, because given that we haven't had great client side tooling or great ways of describing what those interfaces are, the let's always build CRUD style interfaces is something that's naturally happened because it's been a bit of a constraint of we can't express very well different things that don't fit into that. But, having a really expressive way being able to declare what those interfaces are let's you take a more, "Okay, these are just different types of actions or different types of views of the data." They don't necessarily have to correspond to CRUD.

44:57 Michael Kennedy: Yeah, yeah. This portion of Talk Python to Me has been brought to you by Rollbar. One of the frustrating things about being a developer is dealing with errors, ugh, relying on users to report errors, digging through log files trying to debug issues, or getting millions of alerts just flooding your inbox and ruining your day. With Rollbar's full stack error monitoring, you get the context, insight, and control you need to find and fix bugs faster. Adding Rollbar to your Python app is as easy as pip install rollbar. You can start tracking production errors and appointments in eight minutes or less. Are you considering self hosting tools for security or compliance reasons? Then you should really check out Rollbar's compliant SaaS option. Get advanced security features and meet compliance without the hassle of self hosting, including HIPAA, ISO 27001, Privacy Shield, and more. They'd love to give you a demo. Give Rollbar a try today. Go to talkpython.fm/rollbar and check them out. One of the things that struck me the most about looking at this API, is it Python 3 only?

46:00 Tom Christie: It is, yes. So, because of the type annotation.

46:04 Michael Kennedy: Right, yeah. So, the type annotations are the things that struck me the most when I look at it. So, you might have an API method and the method say if you want to have access to the request object you would say request: then http.request. And you basically annotate like this variable is this type. And your framework will go, "Oh, I have one of those types. I'm going to inject it here." There's all these different types of things that can be injected, right? Yes, so the different kinds of components that you can inject they can either be say something that's based on the incoming request, like say request, or you could say... You could have query_params, so the class query_params. If you've got that on you'll get the dictionary of all the query parameters, but you can also do interesting things like as a class the singular, query_param, and if you put on the name that you use for that variable and the class query_param together it will say, "Okay, I'll give you this query parameter." That is so awesome, because like all the time I've just... I'm really excited to see this thing come into existence. So, all the time if you're taking a traditional HTML based framework, Django, Pyramid, Flask, whatever, they're like, "Okay, here's the post dictionary," or, "Here's the data that was submitted," or, "Here's the query string." And you can go dig into that dictionary and find the thing you want. Here are the headers. Dig into the headers and find the thing you want. But on yours you could actually say things like user_agent: http header, and it will go into the headers, find the user agent, pull it out and set that value, so you just... It's as if all of the elements of the request become just function parameters, which is glorious.

47:50 Tom Christie: Yeah, so it's really expressive. It reads really nicely when you look at it, right? It also means that it's really testable, because the interface that you're exposing is exactly the thing you want to be exposing. You don't have to do things like build up a mock request object in order to test your functionality.

48:08 Michael Kennedy: Right, because if you say it takes a query string and the thing is called name, you just pass in the value you want for name in your test and you're done, right? You don't have to fake up a dictionary that holds things, right?

48:19 Tom Christie: Yeah, yeah.

48:19 Michael Kennedy: Nice.

48:21 Tom Christie: And the other thing that's interesting as well is, because the way it works is when you root all of your views with the application, at the point they get rooted it inspects what the function signature is, figures out how to look at what all the different components are, and figures out exactly what's going to need to be run every time it needs to run that view, which ends up meaning that it's really, really fast, because you only pull in exactly the things that you need to pull in. So for instance if you've got a database component, only views which declare that they're using the database component will end up getting wrapped in a transaction, and anything that doesn't will never get seen at all.

49:05 Michael Kennedy: That's fantastic. Yeah, the other thing that I really like about the type annotations is automatic type conversion. So, you can have like user_id:int, and obviously everything comes into the web as a string, but do you like parse that into an int?

49:20 Tom Christie: Yes, that gets handled for you and the type validation on that. So, you can either use the native type for that, or you can use the type system for that, which gives you the more expressive everything that JSON schema gives you stuff that, you know, "Here's a complex object. Here's what it looks like."

49:40 Michael Kennedy: Yeah, that's awesome. Alright, another thing that I found to be interesting was the performance story. So, you already touched on it. This is a super lightweight framework. And I didn't even realize the extent to which you were ensuring that it only does what it needs to do. That's really cool. So, you ran some various tests with like Gunicorn and Meinheld, or even with like some of the shiny new things like Sanic and UV Loop, right? And so the stuff you had on the website, let's see, it says Flask. You talk about scalability and just raw individual request performance. So Flask on your setup you were getting 5,238 requests per second and about a 40 millisecond latency. On Sanic, which is really an interesting framework as well, you were getting like 21,000 requests per second and 10 milliseconds. And with API Star it's 25,008. So, you were like five times faster than flask on both raw individual requests and... That's awesome.

50:46 Tom Christie: Yeah, keep in mind that that's a test type that is not really real world usage, but some really big providers now. We'll talk about that stuff in a bit more detail later, but I interested you, so.

51:00 Michael Kennedy: Well, I was going to say you have to be really super careful about these sorts of performance comparisons, because the way I'm using it and the way someone else using it could be super different. The way I've set up my machine could be really different. But, you were running the TechEmpower Benchmark, which is a particular one which is pretty well respected. So at least in that sense you were running the simple JSON serialization version. Does that mean like if you use API Star your API is five times as fast? No, because you're still probably doing your same implementation stuff, right? But, it means how much overhead are you bringing in with a framework. Like that you could maybe consider, right?

51:35 Tom Christie: Yes, so that's what I was trying to demonstrate when I've included that section in the documentation is not so much this is going to make things faster all the time, but the aspect of the framework can get out of your way if you need it to. So, the being able to declare the response type on the view you can actually have it just return a WSGI response directly. You end up with the server calling into your framework, and a single function call gets made, and that gets returned directly back to the server. So, that's an interesting aspect of it. Now, that doesn't mean for the different test types where different things like asynchronicity come into play that it would perform in the same sort of way. It wouldn't.

52:28 Michael Kennedy: Yeah, of course.

52:29 Tom Christie: But, I think it's still really important to highlight that aspect of once you take this dynamic approach you can work at whatever level is appropriate for the thing that you're building.

52:41 Michael Kennedy: Yeah, very cool. Okay, so I'm sure I've got you a bunch of email from pointing that out. Oh no, let me tell you how you could've optimized this if this said ... Anyway, speaking of performance and asynchrony, there's another project called UVicorn. Tell us what this is. It's like the future version of Gunicorn, right?

53:08 Tom Christie: So, in Python Asyncio land at the moment there's a few frameworks that are existing, Sanic being one, aiohttp being another. There's a few of them

53:24 Michael Kennedy: Pronto. popping up.

53:25 Tom Christie: That's right, and Muffin.

53:29 Michael Kennedy: They're very exciting.

53:30 Tom Christie: kawaiisync, is that right, kawaiisync? Yeah, so that, which is really cool. And in particular Sanic has been interesting, because it's shown just how impressive these two particular components that it uses are, which is UV Loop, which is an event loop that is implemented as a Python binding around either the same event loop that Node uses or something that was originally built for Node. And also an HTTP parsing library that is also a Python implementation, a set of Python wrappers around a super super fast C implementation that again I think was built originally for Node. And the performance of those two components is really impressive. The only problem that we have in the Python landscape at the moment is that all of these Asyncio frameworks that we have are really still at the framework level. There isn't this separation of well, here's the server, here's the framework, and here's what the interface between those two is. We've had that for a long time in our regular old happy synchronous land because we've had WSGI for ages. And now you can't repurpose that for async land because you just can't. The thing that I would like to do is combine the fact that there happens to be an incredibly awesome implementation but that is super fast, and the how do we figure out what the interface between the server and the framework ought to look like, and bridge that gap and have, "Okay, here's just the bare bones. You've got a single callable and it gets this set of data. Now go and build your Asyncio framework on top of that."

55:32 Michael Kennedy: Yeah, yeah, yeah. That's really been the pinch point where we haven't been able to unlock the async stuff on the server easily with existing frameworks.

55:41 Tom Christie: Yes, and I think once we've got that that's really going to set everybody free to start doing interesting things. Where it's difficult is trying to figure out how it fits into the existing landscape. So, for instance one of the things that I would like UVicorn to be able to do is to also be able to interface with Django Projects using channels and be able to provide web socket support but in a different deployment mode to channel tags. So, deployment mode there at the moment is you have your servers, and those sit in front of a channel layer, which all the data gets passed over probably Redis or something like that, and then you have your work is running behind that. You have those three different components. I would like to also be able to provide a different deployment style where you just have a single server process, well, set of server processes that are running that interface with the application framework directly.

56:42 Michael Kennedy: Yeah, that's awesome. Instead of having one more hop and two sets of worker processes and separate implementations and all of that stuff, yeah.

56:50 Tom Christie: And there's different considerations to both different deployment styles, and there's some reasons the tiered approach that channels take can be preferable, as well, but I think it would be... If you can do this build a server framework that would be great for Asyncio frameworks to be built against, that is also able to power an existing really, really widely deployed synchronous framework, as well, and add web socket support to it. I mean, you know, Andrew Godwin has done all of the heavy lifting here. I'm just gluing various bits and pieces together.

57:28 Michael Kennedy: This wire goes to this wire, yeah.

57:30 Tom Christie: Okay, I think there's like low hanging fruit here. If we take this good work from over here and that good work from over there, and yeah, live wire them together. Yeah, so that's where that is at the moment.

57:44 Michael Kennedy: That's super exciting.

57:45 Tom Christie: Yeah, and ultimately that is also a different slice of the puzzle for API Star, because I would like API Star to primarily end up being an Asyncio framework, and in order to do that I want to get a proper separation between the server and the framework for that.

58:05 Michael Kennedy: Yeah, ideally just make those methods all async methods and just await on the things in the middle, and let the server do all the magic, yeah.

58:15 Tom Christie: Sounds easy.

58:15 Michael Kennedy: Yeah, perfect. What is that, next week?

58:20 Tom Christie: We'll see.

58:21 Michael Kennedy: So, let me ask you a question. So, if I want to take API Star and run it on UVicorn in production and send all my traffic and stuff there, is it ready?

58:30 Tom Christie: It would be very enthusiastic.

58:34 Michael Kennedy: How is the readiness of API Star? How is the readiness of this UVicorn?

58:37 Tom Christie: So, I know that API Star is already being used in production by at least a couple of folks. You certainly would have to not mind the fact that there will breaking changes to the API that we provide.

58:49 Michael Kennedy: So, pin your dependencies in requirements.txt?

58:52 Tom Christie: Yeah, that's one thing, and be prepared to move forward as the framework moves forward. The flip side of that is that I think once it's... The reason that it's still slightly in flux is because of the async side of the puzzle and crossing that bridge. Once that bridge has been crossed, then I think that's when it all starts to tie together really nicely. The point at which I'd say, "Yeah, go and do interesting business things in it," will be the point at which you see a proper set of docs available for API Star. At the moment, it's fairly comprehensively documented, but it's still all on the Read Me at the moment. My big signal to everybody of, "Yeah, go, primetime," will be once I've finally got the doc slice, because I'll only launch them when I'm really happy with how the thing is looking.

59:44 Michael Kennedy: Sure, sounds great. And UVicorn?

59:46 Tom Christie: Well, that's a different slice of stuff that you would use it for, at least at its current level, because it's so much lower level. Give it a try. Don't do anything with it in production now, for sure. At the moment, that's in experimental stage. But again, I think that's going to it's not actually that much work that needs to be done on it. So, I think that that's going to become... And actually, that will kind of land before API Star is completely, completely primetime ready, because it's a smaller chunk of work.

01:00:23 Michael Kennedy: Sure, sounds good. So, often people will write to me and say, "Hey, I'm really looking forward to getting into open source. What project would be good?" Are you looking for contributors for any of these projects?

01:00:33 Tom Christie: Yeah, sure. REST framework is quite hard to get involved with unless you have quite a unique quality, which is the quality of being happy to go in, watch the issues list, and as something comes in just try and figure out if it makes sense, comment on the issue does this make sense or not, and figure out if you can reproduce it if it looks like a bug, and all of those sorts of bits of day to day triage work. And they're not terribly shiny or wildly exciting. It's amazing when people do pick it up, and it's amazing when you see folks who do have the quality of being good at that because it's quite rare, but that's more difficult typically for people to get into. There's loads and loads of interesting feature work that's all surrounding stuff that can be done on both API Star and UVicorn a the moment. There's a discussion group for API Star. So, shout out on there if you're interested in it and if there's anything in particular that you think you might be interested in working on. And with UVicorn just raise an issue against the repo and I'll check them out there.

01:01:55 Michael Kennedy: Alright, yeah, sounds good, sounds good. Well Tom, I feel like we've covered a lot of ground in this episode.

01:02:02 Tom Christie: It seems like it, yes.

01:02:06 Michael Kennedy: We could've almost done three shows on this. Maybe we'll come back someday and do that. So, maybe we'll leave it there for those topics, but let's get to the questions. So, if you're going to work on any of these projects, what editor do you open up?

01:02:19 Tom Christie: Atom, always Atom. I had a brain fail there. I was like, "Wait a minute ."

01:02:27 Michael Kennedy: Oh yeah Atom. Cool, cool. And we have over 100,000 packages on PyPI these days. What's one that maybe you ran across recently that was awesome but not everyone knows?

01:02:41 Tom Christie: White Noise. White Noise is.... So, White Noise is a piece of WSGI middleware that serves static files for you. That's all it does. It serves static files for you, and it does it so what is great about it is it's fast enough that you can just put in this bit of pure Python code that handles being able to have your static files served from the same platform that the rest of your application is served from. Don't have to set up S3 and direct some of the stuff over there and make sure that you're uploading your images to S3 at the right time. And it handles all of the right bits of getting the HTTP caching for that right. So, if you want to be really savvy you can deploy it behind a CDN and your application won't get any significant traffic hits because as soon as the CDN got each asset it'll just end up getting served with the CDN. And what I love about White Noise is it's so perfectly scoped. It's one of those projects where it's like, "Okay yeah, you got this right."

01:03:57 Michael Kennedy: That's awesome, that sounds really, really fun. So, thanks for that. I hadn't heard of White Noise. And I like the name, too.

01:04:02 Tom Christie: You got it. Yeah, static files. So, it took me a while to get it. Static files, right?

01:04:07 Michael Kennedy: Yeah, it's cool. Yeah, yeah, that's awesome. Okay, so final call to action. People want to check out your projects, they want to get involved, what do they do?

01:04:16 Tom Christie: Just jump in really. So API Star that's jumping on the discussion group, even if it's just saying, "I'm interested. This is why I'm interested." Then that's a great thing to do.

01:04:27 Michael Kennedy: Was that what your talk at PyCon was about?

01:04:29 Tom Christie: It was, yes, it was.

01:04:30 Michael Kennedy: Yeah okay, so you can check that out at YouTube.com/PyCon2017, and then search for you.

01:04:37 Tom Christie: Yeah. With REST framework the thing that is a really good place to get into is going onto the discussion group and if anybody is having problems trying to help them, or getting stuck in on Stack Overflow. Those are all great things to do. Or, just say hi on Twitter or come and have a chat on IRC. Anything like that is always welcome.

01:04:59 Michael Kennedy: Alright, beautiful. Well Tom, this has been a lot of fun to talk about all these API ideas, and also the funding stuff was really interesting. Thanks for being on the show.

01:05:07 Tom Christie: Great, thank you so much Mike. Have a great day.

01:05:10 Michael Kennedy: You bet, yeah, you too, bye. This has been another episode of Talk Python to Me. Today's guest was Tom Christie, and this episode was brought to you by Linode and Rollbar. Linode is bulletproof hosting for whatever you're building with Python. Get your four months free at talkpython.fm/linode. Just use the code Python17. Rollbar takes the pain out of errors. They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed, until your users complain of course. As Talk Python to Me listeners, track a ridiculous number of errors for free at Rollbar.com/talkpythontome. Are you or a colleague trying to learn Python? Have you tried books and videos that just left you bored by covering topics point by point? Well, check out my online course Python Jump Start by Building 10 Apps at talkpython.fm/course to experience a more engaging way to learn Python. And if you're looking for something a little more advanced, try my Write Python code course at talkpython.fm/pythonic. Be sure to subscribe to the show. Open your favorite podcatcher and search for Python. We should be right at the top. You can also find iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm. Our theme music is Developers, Developers, Developers by Cory Smith who goes by Smixx. Cory just recently started selling his tracks on iTunes, so I recommend you check it out at talkpython.fm/music. You can browse his tracks he has for sale on iTunes and listen to the full length version of the theme song. This is your host Michael Kennedy. Thanks so much for listening. I really appreciate it. Smixx, let's get out of here.