APIs were once the new and enabling thing in technology.
Today, they're table stakes, and getting them right is important.
Today, we'll talk about one of the most popular and mature frameworks in Django REST framework.
You'll meet the creator, Tom Christie, and talk about the framework, API design,
and even his successful take on funding open source projects.
But Tom is not done here.
He's also creating the next-generation API framework
that fully embraces Python 3's features called API Star.
It's all about frameworks on Talk Python to me.
Episode 125, recorded June 8, 2017.
Welcome to Talk Python to Me, a weekly podcast on Python,
the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy.
Follow me on Twitter, where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm,
and follow the show on Twitter via at Talk Python.
This episode is brought to you by Linode and Rollbar.
That's right.
Welcome to Linode, who has joined Talk Python to me as a major sponsor.
Be sure to check out what both of them are offering during their segments.
It really helps support the show.
Tom, welcome to Talk Python.
Hi.
Hey, it's great to see you again, and great to have you on the show finally.
I've been admiring Django REST framework for a long time,
and really a fan of your new work that we're going to get into.
So I'm happy to have you here to talk about it.
Cheers, Mike.
Good to see you again.
Yeah, yeah.
How are you doing, anyway?
I'm doing super well, super well.
So I want to hear how you got into programming and Python and all that, but how was PyCon?
We were both there the last time I saw you.
Even though right now we're like 6,000 miles apart, we were in the same room just a little
while ago.
How did you enjoy it?
It was hectic.
You had a booth, right?
Yeah, we had a booth.
So I haven't done PyCon before, mostly because with it being over in the States, in the past
it's been hard for me to justify taking the time and the cost of the transatlantic flight
and so on.
Since I've been, and I guess we'll talk about this later, but since I've been doing open
source for a living, all of a sudden, okay, I really better step up and get across there.
And I kind of realized slightly later on in the day, oh, wait a minute, they have this
thing where they offer booths to open source projects.
We better do that.
That sounds like a good idea.
And yeah, just about managed to organize getting one of those with a lot of help from Anna,
who I also work with.
And I didn't know what to expect, but it was just nonstop, basically.
That's really good, right?
People are dropping in to talk to you about your project.
Yeah.
And there's so much kind of positive reinforcement coming from people at the stage, but it was
just nonstop talking for about three days together with the poster session as well.
So I have no idea how many, felt like hundreds, but...
Yeah, that's really cool.
We had a booth as well for all the podcasters that we had gotten grouped together and had
the five podcasts all put together in the one booth.
And it was a real similar experience.
Like the positive vibes and the feedback was amazing.
Some of the other guys who hadn't been there before, they were like, wow, this kind of makes
it so real with all these people.
Did you have that similar experience?
Like, you know, there's a number of stars on GitHub and you get a message and stuff,
but like when hundreds of people come up to you and talk to how you've like enabled something,
did that strike you pretty strongly?
I kind of had this odd mixed feelings about, not mixed feelings exactly, but just this strange
sensation because being at the booth and you've got, like you say, all these people coming
up and, you know, saying thanks for the work that you do on the project and all of these
kind of good vibes.
But then at the same time, I'm standing there at the booth and I'm looking over at all the
big, real professional companies that know what they're doing and thinking, hmm, that's
where I'd really like us to get to, you know?
Yeah.
That was interesting.
But what was also nice as well was having a good excuse to only talk to people at the
conference.
You know, the only time that I managed to get upstairs into the talk spaces was for
the talk that I gave and the rest of all of the talk session time, I was just talking to
different people and I really enjoyed that.
I got a lot out of that.
Yeah.
I felt the same way.
I only went to the keynotes and the open spaces.
And other than that, I really just was at the booth talking to people.
It was really, really amazing.
And, you know, thanks to everyone who stopped by my booth and I'm sure you feel the same
way.
It was quite cool.
By the end of the second day, I had basically lost my voice from talking.
Yes.
I should also say that we got a bunch of help from other volunteers who offered to step up
and give us some of their time on the booth, which was awesome as well.
Yeah, that's cool.
Thank you to them.
So, yeah, one positive experience from PyCon and people who are out there listening, be sure
to get your PyCon tickets early because, as always, it sells out.
And then I get messages like, hey, can you get me a ticket?
It's sold out.
Like, nope, no secret tickets.
All right.
So let's get into your story first before we talk about your restful things.
How do you get into programming in Python?
Oh, yeah.
Okay.
Well, let's start with how do I get into programming, I suppose.
So ZX Spectrum 128.
So when I was a little kid, I did a little bit of programming from magazines and stuff.
You know, when it was in the days when they came with a tape on the cover back then.
And my dad bought me a ZX Spectrum and I did a little bit of noodling around then.
Just silly little game stuff.
Well, not even that, really.
At that point, that was just copying stuff out of magazines or trying to figure out little
bits and pieces.
And then I kind of dropped it for a while.
And later on, when I was a teenager, I started getting into QBasic and programming games together
with a friend.
And we had like a little thing going when we did that.
And that got me into it enough to decide to do it for a university course.
And then I've had a big old windy road since then.
So I started out working in speech recognition, stayed in that field for several years.
I then moved into networking and worked with a content distribution network who did a peer-to-peer
content distribution network.
Peer-to-peer, huh?
That's pretty cool.
Yeah.
Well, what they started out with initially was providing BitTorrent caches to ISPs so that
ISPs wouldn't have to pay for the upstream costs to the internet and could instead just cache
the really popular streams, right?
Do you feel like BitTorrent was ruined by people using it too much for piracy?
Like BitTorrent, I think, would have been a really fantastic for all sorts of web content
delivery, video streaming, so many things.
But since it got so associated with pirating MP3s and videos, it's kind of a black mark.
I don't know.
I think if it had been the right solution technically, then it would have won out on its own merits
anyway.
And the legal side of it, I don't think was really the main influencer of that necessarily.
Oh, and really, I guess the reason why I say that is because I was there busy working at
a company which was using BitTorrent and was trying to provide a, here's how you do a CDN
using PID's PIT.
I don't know.
It's too long ago for me to give you a code here.
I'm going to just stop now.
All right.
Sounds good.
All right.
So you did that for a while.
Was that in Python or something else?
So that, see, I can't remember at what point things stepped into Python.
It was either when I was working in speech recognition and at some point all of the sorts of the build
tools that you need for doing things like building the language models, right?
The sorts of things that we were doing when I started there was writing shell scripts and
pull scripts.
And I can't quite remember whether I moved into Python then because it's been long enough
now or whether it was when I then later started doing networking.
But whenever it was, all of a sudden, it was like, oh, wow.
This is so, you know.
This is different.
Having been used to working in C and C++ and either Pearl or Bash for scripting, it was,
yes, such a breath of fresh air.
And it's quite surprising almost to be here after all this time and still be so infused by
the language and not feel like that's really changed.
Yeah, that's really an interesting point.
I've kind of marveled at that myself as well.
So, you know, even if you just look at like some of the growth curves, popularity curves
of Python and the ecosystem, it's been around for over 25 years now, but it's still growing
and growing more quickly now than it did 10 years ago or when it was brand new.
It's really interesting.
Right.
And we're just coming up to potentially a new kind of slice of stuff that it can break into
as well.
You know, now that Python's readoption has finally really started to hit that tipping
point and the asyncio primitives that are in there, all of a sudden there's this new slice
of stuff that Python could be really, really good at that it historically hasn't particularly
been.
Yeah, it's definitely an exciting time.
Yeah, for sure.
All right.
So thanks for the story.
What do you do day to day?
You said you're a professional open source developer.
How awesome is that?
Yes, that's quite a rare, a rare thing, isn't it?
Really?
I love to share these stories.
I've been, you know, been really lucky with the way things have worked out just with timing
and so on.
So, yes, I, as of about 12 months ago, I now work on open source for my living full
time.
So let's approach it this way.
So I started on Rest Framework about five years ago and all of that was as a side project
as it is with most people.
And when we came up to the point where I was thinking about all the things we'd need to do
for Rest Framework 2, I figured that I was going to need some full time on it.
And in order to do that, we launched a Kickstarter project for it.
And that was really successful.
And I managed to get...
What year was that?
Do you remember?
That must have been either 2014 or 2015.
2015.
Okay.
So pretty recently.
Yeah.
Not so long ago.
It probably, it was probably started in 2014 and I was probably working on it up until
2015.
How'd the Kickstarter go, by the way?
Oh, it was awesome.
Yeah.
We were, we got about 35,000 pounds, which I think was about $50,000.
The way that I organized it, I was with a very small web consultancy or very, very into supporting
open source.
The person who's the CTO of the company runs the local Python meetup group and so on.
And we figured out a way that I would be able to take a slice of time out of my working day
in exchange for us doing the Kickstarter through the company.
And I was able to work on the project for, I don't know, maybe about nine months or so as
a result of the Kickstarter, with most of my time being dedicated to it, or about 50% time.
So that was great and was much more successful than my highest estimate would have been for it.
What goal were you asking for?
I guess our starting point was probably something like 5,000 pounds.
And maybe I was hoping for 10,000 and flew way past all of that.
Cool.
Congrats.
Yeah, it was great.
Yeah.
And then more recently, when I was thinking about the next chunk of functionality that I wanted to be able to deliver, I was trying to figure out how to do it.
And launching another Kickstarter was one of the options.
But trying to find a way that we could do it that would remain sustainable rather than just being a big one-off thing.
Yeah.
Kickstarters are nice, but they're very much one-off.
Like, here's a boost.
Now what?
Yeah.
You can't just keep doing Kickstarters.
Yeah, and it has to be – you have to have a really nicely wrapped up bit of functionality.
It's difficult to kind of sell it as triage on the project or open-ended bits of work.
And there's a lot of work as well.
You can't hit go until you're completely confident that you've really nailed the pitch that you want to make, which can make it quite difficult.
Because even if you've got something you think, we really need to do a Kickstarter for this, until you feel that you're at the point that, okay, we've got it just right.
Let's do this.
You keep on wanting to hold off, hold off, and try and – anyway.
But this time around, I started thinking about launching an ongoing funding model instead.
We were really fortunate because it tied in at the same time with when Mozilla launched their MOS awards, so the Mozilla Open Source Awards, where they give awards to projects that they're using in order to help them develop further functionality.
I mean, that comes with some sort of funding, right?
It does.
It does indeed.
So we put forward an application for the project.
And when we were told that we'd been successful, and I can't remember exactly how much it was right now.
It was probably about $35,000, I think, that we'd requested.
When we found out that we'd been successful, that was the point at which I said, okay, well, I better give this a shot now.
I told my company that I was going to go and try working on open source full-time, and we launched the collaborative funding model, is what I tend to call it.
That's cool.
How long ago was that?
How long have you been doing it in this path?
So, yeah, just over a year.
We literally just passed over the year milestone now.
So, yeah.
Are you enjoying it?
Oh, it's great.
It's great.
It's not, you know, being entirely self-driven and being in such an atypical business model, quite an amorphous set of constraints, you know, that can be odd at times.
But I try to spend most of my working time out of co-working space rather than just working from home.
So that helps give things structure, and it's been amazing how much you can push things forward when businesses actually have that relationship of trust to invest a little bit of their money in you in order to give you that space to do that.
So, yeah.
Yeah, I feel like there should be way more of this.
There are businesses, there are billion-dollar businesses whose primary product is built on layer upon layer of these super important open source things, and they contribute nothing to it.
Right.
And at best, they'll do a PR back to it.
But even, you know, maybe not even that, right?
And they would probably get a net benefit financially just by supporting the people to tweak it just a little bit to suit their needs better, you know?
Right.
So, yeah.
And the responsibility of people like myself who are fortunate enough to have managed to find themselves in this position is to prove the business model.
And so there's two sides to it.
One's the marketing angle of it, which is important.
But in a way, it's something that I'm less interested in.
The thing that I really want to demonstrate is when you take a collaborative approach to investing in infrastructure, if it's shared infrastructure, then the rewards that you get for your investment are so outsized to the rewards that you can get by solely investing only within your own company.
Because you've got all of these other companies that are also pitching in at the same time.
So the way that I tend to talk to potential sponsors is I say, hey, I want to work for your business full time for $50 a month or $100 a month or $150 a month.
And they're like, amazing.
How do we do this?
Well, I have a figure I tell them.
You know, how is that not a reasonable business proposition?
Okay, I'm not working on exactly day to day what they tell me, but I am working on trying to support their businesses.
You know, it's not eye in the sky stuff.
Yeah, I'm not coming to your meetings, but I'm going to be working on your business.
Yeah.
Yeah, I think it's a super, super straightforward story.
And I think you're right.
There's people sort of blazing that trail.
And how well it works in the future is going to depend on how well, like the first few successful ones blaze that trail.
Right?
Yeah.
We need to build stuff that makes those companies more successful.
And we also need to sing and dance about the companies that are supporting open source financially.
And do things like try to get the community into a mindset of when you're looking for your next job, look to companies that are supporting open source financially as your first port of call.
And try to drive things that way around as well.
Yeah, I feel like it's something of an awareness thing as well, right?
Like, it's a pretty straightforward sell to go to pick a random company, Fidelity Investments or whatever, and say, look, if you guys paid like 200 bucks or a thousand bucks a month to these five open source projects to keep them going.
And everyone did this.
Like, you would have a much more stable platform.
Your scalability would be better.
And you would be able to, you know, trade half a microsecond or millisecond faster or whatever, right?
That's a really simple story to tell to the business owners.
But it's, I don't know.
It's, I'm not sure how much it's happening yet.
It's step by step, right?
You know, we need to be able to have really great examples where we can point to whether it's frameworks or platforms or projects that have really succeeded because they have been financially successful and say, look, this is what can happen when you do support these projects.
This is what the people who supported it and anyone who's using it actually got out of this experience.
And now we can do it over here and we can do it here and we can do it here.
And also to do things like look retrospectively and say, how much did that cost each business?
When you go and you look at all these bits of, well, anyway.
Yeah, no, I think it's, I think it's really great.
And we should probably talk about your projects in particular as well.
Yeah, I know.
But I really, but I think that this is super interesting to people.
It's very powerful.
And for everyone who's doing it, there's probably a thousand people who wish they could.
And so, like, just the awareness of us talking about it, I think, serves a good purpose.
I think it's going to change.
I'm really optimistic.
I see things have been moving forward.
And I feel like we're on the start of a wave.
I mean, that's easy for me to say because I'm fortunate enough to actually be in that position.
But I do feel like it's starting to change.
Sure.
I definitely think it's something that 15 years ago would have been really, really unexpected.
And now it seems pretty, pretty normal.
It's just something that's got to get moving.
Yeah.
This portion of Talk Python to Me is brought to you by Linode.
Are you looking for bulletproof hosting that is fast, simple, and incredibly affordable?
Look past that bookstore and check out Linode at talkpython.fm/Linode.
That's L-I-N-O-D-E.
Plans start at just $5 a month for a dedicated server with a gig of RAM.
They have 10 data centers across the globe.
So no matter where you are, there's a data center near you.
Whether you want to run your Python web app, host a private Git server, or even a file server,
you'll get native SSDs on all the machines.
It's a 40 gigabit network, 24-7 friendly support, even on holidays, and a seven-day money-back guarantee.
Want a dedicated server for free for the next four months?
Use the coupon code python17 at talkpython.fm/Linode.
You've got a couple of projects that are amazing.
And so let's take them, I guess, from the beginning.
So the first one, which we've mostly been talking about, is Django REST framework.
All right.
This is the thing you started back in 2010 or so?
Yes.
Yeah.
All right.
So, you know, everyone knows what Django is.
They probably know what REST is.
What's Django REST framework?
What is this project?
Django REST framework is, unsurprisingly, an API framework for Django.
And it tries to fill in all of the gaps of where Django was historically designed as a web framework.
What are all the extra bits?
Right.
Here's HTML.
Go into browsers.
You wanted to make a talk to machines, not people, right?
What are all the various bits and pieces that you would want to provide additionally if you wanted to use Django primarily as an API framework?
So, for instance, approaching authentication differently, providing things like throttling and permissioning in a way that it's really easy to use them in different slices of your API in different ways, handling things like HTTP content negotiation,
and all sorts of bits and pieces, providing generic classes of views that give you default functionality for the CRUD case.
So, if you've got an API where you want to provide a set of different URLs and methods on those URLs that are useful, creating records, retrieving records, updating and deleting records, giving you all of those sorts of bits of tools at the bottom.
So, maybe you have like slash API slash users, and then you can do a get to get all the users, a get slash ID to get a particular user, a post to create a user, things like that?
Yes.
And also dealing with the, if you've got some data that you want to present to the user, or rather if you've got some objects that you want to present to the user, what do you want the representation of those objects to look like?
If we're dealing with relationships between the objects, are you presenting those as the plain primary keys, or should they be presented as hyperlinks?
All of those sorts of that.
Do you want to return big nested sets of data where you've got all sorts of interlinked objects all being returned in bulk, or do you want flat data structures?
All of those sorts of things.
Okay, cool.
So, if I'm working with Django ORM, or even SQLAlchemy, or MongoDB, something like this, and I'm getting sort of ORM objects back, does it handle like the serialization of those types of things?
So, being Django-based, it works primarily with the Django ORM.
There's stuff that you could do to integrate it with other backends as well.
The question is, suppose in my web method, I do a query and I get a list of users back from Django ORM.
Can I just return that list and have it magically JSONify it, or XMLify it, or whatever?
Yes and no.
So, yes, you can.
No, you probably don't want to do that.
I shouldn't.
You can, but don't do it.
So, you can ask it to give you the default representation and just say, return all of the fields on this model.
So, give it a model serializer, tell it which model it's pointed at, and say, and I just want all of the fields on that.
And that will give you a default representation that, depending on which class you use, just uses either primary keys or hyperlinks for relationships that that model happens to have.
What you would want to do more typically is be a little bit more precise about the representation.
So, declare the fields that you want returned explicitly and independently of the model itself.
So, list them all out.
So, yeah.
Yeah.
And going the other way, maybe it's even more important.
Like, if I post a JSON body to a method, is there serialization on that side?
Or do I, like, read it?
There's two different aspects to this.
So, there's the taking the ORM object and turning that into plain Python data types that are ready to be serialized, that are ready to be rendered.
That we call the serialization, going between the plain Python data types and ORM objects.
Then there's the parsing or rendering, which is, on the parsing side, taking request data that's coming in and turning that into Python data types.
So, JSON.load.
And on the other side, the rendering, which is taking the data types and turning those into the response body.
So, JSON.done typically.
Yeah.
So, it looks like you have a lot of control over all those different things.
Yes.
So, you can, for instance, you control the serialization style, which is what information do you want in the representation.
But separately to that, you can also control the renderings of that.
So, you could, for instance, have both a JSON and a YAML output for the same data.
Sure.
Did I notice somewhere in your docs that you can basically use, like, a fake or false extension to control that?
So, like, a slash user slash one dot JSON versus slash user slash one dot XML and things like that?
Yeah.
So, there's different ways that you can control which rendering you get.
By default, REST Framework will use content negotiation, where the client sets an accept header in the request, which informs the server which response type it would like,
which content type it would like in the response, or you can have URLs that specifically are for one particular type of rendering using suffix.
So, yes, dot JSON or dot YAML.
Yeah, yeah.
So, the most popular API that I know that does that is the Basecamp API.
Right.
They do, like, thing dot JSON, even though it's entirely not needed.
That's just, like, their style, right?
Some people like that.
Some people don't.
Yeah.
But the thing that's actually drove all of that side of the rendering in the first place anyway was the browsable API.
So, when you go to a URL in REST Framework, if you make that request with curl or a command line client, you will typically get back,
depending on exactly what you configure, but you'll typically get back plain JSON.
When you go to it in the browser and the browser sends an accept header that says, I would prefer HTML, then we're able to return an HTML rendering of the view with the data in line on the page,
but also a whole bunch of extra information around the outside and controls to allow the user to interact with that API endpoint.
That's awesome.
So, it's like self-documenting APIs out of the box?
Yes, exactly.
That's really cool.
Yeah, that's one of the things that I think often gets forgotten on APIs or gets out of date, right?
Like, either there's basically almost no API, like here's an example and it's incomplete or whatever, or here's the documentation, but our API has evolved and it's no longer.
Yeah.
Well, I'm still surprised that there aren't more frameworks that are starting to provide this.
I mean, the browsable API stuff, that was the entire reason that REST Framework came into existence in the first place,
because I was thinking about what, you know, why are APIs not, why can't we interact with them using the web browser?
This is a bit weird.
Yeah.
And that was the entire driver for, okay, well, how would you do that?
Oh, you'd do it like this.
Oh, wait a minute.
This is a framework.
Oh, if I think this is a good idea, then I'd better make it a good framework.
My prototype is almost the thing.
You know, that's awesome.
Yeah.
I feel like that's always been a weak point, or traditionally it's been a weak point of HTTP frameworks or APIs is that they're really easy to interact with,
but it's hard to see the documentation unless somebody's taking the time to do that separate.
Yeah.
So can you give us just like a quick sense, not too much in code, but just like the conceptual steps of like,
how would I create a project and like set it up and add an API method using Django REST framework?
It slightly depends on whether you are building a project that is primarily a CRUD style API or not,
because REST framework gives you an awful lot of tools for building those sorts of APIs with a minimal amount of code if you are,
and that's the most common use case, really.
Do you point that out like your ORM objects and say, here's an ORM object, give it the various operations?
The way around that I would approach a new project with REST framework,
I would start by taking REST framework entirely out of the picture and not think about what tool is this API being built with anyway,
and instead think about the basic, the fundamentals of the API.
So which endpoints do I want to expose?
What do I want the representations of those endpoints to look like?
How do I want relationships between different objects to look?
And try to get that side of design figured out in my head first independently.
Then you've also got the data models as well and thinking about the storage side of things,
which is very closely tied to the representation, but isn't necessarily the same thing exactly.
So once you've got the start of those things in your head, then it's a case of blocking out the Django models,
blocking out corresponding serializers for those models.
So REST framework gives you a lot of different levels of tooling that you can work at.
If you want, you can work at a very high level of abstraction where we provide things called view sets
that bundle up a whole lot of different interactions into a single class
and provide, mostly automatically provide the routing of URLs onto all of those points of interaction.
Now I tend to not work at that level of extraction.
I would usually go down and work at the level of per URL, write either a function-based or a class-based view for that URL,
and write out the methods on each of those individually.
And again, there's still another different couple of layers of abstraction that you can be working at there,
because you can either be using the built-in functionality of give me the automatic create an object function,
or you can go and write that code yourself.
When I'm building an API with REST framework, I tend to prefer the explicit approach.
So if it happens that a listing view or an update view or a delete view is exactly already provided by the framework in the right way,
then I'll use that.
But if it's not, rather than trying to override it at particular hook points within those class-based views,
I'll just override the get or the post or the put or the delete method entirely and write it out explicitly,
because that really keeps everything surfaced up to the top.
I find makes it quite a maintainable approach to building the API.
That's cool.
I really like that you approach it from the outside.
Like, how should this look to the people consuming this API?
Forget the implementation.
And then let's make that happen with Django REST framework, right?
Because really, in the end, that's what's going to determine whether people are happy with the API you've delivered them, right?
Yes.
Yeah, absolutely.
Yeah.
It's not like, well, you do it this way because it was easier.
It was automatic in Django REST framework.
But it should have been this other way, right?
I mean, it started off maybe when the API frameworks and the landscape were younger.
Maybe it was a bit different then because there's also the aspect of it's useful if the frameworks guide you towards what set of design choices should you be making there.
Yeah.
Yeah, but, you know, in 2017, we have more experience building APIs and more examples of good APIs than long ago.
Yeah.
Nice.
So you want to tell us about some of the popular deployments?
Like, who's using it?
Do you know how they are?
Gosh, yeah.
I should probably open up things like the sponsor list to remind myself there.
So let's think.
No, I'm not going to do that.
I'm just going to.
So Mozilla are obviously a really big name.
We're a big sponsor of the Kickstarter.
I can't remember how much they're using it at the moment, but I think that they are.
I know that Red Hat are using it at least within a couple of places.
Heroku use it for something or other.
And, well, we've got about 60 corporate sponsors.
So stacks and stacks of different consultancies and product businesses of all types, really.
It's always interesting when you get a new sign up, finding out what the business is, because there's so many different surprising angles that it's being used from.
Yeah, can you think of one you're like, whoa, you're doing what with my framework?
Not off the top of my head, but that's probably like post-PyCon haze, because, you know, they all went in and then, yes, it all walked over me.
Yeah, excellent.
All right.
So it sounds like a really successful project.
And this is the one that basically let you, through the Kickstarter and the collaborative funding and these things, lets you become a professional open source developer.
Basically create a business around your framework.
And congratulations to you on that.
That is super, super cool.
I think it's so neat.
I'm happy to be able to, like, shine a light on it.
Thanks.
Yeah, it is.
It's wonderful.
Yeah.
So there's another thing that I came across that you just started working on as well that's very exciting.
And I don't know how it ties into the funding stuff, but I certainly think it's cool, is called API Star.
Tell people what API Star is.
This is a new API framework that I started working on.
Rest framework is still my main gig, I guess.
But there are a couple of different things that I think are important to explore at the moment.
And API Star is my start towards doing some of that.
It's an API framework that's not tied in to Django or the Django ORM and has a different approach to the fundamental interface that you use for building your code base.
So rather than having views, which are functions that take a request and return a response, instead you have views where you declare on the view function using type annotations,
the sorts of data and the sorts of components that that function is going to take.
And you also declare what type of data it's going to return as well.
And once you take that different approach, there's a lot of interesting things that you can start to do on top of that.
So to start with, because you've got a plain data interface rather than take a request and return a response,
you can layer on different types of network interface or different ways of interacting with that function on top of it.
So for example, as well as accepting HTTP requests, you could route a command line command onto your view.
Or you can potentially look at being able to build views that accept either HTTP requests or WebSocket connections and that provide a real-time subscription endpoint.
There are ways that you can start to think about doing that or even start to think about can we build views that we can share functionality being exposed both to GraphQL endpoints and regular HTTP calls.
Now, not all of those things are done, but I'm starting to think about how would we be able to do those.
So what we're going to do is do is do a type system that corresponds to the same capabilities as JSON schema.
So we have a type system that you can use to declare what your data structures look like, both for the inputs and for the outputs of these functions.
And once you've got that, you've suddenly got something really powerful where you're able to, for instance, build API documentation based on those type annotations that is guaranteed to always be in sync with your code base.
Not only can you build this API documentation, you can also build interactive API documentation because you've got enough information in those types
that you can build HTML form controls that you can build HTML form controls with appropriate controls given the shape of the data input.
You have this property all of a sudden that you have a really nice one-to-one mapping between your function declarations and an API schema language such as Swagger or Rammel.
So you can have a really clear way of generating Swagger or Rammel from your type declarations.
If you wanted, you could even go the other way and say, okay, I've got my Swagger blueprint.
Build me the stubbed out set of functions that I'm going to want for my project.
You've also, you've even got enough information that once you write out these declarations, even before you've implemented anything,
you could start running or you will be able to start running a mock API.
So you've got all the response types.
You know what the inputs look like and whether they're valid or they're not valid.
So that what you can do is have your team build up all the function declarations and present to your front-end team a mock API
that's already ready to start them build the front-end against,
while your back-end team starts to fill in all the blanks of, oh, well, we better actually implement the behavior here.
Yeah.
So there's also some interesting things you can do.
Oh, and also, what else?
Dynamic client libraries that are able to interact with your functions.
They have enough information being exposed in the schemas that you're able to generate that you can present an interface on the client side
that corresponds to the same thing you have on the server side.
So all sorts of cool stuff once you've got this.
Yeah, that's actually been one of the weak points of HTTP APIs over, say, SOAP services, right?
SoAP services are dreadful.
But one of the things that they were good at is they had this WSDL thing that you could then generate basically clients that know all the methods,
all the types that are exchanged there.
I'm thinking of SUDs in Python or obviously like Java or .NET and their tooling.
But here you're kind of describing enough information.
This is the stuff that goes in.
This is the stuff that goes out that you could actually bring that back in a sense.
Yes.
What this is it's a way of – it's still a restful approach.
It's just a way of providing a better description to what the interface is.
And that also has an impact on what sort of APIs you build, actually,
because given that we haven't had great client-side tooling or great ways of describing what those interfaces are,
the let's always build CRUD style interfaces is something that's kind of naturally happened
because it's been a bit of a constraint of we can't express very well different things that don't fit into that.
But having a really expressive way of being able to declare what those interfaces are lets you take a more,
okay, these are just different types of actions or different types of views onto the data.
They don't necessarily have to correspond to CRUD.
Yeah.
This portion of Talk Python to Me has been brought to you by Rollbar.
One of the frustrating things about being a developer is dealing with errors.
Relying on users to report errors, digging through log files, trying to debug issues,
or getting millions of alerts just flooding your inbox and ruining your day.
With Rollbar's full-stack error monitoring, you get the context, insight, and control you need to find and fix bugs faster.
Adding Rollbar to your Python app is as easy as pip install Rollbar.
You can start tracking production errors and deployments in eight minutes or less.
Are you considering self-hosting tools for security or compliance reasons?
Then you should really check out Rollbar's compliant SaaS option.
Get advanced security features and meet compliance without the hassle of self-hosting,
including HIPAA, ISO 27001, Privacy Shield, and more.
They'd love to give you a demo.
Give Rollbar a try today.
Go to talkpython.fm/Rollbar and check them out.
One of the things that struck me the most about looking at this API, is it Python 3 only?
It is, yes.
So because of the type annotation.
Right.
Yeah.
So the type annotations are the things that struck me the most when I look at it.
So you might have an API method and the method, say, if you want to have access to the request object,
you would say request colon then HTTP.request.
And you basically annotate like this variable is this type.
And your framework will go, oh, I have one of those types.
I'm going to inject it here.
There's all these different types of things that can be injected, right?
Yes.
So the different kinds of component that you can inject, they can either be, say, something
that's based on the incoming request, like you say request, or you could say, you could
have query params.
So the class query params, if you put that on, you'll get the dictionary of all the query
parameters.
But you can also do interesting things like there's a class, the singular query param.
And if you put on the name that you use for that variable and the class query param together,
it will say, okay, I'll give you this query parameter.
That is so awesome because I feel like all the time, I'm just, I'm not really excited to
see this thing come into existence.
So all the time, if you're taking a traditional HTML based framework, Django, Pyramid, Flask,
whatever, they're like, okay, here's the post dictionary.
Or here's the data that was submitted.
Or here's the query string.
And you can go dig into that dictionary and find the thing you want.
Here are the headers.
Dig into the headers and find the thing you want.
But on yours, you could actually say things like user underscore agent colon HTTP header.
And it will go into the headers, find the user agent, pull it out and set that value.
So you just, it's as if all the elements of the request become just function parameters,
which is glorious.
Yeah.
So it's really expressive.
It reads really nicely when you look at it.
Right.
It also means that it's really testable because the interface that you're exposing is exactly
the thing you want to be exposing.
You don't have to do things like build up a mock request object in order to test your functionality.
Right.
Because if you say it takes a query string and it's, the thing is called name, you just
pass in the value you want for name in your test and you're done.
Right.
You don't have to fake up a dictionary that holds things.
Right.
Yeah.
And the other thing that's interesting as well is because the way it works is when you
route all of your views with the application, at the point they get rooted, it inspects what
the function signature is, figures out, has to look at what all the different components
are and figures out exactly what's going to need to be run every time it needs to run that
view, which ends up meaning that it's really, really fast because you only pull in exactly
the things that you need to pull in.
So for instance, if you've got a database component, only views which declare that they're using
the database component will end up getting wrapped in a transaction and anything that doesn't,
that never gets seen at all.
That's fantastic.
Yeah.
The other thing that I really like about the type annotations is automatic type conversion.
So you could have like a user underscore ID colon int, and obviously everything comes into
the web as a string, but do you like parse that into an int?
Yes.
That gets handled for you and the type validation on that.
So you can either use the native types for that, or you can use the type system for that,
which gives you the more expressive, everything that Jason Schema gives you stuff.
You know, here's a complex object.
Here's what it looks like.
Yeah.
Wow.
That's awesome.
All right.
Another thing that I found to be interesting was the performance story.
So like you already touched on it, this is a super lightweight framework, and I didn't
even realize the extent to which you were ensuring that it only does what it needs to do.
That's really cool.
So you ran some various tests with like G Unicorn and Mindheld, or even with like some
of the shiny new things like Sanic and UV loop, right?
And so the stuff that you had on the website, let's see, it says Flask.
You talk about scalability and just raw individual request performance.
Like, so Flask on your setup, you're getting 5,238 requests per second and about 40 millisecond
latency.
On Sanic, which is really an interesting framework as well, you're getting like 21,000 requests
per second and 10 milliseconds.
And with API stars, 25,008.
So you're like five times faster than Flask on both raw individual requests and overall, right?
That's awesome.
Yeah.
Well, keep in mind that that's a test type that is not really real world usage.
There's some really big provisos to make there.
We'll talk about that stuff in a bit more detail later, but I interrupted you.
Well, I was going to say, you have to be really super careful about these sorts of performance
comparisons because the way I'm using it and the way someone else using it could be super
different.
The way I've set up my machine could be really different, but you were running the tech and
power benchmark, which is a particular one, which is pretty well respected.
So at least in that sense, you're running the simple JSON serialization version.
Does that mean like if you use API star, your API is five times fast?
No, because you're still probably doing your same implementation stuff, right?
But it means how much overhead are you bringing in with the framework like that?
You could maybe consider, right?
Yes.
So that's what I was trying to demonstrate when I've included that section in the documentation
is not so much this is going to make things fast for you all the time, but the aspect of the
framework can get out of your way if you need it to.
Yes.
So being able to declare the response type on the view, you can actually have it just
return a WSGI response directly.
You end up with the server calling into your framework and a single function call gets made
and that gets returned directly back to the server.
So that's an interesting aspect of it.
Now, that doesn't mean for the different test types where different things like
asynchronicity come into play that it would perform in the same sort of way, wouldn't it?
Yeah, of course.
But I think it's still really important to highlight that aspect of once you take this
dynamic approach, you can work at whatever level is appropriate for the thing that you're
building.
Yeah, very cool.
Okay.
So I'm sure I've got you a bunch of email from pointing that out.
Oh, no.
Let me tell you how you could have optimized this in this setting.
Anyway, speaking of performance and asynchrony, there's another project called UVicorn.
Yeah, UVicorn.
Tell us what this is.
It's like the future version of G-Unicorn, right?
So in Python asynch.io land at the moment, there's a few frameworks that are existing.
Sanic being one, AIO HTTP being another.
Yep, Jepronto.
That's right.
A muffin.
Yeah.
They're very exciting.
Kaiowaki.
Is that right?
Kaiowaki.
Yeah.
So that's, and which is really cool.
And in particular, Sanic's been interesting because it's shown just how impressive these
two particular components that it uses are, which is UV loop, which is an event loop that is implemented
as a Python binding around either the same event loop that Node uses or something that was
originally built for Node.
And also an HTTP parsing library that is also a Python implementation, you know, a set of
Python wrappers around a super fast C implementation that, again, I think was built originally for Node.
And the performance of those two components is really impressive.
The only problem that we have in the Python landscape at the moment is that all of these
AsyncIO frameworks that we have are really still at the framework level.
There isn't this separation of, well, here's the server, here's the framework, and here's what the interface between those two is.
We've had that for a long time in our regular old happy synchronous land because we've had WSGI for ages.
And now you can't repurpose that for AsyncLand because you just can't.
And the thing that I would like to do is combine the fact that there happens to be an incredibly awesome implementation
that is super fast and the how do we figure out what the interface between the server and the framework ought to look like
and bridge that gap and have, okay, here's just the bare bones.
You know, you've got a single callable and it gets this set of data.
Now go and build your AsyncIO framework on top of that.
Okay?
Yeah, yeah, yeah.
That's really been the pinch point where we haven't been able to unlock the Async stuff on the server easily with existing frameworks.
Yes.
And I think once we've got that, that's really going to kind of set everybody free to start doing interesting things.
Where it's difficult is trying to figure out how it sits into the existing landscape.
So, for instance, one of the things that I would like Uvacorn to be able to do is to also be able to interface with Django projects using channels
and be able to provide WebSocket support.
But in a different deployment mode to channels has.
So deployment mode there at the moment is you have your servers and those sit in front of a channel layer,
which all the data gets passed over, probably Redis or something like that.
And then you have your workers running behind that and you have those three different components.
I would like to also be able to provide a different deployment style where you just have a single server process,
well, set of server processes that are running that interface with the application framework directly.
Yeah, that's awesome.
Instead of having one more hop and two sets of worker processes and separate implementations, all that stuff.
And there's different considerations to both different deployment styles.
And there's some reasons that tiered approach that channels takes can be preferable as well.
But I think it would be if you can do this, build a server framework that would be great for AsyncIO frameworks to be built against,
that is also able to power an existing really, really widely deployed synchronous framework as well and add WebSocket supports.
I mean, you know, Andrew Goswin's done all of the heavy lifting here.
I'm just gluing various bits and pieces together.
This wire goes to this wire, yeah.
Okay, I think there's like low-hanging fruit here.
If we take this good work from over here and that good work from over there, yeah, live wire them together.
Yeah, so that's where that is at the moment.
That's super exciting.
Yeah, and ultimately, you know, that is also a different slice of the puzzle for API star because I would like API star to primarily end up being an AsyncIO framework.
And in order to do that, I want to get a proper separation between the server and the framework for that.
Yeah, ideally, just make those methods all Async methods and just await on the things in the middle and let the server do all the magic, yeah.
Sounds easy.
Yeah, perfect.
What is that, next week?
We'll see.
So let me ask you a question.
So if I want to take API star and run it on UVicorn in production and send all my traffic and stuff there, is it ready?
It would be very enthusiastic.
How's the readiness of API star?
How's the readiness of this UVicorn?
So I know that API star is already being used in production by at least a couple of folks.
You certainly would have to not mind the fact that there will be breaking changes to the API that we provide.
So pin your dependencies in requirements.txt?
Yeah, that's one thing.
And, you know, be prepared to move forward as the framework moves forward.
The flip side of that is that I think, you know, once it's the reason that it's still slightly in flux is because of the Async side of the puzzle.
And get crossing that bridge.
Once that bridge is being crossed, then I think that's when it all starts to tie together really nicely.
The point at which I'd say, yeah, go and do interesting business things in it will be the point at which you see a proper set of docs available for API star.
At the moment, it's fairly comprehensively documented, but it's still all on the readme at the moment.
My big signal to everybody of, yeah, go primetime will be once I've finally got the docs live.
Because I'll only launch them when I'm really, really happy with how everything's looking.
Sure.
Sounds great.
And UVicorn?
Well, it's a different slice of stuff that you would use it for.
So at least at its current level, because it's so much lower level, give it a try.
Don't do anything with it in production now, for sure.
At the moment, that's an experimental stage.
But again, I think that's going to, there's not actually that much work that needs to be done on it.
So I think that that's going to become, and actually that will kind of land before API star is completely, completely like primetime ready because it's a smaller chunk of work.
Sure.
Sounds good.
So often people will write to me and say, hey, I'm really looking forward to get into open source.
What project would be good?
Are you looking for contributors for any of these projects?
Yeah, sure.
Best framework's quite hard to get involved with unless you have quite a unique quality, which is the quality of being happy to go in, watch the issues list, and as something comes in, just try and figure out if it makes sense.
Comment on the issue.
Does this make sense or not?
And figure out if you can reproduce it, if it looks like a bug.
And all of those sorts of bits of day-to-day triage work, and they're not terribly shiny or, you know, wildly exciting.
It's amazing when people do pick it up, and it's amazing when you see folks who do have the quality of being good at that because it's quite rare.
But that's more difficult typically for people to get into.
There's loads and loads of interesting feature work or surrounding stuff that can be done on both API Star and Yuvikon at the moment.
There's a discussion group for API Star, so shout out on there if you're interested in it,
and if there's anything in particular that you think you might be interested in working on.
And with Yuvikon, just raise an issue against the repo, and I'll chat about it there.
All right.
Yeah.
Sounds good.
Well, Tom, I feel like we've covered a lot of ground in this episode.
It seems like it, yeah.
Yeah, yeah.
We could have almost done three shows out of this.
Maybe we'll come back someday and do that.
So maybe we'll leave it there for those topics, but let's get to the questions.
So if you're going to work on any of these projects, what editor do you open up?
Atom.
Always Atom.
I had a brain fail there.
I was like, wait a minute.
What is it even called?
Oh, yeah.
The title says Atom.
Awesome.
Cool, cool.
And we have over 100,000 packages on PyPI these days.
What's one that maybe you ran across recently that was awesome, but not everyone knows?
White Noise.
White Noise is a piece of whiskey middleware that serves static files for you.
That's all it does.
It serves static files for you.
And it does it so.
What is great about it is it's fast enough that you can just put in this bit of pure Python code
that handles being able to have your static files served from the same platform that the rest of your application is being served from.
You don't have to set up S3 and direct some of the stuff over there and make sure that you're uploading your images to S3 at the right time.
And it handles all of the right bits of getting the HTTP caching for that right.
So if you want to be really sad that you can deploy it behind a CDN and your application won't get any significant traffic hits
because as soon as the CDN got each asset, it'll just end up getting served from the CDN.
And what I love about White Noise is it's so perfectly scoped.
It's one of those projects where it's like, okay, yeah, you've got this right.
That's awesome.
That sounds really, really fun.
So thanks for that.
I hadn't heard of White Noise.
And I like the name too.
There you go.
Yeah, static files.
So it took me a while to get it.
Static files, right?
Yeah, yeah.
That's awesome.
Okay.
So final call to action.
People want to check out your projects.
They want to get involved.
What do they do?
Just jump in really.
So API start, that's jumping on the discussion group.
Even if it's just saying I'm interested, this is why I'm interested.
And that's a great thing to do.
Was that what your talk at PyCon was about?
It was.
Yes, it was.
Yeah.
Okay.
So you can check that out at youtube.com slash, what is it?
PyCon 2017 and then search for you.
Yeah.
With REST Framework, the thing that is a really good place to get into is going on to the discussion
group.
And if anybody's having problems trying to help there or getting stuck in on Stack Overflow,
those are all great things to do.
Or just say hi on Twitter or come and have a chat on IRC.
Anything like that is always welcome.
All right.
Beautiful.
Well, Tom, this has been a lot of fun to talk about all these API ideas and also the funding
stuff was really interesting.
Thanks for being on the show.
Great.
Thank you so much, Mike.
Have a great day.
You bet.
Yeah, you too.
Bye.
This has been another episode of Talk Python to Me.
Today's guest was Tom Christie.
And this episode was brought to you by Linode and Rollbar.
Linode is bulletproof hosting for whatever you're building with Python.
Get your four months free at talkpython.fm/Linode.
Just use the code Python17.
Rollbar takes the pain out of errors.
They give you the context and insight you need to quickly locate and fix errors that
might have gone unnoticed until your users complain, of course.
As Talk Python to Me listeners, track a ridiculous number of errors for free at rollbar.com slash
talkpythontome.
Are you or a colleague trying to learn Python?
Have you tried books and videos that just left you bored by covering topics point by point?
Well, check out my online course, Python Jumpstart by Building 10 Apps at talkpython.fm slash
course to experience a more engaging way to learn Python.
And if you're looking for something a little more advanced, try my Write Pythonic Code course
at talkpython.fm/pythonic.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, Google Play feed at /play, and
direct RSS feed at /rss on talkpython.fm.
Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.
Corey just recently started selling his tracks on iTunes, so I recommend you check it out at
talkpython.fm/music.
You can browse his tracks he has for sale on iTunes and listen to the full-length version
of the theme song.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Smix, let's get out of here.
Outro Music.
you
Bye.
you