00:00 Could you write me a Python app for the wide range of platforms out there? Oh, wait, I want them to be native GUI applications. And I need them on mobile (Android, iOS, tvOS, and watchOS) as well as major desktop apps. I also need them to appear indistinguishable from native apps in terms of the way you deploy them (be a .app on macOS, .exe on Windows, etc).
00:00 What technology would you use for this? This week I'll introduce you to a wide set of small, focused and powerful tools that make all of this, and more, possible. We're speaking with Russell Keith-Magee, founder of the Beeware project.
00:00 This is Talk Python To Me, episode 79, recorded October 4th, 2016.
00:00 [music into]
00:00 Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the ecosystem and the personalities. This is your host Michael Kennedy, follow me on Twitter where I am at @mkennedy, keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via @talkpython.
00:00 This episode is brought to you by Rollbar and Hired. Thank them both for supporting the show, check them out at @rollbar and @hired_hq on Twitter and tell them thank you.
01:43 Michael: Hi everyone. I want to take a moment and give you a quick update about my online courses. I just opened access for my Python For Entrepreneurs course for pre-order. If you're thinking about taking this course, you should act soon, right now, we're still finishing the content, you can buy it for just 59 dollars, that's 30 dollars off the retail price, and you'll get early access to the content while it's being developed. Currently, there's about 3.5 hours of content and there's more coming every day. Check it out at talkpython.fm/launch. The early reviews have been great so I'm sure you'll love it too.
01:43 Now, let's hear about the Beware project with Russell. Russell, welcome to the show.
02:21 Russell: Thanks for having me.
02:22 Michael: It's my honor. We're going to talk about so many things in the Beeware project, several of my guests have actually recommended the Beeware project as something that's really standing out in the Python space, and I kind of have to agree with them, you guys are doing some cool stuff, and so I'm looking forward to talking about it.
02:38 Russell: Fantastic, it's been a fantastic ride the last couple of years, and over the last 12 months it sort of just started to get a lot more momentum, partially because we're actually getting to the point where we've got something we can really show that's interesting not just sort of theoretical, so yeah, I'm really enthused to talk about it.
02:55 Michael: Yeah great. Now, before we get to it, let's start with you, what's your story how did you get into Python and programming?
03:02 Russell: The programming story for me started a long, long time ago. My father is a mechanical engineer who kind of saw these computer things at the point when home computing was just starting to become a thing, and he got one of the first Commodore 64 that was available for sale in Australia. Not with any specific plan of what he was going to do with it, he just could see that these computers were going to be a thing and kind of maybe we should play with them a little bit, which is something he's always done with technology, he bought a hideously expensive 3D printer, when it was still hideously expensive, just so he could play with it. So I got introduced to the Commodore 64 when I was eight or ten, somewhere around that sort of territory, and I've kind of been just tinkering with it ever since, it's kind of gone from, "Hey I can write this really, really dumb- my sister is ugly, my sister is ugly, my sister is ugly" loop, through to doing a PhD in computer science overtime. So my undergrad degree actually wasn't in computer science, it's in physics, but I picked up all the computing degrees, all computing courses I could along the way, and they kind of encouraged me to come over to the computer science department for my postgraduate studies, so I did a PhD in computer science. And then it progressed from there, into working in industry, my PhD supervisor and I had a very frank conversation at one point about halfway through my PhD and when we decided mutually that my future did not lie in academia. And so yes, from that point she was happy to try to get me out of academia into industry and I started working professionally as a computer scientist since then.
04:37 Michael: I was in a PhD program and I I know you feel like you're on the path to just keep going into academics, but I feel personally very lucky that I had a broader exposure around that time to see what was possible in industry, because for every opportunity in academics I feel like there's a hundred opportunities that are just so amazing, outside of the confines of a university and a grant. And I think it's great.
05:03 Russell: Yeah, for me it was a slightly more practical thing, as I actually liked doing things, not just kind of the theoretical aspects of it, and I was getting during my PhD I was getting very tied up in playing around with interesting pieces of technology with a view to actually doing something, but it wasn't actually furthering my research at all, it's kind of a precursor to a lot of the stuff that I'm now doing in Beeware in some regards.
05:25 Michael: Excellent, ok so where along the path did you come across Python?
05:29 Russell: I first used Python back in the 1.5 days so, a long time ago,like really nineties, at which point it was mostly a system config language, mostly because red hat was using it in their setup tools. At that point it was kind of, well this is an odd language without syntax, I don't want to use odd syntax, so let's not use it.
05:51 Michael: What's up with this white space thing, huh?
05:53 Russell: Yeah, well and that's the thing, it's the thing that everybody gets caught on but as I sort of grew as a software developer, that interesting property became more and more thing that I actually value in Python. So it's one of these things where once you've read enough incredibly badly formatted Java code, you realize how valuable it is having a language that literally cannot run unless you get the white space indentation right. And so yeah, over time it grew from being just the system's language I kind of have to use, to something that vaguely tinkered around with. Over the whole period, the sort of mid nineties through to about 2005 when I got involved with Django, I've had this series of little background projects that I've tinkered around with and whenever I wanted to learn a new language or a new framework, or whatever, sort of the first project is let's render that in this new language, and at some point, I got to the point where, "hang on, I can do this Python, can I, let's build this in Python and yeah, this is actually working really nice, I like the way this is coming out". That didn't get distracted by, "hang on, maybe this web thing is actually is a thing too, I can use this Python for the web, oh look, here is this Django thing which is brand new", and ten years later I've been still working on the Django project.
07:07 Michael: So you actually became and have been for a while a core developer on a Django project?
07:11 Russell: That's correct, yeah, so I was introduced, I've discovered Django because I was poking around, I had this vague idea that the web was a thing that maybe I should know more about and maybe I could use it to solve some of the technical problems that I had. And I found Django just before they did like the .91 release or around that sort of time frame, so very early days 2005, November 2005 effectively. And it was two things really happened, one because it was a really early stage project, the sort of the barrier to getting involved was a lot lower than it is today, because there's not as much sort of momentum and inertia around what's going on. But also, because it was written in Python; I come from a physics background, I have done theoretical computer science but like all theory it's utterly useless for actually doing things, I've never actually formally studied networking or anything like that. So when I tried to work out how to build a web page with PHP, the PHP website didn't actually help me at all, I didn't understand what it was trying to do, but the Django tutorial got me from, "I don't know how the web really works" to, "Oh yeah I can build a website and this makes sense now" relatively quickly. And when I hit the limits of what Django could do, because it was written in Python I could dive into it and I could modify it relatively easily, I understood what was going on pretty quickly. So within a month or two of picking up Django for the first time, I was submitting patches that would- if you've got a many-to-many relationship between two tables, back in the day you could only go from the model that defined it to the model that was pointing to. I needed to be able to go the other way, so I added that modification into the query language. And, when you start making changes like that, the core team Adrian and Jacob offered me the commit bit and the rest is history from there .
09:05 Michael: Yeah, that's really great. Do you feel like getting started and contributing to Django was easier 10 years ago, because it was a newer, less mature, less sort of solidified project?
09:17 Russel: Absolutely, I mean 10 years ago Django didn't have a backwards compatibility guarantee, there was certainly a preference to not just move things around because, but changing backwards compatible things was at least an option. There wasn't as much- the community was smaller, so it was easier to get consensus with a bunch of people, there wasn't the consideration of a million- if we deploy this change Instagram is going to break because we all of the sudden have changed this API. So yeah, ten years ago was absolutely easier to get involved, there's no doubt about that at all.
09:51 Michael: Yeah, that's really interesting. I think that's a big trade-off right, I mean the value of Django is probably higher now, but it's just all sorts of projects that have been around for so long. So, let's talk about a project that's not been around for quite that long, and that's your project, Beware.
10:06 Russell: Yes.
10:08 Michael: So, Beeware is interesting because it's kind of many things that work together sort of synergistically, and you said that your project follows the Unix philosophy for tools?
10:20 Russell: Yes.
10:21 Michael: Give us a quick overview of what Beeware is and then maybe touch on that?
10:24 Russell: Sure, so the place where I started with, the Unix philosophy is a talk that I gave at PyCon Australia, I think it was four years ago, I'd have to check the exact date. Essentially the theory that I was positing at that point was that the Unix philosophy as a process is quite strong, Unix philosophy is the idea that you don't build one tool that does everything, you build a large number of small tools, each of which has a well-defined interface, because the power comes from the small tools being able to plug together really easily, with having a natural way of plugging together. Now, the unfortunate corollary of the Unix philosophy has become that in order for something to be Unix it must be text-based. So yes, you can run LS and LS you can pipe into sort, and sort you can pipe into unique, and these are really powerful things to do. But at the end of the day, it's all text, and there are many areas where text is not the natural interface, and the example that I gave at the time was testing- if you run a unit test suite or pod test suite or whatever, what you get coming up on the screen as the user interface is .....f....x.... which is completely correct and completely accurate, but not helpful from a user interface perspective.
11:40 What you want to know is which . is running, why did it turn up an f, when something writes to standard output at the wrong time, which test did that, how long has my test suite got left to run. Now these are all questions that you're entirely in a position to answer, or at least provide good guesses for, in case of something like runtime. But when your only user interface is writing something to console, you've just hobbled yourself in terms of what it is you can actually display. So, what I was originally positing with a Beeware project was that what we can do is build tools that are still Unix related, they do one thing, they do one thing well, but one of those one things can be let's display a GUI that enables me to see all these useful things. So, here is where we started was with Cricket, Cricket is a user interface for running test suites, it shows a tree which is your test suite, you press go and it works through your test suite, highlighting the lines of the tree that are currently running as individual tests, as the test suite is running, you can click on other pieces of the tree and see what tests have run and how long they took to run and what output they generated, and whether they passed and failed. And because you know this is how long the test suite is, this is how many tests there are, each test takes an average of x milliseconds or seconds, therefore we can estimate that this entire suite is going to take 20 seconds to run.
13:00 Michael: That's really nice, I think the visual component of it is just, I'm with you, I don't mind working the command line, but we have such a powerful and beautiful OSs these days, I feel like we're using one-tenth of what we can in terms of interactability, right?
13:18 Russell: Yeah, I've bought a four thousand dollar laptop with a retina display that's able to render things at a resolution that my eye can't discern, and I'm using it to show 80 by 25. It just doesn't make any sense.
13:30 Michael: Yeah, it totally doesn't make any sense, but I think that taking this idea of the Unix philosophy apply to like a modern gui based set of tooling, and that's really powerful, and that's one of the things that I find really appealing about your project.
13:44 Russell: Yeah, and that's, like I said, I started initially with Cricket as a proof of concept because unit testing is the thing that I do a lot of, and I enjoy and the unit test suite is difficult to get into it, when all you've got is ..... so here is a user interface that lets you use interrogate manipulate a test suite as it runs. The other is I also had this another tool called Duvet which is a coverage tool, it takes the code results partially from a test suite but also from anywhere else that it's running, and shows you graphically which lines of code have been operated rather than giving you a line 17 to 25 were not covered; well, what's line 17 to 25? Oh it's that for loop.
14:22 Michael: Yeah exactly. 94% coverage is not the same as scrolling through your code and seeing oh there's a red spot, hmm, why is that red?
14:29 Russell: Yeah, exactly. But in order to get that capability, you don't have to swallow the whole of an ide you just have to have a mechanism for displaying that isn't 80 by 25 console.
14:42 Michael: Interesting. And so, one of the things when I was poking around on your projects, it seemed to be a theme, was they seem to be supported on all the major OSs at least the desktop ones, for the most part.
14:54 Russell: Yes, so that's the intention anyway. So, originally I use a Mac as my daily developer box, I do also do work on Linux mostly as a server capacity. I know I have many friends who are using Linux as a desktop and if I write a tool that is useful I don't want to lock them out. My passion for Mac is not because I particularly like Macs, it's because I like the hardware, I think they're there is a usable operating system it is a usable platform where I don't have to mess around if someone wants to use a different platform they should be able to, and my choice of tools shouldn't be affecting their productivity. I've been from the beginning very actively trying to do this as a cross-platform thing, plus also, from my perspective, one of these days I'm anticipating that I probably will end up leaving Mac and going somewhere else, and I want all my tools to keep working please.
15:42 Michael: Yeah, that would definitely lock you in, if you spent five years building an amazing set of tools but you had to stay on your whatever OS do you happen to have been on at the time right. You couldn't possibly leave.
15:53 Russell: And the thing is, at the end of the day so much of what these platforms are doing is not platform-specific, show me a menu, show me a button, show me a pull down, these are generic concepts, originally, so it started like Cricket is still and was originally built using a Tkinter mostly because it is available cross-platform and you can install it saying pip install whatever and it just works. And a lot of where Beeware is being sort of hunting the last sort of three and a half years has been Tkinter, the major limitation here is the TK doesn't have a web widget you cannot display a webpage in TK, and overtime TK is theming on OSX has gone worse and worse but that's kind of secondary, you cannot display webpage, and trying to write a new widget in TK is just pathologically bad. So it turned out for me the easy solution was to write a new widget toolkit, ironically.
16:44 Michael: Yeah I think there's a lot of cross-platform tools or toolchains but not too many of them seem to really focus on the truly native component. And that makes a big difference to me, I remember running some Java apps and just looking at like the file open dialog and go, "this just so does not belong here, what is this thing, it kind of resembles of a file open dialog, but no it's not, like the hot keys don't work" all the various things that you would expect right?
17:12 Russell: Yeah, exactly, there are cross-platform widget toolkits that work to various degrees of work, there are things like swing in Java, which sometimes looks like, or even sometimes doesn't quite, and then you've got things like gtk which yeah, sure works on Mac as long as you're running an x server, and then you get the loving user interface of an x server running on your Mac. And then there are things like a Qt and WX which are relatively mature and being used for a while and have Python bindings, but the Python bindings are kind of a secondary thought, in the case of a Qt there's a whole licensing thing that's going on that's just a complete mess, and they're not as simple as pip install widget set, because there's compiled components and libraries and you've got to install this 150M library and you've got to know which one you've got to have, and it just becomes such a nightmare that the basic thing, I just want this damn thing to run gets in the way of getting stuff done.
18:05 Michael: Absolutely the UIs in your apps look really nice and they are cross-platform, what did you do?
18:09 Russell: I accidentally stumbled upon a couple of things that I didn't realize were actually possible. I was looking a long time ago at PyGame and realized that PyGame actually manages to do this native thing, without actually having a compiled components and the way it does it is because objective C whether you know it or not, is actually a raw C API with some pretty syntax on the outside of it. And Python gives you this thing called C types which lets you call any C API without having to compile something. So, with a little bit of Python class magic and metaclass and descriptors and what have you, you can write a class in Python that is a pure Python class that looks and feels and behaves in the objective C environment like an objective C class, and then when you look at the next option you look at gtk, gtk provides native Python bindings that are actually native Python bindings. So, there you've got two major platforms, where you can write native applications using the native APIs with pure Python without having a binary component involved. And then from there it was kind of a well okay, that means I've got coco and gtk which are the two that I'm sort of covering my bets on, can I do this on windows- well, yeah, because there's a win32 and that's a C API, so that works. What about why is anyone going to use this new platform, it's nice that it doesn't have this extra compiled component, but that's not sort of compelling, what's going to be compelling, and this was again three years ago, it was like well, what if I could use these widget sets on mobile as well, a lot of what I've been doing the last couple years is trying to get Python to run on mobile, both on Android and on iOS.
19:55 so that you can actually then say, yeah I can run this, this is similar API or identical API, on macOS I can run it on gtk on Linux, I can run it on iOS and I can run it on an Android. And that sort of, that's what's been keeping me busy for the last couple of years at this point.
20:11 Michael: Yeah, and that's quite the achievement, especially the mobile part, but I think, you kind of went through a quick but I feel like native on the three major desktops is actually a bigger deal then maybe it sounds at first. I think if you're going to build an app and if it can look and behave literally like a macOS app or a Windows app or a Linux app, that is a really big deal and if you can use Python for that- that's even better. Now, one of the projects, and I'm sorry I don't remember the name of the project, but one of the projects you're working on is bundling that up so you can distribute, you can actually ship sort of a bundled up version of Python with everything it needs to run your app, right?
20:54 Russell: Correct, yeah that's Briefcase. The idea of Briefcase is to say well okay, a macOS application is basically a directory with some metadata in very specific places. And historically, giving someone a tool that is written in Python is kind of well you need to install Python but you need to install it from here and you need to give it your system password, and then you've got to hope that someone hasn't buggered up the site packages directory and and and-
21:22 Michael: Here's the dependency, install those too.
21:24 Russell: Yeah, and then you sacrifice the chicken and waive the blood this way. And hopefully, it works.
21:31 Michael: Unless you're on Windows, then you better find out-
21:34 Russell: Exactly, because every Windows user knows how to set a path variable, but at the end of the day, we're now at the point where a 100Mb application is absolutely nothing, 10 years ago that was a major drama, but now that's just getting up in the morning. So, why not ship the version of Python you actually need with the dependencies you actually need and when you run it, it runs it locally, and so well yeah, I can do that I can template out, I can use cookiecutter, which is a fantastic project from Audrey Roy Greenfeld and Danny Greenfeld to like template out what this application needs to look like, and then your partner script goes here and your dependencies go there, and we can pip install into that directory, zip it up, ship it to someone, and it just runs because it's pure Python, you're giving them the Python interpreter it has to use with the dependencies that they need, and it's running natively. And then from there, you can do similar sorts of things, I haven't done as much work with Linux because installing apps on Linux doesn't quite work as well. I've done some exploratory work with Windows and it should absolutely be possible on Windows, possibly to the point of like actually generating an MSI installer that looks like it's a completely native installer it just happens to spit out something that's not in Python interpreter but you never know that.
22:43 Michael: I think the irony is, we in the Python community will be successful on these desktop systems and mobile systems, if people don't know that they're running a Python application.
22:56 Russell: Absolutely, I completely agree. So, as soon as you know, or you can tell what tools someone used to build something, I think you've lost the game. You need to be able to run a piece of software and just be able to give it to anybody, and it will just start, it will just run, it will behave the way they expected to and they don't get any surprises. Anything less than that, and okay, it may be very useful in various areas or demographics or whatever, but you're not going to get the mass market adoption if you've got to hold your breath and lift your leg in a funny way on a Tuesday when the moon's full in order for it to work.
23:29 Michael: Yep, it might work for developers, it might work inside your company, but it's not going to be something you can broadly distribute, if that's the story, right?
23:36 Russell: Yeah, exactly. But even beyond that, looking at the developers themselves this is part of the great lamentation about why I got involved with Beeware in the first place, you should be in the situation where you know, you have to be this tall to get onto this development project, if you can't just pick up a tool and use it, and pick up a tool and develop it, then you've just set a barrier to entry to anybody helping to make your project better. So the dream of open source is that anyone can get involved, anyone can make their project better, anyone can contribute a patch, but if I can't even get a development environment going without 10 years of computer science experience, then what the hell chance has the random person who has no computer science background got to fixing the tool, they don't. Being able to contribute open-source dream is at best a dream.
24:22 Michael: Yeah, and then you end up with people run into things like JavaScript because, hey we can all start a browser, or something like that, right?
24:28 Russell: Yeah, exactly.
24:28 [music]
24:28 This portion of Talk Python To Me is brought to you by Rollbar. One of the frustrating things about being a developer is dealing with errors, relying on users to report errors, digging through log files trying to debug issues, or a million alerts just flooding your inbox and ruining your day.
24:28 With Rollbar's full stack error monitoring, you'll get the context insights and control your need to find and fix bugs faster. It's easy to install, you can start tracking production errors and deployments in eight minutes or even less. Rollbar works with all the major languages and frameworks including the Python ones such as Django, Flask, Pyramid, as well as Ruby, Javascript, Node, IOS and Android. You can integrate Rollbar into your existing workflow, send error alerts to Slack or Hip chat or even automatically create new issues in Jira, Pivotal tracker and lots more.
24:28 Rollbar has put together a special offer for Talk Python To Me listeners- visit rollbar.com/talkpythontome, sign up and get the bootstrap plan for free for 90 days, that's 300 000 errors tracked for free. But hey, just between you and me, I really hope you don't encounter that many errors. Rollbar is loved by developers, awesome companies like Heroku, Twilio, Kayak, Instacart, Zendesk, Twitch and more. Give Rollbar a try today, go to rollbar.com/talkpythontome.
24:28 [music]
26:05 Michael: Yeah, so you said, well, your website which I'm presuming you had to say what it said, says that the end goal of the Beeware project is to be able to do for mobile and desktop facing software the same thing that Django's done for web software. I think that touches on the point you just made, that like Django, the Django community, Django girls all these different groups seem to have found a way to make it really easy to onboard into dynamic server-based web development; which is not really a simple thing to do, to be honest, there's a lot of moving parts and things you've got to know.
26:38 Russell: Yeah, and that's the Django girls analogy is really the one that really is by yardstick for when this is successful, because what the Django girls have done, and the rest of the detainment with the Django girls as it's been growing has been absolutely amazing because it proves that you don't need to have a computer science degree to be able to do things on the web and to do really meaningful things on the web too. You can take people who are intelligent and engaged with no computer science background and in a day's training course get them to the point where they have a database backed website that is under their control, publicly visible somewhere on the internet. Now there's at some level, that's an absolutely huge achievement, because this is incredibly empowering for those people, but it also having APIs and part of the reason it's possible is because we've got an API we've got Django, we've got a language like Python that is readable and gives reasonably good error messages and enforces reasonably good practices; but we haven't done it by taking something and dumbing it down, we've taken the live tool and just pointed them in the direction of things they have to do. That's powerful.
27:49 Michael: That's super powerful. I think that's also one of the powers of Python is there are simple languages out there, Visual Basic for example, that is super easy to get started with, but they don't go very far; And then there's really powerful languages like C++, Java, C#, whatever, but the first step is like a really big jump. In Python, I think it's just really well designed that it actually spans both of those, and so I'm kind of lucky to be in the space as well.
28:21 Russell: Yeah, I know I agree, Python is a language that is easy enough to teach to beginners, and simple enough and conceivable enough that you can 'hello world' is one line of code, you don't have to teach a whole lot of computer science concept to get going, but that doesn't mean the computer science concepts aren't there, you can do asynchronous decorators and all this kind of really hard core computer science stuff without, but you can opt into it as you need to, over time, and you can get basic stuff done and Django, in something like Django you can provide an API that lets you get stuff done with most of the important, sort of the difficult questions answered for you, but with an option in the future that if it ever starts to become a problem you can dig in and replace that decision with something that's going to scale better, or perform better, or whatever the problem happens to be.
29:09 Michael: Yeah, absolutely and we have a lot of good examples, like hey you can build dropbox or Instagram or whatever, like okay well, whatever you're trying to do is probably not that complicated, so perfect, this will work for you.
29:20 Russell: You could scale from Django girls to dropbox or to Instagram without- yeah okay, yes you've got to level up along the way and the Django that's being used in Instagram is not the same Django that's being used in Django girls in some regards, because so many parts have been swapped out, there was a quote from Cal Henderson I met at the very very first Django con where he sort of points out the frameworks, the companies that are at that top of that performance treat, the top 1% of websites have very unique requirements because they've got very specific traffic patterns and very specific queries that need to be issued and there are some things they have to do a lot of the other people don't have to do a lot of. The good news is that 99% of websites are not in the top 1% of websites. So, for most of those websites, you can just take a common entry point, give them something generic and as long as they've got the power to move over time, then it doesn't matter that you're hiding the detail from them initially.
30:13 Michael: Yeah, yeah that's really great. One of the things that I thought would be fun to do for the audience would be to look at all the different projects that you have going on, because I look to it here and I was just blown away at how many different things you know, following the Unix philosophy that you pointed out, cool little tools that fit together that you guys have here; so maybe we could do like a quick survey of those?
30:34 Russell: Sure, okay, it's interesting that this is actually, reactions would be the wrong way of putting it, it's sort of lessons learned from being involved in Django. The Unix philosophy here is not just about a single tool that does something well and therefore makes it easy to use and powerful to plug into something else, it's also a contribution aid, because if you've got a tool that you know only does one thing, and only does one thing well, the codebase is inherently smaller. Overall the amount of lines of code that are ultimately going to end up in everything that's Beeware is probably going to be as big if not bigger than Django I imagine. But, if you want to just contribute to the coverage tool, you can look at just the code for the coverage tool, you don't have to carry the entire baggage of all of Beeware to be able to understand that one piece, so it makes it easier to consume peace and which makes it easy to get involved, but yes, so a survey of what we have.
31:27 Russell: Sure, also I think that also is helpful from the project management and scope perspective, like somebody proposes a feature for cricket the test runner, l like it really it better be about running tests right, and not like an editor feature or something, like if this was some giant project that did all these things, maybe it makes sense to add this funky drag-and-drop editor thing that somebody is proposing or whatever their idea was right?
31:50 Russell: Yeah exactly. And you can say like yes, this should absolutely be a tool it's just not cricket because cricket is a test run.
31:58 Michael: Excellent, perfect. So just for people listening, you can find all the stuff at pybee.org. And so, let's talk about some of the tools like you built some tools for actually performing checks when people do poll request and you call that Beefore with 2 e, nice.
32:15 Russell: Yes, we're big on puns as well, so there's almost always an inside story or background geg, the person who suggested Beefore to me suggested at the same time a different name that I really am kicking myself I didn't pick which was apis priori, the Latin for bee a priori - apis piropri, but yeah, Beefore is, we haven't actually got that one live yet, basically we're looking for a way to actually run that and integrate, but it's essentially, all it does is listens for a pull request trigger, runs in the same way that you're- pythonsetup.pytest, runs your test suite. Python setup.py beefore will go through and run linters, it'll go through and run spell checkers, it will go through and run any of the pre-commit checks that you might have, which is also not something that's particularly uncommon, but the thing that's different is that it integrates with github so that if you look at like your github pull request if anyone is integrated with something that Travis it will run your pull requests, so run your pull request through travis, run the test suite and then if it passes the tests, give it a little green tick. Beefore will also put a little green tick to say and the linting is fine and the spell checking is fine and this, and this, and this; so just as a sanity check, before anyone pulled or even reviews a pull request, we can go through 15 different checks if we need to about what this code needs to look like and provide the automated feedback to say oh you know, hang on, you've got an indentation problem here on line 17, and a spelling problem in your documentation over here, and so on, and so on. That's actually taking a suggestion from the Rust community that they've discovered, I don't know if it was accidentally or intentionally or if there's something behind it, but if you've got feedback that needs to be given, that is sort of mechanical in nature, it is better for that feedback to come from a robot because people understand that robots don't have emotions and aren't being, they're not picking on this because you're a human, they're picking on this because it's not fitting a rule and so they will work with it. It also means it can be automated which is one more thing that you don't have to do as a developer who is reviewing this code, if I look at Beefore approved patch I know that it passes all the tests and it has been lintered and it doesn't have any spelling mistakes and so on.
34:29 Michael: Yes, great so you save your energy for reviewing things like algorithms and the checks that should be there and whatnot right? Cool, so the next one that's on the list is Duvet, which is your coverage visualization tool, we talked about that. And then there's Briefcase that's the one that lets you package your Python App into a standalone application and I just can't get over the possibility of how cool that is. I know there's been like py2exe and cx_freeze and py2app and they all seem to like almost work, and it's just driving me crazy.
35:03 Russell: Yeah, and that was kind of my- like I'm aware these projects exist and a lot of them work really well for Windows, but don't work on any other platform. So you know, in some cases I will admit I haven't actually done a lot of the packaging work here on Windows and I haven't done any on Linux at this point, but the approach that I'm sort of taking there with the way Briefcase is working is basically well here is a template for what an app needs to look like if only I knew the one script I need to run, put my one script here and the rest goes. It's also, [35:34] got a thing for iOS and Android because in order to get an iOS project running, you can't just drop a Python script somewhere, you actually need to have this great big iOS xcode project and most of that can be automated.
35:46 Michael: Yeah, it actually creates an Xcode project file, right?
35:49 Russell: Yes, exactly, and the intention is in the not-too-distant future hopefully that it will create the xcode project, compile the xcode project, push the xcode project to the device and start it.
35:59 Michael: Yeah, that's fantastic. Like I said, I'm very thrilled about this and this is going to require lots of attention because it looks cool. And the other one that we talked about that's next on the list is Cricket, so that your GUI test runner, and that was your first [36:14] in this whole Beeware plan right?
36:17 Russell: Yeah, that's where I started as a proof of concept that yes you could build a cross-platform GUI tool that was useful for development purposes that you could pip install into your development environment and just use. What is interesting there, that sort of pip install thing, if you've got like test runners aren't unusual in ides, but in order to get them to work like you need to know where PyCharm puts its path setting, you need to know where the environment variables go for the test environment to start up. Django's context is you know you need to set Django settings, you need to know where the Django settings file is, so that the application can start. Working out where to put that in your ide is a complex thing whereas one of the interesting side effects of Cricket is because you pip install it into your environment, when it starts in forks from inside your running Python environment, so you inherit all the environment that it needs to run in. So as long as you can run a test suite, as long as your environment is set up, the way that your command line environment should be set up, Cricket just works, you don't have to, there is no configuration dialog for Cricket.
37:21 Michael: Yeah, that's awesome. I really like it, and you know, the playful icons and the fun names really make it like enticing the ghost [37:31] for these. So the next one speaking of that, is called Bugjar.
37:33 Russell: Yeah, and I'm actually surprised that name hadn't been taken anyway because it seems like an absolutely gimme for a debugger. That's actually like if you dig back into the archives behind before Cricket Bugjar is actually where I really started because what I really wanted pdb is an incredibly powerful tool pdb is the API the way it all works is fantastic. The pdb interface to my mind is toxic and I know that there is I pdb there's pdb ++ and there's a bunch of other tools out there, but they're all console driven and I know there is win pdb but I've never been able to get it to actually work, what I really really wanted was the debugger that was like the one that I had in 1988 when I first got my copy of ball in C but for Python, and I just got a little distracted along the way eventually, I'm going to get back there I swear.
38:24 Michael: That's a great idea, and I really appreciate a visual interactive debugger myself as well. and I know you can do it in the command line, but it's not the same when you're working on a large project in my opinion.
38:37 Russell: Yeah I mean the visual scope and contexts you get by having an arrow pointing at a line of code with a window on the side that says the value of x is 3 versus a command-line interface where you have to type 15 commands to get the same amount of information, it just it seems like a no brainer to me.
38:52 Michael: Yeah, absolutely. So the last one is kind of a quick documentation one called Galley?
38:59 Russel: Yeah, the name on that may actually need to change because we've lost the names space PyPi that was the one that finally pushed me over the edge with TK because TK doesn't have a web widget, I write documentation, I like writing documentation but I appreciate the documentation is important and I need to write it. But Sphinx, as much as markdown and restructured text try to look like markup, they don't. So, you end up writing a bunch of code, compiling it, oh I missed it, I didn't put, there's 1 less underscore that I needed on that particular thing, or I forgot the colon after this bit. So, the idea of my Galley was to provide you a text window on one side where you edit your text, and then actually render it so you can see what it looks like as you modify it. But in order to do that, you need to have a web browser to actually display the content of the code, or the content of what is being generated and you can't do that in TK. The other interesting thing about Gallery is that it actually takes me a little bit closer to the thing that got me involved in Django in the first place, which is an editing tool; there's a whole other podcast in the tool that in my dream of dreams I'm eventually one day going to write that is sort of an editor that is context-sensitive or syntax sensitive, but, yeah that's a talk for another time.
40:16 Michael: Alright, awesome, it sounds great. Alright, so that wraps it up for the tools, let's talk about some of the libraries, and these libraries are each one of them is really ambitious, and it's just one more library, so that's pretty cool. So, the first one is Colosseum; this has to do with the layout and putting stuff basically on a screen, is this for the web or is this for your Python desktop app?
40:40 Russell: This is actually targeting, the reason I started that project is specifically because of Toga, when I started Toga, the original plan was to use something called Cassowary which is if you've ever programmed for iOS it's the constraints algorithm that underpins iOS, so effectively it's a linear algebra approach to describing how things, how widgets appear on a page you know, if you've ever written any widget, any gui code for a native application, you effectively have to say I've got a button and the button, and there's a split pain [41:13] and the split pain is this and so on, and so on. And there's usually some box model it's involved which you know, here is a stack of vertical boxes or here is a greed of some description. The Cassowary is trying to do an algorithmic approach which says I'm going to specify how my desktop lays out, as a series of linear programming constraints. So the left side of this button must be 10 pixels from the left side of the screen, but the right side of the button must be no more than 10 pixels from the right side of the screen, and so on. And then you run cassowary to solve the linear programming constraints and then your widgets appear on the screen. Now, that's incredibly powerful, but it's also an absolute pain in the butt, if when it goes wrong.
41:55 Michael: Yeah, I tried to debug the iOS stuff, I was like what's good on with this, why is this not working?
42:00 Russell: Yeah, when it works, it's great, when it doesn't you are screwed. So, prompted mostly by react, they sort of pointed out that you could lay out like the css box model, yes it's intended for the web, but is actually nothing web specific about it per se, it's basically just a set of algorithms for how you put boxes of content and related boxes of content in a tree like structure. So, what I said originally ported just the version of flexbox that that reactors has developed, and the intention is to put full css2 plus flexbox plus any of the other interesting pieces of the CSS spec, that you can use for any boxes. Now, obviously that immediately translates to buttons, widgets, on a page, but you could also do it things like okay I need a HTML to PDF translator; well okay I can take the DOM tree that I've got and use that to turn out a set of [42:57] instructions, so the intention is that ultimately Colosseum is going to be powerful enough that you can throw an arbitrary piece of CSS and some tree of content and be able to render it. The interesting side effect of that causes that when you actually want to take Toga and deploy it to the web which is something I have spoken about on Django con Europe this year. It means that you've actually got CSS so you've got the same layout mechanism on the web that you're using in your native gui toolkits, so you don't actually have to have a separate layouts game working out there.
43:31 Michael: Yeah, that's cool, I really like it, that's quite ambitious. I definitely think CSS is a fabulous way to lay out things. I've used a lot of Windows rich gui type of layout and I've always gone, if I just had CSS this thing would be maintainable you know?
43:48 Russell: I will admit I do have a bit of a love-hate relationship with CSS more generally, but it's sort of as in terms of a general, it is something that is well known, it is something that is well understood, it is something that is well documented, it is something that is open and that's sort of combination of stuff, but the only it makes it really powerful, but the only thing we don't have is an open implementation that isn't tied to a browser. At some point I need to sit down and spend a whole lot of time just working on Colosseum, at the moment it's fairly limited to what I can do, it is useful enough but not completely CSS.
44:20 Michael: Sure. Yeah. I think part of that frustration is projecting my own frustrations I've had with CSS and things like that, and the design is more about when the implementation varies, it's like why does IE have a line here, a border, and why does Firefox have a space and Chrome has neither of these, like that's not really a frustration with CSS, that's just like with the browsers.
44:44 Russell: Yeah, for sure.
44:44 [music]
44:44 This portion of Talk Python To Me is brought to you by Hired. Hired is the platform for top Python developer jobs. Create your profile and instantly get access to 3,500 companies who will work to compete with you.
44:44 Take it from one of Hired users who recently got a job and said, "I had my first offer on Thursday, after going live on Monday and I ended up getting eight offers in total. I've worked with recruiters in the past but they've always been pretty hit and miss, I've tried LinkedIn but I found Hired to be the best. I really like knowing the salary upfront, privacy was also a huge seller for me."
44:44 Sounds awesome, doesn't it? Wait until you hear about the signing bonus- everyone who accepts a job from Hired gets a $1,000 signing bonus, and as Talk Python listeners it gets way sweeter. Use the link hired.com/talkpythontome and hired will double the signing bonus to $2,000. Opportunity is knocking, visit hired.com/talkpythontome and answer the door.
44:44 [music]
45:54 Michael: So, you mentioned Toga a couple of times, and Toga is a Python native OS native GUI toolkit? Tell us about that.
46:00 Russell: Yes, so that's essentially the bit that comes before the actual application. So the philosophy we've kind of taken is that native is good, native is what
46:00 you're looking for you want something that looks native, feels native, it's not pretending to be agnostic, it's not pretending to be the platform that it's not, it gives you if you want a button, it's a button the way that a button looks on that platform, and it sort of does that by being Python first, Python native, which means that you're not looking at an API that is very obviously C wrapped in Python, it enables you to do things like the classic example is whenever you've got a long-running task in most widget toolkits, there are C or some variant thereof, and so you come up with some small quantum of work that can be evaluated, and you push that quantum of work into some form of timer in a separate thread that gets evaluated at some regular interval. In Toga, you use the same function that's a call back, except you make it a generator you make it yield, regularly, and when it yields what it does is it yields to the event loop and then it resumes inside where it was previously. So, you can exploit the fact that Toga gives you, Python gives you this mechanism to resume where you were inside a method, which isn't a capability you have inside Java and C and C++, and so on, to produce an API that is rich and flexible, you know, when you want to specify list of something you just give it a list of something, when you want a string you just give it a string, and it works out how to make that render in a native fashion, in a native appropriate fashion.
47:36 Michael: Yeah that's really amazing and if if you want to try it out, anyone listening can just pip install Toga, and then toga-demo right, and then boom, you have a running gui right there, it looks totally native to me.
47:50 Russell: Yeah, and it is completely native, because it is, when Toga demo runs it is firing up on Mac, it's an ns window with a ns split pain, with an ns list where it is stable and so on, and so on, when you click on the button that's a dialogue it's an ns alert box or whatever happens to be. And then if you do exactly the same on the gtk it's a gtk window with the gtk split pain with the gtk button in it, and so on, and so on. So if you look behind us it sounds like it's incredibly complicated, but the actual code behind Toga is relatively simple once you've got the bridging to the native [48:21] done on gtk you get there for free because there is a native Python API, and Rubicon, which is the library that lets us get to where objective C is complex but always completely transparent because you just sort of name the classes and name the methods in a slightly funny way, and you can access them. So the native Toga implementation of a button on macOS for example is about 30 lines of code and it's 30 fairly straightforward lines of code, so adding new widgets is relatively simple, and at the end of the day if you do actually want to get into a widget that is very Mac Os specific you can just drop into that level.
48:55 Michael: Ok, yeah, that's really great. Toga looks really amazing, and like I said, I think building out these truly OS native things in Python, mix that with the Briefcase, very lovely right? So, the next one, I'm sure I'm going to mispronounce so can you hit for me?
49:12 Russell: Okay, so that's Ouroboros.
49:14 Michael: Ouroboros,oaky.
49:15 Russel: Yeah. So, Ouroboros, for those who don't know the Greek mythology it's the image of a snake eating its own tail. So, the the idea here is that if we're going to, a part of [49:29] is trying to be cross-platform, so we've got it on MacOS we've got it on gtk, we've got it some parts of it on windows, but they're all just kind of standard Python running in standard Pythonic kind of ways. We've got Python running on iOS and that's actually just CPython compiled for iOS, because iOS is as much as they try to hide it, C under the hood and clang under hood. But on Android, they really, really, really want you to be using Java, so it kind of had to do a bit of an interesting shim [49:55] to get there, which I am sure we'll come to. And, for the Python to the web, if you want Python on the web well you've got to have it written in Javascript, in order to do that, you need an implementation of the Python standard library written so it works on that platform. Now a lot of the Python standard library is written in Python, but there are some parts that aren't, so things like the underlying core of datetime, NK time and time structures t …..things like that are native concepts. So you've got to have a native implementation of those concepts that the rest of the standard library can then build on. And so, Ouroboros is attempting to build as much of the standard library in Python as we can possibly build cribing from CPython when we can and then filling out the rest with, the interface, the API that you need to implement in order to have a datetime working inside of the web browser, or working on an android phone.
50:49 Michael: That's really amazing. Have you combine that with PyPy?
50:52 Russell: I think we have actually taken some pieces with PyPy and I think there is probably is some crossover there in terms of what we're doing, what they're doing. PyPy benefits from having a pure Python version of any module because they can then optimize the entire vm rather than just optimizing parts of it. At some point I don't know if we'll merge with PyPy, PyPy merges with Ouroboros or whatever, but there is a piece there that I think can be shade at the very least.
51:17 Michael: Yeah, absolutely, that definitely seems like that would be great, because if you have it in pure Python and you feed it to PyPy, it should go nice and fast, theoretically .
51:24 Russell: That's the intention. Yeah. And, that is that is ultimately what PyPy is doing a lot of the time, they have re-implemented parts of what is written in C in Python, so that it will run fast because they can pass the full PyCode optimizer over everything is doing rather than just part of what it's doing.
51:41 Michael: Right, absolutely. So that brings us pretty naturally to the next major category which you call Bridges. I love the subtitle "boldly going where no Python has gone before". So, the question from before might be like, well alright great, it's a pure Python standard library, but then you have Batavia which would work really well with that, right?
52:04 Russell: Yeah. So the idea here is that Batavia specifically started as a bit of a joke at PyCon Australia Sprints last year, I don't particularly like JavaScript, I accepted that it exists and it is the lingua franca that is on the web and it's not going anywhere and I don't want a language shame anybody for having knowing Javascript, it is an important language to learn, to know, and there are some very valuable things in there. But, I would be a very happy man if I never had to write another line of JavaScript in my life. So, part of that is that if I'm going to write web software and I want responsive front-end web software, I need to have Python running in the browser; or alternatively, I need to rewrite everything I'm doing on the server in Javascript, which given that I don't want to do that, I need to get Python into the browser somehow. So, what Batavia does is, take a look at the fact that when you compile CPython- Python is a whole, if you want to have a complete Python compiler and then interpreter and everything, that's a big thing to build. And there are people who are doing that, Brython and Sculpt both do that, PyPy.Js takes a slightly different approach-
53:05 Michael: I had Ryan form PyPy.Js on and there's a bunch of cool stuff that is happening, but a lot of those projects are really large or they make big trade-offs, in terms of support.
53:14 Russell: Yes, PyPy.Js I don't know exactly where it's at in terms of size now, but the first time I looked at it was like 20 MB as a compressed download and that's fine if you've got that already cashed, but if you want to build your next Instagram, downloading a 20 meg bootstrap is just not a candidate. So, Batavia treats it at a slightly different angles and saying instead of we don't actually need the full Python interpreter, or the full Python compiling toolchain with a repl and everything else, what we need is the ability to run Python code, CPython compiles to bytecode, bytecode is a simple virtual machine with a 150 odd instructions most of which are push, pop, add value, subtract value, this kind of thing. Can we implement that virtual machine in JavaScript and then just give it Python bytecode; and the answer is yes, and it turns out to be relatively small amount of bytecode as well, you can get Batavia like a fully functioning Batavia instance running in something like 10 kilobytes of compressed Javascript, and I'm sure there's all sorts of opportunities to optimize there as well.
54:15 Michael: That's amazing, and that's totally acceptable for downloading 10, 15 K of Javascropt.
54:19 Russell: Yeah, so that's less than your logo image in most cases, it's easily an achievable goal. And all you're really losing, is really a performance at that point, because it is CPython that is running, it's just running in the browser using Javascript to emulate the pieces that can't be done natively. The other thing I have to share here is, the part of the reason that I know this or I thought this joke actually might be plausible, is a blog article/ book chapter written by Ned Batchelder and Allison Kaptur called "Architecture open-source vol 4" which looks at sort of a bunch of complex problems and says, okay let's solve this problem in 500 lines of Python. And Alison's chapter in that book-
55:07 Michael: Yeah, that's a cool book, it's a very cool book.
55:09 Russell: Yeah, it's an amazing book, and in one of the chapters in there is "here is a Python interpreter, written in 500 lines of Python". And, that blew my mind when I read it, it's amazing stuff. But it also means, hang on, 500 lines of Python, that means I should be able to do it about 500 lines of JavaScript as well, and that's a bad date on [55:27] it's not that far from it. You need a little bit more bootstrapping to say like Javascript's only got numbers you need a concept of integers and floats and they need to have the right way, there's a little bit of bootstrapping you need to do around the outside, but it's achievable, that's the point, and you can do it in sub hundred K of code or sub hundred K of deliverable artifacts which means that if you've got like a, so you've got a Django server that has a form, and the form has a validation function at some point; that validation function is Python code it can be compiled as a code object, that code object has bytecode, that bytecode can be turned into a string, the string can be sent to the browser, the browser can then run that Javascript or so run that bytecode in the JavaScript version of the virtual machine, and then you've got not just similar or transpiled or whatever, exactly the same code running on the client and the server, and you've written it all in Python.
56:20 Michael: Yeah, that's great, so it takes the load off your server, because most of the stuff will happen client-side, and it means you write it once.
56:26 Russell: Yeah it's not just the load, the load is obviously a benefit, but it's also- if you want a rich user interface, you want to validate credit card numbers as they're being typed in, you can't do that server side, because this is something I get acutely aware of because I'm in Australia, you cannot change the laws of physics, a ping time from me to east coast of US is 300 milliseconds, you cannot change that, unless you are willing to accept a 300 milliseconds or longer delay in every user interface action, it has to be running locally. And if you can run it Python then all the better.
57:00 Michael: Yeah, very cool. Those two go together, great that's awesome. So that's Python and Javascript, what if I want to run Python bytecode on say the JVM?
57:12 Russell: Well, I'm glad you asked. So, the next project on the list is VOC, that takes a very similar approach to what [57:21] has done, but instead of saying let's tag a Javascript let's rewrite a CPython interpreter in Javascript, it says okay, CPython when it runs is a stack-based virtual machine. Java when it runs is a stack-based virtual machine, now Ok, to be sure, they are different stack-based virtual machines, they have different requirements, they have different needs, they have different operational characteristics, but they're close enough that if you squint you can turn one into the other. And so, that's what VOC does, VOC says ok let's take this Python code, CPython code, and compile it into directly the equivalent Java class files, not turning it into Java code that gets compiled, turning it into the class file directly. So the output is an a pyc file, the output is a .class file that then runs as if it was Java, because it is Java. And the sort of interesting little side effects of things like that is that when you write your Python code and it throws an exception for example, the exception can point at the line of Python code that generated it. Now, this sort of general approach isn't that unusual, you will get something like Scale, Scala is just a compiler that produces Java class files that happens to be a different functional language. So all we're doing here is effectively a compiler that takes Python input and spits out Java class files. That's critically different than something like Jython which is a full implementation of a Java virtual machine and Java compiler and all the rest, so when you run your Python code through Jython, it is Python code being interpreted in Java at runtime. Which is also- Jython is a great project as well, but it's a different approach.
59:04 Michael: Right, it's almost like instead of using C to implement the virtual machine you are using Java versus actually compiling, jit compiling the steps.
59:14 Russell: Yeah, and it's ultimately the reason that I've kind of started VOC is that you say it's written in Java rather than written in C, except for the parts that aren't, because there are parts of like the compiler toolchain, [59:28] and what have you, that are written using the Java native interface, and so you can't actually, at least not at the moment anyway, you can't compile Jython for android because there are parts that will not run on Android. So I needed a different approach to get things working on android and so VOC was born.
59:47 Michael: Cool, and VOC works on Android.
59:50 Russell: Yeah, the intention was originally it was targeted for android, but a side effect because Java is actually a cross-platform runtime environment. I've nocked out a demo Swing application, you can write a native Swing application to get native Swing APIs in Python, and it will run through Java and you can't tell that it's not Java. It turns out a Java API or Java user interface that looks awful because it's not native, but it is using Swing's native components; you could entirely possibly write- you can't do it yet, but conceptually you could take Django, compile it to Java class files, run it in a J2ee container and satisfy your enterprise deployment requirement that everything is written in Java, because it is, it's just written in Python in Java.
01:00:34 Michael: Yeah, they just care how it executes, right? Interesting
01:00:38 Russell: Yeah, exactly.
01:00:39 Michael: So you have done a lot of like scientific parallel computation type stuff previously and I'm always on the lookout for things that let me dodge the gill, does this let you dodge the gil?
01:00:51 Russell: Well, effectively yes, because it is on when it's compiling it's compiling for the Java virtual machine. So it is producing you know, the variables are all locked the same way they would be in a piece of Java code, so it is a gil-less version of Python with all the language- well, not all yet, because I'm still working on bits of it, but yeah, you can do anything you can do in Python, you can do in VOC. For example, you can do yield statements in VOC, there is no equivalent of a yield statement in Java, but you can generate the Java bytecode that does the functionality of yield, because it's essentially just assembly language at that point, it's just a Java assembly language.
01:01:28 Michael: Ok, very interesting. Alright final one on your Bridge is Rubicon.
01:01:33 Russell: Yeah, so Rubicon, there was originally a Rubicon Java as well which I was looking at using for android but turned out to not be quite plausible for various reasons, but it's mostly Objectives C portion, so I touched on this earlier, that Objective C, for all the funny syntax is ultimately just a series of C APIs, and C types which is part of the Python standard library, let you call any native C function as long as you know its prototype. So you can invoke any objective C API directly from Python without having to compile anything, and then Rubicon sort of wraps that up in a bunch of meta classes and whatnot using the Python 3 type annotation syntax to say, well this argument is an integer and this one is a boolean and it returns an object, map that to the underlying call in Objective C, and you end up with you know, you can instantiate an NS button the same way that you would in Python normally the only difference is that the message passing syntax you place colons with underscores and the method names get really long, but you don't have to pre-compile, pre-declaire, pre- anything about what is in your Objective C environment, you've just got to call the things you need.
01:02:42 Michael: That's fantastic. So you brought a Python 3, what's the story with Python 3 across Beeware in general?
01:02:48 Russell: Ok, so I have taken the wonderfully controversial stance of saying Python 2 is dead. I am developing a set of tools, I started developing a set of tools four years ago, four years ago I was developing stuff in Python 2 primarily, I am now at the point where I'm developing new runtimes, I'm developing new languages or new runtime environments new language interpreters, new libraries for cross-platform, I have made the decision that I am Python 3 and for the most part I don't care about Python 2. Now, if someone else wants to come in and care about Python 2, power to them, I won't turn down the patch, but I am aggressively targeting Python 3 and hoping that that will be enough of an incentive like [01:03:28] time Toga is mature enough that you can seriously do stuff with it that that will be enough of a reason for people to consider building new code in Python. And a part of that is also that it is also a new project, this is a new library, no one is coming to this, with existing codebase, so I can start with the new APIs and so I am.
01:03:47 Michael: That's this brilliant and on behalf of everyone, thank you for moving us all one step closer, that's great. How many projects do you have 15, maybe move us just 15 steps closer, I don't know, but you definitely moved us closer, so thanks. Alright, so we are kind of more or less around Beeware, there is a few more things on here, and one of them is Python Apple support which let me just read the description because I think it really brings home some of the power of what you have here: a meta package for building a version of Python that can be embedded into macOS,IOS, TVOS, or a watch OS project. Of course for Python, written in Python right, that's just so cool that all those OSs are involved in that, right?
01:04:24 Russell: Yes, so this is this is kind of one of these accidents that happens because Apple is trying to make a consistent development environment for themselves, we just happen to be able to horseshoe or crowbar into place. As Apple is pushing Swift they have historically pushed objective C, both of those are built on top of Clang. Clang is a GCC advanced in various ways, so it is under the hood a C compiler. The good news is that CPython is written in C and so with the application of a few patches, you can compile the default CPython sources for any of those platforms. Now the catch is that, and this is sort of a process of long negotiation with the CPython team, is that it's not just a matter of a patch that says hey look you can build for macOS, macOS these days only compiles for one platform, it compiles for X86. When you compile for iOS, you actually need to compile for, depending upon where exactly you're targeting, possibly as many as five or six different platforms, you need armv6, armv7, armv7s, i386, x86_64, arm64, and you need to have all of those compiled and turned into what's called a "fat binary" so it's like one library file that has all of those pieces. And there's a whole bunch of tooling that you need to build all the library separately and then mash them together with this thing [01:05:50 Lipo] to make them into one library. Now, that doesn't fit well into the CPython build chain at the moment, I've submitted a proposal for a patch that would do, that would enable you to manage that, but I don't know if that's likely to ever get merged, there is a whole bunch of sort of technical and political things that go on there. So Python Apple support is essentially just an external project that says, okay take the CPython source code, download it for you apply the patch that I know needs to be applied, build it, build all the dependencies, build Openssl and build [01:06:20] and all the other pieces that need to be built. And spit out a Python.framework that works on iOS.
01:06:28 Michael: That's great.
01:06:28 Russell: And the interesting thing is then, okay well having doing that for iOS, well you can do it for macOS as well because that's basically just, you only have to compile for x86. TV OS is exactly the same thing, except you've only got to complete from 64 and Watch OS is exactly the same thing except you've got to compile it for arm7k. So, it's the same basic infrastructure, you just have to change the key that you're calling into.
01:06:48 Michael: I see, just make it a little more Apple friendly, bundle that altogether. Awesome, and so one of your main goals was to build the Python toolchain that lets you write these UI apps and Android and the various apple OSs as well as Windows. And so you have a bunch of templates as well to help people get started right, these are cookie cutter templates which I haven't talked a lot about cookie cutter but it's really great project and I'm sure we'll have Audrey on the show some time to talk about it, but yeah. So tell us really quickly about your templates before we're out of time.
01:07:19 Russell: Sure, the templates are effectively saying, well you know okay, a Python project is at the end of the day an entry.py that contains print hello world but it could be more than that, but that's all you have to have, but in order for let me to run an iOS you have to have a whole lot more. You have you have your Xcode project and your dependencies, and your icons, and configurations and all this other stuff, but most of that information is relatively constrained, like we know we need the name of the package and the name of the package needs to appear here, here, here, and here, so the templates are saying okay well I know what format it needs to come out as, all I don't know is the name of the package I want to generate; so let's just leave that as a blank, generate it from template and then you've got something you can use. And then Briefcase takes those templates and takes the support libraries and things like that, precompiled versions of the support libraries and spits them out, you run Briefcase of your project, if you've got a project that has your helloworld.py file and it has a setup.py that says that's where the file is and these are the dependencies, it takes all that information,and uses that to generate the template so that you can then just say run this xcode project and off goes your iOS project, or generate the gtk application, or generate the Watch OS application.
01:08:33 Michael: Yeah that's excellent. Yeah, so I think your project is great, I definitely want to encourage people to check it out, we'll probably have to kind of leave it there for that one. So, one thing I did notice is that you have a couple of upcoming events that you're going to be at, were you just in Portland?
01:08:47 Russell: I was not in Portland for PyDX, no, I was in Portland for PyCon this year, and I'll probably be back the next year at this point.
01:08:55 Michael: Alright, yeah I was thinking maybe I just missed you like three days ago in PyDX.
01:08:59 Russell: No unfortunately not.
01:08:59 Michael: Yeah, ok. Maybe tell people about some of the upcoming events that you have, that you're going to be at in case they want to meet you or some of the team members or something like that?
01:09:08 Russell: Sure, absolutely, so I myself will be keynoting PyCon at a Python Brazil on the 15 th of October this year, and I'll be around for the Sprints there as well, so if you're in Florianópolis, please come and see me, I'm very keen to have new people involved. About two weeks after that I'll be in Amsterdam for Django Under the Hood, so that's the Django deep-dive conference that happens once a year in Amsterdam and I'll be there for the full sprint of that as well, probably working on Batavia in some of the more webby parts of the framework at that point. I'm also speaking at Linux conf Australia 2016 in January next year, down in Hobart if you haven't ever been to Australia I can highly recommend Hobart as a venue to go to, and LCA as a conference, it's a great little community, it's sort of the Australian equivalent, it's like the peak open source conference, despite the name Linux conf au, it is a multi-language, there are people that are talking about freebsd, it's kind of a an odd name that's kind of stuck around for historical reasons, rather than being actually just about Linux. So they're the ones immediately, there are a couple of other conferences that are I'm speaking at in the future, some of which I can't talk about, but watch your directories I'm coming to a city near you.
01:10:22 Michael: Awesome, yeah, spreading the good word, excellent. I also noticed that if people need the help you know, if they want to set this up and they've got a big team and they really want to get going, you also offer training so if people want to get a jumpstart then contact you, right?
01:10:35 Russell: Yes.
01:10:36 Michael: Then just maybe give a quick shout out to all the people on it, if you want to just pull up the list.
01:10:38 Russell: Yes, so I haven't specifically got like, it's not like you can come to me and say here's my red card for Python training or Toga training, it's more an aspirational thing about being able to fund the development of Beeware and Python development on a more sustainable basis. I have been working on this as sort of a part-time volunteer capacity for the last couple of years, I've had some amazing help from Katie McLaughlin and Philip James and Chris Swenson who have done amazing work as well, but they definitely are working in fully volunteer capacity, I've recently been started to be paid on a part-time capacity from company called Jump on software, to do bits of Beeware development but I would like to ultimately turn this into a full-time gig for myself and a small team of crack Python developers and also use that as an opportunity to reach out to get new people involved to reach out to do diversity in outreach, to bring interns and pay them for their time and actually have introduced them to open source development and once, you've got to give them a an internship with a crack team of people at the top of their open source contribution game, let them loose out in the world and see what else they can do. So, yes, a training is one access where I am definitely, I'm notionally available for hire, if someone wants to do that but I'm also open to all sorts of other options including if you go to pybee.org there are some options there for corporate sponsorship if you want to become a member of the project and drop somebody in the team regularly so that we can continue to do what we're doing. Because ultimately, the reason I'm doing this is because I like Python, I want to keep using Python, but I don't think we can continue to sustain large-scale Python development the way that it sort of has been on our wishing for the last 10, 15 years. Django development has not been anywhere near as fast or as motivated as it could have been, if we only had a little bit more money or if the Django project had a little bit more money. And given the amount of money that is available in the Django community, the number of people who are making millions of dollars with Django I think it's telling that we haven't been able to find a way to pay the bills a little bit more effectively.
01:12:46 Michael: Yeah, you know, I just wanted to sort of second that thought and say that there are so many companies that are many million dollars of revenue, or billions of dollars that take open source and just go, okay great this is free, this saves us money and there's not a law of giving back right, like if they just put a small portion, a very small portion back into this project, that would make such a difference, so yeah.
01:13:08 Russell: Yeah. I've had made a gag on Twitter a couple of times where, you know I'm offering corporate sponsorship in Beeware and a thousand dollars a month, it sounds like it's a lot of money to you and me as individual developers, but to accompany, it's like not even- it's around their soda budget, but if that's effectively a deci-Russell, a thousand dollars is a deci-Russell, you can get one tenth of me for a thousand dollars. The problem is I need 10 of you to agree to buy a deci-Russell and then I'm working full-time. Now, there are more 
than ten companies in the world who have a very vested interest in Python, I just need to work out who they are.
01:13:42 Michael: Yeah, absolutely. Before we move on to, before we sort of round up the show, there's always two questions I ask at the end, I'll send them to you now. So, first of all I think you may be able to pull a few from your own project, and it's totally legit if you want to do that, but there's over 80,000 packages in PyPi, what one do you think maybe people don't know about, you want to recommend, or say this thing is amazing you should check it out ?
01:14:08 Russell: If I may indulge, I actually have two.
01:14:09 Michael: You may.
01:14:09 Russell: They are not my projects, so I will completely be aboveboard there, there's no nepotism at all going on here, they're both in the sort of the developer space and they are tools that I have found incredibly useful, particularly since one of the downsides to Beeware being lots of little projects is that you have the process of releasing Toga is not trivial, there's a lot of things that need to happen, because there are a lot of pieces that need to be released, and each one of them has a release process, each one of which needs to be tested and all the rest; so the two tools, one is called check-manifest. It is a really really boring thing, all it does is you pip install it in your project, you run check-manifest and it checks that your manifest.in file matches what is checked into your version control, so that you don't accidentally release a package that doesn't have three files that have to be there for the source code to actually run and compile.
01:15:02 Michael: Ok, excellent.
01:15:03 Russell: It is mind numbingly simple, but the number of times I have made that mistake, it is paid for itself, it's paid for its free cost on PyPi. Many, many times over.
01:15:15 Michael: Alright that's number one, cool.
01:15:17 Russell: That's number one. The second one is defpy, which is a version of the Python package index that you can run locally. So it is a fully, it follows the full PyPi protocol and all that sort of stuff, and effectively it just lets you run a local PyPi server that you can talk to, so you can upload a package to PyPi, and see if it works without actually loading it up to PyPi to see what you've broken. So you can sit there and you can upload the half a dozen packages that are part of Beeware, realize that Oh, crap I've forgotten the dependency here between this one and this one, and as a result, repackage Toga version 021 so that it gets the dependency right, or misses the file if you forgot to run check-manifest this time, and tested out in a sandbox environment rather than pushing it to PyPi where once you have uploaded Toga 021 you can't upload it again, you have to release Toga 022, so yeah, that one has also saved my bacon more times than I care to count.
01:16:18 Michael: Yeah, that's a great recommendation. Alright, and favorite editor?
01:16:22 Russell: So, I am a Sublime text person, I like it but I don't love it, and this is one of these areas where one of these days Toga's going to solve this problem for me. I have an intellectual love of Emacs, but I can't bring myself to actually use it; partially because the [01:16:37] syntax does my heading, and secondly because the people who developed Emacs seem to be scientifically opposed to user interfaces that make any damn sense whatsoever. Every application on the planet has decided that command save saves on a Mac box, but not on Emacs. Ok yes, before the Emacs people come out of the woodwork, yes I know you can configure it, but it does not come out of the box usable for development, and that for me is if your tool ships in a form where it is unusable, then you have a problem. My classic one that I absolutely hate about Emacs is that Emacs ships with the tutorial that's fantastic, let's do the tutorial, it is something I need to not let do the tutorial. The first couple of pages of the tutorial teach you about the key combinations for page up, and page down- I have a button on my keyboard to do that. It's page up and page down, and they work really well, so stop trying to teach me how to use my keyboard. Actually teach me how to do the things in Emacs that are interesting and I know there are many things in Emacs that are interesting, but I can't get there because the onboarding process is toxic.
01:17:42 Michael: Oh boy. You also might have a home and an end button as well.
01:17:45 Russell: Yes, yes exactly, and you know, one of these days, the end goal for me I'll be a very happy person when I've been able to build a Toga- Sublime text is great because it's cross-platform, it's bad because there's no source code, it's not open sourced, so I want to write an open sourced programmable editor that is the analog of Emacs but programmed in Python, but open source and native user interface everywhere, that also just happens to plug into all the other really useful Beeware tools because I just need it to be an editor I don't need it to be a debugger, I don't need to be a coverage tool, I need it to edit things really well, and talk out to APIs of other tools when it needs to talk to other tools.
01:18:22 Michael: I think that's a great vision and probably a great place to leave it. So, it's been really great, is there any final call to action, are you guys accepting contributors or anything like that?
01:18:32 Russell: We are always actively seeking for contributors, so this is one of the things I'm moderately proud of in the way that Beeware is operated, we've got an open offer to anyone of any level of experience to contribute to be mentored through the process of contributing to Beeware, and that's not just sort of idle words either, this is something that is actually being fleshed out in reality, so at PyCon US this year, we ended up feeling I think at the peak there was something like six or seven tables full of people who are contributing to Beeware, when anyone contributes to Beeware, there's actually a little coin as a challenge coin, that we've had struck, and essentially as soon as you get a contribution into Beeware you get one of these coins, and you can only get it if you contribute to Beeware. But, we had people who literally learnt to use github two hours earlier who have now got commits in Beeware.
01:19:26 Michael: That's great.
01:19:27 Russel: We have been able to mentor people with very, very little programming experience to be contributors of non-trivial features in Beeware, so if you've got to the Beeware website there is a link in there for how to get involved with contributing, one of the big things that checks off is yes okay, this whole imposter syndrome thing, it's real and by the way the members of the core team have imposter syndrome as well, so yeah, don't worry about the imposter syndrome, we will walk you through this, we want to help you become the best developer you can, and will mentor you through that whole process. So get in touch, and we'll be more than happy to help you get involved.
01:20:01 Michael: Alright that's great. Russell thanks so much for being on the show, it's been fun to talk about a project.
01:20:06 Russell: It's been my pleasure, thank you very much for your time.
01:20:06 Michael: You bet, talk to you later.
01:20:06 This has been another episode of Talk Python To Me.
01:20:06 Today's guest was Russell Keith-Magee and this episode has been sponsored by Rollbar and Hired. Thank you both for sponsoring the show.
01:20:06 Rollbar takes the pain out of errors. They give you the context insight you need to quickly locate and fix errors that might have gone unnoticed in until your users complain of course. And as Talk Python To Me listeners, track a ridiculous number of errors for free at rollbar.com/talkpythontome.
01:20:06 Hied wants to help you find your next big thing, visit hied.com/talkpythontome to get five or more offers with salary inequity presented right up front, and a special listener's signing bonus of 2,000 dollars.
01:20:06 Are you or a colleague trying to learn Python? Have you tried books and videos that just left you bored by just covering topics point-by-point? Check out my online course Python Jumpstart by Building 10 Apps at talkpython.fm/course to experience a more engaging way to learn Python. If you're looking for something a little more advanced, try my Write Pythonic Code course at talkpython.fm/pythonic.
01:20:06 You can find the links from the show at talkpython.fm/episodes/show/78
01:20:06 Be sure to subscribe to the show. Open your favorite podcatcher and search for Python. We should be right at the top. You can also find the iTunes feed at /itunes, Google Play feed at /play and direct RSS feed at /rss on talkpython.fm.
01:20:06 Our theme music is Developers Developers Developers by Cory Smith, who goes by Smixx. Cory just recently started selling his music on iTunes so I recommend you check it out at talkpython.fm/music. You can browse his music there and listen to the full-length version of Developers Developers Developers.
01:20:06 This is your host, Michael Kennedy. Thanks for listening!
01:20:06 Smixx, take us out of here.