#10: Bringing Python to the Masses with Hosting and DevOps at PythonAnywhere Transcript
00:00 It's time to Obey the Testing Goat and fire up some Python in the browser. This is Talk Python To Me, episode number 10 with guest Harry JW Percival Recorded Tuesday, May 12th 2015
00:40 Hello and welcome to Talk Python To Me. A weekly podcast on Python, the language, the libraries, the ecosystem and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I am at @mkennedy, and keep up with the show and listen to past episodes at talkpythontome.com. This episode we will be talking to Harry Percival from Python Anywhere about hosting Python applications and making Python available to the masses, as well as his book in mission TTD in Python.
01:09 I'm thrilled to tell you that this episode is brought to you by Codeship. Codeship is a platform for continuous integration and continuous delivery as a service. We'll talk more about them later in the show. Please take a moment to check them out at Codeship.com or follow them on Twitter where they are @codeship.
01:26 This show is kind of special, because I've traveled all the way from the West Coast of the United States to London to meet with Harry in person; well, maybe I also came here for a conference in central London, which happens to be close to Python Anywhere's headquarters. Either way, it was really fun to do an in person interview with Harry. Now, let's get to the show.
01:48 Harry, welcome to the show.
01:50 Hi Michael, thanks for having me. I'm excited about it.
01:51 This is really an amazing show, it's the first one we're doing in person
01:56 Right. Ok, well I hope I don't mess it up.
01:59 No, you definitely won't. So, I'm here in London, I'm not normally in London, I love this place but I don't live here so it is kind of unique; and I happen to be at a conference like a block away from here- your office, where Python Anywhere is. Why don't we get together and do this in person.
02:16 And here we are.
02:17 Fantastic. All right. So, I really want to talk about Python Anywhere, I want to talk about "Obey The Testing Goat", and all these extra things. But let's just start at the beginning- how did you get started in programming, how did you get started in Python?
02:29 Sure, I guess I started in programming when I was quite young, like a lot of people, maybe when I was 10 years old, on various old bits of hardware from the- this is the sort of early 1990s maybe late 1980s, and I had like an Amstrad PCW, green on black background, I had like an one of the early Olivetti laptops and this thing had- it had not one, but two 3 and a half inch floppy drives
02:56 Oh my gosh...
02:57 No hard disc at all, and you could run you could run MS DOS on one and Microsoft Word, or a really good copy of snake on another one. And then at school, we had them as well so I was programming in Basic, I had a few books from America, and at school we had these things called Thomson TO7s
03:16 I've never heard of that thing-
03:16 Yeah, Ok, so it's like a big ugly box thing that you plug into a television screen and it's got these like rubber keys on and they go like "boop" when you press them, 10 go to toot, go to 20 and we program in Basic, and the cool thing with that actually was that they had this thing called input pen which is like this pen thing that you could press against the screen, and it had some woody then it would be able to register an xy coordinate against where it was on the TV screen. I guess it was calculating where the sort of rasta had got to or something like that.
03:49 It's pretty cutting edge for- when was this?
03:52 This was like 1992 I'm going to say
03:54 Wow. That predates Mozilla.
03:56 I didn't know those computers were already old then, to be fair... And so yeah, we would make like little drawing programs, on them, which is really cool actually, because if you watch kids programming these days on much more vast computers they've got really cool things like Scratch they don't have to go through Basic and Gotos. But they did exactly- I mean I watch kids in schools today and they say do exactly what I did at that age which as soon as you figure you can do some graphics on the screen, you start making a paint program you draw circles and squares and spirals and things like that and so... Yeah yeah, that sort of thing. I'm sure everyone's got the similar toy.
04:27 That's awesome. So, that's Basic, how about Python?
04:31 Yeah, so that was my origins where I was on the right path and then somehow my life went wrong and I started studying economics and I got a job as a management consultant, which was very tedious, and I didn't really do any programming other than kind of over complicated excel spreadsheets.
04:50 Until I was pretty much 30 years old. That was in like 2008, I took the sabbatical from my job which turned out to be just me quitting basically, go off and do a new degree and get a masters in computer science and just learn programming again. And do it properly and then, I went off and that's what I do now.
05:10 Nice. Was it really refreshing to step away from...
05:13 Yeah, well management is often-
05:17 It's a lot of talk, not so much action?
05:19 Yeah, yeah, I mean, so it's not for everyone I think, it wasn't for me. And it was interesting in parts, but I found myself like just if I looked at the part of the job that I enjoyed, it was mostly just making an over complicated Excel spreadsheet, and you know like just a 25 line functions in Excel and like VBA to do things that were like absolutely not required by the project but I thought were cool. So, there was a clear direction I should probably have been going into...
05:46 Instead of taking that job- You know, my programming experience is like that as well, I kind of became an accidental programmer; I was working my Phd in Math, I was working on some sort of graphics like super computers right, and I remember after about a year of working on this project it was just cool through these simulator that I really enjoyed working on except when I got to my math parts. And then after about six month I was like you know maybe I should take this as the clear that I should stop doing math and go doing computers. So very cool and nice to hear that.
06:19 So, I know you because when I was looking for place to host my website and get started I ran across Python Anywhere and I thought Wow, this is a super cool project you guys have, it is amazing and so maybe you could tell me about that?
06:32 It's only moderately amazing.
06:37 It's pretty awesome.
06:37 Well, let's see. So Python Anywhere it's kind of what is this- a place where you can do, you can code Python in your browser. And that could mean a lot of things. And what it means for us eventually, is like- I guess where we are really headed is to try and sort of be as Google doc system Microsoft Word maybe we could try and be that too, installing and running Python on your local computer, right? So if you have Python that runs in the Cloud, with apologies for the buzzword, then what can you do?
07:09 Well, so first you don't have to install any software on your PC, so you go to our website and we will give you a text editor which is going to work and it's going to have syntax highlighting for Python and even maybe some error checking with the PyFlakes. It's going to have a console just like you would have one on your own computer and just like line by line console but like a real terminal emulated, so you could have tab completion you could run vimeo or imax in it.
07:32 We will try and give you those tools that you are used to having on your PC in the cloud, and what it gives you is the ability to kind of run things from anywhere, and it gives you to do a more kind of collaboration things that are a bit harder to do on your own computer. And then, you can do some number crunching that maybe you can't do on your own computer because what we can do is maybe parallelization for you. And then finally, we can do web hosting. So you know, like we are already on servers, it's not much effort for us to then say "ok well we'll give your code some sort of public access to the internet", so you know, we end up having a lot of people just hosting websites in there.
08:07 You know that's really cool. Can you speak of what percentage of users are doing what type of use case.
08:15 It's hard to tell. Because we've got a lot of users, of free users who come along and then like any website they just come and check it out and go "cool" and never come back; and a lot of them will never talk to us, then we are not too sure what are they up to. But, all the ones that pay us and a lot of them do use it for web hosting, because, you know, well so, you know, we built this thing what it came out of was actually an online spreadsheet.
08:42 So the company I work for is called Python Anywhere now, but when I joined it it was called "Resolve a system" and what they were building was a Python spreadsheet. I mentioned I love spreadsheets earlier, and the reason I got this job is because I decided to do my thesis as part of my Masters on spreadsheets. What I thought I'd use I'd phoned a few companies that do spreadsheet software, and like other than Microsoft, right, there is only like a very few of these and they are small; and, I was just "Hey I'm going to do a thesis all about the spreadsheets because I'm crazy" Probably if you guys love spreadsheets like I do, you want to sponsor me, right?"
09:17 And there was one company stupid enough to say yes to that. And that was Resolve a System, and that's how I ended up working there, and they were making a Python spreadsheets and like- just imagine excel which works like excel except that when you want to script it you get to use Python instead of using VBA.
09:31 That's a nice place to be I would think.
09:34 It was really good fun. And a great tool, but ultimately if you want to try and tell people to stop using excel and start using this other tool, it's very high mountain to climb.
09:44 Yeah. You can Google, I think that would be some success with Google docs but even for them and their reach sets a hard story to tell so...
09:49 Yeah, so for a small like five person company out of London, they had delicate of falling of people who loved ...loads of fun. But so that wasn't necessarily working. Fist pivot was let's take that online. And let's build that tool and instead of having a desktop app let's make it web based, and see what do people like that; and they did, and they liked it and sort of number crunching aspect but this was a very sort of lean startup this thing we were doing.
10:16 Let's reinvent this product online. We did that, and so for 3 or 4 months I tried to gather a lot of user feedback, and what people were calling back and saying " you know what, the coolest thing about this isn't the spreadsheet part, it's just the fact that I can actually run Python scripts online, off my computer, in the cloud, I can just go there and it's there."
10:33 And so we said "all right, fine, and what if we drop the spreadsheet and just have Python in the cloud, what would that that look like?" And that was Python Anywhere.
10:39 That really interesting, and there is a parallel to Docker; the thing that was just the foundation of your actual product and the thing that works so well.
10:51 Classics, so it can value pivot although in this case st that it can round about,
10:56 Yeah so it can round about which is like few blocks away...excellent
10:58 Now let me take a minute and tell you about our sponsor for the show- Codeship. Codeship is a hosted continuous delivery service focused on speed, security and customizability. You can set up continuous integration in a matter of seconds and automatically deploy when your test have passed. Codeship supports your GitHub and Bitbucket projects. You can get started with Codeship's free plan today. Should you decide to go with the premium plan, Talk Python listeners can save 20% off any plan for the next three months by using the code: TALKPYTHON. All caps, no spaces. Check them out at Codeship.com, and tell them "thanks" for sponsoring the show on Twitter where they are at @codeship.
12:06 So you guys are hosting your cloud service on the Cloud itself, right?
12:14 Sure. It's hosted on Amazon and I mean we could have picked any other but we picked the big guys, and that works fine.
12:28 Is it on multiple data centers, or is it on one?
12:30 No, at the moment it's just on one, and as we grow we get a few customers going "hey, can we get some in Australia" or "can we get some on the West Coast " or whatever. But US East won, a b c d e
12:41 Somewhere in Northern Virginia ...
12:43 That's it, yeah, I've cousins down there, lovely countryside.
12:46 Fantastic, yeah. So, I think that's a really interesting place to be in; like I said, the way I found about you guys is I am actually hosting my website for the podcast on there and the thing is super responsive, I really like the experience, so I tried to find these other places that would host it, and they are like "Oh yeah, we will support Python 2 and Python 3 but oh we don't support Pyramid."
13:10 Which is one of the framework I use for the website, or "Yeah we support Pyramid but we only do Python 2" and I write in Python 3 and- you know, because it's a brand new project you know, I can write this in like Python 3 that'll be fun. Until I wanted to host it, then it was not so much fun, and I just stumbled across you guys and I think what you are doing is really really cool. You know, you support a whole bunch of different interpreters-
13:34 Well, Python interpreters, yeah-
13:34 Python, of course, so you've got several versions of Python 2, several versions of Python 3, PyPy, which I thought was very- just because you can, right, why not? That is really cool, did you have to do anything special to support that?
13:47 No, no, it's ok, so I mean supporting things from the web hosting point of view means we use 13:56 and some people have it in the background and that just has plugins with various interpreters including non-Python ones. So it is just the matter of activating the ones that we think we can do. So like we could actually if we wanted to go off and make Ruby Anywhere.com, and the trouble is that no one of us knows anything about Ruby so- we could do it technically and then I think we're taking a lot of users that will be asking us questions that we don't have the answers, so...
14:19 Right. And there are places that host Ruby, but you guys are hosting Python really well and...
14:24 And I mean there is lots of people out there you know, so if you are someone going "Ok I want to run my code on some servers," whether that's for web hosting or anything else. You can go around as an astray, you can run your own server in a cupboard you can go to Google App engine or Compute as it is now called, or you can go to Heroku or you could go to the dotCloud, in the old days and so, it's about sort of figuring out where there is a space in that sort of fairly crowded market and I think the one is like "let's focus on Python because that's what we know and there is plenty of business there," and then let's try and say "Ok, well the hard work if you are running your own server is all the kind of sysadmin and configuration and maintenance" there is this kind of trade-off between like the ultimate flexibility as if you do everything yourself,
15:08 open Linux machine, but then it's your Linux machine and you have to babysit
15:14 Right, yeah, so it's more effort. So like more flexibility more effort, and then there is like the other end like Google App engine where they would do a lot of work for you but they were also very restricted. So they would say "right, so you can use App Engine, but these are the rules about what kind of things you can run, as this version of Python, these things won't work and you have to" - all your codes will be killed after x seconds, no database, no nothing. And so we are somewhere between those two, trying to give like enough flexibility that people find it interesting, whilst solving enough of the problems that developers don't like to think about: scaling, backups...
15:50 Yeah, so maybe we could talk about the sort of experience of creating a website in there, and we should also talk about other things like web job type stuff, which I think is interesting. But you go there, you create a website, you do that for free, right?
16:04 So I can create a free website as long as I don't care about the domain name?
16:06 Yes, it'll be .pythonanywhere.com.
16:07 Yeah exactly something.pythonanywhere.com. You are to use your name .pythonanywhere.com.
16:12 That's right.
16:12 And, then I am giving something that's really interesting, it's says "here is your bash shell in the browser to configure your environment " and basically I can start with like virtual environment.
16:28 Yes, exactly. I mean, so we haven't done enough about this, but we were trying and so- people who learn Python- there is sort of first learning curve- trying to get it installed and figuring out how pip works, can you install packages. And we are trying and solve that problem for people by just saying "look, Python is already installed, here it is, you just type Python or you click this button that says which version of Python you want" and then, you know, like how do you install packages "well here is pip and it will just work" and also we pre-install a bunch of them just the popular thing, maybe it's already there.
16:59 What are the popular ones that you have installed?
17:02 We have overly done it. There is like a 1000 packages installed, like SciPy, numpy, Pill, blah blah blah blah Request, and so many of them. But, and web app is the same thing and I go "ok great and I've developed, I want to develop a web application what I do" well now you install this web framework, you need to configure the following settings files you need to have the folder structure that look like this, you need to put your static files over there, and if you are on a web server then you want to configure static file setting here and the database there and the environment there, and we just want to try and do as much as possible to just kind of give users the thing that works, that they can get started with without having to learn all those things. And so we try to do a bit of that, so if you want to start like a Django project you can go to our web tab and start a new project- "Do you want to use Django?", "Yes" and we will install Django for you, and set up basics and run the python.manage.py.project for you, and that will be working Django site there. And you can start with that.
18:04 That's the idea, and you know, we need to do more work on the version of Django we have for Python 2, if I'm nit-picking so ok, you do have to use virtual m but that's a bit more complicated and blah blah blah
18:13 I think the fact you have the virtual environment possibility there means that you can do whatever you want more or less, on the system, but you don't have to manage the infrastructure. And so in my mind, I think you guys are very interesting blend up platform as a service, and infrastructure of the service, so I see it as infrastructure because I 18:33 bash shell, I can really kind of install stuff, I can work with the file system it's kind of like a VM but it's platform is a service, because I don't care what the web server version is, I don't care about Linux, and setting that up, you take care of it for us, right, so...
18:48 Yeah, I guess, so that's right, so some other people will try to hide that stuff from you and just give you a thing where you will write some config files and then...
18:56 Like Google Compute for example.
18:59 Yeah, and then like magic happens. And ours is a little bit maybe it's more about trying to recreate the kind of the environment you have on your own laptop when you are running a dev version of a Flask app or something, you know how that works, making that as much as possible the same on Python Anywhere. Where you just go "Ok well if I want to write a file in bash or whatever, I can" And it's just that this part over here will do the web serving of it. If I want to view my log files, I can just you know, tell my log file I'm here and there it is I don't have to go through some complicated UI to go and retrieve the log file, you know.
19:33 Yeah, that's really cool, I feel like it's a nice miligram 19:37 because
19:39 I'm getting more enthused about my own website by talking to you.
19:41 This is fantastic, I'm here to make you excited. Keep you going because you are doing awesome stuff. So, I feel like you are making me maintain things, but making me maintain the right things. So I have created the virtual environment and everything in that virtual environment is my problem, but that's cool, because I decided when I upgrade Pyramid or Requests or whatever, and that's all in line with what my apps doing, but everything else I don't care about, right. Like if would you upgrade Python 3 for me at some point?
20:10 Like will I just magically switch to a new version of Python?
20:11 You need to rebuild virtual m so if you switch Python...
20:14 Right, of course, that's right.
20:15 But we try to be a bit clever about that, we are trying to think of ways where we can say "Virtual Ms are a bit scary for the beginners", they used to go and ah all right I figure that is command line thing, I know if I type Python I am going to get a Python prompt, ok, fine. And then you go r"ight, but aha, you need a virtual m, and so that when you do your clone you have to check have you activated your virtual m and is it the right one" and this is another point at which some of the beginners are just going to go "what I just- forget it". So, people get that wrong and so that's why we try to give a default no virtual and setup that might work, and then we try to see if there is ways we could maybe build a bit of a gui around the virtual m stuff one day.
20:53 Where you say like a virtual m is a thing that Python Anywhere understands and so knows what your virtual ms are and we can give you a gui that says "this is what your virtual ms are, and if do not want to write in your virtual m bash with you know, pip freeze and go bracket requirements.text maybe we could give you a UI for that and you can select and click on things, upgrade them and...
21:12 You could surface the top ....packages, and they could check them on and
21:17 Yeah yeah exactly. And that's 21:20 so I apologize for talking about it.
21:22 These are good ideas, I think. So web hosting is what a lot of people are paying you people like me, are doing but there is other stuff you can do as well, like one thing I saw us I can run like schedule jobs so if I were to run a script that like stucks down some sort of screen scraping sort of thing I could do that
21:47 A lot of people do that, yeah. So a lot of people use it for web hosting, a lot of people do scraping and so we have this thing schedule task it's basically con jobs, and you get to say "oh yeah every hour every day this sort of time I like to run this script" whatever script you want, and we are go and run that for you, and maybe it's pulling out some data and saving it to a database, which maybe you also have a web app that send us playing that data and show me the train times or something whatever.
22:10 Yeah, so people do it to like run ultimated jobs that don't need to keep their own computers switched on for, and then the third type of users are I guess, who might pay us, are people doing big number crunching. So every so often we get some people coming along and going "right, I want to just download megabytes and megabytes of tweets and like just run massive number crunching sentiment analyses on them" and so, we have a system where if you use a certain amount of like CPU power on the system, we kind of put a quote on that each day and when you go past that quota we don't switch your stuff off, we just put it in the tab where it gets slowed down. And so, if you want to do a little number crunching all day long then you have to pay us to get a 22:52 that it is actually going to run at reasonable speed. That's the deal there.
22:56 Your web apps don't have that restriction now, do they? So they don't count towards your
23:02 Yeah, we find most web apps on big number crunching sources and we do actually have a thing where if a web request hasn't responded within- I think the limit is 5 minutes, then we kill it.
23:13 That seems reasonable.
23:18 So yeah, it's not you could try in a very cleverly circumvent on number crunching restrictions by building a web app if you want, but it will be pretty painful.
23:26 Yeah I know I just more like thinking it's not something I have to worry about with web hosting, it's more of a job's sort of consideration, right?
23:32 That's right.
23:33 Because I was feeling really good like after a half a day I have a lot of hits on my site and I've used 0.00 seconds of my time, and I was like wow I wrote hat thing pretty efficiently but not really, right?
23:45 That's right, we don't count the web hits, and the web more in general is that you know, it's like any what would you call it shed... shed... multitenancy- I don't know. but the point, is that you can like an online booking system; oh my god, I just compared our site to an online booking system, this is the worst PR ever. You know, like we can host a lot of people on a big web server and most of them are like not going to be doing very much and every so often if someone is going to hit the front page of Hacker News- you know, if you run your website on an Amazon micro instance, it's pretty fast, most of the time, and the way Amazon works is that they will do really really fast hits on requests and once you've used about ten seconds of continuous CPU power, they throttle you down to nothing. Right, so it's great if you are running a website that gets a low traffic and it's super cheap it can't be free, but then if you hit Hacker News you website just goes down.
24:39 Yeah. When you need it most, it lets you down. Like the one you finally have gotten the word and ....
24:48 Right, so whereas we say "hey look, we are running you on servers that has 24:51 and the big servers use triple extra larges I forgot which on Amazon, so when someone get hits Hacker News, they just start using out more of the CPU on that instance for getting to front page of Hacker News and we are fine with that, that's part of the plan. If it turns out you are still on the front page of Hacker News after several days we are going to notice and we are going to have a quiet word with you about maybe it's time to start paying us. But in the mean time, it's a nice thing so you don't have to be scared of that sort of getting slashed
25:23 Yeah a friend of mine said, when I told him I was using your service Python Anywhere, and he said well why don't you just go to EC2 and create like a micro Linux and I'm like one of the things I really liked is it's really running on a big machine and has a lot of power, it's just kind of shared but you know, the spikes...
25:42 And so it seems like it's way more responsive than a micro instance for me.
25:48 Yeah, so that's the idea. We can provide that within reason, we can scale up when you get crazy traffic unexpectedly, and if it turns out to be forever, then we are just going to come after your money.
26:01 It seems fair.
26:01 We gently encourage you to upgrade.
26:04 That seems fair. So, other than of course Talk Python To Me, are there other notable clients that you guys can talk about?
26:14 Certainly Talk Python To Me is our favorite-
26:17 I would say so, yeah-
26:16 It's certainly the best one. Well, yeah- so who do we have- so have got sort of funny nation web2py community, so we host web2py.com and we've got lots of users of that site, and then the a case study example we sometimes trot out, which is terrible example again but it's just fun; there is this guy, he was he is a student at the university and he made a website called stereodose.com. And the stereodose.com website is very simple, what it says is like you come into a screen and it says "Pick your mood" and it also says energetic, melo, couple of other choices like funky, and then, screen number 2 says "Pick your drug" and you get to choose acid, mushrooms, ecstasy, weed; and then it is going to auto generate a playlist for to you that is appropriate to your mood and drug of choice...
27:20 Right, so, he is at university, he is at college right, and he is hugely popular among all his friends, and it is a simple website, and he hosted with us and so like he had the equivalent of Hacker News, the equivalent of Hacker News is for the drug induced playlist but he went from like ten hits a day to like 10 000, 100 000 an hour. And we were like well fine,
27:45 The melo hits?
27:46 Yes, some of them were really hyperactive, just keep refreshing the thing.
27:53 That's pretty cool.
27:54 Yeah, so that's an example, he made that thing out of nowhere, he got popular and then we just able to handle that fine, and now he is making some money of advertising, could he pay us x dollars a month. I think yeah, probably he is still one of the quite with no plans, $20 a month and....
28:10 Yeah probably, that's very cool.
28:11 Well we do that, for some of the first customers and then we reward them by keeping them around. The prices are quite frankly cutting our own throats Michael...
28:24 So, we were talking earlier, before we started recording, you said you do a lot of Python sort of end to end the whole thing so they look in and from the outside obviously it is interesting that I can host Python there, but can you talk a little bit about like what the internal usage is and how that goes?
29:31 Python 2, Python 3?
29:33 Well, yes yes, it's in Python 2
29:35 Yeah, of course, well, like literally everybody else. But I think you are doing a good deed to have very wide variety of interpreters like even PyPy is super cool.
29:48 Yeah. I mean yeah, so making the choice to supporting Python 3 for web apps for example, just having Python 3 around, you can install on your own computer in seconds, that's fine, supporting it for a real thing on the internet like an application was- you know, we were looking and again, is this going to be a concern, but then we were saying look, a lot of the people who come to us are beginners or hobbyists or not professional programmers and they the message out there is yeah you should use Python 3, and so to support them well I think we did have to go, even if we are not using it ourselves right now.
30:22 Yeah, that's cool, yeah. I talked to Kenneth Reitz, about Python 2 and Python 3 and you know, it seems like some of the little things that make a big difference; you said that having Django switch their docs by default to Python 3 noticeable bump in the usage for web apps for Python 3 just doing that, so things like have lower ....
30:49 I think that is happening like- I heard the Kenneth episode actually and he was saying "I wish Python.org would just decide which one of the versions they want to push and have that as the one appears when you download and that should be Python 3. And if you want Python 2 you should go and have to search for it." Because, ultimately, I think yeah, beginners are using Python 3, educators are using Python 3, hobbyists are using Python 3 because you know- why would they use the older version like if you don't know any difference. Python 2 is kind of a legacy language, it just happens that 90% of professional developers use it. But yeah, I think we should make every effort to start supporting Python 3 first for that sort of community.
32:29 It's fantastic.
32:30 So for beginners that's great; the learning curve is really nice for Python, the Zen of Python is there. Within the language that's great, that's working; outside the language, there is some peripherals where it is not so good, I sort of someone doing a poster about this at a PyCon three years ago just saying here is what beginners find hard about Python, installing it is hard, like first you have to choose between 2 or 3, and then you install it on your system and I don't know if you have done this on Windows recently-
32:58 I have-
32:58 But like it doesn't add itself to the path and it still doesn't add itself to the path by default by the way, and so then when people go oh just open up a command window and type Python- it doesn't happen and you know, getting Python installed and then people say oh yeah, you need pip to install packages you know. So, just you have to get installed pip or brew install pip and beginners go what, what is this- we've got Windows, everybody's got Windows. And even the ones that have Mac go oh yeah just install homebrew and they go what's homebrew and they find a GitHub page which is a terrifying thing for beginners and there oh yeah you just need to download X code which is 1.1 GB just to be able to install this thing, that's going to help you install this thing, that's going to help you install this thing! Right. Maybe you think...
33:41 What is it like 5 MB or something, I mean how big is pip?
33:47 Right, pip is tiny but like to install it on the Mac people go like oh yeah use homebrew and then you have to, like I say, you have to have this 1.1 GB that is going to help you install this thing which is going to help you install this thing which finally is what you need to install this thing that you hope to use for some programming. Yeah, people have given up by now.
34:04 Yeah, I think people sometimes forget the what is obvious to us now but the things you get hung up on; like the path, you know. It would be no big deal to us, we could easily do it, it's not fun, but the new people they are literally out of luck, right, they are like what do I do?
34:25 And so, you guys make that easier, right?
34:27 Yeah so teachers have to- if you are teaching Python you have to like face that learning curve, and if you got a bunch of students in that school then you go Right, let's get Python on our computers, go! And the school IT department goes well, we are an IT department so we say no to everything... There is great school IT departments out there, of course. But you know, maybe that's going to take time, or maybe whatever.
34:52 Or you are teaching Python to bunch of people out there, maybe adults or they have got bunch of different laptops, some of them have Windows, some of them got Mac and you got to try and make sure each one of them has got Python installed and you got to go and make sure each one of them has got a text editor installed and then you got be like hey if you are using sublime text it looks like this, or if you are using idle, it looks like that.
35:10 And if you are using Mac then type this, and if you are using a PC type that command. And so you know, a lot of people teaching Python use the first days just spend installing Python and they go well...
35:21 Program is not fun, program sucks, it's all about these little settings, right?
35:26 Right, and so if you just say to them ok forget all that if you go to Python Anywhere you need a username and choose a password and go- everyone has a text editor that works and everyone has the same bash prompt and everyone has the same version of Python, and everyone has the same packages available and pre installed. So if you want to do a class at numpy you can hit import numpy you don't even need to explain pip install.
35:45 Yeah you haven't even mentioned that on Windows you might not have like the claim compiler or JCC or VS vars. That will literally stop the beginner... If the path didn't get him down this will definitely. So, that's great, it's all set up, even the stuff that's harder to install like numpy and the things that are C dependencies.
36:09 Yeah. So the teachers have been coming to us for that, and we've recently decided to try and do a little bit more for education. As education in Python worldwide becomes more and more popular, in the UK over here the government have decided to make it part of the national curriculum. They are saying that you have to teach it in the school and this isn't just like you know sort of secondary schools, it's middle schools. So you know, it's 7, 8, 9 year olds, they are going to start to learn programming. And, you know, what can we do to try to like help teachers- some of these teachers they are not computing teachers, these are like general teachers who are just like-
36:48 They might be a science teachers-
36:49 "I don't even like computers that much myself, I don't know if I..." so they are just trying to stay one week ahead of their students, and so if we could do something to try to make their lives easier... So we've been doing that recently and it's been really cool
37:00 Yeah, that's really great. Last episode I talked to a guy named James Curran, and in Australia he is involved in the curriculum there for high schools and he said that in Australia, they're requiring a text based programming language as part of the high school curriculum. And so that you can actually
37:20 We can require two because we are twice as good as Australia.
37:24 Nice, yes, so I think Python is the national choice, you can do Scratch or something like that for beginners but as soon as you like have to do a real programming language, I think Python is a good choice.
37:36 Yeah, and in the UK people tend to teach HTML and Python
37:42 Yeah, my daughter is in the high school class called web coding and they learn HTML.
37:47 I'm not sure HTML, whatever, they are doing something about computers and it's cool.
37:52 CSS is complete, accidentally.
37:57 Nice, Ok. So, we talked about people who started, that's great, talked about the web hosting; another thing that you are really into- you were recently at PyCon, last year right? Or was it this year doing a workshop? On testing, right?
38:15 Sure, so I guess it was in 2009, 2010 that I started with resolve what became Python Anywhere, and I guess I had learned enough Python and Django to be dangerous. And I'd build my first kind of web application for a client; have your read Dive into Python?
38:47 I think I've skimmed it, I haven't read it all the way.
38:48 Dive into Python is a good intro to Python to someone who already knows another programming language. And, in about chapter 10, he starts talking about testing and test element. And he goes "You know we have learned about how Python works but let me tell you about testing, and how you can write test and how you could even write it first" So using a very traditional example which is a Roman numeral translator, you he goes "let's spill a function that can translate Roman numerals into integers, so like, I is going to become 1, and II is going to become 2, III is going to become 3, and IV is going to become 4 and so on. And he goes "so imagine how you do that and let me show you that if you actually you can write tests for this, you can write a thing that says I should = 1 and II should =2 and you can run that as a program which tests you know, the thing you are about to write and then when you write it, you know when you've got the right answer and secondly, you are going to be able to modify your solution and know that it is still correct."
39:41 And I remember reading that, and going "You know what, that is a really good idea, that sounds like something you should really should do, it's healthy, it's good for you, it's like flossing your teeth" And what happened next is like I got my first real client and there were real deadlines and all of my good intentions about TDD just went right out the window, and besides, by then I had had a full seven or eight days of experience with Python and Django and thought I was pretty smart, I figured I could get away without testing and, yeah. I was fine, I was absolutely fine, at first. And then-
40:20 And then you started maintaining the thing?
40:21 Yeah yeah yeah yeah, I mean I, you know, I read a thing about like Don't repeat yourself, and I was like "right, I better not repeat the code" and so I started- I ended up being like sort of an inheritance class higher arche five levels deep...
40:37 Did you hear that siren? Well it's 4 pm on a Tuesday so I guess that means it's a perfect time for a fire alarm test. It was ridiculously loud, now, let's pick up the conversation again after ears stop ringing.
40:51 So it started all fine and then you heard about the DRY principle and you were like "It's a good idea let's do that"
40:58 Yeah yeah let's do that, let's have an inheritance five levels deep, and for some reason I decided, somehow I got to eval statements, so there was eval statements in the code doing stuff. And, you know, like it was no longer a one page app you could just check works by checking that one page, there was 25 pages and they were all interdependent and I change a thing over here and go like "I think I can- oh no, that's going to break that one over there I think, maybe if I overwrite this method..." so pretty soon I had like a nightmare of crafty code and was scared to change it and you know, like all of those things happened to me, so... Then I got a job at Resolve and they went, "we do test of elements" and I'm like "oh really ok maybe that could help" And I am still dragging my feet about it, but yeah, they showed me this thing and so it's an extreme programming shock, everything is pear programming. The CEO is honestly so lucky, for me to learn from these guys who have been doing it for ages. And, you know, if you are pear programmer you just sit next to someone and they start start asking programming and just say "why?" you know, it's just like sponge, just sucking in all this knowledge. So it is an amazing way to learn.
42:07 It'a almost like an apprenticeship, type of thing.
42:10 Right, yeah, I mean, that's really true. And so that's- yeah, so I learned all these testing and I decided to try kind of like regurgitate a little bit, and turn on to some online resources and made I couple blog posts... I rewrote the Django tutorials.
42:27 You have a website, right?
42:27 Yeah, yeah. It's called "Obey The Testing Goat" I used to have a website called TDD Django tutorial, it was official Django tutorial, test driven. And that turned into something like a sort of tutorial I would do in real life and in conferences and it eventually turned into a book, which is published with O"Reilly-
42:49 It's called "Obey The Testing Goat"
43:19 It's fantastic. So what kind of stuff you cover in there?
43:21 So yeah, it's test driven development, but instead of taking that sort of classic slightly artificial example of a Roman numeral calculator which Evan does or banking currency converter, there is always TDD tutorials with like example function and it goes- but it's nothing like the code you write in real life. So...
43:39 It has no dependencies, it doesn't touch the file system, it doesn't have like the huge bunch of code, all that stuff, right
43:46 And so I just did the opposite of that, I went like "Ok well let's test the actual web development and we'll just have all the dependencies. Like so yeah, it will be a web application which needs a web browser and it needs the file system straight away, and there is going to be database and all that is like you know, from the ground up; maybe that's a hideous mistake. But...
44:05 I think it's- I think a lot of those- I do a lot of training for my job and a lot of people have the same reaction that you were saying and it is like it's all good to tell me about these stuff but this is a fake thing, it won't work in my complex company. If you don't see it for real then no one is going to believe it, right?
44:25 So yeah, I mean it's still a fake thing, I go for the development of the web application from scratch but it's a simple web app, right it's a To Do list app because that's everyone's example and I have no imagination. So it's the simplest possible web app but it does have a database and it does use a web browser and it does do things to the file system
44:46 Nice. So, what are the testing frameworks and things like, do you use PyTest, UniTest, Clean Green- like what's the story?
45:00 Ha ha Clean Green.... I met Nathan who did this green test, he is a lovely guy. His GitHub user name is CleanCut and that's because he is just got the most preppy set of teeth. 45:13 I think he is satirizing himself a bit.
45:16 Yeah, so for the book I just used the kind of standard tools, which are the Unittest standard library test framework, and the Django test runner, because that's what most people use. At work we use those as well, and we also started experimenting a little bit with PyTest, so there is a few talk out there if you look saying PyTest versus Nose versus unittest what are the pros and cons and the Nose as PyTest do come off well they say you know like unittest came from the Java world and this sort of like ex unit thing, which involves classes and it involves methods and you they've got this CamelCase method names which is they've got them for historic reasons and people knew that Python will look and go this is weird why using CamelCase they don't look like methods or anything else I've ever seen. And so moving away from that, the PyTest and the Nose is who have more- why have a class, there is no point. Let's just have a functions, because that what we already use these classes for. That has a lot of appeal, and so we've- I've doubled all that, and we've doubled that with work and I think they're kind of cool. So we definitely recommend people take a look at those and particularly I think PyTest has got a lot of- I get the feeling people look at these things PyTest is coming out like the one we are going yeah yeah yeah, this is the one to look at.
46:40 What do you think the benefits are there, is it more Pythonic, easier to use...?
46:44 Yeah... So those two things for example. You know like, so, in Unittest you are going to have well if I run a test without any framework I go assert thing= 3, I do this in the book to explain why would you use a test framework. And you go assert thing =3 and it fails. And you get a thing saying a searching error. But it doesn't say "well thing you expected was 3 was actually equal to 4", that would be useful. So you can write your own error message handler by using the sort of comma, but that's hard work and you know what people have to go to solve some of these problems and there are standard solutions, and one of them is use unittest, put in a test method you get setup and teardown which is a common requirement in tests and you can go self.assert=thing and it is just going to give you a nice error message. And I think that's all very well, but you've gone through trouble of making a class and making a test method and you are like Raymond Hettinger was saying when you write and written a test class you are already three levels down before you have even written any test code. And you are going to go self.assertEqual and there is also self.assertItemsEqual if you just want to compare list items without the order and you've got self.assertIsNone and you've got assertMultiLineEqual for strings, and you've got assert RedJack's maches 48:03 like I can't even remember them, right. And so you have to start looking at the docks whenever you try to compare two things and I think PyTest is superior to that, just say assert and then a thing, and the test framework will figure out what the thing you want to assert and will try to give you the sense of error message without you having to be clever about it.
48:24 Yeah, that's fantastic.
48:25 And I really like that, just the assert key word that you've seen before, and the thing, and other than that you just put your test in methods that start with test as many for class unless you really want one.
48:34 Sure. Very nice. Well, I've been using Unittest, maybe I should be checking out PyTest.
48:40 Yeah, so you can run PyTest on existing Unittest tests and so migration path and 48:49 the weird thing the find the assert key would it will magically tell you what the two values were is the first bit of PyTest magic people raise their eyebrows at a little bit, but a really magic part is the way it does sort of set up and tear down, which is you know in Unittest we write a metrical setup and methrical tear down 49:06 that seems instinctive to us now because we are so used to it, but what we really want, and so the PyTest world is more kind of functional, and what they say ok if you want some setup to be done in the test you are going to have to define a method that does that setup, define the function that's going to do that. So define a function it say like prepare the database for example setup a test database for you, you do some stuff and then if that function is called prepare database you put a decorator on it say this is a fixture for PyTest word for test setup stuff is a fixture, which is slightly different from the way Django uses the term. Pop a decorator on it and then take any of the tests that need that fixture and then you just say add an argument to that test which first of all that function didn't have any arguments, now you have an argument to that test and you just give that argument a same name as the function you defined earlier, and when you run it it will magically find that function, execute it and inject it into your test for you.
50:04 That is magic.
50:05 Yeah, it is magic.
50:07 Crazy convention.
50:08 Or a convention, right. setUp and tearDown, are kind of magic as well, there is something happening to make those happen. And so like is it any more explicit? That's debatable, I think
50:18 No, it sounds great to me
50:19 But then, you can have fixtures inheritance from fixtures, so if you have a fixture function prepared database maybe that inherits from prepare file system and so you can give an argument to a fixture that's also another fixture and you know like...it'll get pretty fun pretty quickly.
50:37 Yeah, that's pretty cool. So, do you discuss code coverage?
50:43 I don't know. I guess we've never used it at work because we've got 50:53 TDD,
50:53 The guy next to you is the code coverage?
50:55 Well, they and the methodology, and we basically don't allow ourselves to write any code in production that hasn't got a test, so we've kind of assumed our coverage is 100%. And, so it would be probably educational for us actually run coverage...But I don't talk about it in the book, I mention it at the end hey here is a thing you should check out.
51:17 That's cool.
51:18 You know, you can't cover everything.
51:19 Yeah, I know, of course. You have to ship even if it is a book you have to ship at some point, so many features.
51:25 Yeah it is a trouble of anyone that is writing a book. But yeah I do talked about being real in the book we build it to do this app, so we go ahead we learn how to database to do lists, we start with app that has like one item, them it has many items abut it's the same list for everyone and there is like multiple lists for multiple people, and then like once yo got that working, I go Ok, well you've made a really ugly website let's do some styling, because like it's not just about Django view that returns horrible HTML, let's put bootstrap on it and CSS and make it look representable; which is something that most guys might not cover. And then that's chapter 7 and chapter 8 and I'm right good, we built a basic webline and it's fine, I know you delete to do items, you can't edit existing once but that is a working to do list app so now we need to ship it. And so like chapter 8 is deploy it to a server, and I'm just going to go how like now you've got your test can you run them against the server, can they actually help you deploy as well. And so yeah, that gets even more real I guess.
52:22 Yeah, very cool.
52:23 And, it fits learning curve.
52:26 Yeah it seems like a good place for people to start. I'll put a link in the show, to the book. So you were talking about databases independence, and I'd like to come back to Python Anywhere just a bit to talk about that a little bit more. You guys have MySql, Postgres, and then I noticed something interesting, I think in an email from one of the guys I think, I remember he was trying to say now you are using containers for your Postgres, is that Docker?
52:59 Yeah. I guess anyone who is building a thing a little bit like we are building it eventually comes across containers. Like this is dotCloud and similar business they invented Docker. So if you are running a lot of peoples untrusted code on your servers you have got to figure out where to isolate them and the way you do that in Unix is with bunch of cool tools that put together a tool this days containerisation. So we use bits of pieces of these tools everywhere, and the thing with MySql is where we provide that is if you host MySql server you can do multi tendency fairly easily. You can create different user accounts for different users, and you can say this user has permissions on this database, and this user has permissions on that database and you are fine. Like they pretty well sand boxed from each other. Postgres isn't so good in that respect it's not redesigned with multitendency modeling mind, and pretty much in Postgres unless you've got super user access to the server you can't redoing a thing. And, none of the cool Postgres stuff is possible without being a super user. So you can't really do multitendency Postgres. So we can't just have one Postgres server that runs the user accounts on the same server, we are going to have to have a multiple servers, that one is going to have to be effectively be a super user in there and since we are doing it it seems to be to run Postgres instance in some sort of container.
54:26 Yeah, that's really cool. A virtual machine probably wouldn't be the right answer?
54:30 Yeah, too slow. I mean this is why everyone is using containers.
54:33 Density on the servers just isn't there. Density matters.
54:41 That was nice, yeah, we use Docker for the first time in production, we use the Py test for the first time in production. So the Py test and Docker like two experiments at the same time, terrible idea, great fun.
54:51 Sounds excellent.
54:51 And, yes, that's how that works, we've been happy with it.
54:56 Cool. And I saw on your features you are considering maybe some time maybe was Mongo DB?
55:06 Yeah, it might be. I think we are looking in the short term to probably outsource that. So that is, I can't remember there is a couple of companies out there that do Mongo Db as a service-
55:15 Sure, Mongo HQ, Mongo LAb, Object Rocket...
55:19 There you go, all of these guys.
55:19 Those are the three that come to my mind about-
55:22 Well you know, since we haven't settle, we haven't picked up any of them in particular, because maybe they will give us a good deal or something.
55:26 Yeah, exactly. It's time for bidding or...
55:30 Yeah so I think what we'll do with Mongo in the short term is pick one of those, try to build the bit of integration, just make it really easy for people to go "ok start a Mongo service ..." and it will actually go out to whoever will be,
55:45 Yeah that's cool. And, a lot of those guys are either on 55:49 so it would be relatively.
55:52 Yes. The sort of fun thing about Python Anywhere actually is if you ever do get clone of the massive repo, and you are sitting there and it's going ok downloading 400MB or more, not everything is fast on Python Anywhere but that is, one of the things it is because it's in Amazon's data centers and GitHub has got some servers running next door. When you get kind of repo it comes down super fast.
56:15 Yeah, it great fun.
56:16 I mean, all the file system things aren't so fast, because we have to have the challenge of running multiple servers needing to give people access to the same file system no matter what server they are on. There is anyfest in the background and that's not fastest thing ever.
56:29 Yeah, of course. That's cool. And that's how I get my source code onto the machines, I just get 56:39 cloned instead of my virtual environment pointed out to get repo. Is there a way to setup continuous delivery?
56:49 Yeah. So you can hack it a little bit. You could do that. But you can setup because like Python Anywhere is essentially a Linux server, you can setup you get repository on it, one that you can push to, so we have got a blog post about this but you can setup a get repository to you can push to, set it opposite a remote in your own local computers one and you can do get push Python Anywhere and you can setup some hooks on Python Anywhere when they see your code coming in maybe they'll even run your tests and then they can do a lot of things that will bounce your web workers and your website will be deployed. So you can sort of hack it. So we haven't built that yet, but you can hack around a thing is you feel like it, yeah.
57:39 It's cool. Is that something you are considering, maybe- maybe not?
57:44 Well, we always have to think you know, we have to serve the beginners and we have to serve the more advanced.
57:50 That what I am describing is not very much in your original mission.
57:53 Right. Yeah. So like, continuous deployment as a thing is not the first thing, it's not the first thing that people work up to. Maybe for now, we would rather like a can we make it easier for people to use virtual ms, or can we make it easier for people to upgrade their version of Django, if they are beginners, we are trying to make that learning curve a bit smoother.
58:15 Yeah, it seems like that kind of stuff would actually be a way more broadly viable.
58:20 Yeah. It's different audiences, right.
58:23 Not this way to me, because I'd like continuous integration as a button, but our continuous delivery, but for most people it seems like that would be the right place to focus.
58:32 Yeah. There is so many things.
58:38 That's a good problem to have, there is always cool stuff you want to do and you have got to pick. So, you know, see what your users are asking from you.
58:49 Yeah, iPython notebooks they want we are trying to ...
58:53 Yeah, that would be cool...
58:52 Yeah. I mean it is getting iPython running in your local computer isn't totally straightforward.
58:58 It's not, so on Windows because you've got some C dependencies somewhere or...Some of the pre packaged-
59:08 It's always the same problem, like there are the things that should be easy, that aren't. And so yeah, we will see what we can do about it.
59:16 So if you make that open your browser, click the button, that would be fantastic.
59:23 Nice. So, I think that's kind of going to be a decent amount stuff we have covered in the show, anything you want to add?
59:32 No, I promised myself I was going to say a thing and it's gone.
59:39 Let me ask you something else and maybe it'll sneak back into your subconscious, so one of the things I ask a lot of the guests somewhere on their way out of the door is what is your sort of favorite package or library in Python? Do you have some favorites?
59:50 You know any type in pip install requirements.text? And it goes no package called this requirements.text. Because you have pip.install/r requirements.text. I've been trying for little while to figure out if I can upload a package to PyPi called requirements.text. somehow. So when you accidentally mistype pip.install.requirements.text installs my package, because setup.py is like you know completely unsecure. I was just going to open up firefox to recall people so that pip install requirements.text just recall you. I think that will be great. I think the PyPi got there ahead of me on that, so it's not possible, but I thought I put a little evil idea out there in the world, so the listeners to the podcast if you guys can figure it out it would be hilarious.
01:00:42 That would be great, maybe slight misspelling of the word requirements. Which is like 10% of the time.
01:00:50 Maybe requirements text without the dot. Maybe will catch a few people like that.
01:00:56 That's fantastic. All right. Final thoughts, have you thought of it?
01:00:59 That was it, that was actually the...
01:01:05 That's good. So, Harry thank you so much for being on the show. It's really interesting, I'm a huge fan of Python Anywhere, is someone's out there listening and they are like that sounds cool I want to check it out, I want to play with this idea, like a bashshell in my browser, whatever, how do they start it?
01:01:23 So, pythonanywhere.com, or you can go to python everywhere that redirects to python anywhere appropriately, you can go to Python.org there is a little Python console on the front page that you can click the button and you get the live console that's supplied by us. If you feel like it. And the other thing I have to plug on my editor will kill me, I have to plug book the book is available at ObeyTheTestingGoat.com, it's available both for free and for purchase, the whole thing is available online for free for the $0, but if you feel like buying you can as well, and yeah, those are the two websites that you have to go to. pythonanywhere.com and obeythetestinggoat.com.
01:02:10 And your testing tutorial you did on PyCon is on YouTube right?
01:02:14 Sure, yeah.
01:02:16 Is that just in the I think i actually have it in my list in one of the previous shows I said like here are the essential presentations from the PyCon 2015 and I think it's in there but I'll put it directly into these show notes as well.
01:02:28 For sure, yeah yeah. I mean there is O'Reilly official video you can buy, there are also free versions of all these things, I'm no great believer in- this is the internet, you can pretend that you can charge for digital things and the people won't find the way of getting them for free if they want them for free. So I'm one of those saying let's have them for free. Your choice is between having it for free without your consent and having it for free with your consent. My choice is with my consent. So help yourself to the stuff for free and pay for it if you feel like it.
01:02:59 Yeah, that's good. I think people are more generous than lot of businesses give them credit for, and you know, they'll buy something if they want it.
01:03:10 Well so I am the member of the pirate party here in the UK and for me it's very well, put my stuff online for free and people will pay for it and so I would love to inspire that as my ideology. I would say that it's a different calculus for me than it is for Stephen King. So, if I put my book online for free, the people that read that while they are reading a book a book test driven development for web development then I'll read- this is not a holiday thriller right, this is a thing they are reading because they want to get better at programming and they are probably doing it for a job for money. So I think people see the very active reading that book is quite a commercial transaction, and when I'm saying hey look read this if you get value of it I'd love it if you payed, and people do well I was reading this in the first place for monetary reasons so I think I should probably pay. Whereas you know, like if Stephen King would put all his books for free out there, you know, you are not reading it for monetary reasons-
01:04:20 Your personal value of it is not the same as if it-
01:04:21 It absolutely works for me.
01:04:30 That's cool. Great. All right, well thank you again so much for being on the show, it was really nice to meet you, and I think it will be a great conversation for everyone to listen to.
01:04:38 Great, thanks Michael.
01:04:39 Yeah, bye.
01:04:39 This has been another episode of Talk Python To Me. Today's guest was Harry Percival. Follow him on Twitter where he is at @hgwp. This episode has been sponsored by Codeship. Please check them out at codeship.com and thank then on Twitter via @codeship. Don't forget the discount code for listeners it's easy, it's TALKPYTHON, all caps no spaces. Remember, you can find the links for the show at talkpythontome.com/episodes/show/10 and if you are feeling generous check out our Patreon campaign at patreon.com/mkennedy. Be sure to subscribe to the show, visit the website and choose subscribe in iTunes or grab the episode RSS feed and drop it into your favorite podcatcher. You'll find both in the footer of every page. This is your host, Michael Kennedy, thanks for listening.