#182: Picture Python at Shutterfly Transcript
00:00 Michael Kennedy: Join Doug Farrell and me as we discuss his career and what he's up to at Shutterfly. You'll learn about the Python stack he's using to work with not just bits and bytes, but physical devices on a production line for creating all sorts of picturesqueue items. You'll also hear how both he and I feel it's a great time to be a developer even if you're on the older side of say 30, 40, or beyond. This is Talk Python to Me, Episode 182, recorded October 4, 2018. Welcome to Talk Python to Me, a weekly Podcast on Python, the language, the libraries, the ecosystem and the personalities. This is your host, Michael Kennedy follow me on Twitter where I'm @mkennedy. Keep up with the show and listen to past episodes @talkpython.fm and follow the show on Twitter via @talkpython. This episode is brought to you by LogRocket and Rollbar. Please check out what they're offering during their segments, it really helps to support the show. Doug, welcome to Talk Python.
01:06 Doug Farrell: Thank you very much Michael, I'm glad to be here.
01:08 Michael Kennedy: I'm glad to have you here, it's great. We've been interacting on email, and on social media and stuff. Talked about some of the articles you've written before and you've sent in recommendations of various things for us, so it's nice to finally have you here.
01:20 Doug Farrell: Yeah, I'd like to drop notes to you and Brian on the Python Bytes Podcast about the articles I've written in Real Python, the site.
01:29 Michael Kennedy: Yeah, they're definitely some good ones and I want to dig into some of those actually as we get further into the show. But let's start with your story about how you got into programming and Python.
01:37 Doug Farrell: Well, I actually got started pretty late. I was probably in my late 20s by the time I got into programming back in the early 80s, late 70s, early 80s. I had failed at Fortran years before the first time I went to college mostly because, back then it was all punch cards and having to go down to the dungeon to hand it to over to somebody who would tell you three days later whether your program worked. I didn't really go...
02:02 Michael Kennedy: Yeah, I could see how you wouldn't be so excited about that. That's not the same as fun GUI editors and compilers and Stack Overflow.
02:11 Doug Farrell: No.
02:13 Michael Kennedy: And all that stuff, right? It's a different world.
02:14 Doug Farrell: Well, plus I saw lots of people who actually left the program, they'd come down the stairs with a giant stack of Cobol cards, drop it, the whole thing would be spread all over the place. They'd leave the program crying because they just couldn't stand to sort them again.
02:29 Michael Kennedy: They just lost their will to be part of that.
02:30 Doug Farrell: That's it. That's it, they'd move on to something else. So I got excited about it when I bought my first personal computer which was back then I bought a RadioShack Color Computer and taught myself some Basic and a little bit of 6809 assembly language for some, you know as most young guys I was thinking I was going to be a game programmer.
03:04 Doug Farrell: The Color Computer was so slow, it was less than a megahertz clock speed, so to do anything fast, you had to really get down and dirty.
03:09 Michael Kennedy: Yeah, but okay so you started there, so that gives you a good foundation of it's only going to get easier from here.
03:15 Doug Farrell: Right, yeah, and had to, actually I was back in school by then going after my physics degree, BS in Physics. My senior project was building a fairly primitive CAT Scan device with some high precision translation tables and using Pascal on this wacky computer called a Terrax. And I really enjoyed that. I spent most on my time doing that bedsides failing French and that gave me enough buzzwords to get my first engineering job as a process control engineer doing automation systems for municipal water systems and gas pipelines.
03:51 Michael Kennedy: Wow, that sounds interesting. So that's a lot of like controlling hardware, right? Which is kind of...
03:55 Doug Farrell: Oh yeah.
03:56 Michael Kennedy: It sounds like from all the stuff that you've been doing.
03:58 Doug Farrell: Yeah, it was kind of a big scale, like a macro version of hardware control where you're doing really huge systems. I think the biggest thing I did was I automated a 35 gallon a day, 35 million gallon a day clean water system.
04:13 Michael Kennedy: Oh, my gosh. I think about there were certain steps in my programming career that they were sort of take your breath away moments. I remember the first time I wrote, this is for some other company, my first eCommerce system and it was doing like $4,000 transactions per transaction. I was like wow, I really better not mess this up. You know, if we do this wrong on scale it would be a problem. Did you have that feeling when you were working on that much water and that much hardware?
04:45 Doug Farrell: I did because, yeah, the scale of this thing was huge. I mean, this was early in its construction, so when I was out there, I was actually myself and the other guy I was working with we were able to ride our bikes inside of the 35 million gallon tank because it wasn't in use yet.
04:59 Michael Kennedy: You had your own velodrome.
05:02 Doug Farrell: Yeah, and we had one disaster, thank God it wasn't me where we had a flood where we managed to wash away a golf course, a putting green down the road.
05:12 Michael Kennedy: Hopefully no one was putting at the time.
05:15 Doug Farrell: Well, it was late at night. The course manager was over screaming at us and then it turned out to be not us, so. That was the nice escape.
05:24 Michael Kennedy: Well, that's really good, that's really good. So what languages were you using to do all this?
05:28 Doug Farrell: This was a propriety language that the company had written. It was a software simulation of hardware, so when you do hardware control, there'd be things like timers and PAD controllers and digital switches and analog inputs and outputs, comparators, all these hardware things that you could wire together and this simulated those. In that language, the variables, you could think of the variables as wire, so anything you connect the variable to was like wiring those two things together. So you could string all these things like an analog input to a comparator, to a set point control, to a PAD loop, to an analog output that would control a valve that would keep a tank from overfilling or underfilling.
06:25 Doug Farrell: Well, I know that that company's still big in the game and they're still around and I think... My experience with municipalities and gas pipeline companies is that they're very reluctant to change, they're always going to stick with something that works. I don't know if you've ever worked with PLC controllers programmable logic controllers?
06:44 Michael Kennedy: Nah, I only have heard the term.
06:47 Doug Farrell: Those are very popular, because they work and people know them. So the same thing applies to municipalities. It works and they know them. I mean, one of my experience when I was there was having a realization that they had our computer stuff in the same depreciation column in their spreadsheet as manhole covers. So, they expected our computer stuff to last like 50 years.
07:10 Michael Kennedy: Oh my gosh.
07:10 Doug Farrell: I was like really? I don't think so.
07:15 Michael Kennedy: This is not going to work out the way you think this is going to work out.
07:18 Doug Farrell: So that was nice. Well, how we going to get parts to replace this? I'm like nah, you're going to replace the whole thing in about 20 years, I think, maybe even less.
07:27 Michael Kennedy: Yeah, how interesting. You moved on from there, right? Towards the web.
07:31 Doug Farrell: Yeah, after I left that company I worked at a high speed pick and place machine. It was kind of a robot that assembled electronics boards by picking up chips, flying 'em over a camera, put them on the board. I don't know the faster one was like 20,000 parts an hour, which is really moving and that was...
07:51 Michael Kennedy: Got to stay away from it, right? Don't get too close to it, huh?
07:53 Doug Farrell: Yeah, there was lots of stuff about don't open the door when it's moving, 'cause you could get really hurt. I went to a trade show and they really had some fast and big machines like that that would scare you to death.
08:05 Michael Kennedy: Yeah, I can imagine.
08:07 Doug Farrell: But, that was like my low-level embedded programming which I really enjoyed and eventually I went to work for a company that did where my internet exposure came in where they were trying to put reference titles like encyclopedias online.
08:22 Michael Kennedy: This is probably pre-Wikipedia, right?
08:26 Doug Farrell: Yeah, this was back in the Encarta days and Grolier Encyclopedia online and I actually worked on the CD ROM version of that, then the web version.
08:35 Michael Kennedy: Yeah, cool. Yeah, and now you're at Shutterfly and it sounds like a really interesting place to be and it also sounds like you're getting to do some of this sort of software meets hardware thing.
08:45 Doug Farrell: Yeah, I really like that part where software meets the real world. It's not just all a bunch of bits moving around in the computer. It has enough of the things that I write. Everything I write is all on our production floor not customer-facing. So, it allows our production people to interact with the presses and other machinery that actually produces a book. Shutterfly is a digital book publisher, where customers can upload their photos and get all kinds of products.
09:11 Michael Kennedy: Make photo books, you can send out for a gifts or printed stuff.
09:14 Doug Farrell: Yeah, that's right.
09:15 Michael Kennedy: Or printed photos out of your digital photos, things like that.
09:19 Doug Farrell: Yeah, pillows and coffee mugs and desk frames and big canvas prints which are all interesting products. Our motto at Shutterfly is Sharing Life's Joys. So it's an interesting way to think about photos and how so many digital photos and then people get so many and they get overwhelmed about what I am going to do with this thing this pile of pictures?
09:40 Michael Kennedy: I just ordered a photo book for my family because we have like 30,000 photos and I'm like we're not going to get through these.
09:46 Doug Farrell: Right.
09:46 Michael Kennedy: It's overwhelming. But if I could put it on the coffee table when friends come to visit, like we could experience them in a different way, that's pretty cool.
09:54 Doug Farrell: Yeah, it makes great gifts, I mean grandparents. I know my daughter sends me a books that she makes with our product and I love that stuff.
10:03 Michael Kennedy: Hey, you printed this for me, you just don't know it.
10:06 Doug Farrell: Yeah. Hopefully, I don't find out that I've printed it and made it wrong.
10:11 Michael Kennedy: That's right. This is messed up, oh I remember that one.
10:13 Doug Farrell: Yeah, oh sorry. So doing that stuff satisfies my kind of need to have the computer touch the real world 'cause we interact with the presses and hardware directly, or a little directly. It's not as dangerous as like high speed stuff where people could get hurt.
10:29 Michael Kennedy: Yeah, you don't want to be putting people on to a board.
10:33 Doug Farrell: No.
10:34 Michael Kennedy: Yeah, this sounds like really fun that software meets hardware thing and you guys use a lot of Python at Shutterfly, right?
11:16 Michael Kennedy: Yeah, of course.
11:17 Doug Farrell: But they're having us come on board. One of the things that happened right away was that we were able to take over a section of gifting products, 'cause Shutterfly has a huge, they do a huge amount of book printing, of course, but the gifting products are becoming is a fast growing market for them and my group that I'm in was able to implement a lot of the gifts and is continuing to implement new gifts all the time in Python. And almost the whole stack for that production is in Python.
11:49 Michael Kennedy: That's really cool. Do you guys see the agility and ability to pip install antigravity to as a real advantage for your team, maybe even within the company? People go, oh yeah, we know those guys can deliver quick.
12:03 Doug Farrell: Yeah, there's two parts to that. There's the we we can be extremely agile and when we first did our first pass at the producing gifts products, I think it was I think it was a real eye opener for people to see how quickly we could actually bring a product to bear into production and then not only get it there to where they could start producing it, but iterate quickly to improve the processes, 'cause the first pass would always have issues, or they'd want changes. And the ability for us to iterate very quickly I think is a huge benefit for not only us but also the company in general.
12:38 Michael Kennedy: Yeah. Yeah, there's an interesting story about how Google acquired YouTube and they had Google Video and a bunch of C++ guys trying to compete with them and YouTube was a little upstart built on Python and they were just outperforming and adding features maybe they needed more servers or something to make it go but it didn't matter, right? Kind of like you, Google or like Shutterfly, Google solved the problem by buying YouTube and that fixed it.
13:02 Doug Farrell: Yeah, and we've been growing because they continue to sort give us the opportunity to add new products sort of subsume some of our products into the way we work in Python and we're doing a lot of infrastructure work. Any growing company has to do a ton of infrastructure work all the time and we're doing a lot of that infrastructure work and architecting based on the experience we've had with Python and some of the tools that we use and interface with using Python, like you said, the pip install thing. We have our own PyPI Server.
13:36 Michael Kennedy: Oh nice.
13:37 Doug Farrell: With modules that we have figured out are safe and to use and build against but it's because we have our own, we have sort of a vetting process to see if new ones we can add, because I know for myself I'm always hey, there's a cool module and twisting people's arms to get into our server so I have access to it.
13:52 Michael Kennedy: That's really cool. So what technology are you using for your private PyPI sever?
13:56 Doug Farrell: Well, we have just a copy of the PyPI server code. It doesn't have the whole front end but it just has the backend where you can do just a PIP install as long as you point at that server with a pip config file.
14:08 Michael Kennedy: Right, so you can configure...
14:09 Doug Farrell: It will just grab files.
14:10 Michael Kennedy: Yeah, you can configure your user profile. So if you say pip install, it first looks there, right?
14:14 Doug Farrell: Yep.
14:14 Michael Kennedy: That's pretty cool.
14:15 Doug Farrell: My own development, we're moving from server side development to a client side where we're using containers, docker containers. And for that stuff it's much we often, this is how stuff gets started, we'll often grab a module from the regular, the public PyPI server PyPI server and then once you see some success in that and people start to review your code, that module get vetted and checked out and moved into our private PyPI.
14:42 Michael Kennedy: Yeah, nothing is more convincing argument than success. I've had so many places where people debate oh, if we do it this way, it would be better. No, that way it'd be better. You know what, we're just going to build, like I just built it this way. I know you said it can't work, but here it's working, can we discuss now whether or not it's still working? Okay, fine, it's working. You're right.
14:59 Doug Farrell: Delivery often wins, being able to deliver wins.
16:24 Doug Farrell: Sure.
16:25 Michael Kennedy: Like some of the tools you're using.
17:34 Michael Kennedy: Oh, that's interesting, yeah.
17:35 Doug Farrell: As a message system. We do a lot of on the backend with the size of like the full stack stuff that I do, there's a lot of processes that we run in Python so we have a lot of Twisted code, we use Twisted code for asyncronous work, we also use a lot of gevent for the same thing. I know I'm more comfortable with gevent than I am with Twisted, although I've written quite a bit of Twisted code, so that we can handle multiple we don't do so much networking code as much as just there's multiple things in motion that take time like as in a production environment where stuff is getting printed and cut and bound and shipped, all those events, they're separated by time so asyncronous code is really handy for that so you can set something, you can set the state and then you get a callback you know, who knows five, 10, 20, an hour later that it's actually finished or moved on to another state. So to be able to handle essentially thousands and thousands of those state-driven processes is really handy.
18:33 Michael Kennedy: That's really cool because if you tried to just wait for that response, that would be a problem.
18:39 Doug Farrell: Oh yeah.
18:40 Michael Kennedy: If you wait for an hour, this fly server.
18:39 Doug Farrell: Oh yeah.
18:44 Michael Kennedy: Just won't scale. You know what I mean. You set the timer.
18:46 Doug Farrell: I have a bunch of production lead guys in my office stamping their feet in a heartbeat, if that was the case.
18:52 Michael Kennedy: I understand. When you talk about doing asyncronous with Flask, is that using say gevent in Flask? Or are you doing like architecture that has queueues or what are you doing there?
19:02 Doug Farrell: We do a couple of things. I use gevent and Flask to try to make that as asyncronous behavior to that. It's still, because of the database access, it still can get paused on a database call. That's one of the reasons why we are moving to the data access is going to another process so that other process now becomes, there's multiples of these and it handles the database. But my call from my application or any other application can be asyncronous to that, it's no longer waiting 'cause now it's going to get a callback when it's done.
19:36 Michael Kennedy: Yeah, that's really cool. And it just queues up on that other system probably won't time out for example, things like that right?
19:43 Doug Farrell: Yeah, those other systems are all also entirely asyncronous so it has a big queue of things that are ongoing that's working on, it'll get back to you when they're done and those come across the message bus as an event.
19:54 Michael Kennedy: I think it's so interesting to look at the different types of architecture choices people make based on what you're trying to do, right? You're here working with with latencies in the minutes or hours so you might choose a really different architecture whereas people say well, the database is kind of slow, it takes 30 milliseconds to get back to us and we want to scale.
20:13 Doug Farrell: 30 milliseconds, I laugh at that.
20:14 Michael Kennedy: People just use, It'd be like a asyncio event loop or something like that would be fine for that, right? So, quite interesting to think about it. So, that's the backend stuff. I guess one thing I did want to ask you about when you talked about gevent and Flask and all that and I don't know that this would I don't think that this would really necessarily be the right application for it, but have you looked at.
20:39 Doug Farrell: I have and I've been following the async stuff like Sanic and Quart and I'm trying to get up to speed on the Python 3.7, the asyncio and async/await stuff because in many ways, I'm moving away from really needing Flask. I mean, it provides a lot of features that I don't use like I don't do any web form validation. I'm doing all either web socket or Ajax calls. REST APIs, so I don't do a lot of form validation, I don't serve a lot of files. I'm mostly creating REST APIs so I just want to make that part faster and those things like that you mentioned would do that, so I'm looking into that to take better advantage of the system.
21:21 Michael Kennedy: Yeah, that makes a lot of sense. There is a lot of stuff in these web frameworks that you just don't need if the response is you know Flask.
21:28 Doug Farrell: Yeah, I mean. In my case, I don't even serve up the web pages the front end anymore myself, that's done by a dedicated web server so my application has strictly turned into a REST API.
21:43 Michael Kennedy: That's a really interesting place for people to work who want to work on the web, but they're not designers. Right? They don't want to try to do all the CSS and make it pretty and put the hero banner on there and all that stuff, right? There's still some really interesting and challenging programming to be done building this.
22:02 Doug Farrell: Yeah, I absolutely agree. I am currently working on this gigantic project which is this REST API that drives a big part of our business and a I find that part, it's challenging 'cause I'm always negotiating with the front end guys about what the payload looks like and what the back and forth looks like. I also enjoy the front end, because it gives me then I'm arguing with myself about what's the best way to get this data across. But also I really enjoy the GUI part. I used to be, for a short time I was a Windows developer and I liked that, that user interaction and trying to figure out what's the best way to present this activity to make my user's life easier and they often thwart me.
22:44 Michael Kennedy: Yeah, what a... People try. What frameworks did you use to build the Win32 apps.
22:50 Doug Farrell: That was way back in the day I was using Windows 32, the Raw Windows 32 API so it was a nightmare.
22:56 Michael Kennedy: Oh, so you were down at the Charles Petzold Win32 layer, huh? Oh yeah. Wow, that's no VB. Yeah, you don't mess around. You go straight to assembly. You go like the Win32 API to GUIs, that's awesome. Okay, so you said also that the front end stuff had been jQuery, jQuery UI, but that you were all moving on to React, right?
23:16 Doug Farrell: Yeah. There's a couple of groups there's, I think another group in the company is doing Angular but the stuff that we're working on for the backend applications is in React and then using a API to manage the conversation back and forth to the actual data representations.
23:34 Michael Kennedy: Right, talking to stuff like the APIs you're building. So, how do you decide upon React? I mean, I actually am still a bit of a fan of jQuery. If I just want to like a little tiny bit of interaction like, jQuery actually is not too bad, it does have all this setup, set the apps, set the models. If you're building real apps, then React and all these things are cool, but I have so much love for jQuery. But tell me about why you chose React.
23:57 Doug Farrell: I'll have to send you a link because I actually just gave a presentation at the last all hands in January about there's still a place in the world for simple jQuery applications. I'll have to send you a link to that. 'Cause I still like that.
23:57 Michael Kennedy: Yeah, yeah, we'll put it in the show notes.
23:57 Michael Kennedy: Yeah, it's almost like the control programming model of like the...
23:57 Doug Farrell: Yes, exactly.
23:57 Michael Kennedy: Systems and stuff, where you like I'm going to buy a grid control and I drop it here, now it does grid stuff.
23:57 Doug Farrell: That appealed to me 'cause I had written a lot of jQuery code, sort of this is the way I like to make my page look with HTML and CSS and then I'm sort of you know, I'm bolting behavior to it.
23:57 Michael Kennedy: It is not obvious where that behavior comes from in jQuery. It's like I know it's hooking an event somewhere and I cannot figure out why.
23:57 Doug Farrell: Well, and I've written some really huge jQuery application and they're still being used and whenever anybody wants to add features to them, I'm cringing like oh, no, please no, because there's so many side effects and ripples when you change things. It gets a ripple after a while.
23:57 Michael Kennedy: I just, reorganize that, put inside this div and it just stopped working.
23:57 Doug Farrell: I know.
23:57 Michael Kennedy: I know. Thanks for that. Let's not do that.
23:57 Doug Farrell: And then...
23:57 Michael Kennedy: awesome, okay.
23:57 Michael Kennedy: Yeah, that's interesting. Have you thought about writing any desktop apps with say Electron JS?
23:57 Doug Farrell: I have, I', looking into that using like in my case, because I'm trying to get smarter in React. I'm thinking of React Native but I've seen some stuff written in Electron and it's all very interesting and I want to get there because I like that stuff.
23:57 Michael Kennedy: It's really interesting. I wish it had an option for a Python backend. And I just discovered one of the listeners from the Python Bytes Podcast sent me this thing that's more of a way to work with Electron that has a Python backend instead of node.js.
23:57 Doug Farrell: Oh, neat.
23:57 Michael Kennedy: And you get it and install it natively on the system.
23:57 Doug Farrell: Wow.
23:57 Michael Kennedy: It's called Python Electron. I'll put that in the show notes. I wish it was called Proton 'cause Electron.
23:57 Doug Farrell: Yeah, I like that.
23:57 Doug Farrell: Oh, I think that would be great, 'cause I've written wxPython applications, which is a lot of fun, it reminds me a lot of my Windows programming days but and they're really cool but they're so hard to port. They're so hard to give to somebody, 'cause there's so much stuff they have to install that's very specific for the target platform.
23:57 Michael Kennedy: Right, make sure you have Python 3.4 and then you're going to need to do this and then you got to pip and like whoa
23:57 Doug Farrell: Yeah, exactly.
23:57 Michael Kennedy: This is not how I get an app. I get it from the App Store or something.
23:57 Doug Farrell: And then if it actually somebody adopts it, any kind of update process to put changes out there is almost impossibly painful.
23:57 Michael Kennedy: Yeah, yeah, yeah. So this Electron Python would have its own auto updating system and stuff, it would be really neat. And you'd package it all up and it's a .app.
23:57 Doug Farrell: Yeah, that would be nice. I'll have to dig into that.
23:57 Michael Kennedy: I'm looking forward to that. Yeah, yeah, I'll put a link in the show notes.
23:57 Doug Farrell: Cool, thank you.
23:57 Doug Farrell: Well, a lot of the stuff I wrote was a whole combination of jQuery plugins. I used jQuery, jQuery UI, data tables, I used handle bars a lot to do templates within jQuery and HTML, so there's a whole pile of stuff. But it gets to be difficult to make the apps look consistent one app to the next. That gets to be hard to manage.
23:57 Michael Kennedy: Yeah, maybe I'll have to check out React. It sounds very good.
23:57 Michael Kennedy: Yeah, TypeScript is interesting. It has a lot of parallels back to the type annotations in Python 3, yeah. But TypeScript, one thing I'm not a fan of with Typescript is it it's a little more serious about the types. Python is like these types are here to help you. If they get in the way, you can kind of just pretend they're not there. Where TypeScript it's like, no you haven't declared this variable type that's in another file. It can be a pain.
23:57 Doug Farrell: Yeah, it gets to start to feel like...
23:57 Michael Kennedy: It reminds me why I like Python.
23:57 Doug Farrell: Like C or Java in that way that it's very strict typing.
23:57 Michael Kennedy: Yeah, yeah, it really is. But there's a lot of potential, a lot of potential especially in the Python stuff, we have Cython now understanding Python 3 types and then Dropbox is now taking about mypyc which will take type annotated Python and it compile it to C code.
23:57 Doug Farrell: I think that you guys mentioned that.
23:57 Michael Kennedy: A lot of interesting stuff.
23:57 Doug Farrell: On either Talk Python to Me or in Python Bytes you mentioned that one time.
23:57 Michael Kennedy: Yeah, I think in Python Bytes we did. Yeah, that's really new and I think that's quite interesting.
23:57 Doug Farrell: Yes, absolutely.
23:57 Michael Kennedy: So I think that leads into another area that you are passionate about is talking about whether it's Python is it slow? Does it perform well? Is it going to solve your problems, right? Like how do you measure performance anyway?
23:57 Doug Farrell: Well, that's the thing. From my embedded days, I know I'm a speed freak. I want this thing to go as fast as possible.
23:57 Michael Kennedy: You started out in assembly, right? I mean, come on.
23:57 Doug Farrell: Yeah, right, every cycle counts on these little tiny machines, but I'm a big believer in there's no sense in optimizing unless you measure and the other part of that is you have to have a target, like what does fast mean to you? How fast does this have to be? And why? Why do you need it to be fast? I think I've proven to myself many, many times more than I care to remember that every place that I think things are slow and I spend time optimizing I'm completely wrong about what makes the code slow by my intuition. It's only by measuring that you find that this is where it's important to speed things up. 'Cause you could spend a lot of time speeding up something that gets called once a day. I've seen that happen plenty.
23:57 Michael Kennedy: It's so easy to get the wrong impression because you see code that it's complicated and you're like that must be it, 'cause that looks super hard for me to understand, but it's no, no, no, you should have used a set instead of a list here. Or something entirely trivial and you know it's just you know, you would know if you had...
23:57 Doug Farrell: And if you're building a 24/7 server, that thing has a long life, so there's lots of opportunities where things, they get called once an hour, once a day, it's not that big a deal and it's very important to say okay, where is this thing spending most of its time? Is it busy? And in my world today, like I said, there's events that are minutes, hours apart, so I need to pay attention to what matters.
23:57 Michael Kennedy: Right, and if the part that instantiates the hour long process takes and extra 10 milliseconds, but it's easier to understand, you know, let's not stress, right?
23:57 Doug Farrell: Yeah, 'cause you could write, even in Python, you could write some obtuse code and I don't really want to read that again.
23:57 Michael Kennedy: Oh yeah.
23:57 Doug Farrell: And the other thing is that computers are so fast. I mean, we long since passed the tipping point where the computer's cycle time is more expensive than mine. My cycle time is much more expensive, in fact my cycle rate is very slow. So being able to develop code quickly is really important.
23:57 Michael Kennedy: It's important. You're absolutely right. I totally agree with you on what are we optimizing for. Are we optimizing for the computer or for the business or the project, or the product, or whatever it is, right? Like, if you can add two more features to your website a month, but you pay an extra $20 in hosting, who cares?
23:57 Doug Farrell: Yeah, exactly.
23:57 Michael Kennedy: Right?
23:57 Doug Farrell: Yeah, if I can get something delivered that
23:57 Michael Kennedy: Yeah, it's quite interesting.
23:57 Doug Farrell: meets some target window, marketing window, or product window, that's far more important than if I'm saving a millisecond here or there by thinking of some crazy way of doing things.
23:57 Michael Kennedy: Right, or say forget it, we're going to Go or forget it, we're going to write it in C. I think there's a couple of interesting options there. One, you say you need to measure and see what's working. Like, you could rewrite the two slow functions if they're you know computationally slow in Cython. And probably make those go nearly to zero, right?
23:57 Doug Farrell: Yeah, exactly.
23:57 Michael Kennedy: Or something crazy like that and just a tiny bit and not have to completely you know, throw the baby out with the bath water.
23:57 Doug Farrell: I often find that idea of adding like the optimization flag to C it gives you like really incremental improvements versus doing algorithmic optimization. Like, I'm improving the algorithm, my data structure is better where you got orders of magnitude improvement in speed.
23:57 Michael Kennedy: Yeah, yeah, you could choose the wrong data structure even, not just the algorithm, even just the wrong data structure and it could be a thousand times slower 'cause you said well, I always put stuff in a list but really what I wanted was a dictionary 'cause I'm randomly looking up stuff. I'm not looking up by index, right?
23:57 Doug Farrell: Exactly.
23:57 Michael Kennedy: Yeah, pretty interesting. This portion of Talk Python to Me is brought to you by Rollbar. Got a queuestion for you. Have you been outsourcing your bug discovery to your users? Have you been making them send you bug reports? You know, there's two problems with that. You can't discover all the bugs this way and some users don't bother reporting bugs at all. They just leave. Sometimes forever. The best software teams practice proactive error monitoring. They detect all the errors in their production apps and services in real time and debug important errors in minutes or hours, sometimes before users even notice. Companies like Twilio and InstaCart, and CircleCI use Rollbar to do this. With Rollbar you get a real time feed of all the errors so you know exactly what's broken in production, and Rollbar automatically collects all the relevant data and metadata you need to debug the errors, so you don't have to sift through logs. If you aren't using Rollbar yet, they have a special offer for you and it's really awesome. Sign up and install Rollbar at talkpython.fm/rollbar and Rollbar will send you a $100 gift card to use at the Open Collective where you can donate to any of the 900 plus projects listed under the Open Source Collective or to the Women Who Code organization. Get notified of errors in real time and make a difference in open source. Visit talkpython.fm/rollbar today. Another thing that you spoke about was that you're not the youngest programmer. You said you started out in the 80s, right? This is something I'm passionate about because running the podcast, people reach out to me a lot and say, "Michael, I'm 40, it's to late for me, I can't do any of this programming stuff." Like programming's for young folks. I myself am not especially young, but I see the programming space as quite vibrant for people.
23:57 Doug Farrell: I teach at the STEM school, STEM education place near me and I teach programming Python to kids there and I've written some material there. I've been trying to I've been twisting the guy who runs the place, I've been twisting his arm to think about that he really should market to older adults, because I think that this is, you know, like they say, like crossword puzzles is a good way to keep your mind sharp. I think this is like the greatest crossword puzzle ever. This kind of programming.
23:57 Michael Kennedy: Yes, it absolutely is. It's so engaging and you have to think deeply. It's better than video games, it's certainly better that Sodoku.
23:57 Doug Farrell: Oh, yeah. Yeah, absolutely and it's with languages like Python and some of the other modern scripting languages so accessible. There's no the whole I would never want to, I mean I really considered myself a very strong C++ programmer, but I would never want to start there again, because the whole idea of the linker and the include files would make you crazy as a programmer.
23:57 Michael Kennedy: Yeah. Remember the times when the header and the .lib would get out of sync?
23:57 Doug Farrell: Yes.
23:57 Michael Kennedy: It would just not behave, like why is this just crashing randomly? Like I don't even understand. Like so glad that problem's gone.
23:57 Doug Farrell: Oh yeah, me too. I like that stuff, but I don't want to go back there.
23:57 Doug Farrell: Yeah. This is a stupid story but years ago I had taught a couple of adult ed classes to help people get into Windows and DOS, this is how far back it goes. And I would be up there, you know, I'd be talking away about something and it would invariably would happen. I would say, okay, I want you to enter this command and I'd turn to the whiteboard and I'd write on the whiteboard and behind me I'd here this beep, beep as people were restarting their computers because they had typed something wrong.
23:57 Michael Kennedy: Oh my gosh.
23:57 Doug Farrell: Because they were so afraid of getting it wrong I'm like no, you can't hurt it, just go with it.
23:57 Michael Kennedy: Yeah, unless you write format c: or something, you're going to be fine.
23:57 Doug Farrell: You're okay. And for me personally, as I came to it late and I committed to that I'm a self-taught programmer and was always, and I've changed languages a lot but I had to learn new languages so I committed to this sort of constant learning idea early and I have stayed with it my whole career and it's really paid off because I'm still able to learn new things, I'm learning new things all the time at work like I'm React for instance and a little bit of Erlang for crying out loud.
23:57 Michael Kennedy: Wow.
23:57 Doug Farrell: But it's because of this habit that I have that I learn constantly that I'm able to stay competitive and be and contribute to at team of mostly younger guys and you know and mentor people based on my experience and able to learn from people so I hope that I'm not overbearing about that but I'm able to learn from other people 'cause there's a lot of people who, they don't even know what assembly language is but they have a valuable skill that I want to learn and know about.
23:57 Michael Kennedy: Yeah, absolutely. Do you feel like being a self-taught programmer was maybe harder in the beginning but has paid off now that you sort of forced yourself to be able to figure it all out?
23:57 Doug Farrell: Yeah, when I moved into software development fulltime from it was the same company I was doing the process control work, I moved to their software group and all those guys had MSs and doctorates in Computer Science and so I was playing catch up all the time on Fortran and C programming so I worked really hard. But one of the things I did, my boss was a very share guy and I would hit a problem, my goal all the time was like, I would work at it, work at it, work at it before I'd go ask his a queuestion. The deal was and I would try to make it so, he didn't answer the queuestion within one minute and in five years I failed every time. But at least I did the work to ask intelligent queuestions to get there.
23:57 Michael Kennedy: Yeah, that's a really good point. I'm also more or less self-taught programmer. I took a couple of programming classes, C++, Fortran in college and stuff but for the most part I studied math and I feel like there was always this little bit of imposter syndrome feeling, like I know are we talking about like you know what happens deep down in the operating system? Well, I didn't take the operating system class I don't know it maybe as well as that person, right? But I know that I taught myself all of this programming stuff and there's basically nothing that scares me about that. Nothing, but not much that scares me because it's like well I've come this far by forcing myself and you just, like you said, you get in this habit of I spent the last 10 years just continuously learning these things and making these steps and so I know to get to wherever you want to be, it's like just more steps.
23:57 Doug Farrell: Absolutely, and those two things, they drive me pretty hard to consonantly learn stuff that I don't know but I also know myself. I'm an applications engineer. I need to apply this stuff so red-black trees and that kind of ABL automatic balancing stuff is like, I mean, yeah, okay, I'm just going to throw a dictionary at that, it solves the problem. So there's gaps in my knowledge that I don't have but on the other hand I haven't run into too many cases in a long career where I've rally needed that stuff. So, for me I need to know the stuff that is applicable to my job.
23:57 Michael Kennedy: Yeah, a lot of people who take computer science have to do things like calculus and differential equations and stuff. I don't know many programmers who still do differential equations, really. I know there are some, but not most of 'em, right? Most of it, it's like well, algebra, maybe a little geometry, maybe.
23:57 Doug Farrell: Well, I have some types, I make this joke at work all the time, like they often lean on math teachers to be the computer teachers at schools and I'm like, yeah I don't know if that's the right choice sometimes I think it should be English majors because most of my life is tearing apart strings all day.
23:57 Michael Kennedy: That's true. It is a lot of text, isn't it?
23:57 Doug Farrell: It's a lot of text.
23:57 Michael Kennedy: So, while we have a little bit of time left, I'd like to talk about some of the articles that you've written.
23:57 Doug Farrell: Oh yeah.
23:57 Michael Kennedy: Because like I said that's how I got to know you. You've said a big part of your day-to-day work is working on these REST APIs with Flask and one, you wrote a couple of articles on realpython.com about Connexion. For REST APIs. Do you want to tells us about the articles and maybe give us a little overview of what connection is and like how it's helpful?
23:57 Doug Farrell: Yeah, it was I started using it 'cause as I was moving away from all of the form validation and stuff like that and getting more into writing REST APIs, I was looking for ways to make that nicer, make that cleaner. And actually, another guy introduced me to Connexion and that is a really nice tool. It does two things that I really like. Besides providing the automatic interactive documentation through the Swagger UI system, the configuration file that drives it is a nice way to think about the APIs so you can sort of structure what the API's going to look at look like and what it's going to accept long before you get into code 'cause you can really go down a rabbit hole where you're like okay, I'm focused on the code for this one endpoint, this URL endpoint. And you can get to a point where you're like oh, if you look at the API, it's just a list of URLs, they can look like a real mess and not make much sense and Swagger or excuse me, Connexion helps you to think about that in, I think, much clearer ways and structure the system. Plus whenever I've shown off whenever I show off my stuff and I show the Swagger UI, everybody's really impressed it's helped the front end guys to be able to look at the API and use it without actually having to write any code themselves and see how it behaves and what it's expecting, what it returns, that kind of thing. And that's been very powerful
23:57 Michael Kennedy: Yeah, the documentation is really cool, actually.
23:57 Doug Farrell: Yeah, it's fantastic.
23:57 Michael Kennedy: Because you know I definitely prefer the REST APIs over other options that had been popular like SOAP services and stuff. One of the things about those bad SOAP services is you could ask it what functions were there and what you're supposed to do with 'em, right? Whereas like an arbitrary HTTP API, it's like well, I better go figure out some URLs and start trying to understand the JSON, but that's pretty much it, right? And so, Swagger gives you a nice view over top that, right?
23:57 Doug Farrell: Yeah, a really very nice consistent standardized UI that I mean, they're constantly reproving it to interact with the API once you get it up and running. And if you spend time, it does take time to build a configuration file because you're defining all the parameters and what they look like and giving them documentation strings but the payoff is the interactive UI that Swagger provides that gives you the documentation for someone who's interested of the parameters, what they mean, what this thing expects, the types of data for that JSON's going to send you that it expects from you. And that's really powerful.
23:57 Michael Kennedy: Yeah, nice. So you have this Swagger YAML file that defines the API basically and generates the help. How do you get the, like where does Connexion and your code all fit together? I know how does it write like a normal Flask API I would put @route and I would return Flask as JSON, but what do I do in this Connexion world?
23:57 Doug Farrell: Well if you, start configuration, the handshake point, the connection point between the configuration and the code, there's one specific parameter. I think it's, what is it? Object ID or something like that in the it's, I can't remember off the top of my head, but that specifies the path to the handler for that particular URL endpoint. So you might have, you know, it's just a string, it'd be like my_code.my_app.my_method within that app and Connexion is smart enough to actually introspect your code and find that. It'll navigate down that path and importing modules as it goes until it finds that handler and connect that to essentially doing the at route. It will create that within Flask to handle that URL when that gets called. So it's a very nice way.
23:57 Michael Kennedy: I see.
23:57 Doug Farrell: So now I can, you know, it autoconnects that this piece, the configuration piece with the code that's going to handle that configuration.
23:57 Michael Kennedy: That's cool, so it finds the Python files and implement it and sort does the connection for ya.
23:57 Doug Farrell: Yeah. And it tells you if you don't have one like I've often misspelled things and it'll tell you, nope! That doesn't exist, no connection there.
23:57 Michael Kennedy: Yeah, too bad. Okay, this looks really nice. So you have a couple of nice articles on that and people can look through there.
23:57 Doug Farrell: Yeah, on Real Python I've published some articles two articles actually on this REST, this Flask, Connexion REST API. The first one was just an in memory structure that it was a long enough article, I just want to talk about the REST and my interpenetration of REST 'cause it's not really a spec, it's just a convention. And then the second article which sort of was spawned by people asking me in the comments, hey, how can I make this thing persist to a database? So, I took the basic API and code and then added a database connection through SQLAlchemy using SQLite as the database.
23:57 Michael Kennedy: Yeah, that's the way to do it, very nice. I love SQLAlchemy.
23:57 Doug Farrell: Oh, it's terrific. I'm not a big SQL guy so I like that Pythonic way of thinking about it.
23:57 Michael Kennedy: Yes, I'm with you on that. So, that was in Real Python. You also did some on dbader.org which is kind of similar because Dan Bader runs both those sites.
23:57 Doug Farrell: Yeah, he runs both sites, right?
23:57 Michael Kennedy: So, it's a little bit of a connection there. You did one on just sort of Python, getting started in Python but one I thought was pretty interesting and I actually used this as a little bit of research for my recent-
23:57 Doug Farrell: Yeah, I was very happy to see that.
23:57 Michael Kennedy: Yeah, the one called Understanding asyncronous programming in Python. Yeah, I had maybe 10 or 15 articles I read to just see what people were doing and this is a nice one. This talks about very much of what you kind of hinted at the beginning with gevent, with Twisted and some of those things, right?
23:57 Doug Farrell: Yeah, this was an article that actually came out of a presentation I gave at one of the Shutterfly all hands meetings and I turned, I had been in touch with Dan about the first article, just the Python introduction and then he and I talked and he was interested in this asyncronous one and I put that together. It's a passion of mine 'cause I like this whole asyncronous way of thinking. I've written threaded code in C and C++ and lived to regret it. And this whole asyncronous way of thinking although at first it's hard to get your head around, it's a really interesting way to work if you have a program that's a mix of CPU and I/O bound processes or tasks. And I talked about that in my... One of the interesting things I find interesting is people I've taught asyncronous programming to my coworkers and talked about it a lot and there's like this real hurdle to get over for people to get their heads around it. And it's interesting to me because, in reality, humans are asyncronous. Our behavior is asyncronous all the time, but to think about it in this linear presentation of code, it's kind of hard to break out of that linear execution model. But, as you mentioned in my article, I had these sort of thought experiments about the different kinds of way that if people were synchronous or how you would handle asyncronous behavior.
23:57 Michael Kennedy: Exactly. As a parent, if you could only cook dinner and then tell your child to get ready for bed and then serve dinner and then, yeah, it doesn't make any sense, right? The real world definitely is asyncronous.
23:57 Doug Farrell: Oh yeah. Like a polling parent when you're running back and forth, checking every two minutes on everything that's going on.
23:57 Michael Kennedy: Is the dinner ready? Is the kid ready? Is the washing machine done? No? Yes? No, no? How interesting. Yeah, so this is the cool one. I would love to see a followup with some asyncio, maybe some aiohttp or something like that, that'd be fun.
23:57 Doug Farrell: Yeah, I'm going to talk to Dan about maybe how he feels about updating existing articles 'cause I want to take that to the next level with asyncio. I haven't had an opportunity to do much of that in my day-to-day work but I'm definitely heading there 'cause I'm thinking, like I said, I was thinking about Quart and the Sanic and I like it that it's native to Python now, the async stuff.
23:57 Doug Farrell: Yeah, it's crazy.
23:57 Michael Kennedy: And it's beautiful, I love it. It's really, really nice.
23:57 Michael Kennedy: Yeah, that is really nice. Yeah, that deeply indented callback stuff, it's hard to deal with if something goes wrong. It's like well, where do I unravel this to? I'm halfway down this hole and I don't know.
23:57 Doug Farrell: What am I doing now?
23:57 Michael Kennedy: Yeah, this is a problem. You got to take your step back, each of them, but yeah, I'll put that in the show notes, people will definitely enjoy that. So, we're down to just a few minutes of time left so, I suppose I'll ask you the two queuestions here at the end.
23:57 Doug Farrell: Sure.
23:57 Michael Kennedy: Alright, so if you're going to write some Python code, what editor do you use?
23:57 Michael Kennedy: Yeah, you and me both. I'm such a fan of it, as well. I think it's really great. So if you're an IntelliJ based person doing PyCharm or something, Brian Oken shared this with me today, this thing called Power Mode II which is one of the plugins which the faster you type, it'll start to shoot flames and sparks from your code as you type.
23:57 Doug Farrell: Oh, cool!
23:57 Michael Kennedy: So if you need to do a presentation showoff, you plug that in, take it to the next level. It's, of course, completely silly but fun.
23:57 Doug Farrell: I might try that, 'cause one of the best things my mom ever made me do was I took a typing course in sixth grade and it's paid off because I type pretty fast. A lot of programmers don't but I type pretty quick so I'll definitely try that out.
23:57 Michael Kennedy: Yeah, you'll have fun with that. Definitely for presentations. Alright, so notable PyPI packages?
23:57 Doug Farrell: Well, I really like, as I mentioned, Connexion which is this REST API. And then the other side of that is Marshmallow. This is a serialization deserialization library that let's you because SQLAlchemy deals with Pyhton objects, you can't just hand those off to something that wants to turn them into JSON like Connexion.
23:57 Michael Kennedy: Right, if you try to return it out of your method, it'll freak out and go I cannot serialize this thing to JSON.
23:57 Doug Farrell: Yeah, there's no matching, there's no matching thing for Python datetime object in JSON, so this Marshmallow thing allows you to, just as you would in SQLAlchemy, build a model that matches your table structure. Marshmallow does the same thing, but it builds a model that tells the system oh, this is the data I'm expecting, this is the structure and this is how I'm going to convert it to something that's JSONable.
23:57 Michael Kennedy: That is cool.
23:57 Doug Farrell: And back and forth.
23:57 Michael Kennedy: Yeah, that's really cool. I should definitely learn about Marshmallow.
23:57 Doug Farrell: Oh, it's sweet. It has the same thing as SQLAlchemy where you have relationships. The one to many, many to many relationships and the data table. You can also set those up in Marshmallow so that if you have a SQLAlchemy object that has, you have on record but it's got many relationship to others, Marshmallow will take care of that, as well. It'll just go down the tree and serialize them all.
23:57 Michael Kennedy: It just does the whole object graph, wow.
23:57 Doug Farrell: Yeah, you just...
23:57 Michael Kennedy: Very nice, okay.
23:57 Doug Farrell: You take that return value and you return it from your call and Connexion will then turn it into a JSON string.
23:57 Michael Kennedy: Okay, I'm loving this one. Good choice. Alright, well.
23:57 Doug Farrell: I hope this was helpful.
23:57 Michael Kennedy: Yeah, this has been really interesting. I really appreciate you being on the show, Doug.
23:57 Doug Farrell: Oh, I'm very glad to be here, it's great. I've really enjoyed it myself.
23:57 Michael Kennedy: Yeah, absolutely. You got a bunch of recommendations and I think a really interesting way to look at the whole async programming thing, so I'm loking forward to your followup articles that you'll write on this.
23:57 Doug Farrell: Yeah!
23:57 Michael Kennedy: Yeah, it'll definitely be fun.
23:57 Doug Farrell: That would be great 'cause I love that little bit of extra complexity and then being able to do many things at once. It's not only important to me, but it's fun, as well.
23:57 Michael Kennedy: Yeah, it's definitely fun. Yeah, we don't need to do crossword puzzles or Sodoku, we can just work on our thread code, right?
23:57 Doug Farrell: Yeah, right. In my classes, I teach the kids using the turtle module because I get them right into drawing on the screen. I do the same thing with adults. It'd be great, I think.
23:57 Michael Kennedy: Yeah, absolutely, it would. Alright, well, thanks for being here, good to talk to you.
23:57 Doug Farrell: Thank you, Michael.
23:57 Michael Kennedy: Bye.
23:57 Doug Farrell: Bye bye.
23:57 Michael Kennedy: This has been another episode of Talk Python to Me. Our guest on this episode has been Doug Farrell and it's been brought to you by LogRocket and Rollbar. LogRocket will help you avoid the back and forth, helping your users with problems on your site. Visit talkpython.fm/logrocket to get started with their free tier and get a pixel perfect replay of what your users saw and the state of your app. Rollbar takes the pain out of errors. They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until your users complain, of course. Track a ridiculous number of errors for free as Talk Python to Me listeners at talkpython.fm/rollbar. Want to level up your Python? If you're just getting started, try my Python Jumpstart by Building 10 Apps or our brand new 100 Days of Code in Python. And if you're interested in more than one course, be sure to check out the everything bundle. It's like a subscription that never expires. Be sure to subscribe to the show, helping your favorite podcast search for Python. We should be right at the top. You can also find iTunes feed at /itunes, Google Play feed at /play and Direct RSS feed at /rss on talkpython.fm. This is your host Michael Kennedy. Thanks so much for listening, I really appreciate it. Now, get out there and write some Python code.