Join Doug Farrell and me as we discuss his career and what he's up to at Shutterfly.
You'll learn about the Python stack he's using to work with not just bits and bytes,
but physical devices on a production line for creating all sorts of picturesque items.
You'll also hear how both he and I feel it's a great time to be a developer,
even if you're on the older side of, say, 30, 40, or beyond.
This is Talk Python to Me, episode 182, recorded October 4th, 2018.
Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy. Follow me on Twitter where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm, and follow the show on Twitter via at Talk Python.
This episode is brought to you by LogRocket and Rollbar.
Please check out what they're offering during their segments. It really helps support the show.
Doug, welcome to Talk Python.
Thank you very much, Michael. I'm glad to be here.
Yeah, I'm glad to have you here. It's great.
We've been interacting on email and on social media and stuff.
I've talked about some of the articles you've written before, and you've sent in recommendations and various things for us.
So it's nice to finally have you here.
Yeah, I'd like to drop notes to you and Brian on the Python Bytes podcast about articles that I've written in RealPython, the site.
Yeah, there are definitely some good ones.
And I want to dig into some of those, actually, as we get further into the show.
But let's start with your story about how you got into programming in Python.
Well, I actually got started pretty late.
I was probably in my late 20s by the time I got into programming back in the early 80s, late 70s, early 80s.
I had failed at Fortran years before the first time I went to college, mostly because, you know, back then it was all punch cards and having to go down to the dungeon to hand it over to somebody who would tell you three days later whether your program worked.
I didn't really get excited about it until it was either.
Yeah, I could see how you wouldn't be so excited about that.
That's not the same as, you know, fun GUI editors and compilers and Stack Overflow and all that stuff, right?
It's a different world.
Well, plus, you know, I saw lots of people who actually left the program, but they'd come down the stairs with a giant stack of COBOL cards, drop it.
The whole thing would be spread all over the place, and they'd leave the program crying, you know, because they just couldn't stand to sort them again.
They just lost their will to be part of that anymore.
That's it.
They'd moved on to something else.
So I got excited about it when I bought my first personal computer, which was back then I bought a Radio Shack color computer and taught myself some basic and a little bit of 6809 assembly language for some, you know, because as most young guys, I was thinking I was going to be a game programmer.
How interesting.
You know, you think of, like, getting started programming.
People say, well, Michael, programming's kind of hard.
I have a hard time wrapping my mind around, like, Python or JavaScript.
Like, starting with a similar.
Like, that is starting.
Seriously.
I still can't do similar.
The color computer was so slow.
It was less than a megahertz clock speed.
So to do anything fast, you had to really get down and dirty.
Yeah.
But, okay, so you started there.
So that gives you a good foundation of it's only going to get easier from here.
Right, yeah.
And I had my – actually, I was back in school by then going after my physics degree, BS in physics.
And my senior project was building a fairly primitive CAT scan device with some high-precision translation tables and using Pascal on this wacky computer called a Terrax.
And that really – I really enjoyed that.
I spent most of my time doing that besides failing French.
And that gave me enough buzzwords to get my first engineering job as a process control engineer doing automation systems for municipal water systems and gas pipelines.
Wow, that sounds interesting.
So that's a lot of, like, controlling hardware, right?
Right, yeah.
Which is kind of a theme, it sounds like, from all the stuff that you've been doing.
Yeah, it was kind of a big scale, you know, like a macro version of hardware control where you're doing really huge systems.
I think the biggest thing I did was I automated a 35-gallon-a-day, 35-million-gallon-a-day clean water system in Arizona.
I think about – there were certain steps in my programming career that they were sort of, you know, take-your-breath-away moments.
And I remember the first time I wrote – this is for some other company – I wrote my first e-commerce system, and it was doing, like, $4,000 transactions per transaction.
I was like, wow, I had really better not mess this up.
You know, if we do this wrong on scale, it's going to be a problem.
Did you have that feeling when you were working on that much water and that much hardware?
Yeah, because the scale of this thing is huge.
I mean, this was early in its construction, so we were actually – when I was out there, I was actually myself and the other guy we were working with.
We were able to ride our bikes inside of the 35-million-gallon tank because it wasn't in use yet.
You had your own velodrome.
Yeah.
And we had one disaster, thank God it wasn't me, where we had a flood where we managed to wash away a golf course putting green down the road.
Hopefully no one was putting at the time.
Well, it was late at night, and we heard – the course manager was over screaming at us, and then it turned out to be not us.
So that was a nice escape.
Well, that's really good.
That's really good.
So what languages were you using to do all this?
This was a proprietary language that the company had written.
It was a software simulation of hardware.
So when you do hardware control, there would be things like timers and PID controllers and digital switches and analog inputs and outputs, comparators, all these hardware things that you could wire together.
And this simulated those.
And in that language, the variables – you could think of the variables as wires.
So anything you connected a variable to was like wiring those two things together.
So you could string all these things like an analog input to a comparator to a set point control to a PID loop to an analog output that would control a valve that would keep a tank from overfilling or underfilling.
Yeah, that's really interesting.
Do you see as much of those sort of custom programming languages now?
Or do you see people adopting things like Python and JavaScript and other languages for that purpose?
Well, I know that that company is still big in the game and they're still around.
And I think my experience with municipalities and gas pipeline companies is that they're very reluctant to change.
They're always going to stick with something that works.
I don't know if you've ever worked with PLC controllers, programmable logic ladder controllers.
Yeah, I only have heard the term.
Those are very popular because they work and people know them.
So the same thing applies to municipalities.
It works and they know them.
I mean one of my experiences when I was there was having a realization that they had our computer stuff in the same depreciation column in their spreadsheet as manhole covers.
So they expected our computer stuff to last like 50 years like a manhole cover.
And I was like, really?
I don't think so.
This is not going to work out the way you think this is going to work out.
So that was nice.
Yeah.
They were like, well, how are we going to get parts to replace this?
I'm like, no, you're going to replace the whole thing in about 20 years, I think.
Maybe even less.
Yeah, how interesting.
You moved on from there, right?
Towards the web?
Yeah, after I left that company, I worked at a high-speed pick-and-place machine.
It was kind of a robot that assembled electronics boards by picking up chips, flying them over a camera, put them on the board.
And about, I don't know, the fast one was like 20,000 parts an hour, which is really moving.
And that was my really—
Kind of stay away from it, right?
Don't get too close to it, huh?
Yeah, there was lots of stuff about don't open the door when it's moving because you could get really hurt.
And we were kind of—that wasn't—I went to a trade show, and they really had some fast and big machines like that that would scare you to death.
Yeah, I can imagine.
But that was sort of like my low-level embedded programming, which I really enjoyed.
And eventually, I went to work for a company that did—where my internet exposure came in, where they were trying to put reference titles like encyclopedias online.
And now I worked at Shutterfly.
This is probably pre-Wikipedia, right?
Yeah, this was back in the Encarta days and Griller's Encyclopedia Online.
And I actually worked on the CD-ROM version of that and then the web version.
Yeah, cool.
Yeah, and now you're at Shutterfly, and it sounds like a really interesting place to be.
And it also sounds like you're getting to do some of this sort of software meets hardware thing.
Yeah, it's satisfying.
I really like that part where software meets the real world.
It's not just all a bunch of bits moving around in the computer.
It has enough of the things that I write.
Everything I write is all on our production floor, not customer-facing.
So it allows our production people to interact with the presses and other machinery that actually produces books.
Shutterfly is a digital book publisher where customers can upload their photos and get all kinds of products.
Like photo books you can send out for gifts or printed stuff or printed photos out of your digital photos, things like that.
Pillows and coffee mugs and desk frames and all kinds of big canvas prints, which are all interesting products.
Our motto at Shutterfly is sharing life's joy.
So it's an interesting way to think about photos and how there's so many digital photos.
And then people get so many and they get overwhelmed about what am I going to do with this thing, this pile of pictures.
You know, I just did that.
I ordered a photo book for my family because we have like 30,000 photos.
And I'm like, we're not going to go through these.
There's just – it's overwhelming.
But if I could put it on the coffee table and friends come to visit, like we can experience them in a different way.
It's pretty cool.
Yeah.
It makes great gifts.
I mean, you know, grandparents.
I know my daughter sends me books that she makes with our product.
And I love that stuff.
Hey, you printed this for me.
You just don't know it.
Yeah.
Usually, hopefully, I don't find out that I printed it and made it wrong.
That's right.
This is messed up.
Oh, I remember that, Bob.
Yeah.
Oh, sorry.
Yeah.
So that's – doing that stuff satisfies my kind of need to have the computer touch the real world because we interact with the presses and hardware directly or more – a little directly.
Yeah.
It's not as dangerous as like high-speed stuff where people could get hurt.
Yeah.
You don't want to be putting people onto a board.
No.
Yeah.
This sounds like really fun.
And it's that software meets hardware thing.
And you guys use a lot of Python at Shutterfly, right?
Yeah.
They actually – they acquired the company I was working for and we were doing all Python stuff.
We were one of their smaller competitors and they acquired us.
And I came on board as a full-time Shutterfly employee, which was great.
And that was their first exposure to Python.
They had been a traditionally and still are a big Java shop.
A lot of the work that they had done and a lot of their infrastructure is in Java.
But through acquisitions, they've become a heterogeneous tech stack company because they have Java, Python, PHP, mobile app development languages, JavaScript, of course, all over the place.
Yeah, of course.
But having us come on board, one of the things that happened right away was that we were able to take over a section of gifting products because Shutterfly has a huge – they do a huge amount of book printing, of course.
But the gifting products are becoming – is a fast-growing market for them.
And my group that I'm in was able to implement a lot of the gifting products and is continuing to implement new gifting products all the time in Python.
And almost that whole – almost the whole stack for that production is in Python.
That's really cool.
Do you guys see the agility and the ability to, you know, pip install anti-gravity to – as a real advantage for your team?
Maybe even within the company, people go, oh, yeah, we know those guys can deliver quick.
Yeah, that's a – there's two parts of that is that we can be extremely agile.
And we first – when we first did our first pass at producing gifts products, I think it was – I think it was a real eye-opener for people to see how quickly we could actually bring a product to bear into production.
And then not only get it there to where they could start producing it, but iterate quickly to improve the process because, you know, the first pass would always have issues or they'd want changes.
And the ability for us to iterate very quickly I think is a huge benefit for not only us but also the company in general.
Yeah.
Yeah, there's an interesting story about how Google acquired YouTube and they had Google Video and a bunch of C++ guys trying to compete with them.
And YouTube was a little upstart built on Python and they were just out, you know, outperforming and adding features.
Maybe they needed more servers or something to make it go, but it didn't matter, right?
So kind of like you, Google – or like Shutterfly, Google solved the problem by buying YouTube and that fixed it.
Yeah, and it's – you know, we've been growing because they continue to – you know, they continue to sort of give us the opportunity to add new products, sort of subsume some of our products into our – the way we work in Python.
And it's – we're doing a lot of infrastructure work.
Any growing company has to do a ton of infrastructure work all the time.
And we're doing a lot of that infrastructure work and architecting based on the experience that we've had with Python and some of the tools that we use and interface with using Python.
And like you said, the pip install thing, we have our own PyPy server.
Oh, nice.
With modules that we have figured out are safe and to use and build against.
But because we have our own, we have a sort of a vetting process to see new ones that we can add because, well, I know for myself, I'm always like, hey, there's a cool module.
I'm twisting people's arms to get it into our server so I have access to it.
That's really cool.
So what technology are you using for your private PyPy server?
Well, we have just a copy of the PyPy server code.
It doesn't have the whole front end, but it just has the back end where you can just do a pip install.
So long as you point at that server with a pip config file.
Right.
So you can configure –
It will just grab files.
Yeah, you can configure your user profile.
So if you say pip install, it first looks there, right?
Yep.
That's pretty cool.
My own development.
We're moving from server-side development to client-side where we're using containers, Docker containers.
And for that stuff, it's much – we often – this is how stuff gets started.
We'll often grab a module from the regular, the public PyPy server, PyPI server.
And then that – once you see some success in that and people start to review your code, that module will get vetted and checked out and moved into our private PyPy server.
Yeah.
Nothing is a more convincing argument than success.
I know.
I've been in so many places where people debate, oh, if we do it this way, it'll be better.
No, that way, it'll be better.
Like, you know what?
We're just going to build – like, I just built it this way.
I know you said it can't work, but here it is working.
Can we discuss whether or not it's still working?
Yeah.
Okay, fine, it's working.
Yeah, you're right.
That would work.
Delivery often wins.
Let's do that.
Being able to deliver wins.
Yeah, it certainly does.
This portion of Talk Python to Me is brought to you by LogRocket.
LogRocket will help you view errors in activity from your user's perspective.
Did they tell you they couldn't log in?
Well, with LogRocket, you'll get a session where you can see a pixel-perfect replay of what your user saw at that moment.
The ability to inspect the state of the application at any point in time.
You'll view every network request and response on their session, as well as a console log of all the JavaScript messages and errors.
And it goes beyond just errors.
You'll learn where your users are getting stuck.
LogRocket intelligently surfaces the key moments of user frustration showing you how to improve your site.
And LogRocket works with your stack regardless of the language or framework you're using.
And they even provide SDKs for popular front-end technologies like Vue.js, React.js, and more.
Do you already use systems such as Rollbar?
Well, you're in luck because they integrate seamlessly with Rollbar and others for great visibility of front-end and back-end errors.
Understand the cause of every bug and customer issue today.
Visit talkpython.fm/LogRocket to get started with their free tier.
Let's look a little bit inside at what you guys are working with, some of the libraries and stuff.
I think it's interesting for people in different industries or not in companies like Shutterfly to just see what kind of apps you're building.
So could you give us just a quick flyover of some of the tools you're using?
A lot of the stuff I build directly that I guess I consider myself a full-stack developer at Shutterfly where I build the back-end server and I also do the front-end code in JavaScript to build web applications.
I've been doing a lot of that with just plain jQuery, but I'm slowly moving to React for my front-end framework.
But for the back-end, I prefer to use Python Flask with gevent and SQLAlchemy as the database interface.
And we try to make our stuff as asynchronous as possible because we have a lot of users, not only users, but we also have a lot of processes that communicate together.
We also use a message bus system as our interconnection between processes.
So rather than like we're rapidly getting to a point where we're decoupling from database access directly to using a message bus to talk to a data service.
So all of the database access is sort of hidden away.
There used to be a lot of like per max, not good, and we're moving away from that.
And then we use like we use RabbitMQ as a database as a message system.
We do a lot of – on the back-end, besides the like full-stack stuff that I do, there's a lot of processes that we run in Python.
So we have a lot of twisted code.
We use twisted code for asynchronous work.
We also use a lot of gevent for the same thing.
I know I'm more comfortable with gevent than I am with twisted, although I've written quite a bit of twisted code so that we can handle multiple – we don't do so much networking code as much as just there's multiple things in motion that take time.
Like as in a production environment where stuff is getting printed and cut and bound and shipped, all those events, they're separated by time.
So asynchronous code is really handy for that.
So you can set something.
You can set a state and then you get a call back, you know, who knows, 5, 10, 20, an hour later that it's actually finished or moved on to another state.
So to be able to handle essentially thousands and thousands of those state-driven processes is really handy.
That's really cool because if you tried to just wait for that response, that would be a problem.
Oh, yeah.
If you wait for an hour, this Flask server just won't scale.
You know, I mean, we set the time on it.
I'd have a bunch of production lead guys in my office stamping their feet in a heartbeat if that was the case.
How interesting.
When you talk about doing asynchronous stuff with Flask, is that using, say, gevent on Flask?
Or are you doing, like, architecture that has queues?
Well, we have a couple things.
I use gevent in Flask to try to make that as – add asynchronous behavior to that.
It's still – because of the database access, it still can get paused on a database call.
That's one of the reasons why we are moving to – the data access is going to another process.
So that other process now becomes – you know, it's handled – there's multiples of those.
And it handles the database.
But my call from my application or any other application can be asynchronous to that.
It's no longer waiting because it's – now it's going to get a call back when it's done.
Yeah, that's really cool.
And it just queues up on that other system probably.
It won't time out, for example, and things like that, right?
Yeah, those other systems are all also entirely asynchronous.
So it has a big queue of things that are ongoing and it's working on and, you know, it'll get back to you when they're done.
And those come across the message bus as an event.
I think it's so interesting to look at the different types of architecture choices people make based on, you know, what you're trying to do, right?
You're here working with latencies in the minutes or hours.
So you might choose a really different architecture, whereas people say, well, the database is kind of slow.
It takes 30 milliseconds to get back to us and we want to go scaling.
30 milliseconds, I laugh at that.
I'll just use an – it'd be like an ASICIO event loop or something like that.
That would be fine for that, right?
So quite interesting to think about.
So that's the backend stuff.
I guess one thing I did want to ask you about when you talked about G-Event and Flask and all that, and I don't know that this would – I don't think that this would really necessarily be the right application for it.
But have you looked at Quart?
I have, and I've been following the ASIC stuff like SANIC and Quart, and I'm trying to get up to speed on the Python 3.7, the ASICIO, and ASIC Await stuff because in many ways, I'm moving away from really needing Flask.
I mean it just – it provides a lot of features that I don't use.
Like I don't do any web form validation.
I'm doing all either WebSocket or AJAX calls, REST APIs.
So I don't do a lot of form validation.
I don't serve a lot of files.
I'm mostly creating REST APIs.
So I just want to make that part faster, and those things that you mentioned would do that.
So I'm looking into that to take better advantage of the system.
Yeah, that makes a lot of sense.
And there is a lot of stuff in these web frameworks that you just don't need if the response is Flask.jsonify.
For my case, I don't even serve up the web pages, the front end anymore myself.
That's done by a dedicated web server.
So my application is strictly – has turned into a REST API.
That's a really interesting place for people to work who want to work on the web, but they are not designers.
Yeah.
Right?
They don't want to try to do all the CSS and layout and make it pretty and put the hero banner on there and all that stuff.
There's still some really interesting and challenging programming to be done building this back-end layer, I think.
Yeah, I absolutely agree.
I am currently working on this gigantic project, which is this REST API that drives a big part of our business.
And I find that part – it's challenging because I'm always negotiating with the front-end guys about what the payload looks like and what the back-and-forth looks like.
I also enjoy the front-end because that gives me – then I'm arguing with myself about what's the best way to get this data across.
But also, I really enjoy the GUI part.
I used to be a – for a short time, I was a Windows developer.
And I like that user interaction and trying to figure out what's the best way to present this activity to make my user's life easier.
And they often thwart me, but –
Yeah.
What –
Hey, we'll try.
What frameworks did you use to build GUI?
Well, that was way back in the day.
I was using Windows 32, the raw Windows 32 API.
So it was a nightmare.
Oh, so you were down at the Charles Petzl Win32 layer, huh?
Oh, yeah.
Wow.
That's no VB.
Yeah, you don't mess around.
You go straight to a simulated learn programming.
You go to, like, the Win32 API to do GUIs.
That's awesome.
Okay.
So you said also that the front-end stuff had been jQuery and jQuery UI, but that you were all moving on to React, right?
Yeah, there's a couple groups.
There's, I think, another group in the company that's doing Angular, but the stuff that we're working on for the back-end applications is in React and then using an API to manage the conversation, but back and forth to the actual data representations.
Right, talking to stuff like the API tree building.
So how do you decide upon React?
I mean, I actually am still a bit of a fan of jQuery.
I just want, like, a little tiny bit of interaction.
Like, jQuery actually is not too bad.
It doesn't have all this setup.
Set the apps.
Set the models.
I mean, if you're building real apps, then React and all these things are cool.
But I still have some love for jQuery.
But tell me about why you chose React.
Well, I'll have to send you a link because I actually just gave a presentation at the last Engineering All Hands about there's still a place in the world for simple jQuery applications.
And I'll have to send you a link to that because I still like that.
Yeah, yeah, we'll put it in the show notes.
Because it's one of the reasons I like it.
It's quick.
You can get a prototype up and running very fast.
And it's not so costly in time that you wouldn't want to throw it away if it gets popular to rebuild it in something else, which is handy.
But React was – I was actually going down the path of Angular.
And then, you know, we sort of switched gears and I jumped on React.
And, well, anybody who's done this kind of work probably has come from another – who's come from another world.
You know, you wait five minutes in the JavaScript world and there's another framework out there.
It took me a long time to actually jump on board.
But I like the conceptual nature of – or the concept behind React where you have components that you can glue together.
And those components can have a visual and a behavior aspect to them.
Yeah, it's almost like the control programming model of, like, the older Windows systems and stuff.
Where you can – you know, I'm going to buy a grid control and I drop it here and now it does grid stuff.
That appealed to me because I had written – you know, writing a lot of jQuery code, you're sort of – this is a way I like to make my page look with HTML and CSS.
And then I'm sort of, you know, I'm bolting behavior to it with a –
It is non-obvious where that behavior comes from in jQuery.
It's like I know it's hooking an event somewhere and I cannot figure that out.
Well, I've written some really huge jQuery applications and they're still being used.
And whenever anybody wants to add features to that, I'm, like, cringing.
I'm like, oh, no, please no.
Because there's so many side effects and ripples when you change things.
So it gets to riddle after a while.
I just reorganized it.
I put inside this div and it just stopped working.
Thanks.
I know.
Thanks for that.
Let's not do that.
But React seems more – I mean, I like the module system so that you can actually break it.
I like JavaScript.
And to me, in many ways, I know people don't like JavaScript who come from other languages.
But to me, it feels very complementary to Python, the whole idea of objects and arrays and the for each and all that kind of stuff.
And now that things like React and Angular, React in particular, has these idea of web components that have look and feel and behavior,
that feels like a real way to program a GUI, especially –
Yeah.
It looks like a web platform as an application platform to deliver stuff.
Yeah.
That's interesting.
Have you thought about writing any desktop apps with, say, Electron.js?
I have.
I'm looking into that using – in my case, because I'm trying to get smarter on React, I'm thinking of React Native.
But I've seen some stuff written in Electron, and it's all very interesting.
And I want to get there because I like that stuff.
It's really interesting.
I wish it had an option for a Python backend.
Yeah.
And I just discovered – one of the listeners from the Python Bytes podcast sent me this thing.
It's more of a way to work with Electron that has a Python backend instead of Node.js.
Oh, neat.
And you get it – install it natively on the system.
It's called Python Electron.
Yeah.
I'll put that in the show notes.
I wish it was called Proton because Electron, Proton, Python.
But a name is missed here.
But it's really quite interesting.
And I think you could take React and the JavaScript and HTML and pair that with this Electron Python backend.
It would be very smart.
Oh, I think that would be great because I've written WX Python applications, which is a lot of fun.
It reminds me a lot of my Windows programming days.
And they're really cool, but they're so hard to port.
They're so hard to give to somebody because there's so much stuff you have to install that's very specific for the target platforms.
Right.
Make sure you have Python 3.4.
And then you're going to need to do this.
And then you've got to pip it like, whoa, this is not how I get an app.
I get it from the app store or something.
And then if actually somebody adopts it, any kind of update process to put changes out there is almost impossibly painful.
Yeah.
Yeah, yeah.
So this Electron Python would have its own auto-updating system and stuff.
It would be really neat.
And you package it all up and it's .app or .exe.
Yeah, that would be nice.
I'll have to dig into that.
I'm looking forward to that.
Yeah, yeah.
I'll put a link.
Cool.
Thank you.
Yeah, so when I was thinking of JavaScript front-end stuff, I'm kind of thinking about Vue.js as probably where I would like to work.
Although I don't have any intention to do tons of rich front-end work right now.
But, yeah, there's a lot of good choices.
But like you said, by the time this episode ships, there's going to be another one.
Well, a lot of the stuff I wrote was a whole combination of jQuery plugins.
I used jQuery, jQuery UI, data tables.
I used handlebars a lot to do templates within jQuery and HTML.
So it was a whole pile of stuff.
But it gets to be difficult to make the apps look consistent, one app to the next.
That gets to be hard to manage.
Yeah.
Maybe I'll have to check out React.
It sounds pretty good.
I think you'll like it.
I know Angular.
I was working with Angular, and I really liked that.
One of the big concepts with Angular is it feels like they're trying to add functionality to HTML.
And one of the differences I see with React is it's the exact opposite.
They're trying to add HTML to functionality.
So it's like embedding HTML into JavaScript.
Conceptually, it's a little different.
But one of the things I had trouble with was in Angular, it's sort of dependent on TypeScript, which I really like TypeScript.
But if you look at the code that's generated by that when it does the transpiling, thank God there's TypeScript because I definitely would not want to write that JavaScript.
Yeah.
TypeScript is interesting.
It has a lot of parallels back to the type annotations in Python 3.
But TypeScript, one of the things I'm not a fan of with TypeScript is it's a little more serious about the types.
Python is like, these types are here to help you.
If they get in the way, you can kind of just pretend they're not there.
Whereas TypeScript, it's like, no, you haven't declared this variable type that's in another file.
You know, it really is.
It can be a thing.
It gets to start to feel like C or Java in that way, that it's very strict typing.
Yeah, it really is.
But there's a lot of potential, a lot of potential with types, especially in the Python stuff.
We have Cython now understanding Python 3 types.
And then Dropbox is now talking about mypyC, which will take type annotated Python and compile it to C code.
I think that you guys mentioned that on either Talk Python to Me or in Python Bytes, you mentioned that at one time.
Yeah.
Yeah, I think in Python Bytes we did.
Yeah, that's really new.
And I think that's quite interesting.
Yes, absolutely.
So I think that leads into another area that you are passionate about is talking about whether it's Python, is it slow?
Does it perform well?
Is it going to solve your problems, right?
Like how do you measure performance anyway?
Well, that's the thing.
I mean, people, you know, from my embedded days, I know I'm a speed freak.
I want this thing to go as fast as possible.
You started out in assembly, right?
I mean, come on.
Yeah, right.
Every cycle counts on these little tiny machines.
But I'm a big believer in there's no sense in optimizing unless you measure.
And the other part of that is you have to have a target.
Like what does fast mean to you?
How fast does this have to be?
And why?
Why do you need it to be fast?
I think I've proven to myself many, many times more than I care to remember that every place that I think things are slow and I spend time optimizing it, I'm completely wrong about what makes the code slow by my intuition.
It's only by measuring that you find out this is where it's important to speed things up because you could spend a lot of time speeding up something that gets called once a day.
I've seen that happen plenty.
It's so easy to get the wrong impression because you see code that is complicated and you're like, that must be it because that looks super hard for me to understand.
But it's no, no, no.
You should have used a set instead of a list here.
Or something entirely trivial and it's just you would know if you had looks.
And if you're building a 24-7 server, that thing has a long life.
So there's lots of opportunities where things don't – they get called once an hour, once a day.
It's not that big a deal.
And it's very important to say, okay, where is this thing spending most of its time?
Is it busy?
And in my world today, there's lots – like I said, there's events that are minutes, hours apart.
So I need to pay attention to what matters.
Right, and if the part that instantiates the hour-long process takes an extra 10 milliseconds but it's easier to understand, let's not stress, right?
Because you could write – even in Python, you could write some obtuse code and I don't really want to read that again.
Oh, yeah.
And the other thing is that computers are so fast.
I mean it's long – we long since passed the tipping point where the computer's cycle time is more expensive than mine.
My cycle time is much more expensive.
In fact, my cycle rate is very slow.
So being able to develop code quickly is really important.
It's important.
You're absolutely right.
I totally agree with you on what are we optimizing for.
Are we optimizing for the computer or for the business or the project or the product or whatever it is, right?
Like if you can add two more features to your website a month but you pay an extra $20 in hosting, who cares?
Yeah, exactly.
Right?
Yeah, if I can get something forever that meets some target window, marketing window or product window, that's far more important than if I'm saving a millisecond here or there by thinking of some crazy way of doing things.
Right.
Or say, forget it.
We're going to go.
Or forget it.
We're going to write it and see.
I think there's a couple of interesting options there.
One, you say you need to measure and see what's working.
Like you could rewrite the two slow functions if they're, you know, computationally slow in Cython and probably make those go nearly to zero.
Yeah, exactly.
Something crazy like that.
And just a tiny bit and not have to completely re, you know, throw the baby out with the bad boy sort of thing.
Oh, I often find that idea of adding like the optimization flag to see gives you like really incremental improvements versus doing algorithmic optimization.
Like I'm improving the algorithm.
My data structure is better where you get orders of magnitude improvement in speed.
Yeah, yeah, you choose the wrong data structure, not just the algorithm, even just the wrong data structure.
And it could be a thousand times slower because you said, well, I always put stuff in a list.
But really what I wanted was a dictionary because I'm randomly looking up stuff.
I'm not looking it up by index.
Right, exactly.
Yeah, pretty interesting.
This portion of Talk Python to Me is brought to you by Rollbar.
Got a question for you.
Have you been outsourcing your bug discovery to your users?
Have you been making them send you bug reports?
You know, there's two problems with that.
You can't discover all the bugs this way.
And some users don't bother reporting bugs at all.
They just leave, sometimes forever.
The best software teams practice proactive error monitoring.
They detect all the errors in their production apps and services in real time and debug important errors in minutes or hours, sometimes before users even notice.
Teams from companies like Twilio, Instacart, and CircleCI use Rollbar to do this.
With Rollbar, you get a real-time feed of all the errors.
So you know exactly what's broken in production.
And Rollbar automatically collects all the relevant data and metadata you need to debug the errors so you don't have to sift through logs.
If you aren't using Rollbar yet, they have a special offer for you, and it's really awesome.
Sign up and install Rollbar at talkpython.fm/Rollbar.
And Rollbar will send you a $100 gift card to use at the Open Collective, where you can donate to any of the 900-plus projects listed under the Open Source Collective or to the Women Who Code organization.
Get notified of errors in real time and make a difference in Open Source.
Visit talkpython.fm/Rollbar today.
Another thing that you spoke about was that you're not the youngest programmer.
You said you started out in the 80s, right?
And this is something I'm passionate about because running the podcast, people reach out to me a lot and say,
Michael, I'm 40.
It's too late for me.
I can't do any of this programming stuff.
Like, programming is for young folks.
I myself am, you know, not especially young.
But I see the programming space as quite vibrant for people.
I teach at this STEM school, STEM education place near me, and I teach programming Python to kids there.
And I've written some material there.
And I've been trying to – I've been twisting the guy who runs the place.
I've been twisting his arm to think about that he really needs – he really should market to older adults
because I think that this is – you know, like they say, like, crossword puzzles is a good way to keep your mind sharp.
I think this is, like, the greatest crossword puzzle ever is kind of programming.
Yes.
And it's –
It absolutely is.
It's so engaging and you have to think deeply.
It's better than video games.
It's certainly better than Sudoku.
Oh, yeah.
Yeah, absolutely.
And it's – with languages like Python and, you know, some of the other modern scripting languages, it's so accessible.
There's no – there's no, you know, the whole – I would never want to.
I mean, I really considered myself a very strong C++ programmer.
But I would never want to start there again because the whole idea of the linker and the include files was just – would make you crazy as a novice programmer.
Remember the times when the header and the .lib file would get out of sync?
Yes.
And it would just not – you're like, why is this just crashing randomly?
Like, I don't even understand.
Like, so glad that problem's gone.
Oh, yeah, me too.
I don't want to – I mean, I like that stuff, but I don't want to go back there.
Yeah, for sure.
I think this is a really interesting idea of having a programming class for older adults.
You know, Philip Guo, who was at UCSD, I had him on Talk Python a while ago about programming in your golden years.
And said a lot of folks who were starting to retire and were learning programming actually were doing so to connect with their grandchildren.
Because their grandchildren were in Minecraft or they wanted to do, like, robots.
And they're like, well, I'd like to do that with them.
Apparently, that requires programming like JavaScript and Python.
So I'm learning it, right?
Yeah.
I had taught – this is a stupid story.
But years ago, I had taught a couple of adult ed classes to help people get into Windows and DOS.
This is how far back it goes.
And I would be up there.
You know, I'd be talking.
I would be talking about something.
And it invariably would happen.
I would say, okay, I want you to enter this command.
And I'd turn to the whiteboard and I'd write on the whiteboard.
And behind me, I'd hear this beep, beep, as people were restarting their computers because they had typed something wrong.
Oh, my God.
Because they were so afraid of getting it wrong.
I'm like, no, you can't hurt it.
Just go with it.
Yeah.
Unless you write format, you know, C colon slash Y or something, you're going to be fine.
You're okay.
And for me personally, you know, I committed to – as I came to it late and I committed to – I'm a self-taught programmer and was always – and I've changed languages a lot.
But I had to learn new languages.
So I committed to this sort of constant learning idea early.
And I've stayed with it my whole career.
And it's really paid off because I'm still able to learn new things.
I'm learning new things all the time.
It works like I'm learning React, for instance.
And a little bit of Erlang, for crying out loud.
Wow.
But it's because of – you know, there's this habit that I have – that I learn constantly that I'm able to stay competitive and be – and contribute to a team of mostly younger guys.
And, you know, mentor people based on my experience and able to learn from people.
So I hope that I'm not overbearing about that.
But I'm able to learn from other people because there's a lot of people who are – they have – you know, they don't even know what assembly language is.
But they have a valuable skill that I want to learn and know about.
Yeah, absolutely.
Do you feel like being a self-taught programmer was maybe harder in the beginning but has paid off now that you sort of force yourself to be able to figure it all out?
Yeah.
When I moved into software development full-time from – it was the same company I was doing the process controller work.
I moved to their software group.
And all those guys had MSs and doctorates in computer science.
And so I was playing catch-up all the time on Fortran and C programming.
And so I worked really hard.
One of the things I did, my boss was a very sharp guy.
And I would hit a problem.
You know, my goal all the time was like I would work at it, work at it, work at it before I go ask him a question.
The deal was – and I would try to make it so he didn't answer the question within one minute.
And I – in five years, I failed every time.
He would – but at least, you know, I did the work to ask intelligent questions to get there.
Yeah.
That's a really good point.
You know, I'm also more or less a self-taught programmer.
I took, you know, a couple of programming classes, C++, Fortran in college and stuff.
But for the most part, I studied math.
And I feel like there was always this little bit of imposter syndrome feeling like, well, I know – are we talking about like, you know, what happens deep down in the operating system?
Well, I didn't take the operating system class.
I don't know maybe as well as that person, right?
But I know that I taught myself all of this programming stuff.
And there's basically nothing that scares me.
Right.
Well, not nothing.
But not much that scares me because it's like, well, I've come this far by forcing myself to do it.
And you just – like you said, you get into this habit of I spent the last 10 years just continuously learning these things and making these steps.
And so I know to get to wherever you want to be, it's like just more steps.
Absolutely.
And those two things, they drive me pretty hard to constantly learn stuff that I don't know.
But I also know myself.
I'm an applications engineer.
I need to apply this stuff.
So, you know, red, black trees and that kind of, you know, AVL automatic balancing stuff is like – I mean, yeah, okay.
I'm just going to throw a dictionary at that.
It solves the problem.
Yeah.
So there's gaps in my knowledge that I don't have.
But on the other hand, I haven't run into too many cases in a long career where I've really needed that stuff.
So for me, I just – I need to know the stuff that is applicable to my job.
Yeah.
You know, a lot of people who take computer science have to do things like calculus and differential equations and stuff.
I don't know many programmers who still do differential equations.
I know there are some.
Yeah.
But not most of them, right?
Most of it, it's like, well, algebra, maybe a little geometry, maybe.
Well, sometimes I make this joke at work all the time.
Like they often lean on math teachers to be the computer teachers at schools.
And I'm like, yeah, I don't know if that's the right choice.
Sometimes I think it should be English majors because most of my life is tearing apart strings all day.
That's true.
It is a lot of text, isn't it?
It's a lot of text.
So, but while we have a little bit of time left, I'd like to talk about some of the articles that you've written.
Because, like I said, that's how I got to know you.
You said a big part of your day-to-day work is working on these REST APIs with Flask.
And you wrote a couple articles on realpython.com about connection with an X for REST APIs.
Do you want to tell us about the articles and maybe give us a little overview of what connection is and how it's helpful?
Yeah.
I started using it because as I was moving away from all of the form validation and stuff like that and getting more into writing REST APIs,
I was looking for ways to make that nicer, make that cleaner.
And actually, another guy introduced me to Connexion.
And that is a really nice tool.
It does two things that I really like.
Besides providing the automatic interactive documentation through the Swagger UI system, the configuration file that drives it is a nice way to think about the API.
So, you can sort of structure what the API is going to look at, look like, and what it's going to accept long before you get into code.
Because you can really go down a rabbit hole where you're like, okay, I'm focused on the code for this one endpoint, this URL endpoint.
And you can get to a point where you're like, oh, you know, if you looked at the API as just a list of URLs, they could look like a real mess and not make much sense.
And Swagger, or excuse me, Connexion helps you think about that in, I think, much clearer ways and structure the system.
Plus, whenever I've shown off, whenever I show off my stuff and I show the Swagger UI, everybody's really impressed.
It's helped the front-end guys to be able to look at the API and use it without actually having to write any code themselves and see how it behaves and what it's expecting, what it returns, that kind of thing.
And that's been very powerful.
Yeah, the documentation is really cool, actually.
Yeah, it's fantastic.
Because, you know, I definitely prefer the REST APIs over other options that have been popular, like SOAP services and stuff.
But one of the things about those bad SOAP services is you could ask it what functions were there and what you're supposed to do with them, right?
Whereas, like, an arbitrary HTTP API, you're just like, well, I better go figure out some URLs and then start trying to understand the JSON because that's pretty much it, right?
And so Swagger gives you a nice view over top of that, right?
Yeah, a really very nice, you know, a consistent standardized UI.
I mean, they're constantly reproving it to interact with the API once you get it up and running.
And I think if you spend time, it does take time to build a configuration file because you're defining all the parameters and what they look like and giving them documentation strings.
But the payoff is the interactive UI that Swagger provides that gives you the documentation for someone who's interested of the parameters, what they mean, what this thing expects, the types of data for that JSON is going to send you or that it expects from you.
And that's really powerful.
Yeah, nice.
So you have this Swagger YAML file that defines the API basically and generates the help.
How do you get the, like, where does connection and your code all fit together?
Well, I know how to write a general, like a normal Flask API.
I would put at route and I would return Flask.jsonify.
But what do I do in this connection on the world?
Well, if you start configuration, the handshake point, the connection point between the configuration and the code, there's one specific parameter.
I think it's, what is it?
Object ID or something like that.
It's, I can't remember off the top of my head, but that is, that specifies the path to the handler for that particular URL endpoint.
So you might have, you know, it's just a string.
It'd be like my code dot my app dot my method within that app.
And Connection is smart enough to actually introspect your code and find that.
You know, it'll navigate down that path, importing modules as it goes until it finds that handler and connect that to essentially doing the at route.
It will create that within Flask to handle that URL when that gets called.
So it's a very nice way.
I see.
You know, it auto connects that, this piece, the configuration piece with the code that's going to handle that configuration.
Oh, that's cool.
So it finds the Python files and implement it and sort of does the connection for you.
Yeah.
I guess it's the name.
And it tells you, you know, it tells you if you don't have one.
Like I've often misspelled things and it'll tell you, nope, that doesn't exist.
No connection there.
Yeah, too bad.
Okay, this looks really nice.
So you have a couple of nice articles on that and people can look through there.
Yeah, on RealPython I've published some articles, two articles actually, on this REST, this FlaskConnection REST API.
The first one was just an in-memory structure that I wanted to talk.
It was a long enough article that I just wanted to talk about REST and my interpretation of REST because it's not really a spec.
It's just a convention.
And then the second article, which sort of was spawned by people asking me in the comments, hey, how could I make this thing persist to a database?
So I took the basic API and code and then added a database connection through SQLAlchemy using SQLite as the database.
Yeah, that's the way to do it.
Very nice.
I love SQLAlchemy.
Oh, it's terrific.
I'm not a big SQL guy, so I like that Pythonic way of thinking about it.
Yes, I'm with you on that.
So that was on RealPython.
You also did some on dbader.org, which is kind of similar because Dan Bader runs both sites.
Yeah, he runs both sites, right.
So it's a little bit of a connection there.
You did one on just sort of Python, getting started with Python.
But one I thought was pretty interesting.
I actually used this as a little bit of research for my recent async course.
Yeah, I was very happy to see that.
Yeah, the one's called Understanding Async Programming in Python.
Yeah, I had maybe 10 or 15 articles I read to just see what people were doing.
And this is a nice one.
And this talks about very much of what you kind of hinted at the beginning with G event, with Twisted, and some of those things, right?
Yeah, this was an article that actually came out of a presentation I gave at one of the Shutterfly All Hands meetings.
And I had been in touch with Dan about the first article, just the Python introduction.
And then he and I talked, and he was interested in this asynchronous one.
And I put that together.
It's a passion of mine because I like this whole asynchronous way of thinking.
I've written threaded code in C and C++ and lived to regret it.
And this whole asynchronous way of thinking, although at first it's hard to get your head around, it's a really interesting way to work if you have a program that's a mix of CPU and I-O bound processes or tasks.
And I talked about that in my – one of the things I find interesting is people have – I've taught asynchronous programming to my coworkers and talked about it a lot.
And there's just like this real hurdle to get over for people to get their heads around it.
And it's interesting to me because in reality, humans are asynchronous.
Our behavior is asynchronous all the time.
But to think about it in this linear presentation of code, it's kind of hard to break out of that linear execution model.
But as you mentioned in my article, I had these sort of thought experiments about different kinds of way that if people were synchronous or how do you handle asynchronous behavior.
Exactly.
You know, if as a parent, you could only cook dinner and then tell your child to, you know, get ready for bed and then serve dinner.
And yeah, it doesn't make any sense, right?
The real world definitely is asynchronous.
Oh, yeah.
I like the polling parent one where you're running back and forth checking every two minutes on everything that's going on.
Is the dinner ready?
Is the kid getting ready?
Is the laundry – is the washing machine done?
No?
Yes?
No, no?
How interesting.
Yeah, so this is a cool one.
I would love to see a follow-up with some async I.O., maybe some A.I.O.H.D.P. or something like that.
That'd be fun.
Yeah, I'm going to talk to Dan about maybe how he feels about updating existing articles because I want to take that to the next level with async I.O.
I haven't had an opportunity to do much of that in my data way of work, but I'm definitely heading that way because I'm thinking, like I said, I was thinking about Quart and Sanic.
And I like it that it's native to Python now, the async stuff.
Yeah, and especially the async and await keywords.
Like you say people have this hard time thinking about async as code and whether it's something like G event or it's JavaScript with their callback mechanisms.
I think it's hard because you have to unravel or turn your code inside out.
I'm going to start this and then it's going to call this other thing to continue it and so on.
But with async and await, you write your code in the serial form and then you just drop in some awaits in the various places.
Yeah, it's great.
And it's beautiful.
I love it.
It's really, really nice.
I've talked to some of my friends who are like the front-end guys I work with who are friends of mine.
You know, they're used to this whole asynchronous method in JavaScript.
But in some ways, they're still sort of in the mind frame of like they're thinking linearly because you see this deeply indented callback hell for their handling asynchronous events.
And it's one of the things I like about React is it helps you break out of that.
Yeah, that is really nice.
Yeah, that deeply indented callback stuff.
It's just hard to deal with if something goes wrong.
It's like, well, where do I unravel this to?
Yeah.
I'm halfway down this hole and I don't know where to go from.
What am I doing now?
Yeah, this is the problem.
So you've got to take your step back each level out.
But, yeah, I'll put that in the show notes.
People will definitely enjoy that.
So we're down to just a few minutes, a time left.
So I suppose I'll ask you the two questions here at the end.
All right.
So if you're going to write some Python code, what editor do you use?
Well, currently I'm an IntelliJ editor user and with the Python module plug-in.
I used to use PYCharm, but now I'm covering so many languages, it's useful to work in IntelliJ so I can jump back and forth.
Sometimes I have to do code reviews in Java and look at JavaScript and some other things.
Right.
So IntelliJ is really great.
I don't understand people who don't want to use an IDE with a debugger.
I just don't get it.
Yeah.
You and me both.
I'm such a fan of it as well.
I think it's really great.
So if you're an IntelliJ-based person doing PYCharm or something, Brian Okken shared this with me today, this thing called Power Mode 2, which is one of the plugins.
The faster you type, it will start to shoot flames and sparks from your code as you type.
So if you need to do a presentation show off, you plug that in and take it to the next level.
It's, of course, completely silly but fun.
I might try that because one of the best things my mom ever made me do was I took a typing course in sixth grade and it's paid off because I type pretty fast.
A lot of programmers don't, but I type pretty quick, so I'll definitely try that out.
Yeah, all fun with that.
Definitely for presentations.
All right.
So notable PyPI packages?
Well, I really like, as I mentioned, Connexion, which is this REST API.
And then the other side of that is Marshmallow, this serialization, deserialization library that lets you, because SQLAlchemy deals with Python objects, you can't just hand those off to something that wants to turn them into JSON like Connexion.
Right.
If you try to return it out of your method, it'll freak out and go, I cannot serialize this thing to JSON.
Yeah, there's no matching thing for a Python date-time object in JSON.
So this Marshmallow thing allows you to, just as you would in SQLAlchemy, build a model that matches your table structure.
Marshmallow does the same thing, but it builds a model that tells us, oh, this is the data I'm expecting, this is the structure, and this is how I'm going to convert it to something that's JSON-able.
That is cool.
And back and forth.
It does both.
Yeah, that's really cool.
I should definitely learn about Marshmallow.
Oh, it's sweet.
It has the same thing as SQLAlchemy, where you have relationships, the one-to-many, many-to-many relationships in the data table.
You can also set those up in Marshmallow so that if you have a SQLAlchemy object that has, you know, you've got one record, but it's got many relationships to others.
Marshmallow will take care of that as well.
It'll just go down the tree and serialize them all.
That just does the whole object graph.
Wow.
Yeah.
Very nice.
You take that return value and you hand it off to – you return it from your call, and Connection will then turn it into a JSON string.
Okay.
I'm loving this one.
Good choice.
All right.
Well, this – yeah.
This has been really interesting.
I really appreciate you being on the show, Doug.
Oh, I'm very glad to be here.
It's great.
I've really enjoyed it myself.
Yeah, absolutely.
You got a bunch of recommendations, and I think a really interesting way to look at that whole async programming thing.
So I'm looking forward to your follow-up articles that you write about this.
Yeah, definitely be fun.
I love that little bit of extra complexity and then being able to do many things at once is not only important to me but fun as well.
Yeah, it's definitely fun.
Yeah, we don't need to do crossword puzzles or Sudoku.
We can just work on our threaded code, right?
Yeah, right.
In my classes, I teach the kids using the turtle module because I get them right into drawing on the screen.
I do the same thing with adults.
It would be great, I think.
Yeah, absolutely.
It would.
All right.
Well, thanks for being here.
Good to talk to you.
Thank you, Michael.
Bye.
Bye-bye.
This has been another episode of Talk Python to Me.
Our guest on this episode has been Doug Farrell, and it's been brought to you by LogRocket and Rollbar.
LogRocket will help you avoid the back and forth, helping your users with problems on your site.
Visit talkpython.fm/LogRocket to get started with their free tier and get a pixel-perfect replay of what your users saw and the state of your app.
Rollbar takes the pain out of errors.
They give you the context and insight you need to quickly locate and fix errors that might have gone unnoticed until your users complained, of course.
Track a ridiculous number of errors for free as Talk Python to Me listeners at talkpython.fm/rollbar.
Want to level up your Python?
If you're just getting started, try my Python jumpstart by building 10 apps or our brand new 100 days of code in Python.
And if you're interested in more than one course, be sure to check out the Everything Bundle.
It's like a subscription that never expires.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, Google Play feed at /play, and direct RSS feed at /rss on talkpython.fm.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Now get out there and write some Python code.
I'll see you next time.