#54: Enterprise Software with Python Transcript
00:00 How often have people asked what language or technology you work in?
00:02 When you answer Python, they get a little confused and ask, what can you actually build in Python? What type of apps?
00:07 Implication being that Python is just a notch above Bash scripts, and that real things aren't built with Python, but rather with Java, C#, Objective-C, and so on.
00:15 Mahmoud Hashemi and I might be able to help put some real evidence behind your responses.
00:19 On episode 54, Talk Python to Me, I talk with Mahmoud about his new online course
00:24 he wrote for O'Reilly called Enterprise Software in Python.
00:27 You'll hear many real-world examples from his experience inside PayPal, and many more throughout the industry.
00:33 This is episode 54 of Talk Python to Me, recorded April 4th, 2016.
00:55 It's about design patterns. Anyone can get the job done. It's the execution that matters.
01:00 Welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystem,
01:06 and the personalities. This is your host, Michael Kennedy. Follow me on Twitter where I'm at,
01:11 mkennedy. Keep up with the show and listen to past episodes at talkpython.fm,
01:16 and follow the show on Twitter via at Talk Python.
01:19 This episode is brought to you by SnapCI and Hired. Thank them for supporting the show on Twitter via
01:25 at Snap underscore CI and at Hired underscore HQ.
01:30 The news this week is all about online courses. You'll want to be sure that you're a friend of the show
01:34 because we have two cool giveaways. Just drop by talkpython.fm and click on Friends of the Show
01:40 in the navbar to become eligible.
01:41 Mahmoud is giving away a copy of his online course which normally sells for $149.
01:46 I'll pick one lucky friend of the show later this week to receive a copy.
01:49 You can find the links to his course in the show notes for this episode.
01:53 Speaking of online courses, my Python jumpstart by building 10 apps is out and people are really loving it.
01:58 I thought I'd join Mahmoud and give away a copy or two this week as well.
02:01 You can see the details of my course at talkpython.fm/course.
02:05 Now let's talk about Enterprise Python.
02:07 Mahmoud, welcome to the show. Welcome back to the show.
02:11 It's great to be back.
02:13 Yeah, I really enjoyed the episodes we've done before and this is actually your third appearance.
02:18 You were on as the main guest in episode four where we talked about the myth of Enterprise Python
02:26 which was based on an article.
02:28 That was sort of when this was kicking all off. That's right.
02:31 Yeah, the 10 myths of Enterprise Python blog posts.
02:35 That's right. This is like the germ or the seed that generated this course that we're going to talk about, right?
02:40 It's been a wild and crazy ride. That's right.
02:43 I bet it has.
02:43 And then you gave some cool inside information on how you guys think about hiring and junior engineers back in episode 41.
02:51 And this is going to be episode 54.
02:53 So I'm instituting a new rule.
02:54 You're not allowed to come back on the show unless there's a four in the number.
02:58 That seems like a fine rule to me.
03:00 I can like, so at least 10 more episodes.
03:04 I can probably drum up something in that time.
03:06 Perfect, perfect.
03:08 All right, so people heard your story back on show four.
03:12 So we won't go sort of through how you got back into programming, even though that's the question I often ask people.
03:17 Instead, I'd like to hear how you got into enterprise programming.
03:21 And what was the step where you went from just, I'm doing programming, I'm doing Python,
03:25 to I'm now working at a company that has more employees than some small towns?
03:31 You know, that's actually something that recently I was forced to sort of ponder a lot on, you know,
03:38 like just had some really deep shower thoughts, as it were.
03:42 And so, yeah, I sort of fell into it, as I'm sure so many do.
03:49 Like you get an internship at a company and they assign you kind of, you know,
03:55 maybe the worst stuff that you can do within three months.
03:59 Like they want you to do something, but they don't want it to be like, you know, super, super glamorous.
04:04 They give that to their star engineers who already have full-time positions.
04:07 So, you know, as an intern, I was doing some really niche log analysis stuff.
04:13 And so, you know, I actually define enterprise software in my course.
04:20 And, you know, one of the main things that makes software enterprise isn't that it's high performance or something like that.
04:26 It's actually that it's very niche, you know, has a very constrained user base that's interested in certain things.
04:31 And so, you know, I just started out doing really niche applications because I love software introspectability so much.
04:38 You know, if you tell me, hey, go analyze these logs, go find out what's slowing this application down,
04:43 that's something that even now I can't really resist.
04:46 Yeah, that's really cool.
04:47 Those types of enterprise apps are pretty different.
04:51 I spent a long time working at a company, I guess probably six, seven years, one of my first jobs,
04:57 working at a company where I think there was maybe a maximum of 25 employees.
05:01 And we were writing internal software to do all sorts of cool stuff at this place where we did eye tracking.
05:08 And we'd write custom apps to sort of, you know, collect various specific studies that the scientists want to do with eye tracking and things like that.
05:16 And I think we wrote some really powerful and beautiful software, but it was for like 10 people.
05:21 You know, and so that's, it's not sort of the scalable sort of advertisement driven concept of like Pinterest and hockey curve growth and all that.
05:31 But it's the people who use that software, like their jobs, like the whole company depends on that software.
05:38 So it's pretty interesting to live in that space.
05:40 They don't really have a, they don't really have a choice, you know.
05:43 So like, I mean, that's kind of what makes a lot of things enterprise.
05:48 That's sort of the stuff that makes a lot of enterprise stuff kind of bad.
05:52 But we can probably go into that later.
05:54 Yeah.
05:55 The whole battleship gray, I don't need to know design, just because I'm a programmer, that kind of thing.
06:00 Yeah, that sort of thing.
06:01 You know, there's no, there's no alternative.
06:04 So you don't spend any time actually making stuff good.
06:07 That's one approach some people take.
06:09 You know, I'm not for that approach personally.
06:11 Yeah.
06:11 I think that used to fly, especially in the 90s when you write enterprise software, it could be seriously unusable.
06:18 But ever since the iPhone came out and people have gotten used to really beautiful, nice computing experiences,
06:29 they're like, why do I have to go to work and have this horrible computing experience, right?
06:32 That has, that has actually left its mark.
06:35 I agree.
06:37 I mean, there are pros to that and there are cons to that.
06:39 Now I see a lot of stuff getting polished a whole lot, but the underlying quality is still lacking.
06:46 So, you know, there are lots of ways to compromise quality.
06:49 And enterprise will always find a new way.
06:52 That's right.
06:53 Yeah.
06:54 So, you know, I have this, like, you know, hopefully we'll get to cover a little bit about, you know, how to sort of mitigate that with the power of Python.
07:02 Yeah, absolutely.
07:03 So, one concept I'd kind of like to introduce near the beginning before we get into the course is just that when you think of enterprise developers,
07:11 and you can maybe put Python to the side for just a moment and just say, like, across all technologies,
07:17 a lot of times there absolutely are super smart people writing amazing software, but maybe they're just going to work and they're just doing their job and they're not going and speaking at the conferences and they're not on Twitter constantly blogging and tweeting about, you know, whatever it is that they're doing.
07:35 They just go do their job and they go away.
07:38 And so I think that we probably underestimate the impact and the number of people that kind of live in this space because they're not necessarily as vocal.
07:48 Yeah.
07:49 And, I mean, I don't know.
07:52 I'm not really for the cult of personality, like, you know, rock star coder type stuff.
07:56 I think that, like, every developer has a rock star within them.
08:02 I know because, I mean, you know, they talk about 10x programmers, like, and some people say, hey, they don't exist.
08:09 And other people say, hey, they do exist.
08:10 And it's, like, a little bit like this big foot of developers.
08:13 And I don't know why it's so controversial, why it has to be like that.
08:17 Everyone has within them, like, a 10x developer.
08:20 If you're struck with the right idea, you know, you will work tirelessly and learn so much within the span of just a couple of weeks.
08:27 And, you know, the rest of the time, like you said, you have a normal work-life balance and you have a day job and that's totally fine.
08:34 But 10x developers do exist and they're within all of us.
08:37 That's what I choose to believe.
08:38 I totally agree with you.
08:39 I think if you were, like you said, struck with inspiration, you can do almost superhuman type of work.
08:48 So focused, so effective.
08:49 And on one hand, it'd be cool if we could always be that focused and effective.
08:53 But, you know, that's how inspiration works, right?
08:56 If you both have the time and the energy and the idea all coming together, you could be that 10x developer for that project or whatever.
09:04 Right.
09:05 I mean, you won't believe this, you know, looking at my GitHub or my Twitter or whatever, right?
09:09 But I do think that there is much more to life than just code.
09:14 So, yeah.
09:16 Yeah, same here.
09:17 But I try to keep it basically to technology on Twitter as well.
09:21 So it's hard to tell.
09:22 So let's talk about your course.
09:26 So it's called Enterprise Software with Python, right?
09:30 And you're distributing this through O'Reilly.
09:32 Is that right?
09:33 That's right.
09:34 So I think that basically because of the 10 minutes of Enterprise Python blog post, they actually contacted me a little bit before we did episode number four.
09:45 And they're like, hey, would you like to expand upon this and do a project on Enterprise Software?
09:53 And, you know, I really hemmed and hawed.
09:55 I'm sort of more of an engineer.
09:57 I'm a builder, you know.
09:58 And I have a lot of reservations around education.
10:02 Like I'm a firm believer in education, but it's very time consuming.
10:06 My dad, you know, he's a professor of virology, right?
10:09 I have really high standards.
10:11 And I know that if I get involved with something like this, I'm going to go all out.
10:14 And so I hemmed and I hawed.
10:16 But then I finally, like, you know, decided to make it happen in August of last year.
10:22 I started working on it.
10:23 And I chose the video format because of how successful I saw that it was inside the company here.
10:31 And so it's a video course.
10:32 In preparation for it, I actually taught over a dozen internal, like, training courses at PayPal.
10:39 And just covering all sorts of different topics, sort of honing in on the right, the best explanations for various topics in Enterprise Software.
10:48 And I ended up with this course that is about, I would say, like, it's half and half.
10:53 It's like half, you know, Enterprise Software, you know, and half, like, how to do it in Python.
10:59 Python just works really well as an example language.
11:02 And, you know, the ecosystem is very full-bodied.
11:05 So, you know, it really fit well.
11:08 But it has whole segments of it that are not at all Python significant.
11:12 It's almost like a case study of Enterprise Software.
11:16 But from, you know, choosing Python as a specific language.
11:20 That's really cool.
11:21 So one of the things that you ask in this class, and I guess your class is, like, in your blog post,
11:29 it was sort of a, people think these things are inappropriate in Python.
11:33 Let me show you all the ways, the 10 ways, where they've got some false conception, right?
11:40 Python is just a scripting language or something silly like this that gets in their way.
11:46 But this is more like, well, we'll talk about those, but let's actually study the whole thing in a more concrete way, right?
11:53 So one of the questions you sort of started out with, which I thought was interesting, is you said,
11:57 is Python suitable for the enterprise?
11:59 And when should you write Python code?
12:02 Right.
12:03 And that is, I mean, it's a valid question, right?
12:06 You're at a company, and, you know, as an engineer, Python, it's a language of choice, you know,
12:13 which means that, like, engineers are going to choose it.
12:15 Your boss isn't going to choose it for you.
12:17 Usually, like, C-level type people do not choose Python for their company.
12:23 And that's kind of something I'm trying to reverse here.
12:27 Python has been shown to be, like, tremendously successful for many very large-scale, you know, companies.
12:34 YouTube, Yelp, Dropbox, and, you know, I believe CERN uses it.
12:41 It's really all over the place.
12:43 So, I mean, it shouldn't be as controversial as it is.
12:47 And so I go down the list of when you, you know, when we've actually seen it used.
12:53 I'm not saying that, hey, here's where I think you should use it.
12:55 I'm like, no, here is where it has been used successfully.
12:58 And then I give a couple of areas where I don't think it's as common, you know, for enterprise development,
13:04 such as you don't really see it that often in web front end.
13:08 You know, that's pretty much all JavaScript and JavaScript derived.
13:10 You know, there's PyJamas.
13:12 That's not really quite there.
13:14 There's Brython.
13:15 And, like, for browser Python, that's not really quite there yet.
13:18 But, yeah, so, but, yeah, Python is absolutely suitable for the enterprise.
13:24 And I don't think that it's just generic organizations that don't understand this.
13:30 Even when we go to conferences and speak with other Python developers, they're kind of appalled to hear that we are, like, you know,
13:39 shooting for 100 microsecond response times.
13:42 You know, 100, 200 microsecond response times.
13:46 You know, as served from CPython.
13:48 You know, this isn't even PyPy.
13:50 And so, yeah, I mean, we can hit those enterprise performance type numbers very consistently with some good engineering.
14:00 And I think that this course lays the groundwork for making that sort of software possible in Python.
14:07 I totally agree.
14:08 I think, you know, you brought up CERN, right?
14:10 I talked to Kyle Kranmer.
14:12 Right.
14:12 I listened to that episode.
14:13 That was so excellent.
14:15 And I'm just thinking if Python is appropriate to help collect the data from the Large Hadron Collider
14:22 and win the Nobel Prize for fighting the Higgs boson, it's probably not too bad to, like, monitor your log files or something.
14:31 The people think they have these amazing, amazing requirements, but a lot of times there's just a bunch of little tools you got to build.
14:39 And, you know, you don't need, I don't know, some super specialized thing.
14:45 Yeah, since the last episode, the main project I've been working on is a sort of a key distribution system
14:53 for securing sensitive data inside of PayPal.
14:57 You know, trust and security is PayPal's, you know, number one, like, selling point.
15:01 That's what the brand is associated with.
15:03 It is the safest way to pay online, that sort of thing.
15:06 And to keep it that way, you know, we have to be innovating in that space.
15:13 And so that's basically as low-level a system as you will find inside of PayPal.
15:18 Like, everything depends on that.
15:21 And, you know, we expect billions of requests per day, three, two, three billion requests per day.
15:28 We're already, like, seeing the adoption is happening, but we've seen days of 500 million requests per day.
15:34 And, you know, the, you know, average median time that we're shooting for
15:39 is less than two milliseconds for the response.
15:43 So it's not just for logs or whatever.
15:46 It is for these critical systems.
15:48 And it's the main reason that we chose Python was because of its consistency,
15:54 our ability to measure how the software is performing, and because we can maintain it with a very small team.
16:02 The security team is not a huge team at PayPal.
16:04 Application security, I should say.
16:06 InfoSec, that's a much larger team.
16:10 But application security is a much smaller team.
16:12 Wow.
16:13 That's really amazing that that app written in Python is at the foundation of all those things.
16:18 Like, billions of requests per day is, that's a number that not so many people deal with.
16:23 That's really, really cool.
16:25 And that's written in Python.
16:26 Is it like an HTTP service, or what's the story there?
16:28 Oh, yeah.
16:29 So that's one of the great things.
16:31 So because of Python's easy, like, I mean, it's not just Python.
16:37 You know, we have a framework that we've built on top of that.
16:40 Most of that is open sourced on PayPal's GitHub.
16:43 It's called support.
16:44 It's sort of a reference framework.
16:47 So using a framework, you know, based on that, we have several protocols that we support.
16:54 HTTP is, like, if you're talking HTTP 1.1, 1.0, is kind of slow to parse.
17:01 So we go for lower overhead protocols.
17:05 We're fans of NetString.
17:06 That's a classic.
17:08 You know, it even has sort of roots in the crypto community, because DJB, Daniel J. Bernstein, you know, defined it so many years ago.
17:15 It has a Wikipedia page.
17:17 It's extremely simple.
17:18 NetString.
17:19 And then we have something even lower level than that, which is similar to NetString, but it's called TLV, tag length value.
17:26 And so you do have to go for, like, lower level protocols there.
17:32 But because it's Python, you know, we can maintain an HTTP interface for development purposes so that people can mock things out.
17:40 without having to worry about the details of TLV.
17:43 I guess if you're going to get two millisecond response time, you're more or less down raw sockets almost, right?
17:49 Yeah, yeah.
17:51 No, we don't shy away from raw sockets.
17:55 We have a little buffered socket that we've written on top of that.
17:57 But when you want to hit sub-millisecond, deep sub-millisecond times, then you do have to basically be counting the, you know,
18:08 bytecode instructions that you're going to be, like, executing.
18:12 So every line does count.
18:14 You know, every line does have its cost.
18:16 But Python makes it easy to gauge that cost.
18:18 And because you can get your features done in time, you can actually start down that performance path much sooner than other stacks.
18:24 And that's the big selling point for us.
18:27 Yeah, you can do the testing and experiment and find the right answer probably before the first prototype would be even finished in C++, right?
18:36 Oh, yeah.
18:37 I mean, in fact, we have had literal, I mean, I wouldn't call them competitions, right?
18:42 We're inside a company.
18:42 We're all one team, right?
18:43 But, like, we've had cases where that exact stuff was happening.
18:48 Like, you know, we were done months ahead of time and already, like, iterating on our performance.
18:53 By the time C++ came around with their stuff, they couldn't even get within an order of magnitude of our scalability.
18:59 Wow, that's awesome.
19:01 So let's take it back to the beginning for a moment.
19:04 You talked about your first enterprise app, and that was kind of that log file monitoring thing, I think, right?
19:10 Right, right.
19:10 And you said there was a couple of lessons you learned from it, and I thought they were really good.
19:15 Maybe you could talk about those a bit.
19:17 Well, so that first version of that log analysis web front end was actually written in PHP.
19:24 You know, that was my last PHP project.
19:27 And, yeah, I got it done in time, but it had some problems.
19:32 Number one is that, like, it was written in PHP, so, like, maintenance wasn't really, you know, on my side.
19:39 And then I actually had to keep fixing bugs and running the service.
19:44 And I basically was left with no time to actually do other projects.
19:49 And so I didn't want to be the guy that just maintains log analysis tool, you know,
19:53 so I shut it down after about a year.
19:55 But, you know, from then on in, I gave a lot more thought to, what is the maintenance curve going to look like?
20:01 Is this something that I'm going to be able to sort of set aside and move on to new things
20:08 so I'll always have something new?
20:10 But that old stuff can keep, like, running.
20:12 I don't want to be, you know, one of those flash-in-the-pan project sort of guys.
20:17 You know, I like to build up a lot of different services that I run that help people in real situations.
20:25 The exact lessons were basically, like, choose your dependencies and avoid doing it all at once.
20:31 So you can, if you can rely on your organization, I mean, you're working at a large organization.
20:36 Find ways to rely on the organization.
20:39 Offload some of that maintenance.
20:40 And finally, like, that longevity, it doesn't come easy.
20:45 It's the result of a lot of critical decisions.
20:48 And it's more than just regular software, I think.
20:50 Like, you could write...
20:53 Yeah, there's a lot of soft skills involved.
20:54 Oh, absolutely.
20:55 So you could write the software if you were kind of doing this little app on your own
21:00 and it could do its own little thing.
21:03 But if it's going to continue to live on and be used, you have to know the sort of invisible connective tissue of the software of your company, right?
21:13 Like, this is the way things are done.
21:14 And this is the way it will actually be useful to people.
21:17 Yes, I should design it this way if it was from scratch.
21:19 But people don't do that.
21:20 They do this.
21:21 And so I'm going to do this other weird thing because then it'll get used for a long time and so on, right?
21:25 Right, right.
21:27 Like, I mean, I'm all for going against the grain a little bit.
21:31 But, and actually innovating, changing things and so forth.
21:34 But if you try to do it all by yourself or all at once, you're setting yourself up for,
21:40 I mean, you know, let's be frank, you're going to burn yourself out, right?
21:44 And you're not going to be able to do those future projects.
21:47 And that's the whole point, you know, is to continue building things because there's a lot of things to be done.
21:52 Continuous delivery isn't just a buzzword.
22:09 It's a shift in productivity that will help your whole team become more efficient.
22:13 With SnapCI's continuous delivery tool, you can test, debug, and deploy your code quickly and reliably.
22:19 Get your product in the hands of your users faster and deploy from just about anywhere at any time.
22:25 And did you know that ThoughtWorks literally wrote the book on continuous integration and continuous delivery?
22:30 Connect Snap to your GitHub repo and they'll build and run your first pipeline automagically.
22:36 Thanks SnapCI for sponsoring this episode by trying them for free at snap.ci slash talkpython.
22:42 Yeah, and you had some really interesting statistics about the types of apps and the numbers of services and so on inside PayPal in your course.
23:02 And it just makes me kind of laugh thinking like, oh my gosh, this is crazy to think, you know,
23:06 So when I'm talking about like, how do people within the company work and how does software normally connect?
23:12 You had like a graph that was basically so dense it was illegible, but showing the interconnections between services.
23:19 Can you talk about that for a sec?
23:20 Sure.
23:21 So PayPal is a service oriented architecture company going back over a decade.
23:27 It started as a monolith.
23:29 And there's nothing wrong with that when you're starting something out.
23:33 Like a monolith is a really good way to get things done.
23:36 But as you are expanding both in terms of customer base, computers running your software, people working on that software, most importantly,
23:48 then you, it starts to become a burden.
23:51 And eventually that hit a technical wall for PayPal where that monolith called WebScr.
23:57 I think I can talk about that.
23:59 W-E-B-S-C-R.
24:01 If you look in some PayPal URLs, you'll still see it.
24:03 It's a C++ executable.
24:06 And it reached over a gigabyte and a half.
24:11 One program.
24:13 And I don't want to talk about that in the course.
24:15 It's C++, not Python.
24:16 So, but it hit like one and a half gigabytes.
24:19 You can't even put debug symbols in there in order to debug what live issues are going to be.
24:24 So, it became completely, it got completely out of hand.
24:30 And they, at that point, like they knew that they had to switch to something more distributed so that they could work on it with more engineers.
24:38 And so, you have these service-oriented architectures sort of for human reasons.
24:44 You know, you want to have a user team.
24:46 You want to have, you know, payments team.
24:48 You want to have a, you know, front-end team or whatever.
24:52 There's, you know, and so you have to start segmenting things by layers and domains.
24:59 And so, at PayPal, we've done this in the extreme.
25:03 We have one huge product.
25:05 You know, Google, they have Gmail.
25:07 They have Maps.
25:08 And even they have sort of monolithic development structure in a way.
25:12 But these are all separate products is what I'd like to emphasize.
25:16 At PayPal, there's not that many different products.
25:18 It's pretty much PayPal.com.
25:20 And so, just for PayPal.com, you have over 3,000 service, over 3,000 service endpoints, logical endpoints.
25:29 You know, there are tens of thousands of machines running the code.
25:32 And just within the mid-tier, over 3,000 endpoints.
25:36 And that's, like, you know, representing hundreds of code bases.
25:40 And so, this is all communicating with a variety of protocols.
25:43 I've listed off a few already.
25:45 But it's pretty much those ones.
25:46 HTTP, you know, RESTful, et cetera.
25:50 Then you have NetString for some of the lower level, older stuff.
25:53 That's pretty much, we have one, like, proprietary one that's really old and will never go away.
25:58 You know, I talk about that in the course, too.
26:01 There's a sort of design permanence.
26:05 It's indelible.
26:06 Yeah, absolutely.
26:08 But, so, yeah, we have these thousands and thousands endpoints.
26:11 And they, you know, rely on each other in really complicated ways.
26:14 And each individual team has to consider how those runtime dependencies and logical dependencies are going to affect the scalability and performance of their application.
26:26 That's a lot to keep in your mind and sort of juggle, right?
26:30 Like, if I call this service and that service and this one calls that one, like, what is the latency and scalability considerations I have to actually worry about?
26:38 Well, I mean, some people don't even get to the point where they're able to worry about that stuff.
26:42 That's kind of the problem.
26:43 You know, due to other decisions, you know, they end up, like, you know, constantly having to fix bugs or work around issues.
26:52 And they don't get to have those important spare cycles to build quality.
26:56 So, you did talk about scalability a bit in your course.
26:59 And I thought that was interesting that you said scalability is not itself a feature of software.
27:06 But it's actually, like, a composition of three or four bits or more sort of atomic units of, like, throughput and so on.
27:13 Yeah.
27:15 So, I thought about it a bunch.
27:16 And I couldn't really find an ontology of what makes up software.
27:23 What are the aspects of software that really applied to what I saw happening at PayPal?
27:28 And so, I sort of had to come up with one of my own.
27:32 So, yeah, we came up with this ontology, talking to coworkers and so forth, of six different software aspects.
27:39 And so, the first is, like, usability.
27:41 You know, that has to be first.
27:44 If your software is not usable, if it doesn't have features to use, then it's really nothing at all.
27:50 So, you have to have usability.
27:52 And then, you know, we have availability.
27:55 That's kind of the reliability.
27:56 We have the introspectability.
27:59 So, that's how good your logging is and other instrumentation.
28:03 Then, we have performance.
28:05 Everyone, when they think of enterprise software, they think about, like, they think I'm talking about performance.
28:10 And that's just one aspect here.
28:12 And within performance, I'm referring to, like, latency, throughput, and utilization.
28:17 There are many aspects within each of these aspects.
28:19 But fifth has to be security.
28:21 Security really makes everything so much harder.
28:25 I can tell you for a fact that one of the reasons why other companies can move, like, faster than how PayPal, like, moves is because they don't have to worry about locking down access to all these secure resources.
28:37 Our deploy process is primarily convoluted because of that security consideration.
28:43 Normal developers do not have access to where their code runs.
28:47 So, that, you know, brings me to my sixth aspect, which is the agility, right?
28:51 Your ability to deploy quickly, change things quickly.
28:55 Everyone talks about, like, oh, everyone on their first day should be able to change one line of code and deploy within one hour or something like that.
29:04 And I'm like, okay, man, like, you know, one line of HTML, you know, changing the styling of a button or something is very different than, like, one line that impacts everyone's, you know, security.
29:17 Yeah, that token, that security service you were talking about that does almost a billion requests per day.
29:23 Two, three billion, yeah.
29:25 Yeah, that one you don't want, like, the new guy to, or girl, to, like, modify necessarily on the first day.
29:33 Right, right.
29:33 And so, I mean, and these six aspects, they're all, like, interconnected, like, better introspectability, we find correlates and causes better performance.
29:45 And because you can actually see the effect that you can actually see the effect that your changes are having.
29:49 And so, these six things sort of create, like, software hexagon or whatever, kind of like the project management triangle, which, aka the pick any two, right?
30:00 Fast, good, and cheap.
30:02 Like, you can make something that is very usable and you can deploy it very quickly, but maybe its availability is not so great because you don't have a good throttling scheme.
30:13 It's like you have to choose a balance of these things.
30:15 And as a result of choosing a good balance of these six aspects, you can build scalable software.
30:20 Yeah.
30:21 Very interesting.
30:22 I think the good, fast, and cheap triad is a very effective mechanism for speaking to people who are not developers and developers as well, but it comes through really well.
30:33 Right.
30:33 But, I mean, good is just so nuanced.
30:36 Fast, that's just a certain amount of time.
30:38 Cheap, that's just a certain amount of money.
30:40 Those are easy to quantify.
30:41 But what does good mean?
30:42 You have to expand that, and that's where these six aspects come from.
30:45 So, you sort of set out to define a spectrum of enterprise software, if you will.
30:51 You said, okay, a thing is not either enterprise software or not enterprise software, but it might live somewhere along a continuum.
30:59 And you have, like, nine points that would push it one way or the other.
31:03 Do you want to talk about those?
31:05 I thought they were really on point.
31:06 Yeah.
31:07 Enterprise, it's either enterprise or it's not enterprise.
31:09 That black and white view is not actually really helpful in helping people find, you know, just corollaries and analogies to the work that they're doing.
31:19 You know, we want to make something that people can actually relate it to what they are working on.
31:24 I created a list of these.
31:27 And, you know, I'll probably release this as a blog post or something like that as well.
31:31 But I think that this one is one of the free clips that you can view.
31:37 But just to run down the list real quickly, you can have any number of these and it sort of defines how much, how enterprise your software is.
31:46 So number one is that it's used by business.
31:48 Number two is that it's sort of tailor made, like it's custom.
31:53 And number three is that you have a specific user base, like we talked about before, that it's a, you know, if there's just two or three people using your software, it's going to have a really different development feed and flow.
32:06 Then you have strict runtime requirements.
32:09 That's where the performance comes in.
32:10 You know, that I need this with an SLA 95 percentile of like, you know, two milliseconds, 200 milliseconds, whatever.
32:18 Then you have that your software is part of a larger system.
32:21 And so at PayPal, you know, if you have to, we have to integrate not just internally with thousands of service endpoints, but also externally with financial processors and, you know, all of these highly regulated, basically legacy systems.
32:37 And so that brings me to the next one, which is legacy integration.
32:42 If you have to integrate with legacy, you know, I just got to give that a special shout out.
32:47 That's enterprise.
32:48 That moves you far down the spectrum to the right, doesn't it?
32:53 Exactly.
32:54 And, you know, so and that feeds into kind of the design permanence.
33:00 And so people worry a lot about making the right decisions.
33:04 And we talk a lot about technical debt.
33:07 But one person's technical debt is another person's like, you know, just software longevity, you know, software that has to run on airplanes.
33:16 And satellites and so forth.
33:18 I mean, it's not technical debt.
33:20 It's just a different kind of investment.
33:24 And so design permanence is definitely a hallmark of enterprise software.
33:28 Because of all of the things involved when you're doing all these integrations, if there are a lot of negotiated timelines and promises being made about, like, you know, delivery dates and so forth.
33:39 You know, some people will just say, hey, that's just waterfall and that's bad.
33:43 But that's how a lot of systems have to work when you have these deep dependency trees of software.
33:49 Like, you know, dependency, I guess.
33:51 Reli.
33:51 People relying on you having something done at a certain time.
33:56 And finally, like, you know, when you are handing software off to another team to be run, you know, if you have a separate operations group, a separate site reliability engineering group, SRE, if you have a separate security group that's going to audit your stuff.
34:12 If there's all these specialized groups, that's another aspect of enterprise software.
34:17 And so if you look at these nine different, like, hallmarks, you can see that in consumer ecosystems, which consumer is the opposite of enterprise, these aren't really the case.
34:28 You know, for instance, specific user base.
34:31 Usually in consumer software, you're just putting an app on an app store and people will either like it or they won't like it.
34:37 There's a market.
34:38 There's choice.
34:39 You know, a user base that's going to be using my first Python project at PayPal, you know, changes the prices of PayPal.
34:46 There's a good reason why we don't want thousands of people using that.
34:50 It basically has 10 users or so who can actually modify the values in there.
34:56 And they're still using my code, you know.
34:59 And it's being maintained by a different group.
35:02 So I consider that enterprise software, even though it only gets hundreds of requests a day at most.
35:08 It has a specific user base.
35:09 It's part of a larger system.
35:11 Legacy integration.
35:12 Tailor-made.
35:13 Right.
35:13 It's tailor-made.
35:14 I mean, I use Django, of course, but at that time.
35:18 And but it had to integrate with a lot of these systems and a lot of these human processes in a way that, you know, is just different than consumer software.
35:28 So just defining something as enterprise based on its performance requirements excludes this other part of the spectrum.
35:36 And that's not really that great.
35:37 I'm not saying that that application is the most enterprise.
35:40 Probably the most enterprise software that can occur to me is the ADA code that runs, you know, on the 737, 747, whatever, you know, airplanes.
35:50 I mean, that's far more enterprise than I will ever write.
35:53 It's even more enterprise than like the security stuff, I think.
35:56 You know, there are lives at stake.
35:58 So medical software.
36:00 I consider that enterprise.
36:02 Yeah, I think I think that's an interesting sort of nine dimensional spectrum that your code can live in.
36:09 And I think it hits a lot of interesting points.
36:12 You know, one thing that sort of struck me when you're talking about that with the tailor-made specific user base and thinking about how it fits into a larger system.
36:23 You know, I think I've heard less of it lately.
36:26 I don't know why, but for quite a while and still to some degree, people make very strong comparisons to architecture as in a person who builds a bridge, an engineer, maybe a civil engineer who builds a bridge.
36:41 Sure.
36:41 And software.
36:42 And they're like, the civil engineer can build a bridge within 10% of time estimates, materials estimates, and so on.
36:50 And, you know, people say, well, why can't we do this with software?
36:53 Well, the reason is software is an immature discipline and we need to get more mature.
36:58 But I think actually what is totally that totally misses the point.
37:01 The thing is, it's the tailor-made part, right?
37:04 Like, right.
37:05 If the software existed and you could just plug it in, then bridges you can't copy.
37:10 If you have 100 bridges that are real similar, you got to have 100 times build it, 100 experiences it.
37:16 Right.
37:16 But if it's software you use, you just you sell it, you know?
37:20 Any web dev shop, any studio can give you a very accurate estimate how long it's going to take to set up your CMS if you do not have, like, really high, you know, customization requirements.
37:36 And, in fact, when people go further, I don't know who's sponsoring these days, right?
37:39 It's Squarespace or whatever, right?
37:41 Like, they've automated it.
37:44 And so the main thing that's different in, like, the real world versus the virtual is the construction aspect, right?
37:51 So, I mean, there's no, like, special construct – there's very little special construction setup, especially in these smaller applications.
37:59 If you go look at ERP integrations, I think those still take, like, you know, a decade.
38:06 I, you know, for better or worse.
38:08 For better or worse.
38:09 Yeah.
38:10 So that is as enterprise as it gets.
38:14 It's maybe more enterprise than it has to be.
38:16 But I'm not going to sit over here and, you know, judge about that.
38:20 Yeah.
38:21 I think what your sort of list here highlights is you might say these are the requirements we have.
38:28 But it has to fit into a very unique and largely invisible bigger structure.
38:32 Right.
38:33 And it's for a specific user base who's never had this piece of software before.
38:36 And so, like, there's just so many unknowns and sort of one-offs that these analogies between, well, why can't we build software like we build bridges?
38:46 Become more obvious when you think about all the stuff going on there.
38:49 Right.
38:50 And I think that we do need to think more about it.
38:52 Like, and we need to talk more about it.
38:54 I don't really like, you know – this is sort of my reservation about what you said about the iPhone, right?
38:59 Like, you know, you can't really reduce every aspect of design to whatever Apple is doing.
39:05 Right?
39:05 Like, Apple doesn't make very much enterprise software.
39:08 And, I mean, internally, I'm sure they have many enterprise systems.
39:12 Don't get me wrong.
39:14 It's not going to represent a whole industry.
39:18 It's just one company.
39:19 Yeah.
39:19 You know?
39:20 So, yeah.
39:22 There's a lot of – I'm all for more variability in designs.
39:26 Yeah, absolutely.
39:38 This episode is brought to you by Hired.
39:40 Hired is a two-sided, curated marketplace that connects the world's knowledge workers to the best opportunities.
39:46 Each offer you receive has salary and equity presented right up front, and you can view the offers to accept or reject them before you even talk to the company.
39:54 Typically, candidates receive five or more offers within the first week, and there are no obligations – ever.
39:59 Sounds awesome, doesn't it?
40:01 Well, did I mention the signing bonus?
40:02 Everyone who accepts a job from Hired gets a $1,000 signing bonus.
40:06 And, as Talk Python listeners, it gets way sweeter.
40:09 Use the link Hired.com slash Talk Python to me, and Hired will double the signing bonus to $2,000.
40:14 Opportunity's knocking.
40:17 Visit Hired.com slash Talk Python to me and answer the call.
40:20 One of the things you talked about in the class that I thought was surprising that you'd bring it up a little ways into it was you said, let's talk about what is Python.
40:36 You said that this actually has several answers, almost depending on who you're talking to, to some degree.
40:42 Yeah, it has to.
40:44 Yeah, pretty deep into the course, I talk about what is Python, and that's because this course is sort of targeted at people who are self-taught, you know.
40:53 And so maybe they learned programming from Python.
40:57 They've taken a beginner course.
40:58 This is an intermediate course.
41:00 So they've taken a beginner course, and they're looking to get into professional software development.
41:04 And at that point, you know, the Python that they learned as a beginner, we need to expand that definition, flesh out that definition to work in the professional setting.
41:15 And, of course, there are also people who are coming to this from an enterprise software perspective, and they are learning more Python from this, and they can use – I mean, I deal with a lot of those – I mean, not deal with.
41:24 I interact with a lot of those people at PayPal who – I mean, you know, they are expanding their skill set.
41:32 They know a lot of Java, C++, JavaScript, and they want to learn Python.
41:36 And so, you know, I give them my definition of Python.
41:39 And so there are three levels at which, you know, we say Python.
41:44 So number one is the language.
41:46 You know, it's the language of choice.
41:48 It's so clear.
41:49 You know, some people don't like the white space stuff, but that's part of the language.
41:53 For many years, that language that we all love, it was directly implemented by more or less just one runtime.
42:03 You know, like CPython, the reference implementation, was synonymous with Python.
42:08 I don't run CPython at my command line.
42:10 I run Python.
42:11 And so many of the benefits of Python are actually benefits of CPython.
42:16 When I talk about the consistency and, you know, rich runtime, those things are – many of those are runtime-specific from CPython, implementation from CPython that may be copied in other runtimes.
42:30 So other runtimes might include, like, Jython, IronPython, PyPy, you know, P-Y-P-Y, and PyPy.
42:36 And, you know, now there are some other ones, too, Pidgin and stuff.
42:40 And then, finally, there's the platform.
42:43 So executives and managers, when you talk about Python with them, they're not really, you know, that interested in what the language looks like.
42:51 Some of them might be, but that's, you know, kind of micromanager-y.
42:55 Anyways, they're more curious about how is Python going to help the organization.
43:01 And so that means Python as a platform, as a community.
43:05 And there are many benefits to Python there.
43:07 You know, you have a lot more education adoption, so you have a lot more sort of pre-training when you're doing your hiring.
43:13 Like, basically, people are coming in already knowing the language.
43:16 And then you have a big open-source ecosystem, you know, 70,000-plus packages on PyPy.
43:24 So there's language, runtime, and platform.
43:26 And when you're having a meeting, it will be most effective if you can be explicit, if you can be clear about which part you're referencing, you know.
43:36 So those three levels of Python.
43:39 Yeah, I think even for experienced developers, a lot of times, if you're talking, like, let's say you're talking to a Java person and maybe a Node.js person and a Python person,
43:51 and they're saying, you know, Python is really good in this way, right?
43:55 You've got – but there's a lot of comparisons that will be made.
43:59 And you have to say, are we comparing the language?
44:01 Are we comparing the implementation with the standard library?
44:03 Are we comparing the entire ecosystem, including PyPI?
44:06 Like, before you say this is this feature or it doesn't or whatever, right, like, we need to know what we're talking about.
44:11 So that was really interesting, I thought.
44:13 Yeah, with Java, I mean, you know, Java is a language.
44:16 Python is a language.
44:17 CPython is a runtime.
44:19 You know, you might be using an IBM JVM.
44:21 That's your runtime.
44:22 And people say – people do compare.
44:26 This is sort of why I brought it up.
44:28 They compare Python to Node.js.
44:29 I'm like, okay, you know, Python is a language.
44:32 JavaScript is a language of Node.js.
44:34 Node.js is the runtime.
44:35 CPython is the runtime.
44:37 Let's, like, you know, compare these things side by side.
44:40 So people say Python is slow.
44:41 I'm like, okay, man, a language can't be slow.
44:43 CPython is not jitted.
44:46 You can use PyPy.
44:47 You can get certain performance benefits.
44:49 You know, Node.js doesn't support threads.
44:53 That's, you know, another aspect of the runtime.
44:57 I think that you could probably make a thread-safe ECMAScript, like implementation or something like that.
45:03 But I haven't really looked too deeply into it, honestly.
45:05 I'm sure it's possible.
45:06 But also not done, right?
45:08 So one of the things I thought was – yeah, maybe it's been done.
45:11 One thing that's interesting is sort of the soft skills story here.
45:16 Like, when you go and talk to executives or managers, it's not just – you don't want to get too technical, right?
45:22 You want to say, like, we're talking about the ecosystem.
45:24 We're talking about – take it as a whole, right?
45:27 And the truth is that businesses are run by people.
45:30 And your sort of skills to work with these people and convince them and speak their language really matter still.
45:35 Yeah.
45:36 So what I usually tell them – and so this is all just expanding that 10 myths thing, just blowing it wide open and approaching it from a more positive aspect.
45:44 It's not just rejecting myths, right?
45:46 It's saying, okay, well, what do you talk about instead?
45:49 And, you know, Python created in 1991.
45:53 It's like it had a nice, long, slow bake all the way until it sort of hockey-sticked for a lot of web things between 2003 and 2006 or so.
46:05 And so as a result, you have this, like, really, like, you know, well-tested, well-understood core reference implementation.
46:14 And you have a huge open source library.
46:16 It's used in education a lot.
46:18 And one that I like the most is – I have to tell people, it's organizationally neutral.
46:23 So a lot of people still haven't heard of Python.
46:25 You would be surprised.
46:26 People paid far more than I am.
46:29 I mean, they haven't heard of it even.
46:32 And one of the reasons for this is that Python is organizationally neutral, where IBM and Sun and Oracle – or, you know, Sun used to, but IBM and Oracle, like, you know, sell and sold Java.
46:44 You know, there's no company that is selling Python.
46:48 You know, there's no joyance of Python.
46:51 I mean, there's Continuum Analytics.
46:53 I mean, they're great.
46:53 You know, there's InThought.
46:55 But these are, like, you know, pretty small companies.
46:57 Your company wouldn't have, like, an enterprise license agreement with Python as a thing, right?
47:04 There are probably some groups that do it.
47:07 But no, it's not like that.
47:09 It's not a sold thing.
47:10 Instead, engineers choose it, and they use it, and they succeed, you know?
47:14 And you can get support contracts for it, but that's not really the Python way.
47:18 The other thing that you kind of touched on in that whole section is the idea of a foundational technology.
47:26 This is – managers especially, but basically everyone has this to a certain degree.
47:31 Like, people get really anxious about doing things the right way.
47:34 Basically, everywhere you look in technology, people are telling you that things are changing, okay?
47:40 This changes it all.
47:41 This is new, and it's going to change it all.
47:43 And, you know, I think we all know this, but we don't really get to think about it.
47:47 It's like a year later, you just don't hear about it ever again.
47:50 We've been fed this line about software being ephemeral and, you know, tools not mattering, only delivery.
47:56 You know?
47:57 This whole ephemeral technology thing, I don't buy it.
48:00 Foundational technologies are real.
48:03 So, specifically, like, Linux, BSD, and in general, the POSIX standard.
48:07 There are people who, if they woke up one day and read the news and said, hey, Linux is over.
48:14 You know, Linux is gone.
48:16 They'd be like, well, yeah, what did you expect?
48:19 I mean, you know, open source.
48:21 And it's just things are crazy.
48:22 You know, it's like they could actually buy that sort of reality.
48:26 But these things are permanent.
48:28 There's, like, long traditions.
48:30 There are long histories.
48:31 We don't usually get time to read into them.
48:33 But these foundational technologies are real.
48:37 Another would be the C programming language.
48:38 And CPython is a really, I mean, Python and CPython are really natural extensions, you know, relying on the C programming language.
48:46 So, I feel confident, you know, grouping Python with these foundational technologies, largely because C is so foundational.
48:54 You know, C is not going away.
48:57 You know, you can take it even further.
48:59 There are people I know, I mean, highly paid people here at my work.
49:02 They'll come up to me.
49:04 They're like, oh, man, but you're like the Python guy.
49:06 You know, you like new programming languages and stuff.
49:09 You know what I think?
49:10 I've heard that this visual programming thing, this is going to be the next big thing.
49:16 And it's like they see one demo and they think that, you know, procedural programming is going to go away.
49:22 That, like, top to bottom execution of code is in the future at some point not going to be something viable within their lifetime, within their work span.
49:31 And it's just such a wasteful conversation to have.
49:34 I talk about it a little bit in the course, that sort of navel-gazing sort of thing.
49:38 Which language is the best language?
49:40 Instead, I try to refocus them.
49:42 You know, I tell them maintenance is real.
49:45 And, like, longevity is possible.
49:46 And the tools and methods matter.
49:48 You have to use stuff that has been baked.
49:50 So the underlying spirit of the course is that if you choose your architecture, dependencies, and practices wisely, you know, you can rest easy thanks to these foundational technologies.
50:00 Yeah, and I think you're more likely to experience or run into the deep history of them at enterprises, right?
50:10 Like, at a startup, all the code is, what, a year old or whatever, besides maybe the package you grab.
50:15 But if you go to a company that's been using a technology for a long time, you could go to source control.
50:21 And the last check-in or the first check-in you would see might be, like, 1995.
50:24 And that's because you switched version controls and couldn't carry the history over or something like that, right?
50:29 Right.
50:29 No, you hit it right on the head, man.
50:31 I mean, like, I was blown away when, I think it was 2012, beginning of 2013.
50:37 I was actually, like, curious.
50:39 This is when I first started getting into the security stuff.
50:41 I was curious how something was implemented.
50:43 We needed to re-implement it in Python to enable running on 64-bit.
50:46 You know, PayPal primarily runs on 32-bit stuff because they have old SOs that are just linked in.
50:53 They're checked into the repo and stuff.
50:55 So, I mean, 64-bit was not a thing when PayPal was written.
51:00 And so, you know, 80% of our, I should say, too, like, 80% of our revenue or something like that is still going through the, quote, unquote, legacy stack, right?
51:10 But, you know, when you're actually, like, there re-implementing and reverse engineering Max Levchin's code, you're like, wow, software really is actually a lot more permanent than I was taught.
51:21 So, you know, by the Internet or whatever.
51:25 Yeah, yeah, yeah.
51:25 By the person with the latest framework that's super excited about it.
51:28 Exactly.
51:29 And so you have to learn to distinguish between a press release and, you know, actual, like, neutral academic whatever, you know?
51:39 There's press releases and then there's real experience, real education.
51:46 Yeah.
51:46 So one of the things you talked about was sort of Greenfield versus Brownfield or, like, brand new software versus legacy software, if you will.
51:54 Yeah.
51:56 I think you sort of convinced me that there's more value in these Brownfield apps and people that work on them and can evolve them and really add features and understand them.
52:07 I deserve maybe more respect than the industry gives them.
52:10 Yeah, that is definitely the case.
52:13 It's just a culture thing within technology.
52:15 The new is always emphasized.
52:17 But the old stuff is what we're using right now.
52:21 That's what's working right now.
52:22 That's what's causing money to move.
52:25 And disrespecting it and, you know, being productivist about it doesn't actually get us any further in technology progress.
52:32 You know, we have a lot of people who come in and they try to, you know, plow the whole field under.
52:38 And, you know, their tractor, their plow, whatever, just, like, you know, hits one rock.
52:44 And then they declare victory with the small patch that they turned, you know, green.
52:50 And they, you know, move on.
52:52 But, you know, and then you end up with this patchwork environment, you know, where it's like, well, that was green.
52:58 That was greened by so-and-so that long ago.
53:01 And, you know, that part over there they never reached.
53:03 And so you just end up with this very untended wilderness.
53:07 It's interesting.
53:09 It's a really fun environment.
53:10 You know, this is why, you know, you can see I get carried away just daydreaming about, like, the complexity inherent in a system as complex as PayPal's.
53:20 So we don't have a lot of time left in the show.
53:22 We're sort of getting to the end.
53:23 But I wanted to maybe talk about some of the stuff farther down, like some of the best practices that you're talking about.
53:32 One of my thoughts was interesting.
53:33 It was sort of a pull request, designer view type of experience.
53:38 Right, right.
53:39 So the way that the course is structured, you know, is I got an intro and then I defined the basics.
53:45 I just do some definitions.
53:46 And then I have this dry segment in the middle about architecture and design.
53:49 And so those first three parts are, like, I mean, especially the architecture and design part, those are just, like, processes and soft skills that, you know, you can use in the enterprise world.
54:00 Then we get to, like, the real meat, which is this, like, 12 segment best practices.
54:06 And this is sort of, like, you know, adjustments to, you know, your knowledge and pointers to how to do things that should be generic for most organizations, most legacy environments, and how you can put Python in them and create new software.
54:25 So once you get past, like, you know, just, you know, editors, dev tools, issue tracking, you know, how to start a Python project and some design patterns.
54:34 You know, people always ask me about design patterns.
54:37 They're coming from, like, a Java and C++ background.
54:39 And so I threw a segment in there.
54:40 It's, like, well, design patterns in Python tend to be, like, so small you barely even see them.
54:45 So I said, here's how to, like, actually look and see Python design patterns.
54:50 But, you know, eventually, like, so the first half of best practices is building new code.
54:54 And the second half is maintaining that code and building quality.
54:58 And I think that the first one of those is code review.
55:01 And this is one of my favorite segments because, you know, I get to do a little bit of, like, black hat, white hat sort of role playing with myself one weekend.
55:10 And so one day I just wrote some really bad code that I tried to build in every anti-pattern that a, you know, budding developer will accidentally stumble upon.
55:21 And I put it up on GitHub.
55:23 And I just, like, had to sleep on it.
55:25 I just, like, went to sleep.
55:26 And the next day I came back and I just, like, tore into myself.
55:29 I just, I mean, well, that's not true, actually.
55:34 Like, I did a precise code review stating why there were problems, where there were problems.
55:41 And I did my best to be nice about it to promote the actual, like, you know, positive interactions you try to have.
55:47 And so all of that is available on the GitHub repo for the class where I sort of have this example project,
55:54 which is useful for about a quarter of the class, just as sort of a reference.
56:00 And so that's github.com forward slash Mahmood forward slash SP metrics, ESPY metrics.
56:07 Yeah.
56:08 And I'll link to the link to the show notes as well.
56:11 Great.
56:11 Great.
56:11 And so there's just a pull request in there that I will never merge because it does everything the wrong way.
56:17 And so I have a bad implementation, all the comments on it, and then a good implementation.
56:22 And you can, you know, sort of do an apples to apples comparison there.
56:26 And, you know, it was interesting.
56:29 Yeah.
56:31 I think that's something that a lot of enterprise developers who maybe haven't spent a lot of time in open source don't get to really experience a lot.
56:38 The sort of GitHub pull request style of work, right?
56:42 That is true.
56:42 When I started at PayPal, we were using, I think, something you've probably heard of.
56:47 We were using ClearCase.
56:48 ClearCase was maybe worse than I could ever imagine for a version control system.
56:56 You know, it was like every file, like it was a file by file thing.
57:00 You could get these evil twins.
57:01 And I just, you know, I know that some people still have to use it.
57:04 I run into them here in like the valley.
57:06 And, you know, it's kind of a shame.
57:08 But, yeah, so we did actually not too long ago get GitHub Enterprise.
57:13 And I think that it's a much, much healthier way to develop.
57:18 You know, you have a decent branching strategy to keep things like sane.
57:23 And then you, this like very useful UI.
57:26 And I'm sure that like Bitbucket has similar stuff.
57:29 Yeah, the code reviews is actually exactly what, this is exactly the sort of thing that we do at PayPal as well.
57:34 And that's one of the reasons I did it because we have an internal DL as part of the community.
57:40 We have an internal listserv where people can request code reviews from other teams.
57:45 You know, and so that's one of the services we provide as the Python infrastructure team at work
57:49 is that we'll code review any Python code you give us.
57:52 And I just did my best to export that in an organizationally neutral way.
57:56 Yeah, so you've had a chance to look at both good and bad examples.
58:00 And you brought them to the class, yeah?
58:03 Well, and yeah.
58:04 And so that is like actually one thing you just reminded me, right, is that a big part of the class is evaluating dependencies.
58:11 I think that that's a great way to learn Python.
58:13 That's a massive way that I learned Python.
58:15 Reading Django code.
58:16 Reading bottle PY code.
58:18 You know, when I was doing all this web development, actually digging a little bit deeper, you know,
58:23 Python ships with its source code and you can see your actual code, like what your dependencies are doing, what you're relying on.
58:30 And so, you know, don't constantly live in the sausage factory looking at how things are made.
58:35 But take your time to evaluate dependencies because it will help you avoid some really horrible architectural nightmares.
58:41 At PayPal, I mean, I think we all know about the left pad thing at this point.
58:45 So anyways, at PayPal, we have a typical Python team.
58:48 It's small and we always have like 10 things to do.
58:53 So we have to make choices that favor stability and to make sure that when we finish something, it stays off of our plate.
58:59 As such, like we're really big advocates of vendoring in libraries.
59:03 So that means that we copy dependency into the project.
59:08 This isn't something you'll see a lot in open source projects because there's the Python packaging index, PyPI.
59:15 But we view like copying in this library at a certain version as sort of the Python equivalent of static linking.
59:21 Yeah, we do that too with our C libraries.
59:23 But we build projects with the dependencies built in.
59:26 And this way, teams can just clone the repository and go.
59:29 And then when you package an RPM or DEB or whatever, there's no install time dependencies.
59:36 And so there are minimal side effects.
59:37 Basically, repeatable deploys are critically important in any environment.
59:41 And whether you achieve them via RPMs, DEBs, containers, whatever, as long as you have those repeatable deploys, you're going to be in a lot better spot.
59:50 Yeah, definitely.
59:51 Repeatable and automated are the key element, right?
59:55 Yeah, I talk about CI in here.
59:57 We definitely use Jenkins and so forth.
01:00:01 I don't know if I'd recommend Jenkins specifically, but another team is running the Jenkins.
01:00:05 That's the thing.
01:00:06 We have a group that manages Jenkins for us.
01:00:09 And so if that's one more thing we don't need to do, that's one more thing of our own that we can do.
01:00:13 Yeah, that's awesome.
01:00:14 All right.
01:00:15 So I think for the course, we'll just have to leave it there.
01:00:18 We're out of time.
01:00:19 Let me ask you.
01:00:20 Oh, wow.
01:00:20 Yeah.
01:00:21 Let me ask the questions I ask everyone.
01:00:23 So first of all, I know I asked you this way back, like a year ago, but maybe it's changed.
01:00:28 Favorite editor?
01:00:29 No, I mean, it hasn't changed.
01:00:31 Emacs.
01:00:32 I still use Emacs primarily in the console.
01:00:36 Yeah, because this is interesting because now there's going to be like Ubuntu on Windows, sort of a GNU NT thing.
01:00:43 So I've actually considered getting a Windows laptop and running Emacs natively in it because we have so many Windows users here at PayPal to help support them.
01:00:53 Yeah, you'll be able to live the life they have to live and sort of experience it.
01:00:56 Yeah.
01:00:56 That was just announced last this week or no, last week at Build.
01:01:01 So yeah.
01:01:02 Yeah, yeah.
01:01:03 Basically, Ubuntu running natively on Windows.
01:01:05 It's weird to be excited about Windows feature.
01:01:07 For me, it's weird to be excited about Windows feature.
01:01:09 I could only get more excited if they told me that they're going to build in Python.
01:01:13 Maybe they will.
01:01:13 That would be intense.
01:01:14 You know, when Windows 10 was under development, I was using the community preview or whatever they called it.
01:01:22 And they had a user voice set up for you could recommend features and get votes.
01:01:26 And I recommended that they install Python and ship Python 3 and Python 2 as built in to Windows.
01:01:35 And I got, I think, over a thousand votes from the community to say, yes, Microsoft, build this.
01:01:40 Absolutely.
01:01:41 And sadly, it went nowhere.
01:01:43 Nowhere.
01:01:44 I mean, maybe somebody heard it.
01:01:46 You know, that's one thing with the enterprise.
01:01:48 Like, you know, there are so many, like, moving parts inside of it.
01:01:51 You know, you can, just because you don't see anything happen on the surface doesn't mean that people aren't hearing it inside.
01:01:56 Yeah.
01:01:57 It doesn't mean it'll go anywhere.
01:01:58 I'm not, I can't make promises for Microsoft.
01:02:00 Right.
01:02:00 Yeah, just the show I just recorded before this one was Steve Dower.
01:02:04 And he works at Microsoft on doing a lot of really interesting stuff with Python and Visual Studio and Windows.
01:02:12 And so there's, there's, there's a chance someone will pick it up.
01:02:16 We'll see.
01:02:16 We'll keep putting the idea out there.
01:02:17 I'm hopeful.
01:02:19 Yeah, absolutely.
01:02:21 All right.
01:02:22 And favorite PyPI package?
01:02:24 What do you, what do you find recently?
01:02:26 That's awesome.
01:02:26 This was all, like, this was really, really tough.
01:02:29 I mean, I, I don't think I, I said this last time.
01:02:33 I mean, G event and Greenlit, we use it so much at work that, like, I just have to give it a shout out.
01:02:38 G event and Greenlit.
01:02:39 But, you know, we have Twisted users here.
01:02:43 Twisted is a very respectable code base as well.
01:02:46 And, you know, I'll give a little shout out.
01:02:48 I'm working on a, a stats post that, like, you know, statistics for engineers and whatnot.
01:02:53 And a lot of that is motivated by a new project I'm working on called Lithoxal, which is sort of next generation logging and instrumentation.
01:03:01 You know, it's not public yet, but I know that some people on the podcast are probably early adopters.
01:03:06 And, you know, it's good to get some feedback.
01:03:08 So, L-I-T-H-O-X-Y-L.
01:03:12 Awesome.
01:03:13 Weird name, good library.
01:03:14 That's awesome.
01:03:16 We're going to give away a copy of your class, right?
01:03:19 Absolutely.
01:03:19 Yeah.
01:03:20 All right.
01:03:20 So, if you're listening and you want to check out the course, it is on O'Reilly.
01:03:24 All you have to do to be eligible to win is be a friend of the show.
01:03:29 So, visit talkbythumb.fm.
01:03:31 Go to the nav bar, click friends of the show in your email address, and then we will randomly choose somebody later in the week and give you a free copy of the course.
01:03:41 If not, how do they find it?
01:03:42 Like, if people want to check out the course, where do they go?
01:03:44 So, there are several places to find it.
01:03:46 So, one major reason I went with O'Reilly is because so many large organizations have Safari.
01:03:51 And so, I waited until the course propagated to all of the Safari-like installations.
01:03:57 So, if you have TechBus or, let's see, I mean, you can just go to Safari Books Online and type in Enterprise Software with Python.
01:04:05 Otherwise, you can go to straight up O'Reilly.com, and it's called Enterprise Software with Python.
01:04:11 And so, you can just search it.
01:04:12 I think it may even be Google-able.
01:04:13 The one thing is it has a price, right?
01:04:17 If you're someone who can't afford it, just get in touch, right?
01:04:21 This is something that I'm passionate about.
01:04:23 And, you know, I want to make sure that everyone has a chance to get into the professional Python software development world.
01:04:29 Yeah.
01:04:30 It's a really great course.
01:04:32 I totally enjoyed going through it.
01:04:34 So, nice work.
01:04:36 You have a course as well.
01:04:37 I do.
01:04:37 I just released one a month ago, Python Jumpstart by Building 10 Apps.
01:04:42 And that's been really well received.
01:04:44 That was cool to do it on Kickstarter.
01:04:45 And I just announced this week, I think, yes, Monday, that I'm working on a new one called Python for Entrepreneurs.
01:04:53 And basically, you know, if you've ever built a business, it's only like 30% product.
01:04:59 It's only 30% technology.
01:05:01 And there's all this other stuff.
01:05:02 Like, how do I accept credit cards?
01:05:03 You know, speaking of PayPal and that sort of thing.
01:05:06 How do I gather emails and send newsletters?
01:05:09 And just all, how do I do deployments?
01:05:11 How do I do SSL?
01:05:12 All that kind of stuff.
01:05:13 Oh, yeah.
01:05:14 It can be intimidating.
01:05:14 Yeah.
01:05:15 When you think you've finished your app and whatnot, you know, you're like, okay, well, I'm almost ready to release it.
01:05:21 No, you're like 30% of what you're done.
01:05:22 So, I wanted to create a class that would help people close that gap.
01:05:25 Great.
01:05:25 Yeah.
01:05:26 No, I've already, I think, bought a copy.
01:05:29 And, you know, we're going to actually have a little powwow with the Python community here at PayPal.
01:05:34 Oh, awesome.
01:05:35 Well, thank you so much.
01:05:36 That's great.
01:05:36 Yeah, it's good stuff.
01:05:38 Cool.
01:05:38 Well, Mahmoud, this is fun as always.
01:05:41 Your original 10 Myths post was super inspirational.
01:05:44 I think to a lot of people, it certainly was to me.
01:05:47 And I was really glad to feature as one of the early shows.
01:05:50 And this is kind of like the rounding out of that story.
01:05:53 So, everyone go check out his course.
01:05:55 It's great.
01:05:55 Yeah.
01:05:56 Thanks, everyone, for all the support, frankly.
01:05:58 Yeah.
01:05:59 I don't think I could have done it without at least some of you.
01:06:04 That's awesome.
01:06:04 That's awesome.
01:06:04 So, yeah, that's great.
01:06:06 All right.
01:06:06 See you later, Mahmoud.
01:06:07 Thanks.
01:06:07 Yeah.
01:06:08 Bye-bye-bye.
01:06:09 Bye-bye.
01:06:10 This has been another episode of Talk Python to Me.
01:06:13 Today's guest was Mahmoud Hashemi, and this episode has been sponsored by SnapCI and Hired.
01:06:17 Thank you guys for supporting the show.
01:06:19 SnapCI is modern continuous integration delivery.
01:06:22 Build, test, and deploy your code directly from GitHub, all in your browser with debugging,
01:06:26 Docker, and parallelism included.
01:06:28 Try them for free at snap.ci slash talkpython.
01:06:30 Hired wants to help you find your next big thing.
01:06:33 Visit hired.com slash talkpython to me to get five or more offers of a salary and equity
01:06:37 right up front and a special listener signing bonus of $2,000.
01:06:41 Are you or a colleague trying to learn Python?
01:06:43 Have you tried boring books and videos that just cover the topic point by point?
01:06:47 Well, check out my online course, Python Jumpstart, by building 10 apps at training.talkpython.fm.
01:06:52 You can find the links from today's show at talkpython.fm/episodes slash show slash 54.
01:06:58 Be sure to subscribe to the show.
01:07:00 Open your favorite podcatcher and search for Python, which should be right at the top.
01:07:03 You can also find the iTunes and direct RSS feeds in the footer of the website.
01:07:07 Our theme music is Developers, Developers, Developers by Corey Smith.
01:07:11 It goes by Smix.
01:07:12 You can hear the entire song on talkpython.fm.
01:07:15 This is your host, Michael Kennedy.
01:07:16 I really appreciate you guys taking the time to listen.
01:07:18 Smix, let's get out of here.
01:07:21 Outro Music Outro Music