#54: Enterprise Software with Python Transcript

00:00 How often have people asked what language / technology you work in and when you answered Python they got a little confused and asked, what can you actually build with Python? What type of apps? The implication being Python is just a notch above Bash scripts. That real things aren't built with Python but rather with Java, C#, Objective-C and so on.

00:00 Mahmoud Hashemi and I might be able to help you put some real evidence and experience behind your response. On episode 54 of Talk Python To Me, I talk with Mahmoud about his new online course he wrote for O'Reilly entitled Enterprise Software in Python. You'll hear many real-world examples from his experience inside PayPal and many more throughout the industry. This is episode 54 of Talk Python To Me, recorder April 4th, 2016.

00:00 [music]

00:00 Welcome to Talk Python To Me, a weekly podcast on Python, the language, the libraries, the ecosystem and the personalities. This is your host, Michael Kennedy follow me on Twitter where I am at @mkennedy keep up with the show and listen to past episodes at talkpython.fm and follow the show on Twitter via @talkpython.

00:00 This episode is brought to you by Snap CI and Hired. Thank them for supporting the show on Twitter via @snap_ci and @hired_hq.

00:00 The news this week is all about online courses. You want to be sure that you are a friend of the show because we have two cool giveaways, so just drop by talkpython.fm and click on friends of the show on the nav bar to become eligible. Mahmoud is giving away a copy of his online course he normally sells for $149. I'll pick one lucky friend of the show later this week to receive the copy. You can find the links to his course in the show notes for this episode.

00:00 Speaking of online courses, my Python Jumpstart by Building Ten Apps is out and people are really loving it, I thought I joined Mahmoud and give away a copy or two this week as well, you can see the details of my course at talkpython.fm/course.

02:06 Michael: Now, let's talk about Enterprise Python. Mahmoud, welcome to the show, welcome back to the show

02:12 Mahmoud: It's great to be back.

02:14 Michael: Yeah, I really enjoyed the episode we've done before, and this is your third appearance, you were on as the main guest in episode 4 where we talked about the myths of enterprise python which was based on an article-

02:28 Mahmoud: That was sort of when this was kicking all off, that's right, the Ten Myths Of Enterprise Python blog post.

02:36 Michael: That's right, this is like the germ or the seed that like generated this course we are going to talk about, right?

02:42 Mahmoud: It's been a wild and crazy ride, that's right.

02:43 Michael: I bet it has. And then you gave some cool inside information on how you guys think about hiring junior engineers back at episode 41 and this is going to be episode 54 so I am instituting a new role, you are not allowed to come back on the show unless there is a 4 in the number.

02:58 Mahmoud: That seems like a fine rule to me, I think like so at least ten more episodes I can probably think of something in that time.

03:08 Michael: Perfect, perfect. All right, so you know, people heard your story back on show 4, so we won't go sort of through how you got back into programming that's the question I often ask people. Instead, I'd love to hear how you got into enterprise programming and like what was the step where you went from just I am doing programming, I am doing Python, to like I am now working at a company that has like more employees than some small towns?

03:32 Mahmoud: You know, that's actually something that recently I was forced to sort of ponder a lot on, like I just had some really deep shower thoughts, as it were, and, so yeah, I sort of fell into it as I am sure so many do, like you get an internship at a company and they assign you a kind of you know, maybe the worst stuff that you can do within three months, like they want you to do something but they don't want it to be super, super glamorous, they give that to their star engineers who already have full time positions. So as an intern, I was doing some really niche log analyses stuff and so I actually defined enterprise software in my course, and one of the main things that makes software enterprise isn't that it's high performance, or something like that, it's actually that it's very niche, it has very constrained user basis interested in certain things. And so, I just started out doing really niche applications because I loved software introspectability so much, if you tell me hey go analyze these logs, go find out what's slowing this application down, that's something that even now I can't really resist.

04:47 Michael: Yeah, that's really cool, those types of enterprise apps are pretty different, I spent a lot of time working at a company 6, 7 years, one of my first jobs working at a company where I think there was maybe maximum of 25 employees, and we were writing internal software, all sorts of cool stuff, at this space where we did i tracking and we'd write custom apps to sort of collect very specific studies what to do with i tracking, and things like that, and I think we wrote some really powerful and beautiful software, but it was for like 10 people, you know, and so that's, it's not sort of the scalable sort of advertisement driven concept of like Pinterest, and hockey curve growth and all that, but it's the people who use that software like, their jobs like the whole company depends on that software, so it's pretty interesting, just living in that space.

05:40 Mahmoud: They don't really have a choice, so like I mean, that's kind of what makes a lot of things and applies that sort of the, that's stuff that makes a lot of enterprise stuff kind of bad, but we can probably go into that later.

05:55 Michael: Yeah, the whole battleship grey I don't need to know design just be a programmer, that king of thing?

06:01 Mahmoud: Yeah, that sort of thing. You know, there is no alternative, so you don't spend any time actually making stuff good, that's one approach some people take, you know, I am not for that approach personally.

06:11 Michael: Yeah, I think that used to fly, especially in the 1990s, when you write enterprise software it could be seriously unusable, but ever since the iphone came out, and people have gotten used to really beautiful nice computing experiences they are like why do I have to go to work and have this horrible experience, right.

06:34 Mahmoud: That has actually left its mark, I agree. I mean, there are pros for that there are cons for that, now I see a lot of stuff getting polished a whole lot but the underlying quality is still lacking, so you know, there are a lot of ways to compromise quality and enterprise will always find a new way.

06:53 Michael: That's right, yeah.

06:55 Mahmoud: So you know, I have, this hopefully will get to cover a little bit about how to sort of mitigate that with the power of Python.

07:02 Michael: Yeah, absolutely. So one concept I kind of like to introduce near the beginning before we get into the course is just that when you think of enterprise developers and you can maybe put Python to the side for just a moment and just say like across all technologies a lot of times there are absolutely are super smart people writing amazing software, but maybe they are just going to work and they are just doing their jobs and they are not going to speak in the conferences and they are not on Twitter constantly blogging and tweeting about whatever it is they are doing, they just go do their job and they go away. So, I think that we probably underestimate the impact and the number of people that kind of live in this space because they are not necessarily as vocal.

07:49 Mahmoud: Yeah, and I mean, I don't know, I am not really for the cult of personality, you know, rockstar coder type stuff, I think that like every developer has a rockstar within them, I know because I mean, they talk about Tenex programmers like and some people say hey, they don't exist and other people say hey they do exist, and it's like a little bit like this big foot of developers and I don't know why it's so controversial, why it has to be like that, everyone has within them like a tenex developer, if you are struck with the right idea, you will work tirelessly and learn so much within the span of just a couple of weeks and you know, the rest of the time like you said, you have a normal work-life balance and you have a day job and that's totally fine, but tenex developers do exist and they are within all of us, that's what I choose to believe.

08:38 Michael: I totally agree with you, I think if you are like you said struck with inspiration, you can do almost super human type of work, you are so focused, so effective, and on one hand it would be cool if you could always be that focused and effective, but you know, that's how inspiration works, right, if you both have the time and the energy and the idea all coming together, like you could be the tenex developer for that project or whatever.

09:04 Mahmoud: Right. I mean, you won’t believe this, you know, looking at my GitHub or my Twitter or whatever, right, but I do think that there is much more to life than just code. [laugh] So, yeah.

09:17 Michael: Yeah, same here. But I try to keep it basically to technology on Twitter as well so it's hard to tell. So let's talk about your course. So it's called Enterprise Software With Python. Right, and you distributed this through O'Reilly is that right?

09:34 Mahmoud: That’s right, so I think that basically because of the enterprise python blog post they actually contacted me a little bit before we did episode number 4 and they were like hey would you like to expand upon this and do a project on enterprise software. And you know, I am sort of more of an engineer, I am a builder you know, and I have a lot of reservations around education, like I am a firm believer in education but it's very time consuming, my dad you know, he is a professor of virology, I have really high standards and I know that if I get involved with something like this I am going to go all out, and so I hand and I 10:16 but then I finally like decided to make it happen in August of last year, I started working on it and I choose the video format because of how successful I saw that it was inside the company here and so it's a video course, in preparation for it I actually taught over a dozen internal training courses at PayPal, and just covering all sorts of different topics sort of honing in on the best explanations for various topics in enterprise software, and I ended up with this course that is I would say it's like half and half, it's like half enterprise software and half like how to do it in Python. Python just works really well as an example language and you know the ecosystem is very full bodied so you know, it really fit well but it has whole segments of it that are not at all python significant.

11:13 Michael: It's almost like a case study of enterprise software but from you know choosing Python as a specific language, that's really cool. So one of the things that you ask in this class, and I guess your class is like in your blog post it was sort of a people think these things are inappropriate in Python let me show you all the ways the ten ways, where they've got some false conception, right, the Python is just a scripting language or something silly like this that gets in their way but this is more like well, we'll talk about those but let's actually study the whole thing in a more concrete way, right, and so one of the questions you sort of started out which I thought was interesting is you said is Python suitable for the enterprise and when should you write Python code.

12:04 Mahmoud: Right, and that is I mean, it's a valid question, right, you are a company and you know, as an engineer Python it's a language of choice which means that like engineers are going to choose your bosses are going to choose it for you, usually like C level type people do not choose Python for their company and that's kind of something I am trying to reverse here, Python has been shown to be like tremendously successful for many very large scale companies, YouTube, Yelp, Dropbox, and you know, I believe CERN uses it, it's really all over the place, so I mean it shouldn't be as controversial as it is. And so I go down the list of when you, when we've actually seen it used, I am not saying that hey, here is where I think you should use it, I'm like no, here is where it has been used successfully, and then I give a couple of areas where I don’t think it's as common for enterprise development such as, you don’t really see it that often in web front end, that's pretty much all Javascript and Javascript derived, you know there is Pyjamas that's not really, there's Brython and that like for browser Python that's not really quite there yet. But, yeah, so, but yeah, Python is absolutely suitable for Enterprise and I don't think that it's just generic organizations that don't understand this, even when we go to conferences and speak with other Python developers, they are kind of appalled to hear that we are like shooting for 100 micro seconds response times. You know, 100, 200 micro seconds response times, you know, as served from CPython. And so yeah, I mean we can hit those enterprise performance type numbers very consistently with some good engineering and I think that this course lays the groundwork for making that sort of software possible in Python.

14:07 Michael: I totally agree, I think you brought up CERN, I talked to Kyle Cranmer.

14:13 Mahmoud: I know, I listened to that episode.

14:13 Michael: That was so excellent and I am just thinking if Python is appropriate to help collect the data from the Hadron Collider and win the Nobel Prize, for finding the Higgs Boson, it's probably not too bad to like monitor your log files, or something like that. People think they have these amazing amazing requirements, but a lot of times there is just a bunch of little tools you've got to build and you know, you don’t need some super specialized thing.

14:44 Mahmoud: Yeah, since the last episode, main project I've been working on is a sort of t distributions system for securing sensitive data inside of PayPal, you know trust and security is PayPal's number one like selling point, that's what the brand is associated with, it is the safest way to pay online, that sort of thing. And, to keep it that way, we have to be innovating in that space and so that is basically as low level a system as you will find inside of PayPal, like everything depends on that, and you know, we expect billions of requests per day, two, three billion requests per day, we are already like seeing the adoption is happening, we have seen days of 500 million requests per day, and you know, the average median time that we are shooting for is less than two milliseconds for the response. So, it's not just what logs or whatever it is for these critical systems and it's the main reason that we chose Python was because of it consistency, the ability to measure how the software is performing, and because we can maintain with the very small team, the security team is not a huge team at PayPal application security I should say. Info sec that's a much larger team, but application security is much smaller team.

16:13 Michael: Wow, that's really amazing that that app written in Python is at the foundation of all those things like billions of requests per day is- it's a number that not so many people deal with, that's really really cool and that's written in Python, is it like an HTTP service or what's the story there?

16:28 Mahmoud: Oh yeah, so that's one of the great things, so because of Python is easy, I mean, it's not just Python, we have a framework that we've built on top of that most of that is open source on PayPal's GitHub, it's called support as sort of reference framework, so using a framework based on that, we have several protocols that we support, HTTP is like if you tie hp 1.1, 1.0, is kind of slow to parse, so we go for lower overhead protocols we are fans of net string, that's a classic, it even has sort of roots in the crypto community because DJB Daniel J Bernstein defined it so many years ago, we have wikipedia page it's extremely simple net string, and then we have something even lower level that that which is similar to that string but it's called TLV tag length value, and so you do have to go for like lower level protocols there but because it's Python you know, we can maintain an http interface for development purposes so the people can mark thing out without having to worry about the details of tlv.

17:44 Michael: I guess if you are going to get two milliseconds response time you are more or less down the raw sockets almost, right?

17:51 Mahmoud: Yeah, yeah. No, we don't shy away from raw sockets, we have a little buffered socket that we written on top of that, but when you want to hit sub millisecond, deep submillisecond times, then you do have to basically be counting the bytecode instructions that you are going to be like executing. So every line does count, every line does have its cost, but Python makes it easy to gage that cost and because you can get your features done in time you can actually start down that performance path much sooner than other stacks and that's the big selling point for us.

18:28 Michael: Yeah. You can do the testing and experiment and find the right answer, probably before the first prototype would have been even finished in C++.

18:37 Mahmoud: Oh yeah. I mean, in fact, we have had literal, I mean, I wouldn't call in competitions, right, inside a company we are all one team, right, but like we've had cases where that exact stuff was happening, like you know, we were done months ahead of time and already like iterating on our performance by time C++came around with their stuff because they couldn't even get within a order of magnitude of our scalability.

19:00 Michael: Wow, that's awesome. So let's take it back to the beginning for a moment, you talked about your first enterprise app and that was kind of log file monitoring thing I think, and you said there was a couple of lessons you learned from it, and I thought they were really good, maybe you could talk about those a bit?

19:17 Mahmoud: well, so first version of that log analyses web front end was actually written in PHP and that was my last PHP project, and yeah, you've got it done in time but you had some problems number one is that like it was written in PHP so that like maintenance wasn't really on my side, and that I actually had to keep fixing bugs and running the service and I basically was left with no time to actually do other projects. And so I didn't want to be the guy that just maintains log analyses tool, so I shut it down after about a year. From then on, I gave a lot more thought to what is the maintenance curve going to look like, is this something that I am going to be able to sort of set aside and move on to new things, we always have something new, but that old stuff can keep like running, I don't want to be one of those flash in the pan project guys I'd like to build up a lot of different services that I run, that help people in real situations. The exact lessons, were basically like choose your dependencies, and avoid doing it all at once so if you can rely on your organization I mean you are working in large organizations, find ways to rely on the organization off load some of that maintenance and finally, like that longevity it doesn't come easy, it's a result of a lot of critical decisions.

20:49 Michael: And it's more than just regular software I think, like you could write-

20:55 Mahmoud: There is a lot of soft skills involved.

20:55 Michael: Absolutely. So you could write the software if you are kind of doing this little app on your own and it could do its own little thing but if it's going to continue to live on and be used, you have to know the sort of invisible connective tissue of the software of your company right, like this is the way things are done and this is the way it will actually be useful to people, yes I should design it this way if it was from scratch, but people don't do that, they do this and so I want to do this other weird thing because then it will get used for long time and so on, right?

21:26 Mahmoud: Right. Like, I mean, I am all for going against the grain a little bit, but and actually innovate and change things and so forth, but if you try to do it all by yourself or all at once, you are setting yourself up for let's be frank, you are going to burn yourself out, right, and you are not going to be able to do those future projects and that's the whole point, you know, is to continue building things because there is a lot of things to be done.

21:26 [music]

21:26 Continuous delivery isn't just a buzzword, it's a shift in productivity that will help your whole team become more efficient. The Snap Ci's continuous delivery tool you can test, debug and deploy your code quickly and reliably. Get your product in the hands of your users, faster and deploy from just about anywhere at any time. And did you know that Thoughtworks literally wrote the book on continuous integration and continuous delivery? Connect Snap to your GitHub repo and they will build and run your first pipeline automagically.

21:26 Thank Snap CI for sponsoring this show by trying them for free at snap.ci/talkpython.

21:26 [music]

22:52 Michael: Yeah, and the- you had some really interesting statistics about the types of apps and the numbers of services and so on inside PayPal, in your course, and I kind of laughed thinking about oh my gosh this is crazy to think, so I am talking about like how do people within the company work and how the software normally connect, you had like a graph that was basically so dense it was illegible but showing the inter connection between services. Can you talk about that for a sec?

23:21 Mahmoud: Sure, so PayPal is a service oriented architecture company going back over a decade; it's started as a monolith and there's nothing wrong with that when you are starting something out like a monolith is a really good way to get things done, but as you are expanding, both in terms of customer base, computers running in your software, people working on that software, most importantly, then you, it starts to become a burden. And, eventually that hit a technical wall for PayPal where that monolith called WEBSCR I think I can talk about that, W-E-B-S-C-R, if you look at some PayPal you will still see it, it's a C++ executable and it reached over a gigabyte and a half, one program. And I don't really talk about that in the course, it's C++ not Python, but it hit like 1,5 GB you can't even put debug symbol in there in order to debug what life issues are going to be so it became completely, 24:30 and at that point they knew that they had to switch to something more distributed so they could work on it with more engineers and so you have this service oriented architectures sort of for human reasons, you know, you want to have a user team, you want to have payments team, you want to have a front end team or whatever, there is, and so you have to start segmenting things by layers and domains and so at PayPal we have done this into extreme, we have one huge product, you know, Google, they have gmail, they have maps, and even they have sort of monolithic development structure in a way but these are all separate products as well, like to emphasize. At PayPal, there is not that many different products, it's pretty much paypal.com and so just for paypal.com you have over 3000 service end points logical end points, you know, there are tens of thousands of machines running the code and over 3000 end points, and that's like representing hundreds of code basis and so this is all communicating with the variety of protocols, I've listed off a few already but this pretty much goes http, restful etc, then you have net string for some of the lower level older stuff, but that pretty much, we have one like proprietary one that's really old and will never go away, you know, I talk about that in the course too. There is a sort of design permanence it's indelible.

26:08 Michael: Yeah, absolutely.

26:09 Mahmoud: Yeah, but so yeah, we have these thousands and thousands of end points and they rely on each other in really complicated ways and each individual team has to consider how those runtime dependencies and logical dependencies are going to affect the scalability and performance of their application.

26:26 Michael: That's a lot to keep in your mind and sort of juggle, like if I call this service and that service and this one calls that one like what is the latency and scalability considerations I have to actually worry about.

26:39 Mahmoud: Well, some people don't even get to the point where they are able to worry about that stuff, that's kind of the problem due to other decisions they end up like constantly having to fix bugs or work around issues and they don't get to have those important spare cycles to build quality.

26:57 Michael: So you did talk about scalability a bit in your course, I thought that was interesting that you said scalability is not itself feature of a software but it's actually like a composition of 3 or 4 bits or more sort of atomic units of like throughput and so on.

27:15 Mahmoud: Yeah, so I thought about it a bunch, and I couldn't really find a anthology of what makes up a software, what are the aspects of software that really applied to what I saw happening at PayPal and so I sort of had to come up with one of my own, so yeah, we came up with this anthology talking to co workers and so forth of six different software aspects and so the first is like usability, you know, that has to be first if your software is not usable it doesn't have features to use then it's really nothing at all, so you have to have usability, and then you know we have availability, that's kind of the reliability, we have the introspectability, so that's how good your logging is and other instrumentation, then we have performance, everyone when they think of enterprise software they think about like they think of talking about performance and that's just one aspect here, and within performance, I am referring to like latency throughput and utilization, there are many aspects within each of these aspects; fifth has to be security, security really makes everything so much harder I can tell you for a fact one of the reasons why other companies can move like faster than how PayPal moves is because they don't have to worry about locking down access to all these secure resources. Our deploy process is primarily convoluted because of that security consideration, normal developers do not have access to where their code runs. That brings me to my sixth aspect, which is the agility, right, your ability to deploy quickly change things quickly, everyone talk about like oh, everyone on their first day should be able to change one line of code and deploy within one hour or something like that and I am like ok man, like you know, one line of html changing the styling of a button or something is very different than like one line that impacts everyone's security.

29:18 Michael: Yeah, that security service you are talking about, that does almost billion requests per day.

29:23 Mahmoud: Yeah. Several Billion.

29:26 Michael: Yeah, that one you don't want like the new guy or girl to modify necessarily on the first day.

29:33 Mahmoud: Right, and so and these six aspects, they are all like interconnected, like better introspectability, we find co-relates and causes this better performance, and because you can actually see the effect that your changes are having. And so, these six things sort of create like software hexagon, or whatever, kind of like the project management triangle which aka pick any two, right, fast good and cheap. You can make something that is very usable and you can deploy it very quickly but maybe its availability is not so great because you don't have a good throttling scheme, it's like you have to choose a balance of these things and as a result of choosing a good balance of these six aspects you can build scalable software.

30:21 Michael: Yeah. Very interesting. I think the good, fast and cheap triade is very effective mechanism for speaking to people who are not developers, and developers as well but it comes through very well.

30:34 Mahmoud: Yeah, but I mean good is just so nuance, fast that's just a certain amount of time, cheap,t hat's also a certain amount of money those are easy to quantify but what does good mean, you have to expand that and that's really a success where it comes from.

30:45 Michael: So, you sort of set out to define a spectrum of enterprise software if you will, you said ok the thing is not either enterprise software or not enterprise software but it might live somewhere along the continuum, and you have like nine points that would push it one way or the other. Do you want to talk about those, I thought they were really on point.

31:07 Mahmoud: Yeah, enterprise it's either enterprise or black and white view is not actually really helpful in helping people find just 31:16 and analogies to the work that they are doing. we want to make something that people can actually relate to what they are working on, I created a list of these, and I'll probably release it as a blog post or something like that as well, but I think that this one is one of the three clips that you can view, but to just run down the list really quickly; you can have any number of these and it sort of defines how much, how enterprise your software is. So number one is that if use by business, number two is that it's sort of tailor made, like it's custom, and number three is that you have a specific user base, like we talked about before, if there are just two or three people using your software it's going to have a really different development feed and flow. Then, you have strict runtime requirements, that's where the performance comes in, you know, I need this with an SLA 95% like in 200 milliseconds, whatever. Then you have that your software is a part of a larger system, and so at PayPal if you have to, we have to integrate not just internally with thousands of service end points but also externally with financial processors, and you know, all of these highly regulated basically legacy systems and so that brings me to the next one which is legacy integration if you have to integrate with legacy I just have to give that a special shout out, that's enterprise.

32:50 Michael: That moves you far down the spectrum to the right, doesn't it?

32:54 Mahmoud: Exactly, and you know, so and that feeds into kind of the design permanence and so people worry a lot about making the right decisions, and we talk a lot about technical dead, but if one person is technical death is another person's software longevity, software that has to run on aeroplanes, and satellites and so forth, it's not technical dead, it's just a different kind of investment and so design permanence is definitely a hallmark of enterprise software because of all of the thins involved, you do all these integrations if there are negotiated timelines and promises being made about like delivery dates and so forth, some people will just say hey that's just waterfall and thus bad, but that' show a lot of systems have to work, when you have these deep dependency trees of software like you know, dependency I guess, people relying on you having something done at a certain time. And finally, like you know, when you are handing software off to another team to be run, if you have a separate operations group, a separate reliability engineering group, if you have separate security group that is going to audit your stuff, if there is all these specialized groups that's another aspect of enterprise software. And so if you look at these nine different like hallmarks, you can see that in consumer ecosystems which consumer is the opposite of enterprise, these aren't really the case, for instance, it's specific user based, usually, in consumer software you are just putting an app on an app store and people whether they will like it or they won't like it, there is a market, there is choice, you know, user base that is going to be using my first Python project to PayPal changes the prices of PayPal. There is good reason why we don't want thousands of people using that, basically has ten users or so who can actually modify the values in there and they are still using my code, you know. And it's been maintained by different group, so I consider that enterprise software even though it only gets hundreds of requests a day at most, it has a specific user base, as part of a larger system legacy integration-

35:13 Michael: Tailor made.

35:14 Mahmoud: Right, it's tailor made I mean I use Django of course but at that time, and but it had to integrate with a lot of these systems, a lot of these human processes, in a way that it's just different than consumer software. So just defining something as enterprise based on its performance requirements, excludes this other part of the spectrum and that's not really that great. I am not saying that that application is the most enterprise, probably the most enterprise software that can occur to me is the either code that runs on the 7 47 whatever, you know aeroplanes, I mean that's far more enterprise than I will ever write, it's even more enterprise than like the security stuff I can think, there are lives at stake. So medical software, I consider that enterprise.

36:03 Michael: Yeah, I think that's an interesting sort of nine dimensional spectrum that your code can live in and I think it hits a lot of interesting points. One thing that sort of struck me when you were talking about that with the tailor made specific user base and thinking about how it fits into a larger system, you know, I think I've heard less of it, lately, I don't know why but for quite a while and still to some degree people make very strong comparisons to architecture, as in person who build a bridge and engineer, maybe a civil engineer who builds a bridge, and software, and they are like the civil engineer can build a bridge within ten per cent of time estimants, material estimants and so on. And, people say well why can't we do this with software? Well, the reason is software is an immature discipline, we need to get more mature, but I think actually what is totally- that totally misses the point, the thing is it's the tailor made part, right, like if the software existed and you could just plug it in, then bridges you can't copy, if you had a hundred bridges that are really similar, you've got to have a hundred experiences, right, but if it's software you just- you sell it.

37:21 Mahmoud: Any web dev shop, any studio can give you a very accurate estimate how long it's going to take to set up your cms if you do not have like really high customization requirements. And in fact, people go further, I don't know who is sponsoring these days, right, the square space so whatever like, they have automated it, and so the main thing that's different, in like the real world versus the virtual is the construction aspect, right, so I mean, there is no like special, there is very little special construction setup, especially in these smaller applications, if you go look at erp integrations I think those still take like a decade, for better or worse. So, that is as enterprise as it gets, it's maybe more enterprise than it has to be but I am not going to sit over here and judge about that.

38:21 Michael: Yeah, I think what your sort of list's here highlights is you might say these are the requirements we have but it has to fit into a very unique and largely invisible bigger structure, and it's for a specific user base who has never had this piece of software before and so on, and like there is just so many unknowns and sort of one offs that these analogies between why can't we build software like we build bridges, become more obvious when you think about all the stuff going on there.

38:51 Mahmoud: Right, and I think that we do need to think more about it, like we need to talk more about it, I don't really like this is sort of my reservation about what you said about the iphone right, like you can't really reduce every aspect of design to whatever apple is doing, right, like Apple doesn't make much enterprise software and I mean, internally, I am sure they have many enterprise systems, don't get me wrong, but it's not going to represent a whole industry, it's just one company. So, yeah, I am all for more variability in designs.

39:27 Michael: Yeah, absolutely.

39:27 [music]

39:27 This episode is brought to you by Hired. Hired is a two-sided, curated marketplace that connects the world's knowledge workers to the best opportunities.

39:27 Each offer you receive has salary and equity presented right up front and you can view the offers to accept or reject them before you even talk to the company.

39:27 Typically, candidates receive 5 or more offers in just the first week and there are no obligations ever.

39:27 Sounds awesome, doesn't it? Well did I mention the signing bonus? Everyone who accepts a job from Hired gets a $1,000 signing bonus. And, as Talk Python listeners, it get's way sweeter! Use the link hired.com/talkpythontome and Hired will double the signing bonus to $2,000!

Opportunity is knocking, visit hired.com/talkpythontome and answer the call.

39:27 [music]

40:26 Michael: One of the things you talked about in the class that I thought was surprising that you would bring it up, was you said let's talk about what is Python. You said, that this actually has several answers almost depending on who you are talking to to some degree. 

40:43 Mahmoud: Yeah, it has to, yeah in course I talk about what is Python and that's because this course is sort of targeted at people who are self taught, you know, so maybe they had learned programming from Python, they have taken the beginner course, this is an intermediate course. So they have taken a beginner course and they are looking to get into the professional software development and at that point, the Python that they learned as a beginner, we need to expand that definition, flash out that definition to work in the professional setting. And, of course, there are also people who are coming to this from enterprise software perspective and they are learning more Python from this and they can use I mean, I deal with a lot of- I mean, not deal with, I interact with a lot of those people at PayPal who I mean, you know, they are expanding their skill set, they know a lot of Java, C++, Javascript and they want to learn Python. And so, I give them my definition of Python and so there are three levels at which we say Python, so number one is the language, you know, it's the language of choice, it's so clear, some people don't like the white space stuff, but that's part of the language. For many years that language that we all love, it was directly implemented by more or less just one runtime, you know, like CPython, the reference implementation was synonimous with Python, I don't run CPython at my command line, I run Python. And so, many of the benefits of Python are actually benefits of CPython, when I talk about the consistency and you know rich runtime, those things are many of those are runtime specific from CPython implementation from CPython that maybe copied in other runtimes, so other runtimes might include like Jython, IronPython, PyPy, and now there are some other ones too, PyJion and stuff. And then, finally there is the platform. So, executives and managers when you talk about Python with them, they are not really that interested in what the language looks like, some of them might be but that's kind of micro managery, anyways, they are more curious about how is Python going to help the organization, and so that means Python is a platform, as a community and there are many benefits to Python there, you have a lot more education adoption so you have a lot more sort of pre training when you are doing your hiring, like basically people are coming in already knowing the language and then we have a big open source ecosystem you know, 70 000+ packages on PyPi so there is language runtime in platform and when you are having a meeting it will be most effective if you can be explicit, if you can be clear about which part you are referencing. So, those three levels of Python.

43:40 Michael: Yeah, I think even for experienced developers, a lot of times if you are let's say talking to a Java person and maybe a NoJs person and Python person, they are saying Python is really good in this way, but there is a lot of comparisons that will be made and you have to say are we comparing the languages, are we comparing the implementation with the standard library, are we comparing the entire ecosystem including PyPi, like before you say this is this feature or whatever like, we need to know what we are talking about, so that was really interesting I thought,

44:14 Mahmoud: Yeah, with java, I mean, you know Java is a language, Python is a language, CPython is a runtime, you know, you might be using an IBM JVM that's your runtime, and people do compare, this is sort of why I brought it up, they compare Python to NoJs, I say ok, Python is a language, Javascript is a language, NoJs is the runtime, CPython is the runtime let's compare these things side by side. So people say Python is slow, I'm like, ok man a language can't be slow, CPython is not JIted, you can use PyPy, you can get certain performance benefits, NoJs doesn't support threads that is another aspect of the runtime. I think that you can probably make threads say ecmascript like implementation or something like that but I haven't really looked too deeply into it honestly.

45:05 Michael: yeah, I'm sure it's possible, but also not done, right. So one of the things I thought was interesting is sort of the soft skills story here, like when you go and talk to executives and managers it's not just, you want to say like we are talking about the ecosystem, we are talking about, take it as a whole, right, and the truth is that businesses are run by people and your sort of skills to work with these people and convince them and speak their language really matters.

45:36 Mahmoud: Yeah, so what I usually tell them, and so this is all just expanding that ten myths, things are wide open and approaching them from a more positive aspects, it's not just rejecting myths, right, it's saying ok, what do you talk about instead, and you know, Python was created in 1991, it's like it had a nice long slow bake, all the way until it sort of hockey sticked through live web things, between 2003 and 2006 or so, and so as a result you have this like really well tested, well understood core reference implementation and you have a huge open source library it's used in education a lot and one that I liked the most is I always tell people it's organizationally neutral so a lot of people still haven't heard of Python, you would be surprised, people paid far more than I am, they haven't heard of it even and one of the reasons for this is that Python is organizationally neutral where ibm and oracle sell and sold Java, there is no company that is selling Python, you know, there is no joints of Python there is Continuum and Linux, they are great, there is Enthought, but these are like pretty small companies.

46:58 Michael: Your company wouldn't have like an enterprise license agreement with Python, right?

47:06 Mahmoud: There are probably some goofs to do it but no it's not like that, it's not a sole thing, instead, engineers choose it and they use it and they succeed, you know. And you can get support contracts but that's not really Python way.

47:19 Michael: The other thing that you kind of touched on in that whole section is the idea of foundational technology.

47:27 Mahmoud: This is managers especially, but basically everyone has this to a certain degree, like people get really anxious about doing things the right way. basically, everywhere you look in technology, people are telling you that things are changing ok, this changes it all, this is new and it's going to change it all, and I think we all know this but we don't really get to think about it, it's like a year later you just don't hear about it ever again. We have been set this line about software being ephemeral and tools not mattering only delivery. This whole ephemeral technology thing, I don't buy it, foundational technologies are real. So specifically, like Linux, BST and in general the posix standard, there are people who if they woke up one day and read the news and said hey Linux is over, Linux is gone, they would be like oh yeah, what did you expect, open source and these things are crazy, this is like they could actually buy that sort of reality but these things are permanent, there is like long traditions, there are long histories, we don't usually get time to read into them, but these foundational technologies are real, another would be the C programming language, and CPython is a really, I mean, Python and CPython are really natural extensions relying on the C programming language so I feel confident, you know, grouping Python with these foundational technologies it largely because C is so foundational, you know, C is not going away. You can take it even further, there are people I know, I mean, highly paid people here at my work, they will come down to me and they will be like oh are you the Python guy- you like new programming languages and stuff, you know what I think, I've heard that this visual programming thing this is going to be the next big thing and it's like they see one demo and they think that procedural programming is going to go away, their like top to bottom execution of code is in the future at some point not going to be something viable within their lifetime, within their work span. I talk about it a little bit int he course, that sort of naval gazing sort of thing, which language is the best language. Instead I try to refocus them I tell them maintenance is real, and like longevity is possible and the tools and methods matter, you have to use stuff that has been baked. The underlined experience of the course is that if you choose your architecture dependencies and practices wisely, you can rest easy, thanks to these foundational technologies.

50:01 Michael: yeah, and I think you are more likely to sort of experience or run into the deep history of them at enterprises, right like at a startup, all the code is what, a year old or whatever, besides maybe the packages but if you go to a company that's been using the technology for a long time, you could go to source control and the last check in, or the first check in you would see might be like 1995 and that's because you switched version controls and couldn't care of the history over, or something like that, right?

50:29 Mahmoud: Right, I was blown away when I think it was 2012, beginning of 2013 I was actually like curious, this was my first targeting to the security stuff, 50:41 stuff was implemented, we needed to re-implement it in Python to enable running on 64 bit PayPal primarily runs on 32 bit, 64 bit was not a thing when PayPal was written and so 80% of our revenue or something like that is still going through the quote unquote legacy stack, right, but you know, when you are actually like there re-implementing and reverse engineering Max Levchin's code like wow, software really is actually a lot more permanent than I was taught. So you know, by the internet, or whatever.

51:26 Michael: Yeah, by the person with the latest framework that's super excited about it. 

51:29 Mahmoud: Exactly, and so you have to learn to distinguish between a press release and actual like neutral academic whatever, there is press releases, and then there is real experience, real education.

51:47 Michael: Yeah, so one of the things you talked about was sort of greenfield versus brownfield, or like brand new software versus legacy software if you will, and I think you sort of convinced me that there is more value in these brown field apps and people that work on them can involve them and really add features and understand them, they deserve maybe more respect than the industry gives them.

52:11 Mahmoud: Yeah. That is definitely the case, it's just a culture thing within technology, the new is always emphasized, but the old stuff that is what we are using right now, that's what is working right now, that's what is causing money to move and disrespecting it, that doesn't actually get us any further in technology progress. We have a lot of people come in, and they try to you know, plough the whole field under and you know, it just hits one rock and then they declare victory with the small 52:49 they turned green and they move on. But, and then you end up with this patchwork environment, where like well that was green so and so that long ago, and that part over there they never reached and so you just end up with this very untended wilderness; it's interesting, it's really fun environment, you know, this is why you can see I get carried away just day dreaming about like the complexity inherent in a system as complex as PayPal's.

53:20 Michael: So, we don't have a lot of time left in the show, we are sort of getting to the end, but I wanted to maybe talk about some of the stuff farther down, like some of the best practices that you are talking about. One of them I thought was interesting was sort of pull request designer view type of experience.

53:40 Mahmoud: Right, so the way that the course is structured, you know, is I got intro and then I define the basics, I just do some definitions, and then I have this dry segment about architecture and design. And so those first three parts are like I mean especially the architecture and design part those are just like processes and soft skills that you know, you can use in enterprise world. Then, we get to like the real meat, which is like 12 segment best practices, and this is sort of like end to end adjustments to your knowledge and pointers to how to do things that should be generic for most organizations most legacy environments and how you can put python in them and create new software. So once you get past like you know, just editors, dev tools, issue tracking, how to start a python projects, and some design patterns, people always ask me about design patterns they are coming from like Java and C++ background, and so I threw a segment in there like well, design patterns in Python tend to be like so small you barely even see them, so I said, here is how to like actually look and see Python design patterns but you know, eventually, like so the first half of the best practices is build a new code and the second half is maintaining that code and building quality. And I think that the first one of those is code review, and this is one of my favorite segments because you know I got to do a little bit of like black hat/ white hat sort of role playing with myself one weekend and so one day I just wrote some really bad code that I tried to build in every antipattern that a budding developer will accidentally stumble upon and I put it up on GitHub I just like had to sleep on it, I just went to seep. And then the next day, I came back and I just torrent myself, I just I mean, well, that's not true actually, like I did a precise code review stating why there were problems where there were problems, I did my best being nice about it, promote the actual like you know positive interaction you should try to have. And so all of that is available on the GitHub repo for the class where I sort of have this example project which is useful for about a quarter of the class just as sort of a reference, and so that's github.com/mahmoud/espymetrics.

56:09 Michael: Yeah, and I'll link to that in the shownotes as well.

56:11 Mahmoud: Great. And so, there is this pull request I will never merge., because it does everything wrong way and so I have a bad implementation all the comments on it and then a good implementation and you can sort of do an apples to apples comparison there and you know, it was interesting.

56:31 Michael: Yeah, I think that's something that a lot of enterprise developers who maybe haven't spend a lot of time in open source don't get to really experience a lot to sort of GitHub, pull request style of work, right?

56:42 Mahmoud: That is true, when I started at PayPal we were using I think something you've probably never heard of we were using clear case, clear case was maybe worst than I could ever imagine, for a version control system, it was like every file- it was a file by file thing, you could get these evil twins and I just know that some people still have to use it, I run into them in the Valley and it's kind of a shame. But, yeah, so we did actually not to long ago a GitHub enterprise and I think that it's much much healthier way to develop, you have a decent branching strategy, to keep things like sane, and then you there is like the useful UI and I am sure the bitbucket has like similar stuff. The code reviews are actually exactly like sort of thing that we do at PayPal as well, and that's one of the reasons I did it because we have an internal 57:39 as part of the community, internal lists serve where people can request code reviews from other teams. And so that's one of the services we provide as the Python infrastructure team at work, is that we'll code review any Python code you give us and I just did my best to export that in an organizationally neutral way.

57:56 Michael: Yeah, so you've had a chance to look at the both, good and bad examples and you brought them to the class, yeah?

58:03 Mahmoud: Yeah, and so that is like actually one thing you just reminded me, is that a big part of the class is evaluating dependencies, I think that that's a great way learn Python, that's a massive way that I learn Python, reading Django code, reading bottle py code, you know, when I was doing all this web development actually digging a little bit deeper python ships with its source code and you can see the actual code like what your dependencies are doing, what you are relying on, and so you know, don't constantly live in the sausage factory looking at how things are made, but take your time to evaluate dependencies because it will help you avoid some really horrible architectural nightmares. At PayPal, I think we all know about the left pad thing at this point so anyways, at PayPal we have a typical Python team, it's small and we always have like ten things to do, so we have to make choices that favors stability, and to make sure that when we finish something it stays off of our plate. We are really big advocates of vendoring in libraries, so that means that we copy dependency into the project, this isn't something you'll see a lot in open source projects because there is the python packaging index, PyPi, but we view like coppying in this library, the certain version as sort of the Python equivalent of static linking. Yeah, we do that too with our C libraries, but we build projects with the dependencies built in, and this way teams can just clone the repository and go. and then when you package an rpm or dev or whatever, there is no install time dependencies, and so there are minimal side effects. Basically repeatable deploys are critically important at any environment, and whether you achieve them the rpms devs containers whatever, as long as you have those repeatable deploys you are going to be in a lot better spot.

59:51 Michael: Yeah, definitely repeatable and automated are the key element, right?

59:56 Mahmoud: Yeah, I talk about ci in here, we definitely use Jenkins and so forth, I don't know if I recommend Jenkins specifically but another team is running the Jenkins, that's the thing, we have this group that manages Jenkins for us, and so that's one more thing we don't need to do, that's one more thing of our own that we can do.

01:00:14 Michael: Yeah,t hat's awesome. All right, so I think for the course we'll just have to leave it there, we are out of time. Let me ask you questions I ask everyone- so first of all, I know I asked you this way back, like a year ago but maybe it's changed- a favorite editor?

01:00:29 Mahmoud: No it hasn't changed, Emacs, i still use Emacs primarily in the console, this is interesting because now there is going to be like Ubuntu on Windows, sort of 60:41 thing so i have actually considered getting a Windows laptop for writing Emacs natively there because we have so many windows users here at PayPal to help support them.

01:00:53 Michael: yeah, you'll be able to live the life they have to live and sort of experience it, yeah. That was just this week, or was it last week at build, so yeah, basically Ubuntu running natively on Windows.

01:01:05 Mahmoud: It's weird to be excited about windows feature, for me it's weird to be excited about windows feature. I can only get more excited if they told me that they are going to build in python, maybe they will, that would be intense

01:01:14 Michael: You know, when windows 10 was under development I was using the community preview or whatever they call it, and they had a user voice setup for you could recommend features and get votes, and I recommended that they installed Python and ship Python 3 and Python 2, as built in to windows, and I got I think over a thousand votes from the community saying yes Microsoft, build this. And sadly it went nowhere.

01:01:41 Mahmoud: Absolutely. Nowhere- maybe somebody heard it, that's one thing with enterprise, there are so many like moving parts inside of it, just because you don't see anything happen on the surface doesn't mean that people aren't hearing it inside. Yeah, it doesn't mean we are going anywhere, I can't make promises for Microsoft.

01:02:01 Michael: Yeah right, the show I just recorded before this one was Steve Dower and he works in Microsoft on doing a lot of really interesting stuff with Python and Visual Studio and Windows and so there is a chance someone will pick it up, we'll see keep putting the idea out there. All right, and favorite PyPi package, what did you find recently that's awesome?

01:02:27 Mahmoud: This is really tough, I don't think I Gevent and Greenlet we use it so much at work that like I just have to give it a shout out, Gevent and Greenlet, but you know, we have twisted users here, Twisted is a very respectable code basis, and I'll give a little shout out, I am working on a stats post that like statistics for engineers and what not, and a lot of that is motivated by a new project I am working on called Lithoxyl which is sort of next generation logging and instrumentation. It's not public yet but I know that some people on the podcast are probably early adopters and you know, it's good to get some feedback so Lithoxyl. Weird name good library.

01:03:14 Michael: Awesome. That's awesome. We are going to give away a copy of your class, right.

01:03:19 Mahmoud: Absolutely.

01:03:21 Michael: All right, so if you are listening and you want to check out the course, it is on O'Reilly all you have to do to be eligible to win is be a friend of the show so visit talkpython.fm go to the nav bar, click friends of the show enter your email address and then we will randomly choose somebody later in the week and give you a free copy of the course. If not, how do they find it?

01:03:42 Mahmoud: There are several places to find it, so one major reason I went with O'Reilly is because so many large organizations have safari and so I waited until the course propagated to all of the safari like installation, so if you have tech buzz or you can just go safari books online and type enterprise software with Python; otherwise, you can go straight up to O'Reilly.com and it's called enterprise software with Python and so you can just search it, I think it may even be Googleable. The one thing is it has a price, right, if you are someone who can't afford it just get in touch. This is something I am passionate about and I want to make sure that everyone has a chance to get into the professional Python software developer world.

01:04:30 Michael: Yeah, it's a really great course, I totally enjoyed going through it so nice work.

01:04:37 Mahmoud: Yeah, and you have a course as well?

01:04:37 Michael: I do, I just released one a month ago. Python Jumpstart By Building Ten Apps and that's been really well received, that was cool to do in a kickstater and I just announced this week, that I am working on a new one called Python For Entrepreneurs. And basically if you've ever built a business, it's only like 30% product, it's only 30% technology and there is all this other stuff like how do I accept credit cards, speaking of PayPal and that sort of thing, and how do I gather emails and send newsletters and just all how do I do deployments, how do I do ssl, all that kind of stuff, so when you think you've finished your app and what not, you are like, ok, well I am almost ready, just to release it, no you are like 30% done. So I wanted to create a class that would help people close that gap.

01:05:26 Mahmoud: Great. Yeah, no, I've already I think bought a copy and we are going to actually have a little parallel with the python community here at PayPal.

01:05:35 Michael: Awesome, thank you so much. That's great.

01:05:38 Mahmoud: Yeah, it's good stuff.

01:05:39 Michael: Cool. Well, Mahmoud, this is fun as always, your original ten myths post was super inspirational I think to a lot of people and certainly it was to me and I was really glad to feature it in one of the early shows, and this is kind of like the rounding out of that story. so everyone go check out the course, it's great.

01:05:56 Mahmoud: Yeah, thanks everyone for all the support frankly. I don't think I could have done it without at least some of you.

01:06:04 Michael: That's awesome.

01:06:06 Mahmoud: Yeah, that's great.

01:06:06 Michael: All right, see you later Mahmoud, thanks.

01:06:06 This has been another episode of Talk Python To Me.

01:06:06 Today's guest was Mahmoud Hashemi and this episode has been sponsored by SnapCI and Hired. Thank you guys for supporting the show!

01:06:06 Snap CI is modern continuous integration and delivery. Build, test, and deploy your code directly from github, all in your browser with debugging, docker, and parallelism included. Try them for free at snap.ci/talkpython

01:06:06 Hired wants to help you find your next big thing. Visit hired.com/talkpythontome to get 5 or more offers with salary and equity right up front and a special listener signing bonus of $2,000 USD.

01:06:06 Are you or a colleague trying to learn Python? Have you tried boring books and videos that just cover the topic point-by-point? Check out my onlne course Python Jumpstart by Building 10 Apps at https://training.talkpython.fm/.

01:06:06 You can find the links from the show at talkpython.fm/episodes/show/54

01:06:06 Be sure to subscribe to the show. Open your favorite podcatcher and search for Python. We should be right at the top. You can also find the iTunes and direct RSS feeds in the footer on the website.

01:06:06 Our theme music is Developers Developers Developers by Cory Smith, who goes by Smixx. You can hear the entire song on talkpython.fm.

01:06:06 This is your host, Michael Kennedy. Thanks for listening!

01:06:06 Smixx, take us out of here.