Talk Python to Me.
Episode number four, with guest Mahmoud Hashemi.
Recorded Tuesday, April 7th, 2015.
Music by Ben Thede
You may have noticed that we have new intro music for this show.
If you're a fan of the Developers, Developers, Developers music by Smix,
don't worry, he's not gone. He'll be back next time.
However, I chose this music for this show because it's actually not a performance,
but rather sounds from an auditory experience for Wikipedia called Listen to Wikipedia.
This is a project written in Python by our guest, Mahmoud.
We'll talk a little bit about that during the show.
The sounds you hear corresponds to edits, additions, and deletions to Wikipedia.
Reading off a few of the topics that cause those sounds are kind of random, but here they are.
Asphalt, Geography of Peru, Subobject, Fox, 2013 European Olympics, Pretty Little Liars, and Carrie's Grammar School.
If you need some sweet Zen sounds to program to, or are just interested in this project,
check it out at listen.hatnote.com and experience Python and Wikipedia and audio all wrapped up into one for yourself.
That's listen.hatnote.com.
Now, on to the show.
Hello, and welcome to Talk Python to Me, a weekly podcast on Python, the language, the libraries, the ecosystems, and the personalities.
This is your host, Michael Kennedy.
Follow me on Twitter, where I'm @mkennedy, and keep up with the show and listen to past episodes at talkpythontome.com.
This episode, we'll be talking to Mahmoud Hashimi about Enterprise Python and large-scale Python projects.
Mahmoud Hashimi is the lead developer of the Python infrastructure team at eBay and PayPal,
where he focuses his development and instruction energies on service frameworks, API design, and system resiliency.
Outside of work, he enjoys coding on his open-source projects, which you can find at github.com/Mahmoud,
as well as creating and maintaining several Wikipedia-based projects, such as Listen to Wikipedia and The Weeklypedia.
You got it.
Excellent.
Mahmoud, welcome to the show.
Oh, yeah.
Thank you.
It's great to be here.
Like I said before, I'm a very big fan of the prior two episodes, just listening to the MongoDB episode on my way here today.
Oh, that's fantastic.
Yeah, we just released that a few hours ago.
It was great.
Yeah, it popped up on my phone.
I listened to it right away.
All right, so it sounds like you have some amazing stuff going on with Python at PayPal.
And, you know, the way that I sort of got to know you, I just ran across this blog post that you did called 10 Myths of Enterprise Python.
And I was just, I was seriously blown away at the detail and sort of the power of your message that Python is this really fantastic, powerful, flexible, and, you know, surprisingly to some people, not to me, but some people, large-scale development platform.
Well, and, I mean, I don't know what took me so long, honestly.
Like that post, you know, we were looking for that post years ago.
And, like, when we were just starting to evangelize Python and, like, work on Python here at PayPal.
But, you know, like, it was instead, it was, like, scattered throughout the whole Internet.
The evidence was there.
I was just looking for something that summed it up in that, like, you know, very sort of, like, modern Internet fashion.
But, you know, and then I guess, like, I think I wrote the first draft of this maybe, like, a year ago and started using it in internal presentations and so forth.
But, I mean, yeah, we really did need it something like four or five years ago.
And what is that quote?
You should be the change you want to see in the world.
Exactly.
Eventually, like, you know, I just went through the whole process to get onto the corporate blog and, like, just get it out there to help, like, you know, anyone else who might be in the same sort of bind.
You know, just looking for, like, a comprehensive set of evidence to sort of quash these very persistent arguments that seem to be coming up in, you know, the sort of corporate meetings that we all know and love.
Yeah.
And what I really love about your article is it's so full of evidence.
Like, you know, every sentence, every, at least every paragraph has a couple of links to other sort of richer articles that back up whatever it is you're talking about.
That's great.
Yeah.
And, I mean, so, I mean, that's the thing.
It's, like, I don't even really feel, like, that responsible for all this.
All of the evidence was there.
It just needed to be collected under one roof and then hopefully have some sort of credible source to it.
I mean, the real, like, seller for me here at PayPal is often that Bank of America has thousands of Python developers.
You know, that's a financial institution, arguably, like, much more important and larger than PayPal.
And they, like, you know, trust Python with all sorts of, like, activities.
So, if it's good enough for them, you know, them, it should be good enough for us.
But the thing is that that's, like, not really advertised that much.
Even though they're at PyCon, like, doing recruiting, looking for people.
If you're looking for a job, maybe call it Bank of America.
So, yeah.
I mean, we'll go through them one by one here, I guess, today.
But for the most part, I guess, I came to write that blog post by way of, like, you know, originally joining, like, PayPal as a PHP developer.
Not for PayPal, mind you.
But that was just what I knew.
Like, coming into the company, I saw that there was sort of this, it was like code archaeology, if you would, right?
Like, there was some evidence of Python usage dating back to the early days of PayPal.
You said prior to HTTP or something like that, right?
And prior to Java.
Quite a way back.
Yeah, no.
Way before.
So, I mean, basically, when, like, original PayPal was all C++.
And I think still the majority of the traffic in the data center is, like, you know, going between, like, C++ services and so forth.
So, yeah.
I mean, there's a long history here going back to, like, 1998.
Pretty much when I came, like, sort of that ancient civilization was, you know, for the most part gone in 2008, 2009.
Where had it moved to?
What were people using?
You know, I mean, people went and founded YouTube and Yelp and LinkedIn and used Python to much, you know, like, more visible good than I guess probably has come from my own role here at PayPal.
But it all has to start somewhere.
Yeah, that's pretty amazing.
Like I was telling you before the show, I just talked to the Chris McDonough from the Pyramid guys.
And he said that also Pyramid is being used at Yelp.
So, a lot of things coming together in these two shows.
That's really cool.
I think YouTube is really one of the major sort of things we can put up on a pedestal and say, before you say that it can't, look at this and see if you could build that with whatever your, you know, pet project or pet technology is.
Another one is Dropbox.
Those guys are really doing interesting stuff, especially with Guido moving over there full time, which is pretty cool.
Yeah, yeah, yeah.
It's pretty neat the sort of, like, energy-inducing effects he has when he sort of, like, moves between these companies.
Like, he was previously at Google and, like, you know, worked on App Engine there and, you know, was, I guess, like, probably a very key, probably subconscious influence, I would guess, in PayPal.
Sorry, in Python becoming sort of the third, like, you know, stack there at Google.
Yeah, that's really cool.
Michael here.
Thank you so much for listening to and spreading the word about Talk Python TV.
The response to the podcast continues to be wonderful and humbling.
I have a quick comment about supporting and sponsoring the show.
I'm still looking to line up stable corporate sponsorships, but I wanted to tell you about a community-based campaign I'm launching to allow listeners to directly support the show.
We are running a Patreon campaign.
You might not have heard about Patreon, but it's kind of like Kickstarter for things like podcasts, which release frequent small deliverables rather than one-off large engineering projects.
Visit patreon.com slash mkennedy, that's p-a-t-r-e-o-n dot com slash mkennedy, and watch the video to see how you can donate as little as $1 per episode to support Talk Python to Me.
This is your chance to ensure that the Python community continues to have a strong public voice.
Consider supporting us today at patreon.com slash mkennedy, and thanks for listening.
Before we move into the myths, let me just ask you, how did PayPal and eBay sort of adopt Python so broadly?
So there was this history, and then some of those people kind of moved on to do minor things like YouTube.
And then it sounds like you guys are doing a lot of Python stuff now, right?
Oh, yeah, absolutely.
So, I mean, basically every aspect of PayPal, and it is like a large business with many different concerns.
So every aspect you can name, we basically have some contingency of like, you know, some contingent of Python in that group.
So whether it is like, you know, data analysis in the risk group, or it is like, you know, sort of, let's see, I'm trying not to give away too much here.
Yeah, don't spill any super secret things you can't talk about, but, you know.
But, yeah, but I mean, you know, admin tools, batch jobs, mid-tier services, even like, let's see, even, I mean, even some web development and web service like front-end type stuff is going on as well,
even though it's primarily shifted to other stacks at the moment.
Sure. Okay. Well, that's really interesting.
Mm-hmm.
I think that's a nice lead-in to sort of some of the myths.
So on your post, you say that you have actually over 50 Python projects running their PayPal, eBay,
and you sort of go through a list of all the different kinds of things.
You know, how many people are working on these projects, and are they all active, and what's the story there?
Well, I mean, Python is sort of, like, you know, kind of renowned for being able to let you move fast,
and it's interesting how one Python developer can sort of sweep through, like, his new team or his new organization
and, like, you know, fix a whole lot of things and get a whole lot of balls rolling.
So, I mean, the 50-plus number is really definitely immediately attributable to, like, Python's own, like, high efficiency.
Right. It's easy to start and complete 50 projects, more so than, say, C++ or something like Java.
Right. I think one developer in 2014 got something like three services launched on, like, the eBay side
using some of our new, like, cloud integration stuff, and that's just one guy.
He's a team of one.
And so...
The meetings are short.
Oh, yeah, yeah, yeah.
So, basically, I mean, when I started out, like, there were maybe...
It started with me making, like, a DL or, like, you know, sort of having an email list on a listserv,
and they...
I think maybe, like, 25 people joined up, right?
And then I think now the DL has on the order of 260 people who use Python on a somewhat regular basis.
And, you know, I mean, I can't take much...
I mean, I can't take all the credit for this because, like, a lot of it is due to the sort of, like,
burgeoning, like, OpenStack wave.
Like, eBay is one of the main contributors to OpenStack, main users of OpenStack.
PayPal has also followed suit there.
And then, also, we've done some acquisitions, you know, of, you know, companies that use Python pretty, like, heavily.
So, I mean, it sort of, like, had a lot of organic growth.
I'd say most of the growth is organic, but a lot of it is also attributable to those external projects as well.
Yeah, sure.
So, some of it came from the outside and sort of pollinated the internal folks, right?
Something to that effect.
Right.
And that's why I think it's really important that PayPal and eBay sort of participate in that, you know,
open source ecosystem so that they can continue to derive benefits from that, as we've seen, like, in the past few years here.
Yeah.
Do you guys have, like, a corporate GitHub place where you're doing things or anything like that?
Absolutely.
GitHub.com forward slash eBay.
GitHub.com forward slash PayPal.
You know, I don't personally maintain the Python SDKs by any means for these companies, but I do know the maintainers quite well.
So, I, you know, I'm willing to take some email spam if you want to email me at mahmoud at PayPal.com.
And, you know, I'll forward your messages on to the folks in charge of those.
But they also are pretty responsive on GitHub issues.
And, I mean, all in all, I think that, like, these Python developers that are in our, like, Githubs are pretty much on the ball when it comes to, like, the whole open source philosophy.
That's cool.
Can you open, like, an issue on GitHub for the team?
Yeah.
Okay.
Very cool.
So, you can participate there.
Even the Python infrastructure team, you know, which is mostly an internal team, now has, like, a repo up there.
It's a mid-tier server framework.
It can be used for Frontier as well, but we primarily use it here for mid-tier stuff.
And it is called support.
So, you just go github.com forward slash PayPal forward slash support.
And, you know, hopefully you'll see me updating the docs.
Fantastic.
Fervently.
Yes.
That sounds really cool.
Mm-hmm.
So, you want to touch on some of the myths?
Absolutely.
That's what we're here for, after all.
Yeah.
I think there's probably 50 myths if we sat down and brainstormed over enough beer and over enough time.
It's true.
I had to cut it down to these 10.
But, yeah.
I think you really did hit some interesting ones.
You know, maybe this is just my sort of myopic view of the world.
But, to me, it feels like Python is becoming quite a bit more popular in the last five years than it has been.
Yeah, absolutely.
And I think that is what is leading into what I think I had as myth number one, which was that Python is a new language.
Exactly.
People are just now starting to hear about it.
They feel like, oh, this thing's taken off.
But it's not new, is it?
Yeah, yeah, yeah.
No, I mean, it's technically older than Java.
You know, its first release happened three years before the first release of Java.
And, I mean, it comes from, like, a really long history as well.
Like, it's not just, you know, sort of.
Yeah, so it's not a, like, sort of, like, designed in a weak kind of language that just, like, appeared out of nowhere.
It's based on, like, Guido's long experience with languages like C and ABC.
Absolutely.
And I think he took, like, two years.
He started in, like, 1989, maybe, I seem to remember.
And it came out in 91, version 1.0, something like that.
Yeah, well, I was, you know, just a wee one, right?
Exactly.
Like, running around, not even thinking about, like, programming.
But, yeah, it is very mature language on top of that.
And, I mean, I don't think I go into this in the post, but it has evolved so remarkably.
Like, all of the changes after, like, version 2.2 with new style classes and all of the, like, you know, rationalizations that have occurred have really, like, proven that it still has the flexibility of a new language, right?
Where it's constantly under development.
And you can see a lot of that in, sort of, like, the interesting, sort of, like, discussions and tensions that arise, like, you know, with the Python 3 issue.
Yeah, the Python 2, Python 3 issues.
We probably have to have a whole show on that.
But, you know, it is interesting.
I actually have some opinions on that.
But, you know, I think it's interesting how much the language is still sort of evolving in a positive way.
So, for example, in, what was it, 3.3, they had added yield from to simplify generator methods, which was really cool.
In 3.4, they added async I.O.
In 3.5, there's talk of, like, type hinting.
I mean, these are major sort of additions that are coming on still with 2025-plus years out.
And many of them have a really nice realistic timeline to them.
Because, I mean, yeah, Python 3 option, it'll take some time.
You'll probably do a whole show on that, and you'll have plenty of time to get it out before Python 3 is the majority.
But, yeah.
The debate will still be raging if we have that show in six months, I'm sure.
Absolutely.
So, yeah.
But, I mean, Python is not a new language.
And probably most of the people on this show, like, or who will be listening to this show are going to know that, I think.
Absolutely.
But one of the goals I kind of have for this show is, obviously, the people that take the time to, like, sign up and listen to a Python podcast generally are knowledgeable about it pretty well already.
But, you know, maybe they work in teams where people aren't.
Like, I have a lot of conversations with folks who are in sort of the more compiled language space.
And they always think of, oh, Python is like this scripting.
It's almost like it's a Bash shell script type of thing.
And I don't think they appreciate sort of the, basically what you laid out in your 10 myths here, which is really cool.
So, speaking of a compiled language, that's myth number two, right?
Well, sure.
And so, myth number two is that Python is not compiled.
This one is a little bit of a, you know, sort of a, it's a little bit of like sort of a leading myth in a way.
People do sort of bring this up.
But the point is that, like, what they're trying to say is that Python is not like C++ and Java.
I mean, it just ends up coming up always.
As a company that uses primarily C++ and Java, like, we're going to have a lot of people who sort of protest at, like, the first time they're seeing a REPL used effectively, you know, when you're doing, like, a live demo or something like that.
So, they say, like, what do you mean there's no compilation?
What happens if I, like, you know, get something wrong?
And it's like, well, you got tests and you'll be okay.
So, like, yeah, Python's not, and the other thing, too, is that Python is technically compiled.
Like, it does get compiled down to bytecode.
It happens fast enough that it can happen sort of, like, you know, right before runtime without incurring too much of, like, you know, overhead for most sizes of projects.
Right, and it does cache that output as the PCH is, right, in the PyCache or whatever it's called in that folder.
The PYCs and the PyCache and Python 3, yes.
And so, yeah.
So, basically, it is a compiled language, and it does execute from bytecode, which is exactly what Java does.
I mean, the JVM, like, also, like, suffers from a thing called typeer, which, I'm not sure if I mentioned this in the post.
It probably did.
But, yeah, it has typeer ratio because it's originally based on a Smalltalk VM, which, and, you know, Smalltalk itself, like, you know, isn't really a, like, statically typed language.
So, basically, effectively, the bytecode that's emitted by Python is kind of on the same level as the bytecode emitted on the Java side, and the main difference between those two runtimes is the JIT, the just-and-time.
Yeah, the JIT and the GC.
That's interesting.
Yeah, the most notable thing that comes to mind when you say that is sort of the Java implementation of generics or templates, where, in the language, it looks like it's, you know, integers and other types of objects, but it's really just, you know, down to objects at the base, right?
When it runs.
The other thing I think is interesting about this is, I'm sure one of the hesitations is, what do you mean you're going to give just the source code away as, you know, bare files, right?
And, you know, Java and .NET and those things, they don't protect you much, right?
Because all you've got to do is throw it into a tool, and you see basically the same thing.
Absolutely.
I don't rely on that for security, like, whatsoever.
So, yeah, it's interesting that these other languages can be decompiled.
And so, in some sense, you're more or less giving away your source code as well.
And although there are obfuscators, if you really want to get to it, it doesn't matter so much.
I think that brings us to the real actually important matter is, is Python secure or not?
Yeah, that's the real question that people should be asking rather than wondering if they should distribute source code.
Is Python secure?
To which the answer is yes, definitely.
To the point that we're willing to, like, actually, like, put a lot of our security onto that platform.
I mean, and it's not just our own assessment.
Like, there have been, like, studies done, surveys done of, like, you know, the actual Python source code that determined that it was, like, actually very safe.
Not really prone to many of the traditional, like, you know, sort of C, the C runtime issues that can occur that we've seen perhaps in, like, openness to sell and so forth.
Right. A lot of the safety of the language, the fact that you can't work with raw bytes and pointers and offsets is part of it.
Then also, you also pointed out that the small surface area you can accomplish so much with so little that you don't have to kind of put so much out there in your code.
So it's easier to guard that smaller code base.
Right. And that's just, like, a good security policy, a general good security policy.
So, but, I mean, no, specifically, like, you know, when we were having our discussions about security, using Python with security here at PayPal, like, we did talk about the implications of using CPython.
And CPython's, like, maturity, as well as, like, the code analysis that has occurred on it, like, you know, was one of the deciding factors.
Like, basically, if you look at Java or something like that, every couple weeks there's a new Java update because it has tried to, like, you know, introduce a new security model into its runtime involving, like, protected memory and that sort of stuff.
So, I mean, maybe I should rephrase that.
Basically, it makes promises that are hard to keep.
And so they end up having, like, a lot of security releases.
You don't see those same sort of security releases nearly as often as CPython, even though it has a very wide usage.
It's on, like, not only all the servers, but also most of, like, you know, consumer computers at this point.
Right. Definitely all the Macs and Linux machines, anyway.
Exactly.
So, I mean, if it were, like, a viable, like, sort of attack, like, if it was sort of, like, a viable vulnerability, if you would, it would probably be exploited by now.
And so, like, it's, it's, and you can never, like, really prove security.
You can just sort of go with what seems to be secure.
And, like, the way that you get a good idea of that is something that has been as widely tested as Python.
Absolutely.
Yeah, you cannot prove security by one example where a person's not hacked, right?
But more and more that it sustains up over time, the more faith we should put in a system.
Absolutely.
So I think number four may be my absolute favorite, or at least the most encountered one for me, the folks that I interact with.
And that's, Python is really just a scripting language.
And this is, this is the original myth, like, you know, in my experience.
I agree.
Yeah.
So, I mean, when I was giving my first, my first Tech Talk, I remember it like it was yesterday.
After I finished, we were demoing a new application that basically controls the prices of PayPal.
It's still used now.
It's, like, before this application was released, it would take on the order of weeks to get new pricing schedules out for, like, you know, certain vendors and that sort of stuff.
And this made it so that it became just a matter of, like, hours or minutes.
And so we were demoing all of this.
And by the time we got to the end, the first question out of the first person's mouth was, so wait a second, you did this in Python, right?
And I'm like, oh, well, yes.
Like, and how do you mean?
And they're like, but Python is a scripting language, you know, so what is this?
Is this, like, CGI or what is this?
Like, how is it that you're, like, talking these complex network protocols with a scripting language?
As though I had done something very irresponsible.
That's really funny.
But, you know, four or five years later, here we are, and it's still, like, doing a bang-up job.
Yeah, that's fantastic.
Yeah.
And so in this myth, I, like, basically go through, as mentioned, like, earlier in the program here, like, you know, I go through all of the different, like, companies that have used Python in so many different ways.
It is a general purpose language as it describes itself.
So it has many, many purposes.
Yeah, I really like your list, and maybe I'll just read a couple of them.
So we've got, like, Twilio doing telephony infrastructure, obviously payments with you guys, neuroscience and psychology with tons of examples.
There's all the numerical analysis stuff with NumPy and SciPy.
Disney, DreamWorks, and LucasArts are all doing animation rendering type stuff.
Games, backends, email.
Let's see what else is here.
I think the security and penetration testing stuff is interesting.
Big data, you know, Hadoop.
And then, obviously, we just talked to the Mongo guys.
There's great support on those types of systems.
Yeah, and we're spinning up a Disco cluster here, too.
I mean, Disco's been around for, like, a little while, but it's sort of like an Erlang and Python-based big data thing.
We're spinning up a cluster here.
It's actually been a pretty interesting experience so far.
Oh, I bet that's an interesting thing to work with.
Definitely.
My favorite example of all this, right, was sort of during sort of a contentious time in the Python infrastructure team's history here.
I think Binds10 came out and announced that they were going to, like, use Python for a good part of DNS, right, which is basically as close to infrastructure as, like, common infrastructure as the Internet has.
So, I mean, I'm not sure where that's at right now.
Yeah, that's pretty cool, though.
Yeah, but it was, like, a really, like, sort of – it drove a stake through the heart of this myth that Python's just a scripting language.
Well, I think one of the other concepts that leads to people thinking that is, well, you can't have a real language without strong typing.
And so people think Python is weakly typed, right?
No, that's – so this is another one that I guess people, like, you know, in a practical meeting start getting into theoretical aspects, right?
And they start getting into type systems, which, I mean, you know, Python is definitely the language of, like, sort of pragmatic doing of things.
And so, you know, that's my way of saying get shit done.
I'm not sure what rating you have on this podcast.
But, you know, I mean, and people will bring up that it's weakly typed.
And the response to this, the real response to this should just be, like, one, it's not.
And even if it was, like, so what?
Like, the thing has already been done by the end of the meeting.
So it's really about getting, like, results.
And Python's type system, while wonderful, I'm a very big fan, like, has very little to do with, you know, it's – I don't want to rule it out.
I don't want to rule it out.
But I, like, in a moment there –
JavaScript is an example that you could actually do without real typing and still get stuff done, you know, the node guys and so on.
I don't know if I'd include that one.
But, yeah.
So, no, it's true, though.
It's – yeah, Python is, like, a strongly dynamically typed language.
And it works for strong, dynamic people.
Yeah, that's fantastic.
So another myth, you know, people think of the scripting language and they think it's just this interpreted thing.
And so, well, obviously interpreted code is slow.
And even if you just focus on CPython, I think the performance story is super nuanced and interesting and non-obvious.
So, you know, take, like, NumPy, for example.
So if you did all of your sort of mathematics in pure Python and then interpreted that, that would probably be slow.
But, of course, they've taken the slow parts and rewritten that in C.
So that's sort of native and that's super fast.
And if you can just kind of orchestrate calling into these low-level C functions, then you're talking about really fast even CPython.
But there's more to it than that, right?
There's a ton of runtimes or implementations.
Definitely.
I mean, Python's demand has led to quite a bit of supply in all sorts of different, like, aspects.
Runtimes definitely being one of them.
So, I mean, you've got CPython, which is the standard.
You've got Jython and IronPython and PyPy.
And then there are, like, you know, more sort of, like, academic ones, you know, for teaching how a runtime works and so forth.
So, but the real key here is that, like, calling it interpreted language is sort of a form of micro-labeling that doesn't actually, like, contribute to the overall, like, engineering discussion that should be happening, which is that Python can have such a huge impact on your workflow.
So, yeah, you can iterate on your projects at such great speed that you end up, like, finding yourself using more advanced techniques or finding out that this area that you would have spent so much time optimizing otherwise is actually not really where the majority of the work of your application is being done.
So, basically, it allows you to focus on what matters.
When you take that to a macro level, looking at the whole ecosystem, there are people out there who have gone through that same workflow and have generalized out libraries that you can then take advantage of all of their optimizations.
So, like, you know, instances where Python has ended up using, like, SIMs, you know, which is, like, sort of vectorized, like, you know, computation that, like, would otherwise be, like, rather difficult to, like, code by hand in C and C++, not to mention distribute.
Yeah, I mean, it's really important to look at the whole, like, process instead of just, like, individual adjectives about how, like, a given, like, single runtime works.
Like, the Python way will lead to faster and more efficient code, you know, that makes a big difference on the overall, like, complexity of your application, which can lead to, like, you know, better maintenance aspects as well.
Yeah, that's really interesting.
I've talked with some companies that are doing amazing stuff and sort of building their almost entire enterprise foundation on top of Python.
And they've got, like, the group I was speaking with, I can't, you know, NDA stuff, I can't really talk about it.
But they had 160 internal enterprise business applications, and they were creating a Python layer to be the foundation of, like, sort of unifying all that data and underlying infrastructure.
And, you know, if you could do that, then you can do some pretty amazing, amazing stuff.
That's not a slow system you decide to do that with.
Yeah.
And, I mean, with, like, honestly, enterprise is not, like, you know, sort of the domain of, like, performance being king.
We end up spending, like, more time than we should talking about it because, I guess, people want to go back to their, like, you know, college roots.
But, honestly, we can afford more machines.
We put more machines on it because, you know, we need the redundancy anyway, right?
And we end up having to, like, spend, we should spend more time talking about, like, you know, just how our developers interact with our development tools.
Because that's really usually where, like, we have a bigger bottleneck, actually, like, getting projects done on time.
If you do it in Python, you end up having extra time.
You can engineer your products for correct behavior, and you profile it.
You know, Python has a decent profile.
It's built right in.
And then you can optimize it as need be.
And, you know, we've even found spare cycles to write some of our hot loops and see.
Yeah, maybe even have some time to write some unit tests to make sure it works right when you have help.
Yeah.
No, quality is what lets us sort of, like, sleep soundly at night.
That's what's going to, like, you know, actually make for, like, you know, successful business in the long term, generally speaking.
Right.
So, that was myth number six, which was that Python is slow.
And I think you have kind of a pair of myths that talk about scaling.
And one of them is kind of, well, number seven is Python does not scale.
You have some really interesting examples on sort of performance scale there.
Well, yeah, because this one is really just so easily quashed by a counter example, right?
YouTube, I think, is, what, the second largest website on the Internet right now, right?
We talk about Dropbox, Disqus, Eventbrite, Reddit, right?
They may have, like, Twilio with its telephony, right?
And Instagram and Yelp.
And, I mean, even games, right?
Like, even online and Second Life, right?
They're actually the areas where you find the most radical, like, scaling stories.
Because unlike enterprise companies like PayPal and Bank of America, they don't just, they're not made of money, right?
Like, the game has to be fun and they run a tight margin.
And they do it for, like, the love of crafting, like, a unique sort of, like, self-contained system.
And they end up, like, you know, creating such technology marvels as stackless and eventlet and all that sort of stuff.
So, well, stackless and eventlet and, you know, Tornado, AsyncIO, all these sorts of things are in the general realm of concurrency and async processing.
And that's your myth number eight, right?
Is that Python, that it does lack good support for sort of concurrency and multithreading?
Yeah.
So, this one, I think, is one area where, like, Python probably gets the most legitimate, like, technology flack.
And that's because, I would say that's because Python sort of has a stated mission of there's only one obvious way to do something.
And in the realm of concurrency, unfortunately, that's not true.
So, CPython, like, you know, by itself is sort of a runtime environment.
When you introduce a concurrency, like, sort of library to that, you change your, like, fundamental, like, runtime behaviors.
You basically have added a layer on top of CPython's sort of native, like, main thread or whatever you want to call it.
Now, either you're working with threads or greenlets or promises or deferreds.
And that really, like, there are so many different opinions that you can have about that, like, area of computing that Python has itself, like, you know, spawned, I think, probably at this point a dozen different ways to do concurrency.
Right.
I think one of the things that people immediately jump to, some people anyway, when they hear, I need more concurrency, is let me kick off a bunch of threads.
And I think, better or worse, the whole Node.js thing that's taken, you know, was really popular sort of coming up a few years ago is showing that you can get really great concurrency with very, very few threads.
If you're willing to sort of put those threads, you know, reuse those threads when they're generally waiting on a web service call or a database call or disk I.O. or something like that, right?
Yeah, absolutely.
And, I mean, it's been really amazing how we've sort of rediscovered these techniques.
One of the guys on the Python infrastructure team here actually worked on the AIM servers, if you remember, AOL Instant Messenger.
And, yeah, I mean, you know, that was all, and they did it all with C and callbacks.
And, you know, like, they actually, like, we ended up doing a lot of the stuff that he did back in the day.
Like, we ended up redoing that sort of now.
It's been really nice having sort of a gray beard, if you will, around, because basically he sort of, like, I think back in the day he was working with SSL BIO and, like, OpenSSL BIOs.
And today we're doing sort of the same thing.
And, you know, back in the day they also had trouble with threads.
And these days we also, we still find trouble with threads.
And these aren't Python-specific troubles.
It's just that threaded programming bears a few risks.
And so in our infrastructure, anyway, we've sort of, like, taken, we've made some decisions for our developers that will allow us to pre-mitigate those risks, you know.
And one of those is to, like, basically sort of limit ourselves to a somewhat fixed number of threads.
You don't want to have, like, one thread per request, for instance, in a server.
Because that means that as your load goes up, your contention and overhead, like, sort of also start going up.
Sure.
Yeah, and even just the pure memory from just the stack space for each thread can start to become significant when you're talking tens of thousands or hundreds of thousands of threads.
It's a problem, yeah.
Right.
And so what we end up with in a lot of cases for applications that have gone with a threaded model, and these aren't, like, Python applications, they end up topping out with sort of a hard stop.
They start falling behind on their work without, like, you know, being able to respond and shed load nicely.
We can sort of chalk up all of our good server behavior to just having the time to actually, like, analyze how our application works, add additional sort of behaviors, and instrument it appropriately.
Because we're not spending all of our time wrangling with a thread per request model or, you know.
But you really do have to, I mean, going back to the original issue of concurrency support,
like, a concurrency library is more than just any other library because you end up having to adopt some aspect of its philosophy.
And there aren't any real easy answers there.
You need to look at what they are, find out what, you know, like, sort of is easily digested by you and your brain.
And then look at examples of how other people have architected their applications if you don't have, like, strong opinions of your own.
It's a learning process, and this is something I feel that maybe, like, they should spend more time on back in most schools where, unfortunately, they mostly just focus on processes and threads, which, while important, aren't the whole story.
Yeah, absolutely.
Well, maybe that's changing in the future.
A friend of mine has a really interesting saying or way of looking at the world.
He says, you know, look, when you're writing this multi-threaded concurrent code, if you try to get too tricky, you're writing code, like, right at the limit of your ability to understand what you're doing.
And debugging code is harder than writing code.
So you're writing code that you literally can't debug.
You know what I mean?
Like, you've just gone right over that barrier.
Now it's like, I've created this monster I've got to live with.
So that's pretty interesting.
And, I mean, the code certainly takes on its own, a soul of its own.
But, like, basically, when you start running it at scale and it's spread across many pools of many machines, like, it has, it ends up, like, having an almost organic nature when you take into account, like, load balancers and, like, you know, variations in the network and so forth.
You need to make time for all of those sort of emergent behaviors that are going to come up when you deploy.
Yeah, absolutely.
So myth number nine is that Python programmers are scarce.
Like, I think I mentioned in the myth, like, this is somewhat true, right?
It depends what you're comparing it to, though.
Because, but like I said before, you know, we have one developer that goes and creates three production services in one year.
Do you need a huge team of Python developers to accomplish some project?
Maybe, maybe not.
Yeah, maybe not.
Actually, almost certainly not.
Almost certainly not as many as you would need for other stacks.
And that's sort of reflected in, like, every Python team that I've seen, not just here at PayPal, but also, like, across the industries that, like, they end up generally being smaller and more effective.
And if you, like, the literature sort of backs up that these are actually not bad things to have.
People worry about people getting hit by buses, and it's nice for, you know, nice of them to be so concerned.
But, but, but, but really, like, one of the keys with Python is that you can, you can learn it very quickly.
And it has a really nice learning curve.
I mean, maybe it's, maybe it's just me, but I had a really nice time transitioning from PHP to Django, digging deeper into the standard library, reading the source code of, like, you know, just sort of, like, modules like Ether tools and collections and these sorts of things.
And learning about Python from examples that are not opaque, like they were in, like, you know, C++ and Java.
I felt that those, like, you know, applications, like, those sort of stacks were more opaque and Python was more open.
And then, like, you know, with the rise of GitHub and so forth, going and learning, like, you know, the fundamentals of web frameworks from Bottle and Django source code.
And basically, like, you sort of, it sort of has a natural learning curve to it that basically without any of, like, you know, official training or standard training, like, you know, we were able to rise to the level of, like, effective infrastructure engineers.
And I had a similar experience, you know, when I first learned C++ way back in the day.
I remember it was this mountain I climbed, you know, and learning Python, I mean, obviously, learning as a second language makes it, a next language, not the first language, makes it easier to do.
But I think even if I had learned it originally, it would have been a much more enjoyable experience.
So, and then, you know, you make some interesting points about, well, there may be not so many Python developers or however many there are, there's going to be more.
Yeah, yeah.
And, I mean, especially when, like, there are changes happening in education right now where Python is becoming, like, I think, the top teaching language.
Exactly.
That's what I was thinking of is that, you know, just the last couple years ago, it flipped, I think, from Java maybe into Python being the most taught programming language in college.
So, a little while, that will have some big effects, you know.
Yeah.
And, well, already we're seeing it.
I mean, if your company has a policy that they'll, like, for a while, PayPal had a policy where they would only hire experienced engineers, not straight out of college.
But, already we're seeing people coming out of college being hired at PayPal whose primary experience is in Python.
So, like, the effect is already there.
And, I mean, frankly, of course, I think it's a good thing.
But, especially because Python has so many language features that are so, like, you know, well-documented and well-thought-through and designed through PEPs and, like, you know, the Python enhancement processes.
So, I mean, it's actually – it's just a more open, approachable language that has so much to teach that even if you're going to end up working in other languages, I recommend studying Python unless it's going to spoil you for other languages, which has been known to occur.
If you don't want to go to work anymore because you have to go back to writing embedded C, you may want to stay away from Python.
That's right.
We like C.
We like C, too.
Anyways.
Okay.
So, I think your final myth actually had some of the most interesting actual statistics in it.
That's that Python is not for large projects, as in large number of lines of code.
Right.
So, I mean, before we talked about, like, scaling traffic, which YouTube and others, like, you know, clearly disprove that you can just, like – Python has a consistent intuitive runtime and it can scale simply.
But scaling the developer side of things is not as simple, right?
People are complex.
And so – but that said, I mean, there have been many, many, like, you know, examples of Python scaling to the enterprise level.
And, you know, here at, like, PayPal and eBay is, you know, we have a lot of small teams, but we have some larger ones, too, with, like, multiple experienced developers and a couple of junior developers and so forth.
So, on the OpenStack side especially.
So, yeah, Bank of America, like I mentioned before, has, like, you know, 5,000 Python developers.
And they just spun that out of nowhere.
Maybe that's why Python developers are scarce.
Yeah, maybe they grabbed them all up.
Yeah, you said they have over 10 million lines of Python code, which is kind of crazy.
I mean, yeah.
And, I mean, they did that, I think the – I mean, either they did that because JP Morgan was doing it or JP Morgan did it because Bank of America was doing it.
Or maybe, like, you know, it was just a coincidence.
But the financial industry has certainly seen, like, you know, a large amount of Python adoption.
And the important thing, too, is that, like, when your company starts getting big enough, heterogeneity becomes a really important aspect of your recruiting strategy.
So, like, if you really do actually need 2,000, 3,000 developers, then maybe you shouldn't be banking on all of them, you know, being from one programming discipline, right?
So, basically, with trends in education and open source dictating a lot of what talents is available, you should focus on having language agnostic protocols and, like, well-designed larger architectures that you can plug many languages' stacks into.
And that's what we have here at PayPal, where we have C++, Java, Python, Node, and there's even been some fledgling work done with, like, Scala and Go.
And, certainly, as we – like, adding – so, doing C++ was, like, hard.
That, like, built PayPal.
Then adding Java was, like, you know, still pretty hard.
When we added Python, like, you know, we actually created reference implementations and standards for a lot of these emergent protocols.
And then other stacks, like, can come and sort of, like, follow suit.
If you're looking – if you're actually someone – if there's actually someone in charge of a big project listening to this podcast right now, you know, as long as you have good talent, good architecture, you can definitely use Python as part of a large project.
We have many processes detailed in the blog post that can, like, you know, lead to good practices, best practices for a variety of environments.
Yeah, that's great.
Can you talk about, like, static code analysis with PyFlakes and other things like that?
It's a big world out there.
It's a big industry.
And Python has a long history and, like, a lot of experience it can bring to enhance basically any – like, a company of any size, if you ask me.
I mean, it's not all about, like, you know, just evangelizing and, as they say, being religious about, like, you know, a given technology.
Python is just a really handy piece that fits into a lot of applications.
Yeah, that's really fantastic.
And so people out there listening, if they're sort of having these debates at work or on their projects and, like, I'd really like to use Python, but people keep laughing at me and say it's not – you know, whatever, right?
Yeah, I really recommend checking out Mahmoud's article, 10 Minutes of Enterprise Python, which I'll link to from the show notes.
And you can share that or you can share this longer form conversation that we had.
I just wish I had a time machine so I could send it back to me a few years ago.
Yeah, I can see that.
What would you say to your former self at work?
I would give him this article.
That's great.
It really would simplify things.
But, yeah.
And also maybe I'd send some stock.
Yeah, exactly.
Of course.
Anyways.
All right, well, I think that's – you know, this has been a super interesting conversation.
And a question I'd like to ask my guests on sort of on the way out the door is so much of Python is driven by open source stuff.
And there's so much great stuff on PyPy and GitHub and so on.
Do you have, like, a favorite thing that, you know, you maybe want to call some attention to?
Favorite project?
I have many favorites.
Like, one of the ones that I remember looking at recently and being really impressed with the implementation was – it was a very small thing.
It's called, I think, NetAdder or something like that.
And it was just a library for working with IPs and IP ranges.
But, you know, as I'm wont to do, I went and I looked at the code and it was just really exquisitely implemented.
So, I mean, it's – for some reason, like, my mind always jumps to that, like, because I was like, you know, as I looked at the code, I was just like, I couldn't do a better job.
I recently saw some guy, I can't remember who, probably in your neighborhood in San Francisco, being interviewed on Bloomberg News about – they had some sort of museum art set up about how code is art and, like, the beauty of algorithms.
And, you know, maybe that sort of part of it.
That's a Glitzman.
He works with you.
Oh, he works with you?
Oh, my gosh.
That's awesome.
Yeah.
So, he was part of one of the acquisitions that we had that used a lot of Python, actually.
And so, he uses our code internally as well.
And, no, I mean, he's a super active guy, lots of great ideas.
I think he's named Benjamin.
Yeah, that sounds correct.
I just – I didn't – I was – it was literally on TV, so I didn't, you know, like, save it.
It was a weird crossover, for sure.
Let's see.
But, I mean, in terms of open source projects, I've been spending a lot of time on.
I mean, I've worked a lot on this one recently called Boltons.
Basically, like, these are sort of things that I wish were built into Python.
Like, you know, over the years, I've just sort of accumulated all of these utils, right, that I've seen, like, you know, implementations of in various, like, you know,
libraries internally, right?
They're just like, oh, like, why doesn't this exist?
They'll throw something together that sort of works for them.
But it's not as well tested and generalized.
So, this sort of, like, you know, it's just a meta toolbox.
It's like a toolbox of all these tools for, like, you know, toolboxes for working with, like, a variety of different things, caches and strings.
And some of them are designed as extensions to, like, Itertools and other built-in, like, modules.
Itertools was certainly one of, like, I mean, if you ask me what my favorite, like, Python modules were, like, the standard library ones, I have a list for sure.
Itertools is up there.
Itertools, collections.
I mean, I'm a big fan of the select module.
Like, there are just so many, like, good standard library things built right in.
That batteries included aspect of Python wasn't just responsible for drawing me to Python in the first place,
but it was, like, responsible for what I feel was almost like a postgraduate education in, like, you know, programming.
Like, it's great.
I'm getting nostalgic now.
Anyways.
So.
Getting emotional about the packages.
Okay.
So, is there any final thing you want to kind of call the attention to for the listeners or give a shout-out to or anything like that?
Well, I mean.
Like you mentioned in the beginning, we've got, like, one of the things, one of my passion projects on the side is I have a little, like, gig.
It's not a gig.
I don't get paid or anything.
It's called Hat Note.
You know, we're working on Wikipedia-based projects.
Maybe you've seen or heard, like, listen to Wikipedia.
Our new thing is, like, a newsletter that just summarizes all the work that Wikipedians, you know, working on.
And so, it's called the Weeklypedia.
You can visit it weekly, like, you know, once per week.
Weekly.hatnote.
That's, like, the thing you wear on your head and the thing you write to a loved one.
Dot com.
That's fantastic.
The other one's at listen.hatnote.com.
Is that right?
Yeah, yeah, yeah.
And so, and these are all Python-based as well, open source.
You can find the code at github.com forward slash hat note.
You know, those are fun weekend projects that I don't have to hold to.
They don't have to process money, for example.
Right, right.
Yeah, yeah.
I was, I mean, they are high-quality projects, I like to think, but they don't have to be held to the same scene as Python.
Right, the consequence of failure is lower, which makes them maybe more relaxing to work on, yeah.
You nailed it.
You got me.
I think that's a wrap.
Thank you so much, Mahmoud, for being on the show.
This has been a super interesting conversation.
My pleasure.
Yeah, and I think the view inside to some of these big companies, like, it really might give people a different perspective on Python.
So, I hope so anyway.
Thank you.
Yeah, it was great being here.
Thanks for having me, Michael.
Yeah, thanks.
Talk to you later.
All right.
Bye-bye.
This has been another episode of Talk Python to Me.
This is your host, Michael Kennedy.
I want to say thank you for listening, and let's let Wikipedia take us on out of here.
know they heard was interesting.
The End of the World