Google runs millions of lines of Python code.
The front-end servers that drive YouTube.com and YouTube's API
are primarily written in Python,
and they serve millions of requests per second.
On this episode, you'll meet Dylan Trotter,
who is working to increase the performance and concurrency
of these servers powering YouTube.
He just launched Grumpy, a Python implementation based on Go,
the highly concurrent language from Google.
This is Talk Python to Me, recorded January 12, 2017.
Welcome to Talk Python to Me, a weekly podcast on Python,
the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy.
Follow me on Twitter, where I'm @mkennedy.
Keep up with the show and listen to past episodes at talkpython.fm,
and follow the show on Twitter via at Talk Python.
This episode has been sponsored by Hired and, as a new sponsor,
pyimagesearch.com.
They're announcing a Kickstarter campaign called Deep Learning for Computer Vision with Python,
launching on Kickstarter right now.
Thank both of these companies for supporting the show by checking out what they have to offer
during their segments.
Dylan, welcome to Talk Python.
Thanks.
Nice to be here.
Yeah, I'm really excited to talk about Grumpy, actually.
Grumpy, your Python project.
It's going to be fun.
Yeah, it's been pretty exciting a couple weeks since the release.
So, yeah, I'm excited to talk about it, too.
Yeah, it's definitely gotten a lot of attention in the open source world on GitHub.
And we're going to dig into a lot of the details behind it.
But let's start with you and your story.
How did you get into programming in Python?
I started programming, I guess, when I was in high school.
I took, like, an intro programming course and kind of got the bug.
And I just kind of took it from there.
I was really into, like, programming little games and stuff like that back then.
I did not do CS in university.
I actually did physics, but I continued to work on programming in my own time a lot.
And after or during university, actually, I got a gig at a sort of summer gig at a software company.
And that gave me a leg up when I graduated, which was pretty lucky.
And so I got a job at a visual effects company doing software there.
So there was a bunch of different things, like a lot of sort of proprietary languages for the different packages.
But Python sort of came out as a front runner in terms of integration with different visual effects packages and stuff like that.
And so that's where I started to dig into Python, especially not so much on the sort of, like, effect side,
but more on the pipeline data management kind of side of things.
So there's a lot of asset management and stuff going on in visual effects studios.
And Python's great for that sort of stuff.
Yeah, that's really cool.
I did a whole episode on Python and, like, game development studios and movies and production and stuff.
I was really surprised how much Python glues all the tooling together for those folks.
Yeah, it's really deep in there.
In fact, when I was working in that area, that's when Python sort of started to come to the fore.
And so, like, Maya, which is a big, like, modeling and animation package, built Python integration around that time.
And Houdini is another one, similar use cases that integrate.
Or actually, I think from pretty early on, Houdini had Python integration.
So, yeah, it sort of became the de facto visual effects integration language.
Okay.
Yeah, yeah, very cool.
And I think that's only growing.
It seems like there's a couple areas where Python is sort of past critical mass.
It's kind of like a black hole now.
It's just sucking everything into it.
Totally.
Yeah, yeah.
And that's a good thing.
So you don't do visual effects anymore, although you kind of work in the video world these days.
Why don't you tell everybody what you're up to?
Yeah, sure.
So I'm at YouTube now.
I started there about seven years ago.
It was kind of a big shift for me.
Visual effects was a fun environment, but it was always kind of a dream to work at Google and stuff.
So I took a gig at YouTube.
And I've worked on a number of different teams there, actually.
I've worked on sort of user-facing features.
I was on the channels team for a long time, working on YouTube channels and stuff around that.
And eventually got more into the infrastructure side.
And so now I'm working on what's called, I guess, the application infrastructure group.
And our team specifically looks after the application server that serves YouTube.com and YouTube APIs and those sorts of things.
Excellent.
So when we're all watching various things on YouTube, be it cat videos or something educational, we have you to thank for keeping those servers running.
Yeah, well, me and a lot of other people.
Yep.
Yeah, yeah, I'm sure.
Well, I'll thank you individually.
So, yeah, awesome.
Yeah, it sounds like a really fun place to work.
Where's YouTube?
Where's the center of the universe for YouTube?
Is that Mountain View or somewhere else?
San Bruno actually is where the main YouTube campus is.
So there's a few different offices around the world.
But the biggest group of the biggest sort of geographical concentration is in San Bruno.
So there's a few buildings there that are YouTube.
Okay.
Yeah.
Nice.
That sounds like so much fun.
So Python actually plays a really important role at YouTube these days.
Let's talk about how it's used now and then how that kind of came to be.
Sure.
Yeah.
So Python is what is running the main application server and a lot of the application code for
the YouTube front end and for that serves like the website and the API service APIs that,
you know, service your phone and those sorts of things.
So it's sort of like the gateway for most user traffic.
Right.
And then maybe the Python code branches back into all sorts of Google services behind the
scenes that are in a variety of technologies or something like that.
Right.
Yeah.
There's a, there's a lot of different technologies and servers involved in the whole thing.
Yep.
Okay.
So YouTube wasn't initially a Google creation, right?
It was created by some other folks.
It was founded in, in 2005, I think by three guys.
One of the one of the one of the one of the one of the ones that I joined in 2009 is Chad Hurley.
I think at the time he was the president or something.
He left shortly after I joined.
But yeah, they, they built it in 2005 and it gained a lot of traction really early on.
And I guess Google took an interest at some point in 2006 and, and ended up buying YouTube in
November, 2006.
Yeah.
I'd say that was a great move for them because it's, it's such a central part of the internet.
Yeah.
I feel like it, it had YouTube.
The idea was something that a lot of people probably had the idea.
It was a thing that clearly should exist.
But when you think of the infrastructure and the bandwidth costs and just the actual act
of creating such, such a huge video network seems prohibitive.
But, you know, once it came into existence, you know, I guess Google jumped on it.
That's cool.
I actually remember thinking what a simple idea it was and, and how like, it seemed so crazy
at the time that I think the acquisition cost was like 1.6 billion or something like that.
And, and I remember reading about that and I was like, good Lord, like, you know, you
could, how, how is something so simple worth so much?
But now that number seems so quaint compared to recent, recent stuff.
So, yeah, yeah, of course, of course.
I mean, a lot of companies go through the thinking, it's easy to go through the thinking of you
could have just built that yourself.
Yeah.
Right.
I mean, Facebook bought Instagram for like an insane amount of money and that's like a team
of 12 people, right?
For whatever it was like 19 billion or something.
They, they could have easily paid 12 people to build another Instagram, but it's, it's also
also got people's interests.
It's got the users.
It's got the momentum.
And that, that's the thing I think people buy.
Absolutely.
That's what you're paying for.
Yeah.
But they didn't write it in Python at first, did they?
No, they, the first implementation I believe was PHP.
And I don't think that lasted very long.
I think it was, most of that was rewritten in Python pretty early on.
Well, before I was there.
Sure.
I suspect the way YouTube looks today with the growth of cloud computing and all the
different APIs and services is probably super different from when, what you guys got back
in 2006, right?
Yeah.
It's, I mean, I think, you know, the company has grown a lot.
The, the use cases have grown a lot.
It's just, I mean, it's kind of night and day.
It's when I, when I first joined, you know, everyone was kind of in one floor of one building.
And since then, you know, there's distributed all over the world.
And so, yeah, it's, it's, it's changed a lot.
Sure.
Wow.
Okay.
So that brings us to today, to YouTube and what this project that you, was this something
that you created this project called grumpy or where did this come from?
Yeah.
I talked about some of the challenges we were having with, you know, running Python at scale
and on, on the blog post.
And it basically, there's a few different aspects that, that affects our ability to run, you know,
that many Python servers.
The CPython runtime, well, it's, it's really great and it's highly optimized and, and it
does a lot of things really well for our use case.
It's never really been a focus for CPython as a project.
You know, we thought, you know, maybe it makes sense to rethink how the runtime is built with
a focus on concurrency and, and running large server applications.
Yeah.
And you're not, you guys are not the first people to have this idea of, well, maybe we
could replace the CPython runtime interpreter with something else.
There's like Jython, there's IronPython, there's PyPy, there's plugin JIT.
So there's a lot of stuff happening there, but nobody's gone in the direction that you went
in, right?
Yeah.
It was, it was, it was an interesting, I mean, it's, you know, in a lot of ways it's kind
of crazy.
And the thing about Go, the Go runtime that Grumpy is based on is that it is kind of designed
for very similar use cases to what we are interested in.
So Go tends to be, tends to be used for writing highly concurrent server applications with,
you know, a lot of like sort of message passing and things within the, within the application
between threads.
It seemed like kind of a good fit.
And once I started to flesh things out and to build out some of the core functionality,
some of the pieces started to fall into place and it started to look actually really compelling.
And you're like, Hey, we could actually do this.
We should stop for just a second.
I don't think we've explicitly said your project is called Grumpy, which is a replacement for
the CPython implementation with a entirely different Go implementation, right?
Yeah, that's right.
Yeah.
So, so very interesting.
I think, you know, Go, obviously it makes sense for Google to be the ones experimenting
with Go, right?
Go comes from Google, doesn't it?
It does.
Yep.
It was developed, I think, originally by, well, Rob Pike and I'm going to mix it up.
It's either Ken Thompson or no, it's, yeah, it's Ken Thompson, I believe.
Yeah.
It was, it was developed for, I guess they had observed that I get similar to, you know, the,
what the observations that we made about running Python programs for Python server programs.
They had made sort of more general observations about writing server applications and how languages
that existed didn't, didn't quite fit what our use cases.
Yeah.
Go is really quite, it's one of the newest languages out there that I would consider a mainstream
language.
It's not as mainstream as C, but it's definitely getting there and came out in 2012 in version
one sort of officially.
So it's born within this world of multi-core microservices, distributed cloud computing stuff,
right?
Yeah.
Okay.
So let's dig into the, what is grumpy?
Let's dig in a little bit.
Like how do I take, so I can take my Python code.
I can write some, presumably some web app or something in a web service, and then I can run
that on grumpy.
Like what, what does grumpy do?
How does it take my Python code and run it?
So grumpy is, takes a little bit of a different tack than CPython.
It's actually a trans compiler and a runtime, whereas you can kind of think of CPython as
a, it's like a virtual machine bytecode interpreter and runtime.
And in that sense, it's kind of like a combination of Cython and, you know, a bundling of Cython and
CPython, except that it's all in Go.
Right.
So Cython takes a flavor of Python and then compiles it to C directly.
And like C, Go is a statically typed compiled language.
And so it's no longer interpreted.
It's not even like JIT compiled like Java or .NET.
It's full on compiled, right?
That's correct.
Okay.
So the sort of runtime side of things is actually like the correspondence is like this Python
C API.
There's actually a Go grumpy API.
And so what it's compiling is code that uses that API to mutate objects, to pull out a state
and those sorts of things.
And so whereas CPython or vanilla CPython uses a bytecode interpreter to actually drive those
API calls, the Grumpy and Cython are actually generating code that drives those API calls.
Okay.
Yeah.
Very, very cool.
Now in your GitHub repo or the blog post, I don't remember where I got this, but you said
it's intended to be a near drop-in replacement for CPython 2.7.
How's that going?
How far are you towards that goal?
That's a pretty big set of APIs to cover.
Yeah.
I'm learning every day like how big Python is.
Nobody told me about this weird case I'm going to have to support.
Oh yeah, totally.
Yeah.
I mean, I've been the amount of sort of spelunking I've done in CPython internals is I did not
expect all that.
But yeah, so it's going pretty well.
The core functionality is there.
So like the basic semantics of the language in terms of attribute access and how types work
and how method dispatch works, all of that functions basically fine.
The basic types are all there.
So lists and dictionaries and things all kind of work.
Do those mostly map directly to the underlying Go structures?
Like does a list in Python map to a slice in Go and things like that?
Or do you have to do more complicated things to map it?
It's more complicated.
And the reason is that Python is so dynamic, right?
Like method dispatch is so dynamic and attribute access.
You can put attributes on just about anything.
You know, if it was just this native Go types, then you wouldn't be able to put an attribute
on a list or on a slice, right?
Right.
So it's actually, there's sort of wrapper types, basically structures that actually map very closely
to CPython's object structures.
Okay.
Yeah, I can see that because you're working with a non-dynamic language and yet it has
to support dynamic capabilities.
So you got to somehow put a shim in there for that, right?
That's right.
Okay.
I guess the biggest kind of gaps in terms of supporting or being a drop-in replacement are
the standard library still needs a lot of work.
So CPython has a lot of its standard library is actually written as C extension modules, which
Grumpy does not support.
So that's one area of significant divergence between the two words.
And we could talk about that more.
That's turned out to be sort of a big kind of beast to slay.
The nice thing is that with, you know, all those other Python runtimes out there, there's
actually, you know, you can find pure Python versions of most things.
So like PyPy, for example, implements a number of libraries that are in Python that aren't implemented
in CPython.
Right.
So you could say, start this transition or this backfilling of APIs by just moving to
pure Python implementations that then get sent through Grumpy that actually get compiled
or run on Go, right?
Yep.
That's exactly right.
And maybe do some profiling and say, well, you know, people use lists a lot.
Let's write that directly in Go or something like this, right?
You can optimize later.
Exactly.
Yep.
Okay.
Yeah.
I suspect that there's a long tail of like stuff.
This doesn't really need to be optimized that last 5%.
Whereas these are the few things that we really should focus on, right?
Yeah.
So right now, you know, I'm kind of focused on getting support for the whole, like I want
to be able to run some common libraries that are written in Python.
Some, I want some program, Python programs that are out there, like open source programs to
be able to just use Grumpy.
So like just getting it to the point where everything runs is the first step and then
you make it fast.
Okay.
Yeah, of course.
Making it work and then making it fast seems like the right order to me as well.
So you said in your blog post that there's going to be some things that Grumpy will never
support and then there's things that it doesn't support yet, but you're working towards.
Yeah.
So one of the things I mentioned already is the C extension support.
The API for CPython is a bit different than the API for Grumpy because it's, well, for one
thing it's a different language, but also the data structures are a little bit different.
The function return values and things are a little bit different.
And so there wasn't a good mapping between those APIs and it would be too constraining for,
you know, to try to make Grumpy map perfectly to the C API.
Sure.
Have you looked at the CFFI stuff that PyPy was using?
Right.
So that's, I have not looked very closely at that.
That is something that we've looked at internally for other reasons as well.
But that is an interesting way to approach the problem.
And, and potentially, you know, there are ways to bridge the two APIs that C and CFFI may
be one of those.
Yeah.
Okay.
Does go must have a C C integration option somewhere, right?
It does.
Yeah.
Okay.
And the other thing you said is not going to support is things like eval.
And again, this is like, it is possible to implement something that's a little bit hokey to support
eval or exec.
Shell out and compile.
Oh yeah, exactly.
I mean, like that's, well, I mean, it's funny you think about it and like, that's, that's actually
what Python is doing, right?
It's like, except that it's a bytecode compiler and then it's executing in a VM.
If you instead are actually doing a, you know, an actual static compilation and then executing
that.
It's not conceptually that much different, except that the tool chain that you have to use to
do the compilation and stuff is much heavier.
So it's going to be slower and it just, it kind of doesn't make a lot of sense.
I think I could see maybe supporting it for, you know, debugging use cases and things like
that.
I don't think I, I kind of want to avoid having to worry too much about like, you know,
making that performant or whatever.
Yeah, sure.
I, I, for one would, don't think I would miss it.
I think it's fine.
Yeah.
The other thing about exec and eval is there's very few cases I've ever come across in all
my years of programming Python where exec or eval was a good idea.
So actually like, I kind of think that it's an unnecessary aspect of that language.
Yeah.
That's interesting.
And you know, it is kind of keeping with go in the sense that go is very strict about
conventions and some of the best practices that it believes.
Like for example, if you have an import of a package and you're not using that package,
that's a compilation error, right?
Things like that.
Right.
Absolutely.
Yep.
Yeah.
So eval skipping eval seems like that's all right.
This portion of talk Python to me is brought to you by hired hired is the platform for
top Python developer jobs, create your profile and instantly get access to 3,500 companies
who will work to compete with you.
Take it from one of hired users who recently got a job and said, I had my first offer on
Thursday after going live on Monday and I ended up getting eight offers in total.
I've worked with recruiters in the past, but they've always been pretty hit and miss.
I tried LinkedIn, but I found hired to be the best.
I really liked knowing the salary upfront.
Privacy was also a huge seller for me.
Sounds awesome.
Doesn't it?
Well, wait until you hear about the signing bonus.
Everyone who accepts the job from hired gets a thousand dollars signing bonus.
And as talk Python listeners, it gets way sweeter.
Use the link hired.com slash talk Python to me and hired will double the signing bonus
to $2,000.
Opportunities knocking.
Visit hired.com slash talk Python to me and answer the door.
Then you said there's a set of things that you're going to support, but it doesn't yet.
What are those?
We talked a little bit about some of this stuff, but like the standard library is not there yet.
There's a subset of the standard libraries is available currently.
Can you give us like a percentage of what, how far you are down that path?
I mean, everyone listening, this, this whole project has been like, I don't get up for three
or four weeks.
So it's not like you should have implemented at all.
It's just curious, like how far you've gotten.
It's really hard to put a percentage on it.
I guess, I mean, I probably could like, you know, compare lines of code or something, but
I think that what's going to happen is you're going to get sort of a core set of libraries
that run all the other libraries and everything will just kind of fall into place.
So I think it's, it's sort of more important to count those core libraries.
And you know, that's, that's things like types and collections and operator and all those
things.
And some of those are already there.
I mean, I, I feel like, yeah, it's hard to put a number on it.
Yeah, sure.
Maybe it's one of those things where it's, it seems like you're not very far and then all
of a sudden it kind of unlocks and things go really quick.
That's the dream.
That's a good way to like, it's a, it's a optimistic view of the future.
That's right.
If you're going to clone the repo off GitHub and try things out, like you may be disappointed
that, that your favorite libraries aren't there.
There's a good chance that if you have a program that's, that's at all, you know, complex that
there are some libraries that are missing for you.
I'd say, I don't know, maybe 20% or something like that.
Okay.
Well, that's good.
And you said you also want to support all the built-ins.
That's right.
Yeah.
That's obviously a good idea.
Yeah.
Those are important.
And again, you know, there's a bunch of stuff that just hasn't a bunch of like functions
like map and, and reduce and things like that, that I haven't got around to, haven't needed
to support them yet, but they're actually pretty straightforward to implement by and large.
So, so I think we're, we're pretty far along on, on that stuff.
So how much of your focus on Grumpy is going to be to make this a project that you guys could
use for your specific use cases at Google and then make that a skeleton or base and people can
come along and add other features and contribute to the open source project to make it more broad
versus how much are you trying to make this like we're trying to re entirely replace CPython.
So I think that we, I want to see, I like, okay.
So, so I put it this way.
I'm interested in, you know, solving some of these concurrent use cases that don't have a great answer
in CPython.
That's the primary focus.
But I, again, it might be my optimism is showing again, but like, I feel like once you kind of have
some of those use cases locked down, now people start to use it for, for things you didn't expect
right away.
I know that like scientific computing is an area where Python has a really well-established
libraries and, and NumPy is, is sort of crucial to some of this stuff.
And that's got, that's C, you know, involves C extensions.
And I think in the near term, I don't see Grumpy being useful for numerical analysis.
And, you know, that's kind of compounded by Go doesn't have too many sort of inroads in that
direction either.
So, but on the other hand, you know, some of the static, the advantages of like being
statically compiled and, and type inferencing and compiling down to native operations, that is
potentially useful for, you know, scientific computing and those sorts of things.
So, so I kind of see, you know, I want to focus on our, our immediate use cases, but I have this
kind of idea that there's.
You know, more opportunities out there once that's, once things are kind of working.
Okay.
Yeah.
That, that seems like a good roadmap to me.
It makes a lot of sense.
So let's talk about the execution engine, which effectively, effectively is the execution engine
of Go versus CPython.
So CPython, the Python code gets converted to bytecode.
Those bytecode instructions are sent to like a super large force for loop switch sort of thing.
And those are interpreted and run.
How does Go work?
Go has a runtime, which is to say that there's code that is running, that's managing things
like Go routines, which are the equivalent of threads and, and Go programs and garbage collection
and things like that.
But much of what is actually happening throughout a Go program is just, is actually, you know,
low level machine instruction.
So the Go program, much like a C program is compiled down to a machine code and actually
executed natively.
Right.
That makes sense.
So you say Go has a garbage collection, which is, is awesome.
Do you know what kind it is?
Is it reference counting or is it like mark and sweep or what, what kind of garbage is it
deterministic?
How's the garbage collector work and go?
This is not my area of expertise.
Nor mine.
But it is not reference counted.
So I believe that it is a, and actually this has changed significantly.
I believe in 1.7, they significantly re retro or sort of retrofitted the garbage collector.
It mostly just around the way that garbage for particular Go routines is managed garbage that
is sort of local to particular Go routines.
And, but it's, it's sort of a traditional, otherwise it's pretty, it's kind of a traditional garbage
collector that much similar to what Java has.
But it's actually much simpler.
Java has a, a number of different algorithms it supports and a lot of tuning parameters.
Go's garbage collection is fairly, is much simpler and is targeted for the use case of,
you know, handling requests in a server application and those sorts of things.
Yeah.
It makes sense.
I suspect they highly parallelize that thinking of Go as well.
One thing you said that's nice about executing ultimately on Go is you said the deployment story
is a little bit simpler.
You know, Python, you do, when you deploy a Python program, you are actually including your like
PY files or at least your PYC files in the deployment.
And so you have to have some way to sort of package them together and ship them off to
production or wherever you're running your program.
Right.
And beyond that, also the dependencies and the runtime, right?
So you got to have all of those things.
That's right.
Which can make it really tricky.
And there's things like PY2 AMP, PY2 XE, CX freeze, the, the Bware project.
There's a lot of project trying to make that something you can ship around, but it's not simple.
Yeah, that's right.
And, you know, so I'm sure people who have run Python in production have run into, you
know, version mismatches, things like that, using the system Python version, which was,
you know, different than the one they were developing on and so on.
The nice thing about statically compiled programs in general is that you, you produce a binary
and you just, you can put that just about anywhere and it'll run.
And that's very true for Go programs.
There's few dependencies in most cases.
Most of the, the runtime is actually compiled or is actually linked into the executable.
Yeah, that's really cool.
What's the size of like a Hello World compiled output?
Do you know?
I have not looked at the size myself.
I think I saw some comment somewhere that said it was something like three megabytes.
So it's, it's pretty substantial, but you know, that, that includes a lot of overhead for
the runtime that, that wouldn't increase significantly if your program grew.
Right.
Absolutely.
Like, you know, the next 10,000 lines add 10 K or something.
Right.
Exactly.
Yeah.
I think three megs is totally fine to get a good deployment story, stability.
You run what you shipped, all those things.
Like if this was 1994, three megs would be a problem, but it's not today, right?
Yeah, that's right.
Nice.
So what sort of optimizations do you think are possible if you run Python on Go, if rather
than as an interpreted system?
This is not an area I've dug into significantly yet.
My thinking is that if you can determine that a particular, for example, a particular integer
counter in a function is only ever an integer type and it only, you know, uses integer operations
like increment or, or whatever, then there's no need to go through the whole Python method
dispatch and creating new integer objects.
Every time you increment that counter, you can actually just use a native integer and increment
using native operations.
So that's a, that's a really simple example, but not, not uncommon.
I think once you kind of broaden that to a whole program optimization, that's when things
start to get interesting because then you can think about like, well, if you know that a
function is only ever called with particular parameters or parameters of a particular type,
then you can make some assumptions and again, use native, maybe use native data types.
Sure.
What about type annotations?
And I know that's more a Python three thing, but would you be able to, or interested in having some flavor that
takes type annotations and then uses that for certain types of optimizations?
Yeah.
I thought about this and, and I'm a little ambivalent because, you know, type annotations, the way that
they are sort of used today, they're not intended to actually, you know, raise or anything if they're
not respected.
Uh, it's mostly for analysis before you ship your program to like, you know, make the linting, the
linters job easier and things like that.
Right.
And so, when, once it actually in CPython, once your type annotations, once you're actually
running your program, the type annotations basically have no effect.
And so I'm a little hesitant to say that grumpy should use these in a more, in sort of a more
strict way, because I think that might have affect programs compatibility and stuff like
that.
Yeah.
It will absolutely do that.
Wouldn't it?
Yeah.
There's some real advantages there.
If you, if you do make them strict, then you say that a type, an argument is an integer,
then yeah, it makes the optimizer's job way easier because it can, you know, it doesn't have
to do any inferencing to determine that relationship.
Obviously it would break the sort of contract with type annotations that these are just for editors
and linters and to help you, but not actually meant to affect the runtime.
That's right.
On the other hand, if, if you could make some part of code that's like really critical go,
you know, 10 or a hundred times faster by putting a type annotation that's strict, you know,
you might be willing to make that trade off.
So I have no, I don't know which way would be the right way to go either, but it's interesting
to think about.
Yeah.
I'm very curious how that sort of evolves.
Yeah.
I'm going to keep an eye on it.
That's cool.
So let's talk about when you launched.
So this should be pretty fresh in your mind, right?
Yeah.
It's not a very old project.
It's about a week, week and a day.
Yeah.
So we, well, I guess we migrated the code to GitHub in mid-December and I spent some time
over the next month kind of cleaning it or the next few weeks cleaning up the code and adding
some functionality for the build system that we were not able to use, obviously the internal
Google build system in the open source project.
So I had to build some of that out and then I guess January 4th.
Yeah.
I guess it was the 4th.
That's eight days ago just for the day of the recording.
Yeah.
We did.
We, we sort of coordinated an open source blog post with, with the actual making the GitHub
repo public and got a little bit of traction on hacker news and, and yeah, it was kind of
astonishing how great the reception was.
Yeah.
It's going like crazy.
Like when I took notes to, for this conversation, like four or five days ago, I had said there
were 5,000 stars in GitHub.
Now, maybe that was three days ago.
Now there's 6,000, almost 6,317 contributors.
That's, that's a pretty serious uptake for a project that's been out for eight days.
Yeah.
Um, I, I think the thing that kind of blew me away most was the number of pull requests that
I got.
I mean, right on day one, people were digging into the code and, you know, doing like it,
it, the code, there are tricky parts to the code and it's not necessarily obvious how you
ought to write certain features.
And people, you know, really dug in and started filling out some of this functionality that's
missing and started talking about, you know, well, how are we going to support programs or
libraries, Python, third party Python libraries out of the box and stuff like that.
So it's been great.
I've had a really good time working with some, some of these people that have been contributing.
Yeah.
I would say that's really cool.
You talked about the code a little bit, looking on GitHub, GitHub thinks it's 77% go
code, 22% Python code and a bit of a make file.
Yeah.
That's about right.
Yeah, that's about right.
And, and a lot of that Python code is actually just tests, and benchmarks and things.
So it's, it, most of it is, is go.
And, and actually, I guess the standard libraries, which most of which are copied from, from
other places like CPython, there's pretty substantial amount of Python, but that's not like, you know,
I don't think about too much about that code since we don't have to write it or maintain it.
Yeah, absolutely.
So how do you ensure compatibility in this?
Like, are you running the standard CPython test?
That's something that we're working to.
So that's sort of milestone number one.
I haven't published a roadmap document yet, but getting to the point where we can run
the unit test library is going to be a huge milestone because it means we can then run
the unit tests that are written for CPython.
That would be a huge milestone just on compatibility.
Exactly.
Before, we get there, we've been writing small tests to that, you know, demonstrate,
um, compatibility concerns and stuff like that.
And then running those in both Python and Crumpy.
Okay.
Yeah.
Very cool.
So let's talk about why you chose Go because are there three sort of officially blessed languages
at Google?
There's Python, there's Go and Java.
Is that?
And C++.
Is that the story these days?
Yeah.
Right.
Of course.
And C++.
So four.
So why did you choose Go?
Like you could have tried Jython or something, right?
Jython is something that we, we've looked into.
Jython is a really great mature product.
Um, it's our experience that it's better to start a project on Jython, than to migrate
to Jython.
There's a number of compatibility issues, not so much like the kinds of compatibility issues
like, oh, on an, on CPython, this function returns, a different type or something like
that more that there are certain constraints of running in the JVM that make certain programs
not work very well or, or those sorts of things.
So like performance issues that sort of crop up in those sorts of things.
It sounds like running on the JVM was not the best concurrency server story as it might've
been running on Go because Go is more focused on concurrency from the beginning and things
like that.
That might be more important to you guys.
Yeah.
I think that was part of it.
I mean, like lightweight Go routines are definitely a big advantage to Go.
So Java has native threads, which have large stacks.
And so it has sort of a different performance characteristics for concurrent workloads.
Um, and so you have to kind of write programs, parallel programs in a slightly different way
for Java, but also, for real time server applications, the JIT actually can be a liability.
It becomes difficult to, you know, reproduce certain kinds of, certain kinds of issues,
debug certain kinds of problems and consistent because consistency of, how requests are handled,
is really important in these kinds of applications.
And, and the JIT can make, you know, identical requests behave very differently depending on
where in the life cycle the program is.
Sure.
Yeah.
That makes a lot of sense.
Being statically typed, you get a little more predictability.
Absolutely.
Well, not, sorry, not statically typed, compiled to like machine instructions rather than digits.
Yeah.
That's right.
Yep.
Hey everyone.
Let me take just a moment and tell you about a new sponsor with a cool and timely offer.
This portion of talk Python to me is brought to you by deep learning for computer vision
with Python, a new book from pi image search.com launching on Kickstarter right now.
Have you ever wondered how Facebook can not only detect your face in an image, but also recognize
and tag you as well.
It's not magic.
Facebook uses specialized machine learning algorithms called deep learning in pi image search wants
to pull back the curtain and show you how these algorithms work.
Their new book is designed from the ground up to help you reach expert status.
Even if you've never worked with machine learning or neural networks before inside deep learning
for computer vision with Python, you'll find super practical walkthroughs, hands-on tutorials
with lots of code and a no fluff teaching style that is guaranteed to cut through all the cruft
and help you master deep learning for visual recognition.
To learn more about this book and back the Kickstarter campaign, just head to pi image search.com
slash Kickstarter.
Yeah.
So how do you run apps on, on Grumpy?
Like if I have Python code and I want to make it, make it go, how do I make it go on Grumpy?
This is sort of a hot topic right now in the, the issue tracker on GitHub because like the
build system that I have and is strictly focused on, you know, getting the internal libraries working.
And so it doesn't have good support for building a program that's outside that directory structure
or using libraries that are in your Python path or anything like that.
And so we're debating kind of how exactly it should be supported.
So right now, if you want to run a program or compile a library, you have to kind of drop
it into that directory structure and the make system will pick up on it and, and, try
to compile it into go.
But, ideally, you know, you have some kind of Python path style construct where it can find
Python code and build it in a sort of standard way.
That's something that we're working towards.
Okay, cool.
Now, if people want to contribute to Grumpy, there's like three major areas that, that make it up.
You want to talk about those three areas so they maybe can use it as a roadmap?
You can kind of think of it as the trans compiler, which is the tool called Grump C and that takes
Python code and it actually uses it's written in Python and it uses the AST module.
So it's kind of cheating.
Another milestone will be when Grump C can compile Grump C and, that takes, the Python
code and spits out some go code.
And then you're going to, the second part is the Grumpy runtime, which is kind of the parallel
of the C API.
The trans compiled go code will depend on that runtime.
So it imports the runtime and uses the, constructs and functions and things in the runtime.
And so that's another sort of component that's written strictly in go.
And that's where all the sort of data structures and things are defined.
Um, and finally there's the standard library that is a mostly written in or actually exclusively
written in Python, but also has some uses some of the Grumpy native extensions to actually
interface directly with go packages and, and functions and things.
Um, so those are, so there's sort of the three areas and there's a lot of work to do in,
in all of those different areas.
I'd say like the standard library is, is the biggest chunk of work to do at this point.
Presumably you guys chose go because of the concurrency story, right?
And if you have Python code running on go, you want to leverage that concurrency.
Do you have to use a different API?
Uh, this is Python two seven.
So you don't have things like async or wait.
How do I interact with the concurrency model of go?
Currently, the way that go routines are made available is through the threading library.
So the standard Python threading library, you create a thread and start it.
And that actually starts a go routine instead of a native thread that will work pretty seamlessly with existing code.
Um, I don't foresee huge problems there in terms of like the differences between those kinds of threads.
And again, like, you know, go has the concept of channels, which are sort of a message passing mechanism.
And whereas in Python, you have a queue, the queue data structure, and this isn't actually implemented, but I plan to implement a queue using channels.
And so you should be able to just write Python concurrent Python code like you always have.
But I think to really take advantage of sort of the concurrency model, you probably, eventually I'd, I'd like to implement the async and await Python construct.
I think that would be a huge win.
Yeah, that would be, that would be a huge win.
And it seems to me like using the threading API is much more coarse grained concurrency than go is really built for.
And while it would work, it's not, not taking full advantage.
The idea with go is you can start a go routine or starting a go routine is extremely lightweight and passing messages back and forth is the way to sort of share state rather than with sharing memory.
Or sharing objects.
So I think that programs that are written with sort of heavyweight threads in mind aren't necessarily going to be the best possible way to express that functionality.
And so, you know, long-term I could see, you know, maybe, well, actually because you can access native go constructs.
For example, you will be able to, in a grumpy program, use go channels directly.
You know, that has upsides and downsides.
It starts to diverge from the Python language and those sorts of things.
Yeah, but it's not unlike, Iron Python or Jython or those things, right?
Where you can reach down into the underlying JVM or CLR or something like that.
That's right.
Yep.
Absolutely.
Okay.
So if you're going towards async and await, what's the story on Python three?
Since I feel like the threading concurrency story is a lot better in Python three.
Yeah.
I'd love to support Python three.
The long-term goal is definitely to support it.
The reason for 2.7 is that we have a large, YouTube had a large existing Python code base and
that was a 2.7.
So that was the main reason for choosing 2.7 out of the gate, but certainly long-term,
I'd like to see all Python three supported.
Right.
Oh, that'd be, that'd be fantastic.
I'd like to see that as well.
I mean, it certainly makes sense if you're working on the YouTube team.
YouTube has a tremendously large and widely adopted deployment of Python two seven.
Like you want to, you know, work where you can have the biggest impact locally, right?
Absolutely.
Yeah.
So reading the tea leaves, does this mean that Grumpy might someday run YouTube?
I want to hedge a little bit on that.
I think there's a sort of a long road ahead before Grumpy's ready to handle the kinds of
large applications that we run on YouTube.
So I wouldn't want to speculate about the long-term outcomes there.
Sure.
Yeah.
Of course.
You know, let me just imagine, let's imagine a world where it did.
That would be, probably the first few weeks that it switched to Grumpy would be a little
bit nerve wracking, right?
Yeah.
It would definitely.
If YouTube goes down and it's your fault, that's going to be a problem.
Yeah, exactly.
I don't want to be that guy.
Exactly.
Here's the four pages we're giving you.
No, just kidding.
But it would, if, if someday that, that came to be, that would be a really cool outcome of
this project.
Yeah, absolutely.
Um, that's, that's sort of the dream.
Excellent.
Okay.
So maybe that's, that's a good place to leave it.
Let me ask you just a couple of questions before we let you out of here.
If you're going to write some code, what editor do you use?
Uh, Vim.
Vim.
All right.
Yeah.
Very cool.
And there's over 96,000 packages on PyPI these days.
And I'm sure you've come across some that are kind of unique.
You're like, Hey, have you heard about this package?
It's pretty cool.
You should check it out.
You got any, coming to mind?
You know, it's funny.
I mean, because I do a lot of my, most of my development inside Google, you know, we
kind of have a different set of tools we tend to use.
I don't have a ton of, experience with a lot of PyPI packages.
Yeah.
So it's a little bit more a dark matter.
We out here in the larger universe don't get to see a lot of the cool stuff you guys get
to use.
I'm sure it's pretty neat though.
Absolutely.
All right.
Awesome.
So how about a final call for action?
Like how can people get started grumpy?
What can they do if they, if this resonates with them, things like that?
Um, yeah.
I mean, we're, we're super interested in, in seeing where the project goes.
I, I don't have, like I said, I would like to see, where grumpy can be useful
besides just, you know, large concurrent server applications.
Community feedback around that is great.
I, people have been filing, issues asking about, you know, support for different things.
And that's been really illuminating seeing where people are thinking about where this might
be useful.
So that's huge.
Um, if, you have the time and the inclination, try it out, just clone the repo and type make
run and, and try out Python and go and, report any issues.
That's really useful to us.
And, and obviously there's a ton of work to do.
Uh, we talked about some of the different things and, you know, contributions, via PR,
pull requests on GitHub are really appreciated.
It's been kind of amazing how much people effort people have put in already.
So that's been, really exciting for us.
Yeah.
It's, it's a cool project.
And I think if we have yet another powerful, flexible runtime that has some different trade
offs that we can make for Python, that's great for everyone.
So congratulations on your project and thanks for sharing it with everyone.
Yeah.
Thanks very much, Michael.
You bet.
Talk to you later.
This has been another episode of talk Python to me.
Today's guest has been Dylan Trotter.
And this episode has been sponsored by hired and pie image search.
Thank you both for supporting the show.
Hired wants to help you find your next big thing.
Visit hired.com slash talk Python to me to get five or more offers with salary and equity
presented right up front and a special listener signing bonus of $2,000.
Struggling to get started with neural networks, deep learning and image recognition.
Pie image search.com can help with that.
To learn more about their new book, deep learning for visual recognition with Python and back the
Kickstarter campaign.
Just head to pie image search.com slash Kickstarter.
Are you or a colleague trying to learn Python?
Have you tried books and videos that just left you bored by covering topics point by point?
Well, check out my online course Python Jumpstart by building 10 apps at talkpython.fm/course
to experience a more engaging way to learn Python.
And if you're looking for something a little more advanced, try my Write Pythonic Code course
at talkpython.fm/pythonic.
Be sure to subscribe to the show.
Open your favorite podcatcher and search for Python.
We should be right at the top.
You can also find the iTunes feed at /itunes, Google Play feed at /play and direct
RSS feed at /rss on talkpython.fm.
Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.
Corey just recently started selling his tracks on iTunes.
So I recommend you check it out at talkpython.fm/music.
You can browse his tracks he has for sale on iTunes and listen to the
full length version of the theme song.
This is your host, Michael Kennedy.
Thanks so much for listening.
I really appreciate it.
Smix, let's get out of here.
Don't forget.
you