WEBVTT

00:00:00.001 --> 00:00:02.740
Have you ever played a massively multiplayer online game?

00:00:02.740 --> 00:00:09.400
My first experience with these types of games was with text-based role-playing games called MUDs back in the early 90s.

00:00:09.400 --> 00:00:11.580
Well, things have come a long way since then.

00:00:11.580 --> 00:00:19.720
Games such as EVE Online have hundreds of thousands of players exploring, trading, and battling within a universe of over 7,000 star systems.

00:00:19.720 --> 00:00:26.000
Gameplay in EVE Online consists of beautiful 3D spaceflight within a dynamic universe with many real-world players.

00:00:26.080 --> 00:00:31.060
You may have played EVE Online, as it's one of the first major MMOs released back in 2003.

00:00:31.060 --> 00:00:39.780
But did you know that Python is at the core of the game, playing a critical role in the backend infrastructure as well as a major role in the client-side game itself?

00:00:39.780 --> 00:00:45.760
On this episode, you'll meet Christian Sigibirgsen from CCP Games to dig into Python at EVE Online.

00:00:45.760 --> 00:00:50.840
This is Talk Python To Me, episode number 52, recorded March 16, 2016.

00:00:50.840 --> 00:01:18.200
Welcome to Talk Python To Me, a weekly podcast on Python.

00:01:18.200 --> 00:01:21.300
The language, the libraries, the ecosystem, and the personalities.

00:01:21.300 --> 00:01:23.400
This is your host, Michael Kennedy.

00:01:23.400 --> 00:01:25.400
Follow me on Twitter, where I'm @mkennedy.

00:01:25.400 --> 00:01:29.280
Keep up with the show and listen to past episodes at talkpython.fm.

00:01:29.280 --> 00:01:31.820
And follow the show on Twitter via at Talk Python.

00:01:31.820 --> 00:01:35.260
This episode is brought to you by Hired and SnapCI.

00:01:35.260 --> 00:01:42.040
Thank them for supporting the show on Twitter via at Hired underscore HQ and at Snap underscore CI.

00:01:42.040 --> 00:01:43.980
Kristen, welcome to the show.

00:01:43.980 --> 00:01:44.780
Thanks for having me.

00:01:44.780 --> 00:01:52.620
We're going to have a really cool conversation about games and massive scale of just online applications in general, right?

00:01:52.620 --> 00:01:52.980
Yep.

00:01:52.980 --> 00:01:53.840
I'm sure that we will.

00:01:53.840 --> 00:01:55.020
You guys are doing amazing stuff.

00:01:55.020 --> 00:01:56.920
Before we dig into that, though, what's your story?

00:01:56.920 --> 00:01:58.460
How did you get into programming in Python?

00:01:58.760 --> 00:02:01.500
I came kind of roundabout way into game programming.

00:02:01.500 --> 00:02:04.560
I actually applied for a position at CSB as a game designer.

00:02:04.560 --> 00:02:08.040
And I started here as a game designer, worked here for two years.

00:02:08.040 --> 00:02:10.620
But I've always kind of been interested in programming.

00:02:11.100 --> 00:02:13.780
So I started poking around, fixing things.

00:02:13.780 --> 00:02:21.500
And then after that, it was kind of no turning back because programmers are always in such a short demand that was so much demand.

00:02:21.500 --> 00:02:25.380
That is that I just kind of pulled all of my time over into programming.

00:02:25.380 --> 00:02:33.240
And then I kind of just kind of worked my way into being my current position, which is technical director of EVE Online.

00:02:33.600 --> 00:02:39.000
And I did that by just attacking every problem that I saw and never giving up on them.

00:02:39.000 --> 00:02:39.760
That's really cool.

00:02:39.760 --> 00:02:41.680
And you guys do mostly Python there.

00:02:41.680 --> 00:02:42.200
Is that correct?

00:02:42.200 --> 00:02:42.800
Yeah.

00:02:42.800 --> 00:02:46.680
Majority of the programmers in EVE work in Python.

00:02:46.680 --> 00:02:52.920
We do have some stuff written in C++ as well as most of our DBA SQL.

00:02:52.920 --> 00:03:00.480
But I'd say like at least 75, 80, 80 percent of our programmers spend like vast majority of the time in Python.

00:03:00.480 --> 00:03:01.200
Okay, excellent.

00:03:01.200 --> 00:03:02.620
And how did you get into that originally?

00:03:02.820 --> 00:03:05.140
Is that something you learned when you first started there?

00:03:05.140 --> 00:03:06.540
Did you know that before you started?

00:03:06.540 --> 00:03:08.500
No, I'd known about CCP.

00:03:08.500 --> 00:03:11.160
I knew they were developing in Python.

00:03:11.160 --> 00:03:17.980
But the only thing I knew about Python was that white space matters, which I thought was ridiculous at first.

00:03:17.980 --> 00:03:23.660
But now I'm thinking it's one of its greatest features because you're going to have the white space that way anyway.

00:03:23.660 --> 00:03:25.260
So you might as well have it mean something.

00:03:25.260 --> 00:03:30.080
It also kind of forces you to be consistent on it.

00:03:30.080 --> 00:03:31.580
Yeah, that's an interesting point.

00:03:31.780 --> 00:03:38.680
Like anytime you look at professional code, it's always structured in a way that resembles the way that Python is structured.

00:03:38.680 --> 00:03:48.560
You know, if you look at C++ or C# or even JavaScript when it's well written, the white space is in a format such that it could matter, but it doesn't.

00:03:48.560 --> 00:03:48.800
Right.

00:03:48.860 --> 00:03:53.220
And so I had exactly the same feeling when I learned Python the first time.

00:03:53.220 --> 00:03:56.400
I'm like, this four space thing, this is insane to me.

00:03:56.400 --> 00:03:58.800
Like, where are the curly braces and the semicolons?

00:03:58.800 --> 00:03:59.700
What is going on?

00:04:00.500 --> 00:04:03.200
And then I worked in Python for a while and I went back.

00:04:03.200 --> 00:04:07.500
I had been coming from C# and I went back to C# and I thought, wait a minute.

00:04:07.500 --> 00:04:09.160
Why am I typing all this crap?

00:04:09.160 --> 00:04:11.340
This is way less fun to type than I remember.

00:04:11.340 --> 00:04:15.940
And it's because I was spoiled by the structure of Python, which really surprises me.

00:04:16.080 --> 00:04:21.100
And you can kind of see it from the code when you read like Java code or C# code and stuff like that.

00:04:21.100 --> 00:04:28.780
When you have a lot of indentation, you also have a lot of like white space beneath it where you have closing all of those brackets, all of those curly brackets.

00:04:29.240 --> 00:04:45.080
And it's also weird because if you see somebody write Java code, for example, and it has an if statement and then two indented lines, but no curly brackets, I can guarantee you that program met both of those clauses to be within the if clause.

00:04:45.080 --> 00:04:45.740
Absolutely.

00:04:45.740 --> 00:04:47.320
There was some bug.

00:04:47.320 --> 00:04:52.300
I think it was in OS X about a year and a half ago.

00:04:52.300 --> 00:04:56.680
A really bad like SSL man in the middle bug.

00:04:56.680 --> 00:04:58.200
And it was exactly that.

00:04:58.280 --> 00:05:03.640
It was like there was an if statement and they didn't have the curly braces, but it was indented.

00:05:03.640 --> 00:05:06.600
So people thought it was in one case, but it was actually in a different one.

00:05:06.600 --> 00:05:06.840
Right?

00:05:06.840 --> 00:05:07.440
Yeah.

00:05:07.440 --> 00:05:12.200
I thought this exact thing and I thought this would never have happened in Python.

00:05:12.200 --> 00:05:13.780
How interesting.

00:05:13.780 --> 00:05:15.420
We'd have different problems, surely.

00:05:15.420 --> 00:05:19.240
But yeah, we might have other problems, but the indentation problem, not so much.

00:05:19.240 --> 00:05:20.280
Yeah.

00:05:20.280 --> 00:05:28.000
And I kind of always liked that about the Python philosophy is that what you see is kind of what you,

00:05:28.000 --> 00:05:31.660
the code, the whole idea is that the code is readable.

00:05:31.660 --> 00:05:34.620
And what you see on the screen is what actually happened.

00:05:34.620 --> 00:05:37.560
Not some like, aha, you forgot the curly brackets.

00:05:37.560 --> 00:05:39.100
I say it a lot on the show.

00:05:39.260 --> 00:05:45.160
Like you hear about things like JavaScript gotchas and, you know, like languages like C, the mistakes

00:05:45.160 --> 00:05:49.020
that people make when they're new to the language that are really bad and not obvious.

00:05:49.020 --> 00:05:52.000
And you just don't hear about that concept in Python.

00:05:52.000 --> 00:05:55.060
Like people make mistakes, but they like don't work or something.

00:05:55.060 --> 00:05:55.320
Right?

00:05:55.320 --> 00:06:01.720
The only thing I can think of, and I love showing people that are new to Python this, is the

00:06:01.720 --> 00:06:02.860
mutable keyword.

00:06:02.860 --> 00:06:04.620
I love that bit.

00:06:05.620 --> 00:06:06.100
Nice.

00:06:06.100 --> 00:06:06.540
Yeah.

00:06:06.540 --> 00:06:08.500
So let's talk a little bit about EVE Online.

00:06:08.500 --> 00:06:17.400
EVE Online is a massive multiplayer online game that has to do with sort of like space exploration,

00:06:17.400 --> 00:06:21.640
colonization, like a science fiction space future, right?

00:06:21.640 --> 00:06:22.180
Right.

00:06:22.180 --> 00:06:22.460
Yeah.

00:06:22.460 --> 00:06:24.760
It's a massively multiplayer online game.

00:06:25.220 --> 00:06:28.900
We focus a lot here on the kind of the player interaction.

00:06:28.900 --> 00:06:32.960
We care a lot about how people interact with each other.

00:06:32.960 --> 00:06:36.460
So we, and we don't try to interfere that much about it.

00:06:36.460 --> 00:06:41.280
But in the end, it's a game where you fly a spaceship, you gather resources, you, you can

00:06:41.280 --> 00:06:45.640
shoot other people, you can shoot non-player controlled ships, NPCs.

00:06:45.860 --> 00:06:49.700
The biggest selling feature is the whole dynamic market.

00:06:49.700 --> 00:06:52.160
We have like a very realistic market.

00:06:52.160 --> 00:06:52.500
Yeah.

00:06:52.500 --> 00:06:57.160
There's kind of like an economy in the game that is live and not programmed, but it's like

00:06:57.160 --> 00:06:58.160
organic in a sense.

00:06:58.160 --> 00:06:58.360
Yeah.

00:06:58.360 --> 00:06:58.800
Yeah.

00:06:58.800 --> 00:06:59.020
Yeah.

00:06:59.080 --> 00:07:02.740
And it mimics the real world economies really.

00:07:02.740 --> 00:07:05.320
And you can see the same behavior there.

00:07:05.320 --> 00:07:10.800
And we actually have like a professor in economics working just analyzing the data.

00:07:10.800 --> 00:07:15.560
We still analyze a lot of the data, like the market activity, just to know the health of

00:07:15.560 --> 00:07:16.640
the game and what's happening.

00:07:16.640 --> 00:07:17.320
Okay.

00:07:17.320 --> 00:07:18.260
That's very cool.

00:07:18.260 --> 00:07:23.160
Do you have like machine learning type stuff going on in the background or is it more procedural

00:07:23.160 --> 00:07:25.100
and you understand it and you tweak the algorithm?

00:07:25.100 --> 00:07:33.000
We had, so for stuff that NPCs like the, just we sold on the market, we had something where

00:07:33.000 --> 00:07:34.120
we controlled the supply.

00:07:34.120 --> 00:07:38.420
We just created the item and sold them so-called like NPC goods.

00:07:38.420 --> 00:07:43.180
And what we did there is that we had something where you bought off the supply.

00:07:43.180 --> 00:07:46.160
We added a new supply, but at a higher price.

00:07:46.160 --> 00:07:50.320
And if it bought a lot of the supplies, we reduced the price.

00:07:50.320 --> 00:07:54.280
So it kind of balanced itself out, which created kind of like trade routes.

00:07:54.680 --> 00:07:58.340
But that's kind of not the most interesting thing about the market.

00:07:58.340 --> 00:08:03.720
The most interesting thing about the market is the player versus player trading where players

00:08:03.720 --> 00:08:05.440
sell items to other players.

00:08:05.440 --> 00:08:07.780
We don't need any machine learning.

00:08:07.780 --> 00:08:09.420
We don't need any fancy mechanics.

00:08:09.420 --> 00:08:14.920
The whole dynamic part of it is something that the players provide themselves just by speculating

00:08:14.920 --> 00:08:16.360
on the market and stuff like that.

00:08:16.360 --> 00:08:16.640
Yeah.

00:08:16.640 --> 00:08:17.240
Very interesting.

00:08:17.240 --> 00:08:21.700
Like you just have a bunch of people interacting and that is like the artificial intelligence,

00:08:21.700 --> 00:08:23.060
although not artificial really.

00:08:23.060 --> 00:08:29.980
That is what creates all the kind of variants and dynamics and the interesting conversations.

00:08:29.980 --> 00:08:36.560
Like just this other day, somebody showed a fake screenshot of an item, a very rare item

00:08:36.560 --> 00:08:42.160
from our test server saying that they were going to be dropped in the next release, which plummeted

00:08:42.160 --> 00:08:43.480
the price on that thing.

00:08:43.580 --> 00:08:49.740
So just by the market speculation that we might be doing something influenced the price on that

00:08:49.740 --> 00:08:50.080
item.

00:08:50.080 --> 00:08:50.500
Wow.

00:08:50.500 --> 00:08:52.440
Did that person go and buy a bunch of them all of a sudden?

00:08:52.440 --> 00:08:53.920
I assume so.

00:08:53.920 --> 00:08:54.280
Yes.

00:08:54.280 --> 00:08:56.680
But people do that a lot.

00:08:56.680 --> 00:09:03.360
And they try to buy all the resources of a certain type just to create a shortage of them.

00:09:03.500 --> 00:09:08.640
And we have amazingly clever group of players that do that.

00:09:08.640 --> 00:09:15.520
Players that have like economist boards and they have IT departments and all sort of stuff.

00:09:15.520 --> 00:09:21.400
Have you had instances where people are selling items in the game, but in real life for real money?

00:09:21.520 --> 00:09:22.940
It's a constant problem, really.

00:09:22.940 --> 00:09:25.400
We don't support it, obviously.

00:09:25.400 --> 00:09:28.340
It's very tricky to stop what we work.

00:09:28.340 --> 00:09:30.620
We try to do it as much as we can.

00:09:30.620 --> 00:09:32.960
And we've had a lot of success in it as well.

00:09:32.960 --> 00:09:34.780
But people always try to sell it.

00:09:34.780 --> 00:09:35.040
Sure.

00:09:35.040 --> 00:09:37.220
I guess that would mess up your algorithm, right?

00:09:37.220 --> 00:09:42.400
Because that's basically extraordinarily high value transaction happening invisible to your

00:09:42.400 --> 00:09:43.640
economy in a sense, right?

00:09:43.640 --> 00:09:45.520
Like a black market to some degree, I guess.

00:09:45.520 --> 00:09:46.200
Yeah, yeah.

00:09:46.200 --> 00:09:47.500
But that's not the only reason.

00:09:47.500 --> 00:09:50.440
It's also we don't like this kind of it.

00:09:50.440 --> 00:09:52.920
It's not secure for our players either.

00:09:52.920 --> 00:09:58.360
We also don't want you to be able to kind of buy a hat in the game and kind of skews the whole thing.

00:09:58.360 --> 00:10:07.440
This is the philosophy of a game that was made before the last five years of in-app purchases in iOS and these types of games, right?

00:10:07.440 --> 00:10:08.820
That has such a different philosophy.

00:10:08.820 --> 00:10:09.480
Yeah.

00:10:09.480 --> 00:10:14.200
The people that are the most successful in it, they just make a game that is fun.

00:10:14.280 --> 00:10:17.040
And then they find a way to monetize that without it being.

00:10:17.040 --> 00:10:17.480
Yeah.

00:10:17.480 --> 00:10:23.200
And then, of course, there are the other games where you just literally have to pay dollars and then you win.

00:10:23.200 --> 00:10:23.700
Yeah.

00:10:23.700 --> 00:10:26.140
That is not the same type of game as the.

00:10:26.140 --> 00:10:26.500
No.

00:10:26.500 --> 00:10:28.020
As even these other ones, right?

00:10:28.020 --> 00:10:28.580
I don't know.

00:10:28.580 --> 00:10:29.780
It's impure in some way.

00:10:29.780 --> 00:10:34.160
So let's talk a little bit more about the gameplay and then we can get into the technical bits.

00:10:34.160 --> 00:10:38.300
So there's this sort of economy and this exchange of goods.

00:10:38.300 --> 00:10:42.800
And there's a lot of like you had like 7000 star systems or something.

00:10:42.940 --> 00:10:44.540
You travel around and you can harvest things.

00:10:44.540 --> 00:10:46.320
So what other games is it like?

00:10:46.320 --> 00:10:51.260
Like does it relate to like Warcraft Starcraft or other games?

00:10:51.260 --> 00:10:52.440
Like can you give some analogies?

00:10:52.440 --> 00:10:54.480
So people who maybe haven't seen it are kind of aware.

00:10:54.480 --> 00:10:56.500
It's like a massive multiplayer online.

00:10:56.660 --> 00:11:04.100
So obviously you're going to draw some comparison with games like World of Warcraft and all the games online and those sort of games.

00:11:04.100 --> 00:11:06.920
Those are all kind of avatar based games.

00:11:06.920 --> 00:11:16.500
We like to think of them as more of a kind of a theme park kind of game where you go into the game and you have a bunch of rides where Eve is more of a sandbox kind of game.

00:11:16.720 --> 00:11:22.600
You can imagine you go to a sandbox and you have a shovel and a bucket and you're just going to have to make some fun out of it.

00:11:22.600 --> 00:11:29.400
Both genres are a lot of fun but kind of maybe cater to different needs and possibly just at different times.

00:11:29.400 --> 00:11:35.120
Yeah it seems to me like Eve is a game that really appeals to people that are like there for the long term.

00:11:35.120 --> 00:11:39.380
Like you're going to build something up over time and sort of grow it and so on.

00:11:39.380 --> 00:11:42.880
Yeah and we do have very loyal subscribers.

00:11:42.880 --> 00:11:43.680
They've been with us.

00:11:43.680 --> 00:11:45.260
Some have been with us for years.

00:11:45.260 --> 00:11:54.680
And even some people that have since then quit playing Eve still show up at the Eve Fan Fest, the annual kind of gathering here in Iceland that we have.

00:11:55.000 --> 00:11:58.600
Just because they're part of the community and like staying part of the community.

00:11:58.600 --> 00:12:03.260
That's really cool that people are like so attached to the community there.

00:12:03.260 --> 00:12:05.000
I mean how many players do you guys have?

00:12:05.000 --> 00:12:06.880
Give us some stats if you can.

00:12:06.880 --> 00:12:10.460
The peak is I think around 30 or 40,000 now.

00:12:10.460 --> 00:12:10.660
Wow.

00:12:10.660 --> 00:12:15.500
We were in the stats of like 300,000, 400,000 users I think.

00:12:15.500 --> 00:12:16.300
Must I know.

00:12:16.300 --> 00:12:16.680
Okay.

00:12:16.680 --> 00:12:19.700
When you say massively multiplayer that's pretty massive.

00:12:19.700 --> 00:12:20.720
Yeah it is.

00:12:20.720 --> 00:12:22.780
I mean there are bigger games there.

00:12:22.900 --> 00:12:29.220
I think at the time we ranked somewhere between a small multiplayer, massive multiplayer and a large one.

00:12:29.220 --> 00:12:29.880
Yeah yeah sure.

00:12:29.880 --> 00:12:31.500
Let's talk about Python.

00:12:31.500 --> 00:12:34.380
Where does Python fit in the story of Eve?

00:12:34.380 --> 00:12:42.040
When they started developing Eve they always wanted like a scripting language to go along with something a bit lower level like C++.

00:12:42.040 --> 00:12:45.560
I think they just it was obvious at the time to choose C++.

00:12:45.940 --> 00:12:48.560
But that was like back in 97 or something.

00:12:48.560 --> 00:12:51.120
It's pre-2000.

00:12:51.120 --> 00:12:53.420
There weren't a whole lot of options.

00:12:53.420 --> 00:13:00.420
They were looking at JavaScript but they eventually found out about Python and started looking into stackless Python.

00:13:00.420 --> 00:13:05.460
And they really liked the idea of the whole micro threads they had in stackless.

00:13:05.680 --> 00:13:11.300
Yeah and when you got started this was like probably Python 2 maybe before that right?

00:13:11.300 --> 00:13:13.980
I think it was Python 1.6.

00:13:13.980 --> 00:13:16.420
Yeah yeah because I think Python 2 was the year 2000.

00:13:16.420 --> 00:13:16.980
Is that right?

00:13:16.980 --> 00:13:17.440
Yeah.

00:13:17.440 --> 00:13:17.780
I think it is.

00:13:17.780 --> 00:13:25.520
So when you actually look at a lot of the decisions that were made at CCP with Python you kind of have to color them in that light.

00:13:25.660 --> 00:13:30.240
You have to look at the code and see okay so when was this written and what was available at the time.

00:13:30.240 --> 00:13:31.080
Right.

00:13:31.080 --> 00:13:37.760
The trade-offs you made have to consider like well should we use VB or this.

00:13:37.760 --> 00:13:40.680
I mean like these are the other technologies that are around at the time right?

00:13:40.680 --> 00:13:41.240
Like JavaScript.

00:13:41.240 --> 00:13:41.800
Yeah.

00:13:41.800 --> 00:13:44.600
There was no concept of this Node.js thing.

00:13:44.600 --> 00:13:45.800
It was kind of a toy right?

00:13:45.800 --> 00:13:47.780
So Python is a really good choice I think.

00:13:47.780 --> 00:13:52.460
Yeah but it was also I think a pretty daring choice because Python was pretty new at the time.

00:13:52.460 --> 00:13:58.980
It wasn't really that stable I would say and it didn't have like all the features that I think they would have needed.

00:13:58.980 --> 00:14:02.320
It didn't have as good IDEs and stuff like that.

00:14:02.320 --> 00:14:07.080
And I think it was daring but then again there wasn't like a whole lot of other options.

00:14:07.080 --> 00:14:10.640
Not for doing like a lot of work in a scripting.

00:14:10.640 --> 00:14:10.860
Right.

00:14:10.860 --> 00:14:15.500
There are actually quite a few Python implementations these days.

00:14:15.920 --> 00:14:23.140
And the show I just had last week was with Brett Cannon from Microsoft talking about Pigeon.

00:14:23.140 --> 00:14:31.340
And we talked about things like PyPy and Pyston and Pigeon and all these different choices for alternate implementations.

00:14:31.340 --> 00:14:34.700
And one I don't think we really spoke about was Stackless.

00:14:34.700 --> 00:14:39.900
But Stackless was one of the few alternative options you had back in that time frame right?

00:14:40.100 --> 00:14:43.540
Yeah yeah there wasn't a lot of other implementations I think.

00:14:43.540 --> 00:14:48.380
Then there was just the Vanilla Python and Stackless maybe a few others but yeah.

00:14:48.380 --> 00:14:54.340
Yeah not many like things like Jython and Iron Python are like circa 2006, 2008 and then onward right?

00:14:54.340 --> 00:14:55.120
So yeah.

00:14:55.120 --> 00:14:57.180
Why don't you tell everyone what Stackless Python is?

00:14:57.180 --> 00:14:57.940
Like what is it?

00:14:57.940 --> 00:14:58.920
What's the advantage of it?

00:14:58.920 --> 00:15:02.680
Why did you guys choose this over just CPython 1.9 or whatever it was?

00:15:02.680 --> 00:15:09.540
The biggest advantage of Stackless Python that people saw here was the kind of the existence of micro threads.

00:15:09.540 --> 00:15:14.840
Just very lightweight worker threads that you can even start up and they could do one thing.

00:15:14.840 --> 00:15:20.040
The idea being that you could be cold would be a lot easier to read basically.

00:15:20.040 --> 00:15:24.160
You could just start up like this one task and it just does this one thing.

00:15:24.160 --> 00:15:27.660
Yeah well if you're dealing with 40,000 concurrent connections.

00:15:27.660 --> 00:15:28.180
Yeah.

00:15:28.420 --> 00:15:35.700
A single threaded you know sort of maybe forked off the 10 or 20 process type of system may well not scale the way you need it right?

00:15:35.700 --> 00:15:38.540
So something like this is really helpful yeah?

00:15:38.540 --> 00:15:43.060
Yeah that was the biggest reason for choosing this over Vanilla Python I believe.

00:15:43.060 --> 00:15:44.040
Are you guys still using it?

00:15:44.040 --> 00:15:46.100
Yeah we still use Stackless.

00:15:46.100 --> 00:15:51.880
We kind of wish that the rest of the world would have followed us into Stackless but sadly that didn't happen.

00:15:51.880 --> 00:16:00.720
And now we have some alternatives in Python so we still are kind of tied to Stackless Python in terms of code.

00:16:00.720 --> 00:16:03.320
Yeah it would probably take a pretty major rewrite yeah?

00:16:03.320 --> 00:16:08.920
I'm not really sure how major it would be but it would be pretty substantial by the look of it.

00:16:08.920 --> 00:16:09.300
Right.

00:16:09.300 --> 00:16:14.120
So maybe let's talk a little bit about the architecture infrastructure type of bits.

00:16:14.120 --> 00:16:19.500
Could you maybe talk about like the different systems and stuff that's going on in the back end?

00:16:19.500 --> 00:16:27.900
I'm assuming that the front end the thing that runs on my OS X machine or Windows that is the graphics and the actual client side game.

00:16:27.900 --> 00:16:29.060
It's mostly C++.

00:16:29.060 --> 00:16:30.360
Is that a fair assumption?

00:16:30.360 --> 00:16:30.760
Right.

00:16:30.760 --> 00:16:37.720
Most of the most of the highest high performing stuff is C++ like the graphics engine is C++.

00:16:37.720 --> 00:16:39.440
The physics engine is C++.

00:16:39.440 --> 00:16:41.480
The DP layer is C++.

00:16:41.480 --> 00:16:51.900
Then we kind of have a nebulous layer that handles like files and various things, clocks and stuff like that called blue which is like our own dimension.

00:16:51.900 --> 00:16:56.540
We have various third party libraries written in C++ as well.

00:16:56.540 --> 00:16:57.380
Yeah.

00:16:57.380 --> 00:16:59.500
So most of that is done in C++.

00:16:59.500 --> 00:17:03.940
Most of the stuff that we for someone saw that would be a problem.

00:17:04.480 --> 00:17:07.000
We also have like a minor things.

00:17:07.000 --> 00:17:21.320
There's a feature called planetary interaction where we kind of have a layer over the planets that shows like resource distributions which is implementing the C++ because we thought it would be very performant.

00:17:21.320 --> 00:17:25.280
And which would have had a performance problem I think in Python.

00:17:25.280 --> 00:17:27.280
But we wrote that in C++.

00:17:27.420 --> 00:17:32.380
So we have some options if we want to do something to perform really really fast we can write in C++.

00:17:32.380 --> 00:17:32.960
Sure.

00:17:32.960 --> 00:17:35.100
And do you have anything on the client side that's Python?

00:17:35.100 --> 00:17:36.500
Or is that all backend stuff?

00:17:36.500 --> 00:17:36.700
Yeah.

00:17:36.700 --> 00:17:36.880
Yeah.

00:17:36.880 --> 00:17:37.060
Yeah.

00:17:37.060 --> 00:17:40.220
The most of the UI implementation is in Python.

00:17:40.220 --> 00:17:47.760
Like it falls on to like C++ some rendering but for the most part UI programmers write stuff in Python.

00:17:47.760 --> 00:17:48.920
Oh that's really interesting.

00:17:48.920 --> 00:17:49.020
Yeah.

00:17:49.020 --> 00:18:02.520
This episode is brought to you by Hired.

00:18:02.520 --> 00:18:08.120
Hired is a two-sided curated marketplace that connects the world's knowledge workers to the best opportunities.

00:18:08.120 --> 00:18:15.620
Each offer you receive has salary and equity presented right up front and you can view the offers to accept or reject them before you even talk to the company.

00:18:16.420 --> 00:18:20.760
Typically candidates receive five or more offers within the first week and there are no obligations.

00:18:20.760 --> 00:18:21.340
Ever.

00:18:21.340 --> 00:18:22.940
Sounds awesome doesn't it?

00:18:22.940 --> 00:18:24.580
Well did I mention the signing bonus?

00:18:24.580 --> 00:18:27.980
Everyone who accepts a job from Hired gets a $1,000 signing bonus.

00:18:27.980 --> 00:18:30.760
And as Talk Python listeners it gets way sweeter.

00:18:30.760 --> 00:18:36.480
Use the link Hired.com slash Talk Python To Me and Hired will double the signing bonus to $2,000.

00:18:36.480 --> 00:18:38.620
Opportunities knocking.

00:18:38.620 --> 00:18:42.080
Visit Hired.com slash Talk Python To Me and answer the call.

00:18:42.080 --> 00:18:52.280
When I thought about how you guys were using Python I of course thought all right the back end is probably a lot of Python and things like that.

00:18:52.280 --> 00:18:54.760
But like I said the front end mostly C++.

00:18:54.760 --> 00:18:56.040
Can you elaborate on that?

00:18:56.040 --> 00:19:05.880
Like are they using a custom sort of plug into your own rendering engine or are they using some kind of widget windowing system or maybe talk about it a bit?

00:19:05.880 --> 00:19:20.280
This is all kind of custom made here at C++ so the low level objects like the drawing of the stuff is in C++ and then we can actually just and I think at least it used to be the case that you would just kind of inherit from them.

00:19:20.280 --> 00:19:23.500
The Python object could inherit from the C objects and you would.

00:19:23.760 --> 00:19:27.820
I can't remember what we were using for it whether it was boost or.

00:19:27.820 --> 00:19:28.140
Sure.

00:19:28.140 --> 00:19:29.580
That sounds really really cool.

00:19:29.580 --> 00:19:40.300
How about on the server side like what type of interaction does the game from the client side actually have with the server and what does a request sort of look like from sort of systems and infrastructure and so on?

00:19:40.300 --> 00:19:40.540
Right.

00:19:40.540 --> 00:19:53.020
We have our own kind of a networking layer which we call Matsunet and it kind of handles the server and client communication as well as communication between nodes on the cluster.

00:19:53.700 --> 00:19:58.880
Both from the proxies to the sole nodes as we call them which are really just the worker nodes where everything happens.

00:19:58.880 --> 00:20:07.020
The proxies sit out in front and receive requests from the client as well as the communication between nodes as well.

00:20:07.020 --> 00:20:09.140
This is all done in Python actually.

00:20:09.140 --> 00:20:09.560
Okay.

00:20:09.560 --> 00:20:09.960
Yeah.

00:20:09.960 --> 00:20:10.620
No, no.

00:20:10.620 --> 00:20:12.200
It was rewritten some time ago.

00:20:12.200 --> 00:20:13.180
C++.

00:20:13.180 --> 00:20:16.860
But with the Python layer on top so you can know this.

00:20:16.860 --> 00:20:17.200
Right.

00:20:17.200 --> 00:20:17.480
Right.

00:20:17.480 --> 00:20:17.960
Very cool.

00:20:17.960 --> 00:20:22.940
And is this in your own set of data centers or are you hosted in like some kind of cloud place?

00:20:23.160 --> 00:20:24.320
Like AWS or something?

00:20:24.320 --> 00:20:24.880
No, no.

00:20:24.880 --> 00:20:28.400
It's in our own data center and we actually moved data centers recently.

00:20:28.400 --> 00:20:30.920
There's a big blog on that on the website.

00:20:30.920 --> 00:20:34.240
It might be interesting for attack might be able to read that.

00:20:34.240 --> 00:20:35.960
It was actually quite an operation.

00:20:35.960 --> 00:20:42.720
We had to both move data centers and we had to buy whole new hardware in it.

00:20:42.720 --> 00:20:45.320
And we were just moving between like London.

00:20:45.840 --> 00:20:49.640
And then I heard you say you were using SQL server somewhere around there.

00:20:49.640 --> 00:20:50.040
Is that right?

00:20:50.040 --> 00:20:50.260
Yeah.

00:20:50.260 --> 00:20:51.160
That's correct.

00:20:51.160 --> 00:20:51.800
Okay, cool.

00:20:51.800 --> 00:20:53.520
Are you hosted on Windows or on Linux?

00:20:53.520 --> 00:20:54.660
We're hosted on Windows.

00:20:54.660 --> 00:20:59.360
We're probably one of the rare instances where we actually use Python and Windows, I guess.

00:20:59.360 --> 00:20:59.740
Yeah.

00:20:59.740 --> 00:20:59.940
Yeah.

00:20:59.940 --> 00:21:00.400
Very cool.

00:21:00.400 --> 00:21:00.960
Okay.

00:21:01.400 --> 00:21:06.300
The world has changed since 1997 in terms of databases pretty dramatically, especially

00:21:06.300 --> 00:21:07.780
in the last five years.

00:21:07.780 --> 00:21:12.200
I'm sure part of the reason that you guys are on a relational database has to do with the

00:21:12.200 --> 00:21:14.040
fact that you started on a relational database.

00:21:14.040 --> 00:21:18.560
But do you think if you could start totally from scratch, would it make more sense to be

00:21:18.560 --> 00:21:22.820
on like a NoSQL database or stick with a RDBMS like SQL server?

00:21:23.340 --> 00:21:29.980
I think if I were forced to choose like a single database, I would probably still go with a

00:21:29.980 --> 00:21:30.760
relational database.

00:21:30.760 --> 00:21:37.920
We've made so many changes recently in terms of this microservices architecture that I wouldn't

00:21:37.920 --> 00:21:40.080
really see a necessarily need to do that.

00:21:40.080 --> 00:21:46.600
I would try to split up the clusters a bit more into these functional areas than when I'm

00:21:46.600 --> 00:21:49.140
storing data that needs a relational database.

00:21:49.140 --> 00:21:55.080
I would store it in a relational database, otherwise a NoSQL database or a graph database

00:21:55.080 --> 00:21:55.780
or whatever.

00:21:55.780 --> 00:22:01.600
But I think if I had like a 1db catch all, I would still go with relational database.

00:22:01.600 --> 00:22:02.420
Makes a lot of sense.

00:22:02.420 --> 00:22:06.800
The world is definitely moving towards this concept of microservices.

00:22:06.800 --> 00:22:11.000
And I don't think I've spoken very much about that on the show.

00:22:11.000 --> 00:22:17.080
I think maybe a little bit when we had the Docker guys on way back in show nine, but also

00:22:17.080 --> 00:22:22.540
had a conversation with Sam Newman that actually the audio went so bad we couldn't put it together

00:22:22.540 --> 00:22:24.240
specifically about microservices.

00:22:24.240 --> 00:22:25.580
I'm sorry about that, Sam.

00:22:25.580 --> 00:22:29.240
But maybe you could just tell us really quickly for those people are like microservices.

00:22:29.240 --> 00:22:30.400
Those sound like small services.

00:22:30.400 --> 00:22:30.920
What's that?

00:22:30.920 --> 00:22:32.520
Maybe describe that a bit.

00:22:32.520 --> 00:22:35.540
It's essentially that it's small services.

00:22:35.540 --> 00:22:43.140
It's this idea that you spin off a service for a specific need that your total ecosystem needs.

00:22:43.140 --> 00:22:46.520
So in terms of EVE, then we, for example, have a market.

00:22:46.520 --> 00:22:48.780
Maybe the market could be its own microservice.

00:22:48.780 --> 00:22:51.320
Maybe it doesn't have to need to connect to the cluster.

00:22:51.320 --> 00:22:55.360
The cluster would only need to know how to talk to that microservice and that Microsoft

00:22:55.360 --> 00:22:57.260
service to talk to that cluster.

00:22:57.260 --> 00:23:02.840
I don't think I would really, really do it justice, but it's really just a matter of that,

00:23:02.840 --> 00:23:07.980
of breaking up your kind of the total application into like multiple smaller services.

00:23:07.980 --> 00:23:12.500
Yeah, the thing that I think that's interesting about that is, you know, I ask you about what

00:23:12.500 --> 00:23:13.660
is the right database?

00:23:13.660 --> 00:23:20.060
And the answer may well be, well, certain types of architectures that are proving to be powerful,

00:23:20.060 --> 00:23:23.600
like these microservices, the answer could be many small ones.

00:23:23.600 --> 00:23:28.700
And then it even matters less because when you're, if you're, let's say, just modeling only the

00:23:28.700 --> 00:23:33.260
economy, which I'm sure is still massive, but just the economy, not every possible thing

00:23:33.260 --> 00:23:37.960
that could go on in the game, then you might have smaller database performance needs and

00:23:37.960 --> 00:23:38.460
so on, right?

00:23:38.460 --> 00:23:38.940
Yeah.

00:23:38.940 --> 00:23:45.200
And in some cases of data, you actually need everything to be always up to date, but maybe

00:23:45.200 --> 00:23:48.420
sometimes you just need like eventual consistency on that data.

00:23:48.420 --> 00:23:55.540
So you can pick database solutions based on the need for that particular feature rather than

00:23:55.540 --> 00:23:59.520
choosing it for everything and hoping it all matches.

00:24:00.100 --> 00:24:05.480
Having said that, I don't suggest that everybody makes changes to their architecture from like

00:24:05.480 --> 00:24:08.420
single monolithic application to a microservices.

00:24:08.420 --> 00:24:13.560
It's something you, you exchange one problem for a set of other problems, but it can make

00:24:13.560 --> 00:24:14.320
a lot of sense.

00:24:14.320 --> 00:24:20.760
And one of the things that I love about it is that it allows you to change, you know, decisions

00:24:20.760 --> 00:24:25.060
that you thought you were kind of rooted into that you thought you would never, ever going

00:24:25.060 --> 00:24:25.500
to change.

00:24:25.940 --> 00:24:27.040
Yeah, that's, that's interesting.

00:24:27.040 --> 00:24:28.280
And it's good advice, right?

00:24:28.280 --> 00:24:34.020
Like design patterns and these larger architectural patterns are super powerful, but they almost

00:24:34.020 --> 00:24:36.420
always have a context that goes along with them.

00:24:36.420 --> 00:24:42.540
If you have a billion lines of code and this level of huge complexity, you can make that complexity

00:24:42.540 --> 00:24:44.060
more manageable by breaking it apart.

00:24:44.060 --> 00:24:48.480
But if you've got like 5,000 lines of code, maybe a microservice just means more servers and

00:24:48.480 --> 00:24:50.040
things to maintain, right?

00:24:50.260 --> 00:24:50.380
Right.

00:24:50.380 --> 00:24:51.280
It's more service.

00:24:51.280 --> 00:24:52.580
It's more things to maintain.

00:24:52.580 --> 00:24:58.620
It's in theory that those servers should require less maintenance as well.

00:24:58.620 --> 00:25:00.820
And they're easier to comprehend.

00:25:00.820 --> 00:25:02.260
But you also need like infrastructure.

00:25:02.260 --> 00:25:04.520
Maybe you need to have a message queue in there.

00:25:04.520 --> 00:25:09.660
Maybe you need to add in a bunch of technology that you wouldn't have needed if you had a monolithic

00:25:09.660 --> 00:25:10.340
application.

00:25:10.340 --> 00:25:10.620
Yeah.

00:25:10.620 --> 00:25:11.760
And performance as well, right?

00:25:11.760 --> 00:25:17.100
Like if previously the call to get data about like the price of an item in your economy

00:25:17.100 --> 00:25:23.660
was within the same machine or, or something to this effect within the same process even,

00:25:23.660 --> 00:25:25.640
but it still hits like a database maybe.

00:25:25.640 --> 00:25:30.020
That's probably a lot faster than actually going across the network, finding another machine,

00:25:30.020 --> 00:25:32.900
doing a whole HTTP thing, and then asking that same question, right?

00:25:32.900 --> 00:25:33.240
Yeah.

00:25:33.240 --> 00:25:33.660
Absolutely.

00:25:33.880 --> 00:25:38.520
I think, yeah, you just have to, you have to tread very carefully.

00:25:38.520 --> 00:25:44.280
I think it's easy to get really, really excited about microservices and, but you really have

00:25:44.280 --> 00:25:47.160
to make sure that's the correct solution for you.

00:25:47.160 --> 00:25:47.840
Yeah, absolutely.

00:25:47.840 --> 00:25:51.720
I certainly see those tied together with Docker and the whole container space.

00:25:51.720 --> 00:25:52.340
But again.

00:25:52.340 --> 00:25:53.100
Absolutely.

00:25:53.100 --> 00:25:55.200
They've made this thing a lot easier.

00:25:55.200 --> 00:25:55.680
Nice.

00:25:55.680 --> 00:25:59.940
So one of the things that you said that you guys had done, I thought was pretty interesting

00:25:59.940 --> 00:26:02.680
is you said you had written your own importer.

00:26:03.160 --> 00:26:08.680
Is that, like if I say import space some module name, like import OS, you rewrote what that meant?

00:26:08.680 --> 00:26:09.720
Well, sort of.

00:26:09.720 --> 00:26:14.220
It's not really maybe such an importer as much as like a bootstrap.

00:26:14.220 --> 00:26:15.260
Okay.

00:26:15.260 --> 00:26:22.000
I don't know actually what the state was in Python 1.6, but I'm thinking maybe it didn't actually

00:26:22.000 --> 00:26:26.300
know how to import packages and that whole dunder in it.

00:26:26.300 --> 00:26:27.260
Oh, interesting.

00:26:27.260 --> 00:26:27.640
Okay.

00:26:27.640 --> 00:26:29.480
So that was pretty interesting.

00:26:29.480 --> 00:26:33.820
And so what they wanted, and they wanted like a shallow namespace for these things.

00:26:33.820 --> 00:26:37.860
So what they actually did was they wrote something that would just loop over the file.

00:26:37.860 --> 00:26:44.460
It would, you know, read, execute that file and import it into like a namespace that they

00:26:44.460 --> 00:26:44.720
created.

00:26:44.720 --> 00:26:47.440
So they basically just injected into CSTOck modules.

00:26:47.820 --> 00:26:48.180
Interesting.

00:26:48.180 --> 00:26:49.020
Okay.

00:26:49.020 --> 00:26:50.460
So what was the benefit there?

00:26:50.460 --> 00:26:50.860
Do you remember?

00:26:50.860 --> 00:26:56.400
One of the benefits was that we'd have auto-reloading of modules when we changed them.

00:26:56.400 --> 00:26:59.920
And it was actually quite nice to work with E at the time.

00:26:59.920 --> 00:27:01.660
And it still works that way.

00:27:01.660 --> 00:27:05.440
It's just now it's a Python solution instead of like a home-cooked one.

00:27:06.120 --> 00:27:11.780
But just changing something in code, you don't have to restart your survey, don't have to

00:27:11.780 --> 00:27:13.460
restart your clients or anything.

00:27:13.460 --> 00:27:16.060
You just re-execute whatever you're doing.

00:27:16.060 --> 00:27:17.960
You click the button again.

00:27:17.960 --> 00:27:18.420
Okay.

00:27:18.420 --> 00:27:19.220
Now you got my attention.

00:27:19.220 --> 00:27:20.600
That's pretty intense, actually.

00:27:20.600 --> 00:27:21.120
Yeah.

00:27:21.120 --> 00:27:23.220
And that's still the case today.

00:27:23.220 --> 00:27:26.740
But now it's like with a Python implementation of it.

00:27:26.740 --> 00:27:27.640
Did you have to do...

00:27:27.640 --> 00:27:28.720
I guess you almost have to.

00:27:28.800 --> 00:27:32.340
Either you say you can't use these types of features or you have to deal with things like

00:27:32.340 --> 00:27:36.400
static variables and static class fields that have been set previously.

00:27:36.400 --> 00:27:39.220
And then if you reload the module, what did you do there?

00:27:39.220 --> 00:27:41.140
It wasn't completely foolproof.

00:27:41.140 --> 00:27:42.320
Like if you...

00:27:42.320 --> 00:27:43.420
And it still isn't.

00:27:43.420 --> 00:27:47.540
If you already bound the method to something, it doesn't actually reload that.

00:27:47.540 --> 00:27:52.280
If you like pass in a method of an object into something else, then it's already bounded

00:27:52.280 --> 00:27:54.060
and it doesn't reload it.

00:27:54.440 --> 00:27:59.080
But for the most part, for the most code you are changing, it actually just changes it.

00:27:59.080 --> 00:28:05.360
I've had issues where some data was actually stored on a module level, like a dictionary

00:28:05.360 --> 00:28:06.300
on a module level.

00:28:06.300 --> 00:28:09.900
And whenever that got re-imported, it would clear that.

00:28:09.900 --> 00:28:11.600
And those sort of things.

00:28:11.600 --> 00:28:13.320
I mean, it happens.

00:28:13.320 --> 00:28:15.220
You need to restart your server and client.

00:28:15.220 --> 00:28:19.460
And when you're trying something out and something funky comes up, that's something what you do.

00:28:19.460 --> 00:28:21.620
You just restart everything and try again.

00:28:21.620 --> 00:28:21.860
Yeah.

00:28:21.860 --> 00:28:22.240
Interesting.

00:28:22.240 --> 00:28:23.200
So you have like two levels.

00:28:23.300 --> 00:28:25.320
You're like, let's just try to hot patch it.

00:28:25.320 --> 00:28:27.460
If that doesn't work, give it a hard kick, right?

00:28:27.460 --> 00:28:29.760
And we've done that in production as well.

00:28:29.760 --> 00:28:34.600
Like if there's a problem that we really want to fix, and that's actually one of the drawbacks

00:28:34.600 --> 00:28:36.420
of having a monolithic application.

00:28:36.420 --> 00:28:41.180
If I have a problem in the market, I need to take everything down to patch it up.

00:28:41.180 --> 00:28:41.500
Right.

00:28:41.500 --> 00:28:44.160
We've actually live fixed stuff like that.

00:28:44.160 --> 00:28:49.500
Just going in console and just creating a new method and attaching that to the class.

00:28:49.500 --> 00:28:49.780
Okay.

00:28:49.780 --> 00:28:51.720
That sounds really, really powerful and interesting.

00:28:51.900 --> 00:28:56.040
Is any of that sort of open source or blogged or written about that people can check out?

00:28:56.040 --> 00:28:57.280
No, no, no.

00:28:57.280 --> 00:28:59.160
None of this is open source.

00:28:59.160 --> 00:29:01.300
I don't think we've ever blogged about this.

00:29:01.300 --> 00:29:01.840
I think.

00:29:01.840 --> 00:29:02.360
Okay.

00:29:02.360 --> 00:29:03.360
Well, it sounds interesting.

00:29:03.360 --> 00:29:03.960
I know.

00:29:03.960 --> 00:29:05.620
I talked to David Beasley.

00:29:05.620 --> 00:29:09.700
I think it's show 12 about modules and packages and this whole concept.

00:29:09.700 --> 00:29:12.660
And it was like, oh, we're not really sure that this is going to work.

00:29:12.660 --> 00:29:13.500
If it can be done.

00:29:13.680 --> 00:29:16.840
So to hear that you guys are doing it on this real app, this is very cool.

00:29:16.840 --> 00:29:21.580
I think people might get kind of caught up in the idea that, well, it's not going to work

00:29:21.580 --> 00:29:22.140
every time.

00:29:22.140 --> 00:29:23.900
But that's okay when you're doing development.

00:29:23.900 --> 00:29:26.440
You just want like rapid feedback, really.

00:29:26.440 --> 00:29:26.680
Yeah.

00:29:26.680 --> 00:29:30.640
If it works 95% of the time and it saves you a ton, then maybe it's worth it, right?

00:29:30.640 --> 00:29:30.980
Yeah.

00:29:30.980 --> 00:29:31.280
Yeah.

00:29:31.280 --> 00:29:35.640
I guess coming back to the microservices thing, if you build for something like microservices

00:29:35.640 --> 00:29:40.600
and you do need to sort of hard restart some piece, it probably has a smaller consequence

00:29:40.600 --> 00:29:42.700
because you're not restarting the entire thing, right?

00:29:42.700 --> 00:29:43.040
Yeah.

00:29:43.040 --> 00:29:48.360
Well, but the flip side of that is that you have to write everything now as in nothing

00:29:48.360 --> 00:29:50.600
is guaranteed to be up.

00:29:50.600 --> 00:29:51.460
Yeah, right.

00:29:51.460 --> 00:29:56.080
If you do that, then you kind of, you're always guaranteed to have something down.

00:29:56.220 --> 00:30:00.540
And if your whole, if nothing works, when one of the bits is wrong, you're going to

00:30:00.540 --> 00:30:01.700
have constant downtimes.

00:30:01.700 --> 00:30:02.240
Yeah.

00:30:02.240 --> 00:30:03.720
That's straight off again, right?

00:30:03.720 --> 00:30:04.000
Yeah.

00:30:04.000 --> 00:30:05.060
It's a trade off.

00:30:05.060 --> 00:30:06.220
Like, it's not a silver bullet.

00:30:06.220 --> 00:30:10.140
And I think people have been pretty, pretty vocal about that.

00:30:10.140 --> 00:30:12.740
Like, the experts on this have been pretty vocal about that.

00:30:12.740 --> 00:30:19.200
And I think every paper and every talk I've seen about microservices always starts with

00:30:19.200 --> 00:30:19.380
that.

00:30:19.380 --> 00:30:22.420
This is not like a silver bullet that will fix all your problems.

00:30:22.420 --> 00:30:23.100
Okay, cool.

00:30:23.100 --> 00:30:29.040
So one of the things that happens in the game is there seems to be a lot of people

00:30:29.040 --> 00:30:31.760
that group together to form like a fleet.

00:30:31.760 --> 00:30:38.020
And then sort of in real time with voice communication, they'll go in and they'll have like these fleet

00:30:38.020 --> 00:30:38.680
battles, right?

00:30:38.680 --> 00:30:41.240
How many people end up in those battles?

00:30:41.240 --> 00:30:42.740
How many different players?

00:30:42.740 --> 00:31:02.760
SnapCI is a continuous delivery tool from ThoughtWorks that lets you reliably test and deploy your code through

00:31:02.760 --> 00:31:06.840
multi-stage pipelines in the cloud without the hassle of managing hardware.

00:31:07.460 --> 00:31:12.180
automate and visualize your deployments with ease and make pushing to production an effortless

00:31:12.180 --> 00:31:13.500
item on your to-do list.

00:31:13.500 --> 00:31:19.380
Snap also supports Docker and M browser debugging, and they integrate with AWS and Heroku.

00:31:19.380 --> 00:31:25.140
Thanks, SnapCI for sponsoring this episode by trying them with no obligation for 30 days by

00:31:25.140 --> 00:31:27.140
going to snap.ci.com.

00:31:27.140 --> 00:31:28.060
SnapCI slash talkpython.

00:31:28.060 --> 00:31:41.740
It can, Lars, I think the largest we had was like 4,000 people, I think.

00:31:41.740 --> 00:31:42.220
Wow.

00:31:42.220 --> 00:31:45.140
Then it can get pretty laggy, but...

00:31:45.960 --> 00:31:49.480
What's the sort of performance characteristics as you add more people?

00:31:49.480 --> 00:31:53.240
I mean, it can't be linear, like 2,000 people versus 4,000.

00:31:53.240 --> 00:31:56.320
It's more like combinatorial or something really expensive, right?

00:31:56.320 --> 00:31:57.600
Like, how's that work?

00:31:57.600 --> 00:32:00.820
We've done a number of optimizations over the years of that.

00:32:00.820 --> 00:32:02.920
It actually performs quite well.

00:32:02.920 --> 00:32:07.500
I'm not sure if it's linear, but it's not N2 or anything like that.

00:32:07.500 --> 00:32:12.360
But it's also very difficult to predict on that because it matters a lot whether everybody

00:32:12.360 --> 00:32:17.280
is shooting at a structure, maybe, that doesn't die for a long time, or if people are shooting

00:32:17.280 --> 00:32:22.220
at each other and people are getting killed and then removed from that solar system.

00:32:22.220 --> 00:32:24.120
That creates more load.

00:32:24.120 --> 00:32:28.000
It matters if people are shooting from guns or using missiles.

00:32:28.000 --> 00:32:32.740
Right, because guns presumably are instantaneous, but missiles sort of cruise through space.

00:32:32.740 --> 00:32:33.220
Yeah.

00:32:33.220 --> 00:32:34.020
Right?

00:32:34.020 --> 00:32:36.860
You've got to track them for everyone and see them go by and so on.

00:32:36.860 --> 00:32:37.320
Yeah, yeah.

00:32:37.320 --> 00:32:41.940
They don't collide really with anyone except the person that they're meant to collide with.

00:32:41.940 --> 00:32:45.000
That's one of the optimizations we did a long time ago.

00:32:45.000 --> 00:32:46.080
There's a lot of factors.

00:32:46.080 --> 00:32:53.080
It's very hard to say, like, you can look at a thousand people fleet fight and see no lag,

00:32:53.080 --> 00:32:55.900
and you can see another one that has a massive amount of lag.

00:32:55.900 --> 00:32:57.840
It's very hard to measure that.

00:32:57.840 --> 00:33:02.920
One of the most reliable kind of metrics we have is the market up.

00:33:02.920 --> 00:33:07.060
We have a single system that, you know how people are, they kind of just all gravitate.

00:33:07.060 --> 00:33:12.080
And because a lot of goods are there for sale, people think it's worth it for them to put

00:33:12.080 --> 00:33:13.460
their goods up for sale.

00:33:13.460 --> 00:33:16.880
So it creates this market up, which is a solar system called Jita.

00:33:16.880 --> 00:33:20.640
And there we have like a constant influx of people.

00:33:20.640 --> 00:33:25.520
We had like a maximum of 3,600 people there the other day.

00:33:25.520 --> 00:33:26.160
That's a lot.

00:33:26.340 --> 00:33:30.520
Because of some server optimization we had been doing, the CPU actually never went above

00:33:30.520 --> 00:33:32.400
50%, I think, on that.

00:33:32.400 --> 00:33:33.020
Yeah, yeah.

00:33:33.020 --> 00:33:33.780
That's really cool.

00:33:33.780 --> 00:33:38.140
So you said one of the challenges you have with some of the scaling has to do with the dreaded

00:33:38.140 --> 00:33:39.820
global interpreter lock or the GIL.

00:33:39.820 --> 00:33:40.900
What's the story there?

00:33:40.900 --> 00:33:47.760
We basically can segment our cluster up into, well, it's probably easier to explain by solar

00:33:47.760 --> 00:33:48.260
system, right?

00:33:48.260 --> 00:33:51.080
That's the smallest unit we can run.

00:33:51.080 --> 00:33:55.880
We can't distribute the load within one solar system to multiple processes, really.

00:33:56.000 --> 00:33:56.160
Right.

00:33:56.160 --> 00:33:56.400
Okay.

00:33:56.400 --> 00:34:00.480
That's going to be a bit problematic because that means we can only use really just one

00:34:00.480 --> 00:34:02.160
core on the machine.

00:34:02.160 --> 00:34:03.180
So it doesn't really matter.

00:34:03.180 --> 00:34:07.960
And it's very problematic because the hardware industry is actually moving more towards more

00:34:07.960 --> 00:34:10.120
core and maybe less clock speed.

00:34:10.120 --> 00:34:13.120
What we really desire is more clock speeds.

00:34:13.120 --> 00:34:13.480
Yeah.

00:34:13.480 --> 00:34:18.340
I mean, the clock speeds, I think, are, if they're not slower, they're definitely not any faster

00:34:18.340 --> 00:34:20.440
than what they were in like 2005, right?

00:34:20.440 --> 00:34:21.540
They're not getting faster.

00:34:21.540 --> 00:34:22.400
I can tell you that.

00:34:22.400 --> 00:34:22.660
Yeah.

00:34:22.660 --> 00:34:23.140
Yeah.

00:34:23.140 --> 00:34:24.260
I mean, they may even be slower.

00:34:24.360 --> 00:34:26.140
I know they got higher and they started to melt.

00:34:26.140 --> 00:34:26.800
They kind of went down.

00:34:26.800 --> 00:34:27.120
All right.

00:34:27.120 --> 00:34:27.540
That's it.

00:34:27.540 --> 00:34:31.440
Eight cores, lower, lower gigahertz, lower clock speed.

00:34:31.440 --> 00:34:31.640
Yep.

00:34:31.640 --> 00:34:32.280
Very interesting.

00:34:32.280 --> 00:34:38.080
Would Docker type stuff or virtualization sort of let you sort of break that up, at least

00:34:38.080 --> 00:34:43.500
not dedicate a whole machine to it, but say, okay, well, we're going to isolate, you know,

00:34:43.500 --> 00:34:44.100
Windows.

00:34:44.240 --> 00:34:48.620
I guess Docker is doing stuff with Microsoft these days, but maybe it's, I should say,

00:34:48.620 --> 00:34:50.280
Windows containers that's coming out soon.

00:34:50.280 --> 00:34:54.600
Would that be a way to partition this so that you can still get everything you want out of

00:34:54.600 --> 00:34:56.800
those machines without dealing with the threading?

00:34:56.800 --> 00:34:57.180
Yeah.

00:34:57.180 --> 00:34:58.880
It's an interesting thought.

00:34:58.880 --> 00:35:03.360
It still requires like a lot of programming work to actually do it, but you'd have to segment

00:35:03.360 --> 00:35:08.220
what the cluster is doing within the solar system into like something logical units,

00:35:08.220 --> 00:35:12.220
which you can scale, which doesn't chapter too much between each other.

00:35:12.460 --> 00:35:16.440
We actually have an idea about that, but yeah, we'll see if we get there.

00:35:16.440 --> 00:35:23.200
As it currently stands, we're not really that pressed for performance, or at least it's

00:35:23.200 --> 00:35:26.780
not as high priority as some other stuff we want to do.

00:35:26.780 --> 00:35:27.140
Right.

00:35:27.140 --> 00:35:31.560
When you're doing performance analysis, like you just look at the slowest thing, the bottleneck.

00:35:31.560 --> 00:35:32.420
Right.

00:35:32.420 --> 00:35:36.980
If the stuff behind it is slower, but still fast, you know, slow, but faster than the bottleneck,

00:35:36.980 --> 00:35:37.700
then who cares, right?

00:35:37.700 --> 00:35:38.640
Yeah, right.

00:35:38.640 --> 00:35:44.820
And there's something to be gained looking before we start looking into splitting really

00:35:44.820 --> 00:35:47.740
this into multiple processes on the same note.

00:35:47.740 --> 00:35:51.520
But we have like a better lead than that.

00:35:51.520 --> 00:35:56.040
So one thing you said you're really into is test-driven development, unit testing, and so on.

00:35:56.040 --> 00:35:58.360
And this code is super old.

00:35:58.360 --> 00:36:03.600
You must have some interesting challenges writing tests for a code from this long ago, yeah?

00:36:03.600 --> 00:36:09.400
Yeah, and the importer or bootstrapper, which we appropriately named Nasty at the time,

00:36:09.400 --> 00:36:17.440
was of course like a huge, huge burden on us for that, because we couldn't really import

00:36:17.440 --> 00:36:22.000
any of the EVE-specific code without using that Nasty importer.

00:36:22.000 --> 00:36:28.340
And that Nasty importer really had to throw through everything and really just run a full-scale

00:36:28.340 --> 00:36:29.520
client or a server.

00:36:29.520 --> 00:36:33.580
Yeah, so you were almost like forced to the point where you're doing integration tests

00:36:33.580 --> 00:36:36.500
because you had to start the whole thing anyway, right?

00:36:36.500 --> 00:36:37.160
Exactly.

00:36:37.160 --> 00:36:44.640
And we did manage to create like an interpreter that did that, but we had me and another program

00:36:44.640 --> 00:36:48.420
who were writing tests, and I think we were up to like 100 tests, and they were taking 45

00:36:48.420 --> 00:36:51.740
seconds to run, which was just terrible.

00:36:51.740 --> 00:36:53.840
Yeah, that's pretty long for unit tests, right?

00:36:53.900 --> 00:36:58.500
Yeah, yeah, and it's unusable if you want to use it for like a quick iteration feedback.

00:36:58.500 --> 00:36:59.160
Yeah.

00:36:59.160 --> 00:37:06.040
When developing, you really need something in less than a second time or like close to it.

00:37:06.040 --> 00:37:09.300
But there was some initiative to actually get rid of Nasty.

00:37:09.300 --> 00:37:12.600
They actually managed to do it in like one summer.

00:37:12.600 --> 00:37:15.240
They managed to do it in, I think, two weeks.

00:37:15.240 --> 00:37:16.080
Oh, nice.

00:37:16.080 --> 00:37:17.720
And now testing is much easier?

00:37:17.720 --> 00:37:19.000
Now testing is a lot easier.

00:37:19.220 --> 00:37:24.600
It's still a lot of coupling, and there are other things other than creating our own

00:37:24.600 --> 00:37:30.200
bootstrapper slash importer that we did, like shoving stuff into buildings and stuff like

00:37:30.200 --> 00:37:32.060
that that make testing harder.

00:37:32.060 --> 00:37:40.080
But usually always you can extract the code that you want to test out, and you write tests

00:37:40.080 --> 00:37:40.420
for them.

00:37:40.420 --> 00:37:40.980
Right, okay.

00:37:40.980 --> 00:37:44.680
Nice to hear that you've got that sort of cracked open so you can test it.

00:37:44.680 --> 00:37:45.200
Yeah, yeah.

00:37:45.200 --> 00:37:51.920
It's been like once that came in and once I started being able to write tests like that,

00:37:51.920 --> 00:37:58.080
it kind of really opened my eyes towards testing and development and automatic testing in general.

00:37:58.080 --> 00:37:58.800
All right, cool.

00:37:58.800 --> 00:38:03.640
So let's talk about a little bit about CCP, the company that makes EVE Online.

00:38:03.640 --> 00:38:03.980
Yeah.

00:38:03.980 --> 00:38:06.600
You guys are primarily based out of Iceland, right?

00:38:06.600 --> 00:38:07.900
Right, mostly out of Iceland.

00:38:07.900 --> 00:38:14.300
We have offices in Núkastúl, Shanghai, and Atlanta, but everybody that works on EVE is

00:38:14.300 --> 00:38:15.320
stationed in Reykjavík.

00:38:15.320 --> 00:38:15.900
Very cool.

00:38:15.900 --> 00:38:17.120
I've never been to Iceland.

00:38:17.120 --> 00:38:19.080
It's definitely one of the places I'd like to visit, though.

00:38:19.080 --> 00:38:23.480
Yeah, it's definitely interesting, although I lived here most of my life, so...

00:38:23.480 --> 00:38:25.020
It seems very natural to you, right?

00:38:25.020 --> 00:38:25.940
Very normal, but...

00:38:25.940 --> 00:38:26.640
It seems very normal.

00:38:26.640 --> 00:38:32.260
Yeah, with all the winter and the volcanoes, and I think that'd be super cool to go check

00:38:32.260 --> 00:38:32.480
out.

00:38:32.480 --> 00:38:33.140
Nice.

00:38:33.140 --> 00:38:35.900
So you guys use an Agile approach, is that right?

00:38:35.900 --> 00:38:37.620
Do you use Scrum, or what's the story there?

00:38:37.920 --> 00:38:41.480
Yeah, well, we started with Scrum, and then we kind of...

00:38:41.480 --> 00:38:45.820
It's morphed into something that's like Scrum, but it's mostly...

00:38:45.820 --> 00:38:52.580
Scrum is really the basis of it, but I think we allow every team to really put their own

00:38:52.580 --> 00:38:55.460
flavor on it and do kind of whatever works for them.

00:38:55.460 --> 00:39:01.300
Agile practices are really funny, because if you don't pay attention to them, they will

00:39:01.300 --> 00:39:04.660
just deteriorate, and you'll start getting worse at them.

00:39:04.660 --> 00:39:07.600
So it's something that just kind of requires constant putting.

00:39:07.600 --> 00:39:13.300
And of course, it's part of the Agile to have retrospective and constantly be trying to

00:39:13.300 --> 00:39:13.880
improve yourself.

00:39:13.880 --> 00:39:14.180
Right.

00:39:14.180 --> 00:39:15.820
I think a lot of people lose sight of that.

00:39:15.820 --> 00:39:21.760
I mean, on one hand, there's not necessarily a single Agile process that is exactly right

00:39:21.760 --> 00:39:22.980
for everyone, right?

00:39:23.000 --> 00:39:26.200
I mean, that's a little bit about being flexible and Agile and so on there.

00:39:26.200 --> 00:39:30.720
But then if you're going to go and sort of walk it up to the edge of the absolute minimum

00:39:30.720 --> 00:39:36.820
amount of documentation, rules, structure, that means if you fall apart just a little bit,

00:39:36.820 --> 00:39:39.640
you're probably falling out of the minimum amount of structure that you need, right?

00:39:39.640 --> 00:39:40.000
Right.

00:39:40.240 --> 00:39:41.720
It's a little bit like that.

00:39:41.720 --> 00:39:44.120
I kind of like starting with Scrum.

00:39:44.120 --> 00:39:49.920
I think Scrum has like, if you understand why the rules of Scrum, then you have the power

00:39:49.920 --> 00:39:51.980
to change them as they make sense to you.

00:39:51.980 --> 00:39:56.900
But I've seen also seen a lot of companies or heard from like a lot of my colleagues here

00:39:56.900 --> 00:40:03.000
around Iceland is that they've implemented Scrum, but they really didn't get what Agile was.

00:40:03.720 --> 00:40:05.160
So it just goes through the motions.

00:40:05.160 --> 00:40:07.460
And then, hey, this Agile thing sucks.

00:40:07.460 --> 00:40:08.080
It doesn't work.

00:40:08.080 --> 00:40:09.020
Yeah, right.

00:40:09.020 --> 00:40:10.860
Yeah, well, maybe you didn't follow it.

00:40:10.860 --> 00:40:16.160
You guys said you also have a kind of interesting learning and development philosophy.

00:40:16.160 --> 00:40:17.120
What's the story there?

00:40:17.120 --> 00:40:18.560
Well, it's not.

00:40:18.560 --> 00:40:25.280
I mean, we do try to foster kind of the culture of having like what we call a community of practice.

00:40:25.280 --> 00:40:32.220
We had a Python community of practice, for example, but we kind of retired that because

00:40:32.220 --> 00:40:34.640
people weren't really showing up that much.

00:40:34.640 --> 00:40:41.220
But it's actually very difficult because people are so invested in their job.

00:40:41.220 --> 00:40:47.560
They're so invested in what they're doing that they don't see that they can spare an hour here

00:40:47.560 --> 00:40:49.160
to do stuff like that.

00:40:49.160 --> 00:40:54.700
What we've been doing is maybe a bit weird, which is different than sending people to conferences

00:40:54.700 --> 00:40:59.720
and getting consultants here is that we've been kind of running this video series.

00:40:59.720 --> 00:41:00.700
Uncle Bob.

00:41:01.040 --> 00:41:01.340
Oh, yeah.

00:41:01.340 --> 00:41:02.540
Robert Martin's awesome.

00:41:02.540 --> 00:41:02.880
Yeah.

00:41:02.880 --> 00:41:06.540
We've been running that and we've been having like talks after it.

00:41:06.540 --> 00:41:09.520
That I think has been very good.

00:41:09.520 --> 00:41:12.520
I definitely recommend for companies to do that.

00:41:12.520 --> 00:41:17.960
It's probably one of the cheapest learning and development you can get, like in terms of bang for buck.

00:41:17.960 --> 00:41:19.700
Yeah, I totally agree.

00:41:19.820 --> 00:41:24.800
I think one of the things that people, companies, teams, individuals, like on all the levels,

00:41:24.800 --> 00:41:31.180
don't necessarily realize is if you just spend like half an hour a week or half an hour twice

00:41:31.180 --> 00:41:34.760
a week making sure you polish your skills, you learn something new.

00:41:34.760 --> 00:41:37.100
After a few years, that really adds up.

00:41:37.240 --> 00:41:37.680
Absolutely.

00:41:37.680 --> 00:41:40.080
There have been other initiatives in the past.

00:41:40.080 --> 00:41:44.980
Like we had, like on Friday, people had code cutters where they would meet up and they would

00:41:44.980 --> 00:41:50.120
get like a task and then go in groups of twos and they would do something like write these

00:41:50.120 --> 00:41:51.580
features, but don't use classes.

00:41:51.580 --> 00:41:52.940
Stuff like that.

00:41:53.540 --> 00:41:55.560
So these initiatives, they come and go.

00:41:55.560 --> 00:42:03.040
I had a kind of a mob programming session, which was all about pulling code out and write tests for it.

00:42:03.040 --> 00:42:03.660
Very interesting.

00:42:03.660 --> 00:42:08.340
You know, I don't think we've talked about mob programming on this show.

00:42:08.340 --> 00:42:11.920
I know what it is, but I don't think we've ever mentioned it.

00:42:11.920 --> 00:42:13.860
Can you just give me like the really quick elevator pitch?

00:42:13.860 --> 00:42:15.220
Like what is mob programming?

00:42:15.220 --> 00:42:21.820
Basically what it sounds like, it's like one guy and a keyboard and he takes a test and people shout at him.

00:42:21.820 --> 00:42:28.020
But of course, there's one driver and he probably knows a bit where he's going and then he kind of just listens to,

00:42:28.020 --> 00:42:29.640
okay, so what do you want to do here?

00:42:29.640 --> 00:42:31.960
Do you want to create a new method here?

00:42:31.960 --> 00:42:33.980
Do you want to create a class or whatever?

00:42:33.980 --> 00:42:38.560
Would you say it's kind of like paired programming, but many people on the pair side?

00:42:38.560 --> 00:42:43.720
Yeah, and probably a lot, lot less efficient, but it sparks up a good conversation.

00:42:43.720 --> 00:42:44.280
Right.

00:42:44.280 --> 00:42:46.580
It's more, it's not necessarily like paired programming.

00:42:46.580 --> 00:42:50.540
The expected output is solid code that you're going to ship.

00:42:50.540 --> 00:42:50.760
Yeah.

00:42:50.840 --> 00:42:54.120
Where I think mob programming is more about mutual understanding.

00:42:54.120 --> 00:42:54.940
Yes.

00:42:54.940 --> 00:42:56.300
And skill development and so on, right?

00:42:56.300 --> 00:42:56.640
Yeah.

00:42:56.640 --> 00:43:01.160
Also just getting people to kind of voice their opinions about how code should be.

00:43:01.160 --> 00:43:03.020
Because that's, we actually just don't do that.

00:43:03.020 --> 00:43:05.380
We write code, we put it up to code review.

00:43:05.380 --> 00:43:11.820
But without me understanding what your agenda or what your philosophy about code is,

00:43:11.820 --> 00:43:16.140
then chances are we're going to disagree on a lot of like fundamental things.

00:43:16.560 --> 00:43:17.480
I would say so.

00:43:17.480 --> 00:43:17.900
Yeah.

00:43:17.900 --> 00:43:20.100
It sounds like a cool approach you guys have going there.

00:43:20.100 --> 00:43:24.820
So one thing I wanted to ask you about, do you have any stories about like when there was

00:43:24.820 --> 00:43:30.100
some giant unexpected rush of new traffic or new customers, like maybe like, you know,

00:43:30.100 --> 00:43:32.860
a new release came out that was really popular or something like that?

00:43:32.860 --> 00:43:38.460
Well, we had the other day, we had a new item come out that was very popular on the market.

00:43:38.660 --> 00:43:44.100
And then that meant that everybody brought to that note, that cheetah note to sell stuff

00:43:44.100 --> 00:43:45.260
and buy stuff on it.

00:43:45.260 --> 00:43:48.740
Way more than what we've usually been expecting.

00:43:48.740 --> 00:43:51.620
The thing is, we have a cup on that note.

00:43:51.620 --> 00:43:56.200
We have like, we allow like max 2,000 people, I think, on it.

00:43:57.000 --> 00:44:04.620
Because if we get more, or when we used to get more, then the game would be pretty unplayable

00:44:04.620 --> 00:44:05.940
for everybody involved.

00:44:05.940 --> 00:44:09.880
So we thought it was better to actually limit the amount of people that could be in there

00:44:09.880 --> 00:44:10.600
at any time.

00:44:10.600 --> 00:44:13.540
And their experience would at least be a little bit better.

00:44:13.540 --> 00:44:16.780
And people can always play and be in other systems.

00:44:17.340 --> 00:44:23.240
But because of a number of performance, like a big performance upgrade we've done, we kind

00:44:23.240 --> 00:44:25.760
of saw that the server wasn't actually...

00:44:25.760 --> 00:44:27.800
So we started increasing the cup.

00:44:27.800 --> 00:44:30.180
We increased it first by 2,500.

00:44:30.180 --> 00:44:32.740
And then we started reaching that.

00:44:32.740 --> 00:44:33.700
So we increased it.

00:44:33.700 --> 00:44:36.240
So we're thinking about increasing it to 3,000.

00:44:36.240 --> 00:44:40.880
But instead, the development director said, like, looking at the graph, I think we should

00:44:40.880 --> 00:44:42.100
just go to 5,000.

00:44:42.100 --> 00:44:44.300
We can support 5,000 based on these numbers.

00:44:44.300 --> 00:44:45.920
So that's what we did.

00:44:45.920 --> 00:44:49.280
And we got 3,600 people there.

00:44:49.280 --> 00:44:51.840
The CPU never went above 50%.

00:44:51.840 --> 00:44:55.640
But the memory was actually getting to the limit.

00:44:55.640 --> 00:44:55.960
Wow.

00:44:55.960 --> 00:44:57.760
And that was actually like a...

00:44:57.760 --> 00:45:04.840
It's a weird thing because we've gone from being memory restricted on a PC to almost not

00:45:04.840 --> 00:45:06.920
having to think about it at any point.

00:45:06.920 --> 00:45:12.100
Eve has this weird thing where we restart the cluster every day at 11 o'clock.

00:45:12.460 --> 00:45:16.600
So whatever memory we have cast, that gets flushed out.

00:45:16.600 --> 00:45:22.120
So the programmers on Eve have never really considered memory so much.

00:45:22.120 --> 00:45:26.740
So we just cast it and then we never release it because we've never really had to release it.

00:45:27.500 --> 00:45:33.860
So it's weird that when you have a lot of people coming in, we just saw the memory increasing and it never went down.

00:45:33.860 --> 00:45:34.420
Oh, wow.

00:45:34.420 --> 00:45:36.020
Nothing ever released that memory.

00:45:36.020 --> 00:45:36.460
Yeah.

00:45:36.460 --> 00:45:48.840
What I think is interesting about that story is when I think of some system getting hit with lots of traffic and having unexpected growth, I see it as an all or nothing type of thing.

00:45:49.000 --> 00:45:52.500
Like the website now has a lot of visitors and so it's busy or not.

00:45:52.500 --> 00:46:00.480
But the way you guys have structured your world, it's like there's a little node over there somewhere where a bunch of people went into that part of the universe and now that part is too busy.

00:46:00.480 --> 00:46:02.700
That's an interesting way to think about scale.

00:46:02.700 --> 00:46:05.280
Like I suspect most places don't.

00:46:05.280 --> 00:46:05.900
Yeah, yeah.

00:46:05.900 --> 00:46:07.440
It's always been a...

00:46:07.440 --> 00:46:08.440
And it's both...

00:46:08.440 --> 00:46:20.260
It's very funny to look at the kind of the overall CPU utilization of our clusters and you can see like it's using 10% CPU on average.

00:46:20.260 --> 00:46:28.840
But still, you know that there are some people, there are some node there that is 100% and people are experiencing lag and you're seeing 10%.

00:46:28.840 --> 00:46:34.760
Yeah, it's almost like you need sort of a mountain range type graph instead of just a level or something like this.

00:46:34.760 --> 00:46:35.480
Very cool.

00:46:35.480 --> 00:46:40.520
And in our old data center, what we used to have is that we had one node that was...

00:46:40.520 --> 00:46:42.120
We called it the Everest node.

00:46:42.120 --> 00:46:52.600
It only ran that one JETA solar system and it was a bit beefier than all the other nodes and it was just dedicated to running that particular node.

00:46:52.600 --> 00:46:52.920
Yeah.

00:46:52.920 --> 00:46:57.200
In the new cluster, I think all the machines are better than that old machine.

00:46:57.200 --> 00:47:01.020
That was the good one and now it's old and crusty.

00:47:01.020 --> 00:47:01.480
Very cool.

00:47:01.480 --> 00:47:04.940
So I do have just a few more questions before we kind of wrap up the show.

00:47:04.940 --> 00:47:05.260
Yep.

00:47:05.260 --> 00:47:11.560
I was told on Twitter, thanks, I think it was Kyle that sent a message that said, I should ask you about the big red button.

00:47:11.900 --> 00:47:14.100
Yeah, I always get that.

00:47:14.100 --> 00:47:18.000
I'm a bit of a, I guess, infamous in the EVE universe.

00:47:18.000 --> 00:47:24.580
So what happened is that I was just happily developing on my local clusters and I needed to do something.

00:47:24.580 --> 00:47:29.140
Normally when I shut down the cluster, I just kill the process and start it off again.

00:47:29.540 --> 00:47:35.300
But there is some logic that happens when you shut down the cluster, some persisting logic that I wanted to test.

00:47:35.300 --> 00:47:41.060
So I go to the web page of the cluster and I shut it down properly or properly.

00:47:41.060 --> 00:47:44.240
It's called an emergency shutdown and I gave it a minute timer.

00:47:44.840 --> 00:47:51.860
So I did that and then I go back to my client and I don't see the normal message that I get when I shut down the cluster.

00:47:51.860 --> 00:47:54.220
Like the cluster is about to shut down.

00:47:54.760 --> 00:48:00.700
So I look at the page again and I realize I'm on the production clusters and I'm shutting down the production clusters.

00:48:00.700 --> 00:48:03.000
Oops.

00:48:03.240 --> 00:48:08.260
So I, and the only way to cancel it is to go to that page again.

00:48:08.260 --> 00:48:12.120
So I did that and I canceled and I didn't know how long time had passed.

00:48:12.120 --> 00:48:16.680
I knew I put a one minute timer there, but I just frantically clicked that button.

00:48:16.680 --> 00:48:19.600
And it turns out I managed to stop the shutdown.

00:48:19.600 --> 00:48:21.380
It never actually shut down.

00:48:21.380 --> 00:48:28.860
But everybody that was logged in got that message that the EVE was shutting down and people really don't like that.

00:48:28.860 --> 00:48:29.740
I'm sure they don't.

00:48:29.740 --> 00:48:31.380
They really don't like that.

00:48:31.440 --> 00:48:35.480
So they went on the forums immediately and said, like, CCP, what the hell?

00:48:35.480 --> 00:48:37.100
Why are you shutting down the server?

00:48:37.100 --> 00:48:38.560
Why do you have a one minute timer?

00:48:38.560 --> 00:48:41.660
Because people want to get their assets to safety and stuff like that.

00:48:41.660 --> 00:48:42.440
Right, of course.

00:48:42.440 --> 00:48:42.860
Yeah.

00:48:42.860 --> 00:48:45.260
So I did what just came naturally to me.

00:48:45.260 --> 00:48:50.320
I just, I just went into that forum thread and saw, look, sorry, this is my bad.

00:48:50.320 --> 00:48:53.700
I thought I was shutting down my development cluster.

00:48:53.700 --> 00:48:56.920
And I'm being ridiculed by my coworkers now.

00:48:56.920 --> 00:48:57.820
Yeah, I'm sure.

00:48:57.820 --> 00:48:59.220
But it's really, really funny.

00:48:59.220 --> 00:49:00.940
Because EVE players are like that.

00:49:01.140 --> 00:49:03.560
They do get very emotional about it.

00:49:03.560 --> 00:49:05.640
But they also are pretty cool people.

00:49:05.640 --> 00:49:09.320
And honest mistake like that, they just let it slide.

00:49:09.320 --> 00:49:09.720
Yeah.

00:49:09.720 --> 00:49:11.020
And they're pretty technically savvy.

00:49:11.020 --> 00:49:12.240
They are technically savvy.

00:49:12.240 --> 00:49:12.860
They understand.

00:49:12.860 --> 00:49:15.160
Some of them probably did this once.

00:49:15.160 --> 00:49:25.580
And I think this even happened like a week after when someone in operations of QA, I can't remember, was going to shut down the test server and shut down that.

00:49:25.760 --> 00:49:27.760
But he didn't go to the forums and apologize.

00:49:27.760 --> 00:49:27.920
Yeah.

00:49:27.920 --> 00:49:31.640
And didn't claim, you know, public responsibility and say something.

00:49:31.860 --> 00:49:35.160
And I think that people like to remember that story as something.

00:49:35.160 --> 00:49:36.000
Yeah.

00:49:37.320 --> 00:49:38.500
This is messed up.

00:49:38.500 --> 00:49:51.520
But I'd like to think that I just, I did the right thing by apologizing to people and explaining just exactly what went wrong and why I was being stupid for doing it.

00:49:51.780 --> 00:49:52.060
Yeah.

00:49:52.060 --> 00:49:53.160
It all happens to us.

00:49:53.160 --> 00:49:55.780
One time I was teaching a training class.

00:49:55.780 --> 00:49:58.980
And I'll try to not put any details about any place.

00:49:58.980 --> 00:50:01.360
And I was somewhere in Chicago, let's say.

00:50:01.360 --> 00:50:04.680
And I was teaching this training class at a financial place.

00:50:04.780 --> 00:50:10.760
And they, for some reason, didn't have a proper room that they could use for all the 20 people that were in my class or whatever.

00:50:10.760 --> 00:50:16.280
So they had given us this weird, like, secondary server room where there's all these servers along the wall.

00:50:16.280 --> 00:50:19.340
But they weren't, it wasn't obviously the main data center.

00:50:19.340 --> 00:50:22.800
It was just, like, servers that ran internals within the company, you know?

00:50:22.800 --> 00:50:23.060
Yeah.

00:50:23.060 --> 00:50:27.740
And I had this little squishy ball thing that I would use to, like, work out your hands or whatever.

00:50:27.740 --> 00:50:28.860
And I was walking around.

00:50:28.860 --> 00:50:30.100
It slipped out of my hand.

00:50:30.100 --> 00:50:31.260
And it was odd shaped.

00:50:31.260 --> 00:50:33.100
So it bounced like a crazy ball.

00:50:33.440 --> 00:50:40.420
And it bounced along the students' desks, over along the wall, hit the wall, fell down, and hit a power strip that killed an entire wall of servers.

00:50:40.420 --> 00:50:42.400
Oh, my God.

00:50:42.400 --> 00:50:44.180
You hear it go quieter.

00:50:44.180 --> 00:50:44.880
Oh, my.

00:50:44.880 --> 00:50:45.520
That is bad.

00:50:45.520 --> 00:50:48.180
So we quick turn them all back on and just go back to the class.

00:50:48.180 --> 00:50:52.840
And about 10 minutes later, some guys rush in, like, what happened to the servers?

00:50:52.840 --> 00:50:54.140
Everyone's like, we don't know.

00:50:54.140 --> 00:50:57.020
So, yeah.

00:50:57.020 --> 00:50:59.400
I've turned off the servers as well on accident.

00:50:59.400 --> 00:51:01.920
Yeah, yeah.

00:51:01.920 --> 00:51:03.140
Luckily, the students had my back.

00:51:03.140 --> 00:51:03.740
All right.

00:51:03.740 --> 00:51:05.520
Two quick questions before we go.

00:51:05.520 --> 00:51:08.380
If you're going to write some Python code, what editor do you use?

00:51:08.380 --> 00:51:10.340
I use PyCharm now.

00:51:10.340 --> 00:51:13.440
I've used Sublime and Wim in the past.

00:51:13.440 --> 00:51:18.880
But I think in terms of, like, a good IDE, I think nothing is better than PyCharm.

00:51:18.880 --> 00:51:20.860
Yeah, I'm with you on that.

00:51:20.860 --> 00:51:21.900
I love PyCharm.

00:51:21.900 --> 00:51:23.120
That's what I use as well.

00:51:23.120 --> 00:51:23.600
Awesome.

00:51:23.800 --> 00:51:30.120
And then if you're going to go and look at PyPI, all the packages, I think there's like 75,000 of them now.

00:51:30.120 --> 00:51:34.420
Are there ones that are super useful that you think maybe people don't know about that they should?

00:51:34.420 --> 00:51:35.700
Well, what to choose from?

00:51:35.700 --> 00:51:37.540
I love the request packets.

00:51:37.680 --> 00:51:42.440
I think it kind of puts the simplicity in making, like, HTTP call.

00:51:42.440 --> 00:51:42.800
Yeah.

00:51:42.800 --> 00:51:48.160
What's interesting about that is it's, like, so simple, you can almost not appreciate how cool it is.

00:51:48.160 --> 00:51:48.580
Exactly.

00:51:48.580 --> 00:51:50.740
I really love that packet.

00:51:50.740 --> 00:51:53.400
Mostly because I have probably used the other libraries.

00:51:53.400 --> 00:51:56.140
And they are probably more flexible and stuff like that.

00:51:56.180 --> 00:51:59.340
But sometimes you just want to make a get request on a resource.

00:51:59.340 --> 00:52:00.500
Yep.

00:52:00.500 --> 00:52:01.300
Request.get.

00:52:01.300 --> 00:52:02.340
And you're good to go.

00:52:02.340 --> 00:52:03.300
Awesome.

00:52:03.300 --> 00:52:06.860
So if people want to get started with EVE Online, how do they do it?

00:52:06.860 --> 00:52:08.320
Visit eveonline.com.

00:52:08.320 --> 00:52:10.080
That would be the first step.

00:52:10.080 --> 00:52:14.620
And I want to highlight a feature of that is that you can download the launcher.

00:52:14.620 --> 00:52:17.360
And it starts downloading the game.

00:52:17.360 --> 00:52:19.200
But it only downloads the stuff it needs.

00:52:19.200 --> 00:52:22.860
So it only downloads the code and, like, a minimum set of resources.

00:52:22.860 --> 00:52:25.980
So you can be within the game in, like, 10 minutes from there.

00:52:26.100 --> 00:52:26.340
Oh, yeah.

00:52:26.340 --> 00:52:27.320
That's awesome.

00:52:27.320 --> 00:52:27.600
Yeah.

00:52:27.600 --> 00:52:29.520
I definitely want to start playing and check it out.

00:52:29.520 --> 00:52:30.280
It sounds really cool.

00:52:30.280 --> 00:52:34.040
And after that, it just downloads the resources as it needs it.

00:52:34.040 --> 00:52:37.180
So if you want to try it out, go to eveonline.com.

00:52:37.180 --> 00:52:40.420
Sign up for a free trial and check it out.

00:52:40.420 --> 00:52:44.900
It should take you less than 20 minutes on a decent minute connection.

00:52:44.900 --> 00:52:45.500
Very cool.

00:52:45.500 --> 00:52:48.860
Kristen, it's been fun to learn about what you guys have going on with Python.

00:52:48.860 --> 00:52:49.960
A lot of cool stories.

00:52:49.960 --> 00:52:50.640
Thanks for sharing them.

00:52:50.640 --> 00:52:50.980
Yeah.

00:52:50.980 --> 00:52:51.400
Thanks.

00:52:51.400 --> 00:52:52.400
Thanks for having me.

00:52:52.400 --> 00:52:52.600
Yeah.

00:52:52.600 --> 00:52:52.920
Take care.

00:52:52.920 --> 00:52:53.220
Bye-bye.

00:52:54.300 --> 00:52:56.600
This has been another episode of Talk Python To Me.

00:52:56.600 --> 00:52:58.700
Today's guest was Kristen Siger-Bergsen.

00:52:58.700 --> 00:53:01.540
And this episode has been sponsored by Hired and SnapCI.

00:53:01.540 --> 00:53:03.500
Thank you guys for supporting the show.

00:53:03.500 --> 00:53:05.660
Hired wants to help you find your next big thing.

00:53:05.660 --> 00:53:10.560
Visit Hired.com slash Talk Python To Me to get five or more offers with salary and equity presented

00:53:10.560 --> 00:53:13.840
right up front and a special listener signing bonus of $2,000.

00:53:13.840 --> 00:53:17.620
SnapCI is modern, continuous integration and delivery.

00:53:18.180 --> 00:53:20.820
Build, test, and deploy your code directly from GitHub.

00:53:20.820 --> 00:53:24.160
All in your browser with debugging, Docker, and parallelism included.

00:53:24.160 --> 00:53:27.100
Try them for free at snap.ci slash Talk Python.

00:53:27.100 --> 00:53:30.020
Are you or your colleague trying to learn Python?

00:53:30.020 --> 00:53:34.600
Have you tried boring books and videos that just cover topics point by point?

00:53:34.600 --> 00:53:40.720
Check out my online course, Python Jumpstart by Building 10 Apps, at training.talkpython.fm.

00:53:40.720 --> 00:53:42.620
And have a great time learning Python.

00:53:42.620 --> 00:53:49.440
You can find the links from today's show at talkpython.fm/episode slash show slash 52.

00:53:49.440 --> 00:53:51.720
And be sure to subscribe to the show.

00:53:51.720 --> 00:53:53.780
Open your favorite podcatcher and search for Python.

00:53:53.780 --> 00:53:55.100
We should be right near the top.

00:53:55.100 --> 00:53:59.160
You can also find the iTunes and direct RSS feeds in the footer of the website.

00:53:59.160 --> 00:54:03.980
Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.

00:54:03.980 --> 00:54:07.040
You can hear the entire song on talkpython.fm.

00:54:07.040 --> 00:54:08.800
This is your host, Michael Kennedy.

00:54:08.800 --> 00:54:10.080
Thanks so much for listening.

00:54:10.660 --> 00:54:12.100
Smix, take us out of here.

00:54:12.100 --> 00:54:14.560
Stating with my voice.

00:54:14.560 --> 00:54:16.340
There's no norm that I can feel within.

00:54:16.340 --> 00:54:17.540
Haven't been sleeping.

00:54:17.540 --> 00:54:19.200
I've been using lots of rest.

00:54:19.200 --> 00:54:22.040
I'll pass the mic back to who rocked it best.

00:54:22.040 --> 00:54:23.820
on first developers

00:54:23.820 --> 00:54:24.600
of color

00:54:24.600 --> 00:54:25.480
ٌ

00:54:25.480 --> 00:54:27.500
For the content