#52: EVE Online: MMO game powered by Python Transcript

00:00 Have you ever played a massively multiplayer online game?

00:02 My first experience with these types of games was with text-based role-playing games called MUDs back in the early 90s.

00:09 Well, things have come a long way since then.

00:11 Games such as EVE Online have hundreds of thousands of players exploring, trading, and battling within a universe of over 7,000 star systems.

00:19 Gameplay in EVE Online consists of beautiful 3D spaceflight within a dynamic universe with many real-world players.

00:26 You may have played EVE Online, as it's one of the first major MMOs released back in 2003.

00:31 But did you know that Python is at the core of the game, playing a critical role in the backend infrastructure as well as a major role in the client-side game itself?

00:39 On this episode, you'll meet Christian Sigibirgsen from CCP Games to dig into Python at EVE Online.

00:45 This is Talk Python to Me, episode number 52, recorded March 16, 2016.

00:50 Welcome to Talk Python to Me, a weekly podcast on Python.

01:18 The language, the libraries, the ecosystem, and the personalities.

01:21 This is your host, Michael Kennedy.

01:23 Follow me on Twitter, where I'm @mkennedy.

01:25 Keep up with the show and listen to past episodes at talkpython.fm.

01:29 And follow the show on Twitter via at Talk Python.

01:31 This episode is brought to you by Hired and SnapCI.

01:35 Thank them for supporting the show on Twitter via at Hired underscore HQ and at Snap underscore CI.

01:42 Kristen, welcome to the show.

01:43 Thanks for having me.

01:44 We're going to have a really cool conversation about games and massive scale of just online applications in general, right?

01:52 Yep.

01:52 I'm sure that we will.

01:53 You guys are doing amazing stuff.

01:55 Before we dig into that, though, what's your story?

01:56 How did you get into programming in Python?

01:58 I came kind of roundabout way into game programming.

02:01 I actually applied for a position at CSB as a game designer.

02:04 And I started here as a game designer, worked here for two years.

02:08 But I've always kind of been interested in programming.

02:11 So I started poking around, fixing things.

02:13 And then after that, it was kind of no turning back because programmers are always in such a short demand that was so much demand.

02:21 That is that I just kind of pulled all of my time over into programming.

02:25 And then I kind of just kind of worked my way into being my current position, which is technical director of EVE Online.

02:33 And I did that by just attacking every problem that I saw and never giving up on them.

02:39 That's really cool.

02:39 And you guys do mostly Python there.

02:41 Is that correct?

02:42 Yeah.

02:42 Majority of the programmers in EVE work in Python.

02:46 We do have some stuff written in C++ as well as most of our DBA SQL.

02:52 But I'd say like at least 75, 80, 80 percent of our programmers spend like vast majority of the time in Python.

03:00 Okay, excellent.

03:01 And how did you get into that originally?

03:02 Is that something you learned when you first started there?

03:05 Did you know that before you started?

03:06 No, I'd known about CCP.

03:08 I knew they were developing in Python.

03:11 But the only thing I knew about Python was that white space matters, which I thought was ridiculous at first.

03:17 But now I'm thinking it's one of its greatest features because you're going to have the white space that way anyway.

03:23 So you might as well have it mean something.

03:25 It also kind of forces you to be consistent on it.

03:30 Yeah, that's an interesting point.

03:31 Like anytime you look at professional code, it's always structured in a way that resembles the way that Python is structured.

03:38 You know, if you look at C++ or C Sharp or even JavaScript when it's well written, the white space is in a format such that it could matter, but it doesn't.

03:48 Right.

03:48 And so I had exactly the same feeling when I learned Python the first time.

03:53 I'm like, this four space thing, this is insane to me.

03:56 Like, where are the curly braces and the semicolons?

03:58 What is going on?

04:00 And then I worked in Python for a while and I went back.

04:03 I had been coming from C Sharp and I went back to C Sharp and I thought, wait a minute.

04:07 Why am I typing all this crap?

04:09 This is way less fun to type than I remember.

04:11 And it's because I was spoiled by the structure of Python, which really surprises me.

04:16 And you can kind of see it from the code when you read like Java code or C Sharp code and stuff like that.

04:21 When you have a lot of indentation, you also have a lot of like white space beneath it where you have closing all of those brackets, all of those curly brackets.

04:29 And it's also weird because if you see somebody write Java code, for example, and it has an if statement and then two indented lines, but no curly brackets, I can guarantee you that program met both of those clauses to be within the if clause.

04:45 Absolutely.

04:45 There was some bug.

04:47 I think it was in OS X about a year and a half ago.

04:52 A really bad like SSL man in the middle bug.

04:56 And it was exactly that.

04:58 It was like there was an if statement and they didn't have the curly braces, but it was indented.

05:03 So people thought it was in one case, but it was actually in a different one.

05:06 Right?

05:06 Yeah.

05:07 I thought this exact thing and I thought this would never have happened in Python.

05:12 How interesting.

05:13 We'd have different problems, surely.

05:15 But yeah, we might have other problems, but the indentation problem, not so much.

05:19 Yeah.

05:20 And I kind of always liked that about the Python philosophy is that what you see is kind of what you,

05:28 the code, the whole idea is that the code is readable.

05:31 And what you see on the screen is what actually happened.

05:34 Not some like, aha, you forgot the curly brackets.

05:37 I say it a lot on the show.

05:39 Like you hear about things like JavaScript gotchas and, you know, like languages like C, the mistakes

05:45 that people make when they're new to the language that are really bad and not obvious.

05:49 And you just don't hear about that concept in Python.

05:52 Like people make mistakes, but they like don't work or something.

05:55 Right?

05:55 The only thing I can think of, and I love showing people that are new to Python this, is the

06:01 mutable keyword.

06:02 I love that bit.

06:05 Nice.

06:06 Yeah.

06:06 So let's talk a little bit about EVE Online.

06:08 EVE Online is a massive multiplayer online game that has to do with sort of like space exploration,

06:17 colonization, like a science fiction space future, right?

06:21 Right.

06:22 Yeah.

06:22 It's a massively multiplayer online game.

06:25 We focus a lot here on the kind of the player interaction.

06:28 We care a lot about how people interact with each other.

06:32 So we, and we don't try to interfere that much about it.

06:36 But in the end, it's a game where you fly a spaceship, you gather resources, you, you can

06:41 shoot other people, you can shoot non-player controlled ships, NPCs.

06:45 The biggest selling feature is the whole dynamic market.

06:49 We have like a very realistic market.

06:52 Yeah.

06:52 There's kind of like an economy in the game that is live and not programmed, but it's like

06:57 organic in a sense.

06:58 Yeah.

06:58 Yeah.

06:58 Yeah.

06:59 And it mimics the real world economies really.

07:02 And you can see the same behavior there.

07:05 And we actually have like a professor in economics working just analyzing the data.

07:10 We still analyze a lot of the data, like the market activity, just to know the health of

07:15 the game and what's happening.

07:16 Okay.

07:17 That's very cool.

07:18 Do you have like machine learning type stuff going on in the background or is it more procedural

07:23 and you understand it and you tweak the algorithm?

07:25 We had, so for stuff that NPCs like the, just we sold on the market, we had something where

07:33 we controlled the supply.

07:34 We just created the item and sold them so-called like NPC goods.

07:38 And what we did there is that we had something where you bought off the supply.

07:43 We added a new supply, but at a higher price.

07:46 And if it bought a lot of the supplies, we reduced the price.

07:50 So it kind of balanced itself out, which created kind of like trade routes.

07:54 But that's kind of not the most interesting thing about the market.

07:58 The most interesting thing about the market is the player versus player trading where players

08:03 sell items to other players.

08:05 We don't need any machine learning.

08:07 We don't need any fancy mechanics.

08:09 The whole dynamic part of it is something that the players provide themselves just by speculating

08:14 on the market and stuff like that.

08:16 Yeah.

08:16 Very interesting.

08:17 Like you just have a bunch of people interacting and that is like the artificial intelligence,

08:21 although not artificial really.

08:23 That is what creates all the kind of variants and dynamics and the interesting conversations.

08:29 Like just this other day, somebody showed a fake screenshot of an item, a very rare item

08:36 from our test server saying that they were going to be dropped in the next release, which plummeted

08:42 the price on that thing.

08:43 So just by the market speculation that we might be doing something influenced the price on that

08:49 item.

08:50 Wow.

08:50 Did that person go and buy a bunch of them all of a sudden?

08:52 I assume so.

08:53 Yes.

08:54 But people do that a lot.

08:56 And they try to buy all the resources of a certain type just to create a shortage of them.

09:03 And we have amazingly clever group of players that do that.

09:08 Players that have like economist boards and they have IT departments and all sort of stuff.

09:15 Have you had instances where people are selling items in the game, but in real life for real money?

09:21 It's a constant problem, really.

09:22 We don't support it, obviously.

09:25 It's very tricky to stop what we work.

09:28 We try to do it as much as we can.

09:30 And we've had a lot of success in it as well.

09:32 But people always try to sell it.

09:34 Sure.

09:35 I guess that would mess up your algorithm, right?

09:37 Because that's basically extraordinarily high value transaction happening invisible to your

09:42 economy in a sense, right?

09:43 Like a black market to some degree, I guess.

09:45 Yeah, yeah.

09:46 But that's not the only reason.

09:47 It's also we don't like this kind of it.

09:50 It's not secure for our players either.

09:52 We also don't want you to be able to kind of buy a hat in the game and kind of skews the whole thing.

09:58 This is the philosophy of a game that was made before the last five years of in-app purchases in iOS and these types of games, right?

10:07 That has such a different philosophy.

10:08 Yeah.

10:09 The people that are the most successful in it, they just make a game that is fun.

10:14 And then they find a way to monetize that without it being.

10:17 Yeah.

10:17 And then, of course, there are the other games where you just literally have to pay dollars and then you win.

10:23 Yeah.

10:23 That is not the same type of game as the.

10:26 No.

10:26 As even these other ones, right?

10:28 I don't know.

10:28 It's impure in some way.

10:29 So let's talk a little bit more about the gameplay and then we can get into the technical bits.

10:34 So there's this sort of economy and this exchange of goods.

10:38 And there's a lot of like you had like 7000 star systems or something.

10:42 You travel around and you can harvest things.

10:44 So what other games is it like?

10:46 Like does it relate to like Warcraft Starcraft or other games?

10:51 Like can you give some analogies?

10:52 So people who maybe haven't seen it are kind of aware.

10:54 It's like a massive multiplayer online.

10:56 So obviously you're going to draw some comparison with games like World of Warcraft and all the games online and those sort of games.

11:04 Those are all kind of avatar based games.

11:06 We like to think of them as more of a kind of a theme park kind of game where you go into the game and you have a bunch of rides where Eve is more of a sandbox kind of game.

11:16 You can imagine you go to a sandbox and you have a shovel and a bucket and you're just going to have to make some fun out of it.

11:22 Both genres are a lot of fun but kind of maybe cater to different needs and possibly just at different times.

11:29 Yeah it seems to me like Eve is a game that really appeals to people that are like there for the long term.

11:35 Like you're going to build something up over time and sort of grow it and so on.

11:39 Yeah and we do have very loyal subscribers.

11:42 They've been with us.

11:43 Some have been with us for years.

11:45 And even some people that have since then quit playing Eve still show up at the Eve Fan Fest, the annual kind of gathering here in Iceland that we have.

11:55 Just because they're part of the community and like staying part of the community.

11:58 That's really cool that people are like so attached to the community there.

12:03 I mean how many players do you guys have?

12:05 Give us some stats if you can.

12:06 The peak is I think around 30 or 40,000 now.

12:10 Wow.

12:10 We were in the stats of like 300,000, 400,000 users I think.

12:15 Must I know.

12:16 Okay.

12:16 When you say massively multiplayer that's pretty massive.

12:19 Yeah it is.

12:20 I mean there are bigger games there.

12:22 I think at the time we ranked somewhere between a small multiplayer, massive multiplayer and a large one.

12:29 Yeah yeah sure.

12:29 Let's talk about Python.

12:31 Where does Python fit in the story of Eve?

12:34 When they started developing Eve they always wanted like a scripting language to go along with something a bit lower level like C++.

12:42 I think they just it was obvious at the time to choose C++.

12:45 But that was like back in 97 or something.

12:48 It's pre-2000.

12:51 There weren't a whole lot of options.

12:53 They were looking at JavaScript but they eventually found out about Python and started looking into stackless Python.

13:00 And they really liked the idea of the whole micro threads they had in stackless.

13:05 Yeah and when you got started this was like probably Python 2 maybe before that right?

13:11 I think it was Python 1.6.

13:13 Yeah yeah because I think Python 2 was the year 2000.

13:16 Is that right?

13:16 Yeah.

13:17 I think it is.

13:17 So when you actually look at a lot of the decisions that were made at CCP with Python you kind of have to color them in that light.

13:25 You have to look at the code and see okay so when was this written and what was available at the time.

13:30 Right.

13:31 The trade-offs you made have to consider like well should we use VB or this.

13:37 I mean like these are the other technologies that are around at the time right?

13:40 Like JavaScript.

13:41 Yeah.

13:41 There was no concept of this Node.js thing.

13:44 It was kind of a toy right?

13:45 So Python is a really good choice I think.

13:47 Yeah but it was also I think a pretty daring choice because Python was pretty new at the time.

13:52 It wasn't really that stable I would say and it didn't have like all the features that I think they would have needed.

13:58 It didn't have as good IDEs and stuff like that.

14:02 And I think it was daring but then again there wasn't like a whole lot of other options.

14:07 Not for doing like a lot of work in a scripting.

14:10 Right.

14:10 There are actually quite a few Python implementations these days.

14:15 And the show I just had last week was with Brett Cannon from Microsoft talking about Pigeon.

14:23 And we talked about things like PyPy and Pyston and Pigeon and all these different choices for alternate implementations.

14:31 And one I don't think we really spoke about was Stackless.

14:34 But Stackless was one of the few alternative options you had back in that time frame right?

14:40 Yeah yeah there wasn't a lot of other implementations I think.

14:43 Then there was just the Vanilla Python and Stackless maybe a few others but yeah.

14:48 Yeah not many like things like Jython and Iron Python are like circa 2006, 2008 and then onward right?

14:54 So yeah.

14:55 Why don't you tell everyone what Stackless Python is?

14:57 Like what is it?

14:57 What's the advantage of it?

14:58 Why did you guys choose this over just CPython 1.9 or whatever it was?

15:02 The biggest advantage of Stackless Python that people saw here was the kind of the existence of micro threads.

15:09 Just very lightweight worker threads that you can even start up and they could do one thing.

15:14 The idea being that you could be cold would be a lot easier to read basically.

15:20 You could just start up like this one task and it just does this one thing.

15:24 Yeah well if you're dealing with 40,000 concurrent connections.

15:27 Yeah.

15:28 A single threaded you know sort of maybe forked off the 10 or 20 process type of system may well not scale the way you need it right?

15:35 So something like this is really helpful yeah?

15:38 Yeah that was the biggest reason for choosing this over Vanilla Python I believe.

15:43 Are you guys still using it?

15:44 Yeah we still use Stackless.

15:46 We kind of wish that the rest of the world would have followed us into Stackless but sadly that didn't happen.

15:51 And now we have some alternatives in Python so we still are kind of tied to Stackless Python in terms of code.

16:00 Yeah it would probably take a pretty major rewrite yeah?

16:03 I'm not really sure how major it would be but it would be pretty substantial by the look of it.

16:08 Right.

16:09 So maybe let's talk a little bit about the architecture infrastructure type of bits.

16:14 Could you maybe talk about like the different systems and stuff that's going on in the back end?

16:19 I'm assuming that the front end the thing that runs on my OS X machine or Windows that is the graphics and the actual client side game.

16:27 It's mostly C++.

16:29 Is that a fair assumption?

16:30 Right.

16:30 Most of the most of the highest high performing stuff is C++ like the graphics engine is C++.

16:37 The physics engine is C++.

16:39 The DP layer is C++.

16:41 Then we kind of have a nebulous layer that handles like files and various things, clocks and stuff like that called blue which is like our own dimension.

16:51 We have various third party libraries written in C++ as well.

16:56 Yeah.

16:57 So most of that is done in C++.

16:59 Most of the stuff that we for someone saw that would be a problem.

17:04 We also have like a minor things.

17:07 There's a feature called planetary interaction where we kind of have a layer over the planets that shows like resource distributions which is implementing the C++ because we thought it would be very performant.

17:21 And which would have had a performance problem I think in Python.

17:25 But we wrote that in C++.

17:27 So we have some options if we want to do something to perform really really fast we can write in C++.

17:32 Sure.

17:32 And do you have anything on the client side that's Python?

17:35 Or is that all backend stuff?

17:36 Yeah.

17:36 Yeah.

17:36 Yeah.

17:37 The most of the UI implementation is in Python.

17:40 Like it falls on to like C++ some rendering but for the most part UI programmers write stuff in Python.

17:47 Oh that's really interesting.

17:48 Yeah.

17:49 This episode is brought to you by Hired.

18:02 Hired is a two-sided curated marketplace that connects the world's knowledge workers to the best opportunities.

18:08 Each offer you receive has salary and equity presented right up front and you can view the offers to accept or reject them before you even talk to the company.

18:16 Typically candidates receive five or more offers within the first week and there are no obligations.

18:20 Ever.

18:21 Sounds awesome doesn't it?

18:22 Well did I mention the signing bonus?

18:24 Everyone who accepts a job from Hired gets a $1,000 signing bonus.

18:27 And as Talk Python listeners it gets way sweeter.

18:30 Use the link Hired.com slash Talk Python to me and Hired will double the signing bonus to $2,000.

18:36 Opportunities knocking.

18:38 Visit Hired.com slash Talk Python to me and answer the call.

18:42 When I thought about how you guys were using Python I of course thought all right the back end is probably a lot of Python and things like that.

18:52 But like I said the front end mostly C++.

18:54 Can you elaborate on that?

18:56 Like are they using a custom sort of plug into your own rendering engine or are they using some kind of widget windowing system or maybe talk about it a bit?

19:05 This is all kind of custom made here at C++ so the low level objects like the drawing of the stuff is in C++ and then we can actually just and I think at least it used to be the case that you would just kind of inherit from them.

19:20 The Python object could inherit from the C objects and you would.

19:23 I can't remember what we were using for it whether it was boost or.

19:27 Sure.

19:28 That sounds really really cool.

19:29 How about on the server side like what type of interaction does the game from the client side actually have with the server and what does a request sort of look like from sort of systems and infrastructure and so on?

19:40 Right.

19:40 We have our own kind of a networking layer which we call Matsunet and it kind of handles the server and client communication as well as communication between nodes on the cluster.

19:53 Both from the proxies to the sole nodes as we call them which are really just the worker nodes where everything happens.

19:58 The proxies sit out in front and receive requests from the client as well as the communication between nodes as well.

20:07 This is all done in Python actually.

20:09 Okay.

20:09 Yeah.

20:09 No, no.

20:10 It was rewritten some time ago.

20:12 C++.

20:13 But with the Python layer on top so you can know this.

20:16 Right.

20:17 Right.

20:17 Very cool.

20:17 And is this in your own set of data centers or are you hosted in like some kind of cloud place?

20:23 Like AWS or something?

20:24 No, no.

20:24 It's in our own data center and we actually moved data centers recently.

20:28 There's a big blog on that on the website.

20:30 It might be interesting for attack might be able to read that.

20:34 It was actually quite an operation.

20:35 We had to both move data centers and we had to buy whole new hardware in it.

20:42 And we were just moving between like London.

20:45 And then I heard you say you were using SQL server somewhere around there.

20:49 Is that right?

20:50 Yeah.

20:50 That's correct.

20:51 Okay, cool.

20:51 Are you hosted on Windows or on Linux?

20:53 We're hosted on Windows.

20:54 We're probably one of the rare instances where we actually use Python and Windows, I guess.

20:59 Yeah.

20:59 Yeah.

20:59 Very cool.

21:00 Okay.

21:01 The world has changed since 1997 in terms of databases pretty dramatically, especially

21:06 in the last five years.

21:07 I'm sure part of the reason that you guys are on a relational database has to do with the

21:12 fact that you started on a relational database.

21:14 But do you think if you could start totally from scratch, would it make more sense to be

21:18 on like a NoSQL database or stick with a RDBMS like SQL server?

21:23 I think if I were forced to choose like a single database, I would probably still go with a

21:29 relational database.

21:30 We've made so many changes recently in terms of this microservices architecture that I wouldn't

21:37 really see a necessarily need to do that.

21:40 I would try to split up the clusters a bit more into these functional areas than when I'm

21:46 storing data that needs a relational database.

21:49 I would store it in a relational database, otherwise a NoSQL database or a graph database

21:55 or whatever.

21:55 But I think if I had like a 1db catch all, I would still go with relational database.

22:01 Makes a lot of sense.

22:02 The world is definitely moving towards this concept of microservices.

22:06 And I don't think I've spoken very much about that on the show.

22:11 I think maybe a little bit when we had the Docker guys on way back in show nine, but also

22:17 had a conversation with Sam Newman that actually the audio went so bad we couldn't put it together

22:22 specifically about microservices.

22:24 I'm sorry about that, Sam.

22:25 But maybe you could just tell us really quickly for those people are like microservices.

22:29 Those sound like small services.

22:30 What's that?

22:30 Maybe describe that a bit.

22:32 It's essentially that it's small services.

22:35 It's this idea that you spin off a service for a specific need that your total ecosystem needs.

22:43 So in terms of EVE, then we, for example, have a market.

22:46 Maybe the market could be its own microservice.

22:48 Maybe it doesn't have to need to connect to the cluster.

22:51 The cluster would only need to know how to talk to that microservice and that Microsoft

22:55 service to talk to that cluster.

22:57 I don't think I would really, really do it justice, but it's really just a matter of that,

23:02 of breaking up your kind of the total application into like multiple smaller services.

23:07 Yeah, the thing that I think that's interesting about that is, you know, I ask you about what

23:12 is the right database?

23:13 And the answer may well be, well, certain types of architectures that are proving to be powerful,

23:20 like these microservices, the answer could be many small ones.

23:23 And then it even matters less because when you're, if you're, let's say, just modeling only the

23:28 economy, which I'm sure is still massive, but just the economy, not every possible thing

23:33 that could go on in the game, then you might have smaller database performance needs and

23:37 so on, right?

23:38 Yeah.

23:38 And in some cases of data, you actually need everything to be always up to date, but maybe

23:45 sometimes you just need like eventual consistency on that data.

23:48 So you can pick database solutions based on the need for that particular feature rather than

23:55 choosing it for everything and hoping it all matches.

24:00 Having said that, I don't suggest that everybody makes changes to their architecture from like

24:05 single monolithic application to a microservices.

24:08 It's something you, you exchange one problem for a set of other problems, but it can make

24:13 a lot of sense.

24:14 And one of the things that I love about it is that it allows you to change, you know, decisions

24:20 that you thought you were kind of rooted into that you thought you would never, ever going

24:25 to change.

24:25 Yeah, that's, that's interesting.

24:27 And it's good advice, right?

24:28 Like design patterns and these larger architectural patterns are super powerful, but they almost

24:34 always have a context that goes along with them.

24:36 If you have a billion lines of code and this level of huge complexity, you can make that complexity

24:42 more manageable by breaking it apart.

24:44 But if you've got like 5,000 lines of code, maybe a microservice just means more servers and

24:48 things to maintain, right?

24:50 Right.

24:50 It's more service.

24:51 It's more things to maintain.

24:52 It's in theory that those servers should require less maintenance as well.

24:58 And they're easier to comprehend.

25:00 But you also need like infrastructure.

25:02 Maybe you need to have a message queue in there.

25:04 Maybe you need to add in a bunch of technology that you wouldn't have needed if you had a monolithic

25:09 application.

25:10 Yeah.

25:10 And performance as well, right?

25:11 Like if previously the call to get data about like the price of an item in your economy

25:17 was within the same machine or, or something to this effect within the same process even,

25:23 but it still hits like a database maybe.

25:25 That's probably a lot faster than actually going across the network, finding another machine,

25:30 doing a whole HTTP thing, and then asking that same question, right?

25:32 Yeah.

25:33 Absolutely.

25:33 I think, yeah, you just have to, you have to tread very carefully.

25:38 I think it's easy to get really, really excited about microservices and, but you really have

25:44 to make sure that's the correct solution for you.

25:47 Yeah, absolutely.

25:47 I certainly see those tied together with Docker and the whole container space.

25:51 But again.

25:52 Absolutely.

25:53 They've made this thing a lot easier.

25:55 Nice.

25:55 So one of the things that you said that you guys had done, I thought was pretty interesting

25:59 is you said you had written your own importer.

26:03 Is that, like if I say import space some module name, like import OS, you rewrote what that meant?

26:08 Well, sort of.

26:09 It's not really maybe such an importer as much as like a bootstrap.

26:14 Okay.

26:15 I don't know actually what the state was in Python 1.6, but I'm thinking maybe it didn't actually

26:22 know how to import packages and that whole dunder in it.

26:26 Oh, interesting.

26:27 Okay.

26:27 So that was pretty interesting.

26:29 And so what they wanted, and they wanted like a shallow namespace for these things.

26:33 So what they actually did was they wrote something that would just loop over the file.

26:37 It would, you know, read, execute that file and import it into like a namespace that they

26:44 created.

26:44 So they basically just injected into CSTOck modules.

26:47 Interesting.

26:48 Okay.

26:49 So what was the benefit there?

26:50 Do you remember?

26:50 One of the benefits was that we'd have auto-reloading of modules when we changed them.

26:56 And it was actually quite nice to work with E at the time.

26:59 And it still works that way.

27:01 It's just now it's a Python solution instead of like a home-cooked one.

27:06 But just changing something in code, you don't have to restart your survey, don't have to

27:11 restart your clients or anything.

27:13 You just re-execute whatever you're doing.

27:16 You click the button again.

27:17 Okay.

27:18 Now you got my attention.

27:19 That's pretty intense, actually.

27:20 Yeah.

27:21 And that's still the case today.

27:23 But now it's like with a Python implementation of it.

27:26 Did you have to do...

27:27 I guess you almost have to.

27:28 Either you say you can't use these types of features or you have to deal with things like

27:32 static variables and static class fields that have been set previously.

27:36 And then if you reload the module, what did you do there?

27:39 It wasn't completely foolproof.

27:41 Like if you...

27:42 And it still isn't.

27:43 If you already bound the method to something, it doesn't actually reload that.

27:47 If you like pass in a method of an object into something else, then it's already bounded

27:52 and it doesn't reload it.

27:54 But for the most part, for the most code you are changing, it actually just changes it.

27:59 I've had issues where some data was actually stored on a module level, like a dictionary

28:05 on a module level.

28:06 And whenever that got re-imported, it would clear that.

28:09 And those sort of things.

28:11 I mean, it happens.

28:13 You need to restart your server and client.

28:15 And when you're trying something out and something funky comes up, that's something what you do.

28:19 You just restart everything and try again.

28:21 Yeah.

28:21 Interesting.

28:22 So you have like two levels.

28:23 You're like, let's just try to hot patch it.

28:25 If that doesn't work, give it a hard kick, right?

28:27 And we've done that in production as well.

28:29 Like if there's a problem that we really want to fix, and that's actually one of the drawbacks

28:34 of having a monolithic application.

28:36 If I have a problem in the market, I need to take everything down to patch it up.

28:41 Right.

28:41 We've actually live fixed stuff like that.

28:44 Just going in console and just creating a new method and attaching that to the class.

28:49 Okay.

28:49 That sounds really, really powerful and interesting.

28:51 Is any of that sort of open source or blogged or written about that people can check out?

28:56 No, no, no.

28:57 None of this is open source.

28:59 I don't think we've ever blogged about this.

29:01 I think.

29:01 Okay.

29:02 Well, it sounds interesting.

29:03 I know.

29:03 I talked to David Beasley.

29:05 I think it's show 12 about modules and packages and this whole concept.

29:09 And it was like, oh, we're not really sure that this is going to work.

29:12 If it can be done.

29:13 So to hear that you guys are doing it on this real app, this is very cool.

29:16 I think people might get kind of caught up in the idea that, well, it's not going to work

29:21 every time.

29:22 But that's okay when you're doing development.

29:23 You just want like rapid feedback, really.

29:26 Yeah.

29:26 If it works 95% of the time and it saves you a ton, then maybe it's worth it, right?

29:30 Yeah.

29:30 Yeah.

29:31 I guess coming back to the microservices thing, if you build for something like microservices

29:35 and you do need to sort of hard restart some piece, it probably has a smaller consequence

29:40 because you're not restarting the entire thing, right?

29:42 Yeah.

29:43 Well, but the flip side of that is that you have to write everything now as in nothing

29:48 is guaranteed to be up.

29:50 Yeah, right.

29:51 If you do that, then you kind of, you're always guaranteed to have something down.

29:56 And if your whole, if nothing works, when one of the bits is wrong, you're going to

30:00 have constant downtimes.

30:01 Yeah.

30:02 That's straight off again, right?

30:03 Yeah.

30:04 It's a trade off.

30:05 Like, it's not a silver bullet.

30:06 And I think people have been pretty, pretty vocal about that.

30:10 Like, the experts on this have been pretty vocal about that.

30:12 And I think every paper and every talk I've seen about microservices always starts with

30:19 that.

30:19 This is not like a silver bullet that will fix all your problems.

30:22 Okay, cool.

30:23 So one of the things that happens in the game is there seems to be a lot of people

30:29 that group together to form like a fleet.

30:31 And then sort of in real time with voice communication, they'll go in and they'll have like these fleet

30:38 battles, right?

30:38 How many people end up in those battles?

30:41 How many different players?

30:42 SnapCI is a continuous delivery tool from ThoughtWorks that lets you reliably test and deploy your code through

31:02 multi-stage pipelines in the cloud without the hassle of managing hardware.

31:07 automate and visualize your deployments with ease and make pushing to production an effortless

31:12 item on your to-do list.

31:13 Snap also supports Docker and M browser debugging, and they integrate with AWS and Heroku.

31:19 Thanks, SnapCI for sponsoring this episode by trying them with no obligation for 30 days by

31:25 going to snap.ci.com.

31:27 SnapCI slash talkpython.

31:28 It can, Lars, I think the largest we had was like 4,000 people, I think.

31:41 Wow.

31:42 Then it can get pretty laggy, but...

31:45 What's the sort of performance characteristics as you add more people?

31:49 I mean, it can't be linear, like 2,000 people versus 4,000.

31:53 It's more like combinatorial or something really expensive, right?

31:56 Like, how's that work?

31:57 We've done a number of optimizations over the years of that.

32:00 It actually performs quite well.

32:02 I'm not sure if it's linear, but it's not N2 or anything like that.

32:07 But it's also very difficult to predict on that because it matters a lot whether everybody

32:12 is shooting at a structure, maybe, that doesn't die for a long time, or if people are shooting

32:17 at each other and people are getting killed and then removed from that solar system.

32:22 That creates more load.

32:24 It matters if people are shooting from guns or using missiles.

32:28 Right, because guns presumably are instantaneous, but missiles sort of cruise through space.

32:32 Yeah.

32:33 Right?

32:34 You've got to track them for everyone and see them go by and so on.

32:36 Yeah, yeah.

32:37 They don't collide really with anyone except the person that they're meant to collide with.

32:41 That's one of the optimizations we did a long time ago.

32:45 There's a lot of factors.

32:46 It's very hard to say, like, you can look at a thousand people fleet fight and see no lag,

32:53 and you can see another one that has a massive amount of lag.

32:55 It's very hard to measure that.

32:57 One of the most reliable kind of metrics we have is the market up.

33:02 We have a single system that, you know how people are, they kind of just all gravitate.

33:07 And because a lot of goods are there for sale, people think it's worth it for them to put

33:12 their goods up for sale.

33:13 So it creates this market up, which is a solar system called Jita.

33:16 And there we have like a constant influx of people.

33:20 We had like a maximum of 3,600 people there the other day.

33:25 That's a lot.

33:26 Because of some server optimization we had been doing, the CPU actually never went above

33:30 50%, I think, on that.

33:32 Yeah, yeah.

33:33 That's really cool.

33:33 So you said one of the challenges you have with some of the scaling has to do with the dreaded

33:38 global interpreter lock or the GIL.

33:39 What's the story there?

33:40 We basically can segment our cluster up into, well, it's probably easier to explain by solar

33:47 system, right?

33:48 That's the smallest unit we can run.

33:51 We can't distribute the load within one solar system to multiple processes, really.

33:56 Right.

33:56 Okay.

33:56 That's going to be a bit problematic because that means we can only use really just one

34:00 core on the machine.

34:02 So it doesn't really matter.

34:03 And it's very problematic because the hardware industry is actually moving more towards more

34:07 core and maybe less clock speed.

34:10 What we really desire is more clock speeds.

34:13 Yeah.

34:13 I mean, the clock speeds, I think, are, if they're not slower, they're definitely not any faster

34:18 than what they were in like 2005, right?

34:20 They're not getting faster.

34:21 I can tell you that.

34:22 Yeah.

34:22 Yeah.

34:23 I mean, they may even be slower.

34:24 I know they got higher and they started to melt.

34:26 They kind of went down.

34:26 All right.

34:27 That's it.

34:27 Eight cores, lower, lower gigahertz, lower clock speed.

34:31 Yep.

34:31 Very interesting.

34:32 Would Docker type stuff or virtualization sort of let you sort of break that up, at least

34:38 not dedicate a whole machine to it, but say, okay, well, we're going to isolate, you know,

34:43 Windows.

34:44 I guess Docker is doing stuff with Microsoft these days, but maybe it's, I should say,

34:48 Windows containers that's coming out soon.

34:50 Would that be a way to partition this so that you can still get everything you want out of

34:54 those machines without dealing with the threading?

34:56 Yeah.

34:57 It's an interesting thought.

34:58 It still requires like a lot of programming work to actually do it, but you'd have to segment

35:03 what the cluster is doing within the solar system into like something logical units,

35:08 which you can scale, which doesn't chapter too much between each other.

35:12 We actually have an idea about that, but yeah, we'll see if we get there.

35:16 As it currently stands, we're not really that pressed for performance, or at least it's

35:23 not as high priority as some other stuff we want to do.

35:26 Right.

35:27 When you're doing performance analysis, like you just look at the slowest thing, the bottleneck.

35:31 Right.

35:32 If the stuff behind it is slower, but still fast, you know, slow, but faster than the bottleneck,

35:36 then who cares, right?

35:37 Yeah, right.

35:38 And there's something to be gained looking before we start looking into splitting really

35:44 this into multiple processes on the same note.

35:47 But we have like a better lead than that.

35:51 So one thing you said you're really into is test-driven development, unit testing, and so on.

35:56 And this code is super old.

35:58 You must have some interesting challenges writing tests for a code from this long ago, yeah?

36:03 Yeah, and the importer or bootstrapper, which we appropriately named Nasty at the time,

36:09 was of course like a huge, huge burden on us for that, because we couldn't really import

36:17 any of the EVE-specific code without using that Nasty importer.

36:22 And that Nasty importer really had to throw through everything and really just run a full-scale

36:28 client or a server.

36:29 Yeah, so you were almost like forced to the point where you're doing integration tests

36:33 because you had to start the whole thing anyway, right?

36:36 Exactly.

36:37 And we did manage to create like an interpreter that did that, but we had me and another program

36:44 who were writing tests, and I think we were up to like 100 tests, and they were taking 45

36:48 seconds to run, which was just terrible.

36:51 Yeah, that's pretty long for unit tests, right?

36:53 Yeah, yeah, and it's unusable if you want to use it for like a quick iteration feedback.

36:58 Yeah.

36:59 When developing, you really need something in less than a second time or like close to it.

37:06 But there was some initiative to actually get rid of Nasty.

37:09 They actually managed to do it in like one summer.

37:12 They managed to do it in, I think, two weeks.

37:15 Oh, nice.

37:16 And now testing is much easier?

37:17 Now testing is a lot easier.

37:19 It's still a lot of coupling, and there are other things other than creating our own

37:24 bootstrapper slash importer that we did, like shoving stuff into buildings and stuff like

37:30 that that make testing harder.

37:32 But usually always you can extract the code that you want to test out, and you write tests

37:40 for them.

37:40 Right, okay.

37:40 Nice to hear that you've got that sort of cracked open so you can test it.

37:44 Yeah, yeah.

37:45 It's been like once that came in and once I started being able to write tests like that,

37:51 it kind of really opened my eyes towards testing and development and automatic testing in general.

37:58 All right, cool.

37:58 So let's talk about a little bit about CCP, the company that makes EVE Online.

38:03 Yeah.

38:03 You guys are primarily based out of Iceland, right?

38:06 Right, mostly out of Iceland.

38:07 We have offices in Núkastúl, Shanghai, and Atlanta, but everybody that works on EVE is

38:14 stationed in Reykjavík.

38:15 Very cool.

38:15 I've never been to Iceland.

38:17 It's definitely one of the places I'd like to visit, though.

38:19 Yeah, it's definitely interesting, although I lived here most of my life, so...

38:23 It seems very natural to you, right?

38:25 Very normal, but...

38:25 It seems very normal.

38:26 Yeah, with all the winter and the volcanoes, and I think that'd be super cool to go check

38:32 out.

38:32 Nice.

38:33 So you guys use an Agile approach, is that right?

38:35 Do you use Scrum, or what's the story there?

38:37 Yeah, well, we started with Scrum, and then we kind of...

38:41 It's morphed into something that's like Scrum, but it's mostly...

38:45 Scrum is really the basis of it, but I think we allow every team to really put their own

38:52 flavor on it and do kind of whatever works for them.

38:55 Agile practices are really funny, because if you don't pay attention to them, they will

39:01 just deteriorate, and you'll start getting worse at them.

39:04 So it's something that just kind of requires constant putting.

39:07 And of course, it's part of the Agile to have retrospective and constantly be trying to

39:13 improve yourself.

39:13 Right.

39:14 I think a lot of people lose sight of that.

39:15 I mean, on one hand, there's not necessarily a single Agile process that is exactly right

39:21 for everyone, right?

39:23 I mean, that's a little bit about being flexible and Agile and so on there.

39:26 But then if you're going to go and sort of walk it up to the edge of the absolute minimum

39:30 amount of documentation, rules, structure, that means if you fall apart just a little bit,

39:36 you're probably falling out of the minimum amount of structure that you need, right?

39:39 Right.

39:40 It's a little bit like that.

39:41 I kind of like starting with Scrum.

39:44 I think Scrum has like, if you understand why the rules of Scrum, then you have the power

39:49 to change them as they make sense to you.

39:51 But I've seen also seen a lot of companies or heard from like a lot of my colleagues here

39:56 around Iceland is that they've implemented Scrum, but they really didn't get what Agile was.

40:03 So it just goes through the motions.

40:05 And then, hey, this Agile thing sucks.

40:07 It doesn't work.

40:08 Yeah, right.

40:09 Yeah, well, maybe you didn't follow it.

40:10 You guys said you also have a kind of interesting learning and development philosophy.

40:16 What's the story there?

40:17 Well, it's not.

40:18 I mean, we do try to foster kind of the culture of having like what we call a community of practice.

40:25 We had a Python community of practice, for example, but we kind of retired that because

40:32 people weren't really showing up that much.

40:34 But it's actually very difficult because people are so invested in their job.

40:41 They're so invested in what they're doing that they don't see that they can spare an hour here

40:47 to do stuff like that.

40:49 What we've been doing is maybe a bit weird, which is different than sending people to conferences

40:54 and getting consultants here is that we've been kind of running this video series.

40:59 Uncle Bob.

41:01 Oh, yeah.

41:01 Robert Martin's awesome.

41:02 Yeah.

41:02 We've been running that and we've been having like talks after it.

41:06 That I think has been very good.

41:09 I definitely recommend for companies to do that.

41:12 It's probably one of the cheapest learning and development you can get, like in terms of bang for buck.

41:17 Yeah, I totally agree.

41:19 I think one of the things that people, companies, teams, individuals, like on all the levels,

41:24 don't necessarily realize is if you just spend like half an hour a week or half an hour twice

41:31 a week making sure you polish your skills, you learn something new.

41:34 After a few years, that really adds up.

41:37 Absolutely.

41:37 There have been other initiatives in the past.

41:40 Like we had, like on Friday, people had code cutters where they would meet up and they would

41:44 get like a task and then go in groups of twos and they would do something like write these

41:50 features, but don't use classes.

41:51 Stuff like that.

41:53 So these initiatives, they come and go.

41:55 I had a kind of a mob programming session, which was all about pulling code out and write tests for it.

42:03 Very interesting.

42:03 You know, I don't think we've talked about mob programming on this show.

42:08 I know what it is, but I don't think we've ever mentioned it.

42:11 Can you just give me like the really quick elevator pitch?

42:13 Like what is mob programming?

42:15 Basically what it sounds like, it's like one guy and a keyboard and he takes a test and people shout at him.

42:21 But of course, there's one driver and he probably knows a bit where he's going and then he kind of just listens to,

42:28 okay, so what do you want to do here?

42:29 Do you want to create a new method here?

42:31 Do you want to create a class or whatever?

42:33 Would you say it's kind of like paired programming, but many people on the pair side?

42:38 Yeah, and probably a lot, lot less efficient, but it sparks up a good conversation.

42:43 Right.

42:44 It's more, it's not necessarily like paired programming.

42:46 The expected output is solid code that you're going to ship.

42:50 Yeah.

42:50 Where I think mob programming is more about mutual understanding.

42:54 Yes.

42:54 And skill development and so on, right?

42:56 Yeah.

42:56 Also just getting people to kind of voice their opinions about how code should be.

43:01 Because that's, we actually just don't do that.

43:03 We write code, we put it up to code review.

43:05 But without me understanding what your agenda or what your philosophy about code is,

43:11 then chances are we're going to disagree on a lot of like fundamental things.

43:16 I would say so.

43:17 Yeah.

43:17 It sounds like a cool approach you guys have going there.

43:20 So one thing I wanted to ask you about, do you have any stories about like when there was

43:24 some giant unexpected rush of new traffic or new customers, like maybe like, you know,

43:30 a new release came out that was really popular or something like that?

43:32 Well, we had the other day, we had a new item come out that was very popular on the market.

43:38 And then that meant that everybody brought to that note, that cheetah note to sell stuff

43:44 and buy stuff on it.

43:45 Way more than what we've usually been expecting.

43:48 The thing is, we have a cup on that note.

43:51 We have like, we allow like max 2,000 people, I think, on it.

43:57 Because if we get more, or when we used to get more, then the game would be pretty unplayable

44:04 for everybody involved.

44:05 So we thought it was better to actually limit the amount of people that could be in there

44:09 at any time.

44:10 And their experience would at least be a little bit better.

44:13 And people can always play and be in other systems.

44:17 But because of a number of performance, like a big performance upgrade we've done, we kind

44:23 of saw that the server wasn't actually...

44:25 So we started increasing the cup.

44:27 We increased it first by 2,500.

44:30 And then we started reaching that.

44:32 So we increased it.

44:33 So we're thinking about increasing it to 3,000.

44:36 But instead, the development director said, like, looking at the graph, I think we should

44:40 just go to 5,000.

44:42 We can support 5,000 based on these numbers.

44:44 So that's what we did.

44:45 And we got 3,600 people there.

44:49 The CPU never went above 50%.

44:51 But the memory was actually getting to the limit.

44:55 Wow.

44:55 And that was actually like a...

44:57 It's a weird thing because we've gone from being memory restricted on a PC to almost not

45:04 having to think about it at any point.

45:06 Eve has this weird thing where we restart the cluster every day at 11 o'clock.

45:12 So whatever memory we have cast, that gets flushed out.

45:16 So the programmers on Eve have never really considered memory so much.

45:22 So we just cast it and then we never release it because we've never really had to release it.

45:27 So it's weird that when you have a lot of people coming in, we just saw the memory increasing and it never went down.

45:33 Oh, wow.

45:34 Nothing ever released that memory.

45:36 Yeah.

45:36 What I think is interesting about that story is when I think of some system getting hit with lots of traffic and having unexpected growth, I see it as an all or nothing type of thing.

45:49 Like the website now has a lot of visitors and so it's busy or not.

45:52 But the way you guys have structured your world, it's like there's a little node over there somewhere where a bunch of people went into that part of the universe and now that part is too busy.

46:00 That's an interesting way to think about scale.

46:02 Like I suspect most places don't.

46:05 Yeah, yeah.

46:05 It's always been a...

46:07 And it's both...

46:08 It's very funny to look at the kind of the overall CPU utilization of our clusters and you can see like it's using 10% CPU on average.

46:20 But still, you know that there are some people, there are some node there that is 100% and people are experiencing lag and you're seeing 10%.

46:28 Yeah, it's almost like you need sort of a mountain range type graph instead of just a level or something like this.

46:34 Very cool.

46:35 And in our old data center, what we used to have is that we had one node that was...

46:40 We called it the Everest node.

46:42 It only ran that one JETA solar system and it was a bit beefier than all the other nodes and it was just dedicated to running that particular node.

46:52 Yeah.

46:52 In the new cluster, I think all the machines are better than that old machine.

46:57 That was the good one and now it's old and crusty.

47:01 Very cool.

47:01 So I do have just a few more questions before we kind of wrap up the show.

47:04 Yep.

47:05 I was told on Twitter, thanks, I think it was Kyle that sent a message that said, I should ask you about the big red button.

47:11 Yeah, I always get that.

47:14 I'm a bit of a, I guess, infamous in the EVE universe.

47:18 So what happened is that I was just happily developing on my local clusters and I needed to do something.

47:24 Normally when I shut down the cluster, I just kill the process and start it off again.

47:29 But there is some logic that happens when you shut down the cluster, some persisting logic that I wanted to test.

47:35 So I go to the web page of the cluster and I shut it down properly or properly.

47:41 It's called an emergency shutdown and I gave it a minute timer.

47:44 So I did that and then I go back to my client and I don't see the normal message that I get when I shut down the cluster.

47:51 Like the cluster is about to shut down.

47:54 So I look at the page again and I realize I'm on the production clusters and I'm shutting down the production clusters.

48:00 Oops.

48:03 So I, and the only way to cancel it is to go to that page again.

48:08 So I did that and I canceled and I didn't know how long time had passed.

48:12 I knew I put a one minute timer there, but I just frantically clicked that button.

48:16 And it turns out I managed to stop the shutdown.

48:19 It never actually shut down.

48:21 But everybody that was logged in got that message that the EVE was shutting down and people really don't like that.

48:28 I'm sure they don't.

48:29 They really don't like that.

48:31 So they went on the forums immediately and said, like, CCP, what the hell?

48:35 Why are you shutting down the server?

48:37 Why do you have a one minute timer?

48:38 Because people want to get their assets to safety and stuff like that.

48:41 Right, of course.

48:42 Yeah.

48:42 So I did what just came naturally to me.

48:45 I just, I just went into that forum thread and saw, look, sorry, this is my bad.

48:50 I thought I was shutting down my development cluster.

48:53 And I'm being ridiculed by my coworkers now.

48:56 Yeah, I'm sure.

48:57 But it's really, really funny.

48:59 Because EVE players are like that.

49:01 They do get very emotional about it.

49:03 But they also are pretty cool people.

49:05 And honest mistake like that, they just let it slide.

49:09 Yeah.

49:09 And they're pretty technically savvy.

49:11 They are technically savvy.

49:12 They understand.

49:12 Some of them probably did this once.

49:15 And I think this even happened like a week after when someone in operations of QA, I can't remember, was going to shut down the test server and shut down that.

49:25 But he didn't go to the forums and apologize.

49:27 Yeah.

49:27 And didn't claim, you know, public responsibility and say something.

49:31 And I think that people like to remember that story as something.

49:35 Yeah.

49:37 This is messed up.

49:38 But I'd like to think that I just, I did the right thing by apologizing to people and explaining just exactly what went wrong and why I was being stupid for doing it.

49:51 Yeah.

49:52 It all happens to us.

49:53 One time I was teaching a training class.

49:55 And I'll try to not put any details about any place.

49:58 And I was somewhere in Chicago, let's say.

50:01 And I was teaching this training class at a financial place.

50:04 And they, for some reason, didn't have a proper room that they could use for all the 20 people that were in my class or whatever.

50:10 So they had given us this weird, like, secondary server room where there's all these servers along the wall.

50:16 But they weren't, it wasn't obviously the main data center.

50:19 It was just, like, servers that ran internals within the company, you know?

50:22 Yeah.

50:23 And I had this little squishy ball thing that I would use to, like, work out your hands or whatever.

50:27 And I was walking around.

50:28 It slipped out of my hand.

50:30 And it was odd shaped.

50:31 So it bounced like a crazy ball.

50:33 And it bounced along the students' desks, over along the wall, hit the wall, fell down, and hit a power strip that killed an entire wall of servers.

50:40 Oh, my God.

50:42 You hear it go quieter.

50:44 Oh, my.

50:44 That is bad.

50:45 So we quick turn them all back on and just go back to the class.

50:48 And about 10 minutes later, some guys rush in, like, what happened to the servers?

50:52 Everyone's like, we don't know.

50:54 So, yeah.

50:57 I've turned off the servers as well on accident.

50:59 Yeah, yeah.

51:01 Luckily, the students had my back.

51:03 All right.

51:03 Two quick questions before we go.

51:05 If you're going to write some Python code, what editor do you use?

51:08 I use PyCharm now.

51:10 I've used Sublime and Wim in the past.

51:13 But I think in terms of, like, a good IDE, I think nothing is better than PyCharm.

51:18 Yeah, I'm with you on that.

51:20 I love PyCharm.

51:21 That's what I use as well.

51:23 Awesome.

51:23 And then if you're going to go and look at PyPI, all the packages, I think there's like 75,000 of them now.

51:30 Are there ones that are super useful that you think maybe people don't know about that they should?

51:34 Well, what to choose from?

51:35 I love the request packets.

51:37 I think it kind of puts the simplicity in making, like, HTTP call.

51:42 Yeah.

51:42 What's interesting about that is it's, like, so simple, you can almost not appreciate how cool it is.

51:48 Exactly.

51:48 I really love that packet.

51:50 Mostly because I have probably used the other libraries.

51:53 And they are probably more flexible and stuff like that.

51:56 But sometimes you just want to make a get request on a resource.

51:59 Yep.

52:00 Request.get.

52:01 And you're good to go.

52:02 Awesome.

52:03 So if people want to get started with EVE Online, how do they do it?

52:06 Visit eveonline.com.

52:08 That would be the first step.

52:10 And I want to highlight a feature of that is that you can download the launcher.

52:14 And it starts downloading the game.

52:17 But it only downloads the stuff it needs.

52:19 So it only downloads the code and, like, a minimum set of resources.

52:22 So you can be within the game in, like, 10 minutes from there.

52:26 Oh, yeah.

52:26 That's awesome.

52:27 Yeah.

52:27 I definitely want to start playing and check it out.

52:29 It sounds really cool.

52:30 And after that, it just downloads the resources as it needs it.

52:34 So if you want to try it out, go to eveonline.com.

52:37 Sign up for a free trial and check it out.

52:40 It should take you less than 20 minutes on a decent minute connection.

52:44 Very cool.

52:45 Kristen, it's been fun to learn about what you guys have going on with Python.

52:48 A lot of cool stories.

52:49 Thanks for sharing them.

52:50 Yeah.

52:50 Thanks.

52:51 Thanks for having me.

52:52 Yeah.

52:52 Take care.

52:52 Bye-bye.

52:54 This has been another episode of Talk Python to Me.

52:56 Today's guest was Kristen Siger-Bergsen.

52:58 And this episode has been sponsored by Hired and SnapCI.

53:01 Thank you guys for supporting the show.

53:03 Hired wants to help you find your next big thing.

53:05 Visit Hired.com slash Talk Python to me to get five or more offers with salary and equity presented

53:10 right up front and a special listener signing bonus of $2,000.

53:13 SnapCI is modern, continuous integration and delivery.

53:18 Build, test, and deploy your code directly from GitHub.

53:20 All in your browser with debugging, Docker, and parallelism included.

53:24 Try them for free at snap.ci slash Talk Python.

53:27 Are you or your colleague trying to learn Python?

53:30 Have you tried boring books and videos that just cover topics point by point?

53:34 Check out my online course, Python Jumpstart by Building 10 Apps, at training.talkpython.fm.

53:40 And have a great time learning Python.

53:42 You can find the links from today's show at talkpython.fm/episode slash show slash 52.

53:49 And be sure to subscribe to the show.

53:51 Open your favorite podcatcher and search for Python.

53:53 We should be right near the top.

53:55 You can also find the iTunes and direct RSS feeds in the footer of the website.

53:59 Our theme music is Developers, Developers, Developers by Corey Smith, who goes by Smix.

54:03 You can hear the entire song on talkpython.fm.

54:07 This is your host, Michael Kennedy.

54:08 Thanks so much for listening.

54:10 Smix, take us out of here.

54:12 Stating with my voice.

54:14 There's no norm that I can feel within.

54:16 Haven't been sleeping.

54:17 I've been using lots of rest.

54:19 I'll pass the mic back to who rocked it best.

54:22 on first developers of color ٌ For the content