Monitor performance issues & errors in your code

#52: EVE Online: MMO game powered by Python Transcript

Recorded on Wednesday, Mar 16, 2016.

00:00 Have you ever played a massively multiplayer online game? My first experience with these types of games was with text-based role playing games called MUDs back in the early 90's. Well, things have come a long way since then. Game such as Eve Online have hundreds of thousands of players exploring, trading, and battling within a universe of over 7,000 star systems. Gameplay in Eve Online consists of beautiful 3D space flight within a dynamic universe and many real world players.

00:00 You may have played Eve Online as it's one of the first major MMOs released in 2003. But did you know that Python is at the core of the game, playing a critical role in the backend infrastructure as well as a major role in the client side game itself!

00:00 On this episode, you'll meet Kristinn Sigurbergsson from CCP games to dig into Python at Eve Online. This is Talk Python To Me, episode number 52, recorded March 16rd 2016.

00:00 [music intro]

00:00 Welcome to Talk Python to Me.
A weekly podcast on Python
the language, the libraries, the ecosystem, and the personalities.
This is your host, Michael Kennedy
. Follow me on twitter where I'm @mkennedy
. Keep up with the show and listen to past episodes at and follow the show on twitter via @talkpython.

00:00 This episode is brought to you by Hired and Snap CI. Thank them for supporting the show on twitter via @Hired_HQ and @snap_ci.

01:43 Michael: Kristinn, welcome to the show.

01:44 Kristinn: Thanks for having me.

01:44 Michael: We are going to have a really cool conversation about games and massive scale just online applications in general, right? I am sure we will, you guys are doing amazing stuff. Before we dig into that though, what's your story, how did you get into programming, Python?

01:59 Kristinn: I came kind of roundabout way into game programming, I actually applied for the position at CCP as a game designer, and I started here as a game designer worked here for 2 years but I have always kind of been interested in programming so I started poking around, fixing things and then after that there was no turning back because programs have always seemed so much demand that I just kind of pulled all of my time into programming and then I kind of just worked my way into being my current position which is technical director of eve online.

02:33 Michael: Wow.

02:34 Kristinn: And I did that just debugging every problem that I saw and never giving up on them.

02:39 Michael: That's really cool and you guys do mostly Python there, is that correct?

02:42 Kristinn: Yeah, majority of the program is- eve is working in Python, we do have some stuff written in C++ as well as most of our BP in SQL but I'd say like at least 75%, 80 % of our program is in Python.

03:02 Michael: Ok, excellent and how did you get into that originally, is that something you learned when you first started there, did you know that before you started?

03:06 Kristinn: No, I had known about CCP, I had known they were developing in Python but the only thing I knew about Python was that white space matters. Ridiculous at first, but now I am thinking it's one of its greatest features because you are going to have the white space that way anyway so this one have mean something. It also kind of forces you to be consistent, on it.

03:31 Michael: Yeah, that's an interesting point like any time you look at professional code it's always structured, in a way that resembles the way that Python is structured, you know if you look at C++ or CSharp or even Javascript one as well written, the white space is in a format such that it could matter but it doesn't, right, and so I had exactly the same feeling, when I learned Python the first time I was like this for space this is insane to me like, where are the curly braces, and the semi colons, what is going on, and then I worked in Python for a while and I went back, I had been coming from CSharp and then I went back to CSharp and I was like wait a minute, why am I typing all this crap, this is way less fun to type, and it was because I was spoiled by the structure of Python which really surprises me.

04:16 Kristinn: Yeah, and you can kind of see it from the code, when you read like Java code or CSharp code, and stuff like that, when you have a lot of indentation you also have a lot of like white space beneath it, while you have closing all of those brackets, all of those curly brackets. And it's also weird if you see somebody write Java code for example, and it has an if statement and then two indented lines but no curly brackets, I can guarantee you that program met both of those closes the 4:44 within the if clause.

04:45 Michael: Absolutely, there was some bug, I think it was in OS X about a year and half ago, really bad like ssl, man in the middle bug and it was exactly that, it was like there was an if statement, and they didn't have the curly braces but it wasn't in and so people thought it was in one case but it was actually in a different one, right.

05:05 Kristinn: Yeah. I thought this exact thing and I thought this would never have happened in Python.

05:13 Michael: How interesting.

05:14 Kristinn: We'd have different problem surely, but...

05:16 Michael: Yeah, we would have other problems, but the indentation problem not so much.

05:21 Kristinn: I kind of always liked that about the Python philosophy is that what you see is kind of what you- I mean the whole idea is that the code is readable and what you see on the screen is what actually happens, not some like aha, you forget the curly brackets.

05:37 Michael: I say it a lot on the show, like you hear about things like Javascript gotchas and like languages like C, the mistakes that people make when they are new to the language, that are really bad and not obvious, and you just don't hear about that concept in python, like people make mistakes but they like don't work or something, right?

05:56 Kristinn: The only thing I can think of and I love showing people is the mutable keyword I love that but.

06:06 Michael: Nice, yeah. so let's talk a little bit about EVE online, EVE online is a massive multiplayer online game that has to with sort of like space exploration, colonization, like a science fiction space future, right?

06:22 Kristinn: Right, yeah. It's a massively multiplayer online game we focus a lot here on the kind of the player interaction we care a lot about how people interact with each other and we don't try to interfere that much about it but in the end it's a game where you fly spaceship if you've got the resources you can shoot other people, you can shoot non player controlled ships and this is the biggest selling feature is the whole dynamic market, we have like a realistic market.

06:53 Michael: Yeah, there is kind of like an economy in the game that is live and not programmed but it's like organic in the sense, right?

06:59 Kristinn: Yeah, and it makes the real world economies really, and you can see the same behavior there and we actually had like a professor in economics working just analyzing the data we still analyze a lot of the data like the market activity, just to know the game and what's happening in it.

07:18 Michael: Ok, that's really cool, do you have like machine learning type stuff going on in the background or is it more procedural and you understand it and then you tweak the algorithm?

07:26 Kristinn: We had, so for stuff the NPCs like the just we sold on market, we have something where we control the supply, we just created the item and sold them, so called like NPC goods, and what we did there is that we had something where you bought office supply we added the new supply but at the higher price. And if they had bought a lot of the supplies we reduced the price, so it kind of balanced itself out. Which created kind of like trade trust, but that is kind of not the most interesting thing about the market, the most interesting thing about the market is the player versus player threading, where players sell items to other players. We don't need any machine learning, we don't need any mechanics, the whole dynamic part of it is something that the players provide themselves, just by speculating on the market.

08:17 Michael; Yeah, very interesting, like you just have a bunch of people interacting and that is like the artificial intelligence although not artificial really.

08:24 Kristinn: That is what creates all the kind of body and dynamics and the interesting conversations like just this other day somebody saw the faked screenshot of the item that are very rare item from our test server, saying that they were going to be brought in the next release, which increased the price on that thing so just by the market speculation that we might be doing something influence the price on that item.

08:50 Michael: Wow, did that person go and buy a bunch of them all of the sudden?

08:53 Kristinn: I assume so, yes. People do that a lot and they try to buy all the resources of a certain type just to create the shortage of them. And we have amazingly clever group of players that do that, players that have like a economist aboard and they have IT departments and all sorts of stuff.

09:15 Michael: Have you had instances where people are selling items in the game but in real life for real money?

09:22 Kristinn: It’s a constant problem really we don't support it obviously it's very tricky to stop what we work, we try to do it as much as we can and we have a lot of success in it as well, but people always try to sell it.

09:34 Michael: Sure, I guess that would mess up your algorithm right, because that's basically extraordinarily high value transaction happening invisible to your economy in that sense, right, like a black market to some degree I guess.

09:45 Kriistinn: Yes, but that's not the only reason, it's also we talked like this kind, it's not a keyword for our players either, we also don't want you to be able to kind of buy a hat in the game and kind of 9:57

09:59 Michael: This is the philosophy of a game that was made before the last five years in that purchases in IOS and these types of gears, right, but it's such a different philosophy.

10:08 Kristinn: Yeah, the people that are the most successful in it they just make a game that is on and then they find the way to monetize that without it being- and then of course there are the other games where you just literally have to pay dollars and then you win.

10:24 Michael: yeah, that is not the same type of game as Eve and these other ones, right. It's impure in some way. So let' talk a little bit more about the game play and then we can get into the technical bits so there is this sort of economy and this exchange of goods and there is a lot of like 7000 stars system or something, you travel around and you can harvest things, so what other games is it like, like does it relate to like warcraft/ starcraft, or other games, like can you give some analogy so people who maybe haven't seen it are kind of aware?

10:54 Kristinn: It's a like a massive multiplayer online, so obviously you can throw some compiler, some of the games like world of warcraft, and those sort of games, those are all kind of avatar based games; we like to think of them as more of the kind of a team park kind of game where you go to a game and you have a bunch of rights where eve is more of a sandbox kind of game, you can imagine you go to a sandbox and you have a shovel and a bucket and you are just going to have to make some fun out of it, all the others are a lot of fun but kind of maybe care to different needs and possibly just at different times.

11:30 Michael: Yeah, it seems to me like eve is game that really appeals to people they are like for the long term like, you are going to build something up over time and sort of grow it and so on.

11:38 Kristinn: yeah and we do have very loyal subscribers, they've been with us- some have been with us for years, and even some people that have since then quit playing eve still show up at the Funfest, the annual kind of gathering here in Island that we have. Just because they are part of the community and like staying a part of the community.

11:59 Michael: That's really cool that people are like so attached to the community, right, I mean, how many players do you guys have? Give us some stats if you can.

12:07 Kristinn: The peak is I think around 30 or 40 thousand now. We were in the stats of like 3, 4 hundred thousand users I think, last I know.

12:15 Michael: Ok. When you say massively multiplayer, that's pretty massive.

12:20 Kristinn: Yeah, it is. I mean, there are bigger games there, I think at the time we ranked somewhere between the small massive multiplayer and a large one.

12:29 Michael: Yeah, yeah, sure. Let's talk about Python, where does Python fit in this story of Eve?

12:35 Kristinn: When they started developing eve, they always wanted like a scripting language to go along with something bit low level like C-++, and I think they just, it was obvious at the time the C++ but that was like back in the 1997 or something, it was pre 2000. The worth the whole lot of options they were looking at Javascript, then Python, and started looking into stackless Python. And they really liked the idea of the whole microthreads they had in stackless.

13:06 Michael: yeah, when you got started this was like probably Python 2 maybe before that, right?

13:12 Kristinn: I think it was Python 1.6.

13:15 Michael: yeah, because I think Python 2 was year 2000.

13:19 Kristinn: Yeah, so when you actually look at a lot of the decisions that were made at CCP with Python you kind of have to look at the code and say ok so when was this written, and what was available at the time?

13:32 Michael: Right, the tradeoffs you made have to consider like well, should we use VB or this, I mean like these are the other technologies that are around at the time, like Javascript, there was no cons of this nojs thing, it was kind of a toy right, so Python was a really good choice I think.

13:48 Kristinn: Yeah but it was also I think a prediterian choice because Python was pretty new at the time, it wasn't really that stable I would say and it didn't have like all the features that I think they would have needed, it didn't have IDs and stuff like that and I think it was daring but then again there wasn't like a whole lot of other options, not for doing like a lot of work, in a scripting.

14:11 Michael: Right, there are actually quite a few Python implementations these days and the show I just had last week was with Brett Cannon form Microsoft, talking about Pyjion and we talked about things like PyPy, and Pyston and Pyjion and all these different things for alternate implementations, and what I don't think we really spoke about was stackless, but stackless was one of the few alternative options you had back in that time frame, right?

14:40 Kristinn: Yeah, yeah, there was not a lot of other implementations, I think, there was vanilla Python and stackless, maybe a few others, but yeah.

14:49 Michael: yeah, not many like things like Jython or IronPython are like circa 2006, 2008 and then onward, right, so yeah. Why don't you tell everyone what stackless Python is, like what is it, what is the advantage of it, why did you guys choose this over just CPython 1.9 or whatever it was.

15:02 Krisitnn: The biggest advantage of stackless Python that people saw here was the kind of the existence of microthreads. And just very light weight work of threads that you could start up and they could do one thing, the idea being that you could be called with 15:18 to read, you could just startup like this one task and it just taught us this one thing.

15:25 Michael: Yeah, well if you are dealing with 40 000 concurrent connections, a single threaded sort of maybe forked off the 10 of 20 process type of system may well not scale the way you need it, right, so something like this is really helpful, yeah?

15:39 Kristinn: Yeah, that was the biggest reason for choosing this over vanilla Python I believe.

15:44 Michael: Are you guys still using it?

15:45 Kristinn: Yeah, we still use stackless, we kind if wish that the rest of the world would have followed us into stackless, but sadly that didn't happen and now e have some alternatives in Python, so we are still our kind of type to stackless Python in terms of code.

16:00 Michael: Yeah, it would probably take a pretty major rewrite, yeah?

16:04 Kristinn: I am not really sure how major it would be but it would be pretty substantial by the look at them-

16:09 Michael: Right, so maybe let's talk a little bit about that architecture, infrastructure, type of bits, could you maybe talk about like the different systems, and stuff that's going on in the back end, I was assuming that the front end the things that runs on my OS X machine or Windows, that is the graphics and the actual client side game, it's mostly C++, is that a fair assumption?

16:30 Krisitnn: Right, most of the highest high performing stuff is C++ like the craft extension is C++, this extension is C++, the 16:39 is C++, then we kind of have a 16:42 layer that hand tools like fires and various things, clocks and stuff like that called blue, which is like our own invention. We have various third party libraries written in C++ as well, yes, most of that is done in C++, most of the stuff that we someone saw that would be a problem, we also have like a minor things, there is a feature called planet retraction where we kind of layer over the planets that shows like resource distributions which is implementing the C++ because we thought it would be very performant and which is, it would have performance problem I think in Python but we wrote that in C++ and we had some options if we wanted really something to perform really fast we can write it in C++.

17:32 Michael: Sure, and do you have anything on the client side that's Python, was that all back end stuff?

17:36 Kristinn: Yeah, yeah, the most of the UI implementation is in Python, like it 17:40 some tools like C++, some rendering but for the most part UI program is written in Python.

17:48 Michael: That's really interesting.

17:48 [music]

17:48 This episode is brought to you by Hired. Hired is a two-sided, curated marketplace that connects the world's knowledge workers to the best opportunities.

17:48 Each offer you receive has salary and equity presented right up front and you can view the offers to accept or reject them before you even talk to the company.

17:48 Typically, candidates receive 5 or more offers in just the first week and there are no obligations ever.

17:48 Sounds awesome, doesn't it? Well did I mention the signing bonus? Everyone who accepts a job from Hired gets a $1,000 signing bonus. And, as Talk Python listeners, it get's way sweeter! Use the link and Hired will double the signing bonus to $2,000!

Opportunity is knocking, visit and answer the call.

17:48 [music]

18:42 Michael: And when I thought about how you guys were using Python I of course thought at the back end there's probably a lot of Python and things like that but like I said the front end is mostly C++, can you elaborate on that, like are they using a custom sort of plug in to your own engine, or they are using some kind of widget windowing system or maybe talk about it a bit?

19:06 Kristinn: This is all kind of custom made in C++, so the low level opt just like the throwing 19:12 is in C++, and then we can actually just- and I think at least it used to be the case that you just kind of inherit from them, the Python objects could inherit from the C objects and you would, I can't remember what we were using for it, whether it was boost or-

19:29 Michael: Sure, that sounds really cool, how about on the server side, like what type of interaction does the game from the client side actually have with the server and what does the request sort of look like from sort of systems and infrastructure and so on ?

19:40 Kristinn: Right, we have our own kind of networking layer which we call matsunet and it kind of 19:48 the server and client communication and that's where our last communication between nodes on the cluster both from the proxies to the sole nodes as we call them which are really just the work nodes where everything happens, the proxies sit out in front and receive request from the client as well as the communication between nodes as well. This is all done in Python actually.

20:10 Michael: Ok, yeah.

20:11 Kristinn: And it was rewritten some time ago in C++. But, with the Python layer on top so you can always-

20:17 Michael: Right, right, very cool. And, is this in your own sort of data centers or you are hosted in like some kind of cloud place like AWS or something?

20:25 Kristinn: No, no, we have our own data center, we actually moved data centers recently, it was a big blog on that on the website, it might be interesting for tech people to read that, it was actually quite an operation, we had to both move data centers and we had to buy whole new hardware in it and we were just moving between like London.

20:46 Michael: And then, I heard you say you are using SQL server somewhere around there, is that right?

20:50 Kristinn: Yeah, that's correct.

20:52 Michael: Ok, cool, are you hosted on Windows or on Linux?

20:53 Kristinn: We are hosted on Windows, we are probably one of rare instances where we actually use Python and Windows I guess.

21:01 Michael: Yeah, yeah, very cool, ok. The world has changed since 1997, in terms of databases pretty dramatically especially in the last 5 years, I am sure part of the reason that you guys are on a relational database has to do with the fact that you started on a relational database, but do you think if you could start totally from scratch, would it make more sense to be on like a no SQL database or stick with the rdb like SQL server?

21:24 Kristinn: I think if I probably would be forced to choose like a single database, I would probably still go with the relational database, we made so many changes recently in terms of microserveces architecture that I wouldn't really see necessarily a need to do that, I would try to split out the cluster a bit more into these functional areas then when I storing data that needs a relational database, I'll store it in a relational database, otherwise a NoSQL database or craft database or whatever, but I think if I had like one deeply cuts all, I would still go with the relational database.

22:00 Michael: It makes a lot of sense, the world is definitely moving towards this concept of microservices and I don't think I've spoken very much about that on the show, I think maybe a little bit when we have the Docker guys on way back on show 9, but also had a conversation with Sam Newman that actually the audio went so bad, we couldn't put it together, I specifically about microservices, sorry about that Sam, but maybe you could just tell us really quickly for this people are like microservices, it sounds like small services, what's that, maybe describe that a bit.

22:33 Kristinn: It's essentially that it's small services, it's this idea that you spin up a service for a specific need that your total ecosystem needs, so in terms of EVE, we for example have a little market, maybe the market could be its own microservice, maybe it doesn't have to meet to connect to the cluster, the cluster would only need to know how to talk to that microservice and that microservice to talk to that cluster. I don't think I would really do adjust this but it's really just a matter of that, the breaking up your currently total application into like multiple smaller services.

23:08 Michael: Yeah, the thing that I think is interesting about that is, you know, I ask you about what is the right database, and the answer may well be, well, certain types of architectures that are proven to be powerful like these microservices the answer could be many small ones and then it even matters less because when you are, if you are let's say just modelling only the economy, which I am sure is still massive, but just the economy not every possible thing that could on in the game, then you might have smaller database performance needs and so on, right.

23:39 Kristinn:: Yeah, and in some case of data you actually need everything to be always up to date, but maybe sometimes you just need like eventual consistency on data so you are going to pick the database solution based on the need for that particular feature rather than choosing it for everything and hoping 23:58 Having said that, I don't suggest that everybody makes changes their architecture from like single monolithic application to microservices, it's something you exchange one problem for other problems, but it can make a lot of sense and one of the things that I like about this is that it allows you to change decisions that you thought you kind of thought you would never ever going to change.

24:26 Michael: Yeah, that's interesting. And it's good advice, like, design patterns and these larger architectural patterns are super powerful but they almost always have a context that goes along with them. If you have a million lines of code and this level of huge complexity you can make that complexity manageable by breaking them apart, but if you write like 5000 lines of code maybe a microservice just means more servers and things to maintain, right?

24:50 Kristinn: Right, it's more service, it's more things to maintain, in theory that those service should require less maintenance as well, and they are easier to comprehend but you also need like maybe you need to have a message queue and maybe you need to opt in a bunch of technology that you wouldn't have need that you have that monolithic application.

25:10 Michael: Yeah, and performance as well, right, like if previously the call to get data about like the price of an item in your economy, was within the same machine or something to this effect within the same process even but it's like a database maybe, that's probably a lot faster than actually going across network finding another machine, doing a whole http thing and then asking that same question, right?

25:33 Kristinn: Absolutely, yeah, I think, yeah, you just have to play very carefully this, I think it's you should get really excited about microservices but you really have to make sure that's the correct solution for you.

25:48 Michael: Yeah, absolutely. I certainly see those tied together with Docker and the whole container space, but again-

25:53 Kristinn: Absolutely, they make this thing a lot easier.

25:56 Michael: Nice, so one of the things that you said that you guys had done I thought was pretty interesting is you said you had written your own importer, is that like if I say import space some module name, like import os you rewrote what that meant?

26:09 Kristinn: Well, sure, it's not really maybe import like a bootstraper, I don't know actually what the state was in Python 1.6 but I am thinking maybe it didn't actually know how to import packages and the whole dunder init.

26:27 Michael: Oh, interesting, ok.

26:28 Krisitnn: So, that was pretty interesting, and so what they wanted then, they wanted like a shallow namespace for these things, so what they actually did was they wrote something they would just look over the file, it would you know, read actually that file and import it into like a namespace that they created, so they basic just injected into cs.modules.

26:47 Michael: Interesting. Ok, so what was the benefit there, do you remember?

26:51 Kristinn: One of the benefits was that we would have how to reload thing on modules when we change them, and it was actually quite nice to work at the time and it still works that way, it's still now it's a Python solution, instead of like home-cooked one but just changing something in code you don't have to restart your server, you don't have to restart your client or anything, just we execute whatever you're doing [indiscernible]

27:18 Michael: Ok, now you've got my attention, that's pretty intense actually.

27:20 Kristinn: Yeah, and that's still the case today, but now it's like with the Python implementation of it.

27:27 Michael: Did you have to do, like I guess you almost have to, either you say you can't use these types of features or you have to deal with things like static variables, and static class fields that had been set previously and then if you reload the module- what did you do there?

27:39 Kristinn: It wasn't completely bulletproof like if you and it still isn't, if you already bound a method to something it doesn't actually reload that if you are like passing the method of an object into something else then it's already bounded and it doesn't reload it but for the most part, for the most code you are changing it, it actually just changes it. I've had issues but it would, some data was actually stored on a module level, like a dictionary on a module level, and whenever that got reimported it would clear that, and those sorts of things, it's I mean, it happens you need to restart your server and client and when you are trying something out of it, something funky comes up, it's something what you do, you just restart everything and try again.

28:22 Micahel: Yeah, interesting, so you have like two levels, you are like let's just try to hot patch it if it doesn't work give it a hard kick, right?

28:27 Kristinn: And we had done that in production as well, if there is a problem that we really want to fix and that's actually one of the drawbacks of having a monolithic application if I have a problem in the market, I need to take everything down to parts, we have actually live fixed stuff like that but just going in console and just creating a new method and that to the class.

28:49 Michael: ok, that sounds really powerful and interesting is any of that sort of open source, or blogged or written about that people can check it out, or is it...?

28:58 Michael: No, none of this is open sourced, I don't think we have ever blogged about this I think-

29:02 Michael: Ok. Well, it sounds interesting, I know I talked to David Beezley I think at show 12 about modules and packages and this whole concept and it was like, I am not really sure that this is going to work if it can be- so to hear that you guys are doing it on this real app this is really cool.

29:17 Kristinn: I think people might get kind of caught up in the idea that while it's not going to work every time, but that's ok when you are doing development, you just work like rapid feedback really

29:27 Michael: Yeah, if it works 95% of the time and it saves you a ton then maybe it's worth it right.

29:31 Kristinn: Yeah,

29:31 Michael: Yeah, I guess coming back to the microservices thing if you build first something microservices and you do need to sort of hard restart some piece, it probably has a smaller consequence because you are not restarting the entire thing, right?

29:43 Kristinn: Yeah, but the flip side of that is that you have to write everything now as nothing is currently to be up.

29:50 if you do that then you kind of, you are always currently to have something down and if your whole- if nothing works when one of the bit is wrong you are going to have constant downtimes, but.

30:02 Michael: Yeah, there's that trade off again right?

30:04 Kristinn: Yeah, it's a trade off like it's not a silver bullet, and I think people have been pretty vocal about that, like the experts on that have been pretty vocal about that and I think every paper and every talk I've seen about microservice always starts with that, this is not like a silver bullet that will fix all your problems.

30:23 Michael: Ok, cool, so one of the things that happens in the game is there seems to be a lot of people that group together the form like a fleet. And then, sort of in real time with fully communication they will go and they will have like these fleet battles, right? How many people end up in those battles? How many different players?

30:23 [music]

30:23 Snap CI is a continuous delivery tool from Thought works that lets you reliably test and deploy your code through multi stage pipelines in the cloud, without the hustle of managing hardware. Automate and visualize your deployments with ease, and make pushing to production and effortless item on your to do list.

30:23 Snap also supports Docker and in browser debugging and hey integrate with AWS and Heroku.

30:23 Thank Snap CI for sponsoring this episode by trying them with no obligation for 30 days by going to

30:23 [music]

31:36 Kristinn: I think the launches we have was like 4000 people I think, then you can get pretty laggy, but

31:47 Michael: What's the sort of performance characteristics as you add more people, it can't be linear like 2000 people versus 4000, it's more like combinatorial, or something really expensive, right, like how does that work?

31:58 Kristinn: We've done a number of optimizations over the years, it actually performs quite well now, I'm not sure if it's linear but it's not n tool or anything like that, but it's also very difficult to protect on that because it matters a lot whether everybody is shooting at the strokes or maybe that doesn't die for a long time, or if people are shooting at each other and people are getting killed and then removed form that sole assistant, that creates more load, it matters if the people are shooting form guns or using missiles-

32:29 Michael: Right, because guns, presumably are instantaneous but missiles sort of cruise through space. You've got to attract them for everyone and see them go by and so on.

32:37 Kristinn: Yeah, yeah, they don't collide really with anyone except the person that they were meant to collide with, that's one of the optimization we did a long time ago, there is a lot of factors, it's very hard to say like you can look at the a 1000 people fleet fight and see no lack, and you can see another one that has a missile multipack it's very hard to measure that, one of the most reliable kind of metric we have is the market up, we have a single system that- you know how people are, they kind of just all gravitate and because a lot of goods are there for sale people think it's worth it for them to put their guns for sale, so it creates this 33:13 which is a solar system called zeta and there we have like a constant influx of people, we had like a maximum of 3600 people there the other day-

33:25 Michael: That's a lot.

33:27 Kristinn: Because of some server optimization we have been doing, the CPU actually never went above 50% I think, on that.

33:33 Michael: Yeah, yeah, that's really cool. So you said one of the challenges you have with some of the scaling has to do with the dreaded global interpreter lock, the gil, what's the story there?

33:40 Kristinn: We basically can sack 33:42 of cluster off into, but while it's probably easiest to 33:46 that's the smallest units we can run, we can distribute the load within one solar system to multiple processes really, that's going to be a bit problematic because that means we can only use really just one core on the machine so it doesn't really matter, and it's very problematic because the hardware industry is moving more towards more core and maybe less clock speed. But we really decided there is more clock speed.

34:13 Michael: Yeah, I mean, the clock speeds I think are, if they are not slower, they are definitely not any faster than what they were in like 2005, right?

34:20 Kristinn: They are not getting faster I can tell you that.

34:23 Michael: Yeah, I mean they may even be slower, I know they got higher and they started to melt, they kind of went down, so right, that's it. 8 cores, lower clock speed. Very interesting. Would Docker type stuff or virtualization sort of let you sort of break that up at least not dedicate a whole machine to it but say ok, well we are going to isolate windows, I guess Docker is doing stuff with Microsoft these days but maybe I should say windows containers are coming out soon. Would that be a way to partition this so that you can still get everything you want out of those machines without dealing with the threading?

34:57 Kristinn: Yeah, it's an interesting thought it still requires like a lot of program work to actually do it, but you have to segment what the cluster is doing within the solar system into like something logically units which you can scale which doesn't chatter too much between each other. We actually have an idea about that but yeah, we'll see if we get there, the currently stands we are not really that for performance or at least it's not as high priority as some other stuff we want to do.

35:28 Michael: Right, when you are doing performance analyses like you just look at the slowest thing, that's the bottle neck and if the stuff behind it is slower but still, you know, slow but faster than the bottle neck then who cares, right?

35:38 Kristinn: Yeah, right, and there's something to begin including before we start looking into a splitting really really into multiple process on the same note. But, we have like a better leads-

35:52 Michael: So one thing you said you are really into is test driven development unit testing and so on, and this code is super old, you must have some interesting challenges writing test for a code from this long ago, yeah?

36:04 Kristinn: Yeah, the import of bootstrapper which we appropriately named nasty at the time wasn't of course like a huge burden on us, because we couldn't really import any of the specific call without using that nasty importer and that nasty importer really had to go through everything and really just run a full scale client or a server.

36:29 Michael: Yeah, so you are almost like forced to the point where you are doing integration tests because you had to start the whole thing anyway, right?

36:37 Kristinn: Exactly, and we did manage to create like an interpreter that did that but we had me and another programmer are writing test and I think we are up to like 100 tests and it will take 45 seconds to run, which is just terrible.

36:52 Michael: Yeah, it's pretty long for unit test, right?

36:54 Kristinn: Yeah, and it's unusable if you want to use it for like a quick iteration feedback. When developing you really need something in less than a second 37:07 but there were some initiative to actually get rid of nasty they actually managed to do it in like one summer than managed to do it I think in I think two weeks.

37:16 Michael: Nice, and now testing is much easier?

37:18 Kristinn: Now testing is a lot easier, it's still a lot of coupling and there are other sins, other than creating our own bootstrappers class importer that we did, like shoving stuff into 37:29 and stuff like that, that make testing harder. But, usually, always you can extract the code that you want to pass out and you write tests for them.

37:40 Michael: Right, ok, nice to hear that you got that sort of cracked open so you can test it.

37:45 Kristinn: Yeah, it's been like one stat came in and it was a stat that being able to like tests like that it can really open my eyes towards faster development, when automated testing in general.

37:58 Michael: All right, cool, so let's talk about a little bit about ccp the company that makes Eve online, you guys are primarily based out of Iceland, right?

38:06 Kristinn: Right, mostly out of Iceland, we had offices in New Castle, Shanghai and Atlanta, but everybody that works on Eve is stationed in Reykjavik.

38:16 Michael: Very cool, I've never been to Iceland and it's definitely one of the places I'd like to visit though.

38:20 Kristinn: Yeah, it's definitely interesting although I lived there through most of my life so

38:23 Michael: it seems very natural to hear that, it's very normal, with all the winter, and the volcanoes and I think that would be super cool to go check out. Nice, so you guys use an Agile approach, is that right, do you use Scrum or what's the story there?

38:38 Kristinn: Yeah, we started with Scrum, and then we kind of, it's moved into something that's like Scrum but it's mostly- Scrum is really the basis of it, but I think we allow every team to really put their own label on it and do kind of whatever works for them, Agile practices are really funny because if you don't pay attention to them they would just deteriorate and you'll start getting worse of them, it's something that just kind of requires constant work and of course it's part of the Agile to behave retrospective and constantly be trying to improve yourself.

39:13 Michael: Right, I think a lot of people lose sight of that, I mean, on one hand there is not necessarily a single Agile process that is exactly right for everyone, right, I mean, that's a little bit of app being flexible and agile and so on there, but then, if you are going to go and sort of walk it up to the edge of the absolute minimum amount of documentation, rules, structure, that means if you fall apart just a little bit, you are probably falling out of the minimum amount of structure that you need, right?

39:40 Kristinn: Right, it's a little bit like that, I kind of like starting with Scrum, I think Scrum has like if you understand why the rules of Scrum then you have the power to change them as they make sense to you but I've seen also seen a lot of companies, or heard from like a lot of my colleagues here along Iceland is that they have implemented scrum but they really didn't get what Agile was.

40:07 Michael: And then hey, this Agile thing sucks, it doesn't work.

40:09 Michael: Yeah, right.

40:09 Michael: Yeah, well. Maybe you didn't follow it. You guys said you also have kind of an interesting learning and development philosophy, what's the story there?

40:18 Kristinn: Well, it's not, I mean, we do try to foster kind of the culture of having like what you call a community of practice, we had a Python community of practice for example, but we kind of retired that because people weren't really showing up that much. But it's actually very difficult because people are so invested in their job, they are so invested in what they are doing that they don't see that they can spare a few an hour here to do stuff like that, what do we do is maybe a bit weird, it's different than sending people to conferences and getting consultancy we have been kind of running this video series uncle Bob-

41:02 Michael: Oh yeah, Robert Martin is awesome.

41:03 Kristinn: Yeah, we have been running that and we have been having like talks afterwards, that I think has been very good, I definitely recommend companies to do that it's probably one of the cheapest learning and development you can get like in terms of bang for buck.

41:18 Michael: yeah, I totally agree, I think one of the things that people, companies, teams individuals, like, on all of the levels, don't necessarily realize is if you just spent like half an hour a week, or half an hour twice a week making sure you polish your skills, you learn something new, after a few years, that really adds up.

41:36 Kristinn: Absolutely. There have been other initiatives in the past, like we had like on Friday people had code meetings where they would meet up and they would get like a task and then they would go in groups of two and they would do something like write these features but don't use classes, stuff like that, and there is all these initiatives they common go, I had a kind of a mob programming sessions which was all about pulling code out and write test for it-

42:03 Michael: Very interesting, you know, I don't think we have talked about mob programming on this show, I know what it is but I don't think we have ever mentioned it, can you just give it like a really quick elevator pitch like what is mob programming?

42:15 Kristinn: Basically what it sounds like, it's like one guy and a keyboard and he takes suggestions people shout at him, but of course, there is one driver and he probably knows a bit where he is going. Obviously what do you want to do here, do you want to create a new method here, do you want to create a class or whatever-

42:35 Michael: Would you say it's kind of like paired programming but many people on the pair side?

42:38 Kristinn: Yeah, and probably a lot, lot less efficient but it sparks up good conversation.

42:44 Michael: Right, it's more, it's not necessarily like paired programming, the expected output is solid code that you are going to ship. Where think mob programming is more about mutual understanding and skill development and so on, right.

42:56 Kristinn: Yeah, also just getting people to kind of voice their private thinkings about how code should be, because that is, we actually just don't do that, we write code we put it up to code review but without me understanding what is your agenda or what is your philosophy about code and chances are we are going to disagree on a lot of like fundamental things.

43:16 Michael: Yeah, I would say so. Now, it sounds like a cool projects you've got going there. So one thing I wanted to ask you about, do you have any stories about like when there was some giant unexpected rush of new traffic or new customers like maybe you know, a new release came out that was really popular or something like that?

43:33 Kristinn: What we had the other day, we had a new item come out that was very popular on the market, that meant that everybody brought to that, 43:41 to sell stuff and buy stuff, way more than what it usually had been accepted. The thing is we have like, we allow like max 2000 people on it, because if we get more or when we used to get more than we, the game would be pretty unplayable for everybody involved so we thought it would be better to actually limit the amount of people that could be in there at any time and their experience would at least be better. And people can always play in be in all the systems. But because it's a number of performance like a big performance you done, we kind of saw that the seller wasn't actually 44:26 we started increasing the cart, we increased it first by 2500 and then we started reaching that so we increased it so we think about increasing it to 3000 so but instead the development directors said I think we should just go to 5000 because 5000 based on these numbers. So that's what we did, we got 3600 people there, the cpu went about 50% but the memory was actually getting to the limit; and it's a weird thing, because we've gone from having been memory swifted on a PC to almost not having to think about it at any point. Eve has this weird thing where we restart the cluster every day at 11 o'clock so whatever memory we have, we would just that gets crossed out. So the program is on Eve I've never really considered memory so much so we just cash it and then we never released it because we never really had to release it. So, it's weird that when you have a lot of people coming in we just saw the memory is increasing and it never went down.

45:36 Michael: What's, what I think is interesting about that story is when I think of some system getting hit with lots of traffic and having unexpected growth, I see it as in all or nothing type of thing, like the website now has a lot of visitors and so it's busy or not, but the way you guys have structured your world it's like there is a little node over there somewhere where a bunch of people went into that part of the universe and then that part is too busy, that's interesting way to think about scale, like I suspect most places don't.

46:06 Kristinn: Yeah, it's always been a and it's very funny to look at the kind of the overall cpu utilization of our cluster and you could see like it's using 10% cpu on average but still you know that there are some people there is some node there that there is at 100% are experiencing lag and you are seeing 10%

46:29 Michael: Yeah, it's almost like you need sort of a mountain range type graph instead of just a level, or something like this, very cool.

46:36 Kristinn: And in our old data center what we used to have is that we had one node that was we called it the Everest node, it only ran that one zeta solar system and it was a bit beefier than all the other all the nodes and it was just dedicated to one in that particular node. In the cluster I think all the machines are better than that old machine.

46:58 Michael: That was the good one and now it's old and crusty. Very cool, so I have just a few more question before we kind of wrap up the show, I was told on Twitter, I think it was Kyle that sent a message that said I should ask you about the big red button.

47:12 Kristinn: Yeah, that's, I always get that, I am a bit of I guess infamous in the Eve, so what happened is that I was just happily developing on my local cluster and I needed to something, and normally when I shut down the cluster I just killed the process and started off again. But, there is some logic that happened when you shut down the cluster, some persisting logic that I wanted to test so I go to their webpage, of the cluster, and I shut it down properly or properly it's called an emergency shutdown and I gave it a timer, so I did that and then I go back to my client and I don't see all the messages that I get when I shut down the cluster, like cluster is about to shut down, so I look at the page again and I realize I am on the production cluster, and I am shutting down production cluster.

48:03 Michael: [laugh] Ooops.

48:03 Kristinn: So I, and the only way to cancel this is to go through that page again, so I did that and I canceled and I didn't know how long time it passed, I knew I put a one minute timer there but I just practically clicked that button, and it turns out I managed to stop the shutdown, I never actually shut down, but everybody that was logged in got the message that Eve was shutting down and people really don't like that.

48:29 Michael: I am sure they don't.

48:31 Krisitnn: They really don't like that so they went to the forums immediately, it was like ccp, what the hell, why you shut down the server, why do you have a one minute timer because people want to get their assets to safety and stuff like that.

48:41 Michael: Right, of course.

48:43 Kristinn: So I did what just came naturally to me, I just went into the forums and said look, sorry, this is my bad, I thought it was shutting down my development cluster, and I am being really cute with my coworkers now.

48:57 Michael: Yeah, I am sure.

48:58 Kristinn: But it ended really funny, because they do get angry but they are also pretty cool people and honest mistake like that, they let it slide.

49:09 Michael: yeah, and they are pretty technically savvy, they understand.

49:11 Kristinn: Some of them probably- and they see it happened like a week after someone in operations was going to shut down the test server and shut down the, but even go to the forums and apologize.

49:27 Michael: You didn't claim public responsibility and say sorry.

49:33 Kristinn: I think that people like to remember that story as some 49:36

49:37 Michael: Ooops, [laugh]

49:40 Krisitnn: But I'd like to think that I just, I did the right thing by apologizing to people and explaining just the exactly what went wrong and why I was being stupid for doing it.

49:53 Michael: yeah, it all happens to us. I one time I was teaching a training class and I'll try to not put any details about any place that I was somewhere in Chicago let's say, and I was teaching this training class at a financial place and they for some reason didn't have a proper room that they could use for all the 20 people that were in my class or whatever and so, they and they had given us this weird like secondary server room, where there is all these servers along the wall, but they weren't, it wasn't the obviously the main data center, it was just like servers that ran internals within the company, you know. And, I had this little squishy ball thing that I would use to like work out your hands or whatever, and I was walking, it slipped out of my hand, and it was odd shape so it bounced like a crazy ball, and it bounced along the students' desks, over along the wall, hit the wall, fell down and hit a power strip that killed an entire wall of servers. Oh my God, that was bad, so we quickly turned them all back on and just go back to the class, and about ten minutes later, some guys rush in like what happened to the servers, and I was like we don't know, [laugh] so yeah, I have turned off the servers as well, on accident.

51:01 Krisitnn: It happens.

51:02 Michael: Yeah, luckily the students had my back. All right, two quick questions before we go, if you are going to write some python code, what editor do you use?

51:08 Kristinn: I use PyCharm now, I had used Sublime and Vim in the past but I think in terms of like a good IDE I think nothing is better than PyCharm.

51:20 Michael: Yeah, I am with you on that, I love PyCharm, that's what I use as well. Awesome, and then if you are going to go and look at PyPi all the packages, I think there is like 75 000 of them now, are there ones that are super useful that you think maybe people don't know about but they should?

51:35 Kristinn: Well, I love the requests package, I think it's kind of puts the simplicity and makes an http call.

51:42 Michael: Yeah, what's interesting about that is it's like so simple that you could almost not appreciate how cool it is.

51:48 Kristinn: Exactly, I really love that package, mostly due it was probably used the other libraries and they are probably more flexible and stuff like that but sometime you just want to make a get request on a resource and...

52:00 Michael: Yeah. Request.get and you are good to go. Awesome, so if people want to get started with Eve online, how do they do it?

52:06 Kristinn: Visit, that would be a first step and I want to highlight the feature that is that you can download the launcher and it starts downloading the game but it only downloads the stuff it needs, so it only downloads the code and like a minimum for users, so you could be within the game in like 10 minutes.

52:27 Michael: Yeah, that's awesome, yeah, I definitely would start playing it and check it out, it sounds really cool.

52:29 Kristinn: Yeah, and after that it just downloads the resources as it needs, so try that, go to sign up for a free trial and check it out, it should take you less than 20 minutes on a decent internet connection.

52:45 Michael: Very cool. Kristinn, it's been fun to learn about what you guys have going on with Python, a lot of cool stories, thanks for sharing them.

52:50 Kristinn: Yeah, thanks for having me.

52:53 Michael: Yeah, take care, bye bye.

52:53 This has been another episode of Talk Python To Me.

Today's guest was Kristinn Sigurbergsson and this episode has been sponsored by Hired and SnapCI.

52:53 Thank you guys for supporting the show!

52:53 Hired wants to help you find your next big thing. Visit to get 5 or more offers with salary and equity right up front and a special listener signing bonus of $2,000 USD.

52:53 Snap CI is modern continuous integration and delivery. Build, test, and deploy your code directly from github, all in your browser with debugging, docker, and parallelism included. Try them for free at

52:53 Are you or a colleague trying to learn Python? Have you tried boring books and videos that just cover the topic point-by-point? Check out my onlne course Python Jumpstart by Building 10 Apps at

52:53 You can find the links from the show at

52:53 Be sure to subscribe to the show. Open your favorite podcatcher and search for Python. We should be right at the top. You can also find the iTunes and direct RSS feeds in the footer on the website.

52:53 Our theme music is Developers Developers Developers by Cory Smith, who goes by Smixx. You can hear the entire song on

52:53 This is your host, Michael Kennedy. Thanks for listening!

52:53 Smixx, take us out of here.

Back to show page
Talk Python's Mastodon Michael Kennedy's Mastodon