EVE Online Departs for Python 3

0:00

01:07:52

Links Episode Deep Dive Transcript

Panelists

Jamie Bannister

Thomas Dähling

Kristinn Þór Sigurbergsson

Every ship in EVE Online eventually undocks and leaves the station. This time, it's the whole game. EVE has run on Python 2 since it launched in 2003, all 2.4 million lines of it, on a custom Stackless interpreter that stopped at 3.8 and was archived last year. Destination: Python 3.12. The route runs through 6,500 lines of division that decide who wins a fight, and 100 gigabytes of pickled Python objects that have to survive the jump intact. Kristinn Sigurbergsson was on this show ten years ago. He's back, with Jamie Bannister, who is flying the EVE Online migration right now, and Thomas Dähling, who took EVE Frontier through first. In EVE, a destroyed ship is gone for good. There are no do-overs, and the universe has to stay online the whole way.

Play on YouTube

Watch the live stream version

Episode Deep Dive

Guests Introduction and Background

Kristinn Þór Sigurbergsson is director of gameplay engineering on EVE Frontier at Fenris Creations, the Reykjavik studio formerly known as CCP Games. He first joined Talk Python for episode 52 in March 2016, when he was technical director on EVE Online, and he mentioned that the Python 3 upgrade was already on the table back then. Since that visit he has been everything from game designer and software engineer to technical director of EVE Online and later technical director of the platform group that maintains the technology shared across the studio's games. He was part of the team that carried EVE Frontier to Python 3 and, as he put it, is now the person who primarily benefits from that work.

Jamie Bannister has been an engineer on EVE Online for about 16 years, mostly on gameplay, which has taken him across almost every part of the game from the client front end to the server back end. He was a dedicated EVE player before joining the studio and used to organize player meetups around the UK. He is currently working on the EVE Online Python 3 migration itself, building on what the Frontier team proved out.

Thomas Dähling is a principal programmer on the platform segment at Fenris Creations. He led the initial Python 3 transition for CARBON, the game engine shared by EVE Online and EVE Frontier, taking it from Python 2.7 through Stackless Python 3.8 and on to Python 3.12. He presented that work at EVE Fanfest 2025 under his developer handle CCP Aporia, and he attended EuroPython this summer to follow where free-threaded Python is heading.

What to Know If You're New to Python

This episode lives mostly below the application code, in the interpreter, the C extensions, and the serialization formats, so a few ideas about how Python actually runs will make the conversation much easier to follow.

Python 2 versus Python 3: Python 3 changed core behaviors, not just syntax. Print became a function, dividing two integers produces a float, strings became Unicode, and old-style classes disappeared. Python 2 reached end of life in 2020, and this episode is about moving a codebase that predates all of that.
Coroutines, Stackless Python, and async/await: Concurrency in Python often means juggling many tasks on one thread by having each one pause while it waits for something. Stackless Python did this with tasklets and no special syntax, while modern asyncio requires marking functions with async and await. That difference is the whole reason EVE could not simply adopt asyncio.
The GIL and free-threaded Python: The Global Interpreter Lock lets only one thread execute Python bytecode at a time. Python 3.13 introduced a build without it, and the guests discuss what 20 years of code that silently assumed the GIL means for adopting it.
C extensions and embedding: Python can call into modules written in C or C++, and a program can embed the Python interpreter inside its own executable. Most of EVE's migration effort was in this C++ layer, because the C API changed between Python 2 and 3, and the Python code cannot even run until those modules import.
Serialization and pickle: The pickle module turns live Python objects into bytes for storage or network transfer, but those bytes are tied to the Python version and class definitions that wrote them. That is why 100 gigabytes of stored objects is a migration problem and not just a data problem.

Key Points and Takeaways

EVE Online is moving 2.4 million lines of Python from 2.7 to 3.12, and it has already started

The center of the episode is the announcement that EVE Online, live since 2003, is actively migrating its roughly 2.4 million lines of Python off Python 2.7. The team is not stepping through every release. The target is Python 3.12, reached by way of an intermediate stop on Stackless Python 3.8, which was already at end of life when they did the work. Thomas said the build pipelines have been reworked so that future upgrades are much easier, with a jump to Python 3.15 planned for later this year. The public post, "The Move to Python 3 Begins," was written like a mission briefing, and Jamie admitted the heading was a little cheeky, since the sister game EVE Frontier is already on Python 3. What is new is that EVE Online itself is now on the path, reusing the engine work proven on Frontier, and the first changes have already shipped to the live game. Kristinn noted that the upgrade was being discussed when he was on the show in 2016, so this moment has been ten years in the making.

Stackless Python was the right call in 2000 and a dead end by 2025

Thomas described Stackless Python as "basically Golang's goroutines, but in Python," years before the coroutine model became popular. The studio built a customized network stack on top of it so that developers could write plain sequential code that never blocked the main process, which is what let thousands of players share a single solar system. The cost was a heavily customized fork of CPython, with the studio's own changes for metrics and memory allocation layered on top, which made every interpreter upgrade painful and left Python 2.3 and 2.5 idioms lurking in the codebase. Stackless also became a niche as asyncio, built on the Twisted model, and gevent, which you can simply import into standard CPython, took over the ecosystem. Michael pointed out that the Stackless repository on GitHub was archived in 2025, and it never went past Python 3.8. Being stuck there meant no Pydantic, no FastAPI, and none of the modern libraries the rest of us take for granted. Thomas mentioned that the new scheduler module that replaced Stackless now serves as the synchronization point between the background I/O worker threads and the main Python thread.

Stage one: make everything valid Python 3 syntax while still shipping on 2.7

Jamie described the first phase as "legacy to 2.7": bring code written with Python 2.4 and 2.5 idioms up to a 2.7 standard, because that is where forward compatibility starts. The team used the fixers from futurize, part of the python-future project, choosing only the subset that keeps the code running on 2.7. A typical fix was replacing the old has_key dictionary method with the in operator. When they started, about 94% of the code was already syntactically valid Python 3, and after the first pass that number is 99.8%. That first batch was released and deployed to the live game a few weeks before the announcement post went out. The next phase forks the Python 2 branch into a "Python 3 unstable" branch, where many tests immediately went red, alongside a "2.7 unstable" branch full of shims and adapters that behaves correctly on both versions but is too slow to ship. That middle branch is a compatibility proof, used to decide which changes can land on 2.7 today and which are breaking and must wait for 3.

Syntax is the easy part; 6,500 lines of division are the hard part

Code that parses on both versions can still behave differently, and division is the classic case. In Python 2, dividing two integers truncates to an integer, while in Python 3 the same expression produces a float. Jamie said EVE has about 6,500 lines featuring a division, and some of them will break if the switch is simply flipped. In a game, those numbers are damage: as Jamie put it, if the math changes, "I should have won this fight and now I've lost it." Michael added that turning integers into floats is also poison for equality comparisons. Each of those lines needs human review rather than an automated fixer, which is exactly why the team keeps a separate unstable branch and a test suite to flush them out.

100 gigabytes of pickled Python objects have to survive the jump

Data at rest and on the network is serialized Python, and Jamie called that out as just as big a job as the runtime code. EVE's NPC agents, which have nothing to do with LLM agents, remember your interactions, and that memory is stored in the database as pickled Python objects going back 20 years, roughly 100 gigabytes of it. A SQL script cannot fix that, because SQL cannot deserialize Python objects. Old-style Python 2 classes serialized and sent to a Python 3 node simply cannot be understood on the other side. Michael brought up the common pattern of pickling objects into Redis and asked about version-independent formats like msgspec, Pydantic, or JSON, and Jamie said the plan is to move to protobuf, or FlatBuffers depending on size, because EVE's internal and external APIs are already built on protobuf. Michael noted the bonus that a node could then pass data straight through without deserializing it at all. Jamie is writing a public dev blog about this exact problem.

EVE Frontier went first, and its first release went down every hour

Frontier forked the EVE Online codebase and shares the underlying engine, which made it the right place to prove the upgrade. Kristinn explained the chicken-and-egg problem: the Python code would not run until the C++ modules were importable on Python 3, and the C++ modules made no sense without the Python glue tying them together. On Frontier, with a small alpha player base, the risk of breaking things was far lower. Even so, the first Python 3 release had hourly downtime, because after an hour and a half or two some process somewhere would stop responding. Kristinn called it a very stressful time and said the root cause turned out to be something really stupid. With no robust unit test suite for a game of this age, the team leaned on the TeamCity build system and pushed further and further into a real build to find problems incrementally. Thomas added that the Python 3 transition is all or nothing, since you cannot put the interpreter on 3 and leave native extensions on 2, which is why they took the intermediate Stackless 3.8 step so they changed only the major version and not the async library underneath.

Mix mode: swap 200 nodes one at a time, never a big deployment day

EVE runs as a cluster of about 200 server nodes, each with particular jobs, plus every player's client, and all of it is on 2.7 today. Rather than one deployment day, Jamie described "mix mode": selectively replace one piece of the cluster with its Python 3 equivalent and spread the risk over time. Jamie has been through the big-bang deployments and called them very stressful. Getting there means building tooling that is not strictly required but pays off later, including the ability to deploy a heterogeneous mix of builds and even mix operating systems, with some nodes on Linux and some on Windows. Michael noted that was a consequence he had not seen coming. The constraint driving all of this is that the server has to be up every single day, and Frontier running ahead proved the approach before EVE Online needed it to work around the clock.

eveonline.com/news/view/the-move-to-python-3-begins

Python was a hiring strategy in dot-com era Iceland

Development started in 1999 and the game shipped in 2003, when Python itself was still rather new. Kristinn explained that programmers were scarce and expensive in the dot-com era, especially in Iceland's small talent pool, so Python let junior people and even game designers write sequential code and speed up development. The performance-critical pieces were always C++: the physics engine Destiny and the graphics engine Trinity. Gameplay logic, the network stack, and the clustering stack are Python, and Michael suggested this is probably the single largest Python application deployment on Windows. Michael observed that EVE predates the two milestones that matured Python around 2006, Django and the NumPy and Jupyter data science stack, so the original developers had to invent their own containers, clustering, logging, and telemetry, which Jamie said now look a lot like popular systems such as Redis and container orchestrators. Kristinn also credited data scientists with pushing Python 3 adoption, for the same reason the studio chose Stackless: hand people who are not primarily programmers a tool for writing code.

No pip, no uv: life with a custom importer and an embedded interpreter

The studio has never had a package manager, though Thomas said they are working on one. Adding a library meant taking the zip file from PyPI, extracting it into their environment, and seeing what happened. Native extensions were worse: one module built for Python 2.7.18 corrupted memory because the PyTypeObject layout had changed in 2.7.6 and their fork did not carry that change. For 20 years they have embedded the interpreter in a custom binary, customized which paths and environment variables it sees, and patched out the system-specific paths in Python's own build process, what Thomas called a 75% solution. Distributing Python tools to designers is a struggle because people change PYTHONPATH or run the wrong Python, and Jamie said new hires often spend their first days fighting the Python bundled with Windows. Thomas sees uv run and uvx making it possible for a developer to install a package straight into the game environment, and now that CARBON is open source, others could pip or uv install engine pieces into a normal interpreter. Kristinn offered the director's view that friction around dependencies is not all bad, since it guards against people flippantly adding a package that becomes a supply chain problem, though it also makes upgrades less likely. One fun detail: since 2011 or 2012 the engine has had an interpreter mode that starts a Python REPL inside the game so you can poke at live data structures.

What Python 3 unlocks: Apple silicon, types, dataclasses, and 15 to 20 percent for free

For Thomas, the trigger was the native Mac client work in the early 2020s. Python 2.7 knew nothing about Apple silicon, macOS handles Unicode in system calls very differently from Windows, and the old Mac client had run through a Wine-like translation layer. Without Python 3, the studio would need a small team dedicated to nothing but maintaining the interpreter. For Jamie, the wins are type declarations and dataclasses, since type hints are inconsistent in 2.7 and EVE hand-rolls lots of small data-packet classes. Michael pointed out that 2.4 million lines is exactly the scenario typed Python and checkers like Pyrefly and ty were built for, asking "is our code still consistent?" in an automated way, and Jamie agreed language-level correctness matters when test coverage is reasonable but incomplete. Thomas reported a 15 to 20 percent performance improvement across the board essentially for free, with string handling the one area where all-Unicode strings carry overhead compared with operating on bytes.

asyncio was never an option, and free-threading is on the radar

Thomas said asyncio is "out of the window" for them. It requires marking every function that can yield with async, and 2.4 million lines were written without that in mind. Michael raised colored functions and TonIO, the multi-threaded async runtime from Granian creator Giovanni Barillari that Michael covered on episode 561. Thomas saw that talk at EuroPython but explained that CARBON already runs all I/O on background worker threads, with the scheduler acting as the synchronization point into the main Python thread, so integrating a new runtime would not be straightforward. Free-threaded Python is on their radar, especially for resource loading, but many places implicitly rely on the GIL and would need to be found first. Michael confessed free-threading scares him, not because of his own code but because of libraries that assumed the GIL and may hide latent race conditions that turn into deadlocks once locks get added. Thomas pointed to the community compatibility tracker, which Michael pulled up on screen, noting its tested-in-CI, PyPI release, and first-supported-version columns and how data science heavy it is. Both agreed asyncio solves I/O concurrency but does nothing for computation, which is why data scientists are once again leading the way.

What EVE Online actually is: one universe, full loss, and an economy that economists study

Jamie described EVE as a classic MMORPG running since 2003, set in a far future where humanity passed through a wormhole and society collapsed and rebuilt in a new form. It is an open-world sandbox with mining, trading, piracy, and a lot of combat, and it is "full loss": a destroyed ship is removed from the game. Everyone plays on a single shard, with about 15,000 characters logged in at a time, and the game holds two Guinness World Records for the most people in a PvP fight. Economists study it because every single transaction is recorded, which Michael called perfect data from a society with zero privacy. Players have built banks, loan schemes, welfare groups, chains of command with scouts and fleet commanders, and in the bigger groups even their own IT departments. Kristinn noted that some people have put running an EVE corporation on their CV and it helped them land a job. Michael connected all of this to Trade Wars on 1990s BBSs, and Jamie traced EVE's lineage to Elite and D&D-style character stats.

EVE Frontier is a fork, not a sequel

Kristinn explained that Frontier forked the EVE Online codebase and shares much of the underlying tech, although the two have diverged enough that the studio has stopped pulling changes across. In EVE Online you arrive in an established society with NPC police and infrastructure, while in Frontier you land in an uninhabited zone and the players have to build everything themselves. It leans toward survival gameplay with more claustrophobic environments, which Kristinn admitted is hard to pull off when your setting is space. Because it is new, Frontier can make dramatic gameplay changes that would be impossible in a game people have played the same way for 20 years. That freedom, and the smaller alpha player base, is exactly why it was the right proving ground for Python 3.

Advice for your own migration: secure the people, integrate constantly, keep it small

Kristinn's advice is to secure the resources and make them untouchable, because a live game team gets derailed by feature work, and "once you start, you have to finish"; pausing is almost wasted effort. For Frontier the studio brought in the contractor Reckon Digital to migrate most of the Python code for exactly that reason. Jamie's team rolled out a new linter this week that they will progressively tighten, so the developers still adding features against 2.7 write forward-compatible code. Thomas's rules are to have good test coverage, migrate in a separate branch, and continuously integrate from the Python 2 side into the Python 3 side so new pain points surface immediately. When his team only merged every other week, it cost a full day each time, which led to his summary: "do it frequent, do it often, keep it small." Kristinn also admitted he went looking for Python 3 war stories before starting and found very few, theorizing that the experience is traumatic enough that nobody wants to write it up afterward, which is part of why this conversation exists.

Interesting Quotes and Stories

"The distinctive factor is the game is full loss. If you kill someone's ship, that's the ship removed from the game. There's no getting things back. Nothing lasts forever. So the game is built on creation and destruction." -- Jamie Bannister

"It's like, what if we lived in a society where there was zero privacy and 100% data sharing? It would be creepy from a human perspective. But from an economics perspective, you have perfect data." -- Michael Kennedy

"To some extent these people are playing the game, but they're not actually in the game, because they are managing the spreadsheets and the communications and all of that." -- Kristinn Sigurbergsson, on the players who run EVE's largest organizations

"The programming resources were scarce and heavily sought after, so very expensive. So part of the reason why they chose Python was to get more junior people and even designers to be able to write sequential code in Python in order to speed up development." -- Kristinn Sigurbergsson

"I think, honestly, we should give the data scientists a little bit more credit for making the two to three change happen." -- Michael Kennedy

"Stackless Python itself was a very great solution at the time. It's basically Golang's goroutines, but in Python, before this coroutine model became so popular as it is in recent years." -- Thomas Dähling

"Oh, it's Git magic. You will find on GitHub repositories where there are commits from Abraham Lincoln and various other historical figures." -- Thomas Dähling, on how the Stackless repo shows as forked from CPython

"It is really amazing how much people can mess with their environment by just changing the Python path, running the wrong version of Python. It's been a struggle for years, honestly." -- Kristinn Sigurbergsson

"If we don't go up to Python 3, we are stuck with having to have a small team that is dedicated to nothing else but maintaining the Python interpreter." -- Thomas Dähling

"It's not my code that scares me. It's all the libraries that are written in Python that, like you all said, a lot of people assumed the GIL was a thing. And how many of those have some little latent race condition that when they figure out what it is, they're going to put locks on it, and then it'll become a latent lock, a deadlock?" -- Michael Kennedy, on free-threaded Python

"If you're doing a lot of damage calculations, you've got one ship shooting another ship. That can make a lot of difference if suddenly your damage changes and now I should have won this fight and now I've lost it because the maths is working out differently." -- Jamie Bannister

"I think there's about 100 gigabytes of agent memory in the DB in the form of pickled Python objects. We can't just write a database script that's gonna do those, because the SQL script isn't gonna be able to serialize and deserialize Python objects." -- Jamie Bannister

"The first release we did, we had hourly downtime, because after like one and a half, two hours, some process somewhere would just stop responding at all. We were a bit in the dark there. It was a very stressful time, and it turned out to be something really, really stupid." -- Kristinn Sigurbergsson

"The transition to Python 3 is an all or nothing step. You can't just put the base interpreter on Python 3 and then keep all your native extensions on Python 2. You need to go all the way. Otherwise, it just doesn't work." -- Thomas Dähling

"I wanted the war stories and I didn't really find a lot of people that had done it. And I'm sure a lot of people had done a very painful Python 3 migration. So my running theory was that nobody really wanted to talk about it after they had done it. It's a traumatic experience, and they just let it go." -- Kristinn Sigurbergsson

"Secure the resources, make them untouchable while you're doing this migration. Because once you start, you have to finish. If you pause it, it's almost wasted effort." -- Kristinn Sigurbergsson

"The more frequent you do these integrations, the less painful they are. If you do it once a month, oh my God. There was a period where we only did it every other week and we would basically spend a whole day just resolving all of the issues. So do it frequent, do it often, keep it small." -- Thomas Dähling

Key Definitions and Terms

Stackless Python: A fork of CPython that adds tasklets, lightweight microthreads scheduled by the interpreter, plus channels for communicating between them. It let EVE developers write sequential-looking code that ran concurrently, but it never advanced past Python 3.8 and was archived in 2025.
Tasklet: The unit of concurrency in Stackless Python. Thousands of tasklets can share one thread, each pausing when it waits on I/O and resuming later, without any async or await keywords in the code.
CARBON: The game engine shared by EVE Online and EVE Frontier, now open source under the MIT license on GitHub. It embeds the Python interpreter and exposes the C++ engine modules to Python.
Destiny and Trinity: EVE's physics engine and graphics engine, respectively. Both are C++, while gameplay logic, networking, and clustering are Python.
Single shard: An MMO architecture where every player shares one universe on one cluster, rather than being split across many independent servers. Any two EVE players who travel to the same place will meet.
Full loss: EVE's rule that a destroyed ship is permanently removed from the game, which is why numerical correctness in combat calculations matters so much.
Fleet commander (FC): The player who gives orders during a large fleet battle, sitting at the top of a chain of command that players built themselves.
Fanfest: The studio's in-person player convention in Reykjavik, where Thomas presented the CARBON upgrade talk in 2025.
GIL (Global Interpreter Lock): The lock in CPython that allows only one thread to execute Python bytecode at a time. Much of EVE's 20-year-old code relies on it implicitly for synchronization.
Free-threaded Python: A build of CPython without the GIL, introduced in Python 3.13, that lets threads run Python code truly in parallel. Adopting it requires finding every place that silently depended on the GIL.
Colored functions: The idea that async functions are a different "color" from regular functions, so anything that calls one must itself become async. This is why asyncio was not viable for a codebase with millions of lines that never used it.
gevent: A coroutine library for standard CPython that monkey patches sockets and threading to run cooperatively. Thomas noted it overtook Stackless in popularity because it could simply be imported.
futurize: A tool from the python-future project, built on 2to3, that rewrites Python 2 idioms into forms that work on both Python 2 and 3. EVE used the subset of its fixers that keep the code shipping on 2.7.
True division versus floor division: In Python 3, the slash operator on two integers returns a float, while Python 2 returned a truncated integer. The double-slash operator does floor division on both.
Old-style classes: Python 2 classes that do not inherit from object. They no longer exist in Python 3, so serialized instances of them cannot be deserialized on a Python 3 node.
pickle: Python's built-in object serialization format. EVE stores roughly 100 gigabytes of NPC agent memory as pickled objects, which have to be readable after the migration.
Protocol Buffers and FlatBuffers: Language-neutral binary serialization formats from Google. EVE already uses protobuf for its APIs and plans to move the pickled data to protobuf or FlatBuffers depending on size.
Mix mode: Jamie's term for running a cluster where individual nodes have been swapped to Python 3 while the rest remain on 2.7, so the migration can be rolled out incrementally rather than on one deployment day.
Embedded interpreter: Running the Python interpreter inside another program's executable rather than as a standalone python command. EVE has shipped this way for 20 years, which is why the standard packaging tools never quite fit.
PyTypeObject: The C structure that defines a Python type in the C API. A change to its layout between 2.7.6 and 2.7.18 caused memory corruption when EVE loaded a module built for the newer version.
PYTHONPATH: The environment variable that tells Python where to look for modules. Kristinn cited it as the classic way developers break their own environments.
uv, uv run, and uvx: A fast Python package and project manager from Astral, and its commands for running a project or a tool in an isolated environment. Thomas is exploring it as the path to a real package manager for the game.
Linter: A tool that checks code for style and correctness problems without running it. Jamie's team just rolled one out to keep new 2.7 code forward-compatible with Python 3.
Dataclass: A Python 3 feature that generates boilerplate for classes that mainly hold data, which Jamie wants because EVE hand-writes many small data-packet classes.
Type hints: Optional annotations describing the expected types of variables and function arguments. Checkers like Pyrefly and ty use them to verify consistency across a large codebase.
TeamCity: The JetBrains continuous integration server Kristinn said helped the Frontier team work through the migration build by build.
MMORPG: Massively multiplayer online role-playing game, the genre EVE has occupied since 2003.
MUD: Multi-user dungeon, the text-based multiplayer games of the 1980s and 1990s that both Michael and Jamie played.
BBS and Trade Wars: Bulletin board systems were dial-up servers from before the web, and Trade Wars was a turn-based space trading game played on them that Michael sees as a spiritual ancestor of EVE.
Elite: A 1984 space trading and combat game that Jamie cited as a major inspiration for EVE.
EuroPython: The largest European Python conference, where Thomas caught up on free-threading and saw the TonIO presentation this July.

Learning Resources

If this episode made you want to go deeper on porting old code, Python concurrency, or adding types to a large codebase, here are some places to start.

Python 3, an Illustrated Tour: A tour of what changed in Python 3 and why it is worth porting Python 2 projects, covering Unicode strings and the other features EVE's team is finally getting to use.
Async Techniques and Examples in Python: Michael's course on the full spectrum of Python concurrency, from async and await and asyncio to threads, multiprocessing, and thread safety. It is the background you want for the Stackless, asyncio, and free-threading discussion.
Rock Solid Python with Python Typing: Everything about type hints and the frameworks built on them, which is the capability Jamie is most looking forward to on a 2.4 million line codebase.
Porting Python 2 Code to Python 3: The official HOWTO that walks through the same staged approach EVE is using, including keeping code running on both versions during the transition.
futurize documentation: The tool EVE used for its first migration stage, with a list of the fixers and which ones stay compatible with Python 2.
Free-threaded Python HOWTO: The official guide to the build without the GIL that Thomas's team is evaluating.
Free-threading compatibility tracker: The community page Michael pulled up on the show, listing which packages with native extensions support free-threaded Python.
Upgrading CARBON to Python 3, EVE Fanfest 2025: Thomas's talk on the Frontier migration that Michael references in the episode.
CARBON engine on GitHub: The open source engine repositories, including the scheduler that replaced Stackless.

Overall Takeaway

Most Python 2 to 3 stories were told years ago, and Kristinn's theory is that the painful ones were never told at all. EVE Online is the exception: a 2.4 million line codebase that predates Django and NumPy, running on a custom interpreter that was archived last year, being moved to Python 3.12 while the game stays online every day. The plan is refreshingly unglamorous. Get the syntax valid while still shipping on 2.7. Hunt down the 6,500 divisions and the 100 gigabytes of pickled objects that parse fine but behave differently. Prove it on Frontier first, then swap the cluster one node at a time.

The lessons travel well beyond games. Secure the people and do not pause once you start. Integrate constantly so the pain stays small. Use a linter to keep new code from digging the hole deeper. And accept that the language work is the easy part, because the real effort lives in the C API, the build system, and the data you serialized twenty years ago. If a studio can take a live universe with 15,000 players through that jump, the old service you have been putting off migrating can probably make it too.

Links from the show

Guests
Jamie Bannister: linkedin.com
Thomas Dähling: linkedin.com
Kristinn Sigurbergsson: linkedin.com

EVE Online: www.eveonline.com
EVE Frontier: evefrontier.com
2.4 million lines of Python: www.eveonline.com
Example of graphics, ships walk-through: www.youtube.com
Stackless: www.stackless.com
futurize: python-future.org
blue: github.com
Dev blog on the Frontier upgrade: evefrontier.com
Upgrading CARBON to Python 3: www.youtube.com
carbon-scheduler: github.com
Scheduling in Carbon: Leaving Stackless Python Behind: www.youtube.com
All of Carbon is on GitHub now: github.com
did a whole course: training.talkpython.fm
Trade Wars BBS (Wikipedia): en.wikipedia.org
TW 3002 - TradeWars 2002, reimagined as a modern browser game: www.reddit.com

Watch this episode on YouTube: youtube.com
Episode #564 deep-dive: talkpython.fm/564
Episode transcripts: talkpython.fm

Theme Song: Developer Rap
🥁 Served in a Flask 🎸: talkpython.fm/flasksong

---== Don't be a stranger ==---
YouTube: youtube.com/@talkpython

Bluesky: @talkpython.fm
Mastodon: @talkpython@fosstodon.org
X.com: @talkpython

Michael on Bluesky: @mkennedy.codes
Michael on Mastodon: @mkennedy@fosstodon.org
Michael on X.com: @mkennedy

Episode Transcript

Collapse transcript

WebVTT format On GitHub

00:00 Every ship in EVE Online eventually undocks and leaves the station.

00:04 This time, it's the whole game.

00:06 EVE is run on Python 2 since its launch in 2003.

00:10 All 2.4 million lines of it on a custom stackless interpreter that stopped at Python 3.8.

00:18 And stackless Python was mothballed last year.

00:21 Now it's destination Python 3 for EVE Online.

00:24 The route runs through 6,500 lines of division that decide who wins a fight

00:29 and 100 gigabytes of pickled Python objects that have to survive the jump intact.

00:34 Kristinn Sigurbergsen was on the show 10 years ago, and he's back, this time with Jamie Bannister,

00:39 who is flying the EVE Online migration right now, and Thomas Darling, who took EVE Frontier through the jump first.

00:47 In EVE, a destroyed ship is gone for good.

00:49 There are no do-overs, and the universe has to stay online the entire time.

00:54 That's a serious migration.

00:56 This is Talk Python To Me, episode 564, recorded Thursday, September 10th, 2026.

01:20 Welcome to Talk Python To Me, the number one Python podcast for developers and data scientists.

01:24 This is your host, Michael Kennedy.

01:26 I'm a PSF fellow who's been coding for over 25 years.

01:31 Let's connect on social media.

01:32 You'll find me and Talk Python on Mastodon, BlueSky, and X.

01:35 The social links are all in your show notes.

01:38 You can find over 10 years of past episodes at talkpython.fm.

01:42 And if you want to be part of the show, you can join our recording live streams.

01:45 That's right.

01:46 We live stream the raw uncut version of each episode on YouTube.

01:50 Just visit talkpython.fm/youtube to see the schedule of upcoming events.

01:54 Be sure to subscribe there and press the bell so you'll get notified anytime we're recording.

01:59 This episode is sponsored by Sentry's Seer.

02:02 If you're tired of debugging in the dark, give Seer a try.

02:05 There are plenty of AI tools that help you write code, but Sentry's Seer is built to

02:09 help you fix it when it breaks.

02:11 Visit talkpython.fm/sentry and use the code talkpython26, all one word, no spaces,

02:17 for $100 in Sentry credits.

02:20 And it's also brought to you by Talk Python Courses.

02:24 Course completion certificates are now live.

02:26 If you finished a course, there's a certificate waiting for you on your account page right now.

02:31 Download it as a PDF or add it to your LinkedIn profile with one click under licenses and certifications.

02:37 Same section as your formal degrees.

02:39 Visit training.Talk Python.fm slash account to see what you've already earned.

02:45 Kristinn, Jamie, and Thomas, welcome to Talk Python To Me.

02:49 Kristinn, welcome back.

02:51 Jamie Thomas, hello.

02:52 Welcome to the show.

02:53 Hello.

02:53 Hello, Michael.

02:54 Hello.

02:55 Great to be back.

02:56 Yeah, yeah.

02:56 Great to have you all here.

02:57 So I'm really excited to be talking about EVE Online again.

03:02 When was it we?

03:03 I think it was 2016, 17.

03:06 It was a while ago.

03:07 I think it was 2016.

03:10 And I think we did even brought up the discussion point of the Python 3 upgrade.

03:16 So it's been on the table for a while.

03:18 Well, we're here to talk about upgrading EVE Online to Python 3.

03:22 Well, Virgin, are you going to start at 3.0 and work your way through all of them,

03:26 or are you going to do a little bit of a skip ahead?

03:29 So we are going to Python 3.12.

03:33 That's what we updated to.

03:34 We first did a step in between to stackless Python 3.8.

03:38 But when we did the work, 3.8 was already end of life, so to speak, and stackless Python didn't go any further.

03:45 So we decided to go to 3.12 and have now worked on our infrastructure pipelines a little bit that any upcoming updates are a lot easier.

03:53 So we have plans to go to 3.15 later this year, for example.

03:57 Oh, fantastic. That's really neat. I'm going to be excited to hear about that.

04:00 Probably some cool performance benefits there and profiling and other types of benefits that you get from those last few versions there.

04:08 We're going to dive into why stackless and all that.

04:12 There's a lot of history here, and it's a really, really big project.

04:15 Before we get to the project itself and the upgrade and so on, let's just start with a quick introduction for each of you.

04:24 I'll let you do that for yourselves.

04:25 Kristinn, you go first.

04:26 We'll go around the squares on the screen here.

04:30 Okay.

04:30 My name is Kristinn Seupersson.

04:32 I'm a director of gameplay engineering for eFrontier, which shares the same technology stack as eOnline.

04:41 I've previously been everything from a game designer to software engineer to technical director of E1 Line and later technical director of platform or SEAR technology segment as well.

04:51 So I was part of the team that did the migration for 2.5.3 for Frontier and now I'm the primary benefit from it.

05:03 So I'm Jamie. I've been an engineer on EVE for about 16 years, mostly on gameplay.

05:11 But that has let me work across almost all the different aspects of the game,

05:15 from the front end of the client all the way through to the back.

05:18 So I've kind of seen all of the guts and the glory inside EVE.

05:23 So I'm at the moment working on the actual migration project for EVE Online itself,

05:28 building on kind of what the others have done previously.

05:31 Excellent. Thomas?

05:33 Yeah, I'm Thomas Delling. I'm a principal programmer on the platform segment for Fenris

05:39 Creations. And I have been leading the initial Python 3 transition for Carbon, like getting

05:45 us out of 2.7 onto Python 3.

05:48 Amazing. What a big project, y'all. What a big project. I think where we should start

05:53 is just what is EVE Online? And I have, of course, EVE Frontier pulled up. But tell us,

06:00 what is EVE Online? Whoever wants to take this one.

06:03 So Eve Online, it's an MMORPG in the classic sense.

06:09 It's been going since 2003.

06:12 So it's a massively multiplayer online game set in the far future where humanity has gone through the wormhole to a new system

06:21 and society has kind of collapsed and then rebuilt in a new form.

06:25 It's a very, what we would call in gaming, an open world sandbox.

06:30 So we support all sorts of gameplay professions from mining, trading, piracy, a lot of combat.

06:38 I think the distinctive factor is the game is full, kind of full loss.

06:44 If you kill someone's ship, that's a ship removed from the game.

06:47 There's no kind of getting things back.

06:50 Nothing lasts forever.

06:51 So the game is built on creation and destruction.

06:54 A big part of it, if I understand it right, is kind of the economy and just the society.

06:59 Like there's a society that forms from the players.

07:03 Yeah, the economy I think, and before I joined to work on the game, I was a big player as well,

07:11 and the economy was a big part of it for me.

07:13 It's been used by, or been studied by economists, because it's, if you think,

07:19 the data that we have on a big, real, on our economy is more accurate and detailed

07:24 than you can kind of get from studying the real world economies, where you've always got imprecise data.

07:29 So we have every single transaction.

07:31 We know everything that's going on.

07:33 You see all the money as it moves around the game.

07:36 That's super interesting.

07:37 Yeah, very, very rich simulation.

07:39 What if we lived in a society where there was zero privacy and 100% data sharing?

07:44 It would be creepy from a human perspective, right?

07:47 Just like governments and alt-territarism and all that kind of stuff.

07:52 But from an economics perspective, you have perfect data, right?

07:56 Yes, yeah.

07:57 and it's interesting how real world things have come up in the game like there's no formal mechanic

08:03 for banks for example but players themselves have built banks they've built uh kind of loan schemes

08:10 organizations have got their own way of paying their members for turning up to operations so they

08:19 they have all the equivalent of like we have welfare groups we have um very centralized

08:24 democratic groups, communist groups, all those kind of different social phenomena have reappeared

08:30 in the game kind of organically. So interesting. And many online games are, you go here, you find a

08:38 server, you join 40 people, and you go have an experience, right? But this is a single, unified,

08:44 one world that everybody is in. They can all interact, all that kind of stuff, right? Yeah.

08:49 So we take place where we call it a single shard.

08:52 So yeah, like you say, it's one universe.

08:55 Any two people in the game, if they go to the same place, they will meet and see each other.

09:00 And so we have, I think our largest kind of player group is about 15,000 characters.

09:10 And so all of those potentially could get together and go to one place at one time and have a huge fight.

09:15 I think we have two Guinness World Records for the most, uh people taking part in a pvp fight wow how do you coordinate that as a as a group you know what

09:26 i mean like how do you coordinate just the conversations amongst the people without it

09:32 just being a cacophony um between in turn ourselves i mean with the players i mean with the players

09:38 if there's 1500 people in like uh some sort of action how can you have conversations and

09:44 communication without it being they built built their own solutions for a lot of this um so they've

09:51 replicated a lot of a chain of command that you get from from real kind of real world and some of

09:55 the bigger groups they have their own it departments uh that take care of of things a lot of groups use

10:02 uh discord for kind of coordination but they have and various other kind of voice uh systems they

10:08 have uh what's called an fc which is like a fleet commander so that's the person uh giving the orders

10:14 and kind of directing the troops.

10:16 And then it kind of, there's a hierarchy, if you like, of communication that goes down.

10:21 So yeah, the bigger groups have built very complex kind of command and control systems

10:28 with themselves.

10:29 I guess that makes sense, but wow.

10:31 And you think of a real army, if there's a thousand people, they don't all just get onto the same audio channel.

10:37 They do have certain people who talk to each other and then they give commands down the line.

10:42 And interesting.

10:42 They replicate the processes like scouts filtering information up.

10:46 They'll have then some people gather that information and feed it into the FC,

10:50 who makes the calls of where they're going to go, what they kind of do next.

10:54 Yeah.

10:54 Kristinn, Thomas, you guys want to add anything to that overview before we dive in?

11:00 Yeah, I would say that this doesn't really happen by its own.

11:03 There are a lot of people that play EVE online that spend an incredible amount of time orchestrating all this.

11:11 not only just the fights themselves, but the reasons behind the fight, gathering everybody up.

11:17 So it is really fascinating to see. And to some extent, these people are playing the game,

11:22 but they're not actually in the game because they are managing the spreadsheets and

11:28 the communications and all of that. And to some extent in certain circles, like people have

11:34 put their experience in managing these larger corporations in EVE onto their CV and it has

11:42 helped them get a job. That's a level of commitment there. Yeah, I can now function as a full

11:50 logistics operation person because I've done it for a thousand people in EVE Online.

11:56 Very much so. Yeah, very much. Crazy. Thomas? Oh, I've got nothing more to add there. I think

12:01 they both described it really well.

12:03 It can sometimes be almost more like a job than a game if you want it to be.

12:07 You learn a lot of valuable things.

12:10 You meet a lot of great people that way as well, right?

12:12 I mean, over the 20 years, so many people that met each other.

12:15 We even had a wedding at one of our FanFests, I believe.

12:20 We've actually had a few, I think.

12:22 Thomas, you gave a great talk at FanFest about upgrading and inspiring the folks.

12:28 That video is online.

12:30 I'll link to it in the show notes.

12:31 We'll maybe get to that in a little bit later.

12:33 So that sounds like a really cool gathering.

12:36 You know, I really, the last couple of years, I've really enjoyed just diving into games.

12:41 And my daughter is getting older.

12:42 And one of the things she really likes to do is sit down and play games together.

12:46 Not even online, it's too big, but sort of cooperative stuff and just sort of exploring.

12:51 And I think that there's, well, I know that studies have shown that there's more loneliness.

12:57 There's fewer third places that people find themselves at and all that.

13:02 Having something like this where you can gather and be with people and immerse yourself in,

13:07 I think it's really neat, honestly.

13:08 As long as you don't take it too far, you know, still go to work and stuff.

13:11 Yeah, yeah.

13:12 I mean, before I joined and as a player, we would organize meetup events at different places around the UK where I was from.

13:22 And so I got to know a lot of people.

13:24 I travel to a lot of places just for the purpose of meeting up with other EVE players.

13:29 So for a lot of people, EVE or a lot of games like this, they can be your primary social connection to a lot of them.

13:38 Let me take us back 35 years or something like that to the days of BBSs.

13:46 Have any of you guys done BBSs? Remember them?

13:49 Yeah, I used to play the MUDs back in the day.

13:51 Oh, I used to play MUDs as well.

13:53 Though that's not what I'm thinking of.

13:54 But actually, MUDs have a really interesting multi-user dungeons.

13:58 They have some really interesting analogies as well.

14:00 But I was thinking of Trade Wars.

14:03 Have any of y'all played Trade Wars from the BBS days?

14:06 I know of it.

14:07 I've never played it.

14:09 It's really interesting.

14:10 It was this multiplayer thing way back in the 90s.

14:13 And you would dial in and you'd have to take turns because only one phone could be connected to the BBS at the time, at a time.

14:21 And it was really, it has some real similar vibes.

14:24 It's, you know, you travel to different planets.

14:26 There's mining, there's an economy, there's trade.

14:28 Anyway, I feel like this is like, what if it could be real?

14:32 Instead of just a little sort of simple game, what if you could make it real with all the nuances

14:37 and intricacies of reality and multiplayer?

14:39 And it's just the realization of what that thing should have become.

14:43 Yeah.

14:45 Because Eve was heavily inspired by a game called Elite, which was one of the very early 3D space games.

14:52 And that had a big world, a big universe of systems.

14:57 Trading was a big part of it.

14:58 And so there's definitely links from there.

15:01 And also classic role-playing games as well.

15:04 Our character stats have similar kind of perception, intelligence, charisma sort of stats.

15:09 Interesting.

15:10 Almost a D&D sort of.

15:12 Very much, yeah.

15:13 Yeah, you can tell there were D&D players in the very early EVE designers, for sure.

15:18 This portion of Talk Python To Me is brought to you by Sentry and Sear AI.

15:23 There are plenty of AI tools that help you write code, but Sentry's Seer is built to help you fix it when it breaks.

15:29 The difference is context.

15:31 Sear isn't just guessing based on syntax.

15:33 It's analyzing your actual Sentry data, your stack traces, logs, and failure patterns.

15:38 Because it has the full context, it can A, spot buggy code in review and help prevent issues before they happen,

15:45 and B, identify the root cause of production errors.

15:49 It can even draft a fix and hand the work off to an agent-like cursor to open a PR for you.

15:54 Seer turns Sentry into a complete loop.

15:56 You have your traces, errors, logs, and replays to see the problem, and now AI to help solve it.

16:01 Join millions of devs at companies like Claude, Disney+, and even Talk Python, who use Sentry to move faster.

16:08 Check them out at talkpython.fm/sentry and use code talkpython26, all one word, for $100 in Sentry credits.

16:17 Thank you to Sentry for supporting Talk Python.

16:20 We're going to also talk about this thing called Eve Frontier, which is a little different,

16:26 but very much inspired, shares a lot of the code base.

16:28 So give us a quick comparison to Eve Frontier, which I'm sure fewer people know about.

16:33 Yeah, sure.

16:34 So what matters for the tech is that we've essentially forked Eve Online, the code base,

16:41 and are sharing a lot of the underlying tech.

16:44 We've made a lot of modifications.

16:45 We haven't been kind of pulling in stuff lately because it's just, it's become too different.

16:53 Where it differs from EVE Online is maybe if you play EVE Online, you very much come into a society.

17:00 Part of it is definitely player built, but part of it is also kind of NBC built.

17:05 The environment is there's a police, there's everything.

17:08 Whereas in Frontier, you're more coming into like an unbuilt, uninhabited zone

17:12 and you're the first kind of writers in that universe.

17:15 And it's up to the players to build that society up and kind of build up all of that infrastructure.

17:23 It doesn't really differ from even a lot of like the design paradigms.

17:29 I think both games aim to give players as much player agency as they possibly can.

17:34 But this also allows us to make like a lot more dramatic changes in the gameplay itself.

17:42 it's very hard to change the gameplay on evil line for example very drastically because honestly

17:47 like after 20 years that's the game that people are playing they don't really like they didn't

17:51 play it for 20 years to change all of it so uh we've been doing that it's more survival

17:58 survival based it's more like uh like a survival game than it is uh uh not necessarily more than

18:05 an mmo but it definitely has a lot more survival elements to it uh yeah and the environments are

18:10 a little bit more kind of claustrophobic and smaller.

18:13 Obviously, that's very hard in a space game because space is pretty vast.

18:17 Yeah, amazing.

18:18 Okay, really, really neat.

18:20 Now let's talk about the reason that I reached out to you, which is I saw this article or post from you all

18:27 that said the move to Python 3 begins.

18:29 And, you know, credit to you all.

18:31 This is a really fun article that it's like, it's told as sort of the next phase of a game

18:37 or something where it's kind of a mission in the game, I guess.

18:42 But a mission for you all, right?

18:44 Yeah, I think that we kind of got a little bit cheeky with the heading because it isn't necessarily the beginning

18:51 because, like we were saying earlier, Frontier itself is already on Python 3.

18:57 But this is specifically for EVE Online itself.

19:00 It's now actively moving to Python 3.

19:02 And a lot of it is building up on the tech that we've developed ready and kind of start to prove out through Frontier. But yeah, this is kind of our big

19:12 formal announcement that the legacy of 20 plus years of code, we're now kind of actively working

19:20 to migrate. I think we should start maybe with a little bit of the architecture and just a big

19:26 tech picture of EVE Online. When we spoke, Kristinn, way back in the day, you were doing a bunch of

19:32 interesting things. I think you must have been the single largest deployment of Python,

19:37 a Python app on Windows?

19:39 Is that correct?

19:40 I would say yes, right?

19:41 What else would it be?

19:42 That seems very plausible.

19:44 I think it was a very kind of like a niche environment.

19:48 And what I'd often run into is that I would try to take some libraries and try to use them and they just wouldn't work on Windows.

19:54 Like somebody broke it and nobody cared because they weren't really using it on Windows,

19:59 except we were.

20:01 Obviously, like at that time, like 10 years ago, we couldn't really use a lot of, especially native Python

20:07 libraries.

20:08 I can't even remember if we had removed-

20:10 we must have removed.

20:11 We had our own importer as well, which made it also difficult to use third party libraries.

20:17 But I think we fixed that.

20:20 Yeah, there's a lot of-

20:21 I mean, it's a very old project.

20:23 And a lot of the early beginnings in the code base were before a lot of the things in Python

20:27 actually being standardized to a degree.

20:30 plus the special circumstances of having a packaged program with its own Python interpreter

20:37 just makes certain things more interesting.

20:39 Like, you know, as Kristen alluded to, like you can't just import a random public package

20:44 because we don't have pip as an example, right?

20:47 Or uv these days.

20:48 But like, there's a lot of interesting problems you need to solve when you have such a large deployment,

20:53 for sure.

20:53 Yeah, there was a lot of custom stuff.

20:55 When did this first get created?

20:56 So I know we spoke in 2016, but it's been around for more like 20 years, right?

21:00 So 2005, 2006?

21:04 I think it's earlier than that.

21:06 It's like the game is released that 2003, the development started in 99, I think.

21:15 They didn't really choose Python.

21:18 I don't know the exact date of that, but like there was, that is an interesting story as well.

21:25 Like why they chose Python, which was actually rather new at the time.

21:29 Yeah, I was actually wondering about that as well.

21:32 I'm glad you chose Python.

21:33 I think it's really cool.

21:34 And I'm sure you all really enjoy working on it.

21:36 And this has got to be an exciting project to say.

21:39 We're moving to 3.15.

21:41 We can use uv.

21:42 We can import libraries, like all the exciting stuff.

21:46 But at the time, it was pretty new.

21:48 And we said, well, we could run this much faster and easier if we wrote it,

21:52 I don't know, with C++, Java, C#, something like that, right?

21:57 I think a lot of the core libraries, they are in C++, so our physics engine, which we call Destiny, that's all C++. Trinity, our graphics engine, similarly. But yeah, most of our kind of higher level, all the gameplay logic, a lot of the network stack that we have, that's all built, or clustering stack, that's all built in Python.

22:25 Yeah, probably add to that as well is that like the time when we started or not me, even though I've been here for a very long time, it hasn't been that long.

22:37 But when they started writing EVE, like that was around the.com era and kind of the older listeners probably remember that era.

22:45 But the programming resources were scarce and heavily sought after.

22:49 So very expensive.

22:51 So part of the reason why they chose Python was to get more junior people and even designers

22:57 to be able to write sequential code in Python in order to speed up development.

23:04 It was all about development speed.

23:06 Especially in Iceland, right, where you've got a much smaller pool of people to work from.

23:12 Honestly, that's kind of a cool success story, right?

23:15 Like bring more people in to help build the thing that we're building.

23:19 Yeah, absolutely.

23:21 Yeah, absolutely. And as Kristin just said, allowing people to write sequential, simple, straightforward Python code as well, which then led to us discovering stackless Python, integrating that one, and then building an abstraction on top that people don't realize that they are writing code that runs asynchronously.

23:40 And it enabled us to scale to the degree that we're able to scale with thousands of people being in the same solar system and so on.

23:47 Yeah, you needed concurrency before Python was inspired by concurrency.

23:52 Maybe we should talk a little bit about free-threaded Python later, as well as the possibilities there.

23:58 But I just want to kind of touch on this historical angle a little bit more.

24:02 To me, there's two really significant milestones that matured Python.

24:07 I'm sure there are more, but these are the ones that stand out in my mind.

24:10 They both are around 2006 or so time frame.

24:14 So one is Django coming to Python, really making it a first class way to build web apps.

24:21 People start building on top of that.

24:23 That puts a lot of pressure to make Python operate a little bit differently, right?

24:28 10 years later, we've got Instagram running the largest Django deployment in the world

24:32 and doing things like trying to juggle weird memory stuff by turning off the GC, for example,

24:39 entirely and other things along those lines.

24:41 The other one around 2006 as well is the invention of NumPy and then later Jupyter and the whole data science stack, which also put really hard computational pressure on Python rather than, say, the parallelism of web apps and so on.

24:56 You all were before both of those, right?

24:58 So it was early, early days.

25:00 Yeah.

25:01 I mean, what we also found was, or what I found in some of work here is the original developers had to invent a lot of things which didn't exist.

25:11 So like the containerized system, a way to kind of cluster an application across multiple

25:19 CPUs and notes.

25:21 There wasn't really a lot of easy off the shelf frameworks for that kind of thing.

25:24 So they had to roll their own.

25:27 Even logging and telemetry systems, there weren't great ones of those.

25:32 So what we found from the early years is we have a lot of in-house tools, which we made

25:39 or the people at the time made, which are now using the models that you see being replicated

25:48 by much more popular systems.

25:50 So whether it be Redis for a storage system or your containers as well.

25:56 The way that maybe we can talk about this a bit more in a bit is the way we distribute

26:00 EVE as an application across about 200 nodes in our server cluster.

26:07 And each of those has certain jobs to do.

26:11 And so we have to deploy and configure all of those as one giant cluster.

26:16 And we have our own coordination tools that have been built over the years.

26:20 Nowadays, if you were starting this from scratch, you would have so many off-the-shelf solutions that we just didn't have the options for.

26:26 Oh, you just set up Kubernetes and then do this auto-scaling thing.

26:30 And it's, oh, yeah, it's just easy.

26:33 I would also like to add, I feel like the data scientists kind of grabbing onto Python and

26:40 I feel like pushing it towards Python 3 because I think without that army going into Python

26:48 3, I feel like it mirrors the rationale of why we chose to go into stackless Python.

26:53 You were giving basically a tool to a bunch of people that aren't really programmers,

26:57 but they need to write code.

26:59 So I feel like that is an interesting thing.

27:03 Yeah, you know, I've forgotten the 2v3 wars and all that.

27:07 Now, yeah, it's easy to forget the details of that from, it's been a while now, actually,

27:12 but you're right.

27:13 It was the data scientists that were going into Python 3, and the rest of the folks were

27:18 kind of dragging their feet because they had existing code bases that needed to keep running

27:23 and keep being migrated, whereas the data science projects are more frequently like,

27:27 we'll start a new project to answer this question or to study this new bit of data.

27:32 So they had many more opportunities to choose the most modern tools.

27:37 And I think, honestly, we should give the data scientists a little bit more credit of making the two to three change happen.

27:44 All right, well, let's talk stackless Python because this is one of the foundations

27:49 that you all had to migrate from.

27:51 And the journey begins, The move to Python 3 begins article talks about upgrading from 2.5 to 2.7 not too long ago.

28:02 And then finally up to 3.8.

28:06 What is it about Stackless Python?

28:07 I know it has strong origins with you all because some of the core developers worked on it and refined it there.

28:14 But this far down the line, I imagine there's a lot of people that don't know what this is.

28:19 Yeah, absolutely.

28:21 Stackless Python itself was a very great solution at the time, right?

28:25 I mean, it's basically Golang's GoRoutines, but in Python, before things like this co-routine model

28:32 became so popular as it is in recent years.

28:35 A big problem that we faced was, on the one hand, that because of the customizations

28:42 we had to do to the network stack to make all the magic of people writing sequential code, but it does end up not blocking

28:49 the main process.

28:51 That was a big learning curve for a lot of people.

28:54 And also always something that even the experienced developers tended to forget about in a moment.

29:00 And they're like, oh, yeah, hang on, there's a DB call.

29:02 Of course, that we yield.

29:03 I need to put a lock here or whatever.

29:06 So the model in itself is great, but it comes with a few learning hurdles, I want to say.

29:15 And it served us really well.

29:17 One problem that we had over the years that had become more and more interesting, especially

29:22 last 15 years, I want to say, is that we relied on a heavily customized version of it. So stackless

29:28 Python itself is a fork of the CPython interpreter with a lot of changes in the interpreter. And

29:35 we put some changes on top because doing a real-time game of sorts requires some solutions that go

29:43 very far down the tech stack to be performant. Like if you want to get certain metrics, if you want

29:47 to control how memory is allocated and so on and so forth right so all these customizations and

29:53 also meant that made it very difficult for us to like always go to the next newer version of python

29:58 as the article current calls out like you know there was some code that was still python 2.5

30:02 when we did the native extension porting to python 3 we discovered things from python 2.3 that we were

30:07 still using that shouldn't have been used but we somehow kept around on our customized version of

30:12 of Stackless Python.

30:14 This portion of Talk Python is brought to you by Talk Python courses.

30:18 Here's the thing that always bug me.

30:19 You finish one of our courses, that's hours of video, a pile of code you actually wrote,

30:24 and real skills you didn't have a month before, and then nothing happens.

30:28 No paper, no credential, nothing to show for it.

30:31 So we fixed it.

30:32 Every Talk Python course now generates a completion certificate automatically.

30:36 Go to your account page in your dashboard section, scroll down to your completed courses and click certificate.

30:42 That's the whole process.

30:44 Two things you can do with these course completion certificates, download the full PDF,

30:48 which is handy if your employer reimburses training or gives you credit for finishing it.

30:53 Or you can make the certificate public and hit share on LinkedIn, which adds it to your LinkedIn profile under licenses and certifications.

31:01 Not a poster that scrolls away in a day, an actual credential sitting on your profile

31:05 where your manager and recruiters can see it.

31:07 Plus, if you've been taking our courses for a while, you've probably earned several of these without even knowing they existed.

31:13 Just visit training.Talk Python.fm slash account and collect them.

31:18 Thanks to all of you who have taken a Talk Python course.

31:21 It's a great way to support the podcast.

31:24 Another aspect certainly is that stack lists had become like a sort of niche project in the ecosystem overall.

31:32 With Python 3, we have async.io built on the twisted model.

31:37 And even in the Python 2 world, G event had become a lot more famous than Stackless Python,

31:43 presumably because you could just import G event in a standard CPython interpreter,

31:47 while Stackless Python is more complicated to get started with.

31:53 So-

31:53 So Stackless Python is basically its own runtime.

31:56 Yeah, it's basically-

31:57 Yeah.

31:57 Yeah, very much that.

31:58 It's like its own Python runtime.

32:00 You kind of need your own.

32:01 Like, it has a stackless lib, so to speak, which basically monkey patches similar to what

32:06 Gvent does all of the threading and socket modules and so on to be able to do them in non-blocking fashion.

32:12 And yeah, all these things.

32:14 So that's really, I mean, that's what made it possible.

32:17 That's really awesome.

32:18 One thing, just, I don't know if you all know this, but how is it possible that it says it's forked from CPython on GitHub

32:25 when CPython itself didn't go to GitHub until like the 20s, you know, 2020, I don't know what it was,

32:31 but maybe some GitHub magic, I don't know.

32:34 Oh, it's git magic.

32:35 You will find on GitHub there are repositories where there are commits from Abraham Lincoln and various other historical figures because you can just make up all the history, right?

32:45 Interesting.

32:45 Yeah, it is nice, though, actually, that it shows it.

32:48 If you go to the repo now, it says that it was archived in 2025, which, you know, not every project is meant to last forever.

32:57 But it made it possible for you.

32:58 Although I'm sure that you would think, you know, all right.

33:02 You're like, wouldn't it be cool if we could use, I don't know, libraries like Pydantic or FastAPI or AIO Redis or I don't know, whatever, right?

33:11 Anything that is new and fancy.

33:13 And then you go back and you look and you're like, well, but not here.

33:17 Not on Stackless, right?

33:18 Like that was one of the major problems, wasn't it?

33:21 Yes, this was a major headache for us.

33:24 But for every native extension-

33:26 or not just native extensions-- for every Python package we took on, we had to take a really close look

33:32 about how can we integrate it in our stack.

33:34 Again, we didn't have a package manager.

33:36 We still don't have it, but we're working on it.

33:39 And that just meant that you would take the zip file from PyPy, for example, extract it into our environment,

33:45 and then see what happens.

33:47 If that then happened to be something that had a native extension, well, good luck.

33:51 Does this support our minimum spec environment?

33:53 does this run on all our environments?

33:55 Does this work with our version of the forked Python interpreter?

34:00 And we have discovered a few cases where we took on modules that were built for 2.7.18,

34:08 and they then corrupted memory because in 2.7.6, the memory layout of the PyType object changed,

34:14 and we didn't have that change.

34:16 And yeah, things like that.

34:17 So we had to fight the way forward.

34:20 Jamie, you want to jump in?

34:21 Yeah, no, I was just going to say when it comes to support as well, because we support Mac and we have in the past,

34:28 but we don't now support Linux as well.

34:30 So sometimes with a native extension, we've got to figure out does it even support all of our clients?

34:38 Our servers run on a Windows cluster.

34:42 Most of our players are running on Windows clients.

34:45 Mac is also kind of a smaller majority.

34:48 And like I say, we used to also support Linux.

34:51 So keeping all of those in sync as well with all of these extensions has been problematic at times.

34:57 Yeah.

34:58 Sure, that's an understatement.

34:59 Earlier, when you guys mentioned how you deploy as an app, or.app,.exe, whatever, or to your server clusters, how do you do that?

35:09 That's honestly one of the biggest challenges of Python that many attempts have been made, but I honestly don't think that there's a good answer, a really foolproof answer yet.

35:18 And until it's built into CPython itself with a --build flag or something, I think it's going to be 90, 95% solutions.

35:28 But you all have something that works.

35:29 What are you doing?

35:31 So effectively, I mean, with Python 3, this all got a lot easier because you can initialize the interpreter a lot more straightforward.

35:38 But what we have been doing for the last 20 years is effectively we are embedding the Python interpreter in a custom binary that we have, a custom library that we use.

35:48 And from that moment onwards, we basically just heavily customize what paths are available to the interpreter

35:55 and what environment variables are considered, all these little things.

36:00 And then we package up the remaining things that we need in a way that it fits this layout that we set up

36:05 in the embedded interpreter.

36:07 But yeah, it's also more like a 75% solution than a real one because the Python build process in itself

36:14 requires some system-specific paths.

36:17 We had to go to great lengths to patch that out.

36:20 Not necessarily in the best way possible either, but it gets us there.

36:24 If you're shipping it, that's off you all.

36:27 That's pretty impressive.

36:28 It was kind of like that we have a solution for our players, but also we built a lot of tools for game designers

36:35 to use and stuff like that.

36:36 And it's convenient to write them in Python because we can share some of the code base of the actual game,

36:41 game logic, for example, when we're creating static data.

36:44 And that has been like a very painful experience of distributing Python,

36:50 like Python, basically a Python script to developers, because there is really amazing how much people can mess with their environment

36:59 by just changing the Python path, running the wrong version of Python.

37:04 It's been a struggle for years, honestly.

37:07 Even if we get a new hire, the first few days are often spent setting the machines up.

37:13 And inevitably, as Windows slightly changes from release to release, it will or won't come with certain things.

37:21 Sometimes we have to disable the built-in Python that ships with Windows just so that we can use our own specific Python 2.7 versus Python 3.

37:31 So that's also quite a lot of fun that we have to get started out.

37:36 What about things like uv Run or uvx in the future?

37:41 There's some interesting possibilities there, potentially.

37:44 Absolutely.

37:45 And as I said, we are working on it.

37:47 We are exploring the angle there.

37:50 Because it is still a bit of a hassle of getting things into our special environment for the game itself.

37:58 We have historically had also the issue that we were using a different built environment

38:02 than the standard Python distributions, right?

38:05 Partially because there's a lot of code that would need to be migrated.

38:08 Partially because it works.

38:11 We don't want to spend resources on that at the moment, partially because, well, especially with Stackless Python,

38:18 you could only go so far until there's a new compiler version that then breaks some internals, and you would have

38:22 to do major rewrites.

38:24 So these issues have existed, but we want to be able to be in a place where a developer can also

38:31 just say, "uv install this package into my game environment," then be able to use it.

38:37 Needs to solve a few things, like if it's a native extension, needs to find the right build environment and so on, which is non-trivial, unfortunately.

38:45 What workflows do you think, do you see becoming possible that were much, much harder once you

38:50 have nice package managers and maybe more reuse, right? For example, you could ship versions of

38:57 things more easily than, you know, as one monolithic element. Yeah, for sure. I mean, for example,

39:03 we recently open-sourced basically the game engine, right? And there is, I can see a world where you

39:09 can just uv install or pip install these things into your custom interpreter to get going or into

39:15 your Python interpreter to get going without requiring our custom interpreter. For our internal

39:21 workflows, I don't think, I mean, sure, the acquisition of an external package will definitely

39:27 get a lot easier, but many of the other workflows probably won't change that much since we,

39:33 I was in 2011 or 12 or so, we added an interpreter mode to the game engine itself, so you can

39:38 can basically start the game engine with a special flag, and then it basically starts

39:42 the Python REPL, and you get as much of the standard Python interpreter behavior as possible.

39:49 That's cool. So you can kind of poke around that in the game memory and values and data

39:54 structures as you want, right?

39:56 I guess the director in me doesn't really, it doesn't see it as all negative that there's

40:03 a bit of a hassle to add a new dependency.

40:07 I was thinking about that too, Kristinn. I was. And you can see these headlines of all the supply chain issues and you're like, well, I mean, we should worry about it, but not to the degree that other people with 200 dependencies do, right?

40:19 Yeah, definitely. And I'm not arguing necessarily for making it hard, but you can see a reality when you have a large development team and people just kind of flippantly add the dependency that causes issue. It's very vulnerable to individual mistakes.

40:38 Yeah, and also just duplication of now there's three libraries that kind of handle the same thing, but not entirely.

40:45 On the flip side, if it's harder to upgrade, you're also less likely to actually upgrade it out of the.

40:53 What libraries or tools or experiences did you all see out there and you're like,

40:58 all right, we have to upgrade.

41:00 Were there certain packages you're like, oh gosh, we really, really would benefit

41:04 from using this one or having this feature like the JIT or something in Python

41:10 that makes such a difference that it's worth all this effort.

41:13 So, I mean, Jamie, maybe you can talk about the Python side a little bit,

41:16 but from the game engine perspective, the main driver for really wanting

41:22 to get the upgrade project pushed through was the native backline that we worked on in the early 2020s.

41:31 It has just like Python 2.7 was not a way of silicon to begin with.

41:36 Then a lot of other changes that had happened, like macOS behaves very different with Unicode

41:42 than Windows does when it comes to system calls and so on.

41:45 Historically, before we had the native MacWords, clientists would go through a translation layer similar to Wine for Linux.

41:52 So we really pushed for it because we just saw that, okay, if we don't go up to Python 3,

41:58 we are stuck with having a small team that is dedicated to nothing else but maintaining the Python interpreter.

42:07 That seemed like it's one possible solution, but not necessarily the best one if you want to just go

42:14 rapidly than to add new features or iterate on other existing functionality.

42:19 I think for me as a gameplay programmer who's mostly using the Python framework that we have,

42:26 things that we've sorely been lacking for a long time is a good way of type declaration.

42:31 It's sort of there in Python 2.7 if you're using the right IDE, but it's not consistent.

42:38 That gets a lot better as you move through the Python 3 versions.

42:41 Another thing we end up making a lot of is what's effectively a data class,

42:46 where we just want to kind of define a small packet of data in a particular structure

42:50 and pass it around and do things.

42:52 Having kind of the language support that at a more fundamental level is kind of something I'm really looking forward to getting.

42:59 Yeah, that's really neat.

43:00 And think about such a large code base.

43:02 Was it 2.4 million lines of code, something like that?

43:06 That's non-trivial.

43:06 I mean, that's kind of the scenario that typed Python is perfect for and could have been invented for before AI was a thing,

43:14 which also helps.

43:15 But having the ability to use tools like Pyrefly or TY and ask the question in an automated way,

43:21 is our code still consistent?

43:23 That's pretty neat, right?

43:24 Yeah, I think given the size of our code base and the fact that we have reasonable test coverage,

43:32 but we don't have complete by any means.

43:34 So the more we can put in a kind of leverage from the language to help prove correctness and find issues,

43:41 that's going to lead to a better product and less issues.

43:44 Yeah.

43:44 Last question before we actually get into the details of the migration and how you all did it.

43:49 What about free-threaded Python?

43:51 You started with this concept of high concurrence, so much so we need concurrency that we're going to get our own runtime and maintain it.

43:59 Are you considering free-threaded Python?

44:01 Is this interesting in any way?

44:03 Or is it all asyncio driven?

44:06 What are you considering here?

44:08 We are looking at it.

44:09 I've actually just been to Europe iPhone back in July and attended some of the very

44:15 interesting presentations on where 3.30 is heading.

44:19 There is a lot of work ahead of us to adopt it, because there are many places which just implicitly rely on the GIL for synchronization.

44:27 We will need to identify those.

44:29 There are some interesting parts where I think we could benefit from it when it comes to our resource loading system, for example.

44:38 But yeah, it's on our radar.

44:40 We are interested.

44:41 It's not necessarily going to be the biggest performance improvement that we could possibly get, I think.

44:49 But once it's there, we also have more possibility or more options to explore as well

44:54 and what we might be able to do different.

44:56 Yeah.

44:56 I imagine that AsyncIO, given how much network, Redis, etc., stuff you do, will give you a pretty mega boost

45:03 as well as just the performance of CPython has gotten so much better, right?

45:08 Yeah, we've gotten, like, what was it, 15% to 20% improvement across the board for most things.

45:14 Like, there's the one-

45:16 like, string handling obviously changed.

45:18 You can't really compare that with Python 2, because now it's all Unicode.

45:22 That always bears a little bit of overhead compared to operating on bytes directly.

45:27 But so far, we have been very happy with the performance improvements that have gotten-- that we basically

45:33 gotten for free, right?

45:35 On the topic of async I/O, though, unfortunately, we can't really benefit from that because you

45:40 need to annotate your functions with async and so on.

45:45 And with the 2.4 million lines of Python code written without that in mind, it had become quite quickly

45:51 quite clear that, unfortunately, that one is out of the window for us.

45:56 Interesting.

45:58 We did discuss it at the time, whether it was just completely out of scope of everything that we were doing.

46:07 You know, one thing in terms of colored functions, that is one thing I just talked to Joe from the Gradient Project

46:14 about Tone.io, which is a multi-threaded async runtime in Python written in Rust, but implemented in Rust.

46:22 But it has this way of running without necessarily decorating the thread.

46:26 Anyway, maybe, I don't know, maybe it's interesting.

46:28 Who knows?

46:29 I attended his talk at EuroPython.

46:31 Oh, yeah, yeah.

46:32 And it looked interesting.

46:36 But on the other hand, we are doing a lot of similar things underneath the hood already.

46:40 So basically, all of the I/O already happens on multiple worker threads in the background.

46:45 Interesting.

46:46 Yeah.

46:47 Really just that we have-

46:48 I mean, the main reason why-

46:50 the main way that we facilitated Stackless back then at the new schedule and what you'll know

46:54 is that it acts as a synchronization point to get the data from those worker threads

46:59 into the main thread that's running Python.

47:00 And I've looked at Toneio a little bit, but even that is not very straightforward to integrate

47:06 in our stack, unfortunately. So doing comparisons is a bit difficult.

47:10 Yeah, honestly, and the free-threaded stuff, honestly, it scares me. I used to do tons of

47:16 concurrent programming, threads, events, signal, locks, all that kind of stuff. It's not my code

47:22 that scares me. It's all the libraries that are written in Python that, like you all said,

47:27 a lot of people assume the GIL was a thing and how many of those have some little latent race

47:33 condition that when they figure out what it is they're gonna put locks on it then it'll become

47:36 a latent lock a deadlock you know what i mean like just having faith that all the foundational

47:41 stuff has really been through the ringer of pre-threading i don't know scares me i'm optimistic

47:46 for it and i want it to exist but it scares me yeah there's a lot of work for sure you know also

47:50 like the other side that the single threaded use case shall not degrade in performance either

47:54 you know it's there are many things but that being said like even with the

47:59 asynchronous programming that we that we use underneath the hood we are already

48:02 in the world of having to be careful what we put a lock on and whatnot right

48:07 for you guys you all are in this world but I think 99% of the Python people

48:13 don't even think about locks and race conditions you know and that's I think that's where

48:18 we're going to have to as a community work through the issues to make these

48:23 dependencies stable Yeah, absolutely.

48:26 There is actually-- I think there is a community effort going on, as far as I remember, where-

48:32 is the URL really rb-free-threaded yet or something?

48:35 So there's a project where they basically track all of the public libraries in PyPy

48:40 and how they behave under free-threaded Python already.

48:43 So we also-- well, I keep an eye on that as well to see where that is happening and where there may be some

48:49 of our dependencies in there.

48:50 There's the compatibility, the free-threading compatibility status checker.

48:54 It's got the tested in CI, the PyPI release, when it was first supported, the free threading bit.

49:00 It's very data science heavy though, isn't it?

49:03 Yeah, I guess it's once again the data scientists that are leading the way here.

49:06 I guess they're also mostly benefiting from a free thread in Python, given that they are doing a lot of number crunching.

49:11 Right, exactly.

49:12 Because I think async.io, even though it may not apply to you all, does solve the problem of I got to talk to the database

49:17 a whole bunch of times concurrently or to microservices concurrently because it's the IO bit,

49:25 but when you got to do computation, it doesn't help you a bit.

49:28 So yeah, you're right.

49:29 I think that's fundamentally why.

49:30 All right, let's talk the actual migration.

49:34 So you all started by just making sure, well, first converting everything to run

49:39 in two and three, right?

49:41 Tell us about that.

49:43 Yeah.

49:45 All right, so I'm going to take this.

49:47 So a lot of the work that kind of came our way was to take our 2.4 million lines of code that we have

49:57 and bring it forward so that we can eventually end up everything on Python 3.

50:01 What we found is, like we mentioned earlier on, we still have a lot of code that was technically 2.4, 2.5,

50:10 or using 2.4 and 2.5 kind of idioms, which are no longer suitable in Python 3.

50:15 So the very first step that we've taken is what we're calling legacy to 2.7,

50:20 which is get all of those legacy bits of code up to 2.7 standard, because a lot of those changes then become more forwards compatible into 3.

50:30 We've actually already released and deployed the first part of that a few weeks ago.

50:36 That was around the time that we wrote that blog.

50:38 And for a lot of that, we use these Futurize scripts, all the fixes from Futurize.

50:44 And what they do is they take one particular idiom of Python and update it to Python 3.

50:51 There's a subset of those which you can run whilst still being 2.7 compatible.

50:58 And so those are the ones that we've done so far.

51:00 Yeah, and you just basically got them all moved to a way that theoretically would run on Python 3

51:07 because for people who don't know, some of the older ways of working in Python 2

51:10 were literally incompatible, like ways of declaring exceptions, handling some of the string stuff.

51:19 Yeah, the big one we found a lot of was testing membership in a dictionary.

51:24 So now you do if X in Y.

51:27 In the older Pythons, you would do if there was a has key function.

51:32 That is no longer supported in Pyth 3.

51:35 But we still have a lot of code, or we did have a lot of code that was using has key.

51:39 So we had to change all of those to using the in state in operator.

51:43 So that's the kind of thing that we've kind of done in this first phase.

51:48 What we found was we were actually 94% of our code was technically Python 3 syntactically valid.

51:58 We've now got up to 99.8%.

52:02 So that's syntactically valid.

52:03 It doesn't mean it's going to do the right thing when you put it in Python.

52:07 The classic example is division.

52:09 Being a game that does a lot of things with a lot of numbers, we have about 6,500 lines that feature a division.

52:17 some of those are going to break if we just switch to Python 3 without doing anything with them

52:23 because the way it changes from integer to float division.

52:28 So that's going to be part of the next phase that we're going to get into.

52:31 Yeah, that was one of the main deals.

52:33 If you have x slash y in Python 2, that would do basically integer division, truncate it.

52:40 They take the floor of it more or less, right?

52:42 And then those just become floating point numbers, which that's terrible for like this thing equal,

52:46 equal that thing and all, it's-

52:48 And if you're doing a lot of damage calculations, you've got one ship shooting another ship,

52:54 that can make a lot of difference if suddenly your damage changes and now I should have won this fight

52:58 and now I've lost it because the maths is working out differently.

53:01 So that's gonna be important for us to get right.

53:04 Yeah, absolutely.

53:05 Okay, so what next, what happened after that?

53:08 You got it working on Python 2, but syntactically Python 3?

53:12 Yeah, okay, so the next phase, which we're now in at the moment, is we've forked the Python 2 branch into a Python 3 branch.

53:24 Immediately, many, many things failed.

53:26 All of our tests started showing up red and so on.

53:30 And so what we're calling that is our Python 3 unstable.

53:33 And so from there, we need to work through, get all of our tests passing, start getting that more stable.

53:40 At the same time, we've got what we're calling a 2.7 unstable branch, where we're kind of introducing a lot of the shims and the adapters.

53:49 So this is where we can have code that's going to maybe fork if 2 do this, if 3 do this.

53:55 The downside of that is it's going to be a lot slower.

53:57 So that's more of a compatibility proof versus a releasable version.

54:02 But that is going to be our middle ground for code that we can then say,

54:06 we can put this into 2.7 and make it 4.0 compatible, or this is going to be breaking compatible,

54:11 so we need to put it on the three side.

54:14 And yeah, that's our next however many months of work.

54:18 But the other big angle of it, which is really worth calling out, is it's not just the runtime code.

54:24 We have a lot of data that is either data at rest or data going across a network that is serialized Python.

54:33 We need to get all of that compatible.

54:34 So if we have, for example, Python 2 Alt-Star classes and we want to serialize that to send it to your Python 3 node.

54:43 Well, that's not going to work because Python 3 is going to, what's this?

54:46 I don't know what to do with it.

54:47 So as well as changing runtime code, there's a lot of data at rest and data on the network that we also need to change.

54:53 Right.

54:53 You did mention that you're using Redis, and one of the real common things to do is just to pickle the objects

54:59 and put them into Redis.

55:01 Yeah.

55:01 But you know what?

55:02 Memory shape changes from version to version.

55:05 I've literally been writing a blog that's going to come out in the next week or two, a public blog,

55:11 talking about exactly that problem that we have here.

55:14 Is that the agent blog?

55:17 Yes.

55:17 So in EVE, we have a bunch of NPCs that we call agents, nothing to do with LLM agents.

55:25 These are just characters and they have memory.

55:28 So when you talk to them, it's going to remember that you talked to them.

55:32 And then later on, you're going to come back and they'll give you a mission or to go,

55:35 go and fetch me this or go and kill this.

55:38 it saves that bit of memory in the database.

55:40 The way it saves it is as a serialized Python block, so that when it comes back, we can kind of reload the memory

55:48 or reload that bit of memory from the disk.

55:51 We have those going back now for 20 years.

55:54 I think there's about 100 gigabytes of Python memory in the DB, of agent memory in the DB in the form of pickled Python objects.

56:02 So that's a big part we need to go through.

56:07 Because we can't just write a database script that's going to do those, right?

56:10 Because a SQL script isn't going to be able to serialize and deserialize Python objects.

56:15 It has no idea, yeah.

56:16 Yeah.

56:16 Are you considering moving to something that's version independent, like msgspec,

56:23 or Pydantic, or JSON?

56:26 Some of these things, what we want to do is change the format to probably protobuf,

56:31 because we use that for a lot of our kind of communications and storage as well,

56:37 either protobuf or flat buffer, depending on the size.

56:41 But yeah, a big part of our internal and external APIs are all built on top of protobuf.

56:47 They would probably use that.

56:48 Yeah, so that'd be really natural.

56:49 Maybe you could even just not even deserialize it, just give me the stuff and shoot it back over the network

56:54 if it wants in that format anyway, digitally, I guess.

56:57 All right, we'll get really short on time.

56:58 Maybe just take us through the remaining arc of what you all planned out,

57:04 What's still in the story to be told?

57:07 So, yeah, so we've got the data formats that we need to change.

57:12 We've got the runtime code.

57:14 Once we've got those worked out, some of the things we want to start doing are what we call a mix mode,

57:19 which is where we take our cluster of 200 nodes plus all of our clients,

57:24 all of which are 2.7 at the moment, and start to be able to selectively replace one piece of those

57:31 with the Python 3 equivalent.

57:33 So rather than doing a big deployment day where we say, okay, this is the moment, we want to incrementally do it,

57:40 kind of spread that risk and that load at the time.

57:42 Can you imagine the stress of just Oh yeah.

57:45 Pushing the whole thing.

57:46 I've been there for those big deployments, very stressful.

57:51 So by doing this, we can spread that risk and that load over a lot of time,

57:55 but it also means we're developing a lot of tools to support this project that aren't necessarily

58:01 Absolutely required, but they're going to give us a lot better abilities.

58:06 So like this ability to deploy a kind of a heterogeneous mix of types of builds and even mixed.

58:15 Do we want to do some nodes on Linux and some on Windows, for example?

58:19 By doing these tools, that's going to give us a lot of these options.

58:21 Yeah, give you more deployment options and so on, right?

58:24 That's pretty interesting.

58:25 It's an interesting consequence.

58:26 I didn't really consider coming along, but yeah, it makes sense.

58:29 All right.

58:29 Kristinn, what else do you want to say about this?

58:31 migration from maybe from the director high level yeah it's i mean i think it helps a lot that we

58:38 did this for frontier first because one of the problems we had when we started this is that like

58:42 we could start migrating the python code but none of the python code would really work without the

58:47 C++ modules being um importable to python as well and we could start in the python C++ modules but

58:55 like they wouldn't really make sense without the python glue kind of tying it together so

59:00 it was really convenient to do this on a project like Frontier where the

59:05 risk of breaking things is a lot less and like the first release we did we had hourly

59:13 downtime because after like one and a half, two hours the process would just

59:18 some of the process, some process somewhere would just stop responding at all and we were

59:22 like a bit in the dark there it was like a very stressful time and it turned out to be something really really stupid so as it obviously would have but so i think

59:36 i think the team city or the build system helped us a lot obviously being a game with you can

59:42 imagine we don't have a robust robust suite of unit tests for all of our code base like

59:49 i mean even automated tests wasn't really like a big when we started doing this and a lot of the

59:54 code was written in the first place.

59:56 So doing like an incremental thing was kind of hard.

01:00:00 But what we did do when we started, we started just running down the build system

01:00:05 and just getting further and further into like an actual build.

01:00:09 So, yeah.

01:00:10 Yeah, it's hard to test graphics and physics and stuff through automated tests to some degree, right?

01:00:17 And when we started, when I, at least when I started, I started talking to Thomas about this

01:00:22 when I was a technical director in Yves and then we were thinking about it for Yves

01:00:26 and it just, the entire concept of doing it just felt so, just so daunting and like big

01:00:33 that I like couldn't imagine how we would do it.

01:00:36 We did have a good plan, I think, and that plan kind of, we more or less did that for Frontier,

01:00:44 but obviously like the stakes are higher for Yves Online.

01:00:47 So like it's a difficult thing.

01:00:49 And I can do a big difference So a critical thing for EVE is it's an online game,

01:00:54 so the servers need to be running every single day.

01:00:58 We can't afford to be taking the server down for a few hours every day.

01:01:05 We need to kind of keep the product up.

01:01:06 So that's where Frontier kind of running ahead of us was really, really helpful

01:01:09 because they could prove a lot of this out before they got to the point where they needed to be online 24-7.

01:01:15 Yeah, it gives you a lot of flexibility to do all that testing, doesn't it?

01:01:20 Yeah, absolutely.

01:01:21 especially because the transition to Python 3 is like an all or nothing step. You can't just put

01:01:25 like one of your-- you can't just put the base interpreter on Python 3 and then keep all your

01:01:30 native extensions to Python 2, right? You need to go all the way. Otherwise, it just doesn't work.

01:01:36 That was actually the thing that gave-- I think that was the initial very biggest concern that we

01:01:40 had, like, can we actually pull this off by doing this push out all at once? And I think it was also

01:01:45 the main reason why we did this intermediate step of going to Python 3.8, stackless Python 3.8 first,

01:01:50 because then we knew, okay, we are only changing the Python major version.

01:01:55 We're not also changing the async library underneath the hood, which a lot of the code still has some timing dependency somewhere, right?

01:02:02 And all these things.

01:02:04 Yeah, 100%.

01:02:05 Well, congratulations for Frontier and good luck on EVE Online.

01:02:10 It sounds, honestly, it sounds like a really exciting project.

01:02:13 Just a lot of fun to work on, even if it means you can't sleep quite as well at night for a little while.

01:02:18 It's a little stressful.

01:02:19 It is exciting.

01:02:23 I am proud of the team for delivering it to Frontieras.

01:02:27 I'm sure I'll be proud of Eve when they deliver this.

01:02:30 What I was fascinated by was that when I started looking into it, I wanted kind of like the war

01:02:36 stories and I didn't really find a lot of people that had done it.

01:02:40 And I'm sure a lot of people have done a very painful Python 3 migration.

01:02:44 So my running theory was that nobody really wanted to talk about it after they had done.

01:02:49 some traumatic experience and just let it go.

01:02:54 Exactly, exactly.

01:02:55 All right, well, let's close out the show.

01:02:58 People who are maybe trying to accomplish the same thing, they have some older code base,

01:03:02 they're trying to migrate even from maybe like old three to new three, something like that.

01:03:07 Give them really quickly just some parting advice and also people interested in EVE online.

01:03:13 Kristi, you wanna go first?

01:03:15 So parting advice for people that wanna do this.

01:03:18 One of the things that I didn't mention is that we actually used a contractor for a lot

01:03:23 of the stuff called Recon Digital.

01:03:26 They deserve a shout as well.

01:03:27 They did a really good job kind of migrating most of the Python code.

01:03:30 And one of the reasons for it was also that developing a live game, we kind of knew that

01:03:36 Frontier, the development team, would get immediately derailed when they had to do some

01:03:42 feature work and not supporting Python 3.

01:03:44 So that was one of the things.

01:03:46 So what I would probably argue there is like secure the resources, like make them untouchable

01:03:52 while you're doing this migration.

01:03:53 Because once you start, you have to finish it.

01:03:55 If you pause it, it's almost wasted effort.

01:03:58 Yeah, that's a good point.

01:04:00 Yeah, I was going to actually say something similar to that.

01:04:03 Because we've got one group working on pushing ahead with the migration, we've still got a

01:04:08 whole bunch of developers adding features to the game every day.

01:04:11 So they're adding those features against our 2.7.

01:04:14 And so we're putting a lot of work into trying to make sure that they have the new code they're

01:04:20 adding is as forward compatible as possible.

01:04:24 So a big thing we've rolled out this week is a new linter.

01:04:28 So we're getting and we're going to progressively ratchet up the strictness of it.

01:04:35 So those kind of tools to kind of do code quality inspection, I think, are going to be

01:04:39 really important for us going forward.

01:04:40 Yeah, 100% agree.

01:04:43 Yeah, very much that.

01:04:44 I would add the age-old advice of have really good test coverage.

01:04:48 It helps a lot.

01:04:50 But also, touching upon what Jamie just said, if you have a product that's live and you need to migrate it,

01:04:56 make sure that you do this in a separate branch but have constantly ongoing migrations from the Python 2

01:05:02 side to the Python 3 side so that you discover all of these new pain points that may pop up as soon as possible.

01:05:08 This helped us on Frontier as well, where sometimes things would come in that were clearly not

01:05:14 Python 3 compatible, and then we could fix it up right away.

01:05:17 And the more frequent you do these integrations, like the less painful they are.

01:05:21 If you do it once a month, oh my god.

01:05:23 Like there was a period where we only did it like every other week, and we would basically

01:05:27 spend a whole day just resolving all of the issues.

01:05:29 So do it frequent, do it often, keep it small.

01:05:32 Keep it small.

01:05:33 Good advice.

01:05:34 Do it in small, small bite-sized bits.

01:05:36 Yeah, absolutely.

01:05:36 All right, you guys, thank you for being here.

01:05:38 Everyone listening, check out Eve Online, eveonline.com.

01:05:41 Very cool game, very cool ecosystem.

01:05:45 Check out the video I linked that says This Is Eve.

01:05:47 It's got some great graphics to inspire you as well.

01:05:50 Bye, everyone.

01:05:50 Thank you.

01:05:51 Thank you.

01:05:52 Thank you.

01:05:53 This has been another episode of Talk Python To Me.

01:05:55 Thank you to our sponsors.

01:05:56 Be sure to check out what they're offering.

01:05:58 It really helps support the show.

01:06:00 This episode is sponsored by Sentry's Seer.

01:06:03 If you're tired of debugging in the dark, give Seer a try.

01:06:06 There are plenty of AI tools that help you write code, but Sentry's Seer is built to help you fix it when it breaks.

01:06:12 Visit talkpython.fm/century and use the code talkpython26, all one word, no spaces,

01:06:18 for $100 in Century credits.

01:06:20 And it's also brought to you by Talk Python Courses.

01:06:24 Course completion certificates are now live.

01:06:26 If you finished a course, there's a certificate waiting for you on your account page right now.

01:06:31 Download it as a PDF or add it to your LinkedIn profile with one click under licenses and

01:06:36 certifications.

01:06:37 Same section as your formal degrees.

01:06:40 Visit training.Talk Python.fm slash account to see what you've already earned.

01:06:44 If you or your team needs to learn Python, we have over 270 hours of beginner and advanced

01:06:50 courses on topics ranging from complete beginners to async code, Flask, Django, HTMX, and even

01:06:56 LLMs.

01:06:57 Best of all, there's no subscription in sight.

01:06:59 Browse the catalog at talkpython.fm.

01:07:02 And if you're not already subscribed to the show on your favorite podcast player, what

01:07:06 are you waiting for?

01:07:07 Just search for Python in your podcast player.

01:07:09 we should be right at the top.

01:07:10 If you enjoyed that geeky rap song, you can download the full track.

01:07:13 The link is actually in your podcast blur show notes.

01:07:16 This is your host, Michael Kennedy.

01:07:18 Thank you so much for listening.

01:07:19 I really appreciate it.

01:07:20 I'll see you next time.

01:07:31 I thought of me.

01:07:34 Can we break the roll?

01:07:37 Upgrade the code.

01:07:39 No fear of getting old.

01:07:42 We tapped into that modern vibe overcame each storm.

01:07:46 Talk Python and me, async is the norm.