Parallel Python Apps with Sub Interpreters

Episode Deep Dive

The Guests

Eric Snow (Microsoft, Python core developer) has worked on Python’s core runtime for over a decade. He is the driving force behind sub-interpreters in Python and co-author of PEP 684 (Per-Interpreter GIL) and PEP 734 (Multiple Interpreters in the Stdlib). His day-to-day includes building performance enhancements such as Faster CPython.

Anthony Shaw (Microsoft, Python advocate, open-source contributor) works on Python tooling, advocacy, and research into parallelism in Python. He's author of CPython Internals and has been experimenting with sub-interpreters in real-world scenarios, from running web apps in parallel to adapting existing libraries for multi-core Python.

What to Know if You're New to Python

• Python is a high-level language that emphasizes readability and simplicity.
• Before diving into parallelization topics like sub-interpreters, you should feel comfortable with fundamental Python structures and modules.
• Understanding how Python handles concurrency (threads vs. async vs. multiprocessing) is beneficial.
• If you want a concise introduction to concurrency in Python, check out Async Techniques and Examples in Python from Talk Python Training.

Key Topics and Takeaways

1. Why Sub-Interpreters Matter
   Eric Snow explained that sub-interpreters allow multiple Python interpreters to coexist within a single process. Each interpreter has its own GIL (Global Interpreter Lock), effectively enabling parallel CPU-bound execution on multiple cores. This new approach addresses some limitations of threads (one shared GIL) and the heaviness of multiprocessing (spawned OS processes).
2. PEP 684 and PEP 734
   • PEP 684 – Per-Interpreter GIL: This proposal lays the groundwork for separate GILs per interpreter. It required significant work to isolate global state in CPython.
   • PEP 734 – Multiple Interpreters in the Stdlib: This upcoming PEP will introduce a high-level Python API for creating and managing sub-interpreters, making it easier to run Python code in parallel without starting separate processes.
     Links:
   • PEP 684
   • PEP 734
3. Parallel Web Applications
   Anthony tested sub-interpreters in a web context by adapting Hypercorn to spawn sub-interpreter workers. While frameworks often rely on multiple processes (e.g., Gunicorn), sub-interpreters potentially reduce overhead and speed up startup. This indicates a future direction for high-throughput Python web apps without resorting to heavyweight processes.
4. Data Sharing and Serialization
   Because each sub-interpreter is isolated, sharing data between them requires explicit communication via queues or pickle-like serialization. Anthony’s exploration revealed that techniques like dill or msgspec can improve serialization speed. They also discussed passing memoryview objects directly without extra copying—useful for NumPy arrays.
5. Integration with Testing
   Anthony’s work on pytest subinterpreters (Pytest Subinterpreters Example) demonstrates how C extensions or packages can be tested in isolated environments. This helps catch hidden bugs in extension modules that assume there’s only one GIL or a single global state.
6. C Extensions and Global State
   A big part of sub-interpreter readiness is ensuring that extensions don’t rely on hidden global variables (e.g., readline, datetime). Anthony and Eric encountered modules that either share state incorrectly or crash when used in sub-interpreters. Future Python releases will include more robust checks and updated modules to handle multi-interpreter usage gracefully.
7. Memory Model and Performance
   Each sub-interpreter allocates its own memory arenas, avoiding direct conflicts and reducing the need for a single global lock. This can improve parallel CPU-bound tasks. While the approach is conceptually similar to multiprocessing, it’s more lightweight and integrates more seamlessly with the existing Python environment.
8. Sub-Interpreter Pool Executors
   PEP 734 envisions a “sub-interpreter pool executor” similar to threading.ThreadPoolExecutor or multiprocessing.Pool. This would let you reuse interpreters across many tasks, minimizing repeated startup overhead. The result is near-process-level parallelism without the cost of new processes each time.

Quotes & Stories

• Anthony on performance trade-offs: “If you create a fresh Python process for a small task, you waste a lot of startup time. Sub-interpreters drastically cut that overhead.”
• Eric on concurrency models: “Once you move beyond one GIL, you need to be deliberate about your data. Sub-interpreters force you to do concurrency the right way.”

Overall Takeaway

Sub-interpreters mark a significant step in Python’s evolution toward multi-core CPU utilization. By giving each interpreter its own GIL, developers can take advantage of parallelism without spinning up full processes. Though still evolving, experiments in web frameworks, testing, and extension modules show promising speedups and better resource usage. With PEP 734 slated for Python 3.13, now is the perfect time to explore this new concurrency model—and be prepared for a future where Python apps run faster and more safely on modern hardware.