Getting Lazy with Python Imports and PEP 690

Episode Deep Dive

Guests Background

Barry Warsaw is a longtime Python core developer with experience going back to 1994. He has worked on influential Python projects, championed various PEPs, and served as a sponsor on PEP 690. Barry currently works at LinkedIn and brings a deep history of the Python ecosystem to the table.

Germán Méndez Bravo is a Python developer at Meta who initially implemented the lazy imports approach inside Instagram’s massive Python codebase. In this conversation, Germán shares insights on large-scale code performance, startup speed, and practical ways to introduce lazy imports without breaking existing applications.

Carl Meyer is a seasoned Python developer at Meta (formerly Facebook). He has contributed to pip (notably the pip uninstall feature), virtualenv, Django’s core team, and the Cinder project. Carl focuses on Python at scale, performance optimization, and fostering new features, including PEP 690’s approach to lazy imports.

What to Know If You're New to Python

When you hear about “lazy imports” and modules, remember that Python executes code at import time, unlike languages that simply compile header files. If you’re not used to Python’s dynamic behavior, you might not expect side effects just by placing an import at the top of a file. Understanding this is key to following the conversation about improving Python’s startup and memory usage through lazy imports. Here are a few links to get you oriented:

• Python for Absolute Beginners: A solid introduction to Python’s syntax, basic concepts, and how modules work.

Key Points and Takeaways

1. PEP 690 and Lazy Imports as the Centerpiece
   The heart of this episode is PEP 690, a proposal to introduce lazy imports into CPython. Instead of executing and loading a module (and its transitive dependencies) as soon as import is encountered, the process is deferred until the code first touches that module’s name. This can yield significant speedups in startup time, especially for large CLI tools or data-heavy applications. The panelists shared real-world examples at Meta (Instagram) showing up to 70% faster startup and 40% less memory usage.

   • Links and Tools:
     • PEP 690 discussion: https://discuss.python.org/t/pep-690-lazy-imports
     • Cinder (GitHub): https://github.com/facebookincubator/cinder
     • Instagram’s engineering blog (background context): https://instagram-engineering.com

2. How Imports Work in Python
   The episode highlights that importing a module in Python is more than “just linking” or “just including files.” Under the hood, Python executes the module top to bottom at import time, which can be costly if you only need a fraction of that module’s functionality. Barry, Germán, and Carl emphasize that code design can inadvertently trigger expensive actions if imports are scattered or have side effects.

   • Tools and Resources:
     • Official Python docs on import: https://docs.python.org/3/reference/import.html

3. Performance Renaissance in Python
   Python 3.11 is 20-40% faster than even 3.10, and PEP 690 is part of a broader movement to optimize Python. The guests tie lazy imports to other initiatives like the Faster CPython project, showing that Python’s ecosystem is actively reducing performance bottlenecks. This holistic approach aims to tackle everything from the core interpreter to libraries that rely on Python’s dynamic nature.

   • Tools and Resources:
     • Faster CPython project: https://github.com/faster-cpython

4. Avoiding Import Side-Effect Pitfalls
   One challenge of lazy imports is that many Python libraries rely on “import side effects,” such as registering a class in another module or creating a global database connection. The panel explains that PEP 690 addresses this by allowing either eager imports in specific spots or application-level toggles so you don’t break libraries that expect immediate initialization. Nonetheless, the best practice is to avoid heavy-lifting code at import time whenever possible.

   • Links and Tools:
     • Code patterns reference: https://docs.python.org/3/tutorial/modules.html

5. Command-Line Tools and Subcommands
   A primary use case for lazy imports is speeding up command-line interfaces. Tools with many subcommands often end up importing a large hierarchy of packages, even for a single command usage. With lazy imports, you pay only for what you use—improving user experience significantly. The approach is transparent for maintainers who already structure their CLI using subcommands.

   • Example: argparse Sub-commands

6. Implementation in Meta’s Cinder
   Germán built the initial lazy imports prototype for Instagram’s server code and integrated it into Cinder, Meta’s performance-oriented fork of CPython. Carl describes how Cinder’s approach uses specialized dictionary lookups in module globals, substituting a placeholder object until the symbol is first accessed. This stealth substitution is what makes lazy imports so transparent and powerful.

   • Links and Tools:
     • Meta’s Cinder: https://github.com/facebookincubator/cinder

7. Handling Errors and Debuggability
   While lazy imports might defer errors, PEP 690 plans to provide clear tracebacks. If a module fails to import, Python will throw the error when you first access the deferred import symbol, but with a traceback referencing the original import statement. This approach preserves Python’s hallmark clarity, ensuring you know where the issue originated.

   • Tools and Resources:
     • Python’s Error Handling Docs: https://docs.python.org/3/tutorial/errors.html

8. Import Cycles Simplification
   The guests mention that lazy imports can naturally eliminate certain import cycles. If two modules import each other but only use certain names inside function bodies, that code can still run without cyclic import failures. This is a relief for large codebases (like at Instagram) that have grown organically and occasionally form complex webs of imports.

   • Example reference: Cyclic Imports in Python docs

9. Opting In at the Application Level
   The PEP proposes that lazy imports remain off by default, at least initially, given the potential to break older code with side effects. Instead, you can enable lazy imports through a command-line flag or via a special code-level entry point. This ensures that library authors do not force lazy behavior on unsuspecting users; the application owner decides based on testing and validation.

   • Tools:
     • Proposed -L flag or programmatic setting (under discussion on discuss.python.org)

10. Cinder, Pyre, and the Road Ahead
    Beyond PEP 690, Carl and Germán’s teams at Meta have produced other performance and tooling enhancements (e.g., Cinder’s JIT). Coupled with type checkers like Pyre or mypy, these developments indicate a broader movement toward more optimized, production-grade Python. The guests underscore that improvements in one corner of Python’s runtime often have knock-on effects that benefit the entire community.

• Links:
  • Pyre type checker
  • mypy

Interesting Quotes and Stories

• Barry Warsaw on C Interop: “It’s both Python’s advantage and hindrance: the ability to write C extensions is what gives Python its amazing ecosystem, but it also makes rewriting the interpreter more complex.”
• Carl Meyer on Lazy Imports: “The idea is that you only pay for what you actually use, which is huge in a codebase with thousands of modules.”
• Germán Méndez Bravo on Real-World Gains: “With lazy imports, we saw CLI tools at Meta go from seconds to sub-second startup and memory usage shrink by a noticeable margin.”

Key Definitions and Terms

• Lazy Imports: A feature where modules are not actually loaded and executed upon the import statement. Instead, a placeholder is created, and the module is only fully imported when first used.
• PEP (Python Enhancement Proposal): A design document that provides information or proposes changes to Python. PEP 690 is one such proposal focusing on lazy imports.
• Side Effects: Unintended or additional operations performed by a module at import time, such as initializing database connections or registering classes.
• Cinder: Meta’s performance-focused fork of CPython, introducing features like a JIT compiler and lazy imports.
• Import Cycles: Situations where two or more modules import each other, which can cause runtime errors unless carefully managed or deferred.

Learning Resources

Below are some resources to help you deepen your knowledge about Python concepts.

• Python for Absolute Beginners: A thorough introduction to Python that includes modules, packages, and how the import system works.
• Python Memory Management and Tips: Once you understand basics, learn how Python’s memory model can be optimized in advanced scenarios.

Overall Takeaway

PEP 690 represents a forward-thinking push to optimize Python’s runtime performance by deferring module imports until absolutely necessary. Barry, Germán, and Carl illustrate that lazy imports can give significant real-world benefits—especially in large, complex applications—while still respecting Python’s principle of clarity and simplicity. As more features like these enter the mainstream interpreter, Python continues to evolve into an increasingly performant, robust, and versatile language for everything from small CLI tools to massive web infrastructures like Instagram.