Building Rust Extensions for Python

Episode Deep Dive

1. Motivation for Using Rust along with Python

• Python’s Performance Boundaries Python is fantastic as a high-level, productive language. But certain parts of an application—especially performance-critical code—can become slow in pure Python.
• Rust’s Advantages over C for Extensions While CPython historically supports C extensions, Rust provides:
  • A modern compiler toolchain and package manager (Cargo).
  • Memory safety at compile time via its “borrow checker,” drastically reducing entire classes of bugs common in C.
  • A strong ecosystem with many high-quality libraries for tasks like JSON parsing, web servers, etc.

2. The Tooling: PyO3, Maturin, and Cargo

• PyO3 A Rust library that simplifies creating Python extension modules. It wraps and exposes Rust functions, classes, and logic to Python. Example benefits:
  • Reduces manual handling of low-level CPython APIs.
  • Automatically manages ref-counts for Python objects in Rust.
• Cargo Rust’s package manager and build system. You specify Rust crates (similar to PyPI packages) in your Cargo.toml, and Cargo handles compilation, dependency resolution, and more.
• Maturin A Python package (installable via pip install maturin) that integrates Cargo with Python packaging conventions. Maturin lets you:
  • Build a wheel (.whl) for your Rust-based extension modules.
  • Automate creation of many-platform wheels.
  • Use commands like maturin develop to compile your Rust code and install it into your active Python environment for testing.

3. Real-World Success Stories

1. Pydantic Core
   • Samuel Colvin rewrote Pydantic’s core internals from Python to Rust, greatly boosting performance and reliability.
   • The Rust-based internals rely heavily on PyO3 + Maturin for packaging.
   • “jiter” library (for fast JSON parsing) also came out of these efforts, allowing Pydantic to parse JSON more quickly than Serde in many use cases.
   • Jiter is used by other major Python projects (e.g. the official OpenAI Python SDK) to parse JSON at scale.
2. Import Linter / Grimp
   • David Seddon built “import linter” for architectural rules in a massive Django monolith (>8 million lines of code, ~400 PR authors per week).
   • Import linter uses a sub-library named Grimp for dependency graph building.
   • Originally written purely in Python, but crucial parts of its dependency-graph algorithms in Grimp were ported to Rust with PyO3, giving ~6x performance boosts on certain tasks.
   • Also replaced a pure-Python Fluent translator for app localization with a Rust-based Fluent library, cutting startup time from minutes to ~30s.
3. Other Notable Rust Wrappers
   • Granian (wrapping Rust HTTP logic, e.g. Hyper).
   • rloop for async event loops.
   • rtoml for TOML parsing, etc.

4. Practical Steps & Workflow

1. Install Rust
   • Use RustUp (the Rust toolchain installer/manager).
   • You’ll now have cargo and rustc available.
2. Create or Add Rust Crate
   • cargo new or maturin new to scaffold a project structure containing a Cargo.toml.
   • Add [dependencies] like pyo3 = "..." or other crates in Cargo.toml.
3. Python Integration
   • Install Maturin via pip install maturin.
   • maturin develop builds and installs the Rust extension into your local Python environment for interactive testing.
   • (Optional) Provide .pyi stubs so type checkers (mypy, pyright, etc.) know signatures of your Rust-implemented Python functions.
4. Publishing
   • Maturin can also build multi-platform wheels and upload them to PyPI (e.g. maturin build && maturin upload).
   • Advanced workflows generate CI config to build 50-60 wheels across CPU architectures, OSes, and Python versions, then push them to PyPI.

5. Advice & Final Thoughts

• Transitioning to Rust
  • Don’t be discouraged if you feel “lost” at first. Rust’s strictness (the borrow checker) can feel daunting, but it largely prevents memory errors at compile time.
  • Writing smaller Rust “subsystems” can already yield big performance gains—no need to rewrite an entire codebase.
• Smoother Developer Workflow
  • Combine Rust’s speed & concurrency with Python’s clarity & ease of iteration.
  • Keep the majority of your app in Python; only “Rustify” performance bottlenecks.
  • Rust can wrap advanced libraries from crates.io that significantly outperform pure Python (e.g., specialized parsing, HPC tasks).
• Rust Ecosystem
  • Has a reputation for thoroughness and well-tested libraries, though like any open-source domain, some crates are abandoned. Evaluate each library’s activity level before adoption.
• Impact on Architecture
  • Tools like import linter (Grimp) and Rust-based functionality keep giant monoliths manageable (enforcing architecture boundaries, speeding up critical code paths).

Overall Takeaway With a handful of readily available tools (PyO3, Maturin, Cargo), Python developers can confidently tap into Rust’s performance and memory safety. Both David’s and Samuel’s experiences show that even partial migrations (e.g., rewriting only the critical hotspots in Rust) lead to notable wins in speed, maintainability, and system stability.