PyO3 in Production: What We Learned

Every Rust extension for Python that reaches real users spends most of its engineering budget outside the Rust. The kernel — the part that makes the thing fast — is usually a few hundred lines and is done in a week. The rest is distribution, compatibility, failure behaviour and the long tail of “it works on my machine except on Windows with Python 3.9.” These are the decisions we keep re-making, written down.

Build against the stable ABI unless you cannot

CPython exposes two extension ABIs. The version-specific one gives you the full C API and requires a separate wheel per interpreter minor version. The stable ABI — abi3 — restricts you to a smaller surface but produces one wheel per platform that keeps working on future interpreters.

For an acceleration shim, abi3 is almost always right. The matrix collapses from “platforms × Python versions” to “platforms,” the release job gets shorter, and a user on a Python version that did not exist when you last cut a release still gets a binary instead of a source build. You give up a handful of APIs, mostly around type object internals, which a shim rarely needs.

The exception is a project that genuinely needs version-specific internals. If you find yourself reaching for those, be deliberate about it and accept that you now owe users a release every time CPython ships a minor.

Publish enough wheels that nobody compiles anything

The single most common reason a Python user abandons a Rust-backed package is a source build. They run pip install, watch cargo start downloading crates, and cancel. The fix is unglamorous: build wheels for the platforms your users are actually on, and build them in CI so the matrix cannot rot.

For the accelerators we publish, that means Linux on x86_64 and aarch64, macOS on both Intel and Apple Silicon, and Windows. maturin handles the packaging; manylinux containers handle the glibc floor. The target to aim for is that a user never learns your extension was written in Rust until they read the README out of curiosity.

A related rule: never make the Rust toolchain a documented prerequisite. The moment rustup appears in your install instructions, you have changed the user’s workflow, which is precisely what a drop-in replacement exists not to do — see drop-in compatibility as a design principle for why we treat that as a hard constraint.

Release the GIL, but draw the boundary carefully

Python::allow_threads is what makes a Rust extension worth writing in a multi-threaded service: while your Rust code runs, other Python threads can execute. The rule for using it safely is simple and unforgiving — inside the closure you may not touch any Python object, because you no longer hold the lock that makes refcounting sound.

In practice this dictates the shape of the API. The function signature takes Python objects, converts them to owned Rust types, drops into allow_threads for the expensive part, and converts back on the way out. If the conversion itself is the expensive part, you have not gained much, which is why the accelerators tend to draw the boundary at bytes: hand the extension a buffer, get a buffer back, do the real work in between with the lock released.

The corollary is that fine-grained calls are a trap. A Rust function that is invoked a million times for a microsecond of work each will be dominated by conversion and call overhead. Batch at the Python boundary, or the speedup disappears into the FFI.

Map errors into tracebacks people can act on

A Rust panic crossing the FFI boundary is undefined behaviour territory in the general case, and an incomprehensible error in the best case. Convert errors explicitly. Every fallible path returns a PyResult, and every error type maps to a Python exception class the caller would have expected from the pure-Python implementation.

This matters more for drop-in replacements than for new libraries, because user code already has except blocks written against the upstream library’s exception types. If your accelerated path raises something the caller does not catch, you have broken a program that was working — and the user will blame the last thing they installed, correctly.

The same logic applies to messages. ValueError: invalid checkpoint payload at key 'messages[3].content' is a bug report someone can act on. PanicException: called Option::unwrap() on a None value is not.

Patch defensively, and always leave a way back

Several of the accelerators work by monkeypatching an upstream library at import time. It is the only way to deliver a speedup with a one-line integration, and it is also the riskiest thing in the design. Three rules keep it survivable.

Detect the upstream version before patching. The functions you are replacing belong to someone else and can change shape in a minor release. Check what you found before you swap it.

Fall back silently to the original. If the version is unrecognised or the attribute is missing, do nothing and let the pure-Python implementation run. A slower program is a vastly better outcome than a broken one.

Make the patch observable. A user debugging an unexpected result needs to know whether the accelerated path was active. An environment variable that disables patching entirely, and a log line or attribute that reports what was patched, turn a mystery into a two-minute check.

Decision	Cheap option	What it costs later
Version-specific ABI	Fewer constraints while writing	A release for every CPython minor
Source distribution only	No CI matrix	Users who never get past pip install
Patch without version detection	Fewer branches	Silent breakage on an upstream minor release
Panic instead of PyResult	Less plumbing	Exception types user code cannot catch

Test against the upstream suite, not only your own

The most valuable test signal for a compatibility shim is the upstream project’s own test suite, run with your extension active. Your tests check that your code does what you meant. Theirs check that you did not change the behaviour anyone else depends on.

That is also where the honest compatibility number comes from. The fast-langgraph catalogue entry records passing 85 of 88 upstream tests rather than claiming full parity, because three of them exercise behaviour the Rust path deliberately does not reproduce. Publishing the ratio is more useful to a prospective user than a green tick, and it is the only version of the claim we can defend.

The maintenance bill

Taking on a compatibility layer means taking on someone else’s release schedule. When upstream ships a change to an internal you depend on, you have a deadline you did not set, and users who are already broken. Budget for it explicitly: a shim is not a one-off project, it is a subscription.

The mitigation is to depend on the narrowest possible surface. Every upstream internal you touch is a future incident. The extensions that have aged best in our catalogue are the ones that replace one well-defined operation — serialise this, hash these, pool those connections — and leave everything else alone.

Frequently asked questions

Is PyO3 worth it for a pure-CPU function? Only if it is called with enough work per call to dominate the conversion cost, and only if you release the GIL. A microsecond of Rust behind a microsecond of FFI overhead is not an optimisation.

Should the Rust crate be usable on its own? Yes, where it is free to do so. It keeps the Python bindings thin and makes the core testable without an interpreter in the loop.

How do you decide what to accelerate? Profile first. The diagnosis method — and the reason most agent stacks are bound by framework overhead rather than by the network — is covered in the GIL ceiling in Python AI frameworks.

The bottom line

Write the Rust last. Decide the ABI, the wheel matrix, the error mapping and the fallback behaviour first, because those are the decisions that determine whether anyone ever runs your code. The rest of the catalogue follows the same order, and the projects that skipped it are the ones that cost the most to maintain.