The GIL Ceiling in Python AI Frameworks

If your agent process pins one core at 100% while the other fifteen idle, you have hit the GIL ceiling. If it sits at 4% CPU and still takes nine seconds per step, you have hit a network wall and the GIL has nothing to do with it. The two failure modes look identical from the outside — throughput stops rising when you add concurrency — and they have completely different fixes. Getting the diagnosis wrong is the most common reason a performance project on a Python agent stack produces nothing.

What the GIL actually serialises

CPython’s global interpreter lock is a single mutex that a thread must hold to execute bytecode or to touch a PyObject refcount. It is released around blocking system calls, so a thread waiting on a socket is not holding it. It is also released by C extensions that explicitly drop it — NumPy does this for large array operations, and any well-written native extension does it for the duration of work that touches no Python objects.

What that leaves under the lock is every line of pure Python. Dictionary lookups. String formatting. json.dumps with a custom encoder. Pydantic validation. Iterating a list of message dicts to count tokens. None of it parallelises across threads, no matter how many cores the box has.

The practical consequence is an inversion of the usual intuition. Adding threads to a Python service usually helps when the service is waiting. It never helps when the service is computing, and past a certain point it hurts, because the interpreter pays a context-switch cost every sys.setswitchinterval to hand the lock around.

The three shapes of work inside an agent step

A single step of an agent — one LLM call plus whatever the framework does around it — decomposes into three kinds of work, and only one of them is the network.

Work	Where the time goes	Holds the GIL?	Scales with threads?
Provider call	Socket wait on the model endpoint	No	Yes
Framework overhead	Serialisation, validation, routing, token counting	Yes	No
Tool execution	Depends entirely on the tool	Sometimes	Sometimes

For a short prompt against a fast model, the provider call can be a few hundred milliseconds. Framework overhead on the same step is often tens of milliseconds, which reads as noise. The moment you scale up — fifty concurrent agents, a fifty-message state object, a ten-step graph — the two curves cross. The provider call stays flat because the endpoint is genuinely parallel on the other side. The framework overhead multiplies, and it multiplies against a lock.

This is why the measured wins on the accelerators we publish cluster around the second row of that table rather than the first. The fast-langgraph figures on the checkpoint benchmark write-up — 737× on checkpoint writes, 151× on reads — describe serialisation, not inference. The end-to-end figure on the same workload is 2.8×, which is the honest number: everything except the model call got much faster, and the model call did not move at all.

Telling the two apart in under five minutes

You do not need a profiler build or a staging environment. Attach py-spy to the running process and look at where the wall clock goes:

py-spy top --pid $(pgrep -f 'my-agent-worker') --subprocesses

Read the output as follows.

• High %Own on framework internals — a serializer, a validator, a dict comprehension inside a rate limiter — means the interpreter is executing your framework’s Python, under the lock. This is a GIL ceiling.
• High %Total but near-zero %Own on a socket read means the thread is parked in a syscall with the lock released. This is an I/O wall. More concurrency will help; a faster serializer will not.
• A flat profile with no dominant frame usually means the cost is spread across object allocation and refcounting. That is still GIL-bound work, and it is the hardest kind to fix incrementally.

Cross-check with the coarsest possible signal: run top and look at the process CPU percentage. A CPython process that cannot exceed roughly 100% of one core while requests queue is the clearest GIL signature there is.

What actually moves the ceiling

There are four honest options, in rough order of how much disruption they cause.

Move the hot path out of Python. A native extension that releases the GIL for the duration of its work turns one serialised region into a parallel one. This is the approach behind the drop-in accelerators described in why Rust accelerates Python AI infrastructure: the Python call signature is unchanged, the work underneath happens in Rust, and other threads keep running while it does. The constraint is that the extension must not touch Python objects while the lock is dropped, which is why the boundary is usually drawn at “give me bytes, return bytes.”

Move the work to another process. Multiprocessing sidesteps the lock entirely, at the cost of serialising everything that crosses the process boundary — which, for an agent carrying a large state object, can cost more than it saves. A task queue is the structured version of this; fastworker exists partly because the per-task overhead of the usual broker-plus-worker arrangement is itself Python-bound.

Reduce the work. The cheapest fix is usually to stop doing something. Do not checkpoint every node if only the graph boundary matters. Do not re-validate a state object that has not changed. Do not count tokens twice on the same prompt.

Use free-threaded CPython. Python 3.13 shipped an experimental build with the GIL removed, and 3.14 continued that work. It is a real answer to this problem in the medium term. It is not a drop-in today: any C extension in your dependency tree must be rebuilt for the free-threaded ABI, and single-threaded performance on that build has historically been lower than on the default build. Treat it as something to test, not something to deploy under a deadline.

Limitations of this framing

Three caveats, because the GIL is over-blamed as often as it is under-diagnosed.

The lock is not the cause of most slow agents. Prompt size, retry storms, and a model that is simply slow account for far more production latency than interpreter contention does. Profile before you conclude anything.

Removing the lock from one hot path moves the ceiling; it does not remove it. Once serialisation is native, the next bottleneck is usually allocation in the surrounding Python code, and the second speedup is always smaller than the first.

And the numbers quoted above are workload-specific. They were measured on particular state sizes and concurrency levels described alongside each project, and they are not a general promise about your graph.

Frequently asked questions

Does asyncio avoid the GIL? No. An event loop runs on one thread and executes Python bytecode under the same lock. What it avoids is thread-switching overhead, and it makes waiting cheap. For a workload that is genuinely waiting, that is the right tool. For a workload that is computing, it changes nothing.

Will more workers fix it? More processes will, up to the memory you can afford. More threads inside one process will not.

Is this specific to agent frameworks? No, but agent frameworks are unusually exposed to it, because their per-step overhead is dominated by serialising and validating large nested structures — exactly the work CPython is slowest at.

The bottom line

Diagnose first with py-spy, and let the CPU percentage settle the argument. If the process cannot saturate more than one core, the ceiling is the interpreter and the fix is to move the hot path off it. If the process is idle and slow, the ceiling is the network and nothing in this post will help you. The whole Neul Labs catalogue of accelerators exists for the first case only, and we would rather say so than sell a Rust extension to someone whose real problem is a retry loop.