Cost Control for Production AI Agents

An agent that costs three times what you forecast almost never got there by choosing a pricey model. It got there by sending a growing conversation back to the provider on every one of eleven steps, retrying four of them, and doing that across a queue nobody was watching. The bill is a symptom. Attribution is the diagnosis, and without it, cost work is guesswork.

Attribute before you optimise

A monthly invoice tells you what you spent. It cannot tell you which workflow, tenant or agent spent it, and that is the only fact that makes an optimisation decision possible.

The minimum useful record is per model call: a task or run identifier, the agent or profile that made the call, the model, input tokens, output tokens, cached tokens if the provider reports them, latency, and whether the call was a retry. Aggregate that by run and you get cost per completed task — the number worth putting on a dashboard, because it is the one that moves when you change something.

This is the problem ringlet was built around: when a team runs several coding-agent CLIs across different providers, each with its own credentials, the spend fragments across accounts and nobody can say which agent or which task consumed it. Consolidating token-level accounting into a single ledger is less glamorous than a caching layer and usually more valuable, because it is what tells you whether the caching layer was worth building.

Two derived numbers are worth tracking alongside it. Cost per successful outcome, which divides total spend by completed tasks rather than by attempted ones — a cheap agent that fails half the time is not cheap. And retry share, the fraction of spend that went into calls that were later superseded. A high retry share is almost always the largest single recoverable line item.

Where the money goes in a multi-step run

Driver	Typical shape	What fixes it
Context re-sent per step	Input tokens grow with step count	Summarise, prune, or checkpoint state outside the prompt
Retries and self-correction	Duplicated calls with near-identical input	Cap attempts; make failures terminal
Tool output pasted verbatim	One large payload inflates every later step	Truncate and reference instead of inlining
Over-wide retrieval	Ten chunks retrieved, two used	Tighten the retriever before enlarging the window
Idle agents in a queue	Cost with no corresponding outcome	Deadlines and cancellation

The pattern across all five rows is the same: the expensive thing is rarely the individual call, it is the multiplication. A tool result that is two thousand tokens costs two thousand tokens once and then costs two thousand tokens again on every subsequent step that carries it along. Fixing the multiplication is worth more than shaving the per-call price.

Context discipline beats model shopping

The cheapest token is the one you do not send. Three habits do most of the work.

Do not carry raw tool output forward. Store the full result outside the conversation and put a short, faithful summary in the context with a handle the agent can use to fetch the rest. This is the single largest saving in most multi-step agents, and it also improves reliability, because a shorter context is one the model is less likely to lose the thread of.

Keep durable state out of the prompt. A graph or workflow framework that checkpoints state to storage does not need to restate it to the model. If your agent’s memory of what it did lives only in the message list, every step pays for the whole history. An external memory layer such as memorg exists to make recall a bounded retrieval rather than an unbounded transcript.

Prune on a rule, not on a feeling. Drop tool calls older than N steps, or collapse them into a running summary at a fixed cadence. A rule you can read is a rule you can measure; an ad-hoc trim is neither.

Caching, in the two senses that matter

Provider-side prompt caching reduces the price of input tokens that have already been seen, and it is close to free to adopt: keep the stable part of the prompt — system instructions, tool definitions, long reference material — at the front, and keep it byte-identical between calls. A prompt that interpolates a timestamp into the preamble defeats this without anyone noticing.

Application-side caching is different and complementary. It answers a request without calling the provider at all, which is the only kind of saving that is 100%. It applies where the same question recurs with the same inputs: a classification over a document that has not changed, an extraction from a fixed schema, an embedding of a string you have embedded before. Key on the inputs, version the key on the prompt template, and invalidate when either moves.

A gateway is the natural place to put both, which is part of why fast-litellm targets the connection and rate-limiting layer of a gateway rather than the model call itself. Caching reduces how many calls you make; the gateway work reduces what each remaining call costs you in local resources. They compose, and the honest framing is that neither one is a substitute for sending less context.

Budgets and rate limits are availability controls

A per-tenant or per-run token budget, enforced before the call, turns an unbounded failure into a bounded one. It is not a cost optimisation in the ordinary sense — it does not make the good path cheaper — but it caps the bad path, and the bad path is where the surprise invoices come from.

Two implementation notes matter more than the policy itself. Enforce the budget where the call is made, not in a nightly report, or it will only ever tell you about money already spent. And make the rate limiter cheap: a naive per-key limiter that allocates a Python object for every key in a high-cardinality space becomes its own resource problem at scale, which is exactly the memory behaviour the fast-litellm work targets.

Batching belongs in the same category. Where the provider offers an asynchronous batch endpoint at a reduced rate and the work is not latency-sensitive — overnight classification, backfills, evaluation runs — the discount is large and the change is small. Do not batch interactive traffic to chase it.

What to build, in order

1. Attribution. Per-call accounting with a run identifier. Everything else is unmeasurable without it.
2. Retry caps and deadlines. The largest single recoverable line item in most systems, and a day of work.
3. Context discipline. Stop carrying tool output and history you do not need.
4. Prompt-cache alignment. Reorder prompts so the stable prefix is stable. Nearly free.
5. Application cache. Only where inputs genuinely repeat.
6. Budgets. Before the call, per tenant and per run.
7. Model selection. Last, because by this point you know which calls dominate and can route only those.

Reversing this list is the common failure. Teams start at seven, switch a model, watch quality drop, switch back, and conclude that cost cannot be reduced.

Limitations

Cost per task is not a stable number across workloads. A figure measured on short prompts and a five-step graph says nothing about a twenty-step research agent with large retrieved documents, and neither says anything about yours until you measure it.

Provider pricing, cache semantics and batch discounts change on the provider’s schedule. Anything you build should read the accounting from the response metadata rather than hard-coding a rate, or your dashboard will be confidently wrong the first time a price moves.

And aggressive pruning trades cost against quality. Every token you remove is context the model no longer has. Measure task success alongside spend, or you will optimise a number while degrading the product — the same discipline described in the agent evaluation research design.

Frequently asked questions

Is a smaller model the fastest win? Rarely, and it is the hardest to reverse. Attribution takes a day and tells you whether the smaller model would even help.

Should I cache the whole agent response? Only for deterministic, input-keyed work. Caching a conversational turn usually returns something subtly wrong for a slightly different question.

How do I account for tool cost? The same way: per call, with the run identifier. Paid tools and paid data have their own bill, and payment-aware tool protocols like mcp-pay exist so that cost is declared rather than discovered.

The bottom line

Measure per task, cap the retries, stop re-sending what the model does not need, and only then reach for caching and model routing. The order is the whole advice. Most of the Neul Labs catalogue touches one rung of that ladder or another, but none of it will help a system that cannot yet say which task spent the money.