Your AI bill at ten times the volume

Two questions, one of them hard: what does a single request to your AI system cost today, and what happens to that number at ten times the volume. Most teams cannot answer the first, which makes the second a surprise scheduled for their best quarter. If you want to reduce LLM costs in production, start by accepting that unit cost is a design property. It is set by architecture decisions made before scale, and retrofitting it after growth means re-engineering a live system while the invoice climbs.

We have seen products where the margin quietly died at exactly the moment the product succeeded: every user message triggered a frontier model with the full conversation history attached, which was invisible at pilot volume and a board topic within two quarters of launch.

How AI bills explode

Nothing about a pilot predicts the production bill. At two hundred requests a day, everything is cheap and nobody looks. What multiplies the number later is structural. Context grows: conversation history, retrieved documents and tool outputs get resent with every call, and you pay for every token, every time. Retries and fallbacks double invisible traffic. Agent workflows fan out, one user action becoming dozens of model calls. And the habit formed in the demo phase, sending everything to the largest model available, hardens into architecture.

None of this shows up as a bug. The system works. It just works at a unit cost that no longer closes.

Four levers that reduce LLM costs in production

Routing comes first because it pays the most. Most production traffic is easy: classify the request, send the easy majority to a small model, and reserve the frontier model for the hard slice. Small models are often ten times cheaper per token or more, and on well-bounded tasks their quality is indistinguishable. Escalate on failure or low confidence rather than defaulting to expensive.

Caching comes second. The major providers bill cached prompt prefixes at a steep discount, but only if your prompts are built for it: the stable parts (system prompt, shared reference material) must actually be stable and sit ahead of the variable parts. This is a restructuring exercise, not a checkbox.

Context discipline is the least glamorous and the most durable. The model does not need everything you have been sending it. Retrieve the relevant passages instead of attaching the corpus, summarize old conversation turns, cap tool outputs. Teams that audit their prompts usually find a large share of the tokens were contributing nothing but cost.

Batching closes the list. Anything that does not need an interactive answer, nightly enrichment, backfills, report generation, belongs on the batch endpoints that major providers price at roughly half the interactive rate. Deciding which work is actually urgent is a product question with a direct line to the bill.

Make cost a metric, not a discovery

The levers only work if you can see. Cost per request, and better, cost per completed task, belongs on the same dashboard as quality, tracked per feature and per model, the visibility argument we make in you cannot run an AI system you cannot see. Add alerts on anomalies: an agent stuck in a loop shows up on the invoice before anyone files the bug.

Pair the dashboard with evals, so that swapping in a cheaper model is a measured decision instead of a hope. Model choice is a per-task decision that changes as prices move, the argument of choosing a model, and without an eval set you cannot take the cheap option safely.

One caution against optimism: falling model prices will not save you. Per-token prices have fallen for years, and bills have risen anyway, because products grow context, traffic and agent complexity faster than providers cut rates. A pricing tailwind is not a cost strategy.

There is also an outside reader for this number now. Investors have learned to open AI conversations with unit economics, and a team that knows its cost per request cold reads very differently in that room, a dynamic we describe from the diligence side in the margin question every AI deal should start with.

Cost engineering is part of the architecture review in every system our custom AI team ships, designed in before launch rather than bolted on after. If your bill is already growing faster than your usage, that review is the place to start, and it is usually a week of work, not a rebuild.