Testing AI agents before production: what the demo cannot tell you

Testing an AI agent before production means testing its actions as well as its answers. A chatbot that produces a weak reply has produced one weak reply. An agent that misreads a ticket and issues a refund, cancels the wrong order or emails a customer has done something a person now has to undo. The demo you watched proved that the agent completes a task on inputs picked for the occasion. It says nothing about the four hundredth input, the malformed one, or the moment a tool times out and the agent decides to try again, and again.

On the systems we ship, the unit under test for an agent is the sequence of tool calls it makes, with the final answer as a secondary check. That shift changes what a test suite looks like, what "passing" means, and who signs off. For assistants that only answer, an eval set of 50 examples in a spreadsheet does most of the work. For agents that act, we build the suite below before go-live.

Scenarios where the expected output is a tool trace

Each scenario is a realistic starting state (a ticket, an order, an inbox, a database row), a goal, and the trace we expect: which tools get called, with which arguments, in what order, and which must never be called. "Refund 40 euros on order 1187 after checking the return was received" expands to a lookup, a status check, a refund call with amount 40, and nothing else. A run that reaches the right outcome through a forbidden path, say a refund issued before the return check, fails. Outcome-only testing would have passed it.

We write the first twenty scenarios with the business owner in the room, from real cases of the last quarter. The suite grows every time production surprises us: the new case gets a scenario.

A sandboxed copy of everything the agent touches

An agent under test needs real tools pointing at fake systems. We stand up a sandbox copy of the CRM, the ticketing tool, the payment provider’s test mode and the mailbox, seeded with the data the scenarios depend on. Mocks that return canned success teach you nothing about what the agent does when the API returns a 500, a partial result or an unexpected field. The sandbox is reset between runs, and every write the agent makes lands somewhere we can inspect afterwards.

This is also where permission scoping gets tested rather than declared. If the agent is supposed to have no delete rights, a scenario tries to make it delete, and the sandbox should refuse.

Adversarial inputs, long runs and the cost meter

Three families of scenario catch most production failures. Malformed and hostile inputs: an empty email, a 40-page PDF where a one-pager was expected, a ticket whose body contains instructions aimed at the agent. Long-horizon runs: tasks that need fifteen or twenty steps, where the agent loses the thread, repeats a call, or retries until something external stops it. And the cost and latency meter on every run, because an agent that solves the task in ninety tool calls has solved it at a price nobody will pay. We set a hard ceiling per scenario on steps, tokens and wall-clock time, and a run that hits the ceiling fails whatever the outcome.

Every agent we have put into production has, at some point in testing, called the same tool with the same arguments several times in a row. The circuit breaker (same call, same arguments, third time, stop) gets built there, before a production loop runs up a bill overnight.

Regression on every change, and a threshold the business agreed to

Prompts, models and tool definitions all change the agent’s behaviour, so the suite runs on every change to any of them, the way a test suite runs on every commit. A model upgrade is treated as a release, with the full suite run, the traces diffed and a sign-off.

Passing is defined before the first run, with the business owner, as three numbers: task completion rate on the suite, zero forbidden actions, and cost and latency within the budget per task. Those are the acceptance thresholds, set per workflow: a drafting agent whose output a person reads can live with a lower completion rate than one that posts to the ledger.

The suite is the gate for widening permissions

The agent goes live with narrow rights, meaning read access, drafts instead of sends and a human approval on every write. Widening any of that is a decision taken on evidence from the suite plus production traces, which is why we log every tool call from day one. The suite is rerun with the broader permissions, the forbidden-action list is updated, and the new threshold is agreed before the switch flips. Teams that skip this join the pilots that proved the demo and never the workflow, with an agent trusted on a demo and demoted after its first expensive mistake.

This is the test discipline every other piece of software already gets, adapted to a component that can act. If you have an agent that performs well in demos and nobody can say what it would do with a hostile ticket, building the suite is a few weeks of work, usually shorter than the argument about whether it is needed.