How we verify AI performance claims that look too good

When a deck claims 94 percent accuracy, the useful question is not whether the number is true. It is: measured on what, by whom, and when. To verify an AI startup’s performance claims, we ignore the slide and rebuild the number from three sources the company already has: its eval sets, a held-out test we run ourselves, and its production logs. All three fit inside a deal window. None of them requires a research project.

Why AI performance claims drift from reality

Benchmarks get chosen after the results come in. Public ones suffer from contamination (the test data leaks into training data) and from task mismatch (the product does not do what the benchmark measures). Internal ones get built by the team whose bonus depends on the number: cases accumulate from the easy end, the hard cases that failed last quarter quietly leave the set, and the whole thing ends up measuring a distribution that production stopped resembling a year ago.

None of this requires bad faith. It is what happens by default when nobody designs the measurement. The result is the same either way: a number that describes the demo, not the product.

Start with the eval set

Ask to see the eval set itself, not a summary of it. How many cases, who wrote them, when it was last refreshed, how it maps to real production traffic, and whether production failures get added to it. A strong team treats its eval set as an asset and talks about it the way a finance team talks about its ledger. The builder’s side of this discipline is covered in evals: the unglamorous work that separates systems from demos.

If there is no eval set at all, stop treating the performance claim as a measurement. It is an impression: someone looked at outputs and felt good. That is the single most common finding we make in AI deals, and it reframes every number in the deck.

Run a held-out test inside the deal window

This is cheaper than it sounds. Assemble 50 to 100 cases the team has never seen: drawn from the buyer’s own domain, or sampled from the target’s live traffic that same day. Run them through the real product, in the same interface customers use, and score them with the target’s own definition of correct. An afternoon, maybe two.

Precision is not the point; the delta is. A product that claims 94 percent and scores 90 on fresh cases is normal. One that scores 60 has either a measurement problem or a claims problem, and both are findings. Watch the failure modes too: wrong-but-plausible answers are a different risk than visible errors, because customers catch the second kind and act on the first.

Read the production logs

The logs tell the truth the deck cannot. Completion rates. Human override and escalation rates. Retries and fallbacks. Latency percentiles. The share of total traffic that actually goes through the AI path, which is sometimes the most clarifying number in the whole deal. Ask for a sample of raw sessions and read real user inputs: they are messier, shorter and stranger than anything in the demo script.

For agent products, the log read is nearly the entire assessment. Task completion rates and cost per completed task matter more than any benchmark, and we covered that in auditing AI agent startups.

Watch the demo choreography

Demos are rehearsed; that is fine and universal. The information is in what happens off script. Ask to drive. Bring your own inputs. Ask for the failure reel: strong teams show failures willingly, because they have catalogued them, built guardrails around them, and can tell you the rate at which each one occurs. The tell is not a failure during the demo. The tell is surprise, because a team surprised by its own product has never measured it.

One more choreography pattern worth knowing: a rules engine with a language model at the front can demo exactly like a learned system. Telling them apart changes how you price defensibility, and it is the wrapper question treated on its own terms.

Verified performance does not need to be perfect. It needs to be known. A team that says "87 percent on this distribution, here are the failure modes, here is the trend" is a better asset than one that claims 99 and cannot show the denominator. Rebuilding that number from evidence is standard scope in our technical and AI due diligence.