Auditing AI agent startups: reliability is the product

Evaluating AI agent startups requires one mental shift: with an agent, the model is a commodity and reliability is the product. Anyone can wire a frontier model to tools and produce a demo that books a meeting or files an expense. The company you are underwriting is the one that made the same behavior work on the ten thousandth messy, real-world case. So the diligence question is never "can it do the task". It is "how often, at what cost, and what happens when it fails".

The demo always works. Founders rehearse it, the happy path is real, and the model is genuinely impressive. The logs tell the truth, and everything below is a way of getting to the logs.

Task completion is the metric that matters

The first number to establish is the completion rate: the share of tasks the agent finishes end to end, correctly, without a human stepping in. Then interrogate the denominator, because the same product can honestly claim ninety-five percent on tasks the routing layer accepted and sixty percent on tasks customers actually submitted. Both numbers are real. Only the second one prices the company.

Ask how completion is defined, who verifies correctness, what share of traffic gets quietly routed to humans, and how the rate has moved over six months as usage broadened. A team that answers from a dashboard has built a product. A team that answers from a benchmark has built a demo, and we treat vendor benchmarks the way we treat any performance claim that looks too good: as marketing until reproduced.

Blast radius: price the failure, not the error rate

Two agents with identical error rates can deserve opposite investment decisions. An agent that drafts replies for human review fails cheap: the cost of an error is an editor’s minute. An agent that touches money, sends messages to customers, or writes to production systems fails expensive, and a two percent error rate against ten thousand daily actions is two hundred incidents a day.

So diligence prices the failure, never the failure rate alone. What is the worst single action the agent can take without a human in the loop? Has it happened? What did the postmortem say? A team that cannot name its worst production incident either monitors nothing or ships nothing autonomous, and both answers should move the valuation.

Guardrails: where agent startups differentiate

Between the model and the real world sits the engineering that makes autonomy survivable, and this layer is where strong agent companies actually differentiate. We look for scoped permissions per task rather than a god-token, approval gates on irreversible actions, execution timeouts and spend caps, an audit trail that reconstructs every action with its reasoning, and a kill switch someone has actually used. The principles mirror what we recommend when clients give agents access to their own systems, and their absence in a company selling autonomy is a finding of the highest severity.

The eval suite belongs here too. An agent startup without a regression suite of real task cases is changing prompts and models blind. Evals are what separate systems from demos, and in agent products the gap is wider than anywhere else.

Cost per completed task

Agent economics hide in the retries. A single completed task may burn multiple model calls, tool invocations, long context windows, and one or two full restarts, and failed tasks consume tokens too, so the real unit cost is total inference spend divided by completed tasks, plus the human minutes spent on escalations. We have reviewed agent products where that number, honestly computed, sat above the price charged per task: negative gross margin dressed as growth.

Ask for the fully loaded cost per completed task, its trend, and its sensitivity to model pricing, the same exercise as the margin question in any AI deal, sharpened by the fact that agents consume an order of magnitude more tokens than chat products.

Liability: who pays when the agent acts

An agent that acts creates obligations no chatbot ever did. Read the customer contracts for what happens when the agent errs: is there a liability cap, an indemnity, a carve-out for autonomous actions? Check whether insurance actually covers agentic failure modes. In regulated sectors, ask who the regulator holds responsible when an automated action breaks a rule, because "the AI did it" is not a defense anyone accepts. Contracts that promise outcomes while the product delivers probabilities are a legal finding wearing a product’s clothes.

None of this is an argument against agent startups; the category contains some of the most defensible companies we have audited, precisely because operational reliability at scale is brutally hard to copy. It is an argument for auditing them on their true product. When we run a technical and AI due diligence on an agentic company, the demo gets an hour and the logs get days, and the deals that survive that ordering are the ones worth doing.