Measure before you roll out, or the debate never ends
Six months after the agent licenses arrive, someone senior asks whether any of it worked. If nobody captured baseline metrics before the AI adoption started, that question has no answer, only a debate. The enthusiasts have anecdotes, the skeptics have different anecdotes, and both positions are unfalsifiable. We have watched this argument run for an hour in a leadership meeting and end exactly where it began.
The fix costs one week, before rollout: capture cycle time, review load and defect rate, per team, from tools you already run. That is the entire recommendation. The rest of this note is how to do it without buying anything.
Baseline metrics before AI adoption: the four numbers
Cycle time, measured twice: ticket opened to change in production, and PR opened to merged. The first describes the whole system, the second describes the loop that agents touch most directly.
Review load: time to first review, total time in review, and how concentrated reviewing is. If two seniors handle most of the reviews, you want that on paper now, because agent volume will land on them first.
Defect rate: escaped bugs per month, revert rate, incident count. The exact definition matters less than consistency. Pick one, write it down, keep it.
Throughput: merged changes per week, with the obvious caveat that a count of PRs is gameable and only means something read alongside the other three.
Resist the urge to add more. Lines of code, commit counts and story points measure activity rather than delivery, and every number you publish is a number someone will eventually optimize. Four metrics that survive scrutiny beat twelve that invite gaming.
The data already exists
Everything above lives in systems you already run: git history, the GitHub or GitLab API, CI logs, the incident tracker. A baseline is a handful of scripts and a spreadsheet. Teams stall because this sounds like a metrics-platform project; in practice it is two days of queries. Where a number is genuinely unobtainable (ticket hygiene sometimes makes ticket-to-production meaningless), record that fact and use the PR-based measure rather than inventing precision.
One optional addition pays for itself: a ten-minute survey asking engineers where a typical week actually goes (writing, reviewing, debugging, meetings, waiting on CI). Self-reported data is soft, but the before-and-after comparison catches shifts the delivery metrics miss, like review time quietly doubling while cycle time improves.
Per team, trailing 90 days
Take a trailing 90 days so one unusual sprint does not define normal. And measure per team, never per individual. Individual metrics poison a rollout before it starts: engineers game them or fear them, and both reactions distort the very numbers you wanted. Delivery is a team property anyway.
Per team also because teams absorb agents differently. A squad on a greenfield TypeScript service and a squad inside a twelve-year-old monolith will get different value from the same tools, and a company-wide average would hide exactly the difference that tells you where to invest.
What the baseline buys you
Three things. First, grounded rollout decisions: pilot with one squad, measure the delta against the baseline, then extend or adjust. The rollout sequence that works depends on that comparison existing.
Second, immunity to vanity numbers. Without your own baseline, the dashboard that ships with the vendor tool becomes the scoreboard, and acceptance rate is a vanity metric. A baseline keeps the conversation on delivery, where it belongs.
Third, calmer politics. The skeptic who predicted quality would collapse either sees the defect rate move or does not. The enthusiast who promised doubled throughput faces the same numbers. The debate ends because it finally can.
The baseline week, as we run it
Days one and two: extraction, scripts against git, the PR API and CI. Day three: sanity check with the tech leads, because raw numbers lie (bot PRs, squash habits, a migration week that skews everything). Days four and five: agree the definitions, write the numbers into a one-page document, and get the people who will argue in six months to accept it now. That last step is the point of the exercise. A baseline nobody signed off on will be relitigated at exactly the moment you need it most.
A baseline week is week one of every agentic engineering engagement we run, before a single license is assigned, because every claim we make afterwards has to survive comparison with it.