Measuring what AI actually changed in your delivery

If you want to measure AI coding productivity, measure delivery, not activity: cycle time, review load, defect rate, deployment frequency, all against a baseline captured before the tools arrived. Everything else on the vendor dashboard (acceptance rates, suggestions served, active users) measures engagement with the tool, and engagement is not the thing you are paying for.

The reason to be strict about this is that the easy numbers all point up while the numbers that matter often do not move. That divergence has a name now.

The AI productivity paradox

The 2025 DORA report found that 90% of technology professionals use AI at work and more than 80% say it makes them more productive. The same research found that higher AI adoption raises software delivery throughput and raises delivery instability with it, and settled on a framing we see confirmed in the field: AI is an amplifier. It magnifies the strengths of well-run organizations and the dysfunctions of struggling ones.

So individual developers are right that they are faster, and the VP looking at quarterly delivery is right that little changed. Both readings are true, because the time saved writing code reappears downstream: in review queues, in rework, in verification, in incidents from changes that shipped faster than the safety net matured. Measuring only the individual side is how companies convince themselves a rollout worked when it moved nothing.

Measure AI coding productivity against a baseline

The instrument is unglamorous: the delivery metrics you should already track, read against a before. Four cover most of it.

Cycle time, from first commit to merge, or ticket start to production. This is where genuine acceleration shows up, and where it visibly fails to show up when AI-written code sits in review for three days.

Review load: time to first review, queue depth, review time per PR. This is the leading indicator, because review is where AI-era delivery breaks first. If cycle time is flat while review queues grow, you have found your constraint.

Defect rate: change failure rate, reverts, hotfixes, escaped bugs. This is the check on the DORA instability finding, and the number that tells you whether speed is real or borrowed from next month.

Throughput: merged PRs or completed tickets per week, per team. Noisy on its own, useful alongside the other three.

None of this needs a new platform. Git, CI and the issue tracker already hold the data; a baseline week before rollout is what makes it interpretable. Without the before, every reading dissolves into "maybe it was the new hire" and the debate never closes.

What to ignore

Acceptance rate, first. It is on every vendor dashboard and it correlates with almost nothing: accepted code gets rewritten, deleted, or ships a bug, and none of that shows in the metric. We took it apart in acceptance rate is a vanity metric.

Lines of code, still, and now with a twist: agents generate volume effortlessly, so the metric rewards exactly the verbosity you should be resisting. Active-user counts tell you the license is used, not that it helps. And self-reported time savings are worth collecting for sentiment, worth nothing as evidence: DORA’s 80% would have declared victory at companies whose delivery data says otherwise.

One number worth adding rather than ignoring: the share of merged work that started as a task delegated to an agent end to end. It is the cleanest signal of whether the team is actually changing how it works or just typing with assistance.

Read the numbers like an engineer

Expect a J-curve: the first two to four weeks of real adoption are often slower while the team pays the learning curve, which is why measuring too early kills programs that were working. Compare pilot squads against their own history, not against other teams with different codebases. Watch instability with the same attention as speed, because the failure mode DORA documents (throughput up, stability down) is precisely the one that flatters a quarterly slide before it burns you. And when a number moves, ask what else moved: smaller PRs, a new deploy pipeline, a reorg. Attribution deserves the same skepticism you would apply to any benchmark.

Measured this way, the results are usually undramatic and real: cycle time down on well-scoped work, review load up until the team redesigns review, quality flat if the guardrails came first. That is what an honest win looks like.

This measurement loop, baseline first and monthly readings after, is built into every agentic engineering engagement we run, because a rollout without it is a story, and stories do not survive contact with a CFO.