Why DORA Metrics Break for AI Coding

Ask a chatbot whether DORA metrics still work once your team adopts AI coding agents, and it will hedge: metrics are still useful, just pair them with quality measures and watch for gaming. That hedge is not wrong, but it dodges the specific failure. The problem is not that DORA is gameable in general. It is that AI shifts the bottleneck, so two of the four DORA metrics start measuring activity that no longer predicts delivered value, while the team's felt productivity climbs at the same time. This page names which metrics go vanity, why, and what an operator should watch instead.

Do DORA metrics still work for AI coding?

DORA's four keys — deployment frequency, lead time for changes, change failure rate, and time to restore — were defined to measure a delivery pipeline whose scarce resource was human authoring time. AI coding agents attack exactly that resource. They make the writing of code, the opening of pull requests, and the drafting of changes cheap and fast. So the metrics tied to upstream output inflate, while the metrics tied to downstream outcome do not move on their own.

Split the four keys by where the bottleneck sits:

• Velocity-coded (inflate with AI): deployment frequency and lead time for changes both improve when more changes are authored faster, regardless of whether those changes were worth shipping.
• Stability-coded (do not improve for free): change failure rate and time to restore measure whether what you shipped held up. More code, faster, can move these the wrong way if review and verification do not scale with output.

DORA still works as a frame. The trap is reading a rising deployment frequency as a productivity win when it is really a rising-output, flat-outcome signal.

Why is delivery flat while output rises?

This is the AI productivity paradox: individual output goes up, team throughput of finished, valuable work does not follow. Three things absorb the extra output before it reaches users.

1. Review becomes the bottleneck. If an agent triples the pull requests a developer opens, but a human still has to read and approve each one, the queue moves at the reviewer's pace. The new code piles up in review rather than in production.
2. Rework hides in the failure metrics. Code produced fast but under-verified comes back as defects, reverts, and follow-up fixes. That shows up in change failure rate and time to restore, not in the velocity numbers the team is celebrating.
3. Perceived speed outruns real speed. People feel faster than they are. In a randomized controlled trial of experienced open-source developers on their own repositories, METR found the developers were about 19% slower with AI tools while believing they had been roughly 20% faster. The gap between felt and actual speedup is the paradox in one measurement.

A dashboard that shows more commits and more PRs while lead-time-to-value and failure rate sit still is showing you the paradox rather than a win.

What should I measure instead?

The fix is to stop counting authored output and start counting output that survived verification and shipped. Watch a small set of outcome-anchored measures alongside the stability half of DORA.

• Accepted change rate: of the changes an agent produces, what fraction is merged and stays merged after a defined window (say, not reverted within a week). This separates volume from delivered work.
• Cost per accepted change: the metered agent cost — tokens times turns — divided by changes that actually landed and held. This is the unit economics of agent work, and it is the spine of Module 7. A falling cost per accepted change is real productivity; a rising raw output with flat accepted changes is not.
• Review latency and rework rate: how long changes wait for human review, and what fraction come back as fixes. These expose the bottleneck and the hidden rework directly.

Avoid leaning on public benchmark scores as a proxy for any of this. Benchmarks measure model capability on fixed tasks rather than your delivery, and even the headline ones are unstable: OpenAI stopped reporting SWE-bench Verified, a benchmark it had featured for earlier models, citing contamination. The accepted-change numbers from your own codebase are what you can stand behind.

The rule

Treat deployment frequency and lead-time-to-merge as vanity metrics once an AI agent is authoring a meaningful share of your changes. They measure how fast the agent produces, which is now cheap and uninformative. Anchor on accepted change rate, cost per accepted change, and the stability half of DORA — change failure rate and time to restore — because those measure work that survived verification. If a metric improves when the agent writes more without anyone confirming the result, it is counting activity, not delivery.

Go deeper

The unit-economics view — cost per completed, verified task, and how to drive it down without inflating the total bill — is the subject of Module 7. Next step: pick one team dashboard, find the velocity-coded metric on it, and add accepted change rate beside it for one sprint to see the gap.