The League Table That Made It Worse

The metric became a target

Deployment frequency is an outcome measure of a system, and pairing it with a league table converts it into a performance measure of people. Once a measure is used for ranking, it stops measuring what it described.

The evidence is already visible

• Team A's 200 deployments at 34% failure means roughly 68 failed deployments a month. Team B's 8 at 6% means about half a failure a month. The ranking says the opposite of what the underlying data says about customer risk.
• Counting documentation updates as deployments is unambiguous gaming, and it tells you the incentive is now stronger than the definition.
• Splitting changes is more ambiguous: small batches are genuinely good, but doing it to inflate a count rather than to reduce risk is not, and from outside you cannot tell the two apart - which is itself the problem.

What happens next if it stays Failures get hidden or reclassified, restore time becomes unattractive to record, teams optimise their own number rather than the value stream's, and the dashboard becomes a fiction that executives use to make decisions. Trust between teams and leadership erodes, which is the expensive part.

What The four DORA metrics plus reliability: lead time for change, deployment frequency, change failure rate, failed-deployment recovery time, and availability against SLO. Add flow efficiency and queue sizes for diagnosing where work waits.

Rules of presentation

• Never a single metric, never a ranking of teams. Speed metrics are always shown with their stability pair; a frequency figure without a failure rate is an invitation to game.
• Trends over time against the team's own baseline, and distributions rather than averages for lead time.
• Segment by service criticality, since a payments service and a marketing site are not comparable.
• Teams own their own dashboards and see the same data leadership sees. Measurement done to people produces the behaviour you have just seen.

Handling the gaming

1. Name it as a system failure, not misconduct. The league table caused the splitting and the doc-update counting; say so publicly and withdraw it, visibly, from a senior level.
2. Reset the baseline with no retrospective penalty, so nobody has to defend past numbers.
3. Keep raw data visible so definitions can be audited, and periodically sample whether a 'deployment' corresponds to real change reaching users.
4. Expect recovery time to be the hardest to instrument - if incident start and end are not captured, start with a lightweight manual log rather than deferring it.

Compare systems, not people. The VP's real question is 'where does extra capacity or money buy the most improvement?', and that is answered by constraint analysis, not by ranking.

Offer:

• Value-stream maps with quantified wait times for the top three or four delivery paths, showing flow efficiency and the largest queues. This directly points at where to invest.
• Lead-time distributions by service, so the conversation becomes 'which services have tails we cannot tolerate?'.
• SLO and error-budget status, which gives an objective health signal per service without judging the team that owns it. A service burning its error budget needs investment; that is a property of the service, not a verdict on its engineers.
• Toil and support load as a capacity input, since a team spending 60% of its time on manual work cannot improve regardless of skill.
• Cost of delay for the work queued behind the constraint, which is the language investment decisions are actually made in.

Be explicit about the trade-off being declined: a team ranking on DORA metrics predicts gaming, not performance - you already have two months of evidence. If the VP needs comparability, give them comparability across services and value streams, with trend against each group's own baseline.