Measurement, Metrics and Reporting
Measure to learn, not to police
Measurement exists to support decisions, expose bottlenecks and verify improvement. Metrics used to rank or punish individuals get gamed and destroyed - Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Prefer small sets of outcome-oriented, transparent metrics shared openly across teams.
Concepts a candidate must know
- The four key delivery metrics (DORA): lead time for changes, deployment frequency, change failure rate, and time to restore service. The first two describe throughput; the last two describe stability. High performers score well on all four, refuting the idea that speed and stability trade off.
- Flow metrics: work in progress, cycle time, throughput, queue time.
- Lagging versus leading indicators: customer satisfaction and revenue lag; deployment frequency and test coverage lead.
- Monitoring and observability: logs, metrics and traces; alerting on symptoms users feel rather than on every internal threshold.
- SLIs, SLOs and error budgets: a service level indicator is a measured signal; the objective is its target; the remaining budget guides how much change risk is acceptable and when to prioritise reliability.
- Vanity metrics (page views, lines of code, number of tickets closed) look good but do not inform decisions.
Traps in questions
- Using velocity, story points or individual utilisation as performance measures across teams - they are not comparable.
- Reporting only averages: averages hide tail latency and long-running work; percentiles and distributions are more informative.
- Measuring deployment frequency in a team that genuinely deploys rarely and concluding the team is bad, rather than examining the constraint.
- Treating the metrics as a scoreboard for management rather than a feedback signal for the team.
Worked example
A team deploys 40 times in 20 working days (deployment frequency = 2 per day), with a median lead time of 6 hours, 4% of changes causing failure, and a median restore time of 30 minutes. Compare with the previous quarter's 0.5 deploys per day, 4-day lead time and 15% failure rate: throughput improved and stability improved together.