Measurement, reliability and learning signals
Measure to learn, not to police
Measurement exists to support decisions, expose bottlenecks and verify improvement. Metrics used to rank or punish individuals get gamed and destroyed - Goodhart's Law: when a measure becomes a target, it ceases to be a good measure. Prefer small sets of outcome-oriented, transparent metrics shared openly across teams.
Concepts a candidate must know
- Delivery metrics and their version: the historical four-key model described deployment frequency, change lead time, change failure rate and service restoration. Current DORA guidance uses five measures: deployment frequency, change lead time, failed-deployment recovery time, change fail rate and deployment rework rate. Its current grouping places the first three with throughput and the last two with instability. Identify the framework version before answering a question about a named set; do not label the legacy four as the complete current model.
- Flow metrics: work in progress, cycle time, throughput, queue time.
- Lagging versus leading indicators: whether a measure leads or lags depends on the outcome and time horizon. Delivery performance can lag engineering practices while leading later business outcomes; deployment count or coverage is not automatically a reliable leading predictor.
- Monitoring and observability: logs, metrics and traces; alerting on symptoms users feel rather than on every internal threshold.
- SLIs, SLOs and error budgets: a service level indicator is a measured signal; the objective is its target; the remaining budget guides how much change risk is acceptable and when to prioritise reliability.
- Vanity metrics (page views, lines of code, number of tickets closed) look good but do not inform decisions.
Traps in questions
- Using velocity, story points or individual utilisation as performance measures across teams - they are not comparable.
- Reporting only averages: averages hide tail latency and long-running work; percentiles and distributions are more informative.
- Measuring deployment frequency in a team that genuinely deploys rarely and concluding the team is bad, rather than examining the constraint.
- Treating the metrics as a scoreboard for management rather than a feedback signal for the team.
Worked example
A team deploys 40 times in 20 working days (deployment frequency = 2 per day), with a median lead time of 6 hours, two of the forty deployments causing failure, giving a5% change-failure rate, and a median restore time of 30 minutes. Compare with the previous quarter's 0.5 deploys per day, 4-day lead time and 15% failure rate: the supplied throughput measures improved and the failure rate fell. A change in recovery time cannot be inferred because no previous recovery time is given.