The Dashboard Everyone Loves
At a sprint review, the QA manager presents a dashboard: 96.2% test pass rate, 88% requirements coverage, 0.8 defects per test case, and an average defect age of 4 days. The numbers are better than last quarter and the manager is praised. A tester privately raises three points. First, 38% of all defects this release were logged in the final five days of the cycle. Second, 11 of the 19 production incidents after the previous release trace back to requirements marked 'covered', where the only linked test case was a single happy-path smoke check. Third, coverage counts a requirement as covered if any test case references it, including one that only opens the page. Meanwhile the team quietly retired 30% of its test cases last sprint, shortly after the pass-rate target was announced.
1. Diagnose the failure mode of each of the four reported metrics.
Read the worked answer
96.2% pass rate. Pass rate measures execution outcomes, not product quality. It rises whenever tests are weakened, retired or written to pass - and the team retired 30% of its cases right after the target was set. That is Goodhart's law in action: the metric became the goal and stopped measuring the thing it was a proxy for. A suite of trivial tests can report 100%.
88% requirements coverage. The definition is 'any test case references it', which counts a page-open smoke check as coverage. This measures traceability paperwork, not test adequacy. The 11 of 19 production incidents traced to 'covered' requirements are the empirical proof: the coverage number was not wrong, it was measuring something else. Coverage also says nothing about negative paths, boundaries or integration, which is where those incidents most likely lived.
0.8 defects per test case. The denominator is arbitrary. Merge three test cases into one and the ratio rises; split one into five and it falls. It is also directionally ambiguous: a high defect-finding rate can mean excellent testing or terrible code, and you cannot tell which without context. Normalising by test case rather than by requirement, feature or thousand lines of code makes cross-release comparison meaningless.
Average defect age of 4 days. A mean hides the tail that matters. If 25 defects close in two days and three criticals sit open for 40 days, the average still looks healthy. Fast closure is also gameable: closing as 'cannot reproduce' or 'works as designed' improves the number without fixing anything.
The real signal in the room: 38% of defects found in the final five days. That late-arrival curve predicts escapes, and every metric on the dashboard was blind to it.
2. Propose replacements with concrete, testable definitions, and say what each one is for.
Read the worked answer
Use a small set of paired leading and lagging indicators; one number can always be gamed.
| Metric | Definition | Purpose |
|---|---|---|
| Requirement test adequacy | A requirement counts as covered only if linked to at least one positive and one negative/exception test case at the appropriate risk level, reviewed by a second person | Leading; replaces the reference-count definition |
| Defect detection phase distribution | % of defects found in requirements review, design, unit, system test, UAT, and post-release | Leading; shows whether testing is front-loaded |
| Escaped defect rate | Confirmed production defects per release, normalised per 100 requirements delivered, split by severity | Lagging; the outcome that matters |
| Late defect arrival | % of defects logged in the final 20% of the cycle, and the shape of the arrival curve | Leading; a rising tail is the strongest pre-release warning |
| Blocked and not-run visibility | Cases executed / blocked / not run / failed, always shown together, never aggregated into one pass figure | Prevents inflated pass rates |
| Pass rate by priority tier | Never report headline pass rate alone; always split P1/P2/P3 and show counts behind the percentages | Makes weighting visible |
| Reopen and verification rate | % of fixed defects reopened, and count of fixes awaiting independent retest | Catches false closure |
| Suite reliability | % of automated runs failing for reasons unrelated to the change | Protects trust in the gate |
For the production incidents, add root-cause category per escape (missing requirement, missing negative test, environment gap, data gap, regression not covered). That is the metric that tells you what to change, and it is the one most teams never compute.
3. How would you present this to management without it sounding like excuse-making, and what would you ask them to approve?
Read the worked answer
Lead with the decision the data supports, not with a defence of the team. Something like: 'The dashboard is telling us the opposite of what it appears to. Our pass rate went up because we removed 30% of the tests, and 11 of the 19 incidents after the last release came from requirements we counted as covered. Here is the one change I want to make, and here is what it will do to the numbers.'
Framing rules:
- Show the mechanism, not the blame. 'A requirement linked to one smoke check counts as covered' is a definition, not an accusation. Nobody has to be at fault for a metric to be wrong.
- Predict the bad news in advance. New definitions will drop reported coverage, possibly sharply. Say so before the next review, and give the reason, so the fall is understood as honesty rather than decline.
- Pair every lagging number with a leading action. Do not just present escaped-defect rate; present the change to requirement test adequacy that is meant to move it, and the date by which you expect a signal.
- Bring one worked example. The VAT or checkout scenario traced from requirement to smoke check to production incident is more persuasive than any chart.
- Kill the composite score. Do not let anyone combine the metrics into a single index; that recreates the same problem with more arithmetic.
Ask to approve: (1) the revised coverage and pass-rate definitions, effective immediately, with a re-baselined dashboard; (2) a two-release pilot of defect root-cause categorisation for every escape; (3) a written rule that no test may be retired without a documented risk rationale and a second reviewer; (4) fifteen minutes in each sprint review for the defect arrival curve rather than a single headline percentage.