Reading Defect Metrics Under Gaming Pressure
Release 3.2 of a logistics back-office suite (about 60 KLOC of changed and new Java code) went live four weeks ago. System test found 412 defects; 38 more were reported by users in the first 30 days. Of the 412 test defects, 61 were reopened after being marked fixed, and the triage board downgraded 27% of the severity-1 defects raised by testers. Average open-to-closed time is 22 days, and 40% of still-open defects are older than 60 days. Testers are measured on 'defects found per tester-day'; developers on 'defects fixed per sprint'. The test manager's dashboard reports defect density of 6.9 per KLOC and calls the release high quality 'because density is above the industry benchmark'. In the retrospective the development lead argues testers inflate severity to hit their metric, and the test lead argues developers close defects without a real fix. Release 3.3 is two weeks away.
1. Which of these indicators are trustworthy, and what do the escape rate, reopen rate and downgrade rate jointly indicate?
Read the worked answer
Defect density of 6.9 per KLOC is not interpretable on its own: 'above benchmark' is meaningless without a stated benchmark, a denominator definition, and a comparison against the same team's own history on comparable modules. A high found-density can mean good testing or a poor product.
The three process indicators are more informative and should be read together:
- Escape ratio = 38 / (412 + 38) = 8.4% of known defects reached users within 30 days.
- Reopen rate = 61 / 412 = 14.8%.
- Severity downgrade = 27% of severity-1 findings.
A high reopen rate means fixes are not verified against the original failure; a high downgrade rate means severity has no shared, objective definition. Together they point at the definition of done for a fix and at the severity taxonomy, not at tester honesty. An 8.4% escape rate with a 22-day average cycle time also suggests escapes are partly a scheduling effect: defects fixed late get little regression time.
Both arguments in the retrospective are symptoms of measuring people on counts. Once 'defects found per tester-day' is a target, low-value defect logging is rational; once 'defects fixed per sprint' is a target, closing without fixing is rational. This is Goodhart's law in operation, and the metrics must be changed before behaviour will.
2. Design a severity and priority scheme plus a triage process that would end the dispute.
Read the worked answer
Separate the two concepts, because most disputes come from conflating them.
Severity (impact, set by the reporter, objective):
- S1: data loss or corruption, financial mis-posting, security exposure, no workaround, service unavailable.
- S2: major function unusable but a workaround exists; degraded performance beyond the stated budget.
- S3: minor function incorrect with an easy workaround.
- S4: cosmetic, wording, or usability observation.
Each level needs a one-line decision rule and a required evidence pack: reproduction steps, environment, expected versus actual, logs or payload. A defect without a reproduction is 'S-unconfirmed', not S3 by default.
Priority (scheduling, set by the triage board): P1 fix before release, P2 next patch, P3 backlog. Priority may differ from severity - an S2 on the peak seasonal path can be P1.
Process:
- A daily 15-minute triage with named roles: test lead, dev lead, product owner. Severity can only be changed in triage, with the reason recorded, never by silent edit.
- Reopened defects are logged as a new linked record and counted against fix quality, not against the tester.
- A state rule: 'fixed' requires a linked passing regression check around the change plus evidence on the original reproduction.
- Weekly report shows reopen rate and downgrade rate as trends, per component, not per person.
3. With two weeks to release 3.3, what actions would you take and what would you measure to know they worked?
Read the worked answer
Two weeks is enough for process changes and targeted testing, not for a new tooling programme.
Actions:
- Fix-verification gate: no defect may be marked fixed without (a) the original reproduction re-run and (b) at least one adjacent check on the changed code path. Reopened defects go back to the same developer with the failing evidence attached.
- Cluster the 412 defects by module and by defect type (data handling, interface, validation, state, timing). If, say, 60% sit in two modules, redirect the remaining test effort there rather than re-running the full suite uniformly. This is the highest-value use of the two weeks.
- Age-based WIP limit: cap the number of open defects older than 60 days that the team will carry, and force a decision (fix, defer with documented risk, or close as won't-fix) on the rest before the release candidate.
- Remove the individual count metrics immediately, replacing them with escape rate, reopen rate and defect age as team-level trends.
Measure, and baseline now so the trend is visible:
- Escape ratio (target below 5% within two releases).
- Reopen rate (target below 5%).
- Percentage of S1/S2 defects with an automated reproduction (the leading indicator).
- Age profile of open defects, specifically the count over 60 days.
- Root-cause category distribution of escaped defects, sampled from the last two releases.
Track these on a trend line across releases rather than as pass/fail on 3.3; a single release is too noisy to prove anyone right.