Scorecard Metric Erosion: Defect Counts, Duplicates and Rising Escapes

The numbers describe count inflation, not improved quality, and they show the measures are invalid as constructed.

Read the indicators together rather than one at a time:

• Recorded defects per tester rose 50% (9.4 to 14.1) while average severity fell (2.4 to 1.6). More items, each individually less significant, is the signature of splitting and re-logging rather than of finding more real problems.
• Duplicates up from 4% to 19% and reopens up from 3% to 11% confirm it: the same underlying problems are being recorded repeatedly and closed prematurely.
• Escapes rose 83% (2.1 to 3.8) - the outcome that matters most moved the wrong way, decisively.
• Test cases up 35% with risk-weighted coverage flat at 87% means more cases were written, but not against the requirements that carry risk.

The mechanism is straightforward: the scorecard pays for countable, tester-controlled outputs, so behavior optimizes those outputs. Defect removal efficiency is additionally corrupted, because it depends on a denominator (total defects present) that is estimated partly from the very counts being inflated.

This is a construct-validity failure, not a motivation failure. The metric never measured quality; it measured logging behavior, and once it had consequences, logging behavior changed. No amount of communication or 'be honest' messaging fixes a metric whose cheapest path to a good score is the opposite of the desired outcome.

Move from countable activity at the individual level to outcomes and in-process quality at the team and organization level.

Stop: individual defect counts, individual DRE, weekly testers' rankings, and the proposed cases-executed-per-day. Each is directly controllable by the person being measured and gameable by splitting or by executing trivial cases - the vice president's proposal repeats the exact error already visible in the data.

Measure instead:

• Outcomes (lagging): field escapes per release, weighted by severity, normalized per KLOC or per thousand transactions; customer-reported severity-1 count.
• In-process (leading): escaped-defect root-cause distribution; review and inspection effectiveness (defects found per review-hour on Class C items); risk-weighted requirement coverage, replacing raw case counts; test-case churn and duplicate rate as a data-hygiene indicator.
• Productivity, if required: use a diagnostic, team-level proxy such as planned-versus-actual effort variance, and never for ranking individuals.

Level and statistics. Aggregate at team level with sample sizes large enough to be stable, and plot on control charts rather than comparing raw month-to-month numbers. With defect counts in the tens per month, most apparent movement is noise; a run or trend rule on a p-chart or u-chart separates signal from variation.

Balance. Pair every quantity measure with a quality guard: if you count defects found, pair it with escapes and reopens; if you count cases executed, pair it with risk coverage and defect detection per case. A measure with no counterweight will be optimized in isolation.

Refuse the addition, but refuse it with evidence and an alternative - and retire the old measures as a phased replacement, not a deletion.

On cases executed per day: it is the same failure mode with a new name. Case count is an input fully under the tester's control; the cheapest way to raise it is to split cases into smaller ones and execute trivial checks. The current data already demonstrate the pattern: cases up 35%, risk-weighted coverage unchanged at 87%. Adding a third individual count would amplify splitting, duplicate logging and premature closure, and it would further depress the escape metric that leadership actually cares about. Offer the substitute: risk-weighted coverage plus defects detected per test-hour on high-risk requirements, reported at team level.

Retiring the old measures without losing the quality message:

1. Announce that the measures are being replaced because they no longer measure what they were intended to measure - say so explicitly, with the escape-rate graph. Framing it as a correction protects credibility.
2. Run old and new in parallel for two quarters so trends can be compared before the old ones are dropped.
3. Publish the new dashboard widely, so quality is visibly more measured, not less.
4. Handle the historical injustice: identify testers whose ratings were depressed by low logging volume, and have calibration sessions review those cases.
5. After two quarters, sample for new gaming patterns - reopens, severity downgrades, requirement-splitting - and treat a gaming signal as a design defect in the metric, not as misconduct by the testers.