The 98.2% Pass Rate on the Release-Readiness Slide
Studio A ships a live-service action RPG on PC and two consoles. Eight months ago the team moved from 6-month milestones to 2-week sprints, and QA was folded into the sprint cadence without changing the regression approach. The suite holds 3,400 manual and 620 automated cases; the manual cases were last reviewed 14 months ago and 410 reference UI screens or crafting steps that no longer exist. In this sprint's full regression, 3,340 cases were marked Pass, giving a 98.2% pass rate that the producer has put on the release-readiness slide. In the same build, open Sev1/Sev2 defects rose from 34 to 61, the crash rate on the newest console SKU doubled week-over-week, and two save-corruption reports arrived from the external playtest. One tester flagged that several Pass cases were signed off with the note no crash observed. Feature freeze is in three weeks.
1. Explain why the 98.2% pass rate is not evidence of release readiness, and list the specific evidence you would collect in the next 48 hours to give the producer an honest picture.
Read the worked answer
Why 98.2% is not evidence
- Pass rate = passed ÷ executed against authored cases. It measures the suite, not the build. If the suite has drifted, a high rate means the tests are easy, not that the game is good.
- 410 cases reference UI and crafting steps that no longer exist. Testers cannot execute them, yet they are recorded as Pass (often with the note no crash observed), so part of the numerator is fiction.
- Fourteen months without review spans the sprint transition; new systems (the season pass, the newest console SKU) have no cases at all, so the denominator is missing the riskiest areas.
- Divergent signals: Sev1/2 count up 34 to 61, crash rate doubling on a new SKU, external save-corruption reports. When a metric disagrees with the defect trend, the metric is wrong until proven otherwise.
Evidence to collect in 48 hours
- Suite audit sample: re-run 100 random Pass cases and record how many are still valid, executable and actually assert something.
- Traceability map: which shipped features have zero cases; which cases map to code changed in the last 8 weeks.
- Crash telemetry by SKU and by scene, compared with the previous build.
- Defect escape rate (bugs found after a milestone ÷ total found) for the last three milestones.
- Per-area pass rate instead of the aggregate, because the average can hide a 100% green UI and a 60% green save system. Present it as a one-page risk table, not an argument about the number.
2. Rewrite the milestone exit criteria and describe how you would reprioritize the regression suite in three weeks without adding headcount.
Read the worked answer
Exit criteria (rewritten)
- Zero open Sev1; Sev2 at or below an agreed threshold per platform, each with a named owner and date, not a single global count.
- Coverage: every milestone feature has at least one passing P0 case executed on the release-candidate build, not a branch build from a week ago.
- Platform matrix: each supported SKU has run smoke, save/load and suspend-resume sets.
- Stability: a defined soak session with crash-free-session rate at or above target and memory growth inside budget.
- No unexplained regression in the automated suite; flaky tests triaged rather than re-run until green.
- Entry criteria too: the build is testable and patch notes name changed areas so regression can be targeted.
Reprioritizing in three weeks, no new headcount
- Archive the 410 stale cases on day one. A smaller true suite beats a large false one.
- Rank by risk = likelihood of change × player impact × platform exposure. Season pass and the new SKU first; tutorial and cosmetic UI get smoke only.
- Automate the top ~200 cases around money paths, loadout and save data so they run every build; spend the freed manual hours on chartered exploratory sessions against recent code changes.
- Cluster the 61 defects by area. Concentration means those systems need depth, not breadth.
- Publish the coverage map weekly so the producer sees what is not covered, which is the honest version of the green slide.
3. The producer asks you to just make the number green again. How do you respond, and which metric set do you propose to replace the pass rate on the readiness slide?
Read the worked answer
Response Reframe: the number is already green; the risk is not. Making it greener by re-running until pass, closing cases as not applicable without evidence, or lowering severities would falsify a quality signal that a release decision depends on. That is a professional-ethics line, not a scheduling preference. Say it once, plainly, then offer the alternative - a producer who is told no without an option will route around QA.
Replacement metric set
- Risk coverage: % of P0/P1 features with a passing case on the release candidate, per platform.
- Defect trend: open Sev1/Sev2 by area, arrival vs. closure rate, age of the oldest Sev2.
- Escape rate: defects found after each milestone ÷ total found. This is the metric that catches a weak suite, because it counts what QA missed.
- Player-facing stability: crash-free sessions and P99 frame time from telemetry, not just from QA hardware.
- Suite health: valid case count, flaky rate, last-reviewed date, % automated.
- Execution honesty: pass, fail, blocked and invalid reported as four separate buckets. Blocked is never reported as pass. On the slide, show red/amber/green per area with one line of what would change the colour. It takes the same space and survives scrutiny.