Regression Debt Before Content Lock
Project Lantern is a 3D action RPG nine months from launch, currently at alpha. Forty developers work in trunk-based development and twelve feature teams merge to trunk daily. QA is four testers and one lead. The nightly automated smoke suite has a 34% failure rate, but roughly 60% of those failures are attributed to known timing flakiness on UI load screens. The manual regression suite is 900 cases and has not had a full pass in five weeks; the last partial pass covered combat and inventory only. There are 1,240 open defects, 180 of them S1 or S2, and the average age of an open S2 is 22 days. No entry or exit criteria exist for the beta milestone, and about 40% of defects are closed by developers without a retest. Production wants to declare content lock in three weeks and begin beta in six.
1. Production asks whether to freeze trunk for one week so the full 900-case manual regression suite can finally be run. Give your recommendation with reasoning, and state what you would do instead.
Read the worked answer
Freezing trunk buys one clean pass and costs twelve teams a week of throughput, and it guarantees nothing because the suite itself has not been maintained. With 60% of the smoke failures classed as timing flakiness, a full run produces a third of its results as noise, so even a completed pass would not be a trustworthy signal.
Do not freeze. Instead:
- Quarantine flaky automation into a non-blocking job and fix the top five offenders (UI load timing) with explicit state polling, so the nightly signal becomes actionable within a week.
- Run a risk-based regression rather than all 900: rank cases by (a) areas changed since the last pass, (b) areas carrying open S1/S2 defects, (c) player-critical paths such as save/load, progression and store transactions. Execute the top 250-300 first and report coverage as a percentage of risk, not of the suite.
- Add a per-team smoke gate so each of the twelve teams catches its own regressions within a day rather than five weeks.
- Fix the 40% close-without-retest rate before anything else. It corrupts every metric downstream: a defect closed without verification is indistinguishable from one that is still open, so the 180 S1/S2 count is not even a known quantity.
If production still wants the full pass, schedule it after content lock when churn drops, not before it.
2. Define the entry and exit criteria you would propose for beta, and explain what makes each one measurable rather than aspirational.
Read the worked answer
Entry criteria for beta, each with a data source and an owner:
- Zero open S1; no more than fifteen open S2, each with a named owner and a fix date.
- Automated smoke suite green on three consecutive nightly runs after quarantine, with the known-flaky list under 5% and every entry ticketed.
- A completed risk-based regression pass within the last ten builds.
- Content locked: no placeholder text, missing audio or temporary assets in the candidate build.
- Crash-free sessions at or above 99.0% over a 24-hour soak on target hardware.
- Save compatibility verified: the build loads saves from the previous two milestones.
Exit criteria for beta:
- Crash-free sessions at or above 99.5% over 72 hours across the supported hardware matrix.
- No open S1 and no open S2 older than fourteen days.
- Regression pass 100% executed, at least 95% pass, every failure triaged.
- Certification-style scenarios (suspend/resume, storage full, user sign-out) executed on retail hardware.
- Performance inside budget on minimum specification.
What makes them measurable is that each has a number, a data source (build system, crash telemetry, test management tool), a named owner and a stated consequence if missed. Words like stable and ready cannot be argued with; a threshold can. A gate with no consequence is decoration.
3. Which three metrics would you put in front of production leadership, and which current metrics would you stop reporting? Justify the removals.
Read the worked answer
Present three:
- Crash-free session rate, split by platform and hardware bucket. It is the closest available proxy to player pain and it does not depend on anyone's judgement about severity.
- Open S1/S2 count with an age distribution, not just a total. 180 S2 defects at a 22-day mean is a trend line telling you the team is falling behind; the same 180 with a two-day mean is a healthy pipeline.
- Escaped defects per milestone: bugs found in the current milestone that already existed in the previous one. This measures whether testing is working, rather than how busy testing looks.
Retire from the leadership dashboard:
- Raw defects found per week, which rewards volume and punishes good triage.
- Test cases executed as a percentage, because 900 executed cases says nothing about risk coverage; 300 well-chosen cases can cover more real risk than 900.
- Average defect age across all severities, where a 200-day cosmetic P4 drowns out the S2 signal that matters.
- Duplicate rate as a headline number; keep it internally as a report-quality check.
Whatever sits on the dashboard gets optimized. These four optimize for the wrong behaviour, so keep them where they drive a specific local action and out of the room where release decisions are made.