Release 7.2 Go/No-Go: Flattening Discovery Curve or Falling Test Effort?
You are the software quality engineer for a 480 KLOC payment-processing product. Release 7.2 is due in nine working days. Execution stands at 2,140 of 2,430 planned test cases (88%). Open defects: 3 severity 1, 11 severity 2, 27 severity 3. Weekly defect discovery over the last five weeks was 52, 47, 44, 31 and 12; test effort in the same weeks was 310, 285, 260, 180 and 95 test-hours, because half the test team was pulled onto a customer escalation. Requirements-to-test coverage is 91%, but the uncovered 9% includes all 14 requirements for the new refund-reconciliation feature. Two severity-1 defects surfaced in the last four days, both in modules that had been unchanged and 'stable' for six weeks. Product management wants a go/no-go answer on Friday and has already told the sales team the ship date. The last three releases each escaped with 9-14 field defects.
1. The test manager argues that the fall from 44 to 12 defects per week shows the product is converging and is ready to ship. Evaluate that claim using the data given.
Read the worked answer
Reject the claim: the apparent convergence is an artifact of declining test effort, not of product maturity.
The right comparison is defect discovery per unit of detection effort, not raw counts:
| Week | Defects | Test-hours | Defects per 100 test-hours |
|---|---|---|---|
| 1 | 52 | 310 | 16.8 |
| 2 | 47 | 285 | 16.5 |
| 3 | 44 | 260 | 16.9 |
| 4 | 31 | 180 | 17.2 |
| 5 | 12 | 95 | 12.6 |
Normalized, the find rate is flat at roughly 17 per 100 test-hours for four weeks and dips only in week 5. The test process is still encountering defects at the same rate; the organization simply stopped looking as hard. Extrapolating the raw curve to a ship date is therefore invalid.
A second check kills the claim independently: with an expectation near 16 defects, a week-5 count of 12 is within about one standard deviation of ordinary Poisson variation, so it is not statistically distinguishable from the flat trend at all. Small weekly counts are noisy.
Finally, reliability growth requires a declining rate under constant or increasing effort plus an improving severity mix. Here severity 1 defects appeared in code that had been unchanged for six weeks (a detection-latency problem, not a stability signal), and the prior three releases each leaked 9-14 field defects, so no historical evidence supports a clean exit.
2. What does the coverage gap and the two late severity-1 defects imply about residual risk, and how would you spend the remaining nine working days?
Read the worked answer
Both findings point to risk that is concentrated, not spread evenly - so spend the nine days on the concentration, not on finishing the case count.
Coverage gap. The missing 9% is not random: it holds 100% of the refund-reconciliation requirements, which are new functionality sitting on money movement. Risk exposure is roughly probability of failure times impact; untested new code with financial impact is the highest cell in that matrix. Finishing 290 low-value cases to reach 100% execution would leave the largest risk untouched.
Late severity-1 defects. Defects appearing in modules 'stable for six weeks' indicate that stability was an assumption rather than evidence. Two plausible explanations must be distinguished before more testing is planned: (a) the recent change to those modules was not regression-tested deeply enough, or (b) the defects are old and were only exposed by newer, more adversarial cases. Check the defect cluster against change history and defect-prone-module data.
Nine-day plan:
- Run all 14 refund-reconciliation requirements end to end, including negative paths, boundary values, rounding and partial-refund sequences, with business sign-off on expected results.
- Do a focused inspection of the highest-risk reconciliation code - a two-hour structured review by three engineers often finds what 40 test cases would.
- Regression-test the exact modules touched by both severity-1 fixes plus their callers, and require fix verification by someone other than the fixer.
- Re-normalize the defect curve weekly so the effort drop does not repeat the same distortion.
3. How would you frame the go/no-go recommendation to product management, and what release criteria would you propose?
Read the worked answer
Give a conditional, evidence-backed recommendation - not a bare yes or no, and not a date chosen by the sales team.
Frame it as residual risk the business is choosing to accept. State plainly: at current effort the defect discovery rate has not declined; 14 high-impact requirements are unverified; and open severity-1 and severity-2 defects exist. Then offer two options with their conditions:
Option A - ship on the date, with controls. Acceptable only if: zero open severity 1; every severity 2 has a documented workaround and named customer-facing owner; the 14 reconciliation requirements are fully tested and signed off; the two recent fix areas are regression-verified; and a staged rollout (5-10% of customers) runs with monitoring, a defined abort threshold and a tested rollback path.
Option B - slip two weeks. Restore dedicated test effort, close the coverage gap, and require two consecutive weeks of declining normalized discovery rate before GA.
Proposed standing release criteria (so future debates are not relitigated by assertion): risk-weighted requirement coverage at 100%; zero open severity 1; severity-2 defect density per KLOC below the trailing baseline; normalized discovery rate declining for two consecutive weeks; no open defect against an untested or unverified requirement; and a documented residual-risk statement signed by product and quality.
State the cost of each option openly. Exiting with the usual 9-14 field defects is a known cost, not an unknown one; quantify it and put it beside the cost of the two-week slip so the decision is made by whoever owns the trade-off, with the risk written down.