Sign-Off After Three Quiet Days
A 22-person insurance-claims SaaS vendor is two days from a scheduled release. The QA lead has executed 640 of 700 planned test cases, and no new defect has been logged in 72 hours. The product owner wants a go/no-go today because a customer demonstration is booked. Four complicating facts sit on the table. First, the tester who owns the policy-calculation module is on leave, leaving 48 cases in that module unexecuted. Second, two previously reported defects in the same module have been fixed but not retested. Third, the test environment holds a copy of production data that is 11 months old, so no policy issued in the last 11 months exists in test. Fourth, the module is the one that computes every premium the company bills. The CTO summarises the situation as 'zero open defects, so we are good to ship', and asks the QA lead to sign the release note.
1. Identify which fundamental testing principles the CTO's reasoning misreads, and state the practical consequence of each misreading for this release.
Read the worked answer
1. Testing shows the presence of defects, not their absence. A 72-hour quiet period is not evidence of quality; it is evidence that nobody executed a test capable of failing. With 60 cases unexecuted and the highest-risk module untouched, the sample is biased toward easy paths.
2. Exhaustive testing is impossible, so risk must drive the choice. The team has finite time and has spent it on 640 cases of unequal value while 48 cases in a money-critical module sit idle. The right question is not 'how many cases passed' but 'which risks did we retire'.
3. Defect clustering (and the pesticide paradox). The policy-calculation module has already produced two defects. Clusters predict more defects in the same code path, so this is precisely where more testing should go, not less. Repeatedly running the same passing tests elsewhere stops finding anything new.
4. A fix without a retest is a hypothesis, not a fix. The two closed defects have unknown status; closing on 'developer says done' inflates the zero-open count.
5. The absence-of-errors fallacy. Even a defect-free build can be unusable. Eleven-month-old data means recent policy vintages, rate tables and endorsements have never been exercised, so 'no defects' is measured against an unrepresentative world.
2. Draft the go/no-go recommendation you would give, including the conditions and the evidence you would require.
Read the worked answer
Recommendation: conditional no-go for production; conditional go for the demo environment, with the decision revisited when the conditions below are met.
Conditions (with owners and a deadline of end of tomorrow):
- Close the calculation gap. Recall the module owner or assign a cross-trained tester. Execute the 48 cases, prioritised: premium calculation, rounding, tax, proration and endorsement handling first. If all 48 cannot run, run the top 15 risk-selected cases plus a two-hour time-boxed exploratory session on the module, and record exactly what was and was not covered.
- Retest both fixed defects and add a regression case for each. Check the neighbouring code path, because fixes in this module have twice introduced follow-on risk.
- Address the data gap. Either refresh or synthesise policies from the last three months, or state explicitly in the release record that post-cutoff vintages are untested and arrange a production-like verification with one recent policy before general availability.
- Publish exit criteria in writing before the meeting, for example: 100% of priority-1 cases executed; zero open critical or high defects; all fixed defects retested and passed; every blocked case listed with a reason. The current criterion ('zero open defects') is not an exit criterion, it is a side effect of not looking.
Evidence pack for the sign-off: executed/unexecuted counts per module, the blocked and not-run lists, retest results, the residual-risk statement, and the named person accepting each residual risk. Sign-off should record risk accepted, not quality asserted.
Why not a flat yes: the demo carries reputational risk only; production carries financial and regulatory risk on every invoice. Those are different decisions and deserve different evidence.