Seventy-Eight Percent and Two Days to Go
Release 4.2 of a banking mobile app ships in two days. Forty-seven defects are open. Triage has: D-201, a 0.01 rounding error in the displayed interest figure, severity Major, priority 1 - raised by the head of retail banking; it only affects accounts in a three-decimal currency, 0.3% of accounts. D-388, a crash when the screen is rotated during a fund transfer, severity Critical, priority 3, marked 'defer' - it reproduces on 2 of 14 supported Android devices, the two oldest, about 6% of the user base. D-402, the 'Remember me' checkbox does not persist, severity Minor, priority 2. The test summary report shows 92% of planned cases executed, 78% passed, defect detection in system test up 30% on release 4.1, and 11 defects still 'Open - awaiting retest'. The release manager says the pass rate is fine.
1. Critique the three severity and priority assignments. What do you recommend for each, and what are you not entitled to decide?
Read the worked answer
Severity is the degree of impact on the system; priority is the urgency of fixing it, driven by business value. They are assigned by different people, and the inconsistency between them is itself a finding.
D-201 - Major/P1. Priority 1 is a business decision and the head of retail banking is entitled to make it; the cost of an executive escalation is real. But the severity is overstated at Major for 0.3% of accounts, and a financial rounding error deserves one extra question before anyone argues: does the error compound? A one-cent display error that is recalculated each period is trivial; a one-cent error posted to the ledger and carried forward is a reconciliation defect with audit consequences. That question changes the answer, and it should be asked in triage rather than assumed.
D-388 - Critical/P3 'defer'. This is the serious inconsistency. A crash mid-transfer can leave a transaction in an unknown state - money debited, transfer not completed - which is a data-integrity failure, not merely an availability failure. 6% of the user base is several thousand people attempting the app's most sensitive operation. Either the severity is wrong or the priority is. My recommendation: escalate to P1/P2 with a decision recorded, or, if genuinely deferred, require a documented workaround (lock orientation on the transfer screen) shipped in this release and a signed risk acceptance naming who accepted it.
D-402 - Minor/P2. Coherent; deferring it is defensible.
What I am not entitled to decide. I can raise severity, evidence and the risk consequence. I cannot unilaterally upgrade a priority to override a business owner, and I cannot defer a critical defect on my own authority. The output of triage is a recorded decision with an owner, not a tester's preference. The absent voice in this meeting is the product owner.
2. Interpret the report's metrics. What do 92% executed, 78% passed and a 30% rise in detection actually tell you - and what is missing?
Read the worked answer
What the numbers say, and do not say
- 78% passed means roughly 14% failed - about 13 cases, presumably the source of the 47 open defects. But the aggregate says nothing about which requirements failed. An 85% pass rate concentrated in the four regulatory-critical modules is far worse than 60% in cosmetic settings.
- 8% not executed is not neutral. Unexecuted cases are unknown risk, and if they are the hard ones - integration, non-functional, the newest features - the effective coverage is far below 92%.
- 'Defect detection up 30%' is genuinely ambiguous. It can mean testing got better, the build got worse, or more of the product was exercised for the first time. The diagnostic is the shape of the curve: detection should rise then flatten as a module matures. If it is still climbing steeply in the final days, testing has not converged and there is more to find.
- 11 'Open - awaiting retest' is a process alarm. Fixes with unknown status mean the defect lifecycle is not closed; the true open count is 47 plus an unknown number of unverified fixes.
What is missing from the report
- Requirement coverage against the risk-based priorities, not the raw case count.
- Severity distribution of the open defects, and the count of open Critical/High with no accepted workaround.
- Defect Detection Percentage: defects found in test ÷ (found in test + found in production) for 4.1, so the 30% rise can be interpreted.
- Defect density by module, to expose clusters and where to look next.
- Status of regression, performance and security results, and the exit criteria table.
A pass rate on its own is a number that cannot be acted on.
3. What is your release recommendation, and what evidence would you require before it becomes a 'yes'?
Read the worked answer
Recommendation: not releasable as things stand - and the reason is the state of the evidence, not the 78%.
What I would say to the release meeting
- The 11 'awaiting retest' must be closed out. Fixes verified is the difference between a defect list and a quality position; until then nobody knows whether the 47 is falling or static.
- Zero open Critical or High-severity defects, unless each has a written workaround and a named acceptor. D-388 is the immediate candidate: fix it, or ship the orientation lock and record the risk.
- Requirement coverage against the risk-based test plan - a statement that every High-risk requirement has been tested and passed, with the residual risks listed.
- An explanation of the 8% unexecuted, case by case: deferred by risk (acceptable, if recorded) or simply not done (not acceptable).
- A closure decision on D-201 after the compounding question is answered.
What would change my answer to yes
Exactly those five things, if the outcome supports it. Notice that none of them is 'more testing days' by default - a release decision is a judgement about known and accepted risk, so a defensible 'yes, ship' can be built on a complete picture even with defects open. What cannot be defended is a 'yes' built on a single aggregate percentage, because it leaves the organisation unable to say afterwards what it knew.
Also worth recording for the next release: the release manager treated a pass rate as a quality measure, and triage met without the product owner. Both are process defects that will produce this same meeting again.