A 94% Pass Rate on the Release That Cannot Slip
Caldera Software must ship a regulatory reporting module by 30 June, with penalties for late filing. The test manager presents the readiness dashboard at the steering committee: 94% pass rate, 88% requirement coverage, 12 open defects (2 critical, 4 high). On investigation, 1,100 of 4,000 tests were not executed this cycle because environments were unavailable, and the tool counts 'not run' as passed in the pass-rate calculation. Of the 88% coverage, 40% of requirements are mapped to a single end-to-end smoke test that asserts only that the page loads. Sixty-three defects were deferred during the cycle, and 19 of those have since been reopened. Triage consumes 90 minutes daily across 11 people, and 70% of the items discussed are duplicates or environment issues. The steering committee meets in two days and wants a go or no-go recommendation.
1. Identify the reporting defects in the dashboard and state what each one conceals.
Read the worked answer
Not-run counted as passed. This is the most serious defect. Roughly 27% of the planned suite was never executed, so the true execution rate is about 73%, and the pass rate should be computed against executed tests only. The 94% figure is arithmetically meaningless and would not survive an audit. What it conceals: coverage of the module is unknown, but it is materially worse than reported.
Requirement coverage by count. 88% sounds robust until you see that 40% of requirements map to a single smoke test asserting page load. A requirement is not covered by a test that cannot fail for the reason the requirement exists. What it conceals: the regulatory-critical requirements may have no substantive verification at all, and the dashboard gives no way to tell.
Open defects without age or trend. Twelve open defects, 2 critical and 4 high, is a snapshot. It hides whether defects are being found faster than they are fixed, and it hides the 63 deferrals.
Deferrals excluded from the headline. Nineteen of 63 reopened implies a 30% reopen rate, which points at triage quality rather than at release risk being under control. What it conceals: the deferred population is unstable and cannot be treated as closed risk.
Triage cost as a signal. Ninety minutes daily across 11 people with 70% noise is roughly 8 person-hours a day spent on duplicates and environment failures, which both explains the low execution rate and inflates the apparent diligence behind the numbers.
2. Give the steering committee a go or no-go recommendation with explicit conditions.
Read the worked answer
Recommendation: conditional go, with a decision point on 23 June. An unconditional go is indefensible on this evidence, and an unconditional no-go is likely worse: a late filing penalty is a certain loss, and the current evidence is too corrupted to prove the module is unsafe.
Conditions that must be met before filing. First, execute the 1,100 unrun tests, prioritised strictly by regulatory-critical requirement, and report pass rate on executed tests only. Second, every regulatory-critical requirement must have at least one substantive test, not a page-load smoke test; reclassify the 40% mapping and close the gaps. Third, zero open critical defects and a named owner with a dated fix plan for each high.
Compensating control if execution cannot finish. For a reporting module, a defensible fallback is to file with manual reconciliation of every reported figure against source data for the first two filing cycles, signed off by a named accountable owner. That converts an unknown technical risk into a known, costed manual effort, which is a decision the committee can actually take.
What to present in two days. Not a single percentage, but a residual-risk statement per regulatory requirement: verified, partially verified, or unverified, with the compensating control for each unverified item. Then let the committee choose between late filing and accepted manual risk. Do not let them choose based on 94%.
3. How should the 63 deferred defects be handled in the go or no-go decision?
Read the worked answer
Treat the deferral list as open risk, not closed work. The headline count of 12 open defects is misleading because deferral is only a scheduling decision, not a resolution. The 30% reopen rate confirms that a large share of deferred items were never properly understood.
Recut the list by regulatory impact, not by severity label. Severity was assigned before anyone knew which requirements were regulatory-critical. Reclassify each deferred defect against the filing obligations: any defect affecting a reported figure, a filing deadline, an audit trail or a data retention rule is effectively open regardless of its severity label or its deferral status.
Require explicit acceptance. Each remaining deferral needs a named acceptor, a rationale, a customer or compliance impact statement and an expiry date. Undated deferrals should be counted as open.
Report the deferral metrics to the committee. Deferred count, reopen rate and average age belong next to the pass rate, because they are the numbers that predict post-release incidents. A 30% reopen rate is a finding in its own right and should trigger a review of the triage process, alongside the duplicate and environment noise that is consuming the daily 90 minutes.
Use it as a leading indicator afterwards. Track whether defects deferred in this release reopen in the first 60 days post-filing; that measures the reliability of the deferral decision itself.