Thirty-Seven Percent Complete and Completely Off Track
Vantage Retail's release 8.3 regression phase was planned as 400 test cases over 10 working days with five testers. One tester is available at 50% because of support duties, so the plan assumes 4.5 full-time equivalents, or 45 tester-days. At the end of day 4 the test manager reports '150 of 400 executed, 37.5% complete, on track'. The detail behind that figure: 72 passed, 48 failed, 30 blocked waiting for a fixed build of the payment gateway. Twenty-five defects are open, nine of them critical; the historical average fix-and-retest turnaround is 2.5 days, and critical defects have historically clustered in the pricing and checkout modules. Sixty of the remaining cases depend on a third-party sandbox that the vendor has confirmed is available only on days 8 and 9. Two testers have been asked to join a parallel hotfix verification for three days next week. The steering committee meets in 48 hours and wants a go/no-go date.
1. Test the claim 'on track'. Show the arithmetic and name the metrics that make the claim misleading.
Read the worked answer
The arithmetic. 150 cases in 4 days is 37.5 cases per day across 4.5 FTE, or about 8.3 cases per tester-day. Capacity remaining: 6 days x 4.5 FTE = 27 tester-days. Remove the hotfix support (2 testers x 3 days = 6 tester-days), leaving 21 tester-days. At 8.3 cases per tester-day that is roughly 174 cases, not the 250 still outstanding.
Then add work the 400-case plan understated:
- 48 failed cases must be re-executed after fixes.
- 25 open defects need retesting, several of them twice given a 2.5-day turnaround (call it 8 to 10 tester-days including triage time).
Subtract those and the executable capacity covers perhaps 120 to 140 of the remaining 250 cases by day 10. Realistic completion is around 65 to 70% of the planned suite, not 100%.
Why '37.5% complete' misleads.
- It counts execution, not outcome. Pass rate is 72 of 150, or 48%. A suite with a 32% failure rate is not 'progressing' in the ordinary sense; it is generating rework faster than it is consuming test cases.
- It ignores blocked work. 30 blocked cases, 20% of everything executed, mean the environment is not delivering.
- It is not weighted by risk. The nine critical defects cluster in pricing and checkout, which are precisely the modules where unexecuted cases carry the most risk.
- It treats all cases as equal. Executing 150 low-risk cases tells you nothing about whether the pricing module is safe to ship.
The claim is arithmetically defensible only if the remaining work is identical in size and risk to the work already done, and if no retesting is needed. Neither is true.
2. Produce a re-plan with concrete numbers: what can be executed, what must be dropped or deferred, and the options you would put to the steering committee.
Read the worked answer
Sequencing. Re-order the remaining 250 cases by risk, not by ID. Pricing and checkout first, because that is where critical defects cluster. The 60 sandbox-dependent cases are fixed in time to days 8 and 9, so their scripts and data must be prepared by day 7 so that the vendor window is not lost to setup.
Capacity allocation over days 5 to 10 (21 tester-days after hotfix support).
- Retesting the 25 open defects and re-running the 48 failures: about 9 tester-days.
- New execution: about 12 tester-days, or roughly 100 cases.
- Total reachable by day 10: 150 already executed plus about 100, so 250 of 400, around 63%.
What to drop. Defer 80 to 100 low-risk cases (reporting, cosmetic configuration, rarely used admin screens) to post-release or to an automated smoke pack. That closes most of the gap without pretending capacity exists.
Options for the committee.
- A - Hold the date, reduce scope: 100% of critical and high-risk modules, about 63% overall, with the deferred list documented. Requires zero open critical defects and a 5-day post-release test window.
- B - Hold the date, add capacity: two additional testers for 5 days to recover roughly 80 cases; costs onboarding time and still leaves the sandbox constraint untouched.
- C - Delay go-live by 5 days: completes the full suite but moves revenue and marketing commitments.
My recommendation is A with conditions: no open critical defects, all pricing and checkout cases executed, sandbox cases completed on days 8 and 9, and hotfix verification resourced separately so it does not consume the regression capacity twice.
3. Which monitoring metrics would you report from now on, and how would you prevent the same misreporting?
Read the worked answer
Report three dimensions, never one number.
Progress: test cases executed, passed, failed, and blocked reported separately; tester-days consumed against tester-days remaining; schedule variance expressed in tester-days rather than percentages.
Quality: defect detection rate per day, open and closed trends, defect severity distribution, fix-and-retest turnaround. A rising detection rate late in the phase is a warning, not a success.
Risk coverage: cases executed as a fraction of cases planned in each risk area, with pricing and checkout shown explicitly. This is the metric that actually answers 'can we ship?'.
Preventing the misreport. The failure was not arithmetic, it was definition: 'complete' meant 'executed'. I would publish a one-page reporting standard with fixed definitions - executed means run at least once with a recorded result; passed excludes anything later invalidated; blocked is its own category and never counted as progress - and state the estimate basis alongside every status (cases per tester-day at the current pass rate, plus retest load).
Cadence. Daily burndown in tester-days, weekly re-forecast compared against the previous forecast so drift is visible. If the forecast moves more than 10% between reports, the report says so explicitly.
The test to apply. If a status line could still read 'on track' while the pass rate is 48% and 20% of executed cases are blocked, the metric is wrong. I would add one line to every report: 'Forecast completion at current rate: X of Y cases by day 10', and let the committee see the gap rather than having to derive it.