The Atlas Billing Report That Says On Track
You take over the Atlas billing release three weeks before go-live. The weekly status report shows 62% of 1,450 planned test cases executed, a 41% pass rate, 130 open defects of which 22 are blockers, and an overall status of 'on track'. Digging into the data, you find that blocked tests are counted as not executed; 180 of the executed tests were run against build 4.2 while the current build is 4.7; 48 defects are more than 30 days old with no assigned owner; the test environment was unavailable for 6 of the last 15 working days; and the defect arrival rate over the last two weeks is 14 per day while the fix-and-verify rate is 9 per day. The business sponsor is under pressure to hit the quarter-end date and the steering committee meets in two days.
1. Recalculate the status and produce a defensible forecast for the remaining three weeks, stating your assumptions.
Read the worked answer
First separate what has been proven from what has been counted. 62% of 1,450 is 899 executed - but 180 of those were against build 4.2 and must be re-executed against 4.7 (at minimum the high-risk ones, since regression risk is unknown). Add the blocked tests, which the current report hides inside 'not executed': those are not progress, they are deferred work whose size must be counted now.
Defensible restatement: roughly 719 executed tests are valid against the current build (about 50% of plan, not 62%), with an unknown but material blocked backlog. A 41% pass rate on a mature build is abnormal and suggests either a genuine quality problem or that tests are being attempted before entry criteria are met - check both before quoting the number.
Forecast using the flow data, not the percentage. Defect arrival is 14 per day and fix-and-verify is 9 per day, so the open backlog grows by about 5 per day; over 15 remaining working days that is roughly 205 additions against a 195 reduction capacity, with 130 already open. Even a perfect environment makes net reduction nil. Add 40% environment unavailability (6 of 15 days lost) and 22 blockers that must be closed and re-tested, and the release date is not achievable for the current scope.
State assumptions explicitly: environment availability improves to at least 90%; no new scope enters; test data for the billing scenarios is available; and defect triage assigns owners within 24 hours. Then present the forecast as a range - for example, best case 2-3 weeks late with reduced scope, worst case 5-6 weeks - rather than a single date.
2. What control actions do you take immediately, and what do you ask the steering committee to decide?
Read the worked answer
Control actions must be ordered by effect on the constraint.
- Fix the report definitions today. Blocked is not the same as not executed; executed against a superseded build does not count as coverage. Publish a one-page metric definition so the next status is comparable.
- Triage all 130 open defects within 48 hours, assigning owners and separating genuine blockers from defects that can be deferred with business agreement. The 48 unowned defects older than 30 days are a process failure, not a testing problem. Introduce short daily triage with development present.
- Attack the environment. Six lost days out of 15 is the single largest recoverable capacity. Escalate for a dedicated environment with an availability target and a named owner; investigate the cause (contention with other projects is typical) and ask for a booking model.
- Freeze scope and prioritise by risk. Re-order the remaining test execution by business risk so that if time runs out, what is untested is low-impact by design, and document that.
- Re-execute the 180 stale tests against 4.7, high-risk first.
Ask the steering committee for four decisions, not a status acknowledgement: approve or reject a reduced go-live scope; mandate environment priority; authorise additional development capacity for fixing (not additional testers - testers cannot outrun a fix rate of 9 per day); and agree the evidence-based go/no-go criteria in advance, for example zero open blockers, all high-risk scenarios executed on the release candidate, and regression green. Presenting criteria before the decision removes the pressure to reinterpret the numbers at the last minute.
3. The sponsor argues that the 62% figure proves the release is close and accuses you of moving the goalposts. How do you handle that conversation?
Read the worked answer
Do not contest the number - contest the meaning, with evidence the sponsor can verify independently. Show the build history: 180 tests executed on 4.2 cannot be evidence about 4.7. Show the blocked list: those tests were counted as pending but are unexecuted work. Show the arithmetic on defect flow: 14 arriving daily against 9 fixed and verified, which anyone can recompute from the defect tracker. The argument is not that the old report was dishonest but that its definitions cannot support a go/no-go decision.
Then reframe the decision in the sponsor's terms rather than the test manager's. The question is not whether 62% has been executed but which business capabilities are proven, which are unproven, and what the cost of being wrong is at quarter end. Offer a risk-tiered view: what is tested and stable, what is untested and high impact, what can go live behind a feature flag, what can be manually worked around for a few weeks.
Give the sponsor a choice they own rather than a verdict they receive: full scope with a later date, or full date with a defined reduced scope and a signed residual-risk statement. Document the options and the decision in the steering minutes. That protects the release, the sponsor and you, and it converts an argument about metrics into a decision about risk appetite - which is the sponsor's accountability, not yours.