Defending a Continuous Testing Budget With the Wrong Numbers
Northwind Logistics has run a continuous testing programme for two years and must now defend its budget. The QA director reports three headline numbers to the CIO: automated regression coverage rose from 18% to 64%, the pipeline executes 1,900 automated checks per merge, and the test team grew from 12 to 31 people. The CFO counters that production incidents rose from 14 to 19 per quarter and that mean time to restore (MTTR) is unchanged at 4.5 hours. Release frequency improved from monthly to fortnightly, but only in two of the five product lines. Developers describe the pipeline as 'a wall we wait on'; testers say they are still finding the same integration defects they were finding in 2022. Budget decisions for next year are due in three weeks, and the CFO has proposed cutting the test team back to 20.
1. Classify the QA director's three headline measures and explain which outcome measures should replace or supplement them.
Read the worked answer
Classification. Coverage growth (18% to 64%) is an output measure: it counts artefacts, not effects. '1,900 checks per merge' is a volume measure and invites exactly the wrong behaviour, since the cheapest way to raise it is to add more checks of the same kind. Headcount is an input. None of the three says whether the organisation detects problems earlier or recovers faster, which is why the CFO's counter-metrics dominate the argument: outcome measures beat output measures in any budget conversation.
Replacements. First, defect escape rate by severity, normalised per 100 changes so it survives the release-frequency change. Second, change failure rate plus MTTR, split by whether the failure was caught by an automated check or by a customer; that split is what shows whether testing is buying detection or merely activity. Third, lead time for changes reported per product line, because the aggregate hides that three of five lines did not improve. Fourth, 90th-percentile test feedback time, since 'a wall we wait on' is a real cost even when every test passes.
Caveat before conceding anything. Incidents rising from 14 to 19 is ambiguous without exposure data. If releases roughly doubled in frequency, the per-release failure rate may have fallen. Normalise before accepting the CFO's framing.
2. The CFO proposes cutting the team to 20. State your recommendation and the reasoning that supports it.
Read the worked answer
Do not defend 31 as a number. The CFO is right that cost rose; the argument to win is about what the spend buys, not about headcount as an end in itself.
Decompose the 31 roles. If roughly a dozen people spend most of their day executing and re-executing manual regression, that headcount is a symptom of missing automation, not a source of safety. Cutting to 20 while leaving the portfolio unchanged moves the bottleneck rather than removing it, and the first thing to go will be the exploratory and environment work that actually prevents escapes.
Use the two-product-line fact. Three of five lines did not improve release frequency. That is where marginal spend is either wasted or blocked by something other than test capacity: environment contention, approval gates, or test data provisioning. Fund a two-week diagnosis there before cutting or adding.
The recurring integration defects are the strongest evidence. The same class of defect appearing as in 2022 points to absent contract or environment-level checks, not to insufficient tester effort.
Counter-proposal. Hold headcount flat for twelve months, move eight roles into automation, test data and environment engineering, and commit publicly to two outcome targets reviewed quarterly: defect escape rate and 90th-percentile feedback time. Accept a reduction to 24 if both targets are missed in two consecutive quarters.
3. The CIO asks for one change to make in the next quarter. What do you choose, and what do you deliberately postpone?
Read the worked answer
Choose: stop counting and start measuring the escape path. Instrument the last twelve months of released defects with three attributes: severity, which automated layer could plausibly have caught it, and how long the check would have taken to run. This is cheap, takes about three weeks, and converts the entire budget debate from opinion into evidence. It also tells you which of the 1,900 checks are load-bearing and which are theatre.
Why this over automation expansion. The team already has 64% coverage and 31 people; the binding constraint is that nobody knows which gap matters. Adding tests before knowing that is how coverage rose while incidents did not fall. The analysis also produces the escape-rate baseline any future target needs.
Deliberately postpone. Do not restructure the team, buy new tooling, or rewrite the flakiest suites this quarter. Each of those is expensive, visible, and hard to reverse, and all three become better-targeted once the escape data exists. Do not chase the MTTR number either; restoring service faster is largely an observability and rollback problem, not a testing one, and claiming it as a test-team outcome will be rebutted.
Deliverable at quarter end. A one-page map from escaped defect classes to the missing checks, with the cost of adding each, so the next budget conversation is about specific purchases rather than headcount.