Faster Tests, More Failures: Was This an Improvement?
A telecoms operator's engineering group reduced automated test execution time by 40% by parallelising its regression suite. Over the following quarter, deployment frequency rose from 20 to 45 per month, but change failure rate rose from 8% to 15% and mean time to restore service rose from 2 to 5 hours. The continual improvement register entry that authorised the work recorded its expected outcome as 'reduce test execution time from 90 to 50 minutes'; it was closed as successful and the team was recognised at the quarterly town hall. The engineering dashboard shows test execution time, number of tests, and pipeline runs, but not failure or restore data. A product owner now argues for more of the same: 'faster pipeline, more deployments, that is what improvement looks like.' A reliability engineer has raised concerns but has been told the data 'is not comparable' because release volume changed.
1. Was the test parallelisation an improvement? Justify your answer using ITIL 4 continual improvement concepts.
Read the worked answer
Not demonstrably, and probably not. ITIL 4 treats improvement as a deliberate, measurable change that adds value for stakeholders; the test of improvement is the outcome, not the speed of an internal activity. Test execution time is an output. The outcome set moved the wrong way: change failure rate nearly doubled and restoration time more than doubled, both of which customers experience directly. The register entry defined success as 90 to 50 minutes, so the team did exactly what it was asked and closed the item as successful - but the criterion was wrong.
The reasoning must also allow for confounders: deployment frequency more than doubled, so the change population may be different, and parallelisation may have reduced effective test coverage or increased flakiness. A fair judgement requires a short analysis of failure causes and test coverage before and after. My conclusion: this is an output optimisation with adverse outcome effects, not an improvement; at best it is a partial change that needs remediation. Recognising it publicly as a learning, not a failure, is part of keeping momentum.
2. What should the team change in how continual improvement register entries are defined and closed?
Read the worked answer
Every register entry should start from a problem or opportunity and a measurable hypothesis, not an activity. Required fields: baseline (current failure rate, restoration time, lead time), desired outcome and target, leading and lagging indicators, guardrail measures that must not degrade (change failure rate, coverage, MTTR), expected benefit and cost, owner, stakeholders, evaluation window, and explicit assumptions. Closure should require an outcome review, not delivery of the task: 'did we get there?' answered with data against the baseline, including unintended effects, and a decision to keep, adjust or revert. A 30/60/90-day review works for changes whose effects take time to appear.
The practice also needs a norm that closing an item as 'not proven' or 'reverted' is a legitimate, valued result. Otherwise teams will keep choosing activity-based success criteria. The CI manager should sample closed items each quarter and check that outcomes were actually measured, and the review board should ask for the guardrail data as a matter of routine.
3. How would you use the ITIL 4 guiding principles to respond to the product owner?
Read the worked answer
Use the principles to redirect rather than stop the work. Focus on value: agree with the product owner that customers value both speed and reliability, and that a faster pipeline which produces more failures is not value. Collaborate and promote visibility: put change failure rate and MTTR on the same dashboard as test execution time, so the trade-off is visible to everyone rather than argued about. Progress iteratively with feedback: keep the parallelisation, add quality gates and a canary or rollback step, and review after one quarter.
Start where you are: the data exists; the argument that it is 'not comparable' should be tested by segmenting by release size and type, not used to dismiss it. Optimize and automate: automate the checks that catch the failures, not just the tests that run fastest; if coverage fell, restore it before increasing volume further. Reframe the next register entry as 'reduce lead time without increasing failure rate above 9%', with both measures in the success criteria, so the product owner's ambition is preserved and the reliability engineer's concern becomes a guardrail rather than an objection.