Helios Retail: high deployment frequency, rising failure rate
Helios Retail moved to continuous delivery eight months ago. Deployment frequency is now 40 per week, but change failure rate rose from 8% to 19% and mean time to restore is three hours. Feature flags are used widely, but nobody knows how many remain active and some are two years old. Canary releases send 5% of traffic for ten minutes with no automated rollback, and the on-call engineer rolls back manually when customers complain. Many teams bypass the pipeline with manual hotfixes. The release manager wants the change advisory board to review every change again. The CTO worries that slowing down will lose the competitive gains that justified the programme, and asks for an independent assessment before deciding.
1. Is reinstating CAB review for all changes the right response to the rising change failure rate? Justify your answer.
Read the worked answer
No. Reinstating universal CAB review would slow delivery without addressing the causes visible in the numbers. Change failure rate more than doubled while deployment frequency rose, and the stale two-year-old flags mean code that was never released is silently changing behaviour. The canary is too small and too short, and there is no automated rollback, so failures are found by customers. Those are engineering and control-design problems, not approval problems. High-velocity IT separates deployment from release: deploying to production is a technical event, exposing a feature is a business decision controlled by flags and progressive delivery. Standard changes should be pre-authorized with automated evidence - tests, security scan, rollback ready - with the CAB reserved for high-risk change. The manual hotfixes also signal that the normal path is too slow or unreliable; tightening approval without fixing the path drives work further underground.
2. Propose a 90-day plan with concrete measures of success.
Read the worked answer
Days 0-30: instrument and classify. Confirm the four flow metrics, and categorize the last 30 failures by cause - test gaps, flags, configuration, dependencies, environment. Days 30-60: automate rollback, and run progressive delivery with SLO-gated canary steps (for example 1% for 15 minutes, then 10%, then 50%) that abort automatically. Inventory every flag, remove or expire stale ones, and add tests where failures cluster. Publish a standard change model with automated checks. Days 60-90: run blameless reviews of the top three failure causes, share the results, widen automated controls, and shrink the CAB's remit to high-risk changes. Measures: change failure rate back below 10%, MTTR under one hour, share of changes handled as standard, active stale flags near zero, and canary detection rate. Hold deployment frequency steady - losing it would hide the problem rather than solve it.