Monthly mega-release at an online retailer: changing cadence without changing risk
A UK online retailer ships one release per month to its e-commerce platform. Each release bundles 60-90 changes into a single release unit: 12 teams commit to one monolithic application and a shared Oracle database. Last quarter 38% of releases caused a severity-1 or severity-2 incident, and mean time to restore was 4 hours 20 minutes because rollback means restoring a database backup. Developers now want continuous deployment. The full regression suite takes nine hours, so it runs once, overnight, before the release. The CTO wants releases twice a week by next quarter. Marketing has a contractual availability obligation for a three-day Black Friday event 11 weeks away, with a code freeze starting 14 days before. Two teams already use feature toggles in a limited way, but the toggle service is not used by the checkout team, which owns the highest-risk code. The release manager must propose a route forward that the CTO and the risk committee will both accept.
1. The CTO wants twice-weekly releases within a quarter. What should the release manager change first, and why does the order matter?
Read the worked answer
The first move is to separate deployment (moving an artifact into production) from release (making it available to users). Twice-weekly deployment is feasible in a quarter; twice-weekly release of a 60-90 change bundle to all customers is not, given a nine-hour regression run and rollback that depends on a database restore.
Order matters because cadence multiplies whatever failure rate already exists. At 38%, moving to twice weekly without fixing recovery would produce roughly eight to ten severe incidents a month instead of one or two.
Sequence:
- Get the checkout team onto toggles so code can deploy dark and be activated separately. Until the highest-risk component is covered, faster deployment has no safety benefit.
- Adopt expand/contract database migrations so schema change stops being a rollback blocker; the 4h20 MTTR is mostly a database problem, not a code problem.
- Split the release unit: 60-90 changes in one unit destroys fault isolation and makes the change failure rate unmeasurable per change.
- Automate a critical-path test subset on every deploy, keep the full suite nightly.
Release authorization remains for the business-facing activation decision, which is where Black Friday risk actually sits.
2. What should the release manager do specifically to protect the Black Friday window, and what should not be changed?
Read the worked answer
Protect the window with a defined freeze from 14 days out covering functional and cosmetic change, but keep an open, pre-agreed route for security patches and infrastructure fixes - a blanket freeze that has no exception path is usually breached informally, which is worse than a narrow one.
Before the freeze: require a release readiness review per release rather than one giant gate; rehearse rollback end to end and time it; ensure every toggle shipped has a documented default of off for anything not needed for trading; pre-scale and load-test with the actual Black Friday configuration.
The event must not be the pilot for twice-weekly cadence. Set a hard date after which cadence experiments stop - enough time to revert to monthly if a phased deployment misbehaves.
What should not change: the change authority for database migrations. Those are the changes that produce the four-hour outages, and they should stay individually reviewed even while code deployments become routine. Nor should the emergency route be suspended; security exposure does not pause for peak trading.
3. Which measures should the release manager report to show the change is working, and what would a naive metric hide?
Read the worked answer
Report change failure rate per deployment and per release activation separately. Once toggles are in place these diverge: code may deploy cleanly 40 times while an activation still breaks checkout. A single blended failure rate would hide that.
Also track: mean time to restore (target is driven by toggle-off time, not restore-from-backup), deployment frequency and lead time, percentage of changes that can be rolled back in under five minutes, toggle inventory with planned expiry dates, and the count of database migrations rolled back versus rolled forward.
A naive metric would be 'number of releases per month' as a proxy for agility. Shrinking the release unit from 60-90 changes to, say, 8 raises that number without any improvement in outcomes, and could even raise the headline incident count simply because there are more events to count. Report incidents per 100 changes and per thousand transactions, not per release, to keep the comparison honest across a changing cadence.