Monthly Release Train at Northwind Retail
Northwind Retail runs a 1.2M-line Java storefront with 11 teams. Releases go through a monthly train: code freeze on day 1, 40 manual regression steps, a change advisory board on day 3, and a production deploy window on day 5. Median lead time from commit to production is 21 days. Deployment frequency is 12 per year, change failure rate 18%, and MTTR after a failed deploy averages 9 hours because rollback requires restoring an Oracle schema from the previous nightly backup. Product owners bundle 30-40 changes per release, so when checkout errors spike the team spends the first four hours diffing release notes. The VP has approved budget for a new deployment orchestration tool and wants it live next quarter. Meanwhile the on-call engineer reports that 60% of incidents trace to configuration drift between staging and production, and nobody can reproduce staging failures locally. Two teams have started arguing about whose changes caused the last two rollbacks.
1. Where is the binding constraint, and what should be fixed first? Justify your sequencing using the numbers given.
Read the worked answer
The metrics point at batch size and rollback capability, not at deployment speed.
- Lead time of 21 days with a 5-day gate means most of the delay is waiting, not working. Bundling 30-40 changes multiplies the search space when something breaks: 40 changes means roughly 40 suspects, and the team burns four hours just attributing failure.
- 18% change failure rate and 9-hour MTTR are coupled. With 12 deploys a year, two of them fail on average, and each failure costs most of a day because rollback is a database restore rather than a redeploy.
- 60% config-drift incidents say the artifact being tested is not the artifact being shipped, so the regression suite gives false confidence.
Sequence:
- Make deploys reversible before making them frequent. Separate schema changes from code (expand/contract migrations), so rollback is a previous-image redeploy, not a restore. This alone attacks the 9-hour MTTR.
- Shrink batch size. Move to weekly or per-service releases with feature flags so deploy and release are decoupled; 40 changes per release is the root cause of slow diagnosis.
- Make environments reproducible (same image, config as code, no hand-edited staging values), which addresses the 60% drift figure.
- Automate the 40 manual regression steps last, not first: automating a gate that should not exist wastes effort.
Fixing orchestration while batches stay at 40 changes automates the waste and leaves CFR and MTTR where they are.
2. The VP wants to buy deployment orchestration. How should you respond, and what would you measure over the next 90 days to prove progress?
Read the worked answer
Respond with respect for the intent and a concrete counter-proposal, not a flat no.
Framing: a deployment orchestrator moves artifacts to servers. Northwind's problem is that the artifact is untested against production-like config, batches are too large to diagnose, and rollback is a database restore. Orchestration would shave minutes off a process whose delay is measured in days. It would also add a tool to operate while the underlying constraint remains.
Proposal: defer the purchase for one quarter and run a bounded experiment on one service (checkout is highest value, but pick a lower-risk service first if checkout is too volatile):
- trunk-based development with short-lived branches;
- same container image promoted through environments, config externalized;
- weekly deploy behind feature flags;
- automated rollback to the previous image, with schema migrations decoupled.
Measure four things, plus counter-metrics:
- lead time for changes (target: 21 days to under 5);
- deployment frequency (12/year to weekly for the pilot service);
- change failure rate (18% to under 10%);
- MTTR (9 hours to under 1 hour).
- Counter-metrics: batch size per release, percentage of changes released via flag, and rollback time measured in a game day, not estimated.
If the pilot hits these numbers, the orchestration tool becomes a justified scaling step rather than a substitute for process change. If it does not, the VP has avoided a quarter-long tooling detour.
3. Which DORA metric is most at risk of being gamed in this organization, and what counter-metric would you pair with it?
Read the worked answer
Deployment frequency is the easiest to inflate here, because the organization already has a release train and a compliance narrative that reward visible activity.
Ways it gets gamed:
- counting redeploys, config pushes, or empty releases as deployments;
- splitting one release into five deploy events for the same 40-change batch;
- deploying to production but keeping everything dark behind a permanent flag so nothing is really released.
Pair it with:
- percentage of deploys carrying a customer-visible change (or a validated feature flag flip);
- batch size (changes per deploy) - frequency without smaller batches is theater;
- change failure rate, but defined by an external incident record, not by the team's own judgement, so hotfixes and rollbacks count against it;
- mean time to detect, which prevents MTTR being gamed by closing tickets early.
Change failure rate also has a known loophole: reclassifying failures as "known issues" or shipping a fix as a normal deploy. Audit against incident tickets rather than deploy records. MTTR is gamed by redefining when the clock starts (from detection instead of from deploy) or by declaring an incident resolved when service is restored but the root cause is untouched - pair it with reopen rate.
The general rule: every speed metric needs a stability counterpart, and both should be derived from systems the team does not solely control.