Monthly Release Window, 38% Change Failure Rate, and the DBA Shadow Process
Northwind Retail runs 1,200 stores on a monolithic order platform. Releases are batched into a monthly window: code freeze on the 1st, deployment on the 14th, and every change - including a one-line config tweak - must pass the Change Advisory Board, which meets weekly and spends about 40 minutes per release. Over the last two quarters delivery lead time has grown to 21 days and change failure rate sits at 38%. Roughly a third of failures trace to configuration differences between staging and production. When a bad release took checkout down for 90 minutes last month, the incident review concluded that "the platform team should test more" and closed with two action items assigned to individuals. Meanwhile developers maintain a shadow list of hotfixes applied directly by DBAs, because waiting three weeks for approval blocks their sprint. Attrition on the platform team is up, and security has flagged that two hotfixes bypassed review entirely.
1. Leadership asks which delivery metric to attack first. Using only the evidence in this setup, argue for a primary metric and explain what would improve as a consequence versus what would not.
Read the worked answer
The tempting answer is change failure rate, but the setup points one level down: batch size. Twelve releases' worth of change is compressed into one deployment every 30 days, and a 40-minute-per-release CAB review means change volume is being absorbed by human review rather than by automated verification. That is why lead time is 21 days and why a single bad batch takes down checkout for 90 minutes.
Reasoning chain:
- Deployment frequency is the lever. One deploy per month is the constraint; the CAB queue is its cause.
- Lead time improves automatically once changes stop waiting for a weekly meeting, because the waiting time, not the coding or review time, dominates the 21 days.
- Change failure rate improves as a consequence of smaller batches: fewer changes per deploy means each failure has a smaller blast radius and is easier to bisect. Adding more testing to a 30-day batch preserves the batch size and therefore preserves the failure rate.
- Time to restore is already reasonable (90 minutes) and is not the pain point described.
So: primary metric is deployment frequency (with lead time as the paired check), and the first experiment is removing CAB review for changes that pass an automated pipeline and are reversible. Measure failure rate before and after; if it rises, the pipeline's verification is the thing to fix, not the batch size.
2. The incident review produced "test more" and two individually-owned action items. Rewrite the outcome as a blameless, systemic set of actions, and explain what makes your version more likely to actually close.
Read the worked answer
"Test more" is not an action because it names no mechanism and no owner of a mechanism. Assigned-to-an-individual items decay when that person changes teams. Replace both with changes to the system that produced the outcome.
Reframe the finding. The 90-minute outage was not caused by a careless engineer; it was caused by a process where a staging/production configuration difference can reach production undetected. The question becomes: what allowed that class of difference to exist?
Systemic actions, each with an observable completion signal:
- Configuration rendered from a single template per environment so staging and production cannot diverge by hand. Done when a diff of the two rendered configs is empty except for a documented allowlist.
- The pipeline runs a pre-deploy config-diff check against production and fails on unexpected differences. Done when a seeded discrepancy blocks the pipeline.
- Rollback rehearsed within the last 30 days and timed. Done when time-to-restore is recorded, not estimated.
- Blast radius bounded by progressive rollout so a bad batch affects a fraction of stores first. Done when a canary stage exists.
- The review itself records contributing factors and detection gap, not a person.
Why this closes: each item is verifiable by a machine or a rehearsal, is owned by a team rather than an individual, and removes the conditions that made the failure possible instead of asking humans to be more careful.
3. The DBA hotfix path bypasses review. Explain why simply banning it would fail, and propose a sequence that removes the need for it.
Read the worked answer
The shadow process exists because it works: the official path costs three weeks, and the DBA's problem is real and urgent. Banning it removes the fix without removing the need, so it will reappear as a differently shaped exception, or the DBA will simply stop telling anyone - which is worse, because security's current visibility into two bypasses becomes zero.
Sequence, ordered by what creates the most safety soonest:
- Make the bypass visible before making it forbidden. Log all direct production changes centrally and review them daily. You cannot manage what you cannot see, and the current two-flag situation is a visibility failure, not a discipline failure.
- Give the hotfix path a real, fast lane. A documented emergency change class: one approver, a time limit, automatic ticket creation, and mandatory retrospective within 48 hours. This converts an unmanaged risk into a managed one and buys credibility for the next steps.
- Reduce the need. Identify the top three change types the DBAs are hotfixing. Each one becomes a candidate for a pre-approved, licensed self-service action - a reviewed, versioned script or pipeline job the DBA can run without a CAB.
- Shrink the CAB's scope. Move low-risk reversible changes (config values, feature flags) out of CAB entirely and into pipeline-gated deploys. A CAB that reviews everything reviews nothing well.
- Track the ratio. Unlicensed changes should trend toward zero as licensed fast paths grow. If it does not, the fast lane is still too slow, and the metric tells you that rather than blaming the DBA.