The Toolchain-First Transformation
A retail bank employs 900 engineers across 60 teams. Fourteen months into a 'DevOps programme' it has spent $4.1M on a CI/CD platform, Kubernetes and a commercial pipeline tool. Outcomes have barely moved: median lead time for change is 22 days, teams deploy to production roughly once every six weeks, change failure rate is 28%, and mean time to restore is nine hours. Engineering says the toolchain itself is adequate. Every production change still passes a weekly Change Advisory Board that reviews about 40 tickets in 90 minutes, and a separate operations manager signs each one. The board's stated purpose is preventing outages, and last year it rejected 3% of submitted changes. The CIO now proposes buying a value-stream management dashboard and an AIOps platform to 'get visibility'.
1. The CIO believes the problem is visibility. Using the evidence in the setup, explain what is actually constraining flow and what you would do in the next 60 days.
Read the worked answer
What the numbers say
- 22 days of lead time against a weekly board reviewing 40 tickets in 90 minutes means work spends nearly all its life waiting. A change submitted the day after the board meets waits six days for a two-minute decision.
- The board rejects 3% of what it sees. A control that passes 97% of its input is not filtering risk; it is queueing it.
- The pipeline exists and works, so tooling is not the binding constraint. Buying visibility into a system whose main delay is a calendar date will produce a very accurate picture of the wait.
First 60 days
- Make the wait explicit. Take 20 recent changes and record time in each state: coding, review, test, waiting for approval, waiting for a deployment window. Expect flow efficiency well under 10%.
- Classify changes by risk. Pre-approve a standard category (low risk, reversible, covered by automated tests and a canary) and let the pipeline be the control, with the audit trail generated by the tooling itself.
- Retain the board only for high-risk and irreversible changes, and shorten the meeting accordingly.
- Pair this with verification, because a 28% change failure rate says the releases themselves are unsafe: automated tests, canary or feature flags, one-step rollback.
- Baseline the four DORA metrics now so the later argument is evidence-based rather than ideological.
2. The head of risk argues the board is the only control preventing outages and will not relinquish sign-off. How do you respond in a way that avoids a fight over authority?
Read the worked answer
Reframe from who approves to what reduces risk, because arguing about authority makes the debate about status and nobody concedes.
Evidence to gather first
- Post-hoc analysis of last year's incidents: how many were caused by changes, and of those, how many had been approved by the board? If most failures came through an approved change, the board demonstrably does not detect the risk it claims to.
- Volume data: 97% approval means the control's discriminating power is near zero.
Proposal to put on the table
- A risk-tiered model. Standard changes are pre-authorised on the condition that automated evidence exists: peer review recorded, tests green, security scan clean, canary healthy, rollback proven.
- The board keeps high-risk and irreversible changes, plus anything outside the standard category.
- Automated controls are documented as controls. Auditors and regulators generally accept automated, tamper-evident evidence better than a signed ticket.
Make it reversible Agree a trial with exit criteria written down in advance: change failure rate no worse than baseline, restore time improving, zero audit findings. If the guardrails are breached, the tier reverts. This gives the risk owner a safe way to say yes, which is usually the real obstacle.
3. Six months later lead time has fallen to four days, but change failure rate has risen from 28% to 33%. A director wants to re-tighten approvals. What is your read and what do you recommend?
Read the worked answer
Do not read a single metric in isolation, and do not revert - but do not dismiss the signal either.
Two competing explanations, both testable
- Detection effect. Deploying far more often means more small releases are individually exposed to production, and monitoring now surfaces failures that previously hid inside a big-bang release. Check whether absolute failure count per unit of user-facing change actually rose, or only the rate.
- Genuine regression. Faster flow exposed a verification gap: changes now reach production before tests or canaries can catch them. Segment failures by class (configuration, dependency, logic, capacity) and by whether the pipeline flagged anything.
What to do
- Keep the loosened approvals. Re-tightening re-creates the queue and, per the earlier data, would not have caught these failures anyway.
- Invest where the failures actually originate: contract tests, canary analysis with automatic abort, feature flags to make releases reversible, and restore-time work (nine hours is the biggest single liability in the original baseline).
- Report the four metrics as a set. A speed gain bought with stability is not a gain, and a stability gain bought by slowing down is not one either.