Weekly deployments and the 11-day board queue
A 200-person online retailer deploys its customer-facing platform weekly, and the platform team raises a change record for every deployment. The change advisory board meets on Tuesdays. In the last two months, 96 changes were raised. The median time from 'ready to deploy' to a board decision was 11 days. Seventeen changes were rolled back, and 11 of those involved the same configuration item, the pricing service. Three of the rollbacks traced to a shared library upgraded by a different team; no dependency mapping existed. Deployments are blocked in the final week of each month for financial reporting. Product managers have begun bundling low-risk content and copy edits into the same deployment as pricing changes so they need only one board cycle. The engineering director wants the board abolished; the head of risk wants more changes reviewed. You are the change manager asked to cut lead time without increasing failed changes.
1. Which changes should not be going to the board, and how would you decide?
Read the worked answer
The test is not size, effort or which team raised it - it is risk and recurrence. A change belongs outside the board if it is well understood, repeatable, reversible within the deployment window, contained within one service's blast radius, and covered by a test that would catch its common failure mode.
On that test, routine content and copy edits almost certainly qualify: frequent, low impact, individually reversible. Make them a standard change with a pre-authorised procedure, automated tests and a deployment record. The pricing service does not qualify: 11 of 17 rollbacks on one CI is the definition of poorly understood and high blast radius.
Order of work:
- Write the standard-change definition, including the evidence required to keep the status (test coverage, rollback time, monitoring).
- Delegate normal changes by service to named change owners, keeping the board for cross-service, first-of-type and high-risk work.
- Keep pricing as a board item until the dependency problem is fixed.
Two cautions. A standard change is not an unrecorded change - traceability to the deployment still matters. And the deeper fix is the 11-day decision latency: if the board only saw high-risk items, it could meet more often and decide faster, which removes the incentive to bundle at all.
2. What do the repeated pricing-service rollbacks indicate, and what would you do before the next deployment?
Read the worked answer
Eleven rollbacks on one configuration item is not a change-control problem; it is a known-error problem that change control keeps re-approving.
The shared library is the clue. The pricing service depends on an artefact another team controls, and no dependency mapping exists, so impact analysis at the pricing service is incomplete by construction. Three rollbacks were caused by someone else's upgrade. Until that is addressed, every pricing change is being assessed against a stale picture of the estate.
Before the next deployment I would:
- Freeze pricing changes that touch the shared library, with a written exception route and a stated reason.
- Map the dependency in both directions, pin or version the library contract, and add a contract test that fails the build when the interface changes.
- Put the shared library on the pricing service's impact list so its upgrades raise a linked change.
- Review the pricing back-out procedure. Eleven rollbacks suggests either weak pre-production testing or a rollback being used as a way to ship untested code.
- Raise a problem record with a named owner. This will not be closed by the next release, and saying so is part of the job.
Communicate it as protecting the weekly cadence, not as a freeze for its own sake.
3. Product managers are bundling low-risk edits with high-risk pricing changes. Is that a discipline problem or a design problem, and what would you do?
Read the worked answer
It is a design problem producing predictable behaviour, and treating it as indiscipline would misread the incentive and drive the practice further underground.
The design flaw is that every change pays the same toll: 11 days of latency regardless of risk. A product manager with a copy edit and a pricing change has one sensible option under that regime - put both in one deployment and pay once. The fix is differentiated change models (standard, delegated normal, high-risk board), so the copy edit costs nothing and the pricing change gets the scrutiny it needs.
That said, bundling creates a real traceability risk: a low-risk wrapper can carry a high-risk change past its proper assessment. Two controls make bundling safe rather than forbidden:
- A deployment manifest listing every change in the release, each mapped to its own change record and risk classification.
- A rule that high-risk items cannot be approved merely by inclusion in a low-risk bundle; their approval must be visible on their own record.
I would also explain the change to the product managers, because the fastest way to lose the improvement is to leave them believing the toll still exists and keep bundling quietly.