The Change That Skipped the CAB
A payments team pushes a configuration change to a production firewall at 23:00 on a Friday to fix a latency issue before a Monday promotion. The change was logged in the ticketing tool but not submitted to the Change Advisory Board, which meets weekly on Tuesdays. At 01:30 the change causes intermittent timeouts for a card-authorisation partner; the on-call engineer rolls it back at 02:15 but does not open a problem record, only a resolved incident. On Monday the partner reports failed transactions worth an estimated £40,000 over the weekend. The change manager discovers the event during a routine audit and finds the team had a verbal approval from a senior engineer, no documented risk assessment, and no rollback plan on file. Leadership asks whether this was a process failure or a people failure, and what to change.
1. Classify what went wrong across the change lifecycle and identify the control gaps.
Read the worked answer
This is primarily a process/control failure, with a behavioural element.
Gaps by stage:
- Assessment: no documented risk assessment or impact analysis for a customer-facing firewall change.
- Authorisation: bypassed the CAB via verbal approval; no emergency change path was used even though one presumably exists for exactly this scenario.
- Planning: no rollback plan on file, though the engineer improvised one.
- Implementation: Friday-night window before a promotion increased exposure with thin support.
- Post-implementation: no problem record, so the root cause and partner impact were never tracked; the incident was closed as resolved.
So the failure is systemic: an emergency route existed but was not used or not known, and closure discipline is weak. Blaming individuals alone would miss that the process made the correct path unclear or unattractive.
2. Recommend changes to the change-enablement process, distinguishing standard, normal, and emergency routes.
Read the worked answer
Make the right path the easy path:
- Standard: pre-authorised, low-risk, repeatable changes (define a catalogue) so routine work needs no CAB.
- Normal: CAB-reviewed with documented risk, impact, and rollback.
- Emergency: a fast-track with a named authoriser on call, retrospective CAB review within, say, 2 business days, and mandatory rollback plan. Publish this route so the payments team uses it instead of a verbal nod.
Specific fixes: require a rollback plan as a gate for any production change; add a post-implementation review trigger for any change rolled back; and make problem creation automatic when a change causes a rollback or customer impact. Communicate that emergency change is legitimate and fast, removing the incentive to go informal.
3. The partner seeks assurance. What governance and reporting actions would you take, and how would you prevent recurrence?
Read the worked answer
Governance actions:
- Raise a problem record now to establish root cause and link the incident, change, and financial impact.
- Report the £40,000 impact and the control breach to leadership and, if required, the risk register, with an owner and remediation date.
- Do not discipline the engineer for using a route the process failed to make clear; address the design, and only escalate individually if the emergency route was deliberately evaded after being offered.
Prevention:
- Track a metric for changes implemented without authorisation and for changes causing rollback.
- Schedule CAB review of emergency changes so learning is captured.
- Add a pre-promotion freeze check so high-risk windows are visible.
- Test the emergency path in a tabletop exercise.
Close the loop by confirming the partner sees the remediation, since assurance is part of the outcome, not just internal paperwork.