The Outage Nobody Planned For
A retail bank's online banking platform suffered a six-hour outage on a Saturday, affecting 400,000 customers. The root cause was a failed certificate renewal that a single engineer normally performed manually; that engineer was on leave and the handover note was out of date. The incident log shows the first alert fired after 90 minutes, the on-call team spent two hours diagnosing, and customer communications began after four hours. The bank's business continuity plan covers data-centre loss and cyber attack but says nothing about certificate or credential expiry. The regulator requires notification of material operational incidents within 72 hours. The board's risk committee meets in five days.
1. Classify and evaluate the risk that materialised. Why did the existing business continuity plan fail to mitigate it, and what does that reveal about the bank's risk process?
Read the worked answer
This is an operational risk event with a technology root cause and a concentration (key-person) dependency. It is not a rare 'black swan': certificate expiry is a foreseeable, dated event, and the fact that it was managed manually by one person made it both predictable and fragile.
The continuity plan failed because it was built around a narrow set of severe scenarios - data-centre loss and cyber attack - rather than around the bank's critical services and their dependencies. A plan that lists threats rather than services will always miss the mundane dependency that actually breaks. Certificate and credential expiry are classic 'known unknowns': low glamour, high impact, easily controlled.
The deeper failure is in the risk process. A single-person manual control with an out-of-date handover note should have been identified by an operational resilience assessment, recorded on a risk register, and mitigated by automation or a documented deputy. The 90-minute detection lag points to monitoring gaps; the two-hour diagnosis points to runbook and knowledge gaps; the four-hour communications lag points to an untested customer communication plan. The event therefore reveals three control failures, not one, and the risk committee should treat it as a test of the whole resilience framework rather than an isolated incident.
2. Draft the core content the risk committee should see in five days, and set out the immediate remediation actions with owners and dates.
Read the worked answer
The committee paper should have four parts.
Incident summary: timeline, customers affected, duration, root cause, and regulatory notification status (the 72-hour clock). State whether notification has been made and, if not, the deadline and the responsible executive.
Control failure analysis: detection (90 minutes), diagnosis (two hours), communication (four hours), and the key-person dependency. For each, state the failed control, why it failed, and whether it was known. Avoid attributing the event to an individual's leave; the control design is the issue.
Remediation plan with owners and dates: automate certificate and credential renewal with alerting (platform lead, 30 days); create and test a documented deputy rota for all manual operational tasks (operations lead, 45 days); reduce detection time with synthetic monitoring and expiry dashboards (monitoring lead, 60 days); rewrite the customer communication plan with pre-approved templates and a named on-call communications officer (communications lead, 30 days); extend the continuity plan to cover service dependencies and expiry events (head of resilience, 90 days).
Residual risk and assurance: state what remains exposed until remediation completes and commission an independent review of all single-person manual controls across critical services. The committee should also be asked to approve a standing item on operational resilience testing so that this is not a one-off response.