Single-Region Cloud Failure at Northbridge Bank

Assess tolerance first. Degraded authorization has lasted 2 hours 50 minutes by 12:00, so the 2-hour degraded-service tolerance is already breached. The 4-hour outage tolerance is not yet breached, but the manual backlog implies a long recovery: 1.2 million transactions at 300 per minute is about 66 hours of manual processing, so the team should assume the outage tolerance will be breached unless capacity changes.

Immediate actions. (1) Declare a major incident and activate the incident command structure; (2) attempt the secondary-region failover with an explicit rollback trigger, because the manual channel cannot clear the backlog and the runbook is the only tested path, even if partially; (3) keep manual processing running in parallel for high-value and time-critical payments, with a documented triage rule; (4) notify the regulator now, since a material payment disruption is likely reportable and waiting for certainty is not defensible; (5) issue a customer communication that states what works, what does not, and when the next update will come; (6) scale the call center and stop marketing traffic that generates avoidable payment volume.

Reasoning. The decision is not 'failover or not' but 'how to reduce customer harm while preserving rollback.' The failed DNS test is a known control weakness, so failover should be time-boxed and monitored. Manual processing alone is mathematically inadequate.

Weaknesses exposed. The bank has a single-provider, single-region concentration for a critical service; the failover runbook is stale and partially untested; manual fallback capacity is roughly 7.5% of normal volume; the bank lacks early-warning indicators from the provider; and the escalation path did not include timely regulatory notification. These are not separate issues: they compound into a long-tail outage.

Ranking. 1. Automated multi-region failover with quarterly full failover tests, including DNS and data consistency checks. This directly reduces the probability and duration of a regional outage. It is expensive and takes 6-9 months, but it addresses the largest single point of failure. 2. Manual capacity and payment triage. Contract a backup processor or pre-authorize a standby mainframe path, and predefine which payment types are prioritized. This can be delivered in 8-12 weeks and reduces customer harm during any failover delay. 3. Third-party observability and exit/contingency planning. Require provider telemetry, contractual notification, and a tested exit plan. This is lower cost but does not by itself restore service.

Constraint. If only one investment can be funded this year, choose the failover program and run a time-boxed manual-capacity pilot in parallel; otherwise the bank remains exposed to the same scenario.

Measures. The review should quantify: time to detect, time to declare, time to failover, total tolerance breach duration, backlog clearance time, failed or duplicated transactions, customer harm (fees, missed payments, complaints), regulatory notification timing, and whether the runbook step that failed had been tested. It should also measure dependency concentration: percentage of critical services in one region, one provider, and one availability zone.

Board use. The board should not receive a narrative-only report. It should receive a remediation plan with owners, dates, cost, and residual risk, plus a decision on whether the impact tolerance remains appropriate. The board should hold management accountable for test evidence, not promises. It should also require scenario testing for a simultaneous provider outage and cyber event, because the current plan assumes one cause at a time.

Reasoning. Post-incident metrics are useful only if they change limits, budgets, and accountability. A board that accepts 'the provider had an outage' without asking why failover was untested is not providing effective oversight.