Data Centre Failover Breach and the Manual Workaround
At 06:40 on a black rainstorm day, the primary data centre of a Hong Kong retail bank lost power. Automatic failover to the secondary site completed at 11:00, four hours twenty minutes after the outage and well beyond the two-hour recovery time objective for core banking. Mobile banking and ATMs were intermittently available; 14,000 customers could not complete transfers. Branch staff recorded 800 transactions on spreadsheets and keyed them in after service resumed, producing six duplicate postings and two missed stop-payment instructions. The business continuity plan was last tested 14 months ago, and that test was a tabletop walkthrough, not a failover. The crisis management team wants to close the incident as 'business as usual' because no money was lost and no data was compromised. The operational risk manager must decide whether to escalate it as a major incident and what to recommend to the BCP owner.
1. Should this be escalated as a major operational risk incident? Apply criteria and address the 'no loss' argument.
Read the worked answer
Yes. Major incident criteria typically include customer impact, duration, RTO breach, control failure, regulatory relevance and potential for harm, not only realised loss. Here 14,000 customers were affected, the core RTO was exceeded by more than two hours, the manual workaround produced six duplicate postings and two missed stop-payment instructions, and the BCP had not been fully tested for 14 months.
The 'no money lost' argument is weak because the missed stop-payments could have caused customer loss and may still produce claims; duplicate postings create reconciliation and reputational risk. The incident should be escalated as major, with a formal incident report and root-cause analysis. It should also be assessed against the bank's regulatory incident-notification threshold. Closing it as business as usual would suppress exactly the information the BCP owner and board need. The incident owner should be the BCP owner or COO, with operational risk providing challenge and tracking actions to closure.
2. What were the control failures in the manual workaround, and what should the BCP test programme change?
Read the worked answer
The manual workaround failed basic control principles: no unique reference for each entry, no maker-checker, no reconciliation before keying, no cut-off, no dual control and no log of who entered what. The duplicate postings and missed stop-payments are the predictable result. Changes: pre-approved manual procedures with numbered vouchers, independent verification, end-of-day reconciliation, system controls to block duplicate references, and a named controller for the workaround.
The BCP test programme should move from tabletop to full failover at least annually, test the actual RTO by process, include manual workarounds and third-party dependencies, and record actual versus target recovery. It should also test loss of the secondary site and simultaneous unavailability of key staff. Test results must go to the board with actions and dates; a 14-month gap between tests is itself a control finding. The BCP owner should confirm that the recovery objective is achievable with current technology and staffing, not just documented.
3. What should be reported to the board and regulator, and how should customer impact be handled?
Read the worked answer
Report to the board: timeline, root cause, customer impact, RTO breach, workaround errors, near misses, remediation and owner. Report to the regulator if the incident meets the HKMA's notification threshold; do not wait for a loss to be realised. Customers affected by duplicate postings or missed stop-payments should be contacted and compensated where appropriate, with a clear complaints route.
The risk register should be updated with the failed BCP test and the manual workaround control gap. The board should also see whether the bank's operational resilience depends on a single data centre and what investment is needed. The operational risk manager should track actions to closure and re-test before closing the incident. Customer communications should be coordinated with compliance and legal so that the bank does not make commitments it cannot meet.