The Data Centre Outage and the Backup That Wasn't
Riverside Microfinance Bank runs its core banking system in a single on-premises data centre. On a Friday night, a cooling failure caused an unplanned shutdown. The IT manager reports that nightly backups run to a tape drive in the same room; the most recent successful restore test was 14 months ago. The bank's recovery time objective (RTO) is 4 hours and recovery point objective (RPO) is 1 hour, as stated in its ICT policy. The outage lasted 31 hours. During that time, branch staff recorded loan repayments on paper; about KES 4.2m in collections is unposted. The bank's regulator requires notification of material ICT incidents within 24 hours. The CEO learns of the outage on Saturday morning and asks the IT manager to 'get it fixed quietly before Monday'.
1. Evaluate the bank's resilience position against its stated RTO and RPO, and identify the control failures.
Read the worked answer
Gap against objectives
- RTO 4 hours vs 31 hours actual - the bank missed its recovery objective by a factor of ~8. There is no evidence of a tested recovery capability that could meet 4 hours.
- RPO 1 hour vs nightly backups - even a successful nightly backup gives an RPO of up to 24 hours, not 1 hour. The policy is unachievable with the current design; the control and the stated objective are inconsistent.
Control failures
- Single point of failure: no secondary site or failover; cooling is a single system with no redundancy.
- Backup co-location: tapes stored in the same room as the servers - a fire, flood or theft would destroy both production and backup. This is a basic 3-2-1 backup principle violation.
- No recent restore test: 14 months without testing means backup integrity is unproven. Untested backups are an assumption, not a control.
- Policy-reality gap: the ICT policy states objectives the infrastructure cannot deliver. Either the design must change or the policy is misleading - both are governance failures.
- No incident response process: the CEO's instinct to 'fix it quietly' shows there is no defined escalation and notification procedure.
Consequence The KES 4.2m in paper collections creates reconciliation risk: duplicate posting, lost records, and fraud opportunity. Without system controls, manual records are the only evidence.
2. What should the CEO actually do in the first 48 hours, and what should be reported to the regulator?
Read the worked answer
First 48 hours - recommended actions
- Do not conceal. Instruct the IT manager to preserve logs and the failed hardware. Concealment converts an operational incident into a governance and possibly criminal matter.
- Notify the regulator within 24 hours as the policy requires. The notification should state: what happened, when, systems affected, customer impact, interim controls, and remediation plan. Early notification with incomplete facts is far better than late notification with complete facts.
- Stabilise collections: freeze posting of the paper records until a controlled reconciliation is done; dual-authorise every manual posting; issue receipts to customers; reconcile against customer passbooks/SMS confirmations.
- Assess backup viability: attempt restore from the most recent tape before resuming normal operations, and verify data integrity.
- Communicate to customers if service is affected - reputational damage is worsened by silence.
- Convene the board risk committee and document decisions.
What to report
- Nature and duration of the incident.
- Root cause (as known) and whether customer funds or data were compromised.
- RTO/RPO breach and the remediation timeline.
- Any suspected fraud or data loss.
- Interim and permanent controls.
Tone The CEO's job is to protect depositors and the institution's licence, not to protect the IT manager's reputation. 'Quietly' is the wrong objective; 'contained and disclosed' is the right one.
3. Recommend a remediation roadmap, prioritising by risk.
Read the worked answer
Priority 1 - immediate (0–30 days)
- Off-site/offline backup copy (cloud or geographically separate site); apply 3-2-1 (3 copies, 2 media, 1 off-site).
- Full restore test with documented results; schedule quarterly thereafter.
- Redundant cooling/power (UPS + generator) with monitoring and alerts.
- Formal incident response and notification procedure, approved by the board.
- Manual posting controls and reconciliation for the KES 4.2m backlog.
Priority 2 - short term (1–6 months)
- Align policy to reality: either invest to meet RTO 4h / RPO 1h, or formally revise the objectives with board approval and disclose the change.
- Introduce replication or a warm standby site to achieve the RPO.
- Change management and access controls around backups (segregation of duties).
- Staff training on incident escalation.
Priority 3 - medium term (6–18 months)
- Business continuity and disaster recovery testing with business units, not just IT.
- Independent review (internal audit or external) of ICT resilience.
- Consider cloud or co-location for core banking with contractual SLAs.
Governance wrap Report progress to the board risk committee monthly until closed. Track each item with an owner and a due date. The lesson is that resilience is a board responsibility, not an IT department chore.