Calder Mutual: a four-hour RTO, a nightly dump that failed silently
Calder Mutual runs a claims portal for 900 brokers. The business has agreed an RTO of four hours and an RPO of 15 minutes with its regulator-facing customers. Today the stack is a single EC2 instance in one Availability Zone of us-east-1 running both the application and a self-managed MySQL 8 database. A mysqldump runs at 02:00 every night into an S3 bucket in the same region. There is no infrastructure-as-code; the runbook is a wiki page last edited in 2022. Eight months ago an AZ-level event took the portal offline for about six hours, and the restore from the previous night's dump lost roughly 40 minutes of claims data because the dump had itself failed silently two days running and nobody checked. Auditors have now raised the disaster recovery plan as a finding. The CTO has budget for a modest increase of about 30 percent, not 300 percent.
1. Produce the gap analysis against the stated RTO and RPO, showing which evidence supports each gap.
Read the worked answer
Test every objective against what exists, not against what the diagram claims.
RPO of 15 minutes: the only recovery point is a nightly dump, so the worst-case data loss is close to 24 hours - two orders of magnitude off. Worse, the dump failed silently for two days and nobody noticed, so even the 24-hour promise was not real. Gap: roughly 24 hours actual versus 15 minutes required, with no integrity verification.
RTO of four hours: recovery depends on a single EC2 instance plus a manual restore from S3, guided by a wiki page last edited in 2022, with no infrastructure-as-code to rebuild the environment. The only observed test - the AZ event - took about six hours and did not include a full rebuild. Gap: unproven at best, and the absence of automation makes a four-hour rebuild implausible.
Regional resilience: the application, the database and the backups all sit in one region. A regional event loses the workload and its backups together. The requirement does not name a regional scenario, but customers will assume one.
Single AZ: already demonstrated as a six-hour outage, so the availability posture alone breaches the RTO before any disaster is considered.
2. Which disaster recovery strategy fits a 30 percent budget increase, and why not the cheaper or more expensive options?
Read the worked answer
Pilot light, moving toward warm standby, is the fit.
Backup and restore is what exists now and is the cheapest option, but it cannot meet a four-hour RTO reliably because recovery is a manual rebuild from a runbook. It stays as the last line, not the strategy.
Pilot light: keep data replicating continuously - a managed database with a cross-region read replica - and keep the core application defined as infrastructure-as-code with machine images kept current, but run nothing else until invoked. Costs scale with data and stored images rather than with a full duplicate environment, which fits a 30 percent budget, and a four-hour RTO is achievable if the rebuild is automated and rehearsed.
Warm standby would give a faster RTO by keeping a scaled-down but running copy of the application, at a higher monthly cost. Worth it only if the RTO tightens. Multi-site active-active is out of proportion to a four-hour RTO and to the budget.
The decisive constraint is not the strategy name but automation: without infrastructure-as-code, pilot light and backup-and-restore converge on the same multi-hour manual rebuild, so the IaC work is part of the DR project, not a follow-up.
3. What single change most improves the RPO, and what must accompany it so the improvement is real rather than theoretical?
Read the worked answer
Replace the nightly dump with continuous replication into a managed database engine, and rely on automated backups with point-in-time recovery for the local case plus a cross-region read replica for the regional case.
The reasoning: an RPO of 15 minutes cannot be met by any scheduled dump unless it runs every 15 minutes, which the current approach cannot do safely. Continuous replication and transaction log archiving reduce the exposure to seconds or minutes rather than hours, and point-in-time recovery lets you restore to a chosen moment rather than to a nightly boundary.
What must accompany it, or the improvement is cosmetic:
- Verification. The silent dump failure is the real lesson. Automate a restore test on a schedule, restore into an isolated environment, and alarm if it fails or if replica lag exceeds the RPO. An unmonitored backup is not a backup.
- Ownership and a runbook that is exercised, not a wiki page from 2022.
- Alerting on replication lag and on backup job status, routed to someone on call.
Be explicit with the auditors that replication lag, not the backup schedule, is now the thing that defines the RPO - and that it is measured, not assumed.