Warm Standby That Cannot Meet a Five-Minute RPO
A payments processor handles 2.2 million transactions per day in a primary region and maintains a warm standby in a second region. The board approved an RTO of 30 minutes and an RPO of 5 minutes. The primary database is 14 TB, backed up with a daily full backup and hourly transaction logs; the last successful restore test was 11 months ago and took 6 hours. The failover runbook references a jump host that was decommissioned, DNS records have a 3,600-second TTL, and application secrets exist only in the primary region. The queueing layer replicates asynchronously but the replication lag alarm is disabled. A recent regional network event caused a 55-minute outage because the team could not confirm whether the standby was current. The company wants a tested DR capability before peak season in 120 days and cannot run a full duplicate production stack.
1. Identify the gaps against RTO/RPO and recommend a DR strategy that fits the constraint against a full duplicate stack.
Read the worked answer
Reasoning: The current design cannot meet RPO 5 minutes because hourly log backups allow up to 60 minutes of loss. A 6-hour restore also misses RTO 30 minutes, and a warm standby that cannot be validated is not a recovery capability.
Recommendation: keep the warm standby but make it a managed, tested recovery environment. Replicate the 14 TB database continuously or ship logs every few minutes, monitor lag, and alert at 3 minutes so the 5-minute RPO has margin. Replicate secrets, DNS configuration, and infrastructure as code to the secondary region. Reduce DNS TTL to 60 seconds before peak season. Define a tenant-level and service-level failover order. Automate promotion with a runbook that checks replication lag, queue depth, and application health before switching traffic. If lag exceeds 5 minutes, declared data loss must be approved by finance and compliance; do not silently fail over. Run a controlled failover every quarter and a full DR exercise before peak season.
2. How should backup validation be redesigned to prove recoverability rather than just backup completion?
Read the worked answer
Reasoning: A backup is not a recovery point until it is restored and the application is validated. The 11-month-old test is stale, and the 6-hour restore time is the real constraint.
Recommendation: restore into an isolated recovery account quarterly, and measure time to first query, time to full application health, and data consistency. Validate row counts, checksums, referential integrity, and queue offsets against the primary. Automate the restore with infrastructure as code so the test is repeatable. Keep at least two restore points: a recent point for RPO and a known-good point before major releases. Record evidence and assign an owner for remediation. Run a table-level restore test monthly for the most critical tables, because a full 14 TB restore will not always be the first action. If restore exceeds 30 minutes, add a warm standby or snapshot-based recovery rather than assuming the backup meets RTO.
3. Rewrite the failover decision and runbook approach so the next event does not repeat the 55-minute outage.
Read the worked answer
Reasoning: The 55-minute outage was a decision and evidence problem, not only a technology problem. The team could not confirm standby currency, and the runbook had stale dependencies.
Recommendation: rewrite the runbook as a decision tree with three branches: controlled failover, degraded service, and no failover. Put the replication-lag check first and require a screenshot or automated evidence. Remove the decommissioned jump host and replace it with a bastion in the recovery account. Automate DNS cutover and secret retrieval; do not rely on manual console steps. Define a failback path before failing over, including data reconciliation. Run a tabletop exercise in 30 days and a live failover in 90 days. Give the incident commander authority to fail over when criteria are met, and require a communication template for customers and regulators. Track time to detect, decide, and recover separately.