Active-Passive Failover with a 12-Minute Replication Lag
An active-passive service runs on Red Hat OpenShift on IBM Cloud in Dallas, with a warm standby in Frankfurt. Cloud Internet Services global load balancer health checks move traffic when the Dallas endpoint fails. IBM Cloud Databases for PostgreSQL replicates asynchronously to Frankfurt; at peak, replication lag has reached 12 minutes. The stated objectives are RTO 30 minutes and RPO 5 minutes, and the last full failover test was nine months ago. The runbook requires a manual DNS change with a 300 s TTL and a database promotion step. The monthly SLO is 99.9%, and 70% of the error budget was consumed in the first 10 days by a bad release. Product management wants to ship a major feature next week and asks whether the DR posture is acceptable.
1. Can the current design meet RTO and RPO? Break down the failover timeline and identify the largest gaps.
Read the worked answer
RPO is already violated at peak. Asynchronous replication lag of 12 minutes means a failover can lose up to 12 minutes of committed transactions, against a 5-minute objective. RTO is plausible but unproven. A rough timeline: health-check detection (interval times unhealthy threshold) plus DNS TTL 300 s plus manual decision and database promotion plus application warm-up and validation. If each manual step takes 5–10 minutes, the total can approach or exceed 30 minutes, especially if the operator has not run the procedure recently. The largest gaps are data loss (RPO) and human/runbook latency (RTO), not the standby cluster's compute. A nine-month-old test does not prove either objective. The answer to product management is: the posture is not acceptable for a 5-minute RPO claim; either reduce lag, redefine RPO, or add a synchronous or near-synchronous option for critical writes.
2. Given the error budget, what release and reliability decisions do you recommend?
Read the worked answer
The 99.9% monthly SLO allows about 43.2 minutes of full outage equivalent. Consuming 70% in 10 days is a strong signal that the service is not stable enough for normal release cadence. Invoke the error-budget policy: pause feature releases that change the checkout path, keep low-risk fixes and reliability work, and require an exception review for the major feature. Use the remaining budget to fund the failover test, replication-lag reduction, and runbook automation. If product insists on shipping, require a canary with automatic rollback, a freeze on unrelated changes during the rollout, and a documented risk acceptance. The decision is not 'no releases forever'; it is 'earn back budget before adding risk.' Set a re-entry threshold, such as two consecutive weeks with no severity-1 incidents and lag under the RPO target, then resume.
3. Design a game day to validate failover and state what you would measure.
Read the worked answer
Run a planned, reversible game day with a named incident commander, a scribe, and a rollback authority. Pre-stage: confirm backups, record current replication lag, verify DNS and health-check settings, and notify stakeholders. Exercise: simulate Dallas failure by disabling the endpoint or blocking traffic, then follow the runbook exactly, including the manual DNS change and database promotion. Measure detection time, decision time, DNS propagation, database promotion duration, application readiness, error rate during the transition, and actual data loss by comparing a known write sequence. Capture every manual step and any point where the runbook diverges. Afterward, compare measured RTO and RPO with objectives and file actions: reduce lag, automate promotion, lower TTL or use a load-balancer failover that does not depend on DNS, and add a pre-flight check. Schedule the next test before the next quarter, and do not combine it with a feature release.