Cross-Region Failover Misses a Five-Minute RTO
A global SaaS provider runs an active-passive setup between Dallas and Frankfurt on IBM Cloud VPC. IBM Cloud Internet Services (CIS) global load balancer sends traffic to the Dallas VPC application load balancer; Frankfurt is standby. DNS TTL is 300 seconds. The CIS health check probes every 30 seconds and marks unhealthy after 3 failures. During a test, Dallas is taken down at 10:00:00. The team expects failover in about 6 minutes but users report errors until 10:11. Some users with long-lived connections stay on failed sessions. Frankfurt has capacity for the load at 55% average CPU, but its database connection pool is capped at 200 and the app would run 40 instances with 10 connections each. The RTO is 5 minutes.
1. Calculate the expected failover time from the configured health check and DNS TTL, and explain the gap to the observed 11 minutes.
Read the worked answer
The health check needs 3 failures at 30-second intervals, so detection takes about 90 seconds in the best case, plus the probe timeout. DNS records have a 300-second TTL, so a resolver that cached the record just before failure may keep sending traffic to Dallas for up to 5 minutes. The theoretical best case is 90 + 300 = 390 seconds, or 6.5 minutes, which already exceeds the 5-minute RTO. The observed 11 minutes is longer because TTL is not a guarantee: some resolvers and clients cache longer, and long-lived connections or session affinity keep users pinned to the failed region. Additional time may come from global load balancer health propagation, connection draining, application startup in Frankfurt, and database connection contention once traffic arrives. The lesson is that DNS-based failover cannot meet a 5-minute RTO with a 300-second TTL and a 90-second detection window.
2. Recommend changes to the traffic management and capacity design to meet a 5-minute RTO without doubling cost.
Read the worked answer
Reduce detection and propagation time first: lower the CIS health check interval to 10 seconds with 2 failures, and reduce DNS TTL to 60 seconds for the failover records. That gives roughly 20 seconds of detection plus 60 seconds of DNS cache, well inside 5 minutes, before application and database time. Use a warm standby rather than a cold standby: keep a small Frankfurt footprint running and scale it on failover, with pre-built images and tested startup. Externalize sessions to a shared store or use stateless tokens, and enable connection draining on the load balancer. For capacity, 40 instances x 10 connections = 400 connections, but the pool cap is 200, so Frankfurt would fail under full load. Either reduce connections per instance, add a connection pooler, or raise the database limit with tested headroom. Validate the whole path with a failover game day, because CPU headroom alone does not prove the database can accept the traffic.
3. What would you verify in the next failover test to prove the design meets the RTO and does not cause a second outage?
Read the worked answer
Measure from the customer perspective, not only from the control plane. Use synthetic probes from multiple regions to record the first failed request, the first successful request in Frankfurt, and the error rate during the transition. Capture DNS resolution time and the TTL actually observed by resolvers. Check whether existing sessions fail cleanly and whether users are asked to reauthenticate rather than being silently broken. On the Frankfurt side, verify database replication lag before cutover, connection pool saturation, CPU and memory headroom, and application startup time. Confirm that autoscaling responds within the test window and that the load balancer drains connections correctly. Finally, test failback, because a failed failback can create a second outage. Record all timings in the incident timeline and compare them with the 5-minute RTO and the service SLO.