CCOA (IJCAHPO): Active-Passive Payments API With a 47-Minute Failover

Time the path segment by segment.

Detection is ALB health checks (3 x 30 s = 90 s). Decision is a human approval in chat, measured at roughly 12 minutes and unbounded. DNS is a 300 s TTL, so up to 5 minutes of clients still resolve to the dead region. Aurora global promotion is usually under a minute of lag but was observed at 4 minutes. Application rollout with a cached endpoint is about 6 minutes. Verification adds several minutes.

Interleaved, that is roughly 90 s + 12 min + 5 min + 4 min + 6 min, about 28 minutes. The path is dominated by decision and DNS, not by the database.

RTO breaches on their own:

1. Manual approval. 12 minutes plus a timezone-dependent approver consumes the entire budget before any technical step runs, and it has no upper bound.
2. DNS TTL 300 s. Five minutes is a third of the budget, and a fraction of clients stay on the failed region after cutover.
3. Cached connection string. Recovery is not complete until every process restarts, so partial failure persists and some requests keep failing after cutover.

RPO risk: Aurora Global Database is typically sub-second, so the database alone probably meets 5 minutes unless promotion happens during a replication lag spike. Alarm on replication lag at 30 s. The genuine RPO exposure is the statement objects written only to the primary bucket: whatever is not yet replicated is lost, and that lag is neither bounded nor monitored.

Ranking: automate the decision first, then make the DNS record health-checked with a short TTL, then remove the cached endpoint, then add monitored cross-region replication for S3. Fixing DNS before the approval buys nothing, because the approval still dominates.

Automate the decision, not just the actions. Replace the chat approval with a Step Functions workflow triggered by a Route 53 health check or an operator command. The workflow must: verify replication lag is under threshold, detach the secondary cluster, promote it, poll until the writer endpoint accepts connections, then flip traffic. Record each step timestamp so the path is measurable. Keep a human approval only for a destructive branch, never on the critical path.

Pre-provision the routing. Create both DNS records ahead of time and use failover routing with health checks; the standby record should be secondary with its own health check. Set TTL to 30-60 s, which is the practical floor given resolver caching, and accept that some clients will need retries.

Pre-provision capacity. The standby runs two tasks and has never taken traffic. Either keep it at a validated minimum (for example 50% of required tasks, with Fargate fast scaling covering the rest in under two minutes) or accept a longer recovery and document it. Validate that the task role, secrets, and subnet capacity exist in eu-west-1; a warm standby that cannot start tasks is a cold one.

Change the application. Read the endpoint at startup from a regional SSM parameter path or Secrets Manager regional secret, and refresh on connection failure rather than caching for process lifetime. Make the statement writer idempotent and dual-region aware, and enable S3 Cross-Region Replication with replication metrics, or write to a global endpoint.

Remove hidden dependencies. Audit for anything else pinned to us-east-1: KMS keys, SQS queues, third-party webhook URLs, IAM roles that only exist in one account. Each is a silent RTO breaker.

Progressive validation in four tiers.

1. Component tests, run continuously in CI: promote a scratch Aurora global cluster, confirm the app starts against the new writer, confirm the DNS health check flips. This costs minutes and catches config drift weekly rather than semi-annually.
2. Traffic-shifted canary: use weighted routing to send 1% of production traffic to the standby for 30 minutes during low season, then 10%. This exercises real capacity and real data paths with a bounded blast radius. Roll back by shifting weight, not by failing over.
3. Isolated game day: promote the secondary in a non-production account that mirrors production shape, with the same Terraform and the same runbook automation. Time every step. Do this monthly.
4. Full failover: quarterly, in production, announced, with a written abort criteria list (replication lag above threshold, capacity below minimum, any dependency unverified).

What to measure: time-to-detect, time-to-decision, time-to-promote, time-to-first-successful-transaction, and time-to-full-capacity. Track each as a time series, not a single number, because the RTO is the sum and regressions hide in one segment. Also track error-budget consumption during the exercise and the count of undocumented manual steps performed.

Prove it stays ready: run nightly conformance checks that compare standby configuration, IAM, and secrets against the primary and alert on divergence, plus a synthetic canary transaction that runs in both regions continuously. A DR plan that is only tested twice a year is a hypothesis; weekly component checks turn it into a measurement.