Checkout API Across Two Continents
An online retailer runs its checkout API in one region across three availability zones. The API has a 99.95% availability SLO, p95 latency of 180 ms, and 12,000 requests per second at peak. The product catalog is read-heavy and served from a managed relational database; order writes must be durable. Users are concentrated in North America (70%) and Europe (30%), and the business wants p95 under 250 ms on both continents. Finance will fund at most 2.2x the current infrastructure spend. The platform team has six engineers, no dedicated database administrator, and a five-minute recovery time objective with a one-hour recovery point objective. A recent zone failure caused 40 minutes of degraded checkout because the connection pool exhausted during failover. Leadership asks for a resilient design that does not double operational complexity.
1. Recommend a target architecture. State what you would deploy in each region, how traffic is routed, and which constraint most strongly shapes the decision.
Read the worked answer
Constraints first. The 99.95% SLO allows about 22 minutes of unavailability per month, so multi-AZ alone is not enough for a region-level event; the five-minute RTO rules out manual rebuild. The 2.2x budget cap and six-engineer team rule out a fully active-active multi-writer database. The 70/30 user split and 250 ms p95 target mean Europe cannot send every read to North America.
Decision. Run the stateless checkout API in both regions behind a global load balancer with health-based routing. Keep the relational primary in North America and add an asynchronous cross-region read replica in Europe. Serve catalog reads from a local read replica in each region. Route order writes from Europe to the North American primary through a regional write proxy, so checkout remains functional with one cross-region write hop, not a full page load. On region loss, promote the European replica and shift writes.
Why not other options. Active-active writes add conflict handling and operational burden the team cannot absorb. Single-region multi-AZ misses the RTO for a regional failure. A cold standby would require rebuild time and violate the five-minute target unless heavily automated.
Tradeoff. European checkout writes see higher latency than reads, but the write path is a small fraction of requests. The design adds a failover runbook and replication-lag monitoring, not a second write topology.
Validation. Test a forced region failover and measure time to first successful order, not just DNS change.
2. Design the data-tier failover. Specify replication mode, promotion procedure, and how you would prevent the connection pool exhaustion seen in the zone failure.
Read the worked answer
Use asynchronous replication to a warm standby in the second region. Asynchronous replication is acceptable because the RPO is one hour, but lag must be measured continuously; a failover is unsafe if lag exceeds the RPO and the primary is still reachable.
Promotion procedure: (1) declare region failure from health checks and on-call confirmation; (2) fence the old primary to prevent split brain; (3) wait until replica lag is at or below the agreed threshold or the RPO window expires; (4) promote the replica; (5) update the write endpoint through a controlled DNS or proxy switch; (6) drain and restart application pools; (7) verify checkout writes with synthetic transactions.
Preventing pool exhaustion: size each regional pool against the database max-connection limit, reserve admin connections, and use a connection pooler with bounded queues and timeouts. Pre-warm standby application instances so failover does not create a thundering herd. Add retry with jitter and a circuit breaker for connection acquisition, and alert on pool wait time and checkout errors separately. During the last zone failure, the pool likely retried without a budget; cap retries and fail fast.
Evidence: record lag at failover, time to promotion, time to first successful write, and connection wait percentiles.
3. What evidence would you collect in the first 30 days to show the design meets the five-minute RTO and the 99.95% SLO?
Read the worked answer
Collect three classes of evidence. Failure injection: run a controlled regional failover during low traffic; record the timestamp of injection, detection, promotion, DNS/proxy switch, and first successful synthetic checkout. The RTO is the end-to-end interval, not the database promotion alone. Repeat for a zone failure to confirm the pool fix.
SLO measurement: define the SLI at the load balancer as successful checkout responses under one second divided by valid requests, measured per region and globally. Report 99.95% monthly and the error budget. Do not use average latency; use p95 and p99, because the SLO is about user experience.
Supporting signals: cross-region replication lag, connection-pool wait time, retry counts, saturation, and dependency error rates. Tie each alert to an SLO burn rate so the team knows whether to page.
Decision rule: if the first failover test misses five minutes, do not claim the design meets the RTO; fix automation or pre-warming and retest. If the test passes but p95 exceeds 250 ms in Europe, revisit the write-proxy placement.