Token Vault Across Two Regions Under a Fixed Budget
NorthPay operates a card-tokenization platform in a single IBM Cloud VPC in Dallas. It handles 2,300 transactions per second at peak, uses Db2 for the token vault, and keeps session state in Redis. After a four-hour regional outage at a co-location provider, the regulator now expects an RTO of 30 minutes and an RPO of 5 minutes for tokenization. The CIO wants active-active across Dallas and Frankfurt to avoid paying for idle capacity, but security notes that the token vault is the system of record and must remain consistent. Shared services such as central logging, DNS, and key management are Dallas-only today. The team has 14 weeks and a fixed budget; a second full production footprint in Frankfurt would consume 70% of the remaining budget. Frankfurt-to-Dallas latency is about 95 ms, and the token vault replication license is priced per replica.
1. Recommend an architecture pattern (active-active, active-passive, or warm standby) for the token vault and the surrounding platform. Justify the choice against the RTO, RPO, data consistency, latency, and budget constraints.
Read the worked answer
Reject active-active for the token vault. The vault is a write-consistent system of record; synchronous cross-region writes at 95 ms RTT would add unacceptable latency to a 2,300 TPS authorization path, and an active-active write-conflict model would require application redesign that 14 weeks cannot absorb. A warm standby in Frankfurt is the right judgment: pre-provisioned VPC, runtime, Db2 standby, and replicated Redis, with a Global Load Balancer failing over on health checks. RTO 30 minutes is achievable only if the standby is running and the runbook is automated; a cold backup-and-restore site cannot meet it. RPO 5 minutes requires asynchronous replication with lag monitoring. Failover should be allowed only when measured lag is below 5 minutes; otherwise the incident commander must choose between stale tokens and extended outage, and that decision must be documented. Cost matters: a warm standby at roughly 40-50% of production fits better than a second full active footprint at 70% of budget. Cap Frankfurt capacity so it can carry production during failover but not full peak indefinitely.
2. Redesign the shared services so Dallas is not a single point of failure for the Frankfurt failover path. What dependencies must be removed or replicated, and how would you test it?
Read the worked answer
Treat Dallas-only shared services as a second failure domain. DNS should be external to both regions with low TTL and health-checked records; do not let Frankfurt pods resolve critical names through a Dallas resolver. Logging should use regional forwarders that write to a COS bucket in the same geography, with cross-region replication only for retention, so a Dallas outage does not blind Frankfurt. Key management needs a Frankfurt instance and a documented key-import or backup-and-restore path; runtime should not make a synchronous call to Dallas for every decrypt. If the regulator requires single custody, use cross-region key backup under dual control, not an active dependency. Secrets Manager should have regional replicas or a break-glass procedure. The principle is simple: the failover architecture must not depend on the failed region for identity, DNS, keys, or logs. Test by simulating Dallas loss and verifying that Frankfurt can start, authenticate, decrypt, and emit audit evidence without Dallas.
3. What evidence would you present to prove the RTO and RPO, and what would you change if failover consistently takes 50 minutes instead of 30?
Read the worked answer
Present a measured game day, not an architecture diagram. Instrument five timestamps: injection of failure, detection by health check, decision to fail over, database promotion and application readiness, and Global Load Balancer traffic shift. Measure data loss by comparing token sequence numbers or replication lag at cutover. Include DNS TTL behavior and client retry storms, because a 30-minute RTO can be consumed by cached DNS. If failover consistently takes 50 minutes, do not simply raise the RTO. Find where the extra 20 minutes go; common causes are manual approval, database promotion scripts, DNS TTL, and missing pre-pulled images. Automate the runbook, lower TTL before the window, pre-stage capacity, and pre-warm caches. If database promotion is the bottleneck, consider a pre-provisioned standby with automated fencing and a tested split-brain procedure. Then rerun the game day and report the percentile, not the best case. The regulator should see both the target and the measured result, with remediation dates.