Tenant-Pinned Active-Active for a Regulated Ordering Platform
A B2B ordering platform serves 12,000 requests per second at peak from North America, Europe, and Asia. The current single-region deployment has a p95 latency of 180 ms for EU users and 240 ms for APAC users, against a 150 ms target. The board wants active-active across two regions within nine months. Orders and invoices contain EU personal data, and contracts require EU tenant data to remain in the EU. The primary database is an Oracle cluster that the team plans to keep for 18 months. Finance requires RPO of 1 minute and RTO of 15 minutes. The platform team has 14 engineers, limited Kubernetes experience, and a fixed annual cloud budget that cannot grow by more than 12%. Marketing cannot accept an outage during the launch window.
1. Recommend a topology that satisfies the latency target, EU data residency, RPO/RTO, and the 12% budget cap. Explain why the obvious alternatives fail.
Read the worked answer
Reasoning: A naive two-region active-active for all writes would either replicate EU tenant data outside the EU or impose cross-region write latency, so it fails residency or the 150 ms target. A pure active-passive warm standby is cheaper but leaves APAC reads slow and makes failover a 15-minute event that has not been exercised. The 12% budget cap rules out three full write cells.
Recommendation: build two tenant-pinned active-active cells, EU and US, with a shared global routing layer. Each tenant has a home region; its reads and writes stay in the home region, and a read-only replica is placed only where residency permits. APAC users hit edge caches and a read-only API tier, not a third full write cell. Within each cell, use three availability zones and synchronous replication for RPO 0 inside the region. Cross-region replication is asynchronous, monitored for lag under 60 seconds, and used for DR, not normal writes. Failover of a tenant to the paired region is a controlled DR action with legal approval, not automatic for EU data. This meets latency for EU and US users, keeps EU data in the EU, and stays within budget by avoiding a third write cell.
2. How should tenant and order data be partitioned and replicated during the 18-month Oracle transition? Include write-path and failure handling.
Read the worked answer
Reasoning: A single global database cannot satisfy residency and latency without cross-region writes. A per-tenant database per region increases operational load but aligns with tenant pinning. Oracle must remain authoritative until reconciliation proves the new store.
Recommendation: shard by tenant_id, with all rows for a tenant in its home region. Partition orders by tenant_id and order_month; keep invoices immutable and append-only. Put a write API in front of Oracle and the new store. Use a transactional outbox plus change data capture from Oracle rather than dual writes, and make consumers idempotent with order_id and event_id. During transition, Oracle is system of record; the new store is a shadow read model. Cut over tenant by tenant only after reconciliation shows zero lost orders and less than 0.1% mismatch for 14 days. On write failure, fail within the home region first; do not silently route EU writes to the US. Cross-region replication is for DR and read-only serving where allowed.
3. Sequence the migration and define rollback gates that protect the launch window.
Read the worked answer
Reasoning: The team has limited Kubernetes experience and a launch freeze, so a big-bang cutover is the main risk. EU tenants are the most constrained and should not be first unless the legal path is proven.
Recommendation: Phase 0 builds the landing zone and observability. Phase 1 migrates a low-risk US tenant cohort to the new cell read-only. Phase 2 enables writes for that cohort with Oracle shadow reads. Phase 3 onboards five EU pilot tenants with legal sign-off and data-residency checks. Phase 4 expands by cohort. Gates: p95 under 150 ms for the cohort, replication lag under 60 seconds, error rate under 0.5%, reconciliation mismatch under 0.1%, and a successful tenant-level failover drill. Rollback is a routing change back to Oracle plus disabling the new write path; keep Oracle writable for 30 days after each cohort. Freeze all changes two weeks before the marketing launch.