Multi-Cluster Failover and Data Residency
A SaaS company runs the same application in three Kubernetes clusters: us-east, eu-west, and ap-south. Customer data must stay in the region where it was created (GDPR for EU customers). A global load balancer routes users to the nearest healthy cluster. The team wants active-active across all three for availability, with automatic failover if a cluster goes down. The database is a single primary in us-east with read replicas in the other two. During a planned us-east maintenance, the team fails traffic over to eu-west, but EU customer writes now land in eu-west and are replicated back to us-east, violating residency. Also, the failover took 12 minutes because DNS TTL is 300s and health checks are slow.
1. Identify the architectural conflict between active-active availability and data residency, and explain why the current failover violated residency.
Read the worked answer
The conflict: active-active for availability wants any cluster to serve any user, but data residency requires EU customer data to be written and stored only in the EU. With a single global primary in us-east, all writes - including EU customers' - go to us-east, which already violates residency even before failover. When traffic failed over to eu-west, EU writes were served locally but then replicated back to us-east, so EU data left the EU. The residency requirement and the single-primary topology are fundamentally incompatible.
The failover didn't cause the violation; it exposed it. The correct model is data partitioning by region: each region has its own primary for its own customers (or a regional shard), and cross-region replication only carries non-resident data or is avoided entirely. The global load balancer must route each user to the cluster that owns their data, not merely the nearest one.
2. Propose a topology and failover design that satisfies residency while meeting the availability goal, and address the 12-minute failover.
Read the worked answer
Topology: Move to regional data ownership. Each cluster (us-east, eu-west, ap-south) hosts a primary database for customers in its region. A routing layer (e.g., a geo-aware gateway or per-tenant DNS/anycast with a tenant→region map) sends each request to the owning region. Cross-region replication is limited to non-resident metadata or eliminated. This gives active-active per region while keeping EU data in the EU.
Availability within a region: Use multi-AZ replicas inside each region so a zone failure doesn't take down the region's primary. Cross-region failover is only for that region's customers, and only to a standby that is legally permitted to hold their data (e.g., an EU standby in another EU region).
Failover speed: The 12 minutes came from DNS TTL (300s) plus slow health checks. Reduce TTL to 30-60s, use active health checks with short intervals, and prefer an anycast/L4 load balancer that can withdraw a region's route in seconds rather than waiting on DNS. Pre-warm standby capacity so promotion doesn't wait on pod startup. Target <2 minutes for regional failover, and document that failover for EU customers stays within EU regions.