Three-AZ Billing Database: Synchronous Replication and Quorum Placement
A private cloud region has three availability zones. AZ1 and AZ2 are 0.4 ms apart; AZ1 to AZ3 is 1.8 ms; AZ2 to AZ3 is 0.6 ms. Each AZ contains eight compute hosts and a three-node distributed storage cluster. A core billing database needs 2 TB, 60k IOPS, RPO 0 and RTO 5 minutes. An analytics cluster can tolerate 30 minutes of data loss. The storage team proposes one stretched storage cluster across all three AZs. During a prior test, isolating AZ3 made 14 volumes read-only for 22 minutes because two of the three storage monitor nodes were hosted in AZ3. The database team wants local storage in AZ1 plus asynchronous replication to AZ2. Management insists that no single AZ failure may affect storage quorum or cause split-brain. The billing system writes 4,000 transactions per second at peak and must survive one compute host failure without manual intervention.
1. Choose the replication and quorum design for the billing database and the analytics cluster. Justify the choice using the latency and RPO/RTO figures, and state what must not be placed in AZ3.
Read the worked answer
Billing database: use a two-AZ active-passive design with synchronous storage replication between AZ1 and AZ2, plus database-level HA for automatic failover. RPO 0 requires synchronous acknowledgement before commit. AZ1-AZ2 latency is 0.4 ms, so the added round trip is roughly 0.8 ms; at 4,000 transactions per second this is workable if replication bandwidth and queue depth are monitored. AZ1-AZ3 at 1.8 ms adds about 3.6 ms round trip and more jitter; making AZ3 a synchronous replica would raise tail latency and risk false failovers.
Analytics: 30-minute data loss tolerance does not justify synchronous replication. Use asynchronous replication or scheduled backup/restore into AZ3. This keeps the low-latency path for billing and prevents analytics I/O from competing with transaction commits.
Quorum placement: use five monitor nodes: two in AZ1, two in AZ2, and one lightweight witness in AZ3. Any single AZ loss leaves at least three of five monitors, preserving quorum. The witness must not store user data and must have independent power and network paths. Do not place two data-bearing monitors in AZ3; the prior 22-minute outage is exactly the failure mode to avoid. Also place the billing VM and its storage replicas on different compute hosts so one host failure does not remove both copies.
2. The storage team argues that a three-AZ stretched cluster is simpler. Give specific failure scenarios that disprove this for the billing workload, and define failover test acceptance criteria.
Read the worked answer
A three-AZ stretched cluster is simpler only on paper. For this workload it creates three concrete risks:
- Quorum loss during an AZ outage: if two of three monitors sit in AZ3, losing AZ3 leaves one monitor and the cluster goes read-only, as already happened for 22 minutes. Placing data replicas in all three AZs does not fix the monitor majority problem.
- Tail latency and false failover: a synchronous write must wait for the slowest required acknowledgement. AZ3 at 1.8 ms plus network jitter can push commit latency beyond the database timeout, causing application retries, connection storms and unnecessary failovers even when AZ1 and AZ2 are healthy.
- Split-brain risk during asymmetric partitioning: if AZ1 and AZ2 lose their direct link but both can still reach AZ3, a data-bearing three-node quorum can produce competing majority claims. A witness-only tie-breaker avoids this; a full data replica in AZ3 does not.
Acceptance criteria: isolate each AZ in turn and prove the billing database remains writable or fails over within 5 minutes with zero committed-transaction loss. Kill one compute host and prove automatic restart with no manual step. Break the AZ1-AZ2 link and prove exactly one side remains writable. Measure p99 commit latency during each test and require it stays within the application timeout. Run the tests twice, including during a storage rebuild, and record recovery time and data-loss evidence.