Stretched SDN Controller Cluster Across a High-Latency WAN
An enterprise deploys a three-node SDN controller cluster for a VMware-based private cloud with 6,000 VMs across two sites: DC-A (4,200 VMs) and DC-B (1,800 VMs). All three controller nodes sit in DC-A; the inter-site link is 300 ms RTT and has had two 20-minute outages in the last quarter. The platform manages 900 tenants and 12,000 security-group policies, and policy pushes are API-driven. After the last WAN outage, DC-B workloads kept running but new policy could not be enforced, and when the link returned the controller logged 1,100 conflicting policy versions. The team proposes stretching the controller cluster across both sites. The CIO asks whether that reduces or increases risk and what reconciliation design is needed. The platform team has no local control plane in DC-B and relies on the WAN for all policy reads and writes.
1. Evaluate whether stretching the three-node controller cluster across both sites reduces or increases risk, including quorum behavior at 300 ms RTT.
Read the worked answer
Stretching the three-node cluster across a 300 ms link increases risk. A three-node quorum needs two nodes. If DC-A has two nodes and DC-B one, a WAN partition leaves DC-B with one node and no quorum; if DC-A loses one node during the partition, the whole cluster loses quorum. Worse, consensus systems generally require single-digit-millisecond RTT; 300 ms causes leader-election churn, slow writes, and false failure detection. The 1,100 conflicting versions already show what happens when both sides accept writes during a partition. I would keep the control cluster in DC-A with three nodes and deploy a local control-plane relay or a small secondary controller in DC-B for read-only policy and local enforcement. If the business requires site-survivable policy writes, use two separate clusters with a federated policy layer, not one stretched quorum. That gives DC-B a defined failure domain and avoids split-brain, at the cost of explicit reconciliation.
2. Design a reconciliation approach for the 1,100 policy conflicts and for future partitions.
Read the worked answer
Design reconciliation around intent, not last-writer-wins. Each policy object needs a tenant-scoped generation number, a version vector or causal token, and a tombstone for deletes. DC-B should be read-only during a partition unless a break-glass local policy is explicitly marked; otherwise both sides will create conflicting security groups. On reconnect, compare generation vectors, auto-merge non-conflicting objects, and quarantine conflicts for approval, especially anything that changes PCI or production access. The 1,100 conflicts should be treated as evidence that the API accepted writes without a partition check. Add idempotency keys, a staging area, and a dry-run diff before enforcement. For enforcement during the outage, cache the last-known-good policy locally and log every local decision so it can be audited. The target is not zero conflict; it is bounded, explainable conflict with a rollback.
3. What failure test would you require before production, and what result would make you reject the design?
Read the worked answer
Test the partition, not just the node failure. In a lab or maintenance window, cut the WAN while DC-B is serving production, confirm that local enforcement continues, and verify that DC-B cannot write conflicting policy. Measure how long policy reads are served from cache and how many flows fail closed versus open. Then restore the link and replay the reconciliation: how many objects conflict, who approves them, and how long it takes. Also test a two-node failure in DC-A to see whether the cluster goes read-only cleanly or hangs. The test should produce an RTO for policy changes and an RPO for policy state; if the business cannot accept a read-only DC-B for the duration of a WAN outage, the architecture is wrong, not just the runbook.