INC-mode HA pair reaches dual-primary after an inter-DC path outage
Northwind Retail runs two Citrix ADC MPX 5901 appliances in high availability across two data centers. The pair is configured in INC mode because the NSIPs must sit in different management subnets: 10.10.10.11/24 in DC1 and 10.20.20.11/24 in DC2. The VIP 192.168.50.100/24 serves store order traffic and is advertised by OSPF from both sites. Each node has its own default gateway. During a scheduled WAN maintenance window, the routed path between the two NSIPs was down for 11 minutes. Afterward, operators found both nodes reporting state Primary. Store clients experienced intermittent TLS resets and duplicate order submissions, and the persistence tables on the two nodes differed. The WAN link is back, but the pair has not returned to a stable primary/secondary relationship. The team wants to know whether INC mode is appropriate and how to prevent a repeat.
1. Why did both nodes become Primary, and what makes that state unsafe for the store traffic?
Read the worked answer
Both nodes became Primary because each one lost sight of its peer's HA heartbeat/hello traffic and independently concluded that the peer was dead. INC mode changes which network settings are local to each appliance, but it does not remove the need for IP reachability between the NSIPs for HA heartbeats. When the routed path failed for 11 minutes, each node promoted itself, producing a split-brain condition.
That state is unsafe because both nodes can own and answer for the VIP, and both may advertise reachability. Clients can land on either node, but persistence entries, SSL session state, and connection tables are not continuously synchronized. The result is exactly what was observed: TLS resets on connections that move between nodes and duplicate order submissions when a retried request is processed by the other appliance. This is not a supported steady state.
Verification should include show ha node on both appliances, stat ha, show interface, show route, and show ip. Look for peer state, heartbeat counters, and whether the nodes can reach each other's NSIP. Recovery should be controlled rather than accidental: force one node out of the Primary role, restore heartbeat reachability, and then force a synchronization before returning the pair to service. Do not simply wait for the link to come back and assume the pair will self-correct.
2. What configuration and design changes would you make to ensure deterministic failover if the WAN path fails again?
Read the worked answer
The main design flaw is that HA peer health depends on the same WAN path that carries production traffic. The fix is to give HA its own reliable path and to make failover decisions explicit.
First, add a dedicated HA heartbeat link between the two appliances, preferably a direct L2 segment or a dedicated VLAN that does not traverse the WAN. In INC mode the NSIPs can remain in separate subnets, but the heartbeat path must be redundant and reachable. Second, use an HA monitor/failover interface set so the appliance reacts to loss of critical interfaces rather than relying only on peer heartbeat timers. Configure add ha monitor for the relevant interfaces and enable HA monitor on the node with set ha node -hamonitor ON. Third, ensure each node has a redundant route to the peer NSIP and test failover by disabling the primary's uplink, not by unplugging the heartbeat.
If the two sites can be made L2-adjacent, evaluate NCC mode, which keeps network configuration consistent. If INC is required, document that routes, VLANs, and default gateways are configured per node. Validate with show ha node, stat ha, and a controlled failover test during a maintenance window.
3. A colleague wants to add the VIP return route only on the current primary. Why is that unsafe in this deployment?
Read the worked answer
That proposal is unsafe because INC mode does not synchronize network configuration the way many administrators expect. Routes, VLANs, interfaces, and default gateways are local to each appliance. If the VIP return route exists only on the current primary, the standby node may be able to take over the VIP but may not know how to return traffic to the client network or to the VIP's subnet.
The failure would not necessarily look like a clean outage. The new primary could accept connections but send return traffic through the wrong gateway, producing asymmetric routing, failed TCP handshakes, or connections that establish and then stall. Troubleshooting would be confusing because the VIP is up and the service is marked UP, yet clients fail.
Instead, configure the required routing identically on both nodes, verify with show route on each appliance, and include route reachability in the failover test plan. If a route is needed only for the VIP, confirm that both nodes have it and that the upstream routers can reach the VIP from either site. In INC mode, treat every network change as a two-node change, even though the service configuration is synchronized.