HA Cluster Failover Storm After a Switch Stack Upgrade

The cluster is obeying its configuration, not misbehaving. port3 is in the monitored-interface list, so a link-down on the primary forces an immediate release of the primary role, regardless of the health of port5, the heartbeats, or the fact that the LAN path itself is broken.

Why this is the wrong trade

• Failover cannot restore connectivity for traffic that needs port3: the physical path to the switch stack is down either way. The cluster pays a cluster-wide cost for a single-link problem.
• Each role change re-ARPs the WAN and LAN addresses, so every session and tunnel that was riding the healthy WAN is disturbed. That is the 40 seconds and the 30 tunnel re-negotiations.
• Roles reverse about 90 seconds later because the flap moves to whichever unit is now primary; with override disabled unit 1 does not pre-empt back, so the second event is a genuine failure of the new primary's port3, not a pre-emption.

Judgement A monitored interface should be one whose failure means the unit genuinely cannot forward traffic. A single access-switch uplink that flaps under LACP/STP churn is a link-quality problem to fix at layer 2, not a trigger to move the whole cluster. Separating 'this unit is broken' from 'this cable is flapping' is the point of tuning the monitor list.

Collect evidence first

• get system ha status - confirm the failover count, priorities and the monitored interface list.
• diagnose sys ha history read - reconstruct the sequence and timestamps of each role change.
• Interface counters on port3 and its aggregate (diagnose hardware deviceinfo nic port3) - CRC/error growth points at optics or the stack uplink rather than firmware.
• Switch-side LACP and STP logs for the same seconds; correlate them with the HA events.

Immediate mitigation Remove port3 from monitor-interface so a LAN link flap no longer moves the cluster, and optionally enable set link-failed-signal enable so a genuinely failed unit shuts its monitored interfaces down, forcing downstream devices to converge instead of black-holing.

Risk you accept explicitly With only port5 monitored, a unit whose LAN interface is dead but whose WAN is alive will stay primary and silently black-hole all LAN traffic. That is an acceptable short-term trade only because the LACP problem is being fixed in parallel and because the current behaviour already causes whole-cluster outages. Write the decision down and set a review date; do not leave it as permanent configuration.

Permanent design

• Terminate the LAN into a LAG with two members across the stack; monitor the aggregate interface, not the physical members, so a single member flap is invisible to FGCP.
• Keep heartbeat and session sync off the data path (dedicated ports) and keep the two heartbeat links on different switches.
• Keep override disabled with priorities 200/100 so pre-emption cannot add a second outage after every recovery.
• Review the session-pickup settings: session-pickup covers TCP state, and session-pickup-connectionless is needed for the UDP flows in this environment (SIP, RADIUS, syslog) that otherwise break silently even when TCP survives.

Proving it Run a controlled test in a window: hold an SSH session, a long HTTPS download, a ping stream and a SIP registration, then trigger execute ha failover set 1. Compare against the same test done today. Check diagnose sys session filter before and after the switch to see whether the session ID persists and whether the packet counters continue incrementing.

Expected result TCP flows survive with a sub-second interruption; UDP flows survive only if connectionless pickup is enabled; IPsec tunnels should stay up because the SAs are synchronised - if they still drop, the phase1 binding or the WAN ARP change is the next thing to examine, not the HA timers.