HA pair flaps after a switch replacement

Heartbeat path FGCP heartbeats are time-sensitive hello packets. When they travel through an access switch, they share fate with the switch's control plane. After the RSTP replacement, topology changes, root elections, and MAC table flushes can drop or delay heartbeat packets long enough to exceed the lost-heartbeat threshold. Once the secondary stops hearing the primary, it promotes itself.

Why override turns it into a loop With override enabled, a unit with higher priority and a stable heartbeat reclaims the primary role. So the sequence becomes:

1. Heartbeats are delayed, the secondary promotes.
2. The switch reconverges and heartbeats resume.
3. The original primary sees the peer and, because it has priority 200 and override is enabled, preempts and takes over again.
4. Another topology event repeats the cycle.

The uptime reset on the original primary in get system ha status is consistent with it repeatedly losing and regaining the primary role, not with a hardware fault. User VLANs sharing the same switch also mean broadcast traffic and CPU spikes on that switch affect heartbeat delivery.

Separate the heartbeat plane from user traffic

• Connect port3 and port4 directly between the two units, or through a dedicated non-RSTP switch that carries no user VLANs. Two heartbeat links give redundancy.
• Keep both ports listed as heartbeat devices so the loss of one cable does not cause a failover.

Stabilise election behaviour

• Disable override on the primary so a recovering unit does not preempt. Use priority only to decide which unit becomes primary at boot, not to force a takeover later.
• Keep only genuinely critical interfaces in the monitor list. Monitoring an interface that legitimately flaps converts a WAN event into a cluster failover.

Validate

1. get system ha status on both units: confirm the same group, correct role, and that heartbeat and session pickup are reported as enabled.
2. diagnose sys ha checksum show on both: identical checksums confirm configuration sync.
3. Pull one heartbeat cable during a maintenance window and confirm no failover; pull both and confirm a single clean failover with no immediate preemption.
4. Watch the HA event log for 24 hours after the change to confirm the 10 to 15 minute cycle has stopped.

Checks

1. Session pickup: confirm it is enabled on both units and reported in get system ha status. Then place a test call and inspect the session table on the secondary with a filter for the SIP and RTP addresses; if the SIP signalling session exists but the RTP pinholes do not, the ALG-created state is not being carried across.
2. Hardware offload: check whether the voice sessions are offloaded. Offloaded sessions may not be represented in the same way on the standby, so the new primary has to relearn forwarding state; this is a familiar source of brief media loss on failover.
3. ARP and reconvergence: after failover the new primary must take over the MAC and IP. Measure how long SIP re-registration takes; if the provider requires re-registration, brief loss is expected regardless of session state.
4. Timers: confirm SIP registration and RTP timeouts are not so short that a one- to two-second interruption ends the call.

What to tell the business HA protects against device failure; it is not a zero-packet-loss guarantee. An active-passive failover interrupts forwarding for a short period, and stateful protocols can still notice it. For voice, pair the cluster with provider-side redundancy, tune registration timers, and test planned failovers during change windows so the residual interruption is measured and accepted rather than discovered during an incident.