Nightly Failover, Dropped ERP Sessions, and a Split Brain

The firewall is doing exactly what it was told; the monitoring target is wrong.

Mechanism

• Path monitoring probes 10.99.0.1 every 3 seconds with 5 failures - 15 seconds of loss triggers a failover. That host sits on the backed-up segment, so during the backup window its CPU and network are saturated and its ICMP replies are delayed or dropped.
• The device concludes the forwarding path is broken and hands over. Twenty minutes later the backup finishes, the probe recovers, and since preempt is off it fails back only on the next transition.
• Because the probe measures a server, not a path, the conclusion is invalid.

Correcting the design

• Monitor something that reflects forwarding health: the upstream next hop on each data interface, one target per egress path, not a host deep inside a busy segment.
• Raise the threshold to something that survives a 15-second blip - e.g. probe every 5 seconds, 10 failures, and require the condition to persist before acting.
• Remove backup-included hosts from any monitoring target list, and add a change-control note so the next person does not add one back.
• Cross-check with link monitoring on the physical interfaces; if the link is up and the upstream gateway answers, a deep-segment host timing out is not a failover reason.

Enabling sync is not the same as syncing the right things, and a 1 Gbps HA2 that shares an access switch with production is a common place for sync packets to be dropped under load.

Candidates, in the order I would check them

1. The new virtual system. If HA is configured per virtual system and the new vsys was not added to the HA pair, none of its sessions were ever synchronised. This is the most likely cause given the six-week timeline.
2. 'Synchronize all sessions' off. Only a subset of sessions is mirrored. Long-lived idle ERP connections are exactly the profile that ends up unsynchronised, and they are exactly what a user notices.
3. HA2 quality. 1 Gbps shared with data through the same switch, during a backup window, is where sync packets get dropped silently.

How to confirm

• Session-sync statistics on both nodes: nonzero on the active and matching counts on the passive. If the passive shows nothing for the ERP zone, it is a scoping problem.
• Compare the session table on both nodes for the ERP destination while the pair is healthy - an unsynchronised session exists on one side only.
• Then resync cleanly: enable all-session sync, verify the counts, and plan a controlled failover in a maintenance window rather than discovering it at 02:05.

The reboot removed both heartbeat paths simultaneously, so each node concluded its peer was dead. With preempt off and no tiebreaker, both had a legitimate claim to active. That is a design defect, not bad luck.

Changes

• Physically separate the HA paths. A direct cable between the pair, plus a second path on a different switch, so no single device carries both HA1 and HA2. Sharing the data switch defeats the purpose of a redundant pair.
• Data-path link monitoring with a defined action. If the HA links drop but the data interfaces are up, decide explicitly whether the node should hold its state or yield. Ambiguity is what produces two actives.
• Deterministic election. Set device priority so one node wins, and document which one is intended to be active at each site.
• Heartbeat encryption and authentication so a stray device cannot join the pair, and so a mismatched peer is rejected rather than forming an unintended cluster.
• Alerting on the two-active condition. This must page someone within a minute; four minutes of dual-active with the same IPs and MACs causes ARP conflicts and silently dropped traffic that is very hard to diagnose after the fact.

Test it: reboot the access switch in a maintenance window and confirm exactly one node is active throughout.