Both Chassis Say Master: A Split-Brain After a Heartbeat Cable Swap
Two USG6650E firewalls run active/standby hot standby. HRP heartbeat uses GE1/0/7 on each chassis, joined by a single direct cable. VRRP groups 1 (trust) and 2 (untrust) provide gateway redundancy, and NAT Server rules publish two web servers. During a maintenance window an engineer replaced the heartbeat cable and then pushed a policy change from FW1. Afterwards FW1 reports VRRP master on both groups and FW2 also reports master. Users see intermittent resets, the core switch logs MAC flapping between two ports, and syslog shows duplicate-IP ARP conflicts for the untrust virtual address. On FW1, display hrp state shows HRP_M[active] with the heartbeat interface not ready; on FW2 it shows HRP_S[active]. The two devices now hold different policy revisions.
1. Explain what state the pair is in, why it produces the observed symptoms, and the order in which you would restore service.
Read the worked answer
State: the HRP channel is down while both chassis still hold VRRP master. HRP synchronises configuration and session tables; VRRP decides who owns the virtual MAC and IP. They have decoupled, so each firewall independently believes it is the active device - a split-brain, not a failover.
Symptoms follow directly: both devices answer ARP for the untrust virtual address, so the upstream switch sees the same IP/MAC pair move between two ports (duplicate-address detection and MAC flapping). Return traffic is load-shared unpredictably between two chassis that each have half the sessions, so TCP flows reset. NAT Server traffic is worse: both devices own the server-map entry and either may answer for the published address, with no shared session state.
Restore in this order:
- Decide which chassis should be master - normally the one with the fresher, verified configuration and the live sessions (FW1 here, since the change was pushed from it). Do not let 'whoever is master' decide.
- Shut down the VRRP/master role on the loser first (raise VRRP priority cost or disable the VRRP groups) so only one device answers ARP. This stops the flapping and the resets immediately.
- Repair or replace the heartbeat path and confirm the interfaces are up and in the correct zone on both sides.
- Reconcile configuration before re-enabling HRP: batch sync from the chosen master, then verify the peer's policy and NAT revisions match. Re-enable HRP and confirm
display hrp stateshows HRP_M/HRP_S on the correct chassis, then re-enable VRRP with preemption. - Validate with a session-table comparison and a controlled failover test.
2. What configuration and process controls would have prevented this, and which 'fix' would make it worse?
Read the worked answer
Prevention is mostly about not having a single, unmonitored heartbeat path.
- Redundant heartbeat: run HRP over an Eth-Trunk of two links (or two independent interfaces) so a cable swap cannot silently kill the channel.
display hrp interfaceshould be part of the standard post-change checklist. - Track the path: bind an HRP track to the heartbeat link and to the uplinks so that losing the heartbeat forces a defined, visible state rather than leaving two masters.
- Preempt delay: configure a VRRP preempt delay long enough for HRP batch backup to finish, so a recovering chassis does not seize the master role with a stale configuration.
- Change discipline: never push a policy change from the active chassis while the standby is out of sync, and never work on the heartbeat path during a policy window. Verify
display hrp stateand both configuration revisions before and after every change.
The fix that makes it worse: 'just reboot one firewall'. A rebooted chassis comes back with VRRP preemption enabled, grabs master immediately, and if HRP has not finished batch backup it will forward traffic with an outdated policy or an empty session table - turning a partial outage into a full one. Similarly, disabling VRRP preemption globally to stop the flapping hides the symptom and leaves the pair unable to fail back predictably.