VSX Pair Split-Brain During Core Maintenance

Cause. VSX synchronization depends on the ISL carrying control/sync traffic; the keepalive is only used to detect peer liveness. On reload, the primary came back and did not successfully re-form the ISL LAG (e.g., LAG member mismatch, LACP not negotiated, or the ISL VLAN/trunk config drifted). With the ISL down but keepalive up, each node sees the peer alive but cannot sync - depending on configuration and timing, both can believe they are primary.

Immediate risk. Both cores now forward independently: duplicate MAC learning, split default gateway behavior, and (with MSTP) potential loops if the distribution layer's redundant links to both cores are not blocked. Intermittent loss matches asymmetric forwarding and MAC flapping.

First action: restore a single authoritative primary. Bring down one side's core-facing links or administratively disable the secondary's VSX role until the ISL is healthy, then verify show vsx status and show vsx configuration on both before re-enabling.

Verify/fix the ISL.

• Confirm the ISL LAG members are up and LACP is negotiating on both sides.
• Confirm the ISL carries the required VSX VLANs and that MTU and trunk allowed lists match.
• Confirm keepalive is on a separate path from the ISL (it is) and that keepalive is not accidentally riding the ISL.
• Add monitoring/alerts on vsx status and ISL health so a desync is caught before users notice.

Routing design.

• BGP/OSPF to the upstream should be anchored so that only the active VSX primary advertises the shared gateway/prefixes, or use a design where both advertise but the path is deterministic. Duplicate advertisement from both halves during split-brain is what causes black-holing and flapping.
• Use VSX active-gateway for the default gateway so the virtual MAC is consistent, and ensure routing protocols do not form independent adjacencies that survive a split-brain in a harmful way.
• Consider BFD with short timers to fail over fast and to detect a desynced peer.

Change control. Reload of a VSX member should be a documented procedure: verify peer health first, confirm ISL re-forms after reload, and only then resume normal operation. Add an explicit post-reload checklist.

During split-brain. With the ISL down, the two cores are no longer one logical bridge; MSTP must block one of the redundant distribution-to-core paths or a loop forms. Intermittent loss is consistent with MSTP reconverging (or failing to) while both cores forward.

Considerations.

• Confirm MSTP region config (name, revision, VLAN-to-instance mapping) is identical on all switches; a mismatch puts links into different regions and can create unexpected blocking.
• Use root guard on core-facing distribution ports so a misbehaving downstream cannot become root, and BPDU guard on edge ports.
• Ensure the distribution layer's uplinks to the two cores are in the same MST instance so one is cleanly blocked/alternate.
• Loop protection (e.g., loop-protect on edge, and consistent STP priorities) should make the core pair the deterministic root.

After recovery. Verify the topology is back to the intended single-root design, check that no ports are unexpectedly blocked, and confirm MAC tables are clean (no duplicates). Add a post-incident check that MSTP root and VSX primary agree with the documented design.