I85PREP
⌘ K
Aruba ACSP study materials
Aruba ACSP study resources

Split Brain at 02:00

Aruba ACSP · Put your knowledge to work in context.

All case studies

Split Brain at 02:00

Two 8325s run VSX. The ISL is a 2x100G LAG across two line cards. Keepalive runs over a shared 1G management switch that also carries OOB and the automation host. Servers dual-home into the pair as VSX LAGs with LACP. A change window at 02:00 scheduled fiber cleaning on the ISL and a firmware reboot of the management switch, overlapping by about 70 seconds. When the ISL dropped, both peers stayed primary, both kept their VSX LAG member ports up, and the core LAG stayed up on both. Monitoring recorded duplicate frames, MAC flapping on the core, and a two-minute outage for the dual-homed servers. Both nodes came back in-sync after the window without manual intervention.

1. What should the pair have done, and which single design change matters most?

Read the worked answer

Normal VSX behavior: if the ISL fails while keepalive is alive, the secondary detects that it can no longer reach the primary over the ISL for a short hold time, marks itself out-of-sync, and takes its VSX LAG member ports down. Servers then home only to the primary. The cost is a brief outage; the benefit is that no two devices forward for the same anycast MAC and VLAN.

What happened here: the two failure-detection paths shared fate. The ISL went down and the keepalive path rebooted at the same time, so neither peer could prove the other was still alive. Both stayed primary and both kept VSX LAG members up, producing a split brain: duplicate anycast gateway responses, MAC flapping at the core, and two-minute impairment until the management switch returned.

The change that matters most is making keepalive independent of both the ISL and the management plane it protects - a dedicated direct link between the peers, or a separate OOB switch on separate power, ideally diverse fiber. Everything else is secondary:

  • Tightening hold timers only shrinks the window toward detection; with a shared-fate failure it can even lengthen it by triggering false failures.
  • vsx split-recovery is a backstop that can auto-recover one side after both paths fail, but it may reboot a production node, so it must be a deliberate decision, not a substitute for a reliable keepalive.

2. What would you verify on both peers immediately after service is restored, and what would you look for in the MAC tables?

Read the worked answer

Per-peer state, not just 'it pings'.

  • show vsx status and show vsx brief on both: ISL up, keepalive up and reachable in both directions, one primary and one secondary, and config sync reported in-sync rather than in-progress or diverged.
  • show lacp interfaces for every VSX LAG: both members collecting and distributing, partner in sync, no member stuck in a defaulted or detached state after the ports bounced.
  • show interface on the ISL and core uplinks for CRCs, input errors and discards from the event.

MAC tables

  • The same MAC appearing on both peers' core uplinks, or repeatedly moving between the ISL and a core port, indicates a residual split brain or a stale entry, not normal learning.
  • Large numbers of entries now reachable only via the ISL after recovery suggest traffic is hairpinning unexpectedly; confirm it settles once the peers are truly primary/secondary.
  • Confirm the anycast gateway MAC is owned consistently and that ARP/ND entries for the gateway resolve on both peers.

Also compare running config against the pre-window backup, because a peer that reloaded may have come back with unsynced or partially applied VSX-synced configuration.

3. Was the maintenance window itself the fault, and what would you change procedurally?

Read the worked answer

Yes - the window design was the direct cause.

Two changes that each remove redundancy were scheduled to overlap. Each was individually low risk; together they eliminated every independent failure-detection path. A VSX pair tolerates losing the ISL, and it tolerates losing the management path, but not both simultaneously with no out-of-band visibility.

Procedural changes:

  1. Treat the ISL and keepalive path as a shared-fate pair in the change process. The risk assessment must ask explicitly: does this change remove the same redundancy as another change in the same window?
  2. One redundancy-removing change per window, with a verification gate - confirm show vsx status is healthy on both peers before starting the second.
  3. Before the window, confirm which peer is primary and how to force traffic onto the survivor; have console access to both peers independent of the management switch.
  4. Save a config backup and record the pre-change VSX state, so post-change drift is measurable.
  5. Decide the split-recovery policy in advance and document why it is enabled or disabled.
  6. Test the scenario once: pull the ISL in a lab window and confirm the secondary really goes out-of-sync and drops its VSX LAG members.