Failsafe on the Heartbeat VLAN Moves Both Traffic Groups
An active/standby pair (BIG-IP A and B) runs two traffic groups: tg-portal (self IP 10.20.1.100) and tg-partner (self IP 10.20.1.101). tg-portal was active on A; tg-partner was configured to be active on B and had served traffic there for two years, so the pair effectively runs active/active for different applications. The device group is sync-failover with Auto Failover enabled. Network failover runs on VLAN 500, which carries heartbeats only and no data traffic, and VLAN failsafe is also enabled on VLAN 500 with a 5-second timeout. At 02:14 a switch serving VLAN 500 reloaded during a firmware upgrade; at 02:14:07 both traffic groups were active on B. B's CPU reached 98% and the portal was unusable for 40 minutes. When A came back, it did not take the portal back. After the incident about 30% of portal clients still time out on the VIP, although the VIP answers normally from a directly connected test host. MAC masquerade is not configured, and the core router holds a static ARP entry for 10.20.1.100 pointing at A's MAC.
1. Why did both traffic groups move to B, and what do you change so a heartbeat-only VLAN cannot trigger this again?
Read the worked answer
This was VLAN failsafe, not network failover. Network failover exchanges heartbeats between peers; VLAN failsafe watches for any traffic on a specific VLAN and, when the timer expires without traffic, marks that VLAN's failover objects as failed. VLAN 500 carries only heartbeats, so when the switch reloaded nothing crossed it, the 5-second timer expired, and every traffic group listing VLAN 500 as a failover object was forced to the peer. Neither device failed - which is why A came back healthy and did nothing.
Changes:
- Remove VLAN 500 from the failover-object list of both traffic groups and disable VLAN failsafe on it. A heartbeat VLAN is exactly where failsafe should not be enabled.
- Keep network failover, but remember it has the same exposure: if heartbeats stop for longer than the failover timeout, Auto Failover will still move traffic groups. Carrying heartbeats over a direct link between the two units, or over redundant switch paths, removes the shared failure domain.
- Add failure detection only where it means something: gateway ICMP monitors or self-IP tracking on the data VLANs that actually carry user traffic.
- Re-examine the active/active arrangement. Both traffic groups landing on one device is survivable only if that device has the capacity for both, and 98% CPU says it does not.
2. Why didn't A take the portal back, and how do you return it safely?
Read the worked answer
Traffic groups do not fail back automatically. Once tg-portal moved to B it stays active there until something forces it off: the device fails, an administrator forces standby, or the traffic group is explicitly moved. Auto Failover governs whether a device that goes offline is failed over from - it is not a 'return when healthy' setting.
Also note what happened when tg-portal joined tg-partner on B: B was suddenly running both applications. That, not the seven-second blip, is why the portal was degraded for 40 minutes. Two years of active/active by traffic group means the surviving unit must be sized to carry everything, and this one was not.
Safe failback, in a maintenance window:
- Confirm A is healthy: reachable over network failover, config in sync (show cm sync-status), monitors green, and no pending changes on B.
- Fix the ARP problem described by the 30% timeouts first, or the failback recreates the outage in reverse.
- Drain the portal pool members on B before moving, so live connections finish rather than reset.
- Move only tg-portal (run sys failover standby with the traffic-group argument on current releases) rather than failing B over completely, which would also move tg-partner and recreate the overload on A.
- Watch both devices for 15 minutes, then confirm persistence entries and monitor status on the new active unit.
3. Explain the 30% of portal clients that still time out, and give two fixes with their trade-offs.
Read the worked answer
The split is the signature of a stale ARP mapping, not a BIG-IP fault. On failover the newly active device sends gratuitous ARP for the floating self IP, but the core router holds a static ARP entry tying 10.20.1.100 to A's MAC, and static entries ignore gratuitous ARP. The router keeps forwarding portal traffic to A, which is now standby. Hosts on the same subnet learn from the GARP and work fine - exactly the pattern observed.
Fix 1 (correct one): delete the static ARP entry from the core router and let dynamic ARP resolution work. Cheap, immediate, and it removes a permanent dependency on GARP propagation. Trade-off: the router must not depend on a static entry for any other reason (some environments pin ARP to defeat spoofing), so involve the network team.
Fix 2 (make failover transparent): configure MAC masquerade on tg-portal so both devices advertise the same MAC for that traffic group. The shared MAC moves with the group, so an upstream cache - static or stale - still points at something that answers. Trade-offs: the MAC must be unique in the L2 domain, the VLANs in the traffic group must share an L2 path, switch features that pin MACs to ports or enforce port security must be adjusted, and you should still test a real failover in a window.
Either way, validate by failing over deliberately and checking the router's ARP table and client behaviour, rather than waiting for the next unplanned event.