A Failover Test That Passed Two VLANs Out of Three

Link up plus a pingable local address prove only L2 to the switch. They say nothing about whether a floating address is owned or forwarded.

1. 10.0.2.5 is not a floating IP in the vrid on VLAN 300. Most common when one VLAN fails and others are fine: the address may exist only as a static VIP on A, giving B no reason to answer. Check: vrid and floating-IP inventory on both units.
2. VRRP-A not participating on VLAN 300. The vrid may exist but not track that VLAN, so no ownership transition happens there. Check: per-vrid state on both units before and during the test.
3. The other team's trunk does not permit VLAN 300 toward B. The local port shows up because the trunk is up, but tagged frames are dropped. Check: allowed VLAN list on both uplinks and the core's MAC table for 10.0.2.5.
4. Stale ARP on the core. After failover the switch still maps 10.0.2.5 to A's MAC. Check: clear the entry during the failed state and see whether traffic recovers - if it does, that is your cause.
5. Asymmetric return path, where the core routes the VIP back toward A.

Discriminate first: a ping from the core to 10.0.2.5 during the failure separates 1 and 2 (no answer at all) from 4 (answers, but converges slowly). That single test halves the list before you change anything.

Convert 'the test discovered it' into 'the checklist discovered it':

• Inventory every VIP against the HA configuration, per VLAN. Does the address exist as a floating IP inside a vrid, on both units? Run it as an exported diff, not by eye - the failure here was an address that existed on only one unit.
• Confirm L2 on both units for every service VLAN. Trunk allowed lists on both uplinks, not link state on the local port. This is where a configuration owned by another team hides.
• Confirm VRRP-A state per vrid before the window, and that set-id, device-ids, priorities and preemption settings agree on both units.
• Prepare one synthetic probe per VIP, sourced from the same vantage point as real users - through the core, not from the appliance.
• Write pass criteria in advance: probe success within N seconds of transition, plus evidence of ARP convergence.
• Have a rollback ready: re-enable VRRP-A on A and confirm it resumes; know the command before the window opens.
• Invite the trunk team. The fault lives in their configuration; discovering that at 02:00 without them turns one incident into two.

Where possible test one VLAN at a time so a failure stays isolated, and record actual transition times - they become the SLA evidence for the next review.

HA-specific risks:

• The pair must be able to form and synchronize configuration. If the target release does not interoperate with the running peer, the pair is broken the moment the first unit reboots, and the survivor carries everything with zero redundancy.
• Config synchronization has a direction. Know which unit is authoritative before touching either, and take a full configuration backup of the active first.
• Floating IP ownership and vrid state must be verified after every step, not merely 'both units are up'. An upgrade that boots cleanly but loses a floating IP looks healthy in a status summary.

Order:

1. Read the release notes and compatibility guidance for this specific current-to-target pair; determine whether a mixed-version pair is supported at all.
2. Back up both configurations and record versions, the vrid and floating-IP inventory, and which unit is active.
3. Upgrade the standby first - it is out of the data path, so a failure costs time, not service.
4. Verify it boots, receives configuration, and reports healthy standby. Do not fail over until that is verified.
5. Fail over deliberately, run the per-VIP synthetic probes, then upgrade the former active unit.
6. Confirm both units agree, and fail back only if the design requires it.

If mixed-version pairing is not supported, say so up front and schedule a window with a single-unit service period. That is a planning decision; discovering it mid-upgrade is an outage.