MCLAG pair loses half the rack when the ICL flaps

Why switch 2 blocked An MCLAG pair presents one logical switch to downstream devices, and it can only do that while the two peers can talk over the ICL. If the ICL drops and both peers keep forwarding independently, you get duplicate BUM traffic, MAC flapping between chassis, and in a routed design two active gateways - a split-brain. The MCLAG implementation therefore protects itself: when the peer relationship is lost, the secondary chassis disables (or blocks) its MCLAG member ports, while the primary keeps forwarding. Switch 1 staying up and switch 2 blocking is the designed split-brain protection, not a fault. The three flaps in 40 seconds meant this happened repeatedly, so ports bounced rather than settling.

Why 40-60% loss on switch 1's hosts The VMware bond is IP-hash, not LACP. IP-hash hashes flows across the two NICs and has no standard signalling to detect that the peer switch port is blocked - the host's NIC on switch 2 still shows link up, because the physical link never went down. So roughly half of the host's flows (by hash) were black-holed into switch 2's blocked ports while the other half went out through switch 1. That is exactly the 40-60% signature, and it persists until the hash changes or the port returns.

Why switch 2's hosts lost everything Both NICs of those hosts land on switch 2's member ports (or their second NIC's peer was also affected), so when the secondary blocked its MCLAG ports the hosts had no forwarding path at all.

Change board answer: the switches did not misbehave; the design had a single ICL and a bond that cannot detect a blocked peer. The failure was in redundancy design and in monitoring, not in MCLAG.

Immediate (this week, no downtime)

1. Replace the failing DAC and re-seat/clean the optics; capture pre-replacement CRC and input-error counters on ports 51/52 so you can prove the fix afterwards (counters must stop incrementing, not merely reset).
2. Make the ICL a two-member LAG on ports from different ASIC groups where the platform allows it, so a single transceiver or ASIC cannot isolate the chassis pair. Confirm both members are up and the ICL is forwarding before touching anything else.
3. Align the host bond with the switch: move VMware to LACP (IP-hash with LACP, or route-hash with LACP) so the host reacts to LACP state instead of hashing blindly into a blocked port. Do one host first, verify with a failover test, then roll.
4. Monitoring: alert on ICL member state change, LACP state change, CRC/input errors, and MCLAG member-port blocking events. A 40-second ICL flap that only surfaces as user complaints is a monitoring gap.

In the maintenance window (planned outage)

1. Change-failover test: administratively disable one ICL member; the pair must stay converged with no host loss. Re-enable, then repeat for the second member. Then test a full ICL outage and record the expected behaviour (secondary blocks) so the next on-call engineer recognises it.
2. Confirm MTU consistency across ICL, MCLAG members, and host ports if jumbo frames are in use - an MTU mismatch on a new member is the classic reason a repaired ICL 'comes up but drops'.
3. Document the design limit: with a single ICL, half the rack's ports are designed to block during an ICL failure. If that is not acceptable for the business, the answer is a second ICL path or a different topology, and that is a budget decision, not a config tweak.

Proof of completion: zero ICL state changes for 14 days, zero incrementing error counters, one successful planned ICL-member failover with no application impact, and an alert that fired during the test.

At 02:12 - in this order, because each step narrows the next

1. ICL state first. Check trunk membership and per-member status on both chassis: are both members up, is one flapping, is there a LACP/static mismatch, and are CRC/input errors climbing? A flapping member with rising CRC is a physical-layer fault until proven otherwise.
2. MCLAG role and peer state. Confirm which chassis is primary and whether the peer relationship is up or down. Peer down with secondary member ports blocked is the expected split-brain protection, not a new fault - do not start changing VLANs or trunks.
3. Member trunk state per host. For each MCLAG member trunk, confirm the member ports on the primary are forwarding and note which ones are blocked on the secondary. Map that back to each host's bond so you know exactly which hosts have one live path, two live paths, or none.
4. MAC address table symmetry. Compare the MAC table on both chassis for the server VLAN. MACs learned on the wrong side, or the same MAC alternating between chassis, confirms a peer/forwarding inconsistency rather than a server problem.
5. Only then look at the hosts - ESXi NIC teaming state, which uplink is used per vSwitch, and whether the bond reported a failover. If the switch side is consistent and the host still hashes into a blocked port, the fix is on the host, not the switch.

Evidence to capture before you fix anything

• Timestamped trunk and MCLAG state from both chassis, including the three flaps.
• Port error counters (CRC, input errors, discards) on both ICL members and all affected host ports.
• MAC table snapshots per VLAN from both chassis.
• Switch event log and syslog extracts for the 02:10-02:14 window, plus any SNMP traps.
• Host-side bond configuration and vSwitch failover events.
• The exact recovery sequence: did it recover because the ICL came back, or because something on the host side rehashed?

Pitfall: do not 'fix' the blocking on switch 2 by forcing ports up while the ICL is unstable - that reinstates the split-brain the mechanism exists to prevent, and turns a four-minute outage into a broadcast storm with MAC instability across the rack.