ESI-LAG Blackhole After Leaf2 Upgrade
A dual-homed database server connects with a 2-member ESI-LAG to leaf1 and leaf2 in an EVPN multihoming all-active fabric. The server uses VLAN 110, VNI 5010, and sends both unicast and multicast traffic. After leaf2 was upgraded and rebooted, the server experiences about 50% packet loss to a remote subnet; local ping from leaf1 to the server is clean. show evpn multihoming esi on leaf1 shows leaf1 as designated forwarder for VLAN 110 and leaf2 as non-DF; leaf2 shows the opposite after it comes back. The ESI is 00:aa:bb:cc:dd:ee:ff:00. The upgrade was performed with a single commit and no commit confirmed; the leaf2 configuration was restored from a backup that predates an MTU change on the uplink. The fabric underlay is stable.
1. Distinguish between DF election and ESI mismatch as causes. Which is more likely given the symptoms?
Read the worked answer
The DF election state shown is probably not the primary cause. DF election controls which ESI-LAG member forwards BUM traffic toward the server; it does not explain 50% loss of unicast to a remote subnet if both leaves have consistent ESI and VNI configuration. A 50% loss pattern points to inconsistent ESI or MAC advertisement: leaf2 was restored from a backup that predates the MTU change, so it may also be missing the ESI or have a different ESI or VNI mapping. If leaf2 advertises the server MAC without the same ESI, the remote leaf sees one MAC from two different multihoming groups, treats it as a MAC move, and installs one path at a time; roughly half the flows blackhole until the next move.
Local ping from leaf1 works because leaf1 is local and correctly configured. Check show evpn database extensive for the ESI and sequence number, show configuration interfaces ae0 esi, and compare leaf1 and leaf2. Verify DF state separately with show evpn multihoming esi detail, but do not stop at DF.
2. What verification commands and configuration fixes restore all-active multihoming?
Read the worked answer
First restore parity: confirm the same ESI on both leaves, the same LACP system-id, the same VLAN-to-VNI mapping, and the same MTU on the server-facing AE and the uplink. On leaf2, reapply the missing ESI and MTU change with commit confirmed 5; if the server loses connectivity, the commit rolls back automatically.
Use show lacp interfaces ae0 to confirm LACP is up on both members, show evpn multihoming esi detail to confirm both leaves are in the same ESI and the DF roles are as expected, and show evpn database extensive to confirm the server MAC is installed with the same ESI on both leaves. Then clear the server ARP and test from a remote host, not only from leaf1. If the MAC still moves, check for a duplicate MAC or a stale static MAC on another interface.
3. How would you prevent this during future upgrades?
Read the worked answer
Prevent recurrence with a controlled upgrade workflow. Use commit confirmed for every leaf change, keep out-of-band management, and run pre and post checks for ESI, VNI, MTU, BFD, and EVPN route counts. Automate configuration comparison against a known-good template with PyEZ or Ansible, and fail the job if the ESI or MTU differs. Upgrade one leaf at a time from the same ESI-LAG pair, wait for LACP and EVPN convergence, and only then move to the next leaf.
Take a backup immediately before the change and store it with the running config version, not an old backup. Add a health check that pings a remote subnet from a dual-homed server and verifies that both leaves show the same ESI before declaring success. This turns a stale-restore error into a caught deployment failure.