EVPN-VXLAN ESI-LAG Blackhole After Leaf Reboot
A media company runs an EVPN-VXLAN fabric with 4 spines and 16 leaves. The underlay is eBGP with default BGP timers and no BFD; the overlay uses iBGP with route reflectors on the spines. There are 240 dual-homed servers using ESI-LAG, and VLAN 120 (rendering) is stretched across all leaves with a distributed anycast gateway. ARP suppression is enabled. After a leaf reboot, servers on that leaf lose connectivity to remote subnets for 60–90 seconds; same-leaf traffic continues to work. Logs show EVPN Type-2 routes withdrawn and re-advertised, and the ESI designated-forwarder election changes. Each leaf holds about 12,000 MAC and 800 ARP entries. The team wants sub-5-second recovery and asks whether to disable ARP suppression, tune BGP, or change the multihoming design.
1. What is the most likely cause of the 60–90 second blackhole, and what evidence would confirm it?
Read the worked answer
The 60–90 second window matches the default BGP hold time, not a normal EVPN MAC withdrawal. Without BFD, the underlay and overlay can take up to 90 seconds to detect the rebooted leaf's session loss and reconverge. During that window, remote leaves may still have stale EVPN Type-2 routes, and local ARP suppression entries can proxy ARP with the anycast gateway MAC while no valid EVPN path exists, so traffic is dropped. The designated-forwarder change affects BUM traffic, but the reported failure is unicast to remote subnets. Confirm with show bgp summary (peer state and uptime), show evpn arp-cache (stale entries pointing to the anycast MAC), show ethernet-switching table (MACs still remote), and EVPN/BGP traceoptions around the reboot. Also check whether the rebooted leaf's ESI-LAG peer took over forwarding correctly; if not, that is a separate multihoming issue.
2. Recommend design changes to meet sub-5-second recovery for dual-homed servers without adding significant hardware.
Read the worked answer
Enable BFD on every underlay eBGP session with 300 ms × 3 (about 1 second detection) and reduce BGP hold time to 9 seconds or less. Keep the EVPN overlay on the same fast-detection behavior by running BFD on the overlay peer sessions where supported, or rely on underlay BFD to withdraw next hops quickly. Ensure the ESI-LAG is all-active, the anycast gateway MAC and IP are identical on both peer leaves, and the peer leaf is preconfigured with the same EVPN instance and VLAN. Add BGP add-path if route reflector withdrawal causes slow re-advertisement of the best path at scale, but first measure whether 12,000 MACs and 800 ARPs justify it. For sub-5-second recovery, the critical path is underlay detection, BGP withdrawal, EVPN route update, and ARP cache refresh; BFD plus a shorter ARP aging timer gets there. Do not rely on changing ESI DF election alone.
3. Should ARP suppression remain enabled on VLAN 120? Explain the trade-off and a realistic policy.
Read the worked answer
ARP suppression reduces ARP flooding and scale on a 16-leaf fabric with stretched VLANs, but it creates stale proxy entries during reconvergence. With anycast gateway, the proxy reply points to the anycast MAC, so the packet depends on a valid EVPN route; if the route is missing, the packet blackholes until ARP ages out or the route returns. For VLAN 120, where rendering workloads move and the team has already seen 60–90 second outages, disable ARP suppression on that VLAN or shorten ARP aging substantially. Keep suppression on stable VLANs with low MAC mobility, such as management or backup networks. If the platform supports it, enable suppression only after BFD and fast EVPN withdrawal are in place, and monitor proxy ARP counts. The design principle is to use suppression as a scale optimization, not as a default on mobility-heavy VLANs.