EVPN ESI-LAG Blackhole After Leaf Reboot
A 2-spine, 4-leaf EVPN-VXLAN fabric uses QFX5120 switches. Leaf L1 and leaf L2 are in the same rack and dual-home a server with an all-active ESI-LAG. The server bonds two 25G links, one to each leaf. BGP is iBGP for the EVPN overlay; hold-time is 90 seconds, keepalive is 30 seconds, and BFD is not enabled. OSPF is the underlay and converges in about 2 seconds. L1 reboots for 6 minutes. During the reboot, server traffic to a remote VLAN 200 subnet blackholes for about 90 seconds, then recovers. L2's ESI state remains up, and the server's LACP link to L2 stays up. On a remote leaf, show route table bgp.evpn.0 still lists a Type-2 route from L1 during the blackhole.
1. Explain why ESI-LAG did not provide sub-second failover and why the outage lasted about 90 seconds.
Read the worked answer
ESI-LAG protects the server's local attachment, but it does not bypass EVPN route convergence on remote leaves. When L1 reboots, OSPF withdraws the underlay route to L1's loopback in about 2 seconds. However, the remote leaf still has the EVPN Type-2 MAC route from L1 because the BGP session has not yet timed out. The BGP hold-time is 90 seconds, and without BFD the remote leaf has no fast failure signal. It continues to encapsulate traffic toward L1's VTEP, but the underlay route to that VTEP is gone, so the packets are dropped. After the hold-time expires, the Type-2 route from L1 is withdrawn and traffic uses L2 or floods, so connectivity recovers at roughly 90 seconds.
The server's LACP link to L2 staying up is expected and does not fix the remote leaf's stale EVPN route. The fix is faster failure detection for BGP, not a larger ESI hold timer.
2. Design the failure-detection change for BGP and the underlay. Include a practical timer choice and explain the trade-off.
Read the worked answer
Enable BFD for the BGP EVPN sessions between leaves and spines, and consider BFD or fast hellos for OSPF underlay. A practical starting point is BFD minimum-interval 300 ms with multiplier 3, giving roughly 900 ms detection, plus BGP keepalive 10 and hold-time 30 as a fallback. Example:
set protocols bgp group EVPN bfd-liveness-detection minimum-interval 300
set protocols bgp group EVPN bfd-liveness-detection multiplier 3
set protocols bgp group EVPN keepalive-interval 10
set protocols bgp group EVPN hold-time 30
Apply on both ends. The trade-off is that aggressive BFD can flap under RE CPU pressure, especially during a reboot or high control-plane load. Do not use 50 ms × 3 on a shared RE without testing. Validate CPU and BFD stability before tightening timers further. If the spines are third-party, confirm BFD echo and authentication interoperability.
3. What would you verify after the change, and what residual risk remains?
Read the worked answer
Verify that BFD sessions are up and that BGP detects a leaf failure quickly:
show bfd session
show bgp neighbor
show evpn database
show route table bgp.evpn.0
show log messages | match BFD
Perform a controlled test by disabling the EVPN BGP session or rebooting a lab leaf, then measure how long the remote leaf keeps the Type-2 route. Confirm traffic uses the remaining ESI member. Residual risk: BFD can cause a false positive if the RE is too busy; keep a fallback BGP hold-time and monitor BFD flaps. Also ensure the underlay route withdrawal and EVPN route withdrawal are both fast; fast BGP without underlay convergence can still blackhole briefly.