BFD Flap on a Healthy Underlay Blackholes VXLAN Traffic
A 4-spine, 16-leaf EVPN-VXLAN fabric uses an eBGP underlay. Each spine is in AS 65000, each leaf is in a unique private AS, and every leaf advertises its VTEP loopback /32. BFD is enabled on every spine-leaf session with a minimum interval of 300 ms and multiplier 3. During a batch job, spine-02's control-plane CPU reaches 98% for 40 seconds while its PFE continues to forward at line rate. BFD sessions on 12 leaves flap repeatedly. BGP withdraws spine-02's loopback, and each leaf's ECMP set shrinks. Some leaf-to-leaf VXLAN flows blackhole for 2-4 seconds per flap even though the physical links stay up. Non-VXLAN traffic does not show the same loss. The network was stable for months; the only recent change is a new telemetry collector subscription on spine-02.
1. What is the failure mechanism, and why does data-plane traffic fail if the PFE is healthy?
Read the worked answer
BFD is a control-plane liveness check. If the routing engine cannot schedule BFD hellos because telemetry processing is consuming CPU, the multiplier expires even though the PFE is forwarding. BGP then withdraws spine-02's loopback, so every leaf removes that spine from the ECMP set for VTEP reachability. VXLAN traffic that was hashed to spine-02 is dropped until BGP reconverges; the 2-4 second blackhole matches the withdrawal and reconvergence time. The PFE being healthy is not enough, because the route is gone from the control plane. Confirm with show bfd session extensive on several leaves for interval, multiplier, and last-down reason; show log messages for BFD and BGP events; show chassis routing-engine and show system processes extensive on spine-02 around the event. Check whether the telemetry subscription is exporting a large sensor path at a short interval. If the CPU spike and BFD flaps align, you have a false positive caused by control-plane overload, not a link failure.
2. What changes would you make to BFD and BGP to prevent false positives without hiding real failures?
Read the worked answer
Tune BFD to the control plane, not to the link speed. A 300 ms x 3 timer is aggressive for a routing engine that also handles telemetry, BGP, and EVPN. Increase to 500-1000 ms with multiplier 5-8, add no-adaptation so the timers stay predictable, and add a hold-down or dampening so a single missed hello does not immediately withdraw the route. Enable BGP graceful restart and long-lived graceful restart so forwarding can continue during a control-plane restart; ensure the helper mode is supported on the leaves. Rate-limit or move the telemetry subscription off the critical RE path, and avoid exporting high-cardinality sensors at sub-second intervals. Do not disable BFD globally; that hides real failures. Also consider BFD echo mode if the platform supports it, because echo tests the forwarding path rather than only the control plane. Validate the new timers in a maintenance window by loading the RE and confirming that BFD stays up while a real link failure still triggers within the desired detection time.
3. How would you confirm whether the telemetry change caused the CPU spike, and what monitoring would you add?
Read the worked answer
Use show chassis routing-engine for CPU history, show system processes extensive and monitor top to identify the telemetry process, and show services analytics or the equivalent sensor configuration to see the subscription. Compare the CPU before and after the new collector was added; if the spike starts with the subscription, disable it and retest. Add streaming telemetry for routing-engine CPU, BFD session state, and BGP peer state, but export at a sane interval and from a collector that does not overload the device. Create an alarm on BFD session down and on RE CPU above 80% for more than 60 seconds. Use commit confirmed for the telemetry change and keep a rollback. The goal is to distinguish a real forwarding failure from a control-plane scheduling artifact before changing BFD timers.