BGP Route Reflector Redundancy and a Silent Session Failure
An Arista EVPN/VXLAN fabric uses two spine route reflectors (RR-1, RR-2) for the underlay and overlay BGP. All 20 leaves peer with both RRs. A maintenance task reboots RR-1. During the reboot, monitoring shows that 6 of the 20 leaves lost all BGP sessions to both RRs simultaneously for about 90 seconds, causing EVPN route withdrawal and brief tenant outages on those leaves. The other 14 leaves were unaffected. RR-2 stayed up the entire time. The team assumed dual RRs provided redundancy and is confused why six leaves lost both sessions. The six affected leaves are all in the same rack and share a PDU.
1. Given RR-2 never went down, what is the most plausible explanation for six leaves losing both BGP sessions at once?
Read the worked answer
The common factor is physical/power, not BGP: the six leaves share a PDU in one rack. The most plausible cause is that the maintenance event (or an unrelated power event) interrupted that PDU, power-cycling the six leaves. When a leaf reboots, it loses all BGP sessions - to RR-1 and RR-2 - regardless of RR redundancy, because the sessions are local to the leaf. RR-2 being up is irrelevant if the leaf itself is down. The 14 unaffected leaves are on different power. This is a correlated-failure/domain-of-failure problem: dual RRs protect against RR failure, not against leaf power loss. The 90-second outage matches a leaf reboot plus BGP/EVPN reconvergence, not an RR-driven reconvergence.
2. What design and operational changes would reduce the blast radius and detection time for this failure mode?
Read the worked answer
Design:
- Diversify power domains: dual PDUs / A-B feeds per rack so a single PDU event cannot take out a full set of leaves; distribute leaves across racks/power zones so a rack event is not a correlated leaf outage.
- Confirm RR redundancy is real: ensure each leaf's two RR sessions are on diverse paths and that BGP timers/
bfdare tuned so a genuine RR failure reconverges fast (BFD for sub-second detection). - EVPN fast convergence: verify
bgp additional-paths,next-hoptracking, and that leaves have redundant uplinks to both spines so a single leaf reboot doesn't isolate tenants longer than necessary.
Operational:
- Correlate alerts: monitoring should flag simultaneous multi-leaf session loss as a likely power/rack event, not 20 independent BGP problems.
- Maintenance awareness: before rebooting RR-1, check whether any PDU/power work overlaps; schedule power events separately from control-plane maintenance.
- Post-incident: pull PDU and switch boot logs to confirm the reboot hypothesis rather than assuming a BGP bug.