BGP Route Reflector Redundancy During a Data Center Fabric Upgrade
A financial services firm operates a data center fabric of 24 leaf switches and 4 spine switches running eBGP underlay with iBGP overlay. Two route reflectors (RRs) serve the overlay; both are virtual machines on the same hypervisor host. During a firmware upgrade window, the team plans to upgrade the spines one at a time. Change review flags that the RRs share a host, that the overlay has no route reflector client redundancy policy, and that the maintenance window is only 90 minutes for all four spines. The trading application has a 200 ms failover SLA. A previous upgrade caused a 4-minute outage when both RRs rebooted simultaneously due to a hypervisor maintenance event.
1. What is the primary availability risk in this design, and what should the architect require before the upgrade proceeds?
Read the worked answer
The primary risk is correlated failure of both route reflectors. They are VMs on one hypervisor host, so a single host maintenance event, storage outage, or hypervisor reboot takes out both RRs at once - exactly what happened before. The 4-minute outage exceeded the 200 ms SLA by orders of magnitude, and the 90-minute window leaves no room for a second surprise.
What the architect should require before proceeding:
- Anti-affinity or physical separation. Move the two RRs to different hypervisor hosts (and ideally different racks/power domains) with a VM anti-affinity rule enforced. This removes the single point of failure.
- RR client redundancy. Ensure every leaf peers with both RRs and that the overlay converges without one RR. Verify with a controlled test: shut one RR in a lab or during a low-risk window and measure failover time against the 200 ms SLA.
- BGP timers and graceful restart. Confirm BFD or tuned BGP timers are in place so failure detection is sub-second, and that graceful restart is configured so forwarding continues during RR restart.
- Upgrade sequencing. Upgrade RRs one at a time, verifying overlay stability between each, before touching spines. Then upgrade spines one at a time with a health check gate.
- Window realism. A 90-minute window for four spines plus RR work is tight; either split the change into two windows or have a tested rollback plan.
The exam-style trap is answering "add a third RR" without addressing the shared-host correlation - a third VM on the same host does not help.
2. How should the architect sequence the spine upgrades to preserve the 200 ms failover SLA, and what evidence proves the SLA was met?
Read the worked answer
Sequencing must never remove redundancy from more than one layer at a time, and each step needs a measurable gate.
Recommended sequence:
- Pre-checks. Confirm both RRs are on separate hosts, all leaves have two RR sessions Established, and BFD/timers are tuned. Record baseline convergence time.
- Upgrade RRs first, one at a time. Take RR-A down, confirm all leaves remain Established to RR-B and the overlay is stable, then upgrade and restore RR-A. Repeat for RR-B. This validates RR redundancy before the riskier spine work.
- Upgrade spines one at a time. For each spine: drain it (adjust BGP preferences or use maintenance mode so traffic shifts to remaining spines), upgrade, verify sessions re-establish, then restore it to service. Never upgrade two spines concurrently.
- Health gate between steps. After each spine, check leaf-to-spine session counts, route counts, and application latency. Proceed only if all are within baseline.
Evidence that the SLA was met:
- Convergence telemetry from BGP/BFD logs showing sub-200 ms detection and reconvergence per event.
- Continuous synthetic probes to the trading application throughout the window, with latency and loss recorded at fine granularity.
- Session and route-count dashboards showing no unintended flapping.
- A post-change report comparing measured failover to the 200 ms SLA, retained for audit.
The judgement tested: sequencing changes so redundancy is preserved at every step, and proving SLA compliance with measurements rather than intent.