Two Route Reflectors, 2.09 Million Prefixes, and a 90-Second Failover
Two route reflectors on 16 GB, 8-core appliances serve 900 PEs in a single cluster: 1.35 M IPv4 unicast prefixes, 260,000 VPNv4, and 480,000 IPv6. Each PE peers only with the two RRs. During quarterly policy changes the BGP process on the RRs reaches 100 percent CPU, sessions to about 40 PEs reset, and memory climbs to 14 GB. A full RR failover takes roughly 90 seconds: 900 sessions re-establish and the surviving RR must regenerate about 2.09 M prefixes toward 900 peers. The business plans to add 300 PEs and wants BGP add-path for a 5G slicing use case that requires two diverse paths per prefix. No new hardware is approved for nine months, and no change may require a maintenance window on more than 20 PEs at a time.
1. A colleague proposes a third route reflector on identical hardware. Explain why that alone does not fix the problem, and name the actual scaling bottlenecks.
Read the worked answer
A route reflector is a fan-out engine. Its dominant cost is not receiving 2.09 M prefixes once; it is generating and maintaining an Adj-RIB-Out of roughly the same prefix count for every one of 900 peers, which is on the order of 1.9 billion prefix entries plus their path attributes. Adding a third RR splits the client set, which improves failover time and redundancy, but every RR still holds the full Loc-RIB and still builds a per-peer outbound table for its share of clients. Per-peer memory on a 16 GB box does not shrink just because there are three boxes; if a single RR keeps 450 clients at 2.09 M prefixes it is barely better off than today.
The real bottlenecks are:
- Outbound update generation, especially on a policy change, where every affected prefix must be re-evaluated and re-advertised per peer. That is the 100 percent CPU event and the session resets.
- Adj-RIB-Out memory scaling linearly in peers x prefixes, amplified by add-path (2 paths can mean 2x the outbound entries).
- Bestpath computation and inbound policy for 900 sessions on a single process.
- VPNv4 without RT constraint, where every PE that participates in any VPN is a candidate for every VPN route.
So the answer is to shrink what the RR holds and sends, not just to add boxes.
2. Design an interim plan that fits the no-new-hardware and 20-PE window constraints. Explain which change gives the largest reduction and why.
Read the worked answer
Sequence the changes by bytes removed per unit of risk.
- Stop sending the full internet table to PEs. The 1.35 M IPv4 prefixes exist on the ASBRs, not on the PEs. If edge PEs receive a default plus customer and infrastructure routes, the RR's IPv4 Adj-RIB-Out collapses by well over an order of magnitude. This is the single largest win and it is a client-side inbound policy change, so it rolls 20 PEs at a time without touching the RRs.
- Enable RT constraint on the RRs and PEs for VPNv4. PEs then advertise RT membership and the RR only forwards VPN routes a PE actually imports, which typically removes most of the 260,000 VPNv4 prefixes from each PE's outbound table. Do this before adding any new PEs.
- Split address-family roles across the two boxes logically: VPNv4 RR pair on one, IPv4/IPv6 on the other, so a policy change in unicast cannot disturb VPN sessions. Accept the reduced redundancy until hardware arrives.
- Defer add-path. For the slicing requirement, BGP PIC edge (or per-VRF multipath where supported) already provides a pre-installed backup path; full add-path multiplies outbound state and should wait until the table reduction is measured.
- Add max-prefix and per-VRF route limits with alarms so the next growth step is visible before memory reaches 14 GB again.
Measure Adj-RIB-Out memory per peer after step 1 and 2 before committing to 300 new PEs; the projection, not the plan, decides whether 300 is feasible.
3. How would you introduce add-path and the 300 additional PEs while honoring the 20-PE maintenance window, and what would you monitor to know it is safe?
Read the worked answer
Order matters: reduce, then add, then multiply.
First finish the table reduction (default-originate to PEs, RT constraint). Record baseline per-peer memory, RR CPU at idle and during a full route refresh, and convergence after an RR restart. Without those numbers you cannot tell whether add-path is affordable.
Then enable add-path in stages, and only for the address family the slicing use case needs (probably VPNv4), not globally. The RR and the receiving PEs must both support and be configured for additional paths; a mixed state simply falls back to a single best path, which is safe but gives no benefit. Restrict the additional-path selection to the paths that matter (for example, two paths from distinct PEs in distinct IGP regions) rather than advertising every path, and cap it at two. Roll it to 20 PEs, wait a full business cycle, then continue.
For the 300 new PEs, onboard in batches matched to the RRs' measured capacity, using the same template so that inbound/outbound policies and max-prefix values are identical and auditable.
Monitor: RR BGP process memory and headroom, Adj-RIB-Out entries per peer, update-generation rate and CPU during a scheduled route refresh, peer flapping counters, and the time to re-establish 900 sessions after a controlled RR restart. Define a stop condition in advance, such as memory above 80 percent or failover above 30 seconds, and roll back to single-path before the next batch.