iBGP Egress Determinism With Three Providers and Two Route Reflectors

This is an iBGP artifact, not an eBGP one. All reflected routes arrive with the same local preference, so best-path selection descends to IGP metric to the next hop, and next hops resolve through Dallas because that is where the reflectors and the border routers sit. Singapore therefore prefers whatever Dallas prefers, regardless of the local provider C session on its own border router.

The fix is inbound policy setting local preference by region, applied before any other attribute matters. Local preference dominates AS-path length and MED, so it must be the primary lever: for example 200 for a locally connected provider, 150 for the regional backup, 100 for anything learned over the WAN. Encode the decision in BGP communities so the policy lives in one place and the same tag drives outbound prepending later. Once set, Singapore picks C locally without needing C's full table - which matters, because C is partial.

Do not confuse this with MED. MED is a hint to a neighboring AS and is not comparable across providers; it cannot create regional determinism inside your own AS.

Verify with a looking-glass equivalent from each region: for a sample of 200 destinations, the chosen egress AS and the next-hop region should match the intended policy, and the check should be automated, not a manual show command during an incident.

No. Two reflectors in one data center is a single failure domain for the entire 12,000-prefix iBGP topology, and it is also the reason every region resolves next hops through Dallas.

Options considered:

• Full mesh: 60 sites is roughly 1,770 sessions; operationally untenable and it does not fix the metric distortion.
• Confederation: adds sub-AS configuration and inter-AS policy for little benefit inside one administrative domain.
• More reflectors, placed regionally: the right answer.

Recommend four reflectors, two per data center, all four in a full mesh among themselves with a distinct cluster ID per pair. Each site peers only with the two nearest reflectors. Use next-hop-unchanged where the IGP carries loopbacks, so traffic does not hairpin through a reflector. Keep reflectors out of the forwarding path as a design rule.

The survival test is concrete: with all Dallas reflectors down, Amsterdam sites must already hold sessions to the Amsterdam reflectors and reconverge in seconds. If a site only has sessions to Dallas, losing Dallas means waiting for TCP hold-down and BGP reconvergence - minutes, not seconds. Schedule the failure test in a maintenance window and measure the actual reconvergence time before declaring the design done.

You cannot advertise 12,000 prefixes to C, so design for aggregation. Advertise a small regional PA block set - six to ten prefixes - with more-specifics only where traffic engineering justifies them. Determinism then comes from your outbound aggregation plus the communities that set local preference on the return path, not from prefix count.

For the partial table, do not treat C as a general transit provider. Accept C's routes only for its own customer cone plus a default, and ensure that default never outranks A's specific routes in Europe by tagging it with a low local preference and matching on the community. Otherwise the partial table creates black holes for destinations C does not carry.

Singapore then uses C for the regional destinations C actually serves and falls back to A for everything else, which is the intended behavior and is stable under C's session loss.

Residual risk worth stating: with only 200 prefixes inbound and a partial table, C is a regional optimization, not a resilience component. The design must remain correct if C disappears entirely, and the test for that is a session shutdown in a lab or maintenance window, not a paper assumption.