I85PREP
⌘ K
CNEn study materials
CNEn study resources

Dual-Homed BGP with a Provider-Assigned /24

CNEn · Put your knowledge to work in context.

All case studies

Dual-Homed BGP with a Provider-Assigned /24

A regional SaaS provider runs 203.0.113.0/24 from AS 65010 and is dual-homed. ISP-A (AS 64500) provides 200/50 Mbps and the address space is provider-aggregatable space allocated by ISP-A. ISP-B (AS 64501) provides 100/100 Mbps. Outbound traffic splits roughly 70/30 after local preference 200 toward ISP-A, but inbound connections to the public VIP still arrive almost entirely through ISP-A. Since ISP-B was added, users report random 20–30 second stalls during HTTPS uploads, while downloads and ping are clean. The stateful firewall pair sits behind both border routers on one shared outside VLAN and performs static NAT to an internal load balancer. ISP-A sends a full table of about 950,000 routes; ISP-B sends only a default. ISP-B's access link uses PPPoE with a 1492-byte MTU; ISP-A is 1500. The firewall blocks all ICMP except echo.

1. Why does inbound traffic prefer ISP-A, and what does that mean for inbound traffic engineering?

Read the worked answer

Because the /24 is ISP-A's PA space, ISP-B cannot originate it. ISP-B learns 203.0.113.0/24 from ISP-A (or from its peers via ISP-A), so its AS path is at least 64501–64500, while ISP-A advertises it as a directly connected customer prefix. Best-path selection on the global internet therefore lands on ISP-A almost everywhere.

Consequences for traffic engineering:

  • AS-path prepending on ISP-A cannot move inbound traffic to ISP-B. Prepending makes the ISP-A path longer, but the alternative through ISP-B still traverses ISP-A, so the traffic either stays put or takes a worse path.
  • Advertising a /25 through ISP-B is the wrong fix. Most providers filter prefixes longer than /24, it fragments the prefix in a way that breaks reputation and geolocation, and partial deployment produces unpredictable results.
  • MEDs are not comparable across providers and ISP-B has no route of its own to prefer.

Realistic options: obtain PI space and renumber (a project with a real cost and DNS/TLS dependency), front the service with a global load balancer or anycast, or accept ISP-B as outbound-plus-standby and design for that explicitly. The judgement call is to stop tuning BGP attributes and fix the addressing model.

2. Diagnose the 20–30 second HTTPS upload stalls and specify the fix, including where it must be applied.

Read the worked answer

The pattern - small packets and downloads fine, large uploads stalling, and only on a fraction of flows - points to a path-MTU black hole, not to congestion or BGP instability.

Reasoning chain.

  1. Outbound is per-flow load-shared 70/30, so roughly 30% of upload sessions egress ISP-B.
  2. Those sessions build a 1500-byte IP packet with DF set and hit the PPPoE 1492-byte link.
  3. The ISP-B router should return ICMP Fragmentation Needed, but the firewall drops all ICMP except echo, so PMTUD fails.
  4. TCP retransmits the same oversized segment several times before the connection recovers or resets; 20–30 seconds is a typical retransmission backoff pattern. Flows hashed to ISP-A are clean, which explains the randomness.

Fix.

  • Clamp TCP MSS on the ISP-B CE interface to 1452 (or 1400 for headroom) in both directions; this is applied at the ISP-B edge, not at the server.
  • Permit ICMP type 3 code 4 inbound on both firewall outside interfaces. Blocking all ICMP is the root enabler here.
  • Verify with a capture on the CE: look for repeated identical-length segments from the client, then confirm they shrink after clamping.
  • Consider policy-routing upload traffic from the public VIP subnet to ISP-A until the MSS fix is validated, so the user-visible symptom stops during the change.

Do not treat this as a BGP problem: the routing table is stable, and the clean download path proves the session itself establishes.

3. What long-term design changes would you recommend, and what are the trade-offs?

Read the worked answer

Option 1 - Renumber to PI space and run true multihoming. Advertise a PI /24 to both ISPs, use local preference for outbound and MED plus selective prepending for coarse inbound control. Pros: both links usable in both directions, clean failover, no provider lock-in. Cons: renumbering a public VIP fleet touches DNS TTLs, TLS certificates, customer allow-lists, and any partner integrations; plan a 6–12 month dual-addressing period and expect a small share of hardcoded clients to break.

Option 2 - Keep the PA /24 and accept asymmetric inbound. ISP-A carries inbound; ISP-B carries outbound only. Pin long-lived upload flows to ISP-A and keep ISP-B for failover. Pros: no renumbering, low cost, predictable. Cons: ISP-B's 100 Mbps upload capacity is largely wasted, and an ISP-A inbound outage takes the service down regardless of ISP-B.

Option 3 - Put a global load balancer or anycast front end in front. Health-checked endpoints across both providers make inbound independent of BGP path selection. Pros: real inbound resilience and better global latency. Cons: added cost and a new operational dependency.

Whichever option is chosen, add BFD on both eBGP sessions, prefix limits, and a maximum-paths policy so that a flapping session does not cause repeated reconvergence. The recommendation for a SaaS provider with paying customers is Option 3 plus Option 1 in parallel, retiring the PA space as contracts allow.