A Saturated CDN Peer and the Limits of Prepend and MED
A wholesale provider has four transit providers and 14 private peers across two IXPs. Egress selection follows the usual hierarchy: customer routes first, then peers, then transit, with IGP metrics breaking ties inside the peer tier. One IXP peer, AS 64500, delivers a large video CDN and accounts for 38 percent of traffic; its 2x100G LAG runs at 42 to 60 Gbps at peak while two transit links sit at 20 percent utilization. Last quarter an engineer prepended AS 65001 four times toward AS 64500 and set a lower MED on that session; traffic did not move. A second attempt lowered the IGP metric on the transit-facing links, which shifted about 12 Gbps but caused a three-minute outage when 1.2 M BGP routes were re-evaluated and RSVP-TE LSPs re-optimized. Change control now demands a rollback plan and impact analysis for anything touching IGP or BGP policy.
1. Determine why the prepend and MED attempt produced no change. State what information you need before choosing any lever.
Read the worked answer
The first question is direction. The congestion is on the provider's egress toward AS 64500, so the lever must change the provider's own outbound bestpath selection. Prepend and MED are inbound-traffic tools: prepending makes AS 65001 look less attractive to AS 64500, and MED is a hint to a neighboring AS about preferred entry points. Neither changes what the provider itself selects for traffic leaving toward the CDN. If outbound selection is what is broken, prepend and MED were aimed at the wrong side of the conversation.
Second, even for inbound traffic the tools have narrow scope. MED is only compared between routes received from the same neighboring AS; it is not a cross-peer or cross-transit tiebreaker, and many networks reset or ignore it at their boundary. Prepend length competes against the peer's own local preference, and a large content network with a settled inbound policy may not re-optimize for four prepends. Without knowing AS 64500's inbound policy you cannot predict the result.
What you need before acting: a per-prefix and per-peer traffic matrix showing which direction each congested flow takes; the router's actual bestpath for the CDN prefixes (local preference, AS path length, MED, IGP cost to next hop); the peer's contracted capacity and any peering agreement that specifies traffic ratios; and the list of CDN prefixes, preferably validated against RPKI or IRR data so a policy match is precise.
2. Evaluate the IGP-metric approach that was tried and propose a targeted alternative consistent with change control.
Read the worked answer
IGP metric tuning is the bluntest instrument available in a service provider network. It does shift egress, but it shifts everything: all traffic whose tiebreak is IGP cost moves at once, not just the CDN prefixes. In an MPLS backbone it also moves the LDP and RSVP-TE shortest path, so label-switched paths are re-signaled and can re-optimize, which is the likely source of the three-minute outage when 1.2 M routes were re-evaluated. It also risks asymmetric forwarding, where return traffic arrives on one path and leaves on another, which breaks stateful edge devices and complicates capacity planning.
The targeted alternative is BGP policy at the edge, applied to a named prefix set:
- Build a prefix-list or as-path access-list for the CDN, cross-checked against IRR/RPKI so the match set is defensible.
- Tag those prefixes with a community at ingress (for example 65001:900) using an inbound route-map, then apply local preference in the egress policy to prefer the intended exit, or use a lower local preference on the congested peer session for that community only.
- Leave the IGP untouched, so LDP/RSVP-TE topology and all other traffic are unaffected.
- Where BGP policy cannot express the requirement (per-flow or per-application steering), use SR-TE or policy-based routing for that specific class rather than reshaping the IGP.
- In parallel, open a commercial conversation about a port upgrade or additional peering capacity; engineering around a 60 Gbps peak on 2x100G is a stopgap, not a design.
The policy approach changes a bounded prefix set, is reversible with a route refresh, and has an impact statement that reads "no IGP or TE change".
3. Write the rollout and rollback plan you would present to change control, including the monitoring that proves the change did what was intended.
Read the worked answer
Pre-change evidence: capture per-prefix next hops and byte counters for the CDN prefix set on all peering and transit edge routers, plus per-link utilization sampled at one minute for 24 hours, so the baseline reflects the actual diurnal peak rather than an average.
Staging: render the policy in the source-of-truth template, diff it against running config, and test in the lab or digital twin against a session carrying the same prefix count. Apply to one edge router that carries a minority of the CDN traffic, during a window, with the old configuration saved and a one-command revert prepared. Verify with show bgp <prefix> that the next hop changed as intended, that the peer session stayed up, that no unexpected prefixes matched the community, and that the reverse path is unchanged.
Rollback criteria, agreed in advance: peer or transit session reset, any prefix outside the match set changing next hop, packet loss above the baseline on the monitored flows, or CPU on the edge router above its normal policy-application ceiling. Rollback is a config replace plus route refresh, not a reboot, and it must be timed in the lab.
Monitoring after the change: per-link utilization on the peer LAG and the transit links at one-minute resolution, CDN prefix next-hop distribution, BGP update rate for five minutes after applying policy, and end-user latency or video startup metrics if available. Hold the change for one full peak cycle before declaring success, and record the achieved shift against the 12 Gbps the IGP change produced, so the next decision is made on measured results.