MPLS Flap, Backup Tunnel Up, Nobody Online
An HQ site runs an MPLS primary (10 Gbps) and an internet IPsec backup. The MPLS provider runs BGP; HQ advertises an aggregate 10.10.0.0/16 and receives a default. The backup is a static default to the ISP gateway with a higher administrative distance plus a route-based tunnel. Internet-bound source NAT uses a dynamic IP-and-port pool bound to the MPLS egress interface. When the MPLS circuit flapped for nine minutes last week, browsing failed for the first 45 seconds, then users reached some sites but many sessions stalled mid-transfer. The backup tunnel reported up with 0% loss, yet almost no traffic traversed it. Return traffic was observed arriving on the tunnel interface and leaving via MPLS. Branch OSPF adjacencies dropped for 40 seconds after the flap.
1. Account for the 45-second black hole. Where did the time go and how do you remove it?
Read the worked answer
Ninety percent of this is a design choice, not a bug. The static backup route has a higher administrative distance, so it only enters the table after BGP withdraws - that is the intended behaviour, but it means the backup path was never kept warm.
Where the 45 seconds go
- Route withdrawal and reconvergence. The provider's BGP timers plus local processing decide when the default disappears.
- Dead peer detection on the tunnel. If the backup tunnel is idle, its SA may have expired or DPD may need several retries before IKE re-establishes. IKEv2 with a fresh DH exchange plus child SA setup is realistically 10-40 seconds.
- ARP and session setup on the new egress, plus every existing session being invalid because its egress interface changed.
How to remove it
- Keep the tunnel alive with DPD at a short interval and a matching child SA lifetime, so there is nothing to rebuild on failover.
- Do not let a static route be the failover mechanism if you can avoid it: run the tunnel as a second BGP path (or a floating static tied to path monitoring of an MPLS-only target), with BFD on the MPLS session so withdrawal is sub-second.
- Measure it rather than guess: timestamp the last successful packet, the route-table change, the IKE rekey, and the first successful packet after recovery. That tells you which of the three stages to attack.
2. Why did sessions stall after the route converged, and how does NAT fit in?
Read the worked answer
Two independent faults compound here.
NAT binding. The source NAT rule is tied to the MPLS egress interface. When traffic leaves via the tunnel interface, that rule does not match, so packets depart with 10.10.x source addresses and are dropped or never return. Users get the classic 'it connected, then hung'. Fix: express NAT by source zone/address with the tunnel interface included as an egress, or use interface-independent NAT with a pool per egress so each path translates to an address that can actually return.
Asymmetry. You observed return traffic on the tunnel leaving via MPLS. The firewall is stateful: a packet arriving on the tunnel that belongs to a session established out the MPLS interface is discarded as out-of-order. Even correct NAT will not save you if the return path is chosen by a different table than the forward path.
How to make it symmetric
- Ensure that while MPLS is down, the tunnel is the only viable default - no partial route for 10.10.0.0/16 pointing at MPLS remains in the table.
- NAT to the egress interface's own address on each path, so return traffic is naturally drawn back to the same interface.
- Verify with a traceroute in both directions and a session-table dump showing the same ingress and egress interface for forward and return packets.
3. How would you make failover deterministic and observable for the next flap?
Read the worked answer
Move from 'it usually comes back' to a designed and monitored transition.
Design
- BGP for both paths, or a floating static plus path monitoring whose target is reachable only over MPLS - monitoring a host that is reachable both ways proves nothing.
- BFD on the MPLS session; if unsupported, timers no slower than hello 300 / hold 900 ms on the LAN-facing adjacencies and the provider's agreed values on WAN.
- Preempt/failback on a hold-down (e.g. 5-10 minutes stable) so a flapping circuit does not cause repeated transitions.
- Explicitly decide whether the tunnel carries all traffic during an outage or only selected applications; a partial-coverage backup is what produced the 'some sites work' symptom.
Observability
- Alert on: route-table change, tunnel state change, NAT pool exhaustion, and a synthetic probe to three external targets every 30 seconds.
- Log the failover event with the triggering reason (BFD down, path monitor down, interface down) so the post-incident review does not depend on memory.
- The 40-second OSPF drop is a separate tuning item: confirm whether the branch adjacency failed because the tunnel carried the OSPF hello and then stopped, and if so keep routing protocol traffic off the backup path.