Failover Takes 90 Seconds and the Tunnel Comes Back Half-Dead
A hub firewall terminates 40 route-based IPsec tunnels (IKEv2, PSK, /24 selectors) from branch offices and runs BGP with the branches over the tunnels. Branch 7 has a 150/150 Mbps fiber primary and an LTE backup behind carrier-grade NAT; the LTE tunnel is a secondary path with worse BGP local preference. When fiber fails, Branch 7 fails over in about 90 seconds; during that window voice calls drop and several TCP sessions reset. Over the LTE path, the hub cannot initiate connections to branch devices, and large HTTPS uploads stall at roughly 1400-byte segments. After failover the tunnel sometimes shows as established but no traffic passes until it is manually reset.
1. Explain the 90-second failover and why sessions reset. What would you change to shorten it, and what cannot be fixed at the network layer?
Read the worked answer
Ninety seconds is a timer stack-up, not a link-detection problem. Typical contributors: DPD at 30 s interval with 3 retries (90 s before the peer is declared dead), BGP keepalive 60 / hold 180, and IKE SA rekey or stale SA cleanup adding delay. The fiber interface may go down instantly, but nothing tells the tunnel or BGP that the path is gone until DPD expires.
Changes:
- Tune DPD to something like 5 s interval, 3 retries (15 s detection) on both hub and branch, and make sure both ends send DPD, not just one.
- Tune BGP: keepalive 5 / hold 15, or add BFD if supported, so the route withdraws as soon as the tunnel is declared down. Do not let BGP wait for its own hold timer after DPD has already told you the peer is gone.
- Keep both tunnels up simultaneously where possible (active/active with BGP multipath or floating static with tracking), so failover is a route preference change rather than a new IKE negotiation. Re-negotiating IKE and IPsec from scratch is a large part of the outage.
- Stagger SA lifetimes and rekey margins so a rekey does not coincide with a failover event.
What cannot be fixed: TCP sessions bound to the old path will reset when the source address or NAT translation changes, and voice calls will drop unless the application handles reconnection. IPsec without a mobility/multipath layer cannot preserve a flow across an address change. The honest expectation is sub-15-second recovery with fast DPD/BFD plus pre-established tunnels, with application-level reconnection for voice and long-lived TCP.
2. Why can the hub not initiate to Branch 7 over LTE, and how should the design account for it?
Read the worked answer
Branch 7's LTE connection is behind carrier-grade NAT, so the branch has no publicly reachable address or stable port. The hub cannot send IKE to it, and once an SA exists the branch's NAT binding must be kept alive or the mapping expires and inbound packets are dropped. Any design that assumes the hub can poll, probe, or push to the branch over the backup path will fail.
Design implications:
- The branch must always be the IKE initiator for the LTE path, and the hub must be configured to accept and match that peer (PSK/identity based on ID, not source IP, since the public IP is unpredictable).
- Enable NAT-T (UDP 4500 encapsulation) and set keepalives short enough to hold the binding open - typically 10-20 s - and make sure the hub honors them.
- Hub-side monitoring must be passive or branch-initiated: reverse-direction probes, or a branch-side agent that reports health, rather than hub-initiated ICMP/SNMP to the branch. Otherwise the hub will declare the branch dead and the monitoring itself is wrong.
- Log the observed public IP/port and do not build ACLs on it; it can change.
Also confirm the LTE provider does not block UDP 500/4500; if it does, the backup path needs a TCP-based alternative or a different transport.
3. The tunnel is 'up' but passes no traffic after failover, and uploads stall at ~1400 bytes. Diagnose both and give the fix.
Read the worked answer
Two separate issues.
Half-dead tunnel: the IPsec/IKE SA survived the path change but the state is stale - the hub still has a route or SA referencing the old path, or the branch's NAT binding changed and the hub is sending to the old UDP port. Symptoms match because control-plane liveness (DPD) may be satisfied by keepalives while data-plane selectors or routing point at the wrong next hop. Fixes: enable DPD on both ends with short timers so a stale SA is torn down rather than lingering; ensure there is only one active route per prefix and that BGP withdrew the old one; verify SA selectors and traffic selectors on both sides match the actual subnets; consider shortening SA lifetime/rekey margin so a stale SA ages out quickly. Add a post-failover check that validates data-plane reachability (e.g., a probe sourced from the tunnel interface), not just IKE state.
1400-byte stall: MTU black hole again. LTE paths commonly have 1400-1420 usable MTU after encapsulation. When IPsec (and NAT-T, adding more header) is layered on a 1400-byte path, the effective MSS must drop. Set MSS clamping on the tunnel interface: for a 1400-byte path, MSS = 1360; if IPsec/NAT-T overhead pushes the usable MTU to around 1360, clamp to 1320. Also verify the DF-set probe through the tunnel, not just to the tunnel endpoint. Confirm by capturing on the tunnel: repeated retransmission of a maximum-size segment with no ACK, while the handshake completes. Fix is MSS clamp plus, if the platform allows, setting the tunnel MTU to the measured value.