Branch IPsec Tunnels Rebuilding and Stalling Large Transfers

Ranking (most to least likely):

1. SA lifetime / rekey mismatch between the two ends after the upgrade. Default IPsec SA lifetime is 3600s, so a ~40-minute period means a non-default lifetime was introduced on one side only. When both peers rekey independently you get rekey collisions and a torn-down SA.
2. DPD / IKE keepalive mismatch on the LTE path. NAT-T UDP 4500 state expires on carrier NAT, DPD fires, and the tunnel is rebuilt; this can present as periodic flaps even before failover.
3. HQ HA failover or a single-IP constraint. With one public IP per ISP, both peers can end up pointing at the standby unit after a failover, causing repeated re-establishment.

Evidence before changing anything:

• show crypto ikev2 sa detail and show crypto ipsec sa on both ends at the moment of the flap - compare negotiated lifetimes and SPI counters.
• Timestamped syslog for IKEv2 events; correlate flap times across 3-4 branches to see whether they are synchronized (points to a timer) or random (points to path/NAT).
• A capture on the HQ outside interface filtering UDP 500/4500 for one branch, to see who initiates the delete.
• Confirm the ISAKMP/IPsec lifetimes actually configured on both versions after the upgrade, rather than assuming defaults.

Diagnosis: this is a classic MTU/DF black hole, not a bandwidth or QoS problem. Inside hosts send 1500-byte segments with the DF bit set. IPsec encapsulation adds roughly 50-73 bytes (ESP header/IV/trailer, plus outer IP and UDP 8 bytes when NAT-T is in use). The encapsulated packet exceeds the underlay MTU and is dropped by the first device that cannot fragment a DF packet. ICMP 56-byte pings traverse fine, and TCP handshakes and small GETs complete, so the tunnel looks 'up'. Bulk TCP stalls because the PMTUD ICMP 'fragmentation needed' is filtered somewhere in the path, so the sender never reduces MSS.

Fix:

• Clamp TCP MSS on both branch and HQ FTDs: sysopt connection tcpmss 1360 via FlexConfig on FTD (there is no native MSS-clamp object). 1360 leaves headroom for NAT-T + ESP + outer IP.
• On route-based/VTI tunnels, also set the tunnel interface MTU to 1400 and let TCP MSS derive from it.
• Verify with show run all sysopt | include tcpmss and by re-testing a >50 MB transfer; confirm with a capture showing the peer's SYN-ACK advertising MSS 1360.

Apply on branches first (freely changeable), then push the HQ FlexConfig inside the 15-minute window. Clamping only affects TCP; if UDP-heavy apps are in scope, add a per-tunnel MTU adjustment as well.

Decision: do not use floating statics with IP SLA for this. Use BGP on both tunnels with local preference, so fiber is preferred while both tunnels stay up, and a failed path is simply withdrawn rather than requiring a new tunnel to be built cold.

Design:

• Two VTIs per branch; eBGP peering over each, with local-preference raised on the fiber-learned routes and default on LTE.
• Enable BFD on the fiber VTI (bfd interval 300 min_rx 300 multiplier 3) so failure is detected in under a second instead of waiting for DPD or BGP hold-down.
• Add a preemption delay (route-map with a set local-preference applied only after the fiber tunnel has been stably up for 5 minutes, e.g. via EEM or a route-map timer) so a flapping fiber does not pull traffic twice.
• Add BGP dampening or a maximum-prefix guard on the LTE peer so a flapping branch does not churn HQ routing.

Why the current 6-8 minute recovery is wrong: it strongly suggests the failback is driven by SLA probe polling plus a newly negotiated IKEv2 SA (and possibly a rekey of the HQ-side cluster). BFD + prepared but idle backup tunnels cut detection to seconds.

Sequencing: roll branches first, verify on 3 pilot sites, then use the HQ window only for the MSS FlexConfig and any HA-side BFD timers.