Harbor Logistics: Tunnel MTU, Rekey Flaps and False Redundancy

Diagnosis

The signature - small transactions and ICMP fine, large transfers hanging after roughly half a minute - is a path-MTU black hole, not an application fault. The tunnel interface on the customer firewall is 1500, leaving no room for the IPsec header (typically 50–73 bytes for ESP plus a new outer IP header). A large TCP segment leaves the file server with the Don't Fragment bit set, exceeds the real tunnel payload, and is dropped. Path MTU Discovery should recover it, but the ICMP fragmentation-needed message is either blocked by policy or lost, so the session hangs until the application gives up. The 20–40 second freeze is TCP retransmission back-off, which is why it looks like a timeout rather than an error.

Confirmation

1. From a client behind Prisma Access, send DF-bit pings of increasing size (for example 1400, 1450, 1472) to the file server and find the payload at which replies stop.
2. In the traffic logs, check whether the same flow shows small packets succeeding and the session eventually timing out with no reset.
3. Check the firewall's security policy and any upstream ACL for ICMP type 3 code 4 - if it is dropped, PMTUD cannot work at all.

Remediation

• Set the tunnel MTU below the path limit. 1400 is a common working value; a smaller value such as 1350 is needed where the underlying path already carries PPPoE or additional encapsulation.
• Enable TCP MSS adjustment (clamping) on the customer firewall so endpoints negotiate a segment size that fits, rather than relying on PMTUD.
• Permit ICMP fragmentation-needed in both directions as a safety net.
• Repeat the DF-bit sweep after the change and confirm a full-size document transfer completes.

What the timing tells you

A drop with that regularity is not random loss; it is a timer. Two lifetimes are in play - Phase 1 at 86400 s on the customer side and Phase 2 at 28800 s, while the Prisma Access side proposes its own values. In IKEv2 the responder generally honours the initiator's proposal, but when the timers do not match, one peer can delete a security association while the other still believes it is valid. Traffic is dropped or black-holed until the peer notices and rekeys - exactly the observed window.

Checks

1. Read the IKE and IPsec event logs on both ends around the drop time and identify which side sent the delete and which SA expired.
2. Compare Phase 1 and Phase 2 lifetimes, DPD interval and retry count, and IKE identity and PSK settings on both sides.
3. Confirm whether rekey succeeds but BGP simply takes time to re-establish - a 180 s hold with 60 s keepalive means the session will not be declared down quickly, so the visible outage is the BGP reconvergence, not the tunnel itself.
4. Check for a NAT or session timeout on the ISP path that could also expire the IKE SA.

Changes

Align Phase 1 and Phase 2 lifetimes on both peers so neither side expires first. Reduce BGP timers or enable BFD so a failed tunnel is detected in under a second rather than minutes, and make sure BGP is actually configured for graceful restart or that both tunnels carry the prefixes with differing local preference so traffic shifts instead of stopping. Finally, confirm the tunnels are not both tearing down simultaneously for the same reason; if they are, the fault is shared configuration, not path quality.

What the design actually protects against

Dual routers on dual circuits eliminate a single device failure and a single link failure. They do not protect against the ISP, the local loop, the building, or a shared upstream aggregation device. If both circuits are delivered over the same fibre into the same meet-me room, a backhoe or a provider outage takes both tunnels down at once, and every mobile user and branch in the estate loses data-centre reachability. The configuration also advertises identical prefixes with equal weight, so verify that return traffic from the data centre is not preferring one router while the forward path prefers the other - asymmetric routing will not break TCP in a tunnel design, but it doubles state and makes troubleshooting ambiguous.

Options, in order of effort

1. Add path diversity. A second circuit from a different ISP, terminating on a different edge device with its own service connection tunnel, is the only change that addresses provider and physical-plane failure. Verify the new circuit enters the building by a different route.
2. Second site or regional second service connection if the data centre has a secondary facility.
3. Preference engineering. Set BGP local preference so one tunnel is primary and the other hot-standby, making failover deterministic and avoiding equal-cost load sharing across unequal paths.
4. Test it. Periodically shut one tunnel during a maintenance window and measure BGP reconvergence and user impact; an untested standby is a hypothesis, not redundancy.

What to tell the business

Current redundancy is device and link level only. Until path diversity exists, the realistic recovery objective for a provider outage is the ISP's own restoration time, not sub-minute failover - and that should be written into the service tier expectation.