30 Branches, Two Overlapping Subnets, and Transfers That Stall
Thirty branches terminate route-based IPsec tunnels (IKEv2) at a hub NGFW with a single public IP. Two acquired sites still use 192.168.10.0/24 internally, and HQ uses the same range. Branch users report that pings and short HTTPS requests succeed but large file copies and TLS handshakes stall, and an end-to-end ping of 1500 bytes with the DF bit set fails. Tunnels show up with 0% keepalive loss. ESP is NAT-T encapsulated on four tunnels whose branches sit behind carrier-grade NAT, and one of those drops every 30 minutes. The ISP filters inbound ICMP type 3 code 4.
1. Explain the stalling transfers and give the fix, including what to validate afterwards.
Read the worked answer
This is a PMTU black hole, and every symptom lines up.
Mechanism
- ESP adds roughly 50-73 bytes of overhead, and NAT-T adds another 8 for the UDP 4500 header. Effective payload MTU on the tunnel is around 1350-1400, not 1500.
- A 1500-byte DF ping failing is the proof: the packet cannot be fragmented in flight and must be fragmented by the sender after receiving an ICMP 'fragmentation needed'.
- Your ISP filters ICMP type 3 code 4, so the sender never learns. Small packets pass; the TLS handshake with a full certificate chain (several KB) and large file segments are silently dropped. Retransmissions look like a stall, then a timeout.
Fix
- Set the tunnel interface MTU to the computed value and clamp TCP MSS on the tunnel to about 1350 (1340 for the NAT-T tunnels), applied to forwarded traffic in both directions, not just locally originated sessions.
- Permit the relevant ICMP on the tunnel so PMTUD can work where it is available at all.
Validation
- DF-set pings at 1400 and at MSS+40 from a branch host to an HQ host and back - both directions must pass.
- A packet capture at the branch showing the SYN, ClientHello and a full-size segment arriving, with no repeated identical retransmissions.
- Re-test one of the NAT-T tunnels separately; its overhead is larger and it is the most likely to still fail.
2. How do you handle the overlapping 192.168.10.0/24 ranges without destabilising the hub?
Read the worked answer
There is no clean technical trick here; there are three options with very different costs, and the judgement is picking one deliberately.
Option A - re-IP the branch (recommended). For a site of a few dozen devices this is a planned change window plus DHCP/DNS updates. It removes a permanent class of incident and needs no ongoing documentation. Cost is organisational, not technical.
Option B - address translation across the tunnel. Map branch 192.168.10.0/24 to a unique range (e.g. 172.20.10.0/24) with source and destination NAT on both peers. It works for client-server traffic but breaks anything carrying IP literals in the payload, complicates split DNS, doubles the NAT rule count, and must be mirrored exactly on the far peer or the return path fails. If you take this route, record it in the tunnel documentation and add monitoring so a silent NAT rule deletion is caught.
Option C - do nothing and rely on longest-prefix matching. It will appear to work in testing and then fail unpredictably, because HQ's own 192.168.10.0/24 is directly connected and wins for local destinations.
Whichever you choose, the hub-side route for the translated or real range must be more specific than the aggregate you advertise into the tunnel, and you should verify with a traceroute from the branch to an HQ host plus a session-table dump showing the translation.
3. One CGNAT branch drops its tunnel roughly every 30 minutes and recovers on its own. What do you check?
Read the worked answer
Self-healing drops at a fixed interval point at a state timeout somewhere in the middle, not at a misconfiguration on either peer.
- CGNAT UDP mapping lifetime. Carrier NAT commonly expires UDP mappings in 30 seconds to 30 minutes. If the tunnel is idle, the mapping disappears and return ESP is dropped. Verify by watching whether DPD/keepalive traffic is actually traversing at an interval shorter than the mapping lifetime.
- NAT-T forced on both peers and keepalives at 10-20 seconds so the mapping stays warm. Confirm on the wire on port 4500, not just in the GUI.
- Peer identification. The branch's public IP is not stable; the hub must identify that peer by IKE ID (or certificate subject) rather than by IP address, otherwise a re-mapping produces a duplicate or rejected peer.
- Lifetime mismatch. Make sure the IKE SA lifetime is shorter than or equal to the child SA lifetime, and that rekeying works - a missed rekey can look like a periodic drop.
- Logs at both ends. Correlate the exact drop timestamps with IKE debug on the hub. If the hub sees no rekey request at all, the packets are not arriving and the carrier mapping is the cause.