Hub-and-Spoke IPsec with CGNAT Branches and Overlapping Subnets
A retail chain runs 80 stores, each with a USG6000E connecting over IPsec to an HQ USG6650E in a hub-and-spoke topology. IKEv2 with pre-shared keys is used, and 30 stores have a 4G/LTE backup link that sits behind carrier-grade NAT. Store staff report that stock queries and small web pages work, but the nightly POS batch uploads, which are HTTPS POSTs of 5 to 40 MB, stall at 60 to 70 percent and then time out. The POS appliance never fragments and sets the DF bit. The HQ firewall shows Phase 1 and Phase 2 up for every store, and a 1400-byte ping from HQ to store succeeds. After a recent acquisition, eight new stores use 10.20.0.0/16, which already exists at HQ.
1. Phase 1 and Phase 2 are up and small packets pass, yet large uploads stall. Explain the most likely cause and the verification you would perform.
Read the worked answer
The pattern is a classic MTU black hole, not an IKE failure. IPsec adds encapsulation overhead, and on the 4G path there may be additional NAT and carrier tunneling overhead. A 1400-byte ICMP test can pass because the path MTU is just above it or because ICMP is handled differently, while a 1500-byte TCP segment with DF set is dropped by an intermediate link. TCP retransmits the same large segment, so the transfer stalls near completion rather than failing immediately.
Verification: from HQ, ping the store with DF and increasing sizes to find the actual path MTU, for example 1400, 1420, 1450 and 1472 bytes. Check display ipsec sa for the tunnel and confirm which path the store is using, 4G or fiber. Capture on both sides and look for TCP segments that leave HQ but never arrive at the store. Confirm the POS appliance sets DF and does not honor ICMP 'fragmentation needed'.
2. What configuration changes would you apply to make the batch uploads reliable without reducing security? Include the CGNAT-specific consideration.
Read the worked answer
Apply MSS clamping on the IPsec tunnel path so TCP endpoints never emit segments larger than the tunnel can carry: set tcp adjust-mss to a value that accounts for ESP, UDP encapsulation and any carrier overhead, typically 1360 or lower on the 4G links, and test upward. For traffic that must not be fragmented, clear the DF bit on the tunnel interface so the firewall can fragment after encryption where policy allows. Keep IKEv2 NAT traversal enabled and verify UDP 4500 keepalives and DPD are working, because CGNAT mappings expire and a dead peer that looks 'up' will black-hole traffic. Prefer certificate or strong pre-shared-key authentication and do not weaken Phase 1 or Phase 2 proposals to compensate.
Also confirm the POS application honors MSS from the SYN and does not pin its own segment size. If it does, a TCP proxy or a smaller interface MTU at the store side may be needed. Validate with a full 40 MB upload over 4G, not a single ping.
3. How would you integrate the eight acquired stores whose 10.20.0.0/16 overlaps HQ without re-addressing them?
Read the worked answer
Overlapping address space cannot be routed as-is, because HQ would send 10.20.x.x traffic to its local network instead of the tunnel. Choose one side to translate and make the translation explicit in the crypto ACL and policy. The usual choice is to translate the acquired stores' source addresses at their gateway into a unique range, for example 172.30.0.0/16, using source NAT before IPsec, and define the tunnel interesting traffic as 172.30.0.0/16 to HQ. HQ then sees only unique addresses, and return traffic is translated back at the store.
Alternatively translate at the HQ hub, but that concentrates state and complicates troubleshooting. Ensure DNS and application configurations use the translated addresses, update firewall policies on both sides, and document the mapping. Verify with display nat session and display ipsec sa that the translated flow matches the tunnel selectors, and test bidirectional application flows, not just ICMP.