DSVPN Branches That Register but Cannot Talk to Each Other
HQ runs a USG6650E on a static address, 203.0.113.10. Forty branches use USG6510E firewalls with PPPoE dial-up, so their public addresses change daily. The design is DSVPN: an mGRE tunnel interface on the hub acting as NHS, NHRP registration from the spokes, and an IPsec profile protecting the tunnel. Branch-to-HQ traffic works on all 40 branches. Branch-to-branch traffic does not: pings between 192.168.50.0/24 and 192.168.60.0/24 time out, while the hub reports 40 registered NHRP peers. OSPF runs over the tunnel with the default network type and all branch LANs are in area 0. Each spoke's IPsec profile references an ACL that permits its own LAN to the HQ LAN 10.10.0.0/16 only. The hub tunnel interface has no NHRP redirect configured, and several branches sit behind carrier NAT.
1. Identify the independent reasons branch-to-branch traffic fails and explain how you would confirm each one, in order.
Read the worked answer
Start from the fact that registration works. 40 NHRP peers means the spokes can reach the NHS, so IKE, the tunnel and NHRP authentication are fine. The fault is in resolution and in what the IPsec SA is allowed to carry.
Reason 1 - no direct path is ever built. With no nhrp redirect on the hub and no shortcut behaviour on the spokes, every branch-to-branch packet goes hub-and-spoke. That is suboptimal, not fatal - so if the traffic fails entirely, the hub path itself is broken. Check display nhrp peer all for a resolved peer for the remote branch, and check that the hub's tunnel routing has both branch LANs via OSPF. If the routes exist on the hub but not on the spokes, OSPF over a broadcast-type tunnel with a single DR is a likely contributor; verify adjacencies per spoke.
Reason 2 - the IPsec ACL is too narrow. Each spoke's ACL only permits its LAN to 10.10.0.0/16. A spoke-to-spoke SA has selectors that match neither side, so even a correct NHRP resolution cannot be encrypted and the packet is dropped as 'no matching SA'. Confirm with display ike sa and display ipsec sa while pinging: a new SA pair should appear; if it does not, the selector is the reason.
Reason 3 - carrier NAT. A spoke behind NAT cannot accept an inbound IKE negotiation. Only the spoke that initiates can succeed, and NAT-T must be detected on both ends (display ike sa shows the NAT-T flag).
2. What would you change, and how would you de-risk the change across 40 live branches?
Read the worked answer
Widen the crypto before touching NHRP. Change the spoke IPsec profile ACL to permit 0.0.0.0/0 → 0.0.0.0/0 (the GRE tunnel is already the only traffic on that interface, so the selector does not leak anything), and mirror it on the hub. This alone restores branch-to-branch through the hub and is the safest first step: hub-and-spoke forwarding is deterministic and independent of NHRP resolution.
Then optimise, do not fix. Once traffic passes, enable nhrp redirect on the hub tunnel and shortcut behaviour on the spokes so that spoke-to-spoke flows can cut through the hub. Because several branches are behind carrier NAT, shortcut will succeed only when the NATed side initiates; do not promise symmetric shortcut. Keep hub-and-spoke as the fallback and verify with display nhrp peer all and per-flow SA checks after enabling.
De-risking across 40 branches: pilot on two branches (one NATed, one public) during a window; change the hub last, since hub changes break everyone at once; keep the old ACL saved for rollback; add DPD on the tunnel with a timer short enough to detect a dead spoke but not so short that PPPoE re-dials cause flaps; clamp TCP MSS on the tunnel (typically 1360-1400) so large packets do not black-hole after GRE and IPsec headers are added; confirm the OSPF interface cost and tunnel network type so branch LAN routes stay stable through the change. Document that dial-up addresses, not configuration, will be the day-to-day support load.