ADVPN Shortcut Failure Across CGNAT Spokes and Dual-Hub Failover
A service provider uses two FortiGate 1800F hubs and 300 FortiGate 60F spokes. Spokes are on business DSL and 4G, many behind CGNAT. ADVPN with IKEv2 and BGP is deployed; hubs are route reflectors; spoke-to-spoke traffic should form shortcuts. After adding 60 spokes behind a new CGNAT provider, shortcuts fail for those spokes; traffic hairpins through hub1, latency rises from 30 ms to 180 ms, and hub1 CPU hits 85%. BGP sessions are up and all spokes receive remote prefixes. Some spokes show a shortcut for a few minutes, then it disappears. During a hub1 maintenance window, 40 spokes do not fail over to hub2; their BGP sessions stay up but traffic blackholes. The provider cannot give public IPs to the new CGNAT spokes. You must determine what to check and how to make the design resilient without changing all 300 spokes manually.
1. What checks and configuration changes would you use to determine why shortcuts are not forming for some spokes?
Read the worked answer
Start with shortcut prerequisites, not with BGP. ADVPN shortcut requires IKEv2, auto-discovery-receiver on the spoke, auto-discovery-sender on the hub, and a BGP route whose next-hop is the remote spoke's tunnel address rather than the hub. Check diagnose vpn ike gateway list and get vpn ipsec tunnel details; then diagnose debug application ike -1 filtered to the spoke while generating traffic. If the hub's BGP neighbor group is configured with next-hop-self, the spoke sees the hub as next-hop and never tries a shortcut; remove that for the ADVPN group or use a route-map that preserves the spoke next-hop. Also verify the shortcut is not disabled on the phase1 and that both hubs can forward discovery. For the 60 CGNAT spokes, inspect whether the provider preserves source port; if it uses symmetric NAT, a direct UDP 500/4500 shortcut is impossible. Those spokes should be treated as hub-and-spoke only: keep them on the hub path, adjust the latency expectation, and avoid flapping shortcut attempts. The intermittent shortcuts suggest an idle-timeout or DPD mismatch; compare idle-timeout, dpd, and MTU on both ends. Fix next-hop and NAT classes first; changing BGP timers will not create a shortcut.
2. Why do 40 spokes blackhole during hub1 maintenance even though BGP sessions remain up, and how would you make hub failover reliable?
Read the worked answer
Hub failover is a BGP and tunnel-liveness problem. Confirm each spoke has two phase1 tunnels and two BGP sessions; if a spoke only has a session to hub1, it cannot fail over. Check get router info bgp summary for the hub2 neighbor state and diagnose vpn ike gateway list for the hub2 tunnel. The common cause of BGP up but blackhole is that the spoke still prefers routes learned from hub1 because the hub1 session has not been declared down; DPD must detect the dead hub, and BGP hold time must expire. Use IKEv2 DPD with a short but not aggressive interval, and BGP keepalive/hold timers that match the transport. On the hubs, ensure both advertise the same summary; on the spokes, use local preference to prefer hub1 and allow hub2 as backup. Do not use static routes to both hubs with the same distance. Test by shutting hub1's tunnel interface, not by pulling the whole hub, and watch diagnose debug application ike and get router info bgp. Also check that hub2 has a route back to the spoke's summary; asymmetric advertisement is a common blackhole. Automate the 300 spokes with FortiManager or a template so the fix is consistent.
3. How should the design treat CGNAT spokes so that the lack of shortcuts does not destabilize the hubs or violate latency expectations?
Read the worked answer
CGNAT changes the design constraint: a shortcut is a direct IPsec tunnel between two spokes, so it needs at least one side to be reachable on a stable UDP port. Symmetric CGNAT breaks that. Do not force nattraversal forced and hope; classify the spokes. For CGNAT spokes, use the hubs as permanent relays, and make that explicit in policy and expectation: spoke-to-spoke traffic traverses hub1 or hub2 with dual-hub redundancy. If the CGNAT provider offers a static public IP or port-preserving NAT for a subset, those spokes can attempt shortcuts. To protect hub CPU, apply per-spoke bandwidth limits, avoid full-mesh shortcuts for latency-insensitive traffic, and use hub2 for half the relays. If direct spoke-to-spoke is required for voice, put those spokes on business broadband with public IPs or use a rendezvous relay. Also verify that auto-discovery-forwarder is enabled on hub-to-hub links so hub1 can tell hub2 about a shortcut. Document that shortcuts are an optimization, not a requirement; the design must remain functional when they fail.