Overlay Tunnel Scale and Encryption Overhead
A manufacturer is deploying SD-WAN across 60 factories, each with two WAN links (fiber + cable), plus two data centers and one cloud on-ramp. The design calls for a full-mesh overlay among all sites for direct site-to-site traffic. Security requires IPsec on every tunnel. The smallest edge appliance in the fleet has 2 Gbps of IPsec throughput and 4 GB of RAM. A pilot at one factory shows tunnel establishment taking 90 seconds after a WAN flap, and CPU pegged at 95% during rekey.
1. Calculate the number of tunnels in the proposed full mesh and explain why this is a scaling problem for the smallest appliance.
Read the worked answer
Sites: 60 factories + 2 DCs + 1 cloud = 63 sites.
Full mesh of sites: n(n−1)/2 = 63 × 62 / 2 = 1,953 site-to-site adjacencies.
With two WAN links per factory and IPsec per tunnel, each factory edge may need tunnels to every other site over potentially both links - up to ~2,000+ IPsec tunnels per edge in the worst case if per-link tunnels are built.
For the smallest appliance (4 GB RAM, 2 Gbps IPsec):
- Each IPsec SA consumes memory for SAD/SPD entries and state; thousands of SAs plus BGP/BFD sessions exhaust RAM and session tables.
- CPU at 95% during rekey means the appliance cannot absorb the rekey storm when many tunnels rekey simultaneously after a flap.
- 90s establishment is consistent with serialized tunnel setup under CPU pressure.
Full mesh is not viable at this scale on the smallest appliance.
2. Propose an overlay topology that preserves direct site-to-site paths where they matter without full mesh, and justify the choice.
Read the worked answer
Hub-and-spoke with regional hubs plus selective partial mesh:
- Designate the 2 DCs and ~4 regional hubs as overlay hubs. Factories build tunnels to their regional hub and to both DCs (for central services).
- Partial mesh only for high-volume factory-to-factory flows identified in traffic analysis (e.g., a supplier cluster).
- Result: each factory edge holds ~3–6 tunnels instead of 2,000+.
Justification:
- Preserves direct paths for latency-sensitive or high-volume pairs via targeted partial mesh.
- Keeps control plane scale manageable on the small appliance.
- Rekey storms are bounded because each edge has few SAs.
- Hub appliances must be sized for aggregate spoke traffic - verify hub IPsec throughput against sum of spoke demands.
Alternative: dynamic mesh on demand where tunnels are built only when a flow requires them, then torn down after idle - but this adds setup latency and must be tested against the 90s establishment problem.
3. What operational changes reduce the 90s establishment and 95% CPU rekey problem regardless of topology?
Read the worked answer
- Stagger rekey timers across tunnels (jitter the SA lifetime) so rekeys do not coincide.
- Increase SA lifetime moderately to reduce rekey frequency, balanced against security policy.
- Enable hardware offload for IPsec on the edge; verify the 2 Gbps figure is with offload, not software crypto.
- Reduce crypto cost: confirm the cipher suite (e.g., AES-GCM) is offload-friendly; avoid suites that fall back to software.
- Tune BFD timers so a WAN flap does not tear down and rebuild all SAs - use tunnel keepalive separate from SA lifetime.
- Pre-shared key vs certificate: certificates add CPU at establishment; consider PSK where policy allows.
- Monitor CPU and SA count with alerting; capacity-plan the smallest appliance against actual tunnel count, not nameplate throughput.