SD-WAN overlay across dual internet and capped LTE for 80 grocery stores
FreshMart operates 80 grocery stores and two data centers. Thirty stores still have 20 Mbps MPLS; all stores have cable or fiber internet with dynamic public IPs, and 25 stores sit behind carrier CGNAT. LTE is available as a backup but with a 50 GB monthly cap. The company is adopting Teams for voice and moving procurement to SaaS, while store inventory remains in the data centers. Peak store traffic is 30 Mbps; checkout transactions are small but latency-sensitive. The network team wants per-application steering, forward error correction for voice, and sub-second failover, but the MPLS contract runs for another 18 months. No store has a public IP, and the data centers use SRX4600 clusters with two ISP links each. The stores use a mix of Juniper and third-party routers, and the current MPLS has a 99.9% SLA.
1. Design the overlay topology and control plane. Address dynamic/CGNAT addresses, hub redundancy, and how branches will reach both data centers.
Read the worked answer
Use a hub-and-spoke overlay with two hub endpoints, one in each data center. Because 25 stores are behind CGNAT and all stores have dynamic public IPs, the spokes must initiate tunnels outbound to public, static hub addresses. The hubs cannot dial the spokes. That single constraint drives the topology.
Each store builds IPsec tunnels to both data-center hubs over its available underlays: internet primary, LTE backup, and MPLS where present. The spoke advertises its store LAN into the overlay; the hubs advertise the data-center prefixes. Use BGP over the tunnels with per-tunnel next-hop and BFD, or a controller-based overlay that abstracts the tunnel state. Avoid overlapping store subnets; if they already overlap, use a NAT or VRF-aware design before cutover.
Hub redundancy: advertise the same data-center prefixes from both hubs with a consistent routing policy, and let the spoke prefer the nearest hub while keeping a secondary tunnel. For the MPLS remainder, keep it as a third underlay for inventory and POS until contract expiry, but do not stretch L2 across it.
Control-plane rules: spoke-originated tunnels, no inbound dependency on store IP, unique tunnel endpoints per hub ISP, and route summarization at the hubs. Document that CGNAT breaks inbound hosted services and IPsec responder mode; any store requiring inbound access needs a public IP or a relay.
2. How do you satisfy sub-second failover and FEC without breaking the existing MPLS? Include tunnel monitoring parameters and policy.
Read the worked answer
Sub-second failover comes from continuous tunnel monitoring plus BFD, not from routing convergence alone. Probe each underlay path every 500–1000 ms with loss, latency, and jitter thresholds. Example policy: voice path fails when loss exceeds 1%, one-way latency exceeds 150 ms, or jitter exceeds 30 ms for three consecutive probes. BFD at 300 ms x 3 gives roughly 900 ms detection, which meets sub-second if the steering policy reacts immediately.
Use application-aware steering: Teams voice prefers the internet path with FEC enabled; checkout/POS prefers MPLS while it exists, then internet, then LTE; inventory can use internet primary. Keep LTE last because of the 50 GB cap, and rate-limit non-critical traffic on LTE. FEC adds bandwidth overhead, so apply it only to voice and real-time video, not to bulk inventory updates.
Do not break MPLS: keep the existing MPLS CE routing and use the overlay as an additional path. Advertise specific data-center prefixes over the tunnels and let the MPLS path remain preferred for POS via local preference or route policy. Test asymmetric routing and return paths before enabling per-app steering. Also validate that the SRX4600 hub clusters can handle tunnel scale from 80 stores plus MPLS and that syslog/telemetry does not depend on the failed path.