Dual-ISP Site-to-Site VPN for a Long-Lived SQL Session
Site A (HQ, AS 65001) has 500/500 Mbps fiber with static address 203.0.113.10 and a 200/20 Mbps coax link behind carrier NAT. Site B (plant, AS 65002) has 100/100 Mbps fiber with a static address and a 4G LTE modem behind carrier NAT as backup. The plant's ERP client holds long-lived SQL sessions to HQ; the business tolerates at most a two-second outage and wants fiber preferred. The carrier blocks GRE, BGP is allowed only between the two sites (never with the carriers), and each site has one PA-440 with a single public address per link. NAT-T works, but inbound connections cannot reach either CGNAT address.
1. Specify the tunnel topology: how many SAs, route-based or policy-based, and which side initiates each one.
Read the worked answer
- Route-based (tunnel interfaces). You need BGP or route metrics across paths, multiple SAs per peer, and policy written on zones, not on individual tunnels. Policy-based VPN cannot express failover cleanly here.
- The four-link matrix is not four tunnels. Two addresses are CGNAT, so nothing can connect inbound to them:
- A-fiber (static) ↔ B-fiber (static): either side can initiate. Usable.
- A-fiber (static) ↔ B-LTE (CGNAT): B must initiate. Usable, and it is the plant's backup path.
- A-coax (CGNAT) ↔ B-fiber (static): A must initiate. Usable, but the 20 Mbps coax upload is a poor ERP path.
- A-coax (CGNAT) ↔ B-LTE (CGNAT): both unreachable - no rendezvous point. Skip it unless you add a broker or a public relay.
- Identification: with NAT in the path, peer identification cannot rely on source IP. Use pre-shared keys with explicit local and peer identification, and confirm the PAN-OS release supports a dynamic/FQDN peer address for the responder side before designing around it.
- Keepalives must be frequent enough to hold the CGNAT mapping open on coax and LTE, or the SA silently disappears between failures.
2. How do you make fiber preferred, and detect failure inside two seconds? What are the trade-offs?
Read the worked answer
- Preference: run BGP over the tunnel interfaces between AS 65001 and AS 65002 (permitted - it is site-to-site, not with the carriers). Prefer fiber with local preference or AS-path prepend on the coax/LTE tunnels; if you avoid BGP, use static routes with different metrics plus tunnel monitoring.
- Detection: default DPD is far too slow, and a tunnel monitor at 3s interval / 3 failures is nine seconds. For a two-second budget use BFD on the tunnel interfaces (e.g. 300 ms × 3) or a tunnel monitor at 1s with a 3-count failure.
- Trade-offs: aggressive timers flap on the LTE and coax links - jitter, bufferbloat on a 20 Mbps uplink, and CGNAT queuing will trip them. Use per-tunnel timers (aggressive on fiber, lazy on LTE) plus damping/hysteresis, and monitor the underlying ISP next-hop so the underlay fails over first.
- The end-to-end budget includes underlay re-convergence and the new SA being ready. Pre-establish the backup SA and hold it up; a tunnel proposed on demand will never meet two seconds. Note that BFD over a tunnel is itself dependent on the tunnel being up - pair it with underlay path monitoring.
3. Will the ERP SQL sessions survive a tunnel change? What do you do to maximise the chance?
Read the worked answer
Why route-based helps: the inner 5-tuple is unchanged when traffic moves from one tunnel interface to another, so the SQL server sees the same connection from the same inner source address. That is the main argument for route-based here.
Conditions you control:
- Keep all tunnel interfaces in one zone. If a route change also changes the egress zone, the session is killed.
- Ensure the alternate SA is already established; do not rely on on-demand negotiation inside the outage budget.
- Failover must complete inside the client's TCP retransmit window. Two seconds with BFD usually survives a retransmit of the first lost segment; a nine-second DPD timeout usually does not.
- Keep NAT consistent: inside the tunnel the source is the inner address, so post-NAT identity does not change with the path.
- Do not run the two paths active/active with ECMP for this flow - reordering across tunnels breaks TCP far more painfully than a clean two-second switch.
Where it still breaks: anything the plant initiates to the internet during an LTE failover (outer source changes), and application state outside TCP - SQL connection pools must reconnect even when the socket survives.