Dual-ISP Cutover and Stateful Firewall Drops
A regional logistics firm runs a flat 10.40.0.0/16 internal network with 1,800 endpoints. In March it added a second ISP circuit-400 Mbps fiber, with a 150 Mbps LTE backup-to an existing 1 Gbps primary link. The edge firewall is stateful and dual-homed. Since cutover, warehouse users report that the order-tracking application at 10.40.12.30 times out on roughly every third request, although pings to the same host succeed. The help desk also sees HTTPS sessions to a SaaS vendor reset after about 30 seconds. Firewall logs show many 'TCP packet out of state' and 'no matching session' drops on the secondary interface. The network team confirms outbound traffic uses a policy route preferring the primary link, but return traffic from the SaaS provider sometimes arrives on the secondary link after upstream BGP local-preference changes. The firm cannot use carrier-grade NAT and must keep both links active for capacity.
1. What is the most likely cause of the intermittent timeouts, and which evidence would you collect to confirm it before changing the firewall?
Read the worked answer
The pattern points to asymmetric routing through a stateful firewall, not an application or DNS fault. The firewall builds a session when the client's SYN leaves via the primary link, but return packets from 10.40.12.30 or the SaaS provider sometimes arrive on the secondary interface. Because the 5-tuple and ingress zone no longer match the session, the firewall drops them as out of state. Pings can still succeed because ICMP may be permitted statelessly or handled by a separate policy, and HTTPS resets fit a mid-session path change.
Confirm by:
- Capturing the same TCP flow on both ISP-facing interfaces and comparing sequence numbers and timestamps.
- Checking the firewall session table for the client/server 5-tuple and the ingress interface recorded at session creation.
- Correlating drops with BGP local-preference changes or route updates on the secondary link.
- Testing a long-lived TCP session while forcing return traffic to each path.
Do not disable stateful inspection as a test; that masks the routing defect and weakens security. The fix belongs in routing and NAT design, not in a global firewall rule.
2. Recommend a design that keeps both links active and preserves stateful inspection. Include at least two options and their trade-offs.
Read the worked answer
Option A: active/standby links with the firewall cluster sharing session state. Both circuits stay up, but only one carries traffic; failover is fast and return paths are symmetric. Trade-off: the 400 Mbps circuit is idle most of the time, so the capacity that justified the second link is not used.
Option B: per-flow load balancing with source-based NAT. Policy routes send each new flow out one ISP and SNAT it to that ISP's address pool, which pins return traffic to the same path. A firewall cluster or SD-WAN edge must synchronize state and health-check both paths. Trade-off: inbound services become harder because connections may arrive on either link; per-ISP NAT also complicates logging and third-party allowlists. If inbound services exist, keep inbound on one provider and use the other for outbound only.
Option C: provider-independent address space with BGP advertised to both ISPs and a firewall cluster that shares state. This gives true active/active but requires an ASN, address space, and careful BGP policy; a route change can still break existing sessions unless state is synchronized.
Whatever the choice, verify with packet captures on both edges and monitor out-of-state counters during a forced failover.
3. What operational checks would you put in place to prevent recurrence?
Read the worked answer
Add synthetic TCP transactions that traverse each ISP path and measure completion time, not just ICMP reachability. Alert on firewall out-of-state and no-matching-session counters, baselined per interface, because a rising rate is an early signal of path asymmetry. Track BGP route changes and correlate them with application timeouts. Keep a configuration test that confirms policy routing and NAT pools match the intended egress path after every change. During failover drills, run long-lived HTTPS and database sessions, not only pings. Finally, document the supported traffic pattern in the change process so a future link addition does not reintroduce asymmetric return paths.