Asymmetric ISP routing breaks inbound NAT on VS9
VS9 on a VSX cluster hosts a customer with two ISP links. ISP1 is the default route with metric 10; ISP2 is a backup with metric 20. Inbound NAT publishes 203.0.113.50:443 to internal 10.9.0.20 on ISP2. Outbound traffic from 10.9.0.20 uses ISP1 because of the lower metric. The customer reports that HTTPS connections to 203.0.113.50 sometimes complete the TCP handshake but then hang during TLS; other times they time out. Logs on VS9 show 'TCP packet out of state' and 'first packet isn't SYN' for return traffic from 10.9.0.20 to public clients, with the packet leaving via ISP1. The customer wants both ISPs used and does not want to buy another VS. A junior engineer suggests disabling stateful inspection on VS9.
1. Explain the failure mechanism. Why does the handshake sometimes complete?
Read the worked answer
The inbound connection arrives on ISP2 and is NATed to 10.9.0.20. The server's SYN-ACK is routed by its default route, which points to ISP1, so the reply leaves through a different ISP and possibly with a different public source address than the one the client connected to. The client or the upstream ISP drops that reply, and the firewall logs out-of-state because the return path does not match the connection's expected interface and translation. The handshake can sometimes complete when a more specific route, a temporary ISP1 failure, or a stateful NAT coincidence sends the reply back through ISP2, which is why the symptom is intermittent rather than a clean total outage. The root cause is asymmetric routing and NAT source selection, not a missing permit rule.
2. What are the two viable fixes that preserve both ISPs without another VS?
Read the worked answer
First, make the server's return traffic follow the ISP that carried the inbound connection. The simplest reliable option is a policy-based route or a dedicated routing table entry that sends traffic sourced from 10.9.0.20 out ISP2, while other internal networks continue to use ISP1. This sacrifices ISP1 for that server's outbound traffic but keeps both links in use for the rest of the VS. Second, if the business needs the server to use both ISPs, publish the service on both ISP addresses and ensure return traffic is selected per connection, not solely by the lower default metric; this requires careful PBR and NAT design and testing. Disabling stateful inspection is not a fix: it weakens security and does not correct the source address mismatch. Choose the first option unless the customer explicitly needs per-connection load sharing.
3. Why is disabling stateful inspection a poor choice, and what would you check in the logs to prove the fix worked?
Read the worked answer
Disabling stateful inspection removes connection tracking and opens the VS to spoofed and out-of-sequence traffic, but it does not change the routing table or the NAT source address. The client can still receive a SYN-ACK from the wrong public IP, so the connection can still fail. It also makes the original 'out of state' evidence disappear, which hides the real problem. After applying PBR or the routing change, verify that the SYN arrives on ISP2, the SYN-ACK leaves on ISP2 with the expected NAT translation, and subsequent packets stay on ISP2. The log should show accepts on the correct interface with no out-of-state messages for that flow. Test from an external client, not only from inside, and repeat during a period when ISP1 is preferred for other traffic to confirm the exception is scoped correctly.