Two-Arm Cutover with Reset on Every New Connection
A retailer has just moved its checkout web tier from a one-arm to a two-arm Citrix ADC VPX deployment. The VPX has NSIP 10.20.5.10/24 and SNIP 10.20.5.20/24. The client-facing VIP 192.0.2.100 is on VLAN 100; the 12 backend web servers are on 10.20.6.0/24. The server team left each server's default gateway pointing at the core switch 10.20.6.1, not the ADC, because other applications on those servers use that gateway. ADC health checks against TCP 443 pass, and show service shows all 12 servers UP. However, since cutover, most browser requests to the VIP fail with connection reset; nstrace shows the ADC forwarding client SYNs to a server, but the ADC never sees the corresponding SYN-ACK. A few long-lived connections established before the change still work. The business requires client IP addresses in application logs for fraud checks, and the server team says changing 12 default gateways needs a separate maintenance window. The ADC currently has Use Source IP enabled on the services.
1. Explain why health checks pass while client requests fail, and name the configuration change that would most directly restore service.
Read the worked answer
The failure is asymmetric routing caused by Use Source IP (USIP) being enabled while the servers still default-route through the core switch. The ADC terminates the client TCP connection on VIP 192.0.2.100 and opens a separate connection to the backend. With USIP on, the source of that backend connection is the real client IP, not the ADC SNIP. The server therefore sends its SYN-ACK toward the client IP. Because its default gateway is 10.20.6.1, the packet goes server -> core -> client and never traverses the ADC. The client is waiting for a SYN-ACK from the VIP, so it resets or drops the unexpected packet; the ADC connection table shows the backend half-open. Health checks still pass because monitor traffic originates from the ADC itself (SNIP 10.20.5.20) and the core knows how to return that traffic to the SNIP. The most direct restoration is to disable USIP on the affected services/service group so backend connections are sourced from the SNIP and the return path comes back through the ADC. That fixes reachability but removes the real client IP from the server's L3 view.
2. Given the fraud-log requirement and the separate gateway window, recommend an immediate and a long-term fix. Include the trade-offs and a header-based mitigation.
Read the worked answer
Immediate: disable USIP on the affected services, confirm the core route for 10.20.5.0/24 points to the ADC, and add an application-visible client IP header. On the ADC, a rewrite policy can insert X-Forwarded-For or Client-IP using the client source IP; the app must be changed to log that header, and the header must be stripped or overwritten at the edge so clients cannot spoof it. This restores service in the current window without touching server gateways. Trade-off: the app socket peer is now the SNIP, so any fraud logic that only reads the socket source breaks until it reads the header.
Long-term: either change the 12 servers' default gateway to the ADC SNIP and re-enable USIP, or add policy-based routing on the servers for the client subnet via the ADC. That restores true source IPs at L3 and removes the header dependency. The gateway change is cleaner but affects other applications on those servers, so it needs the separate window. Do not treat MAC-based forwarding as a substitute for a symmetric return path.
3. What commands and packet captures would you use to prove the diagnosis and validate the fix?
Read the worked answer
Use nstrace on the ADC with a filter for the VIP and one backend server. A healthy flow should show client SYN to VIP, ADC SYN to server, server SYN-ACK back to the ADC, and ADC SYN-ACK to client. In the broken state you will see the ADC SYN to the server and no SYN-ACK arriving at the ADC; show connectiontable will show the backend connection stuck in SYN_SENT. From a backend server, traceroute to the client IP shows the path leaving via 10.20.6.1 rather than 10.20.5.20. show service and stat service confirm the services are UP and receiving hits; show lb vserver confirms the VIP bindings. After disabling USIP, repeat nstrace: the backend SYN should now source from the SNIP, and the SYN-ACK should return to the SNIP. In the app, curl the VIP and confirm the X-Forwarded-For header contains the client IP while the socket peer is the SNIP.