Uneven load on a WebSocket service group behind a shared NAT
A B2B portal runs on six Linux application servers behind a Citrix ADC VPX pair. The service group uses the Least Connection method, source-IP persistence with a 60-minute timeout, and a TCP monitor on port 443. About 2,000 concurrent users are connected during business hours, and approximately 70% of them reach the portal through a partner's shared egress proxy at 203.0.113.10. The ADC terminates TLS. Monitoring shows one application server at 95-100% CPU and 4,000 established connections, while the other five sit near 30% CPU and 1,200 connections each. The application team also reports that a server returning HTTP 503 for a licensing error stayed in rotation for 20 minutes. The business cannot take an outage, but partner traffic must remain on the same backend server for the duration of a login session.
1. Diagnose the uneven distribution and propose a persistence and LB-method change that preserves session affinity.
Read the worked answer
The distribution is not a Least Connection failure; persistence is evaluated before the load-balancing method for new sessions. Source-IP persistence hashes the partner proxy address 203.0.113.10 to one service, so roughly 70% of users are pinned to the same backend. Least Connection only influences new persistence entries, and most of these are long-lived WebSocket sessions, so the imbalance persists.
Move to cookie-based persistence, because the ADC terminates TLS and the application is HTTP. An inserted HTTP cookie gives affinity per browser session rather than per NAT address. Configure the service group or vserver with cookie persistence, set a sensible timeout, and decide whether the application's own cookie should be used passively or rewritten. Keep source-IP persistence only as a fallback if some clients cannot accept cookies, not as the primary method.
Then review the LB method. Least Connection counts connections, not CPU or memory, and long-lived WebSockets can be idle or heavy. For six servers, weighted Least Connection can reflect different server capacities, and custom load via SNMP can be used if the application team can expose CPU or memory. Validate with stat lb vserver, stat servicegroup, and show lb persistentSessions after changing persistence.
2. Fix the health check so the 503 server is removed quickly without causing false positives during normal latency.
Read the worked answer
A TCP monitor on 443 only proves that the port is listening. It cannot see that the application is returning 503, so a server with a licensing error remains in rotation for 20 minutes. Replace it with an HTTP-ECV or HTTP monitor that requests a lightweight health endpoint and expects an HTTP 200 response.
Use a short interval and a small number of retries so detection is fast but tolerant of normal latency. For example, interval 5 seconds, response timeout 3 seconds, and retries 3 gives roughly 15-20 seconds to mark a consistently failing server DOWN. The request should include the correct Host header and should be sent over TLS if the backend expects HTTPS. Bind the monitor to the service group and test it by stopping the application, not by shutting down the server.
Enable down-state flush if stale persistence entries would otherwise keep sending users to the failed server. For planned maintenance, use graceful shutdown so existing sessions drain. Do not use a WebSocket upgrade endpoint as the health check; use a simple application health URL that verifies dependency readiness without creating long-lived sessions.
3. Why might Least Connection still be a poor choice even after cookie persistence, and what metric would you choose?
Read the worked answer
Cookie persistence fixes the shared-NAT skew, but Least Connection still measures connection count rather than work. A WebSocket connection can be idle for minutes and consume almost no CPU, while another connection can carry heavy message traffic. Six servers with similar connection counts can therefore have very different CPU and memory pressure.
If the servers are uniform, Least Connection may be acceptable once persistence is corrected. If they differ in capacity, weighted Least Connection is the simplest improvement: give larger servers higher weights and smaller servers lower weights. If the application team can expose CPU or memory through SNMP, custom load is more accurate, but it adds monitor and threshold maintenance. Least Bandwidth is useful when throughput is the dominant cost, but it can still be misleading for bursty WebSocket traffic.
Choose the metric that matches the bottleneck. For this portal, start with weighted Least Connection, confirm that no single server exceeds a defined CPU threshold, and use stat servicegroup plus application metrics to decide whether custom load is justified. Avoid changing persistence and LB method at the same time without a rollback plan.