Four Unequal Backends, Sticky Carts and Three NAT Gateways

Round-robin is the defect: it distributes connections, not capacity. 10.0.1.13 has roughly a quarter of the vCPU of the others yet receives 25% of new connections, so it saturates first - and because its sessions run slower, they also pile up.

Recommendation:

• Method: weighted least-connection. It combines declared capacity (weight) with real-time queue length, so a slow or batch-loaded server sheds new connections automatically instead of waiting for an operator.
• Weights 4 / 4 / 1 / 4, approximating the vCPU ratio. Keep .13 in the group at weight 1 rather than removing it - at 12,000 concurrent connections you want the fourth host's capacity available.
• Enable slow-start/ramp-up so a recovered server is not handed a full share in the same second it comes up.

Confirmation metric: not CPU alone. Watch per-server current connections and response time at peak. Success looks like the four servers converging on similar latency with connection counts proportional to weight; failure looks like .13 climbing again while the others idle, which would mean weight 1 is still too high during batch hours. In that case remove .13 from the group for the batch window instead of tuning further - a scheduling answer, not a balancing one.

Source-IP persistence is the obvious answer and the wrong one. Sixty percent of traffic arrives from three NAT addresses. Source-IP persistence would pin each gateway's several thousand users to a single backend - three hot servers, one idle, a very large affinity table, and the original distribution problem made permanent. It is defensible only when the NAT population is small or the application tolerates it.

Better design, given TLS terminates on the ADC:

• Cookie-insert persistence bound at the virtual server. Name the cookie distinctly (e.g. ACOSCART) and set a path of / so it is presented on every request.
• If the application already issues its own session cookie, use cookie-hash/passive on that name instead, so users do not carry a second, redundant cookie.
• Fall back to source-IP only for clients that present no cookie, and document that this path is best-effort.

Timeout: slightly above the application's 20-minute idle timer - 25 minutes, for example. Shorter silently re-breaks the cart mid-session; much longer inflates the persistence table and pins clients to servers that no longer deserve them.

Verification: after the change, all four backends should receive load even from a single NAT address, and a scripted multi-request session from one client must land on the same server throughout.

The requirement is that .13 stops receiving new connections while established sessions finish. Removing it from the group, or disabling it abruptly, drops live carts and re-creates the complaint.

Procedure:

1. Check headroom first. With .13 out, the remaining three carry the full peak, not ~75% of it. If their combined headroom is thin, move the drain to off-peak - that is a business decision, not a technical one, and it should be raised before 09:55.
2. Mark the server/port disabled so the load balancer stops selecting it, with graceful drain rather than immediate removal, so established connections complete.
3. Watch per-server connection counts decay toward zero. If they do not, the drain is not actually in effect and the traffic is still arriving.
4. Leave monitoring in place so state reporting stays accurate, and do not correlate the drain with a monitor change in the same window - you will not know which caused what.
5. Confirm the remaining members' response time and error rate stay flat throughout.

Re-entry: enable only once CPU is stable, with ramp-up on, so the server does not take a full share instantly. Record the drain duration - it becomes the estimate for the next maintenance window.