Source Address Persistence Behind Carrier NAT

Persistence, not load balancing, is choosing the member. Source Address persistence hashes the client IP; with three carrier NAT addresses dominating the traffic, essentially all mobile users collapse onto at most three hashes, and the 1,840-connection member is serving an entire carrier's subscriber base while the near-idle members only see the minority of clients with unique public IPs.

The same hash explains the logout storm. Disabling the member removed every persistence entry that pointed at it, so all clients behind the big NAT address were immediately re-load-balanced - most likely to the member with the fewest connections - and found no session there, because sessions live in node-local memory with a 30-minute timeout. Healthy peers cannot help: there is no state to share.

Finally, note that the connection counts are a poor load signal here. With OneConnect enabled, 1,840 connections is not 1,840 concurrent users, and a member showing 8 connections may be doing nothing at all while the busy member multiplexes thousands of requests over a few pooled connections. Read CPU and response time, not connection counts, when OneConnect is in the path.

Replace IP-based affinity with cookie-based affinity so stickiness follows the user instead of the carrier gateway.

1. Create an HTTP Cookie persistence profile in Insert mode with a distinct cookie name and an Expiration set to a specific interval slightly longer than the app's 30-minute session timeout (roughly 1,800-2,100 seconds). Do not use 'session cookie': mobile browsers discard those when a background tab is evicted, and the user logs in again on resume.
2. Attach it to the virtual server ahead of the Source Address profile. Persistence profiles are evaluated in order, so the cookie wins when present and Source Address remains as a fallback for clients that refuse cookies (health checkers, some API clients).
3. Change one variable only: leave OneConnect as it is for this window.
4. Roll out of hours. Existing Source Address entries expire within 180 seconds, so the migration completes in about three minutes, but every user whose entry pointed at the busy member is re-balanced once - warn support to expect a short login spike.

Verification: watch the persistence table and the per-member distribution. The 1,840 / 12 / 8 / 5 pattern should flatten toward even quarters within minutes. Cookie insert also survives Wi-Fi-to-cellular handovers, which Source Address cannot.

It makes the situation worse. The hash does not change, so 12,000 users behind the busiest NAT address stay pinned to the same member for an hour instead of three minutes. Every pool membership event - patch, monitor flap, scale-out - now costs that member's entire population a re-login rather than a slow trickle, and the window in which a recycled carrier IP inherits a stranger's session stretches from minutes to an hour.

It also does nothing for the failure case that actually hurt: entries are still invalidated the instant the member goes down, regardless of the timeout.

The timeout was never the problem; the key is. Sequence it this way: cookie insert first, so affinity is per user and load spreads across all four members; then push the app team toward a shared session store so affinity becomes a performance optimisation rather than a correctness requirement. Once state is shared, a member failure costs a re-authentication instead of a lost cart, and the timeout can be tuned for efficiency rather than survival.