Source IP persistence collapses onto two members behind forward proxies

Why it collapses: Source IP persistence keys the persistence entry on the client source address. With PAT, hundreds of users share 10.50.0.5, so they all map to the same persistence key and therefore the same member, until that entry expires after 1800 seconds and a new one is created. Two proxies mean at most two effective persistence keys - so at most two hot members. Widening the persistence mask to a subnet would make it worse, not better, because a /24 containing both proxies would pin everyone to a single member.

Recommended: HTTP cookie persistence in Insert mode. The BIG-IP inserts its own cookie (for example BIGipServer_portal_pool) carrying the member encoding, so each user session is a separate persistence entry regardless of network path. Set the cookie expiration to match the application's session timeout, and enable cookie encryption if the app must not be able to decode member addressing.

Caveats:

• If the application already sets cookies on the same path, use a distinct cookie name and verify the app does not rewrite Set-Cookie in a way that breaks reinsertion.
• If the application spans several virtual servers, enable Match Across Services (or Match Across Virtual Servers) so the same member is chosen everywhere; without it, cookie persistence is per virtual server.
• Persistence does not fix in-memory session state. If a member fails, its users lose their sessions; persistence gives affinity, not replication. The durable fix is a shared session store.
• Avoid SSL session ID persistence here - modern clients resume with session tickets and the identifier is not stable enough to rely on.

What happened: the HTTP monitor's receive string ('OK') is matched against the response body after the headers. The release changed the body to {"status":"ok"}, so the match fails on every member. After the monitor's interval/timeout cycle, each member moves to down. With no available members, the virtual server has nothing to load balance to and returns 503 (unless a fallback host is configured).

Why it was instant and total: all six members were monitored by the same string with the same interval - one release and the whole pool goes red simultaneously. That is a health check coupled to a payload the application team can change at will.

Fixes:

• Update the receive string to the new body, quoted properly, e.g. "status":"ok", or use a short stable substring.
• Better: clear the receive string and rely on the HTTP status code, so any 2xx/3xx is up. The check then verifies reachability and correct routing rather than payload text.
• Best: give the application a dedicated health endpoint with a frozen contract, documented as part of the service.

How to fix it safely:

1. Test from the BIG-IP shell against one member with the exact Host header the monitor uses: curl -sv -H 'Host: portal.example.com' http://10.0.0.31/health. If curl sees a 200 with the new body, the monitor is what is wrong.
2. Create the corrected monitor, attach it to one member only, and confirm that member goes green before touching the others.
3. Apply pool-wide, then confirm the virtual server stops returning 503.
4. Tune Time Until Up / Time Until Down so a single bad response does not flap the pool.

Cause: IP::client_addr is the source IP of the TCP connection as the BIG-IP sees it. Because the forward proxies PAT the branch traffic, the BIG-IP sees the proxy, not the user. X-Forwarded-For is just an HTTP convention - if the proxy does not insert it, no amount of iRule logic can recover the original client address. Note this is also why source IP persistence keys on the proxy.

Corrected iRule (append, do not blindly overwrite):

when HTTP_REQUEST {
    if { [HTTP::header exists "X-Forwarded-For"] } {
        HTTP::header replace X-Forwarded-For "[HTTP::header value X-Forwarded-For], [IP::client_addr]"
    } else {
        HTTP::header insert X-Forwarded-For [IP::client_addr]
    }
}

Appending preserves the chain if any upstream device already added the real client. HTTP::header insert alone would create a second, duplicate header line that many applications ignore or parse inconsistently.

Required upstream change: the proxies must send the true client address in X-Forwarded-For (or move to PROXY protocol, which is not spoofable in the same way). Until they do, the header will keep showing proxy addresses and the application cannot use it for anything.

Trust caveat: if the virtual server is also reachable by untrusted clients, they can forge X-Forwarded-For. Only trust it when the connection comes from a known proxy source range - check IP::client_addr against a datagroup before honouring the value, and never use an unvalidated XFF value for authorization or persistence decisions.