SNAT Port Exhaustion During a Flash Sale
A flash-sale VIP uses a SNAT pool with eight translation addresses. Each address provides about 64,000 ports. OneConnect is disabled, and each client TCP connection consumes a SNAT port for the life of the connection. During the sale, the BIG-IP reports about 3% of requests failing with 'SNAT translation failure', while all pool members remain healthy and CPU is 55%. Many users are behind carrier-grade NAT, so thousands of clients share a small number of source IPs. Security requires that backend servers can identify the original client for fraud checks; the application team can read X-Forwarded-For. The current configuration uses a TCP profile with a 300-second idle timeout, and clients hold connections open for keep-alive. The network team can add up to eight more SNAT addresses, but procurement takes a week.
1. Explain why the SNAT pool is exhausted despite healthy pool members, and what evidence you would collect.
Read the worked answer
Each server-side TCP connection needs a unique SNAT IP:port tuple. Eight addresses give roughly 512,000 tuples, but active connections and TIME_WAIT connections both consume them. With a 300-second idle timeout and keep-alive clients, a large fraction of tuples stay occupied after the application exchange ends. At peak, the BIG-IP can run out of free tuples even though CPU and pool members are fine; that produces 'SNAT translation failure' rather than a member health failure.
Collect tmsh show sys connection counts grouped by SNAT address, SNAT pool statistics, and the exact failure timestamps. Confirm that pool member health and response times are normal. Check whether the failures correlate with bursts of new connections from CGNAT ranges. The CGNAT detail matters because many clients share source IPs, but SNAT consumption is per connection, not per client IP.
2. What changes would you make this week, before the new SNAT addresses arrive?
Read the worked answer
Enable OneConnect on the virtual server with a short idle timeout, for example 60 seconds, and a sane maximum reuse per server-side connection. This multiplexes many client connections onto fewer server-side connections and directly reduces SNAT port consumption. Also reduce the server-side TCP idle timeout from 300 seconds to 60 seconds so idle connections release tuples sooner. Tune the client-side keep-alive so clients do not hold idle connections unnecessarily.
If spare self-IPs are available and security accepts their use, add them to the SNAT pool temporarily. Do not switch to automap without confirming that self-IP port capacity is sufficient and that exposing self-IPs is acceptable. Keep X-Forwarded-For insertion for fraud checks. Validate with a load test that measures SNAT pool utilization and failure rate at peak connection churn.
3. How would you size the permanent solution and verify it?
Read the worked answer
Estimate peak concurrent server-side connections plus TIME_WAIT, not just active users. Use the observed peak connection rate, the connection lifetime, and the TCP idle timeout to calculate the required tuple count, then add headroom for bursts and failover. If the calculation still lands near the pool limit, add SNAT addresses or reduce server-side connection count with OneConnect.
Verify by monitoring SNAT pool usage during a controlled peak, checking tmsh show ltm snatpool and connection counts, and confirming no translation failures. Test a single SNAT address failure to ensure the pool still has capacity. Document the sizing formula so the next sale is planned from data rather than from the current pool size.