Intermittent 503s and resets at the morning peak
A hospital's patient portal uses a BIG-IP virtual server with a TCP monitor and a pool of six web servers. At 09:00 each weekday, users report intermittent 503 responses and connection resets; incidents last 10 to 15 minutes. Pool members show normal CPU, but the BIG-IP's SNAT pool contains four addresses and the virtual server uses a 600-second idle timeout. At peak, the portal carries about 45,000 concurrent connections and 500 new connections per second. The application team says most sessions are short-lived API calls, but some clients hold connections open. The network team sees no packet loss. The operations team must find the bottleneck without changing the application, and it needs a monitoring signal that will catch the problem before users do. The pool members are all in one data center.
1. What is the most likely bottleneck, and what evidence would you collect to confirm it?
Read the worked answer
The leading suspect is SNAT port exhaustion. Four SNAT addresses provide roughly 64,000 ports each, but ports are held for the idle timeout and for connection teardown. At 500 new connections per second with a 600-second idle timeout, the potential demand is 300,000 concurrent SNAT bindings, which exceeds the pool even though the measured concurrent count is 45,000. The morning peak is when enough long-held or idle connections accumulate to exhaust the available ports. When SNAT allocation fails, clients can see resets, and the BIG-IP may return an error or drop the connection.
Collect evidence rather than assuming. Check /var/log/ltm for SNAT-related errors, look at connection counts per SNAT address, and compare the incident window with virtual-server connection statistics. Confirm that pool members are up and responding; the TCP monitor only proves the port is open, not that the application is healthy. Capture a packet trace during the incident to see whether the BIG-IP is resetting connections before they reach a pool member. If SNAT counts are not near the limit, investigate pool-member connection limits and application worker exhaustion next.
2. What mitigations would you apply without changing the application, and what are the risks?
Read the worked answer
Add SNAT addresses to the SNAT pool, or move from automap to an explicit SNAT pool with more addresses. This directly expands the port supply and is the least invasive fix. Before adding addresses, confirm that upstream firewalls and the application's access controls accept the new source addresses.
Reduce the idle timeout to a value that matches the application's actual idle behavior, for example 300 seconds or lower, so idle connections release ports sooner. The risk is dropping legitimate long-lived sessions, so identify which clients hold connections open and test the change. Do not rely on OneConnect to fix this: OneConnect reduces server-side connections but does not create client-side SNAT ports.
Replace the TCP monitor with an HTTP monitor against a real health endpoint so the pool does not send traffic to a server that is accepting TCP but failing requests. If the application has connection limits per server, check and raise them only with the application team's agreement.
3. What monitoring would catch this before 09:00?
Read the worked answer
Monitor SNAT port utilization per address and alert at 70 to 80 percent, not at exhaustion. Track new connections per second, concurrent connections, and resets or 503s on the virtual server. A synthetic login or API check every minute gives a user-facing signal that is independent of the TCP monitor.
Use the BIG-IP's analytics and telemetry to chart the morning ramp and correlate SNAT counts with error rates. Set alerts on the rate of change, not only on a static threshold, because the problem builds quickly at 09:00. Include pool-member health and response time so an application bottleneck is not misdiagnosed as a network problem.
After a change, keep the alert thresholds and review them weekly. The goal is to see the SNAT pool approaching its limit while there is still time to add addresses, rather than learning about it from user reports.