Intermittent 5xx on a Two-Data-Center Order Pool
Your company runs a pair of BIG-IP i5800 appliances in active/standby for an order-entry application. The virtual server vs_orders_443 uses a pool of six members split evenly across two data centers, an HTTP monitor (GET /health, interval 5s, timeout 16s, receive string "OK"), a one-connect profile, and SNAT automap. Users report intermittent 502 and 503 errors, roughly 2 percent of requests, concentrated around 10:00 and 14:00. The pool member status page shows members toggling down and back up within 30 to 60 seconds several times a day. The application team says /health returns 200 with body "OK" but occasionally takes 18 to 22 seconds during garbage collection. Appliance CPU sits at 40 percent, and the standby unit has been healthy for 90 days. A change freeze starts Friday, and the application owner wants the errors gone before the Monday peak.
1. Before changing any configuration, which evidence would you collect to separate a monitor-configuration problem from an application-capacity problem, and what would each outcome tell you?
Read the worked answer
Collect three streams of evidence and correlate them on the same timeline.
- Monitor log on both units -
/var/log/ltm, filtered for monitor entries around 10:00 and 14:00. Look for the exact failure text ("no response within 16 seconds", "unable to connect", or a receive-string mismatch). - Live member state -
tmsh show ltm pool <pool_name> membersandtmsh show ltm monitor http <monitor_name>, which show configured versus observed values and the last transition time. - Application-side timing - the app team's /health response-time graph around the same windows.
The diagnostic logic: if the appliance logged "no response within 16 seconds" while the application recorded a 200 in 18 to 22 seconds, the monitor fired before the application answered - a configuration mismatch, not capacity. If instead the appliance logged connection refused or a non-200 status, or the errors cluster on one data center's members, the problem is the application or the path. The 10:00 and 14:00 clustering also raises the question of a scheduled batch job or GC cycle; confirm that before assuming the monitor is at fault.
2. Recommend a specific monitor change and justify the numbers. What new risk does the change create, and how do you bound it?
Read the worked answer
The current interval/timeout pair is 5s/16s, which is at the edge of F5's usual guidance of timeout = (3 x interval) + 1. An endpoint that legitimately takes 22 seconds will therefore be marked down on every slow GC cycle. Recommended change: interval 10s, timeout 31s, which preserves the 3x+1 relationship and covers the observed 22-second worst case with margin. Alternatively, keep a fast TCP monitor for availability and attach a separate, more tolerant HTTP monitor for content health; the pool uses the AND of the two.
The new risk is slower failure detection. A member that genuinely dies will continue receiving traffic for up to 31 seconds. Bound it with one-connect idle timeouts, a conservative max connections or connection-rate limit per member so a dead node cannot accumulate sessions, and an alert on repeated monitor transitions so a real failure is not masked by the tolerant timeout. Validate by watching tmsh show ltm pool members through the next two batch windows.
3. The change freeze starts Friday. Describe the rollout and rollback sequence you would run during the permitted window.
Read the worked answer
Treat this as a standard, reversible configuration change, not an emergency.
Pre-change: capture a UCS archive (tmsh save sys ucs pre_monitor_change_<date>), record the current monitor settings, confirm the standby is in sync, and get written application-owner sign-off for the window.
Change: apply the new monitor parameters (tmsh modify ltm monitor http <name> interval 10 timeout 31), then attach it to the pool. Change one attribute at a time and stay off the console for the initial 10 minutes.
Verify: watch tmsh show ltm pool members and /var/log/ltm for at least two monitor cycles past the next expected GC event; confirm no transitions and no new 5xx in the application logs.
Rollback: if transitions continue, restore the prior monitor values or load the UCS. Because the UCS replaces the whole configuration, note that any unrelated change made after the archive - including by another team - would be lost; that is the argument for restoring the two monitor values rather than the full archive, and for doing the change in a quiet window.