The Pool Is Down but the Application Is Up
A pool of three nodes on port 8080 uses a custom HTTP monitor: send 'GET /health HTTP/1.0\r\n\r\n', receive '200 OK', interval 5, timeout 16. The application team deploys a new version that requires a Host header and redirects HTTP to HTTPS with a 301 for any request without Host. The BIG-IP marks all three members down within a minute; the virtual server's action on service down is Reject, so users get connection resets. From a jump host, curl -H 'Host: app.example.com' http://10.0.0.11:8080/health returns 200; curl without Host returns 301. The application is healthy.
1. Explain why the monitor marks healthy members down. Include what the BIG-IP actually receives and how the receive string is evaluated.
Read the worked answer
Root cause
The custom monitor sends an HTTP/1.0 request with no Host header. The new application version treats a request without Host as invalid and returns a 301 redirect to HTTPS. The BIG-IP connects successfully and receives an HTTP response, so this is not a TCP or port failure. However, the monitor's receive string is '200 OK'. When the response contains '301 Moved Permanently' instead, the receive string is not found and the monitor marks the member down.
The jump-host test proves the application is healthy: the same URL returns 200 when the Host header is present. The monitor is testing a request path that no longer represents a valid health check.
Because all members are marked down and the virtual server's action on service down is Reject, the BIG-IP resets client connections. That converts a monitoring misconfiguration into a full outage.
2. Write the corrected monitor configuration and explain each choice.
Read the worked answer
Corrected monitor
- Type: HTTP
- Send string:
GET /health HTTP/1.1\r\nHost: app.example.com\r\nConnection: close\r\n\r\n - Receive string:
200 OK - Interval: 5 seconds
- Timeout: 16 seconds
Why
HTTP/1.1 requires a Host header. Adding Host: app.example.com makes the request match what real clients send, so the application returns 200 instead of 301. Using Connection: close avoids relying on keep-alive behavior and makes the response complete predictably. The receive string still checks for the success status, so the monitor fails if the app returns 500 or a redirect.
The timeout is 16 seconds, which is consistent with the common guidance of at least three times the interval plus one second for a 5-second interval. That avoids flapping when the application is briefly slow. Do not set the receive string to '301', because that would treat a redirect as healthy and could mask an application failure. If the health endpoint is only valid over HTTPS, use an HTTPS monitor with the same Host header and expect 200.
3. What immediate mitigation would you apply, and what should be changed afterward to reduce blast radius?
Read the worked answer
Immediate mitigation
Fix the monitor first, because the application is healthy and the monitor is wrong. Apply the corrected send string to the monitor and force a re-check. Members should come back up within one interval. If there is a delay before the change can be made, temporarily set the virtual server's action on service down to None so traffic is still forwarded, but treat this as a short-term bridge: it removes protection if a member is genuinely down.
Reduce blast radius
- Do not use Reject as the action on service down unless the business prefers a fast failure over sending traffic to unverified members. Decide deliberately: Reject gives clients an immediate reset, None keeps trying, and Drop causes timeouts.
- Use a pool availability requirement that reflects the design, for example 'at least 1 of' rather than 'all', if the application can run on a single member.
- Monitor a dedicated health endpoint that does not redirect, and make the monitor part of the application's API contract so a release cannot silently break it.
- Test monitor changes with a real request from the BIG-IP before trusting the result, and alert on pool member state changes so a false down is caught quickly.