Monitors and health checking
Domain 5 covers health checking: how a monitor instance is scheduled, what makes it fail, and how its verdict rolls up into node, member and pool status. Questions often hinge on a member being marked down for a reason unrelated to application health.
How a check runs
- Each monitor type has an interval, a timeout, and optional send and receive strings.
- Defaults for common types (http, tcp, icmp, gateway_icmp) are interval 5 seconds and timeout 16 seconds; the rule of thumb is timeout ≥ (3 × interval) + 1.
- Transparent mode disables implicit formatting of the send string; Reverse inverts success; Alias sends the check to a different address or port.
- In-band monitors judge health from live production traffic instead of synthetic probes.
- Adaptive monitoring lengthens the interval during outages to reduce load.
Aggregation
- Monitors can be bound to a pool member, to a node, or both; a node marked down makes every member on it down regardless of member monitors.
- Pool status is driven by member availability, and virtual server availability follows the pool.
Worked example
An HTTP monitor sends GET /health HTTP/1.1 with a Host header and expects 200 OK. The member goes down after a release even though the site is healthy. Reasoning: /health now answers 302 Found to the login page, so the response never contains 200 OK and the check fails. Options: point the monitor at an endpoint returning 200, change the receive string to match the redirect, or match only a status line the application reliably returns. Verify with curl -i from a client following the same path the monitor takes.
Traps
- A timeout shorter than the interval, producing constant flapping.
- Forgetting a member is only marked down after the timeout expires, which looks like slow detection.
- Case-sensitive receive strings and stray whitespace in send strings.
- Binding a monitor to the node and then wondering why every member on it shows down.
- Assuming "unknown" means a failed check; it usually means the check never completed or monitoring is disabled.