A Nightly Total Outage and a Monitor That Never Notices Real Failure

This is a correlated failure, not six independent ones. Something at 02:00 - log rotation, an application pool recycle, a backup - makes / slow or briefly unavailable on all members simultaneously. With interval 5 s, timeout 3 s and 3 retries, each server is declared down within roughly 15–20 seconds, so all six leave together and the VIP has nothing to serve. Hence the 12-minute 503.

The monitor is testing the wrong thing: / is a heavyweight path that shares fate with the recycling pool.

Redesign:

• Point the check at a dedicated, dependency-light endpoint such as /healthz that stays fast during housekeeping, and match on a response body pattern as well as status 200.
• Raise the timeout above the observed p99 for that endpoint. One slow poll should never remove a server.
• Keep retries at 3 or more, so detection is a trend rather than a single sample.
• Stagger or exclude members during the 02:00 job so no more than one can be out at a time.
• Give the VIP a defined all-down behaviour - a maintenance page or redirect - so a correlated event degrades gracefully instead of showing a bare 503.

The target is detecting real failure, not noticing housekeeping. Track down/up events per member after the change; six simultaneous transitions are the signature of the fault recurring.

Detection time is roughly interval × retries, plus the timeout of the final probe.

• 5 s interval with 3 retries is 15–20 s. Too slow for the stated goal.
• 1 s interval, 1 s timeout, 3 retries is about 4 s - but that is six probes per second per server, and any three dropped packets, microburst, SYN backlog or garbage-collection pause removes a healthy server. At six members this fires often, and a false removal pushes its traffic onto the survivors, raising their load and their own chance of failing.

Deploy two tiers:

• A shallow, cheap gate - TCP connect or HTTP on a static file - at 2 s interval, 1 s timeout, 2 retries. Fast enough to matter, not twitchy.
• The dependency check on /healthz at 10 s, used for alerting and paging, not for instant traffic removal.

Then push the 5-second requirement back as a design question. The ADC can only remove what the application admits to; a server that accepts a connection while its database pool is dead is an application readiness problem, best solved with circuit breaking and fast failure in the app itself.

Finally, any change that can mark all members down at once deserves change review, because the failure mode is a total outage, not a slow response.

Health monitors are shared, named objects referenced by service groups. Editing hm-http changes the check for every group bound to it, on every virtual server, and the check itself evaluates a member's address and port, not a VIP. One edit therefore propagates fleet-wide, which is exactly how a change made for one service took down another.

Consequences worth stating explicitly:

• Interval, timeout and expect changes are global to the monitor; they belong in change control with a test against every consumer.
• The same server can be up in one group and down in another if they use different monitors - usually desirable when the services have different dependencies.
• If the portal needs a fast shallow check and the API needs a deep dependency check, define two monitors rather than compromising one shared object.

Safe rollout:

1. Create the new monitor under a new name; leave the existing one untouched.
2. Bind it to one low-risk group and watch down/up events for at least one full business cycle, including the 02:00 window.
3. Confirm the member state matches reality by testing a server manually.
4. Rebind remaining groups one at a time, expecting a brief state re-evaluation on each rebind.
5. Keep the old monitor defined until the new one has survived a month, then delete it deliberately.