Load Balancer Health Checks Behind a Content Delivery Layer
A media company serves a video-on-demand site through a content delivery network in front of an application load balancer. Origin servers are ECS instances in two zones. Users in one region report intermittent 502 errors that last 30–90 seconds, then clear. The load balancer's health check is HTTP GET /health on port 80 with a 5-second timeout, 2-second interval, and unhealthy threshold of 2. The /health endpoint queries the database. During peak traffic the database connection pool saturates, and /health begins taking 6–8 seconds. Meanwhile, cache hit ratio on the CDN is 92%, and the CDN is configured to forward all cookies and query strings to origin.
1. Explain the 502 bursts using the interaction between the health check design and database behavior, and propose a corrected health check.
Read the worked answer
Mechanism. The health check timeout is 5 seconds, but under pool saturation /health takes 6–8 seconds. Two consecutive timeouts (threshold 2, interval 2s) mark an instance unhealthy in roughly 9–14 seconds. The load balancer removes it from rotation; if enough instances fail simultaneously - which they will, because they share the same saturated database - the load balancer has no healthy targets and returns 502 to clients. When database pressure eases, /health recovers in seconds and the instances return, so the errors are intermittent bursts.
This is a cascading failure amplified by a deep health check: the health check depends on a downstream resource, so downstream slowness evicts healthy application servers, reducing capacity and worsening the saturation.
Corrected design:
- Make /health shallow - verify the process is up and able to accept connections; do not query the database on every probe. Move dependency checks to a separate, less frequent readiness endpoint or external synthetic monitor.
- Loosen thresholds to tolerate transient slowness: longer timeout (e.g., 10s), higher unhealthy threshold (e.g., 3–5), and a slow-start/grace period so recovered instances are not immediately re-probed into failure.
- Add connection-pool and database-saturation alarms so the real cause is alerted rather than masked by health-check churn.
- Consider circuit-breaking in the app so a saturated pool fails fast instead of hanging the probe thread.
2. The CDN forwards all cookies and query strings to origin. Quantify the likely effect on origin load and explain how fixing this changes the failure mode.
Read the worked answer
Effect. A 92% hit ratio means only 8% of requests reach origin - but forwarding all cookies and query strings prevents caching of anything that varies per user. If the cacheable surface is smaller than assumed, the true origin share could be far above 8%. Every request carrying a session cookie is effectively unique from the CDN's perspective, so the CDN degenerates into a pass-through for logged-in users, and origin absorbs their full request rate. That is exactly the load that saturates the database pool and triggers the health-check cascade.
Fix. Configure cache keys deliberately: strip cookies except those that genuinely change the response, normalize or exclude cache-busting query parameters, and set long TTLs on static media segments. Keep a separate behavior for authenticated API paths that must reach origin.
How the failure mode changes. With a higher effective hit ratio, origin request rate drops, the database pool stops saturating, /health stays fast, and the 502 bursts disappear. But note the trade-off: a more aggressive cache key means stale content risk. If personalization leaks into cached responses, users could see another user's data. So the fix must be paired with correct cache-control headers (private/no-store on authenticated responses) and verification that no personalized content is cached at the edge.
The exam judgement: the 502s look like a load-balancer problem but are a cache-key and health-check problem, and fixing caching without fixing cache safety creates a worse incident.