I85PREP
⌘ K
ACE-Associate study materials
ACE-Associate study resources

Intermittent 503s Behind a Load Balancer

ACE-Associate · Put your knowledge to work in context.

All case studies

Intermittent 503s Behind a Load Balancer

An HTTP(S) load balancer fronts a regional managed instance group of 6 VMs running a Java service. Users report intermittent 503s during traffic peaks. The backend service health check is GET /healthz expecting HTTP 200; the app returns 200 only after JVM warmup (~40s). The MIG uses an autoscaler with target CPU 0.6. During peaks, CPU hits 0.9 on all instances, new instances take ~90s to become healthy, and the load balancer logs show backend_connection_closed_before_data_sent_to_client and health check failures on newly added instances.

1. Diagnose the most likely causes of the 503s during peaks.

Read the worked answer

Two compounding causes:

  1. Capacity lag. Autoscaling on CPU at 0.6 reacts after saturation. With a 90s cold-start (40s warmup + boot), new capacity arrives after the peak has already caused failures. CPU-based autoscaling is a lagging signal for request-driven load.
  2. Health check vs. warmup. New instances fail /healthz for ~40s while warming up. If the health check's unhealthyThreshold is short, instances flap; and if the app starts accepting connections before warmup completes, in-flight requests get dropped - consistent with backend_connection_closed_before_data_sent_to_client.

The 503s are the LB having no healthy backends to route to, not a code bug.

2. Propose fixes and explain which metric you would autoscale on instead of CPU.

Read the worked answer

Fixes:

  • Autoscale on a leading indicator. Use https/backend_request_count or loadbalancing.googleapis.com/https/request_count per instance (or concurrency) with a target that keeps instances below saturation, so scale-out starts before CPU pins. Add a small minimum and a scale-out stabilizer (short) with a longer scale-in window to avoid flapping.
  • Separate liveness from readiness. Make /healthz return 200 only when the app can serve traffic; set initialDelaySec and a health-check unhealthyThreshold that tolerates the 40s warmup without marking instances unhealthy.
  • Optionally pre-warm or use a startup probe pattern so the LB doesn't route to warming instances.

Verification: replay a peak load test and confirm no 503s and that scale-out begins before CPU crosses ~0.7.