LFCA: Two Percent of Requests Return 502 at Peak

Two Percent of Requests Return 502 at Peak

A Django API runs under gunicorn in one container on a 2 vCPU VM behind nginx, started with --workers 4 --threads 2 --timeout 30 (the default). Baseline p95 is 180 ms, but p99 is 4.6 s, and during the 09:00-10:00 peak about 2.1% of requests return 502. The nginx error log shows repeated upstream prematurely closed connection while reading response header from upstream and occasional connect() to 127.0.0.1:8000 failed (111: Connection refused). Gunicorn logs [CRITICAL] WORKER TIMEOUT followed by worker kill and respawn. The application sets CONN_MAX_AGE=0, so every request opens a new PostgreSQL connection; Postgres has max_connections=100 and three other services already use about 60 of them. Deploys happen three times a day and are done with docker restart. The person on call says "it works locally, so it's infrastructure".

1. Explain the causal chain from a slow endpoint to a 502, and what each of the two nginx messages tells you.

Read the worked answer

One code path is slow enough to exceed 30 seconds occasionally. When that happens, gunicorn's arbiter sends a signal to the worker and replaces it.

upstream prematurely closed connection while reading response header: nginx had an established connection to a worker that was killed mid-request, so the socket closed with no response written. That is the worker-timeout path.
connect() ... Connection refused: for an instant there was no process listening on 127.0.0.1:8000 - typically when several workers had been recycled and the listen backlog was drained, or during the docker restart deploy gap. That is why the two messages appear together but have different causes.

2.1% of peak traffic maps to requests that hit the pathological endpoint often enough to tie up all four workers simultaneously. With --threads 2, a blocked worker also stalls its sibling thread, so a handful of slow requests can starve the whole pool faster than the worker count suggests.

There is a second, independent amplifier: CONN_MAX_AGE=0 means every request opens a Postgres connection. Under retry storms after 502s, connection churn can reach max_connections and produce FATAL: sorry, too many clients already, which looks to the user like yet another 502.

2. What would you collect to prove this, and how would you divide the fix between the application team and the platform team?

Read the worked answer

Evidence to collect: per-endpoint latency histograms from structured request logs (nginx $request_time and $upstream_response_time are available today); a count of WORKER TIMEOUT per hour correlated with the 09:00 peak; pg_stat_activity sampled during peak for connection counts and long-running queries; and log_min_duration_statement=500 to identify the offending query.

Split so the ticket does not bounce:

Application team owns the slow path: add the missing index, remove the N+1, or move the work to a background queue. They also own a defensible request budget - a statement timeout and an explicit error instead of an unbounded wait.
Platform team owns the failure radius: worker/timeout configuration, graceful reloads, a health endpoint that does not touch the database, and connection pooling.

Agree on a written acceptance criterion up front, for example "p99 under 2 s and zero worker timeouts in a 24-hour peak", so the change can be judged instead of debated.

3. Propose the configuration and architecture changes, and explain why raising --timeout is the weakest option.

Read the worked answer

Raising --timeout only lets a slow request hold a worker longer; it converts fast failures into a fully congested pool and delays detection. It is a valid temporary measure only alongside a hard cap on the slow path.

Better set:

Bound the work server-side. Set statement_timeout and an application-level deadline so a slow request fails at, say, 8 seconds with a clear 504 rather than occupying a worker for 30.
Connection management. Turn on CONN_MAX_AGE with a conservative value, or put PgBouncer in transaction-pooling mode in front of Postgres. Trade-off: pooling changes session semantics (no session-level advisory locks or SET persistence) and adds a hop.
Recycle deliberately. Use --max-requests with jitter so workers do not all restart together, and --graceful-timeout so in-flight requests finish.
Deploy without a gap. Replace docker restart with a SIGHUP graceful reload, or run two containers behind the proxy and drain one. Trade-off: needs orchestration or a proxy config change.
Health checks. A /healthz that does not touch the database prevents the orchestrator from killing pods during a database blip.

The order matters: cap the slow request first, then size the pool, then fix the deploy gap.