Intermittent ELB 504s During the Settlement Window
A retail platform runs its order API on six ECS instances (c7.xlarge) behind a dedicated load balancer, spread evenly across two AZs in one region. Health checks are TCP:8080 every 5 s. Between 09:00 and 11:00 daily the ELB reports 0.2-0.4% HTTP 504 responses, yet application logs show the affected requests were still executing when the load balancer stopped waiting, and backend CPU never exceeds 35%. In the same window the batch settlement job fails with connection timeouts when calling a partner payment API over the internet. That traffic leaves through one NAT gateway provisioned at the smallest specification, and CES shows its SNAT connection count pinned at the ceiling whenever the outbound failures occur. The application opens a fresh HTTPS connection for every partner call, uses a 60 s socket timeout, and its request thread pool is fixed at 200 threads shared by inbound and outbound work. No ELB configuration has changed in the past month.
1. Identify the primary fault domain and explain how the two symptoms are causally connected. Cite the evidence that discriminates your explanation from a load balancer or application performance problem.
Read the worked answer
Primary fault domain: the egress path, specifically SNAT connection exhaustion on the NAT gateway - not the ELB, not backend CPU.
Causal chain:
- The application builds a new TCP connection for every partner API call (no keep-alive, no pooling).
- At peak, settlement drives new connection attempts at a rate multiplied by the 60 s hold time; the NAT gateway must keep a SNAT entry per concurrent flow.
- The smallest NAT specification's SNAT connection ceiling is reached, so new outbound flows cannot be allocated and
connect()blocks until the 60 s socket timeout. - Each blocked call occupies one of the 200 shared request threads. Within minutes the pool is saturated, so inbound order requests and even health-check probes to :8080 queue behind blocked outbound calls.
- The ELB's backend timeout expires on queued requests and it returns 504, even though the app is still processing.
Discriminating evidence: CPU is only 35% (not a compute bottleneck); the failing interval tracks the batch job, not user traffic; the same window shows the SNAT metric pinned at its ceiling. That correlation, not the ELB config, is what identifies the real fault domain. The 504s are collateral damage from thread starvation caused by blocked egress.
2. An engineer proposes raising the ELB backend timeout from 60 s to 300 s. Evaluate that change, then give the remediation set ordered by impact.
Read the worked answer
Raising the ELB timeout to 300 s does not fix anything and makes the incident worse. It does not free SNAT entries, does not reduce connection churn, and simply keeps client connections and backend threads occupied five times longer. The 504 becomes a 300 s hang - worse user experience and a larger queue backlog. Timeout tuning is a symptom treatment, and only valid after the resource shortage is gone.
Remediation, ordered by impact:
- Break the coupling. Give partner-API calls their own bounded thread pool and connection pool, enable HTTP keep-alive so a connection is reused instead of re-created per call, and cap concurrent outbound calls. Reuse removes most SNAT churn outright.
- Size the egress correctly. Required SNAT entries ≈ peak new connections/s × average hold time. Either charge the NAT gateway to a larger specification, or better, deploy one NAT gateway per AZ and route each subnet to the NAT in its own AZ - NAT gateways are AZ-scoped, so a single gateway is also an AZ-failure risk.
- Fail fast. Set the application socket timeout below the ELB backend timeout, add a circuit breaker with backoff, and make settlement skip-and-retry rather than block.
- Then tune ELB timeout and health-check interval.
Keep the ordering: step 1 alone may drop SNAT usage below the ceiling, which is why it precedes buying capacity.
3. Which metrics and alarms would you put in place so the next occurrence is detected before customers report it?
Read the worked answer
Monitor along the causal chain, not just at the symptom:
- SNAT connection count and new-connection rate on the NAT gateway in CES, alarm at ~70% of the specification ceiling - this is the earliest indicator and fires well before thread starvation.
- Application thread pool: active threads, queue depth and rejection count via AOM; alarm on queue depth growth trend rather than a fixed number.
- Outbound dependency health: p99 latency and error rate of the partner API, ideally via a synthetic probe every minute, so an external degradation is distinguished from internal capacity.
- ELB: 504 count, upstream response time and unhealthy backend count - leading signal of queueing, but treat it as a consequence metric.
- Correlation: build one AOM dashboard that overlays the settlement job schedule with SNAT usage and thread pool depth. The schedule overlay is what would have shown the pattern on day one.
Set the alarms on the earliest link (SNAT and queue depth), not only on 5xx, and review thresholds after every seasonal peak.