I85PREP
⌘ K
IBM Cloud Professional SRE study materials
IBM Cloud Professional SRE study resources

Intermittent 502s at 02:00 UTC on a Dallas IKS Cluster

IBM Cloud Professional SRE · Put your knowledge to work in context.

All case studies

Intermittent 502s at 02:00 UTC on a Dallas IKS Cluster

A checkout service runs on IBM Cloud Kubernetes Service in us-south across three zones: 12 worker nodes (8 vCPU/32 GB), 60 pods of a Node.js API behind a VPC Application Load Balancer. The SLO is 99.95% successful requests over 30 days. Every Tuesday around 02:00 UTC, for 4–7 minutes, the ALB returns 502s to about 3% of requests; there is no deployment at that time. IBM Cloud Monitoring (Sysdig) shows one or two backend pods with p99 latency of 8 s and a drop in ready endpoints; node CPU stays below 45%. The ALB health check is HTTP GET /healthz with a 2 s timeout and 2 retries. A nightly in-process report job runs at 02:00 in every pod with random jitter, and a log-agent DaemonSet ships to IBM Cloud Logs. The on-call pages on a 5xx-rate alert with no pod or zone attribution and has twice restarted the deployment, which cleared the symptom.

1. Using only the telemetry described, how would you localize the cause and confirm it before changing anything? List the signals and the observation for each.

Read the worked answer

Reasoning: a 502 from the ALB means the proxy could not get a valid response from a backend, so first separate ALB health-check failures from application 5xx.

  1. Scope by dimension: in Sysdig, split the 502 rate by pod, zone, and kube_service, and compare with ready-endpoint count. If failures cluster on one node or zone, suspect node I/O or the DaemonSet; if they follow a 4–7 minute cadence across zones, suspect the in-process job.
  2. Correlate with container CPU throttling (container_cpu_cfs_throttled_seconds_total), event-loop lag, GC pauses, and /healthz latency percentiles. A 2 s probe timeout with a blocked event loop will mark a pod unready even though the process is alive.
  3. Check Kubernetes events for readiness-probe failures and endpoint removals at the same timestamps, and IBM Cloud Logs for report-job start/end and ECONNRESET entries.
  4. Confirm with a controlled experiment: disable the job in one canary Deployment cohort, then compare its 502 rate with the control group. Do not restart first; restarts destroy pod-level evidence and only move traffic to other pods that will hit the same job.

2. What remediation and alerting changes would you make, and in what order?

Read the worked answer

Order matters: fix the mechanism, then fix detection. First move the report job out of the request-serving container into a Kubernetes CronJob with its own resource requests/limits and concurrencyPolicy: Forbid. If it must remain in-process, make /healthz a cheap liveness check and use a separate readiness check that reflects ability to serve, with a timeout above the observed p99 but below the ALB unhealthy threshold. Do not simply raise the timeout as the primary fix; that hides saturation. Add a PodDisruptionBudget and topology spread constraints so a node or zone loss does not remove too many endpoints. Second, replace the flat 5xx alert with multi-window burn-rate alerts on the request-based SLI, carrying pod and zone labels, plus an external synthetic check. Pair 502s with ready endpoints and probe latency on one dashboard. The remediation addresses the trigger; the alerting change reduces MTTR because the first page names the failing dimension instead of prompting a restart.

3. How does this affect the 99.95% SLO, and what do you tell the product owner?

Read the worked answer

Calculate the budget: 99.95% over 30 days allows about 21.6 minutes of full outage equivalent. The SLI is request-based, so convert the incident: 3% errors for about 6 minutes per week is roughly 0.18 'full-outage-equivalent' minutes per week, or under 1 minute per month. That is small against 21.6 minutes, so this is not yet a budget crisis. The larger signals are the 8 s tail latency, repeated paging, and the fact that restarts are being used as a fix. Tell the product owner: the average is hiding a chronic defect; the error budget is not the only risk measure. Prioritize the CronJob separation and the alert redesign, and track weekly 502 count and p99 as leading indicators. If the job cannot be isolated before the next Tuesday, schedule a temporary traffic shift or increase probe tolerance with a documented expiry, not a permanent timeout increase.