Node Memory Pressure and Eviction Correlation in EKS
Your team runs a 48-node EKS cluster of m5.2xlarge instances with 32 GiB each, spread across three availability zones, hosting roughly 1,400 pods. Both the infrastructure agent and the Kubernetes integration are deployed. Two problems surfaced this week. First, node ip-10-0-14-212 reported 96% memory used for five hours, and the kubelet evicted two pods from the checkout namespace, yet the sum of container working sets on that node was only about 46% of capacity. Second, a host-not-reporting alert fired three times in 24 hours for three different nodes, each time for four to six minutes, and each cleared on its own; the platform team patched those nodes nightly. Checkout p95 latency rose from 240 ms to 900 ms during the eviction window, and APM shows the slow instances running on that node.
1. How do you explain 96% node memory used against a 46% sum of pod working sets, and what should the alert actually be?
Read the worked answer
Memory used on a Linux node includes page cache, which the kernel reclaims on demand; it is not the same thing as memory pressure. A node at 96% used with pod working sets summing to roughly 15 GiB of 32 GiB leaves 12 to 13 GiB outside the containers: page cache from heavy file or log writes, tmpfs, kernel slab, or a process running directly on the host.
Check before changing anything:
- Filesystem growth. A chatty pod writing logs faster than rotation can keep up, or a container runtime directory ballooning, is the classic cause of eviction with low working sets.
- Kubelet eviction thresholds and the memory available signal. Kubelet evicts on its own calculation, which can trigger on unreclaimable cache plus working set even when pods are modest.
- Pod requests versus limits. Evicting a pod with inflated requests is a scheduling and rightsizing problem, not a memory problem.
- Reporting freshness. If the infrastructure agent and the Kubernetes integration disagree, or one is stale, your ratio is fiction.
Alert on pressure, not percentage: kubelet eviction events, the MemoryPressure node condition, pods pending for insufficient memory, and node not-ready. Keep a low-urgency informational alert at 90% used for trend with a named owner, and never page a human on reclaimable cache.
2. Host-not-reporting alerts fire for a few minutes on patched nodes. How do you distinguish an agent restart from a real node failure, and how do you reduce the noise?
Read the worked answer
The question is whether the node is unhealthy or only your visibility of it is missing.
- Determine which signal disappeared. If the infrastructure agent stopped while kubelet and pod metrics continued, the agent process restarted, likely from patching, package upgrade, certificate rotation, or the agent itself being OOM-killed. The node was fine. If both stopped while pod logs and APM traffic continued, suspect a network path or endpoint change rather than the host.
- Correlate timing. Four to six minute gaps recurring at the same hour on nodes patched the same night is the signature of an agent restart plus heartbeat gap. Query reporting TIMESERIES 1 minute and compare against the patch schedule and against a node that was not patched.
- Verify independently. Cloud provider node status, the NodeReady condition, and APM throughput from instances on that node together tell you whether workloads kept serving.
- Reduce noise without losing coverage. Use a short agent-not-reporting condition with an appropriate threshold, a separate longer node-down condition for true unavailability, declared maintenance windows for patch cycles, and a distinct alert for nodes whose pods stop serving traffic.
The test of the redesign: a genuine node loss still pages within minutes, while a controlled agent restart during a change window does not wake anyone.
3. Checkout p95 rose from 240 ms to 900 ms during the eviction window. How do you establish whether eviction caused the latency or both share the node as a root cause?
Read the worked answer
Correlation in time is not causation, and the ordering of events decides the story. Start by bounding the window precisely with eviction timestamps from the Kubernetes data, then query SELECT percentile(duration, 95) FROM Transaction FACET host TIMESERIES 1 minute for instances on the affected node against instances on healthy nodes. Latency confined to pods on that node points at the node rather than the code.
Use entity relationships rather than hostname string matching. From the Kubernetes node entity, open the cluster explorer to see which pods were scheduled there, then move from those pods to their APM entities. That chain maps pod to service instance to transaction latency without manual IP joins and keeps working as pods reschedule.
Then check mechanism and order. If latency rose before eviction, the driver was memory pressure: reclaim, page faults, GC, and CPU throttling were already hurting the pods, and eviction was a symptom. If latency rose only after, look at rescheduling effects, cold caches, connection pool re-establishment, and reduced replica capacity. Also check container CPU throttling on surviving pods, because heavy reclaim throttles everything on the node.
Recommendation: raise the incident against the node-level cause, not the checkout service, and keep alerting on eviction events plus per-node latency regression so the next occurrence surfaces before customers notice.