EKS Memory Pressure: The Node Percentage Is Misleading

The two numbers measure different things. Node memory used can include page cache, buffers, kernel memory, system daemons, the container runtime, the infrastructure DaemonSet, and memory reserved for system processes. Pod working sets exclude some of that and are not a clean partition of node capacity. The percentages may also come from different sources with different denominators: node total versus allocatable memory, and working set versus requested memory.

Eviction makes the gap operationally important. The kubelet triggers eviction based on available memory and configured thresholds, not on a nice round 85% used. If requests are far below actual usage, the scheduler believes the node has more room than it really does, so it packs pods onto a node that is already close to pressure. A 92% figure with evictions is therefore a real signal even if the pod sum looks low.

Raising the threshold to 95% would hide the warning and postpone eviction, making the failure more abrupt. The right response is to alert on memory pressure and available memory, then fix requests, limits, and scheduling.

Alert on conditions that directly precede or indicate eviction:

• Available memory as a percentage of total, for example node_memory_MemAvailable_bytes divided by node_memory_MemTotal_bytes, with a threshold that reflects the kubelet eviction reserve rather than a generic 85%.
• The Kubernetes node condition MemoryPressure, because it is the control plane's own view of pressure.
• Eviction events and pod restarts, especially repeated restarts on the same node, from Kubernetes events or kube-state-metrics.
• Container memory working set against limit for pods that have limits, and against request for pods that do not.

Keep a diagnostic alert on the node memory percentage, but do not page solely on it. Page on a combination of available memory, memory pressure, and restart or eviction rate. This preserves early warning while avoiding pages caused by cache or by a large DaemonSet. The alert should include the node, the top memory consumers, and the affected pods so the responder can act immediately.

Start with requests and limits. Set memory requests close to observed p95 working set, not to a low guess, so the scheduler stops overcommitting nodes. Add limits where appropriate, but recognize that limits can cause OOMKills when set too low; for critical services, consider Guaranteed QoS with requests equal to limits. Raise requests gradually and watch scheduling pressure.

Then improve scaling and placement. HPA on CPU alone will not help a memory-bound service; add memory or custom metrics where they reflect the real bottleneck. Enable cluster autoscaler or Karpenter so new nodes arrive before pressure becomes eviction. Use priority classes so critical pods are not evicted first, and spread large consumers with pod anti-affinity.

Finally, verify with data: node available memory, eviction count, OOMKilled count, and restart rate by service. If pages stop only because the threshold moved, the system is still failing; the success criterion is fewer evictions and stable memory headroom under peak load.