Eviction Order on a Memory-Pressured Node

Eviction order is driven by QoS class first, then usage relative to request:

1. batch is BestEffort (no requests, no limits) and is evicted first, in order of largest memory usage among BestEffort pods.
2. web is Burstable (requests < limits). Burstable pods are ranked by how far their usage exceeds their request - here 3.5 GiB used against a 2 GiB request, a 1.75x ratio.
3. cache is Guaranteed (requests == limits for every container) and is evicted last, and in practice only under severe pressure.

The lead's claim is wrong on two counts. Absolute consumption is not the ranking key - cache uses more per pod than nothing else in the Burstable tier and is still safest. And eviction protects the node, not the workload's business value: batch looks "small" but is the cheapest to lose from the kubelet's point of view, which may still be the wrong business outcome. If batch matters, give it requests and limits so it stops being BestEffort, and if web must survive pressure, raise its requests to equal its limits to move it into Guaranteed - at the cost of schedulability, since a 3.5 GiB request per pod is much harder to place than a 2 GiB one.

Three separate mechanisms are being conflated.

• Container OOMKill is a cgroup limit event: the container exceeds its own limits.memory and the kernel kills the process inside it. Raising limits does reduce this, but it does nothing for the node-level problem.
• Node eviction is what is actually happening here (MemoryPressure). It is driven by node allocatable memory and pod QoS class, and raising limits while leaving requests alone makes the node more overcommitted, not less. The sum of limits can legitimately exceed node capacity; that is normal overcommit, but it means limits are not a capacity guarantee.
• Scheduling uses requests only. If the engineer raises limits without requests, schedulability is unchanged and the node pressure gets worse; if they raise requests too, most of these pods will stop fitting at all.

The durable fix is to set requests from observed steady-state usage (for web, somewhere near 2.5-3 GiB rather than 2 GiB), keep limits modestly above requests for burst headroom, and give batch real requests so it is neither BestEffort nor an invisible consumer of node memory. Also configure eviction thresholds and consider priority classes if some workloads must outrank others.

The scheduler does not read kubectl top. It filters nodes using sum of pod requests against node allocatable, after subtracting the kubelet's reserved capacity. top reports actual working sets, so the two numbers answer different questions: a node can have 3 GiB of genuinely free RAM and still have zero requested headroom, because pods have reserved memory they are not currently using.

What to check, in order:

1. kubectl describe node <node> - look at Allocated resources for memory requests versus allocatable, and at the Insufficient memory taint/condition.
2. Whether the pod's request (2 GiB for web) exceeds the largest free request slot on any node. Requests are atomic per pod - the scheduler does not split a pod across nodes - so 3 GiB free spread as 2 x 1.5 GiB places nothing.
3. Whether DaemonSet requests are consuming far more than the assumed 5 GiB.

Mitigations are: lower the request if 2 GiB is inflated, enable the descheduler or rebalance, add a node, or use a topology/preemption-aware PriorityClass so a critical pod can preempt a lower-priority one. Reducing the container's usage will never move a Pending pod; only requests and allocatable matter.