Nightly Evictions in a Cluster That Is Only 71% Committed

Scheduling is driven by requests only. The scheduler sums requests against allocatable; limits never enter the scheduling decision and only constrain runtime. The 40 batch pods contribute zero to the 71% figure, so the scheduler believes there is headroom that does not exist in practice. That is why the cluster looks comfortable and behaves as if it is full.

Eviction is driven by usage above request. When memory.available drops below evictionHard, the kubelet ranks pods by how far their working set exceeds their request, then by QoS class, then by Priority. A batch pod with no memory request has an effective request of 0 and a 900 MiB working set, so it is ranked first, far ahead of an api pod using roughly what it requested. The three api kills happened because pressure was severe enough that rank ordering ran out of batch victims.

The 900 MiB number exposes a second bug. The 512Mi limit is below real peak, so pods are OOMKilled and restart repeatedly, which itself churns memory and amplifies pressure. The limit is not protecting the node; it is generating restarts. The fix is not a bigger limit, it is a request near p95 usage (say 1Gi) with a limit at or modestly above it.

1. Set honest requests. Memory request 900Mi-1Gi per batch pod (p95 of observed peak), CPU request 200-300m. Keep memory limit equal to request unless there is evidence of drift, so OOM behaviour stays predictable.
2. Make the accounting unavoidable. A LimitRange with default request/limit plus a ResourceQuota in batch that rejects pods without requests. Otherwise the next CronJob repeats the mistake.
3. Cut concurrency before adding hardware. concurrencyPolicy: Forbid and a bounded startingDeadlineSeconds on the CronJobs; check whether fan-out can be staggered across the night window.
4. Protect the latency-sensitive tier. A PriorityClass with high value on api and a low one on batch means the scheduler preempts batch pods rather than letting the kubelet evict api.
5. Re-run the arithmetic. 40 x 1Gi = 40 GiB of new requests against roughly 9 x 31 GiB = 279 GiB allocatable and 198 GiB already requested. The honest number may show one or two nodes are genuinely required -- that is a valid outcome, but now it is justified by a real figure rather than a guess.

Monitor kube_pod_container_resource_requests versus allocatable per node, eviction counts, OOMKilled restart counts, and Pending pod duration.

It changes the symptom, not the cause. A 400Mi limit against a 900 MiB working set guarantees OOMKills: the pods will die by SIGKILL on nearly every run, jobs will retry, and because the container never holds a large working set the node may look calmer while the workload stops completing. The eviction signal is replaced by a restart signal plus failed CronJob history.

It also leaves the scheduler blind in exactly the same way. With no request, those pods still contribute zero to the committed calculation, so any other workload that does request resources can be scheduled onto the node and the node will again cross evictionHard. The underlying defect -- runtime demand invisible to the scheduler -- is untouched.

A defensible version of the colleague's idea would be to genuinely reduce the workload's memory appetite (smaller batch chunks, streaming instead of buffering), measure the new p95, and then request that value. Shrinking the limit alone is a self-inflicted outage with better monitoring.