CKA (Other exams): A Deployment stuck one replica short: taint, requests, and a priority class that cannot help

Compute free CPU per node from requests, not from usage:

• node-a: 8 vCPU allocatable − 2000m taken by DaemonSet/system pods = 6000m free → 6000/1500 = 4 pods fit.
• node-c: 4 vCPU − 2000m = 2000m free → 1 pod fits (and only 1 - the second would need 3000m).
• node-b: 6000m free but tainted, so it is scored as unavailable for this workload → 0 pods.

4 + 1 + 0 = 5, which is exactly what is running. The sixth replica fails because node-a has 6000m and 4 × 1500m = 6000m leaves zero left for a fifth pod on that node.

Read the message carefully: the scheduler lists per-node reasons, and only one node is tainted. So the taint is not the sole cause - even if node-b vanished from the cluster entirely, you would still have 5 runnable replicas. Confirm the numbers rather than trusting them: kubectl describe node node-a | sed -n '/Allocated resources/,/Events/p' shows the summed requests, kubectl get pods -A -o json summed against status.allocatable, and kubectl top pod -n batch shows whether 1500m bears any relation to real usage.

Arithmetically yes: node-a 6000/1200 = 5 pods, node-c 2000/1200 = 1 pod → 6 total. But it is a fit with zero headroom, and you should say so before shipping it:

• node-a lands at exactly 6000m of 6000m free. Any DaemonSet rollout that briefly runs a second pod, any node drain, or a kube-system sidecar growing its request pushes a replica back to Pending.
• node-c can host exactly one replica, and it is also labelled disk=slow, so that replica is on the weakest node with no redundancy.
• Memory is not the constraint here (3Gi × 5 = 15Gi against ~28Gi free on node-a), but lowering only CPU makes the pod's request profile meaningless; check memory too before assuming it is free.
• Requests also drive QoS. With no limits the pods stay Burstable either way, so the change does not improve eviction behaviour - it only games the scheduler.

A defensible version: measure with kubectl top pod -n batch over a full report cycle, set the request near the observed p95 (if that is ~900m, a 1000m request leaves real slack), and add a CPU limit above the request so a runaway pod does not starve its neighbours. Document the headroom calculation in the change - the next person to add a DaemonSet needs to know the node is full.

Preemption is not extra capacity; it is a swap. When the scheduler fails to place a pod on every node, it re-runs the filter with lower-priority pods on that node hypothetically removed and checks whether the pod would then fit. If the failure is a node selector or taint mismatch, removing pods changes nothing - there is no victim that makes a tainted node admissible. So on node-b, priority 100000 buys exactly nothing.

Where it could work: node-a. If nightly requested, say, 1000m, the scheduler would find node-a full of priority-0 report-gen pods and nominate one as a victim. Things that still block or slow it:

• Pods with priority greater than or equal to the incoming pod are never victims. kube-system DaemonSet pods carry system-node-critical (2000000000) and are untouchable.
• Preemption honours PodDisruptionBudgets on a best-effort basis; if the nominated victims are covered by a PDB that would be violated, the scheduler may pick another node or not preempt at all.
• Victims are terminated gracefully (default 30s), so the incoming pod waits, and in the meantime both exist on paper.

Diagnose the actual failure before assuming preemption is the answer: kubectl describe pod nightly-... and kubectl get events -n batch --field-selector reason=FailedScheduling distinguish Insufficient cpu (preemption can help) from node(s) had untolerated taint (it cannot).