DiskPressure, stuck Terminating pods and Pending replacements

(a) The kubelet's eviction manager sees nodefs.available/imagefs.available below threshold and tries image garbage collection. GC cannot free anything because containerd's content store is full on the same filesystem - reclaiming needs metadata writes and there is no headroom, hence freed 0B. The kubelet also cannot reliably write its own state under /var/lib/kubelet. Once the node controller stops receiving healthy status updates it marks the node NotReady and applies the node.kubernetes.io/disk-pressure:NoSchedule taint.

(b) Pod deletion is a two-party protocol: the API server sets deletionTimestamp, and the kubelet must stop the containers, tear down the sandbox, then post the final status. containerd on a full disk cannot remove snapshots, so that final update never arrives and the object sits in Terminating long after the 30 s grace period.

kubectl delete node removes only the API record. The kubelet process on the host is still running, re-registers within seconds against a still-full disk, and the pressure returns - while you have destroyed the diagnostic state (describe node, its conditions, its events). It does not free a single byte of disk.

Host first, API second.

1. Diagnose without changing anything: crictl df, du -sh /var/lib/containerd/*, journalctl -u kubelet -n 200. Decide whether it is images or writable snapshots.
2. Reclaim the safe way: crictl rmi --prune removes images not used by a container. Do not rm -rf under /var/lib/containerd/io.containerd.* - you will corrupt the content store and every image pull on that node will fail verification. Container logs under /var/log/pods live on the other volume here, so truncating them buys nothing.
3. Get to roughly 15% free, then systemctl restart kubelet, confirm DiskPressure False, then kubectl uncordon worker-4.
4. For the stuck pods: once crictl ps -a shows the containers are gone, kubectl delete pod <name> --grace-period=0 --force is safe. Forcing while the runtime still reports them creates a ghost pod that holds its name and IP and blocks the replacement.

Settings that would have softened this: imageGCHighThresholdPercent: 75 and imageGCLowThresholdPercent: 65 (GC runs earlier and harder), evictionHard: nodefs.available: 15%, and explicit kubeReserved/systemReserved so batch pods cannot consume the space the kubelet needs. Longer term, move containerd's root to a dedicated larger volume so image churn never competes with node state.

No. The taint exists precisely to stop more containers being created on a filesystem with no space left. Tolerating it means the kubelet accepts pods it then cannot pull images for - ImagePullBackOff, no space left on device in the runtime, and a broken node pushed further from recovery. It also hides the signal that a node needs capacity attention.

The fix is capacity: reclaim the disk or add a node. When the eviction manager observes free space above the threshold the condition clears and the taint is removed automatically - there is nothing to untaint by hand, and manually removing a pressure taint on a still-full node is strictly worse than tolerating it.

If a critical workload genuinely must run during the incident, the right lever is scheduling it elsewhere (node affinity, or a PriorityClass so it preempts batch work on healthy nodes), not tolerating a pressure taint. A temporary toleration is legitimate for one purpose only: draining a node you have already cleaned and verified.