CKA (Other exams): Node NotReady after reboot: kubelet running, container runtime disabled

Chain the three observations:

1. crictl cannot reach the socket and containerd is inactive → there is no CRI endpoint at all.
2. The kubelet is active (running) because kubelet treats the runtime as an external dependency: it starts, fails its CRI handshake, and retries in a loop rather than exiting. That is why systemctl status kubelet is misleading here.
3. With no runtime, the pod lifecycle events and the runtime readiness checks fail (PLEG is not healthy), the kubelet posts Ready=False with KubeletNotReady, and the node controller marks the node NotReady.

The root cause is upstream of Kubernetes: is-enabled containerd says disabled, so the unit was not started at boot. Someone installed or reconfigured containerd without systemctl enable.

Fix and verify on the node:

systemctl enable --now containerd
systemctl is-active containerd      # expect: active
crictl info                        # expect: JSON, not connection refused
systemctl restart kubelet          # optional; kubelet retries on its own
journalctl -u kubelet -f

Then from the control plane: kubectl get node node-4 -w until Ready (usually under a minute), kubectl get pods -A -o wide --field-selector spec.nodeName=node-4 to see what is being restarted, and kubectl describe node node-4 to confirm no lingering RuntimeReady=False. Do not reboot again to "see if it clears" - it will not, the unit is still disabled.

It is not useful, and the reasoning matters more than the answer.

When a node goes NotReady, the node controller applies the taint node.kubernetes.io/not-ready:NoExecute. Pods carry a default toleration for that taint with tolerationSeconds: 300, so after roughly five minutes the pods are evicted from the API's point of view - the ReplicaSet controller sees a missing pod and creates a replacement elsewhere. Twelve minutes have elapsed, so web should already have a replacement scheduled on one of the other four nodes. Check kubectl get pods -o wide -l app=web and kubectl get events --field-selector reason=TaintManagerEviction before doing anything.

The real hazard is the opposite one: because the runtime is down, the kubelet cannot stop the container on node-4. The API object may be gone while the process keeps running and keeps serving or writing. Deleting web-7d9f by hand removes a record, not a process, and does nothing to speed the replacement, which the ReplicaSet controller was already going to create.

The PDB is a red herring: minAvailable: 2 is enforced by the Eviction API (what kubectl drain and autoscalers use). Node-controller evictions for not-ready/unreachable bypass PDBs entirely. So the PDB neither blocked the eviction nor protects you here - do not read a healthy kubectl get pdb output as evidence that the pods are fine.

cache-1 is the pod that will not recover by itself. Its PV has nodeAffinity for node-4 only - that is the whole point of a local volume, the data physically lives on node-4's disk. Two consequences:

• The PV's node affinity is projected onto the pod as a scheduling constraint. A replacement pod for cache-1 will stay Pending with node(s) didn't find available persistent volumes to bind or a volume-node-affinity conflict, because no other node has that volume.
• There is no automatic migration. Kubernetes will not copy a local volume. If node-4 never comes back, the data is unreachable through the cluster; recovering it means mounting the disk elsewhere, rebuilding the PV with new nodeAffinity, and accepting a manual, error-prone procedure. Retain preserves the data but not the service.

Fix order therefore follows from the diagnosis: restoring containerd on node-4 is the only fast path for cache-1, so do that before anything else and let the node rejoin. Then verify cache-1 actually restarted against the same volume by exec-ing in and checking a known key or checksum - after a runtime outage, do not assume the mount and the data are what you expect.

DR implication: if cache-1 must survive a node loss, local volumes are the wrong tool. Options are replicated storage (a CSI driver with topology-aware replicas, or a network volume that any node can attach), or application-level replication with the local volume treated as a cache. Also treat systemctl is-enabled on kubelet, containerd, and the kube-proxy/CNI units as part of the node bootstrap check - this entire incident is one missing enable at image build time.