I85PREP
⌘ K
KCA study materials
KCA study resources

Control-plane upgrade from 1.25 to 1.29 goes wrong on a live kubeadm cluster

KCA · Put your knowledge to work in context.

All case studies

Control-plane upgrade from 1.25 to 1.29 goes wrong on a live kubeadm cluster

A team must take a kubeadm cluster from v1.25.9 to v1.29.x. Topology: 3 control-plane nodes, 5 workers, containerd 1.6, Calico 3.25. Their runbook says: unhold and apt-install kubeadm=1.29.x, kubelet=1.29.x and kubectl=1.29.x on every node, then run kubeadm upgrade apply v1.29.x on one control-plane node, then drain and repeat on the workers.

Manifests in Git (last edited 18 months ago) include an HPA at autoscaling/v2beta2, a CronJob at batch/v1beta1 and a PodDisruptionBudget at policy/v1beta1. A GitOps controller syncs the repo every 5 minutes with pruning enabled.

In a staging rehearsal, right after kubeadm upgrade apply, the controller fails every sync with no matches for kind "HorizontalPodAutoscaler" in version "autoscaling/v2beta2", yet kubectl get hpa -A still lists the existing HPA objects. One worker's kubelet then refuses to start with failed to run Kubelet: the Kubelet requires the systemd cgroup driver.

1. What is wrong with the runbook's ordering, and what would actually happen at each step on the real cluster?

Read the worked answer

Three separate problems.

Version jumping. kubeadm does not support skipping minor versions; kubeadm upgrade apply v1.29 from a 1.25 control plane is refused. You must walk 1.25 → 1.26 → 1.27 → 1.28 → 1.29, running kubeadm upgrade plan and kubeadm upgrade apply v1.26.x and so on, finishing each minor before starting the next.

Wrong upgrade unit. The documented order is: upgrade the kubeadm binary on one control-plane node, kubeadm upgrade plan, kubeadm upgrade apply, then upgrade kubelet and kubectl on that node; repeat across control planes; only then, per worker: kubectl drain --ignore-daemonsets --delete-emptydir-data, upgrade kubeadm, kubeadm upgrade node, upgrade kubelet, systemctl restart kubelet, kubectl uncordon. Installing 1.29 kubelet packages on the workers up front leaves kubelets newer than the API server, which violates skew policy: kubelet may be up to three minor versions older than kube-apiserver, never newer.

The manifests are already broken. batch/v1beta1 CronJob and policy/v1beta1 PodDisruptionBudget were removed in v1.25, so those two have been failing to sync since the cluster reached 1.25 - not caused by this upgrade. Only the HPA is new breakage (removed in v1.26). Existing HPA objects still list because stored objects are converted and served through the internal version; what fails is applying the old manifest - and with pruning on, the controller may delete objects it can no longer map.

2. The worker kubelet will not start with the systemd cgroup driver error. What is the cause, the exact fix, and how would you have caught it before the change window?

Read the worked answer

Cause: a driver mismatch. kubeadm-generated kubelet configs use cgroupDriver: systemd (the default since 1.22 when the host init is systemd), while containerd's CRI plugin still has SystemdCgroup = false - the value containerd config default writes. Mixing cgroupfs and systemd drivers produces duplicate cgroup hierarchies and unreliable CPU/memory limits, so recent kubelets treat it as fatal instead of a warning.

Fix on the node: edit /etc/containerd/config.toml, set SystemdCgroup = true under [plugins."io.containerd.grpc.v1.cri".containerd.runtimes.runc.options], then systemctl restart containerd && systemctl restart kubelet. Confirm with kubectl get nodes -o wide (CONTAINER-RUNTIME) and by checking that a pod with a memory limit lands in a kubepods-burstable-pod...slice.

Catch it earlier: the reason it surfaced is that the node was rebuilt from an older image, so the config drifted from the rest. Validate the node image, not just the manifest - run kubeadm upgrade node --dry-run (or kubeadm config validate) against the exact image you will boot, and diff /etc/containerd/config.toml across the fleet before the window. A five-line Ansible/Packer assertion that SystemdCgroup = true is present would have caught this in staging, not mid-drain.

3. Before touching production, which commands against the live cluster find everything that will break?

Read the worked answer

  1. Real traffic on deprecated APIs - highest signal, because it measures usage rather than guesses: kubectl get --raw /metrics | grep apiserver_requested_deprecated_apis Any non-zero series with group="autoscaling",version="v2beta2" means a live client still calls it. (--audit-log with a policy at Metadata level gives per-user attribution.)

  2. Served versions vs. what you apply. kubectl api-resources lists what the server offers; kubectl get hpa.v2beta2.autoscaling -A returns an error once that version is unserved, proving existing objects are still reachable through v2 but the manifest is not.

  3. Static scan of the repo, where the failure actually lives. kubent, pluto detect-files -d ., or kubectl apply --dry-run=server -f manifests/ against a 1.29 API server. --dry-run=server runs the full admission chain, so it also catches Pod Security and CRD schema changes that a client-side dry run misses.

  4. kubeadm upgrade plan on each control plane: it prints current vs. target versions, the kubelet versions in the cluster, and component-config deprecations.

  5. Add-on compatibility, explicitly: Calico 3.25 against the target Kubernetes version, the bundled CoreDNS version, and the CSI/cloud driver's supported range. The CNI is the component most likely to make nodes NotReady after a minor bump.