I85PREP
⌘ K
CKA study materials
CKA study resources

CKA (CNCF): The Approved Upgrade Plan Skips a Minor Version

CKA · Put your knowledge to work in context.

All case studies

The Approved Upgrade Plan Skips a Minor Version

Your team runs a kubeadm cluster with three stacked-etcd control-plane nodes on v1.28.4 and six worker nodes whose kubelet and kubeadm packages are already at v1.29.2. CNI is Calico, and CoreDNS is the only add-on this change touches. Change control has approved moving the whole cluster to v1.30.2 this weekend. A colleague has already installed kubeadm v1.30.2 on cp-1 (apt-get install -y kubeadm=1.30.2-1.1), but nothing has been executed yet. The written plan reads: run kubeadm upgrade apply v1.30.2 on cp-1, then repeat on cp-2 and cp-3, then upgrade the six workers. The cluster is in production, and the on-call engineer wants a go/no-go on that plan before the window opens.

1. Can `kubeadm upgrade apply v1.30.2` succeed on cp-1 tomorrow? Explain what blocks it and what the correct target sequence is.

Read the worked answer

No. Two independent problems stacked on top of each other.

  1. kubeadm does not skip minor versions. The kubeadm on the node must be the same minor as the cluster it manages, and kubeadm upgrade plan only offers the next minor. On a 1.28 cluster you will not get 1.30 as an option, and even with the 1.30 binary installed the control plane would have to leap two minors in one apply - unsupported and untested against your Calico and CoreDNS versions.
  2. The colleague installed the wrong binary. Installing kubeadm 1.30.2 on a 1.28.4 control-plane node makes the local kubeadm newer than the cluster it is managing. kubeadm upgrade plan will refuse or warn; the supported state is kubeadm == cluster minor.

Correct sequence: 1.28.4 → 1.29.x → 1.30.2, treating each hop as its own change window if you can.

# hop 1
apt-get install -y kubeadm=1.29.<patch>
kubeadm upgrade plan
kubeadm upgrade apply v1.29.<patch>
apt-get install -y kubelet=1.29.<patch> kubectl=1.29.<patch>
systemctl daemon-reload && systemctl restart kubelet
# hop 2, after the whole cluster is 1.29
apt-get install -y kubeadm=1.30.2-1.1
kubeadm upgrade apply v1.30.2

Before the window: run kubeadm upgrade plan and read the add-on upgrade section, confirm the Calico and CoreDNS releases you run support 1.30, and roll back the stray 1.30.2 kubeadm package on cp-1.

2. The workers already run kubelet v1.29.2 while the API servers are v1.28.4. Is that a supported state, and what does it imply about the order of operations?

Read the worked answer

It is not supported, and it already exists - before the upgrade starts.

The Kubernetes version skew policy is asymmetric: a kubelet may be up to three minors older than the kube-apiserver it talks to, but it may never be newer. Right now kubelet 1.29.2 is talking to an apiserver 1.28.4, which is the forbidden direction.

Implications:

  • This is a latent incident, not a cosmetic warning. Any 1.29 kubelet behaviour that expects a 1.29 API surface is undefined against 1.28. You are one API call away from a non-obvious failure, so the upgrade window is now more urgent than the change ticket suggests.
  • Do not 'fix' it by downgrading the workers. The fix is to move the control plane forward.
  • The control plane must go first. Upgrading the API server to 1.29 (and later 1.30) brings the skew back inside policy immediately, and the workers then follow at their own pace.
  • Never install the new kubelet package before the control plane is upgraded. kubeadm preflight on a control-plane node compares the local kubelet version against the target version and errors when the kubelet is newer than the version being applied. If you bump kubelet before kubeadm upgrade apply, you can wedge the very node you are upgrading.

Order: control plane (one node at a time) → workers (one node at a time). Kubelet package upgrades always come after that node's own kubeadm upgrade step.

3. Give the exact per-node order of operations for the first control-plane node, and state the one thing that must happen before you drain anything in a three-member stacked-etcd cluster.

Read the worked answer

Before any drain: take and verify an etcd snapshot, off the node.

ETCDCTL_API=3 etcdctl --endpoints=https://127.0.0.1:2379 \
  --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  snapshot save /var/backups/etcd-pre-upgrade.db
ETCDCTL_API=3 etcdctl snapshot status /var/backups/etcd-pre-upgrade.db

Then copy it off the host - a snapshot on the disk of the node you are about to restart is not a backup. Also confirm before starting: kubectl get --raw='/readyz?verbose' is clean, all three etcd members pass etcdctl endpoint health --cluster, and no other control-plane node is mid-upgrade.

Per control-plane node, one at a time:

kubectl drain cp-1 --ignore-daemonsets --delete-emptydir-data
apt-get install -y kubeadm=1.29.<patch>
kubeadm upgrade plan
kubeadm upgrade apply v1.29.<patch>        # FIRST control-plane node only
# cp-2 and cp-3 instead run: kubeadm upgrade node
apt-get install -y kubelet=1.29.<patch> kubectl=1.29.<patch>
systemctl daemon-reload && systemctl restart kubelet
kubectl uncordon cp-1

Then verify before touching the next node: kubectl get nodes shows cp-1 Ready at the new version, etcdctl endpoint health --cluster is green, and kubectl get pods -n kube-system shows static pods and CoreDNS healthy.

Why the ordering matters so much: a 3-member stacked etcd tolerates exactly one member down (quorum is 2). Restarting a second control-plane node before the first is healthy takes the cluster below quorum, and etcd will not accept writes - the API server goes read-only at best. kubeadm upgrade apply also renews the control-plane certificates, so if you have been putting this off, this window fixes expiry too.

Workers afterwards, same shape: drain one worker → kubeadm upgrade node → kubelet/kubectl packages → restart kubelet → uncordon → wait for Ready before the next.