CKA (Other exams): kubeadm upgrade of a stacked-etcd HA control plane with version skew and expiring certificates

Fix the skew first. Kubernetes supports a kubelet up to three minor versions older than the kube-apiserver, never newer. worker-2 runs kubelet v1.31.0 against an apiserver at v1.29.3 now, and would still be newer than the v1.30.1 apiserver after the upgrade. Downgrade worker-2's kubelet and kubectl to 1.29.x before touching the control plane, or cordon it and exclude it from the window.

Why parallel apply is wrong. kubeadm upgrade apply is the cluster-wide phase: it writes the new cluster configuration, upgrades the static pod manifests of the node it runs on, and upgrades that node's local etcd member. On a stacked etcd cluster, running it on all three nodes at once restarts three etcd members concurrently - quorum is lost, the API server becomes unavailable, and the upgrade itself may hang. Nodes 2 and 3 do not need apply; they use kubeadm upgrade node, which only refreshes local manifests, local etcd, and kubelet config.

Order (one node at a time, always keeping 2/3 etcd members up):

1. kubeadm upgrade plan on cp-1 (dry run first if you want).
2. cp-1: kubeadm upgrade apply v1.30.1 → kubectl drain cp-1 --ignore-daemonsets → upgrade kubelet/kubectl → systemctl daemon-reload && systemctl restart kubelet → kubectl uncordon cp-1.
3. cp-2 and cp-3: kubeadm upgrade node → drain → kubelet upgrade → restart → uncordon.
4. Workers last, same drain/upgrade/uncordon cycle.
5. Only after every kubelet is on 1.30.x may you raise the control plane again in a future window.

It changes the sequence within the control plane nodes, not the shape of the procedure. Do not restart the leader first if you can avoid it: upgrade a follower (cp-3 or cp-1) first so that if the leader election has to happen, it happens against a healthy two-member quorum rather than mid-restart of the node that owns it. etcd handles leader election on its own within seconds, but every restart is a window where you want the other two members fully healthy.

Verify before and after each control plane node, from any control plane node:

kubectl -n kube-system exec -it etcd-cp-1 -- sh -c \
 'ETCDCTL_API=3 etcdctl --cacert=/etc/kubernetes/pki/etcd/ca.crt \
  --cert=/etc/kubernetes/pki/etcd/server.crt \
  --key=/etc/kubernetes/pki/etcd/server.key \
  member list -w table'

and endpoint health --cluster against the three members. You want all members listed, all started, and a leader visible. If endpoint health shows one member unreachable before you start, stop and fix that - you would be upgrading a cluster that is one restart away from losing quorum. Also check kubectl get --raw='/readyz?verbose' on the API server; etcd and poststarthook lines must be ok.

kubeadm upgrade apply and kubeadm upgrade node renew the kubeadm-managed certificates on the node they run on, so the apiserver serving cert and the etcd peer cert are refreshed as a side effect of doing the upgrade now. That is convenient but it is not a plan: if the window slips past day 19 the apiserver will fail to serve and the whole control plane falls over.

If the upgrade might slip, run explicitly on each control plane node:

kubeadm certs check-expiration
kubeadm certs renew all

then restart the static pods (move the manifests out of /etc/kubernetes/manifests and back, or kubectl -n kube-system delete pod kube-apiserver-cp-1 ...) so the new certs are loaded, and re-copy /etc/kubernetes/admin.conf to $HOME/.kube/config because renew all regenerates it.

Reasoning about what is not covered: kubelet client and serving certificates are not managed by kubeadm certs - they are issued by the API server and rotate automatically (check rotateCertificates: true and serverTLSBootstrap behaviour). The service-account keypair (sa.key/sa.pub) is also never rotated by kubeadm; leave it alone unless you have a deliberate, cluster-wide token-invalidation plan. Finally, a cert renewed at 19 days does not remove the need to re-check check-expiration on every control plane node - each node has its own copies.