Upgrade a kubeadm Control Plane to the Next Patch Release

Original Prepium guide · Written for Kubernetes v1.35 and checked against the official documentation on 6 October 2026. Use a disposable practice cluster. Package and kubelet changes affect the whole node, so practise on throwaway VMs and run the read-only checks in step 1 before changing anything.

Upgrade order, package holds and version skew in a kubeadm cluster

A kubeadm upgrade is really three upgrades that must happen in a fixed order: the kubeadm tool, the control-plane components it manages as static Pods, and finally the kubelet that the OS package manager owns. The order exists because of the version skew policy, and the package holds exist so that a routine apt upgrade never moves a node out of that order by accident.

1. kubeadm package     ──→ kubeadm upgrade plan / apply   (first control plane)
                       ──→ kubeadm upgrade node           (other control planes, then workers)
2. per node: drain ──→ kubelet + kubectl packages ──→ daemon-reload + restart kubelet ──→ uncordon
rule: kubelet never newer than kube-apiserver

1. Inspect versions and holds (read-only)

kubectl version                      # client and server versions
kubectl get nodes                    # VERSION column = each node's kubelet
kubeadm version
apt-mark showhold                    # which packages are pinned
apt-cache madison kubeadm            # versions the configured repo offers

A hold marks a package so apt will not install, upgrade or remove it automatically. You must apt-mark unhold a package before changing its version and apt-mark hold it again afterwards. If madison lists nothing newer for a minor upgrade, the repository itself is the problem: pkgs.k8s.io publishes one repository per minor release, so moving from 1.34 to 1.35 means switching the repo definition first.

2. Know what the skew policy allows

• kube-apiserver goes first and must not skip minor versions.
• The kubelet must never be newer than kube-apiserver; it may lag by up to three minor versions.
• kubectl is supported within one minor version of kube-apiserver, older or newer.
• kubeadm itself can manage a kubelet that is the same version or up to three minor versions older.

This is why the kubelet package is always upgraded last on each node: doing it first would briefly make it newer than the API server.

3. Walk one worker through it

Assume the control plane of a practice cluster is already at 1.35 and worker practice-w1 still runs 1.34. On a worker, kubeadm upgrade node only fetches the cluster configuration and upgrades the local kubelet configuration; there are no control-plane manifests to touch. From a machine with kubectl access, run kubectl drain practice-w1 --ignore-daemonsets. Drain cordons the node and evicts its Pods; the flag is needed because DaemonSet Pods would simply be recreated. Then, on the worker, upgrade the kubeadm package (unhold, install a 1.35.x-* version, hold), run kubeadm upgrade node, upgrade the kubelet and kubectl packages the same way, and restart the service:

systemctl daemon-reload && systemctl restart kubelet
systemctl status kubelet       # active (running)?
journalctl -xeu kubelet        # if not, read why

Finish with kubectl uncordon practice-w1. On a control plane, the difference is the middle step: kubeadm upgrade plan checks that the cluster is upgradeable and lists targets, then kubeadm upgrade apply v<target> rewrites the static Pod manifests. kubeadm upgrade diff and apply --dry-run preview that without changing state.

4. Recover from a half-finished run

kubeadm upgrade is idempotent, so rerunning it after an interruption is the documented recovery. It also keeps backups of the old manifests (and local etcd data) under /etc/kubernetes/tmp, which you clean up manually later.

Common mistakes

• Installing over a hold without unholding first: with -y, apt-get aborts rather than change a held package, so the version check still shows the old release.
• Skipping daemon-reload and the restart: the new binary is on disk but the running kubelet, and the VERSION column, stay old.
• Forgetting to re-hold: a later system update can move the packages out of step.
• Leaving the node cordoned: STATUS still shows SchedulingDisabled next to Ready, and new Pods avoid it.

Cleanup: rebuild the practice VMs, or remove old backups under /etc/kubernetes/tmp. References: Upgrading kubeadm clusters, Version Skew Policy, Safely Drain a Node. Next, try the CKS upgrade drill or joining a worker.