Fix a Broken Control Plane: Three Static Pod Faults

Original Prepium guide · Written for Kubernetes v1.35 and checked against the official documentation on 6 October 2026. Use a disposable practice cluster. Control-plane manifests can take down the whole cluster, so practise on a throwaway kubeadm VM, and break only the test Pod created below, never a real component.

Static Pods: how the kubelet runs the control plane, and how to debug it when kubectl is gone

On a kubeadm control-plane node, kube-apiserver, kube-controller-manager, kube-scheduler and etcd are static Pods: plain Pod manifests on disk that the kubelet runs by itself, with no API server involved. That independence is what lets the API server start at all, and it is also why, when a manifest is wrong, kubectl cannot help. You debug from the node with the file, the runtime and the kubelet's own logs.

/var/lib/kubelet/config.yaml (staticPodPath) ──→ kubelet scans /etc/kubernetes/manifests
   file valid?   no ──→ nothing starts; reason is in the kubelet journal
                 yes ──→ container via CRI ──→ exits? reason is in crictl logs
                       └──→ mirror Pod "<name>-<node>" shown by the API server (read-only)

1. Find the directory and look around (read-only)

grep staticPodPath /var/lib/kubelet/config.yaml
ls -la /etc/kubernetes/manifests/
kubectl -n kube-system get pods -o wide | grep "$(hostname)"

The kubelet processes every file in that directory whose name does not start with a dot, regardless of extension. Mirror Pods in the API carry the node name as a suffix; deleting one with kubectl does not stop the real static Pod. The only way to change a static Pod is to change its file.

2. Watch a valid static Pod appear

Write our own test manifest as /etc/kubernetes/manifests/prepium-static.yaml:

apiVersion: v1
kind: Pod
metadata:
  name: prepium-static
  namespace: default
spec:
  containers:
  - name: ticker
    image: busybox:1.36
    command: ["sh", "-c", "while true; do date; sleep 10; done"]

The kubelet scans the directory periodically, so within a short while crictl ps on the node lists a ticker container, and kubectl get pods shows prepium-static-<node>. Move the file out of the directory and both disappear.

3. Break it two ways and read the evidence

A file the kubelet cannot decode. Indent containers: wrongly, or misspell apiVersion, and save. No container starts and no mirror Pod appears, and nothing in the API explains why. Look for the reason in the kubelet journal:

journalctl -u kubelet --since "10 min ago" | grep -i prepium-static

A container that starts and dies. Restore valid YAML but change the command to run a binary that does not exist in the image. Now the Pod exists, yet the container keeps exiting and its ATTEMPT count rises:

crictl ps -a | grep ticker          # STATE Exited, ATTEMPT increasing
crictl logs <container-id>          # last output before it died

A broken control-plane component behaves the same way: either it never appears in crictl ps -a (look at the kubelet journal) or it appears as Exited with a climbing attempt count (look at crictl logs for whatever the component printed before it exited). Fix the manifest, save, and wait for the kubelet to recreate the container. When the API server is back, kubectl get nodes answers again.

4. Edit safely

Keep backups outside the directory, for example in /root/manifest-backups/. The official docs warn that a copy such as kube-apiserver.yaml.bak left beside the original is read too, and two files defining the same Pod name give undefined behaviour, possibly running the stale copy.

Common mistakes

• A backup left in the manifests directory: the kubelet may run the outdated spec instead of your fix.
• Editing the mirror Pod with kubectl edit: it is a reflection of the file, not the source of truth.
• Reading only crictl ps without -a: crashed containers are hidden and the component looks simply absent.
• Fixing one error and stopping: a component can fail on the next flag too, so recheck the logs after every change.

Cleanup: rm /etc/kubernetes/manifests/prepium-static.yaml; the kubelet removes the Pod. References: Create static Pods, Debugging Kubernetes nodes with crictl, kubeadm implementation details. Next, try container runtimes and CRI or kubeadm upgrades.