Node Scheduling & Maintenance

Original Prepium workshop · Tested on Kubernetes v1.35.0 on 1 October 2026 in a kind cluster with one control-plane node and two workers. Use a disposable cluster with at least two schedulable workers: this workshop taints a node, and a single-node cluster cannot show where Pods go instead.

Taints repel, tolerations permit, selectors attract

A taint on a node keeps Pods away unless they carry a matching toleration. A toleration does not pull a Pod towards the node; it only removes the objection. To place a Pod on a dedicated node you need both: a toleration so it is allowed there, and a nodeSelector or node affinity so it is not allowed anywhere else.

                       taint reserved=batch:NoSchedule   label workload=batch
no toleration, no selector ──✗──→ tainted worker     ✓ other nodes
selector only              ──✗──→ tainted worker     ✗ other nodes  → Pending
toleration + selector      ──✓──→ tainted worker     ✗ other nodes  → Running there

1. Reserve one worker

kubectl create namespace prepium-tutorial
WORKER=$(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' -o jsonpath='{.items[0].metadata.name}')
kubectl label node "$WORKER" workload=batch
kubectl taint node "$WORKER" reserved=batch:NoSchedule
kubectl describe node "$WORKER" | grep -A2 Taints

The taint has three parts: key reserved, value batch, effect NoSchedule. A toleration must match the key, the effect and (with operator: Equal) the value.

2. Ordinary Pods avoid it

kubectl -n prepium-tutorial create deployment spread --image=registry.k8s.io/pause:3.10 --replicas=6
kubectl -n prepium-tutorial get pods -l app=spread -o wide

None of the six Pods lands on the reserved worker. They all go to the other worker, because the control-plane node carries its own NoSchedule taint too.

3. A selector alone is not enough

Save this as taint-pods.yaml. Both Pods select the reserved node; only the second tolerates its taint.

apiVersion: v1
kind: Pod
metadata:
  name: selector-only
  namespace: prepium-tutorial
spec:
  nodeSelector:
    workload: batch
  containers:
    - name: pause
      image: registry.k8s.io/pause:3.10
---
apiVersion: v1
kind: Pod
metadata:
  name: selector-and-toleration
  namespace: prepium-tutorial
spec:
  nodeSelector:
    workload: batch
  tolerations:
    - key: reserved
      operator: Equal
      value: batch
      effect: NoSchedule
  containers:
    - name: pause
      image: registry.k8s.io/pause:3.10

kubectl apply -f taint-pods.yaml
kubectl -n prepium-tutorial get pods -o wide
kubectl -n prepium-tutorial describe pod selector-only | tail -3

selector-and-toleration runs on the reserved worker. selector-only stays Pending, and the scheduler explains why in one line: 0/3 nodes are available: 1 node(s) didn't match Pod's node affinity/selector, 2 node(s) had untolerated taint(s). Read it as a filter: the one node without the label was excluded by the selector, and the two remaining nodes (the reserved worker and the control plane) were excluded by taints.

The reverse case, a toleration without a selector, schedules fine but may land on any node. That is the usual reason a "dedicated node" Pod ends up somewhere else in practice: the toleration made the node possible, nothing made it required.

4. NoExecute evicts what is already running

kubectl taint node "$WORKER" maintenance=true:NoExecute
kubectl -n prepium-tutorial get pods -o wide --watch

NoSchedule only affects scheduling decisions; Pods already on the node stay. NoExecute also evicts running Pods that do not tolerate it, so selector-and-toleration disappears: it tolerates reserved, not maintenance. A toleration for a NoExecute taint can add tolerationSeconds to stay for a limited time before eviction.

5. Remove the taints

kubectl taint node "$WORKER" maintenance=true:NoExecute- reserved=batch:NoSchedule-
kubectl -n prepium-tutorial get pod selector-only -o wide
kubectl label node "$WORKER" workload-

The trailing - removes a taint or label. As soon as the taints are gone, selector-only schedules onto the worker, because its label still matches: the taint was the only thing blocking it. Removing the label afterwards does not evict it. A nodeSelector is checked when the Pod is scheduled, not continuously.

Cleanup: kubectl delete namespace prepium-tutorial, and confirm with kubectl describe node "$WORKER" | grep Taints that the worker shows <none>. Reference: Taints and Tolerations. Next, try Pod priority and preemption.