Pod Priority & Preemption

Original Prepium workshop · Tested on Kubernetes v1.35.0 on 1 October 2026 in a kind cluster with two workers. Use a disposable cluster: step 3 deliberately fills one worker's CPU requests so that the scheduler has to preempt a Pod.

PriorityClass: how a Pod gets its priority, and what preemption looks like

A PriorityClass is a cluster-scoped object that maps a name to an integer. A Pod refers to it by priorityClassName, and at admission Kubernetes copies the number into the Pod's spec.priority. The scheduler uses that number twice: higher-priority Pods are tried first, and when a high-priority Pod cannot fit anywhere, the scheduler may preempt (evict) lower-priority Pods to make room.

1. Read what already exists

kubectl get priorityclass

A fresh cluster has two built-in classes, system-cluster-critical (2000000000) and system-node-critical (2000001000), reserved for components the cluster cannot run without. User-defined classes must stay at or below 1000000000. Check the GLOBAL-DEFAULT column too: if a class has globalDefault: true, Pods without a priorityClassName get its value instead of 0.

2. Create two classes

Save this as priorityclasses.yaml:

apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tutorial-low
value: 1000
globalDefault: false
description: "Batch work that may be preempted."
---
apiVersion: scheduling.k8s.io/v1
kind: PriorityClass
metadata:
  name: tutorial-high
value: 100000
globalDefault: false
description: "Latency-sensitive work that may preempt lower priorities."

kubectl apply -f priorityclasses.yaml
kubectl get priorityclass

3. Watch a high-priority Pod preempt a low one

The Pods below are pinned to one worker and each requests 60% of its allocatable CPU, so the node cannot hold both. The scheduler decides by requests, not actual usage, which is why pause containers that use no CPU are enough.

kubectl create namespace prepium-tutorial
WORKER=$(kubectl get nodes -l '!node-role.kubernetes.io/control-plane' -o jsonpath='{.items[0].metadata.name}')
ALLOC=$(kubectl get node "$WORKER" -o jsonpath='{.status.allocatable.cpu}')
case "$ALLOC" in *m) ALLOC_M=${ALLOC%m};; *) ALLOC_M=$((ALLOC*1000));; esac
REQ=$((ALLOC_M*6/10))
for p in low:tutorial-low high:tutorial-high; do
cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata: {name: ${p%%:*}, namespace: prepium-tutorial}
spec:
  priorityClassName: ${p#*:}
  nodeSelector: {kubernetes.io/hostname: $WORKER}
  containers:
  - name: pause
    image: registry.k8s.io/pause:3.10
    resources: {requests: {cpu: ${REQ}m}}
EOF
[ "${p%%:*}" = low ] && kubectl -n prepium-tutorial wait --for=condition=Ready pod/low
done
kubectl -n prepium-tutorial get pods -o wide --watch

Before the second Pod arrives, check kubectl -n prepium-tutorial get pod low -o jsonpath='{.spec.priority}': it prints 1000, copied from the class. Then high cannot fit, the scheduler evicts low, and high starts on the same node. The evidence is in the events: kubectl -n prepium-tutorial get events --field-selector reason=Preempted shows Preempted by pod ... on node .... A bare Pod that is preempted is gone for good; a Pod owned by a Deployment would be recreated and then sit Pending until room frees up.

4. Get the name wrong once

kubectl -n prepium-tutorial run ghost --image=registry.k8s.io/pause:3.10 \
  --overrides='{"spec":{"priorityClassName":"does-not-exist"}}'

A bare Pod is rejected at once: no PriorityClass with name does-not-exist was found. The same typo in a Deployment's Pod template is more dangerous, because the Deployment itself is accepted. The error surfaces only as a FailedCreate event on the ReplicaSet, while the Deployment quietly shows fewer ready replicas. After changing priorityClassName on a workload, check kubectl get events and the rollout, not just the apply output.

Changing the class on a Deployment also only affects new Pods. Priority is fixed when a Pod is admitted, so the template change triggers a rollout, and the new Pods carry the new value.

Cleanup: kubectl delete namespace prepium-tutorial and kubectl delete -f priorityclasses.yaml (classes are cluster-scoped). Reference: Pod Priority and Preemption. Next, try taints and tolerations.