Debug a CrashLoopBackOff and a Container Without a Shell

Original Prepium guide · Written for Kubernetes v1.35 and checked against the official documentation on 6 October 2026. Use a disposable practice cluster.

Read the evidence of a crash loop, then debug a container that has no shell

A container in CrashLoopBackOff is not stuck: the kubelet keeps restarting it, waiting longer each time (10s, 20s, 40s and so on, capped at five minutes). The reason it died is usually printed by the container that already exited, so the skill is collecting that evidence before you change anything. The second skill is getting a shell next to an application whose image ships without one, which is what ephemeral containers are for.

container exits ──→ kubelet waits (back-off) ──→ restarts ──→ exits again
        │
        └── evidence: kubectl logs --previous, describe (Last State, Exit Code, Events)

1. Build a crash loop you understand

Save this as ckad-debug-banner.yaml. The script exits with code 3 when GREETING is empty. The configMapKeyRef points at a key name that does not exist, and because the reference is marked optional: true, Kubernetes starts the container anyway with no value instead of refusing to start the Pod.

apiVersion: v1
kind: ConfigMap
metadata:
  name: banner-settings
  namespace: prepium-tutorial
data:
  greeting.text: "hello from a ConfigMap"
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: banner
  namespace: prepium-tutorial
spec:
  replicas: 1
  selector:
    matchLabels:
      app: banner
  template:
    metadata:
      labels:
        app: banner
    spec:
      containers:
      - name: web
        image: busybox:1.28
        command:
        - sh
        - -c
        - |
          if [ -z "$GREETING" ]; then echo "fatal: GREETING is not set"; exit 3; fi
          echo "$GREETING"; sleep 3600
        env:
        - name: GREETING
          valueFrom:
            configMapKeyRef:
              name: banner-settings
              key: greeting_text   # deliberate typo: the real key is greeting.text
              optional: true

kubectl create namespace prepium-tutorial
kubectl apply -f ckad-debug-banner.yaml
kubectl -n prepium-tutorial get pods -l app=banner --watch

Within a minute or two the STATUS column shows CrashLoopBackOff and RESTARTS keeps climbing.

2. Collect the evidence first

POD=$(kubectl -n prepium-tutorial get pod -l app=banner -o jsonpath='{.items[0].metadata.name}')
kubectl -n prepium-tutorial logs "$POD" -c web --previous > banner-crash.txt
kubectl -n prepium-tutorial describe pod "$POD"
kubectl -n prepium-tutorial get configmap banner-settings -o yaml

--previous prints the log of the earlier instance of the container in the same Pod. Save it now: once a rollout replaces the Pod, that history is gone. In describe, the container shows State: Waiting with reason CrashLoopBackOff, and Last State: Terminated carries the exit code. Comparing the key in the Pod template with the keys under data exposes the typo.

3. Fix the reference, not the symptom

Change key: greeting_text to key: greeting.text in the file and apply it again. The Deployment rolls out a new Pod whose variable is filled from the ConfigMap.

kubectl apply -f ckad-debug-banner.yaml
kubectl -n prepium-tutorial rollout status deployment/banner
kubectl -n prepium-tutorial logs deployment/banner

Had the reference not been optional, the missing key would have stopped the Pod from starting at all, which is a different symptom with the same root cause. Also remember that environment variables taken from a ConfigMap are read when the container starts; editing the ConfigMap later does not change running Pods until they restart.

4. Attach an ephemeral container to a shell-less Pod

kubectl -n prepium-tutorial run quiet --image=registry.k8s.io/pause:3.1 --restart=Never
kubectl -n prepium-tutorial exec -it quiet -- sh        # fails: no shell in the image
kubectl -n prepium-tutorial debug -it quiet --image=busybox:1.28 -c inspect --target=quiet
/ # ps

kubectl debug adds a container to the running Pod through the API's ephemeral containers subresource (stable since v1.25). --target places it in the process namespace of the named container, so ps lists the pause process. Without it you would only see your own shell, because Pods do not share a process namespace unless shareProcessNamespace is set. The container runtime must support targeting; if it does not, the debug container may fail to start or may run isolated.

Common mistakes

• Fixing first and reading logs later. After the rollout the crashed instance has been replaced and --previous has nothing left to show.
• Treating every crash as an application bug. An exit code of 137 with reason OOMKilled in Last State points at the memory limit instead.
• Omitting --target and concluding the app is not running because ps is nearly empty.
• Trying to delete an ephemeral container or add one with kubectl edit. Neither is possible: once added it stays in the Pod spec until the Pod is deleted.

Cleanup: kubectl delete namespace prepium-tutorial. References: Debug Running Pods, Ephemeral Containers, Pod Lifecycle, Configure a Pod to Use a ConfigMap. Next, try health probes or fixing a broken Deployment.