Make Containers Immutable: Non-Root and Read-Only Root FS

Original Prepium guide · Written for Kubernetes v1.35 and checked against the official documentation on 6 October 2026. Use a disposable practice cluster.

Make a container immutable at runtime without breaking the app

An "immutable" container cannot change its own image contents while it runs. Three settings carry most of the weight: a read-only root filesystem, so an intruder cannot drop tools or edit binaries; a fixed non-root user, so a process escape starts without root; and allowPrivilegeEscalation: false, so a setuid binary cannot raise privileges again. Applications still need somewhere to write temporary files, which is what an emptyDir volume is for. On the CKS exam the hard part is rarely the fields; it is keeping the workload running afterwards.

image layers (read-only, readOnlyRootFilesystem: true)
   └─ /tmp ← emptyDir (writable, lives as long as the Pod)
process: uid 31000, runAsNonRoot checked by kubelet, no_new_privs set, capabilities dropped

1. Apply the hardened Deployment and watch it fail

Save this as cks-immutable-deploy.yaml. The container appends a timestamp to /tmp every few seconds and exits if the write fails.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: ledger
  namespace: prepium-tutorial
spec:
  replicas: 1
  selector:
    matchLabels:
      app: ledger
  template:
    metadata:
      labels:
        app: ledger
    spec:
      securityContext:
        runAsNonRoot: true
        runAsUser: 31000
        runAsGroup: 31000
      containers:
        - name: writer
          image: busybox:1.36
          command: ["sh", "-c", "while date >> /tmp/beat.log; do sleep 5; done; echo cannot write; exit 1"]
          securityContext:
            readOnlyRootFilesystem: true
            allowPrivilegeEscalation: false
            capabilities:
              drop: ["ALL"]
          # step 2 adds a volumeMount for /tmp here

kubectl create namespace prepium-tutorial
kubectl apply -f cks-immutable-deploy.yaml
kubectl -n prepium-tutorial get pods -l app=ledger -w
kubectl -n prepium-tutorial logs deploy/ledger

The Pod starts, writes fail against the read-only root, the script prints cannot write and exits, and the Pod settles into CrashLoopBackOff. This is the normal first result of turning on a read-only root filesystem: the setting works, and the app was relying on a writable path you had not noticed.

2. Give the app only the writable paths it needs

Edit the file: add this under the container, at the level of securityContext,

          volumeMounts:
            - name: scratch
              mountPath: /tmp

and this under the Pod spec, at the level of containers:

      volumes:
        - name: scratch
          emptyDir:
            sizeLimit: 64Mi

Apply again and wait for kubectl -n prepium-tutorial rollout status deployment/ledger. An emptyDir starts empty, survives container restarts, and is deleted when the Pod leaves the node, so nothing written there becomes part of the image. The sizeLimit keeps a compromised process from filling the node's ephemeral storage. Mount only the directories the app really writes (common ones are /tmp, a cache directory and a PID or socket directory).

3. Prove each property from inside

kubectl -n prepium-tutorial exec deploy/ledger -- id
kubectl -n prepium-tutorial exec deploy/ledger -- sh -c "grep ' / ' /proc/mounts"
kubectl -n prepium-tutorial exec deploy/ledger -- tail -2 /tmp/beat.log
kubectl -n prepium-tutorial exec deploy/ledger -- grep -E 'NoNewPrivs|CapEff' /proc/1/status

Expect uid 31000 from id, an ro option on the root (/) mount, fresh timestamps in the log, NoNewPrivs: 1 (the kernel flag behind allowPrivilegeEscalation: false) and an all-zero CapEff after dropping every capability.

4. Understand runAsNonRoot versus runAsUser

runAsUser chooses the UID. runAsNonRoot: true chooses nothing; it makes the kubelet check that the container would not run as UID 0 and refuse to start it otherwise. Remove runAsUser from a copy of the manifest and use an image whose default user is root: the Pod is created, but the container never starts and its events explain the refusal. Setting a numeric non-zero runAsUser satisfies the check. Values in the Pod spec take precedence over the user baked into the image, which is useful for images you do not control.

Common mistakes

• Mounting the emptyDir over a directory that holds files the app ships with. The mount hides them; the app then fails in new ways.
• Putting readOnlyRootFilesystem or allowPrivilegeEscalation at Pod level. They exist only on containers, so set them on every container, init containers included.
• Choosing a UID that cannot read the app's files. The container starts but the app logs permission errors.
• Editing only a manifest file and never applying it, or editing the live object and leaving the file stale. Make both agree.

Cleanup: kubectl delete namespace prepium-tutorial. References: Configure a Security Context, emptyDir volumes, Linux kernel security constraints for Pods and containers. Next, try the restricted Pod Security Standard or Pod versus container securityContext.