CrashLoopBackOff
The container keeps exiting and the kubelet restarts it with growing delay; read the previous logs and the exit code to find out why.
The problem
The container starts, exits, and the kubelet restarts it again and again with a growing pause. CrashLoopBackOff is not an error in itself; it is the kubelet telling you the container keeps terminating. The real cause is in the last run's logs and exit code, which are gone after the next restart unless you ask for the previous instance.
How it works
- Symptom: STATUS `CrashLoopBackOff` (or `Error` between attempts), RESTARTS climbing, event `Back-off restarting failed container`. The kubelet restarts with an exponential delay of 10s, 20s, 40s and so on, capped at 5 minutes, and resets the delay after the container has run cleanly for 10 minutes. Applies with `restartPolicy: Always` or `OnFailure`.
- First look: `kubectl logs <pod> -c <container> --previous` for the crashed run, then `kubectl describe pod <pod>` and read `Last State: Terminated` (reason, exit code, started/finished times). A container that dies within a second usually fails at startup; one that runs for exactly the probe timeout was likely killed by a liveness probe.
- Common causes (the process fails): application crash or unhandled exception (exit 1), bad config or missing env var, unreachable database or dependency at startup, wrong `command`/`args` or entrypoint (exit 127 not found, 126 not executable, `exec format error` for a wrong CPU architecture), permission denied from `securityContext` (non-root user writing to a root-owned path, read-only root filesystem).
- Common causes (Kubernetes kills it): a liveness probe that fails on a healthy but slow-starting app (the log ends normally and the exit code is 137 or 143; the event says `Liveness probe failed`), OOM kill (exit 137, `Reason: OOMKilled`), or a process that exits 0 because it daemonised or finished its work, which `restartPolicy: Always` still restarts. A missing Secret or ConfigMap key is not this: it shows as `CreateContainerConfigError`.
- Fix: act on the exit code. 1 or 2: fix the app or its config using the previous logs. 126/127: correct `command`, `args`, image or file permissions. 137: see `oomkilled` or loosen the probe. 0: make the process stay in the foreground, or use a Job. For slow starters add a `startupProbe` instead of a huge `initialDelaySeconds` on liveness.
- Debugging tricks: override the command to keep the container alive (`command: ["sleep", "3600"]`), then `kubectl exec` in and run the real entrypoint by hand; or `kubectl debug <pod> -it --copy-to=<pod>-debug --container=<container> -- sh` to get a copy with a changed command; or attach an ephemeral container (`kubectl debug -it <pod> --image=busybox:1.36 --target=<container>`) to inspect a Pod that is up between crashes.
- Verify: RESTARTS stops increasing and stays flat across several minutes, READY becomes `1/1`, and `kubectl logs <pod> -f` shows normal operation. Note the restart count does not reset by itself until the Pod is recreated, so compare it over time rather than expecting zero.
- Prevent: validate config at CI time, make the app fail fast with a clear message, log to stdout/stderr, use `startupProbe` for slow boots, keep liveness checks cheap and independent of dependencies (a database outage should not restart every app Pod), and alert on `rate(kube_pod_container_status_restarts_total[15m]) > 0`.
Examples
Previous logs and the exit code of the last termination.
bash
kubectl logs "<pod>" -n "<ns>" -c "<container>" --previous --tail=200
kubectl get pod "<pod>" -n "<ns>" \
-o jsonpath='{.status.containerStatuses[*].lastState.terminated}'
# Look for: reason, exitCode, startedAt, finishedAt
kubectl describe pod "<pod>" -n "<ns>" | grep -E -A6 'Last State|Liveness'
Debug without touching the live Pod: a copy with a shell, or an ephemeral container.
bash
# Copy of the Pod whose container just sleeps, so you can exec and try things:
kubectl debug "<pod>" -n "<ns>" -it --copy-to="<pod>-debug" \
--container="<container>" -- sleep 3600
kubectl exec -it "<pod>-debug" -n "<ns>" -c "<container>" -- sh
# Or attach an ephemeral container to a Pod that is currently up:
kubectl debug -it "<pod>" -n "<ns>" --image=busybox:1.36 --target="<container>"
kubectl delete pod "<pod>-debug" -n "<ns>"
A slow-starting app: startupProbe holds off liveness; command override shown as a comment for debugging.
yaml
apiVersion: v1
kind: Pod
metadata:
name: slow-api
spec:
containers:
- name: api
image: registry.example.com/api:1.4.2
# Debug only: keep the container alive, then exec in.
# command: ["sleep", "3600"]
startupProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 5
failureThreshold: 36
livenessProbe:
httpGet:
path: /healthz
port: 8080
periodSeconds: 10
failureThreshold: 3
In practice
- kubelet — Owns the restart back-off (10s doubling to a 5 minute cap, reset after 10 minutes of successful running). The delay is why a fix can seem slow to take effect; delete the Pod to restart immediately.
- kubectl debug — Creates ephemeral containers or Pod copies for debugging. Ephemeral containers are stable in current Kubernetes; they share namespaces with `--target` but cannot be removed from the Pod.
- kube-state-metrics / Prometheus — `kube_pod_container_status_restarts_total` and the waiting-reason metric are the standard inputs for CrashLoopBackOff alerts.
- In practice — A frequent cause is a liveness probe that checks a dependency: when the database blips, every replica restarts together and the outage gets longer.
Trade-offs
Pros
- The back-off protects the node and registry from a tight crash loop.
- `--previous` plus the exit code usually identifies the cause in minutes.
Cons
- The growing delay (up to 5 minutes) slows the feedback loop while you iterate on a fix.
- Previous logs only cover the last instance, and are lost if the Pod is deleted or evicted; central logging is needed for history.