Skip to content

Triaging CrashLoopBackOff to its actual cause

Uses kubectl describe pod's Last State reason, exit code and Events to sort a CrashLoopBackOff into one of five common causes and points at the right next step for each. It deliberately does not fix any of them in depth — the OOM branch, for instance, defers to the dedicated OOMKilled playbook — and it does not cover ImagePullBackOff, FailedScheduling or CreateContainerConfigError, which are different pod conditions with their own playbooks.

Unverifiedno reproductions yetWhy this confidence?
Revision 1published by DevYou curationrevision history
Run the diagnosisEvidence and compatibility

Symptoms

The diagnostic path

8 steps. Every step is written out below in full — the interactive version simply follows the branches for you.

  1. Step 1 · StartPod is stuck in CrashLoopBackOff

    This triages a CrashLoopBackOff down to one of five common causes — OOM kill, failing liveness probe, a bad command/args override, a missing config value or secret, or an image whose entrypoint isn't a long-running process — using the pod's own Last State and Events rather than guessing from the symptom name alone.

    What happens next

    • passed step 2, Read the Last State reason and exit code
  2. Step 2 · ObservationRead the Last State reason and exit code

    kubectl describe pod is more reliable here than logs — the Last State block records the exit code and OOM flag even after the container has already been replaced. Exit Code 137 + Reason OOMKilled means memory. Exit Code 0/143 with repeated 'Unhealthy' Warning events before each restart means the liveness probe, not the process, is killing it. Exit Code 127, or an event reading 'OCI runtime exec failed: exec: "<cmd>": executable file not found in $PATH', means the configured command/args don't match a binary the image actually has. An app-level error in kubectl logs --previous about a missing environment variable, file, or connection string means missing config. Exit Code 0 with no OOM or probe warnings, and the container only staying up a few seconds each time, means the image's own entrypoint isn't a long-running foreground process.

    Read-onlysh
    kubectl describe pod <pod-name> -n <namespace>

    Expected result

    State:          Waiting
      Reason:       CrashLoopBackOff
    Last State:     Terminated
      Reason:       OOMKilled
      Exit Code:    137
    Ready:          False
    Restart Count:  6
    Events:
      Warning  BackOff  6s (x12 over 4m)  kubelet  Back-off restarting failed container

    What happens next

    • different output step 3, Root cause: the container was OOM-killed
    • different output step 4, Root cause: the liveness probe is killing an otherwise-working container
    • different output step 5, Root cause: the container's command or args are wrong
    • different output step 6, Root cause: the app is crashing on a missing config value or secret
    • different output step 7, Root cause: the image's entrypoint isn't a long-running process
    • unknown step 8, None of these match — treat it as an application bug
  3. Step 3 · Root causeRoot cause: the container was OOM-killed

    Last State Reason is OOMKilled and Exit Code is 137 — the kernel's cgroup OOM killer sent SIGKILL because the container tried to use more memory than its configured limit. This branch is confirmation, not the fix: the dedicated OOMKilled playbook covers sizing the limit correctly and telling a genuine leak apart from an undersized limit.

  4. Step 4 · Root causeRoot cause: the liveness probe is killing an otherwise-working container

    Exit Code is usually 0 or 143 and Events show repeated 'Unhealthy' warnings from the liveness probe before each restart — kubelet is restarting the container because the probe failed, not because the process crashed on its own. Check initialDelaySeconds, timeoutSeconds and failureThreshold on the pod's livenessProbe: a probe that fires before the app finishes starting, or that has too short a timeout under load, produces exactly this pattern. A missing readinessProbe alongside it means traffic also hits the pod during the same slow-start window.

    Read-onlysh
    kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[0].livenessProbe}'

    Expected result

    {"httpGet":{"path":"/healthz","port":8080},"initialDelaySeconds":2,"timeoutSeconds":1,"failureThreshold":1,"periodSeconds":5}
  5. Step 5 · Root causeRoot cause: the container's command or args are wrong

    Exit Code 127, or the event text 'OCI runtime exec failed: exec: "<cmd>": executable file not found in $PATH', means the command/args set in the pod or container spec don't match a binary that actually exists in the image — commonly a typo, a path that changed between image versions, or a debugging override that was never removed. Compare the spec against what the image actually ships.

    Read-onlysh
    kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.containers[0].command}{" "}{.spec.containers[0].args}'

    Expected result

    [/app/bin/servr] []
  6. Step 6 · Root causeRoot cause: the app is crashing on a missing config value or secret

    The container starts, runs briefly, then exits with an application-level error referencing a missing environment variable, config file, or connection string — not an OOM or probe signal. Usually a ConfigMap/Secret key referenced by envFrom or env.valueFrom doesn't exist or wasn't updated after a rename, or a required setting simply wasn't provided in this environment. This is distinct from CreateContainerConfigError, a Waiting-state condition shown before the container ever starts at all — if you see that reason instead, the fix is the same (check the referenced ConfigMap/Secret) but the pod never actually ran.

    Read-onlysh
    kubectl logs <pod-name> -n <namespace> --previous

    Expected result

    Error: environment variable DATABASE_URL is required but was not set
        at loadConfig (/app/src/config.js:14:11)
  7. Step 7 · Root causeRoot cause: the image's entrypoint isn't a long-running process

    Exit Code is 0, there are no OOM or probe warnings, and the container only stays up for a few seconds each time — the process the image runs completes and exits normally, which Kubernetes treats as a crash because a pod's container is expected to keep running in the foreground. This happens most often with an image built for a one-shot task (a migration, a setup script) deployed as a Deployment/ReplicaSet instead of a Job, or a custom command that backgrounds the real process and returns immediately. Run it as a Job/CronJob if it's meant to finish, or make the entrypoint exec the long-running process in the foreground.

  8. Step 8 · EndNone of these match — treat it as an application bug

    If the exit code and events don't fit any of the patterns above, kubectl logs <pod-name> -n <namespace> --previous is the next step — the previous container's stdout/stderr almost always contains the real application error. From here this stops being a generic Kubernetes problem and becomes whatever that error says.

Sources

Why this confidence?

What would strengthen it: 6 more independent reproductions. Reproductions from 3 more distinct environments.