Skip to content

Diagnosing and fixing an OOMKilled container

Confirms a genuine OOM kill from kubectl describe pod, then distinguishes a memory limit that's simply undersized from a leak-shaped growth pattern that raising the limit would only delay. Covers raising the memory request/limit as a concrete fix; it does not do in-app memory profiling or cover node-level memory pressure evictions, which are a different mechanism from a per-container limit kill.

Kuberneteskubectl
Unverifiedno reproductions yetWhy this confidence?
Revision 1published by DevYou curationrevision history
Run the diagnosisEvidence and compatibility

Symptoms

The diagnostic path

7 steps. Every step is written out below in full — the interactive version simply follows the branches for you.

  1. Step 1 · StartContainer is repeatedly OOMKilled

    Before touching the limit, confirm this is actually an OOM kill and work out whether usage is undersized-from-the-start or grows over time — raising the limit fixes the first and only delays the second.

    What happens next

    • passed → step 2, Confirm OOMKilled and read the configured limit

  2. Step 2 · TestConfirm OOMKilled and read the configured limit

    Check that Last State really is Terminated/OOMKilled with Exit Code 137, and note the memory limit the container is actually running under.

    Read-onlysh
    kubectl describe pod <pod-name> -n <namespace>

    Expected result

    Last State:     Terminated
  Reason:       OOMKilled
  Exit Code:    137
Limits:
  memory:  256Mi
Requests:
  memory:  256Mi

    What happens next

    • passed → step 3, Does memory usage grow across the container's lifetime before the kill, or is it high from the moment the container starts?
    • failed → step 4, This isn't an OOM kill
    • unknown → step 4, This isn't an OOM kill

  3. Step 3 · TestDoes memory usage grow across the container's lifetime before the kill, or is it high from the moment the container starts?

    Watch kubectl top pod --containers over several minutes (or a metrics dashboard if you have one). A steady climb toward the limit is leak-shaped; being close to the limit within seconds of starting is not.

    Read-onlysh
    kubectl top pod <pod-name> -n <namespace> --containers

    Expected result

    POD              NAME   CPU(cores)   MEMORY(bytes)
<pod-name>       app    120m         248Mi

    What happens next

    • passed → step 5, Root cause: memory grows over the container's lifetime (leak-shaped)
    • failed → step 6, Fix: raise the memory request and limit to match the real working set
    • unknown → step 7, Can't tell yet which pattern this is

  4. Step 4 · EndThis isn't an OOM kill

    Last State Reason wasn't OOMKilled or Exit Code wasn't 137 — this playbook only covers genuine OOM kills. Re-read the Last State block; a different reason belongs in the CrashLoopBackOff triage playbook instead.

  5. Step 5 · Root causeRoot cause: memory grows over the container's lifetime (leak-shaped)

    This looks like the app not releasing memory — unbounded caches, accumulating event listeners, or a queue with no eviction — rather than the limit being wrong. Raising the limit only delays the kill. From here it's an application-level memory investigation (heap/allocation profiling appropriate to the runtime, or comparing against a previous known-good build), not a Kubernetes-level fix.

  6. Step 6 · FixFix: raise the memory request and limit to match the real working set

    If usage is high immediately and stays roughly flat rather than climbing, the limit is simply set below what the container needs. Raise both request and limit — leaving the request far below the limit is what let the pod get scheduled onto a node that then couldn't actually give it the memory it needed.

    Changes statesh

    Changes system or service state. Review before running.

    Patches the deployment's pod template with the new memory request and limit, which triggers a rolling restart of the deployment's pods.

    kubectl set resources deployment/<deployment-name> -c=<container-name> --limits=memory=512Mi --requests=memory=512Mi -n <namespace>

  7. Step 7 · EndCan't tell yet which pattern this is

    If there isn't enough history to tell growth-over-time from high-from-the-start, temporarily raise the limit and watch memory over a longer window, or add basic heap/allocation logging in the app before deciding which branch applies.

Sources

Why this confidence?

What would strengthen it: 6 more independent reproductions. Reproductions from 3 more distinct environments.