Skip to content

Diagnosing containerd task-creation timeouts

Narrows a context-deadline-exceeded failure to node-level resource pressure versus containerd being slowed by a large pile-up of stale containers/images on one node. It does not cover a genuinely crashed containerd daemon (a different, more obvious failure mode) or CNI-level sandbox failures that don't mention containerd task creation specifically.

Unverifiedno reproductions yetWhy this confidence?
Revision 1published by DevYou curationrevision history
Run the diagnosisEvidence and compatibility

Symptoms

The diagnostic path

8 steps. Every step is written out below in full — the interactive version simply follows the branches for you.

  1. Step 1 · Startcontainerd task creation is timing out

    This deadline is a CRI-level timeout, not an application error — something is slowing down containerd's own container-creation path on the affected node(s).

    What happens next

    • passed step 2, Is this happening on one node or across the cluster?
  2. Step 2 · TestIs this happening on one node or across the cluster?

    Check which node(s) the failing pods actually land on before diagnosing a specific machine.

    Read-onlysh
    kubectl get events -A --field-selector reason=FailedCreatePodSandBox -o wide

    Expected result

    NAMESPACE   OBJECT       REASON                    NODE
    default     pod/app-7f  FailedCreatePodSandBox    node-8

    What happens next

    • passed step 3, Check the affected node's resource pressure and containerd's own health
    • failed step 4, Root cause: likely a cluster-wide containerd/CRI or network issue
    • unknown step 3, Check the affected node's resource pressure and containerd's own health
  3. Step 3 · TestCheck the affected node's resource pressure and containerd's own health

    Look for DiskPressure, MemoryPressure or PIDPressure on the node, and whether the containerd service itself is degraded.

    Read-onlysh
    kubectl describe node <node-name> | grep -A5 Conditions

    Expected result

    Conditions:
      Type             Status
      DiskPressure     True
      MemoryPressure   False
      PIDPressure      False
      Ready            True

    What happens next

    • passed step 5, Root cause: the node is resource-starved, usually on disk
    • failed step 6, Is there a large pile-up of stale containers or images on this node?
    • unknown step 6, Is there a large pile-up of stale containers or images on this node?
  4. Step 4 · Root causeRoot cause: likely a cluster-wide containerd/CRI or network issue

    This is happening broadly across most or all nodes, not one sick machine — that points at something affecting every node identically: a containerd/CRI version mismatch after an upgrade, a shared registry/network path every node depends on, or a cluster-wide config change. This needs a broader investigation than a single node's resource state.

  5. Step 5 · Root causeRoot cause: the node is resource-starved, usually on disk

    The node shows DiskPressure, MemoryPressure or PIDPressure True, or containerd's own service is degraded/restarting — containerd can't finish creating a task within the CRI deadline because the node itself is starved. Overlay/snapshot setup is disk-heavy, so disk contention is the most common of the three. Find what's actually consuming the disk before deciding whether to free space, move pods off the node, or add capacity.

    Read-onlysh
    df -h

    Expected result

    Filesystem      Size  Used Avail Use% Mounted on
    /dev/sda1       97G   95G  1.2G  99% /var/lib/containerd
  6. Step 6 · TestIs there a large pile-up of stale containers or images on this node?

    A very large number of stopped containers or dangling images is a known way to slow every containerd operation, including task creation, because containerd has to walk more state on each call.

    Read-onlysh
    crictl ps -a | wc -l && crictl images | wc -l

    Expected result

    3812
    214

    What happens next

    • passed step 7, Fix: remove the stale exited containers
    • failed step 8, Node and containerd both look healthy
    • unknown step 8, Node and containerd both look healthy
  7. Step 7 · FixFix: remove the stale exited containers

    An unusually large exited-container count is slowing containerd down. Prune them on this specific node.

    Destructivesh

    Can delete data or break a running service. This is not reversible.

    Permanently removes every exited container tracked by containerd on this node, freeing the state containerd scans on each operation. It only affects Exited containers, not Running ones, but you lose their logs and exit codes once removed.

    crictl rm $(crictl ps -a -q --state Exited)
  8. Step 8 · EndNode and containerd both look healthy

    Resource conditions are clear and there's no unusual pile-up — this needs a lower-level trace (containerd/runc debug logs, or checking for a known runtime bug matching your containerd version) beyond a generic checklist.

Sources

Why this confidence?

What would strengthen it: 6 more independent reproductions. Reproductions from 3 more distinct environments.