Skip to content

Fixing ImagePullBackOff caused by private-registry credentials

Walks from the exact pull error through the pod's imagePullSecrets to the actual Secret content, fixing a missing secret reference, an expired token, or a wrong registry host/namespace. It explicitly does not cover a missing or mistyped image tag ('manifest unknown', 'repository does not exist') — that's a different error and a different fix — and it is not a general Docker registry-login guide: a successful `docker pull` on a developer's laptop proves the credentials are valid outside the cluster, it does not prove the pod itself has them wired up.

Unverifiedno reproductions yetWhy this confidence?
Revision 1published by DevYou curationrevision history
Run the diagnosisEvidence and compatibility

Symptoms

The diagnostic path

8 steps. Every step is written out below in full — the interactive version simply follows the branches for you.

  1. Step 1 · StartPod stuck in ImagePullBackOff pulling from a private registry

    A working docker pull of the same image on a developer's laptop is a red herring for this branch of the problem — it only proves the registry credentials are valid somewhere, not that the pod has them. Kubelet pulls images using the pod's own imagePullSecrets (or its ServiceAccount's), which is a completely separate credential path.

    What happens next

    • passed step 2, Read the exact pull error
  2. Step 2 · TestRead the exact pull error

    Confirm the Events line is actually an authentication/authorization failure and not a different kind of pull error.

    Read-onlysh
    kubectl describe pod <pod-name> -n <namespace>

    Expected result

    Warning  Failed  8s (x4 over 47s)  kubelet  Failed to pull image "registry.example.com/team/app:1.4.2": rpc error: code = Unknown desc = failed to pull and unpack image "registry.example.com/team/app:1.4.2": failed to resolve reference: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

    What happens next

    • passed step 3, Confirm the pod actually references an imagePullSecrets entry
    • failed step 4, This isn't a credentials problem
    • unknown step 4, This isn't a credentials problem
  3. Step 3 · TestConfirm the pod actually references an imagePullSecrets entry

    Check whether the pod spec (directly, or via its ServiceAccount) has any imagePullSecrets configured at all.

    Read-onlysh
    kubectl get pod <pod-name> -n <namespace> -o jsonpath='{.spec.imagePullSecrets}'

    Expected result

    []

    What happens next

    • passed step 5, Check the referenced secret exists in this namespace with the right registry host
    • failed step 6, Fix: attach an image pull secret to the namespace's default ServiceAccount
    • unknown step 6, Fix: attach an image pull secret to the namespace's default ServiceAccount
  4. Step 4 · EndThis isn't a credentials problem

    If the error instead reads something like 'manifest unknown', 'repository does not exist' without an authorization message, or names a tag that clearly doesn't exist, the image reference itself is wrong. Fix the image name/tag in the pod spec — the credential-wiring steps below won't help.

  5. Step 5 · TestCheck the referenced secret exists in this namespace with the right registry host

    imagePullSecrets are namespace-scoped — a secret created in one namespace is invisible to pods in another, which is one of the most common reasons this looks broken despite 'having a secret'.

    Read-onlysh
    kubectl get secret <secret-name> -n <namespace> --type=kubernetes.io/dockerconfigjson -o jsonpath='{.data.\.dockerconfigjson}' | base64 -d

    Expected result

    {"auths":{"registry.example.com":{"username":"deploy-bot","password":"***","auth":"ZGVwbG95LWJvdDoqKio="}}}

    What happens next

    • passed step 7, Fix: the token is stale — regenerate the secret with current credentials
    • failed step 8, Fix: the secret is missing, in the wrong namespace, or has the wrong registry host
    • unknown step 8, Fix: the secret is missing, in the wrong namespace, or has the wrong registry host
  6. Step 6 · FixFix: attach an image pull secret to the namespace's default ServiceAccount

    If nothing is referenced at all, the most durable fix is attaching the secret to the ServiceAccount so every pod in the namespace picks it up automatically, rather than editing every pod spec individually. Any pod already running still needs to be deleted and recreated to pick this up.

    Changes statesh

    Changes system or service state. Review before running.

    Adds the image pull secret to the namespace's default ServiceAccount; pods created after this pick it up automatically, but the current failing pod still needs to be deleted (or its Deployment rolled) to actually retry the pull with it.

    kubectl patch serviceaccount default -n <namespace> -p '{"imagePullSecrets": [{"name": "regcred"}]}'
  7. Step 7 · FixFix: the token is stale — regenerate the secret with current credentials

    The secret exists, is in the right namespace, and names the right registry host, but the pull still fails as unauthorized — the embedded token/password has most likely expired or been rotated since the secret was created. Recreate it with a currently valid credential.

    Handles credentialssh

    Touches credentials or secrets. Never paste real secrets into a shared terminal.

    Overwrites the existing 'regcred' Secret in this namespace with the newly supplied registry credentials; anyone able to read Secrets in this namespace can read the embedded password after this runs.

    kubectl create secret docker-registry regcred \
      --docker-server=registry.example.com \
      --docker-username=<username> \
      --docker-password=<token> \
      --docker-email=<email> \
      -n <namespace> \
      --dry-run=client -o yaml | kubectl apply -f -
  8. Step 8 · FixFix: the secret is missing, in the wrong namespace, or has the wrong registry host

    Recreate the secret in the correct namespace with --docker-server set to exactly the host used in the image reference (for example ghcr.io if the image is ghcr.io/org/app:tag) — a mismatched host is silently ignored by kubelet's credential lookup.

    Handles credentialssh

    Touches credentials or secrets. Never paste real secrets into a shared terminal.

    Creates or overwrites the 'regcred' Secret in this namespace with the supplied registry credentials; anyone able to read Secrets in this namespace can read the embedded password after this runs.

    kubectl create secret docker-registry regcred \
      --docker-server=<exact-registry-host> \
      --docker-username=<username> \
      --docker-password=<token> \
      --docker-email=<email> \
      -n <namespace> \
      --dry-run=client -o yaml | kubectl apply -f -

Sources

Why this confidence?

What would strengthen it: 6 more independent reproductions. Reproductions from 3 more distinct environments.