Skip to content

Revision 1 — the current text. The evidence on this page is this revision’s own — it has not been carried forward from, or to, any other revision.

Fixing CoreDNS SERVFAIL from a detected forwarding loop

Confirms the crash is genuinely the loop plugin (not an unrelated SERVFAIL cause), traces it to the node's resolv.conf in the common case, and fixes it by forwarding to explicit upstream resolvers instead. It does not cover SERVFAIL caused by an actually-unreachable upstream DNS server, NetworkPolicy blocking egress from CoreDNS, or CoreDNS being resource-starved — those don't log the loop-plugin message and need a different diagnosis.

CoreDNSKuberneteskubectl
Unverifiedno reproductions yetWhy this confidence?
Revision 1published by DevYou curation

Symptoms

The diagnostic path

8 steps, exactly as this revision published them.

  1. Step 1 · StartCoreDNS crashlooping, DNS returns SERVFAIL

    CoreDNS's own loop plugin deliberately crashes it rather than serving an infinite loop — that's a safety feature, not the bug itself. The bug is whatever is making CoreDNS query itself in the first place.

    What happens next

    • passed → step 2, Confirm it's the loop plugin

  2. Step 2 · TestConfirm it's the loop plugin

    Check the CoreDNS pod logs for the specific FATAL loop message before assuming the cause.

    Read-onlysh
    kubectl logs -n kube-system -l k8s-app=kube-dns --tail=20

    Expected result

    [FATAL] plugin/loop: Loop (127.0.0.1:52561 -> :53) detected for zone ".", see https://coredns.io/plugins/loop#troubleshooting. Query: "HINFO 4547991809047724227.400057319072097461."

    What happens next

    • passed → step 3, Find where the loop actually originates: the node's resolv.conf
    • failed → step 4, This SERVFAIL isn't the loop-plugin kind
    • unknown → step 4, This SERVFAIL isn't the loop-plugin kind

  3. Step 3 · TestFind where the loop actually originates: the node's resolv.conf

    The most common source is the node's own /etc/resolv.conf, which kubelet copies into the CoreDNS Corefile's default upstream (forward . /etc/resolv.conf). If it contains a loopback stub such as 127.0.0.53 (systemd-resolved), CoreDNS ends up forwarding to itself indirectly through that stub.

    Read-onlysh
    cat /etc/resolv.conf

    Expected result

    nameserver 127.0.0.53
options edns0 trust-ad
search example.internal

    What happens next

    • passed → step 5, Fix: forward to explicit upstream resolvers instead of the node's resolv.conf
    • failed → step 6, Root cause: the loop originates somewhere other than the node's resolv.conf
    • unknown → step 6, Root cause: the loop originates somewhere other than the node's resolv.conf

  4. Step 4 · EndThis SERVFAIL isn't the loop-plugin kind

    No FATAL loop message in the logs — treat this as a different SERVFAIL cause: upstream DNS actually unreachable, a NetworkPolicy blocking egress from CoreDNS's namespace, or CoreDNS being throttled/OOM-limited under load.

  5. Step 5 · FixFix: forward to explicit upstream resolvers instead of the node's resolv.conf

    Replace forward . /etc/resolv.conf in the CoreDNS Corefile with explicit, known-good upstream resolvers so CoreDNS stops depending on whatever loopback stub the node happens to have configured, then restart CoreDNS to pick up the change. The alternative fix — pointing kubelet's --resolv-conf flag at /run/systemd/resolve/resolv.conf, which has the real upstream servers rather than the stub — requires node-level access and a kubelet restart, so this Corefile edit is the lower-risk, cluster-only option.

    Changes statesh

    Changes system or service state. Review before running.

    Changes what upstream DNS servers every CoreDNS pod forwards unresolved queries to, and restarts every CoreDNS pod to pick up the change; there is a brief window of reduced DNS capacity during the rollout.

    kubectl edit configmap coredns -n kube-system
# change: forward . /etc/resolv.conf
# to:     forward . 1.1.1.1 8.8.8.8
kubectl rollout restart deployment coredns -n kube-system

    What happens next

    • passed → step 7, Confirm DNS resolves again

  6. Step 6 · Root causeRoot cause: the loop originates somewhere other than the node's resolv.conf

    resolv.conf on the node doesn't show a loopback address, so the self-reference is coming from elsewhere — most often a custom Corefile forward target that itself resolves back to CoreDNS's own ClusterIP. Compare the Corefile's forward line against the kube-dns Service's ClusterIP; if they point at each other, that's the loop, and it needs to be traced by hand rather than fixed with the resolv.conf workaround below.

    Read-onlysh
    kubectl get configmap coredns -n kube-system -o yaml | grep forward

    Expected result

        forward . 10.96.0.10

  7. Step 7 · Verify the fixConfirm DNS resolves again

    Run a lookup from inside the cluster to confirm CoreDNS is answering normally.

    Read-onlysh
    kubectl run -it --rm dnstest --image=busybox:1.36 --restart=Never -- nslookup kubernetes.default

    Expected result

    Server:    10.96.0.10
Address:   10.96.0.10:53

Name:      kubernetes.default.svc.cluster.local
Address:   10.96.0.1

    What happens next

    • passed → step 8, DNS restored
    • failed → step 6, Root cause: the loop originates somewhere other than the node's resolv.conf
    • unknown → step 6, Root cause: the loop originates somewhere other than the node's resolv.conf

  8. Step 8 · EndDNS restored

    Lookups succeed and CoreDNS pods are no longer restarting. Nothing further to do.

Sources

Why this confidence?

What would strengthen it: 6 more independent reproductions. Reproductions from 3 more distinct environments.

This counts only what was recorded against revision 1 itself. Nothing reported against another revision is included here — see the revision history for why.