Skip to content

Diagnosing ingress-nginx 504 Gateway Time-out

Splits 504s into a slow-backend branch (raise the per-Ingress timeout for a legitimately slow endpoint, or fix backend contention if it's overload-shaped) and a no-live-upstream branch (no Ready endpoints, or a Service/NetworkPolicy blocking traffic to an otherwise-healthy pod). It does not cover 502/503 responses, which usually mean the backend actively refused or reset the connection rather than never answering, or client-side timeouts that never reach the ingress at all.

NGINX Ingress ControllerKuberneteskubectl
Unverifiedno reproductions yetWhy this confidence?
Revision 1published by DevYou curationrevision history
Run the diagnosisEvidence and compatibility

Symptoms

The diagnostic path

8 steps. Every step is written out below in full — the interactive version simply follows the branches for you.

  1. Step 1 · StartIngress returns 504 Gateway Time-out under load

    504 from ingress-nginx means nginx itself gave up waiting for (or never found) a working backend — it's not the backend actively returning an error. The exact log message splits this into two genuinely different causes.

    What happens next

    • passed → step 2, Check the ingress-controller's error log for the exact upstream message

  2. Step 2 · TestCheck the ingress-controller's error log for the exact upstream message

    'upstream timed out' means nginx reached a backend but it didn't answer in time. 'no live upstreams' (or connection-refused style messages) means nginx had nothing ready to send the request to at all.

    Read-onlysh
    kubectl logs -n ingress-nginx <ingress-controller-pod> --tail=200 | grep -i 'upstream timed out\|no live upstreams'

    Expected result

    2026-08-05T18:44:02+00:00 [error] 34#34: *8821 upstream timed out (110: Connection timed out) while reading response header from upstream, client: 10.2.1.4, server: api.internal, request: "POST /v1/orders HTTP/1.1", upstream: "http://10.2.3.9:8080/v1/orders"

    What happens next

    • passed → step 3, Is the backend genuinely slow, or is nginx's timeout just too low for it?
    • failed → step 4, Are the backend pods actually Ready?
    • unknown → step 4, Are the backend pods actually Ready?

  3. Step 3 · TestIs the backend genuinely slow, or is nginx's timeout just too low for it?

    Compare the backend's real response time for this endpoint (bypassing the ingress) against the configured proxy-read-timeout, which defaults to 60 seconds.

    Read-onlysh
    kubectl exec -n <namespace> <debug-pod> -- curl -w '%{time_total}\n' -o /dev/null -s http://<backend-service>.<namespace>.svc.cluster.local:<port>/<path>

    Expected result

    63.912000

    What happens next

    • passed → step 5, Fix: raise this Ingress's proxy-read/send-timeout
    • failed → step 6, Root cause: backend contention under load, not a timeout setting
    • unknown → step 6, Root cause: backend contention under load, not a timeout setting

  4. Step 4 · TestAre the backend pods actually Ready?

    'no live upstreams' means the Service behind this Ingress currently has no Ready endpoints for nginx to send traffic to.

    Read-onlysh
    kubectl get endpoints <service-name> -n <namespace>

    Expected result

    NAME              ENDPOINTS   AGE
<service-name>    <none>      41d

    What happens next

    • passed → step 7, Root cause: the Service has no Ready backend pods
    • failed → step 8, Root cause: endpoints exist but traffic can't reach them
    • unknown → step 8, Root cause: endpoints exist but traffic can't reach them

  5. Step 5 · FixFix: raise this Ingress's proxy-read/send-timeout

    The endpoint is legitimately slower than nginx's default 60-second timeout for reasons that are expected (a report-generation endpoint, for example). Raise the timeout on this specific Ingress rather than cluster-wide, so a genuinely hung backend elsewhere still fails visibly.

    Changes statesh

    Changes system or service state. Review before running.

    Changes how long this specific Ingress's nginx location block waits for the backend before returning 504. ingress-nginx hot-reloads its configuration, so this takes effect within seconds without restarting the controller pod.

    kubectl annotate ingress <ingress-name> -n <namespace> nginx.ingress.kubernetes.io/proxy-read-timeout=120 nginx.ingress.kubernetes.io/proxy-send-timeout=120 --overwrite

  6. Step 6 · Root causeRoot cause: backend contention under load, not a timeout setting

    The backend responds quickly outside the load spike, so the slowness only appears under concurrent load — this is capacity or contention (thread pool exhaustion, a database connection pool maxing out, insufficient replicas). Raising the ingress timeout just makes clients wait longer for the same overload; the fix is scaling the backend or fixing whatever it blocks on under load, which is outside ingress-nginx's control.

  7. Step 7 · Root causeRoot cause: the Service has no Ready backend pods

    ENDPOINTS is empty — nginx genuinely has nothing to proxy to. Check the Deployment's readiness probe (failing readiness keeps a pod out of Endpoints even while Running), a label selector mismatch between the Service and the pods, or the Deployment being scaled to zero.

  8. Step 8 · Root causeRoot cause: endpoints exist but traffic can't reach them

    Endpoints ARE populated and the pods look Ready, yet nginx still can't get a response — check for a Service/Ingress backend port mismatch, or a NetworkPolicy blocking traffic from the ingress-controller's namespace to the backend's namespace. Kubernetes considers the pod healthy from a readiness-probe standpoint even though packets from the ingress controller specifically never arrive.

Sources

Why this confidence?

What would strengthen it: 6 more independent reproductions. Reproductions from 3 more distinct environments.