Finding and fixing a node-postgres connection leak

Symptoms

• Fine at low traffic; after sustained requests, new queries start throwing `timeout exceeded when trying to connect` or hang
• Restarting the process temporarily fixes it, and it recurs after roughly the same amount of traffic
• The affected code calls pool.connect() directly rather than pool.query()

The diagnostic path

8 steps, exactly as this revision published them.

1. Step 1 · StartThe pool exhausts gradually under sustained normal traffic

   Unlike a startup burst, this climbs steadily over time under otherwise ordinary load and doesn't recover on its own.

   What happens next

   • passed → step 2, Watch the pool counters over time, not just once

2. Step 2 · TestWatch the pool counters over time, not just once

   Sample repeatedly across several minutes of normal traffic rather than at a single instant.

   Read-onlyjavascript

   Copy

   console.log({ total: pool.totalCount, idle: pool.idleCount, waiting: pool.waitingCount });

   Expected result

   { total: 10, idle: 0, waiting: 37 }

   What happens next

   • passed → step 3, idleCount stays at 0 and totalCount never drops — a leak, not a burst
   • failed → step 4, Doesn't match the leak signature
   • unknown → step 4, Doesn't match the leak signature

3. Step 3 · Root causeidleCount stays at 0 and totalCount never drops — a leak, not a burst

   If idleCount stays pinned at 0 and waitingCount keeps growing across ordinary, non-bursty traffic, connections are being checked out and never returned. Every pool.connect() must be matched with client.release(), including on every error path — a thrown/rejected query, an early return, or an unhandled exception between connect() and release() leaves that client permanently checked out, and the pool has no way to know it was abandoned. Note: pool.query(...) used without .connect() doesn't leak this way, since it acquires and releases a client for you automatically — the leak is specific to code that manually calls pool.connect().

   What happens next

   • passed → step 5, Wrap every manual checkout in try/finally

4. Step 4 · EndDoesn't match the leak signature

   If idleCount recovers between bursts and totalCount doesn't stay pinned at max, this may be genuine undersized capacity for a concurrency spike rather than a leak — see the connect-timeout playbook.

5. Step 5 · FixWrap every manual checkout in try/finally

   Guarantee release() runs on every exit path, including exceptions.

   Read-onlyjavascript

   Copy

   const client = await pool.connect();
   try {
     const result = await client.query("SELECT * FROM orders WHERE user_id = $1", [userId]);
     return result.rows;
   } finally {
     client.release();
   }

   What happens next

   • passed → step 6, Re-run the same sustained load and check the counters stay bounded

6. Step 6 · Verify the fixRe-run the same sustained load and check the counters stay bounded

   idleCount should fluctuate rather than stay pinned at 0, and waitingCount should stay near 0.

   Read-onlyjavascript

   Copy

   console.log({ total: pool.totalCount, idle: pool.idleCount, waiting: pool.waitingCount });

   Expected result

   { total: 10, idle: 6, waiting: 0 }

   What happens next

   • passed → step 7, Resolved
   • failed → step 8, Search for other manual .connect() call sites
   • unknown → step 8, Search for other manual .connect() call sites

7. Step 7 · EndResolved

   Pool usage no longer climbs unbounded under sustained traffic.

8. Step 8 · FixSearch for other manual .connect() call sites

   A single other place in the codebase that calls pool.connect() without a matching try/finally reproduces the exact same symptom. Search the whole codebase for pool.connect( rather than assuming the one fixed instance was the only one.

Sources

• pg.Pool— node-postgres
• pg.Client— node-postgres