[{"data":1,"prerenderedAt":6},["ShallowReactive",2],{"post-content-mysql-too-many-connections-pool-math":3},{"content":4,"lastModified":5},"\u003Cfigure>\n  \u003Cimg src=\"https:\u002F\u002Fimages.pexels.com\u002Fphotos\u002F37346951\u002Fpexels-photo-37346951.jpeg?auto=compress&cs=tinysrgb&w=1200\" alt=\"People filing through the single entrance of a large building, more traffic than doors\" loading=\"lazy\" \u002F>\n\u003C\u002Ffigure>\n\n\u003Cp>Two Fridays ago our API started throwing 500s at 9:40am. Every request that touched MySQL failed with \u003Ccode>ERROR 1040 (HY000): Too many connections\u003C\u002Fcode>, while CPU on the database box sat at 12%. Nobody had deployed anything that morning. The server was mostly idle and still refusing to talk to us.\u003C\u002Fp>\n\n\u003Cp>The reason showed up in the first query I ran.\u003C\u002Fp>\n\n\u003Ch2>Connected is not the same as busy\u003C\u002Fh2>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SHOW GLOBAL STATUS WHERE Variable_name IN (\n  'Threads_connected', 'Threads_running', 'Max_used_connections',\n  'Max_used_connections_time', 'Connection_errors_max_connections', 'Aborted_clients'\n);\n\n-- Threads_connected                    152\n-- Threads_running                        3\n-- Max_used_connections                 152\n-- Max_used_connections_time            2026-10-02 09:41:07\n-- Connection_errors_max_connections   2140\n-- Aborted_clients                    18326\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>152 connections open, 3 of them running a query. MySQL accepts 151 by default, and it had refused 2,140 attempts since the last restart. We were pinned at the ceiling while the database did almost nothing. That gap between \u003Ccode>Threads_connected\u003C\u002Fcode> and \u003Ccode>Threads_running\u003C\u002Fcode> is the entire diagnosis: the connections existed, the work did not. Something was holding connections open while idle.\u003C\u002Fp>\n\n\u003Ch2>Find out who is holding them\u003C\u002Fh2>\n\n\u003Cp>\u003Ccode>SHOW PROCESSLIST\u003C\u002Fcode> is the reflex, and with 152 rows it tells you nothing useful. Group it and the answer usually shows up on the first line.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SELECT user,\n       SUBSTRING_INDEX(host, ':', 1) AS client_host,\n       db,\n       command,\n       COUNT(*)  AS conns,\n       MAX(time) AS longest_idle_seconds\nFROM information_schema.processlist\nGROUP BY user, client_host, db, command\nORDER BY conns DESC;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Ours came back with 118 rows of \u003Ccode>command = 'Sleep'\u003C\u002Fcode> from three pod IPs, the longest one idle for 4,700 seconds. No leak and no runaway query. Just pools far larger than the work they had to do.\u003C\u002Fp>\n\n\u003Cp>\u003Ccode>Sleep\u003C\u002Fcode> is not automatically a bug, by the way. Idle pooled connections are the point of pooling. What is not normal is 118 of them on a server where 3 queries are running, since that is the shape of a pool sized by guesswork rather than arithmetic.\u003C\u002Fp>\n\n\u003Ch2>The arithmetic nobody runs\u003C\u002Fh2>\n\n\u003Cp>Every process gets its own pool and the pools know nothing about each other. What the database sees is:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-text\">pool size x worker processes x instances  (+ queue workers, cron, migrations)\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Ours was 10 x 4 workers x 3 pods, which is 120, plus a background worker carrying its own pool of 10 per pod. That makes 150. The migration step in our deploy pipeline took the last slot and landed us on 151, which is the default limit exactly.\u003C\u002Fp>\n\n\u003Cp>That 10 came from a tutorial, and in the config file it looked harmless. Multiply it by the runtime and it stops being a detail. This is also why raising \u003Ccode>max_connections\u003C\u002Fcode> is the wrong first move. Doubling it buys you a week, then the same Friday comes back with twice as many connections to untangle.\u003C\u002Fp>\n\n\u003Cp>Node makes the shortage harder to see, because \u003Ccode>mysql2\u003C\u002Fcode> hides it instead of failing. In \u003Ccode>mysql2\u003C\u002Fcode> 3.x the pool defaults are \u003Ccode>connectionLimit: 10\u003C\u002Fcode>, \u003Ccode>maxIdle: 10\u003C\u002Fcode>, \u003Ccode>idleTimeout: 60000\u003C\u002Fcode>, \u003Ccode>waitForConnections: true\u003C\u002Fcode> and \u003Ccode>queueLimit: 0\u003C\u002Fcode>. That last one is an unbounded queue: when all ten connections are busy, request eleven waits quietly instead of erroring. Your p99 climbs for weeks and nothing in the logs says why.\u003C\u002Fp>\n\n\u003Cp>So I started logging the pool state every 30 seconds. The \u003Ccode>enqueue\u003C\u002Fcode> event fires exactly when a request has to wait for a connection, which makes it the honest counter:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-javascript\">import mysql from 'mysql2\u002Fpromise'\n\nconst pool = mysql.createPool({\n  host: process.env.DB_HOST,\n  connectionLimit: 10,\n  queueLimit: 50,     \u002F\u002F default is 0, which means wait forever\n  idleTimeout: 30000,\n})\n\nlet enqueuedTotal = 0\npool.on('enqueue', () => { enqueuedTotal++ })\n\nsetInterval(() => {\n  const core = pool.pool  \u002F\u002F internal, but it is where these numbers live\n  console.log({\n    open: core._allConnections.length,\n    free: core._freeConnections.length,\n    waiting: core._connectionQueue.length,\n    enqueuedTotal,\n  })\n}, 30000)\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>That log settled the argument in one deploy. Waiting sat at zero until a reporting endpoint ran, then it jumped to 40 and stayed there. Ops had been filing a vague latency complaint about that endpoint for two months and nobody could pin it on anything.\u003C\u002Fp>\n\n\u003Ch2>The fix, and the trap in the standard advice\u003C\u002Fh2>\n\n\u003Cp>We cut the pool to 4 per worker, dropped \u003Ccode>maxIdle\u003C\u002Fcode> to 2 so idle connections actually get reaped, and gave the batch worker its own database user with a hard cap:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">ALTER USER 'app'@'%' WITH MAX_USER_CONNECTIONS 120;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>That cap is the part I would keep even at a comfortable connection count. When a batch job misbehaves now, it fails against its own ceiling with error 1203 instead of taking the site down with it.\u003C\u002Fp>\n\n\u003Cp>Now the trap. Most of the posts I read about error 1040 tell you to shorten \u003Ccode>wait_timeout\u003C\u002Fcode>, which defaults to 28800 seconds and does look absurd. Set it to 60 with a pool that idles at 60 and you have built a race. The server closes an idle connection at the same moment your pool considers it free, then hands it to a request. \u003Ccode>mysql2\u003C\u002Fcode> does not ping a connection before reuse, so that query dies with \u003Ccode>MySQL server has gone away\u003C\u002Fcode> (2006) or \u003Ccode>Lost connection to MySQL server during query\u003C\u002Fcode> (2013), and the user sees a random failure they cannot reproduce. Keep \u003Ccode>wait_timeout\u003C\u002Fcode> comfortably above the pool's idle timeout instead of pulling it down into the same neighbourhood.\u003C\u002Fp>\n\n\u003Cp>One more thing worth knowing before the next incident. MySQL permits \u003Ccode>max_connections\u003C\u002Fcode> plus one connection, and that extra slot is reserved for accounts holding the \u003Ccode>CONNECTION_ADMIN\u003C\u002Fcode> privilege. That is where my 152nd connection came from. When our app user could not get in, the admin account still could, which is the only reason we got a processlist to look at. Keep that privilege on a human account and off the application account.\u003C\u002Fp>\n\n\u003Cp>The status query at the top of this post lives in \u003Ca href=\"\u002Fsnippetark\u002F\">Snippet Ark\u003C\u002Fa> now, next to the grouped processlist one, because at 9:40 on a Friday I do not want to be recalling column names. If your problem is one slow query instead of many idle connections, that is a \u003Ca href=\"\u002Fposts\u002Fmysql-slow-query-debugging-workflow\u002F\">different workflow entirely\u003C\u002Fa>. Here every query was fast. There were just far too many connections sitting open doing nothing.\u003C\u002Fp>\n","2026-10-14",1792462273107]