[{"data":1,"prerenderedAt":6},["ShallowReactive",2],{"post-content-mysql-deadlock-found-how-to-read-the-log":3},{"content":4,"lastModified":5},"\u003Cp>Six failed checkouts in four minutes, all at 2am, all with the same MySQL error in the log: \u003Ccode>ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction\u003C\u002Fcode>. The dashboards were green, and a few customers had been told their order could not be placed.\u003C\u002Fp>\n\n\u003Cp>My instinct was the wrong one, and it is the one almost every team has: restart the API, watch the errors stop, go back to bed. They stop because the two transactions that collided are gone. They come back a night later in a different shape, because the lock order has not changed.\u003C\u002Fp>\n\n\u003Cimg src=\"https:\u002F\u002Fimages.unsplash.com\u002Fphoto-1550439062-609e1531270e?w=1200&q=80\" alt=\"A developer at a three-monitor desk setup at night, seen from above\" loading=\"lazy\" \u002F>\n\n\u003Ch2>Deadlock and lock wait timeout are not the same ticket\u003C\u002Fh2>\n\n\u003Cp>Two InnoDB errors get filed together and they need opposite fixes.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-text\">ERROR 1213 (40001): Deadlock found when trying to get lock; try restarting transaction\nERROR 1205 (HY000): Lock wait timeout exceeded; try restarting transaction\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>1205 is a patience problem. One transaction held a lock past \u003Ccode>innodb_lock_wait_timeout\u003C\u002Fcode>, which defaults to 50 seconds, and the waiter gave up. That is usually a forgotten \u003Ccode>COMMIT\u003C\u002Fcode>, or an HTTP call to a payment provider inside an open transaction.\u003C\u002Fp>\n\n\u003Cp>1213 is a shape problem. Two transactions each hold something the other wants, InnoDB spots the cycle and rolls one back. The MySQL manual says the victim is chosen by size, measured in rows inserted, updated or deleted, so the cheaper transaction dies. You cannot outwait it.\u003C\u002Fp>\n\n\u003Ch2>Turn on the log that keeps history\u003C\u002Fh2>\n\n\u003Cp>\u003Ccode>SHOW ENGINE INNODB STATUS\u003C\u002Fcode> only shows the most recent deadlock. With retries in the app and someone reading the output next morning, that section has already moved on.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SET GLOBAL innodb_print_all_deadlocks = ON;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>No restart needed. Every deadlock from then on goes to the error log with the full lock dump. Check \u003Ccode>SELECT @@log_error\u003C\u002Fcode> first, since inside a container that often resolves to stderr and the dump lands in \u003Ccode>docker logs\u003C\u002Fcode> instead of the file you were grepping. Turn it off again when you are done, because a deadlock storm fills that disk fast.\u003C\u002Fp>\n\n\u003Cp>Each dump is sixty-odd lines of lock detail wrapped around the four lines you need, and they keep landing in an error log that is already large for other reasons. At that size I stop grepping and read them in \u003Ca href=\"\u002Fstreamlog\u002F\">Streamlog\u003C\u002Fa> instead, filtered down to the \u003Ccode>index\u003C\u002Fcode> lines. Two or three in, the repeated index pair usually stares at you.\u003C\u002Fp>\n\n\u003Ch2>What the dump is actually telling you\u003C\u002Fh2>\n\n\u003Cp>The block reads like a small crime report: for each transaction, the statement it ran, the locks it holds, the lock it waited for, then a line naming the one InnoDB killed. Those parts are always there, twice, plus the verdict.\u003C\u002Fp>\n\n\u003Cp>Mine, stripped to the lines that mattered:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-text\">*** (1) TRANSACTION:\nUPDATE orders SET status = 'paid' WHERE user_id = 88214\n*** (1) HOLDS THE LOCK(S):\nRECORD LOCKS ... index idx_user_id of table `shop`.`orders` lock_mode X\n*** (1) WAITING FOR THIS LOCK TO BE GRANTED:\nRECORD LOCKS ... index idx_status of table `shop`.`orders` lock_mode X locks gap before rec waiting\n\n*** (2) TRANSACTION:\nUPDATE orders SET status = 'expired' WHERE status = 'pending' LIMIT 200\n*** (2) HOLDS THE LOCK(S):\nRECORD LOCKS ... index idx_status of table `shop`.`orders` lock_mode X\n*** (2) WAITING FOR THIS LOCK TO BE GRANTED:\nRECORD LOCKS ... index idx_user_id of table `shop`.`orders` lock_mode X waiting\n\n*** WE ROLL BACK TRANSACTION (1)\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>The two index names are the whole story. Checkout walked in through \u003Ccode>idx_user_id\u003C\u002Fcode>. The expiry sweep walked in through \u003Ccode>idx_status\u003C\u002Fcode> and, not being a unique lookup, took a gap lock across the range it scanned under the default REPEATABLE READ. Each then needed what the other held.\u003C\u002Fp>\n\n\u003Cp>Read the \u003Ccode>lock_mode\u003C\u002Fcode> line too. \u003Ccode>locks rec but not gap\u003C\u002Fcode> means a plain row lock, so you have an ordering problem. \u003Ccode>locks gap before rec\u003C\u002Fcode> means a range scan, so a query locking more than it reads.\u003C\u002Fp>\n\n\u003Ch2>The fix, and the version that made it worse\u003C\u002Fh2>\n\n\u003Cp>I stopped the sweep from holding a range lock while it waited on rows checkout already had. It now collects ids first and updates by primary key, in order, putting both writers on the same path through the clustered index.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SELECT id FROM orders WHERE status = 'pending' ORDER BY id LIMIT 200;\n\nUPDATE orders SET status = 'expired' WHERE id IN (88190, 88214, 88233);\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>The attempt before that made things worse. I added a composite index on \u003Ccode>(status, user_id)\u003C\u002Fcode> so the sweep would stop scanning. Deadlock-free for a night, then back with the names swapped, because the sweep was walking through the new index. An index is part of your lock order, not just a speed knob.\u003C\u002Fp>\n\n\u003Cp>If the dump shows two \u003Ccode>INSERT\u003C\u002Fcode> statements colliding on one unique key, the cause is different. A duplicate-key error takes a shared lock on the duplicate record, and two sessions holding shared locks on one key cannot both upgrade to a write. Switching those writes to \u003Ccode>INSERT ... ON DUPLICATE KEY UPDATE\u003C\u002Fcode> sidesteps it, because that duplicate-key error takes an exclusive lock instead.\u003C\u002Fp>\n\n\u003Ch2>Retry the transaction, not the statement\u003C\u002Fh2>\n\n\u003Cp>Our retry helper was re-running the failed \u003Ccode>UPDATE\u003C\u002Fcode> on its own. MySQL does not roll back a statement when it picks a victim, it rolls back the transaction, and the reads inside it are gone. That attempt ran against a different world, which is how a deadlock became an order in the wrong state.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-php\">for ($attempt = 0; $attempt < 3; $attempt++) {\n    try {\n        $pdo->beginTransaction();\n        \u002F\u002F read, decide, write: all of it reruns\n        $pdo->commit();\n        break;\n    } catch (PDOException $e) {\n        if ($pdo->inTransaction()) {\n            $pdo->rollBack();\n        }\n        if ($e->errorInfo[1] !== 1213 || $attempt === 2) {\n            throw $e;\n        }\n        usleep(random_int(50, 200) * 1000);\n    }\n}\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Because InnoDB kills the smaller transaction, the retry usually lands: the winner has committed by the time you come back.\u003C\u002Fp>\n\n\u003Ch2>Watch the waits before they turn into deadlocks\u003C\u002Fh2>\n\n\u003Cp>For the 1205 half of the problem, one view hands you the answer.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SELECT waiting_pid, waiting_query, blocking_pid, blocking_query,\n       wait_age_secs, sql_kill_blocking_connection\nFROM sys.innodb_lock_waits\\G\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>If \u003Ccode>blocking_query\u003C\u002Fcode> is NULL while \u003Ccode>blocking_pid\u003C\u002Fcode> is still live, nobody is running a slow query. Someone is holding an open transaction and has gone idle: the same forgotten \u003Ccode>COMMIT\u003C\u002Fcode> wearing a different hat.\u003C\u002Fp>\n\n\u003Cp>One last thing, since this is the advice passed around most confidently and least accurately. Dropping to READ COMMITTED removes the gap locks, so range-scan cycles like mine stop happening. It does nothing for two transactions locking the same records in opposite orders, and the MySQL manual says the possibility of deadlocks is not affected by the isolation level.\u003C\u002Fp>\n\n\u003Cp>The retry wrapper stays forever now. A few deadlocks a week is noise, ten a second is a design problem, and the dump tells you which one you have. I keep the trimmed dump layout and the retry loop in \u003Ca href=\"\u002Fsnippetark\u002F\">Snippet Ark\u003C\u002Fa> for the next 2am version of me. For the speed problem instead of the contention one: \u003Ca href=\"\u002Fposts\u002Fmysql-slow-query-debugging-workflow\u002F\">the slow query workflow\u003C\u002Fa> and the \u003Ca href=\"\u002Fposts\u002Fhow-i-cut-database-query-time-mysql-indexing\u002F\">index design behind it\u003C\u002Fa>.\u003C\u002Fp>\n","2026-09-21",1789991458756]