[{"data":1,"prerenderedAt":6},["ShallowReactive",2],{"post-content-mysql-replication-lag-stale-reads":3},{"content":4,"lastModified":5},"\u003Ch2>The save button that lied\u003C\u002Fh2>\n\n\u003Cp>A ticket came in last month that I could not reproduce. Customer edits a product price in the admin panel, hits save, the page reloads, and the old price is still there. Refresh once more and it is correct. Two of us tried, both saw it intermittently, neither could trigger it on demand.\u003C\u002Fp>\n\n\u003Cp>It was plain MySQL replication lag. The write hit the source, the admin panel's read hit a replica, and for a few hundred milliseconds the replica had not caught up. Nothing was corrupt. The user was reading the past, and the app did not care.\u003C\u002Fp>\n\n\u003Cp>Almost every metric you reach for first says the database is fine, so here is my order.\u003C\u002Fp>\n\n\u003Cimg src=\"https:\u002F\u002Fimages.pexels.com\u002Fphotos\u002F4508751\u002Fpexels-photo-4508751.jpeg?auto=compress&cs=tinysrgb&w=1200\" alt=\"Aisle between rows of server racks in a data center\" loading=\"lazy\">\n\n\u003Ch2>SHOW SLAVE STATUS does not run any more\u003C\u002Fh2>\n\n\u003Cp>Out of muscle memory, I typed:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SHOW SLAVE STATUS\\G\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Syntax error. Those boxes had been upgraded to MySQL 8.4, where the master\u002Fslave vocabulary was removed, not deprecated. What still runs:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SHOW REPLICA STATUS\\G\n-- the column is Seconds_Behind_Source now\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>I had written up \u003Ca href=\"\u002Fposts\u002Fmysql-8-0-to-8-4-upgrade-gotchas\u002F\">the 8.0 to 8.4 upgrade notes\u003C\u002Fa> after hitting this during the migration. What I had not thought through is monitoring. A cron check that greps \u003Ccode>SHOW SLAVE STATUS\u003C\u002Fcode> output now gets a syntax error, the wrapper swallows stderr, and the alert channel goes quiet. Quiet looks exactly like healthy. Mine had been dead six days.\u003C\u002Fp>\n\n\u003Ch2>A number that reads zero while the replica is stale\u003C\u002Fh2>\n\n\u003Cp>With the right command, the output said \u003Ccode>Seconds_Behind_Source: 0\u003C\u002Fcode>, at the exact moment the admin panel was serving stale rows.\u003C\u002Fp>\n\n\u003Cp>That column is not how far behind the source you are. It is the gap between the applier thread and the receiver thread inside the replica. When the applier is the slow part, you get real numbers. When the receiver itself is behind, because the link is slow, the applier has nothing to be late on and the number sits at zero. The manual spells it out: it measures the distance between the SQL thread and the I\u002FO thread. A health check for one stage, wearing the name of an end to end metric.\u003C\u002Fp>\n\n\u003Ch2>Use a heartbeat instead\u003C\u002Fh2>\n\n\u003Cp>The number I trust is a row stamped on the source and read back on the replica. Percona's \u003Ccode>pt-heartbeat\u003C\u002Fcode> does this properly, but the short version works on managed instances too.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">CREATE TABLE heartbeat (\n  id TINYINT PRIMARY KEY,\n  ts DATETIME(6) NOT NULL\n) ENGINE=InnoDB;\n\nINSERT INTO heartbeat (id, ts) VALUES (1, NOW(6));\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Stamp it every second from cron on the source, then read it on the replica:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">-- source\nUPDATE heartbeat SET ts = NOW(6) WHERE id = 1;\n\n-- replica\nSELECT TIMESTAMPDIFF(SECOND, ts, NOW(6)) AS lag_seconds FROM heartbeat;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>The timestamp comes from the source and travels the same path as your data, so this keeps working when the applier stalls or replication breaks. Two conditions: both hosts need NTP, since clock drift reads as fake lag. And run the updater from system cron, not the MySQL event scheduler, which shares a fate with the server it is watching.\u003C\u002Fp>\n\n\u003Ch2>Which stage owns the lag\u003C\u002Fh2>\n\n\u003Cp>Where does the lag actually live? MySQL 8 keeps commit and apply timestamps per applier worker:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SELECT worker_id,\n  TIMESTAMPDIFF(SECOND, last_applied_transaction_original_commit_timestamp,\n                last_applied_transaction_end_apply_timestamp) AS apply_seconds,\n  TIMESTAMPDIFF(SECOND, applying_transaction_original_commit_timestamp, NOW(6)) AS txn_age\nFROM performance_schema.replication_applier_status_by_worker\nORDER BY worker_id;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Small \u003Ccode>apply_seconds\u003C\u002Fcode> with a big heartbeat lag means the receiver is the bottleneck, so look at network throughput and binlog generation rate. One worker with a huge \u003Ccode>txn_age\u003C\u002Fcode> while its siblings idle means a single transaction is hogging the applier. All of them climbing together means the replica cannot keep up with the write rate, and you have a capacity problem.\u003C\u002Fp>\n\n\u003Ch2>One large UPDATE beats every tuning knob\u003C\u002Fh2>\n\n\u003Cp>A maintenance job did it: somebody ran a single \u003Ccode>UPDATE\u003C\u002Fcode> over about 1.8 million rows to backfill a column. One transaction, nothing to parallelize, so the replica applied it as a unit while everything else queued behind it. The admin panel served stale prices for eleven minutes.\u003C\u002Fp>\n\n\u003Cp>Batch writes get chunked now, no exceptions:\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">DELIMITER $$\nCREATE PROCEDURE backfill_orders()\nBEGIN\n  DECLARE done INT DEFAULT 1;\n  WHILE done > 0 DO\n    UPDATE orders SET archived = 1\n     WHERE created_at \u003C '2025-01-01' AND archived = 0\n     ORDER BY id\n     LIMIT 5000;\n    SET done = ROW_COUNT();\n    DO SLEEP(0.05);\n  END WHILE;\nEND$$\nDELIMITER ;\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Five thousand rows per transaction plus a breather, and the replica never has more than a second of work waiting. \u003Ccode>DELIMITER\u003C\u002Fcode> is only needed in the mysql client. Same rule for migrations, since a long \u003Ccode>ALTER\u003C\u002Fcode> replays as one blocking event.\u003C\u002Fp>\n\n\u003Cp>Most tuning advice for the parallel applier is out of date. Since 8.0.27 the replica is multi threaded by default: four workers, LOGICAL_CLOCK scheduling, commit order preserved. If the source runs 8.0 with the stock \u003Ccode>binlog_transaction_dependency_tracking=COMMIT_ORDER\u003C\u002Fcode>, raising workers to sixteen buys almost nothing, because the binlog only marks transactions as independent when their commit windows overlapped. Switch the source to WRITESET first. On 8.4 the variable is gone and write set tracking is always on.\u003C\u002Fp>\n\n\u003Ch2>The part that actually fixed the stale reads\u003C\u002Fh2>\n\n\u003Cp>Chunking killed the eleven minute stalls. It did not close the two hundred millisecond window the ticket was about. That one needs an application fix, and the honest version is: after a user writes, do not read it back from a replica until you know the write landed.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">-- source, same connection, right after the commit\nSELECT @@global.gtid_executed;\n\n-- replica, before the dependent read\nSELECT WAIT_FOR_EXECUTED_GTID_SET('3e11fa47-71ca-11e1-9e33-c80aa9429562:1-184203', 1);\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Returns 0 when the set has been applied, 1 on timeout, NULL on error. Position based setups have \u003Ccode>SOURCE_POS_WAIT('mysql-bin.000418', 918273, 1)\u003C\u002Fcode>, which returns the event count waited or -1 on timeout. Both replace the MASTER_* names.\u003C\u002Fp>\n\n\u003Cp>In the admin panel we did something cruder: for thirty seconds after a write, that user's session reads from the source. Sticky routing off a cookie. Ugly, cheap, and it worked.\u003C\u002Fp>\n\n\u003Cp>One thing I would not lean on: semisynchronous replication. The ack happens when a replica receives the event, not when it applies it, so a row can be acknowledged and still invisible to a read.\u003C\u002Fp>\n\n\u003Cp>The queries from this post are in my \u003Ca href=\"\u002Fsnippetark\u002F\">Snippet Ark\u003C\u002Fa> under \"replication\". If your lag hunt turns out to be a query problem instead, the \u003Ca href=\"\u002Fposts\u002Fmysql-slow-query-debugging-workflow\u002F\">slow query workflow\u003C\u002Fa> covers that half.\u003C\u002Fp>\n","2026-11-08",1794383305989]