[{"data":1,"prerenderedAt":6},["ShallowReactive",2],{"post-content-mysql-8-0-to-8-4-upgrade-gotchas":3},{"content":4,"lastModified":5},"\u003Cp>We moved production off MySQL 8.0 last weekend. The upgrade cost six minutes of downtime. Getting to those six minutes cost most of a Saturday, because a MySQL 8.0 to 8.4 upgrade is not a patch bump. Under the LTS scheme, features are only added or removed at the start of a series, so everything Oracle deprecated during 8.1 through 8.3 is simply gone in 8.4.\u003C\u002Fp>\n\n\u003Cp>Staging went first and behaved perfectly. That was the misleading part. Its my.cnf is nine lines long; production's has been edited by four people since 2019, and one of them left the company.\u003C\u002Fp>\n\n\u003Cfigure>\n  \u003Cimg src=\"https:\u002F\u002Fimages.unsplash.com\u002Fphoto-1520869562399-e772f042f422?auto=format&fit=crop&w=1200&q=80\" alt=\"The rear of a live rack unit with orange and black patch cables plugged into a switch, port LEDs lit\" loading=\"lazy\" \u002F>\n\u003C\u002Ffigure>\n\n\u003Ch2>Check first, restart second\u003C\u002Fh2>\n\n\u003Cp>Any 8.0.x to 8.4 LTS in-place upgrade is supported, straight from whichever 8.0 patch you are on. A series cannot be skipped, so 5.7 has to land on 8.0 first. MySQL Shell ships an upgrade checker that inspects a running server against a target version.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">mysqlsh -- util check-for-server-upgrade app@db-prod:3306 \\\n  --target-version=8.4.0 --output-format=JSON > upgrade-report.json\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>The report is where I should have started. One of the three surprises below was already in it, in plain text, before I touched a binary.\u003C\u002Fp>\n\n\u003Ch2>The config file that outlived its variables\u003C\u002Fh2>\n\n\u003Cp>The new binary came up and died. Not a warning, not an unknown parameter logged and ignored. A refusal to start, because several parameters that 8.0 accepted were removed in 8.4 and now count as unknown options. MySQL's server version reference lists \u003Ccode>default_authentication_plugin\u003C\u002Fcode>, \u003Ccode>binlog_transaction_dependency_tracking\u003C\u002Fcode>, \u003Ccode>avoid_temporal_upgrade\u003C\u002Fcode>, \u003Ccode>no-dd-upgrade\u003C\u002Fcode>, and the old \u003Ccode>ssl\u003C\u002Fcode>, \u003Ccode>skip-ssl\u003C\u002Fcode> and \u003Ccode>admin-ssl\u003C\u002Fcode> flags among the removals in 8.4.0.\u003C\u002Fp>\n\n\u003Cp>Ours was \u003Ccode>binlog_transaction_dependency_tracking\u003C\u002Fcode>, set to \u003Ccode>WRITESET\u003C\u002Fcode> back in 2021 for better replica parallelism, and nobody had looked at it since. Validating the config against the new binary finds these without starting anything or taking downtime.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-bash\">mysqld --validate-config --defaults-file=\u002Fetc\u002Fmy.cnf\necho $?   # 0 clean, 1 the server will not start\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Ch2>The accounts nobody thought about\u003C\u002Fh2>\n\n\u003Cp>With the server finally up, the application could not connect. The app log said \u003Ccode>ERROR 1045 (28000): Access denied\u003C\u002Fcode>, which sent me hunting for a password problem for twenty minutes. Wrong direction. The server error log said the auth plugin was not loaded.\u003C\u002Fp>\n\n\u003Cp>\u003Ccode>mysql_native_password\u003C\u002Fcode> was deprecated in 8.0.34, disabled by default in 8.4, and removed as of 9.0.0. Every account still on it is refused at connect time, and recreating the user fails too: any \u003Ccode>CREATE USER\u003C\u002Fcode> or \u003Ccode>ALTER USER\u003C\u002Fcode> naming the plugin returns \u003Ccode>ERROR 1524 (HY000): Plugin 'mysql_native_password' is not loaded\u003C\u002Fcode>.\u003C\u002Fp>\n\n\u003Cpre>\u003Ccode class=\"language-sql\">SELECT user, host, plugin FROM mysql.user\nWHERE plugin = 'mysql_native_password';\n\nALTER USER 'app'@'%' IDENTIFIED WITH caching_sha2_password\nBY 'the-same-password-you-already-have';\u003C\u002Fcode>\u003C\u002Fpre>\n\n\u003Cp>Keeping the password string identical means the application config does not change, which is the whole trick. Then confirm your driver speaks \u003Ccode>caching_sha2_password\u003C\u002Fcode>: over an unencrypted TCP connection the password has to be exchanged with an RSA key pair, so an older library will fail to connect no matter what the account says.\u003C\u002Fp>\n\n\u003Ch2>A foreign key 8.0 accepted and 8.4 rejects\u003C\u002Fh2>\n\n\u003Cp>The one that actually cost us time. A teammate's migration created a foreign key against a non-unique index. InnoDB has always allowed that as an extension of standard SQL, and 8.0 was happy with it. On 8.4 the same \u003Ccode>CREATE TABLE\u003C\u002Fcode> failed with \u003Ccode>ER_FK_NO_INDEX_PARENT\u003C\u002Fcode>.\u003C\u002Fp>\n\n\u003Cp>The cause is a new system variable, \u003Ccode>restrict_fk_on_non_standard_key\u003C\u002Fcode>, new in 8.4 and on by default, which blocks non-unique and partial keys as foreign key targets. You can disable it with \u003Ccode>SET @@session.restrict_fk_on_non_standard_key=OFF\u003C\u002Fcode>. Here is the part I would read twice: the variable is itself deprecated, and the manual says support for nonstandard keys should be expected to disappear in a future release. The fix that survives is adding the unique key the constraint should have pointed at.\u003C\u002Fp>\n\n\u003Cp>One replication detail, because it makes this asymmetric. When the primary has the variable off and creates such a foreign key, the statement succeeds on the replica regardless of the replica's own setting.\u003C\u002Fp>\n\n\u003Ch2>The defaults that moved while nobody was looking\u003C\u002Fh2>\n\n\u003Cp>No error message on this one, and it is easy to walk past. A long list of InnoDB defaults changed between 8.0 and 8.4, so any of them you never set explicitly is now a different number. The adaptive hash index went from on to off. Change buffering went from all to none, the log buffer from 16 MiB to 64 MiB, and on Linux the flush method now prefers \u003Ccode>O_DIRECT\u003C\u002Fcode> over \u003Ccode>fsync\u003C\u002Fcode>. Buffer pool instances, page cleaners and the IO threads are computed from CPU count and buffer pool size instead of fixed at 8 and 4.\u003C\u002Fp>\n\n\u003Cp>The loudest one for us was \u003Ccode>innodb_io_capacity\u003C\u002Fcode>, which moved from 200 to 10000. InnoDB's idea of what the disk can absorb changed by fifty times, and on a network-backed volume that shows up as much more aggressive background flushing.\u003C\u002Fp>\n\n\u003Cp>The clean way to see what moved is the Performance Schema \u003Ccode>variables_info\u003C\u002Fcode> table, which records where each variable was last set. If the source is not your config file, you are looking at a new default. Diffing \u003Ccode>SHOW GLOBAL VARIABLES\u003C\u002Fcode> before and after the swap works too.\u003C\u002Fp>\n\n\u003Ch2>Order of operations\u003C\u002Fh2>\n\n\u003Cp>Back up first, and treat it as the rollback plan rather than a precaution. Downgrading from 8.4 to 8.3, or from one 8.4 release to an earlier one, is not supported, and restoring that backup is the only alternative the manual offers. Then run the checker, fix the config with \u003Ccode>--validate-config\u003C\u002Fcode>, migrate the accounts, and rehearse the schema changes that touch foreign keys. Only then swap the binaries.\u003C\u002Fp>\n\n\u003Cp>The six minutes were fine. Every minute before them was the work, and two of the three problems were already in the checker's report. The audit queries and the validate-config step live in \u003Ca href=\"\u002Fsnippetark\u002F\">Snippet Ark\u003C\u002Fa> now, so the next window starts from a checklist. If you are still on 8.0, run the checker and read the report before you schedule anything. The \u003Ca href=\"\u002Fposts\u002Fhow-i-cut-database-query-time-mysql-indexing\u002F\">indexing work\u003C\u002Fa> and the \u003Ca href=\"\u002Fposts\u002Fmysql-slow-query-debugging-workflow\u002F\">slow query loop\u003C\u002Fa> from earlier posts still hold on 8.4.\u003C\u002Fp>\n","2026-10-02",1791168103940]