MySQL 8.0 to 8.4 LTS: What Broke in My Upgrade Window

We moved production off MySQL 8.0 last weekend. The upgrade cost six minutes of downtime. Getting to those six minutes cost most of a Saturday, because a MySQL 8.0 to 8.4 upgrade is not a patch bump. Under the LTS scheme, features are only added or removed at the start of a series, so everything Oracle deprecated during 8.1 through 8.3 is simply gone in 8.4.

Staging went first and behaved perfectly. That was the misleading part. Its my.cnf is nine lines long; production's has been edited by four people since 2019, and one of them left the company.

Check first, restart second

Any 8.0.x to 8.4 LTS in-place upgrade is supported, straight from whichever 8.0 patch you are on. A series cannot be skipped, so 5.7 has to land on 8.0 first. MySQL Shell ships an upgrade checker that inspects a running server against a target version.

mysqlsh -- util check-for-server-upgrade app@db-prod:3306 \
  --target-version=8.4.0 --output-format=JSON > upgrade-report.json

The report is where I should have started. One of the three surprises below was already in it, in plain text, before I touched a binary.

The config file that outlived its variables

The new binary came up and died. Not a warning, not an unknown parameter logged and ignored. A refusal to start, because several parameters that 8.0 accepted were removed in 8.4 and now count as unknown options. MySQL's server version reference lists default_authentication_plugin, binlog_transaction_dependency_tracking, avoid_temporal_upgrade, no-dd-upgrade, and the old ssl, skip-ssl and admin-ssl flags among the removals in 8.4.0.

Ours was binlog_transaction_dependency_tracking, set to WRITESET back in 2021 for better replica parallelism, and nobody had looked at it since. Validating the config against the new binary finds these without starting anything or taking downtime.

mysqld --validate-config --defaults-file=/etc/my.cnf
echo $?   # 0 clean, 1 the server will not start

The accounts nobody thought about

With the server finally up, the application could not connect. The app log said ERROR 1045 (28000): Access denied, which sent me hunting for a password problem for twenty minutes. Wrong direction. The server error log said the auth plugin was not loaded.

mysql_native_password was deprecated in 8.0.34, disabled by default in 8.4, and removed as of 9.0.0. Every account still on it is refused at connect time, and recreating the user fails too: any CREATE USER or ALTER USER naming the plugin returns ERROR 1524 (HY000): Plugin 'mysql_native_password' is not loaded.

SELECT user, host, plugin FROM mysql.user
WHERE plugin = 'mysql_native_password';

ALTER USER 'app'@'%' IDENTIFIED WITH caching_sha2_password
BY 'the-same-password-you-already-have';

Keeping the password string identical means the application config does not change, which is the whole trick. Then confirm your driver speaks caching_sha2_password: over an unencrypted TCP connection the password has to be exchanged with an RSA key pair, so an older library will fail to connect no matter what the account says.

A foreign key 8.0 accepted and 8.4 rejects

The one that actually cost us time. A teammate's migration created a foreign key against a non-unique index. InnoDB has always allowed that as an extension of standard SQL, and 8.0 was happy with it. On 8.4 the same CREATE TABLE failed with ER_FK_NO_INDEX_PARENT.

The cause is a new system variable, restrict_fk_on_non_standard_key, new in 8.4 and on by default, which blocks non-unique and partial keys as foreign key targets. You can disable it with SET @@session.restrict_fk_on_non_standard_key=OFF. Here is the part I would read twice: the variable is itself deprecated, and the manual says support for nonstandard keys should be expected to disappear in a future release. The fix that survives is adding the unique key the constraint should have pointed at.

One replication detail, because it makes this asymmetric. When the primary has the variable off and creates such a foreign key, the statement succeeds on the replica regardless of the replica's own setting.

The defaults that moved while nobody was looking

No error message on this one, and it is easy to walk past. A long list of InnoDB defaults changed between 8.0 and 8.4, so any of them you never set explicitly is now a different number. The adaptive hash index went from on to off. Change buffering went from all to none, the log buffer from 16 MiB to 64 MiB, and on Linux the flush method now prefers O_DIRECT over fsync. Buffer pool instances, page cleaners and the IO threads are computed from CPU count and buffer pool size instead of fixed at 8 and 4.

The loudest one for us was innodb_io_capacity, which moved from 200 to 10000. InnoDB's idea of what the disk can absorb changed by fifty times, and on a network-backed volume that shows up as much more aggressive background flushing.

The clean way to see what moved is the Performance Schema variables_info table, which records where each variable was last set. If the source is not your config file, you are looking at a new default. Diffing SHOW GLOBAL VARIABLES before and after the swap works too.

Order of operations

Back up first, and treat it as the rollback plan rather than a precaution. Downgrading from 8.4 to 8.3, or from one 8.4 release to an earlier one, is not supported, and restoring that backup is the only alternative the manual offers. Then run the checker, fix the config with --validate-config, migrate the accounts, and rehearse the schema changes that touch foreign keys. Only then swap the binaries.

The six minutes were fine. Every minute before them was the work, and two of the three problems were already in the checker's report. The audit queries and the validate-config step live in Snippet Ark now, so the next window starts from a checklist. If you are still on 8.0, run the checker and read the report before you schedule anything. The indexing work and the slow query loop from earlier posts still hold on 8.4.