You read this right, symbolic links (symlinks) are unsafe in MySQL since at least 8.0.39. As always, it is a little more complicated than that, but if you are using symbolic links and in certain conditions, you risk a crash. I think it is important to raise awareness on this, hence this post.
My attention was brought to this via the now private Bug #120156: MySQL 8.0.39/8.0.42/8.4.8 crashed due to Assertion failure error when running TRUNCATE. This bug was submitted on the 25th of March 2026 by IVAN HO, and on the same day, Yakir Gibraltar commented that this is probably a duplicate of the still public, but for how long, Bug #119554: CRASH SERVER after Truncate. Yakir also contributed a patch on the 27th of March, but when the bug was verified on March 31st, it was made private. Yakir mentions his patch on LinkedIn (direct link to the fix on his GitHub).
I will come back to the subject of this bug being private further down below. In Annex #1, I show how to reproduce the bug (with a symbolic link on a schema / database directory) and I give two workarounds (running OPTIMIZE TABLE on the affected table before TRUNCATE, or CREATE TABLE LIKE followed by RENAME and DROP which simulates a TRUNCATE — both of these workarounds being mentioned by IVAN in his Bug #120156).
Upgrading with symlinks from 8.0.37- to 8.0.39+
(including 8.4) also risks a crash
Interestingly, there is no crash in MySQL 9.7.0-er2, but I do not see any reference to this fix in the Release Notes. Hopefully, the fix will be back ported in 8.0 and 8.4.
In Annex #2, we see that a symbolic link on the data directory does not trigger a crash. And this is partially why I wrote in the introduction that "it is a little more complicated than that". So far, the only crashing condition I found is when using a symbolic link for a schema directory, but there might be others (I have some other test ideas, but I lack time to be exhaustive in my testing).
don't use symlinks
As I wrote above, Yakir Gibraltar commented in Bug #120156 (private) that it is probably a duplicated of Bug #119554, so let's follow this path. Bug #119554 was submitted by Alexis Amoudru on the 15th of December 2025, and on December 18, MySQL Verification Team mentioned the error looking the same as in Bug #117660: MySQL 8.4.2: InnoDB Assertion Failure..., which was submitted on the 11th of March 2025 and was not repeatable. But the next day, and without visible input from Alexis, MySQL Verification Team comes back with the information that this is a duplicate of an "internal" verified bug (Bug #38169053) with a link to private Bug #117210. By looking at the submission date of Bug #117211 and Bug #117209, we can infer that Bug #117210 was submitted on the 15th of January 2025, but we cannot exactly know when this crash was attributed to symbolic links, even though the absence of input in Bug #117660 makes me think it was known before December 18.
Continuing the exploration of Bug #119554: CRASH SERVER after Truncate, we can read below comment from MySQL Verification Team on December 24:
don't use symlinks
This is not super helpful. Moreover, the usage of symbolic links allowing me to reproduce the crash is exactly what is described in the MySQL Reference Manual page about Using Symbolic Links for Databases on Unix. If we look at the parent page (Using Symbolic Links), we can read...
For InnoDB tables, use the DATA DIRECTORY clause of
the CREATE TABLE statement instead of symbolic links
...and the above text comes with a link to Creating Tables Externally. So we can understand that something better than symbolic links exists, but that this only applies to table creation (there is nothing so far about moving an existing table). If we continue reading the documentation, we find below (in Creating Tables Externally):
Using the DATA DIRECTORY clause [...]
is an alternative to using symbolic links,
which InnoDB does not support
Wow, it is the first time I see this, let me quote this again to make it unambiguously clear:
InnoDB does not support symlinks !
Symbolic Link Alternatives
This should have been clearly stated in the page about Using Symbolic Links. I am wondering how many DBAs, after having disk space problems on Unix and reaching a page in the MySQL Reference Manual describing a solution, are running InnoDB on unsupported symbolic links. And doing some archeology by looking at archived version of the Manual, InnoDB not supporting symbolic links is mentioned at least since 5.5, which is new to me ! And for that and more, I opened Bug #120268: Clarify Symbolic Link Usage and Alternative for InnoDB. So far, we know how to create a table externally, but we do not know how to move an existing table (it was possible with symbolic links, but needed a MySQL restart), and fully copying the data (INSERT SELECT, pt-osc or gh-ost) would be tedious on a large table. So let's continue reading the documentation.
We were at Creating Tables Externally, which mentions the InnoDB Parameter innodb_directories, in which we can find a link to Moving Tablespace Files While the Server is Offline (there is also a link to Tablespace Discovery During Crash Recovery, which is related to subjects I explored in the past: InnoDB Tablespace Duplicate Check and InnoDB Tablespace Validation). So there it is, with Moving Tablespace Files While the Server is Offline, we have feature parity with Using Symbolic Links. But before covering the topic of our bug being made private, I would like to bring your attention to below in Creating Tables Externally, as this feature might be complicated to use in some replication environment.
The DATA DIRECTORY clause is not supported in a replication environment where the source and replica reside on the same host.
Private Bug and Rant
I fail to see why this bug was made private. A crashing condition is not IMHO enough to justify hiding important information from MySQL Users. For justifying that, the crashing condition should be exploitable by a malicious attacker, and in this case, I do not think it is because it is quite hard for an attacker to "move" a schema to a symbolic link.
Moreover, this bug contains important information, like workarounds, a clear warning to people thinking about upgrading past 8.0.37, and a clear warning to not use symbolic links on schema directories, all of which are the reasons I am writing this post.
Before writing this post, I asked the Oracle MySQL Community Team in a private channel if there was a reason I should consider for keeping this bug private, and I got the corporate prose bellow.
We are working diligently on a plan to increase transparency and communications related to bugs in general and specifically on security related bugs to convey the status and progress. This particular example relates to a crash/security bug and is being worked according to current policy (not publicly communicated/documented). This is something we plan to evolve as part of our next public discussion and have plans in place to improve not only the process and policy but also the communication of that process and policy, with changes being visible as soon as the product release of CE later this month.
I am looking forward to the improved transparency and communications, because right now, Oracle/MySQL does not shine here.
Annex #1 : Reproduction and Workarounds
# Create a sandbox for our tests. # (below is done for MySQL 8.4.8, but 8.0.45, 8.0.39 and 9.6.0 have similar results crashes) # (there is no crash with 8.0.37, and remember that 8.0.38 had the thousand tables crash) # (also, there is no crash in 9.7.0-er2, hopefully the fix will be in all other versions) # (the pv command is a trick to time command execution) { v=mysql_8.4.8; d=${v//./_} dbdeployer deploy single $v | pv -tN dbdepl. > /dev/null cd ~/sandboxes/msb_$d } dbdepl.: 0:00:41 # Create two schemas containing tables for our tests. # (one table in the first schema and two in the second, the reason will become obvious) ./use <<< " CREATE DATABASE test_jfg; CREATE DATABASE test_jfg2; CREATE TABLE test_jfg.t1(id INT PRIMARY KEY); CREATE TABLE test_jfg2.t1 LIKE test_jfg.t1; CREATE TABLE test_jfg2.t2 LIKE test_jfg.t1" # Let's move test_jfg2 to a symlink on another filesystem. # (there will not be a crash if the symlink is on the same filesystem) # (my directory ~/sandboxes is usually a symlink to /mnt/jgagne_sandboxes, ...) # (..., and for these tests, I am using ~/sandboxes as a "real" directory) { ./stop > /dev/null mkdir /mnt/jgagne_sandboxes/$d mv data/test_jfg2 /mnt/jgagne_sandboxes/$d ln -s /mnt/jgagne_sandboxes/$d/test_jfg2 data/ df data/test_jfg* ./start > /dev/null } Filesystem 1K-blocks Used Available Use% Mounted on /dev/nvme0n1p1 8025124 3015296 4580620 40% / /dev/nvme1n1 9371648 98552 9273096 2% /mnt/jgagne_sandboxes # Let's truncate one table in each schema, the 2nd crashes. # (no crash in 8.0.37 or before, nor in 9.7.0-er2). { ./use -N test_jfg <<< "TRUNCATE TABLE t1; SELECT 'ok'" ./use -N test_jfg2 <<< "TRUNCATE TABLE t1; SELECT 'ok'" } ok ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query # Checking the exact error. grep -e ERROR data/msandbox.err 2026-04-14T15:55:38.830657Z 9 [ERROR] [MY-013183] [InnoDB] Assertion failure: ha_innodb.cc:11705:strlen(m_remote_path) != 0 thread 139678848059072 # The restart does not remove the crashing condition. ./use -N test_jfg2 <<< "TRUNCATE TABLE t1; SELECT 'ok'" ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query # OPTIMIZE TABLE avoids future crashes... # (also showing the 2nd table crashes for next test) { ./use -N test_jfg2 <<< "OPTIMIZE TABLE t1; TRUNCATE TABLE t1; SELECT 'ok'" ./use -N test_jfg2 <<< "TRUNCATE TABLE t2; SELECT 'ok'" } test_jfg2.t1 optimize note Table does not support optimize, doing recreate + analyze instead test_jfg2.t1 optimize status OK ok ERROR 2013 (HY000) at line 1: Lost connection to MySQL server during query # ...so does truncating in a smarter way to avoid the crash. { ./use -N test_jfg2 <<< " CREATE TABLE tmp LIKE t2; RENAME TABLE t2 TO tmp2, tmp TO t2; DROP TABLE tmp2; SELECT 'ok'" } ok
Annex #2 : No Crash Linking the Data Directory
# Create a sandbox for our tests. { v=mysql_8.4.8; d=${v//./_} dbdeployer deploy single $v | pv -tN dbdepl. > /dev/null cd ~/sandboxes/msb_$d } dbdepl.: 0:00:41 # Create two schemas containing one a table for our tests. ./use <<< " CREATE DATABASE test_jfg; CREATE DATABASE test_jfg2; CREATE TABLE test_jfg.t1(id INT PRIMARY KEY); CREATE TABLE test_jfg2.t1 LIKE test_jfg.t1" # Let's move the data directory to a symlink on another filesystem. { ./stop > /dev/null mkdir /mnt/jgagne_sandboxes/$d mv data /mnt/jgagne_sandboxes/$d ln -s /mnt/jgagne_sandboxes/$d/data df . data ./start > /dev/null } Filesystem 1K-blocks Used Available Use% Mounted on /dev/nvme0n1p1 8025124 3619756 3976160 48% / /dev/nvme1n1 9371648 305844 9065804 4% /mnt/jgagne_sandboxes # Let's truncate both tables. { ./use -N test_jfg <<< "TRUNCATE TABLE t1; SELECT 'ok'" ./use -N test_jfg2 <<< "TRUNCATE TABLE t1; SELECT 'ok'" } ok ok
No comments:
Post a Comment