Showing posts with label Replication. Show all posts
Showing posts with label Replication. Show all posts

Tuesday, August 4, 2026

MySQL 8.0.17 GTID Crash Safety Improvement

I have known for some times that there is an interesting improvement in MySQL 8.0.17 regarding GTID Crash Safety, but I have not had the time nor the need to look into it before.  When writing my last post (Understanding MySQL Replication "fatal error 1236": [...]), I saw something interesting related to this, and it is now time to cover this on my blog. From my point of view, this change is very interesting, but the documentation about it should be improved (and more), so there will be bug reports.  Let's explore all this.

Monday, August 3, 2026

Understanding MySQL Replication "fatal error 1236": "Replica has more GTIDs than the source has, using the source's SERVER_UUID"

This MySQL replication error — fatal error 1236 / Replica has more GTIDs than the source has, using the source's SERVER_UUID — shows the importance of thinking before acting.  I am glad a non-DBA Colleague asked me about it, because if he had restarted replication, it would have caused a much bigger mess.

Often, we are tempted — or pushed — to just restart things as quickly as possible.  In this specific case, restarting replication is unsafe as it leads to a replication breakage in the best case, and to silent data corruption in the worse case.  It is a great example of the following two advice : 1) do not make a bigger mess, do not failover, do not reboot, and 2) we cannot just bring the database back up, we have things to do first to make this safe.  Let's explore all these.

Thursday, March 26, 2026

Binary Log Compression is Safe since MySQL 8.0.34

This is a quick one.  My attention was recently brought (thanks Simon) on a relatively recent comment (25 Nov 2025) in Bug #103672 - Binlog compression transaction payload event exceeds max allowed packet :

The underlying server bug was fixed in 8.0.34 in BUG#33588473. The server now falls back to writing the transaction without compression, if the compressed size would exceed 1 GiB.

So Binary Log Transaction Compression is safe since MySQL 8.0.34.  But this also means that it is unsafe before, as also clearly mentioned in Bug #103672 :

It is true that an 8.0.33 server (or older) will produce a corrupted binary log and that replicas may have to recover from backup. It would be better to document this clearly and to advise to not use compression on 8.0.33 and older.

However, even if updating the documentation is mentioned in the above quote, my quick check did not find any reference to the risk of using binlog compression before 8.0.34 (including in the documentation of the global variable binlog_transaction_compression).  I thought this deserved wider knowledge, hence this post.

Wednesday, June 4, 2025

Interesting Troubleshooting of a MySQL Crash : filling then freeing the disk

I recently troubleshoot an interesting MySQL crash, and I think it is worth sharing (with the related bugs).  MySQL crashed when the disk was full, you can see the free disk graph below.  The Y-axis is in the tens of GiB scale and the X-axis is in the hour scale.  Can you guess what happened ?

Monday, May 26, 2025

Interesting Binary Logging Optimization in MariaDB

As I wrote in a LinkedIn post, I am working on a blog post related to binary logging of big transactions.  I thought I would split this post in two, so here is the first part where I cover the recent binary logging optimization in MariaDB and its unspoken advantage.

Monday, May 22, 2023

Avoiding a STOP SLAVE Crash with MTR in Percona Server older than 5.7.37-40

I am finalizing my Percona Live talk MySQL and Vitess (and Kubernetes) at HubSpot.  In this talk, I mentioned that I like that Percona is providing better MySQL with Percona Server.  This comes with a little inconvenience though: with improvements, sometimes comes regression.  This post is about such regression and a workaround I implemented some time ago (I should have shared it earlier).  This regression is fixed in Percona Server 5.7.37-40, so this is mostly interesting for historical purposes, or if you are running a Percona Server version older than that.

Tuesday, May 31, 2022

Triggering Replication Lag for Testing a Script

I am currently working on a script to auto-enable parallel replication / multi-threaded replication (MTR) when there is replication lag.  For testing this script, I need to trigger replication lag that would disappear after enabling MTR.  I came-up with a simple solution for that, and I thought it could be useful to more people, so I am writing this blog post about it.  Read-on for the details.

Wednesday, January 19, 2022

Crashing MySQL with Malicious Intent and a lot of Determination

A year ago, I blogged about An Unprivileged User can crash your MySQL Server.  At the time, I explained how to protect yourself against this problem.  A few weeks ago, I revisited this vulnerability in a follow-up post in which I explained the fix, claimed that the MySQL 5.7 default configuration for Group Replication is still problematic, and explained a tuning to avoid the vulnerability.  In this last post in the series, I explain how to exploit this vulnerability to crash older version of MySQL (or untuned Group Replication in 5.7), but this needs a lot of determination.

Tuesday, November 30, 2021

Follow-up on an Unprivileged User can Crash your MySQL Server

A year ago, I blogged about An Unprivileged User can Crash your MySQL Server.  At the time, I presented how to protect yourself against this problem without explaining how to generate a crash.  In this post, I am revisiting this vulnerability, not giving the exploit yet, but presenting the fix.  Also, because the default configuration of Group Replication in 5.7 is still vulnerable (it is not in 8.0), I show the adjustment to make to avoid problems.

Monday, January 27, 2020

A Legacy Behavior of MySQL Corrupting Restored Backups (replicate-same-server-id = OFF)

In my previous post (Puzzled by MySQL Replication), I describe a weird, but completely documented, behavior of replication that had me scratching my head for hours because it was causing data corruption.  I did not give too many details then as I also wanted allowing you to scratch your head if you wished.  In this post, I describe this behavior in more details.

Thursday, January 9, 2020

Puzzled by MySQL Replication (War Story)

Recently, I was puzzled by MySQL replication !  Some weird, but completely documented, behavior of replication had me scratching my head for hours.  I am sharing this war story so you can avoid losing time like me (and also maybe avoid corrupting your data when restoring a backup).  The exact justification will come in a follow-up post, so you can also scratch your head trying to understand what I faced.  So let's dive-in.

Tuesday, July 16, 2019

MySQL Master Replication Crash Safety Part #5a: making things faster without reducing durability - using better hardware

This is a follow-up post in the MySQL Master Replication Crash Safety series.  In the previous posts, we explored the consequences of reducing durability on masters (different data inconsistencies after an OS crash depending on replication type) and the performance boost associated with this configuration (benchmark results done on Google Cloud Platform / GCP).  The consequences are summarised in the introduction of Part #4, and the tests are the subject of this last post.  Also in this last post, I mentioned that my results for high durability are limited by the sync latencies of GCP persistent disks.  As I found a system with better latencies, I am able to present new results.  And this system is a vm in Amazon Web Services (AWS) with local SSD.

Tuesday, July 9, 2019

MySQL Master Replication Crash Safety Part #4: benchmarks of high and low durability

This is a follow-up post in the MySQL Master Replication Crash Safety series.  In the three previous posts, we explored the consequence of reducing durability on masters (including setting sync_binlog to a value different from 1).  But so far, I only quickly presented why a DBA would run MySQL with such configuration.  In this post, I present actual benchmark results.  I also present a fundamental difference between on-premise servers and cloud virtual machines as my tests are done in Google Cloud Platform (GCP).  But before going further, let's summarise the previous posts.

Tuesday, February 26, 2019

MySQL Master High Availability and Failover: more thoughts

Some months ago, Shlomi Noach published a series about Service Discovery.  In his posts, Shlomi describes many ways for an application to find the master.  He also gives detail on how these solutions cope with failing-over to a slave, including their integration with Orchestrator.

This is a great series, and I recommend its reading for everybody implementing master failover, with or without Orchestrator, even if you are not fully automating the process yet.  Taking a step back, I realized that service discovery is only one of the five parts of a full MySQL Master Failover Strategy; this post is about these five parts.  In some follow-up posts, I might analyze some deployments using the framework presented in this post.

Tuesday, February 12, 2019

MySQL Master Replication Crash Safety Part #3: GTID

This is a follow-up post in the MySQL Master Replication Crash Safety series.  In the two previous posts, we explored the consequence of reducing durability on masters (including setting sync_binlog to a value different than 1) when slaves are using legacy file+position replication.  In this post, we cover GTID replication.  This introduces a new inconsistency scenario with a potential replication breakage that depends on transaction execution on the master and timing on the slave.  Before discussing this violation of ACID, we start with some reminders about the last posts and with some explanations about GTIDs.

Thursday, November 15, 2018

MySQL Master Replication Crash Safety Part #2: lagging slaves

This is Part #2 of the MySQL Master Replication Crash Safety series.  In the previous post, we explored the consequence of reducing durability on masters with slaves using legacy file+position replication.  The consequences are data inconsistencies with a clear warning sign: the slaves stop replicating and report an error.  In this post, we extend our understanding of the impact of running a master with sync_binlog != 1 by considering lagging slaves that are using file+position replication.  This introduces a new inconsistency scenario without replication breakage, which is trickier to detect.  But let's start with some reminders.

Tuesday, October 30, 2018

On the consequences of sync_binlog != 1 (part #1)

A well-known performance booster in MySQL is to set sync_binlog to 0.  However, this configuration alone comes with serious consequences on consistency and on durability (the C and D of ACID); I explore those in this series.  In this post, I give some background on the sync_binlog parameter and I explain part of the problem with setting it to 0 (or to a value different from 1).  The other problems — including the behaviour with GTIDs (I am limiting the current scope to legacy file + position replication) — and some solutions are the subject of the upcoming posts.

Tuesday, September 11, 2018

Unforeseen use case of my GTID work: replicating from AWS Aurora to Google CloudSQL

A colleague brought an article to my attention.  I did not see it on Planet MySQL where I get most of the MySQL news (or it did not catch my eye there).  As it is interesting replication stuff, I think it is important to bring it to the attention of the MySQL Community, so I am writing this short post.

The surprising part for me is that it uses my 4-year-old work for online migration to GTID with MySQL 5.6.  This is a completely unforeseen use case of my work as I never thought that my hack would be useful after Oracle include an online migration path to GTID in MySQL 5.7 (Percona did something similar for MySQL 5.6).

Thursday, April 19, 2018

Some bugs and spring pilgrimage to Percona Live Santa Clara 2018

I am now in an airport, waiting for one of the four flights that will bring me to Percona Live Santa Clara 2018.  This is a good time to write some details about my tutorial on parallel replication.  But before talking about Percona Live, I will share thoughts on MySQL/MariaDB bugs that caught my attention in the last weeks/months (Valeriy: you clearly have an influence on me).