MySQL Master High Availability and Failover: more thoughts

Some months ago, Shlomi Noach published a series about Service Discovery.  In his posts, Shlomi describes many ways for an application to find the master.  He also gives detail on how these solutions cope with failing-over to a slave, including their integration with Orchestrator.

This is a great series, and I recommend its reading for everybody implementing master failover, with or without Orchestrator, even if you are not fully automating the process yet.  Taking a step back, I realized that service discovery is only one of the five parts of a full MySQL Master Failover Strategy; this post is about these five parts.  In some follow-up posts, I might analyze some deployments using the framework presented in this post.

MySQL Master Replication Crash Safety Part #3: GTID

This is a follow-up post in the MySQL Master Replication Crash Safety series.  In the two previous posts, we explored the consequence of reducing durability on masters (including setting sync_binlog to a value different than 1) when slaves are using legacy file+position replication.  In this post, we cover GTID replication.  This introduces a new inconsistency scenario with a potential replication breakage that depends on transaction execution on the master and timing on the slave.  Before discussing this violation of ACID, we start with some reminders about the last posts and with some explanations about GTIDs.

Unforeseen use case of my GTID work: replicating from AWS Aurora to Google CloudSQL

A colleague brought an article to my attention.  I did not see it on Planet MySQL where I get most of the MySQL news (or it did not catch my eye there).  As it is interesting replication stuff, I think it is important to bring it to the attention of the MySQL Community, so I am writing this short post.

The surprising part for me is that it uses my 4-year-old work for online migration to GTID with MySQL 5.6.  This is a completely unforeseen use case of my work as I never thought that my hack would be useful after Oracle include an online migration path to GTID in MySQL 5.7 (Percona did something similar for MySQL 5.6).

Many thanks Oracle for implementing RESET MASTER TO

MySQL 8.0.1 is out and it includes an implementation of my feature request (Bug #77438).  This extension to RESET MASTER allows to simplify master promotion with Binlog Servers.  Let's see how it works:

# mysql -N <<< "SHOW MASTER STATUS"
binlog.027892   3006935
# mysql -N <<< "RESET MASTER TO 12345; DO sleep(rand()*10); SHOW MASTER STATUS"
binlog.012345   92773
# mysql -N <<< "RESET MASTER TO 12345678; DO sleep(rand()*10); SHOW MASTER STATUS"
binlog.12345678 24795
# mysql -N <<< "RESET MASTER TO 1234567890; DO sleep(rand()*10); SHOW MASTER STATUS"
binlog.1234567890       13987
# mysql -N <<< "RESET MASTER TO 12345678901; DO sleep(rand()*10); SHOW MASTER STATUS"
ERROR 3567 (HY000) at line 1: The requested value '12345678901' for the next
binary log index is out of range. Please use a value between '1' and '2147483647'.
# mysql -N <<< "RESET MASTER TO $RANDOM; DO sleep(rand()*10); SHOW MASTER STATUS"
binlog.013529   89880
# mysql -N <<< "RESET MASTER TO $RANDOM; DO sleep(rand()*10); SHOW MASTER STATUS"
binlog.000831   22961
# mysql -N <<< "RESET MASTER TO $RANDOM; DO sleep(rand()*10); SHOW MASTER STATUS"
binlog.023089   107764
# mysql -N <<< "RESET MASTER TO $RANDOM; DO sleep(rand()*10); SHOW MASTER STATUS"
binlog.003433   67903

Many thanks Oracle for implementing my feature request, and a special mention to Daniël van Eeden for providing a patch in the bug report.

JFG proposed sessions for Percona Live Santa Clara (and Community voting)

This year, Percona introduced Community Voting for Percona Live submission.  This is what you can read on the conference website:

In an effort to involve the larger community in the selection of speaking sessions for the 2016 Percona Live Data Performance Conference, we’ve implemented a community voting process. After a speaker submits a proposal we encourage sharing to the community and social networks for a vote. The more highly ranked proposals will continue onto the next phase of the voting process with the conference committee.

Binlog Servers for Simplifying Point in Time Recovery

A common way to implement point in time recovery capability is:

1. to regularly do a full backup of a database,
2. and to save the binary logs of that database (or from its master if doing backups on a slave).

When point in time recovery is required you need to:

1. restore a backup,
2. and apply the binary logs up to the point of recovery.

(Step # 2 and # b above are the ones that will be simplified by using Binlog Servers.)

Abstracting Binlog Servers and MySQL Master Promotion without Reconfiguring all Slaves

http://blog.booking.com/abstracting_binlog_servers_and_mysql_master_promotion_wo_reconfiguring_slaves.html

Follow the link above to read my latest article on the Booking.com Developer Blog.  It is about Binlog Servers and how to promote a slave as the new master without reconfiguring all slaves.

This is also a good opportunity to remind you of my next talks:

• I’ll be giving a talk about Binlog Servers and another talk about Binary Logs in general at Percona Live Amsterdam. Feel free to grab me after the talk, catch me at Booking.com booth (# 205) or share a drink with me at the Community Dinner to exchange thoughts about Binlog Servers of other MySQL subjects. 
• I'll also be giving a talk about Binlog Servers at Oracle Open World in San Francisco at the end of October.
• Finally, I'll also be animating a  Birds-of-a-Feather (BoF) session at Oracle Open World: Riding the Binary Logs: Forthcoming Evolution in the Replication Stream.

Looking forward to seeing you in Amsterdam and/or in San Francisco.

MariaDB 10.0 Parallel Replication Benchmark Results (and PLAMS and OOW).

My latest post is online on the Booking.com blog: Evaluating MySQL Parallel Replication Part 3: Benchmarks in Production.  In this post, I present benchmark results on MariaDB 10.0 parallel replication on four Booking.com production workloads.

This post is also the opportunity to promote my two talks at Percona Live Europe, taking place in Amsterdam from September 21 to 23:

• Binlog Servers at Booking.com
• Riding the Binlog: an in Deep Dissection of the Replication Stream

I will also be at Oracle Open World in San Francisco (from October 25 to 29) giving a talk and animating a Birds-of-a-Feather on similar subjects:

• Binlog Servers at Booking.com (talk)
• Riding the Binary Logs: Forthcoming Evolution in the Replication Stream (BoF)

Looking forward to see you in Amsterdam and/or in San Francisco.

Self-Critic and Slides of my PLMCE Talks

The link to the slides of my talks can be found at the end of this post but first, let me share some thoughts about PLMCE.

Talking with people, I was surprised to be criticized of presenting only the good sides of my solution without giving credit to the good side of the alternative solutions.  More than surprised, I was also a little shocked as I want to be perceived as objective as possible.  Let me try to fix that:

MaxScale Binlog Server HOWTO: POC for Master Promotion without Touching any Slave

Note: DO NOT use this procedure in production, this is a proof of concept (POC).  MaxScale 1.1.0 does not yet fully support that procedure and things could go wrong in some situations (see at the end of the post for the details).

In my talk at PLMCE 2015, I presented an architecture to promote a slave as a new master without touching any other slave and I claimed that I tested it.  This HOWTO will show you how I did my test so you are able to reproduce my results.

MaxScale Binlog Server HOWTO: Operations (including Chaining)

In the Install and Configure HOWTO, we learned how to install and configure a MaxScale Binlog Server.  In this HOWTO, I will present the common operations that you might need to perform when using this software.  Those operations include:

• Purging Binary Logs,
• Chaining Binlog Servers,
• Saving Binary Log Files in the Non-Default Directory,
• Downloading Binary Logs other than First,
• Listing Connected Slaves,
• Disconnecting one or all Slaves,
• Differentiating a MaxScale from a MySQL Server,
• Getting More Information about Slaves (and more),
• Recovering After a Crash.

MaxScale Binlog Server HOWTO: Install and Configure

Updated 2015-04-25: add the link to the slides of my PLMCE talk and a link to a bug number.

MaxScale 1.1.0 is out and includes the new Binlog Server module.  This is the first post in s series of three.  The two others are about Operations and High Availability.  The links to the 2 other posts are at the end of this page.

In this post, I present how to install and configure MaxScale as a Binlog Server using the Binlog Router plugin.

Even Easier Master Promotion (and High Availability) for MySQL (no need to touch any slave)

Dealing with the failure of a MySQL master is not simple.  The most common solution is to promote a slave as the new master but in an environment where you have many slaves, the asynchronous implementation of replication gets in your way.  The problem is that each slave might be in a different state:

• some could be very close to the dead master,
• some could be missing the latest transactions,
• and some could be far behind (lagging, delayed slaves, or slaves in maintenance).

Promise from PLUK 2014 finally fulfilled: Better Parallel Replication for MySQL

http://blog.booking.com/better_parallel_replication_for_mysql.html

Follow the link above to read my latest article on the Booking.com developer blog.  It is about MySQL Parallel Replication and Binlog Servers.

This fulfills my promise made at Percona Live London 2014 during my talk High Availability, Disaster Recovery and Extreme Read Scaling using Binlog Servers: I finally took the time to write about slide #25.

This post is also a good opportunity to remind you that I will speak at Percona Live Santa Clara 2015 about Binlog Servers at Booking.com.  More to come about the content of the talk soon.

MySQL Crash-safe replication, Binlog Servers and Percona Live London

I just publish a post on the Booking.com blog: http://blog.booking.com/better_crash_safe_replication_for_mysql.html  Spoiler: it uses Binlog Servers.

This is also the opportunity to tell you that I will be at Percona Live London at the beginning of November, and that I will give a talk about Binlog Servers: High Availability, Disaster Recovery and Extreme Read Scaling using Binlog Servers.  I will not talk too much about Binlog Server for crash-safe replication, but I will present a new use-case for Binlog Servers that I did not blog about yet.  I am looking forward to meet you there.