Failover

At 8:41 on a Friday, the machine hosting Gatepass's primary database stops responding. The replicas are healthy and have nearly all the data. Until one of them becomes the new primary, nobody can buy a ticket. Failover is the process of making that switch, and nearly everything that can go wrong in it comes from a single question: how sure are you that the old primary is really gone?

What failover means

With single-leader replication there is one place that accepts writes. If it fails, writes stop. Failover promotes a replica to be the new primary and points everyone at it. It has five steps:

1. Detect that the primary is down.
2. Choose a replica, usually the one that has applied the most changes.
3. Promote it so it starts accepting writes.
4. Repoint the apps, proxies and other replicas to the new primary.
5. Deal with the old primary if it ever returns.

Where the time goes

Failover isn't instant, and each step has its own delay:

• Detection is usually the biggest. The system has to be confident the primary is dead, not just slow. Typical settings: ping every second and declare death after 10 consecutive failures. Shorter is faster but causes false alarms, the same trade-off as health checks.
• Promotion takes seconds: finish replaying the log it has already received, switch from read-only to read-write.
• Repointing takes more seconds: a virtual IP moves, a proxy updates, or DNS changes (and DNS can linger in caches for minutes, which is why serious setups avoid depending on it).
• Choosing a human. If a person must do it, add paging, waking up, logging in and checking, easily 5 to 15 minutes.

Try it

Kill the primary under the default settings and note the numbers. Then switch to an on-call human, then turn on synchronous replication, then bring the old primary back with fencing off.

Interactive

The primary dies

One primary, one replica, 100 writes per second. Kill the primary and follow the clock. One simulated second takes about 0.15 s.

Who decides

AutomaticOn-call human

Failure detected after

3 s10 s30 s

Synchronous replicationPrimary waits for the replica before confirming a write.

Replica lag when it died500 ms

Fence the old primaryCut it off so it can't accept writes if it comes back.

Primary dead, nobody knows (10 s) Promote replica (5 s) Repoint apps (3 s) Normal again (8 s)

Phase

healthy

Clock

–

would last 18 s

Writes rejected

0

apps got errors

Writes lost

–

confirmed, never reached replica

Kill the primary Old primary comes back Reset

What you should see:

• Automatic failover has an outage of around 18 seconds here, mostly detection. The human version is over two minutes even with an unusually fast human.
• With asynchronous replication, writes that the primary confirmed but hadn't yet sent are gone. At 100 writes per second and 500 ms of lag that's about 50 confirmed orders that never happened, as far as the new primary is concerned.
• With synchronous replication, nothing confirmed is lost. The price is that every write waits for the replica, so writes are slower, and if the replica is the one that fails, the primary has to stop or fall back.
• Without fencing, an old primary that wakes up after a freeze is still convinced it's the leader.

The two nasty failure modes

Lost writes. Asynchronous replication confirms writes before replicas have them. Customers got "Booked!" for orders that the new primary has never heard of. Options: accept it for non-critical data, use synchronous (or semi-synchronous, where at least one replica must confirm) replication for important data, and keep the old primary's disk. When it comes back you can often recover the missing writes by hand, which is painful but better than silence. Never let it silently rejoin.

Split brain. The old primary wasn't dead, only unreachable: a network partition, a long pause, a frozen virtual machine. The monitoring system promoted a replica, and then the old primary comes back still believing it's the leader. Now two nodes accept writes, each with orders the other doesn't have. Merging them is a nightmare.

Defences against split brain:

• Fencing. Make sure the old primary can't accept writes: cut its network, remove its storage access, or shut it down (the industry's blunt name for this is STONITH, "shoot the other node in the head").
• A quorum. The decision to promote is made by a majority of independent nodes (three or five voters), so a node that is cut off from the majority knows it has lost leadership and steps down. This is what tools like Patroni with etcd, or managed cloud databases, do for you.
• Epoch numbers. Each leader has a number that increases on every election. Replicas and clients reject writes from a lower number.

Check yourself

Gatepass uses asynchronous replication with about 800 ms of lag at 60 writes/s. The primary's host crashes and a replica is promoted. Roughly how many confirmed orders can be missing?

ANone, because the replica had all the dataBAbout 48CAbout 800DAll orders from the last hour

Your turn

Write the failover plan for Gatepass's database: who triggers it, how you prevent split brain, and what you do about the writes that might be lost.

01

02

03

04

Compare with an expertWrite at least 3 first.

The URL shortener drill at /drills/url-shortener lets you kill a database node during the load test, so you can measure for yourself how much traffic fails while a replica takes over.

Key takeaways

• Failover = detect, choose, promote, repoint, and handle the old primary. Downtime is the sum of the delays.
• Automation turns an outage of many minutes into seconds. Detection speed is a trade-off against false alarms.
• Asynchronous replication can lose confirmed writes at failover. Synchronous or semi-synchronous replication prevents it at the price of slower writes.
• Split brain happens when two nodes both think they are the leader. Prevent it with quorum decisions, fencing and epoch numbers.
• Rehearse failover regularly. A plan nobody has run is a guess.