Module 5 · Lesson 2 of 5
Replication lag and read-your-writes
Why a fresh write can vanish, and three ways to stop it.
Maya buys a ticket for Saturday's comedy night. The payment page says "Booked!", she taps "My tickets", and the page says she has none. She buys again. Nothing is wrong with the code, and nothing is lost. The write went to the primary and the read went to a replica that hasn't heard about it yet.
Replication takes time
With read replicas, the primary applies a write and then streams it to the replicas. Streaming is asynchronous by default: the primary confirms the write to the app as soon as it's safely stored locally, without waiting for replicas. That keeps writes fast, and it has a price. For a short window, the replica is behind.
That window is replication lag. On a healthy network it is often a few milliseconds. It grows when:
- the replica is busy running a heavy report, so it applies changes slowly;
- a big batch (say, a bulk import of 2 million rows) makes a long burst of changes;
- the network between regions is slow or flaky;
- the primary is writing faster than a single replica can replay.
Notice that nothing here is a bug in the database. The system is eventually consistent: if writes stop, every replica ends up identical. It is just not consistent at every instant.
Try it
Set the lag to about a second, choose "No fix" and press the quick button. You will see the missing ticket. Then switch through the fixes with the same lag.
Interactive
Buy a ticket, then open My tickets
The primary has the new ticket instantly. The replica gets it after the lag. Use the quick button, then compare the fixes.
Fix
Every read goes to a replica, even right after a write.
Primary
0
tickets stored
Replica
0
caught up
- No reads yet. Buy a ticket and open the page.
Recent reads
0
Stale (recent)
0
user saw old data
Which fix worked every time? What did the others cost? Set the lag to 0 for a moment and notice that the problem vanishes. That is why this bug hides in development, where there's one database, and appears in production under load.
The guarantee you actually want
The user's complaint has a name: read-your-writes (or read-after-write) consistency. After I change something, my next read should show my change.
Notice how narrow this is. You don't need every reader in the world to see Maya's ticket instantly. Another user seeing it 800 ms late is fine. Only Maya must see it right away. That's what makes the fixes cheap: they only treat the person who just wrote as special.
A second, related guarantee is monotonic reads. With two replicas at different lags, Maya could load her page from replica A (has the ticket), refresh onto replica B (doesn't), and watch her ticket vanish. Time appears to go backwards. Pinning each user to the same replica for a session prevents it.
Three ways to get read-your-writes
- Read from the primary for a while after a write. When a user writes, set a short-lived flag (a cookie or session entry) such as "read primary for 5 seconds". The window must be longer than your normal lag, including bad days. It is simple and robust. The cost is a bit of extra primary load, only from users who just wrote.
- Write through a cache. When creating the ticket, also store it in a cache such as Redis. "My tickets" checks the cache first, so it can see the ticket before any replica does. It is fast, but it only covers what you remembered to cache. Other reads of the same data still go to replicas.
- Check the replica's position. The primary tells you the log position of your write (Postgres calls it an LSN). Save it with the user's session. Later, only use a replica that has already applied that position, otherwise fall back to the primary. This is precise, with no guessing about window lengths, but it takes plumbing: a position per session and a replica position check.
Some systems also use a sensitive-read rule by query: the seat availability check inside a booking transaction always goes to the primary, because a stale answer there sells the same seat twice. That is not about the user who wrote. It is about the cost of being wrong.
Check yourself
After booking, Maya's 'My tickets' page is empty for about a second. Which change fixes this with the least extra load on the primary?
When lag is fine
Not every read needs a fix. Ask what happens when this read is a second old:
- The public event listing shows 118 seats left instead of 117. Harmless.
- An analytics dashboard is 30 seconds behind. Harmless.
- A confirmation email says "ticket booked" before the replica can show it. Build emails from the primary's data or from the event itself, not from a replica read.
- Checking whether a coupon has already been used. Dangerous: a stale read allows a second use. Read from the primary.
The skill is sorting your reads into "must be fresh" and "can be a little old", and spending effort only on the first group.
Your turn
List three Gatepass reads. For each, say whether it can use a replica, and if it must be fresh, which fix you'd use.
To practise this, open the URL shortener drill at /drills/url-shortener and design where its reads and writes go, so a freshly created link still resolves right away.
Key takeaways
- Replicas apply changes asynchronously, so a read right after a write can miss it. The size of the gap is replication lag.
- The guarantee users expect is read-your-writes: after I write, I see my write. Monotonic reads stops data appearing then disappearing.
- Fixes: read the primary for a short window after a write, write through a cache, or check the replica's log position.
- Sort reads into must-be-fresh (checkout, coupons) and can-be-a-little-old (listings, dashboards).
- Monitor lag and take replicas that fall too far behind out of rotation.
Finished reading?
Mark it complete to tick it off in the course.