Module 3 · Lesson 6 of 7
Thundering herds
When a hot key expires and ten thousand requests hit the database at once.
The cache has been holding the band's event page all evening, absorbing 6,000 reads a second. At 19:42:30 its 30-second TTL runs out. In the next 250 ms, while the first miss is still rebuilding the page, 1,500 more requests also miss, and every one of them sends the same heavy query to the database. This is a thundering herd (also called a cache stampede), and it can take down a database that was coping fine a second earlier.
Why one expiry can cause an outage
Cache-aside has a gap built into it. Between "the key is gone" and "someone has put it back", every request for that key is a miss, and each miss independently decides to rebuild the value. For an ordinary key read twice a minute, that gap doesn't matter. For a key read 6,000 times a second, a 250 ms gap is 1,500 identical database queries.
It gets worse, because the gap isn't fixed. The rebuild query takes 250 ms on an idle database. With 1,500 copies of it arriving at once, the database is no longer idle. Each query slows down, so the gap gets longer, so more requests miss, so the database gets busier still. This positive feedback loop is what turns a blip into a pile-up.
There's a second version of the same problem that doesn't need a hot key at all. Suppose a deploy flushed the cache at 19:42:00 and the first requests refilled all 41 popular pages within a second, each with a 30-second TTL. They will all expire together at 19:42:30, and again at 19:43:00, and so on. Individually each is a small miss. Together they're a synchronised wave.
Watch it happen
The simulation below replays two minutes of announcement-night traffic after a deploy. With no protection, each wave of expiries pushes the database far over its limit. Turn the three defences on one at a time, then in pairs.
Interactive
A hot key expires
Two minutes of Gatepass traffic after a deploy cached everything at once: the band's page at 6,000 reads/s plus 40 other event pages. TTL is 30 s. Turn on protections and watch the database.
Peak DB load
12,300/s
Time overloaded
9.9 s
Queries refused
92,070
beyond capacity
DB queries total
154,800
over 2 minutes
Each expiry sends a wall of misses to the database. It slows down, rebuilds take longer, and the misses keep coming.
Things to try:
- Jitter alone spreads the 40 ordinary pages out, but the band's page still expires on its own and causes the same overload. Jitter fixes synchronised expiry, not hot keys.
- Coalescing alone turns 6,000 misses a second into a trickle: one rebuild per key per server. The spikes remain, because all keys still expire together, but they stay below capacity.
- Early refresh removes the misses altogether. Popular keys never actually expire.
- Coalescing plus jitter is the cheapest combination that makes the chart boring, which is what you want.
Defence 1: request coalescing
Only one request should rebuild a key; everyone else should wait for its result. This is request coalescing (sometimes called single-flight).
Within one app server it's easy: keep an in-memory map of "rebuilds in progress". The first miss for event:band starts the query and stores a promise in the map. Later misses for the same key find the promise and await it. When it resolves, all of them get the same result, and only one query reached the database.
Across 20 servers you get at most 20 rebuild queries per key, which is usually fine. If it isn't, use a short lock in the cache itself:
- On a miss, try
SET lock:event:band 1 NX PX 2000(set only if absent, expire in 2 s). - If you got the lock, rebuild the value, store it, delete the lock.
- If you didn't, wait 50 ms and read the cache again. Give up and query the database (or return a fallback) after a few tries.
The lock's expiry matters: if the server holding it crashes, nobody else could rebuild until the lock expires.
Defence 2: refresh before expiry
Better still, never let a hot key expire while people are reading it. Three ways to do that:
- Background refresh. A job refreshes known-hot keys every 20 seconds, before their 30-second TTL is up. Simple and predictable; you need a list of what's hot.
- Probabilistic early expiry. Each read that is close to the TTL has a small chance of rebuilding early, and the chance rises as expiry approaches. With thousands of reads a second, one of them almost certainly rebuilds a little early, while everyone else keeps getting hits. This needs no list of hot keys.
- Serve stale while revalidating. Keep the value past its "fresh" time but mark it stale. The first reader after that triggers a rebuild in the background and everyone, including that reader, keeps getting the old value until the new one lands. CDNs do exactly this with the
stale-while-revalidateheader, which you'll meet in the next lesson.
All three trade a little freshness for never having a gap.
Defence 3: jittered TTLs
Whenever many keys are set at the same moment (after a deploy, a cache restart or a bulk import), give them slightly different TTLs: instead of exactly 30 seconds, 30 seconds ± 20%. It costs one line of code:
const ttl = Math.round(30 * (0.8 + Math.random() * 0.4));
await cache.set(key, value, { EX: ttl });
Expiries that would have landed in one second now spread over twelve. It does nothing for one very hot key, which is why it's a complement to coalescing, not a replacement.
Coalescing stops many requests rebuilding the same key. Jitter stops many keys expiring at the same moment. Early refresh stops hot keys expiring at all.
Check yourself
After a Redis restart, Gatepass's database overloads in waves every 60 seconds, then settles after about 10 minutes. Which fix targets this specific pattern?
The cold start
The worst herd of all is an empty cache at peak traffic: after a Redis failover, a flush, or a deploy that changed every key's version. Every key is a miss at once. Coalescing still helps, because it limits each key to one rebuild, but thousands of different keys can still overwhelm the database. Plans that help:
- Warm the cache before you need it. Before the 10:00 sale, run a script that loads the top few hundred pages.
- Don't change every key at once. Roll out a new key version gradually, or avoid deploys that invalidate everything during peak hours.
- Protect the database. A cap on concurrent database queries (lesson 5.4 covers connection pools) turns overload into fast failures for some requests instead of slow failures for all.
Your turn
Tickets for the band go on sale at 10:00 tomorrow. Write a three-point plan so the event page's cache can't cause a stampede during the sale.
Key takeaways
- When a hot key expires, every request in the rebuild window misses and runs the same query. The database slows, the window widens, and it feeds itself.
- Request coalescing lets one request rebuild while the rest wait: in memory per server, or with a short lock across servers.
- Early refresh (background, probabilistic or stale-while-revalidate) keeps hot keys from ever expiring.
- Jittered TTLs stop keys that were set together from expiring together.
- Treat an empty cache at peak as a known risk: warm it, avoid mass invalidation, and cap database concurrency.
Finished reading?
Mark it complete to tick it off in the course.