Module 3 · Lesson 1 of 7
Why caching works
Hot data, hit rates, and the 80/20 shape of real traffic.
At 6 pm a band with four million followers announces one show, sold through Gatepass. Within a minute, 20,000 people a second are opening the same event page, and your database is answering the same question 20,000 times a second. The answer hasn't changed once.
The same question, thousands of times
Here is what one page view costs today. The app server receives GET /events/the-band-mumbai, runs a query that joins the event, the venue, the ticket tiers and the remaining seat counts, turns the rows into JSON and sends it back. On a quiet database that query takes about 25 ms.
Twenty-five milliseconds sounds cheap. Multiply it out: 20,000 requests a second × 25 ms is 500 seconds of database work every second. Even with good indexes and a large machine, your Postgres can do maybe 4,000 of these a second. The rest wait in a queue, the queue grows, and pages that took 25 ms start taking 3 seconds, then time out.
The frustrating part is that almost all of this work is repeated. The event's name, date, venue and poster are fixed. The ticket tiers change a few times a day. Even the "seats left" count only matters to the nearest few dozen until the very end. You are paying full price to recompute an answer you computed a millisecond ago.
What a cache is
A cache is a small, fast store that keeps answers you've already computed, so the next request can reuse them. For Gatepass it will usually be an in-memory key-value store like Redis or Memcached, sitting next to your app servers. You ask it for event:the-band-mumbai; if it has the value, you get it back in under a millisecond. If not, you do the slow work once and put the result in the cache for everyone after you.
Two words you'll use constantly:
- A hit is a request the cache could answer.
- A miss is a request it couldn't, which falls through to the database.
Memory is fast because it is close and simple. Reading a value from Redis on the same network is roughly 0.5–1 ms, almost all of it network. The database query was 25 ms, and that was before it was overloaded. The cache doesn't make the database faster. It makes the database unnecessary for most requests.
A cache doesn't speed up your database. It stops most requests from reaching it.
Hit rate is the number that matters
Everything about a cache comes down to one ratio: hits divided by total requests. That hit rate decides how much load is left for the database, and the relationship is harsher than it looks.
| Hit rate | Miss rate | Queries reaching the database (of 20,000/s) |
|---|---|---|
| 50% | 50% | 10,000 |
| 90% | 10% | 2,000 |
| 99% | 1% | 200 |
| 99.9% | 0.1% | 20 |
Going from 90% to 99% sounds like a 10% improvement. For the database it is a 10× drop in load. Think in misses, not hits: the miss rate is what your database has to survive.
Average latency follows the same shape. With a 1 ms cache and a 25 ms database, a 90% hit rate gives you about 3.5 ms on average. At 50% it's about 13.5 ms, if the database isn't overloaded, which at 10,000 queries a second it would be.
Check yourself
Your hit rate drops from 98% to 96% after a release. What happens to database load?
Traffic is lumpy, and that's good news
You can't cache everything. Gatepass has 5,000 live event, artist and venue pages, and the cache is far smaller than the database. So how can a small cache answer most requests?
Because real traffic is never spread evenly. A few pages get most of the views and a long tail gets almost none. This shape is called a power law (or Zipf distribution): the most popular page gets about twice the traffic of the second, three times the traffic of the third, and so on. It is the same shape behind the "80/20 rule", and you'll find it in product pages, search terms, videos and short links.
On announcement night the curve gets even steeper. One page, the band's show, is suddenly worth more than the next thousand combined. That's the best case for a cache: hold a handful of pages and you've absorbed most of the load.
Try it. Start with a cache of 100 pages and typical traffic, then slide the skew to "Announcement". Then try "Flat" and see how much memory you'd need to protect the database.
Interactive
Cache hit-rate explorer
Gatepass serves 20,000 page reads a second across 5,000 pages. Choose how many pages the cache holds and how skewed the traffic is.
The classic power law most real traffic follows.
Hit rate
57%
DB queries/s
8,592
limit 4,000
Avg page read
timeouts
Cache memory
800 KB
2% of pages
The misses alone are more than the database can answer. Queues build, requests time out, and the cache's hits can't save the page.
A few things to notice:
- With typical traffic, the top 1,000 pages (20% of them) take about 80% of requests. With announcement traffic, the top 100 take nearly as much.
- The curve flattens. Each extra page you cache is less popular than the last, so doubling the cache does not double the hit rate. Past a point, memory buys you almost nothing.
- The database cliff is sharp. Between "comfortable" and "timeouts" there are only a few points of hit rate, because queueing makes a busy database slow down much faster than its load rises.
What a cache costs you
A cache is not free, and it's worth naming the price before you add one.
- Stale data. The cache holds a copy. When the real data changes, the copy is wrong until you replace it. Lesson 3.3 is all about this.
- A second system to run. Redis needs memory, monitoring and a plan for when it restarts. When it comes back empty (a cold cache), every request is a miss and the database takes the full load at once.
- More code paths. Every read now has two outcomes, hit and miss, and both need to be correct and tested.
- Hidden dependence. Once the database is sized for 2,000 queries a second because the cache absorbs the rest, the cache is no longer an optimisation. It is load-bearing. If it dies, the database dies with it.
That last point surprises people. A cache starts as "make it faster" and quietly becomes "required to stay up". Plan for it the way you'd plan for any other critical component.
Your turn
List three things in Gatepass you would cache, and one you would never serve from a cache. Say why.
Stuck? Ask of each piece of data: how often is it read, how often does it change, and what happens if a user sees an old copy for a few seconds?
The URL shortener drill at /drills/url-shortener is a near-perfect caching problem: a few writes, billions of redirects, and a small set of very popular links. You'll add a cache there and watch the database load fall in the load test.
Key takeaways
- A cache stores answers you've already computed, so repeated reads skip the slow work. It removes load from the database rather than speeding it up.
- Hit rate is the number to watch, but think in misses: going from 90% to 99% hits cuts database load by ten times.
- Real traffic follows a power law, so a small cache holding the hottest keys absorbs most requests. Each extra key is worth less than the last.
- A cache brings stale data, a cold-start risk and a new critical dependency. Once the database relies on it, losing the cache is an outage.
Finished reading?
Mark it complete to tick it off in the course.