Limiting across many servers

Gatepass ships a token bucket for bookings: 10 per user per second, counted in each server's memory. In testing, on one server, it works perfectly. In production there are eight servers behind the load balancer, and the scalper's script gets 80 requests a second through without a single 429. Every server is enforcing the limit correctly. None of them can see the other seven.

Eight servers, eight limits

A limit is a statement about a user, but an in-memory counter is a statement about a server. With round-robin load balancing, one user's requests are spread across every server, so each server sees only a slice, and each slice stays under the limit. The real limit becomes the per-server limit times the number of servers, and it grows every time autoscaling adds a machine.

The obvious patch, giving each server limit ÷ N, fails the other way. Load balancers don't spread one user's requests evenly; over a short burst, one server might get five of a user's requests and another none. The user gets refused while well under the real limit, and the error depends on luck.

To enforce a limit per user, every server must see the same counter.

A shared counter in Redis

The standard answer is Redis: in memory, sub-millisecond, and it has atomic operations. Every API server talks to the same Redis, and the counter for user 81 lives in one place.

The race: read, decide, write

The first version most people write is three steps:

count = GET rl:user81
if count < 10: SET rl:user81 count + 1; allow
else: reject

Run that from eight servers at once. Server 1 reads 7. Before its SET lands, server 2 also reads 7. Both allow, both write 8. Two requests were allowed and the counter went up by one. Under a burst, dozens of requests can read the same value in the same millisecond, and the limiter lets through two or three times the limit. The slower your network round trip to Redis, the wider the window for the race.

The fix is to make the check and the update one atomic operation inside Redis, so no other request can run between them:

• Fixed window: INCR rl:user81:<window> returns the new count after adding one. If it's over 10, reject. Set an expiry the first time (EXPIRE when the result is 1, or SET … NX EX to create it) so old windows clean themselves up.
• Token bucket or sliding window: the logic needs several reads and writes, so run it as a Lua script (or a Redis function). Redis runs each script to completion before anything else, so the whole read-decide-write sequence is atomic. One round trip, no race.

Try it: where should the counter live?

Start with local counters and add servers. Switch to “limit ÷ N” and watch real requests get refused. Then try the naive Redis read-then-write, and drag the round trip up. Finish with atomic INCR.

Try it

One user, a burst, many servers

The limit is 10 requests per second per user. A script fires a burst within 40 ms and the load balancer spreads it across servers. How many get through?

Local countersLocal, limit ÷ NRedis GET then SETRedis atomic INCR

Each server counts only what it sees. With N servers, the user really gets N × the limit.

API servers4

Requests in the burst40

Redis round trip2 ms

server 1

5 ok · 0 429

server 2

10 ok · 0 429

server 3

10 ok · 4 429

server 4

10 ok · 1 429

Allowed in total35 / limit 10

Versus limit

+25 over

limit not enforced

Redis calls / request

0

no network hop

If Redis is down

unaffected

but inaccurate

The cost of sharing

Atomic Redis counters are accurate. They also put a network call and a shared dependency in front of every request:

• Latency. Each request now waits for a Redis round trip, typically 0.3–1 ms inside a data centre. Fine for bookings, worth thinking about for an endpoint that serves 50,000 requests a second.
• Load on Redis. Every request is a Redis operation. A single Redis node handles roughly 100,000 simple operations a second. Beyond that, shard the counters across several Redis nodes by key (user 81's counter always lives on the same node, so it stays atomic).
• Availability. When Redis is slow or down, what does the limiter do? Fail open (allow everything) keeps the site up but removes protection, which is usually right for general API limits. Fail closed (reject everything) is right where abuse is expensive, like OTP sending. Decide per endpoint, and always use a short timeout on the Redis call so a slow Redis can't stall every request.

Approximate limits are often fine

Not every limit needs to be exact. The question is how much overshoot you can live with:

• Local first, shared second. Each server keeps a local token bucket and syncs its usage to Redis every 100–500 ms in a batch. Redis load drops by orders of magnitude, and the limit can overshoot by about one sync interval's worth of traffic.
• Coarse limits on the edge. Your CDN or API gateway can enforce a rough per-IP limit (say 100 a second) with its own distributed counters. That stops the flood before it reaches you, and your precise per-user limits run behind it on far less traffic.
• Exact where it matters. OTP sends and login attempts are low-volume and costly to get wrong, so they get atomic, fail-closed limits.

Check yourself

Gatepass's limiter does GET, compares with the limit, then SET count + 1. Under a burst, users get well over the limit. What's the fix?

AAdd more Redis replicasBUse a shorter windowCMake check-and-increment atomic: INCR and compare its result, or run the logic in a Lua scriptDMove the counter back into each server's memory

Key takeaways

• In-memory counters limit per server; with N servers a user gets up to N times the limit.
• Splitting the limit across servers refuses real users, because one user's requests aren't spread evenly.
• A shared Redis counter works only with atomic updates: INCR for windows, a Lua script for token buckets and sliding windows.
• Shared counters add latency, load and a dependency. Set short timeouts and decide per endpoint whether to fail open or closed.
• Use exact limits where abuse is costly and cheaper approximate limits (local plus batched sync, edge limits) elsewhere.