Module 4 · Lesson 4 of 6
Health checks and redundancy
Noticing a dead server, and having enough left when one dies.
At 10:07, in the middle of the sale, API 2's machine loses its disk. The process is gone, but the load balancer doesn't know that yet. It keeps sending API 2 a third of all requests, and each of those fans sees an error. Having three servers only helps if the system notices quickly that it has two, and if two are enough.
Redundancy is two problems
Scaling out gave Gatepass several servers, which means one can fail without taking everything down. But that's only true if two things hold:
- Detection. The load balancer must notice the dead server and stop sending it traffic. Until then, a share of requests fails.
- Capacity. The remaining servers must be able to carry the full load. If they can't, removing the dead one just moves the failures around.
This lesson is about both. The first is solved by health checks, the second by planning for N+1.
Health checks
A health check is a small request the balancer sends to every backend on a schedule, typically GET /healthz every few seconds. A fast 200 OK means healthy. An error, a refused connection or no answer within a timeout counts as a failure. After a set number of failures in a row the balancer removes the server from its pool. After a set number of successes, it adds it back.
The two knobs, interval and threshold, trade speed against false alarms:
- Detection delay, worst case, is roughly interval × failures required. Every 10 seconds with 3 failures needed is up to 30 seconds of errors.
- Checking too eagerly has its own failure. With a 1-second interval and a threshold of 1, one slow garbage-collection pause or one dropped packet removes a perfectly good server. Under heavy load, servers get slow, fail checks, get removed, and the remaining ones get even more load, which is how a balancer can turn "busy" into "down".
Common settings are an interval of 5–10 seconds, 2–3 failures to remove and 2–5 successes to restore.
Try it
Kill API 2 and watch the red bars. Then try a shorter interval, a lower threshold, and no health checks at all. Finally, push the traffic to 75% and kill a server again.
Interactive
Kill a server
Three API servers behind a balancer, each good for 100 req/s. Each second of the simulation runs in 0.2 s. Kill API 2 and count the failed requests before the balancer notices.
Check every
Failures before removal
API 1
running
in pool
API 2
running
in pool
API 3
running
in pool
Failed requests
0
Time to remove
–
worst case 30 s
Survivors can absorb the load.
What you should notice:
- Without health checks, a third of requests fail until someone notices and fixes it by hand, which on a good night is minutes.
- With checks, failures stop after the detection delay. Faster checks mean fewer failed requests.
- At 75% load, removing the dead server doesn't stop the failures. Two servers at 112% each drop the excess. You've detected the problem perfectly and still have an outage.
What /healthz should check
A health endpoint has to answer one question: "should the balancer send me traffic?" Get that wrong in either direction and it hurts.
- Too shallow: an endpoint that always returns
200as long as the process is running. It misses a server whose database connection pool is broken or whose disk is full. - Too deep: an endpoint that checks the database, Redis and the payment provider on every call. When the database has a 2-second hiccup, every server fails its check at once, and the balancer removes all of them. A shared dependency failing is not a reason to take every server out of the pool, since none of them would be better.
A sensible split, used by Kubernetes and most platforms:
- Liveness: "is this process stuck?" Very cheap. If it fails, restart the process.
- Readiness: "can this server serve requests right now?" Checks things that are local to the server: finished starting up, connection pool usable, not shutting down. If it fails, take it out of the pool, but don't restart it.
N+1: having enough left
Detection only helps if the survivors can carry the load. The rule of thumb is N+1: if you need N servers at peak, run N+1, so losing any one leaves you at full capacity. Critical systems run N+2, so they can lose one during a deploy and one more by accident.
The arithmetic is worth doing explicitly. With three servers, each at 60% at peak, losing one puts the other two at 90%. That's tight but survivable. At 70% each, the survivors would need 105%, which fails. With ten servers at 70%, losing one puts the other nine at 78%. The more servers you have, the less headroom each needs.
Redundancy you haven't sized for is decoration. Check what the survivors would carry if any one server died at peak.
Two more places where one failure takes out "redundant" servers:
- Same rack, same zone. If all three servers run in one data centre (one cloud availability zone), a power or network fault there kills all three. Spread servers across at least two zones.
- Same bug. A bad deploy goes to every server. Rolling deploys, one server at a time with health checks gating each step, limit the damage to one server.
Check yourself
Gatepass runs 4 API servers at 80% CPU during the sale, with health checks every 5 s. One dies. What happens?
Your turn
Design the health-check setup for Gatepass's API during the sale: interval, thresholds, what /healthz checks, and how many servers you'd run if the peak needs 12.
In the URL shortener drill at /drills/url-shortener you can add a second server and a load balancer, then see how the design holds up when a node fails during the load test.
Key takeaways
- Redundancy needs detection (stop sending traffic to the dead server) and capacity (survivors can carry the load).
- Health checks remove a server after a number of consecutive failures. Worst-case detection is roughly interval × threshold.
- Checks that are too eager cause flapping. Checks that test shared dependencies can remove every server at once.
- Separate liveness (restart me) from readiness (route to me).
- Plan for N+1 or N+2 servers and spread them across zones. Do the arithmetic for losing one at peak.
Finished reading?
Mark it complete to tick it off in the course.