Module 4 · Lesson 1 of 6
Bigger box or more boxes?
Vertical scaling, horizontal scaling and where each one stops.
Tickets for the band go on sale at 10:00. The cache is doing its job, so the database is calm, but the single API server running Gatepass is not. At 10:00:01 its CPU hits 100%. Requests that took 40 ms now wait in a queue, and by 10:00:05 the queue is 30,000 deep. You need more computing power in the next few minutes, and there are exactly two ways to get it.
Two ways to grow
- Scale up (vertical scaling): replace the server with a bigger one. More CPU cores, more memory, faster disks. Same single machine, just stronger.
- Scale out (horizontal scaling): add more servers of the same size and spread requests across them with a load balancer.
Scaling up: simple, until it isn't
Scaling up is the right first move far more often than people admit. You change one setting in your cloud console, reboot, and the server is twice as strong. Your code doesn't change. There's still one machine to deploy to, one set of logs, one place where in-memory state lives. A modern 64-core machine with hundreds of gigabytes of memory can serve a surprising amount of traffic.
But it has three hard limits:
- Diminishing returns. Doubling the cores doesn't double throughput. Threads contend for locks, share memory bandwidth and wait on each other. A 64-core machine might do 5–6× what an 8-core one does, not 8×.
- A ceiling. There's a largest machine you can rent. When traffic outgrows it, scaling up has nowhere left to go.
- One machine is one failure. When it crashes, or you reboot it to resize it, or you deploy a bad build, Gatepass is down. All of it.
There's also a cost curve: the largest machines often cost more per core than mid-sized ones, because they're rarer and more in demand.
Scaling out: limitless, with homework
Scaling out puts several ordinary servers behind a load balancer, which spreads incoming requests across them (next lesson). Need more capacity? Add a server. In principle there's no ceiling: big sites run thousands of identical API servers.
It also fixes the availability problem. If one of ten servers dies, you lose 10% of capacity, not 100%, and the balancer stops sending it traffic. You can deploy one server at a time with no downtime.
The homework is that your servers must be interchangeable. Anything a server keeps in its own memory or disk, like a user's login session or an uploaded poster, now exists on one server only, and the next request may land on a different one. Lessons 4.3 and 4.5 deal with exactly that. You also take on a load balancer, health checks, and deploys across many machines.
Compare the two
Drag the load from a quiet Tuesday (500 requests/s) to the band's on-sale peak (25,000+). Watch the monthly cost, how busy each machine is, and what happens if one machine dies.
Interactive
Bigger box or more boxes?
Drag the load. Left: one API server, upgraded to the smallest size that stays under 70% busy. Right: as many 8-vCPU servers as needed behind a load balancer.
Scale up
1 × 32 vCPU
$1,120/month
- Handles up to 6,660 req/s comfortably.
- If it dies or reboots for an upgrade: 100% of traffic fails.
Scale out
3 × 8 vCPU + load balancer
$865/month
- Add a server whenever load grows. No upper limit from hardware.
- If one dies: the other 2 run at 85%.
What the numbers say:
- At low load, scaling up is cheaper. One 4-vCPU box ($140) beats two 8-vCPU boxes plus a balancer ($585). Many products never outgrow one big machine.
- In the middle, they cost about the same, but the scaled-out setup survives a server loss.
- At high load, scaling up runs out. Even the 128-vCPU machine tops out around 21,800 requests/s at 70% busy, and costs nearly twice as much as ten 8-vCPU servers that handle the same load.
Scale up until it gets expensive or risky. Scale out when you need more than one machine can give, or can't afford one machine's failure.
Why "70% busy" and not 100%
Both strategies in the visual aim to keep machines at or below 70% CPU at peak. That isn't waste:
- Queues explode near 100%. As a server approaches full utilisation, waiting time grows much faster than load (the same effect you saw on the database in lesson 3.1). At 95% busy, a small burst means seconds of delay.
- Losing a machine moves its load to the others. With three servers at 70%, losing one pushes the other two to 105%. With ten at 70%, the other nine go to 78%. More, smaller machines make each loss gentler.
- Traffic forecasts are wrong. The band might be more popular than you guessed.
Check yourself
Gatepass's admin dashboard is used by 40 organisers and runs on one 2-vCPU server at 15% CPU. A teammate wants to move it to five servers behind a load balancer 'for scale'. What's the best response?
The usual path
Most systems, Gatepass included, follow the same order:
- One modest server. Fine for the first fest.
- Scale up when CPU gets high. Cheap, fast, no code changes.
- Make the app stateless, then scale out to two or three servers behind a load balancer, mainly for availability.
- Add servers as traffic grows, ideally automatically (lesson 4.6).
- Meanwhile, keep the database on one big, scaled-up machine for as long as possible, because scaling it out is much harder (module 5).
The database line is worth repeating. API servers are easy to scale out because they don't hold the data. The database does, and two copies of data that both accept writes is a hard problem. That's why "add more API servers" is the common move and "add more databases" is the one you put off.
Your turn
Gatepass expects 25,000 requests/s at the peak of the band's sale and about 2,000 the rest of the week. Sketch a server plan: how many machines, what size, and what you'd do after the sale.
Stuck? Use the visual's numbers: an 8-vCPU server handles about 2,900 requests/s, and you want headroom for one to fail.
The URL shortener drill at /drills/url-shortener lets you try both: resize the single server first, then add more servers behind a balancer, and see which one survives the load test at each traffic level.
Key takeaways
- Scaling up means a bigger machine: no code changes, but diminishing returns, a hard ceiling and a single point of failure.
- Scaling out means more machines behind a load balancer: no ceiling and graceful failures, but servers must be interchangeable.
- Small systems are usually cheaper on one bigger machine. Large or must-stay-up systems need several.
- Plan for machines at about 70% busy at peak, so queues stay short and losing one machine is survivable.
- API servers are easy to scale out. Databases are not, so keep them scaled up for as long as you can.
Finished reading?
Mark it complete to tick it off in the course.