Autoscaling

At 9:00 on Friday, tickets for a headline gig go on sale. Gatepass normally runs on 4 API servers. By 9:01 traffic is five times normal, the servers sit at 100% CPU, and the new machines you ordered by hand are still booting. Autoscaling is the habit of letting the platform add servers when load rises and remove them when it falls, so nobody has to be awake for it.

What autoscaling does

Because the API servers are stateless, adding one is cheap: start a copy, wait for its health check to pass, and the load balancer begins sending it traffic. Autoscaling puts a loop around that:

1. Measure a signal, such as average CPU across the group.
2. Compare it to a target, say 60%.
3. Adjust the number of servers up or down, within a minimum and maximum you set.

The idea is simple. The details are where autoscaling goes wrong.

Pick a signal that matches the bottleneck

CPU is the default, and for a CPU-bound API it works. But a server can be in trouble with low CPU. If each request waits 400 ms on the database, the server has idle CPU and a growing pile of waiting requests. Better signals depend on what limits you:

• CPU or memory: good for compute-heavy work.
• Requests per server: good when each request costs about the same. Target 200 req/s per server and divide.
• Queue depth or age of oldest job: the right signal for workers. If the oldest ticket email has waited 5 minutes, add workers, whatever their CPU says.
• Latency (p95): close to what users feel, but noisy, and adding servers does not help when the real bottleneck is the database.

Lag: the slow part

Reacting is not instant. Count the delays from a spike to a served request:

• The metric is averaged over a window, often 1 minute.
• The autoscaler checks every 15–60 seconds and may require a few readings in a row.
• The cloud provider launches a machine: 30–90 seconds for a virtual machine, a few seconds for a container on a warm host.
• The app starts, warms its caches and passes health checks: 10–60 seconds.

Add it up and a new server can easily arrive 2–4 minutes after the spike started. A ticket sale that sells out in 90 seconds is over by then. That's why reactive scaling alone fails for events you know about.

Check yourself

Tickets for a gig go on sale at 9:00 and normally sell out by 9:02. Which approach works best?

AAutoscale on CPU with a target of 60%BScale to the expected size at 8:45, then let autoscaling handle the surprisesCRun at the maximum size permanentlyDScale up after the first errors appear

Scale up fast, scale down slowly

Adding too late costs you customers. Removing too early costs you a second outage. So the two directions get different rules:

• Scale out quickly. A short window, a big step ("add 50% of current size, at least 2"), no waiting.
• Scale in slowly. Wait for load to stay low for 10–15 minutes, remove one or two servers at a time, and use connection draining so in-flight requests finish before a server stops.

Without that asymmetry you get flapping. Load dips, the autoscaler removes servers, load per server jumps back above target, it adds them again, and you pay for constant churn while latency wobbles. A gap between the scale-out and scale-in thresholds (say out above 65% and in below 35%) prevents it. A cooldown after each change also gives the new size time to show up in the metrics.

Limits and cost

Always set a minimum and a maximum.

• The minimum is your resilience floor. With a minimum of 1, one failure is an outage, so it should be at least the N+1 count from the health checks lesson, spread across zones.
• The maximum is your wallet's safety belt. A bug, a bot or an attack can drive load without limit, and an autoscaler will happily buy 400 servers to serve junk. A cap turns a surprise bill into a degraded service, which is usually better. Rate limiting (later in the course) deals with the junk itself.

Money is also why autoscaling pays off. If Gatepass needs 20 servers for two hours on sale days and 4 otherwise, running 20 always costs five times as much for the quiet 22 hours.

Other gotchas:

• Startup time. A 3-minute boot makes every rule worse. Keep images small, do heavy warm-up in the background, and pre-build the machine image.
• Cold caches. Fresh servers have empty local caches and fresh connections. Ten new servers connecting to the database at once is its own load spike (see connection pooling later in this module).
• Stateful pieces don't scale this way. Databases need other techniques. That's the rest of the module.

Your turn

Design the autoscaling policy for Gatepass's API and its email workers for sale days. State the signal, the minimum and maximum, and how you handle the 9:00 spike.

01

02

03

04

Compare with an expertWrite at least 3 first.

Key takeaways

• Autoscaling is a loop: measure a signal, compare to a target, adjust the group size within a minimum and maximum.
• Pick a signal that reflects the real bottleneck: CPU, requests per server, or queue age for workers.
• Reaction takes minutes. For known rushes, scale on a schedule before the event.
• Scale out fast and in slowly, with a gap between thresholds and a cooldown, to avoid flapping.
• Set a floor for resilience and a ceiling for cost. Autoscaling the app tier can't fix a database bottleneck.