System Design · Lesson 2 of 11

Load Balancing

One address in front of many machines, and the rules for choosing between them.

The idea

A load balancer is one address in front of many machines. Clients see a single endpoint; the balancer decides which instance actually serves each request.

It does two jobs, and the second is the one that matters more in practice.

Distribution spreads load so no machine is overwhelmed while others idle. The algorithm choices differ mainly in how much they know: round robin knows nothing and works when requests are uniform; least-connections knows how busy each server is and handles the far more common case of uneven request cost; hashing sends the same key to the same place, which keeps caches warm at the cost of even spread.

Health checking removes dead machines from rotation. This is what turns a crashed server from an outage into a blip. It also sets the floor on your failover time: a 10-second health check means up to 10 seconds of requests going into a hole, no matter how fast the machine actually died.

The distinction worth being precise about is L4 versus L7. An L4 balancer routes packets by address and port without reading them — fast, works for any protocol, and blind to content. An L7 balancer parses the HTTP request, so it can route /api to one pool and /static to another, terminate TLS, retry idempotent requests, and rate limit. That capability costs CPU and makes it a more interesting thing to operate.

And the balancer itself is a single point of failure unless you plan otherwise — which is why real deployments run several behind a DNS record or an anycast address.

Walkthrough

No walkthrough for this topic yet — the guided exercises below run the same way.

When to reach for it

Rather than the obvious alternative

DNS round robin

Free and requires no infrastructure, but clients cache DNS and it has no health checking — a dead machine keeps receiving traffic until TTLs expire. Fine for coarse geographic distribution, not for failover.

A service mesh

Gives per-service load balancing, retries and observability without a central hop, at the cost of substantial operational complexity. Worth it at many services, overkill at three.

Key terms

Load balancer
Distributes incoming requests across a pool of servers.
Reverse proxy
A server that fronts backends and forwards requests to them.
DNS
Resolves names to addresses, with caching at every layer.
API gateway
A single entry point handling routing, auth, and rate limiting for many services.
Health checks
An endpoint reporting whether an instance should receive traffic.

How to spot it

Where it goes wrong

The balancer as a single point of failure

Putting one box in front of a redundant fleet moves the outage rather than removing it. Run several, fronted by DNS or anycast.

Health checks that only prove the process is running

A server that answers /health but cannot reach the database is worse than a dead one — it fails every real request while looking healthy.

No connection draining

Terminating an instance immediately kills in-flight requests. Drain first, then stop.

Sticky sessions by default

They defeat even distribution and make instance loss user-visible. Use them only when the alternative has been genuinely ruled out.