← System Design · Lesson 2 of 11
Load Balancing
One address in front of many machines, and the rules for choosing between them.
The idea
A load balancer is one address in front of many machines. Clients see a single endpoint; the balancer decides which instance actually serves each request.
It does two jobs, and the second is the one that matters more in practice.
Distribution spreads load so no machine is overwhelmed while others idle. The algorithm choices differ mainly in how much they know: round robin knows nothing and works when requests are uniform; least-connections knows how busy each server is and handles the far more common case of uneven request cost; hashing sends the same key to the same place, which keeps caches warm at the cost of even spread.
Health checking removes dead machines from rotation. This is what turns a crashed server from an outage into a blip. It also sets the floor on your failover time: a 10-second health check means up to 10 seconds of requests going into a hole, no matter how fast the machine actually died.
The distinction worth being precise about is L4 versus L7. An L4 balancer
routes packets by address and port without reading them — fast, works for any
protocol, and blind to content. An L7 balancer parses the HTTP request, so it can
route /api to one pool and /static to another, terminate TLS, retry idempotent
requests, and rate limit. That capability costs CPU and makes it a more
interesting thing to operate.
And the balancer itself is a single point of failure unless you plan otherwise — which is why real deployments run several behind a DNS record or an anycast address.
Walkthrough
When to reach for it
- You have more than one application instance — which is the moment you need one.
- You want zero-downtime deploys, where traffic drains from old instances as new ones come up.
- Requests vary a lot in cost, which is where least-connections beats round robin.
- You need TLS termination, path routing, or retries in one place rather than in every service.
Rather than the obvious alternative
DNS round robin
Free and requires no infrastructure, but clients cache DNS and it has no health checking — a dead machine keeps receiving traffic until TTLs expire. Fine for coarse geographic distribution, not for failover.
A service mesh
Gives per-service load balancing, retries and observability without a central hop, at the cost of substantial operational complexity. Worth it at many services, overkill at three.
Key terms
- Load balancer
- Distributes incoming requests across a pool of servers.
- Reverse proxy
- A server that fronts backends and forwards requests to them.
- DNS
- Resolves names to addresses, with caching at every layer.
- API gateway
- A single entry point handling routing, auth, and rate limiting for many services.
- Health checks
- An endpoint reporting whether an instance should receive traffic.
How to spot it
- The design has more than one instance of anything.
- The question mentions zero-downtime deploys or rolling releases.
- Requests need routing by path, region, or tenant.
- The interviewer asks what happens when a server dies mid-request.
Where it goes wrong
The balancer as a single point of failure
Putting one box in front of a redundant fleet moves the outage rather than removing it. Run several, fronted by DNS or anycast.
Health checks that only prove the process is running
A server that answers /health but cannot reach the database is worse than a dead one — it fails every real request while looking healthy.
No connection draining
Terminating an instance immediately kills in-flight requests. Drain first, then stop.
Sticky sessions by default
They defeat even distribution and make instance loss user-visible. Use them only when the alternative has been genuinely ruled out.