← System Design · Lesson 8 of 11
Rate Limiting
Decide who gets turned away, before load decides it for you.
The idea
Rate limiting decides, deliberately, who gets turned away. Without it the decision still gets made — by whichever resource exhausts first, and usually in the worst possible way, with every user degraded rather than the abusive one stopped.
The algorithms differ mainly in how they treat bursts:
Fixed window counts requests per calendar minute. It is the easiest to implement and has a real flaw: a client can send the full allowance at 11:59:59 and again at 12:00:00, so the actual peak is double the limit.
Sliding window log keeps timestamps and is exactly correct, at the cost of memory proportional to the request rate.
Sliding window counter blends the current and previous window by how far through you are. It is very nearly exact and costs two numbers, which is why it is the common production choice.
Token bucket refills tokens at a fixed rate up to a maximum. A request costs a token. This permits bursts up to the bucket size, which is usually what you want — a client that has been idle should be allowed to catch up.
Leaky bucket drains at a constant rate and permits no burst at all. Use it when the thing you are protecting genuinely cannot absorb a spike.
Two design questions matter as much as the algorithm. What is the key? Per IP punishes everyone behind a NAT; per API key is precise but requires authentication; per user is right for logged-in traffic. Most real systems layer several. And where does the state live? In-memory per instance is fast and wrong — with ten instances the effective limit is ten times what you configured. Shared state in Redis is the standard answer.
When you reject, say so properly: 429, with Retry-After. A client that
knows when to come back stops hammering; one that gets a generic 500 retries
immediately and makes it worse.
Walkthrough
When to reach for it
- Any public API — the limit is what stops one client consuming the capacity of all of them.
- Protecting an expensive downstream: a payment provider, an LLM, an email service with its own quota.
- Login and password-reset endpoints, where the limit is a security control against brute force.
- Fair sharing between tenants, so one customer's batch job does not starve the rest.
Rather than the obvious alternative
Autoscaling
Adds capacity for legitimate load, and happily scales up to serve an attack at your expense. Use both: limits decide who deserves capacity, scaling provides it.
A queue
Delays work rather than rejecting it, which is right when the work must eventually happen. Rate limiting is for when the correct answer is "no".
A circuit breaker
Protects *you* from a failing dependency; rate limiting protects a dependency from you. They solve opposite directions of the same relationship.
Key terms
- Rate limiting
- Capping request rate per client.
- Authentication vs authorization
- Who you are versus what you may do.
- Least privilege
- Granting only the access required.
- Defence in depth
- Layering controls so one failure is not fatal.
How to spot it
- The system exposes a public API.
- A downstream dependency has its own quota or a per-second cost.
- The interviewer raises abuse, scraping, or credential stuffing.
- Multiple tenants share infrastructure and one could starve the others.
Where it goes wrong
Per-instance counters
Ten instances each allowing 100 requests per minute is a limit of 1000, not 100. Share the state or divide the budget explicitly.
Limiting by IP alone
One corporate NAT is thousands of users on one address. Key by user or API key where you can, and treat IP as a coarse backstop.
Returning the wrong status
A 500 tells the client to retry immediately. A 429 with Retry-After tells it when to come back.
Rate limiting after the expensive work
The check has to happen before the database query it is meant to protect, or it protects nothing.
Build it
You have read how it works. This is where you write one — across several files, one rule at a time, run against tests in your browser.