Scaling Reads and Writes
Rate Limiting and Backpressure
Token buckets, leaky buckets and sliding windows; bounded queues; and shedding load before overload turns into collapse.
By the end of this chapter
- Implement a token bucket and explain how its two parameters shape traffic
- Compare fixed window, sliding window and bucket algorithms on accuracy, memory and burst behaviour
- Explain why an unbounded queue under sustained overload makes every request fail
- Choose between rejecting, queuing and shedding when demand exceeds capacity
Two problems, one family of tools
Rate limiting answers 'how much may this client use?' — protection against abuse, runaway scripts and one tenant starving the rest. Backpressure answers 'what happens when everyone together asks for more than we can do?'. Both come down to the same move: decide early, and cheaply, which work will not be done.
The algorithms
Fixed window
Count requests per client per calendar minute; reject once the count passes the limit. One counter per client, trivially cheap. The flaw is at the boundary: with a limit of 100 per minute, a client can send 100 requests at 0:59 and 100 more at 1:00 — 200 requests in about a second, all allowed.
Sliding window
A sliding window log stores each request's timestamp and counts those within the last 60 seconds. It is exact, and costs memory per request. A sliding window counter approximates it with two fixed-window counts: current count plus the previous window's count weighted by how much of it still overlaps.
Limit 100 per minute. The previous minute had 80 requests; the current minute has 30 so far, and we are 15 seconds (25%) into it.Sign in to keep a note here. It is free, and nobody else can see it.
Sign inYou’ve reached the end
What this chapter was for:
- A rate limit protects capacity and fairness; it is a promise about what one client may consume.
- Token bucket allows bounded bursts with a steady average; leaky bucket smooths output to a fixed rate.
- Fixed windows allow double bursts at window boundaries; sliding windows fix that at some cost.
- Bounded queues and early rejection keep latency sane; unbounded queues turn overload into universal timeouts.
- Retries without backoff and jitter amplify the overload they are reacting to.