Request Rate Limiter

Functional requirements

• Allow or reject a request against a per-caller quota
• Support several named policies with different limits and windows
• Tell the caller when they may retry
• Apply limits per API key, per user and per IP independently

Non-functional requirements

• The check must add no meaningful latency to a request
• Limits should hold across many application instances
• The limiter failing must not take the API down with it
• Counting must be accurate enough to be fair, not perfectly exact

Scale assumptions

Worked estimates for this brief, not measurements of any real system. Disagree with them out loud — that is the exercise.

Distinct keys

~2 million active in any window

State per key

~100 bytes (counter, window start)

Latency budget

under 1 ms, so at most one network hop

Requests checked

50,000/second across the fleet

Total working set

~200 MB — comfortably in memory

Suggested API surface

• internalcheck(key, policy)

  Returns allowed, remaining, resetAt

• GET/v1/limits/{key}

  Current consumption, for support and debugging

• PUT/v1/policies/{name}

  Define or update a policy

Decisions you should be able to defend

• Which counting algorithm
• Where counters live
• Behaviour when the limiter is unreachable
• Exactness of counting