Loading
Loading
Foundations
Three numbers that are constantly confused, and the reason averages lie about all of them.
By the end of this chapter
Latency is how long one request takes. Throughput is how many requests complete per second. They feel like the same thing and they are not — a system can improve one by sacrificing the other, and frequently should.
Concept
A courier carrying one envelope across a city has low latency and terrible throughput. A lorry that leaves once a day carrying a hundred thousand envelopes has enormous throughput and dreadful latency. Both 'deliver post'. Neither substitutes for the other.
Batching is the clearest case. Group a hundred writes into one transaction and total throughput climbs sharply, because the fixed cost is paid once. But the first write in the batch now waits for the ninety-ninth, so its latency got worse. That is a real trade and often the right one — it is only a mistake when it is made accidentally.
Suppose a hundred requests: ninety-nine take ten milliseconds and one takes two seconds. Mean latency is about thirty milliseconds. No request took thirty milliseconds. The number is arithmetically correct and describes no user's experience.
| Measure | Says | Use it for |
|---|---|---|
| p50 (median) | The typical request | Is the common path healthy? |
| p95 | The slow twentieth | Where degradation starts |
| p99 | The slow hundredth | What your worst-served users get |
| max | The single worst | Finding pathological cases, not SLOs |
Availability is usually quoted as a number of nines. The useful move is to convert it immediately into time, because time is something a team can reason about.
| Target | Downtime per year | Per month | Feels like |
|---|---|---|---|
| 99% | ~3.7 days | ~7.2 hours | Noticeably unreliable |
| 99.9% | ~8.8 hours | ~44 minutes | One bad afternoon a year |
| 99.99% | ~53 minutes | ~4.4 minutes | Needs redundancy everywhere |
| 99.999% | ~5 minutes | ~26 seconds | Needs automation, not people |
The reason to phrase it as a budget rather than a promise is that it makes the engineering decision explicit. Ninety-nine point nine percent means you may be down for forty-four minutes a month. If a deploy takes the system down for two minutes, you can afford twenty-two deploys — or you can invest in deploys that do not cause downtime. That is a conversation with a number in it.
A system that responds instantly with the wrong answer is available and unreliable. One that is occasionally down but never wrong is reliable and less available. They pull in different directions: the usual way to raise availability is to serve something when the authoritative source is unreachable, and the usual thing you serve is stale.
Deciding which one matters more is a product question, not an engineering one, and it differs per endpoint inside the same system. A stale follower count is fine. A stale account balance is not.
Sign in to keep a note here. It is free, and nobody else can see it.
Sign inWhat this chapter was for: