Reliability
Failure Modes and Disaster Recovery
Containing the failures you cannot prevent, and getting back from the ones you could not contain.
By the end of this chapter
- Limit blast radius with cells, staged rollouts and isolated dependencies
- Stop a cascading failure with circuit breakers, bulkheads and load shedding
- Set RPO and RTO targets and pick a recovery strategy that meets them
- Explain why replicas are not backups, and what multi-region actually costs
Failure is the normal case
A single server failing is an unusual event. A fleet of thousands of components having something broken right now is the usual state of affairs. As an illustrative assumption, say each of 1,000 disks has a 2% chance of failing in a year. You should then expect around 20 disk failures a year, roughly one every two to three weeks, on a completely healthy fleet.
The goal therefore is not a system in which nothing fails. It is a system in which a failure stays small, is noticed quickly, and is recovered from without heroics.
Blast radius
Concept
Blast radius
The share of users or functionality affected when one thing goes wrong. A bad deploy that breaks every customer has a blast radius of one hundred percent; the same bug rolled out to one cell of twenty has five.
- Cells: split users into independent copies of the stack, each with its own databases and queues, so a fault in one cell does not reach the others.
- Staged rollouts: ship code and configuration to a small slice first, watch error rates, then widen. Configuration changes deserve the same care as code; they cause a large share of outages.
- Regional and zonal isolation: a dependency in one zone should not be able to take down callers in another.
- Limit shared fate: one shared cache cluster, one shared auth service or one global config push quietly joins every component's fate together.
Sign in to keep a note here. It is free, and nobody else can see it.
Sign inYou’ve reached the end
What this chapter was for:
- At scale something is always broken. Design so that a failure is small, not so that failures do not happen.
- Cascades spread through waiting. Circuit breakers stop the waiting; bulkheads stop it spreading.
- RPO is how much data you can lose; RTO is how long you can be down. Each recovery strategy buys a point on both.
- A replica faithfully copies your mistakes. Only a point-in-time backup undoes them, and only a tested restore proves you have one.