Reliability
Consensus and Leader Election
Getting several machines to agree on one fact, even when some are dead and the rest cannot tell which.
By the end of this chapter
- Explain why agreement is hard when slow and dead are indistinguishable
- Use majority quorums to reason about how many failures a cluster survives
- Describe leases, terms and fencing tokens, and the failure each one guards against
- State what a Raft-style protocol guarantees and what it costs
Why agreement is hard
'CAP and Consistency Models' warned that coordination is expensive. This chapter is about why it is also subtle. Imagine three servers that must agree on which of them is the primary for a database. If messages were instant and machines never failed, one would announce itself and that would be that.
Real networks delay and drop messages, and real machines pause: a long garbage-collection pause, a VM migration, a disk that hangs for twenty seconds. A node that has not heard from the primary cannot tell whether the primary is dead, slow, or simply cut off from it. Any rule like 'if I have not heard from the leader in five seconds, I become leader' can produce two leaders, because the old one may be very much alive on the other side of a partition.
Concept
The core difficulty
In a fully asynchronous network, where messages can take arbitrarily long, no deterministic protocol can guarantee that agreement is always reached if even one node may crash. Practical protocols respond by never compromising safety - they never decide two different things - and by relying on timeouts for progress, which they make whenever the network behaves reasonably.
What consensus provides
A consensus protocol lets a group of nodes agree on a value, or on a sequence of values, such that:
- Agreement: no two nodes ever decide different values.
- Validity: the decided value was actually proposed by someone, not invented.
Sign in to keep a note here. It is free, and nobody else can see it.
Sign inYou’ve reached the end
What this chapter was for:
- Any two majorities overlap, which is why a majority can make a decision that no other group can contradict.
- A cluster of 2f + 1 nodes survives f failures. Even-sized clusters add cost without adding tolerance.
- A leader can be deposed without knowing it, so storage must reject stale leaders by term or fencing token.
- Consensus buys safety at the price of a majority round trip per decision. Use it for small, important metadata, not for your bulk data path.