Scaling Reads and Writes
Sharding Strategies and Consistent Hashing
Hash versus range sharding, why hash-mod-N is a trap, and how to survive hot keys and resharding.
Hard14 min read
By the end of this chapter
- Choose between hash, range and directory-based sharding from the query patterns
- Calculate how many keys move when a shard is added under modulo and consistent hashing
- Mitigate a hot key without resharding the whole system
- Outline a live resharding plan that never stops serving
Picking up where partitioning left off
The Data section made the case for partitioning — it is the only technique that raises write throughput — and warned that the key is nearly irreversible. This chapter is about the mechanics: given a key, which machine owns it, and what happens to that answer when the number of machines changes.
Three ways to assign a key to a shard
| Strategy | How a key is placed | Good at | Bad at |
|---|---|---|---|
| Range | Each shard owns a contiguous key range | Range scans, ordered reads | Hot spots when new keys cluster (timestamps, sequential ids) |
| Hash | Shard chosen from a hash of the key | Even spread of keys and load | Range queries, which must ask every shard |
| Directory | A lookup table maps key (or tenant) to shard |
Sign in to keep a note here. It is free, and nobody else can see it.
Sign inYou’ve reached the end
What this chapter was for:
- Hash sharding spreads load evenly and breaks range queries; range sharding keeps ranges and invites hot spots.
- Hash mod N reshuffles most keys when N changes; consistent hashing moves roughly 1/N of them.
- Many small logical partitions mapped onto fewer machines make rebalancing a matter of moving whole partitions.
- A single hot key defeats any sharding scheme and needs its own answer.
- Resharding is a migration: double-write, backfill, verify, cut over.