Reliability
Timeouts, Retries and Idempotency
A remote call can succeed, fail, or leave you not knowing. Most reliability bugs live in the third case.
By the end of this chapter
- Set a timeout from a latency budget rather than from a library default
- Retry with exponential backoff and jitter, and only the failures worth retrying
- Explain how retries at several layers multiply into a retry storm
- Make a non-idempotent operation safe to retry with an idempotency key
Every remote call has three outcomes
An in-process function call either returns or throws. A call over the network has a third outcome, and it is the one that causes the trouble: you sent the request, and nothing came back in time. The request may never have arrived. It may have arrived and failed. Or it may have arrived, succeeded completely, and the response is what got lost.
Concept
The unknown outcome
From the caller's side, a lost request and a lost response look identical. A timeout therefore never tells you the work did not happen. Any design that treats 'timed out' as 'failed' will, eventually, do something twice.
This chapter is about living with that. Timeouts decide how long you are willing to not know. Retries decide what you do next. Idempotency is what makes doing it again harmless.
Choosing a timeout
A call with no timeout waits forever, and a thread or connection waiting forever is a resource that never comes back. Many client libraries default to no timeout, or to one measured in minutes, which in practice is the same thing during an incident.
Two numbers decide a sensible value. From below: the downstream's own latency, usually its p99 or p99.9 plus some headroom, so that healthy-but-slow requests are not cut off. From above: the deadline of whoever called you. There is no point waiting two seconds for a dependency when your own caller gave up after one.
Sign in to keep a note here. It is free, and nobody else can see it.
Sign inYou’ve reached the end
What this chapter was for:
- A timeout does not mean the operation failed. It means you do not know, and your design must allow for it having succeeded.
- Back off exponentially and add jitter, or every client retries in lockstep and the recovery gets knocked over again.
- Retry at one layer, with a budget. Retries at every layer multiply.
- Retries are only safe when the operation is idempotent, and an idempotency key is how you make it so.