Distributed Systems
Loading
Loading
Distributed Systems
Taking work off the request path: what it buys, and the four things you must then handle.
By the end of this chapter
If a user action triggers work they do not need to wait for — send the email, transcode the video, rebuild the index — putting it behind a queue returns the response immediately and lets the work happen at whatever rate the workers manage.
Components (5): Client [client]; API [api] — returns immediately; Job Queue [queue]; Workers [worker] — scale independently; Database [database] Connections (5): Client -> API; API ~> Job Queue (enqueue); Job Queue ~> Workers (consume); Workers -> Database; API -> Database Legend: -> synchronous, ~> asynchronous, => replication.
The second benefit is absorption. A traffic spike that would overwhelm synchronous processing instead lengthens the queue; the work still happens, just later. The queue converts a capacity problem into a delay problem, which is usually the better problem.
| Problem | Why it appears | Usual answer |
|---|---|---|
| Duplicates | At-least-once delivery retries on doubt | Idempotent consumers keyed on a message id |
| Ordering | Parallel consumers finish out of order | Order per partition key, or design for commutativity |
| Poison messages | A message that always fails, retried forever | Dead-letter queue after N attempts |
| Invisible failure | Nobody is waiting for the result | Queue depth and consumer-lag alerts |
Sign in to keep a note here. It is free, and nobody else can see it.
Sign inWhat this chapter was for: