Load Balancers, Proxies and CDNs

Load Balancers, Proxies and CDNs — Tech Epitome

Three boxes, three jobs

They are often the same piece of software configured differently, which is exactly why the distinction is worth holding onto — the question is never 'which product' but 'which responsibility'.

	Question it answers	Typically knows about
Load balancer	Which instance handles this?	Your fleet's health
Reverse proxy	What should happen to this request?	Auth, rate limits, routing, TLS
CDN	Can this be served from nearer the user?	Cacheability, geography

Same shape on a diagram, different reasons to exist.

The CDN answers what it can. Everything else continues to your fleet.

Describe this diagram in words

Components (4): Client [client]; CDN [cdn] — static + cacheable; Load Balancer [load_balancer]; API Servers [api] Connections (3): Client -> CDN; CDN -> Load Balancer; Load Balancer -> API Servers Legend: -> synchronous, ~> asynchronous, => replication.

Choosing a strategy

How a balancer picks an instance matters more as requests become less uniform.

Strategy	Picks	Breaks down when
Round robin	The next one in order	Requests differ wildly in cost
Least connections	The least busy	Connections are long-lived but idle
Hash on key	Deterministically by user or key	The fleet changes size
Random with two choices	Better of two at random	Rarely — a strong default

Strategies, and the workload that defeats each.

Health checks are the point

A load balancer without health checks is a hazard: it will keep routing a share of traffic to an instance that is dead, and it will do so with perfect fairness. The health check is what turns balancing into availability.

Sticky sessions

Pinning a user to an instance lets you keep their session in that instance's memory. It is a real technique with a real cost: instances stop being interchangeable, so removing one drops those users' sessions, and load can no longer be evened out.

The alternative is to move session state out — into a cache or a signed token the client carries — which makes every instance identical again. That is usually the better trade, and it is the same principle as the previous section: stateless components scale horizontally almost for free.