What Scale Actually Means

What Scale Actually Means — Tech Epitome

Scaling is a sentence with two halves

People say a system scales as though it were a property of the code. It is not. Scaling is a claim with two halves, and a claim missing either half cannot be checked.

The first half is what grows: users, requests per second, stored bytes, concurrent connections, size of a single item. The second half is what must stay constant: response time, correctness, cost per request, operational effort.

Concept

A usable definition

A system scales along some dimension if you can grow that dimension by adding resources, without the properties you care about degrading and without the cost per unit rising.

Write it out for a real system and the vagueness disappears. "This service scales" becomes "this service can go from one thousand to one hundred thousand writes per second by adding API instances, keeping the ninety-ninth percentile under two hundred milliseconds, at roughly linear cost." That is a sentence somebody can disagree with, which is what makes it useful.

Two directions, two different ceilings

There are only two ways to give a system more resources, and they fail for different reasons.

	Vertical	Horizontal
What you do	Bigger machine	More machines
Code changes	Usually none	Often substantial
Ceiling	The largest machine available	The part that cannot be split
Failure domain	One machine, all of it	One machine, a fraction of it
Cost curve	Superlinear at the top end	Roughly linear, plus coordination

The two directions, and where each one stops.

Vertical scaling is genuinely underrated. It requires no distributed systems, no partitioning scheme, no consensus, and a single large machine today is very large indeed. The reason it is not the end of the story is that its ceiling is fixed by what the market sells, and that one machine is a single failure domain: when it goes, everything goes.

Horizontal scaling removes that ceiling and replaces it with a harder question — what happens to the parts of the system that cannot simply be duplicated?

The bottleneck always moves

A system has one bottleneck at a time. Remove it and you do not get a system without a bottleneck; you get a system whose bottleneck is somewhere else.

Adding API servers is cheap. It moves the pressure onto the database, which is where it stays.

Describe this diagram in words

Components (4): Client [client]; Load Balancer [load_balancer]; API Servers [api] — stateless, easy; Database [database] — the real limit Connections (3): Client -> Load Balancer; Load Balancer -> API Servers; API Servers -> Database Legend: -> synchronous, ~> asynchronous, => replication.

This is why the honest answer to "how would you scale this?" is a sequence rather than an architecture. First the web tier saturates and you add instances. Then the database's connection count saturates and you pool. Then read throughput saturates and you add replicas. Then write throughput saturates, and now you are partitioning, which is a different kind of problem with a different kind of cost.

How to use this in practice

Say what grows and what must hold. If you cannot, you do not yet have a design problem, you have a slogan.
Start with the simplest thing that meets today's numbers, and know which bottleneck you expect to hit first.
When you add a component, say which bottleneck it removes and which one it exposes.
Treat 'we could scale vertically for now' as a legitimate answer, not a cop-out.