Search Tech Epitome

Jump to a section

Jump to a section

System design·Hard

File Sync and Storage

Keep the same folder identical on every device, uploading each byte only once.

Brief

System design·Hard

Keep the same folder identical on every device, uploading each byte only once.

Brief

Resume an interrupted upload without starting over

Sync changes made on one device to the user's other devices automatically

Keep previous versions of a file and allow restoring one

Detect when two devices edited the same file while offline and keep both copies

Share a folder with another user

A stored file must never be corrupted or silently lost
A small edit to a large file should upload roughly the changed bytes, not the whole file
Changes should reach other online devices within a few seconds
Metadata operations like listing a folder must stay fast regardless of file sizes

Worked estimates for this brief, not measurements of any real system. Disagree with them out loud — that is the exercise.

Chunk size: ~4 MB average; a 1 GB file is ~256 chunks
Data per user: ~10 GB average, so ~1 EB logical before deduplication
Upload volume: if a change averages 2 MB of new chunks, ~400 TB/day (~4.6 GB/second) inbound
Changes per day: ~20 file changes per active user, so 200 million (~2,300/second)
Registered users: 100 million, with 10 million active daily
Metadata per file: ~500 bytes, so ~1,000 files × 100 million users ≈ 50 TB of metadata

POST/v1/files/{path}/commit
Propose a new version as an ordered list of chunk hashes; returns which hashes are missing
PUT/v1/chunks/{hash}
Upload one missing chunk; the server verifies the hash
GET/v1/changes?cursor=
Long-poll for metadata changes since a cursor
GET/v1/chunks/{hash}
Download one chunk, usually through a short-lived signed URL
GET/v1/files/{path}/versions
List previous versions for restore

Add components, connect them, and note why each one is there.