Blob Storage and Media Uploads

Two kinds of data in one file

A user's profile photo is two things: a few hundred bytes of facts — owner, size, content type, upload time, processing state — and a few megabytes of pixels. The facts are queried, joined and updated. The pixels are written once and read whole. Those are opposite access patterns, and they belong in different stores.

Uploads that bypass your servers

The naive upload sends the file to the API, which forwards it to storage. Every byte crosses your servers twice, and each slow mobile upload holds a connection and memory for its whole duration. The better shape is to have the API hand out permission and let the client send the bytes straight to storage.

1. The client asks the API for an upload, stating size and content type.
2. The API checks permissions and limits, creates a metadata row in a pending state, and returns a presigned URL: a storage URL carrying a signature that permits exactly one PUT to one key, for a few minutes.
3. The client uploads directly to object storage. Large files use multipart uploads, so a dropped connection resumes from the last part instead of from zero.
4. Storage emits an event when the object lands. A worker validates the file, produces derived versions, and marks the row ready.
5. Reads go through a CDN, which fetches from storage on a miss and serves from the edge after that.

The CDN in front

Media is the ideal CDN workload: large, read many times, and rarely changed. The one thing that makes it hard is changing an object in place. If a user replaces their avatar and the new file is written under the same key, every edge that cached the old one keeps serving it until its cache entry expires or is explicitly purged.

• Private media: serve through short-lived signed URLs (or signed cookies) so the CDN can cache the bytes while still checking that each request was authorised.
• Validate on the server, not the client: the worker should verify the real content type and size of what arrived, since the client chose what to send.
• Abandoned uploads: pending rows whose object never arrived, and objects whose row was never created. A periodic sweep plus a storage lifecycle rule on an uploads prefix cleans up both.

Recognising it

• Anything users upload — images, video, documents, exports: metadata row plus object.
• Large or resumable uploads: multipart, directly to storage.
• Heavy read fan-out on unchanging files: immutable keys behind a CDN.
• Processing after upload: an event from storage into a queue, not work inside the upload request.