Serving user images without burning money
A design review of a Google Cloud image pipeline: signed uploads, fixed variants, immutable names and a CDN, with failure and deletion paths worked through.
Take a hypothetical product where users upload photos: avatars, listings, posts. The first version is nearly always the same. The client posts a file to the API, the API writes it to a bucket, and the frontend renders the bucket URL in an <img> tag. A phone photo is 2 to 5 MB, so a feed that shows twenty of them as 300-pixel thumbnails moves tens of megabytes to draw one screen, you pay egress on every byte, the API has proxied uploads it never needed to touch, and the files still carry the GPS coordinates of someone's kitchen. This review takes that product and designs the image path properly: what it has to guarantee, where the money goes, the pipeline, and then the parts that break under load, failure and deletion. I met this problem on a large-scale social platform I'm building, where avatars and uploaded media appear on nearly every screen.
What the design has to guarantee#
- Egress is the bill. Storage is cheap; bytes leaving the cloud are not. Every decision is judged by how many bytes it puts on the wire.
- The API never carries image bytes. Proxying uploads burns CPU, memory and request time on the most expensive tier.
- Uploads are hostile until proven otherwise. Wrong MIME types, decompression bombs, polyglot files, location metadata.
- Deletion is real. Under GDPR, a deleted photo has to stop being served, including from caches, within a bounded time.
- The marginal request is a cache hit. Nothing in the serving path costs more per request as traffic grows.
Where the money goes#
On Google Cloud there are five meters (check Cloud Storage pricing and Cloud CDN pricing for current rates in your region):
- Storage, per GB-month by class. Nearline, Coldline and Archive hold data more cheaply but charge retrieval fees and have minimum storage durations of 30, 90 and 365 days.
- Operations. Class A (writes, lists) cost roughly ten times Class B (reads). Six variants per upload is six Class A writes; serving from the bucket is one Class B read per view.
- Egress from Cloud Storage to the internet, tiered by monthly volume. For image-heavy products this is the largest line by far.
- CDN charges if you add Cloud CDN: cache egress, cache fill when the CDN reads from the bucket, a per-request lookup fee, and the load balancer's hourly forwarding rule.
- Transformation compute: the CPU time to decode, resize and re-encode. AVIF encoding is the heavy one.
Two levers follow. The biggest is bytes per view, decided by which variant you serve, not by the CDN. The second is how often the origin is touched, decided by object naming and caching headers. Everything below serves those two.
Choosing where to transform#
Fixed variants made at upload, on-the-fly resizing and a third-party image service differ mostly in what they cost per cache miss and who operates them:
| Option | Transform model | Cost shape | Good fit |
|---|---|---|---|
| Public bucket, direct URLs | At upload (your code) | Origin egress on every view | Low traffic, internal tools |
| Cloud CDN with a backend bucket | At upload (your code) | Cache egress, lookups, fill, small fixed load balancer cost | Most products on GCP at meaningful traffic |
| On-the-fly resizer on Cloud Run behind a CDN | Per unique URL, on first request | Compute on every miss | Many layouts needing arbitrary sizes |
| Third-party image CDN | On request, managed | Per image stored, delivered or transformed | Small teams who want to buy the whole problem |
On-the-fly resizing has a nasty property: every distinct width and format is a cache miss that costs compute, and if the width comes from the URL, anyone can generate unlimited misses by iterating ?w=301, ?w=302. Allow-listing the sizes turns it into fixed variants with worse first-view latency. In my own system I resize once at write time, in the same async worker path that already handles other upload side effects through Pub/Sub, because uploads are rare and views are common, and I want the compute attached to the rare event. Firebase Hosting is not in the table because it serves deployed files, not user content.
So the design generates a small fixed set of variants at upload time, stores them in a public bucket under immutable names, and serves them through Cloud CDN with a one-year Cache-Control. Originals never become public: the client uploads straight to a private staging bucket with a V4 signed URL, an event-triggered function validates and processes the file, and the staging object is deleted.
The variant set is three widths (320, 768 and 1536 pixels) in two formats (AVIF and WebP): thumbnails, in-feed images and full-screen views on dense displays. Six objects per image trades storage and upload-time writes for never sending 3 MB to fill a 300-pixel box.
Upload without touching the API#
The API authenticates the user, decides what they may upload, and returns a short-lived URL that permits one PUT of one object with one content type. The V4 signing process binds the method, object name, expiry and signed headers into the signature.
import { randomUUID } from "node:crypto";
import { Storage } from "@google-cloud/storage";
const storage = new Storage();
const stagingBucket = storage.bucket(process.env.STAGING_BUCKET!); // validate in your config module
const MAX_BYTES = 10 * 1024 * 1024;
const ALLOWED_TYPES = new Set(["image/jpeg", "image/png", "image/webp", "image/avif"]);
export async function createUploadUrl(userId: string, contentType: string, declaredSize: number) {
if (!ALLOWED_TYPES.has(contentType)) {
throw new Error("unsupported_type"); // map to a 400 in your framework
}
if (!Number.isInteger(declaredSize) || declaredSize <= 0 || declaredSize > MAX_BYTES) {
throw new Error("too_large");
}
const uploadId = randomUUID();
const objectName = `incoming/${userId}/${uploadId}`;
const [url] = await stagingBucket.file(objectName).getSignedUrl({
version: "v4",
action: "write",
expires: Date.now() + 10 * 60 * 1000, // 10 minutes
contentType,
extensionHeaders: {
"x-goog-content-length-range": `0,${MAX_BYTES}`,
},
});
return {
uploadId,
url,
headers: {
"Content-Type": contentType,
"x-goog-content-length-range": `0,${MAX_BYTES}`,
},
};
}What to know about this code:
- On Cloud Run there is no key file to sign with. The client library falls back to the IAM
signBlobAPI, so the runtime service account needsroles/iam.serviceAccountTokenCreatoron itself. A first deploy without that grant fails on signing, not on upload. - The client must send exactly the signed headers. A browser that sends a different
Content-Typegets403 SignatureDoesNotMatch. Return the headers with the URL so the frontend cannot get them wrong. - The declared size is advisory. The signed
x-goog-content-length-rangeheader asks Cloud Storage to reject bodies outside the range; for a guarantee enforced purely by policy, a V4 signed POST policy with acontent-length-rangecondition is the stricter option. Either way, the processing function re-checks the real size. - The staging bucket is private, with uniform bucket-level access, public access prevention enforced, and CORS allowing
PUTfrom your web origin.
The object name holds the user ID and a random upload ID. It is never shown or served, so it does not need to be pretty.
Validate and resize in a separate trigger#
An Eventarc trigger on google.cloud.storage.object.v1.finalized for the staging bucket invokes a Cloud Run function. The staging bucket must be separate from the outputs: if the function wrote into the bucket that triggers it, every write would fire another event.
The function uses sharp, which wraps libvips and is fast and memory-efficient for this job.
import { createHash } from "node:crypto";
import { cloudEvent } from "@google-cloud/functions-framework";
import { Storage } from "@google-cloud/storage";
import sharp from "sharp";
type StorageObjectData = { bucket: string; name: string; size?: string };
const storage = new Storage();
const PUBLIC_BUCKET = process.env.PUBLIC_BUCKET!;
const MASTER_BUCKET = process.env.MASTER_BUCKET!;
const MAX_BYTES = 10 * 1024 * 1024;
const WIDTHS = [320, 768, 1536] as const;
const FORMATS = ["avif", "webp"] as const;
const ACCEPTED = new Set(["jpeg", "png", "webp", "avif"]);
const IMMUTABLE = "public, max-age=31536000, immutable";
cloudEvent<StorageObjectData>("makeVariants", async (event) => {
const { bucket, name, size } = event.data!;
if (!name.startsWith("incoming/")) return;
const source = storage.bucket(bucket).file(name);
if (Number(size ?? 0) > MAX_BYTES) {
await source.delete({ ignoreNotFound: true });
return;
}
let input: Buffer;
try {
[input] = await source.download();
} catch (err: any) {
if (err?.code === 404) return; // redelivered event, already processed
throw err;
}
// Reject anything that is not really an image we accept, and cap decoded
// pixel count so a tiny file cannot expand into gigabytes of memory.
const opts = { limitInputPixels: 50_000_000, failOn: "error" as const };
const meta = await sharp(input, opts).metadata().catch(() => null);
if (!meta?.format || !ACCEPTED.has(meta.format)) {
await source.delete({ ignoreNotFound: true });
return;
}
const [, userId, uploadId] = name.split("/");
// Deterministic per upload: a retried event produces the same names.
const imageId = createHash("sha256").update(uploadId).update(input).digest("base64url").slice(0, 22);
// autoOrient() applies EXIF orientation. sharp drops EXIF
// (including GPS) from output unless you ask it to keep metadata.
const base = sharp(input, opts).autoOrient();
const jobs = WIDTHS.flatMap((width) =>
FORMATS.map(async (format) => {
const resized = base.clone().resize({ width, withoutEnlargement: true });
const encoded =
format === "avif" ? resized.avif({ quality: 50, effort: 4 }) : resized.webp({ quality: 75 });
const buffer = await encoded.toBuffer();
await saveOnce(PUBLIC_BUCKET, `v/${imageId}/${width}.${format}`, buffer, `image/${format}`, IMMUTABLE);
}),
);
const master = await base.clone().resize({ width: 2560, withoutEnlargement: true }).webp({ quality: 90 }).toBuffer();
jobs.push(saveOnce(MASTER_BUCKET, `m/${userId}/${imageId}.webp`, master, "image/webp", "private, no-store"));
await Promise.all(jobs);
// Record { uploadId, userId, imageId, width: meta.width, height: meta.height } in your database here,
// using uploadId as the document key so retries are idempotent.
await source.delete({ ignoreNotFound: true });
});
async function saveOnce(bucket: string, name: string, data: Buffer, contentType: string, cacheControl: string) {
try {
await storage.bucket(bucket).file(name).save(data, {
resumable: false,
contentType,
metadata: { cacheControl },
preconditionOpts: { ifGenerationMatch: 0 }, // never overwrite an existing object
});
} catch (err: any) {
if (err?.code !== 412) throw err; // 412: already written by an earlier attempt
}
}The details that earn their keep:
metadata()is the real type check. The declared content type came from the client; the decoded format comes from the bytes. Anything sharp cannot decode, or decodes as a format you did not accept, is deleted.limitInputPixelsguards against decompression bombs, such as a small PNG that claims to be 50,000 by 50,000 pixels. sharp has a default limit; set one that matches what a phone camera produces.- Metadata is stripped by default. sharp does not copy EXIF, ICC or XMP to the output unless you ask for it, so location data is gone before anything becomes public.
- Retries are harmless. Events can arrive more than once; deterministic names,
ifGenerationMatch: 0and a database write keyed by upload ID make a repeat a no-op. - Memory. A 1536-pixel AVIF encode of a 12-megapixel photo needs a few hundred megabytes. Give the function 1 GiB, keep concurrency at one or two per instance, and measure before tuning.
- HEIC is not on the accepted list. The prebuilt sharp binaries do not decode HEVC-compressed HEIC because of patent licensing. Browsers usually convert to JPEG when a file input is used; native apps should convert before uploading.
Names that never change#
The public URL of a variant is https://img.example.com/v/{imageId}/768.avif. The imageId derives from the upload and its content, and an object at that name is never overwritten. That is what makes Cache-Control: public, max-age=31536000, immutable safe: a new image, or a re-encode, always gets a new name. Changing encoder settings later means writing under a new prefix (v2/) and updating references, never mutating in place. This single decision turns the origin from something read on every view into something read once per edge location.
Immutable caching is a contract, and I learned that on my own landing page before it could bite in the app: replace bytes in place under an unchanged URL and some visitors keep the old version for up to a year, with nothing you can do remotely. So the rule is mechanical and lives in my deploy checklist, not my memory. Any in-place asset swap bumps a ?v= parameter on every tag that references it.
The database stores the filename, never the full URL, and each service builds the URL through one small helper at its response boundary. I denormalised full URLs onto some records early on, and unwinding that showed me why: a stored URL bakes in the domain, the path scheme and the version parameter, so switching CDN would have meant a migration over millions of documents. With filenames it is a config change and a deploy.
I do not deduplicate identical files across users. Pure content addressing would let two users who upload the same meme share one object, and deleting it for one of them would then need reference counting. Mixing the upload ID into the hash keeps names unique per upload and deletion simple.
On the page, the browser needs to choose among the variants. A <picture> element with an AVIF <source> and a WebP fallback, each with a srcset listing the three widths and an accurate sizes attribute, does it; width and height on the <img> reserve space so layout does not shift. The one part people get wrong is sizes, which must describe the rendered width or the browser picks the 1536-pixel file for a thumbnail. The MDN guide to responsive images covers the syntax.
The CDN in front#
A backend bucket with Cloud CDN behind a global external Application Load Balancer:
# The variants bucket must be readable by the CDN; only derived variants live here.
gcloud storage buckets add-iam-policy-binding gs://example-img-variants \
--member=allUsers --role=roles/storage.objectViewer
gcloud compute backend-buckets create img-backend \
--gcs-bucket-name=example-img-variants \
--enable-cdn \
--cache-mode=CACHE_ALL_STATIC \
--default-ttl=86400 \
--max-ttl=2592000
gcloud compute url-maps create img-map --default-backend-bucket=img-backend
gcloud compute ssl-certificates create img-cert --domains=img.example.com --global
gcloud compute target-https-proxies create img-proxy --url-map=img-map --ssl-certificates=img-cert
gcloud compute addresses create img-ip --global
gcloud compute forwarding-rules create img-https \
--global --load-balancing-scheme=EXTERNAL_MANAGED \
--address=img-ip --target-https-proxy=img-proxy --ports=443The 30-day --max-ttl is about deletion, not performance. In the TTL override documentation, max TTL applies only in CACHE_ALL_STATIC mode and caps how long the CDN keeps a response whose origin max-age is larger. It does not change the max-age sent to the client. So a deleted image can linger at the edge for at most 30 days even if an invalidation is missed, while any browser that already fetched it still holds its own copy for up to a year.
What the bill looks like#
Assume 50 million image views a month, mostly thumbnails and feed images, with rates rounded from the public price lists for a European region and a single volume tier. Use the pricing calculator with your own traffic before deciding anything.
| Scenario | Bytes delivered | Dominant charges | Rough monthly order |
|---|---|---|---|
| Originals served from a bucket (avg 2.5 MB) | ~125 TB | Origin egress | Five figures in USD |
| Variants from the bucket, no CDN (avg 60 KB) | ~3 TB | Origin egress, Class B reads | Low hundreds |
| Variants through Cloud CDN, 95% hit ratio | ~3 TB | Cache egress, lookups, LB fixed cost | Low hundreds, lower latency |
The shape is the lesson. Serving the right-sized variant cuts bytes roughly fortyfold, which dwarfs everything else; a 4 MB upload shown as a 90-pixel avatar would otherwise pay about forty times the egress its pixels justify, on every view. I watch the cache hit ratio the way I watch an error rate: at 95% one request in twenty reaches the origin, and at 99% origin traffic falls by a further factor of five. At this volume the CDN is close to cost-neutral: cache egress is cheaper per GB than origin egress, but you add lookup fees and an hourly forwarding rule. You buy it for latency and for how it scales into higher volume tiers, not for a dramatic saving at small scale. Storage for a million images, with six variants plus a master, is a few tens of dollars a month. Transformation compute depends on your image mix, so measure the CPU seconds of the function on real uploads.
What breaks#
Deletion and GDPR. When a user deletes an image, order matters. Remove the database record first so no page references it. Delete the variants (gcloud storage rm "gs://example-img-variants/v/Qm3x/**" or the library call) and the master. Then invalidate the CDN path with gcloud compute url-maps invalidate-cdn-cache img-map --path="/v/Qm3x/*" --async. Invalidations are rate-limited, so for a whole-account deletion invalidate a per-user path where the URL structure allows it, or accept the --max-ttl bound and document it. If the invalidation step fails silently, the image lives at the edge until that bound, so log and retry it like any side effect. Browser caches on other people's devices are out of your reach; what you control is that your URLs stop resolving. GDPR deletion as a distributed-systems problem covers running this across every store.
Soft delete. Buckets retain deleted objects for seven days by default, which is billed as storage and keeps the data recoverable. That is useful on the master bucket. On the variants bucket I would disable it or shorten it (gcloud storage buckets update gs://example-img-variants --clear-soft-delete), and either way the retention belongs in the privacy policy.
Abandoned uploads. Lifecycle rules clean up what the pipeline leaves behind. On staging, anything still under incoming/ after a day was abandoned or rejected:
{
"rule": [
{ "action": { "type": "Delete" }, "condition": { "age": 1, "matchesPrefix": ["incoming/"] } }
]
}Apply it with gcloud storage buckets update gs://example-img-staging --lifecycle-file=staging-lifecycle.json. Masters are only read when you regenerate variants, so they can move to Coldline after 30 days, but Coldline has a 90-day minimum and a retrieval fee, so that only pays off if regenerations are rare batch jobs. Keep the variants bucket on Standard: cache fills read it constantly and retrieval fees would cancel the saving.
The processing gap. For a second or two after upload the variants do not exist. The client needs a pending state (show the local preview from the file input) and a way to learn the image is ready, by polling the upload ID or subscribing to a database document.
Redesigns cost a backfill. A layout that needs a 1024-pixel width means regenerating from masters, which is a batch job and the reason masters are kept at all. AVIF at high effort values can dominate function time; effort: 4 is a reasonable middle.
A public bucket is public. Anything written there is readable by anyone with the URL. The function must never write masters or originals to it; an organisation policy restricting allUsers bindings to labelled buckets is a cheap guard. Signed upload URLs are bearer credentials for their lifetime, so keep expiries short and scope each to one object name.
Too much machinery. At a few thousand images a month, a private bucket with short-lived signed read URLs, or a single public bucket with pre-sized uploads, is enough, and the load balancer, certificate and function pipeline are weight you do not need yet. Genuinely private images (ID scans, medical documents) invert the whole design: no shared caching, signed URLs or an authenticated proxy with Cache-Control: private, no-store. And a team with more layouts than engineers may do better buying a managed image CDN that transforms on request, trading per-image cost for attention.
Related articles
All articles →A static bundle in Cloud Storage is cheap to run and easy to get wrong: which Cache-Control values to set, how to upload them, and how Cloud CDN behaves.
8 min
A scheduled sync pays to refresh every record, including the ones nobody reads. Refreshing stale data when someone looks makes cost follow attention.
15 min
Start with a budget that emails a monitored channel, then add a Pub/Sub relay, usage alerts and a heartbeat that notices silence when that is worth building.
10 min