Writing / Problems solved

Designing a read-through image pipeline at 50M+ images per day

A Cloudflare Workers and R2 image path with canonical cache keys, conditional requests, origin fallback, signed invalidation, and explicit consistency rules.

Image delivery usually gets introduced as a read problem: fetch bytes, return bytes, cache the response.

At scale, the hard operation is lifecycle change. What happens when an image is replaced? Which resized and watermarked variants are now stale? Which edge locations still hold the old response? Can a cold location recover without asking the origin to regenerate the same object? What does a browser do when it already has a valid copy?

The pipeline I worked on answered those by giving each layer exactly one job:

  • the origin stayed the source of truth and the variant generator
  • the Worker normalized requests and owned the read contract
  • edge cache handled hot responses
  • R2 provided durable edge-side backing storage
  • D1 tracked the key inventory needed for lifecycle operations

Requirements

Requirement

Design implication

Origin fallback

A missing edge object must be recoverable from the origin.

Edge reads

Hot images must not wake application servers.

R2 reads

A cold edge should usually stop at object storage.

Conditional requests

Valid browser copies should get 304 Not Modified.

Variant invalidation

All derived keys for one logical image must be discoverable.

Privileged lifecycle

Purge and deletion require an authenticated control path.

Layer visibility

Each response should say whether it came from browser, cache, R2, or origin.

The read path

sequenceDiagram
  participant B as Browser
  participant W as Worker
  participant C as Edge cache
  participant R as R2
  participant O as Origin

  B->>W: GET image + validators
  W->>C: match canonical key
  alt edge hit
    C-->>W: response + metadata
    W-->>B: 200 or 304
  else edge miss
    W->>R: get object by variant key
    alt R2 hit
      R-->>W: bytes + metadata
      W->>C: store response
      W-->>B: 200 or 304
    else R2 miss
      W->>O: fetch or generate variant
      O-->>W: bytes + metadata
      W->>R: store object
      W->>C: store response
      W-->>B: 200
    end
  end

The cache hit isn't the interesting part. The miss path is — it has to avoid making the origin the default source for every cold edge. R2 gives a durable copy that can warm a location after an edge-cache miss.

Canonical keys are a correctness boundary

The cache key has to represent the semantic image variant, not incidental URL differences:

type ImageVariant = {
  imageId: string;
  width?: number;
  height?: number;
  format: "avif" | "webp" | "jpeg" | "png";
  watermark: "none" | "default";
};

function keyFor(variant: ImageVariant): string {
  return [
    "image",
    variant.imageId,
    variant.width ?? "auto",
    variant.height ?? "auto",
    variant.format,
    variant.watermark,
  ].join("/");
}

An incomplete key produces two classes of bug. Include irrelevant request data and you fragment the cache. Omit a meaningful variant dimension and different images collide, serving the wrong bytes. The same canonical key should drive edge cache lookup, R2 storage, invalidation records, and observability — otherwise the system ends up with several definitions of identity.

Conditional requests move work out of the path

Browsers may already have the image body. The Worker should compare If-None-Match with the current ETag and return 304 when the representation hasn't changed:

function notModified(request: Request, etag: string): boolean {
  return request.headers.get("if-none-match") === etag;
}

The response then carries validators and cache metadata without sending another image body. That saves bandwidth and keeps the browser from queueing repeated bytes behind useful requests. The ETag has to describe the representation the browser will actually receive — a logical image identifier isn't enough if the image can change without that identifier changing.

Invalidation requires an inventory

An image is usually a family of objects:

avatar-42 / original / png
avatar-42 / 320x320 / webp
avatar-42 / 640x640 / avif
avatar-42 / 640x640 / webp / watermark

Deleting the original doesn't delete the derived objects. A lifecycle event has to discover every key tied to the logical image and remove them from durable storage and reachable edge caches.

flowchart TD
  Change["Origin lifecycle event"] --> Auth["Verify signed request"]
  Auth --> Lookup["Find variant keys in D1"]
  Lookup --> R2Delete["Delete R2 objects"]
  Lookup --> Purge["Purge edge keys"]
  Lookup --> IndexDelete["Delete inventory rows"]
  R2Delete --> Repopulate["Next read repopulates"]
  Purge --> Repopulate

D1 isn't the image store. It's the inventory that makes invalidation a normal indexed operation instead of an incident-time reconstruction exercise.

Consistency needs an explicit winner

The layers have different roles:

Layer

Role

Failure assumption

Origin

Authoritative generator

Source of truth is available eventually.

R2

Durable edge-side copy

Object may be absent or stale until lifecycle completes.

Edge cache

Hot per-location copy

Locations converge at different times.

Browser

Local copy

Validators decide whether the body is reusable.

The Worker needs an explicit policy for disagreement. A cache miss isn't proof the origin deleted the image. A stale cache hit isn't proof R2 is stale. The lifecycle path has to establish the order of deletion, purge, and repopulation.

Privileged operations need a separate boundary

Image reads can be public. Cache purge and object deletion can't be.

A short-lived signed request from the origin to the Worker is enough for the control path:

signature = HMAC(method + path + timestamp + body_digest)

accept only when:
  timestamp is inside the allowed window
  body_digest matches the request body
  signature verifies against the deployment secret

The exact signing algorithm is a choice. The boundary isn't: lifecycle operations require authentication, replay protection, and logs that identify the actor and the affected keys.

Measured effect

With the cache path enabled, the image path measured roughly 250 ms at p50, 670 ms at p95, and 720 ms at p99. When requests bypassed both edge cache and R2 and fell through to the origin, p95 rose to roughly 1.9–2.1 seconds and p99 to roughly 3.8–4.0 seconds. Origin CPU for the image path fell from around 90% to below 15%, while the pipeline served more than 50M images per day.

None of that came from one magic cache setting. It came from putting repeated work on the right layer and making the lifecycle operations explicit. At high volume, serving bytes is the easy part. Serving the correct representation after a replacement, deletion, variant change, or cache disagreement is the actual engineering problem.

Bhavin Kamani

A small command line for a curious visitor.