Writing / Engineering tradeoffs

The cache hit isn't the interesting part

A US consumer app was serving more than 50 million images a day, with image-path origin CPU around 90%. What we changed, and what a founder or CTO should check when the origin is doing repeat work.

ElevenLabs' status-page write-up for INC-465 describes what happened on 18 and 19 November 2026. On 18 November, from 14:49 to 20:55 UTC, "a burst of concurrent requests triggered repeated full cache rebuilds on the voice caching cluster, exhausting its memory and client buffers." Backend services had no upper bound on cache connections. Slowed responses caused application servers to open connections until the cluster hit its maximum client limit, failed health checks, and entered "a repeating restart cycle with an empty cache." Queued requests then spread that connection exhaustion to other caching clusters.

The write-up does not use the word "stampede". What it describes has that shape: a burst of concurrent requests set off repeated full cache rebuilds, the cluster kept restarting with an empty cache, and requests queued on application servers spread the connection exhaustion to other caching clusters.

At peak, between 15:45 and 17:15 UTC, ElevenLabs reported that 12%–50% of requests failed across Voices, Creative, Studio, Music, Reader and login/workspace routes. Text to Speech had elevated time-to-first-audio and a 3.4% failure rate. Impact was primarily in the US. A smaller, related disruption the next day, from 14:10 to 15:20 UTC, left stale active-call counts in the affected cache. About 10% of new conversations overall were refused between 14:10 and 15:20 UTC. ElevenLabs cleared the stale counts at 15:17.

The facts above are from ElevenLabs' own write-up of INC-465. The page does not show a publish date.

A US consumer app I worked on served more than 50 million images a day. Repeated image delivery put substantial load on the origin, even when the underlying image had not changed. Image-path CPU sat at around 90%. Hot images were still waking application servers.

The obvious move is to add a cache. That helps, but it does not decide whether the path is safe. Once images are cached at the edge, the hard questions are about change: what happens when an image is replaced or deleted, which resized and watermarked variants are now stale, and whether a cold location can recover without asking the origin to regenerate the same object.

Those are the questions this pipeline had to answer. I wrote the engineering detail in Designing a read-through image pipeline at 50M+ images per day. This piece is about what a founder or CTO should look at in their own stack when the origin is spending its CPU on repeat image delivery.

What was actually going wrong

The origin was still the source of truth, and it still needed to generate missing variants. That was correct. The problem was that it was also serving the same bytes over and over.

A request that can be answered from a hot edge cache, or from a durable copy in R2, should not reach an application server. In this system, too many of those requests still did.

What we changed

I gave each layer one job:

  • the origin stayed the source of truth and the variant generator
  • a Cloudflare Worker normalized requests and owned the read contract
  • the edge cache handled hot responses
  • R2 held durable copies so a cold edge usually stopped at object storage
  • D1 tracked the key inventory needed when an image changed

Requests reached the origin only when the required variant was missing. With that path in place, image-path origin CPU fell from around 90% to below 15%, while the pipeline kept serving more than 50 million images a day.

The origin stayed in the path for misses and generation. What changed was the default: repeated work no longer landed there first.

The miss path is the interesting part

If every edge-cache miss falls through to the origin, a cache flush, a new location, or a wave of cold variants will put the same load back on the application servers you thought you had protected.

R2 was in this design for that reason: a durable copy that can warm a location after an edge-cache miss, without waking the origin each time.

ElevenLabs' write-up is a recent, public account of what happens when nothing bounds the traffic behind a cache. In their case backend services had no upper bound on cache connections, the cache cluster was overwhelmed and kept restarting empty, and queued requests spread the failure to other caching clusters. The case study and write-up for this image pipeline do not say it was tested against a burst of that size, and I am not claiming it was. The connection is the design problem, not a prediction about how this path would have behaved under the same load.

The other easy-to-miss piece is the browser. Browsers may already have a valid copy. If the path always sends the body again, you spend bandwidth and queue bytes behind useful work. Conditional requests, comparing If-None-Match with the current ETag, let a valid browser copy receive a 304 Not Modified response. The ETag has to describe the representation the browser will actually receive. A logical image identifier is not enough if the image can change without that identifier changing.

Invalidation needs an inventory

An image is usually a family of objects: the original, several sizes, several formats, sometimes a watermark. Deleting the original does not delete the derived objects. If you cannot discover every key tied to one logical image, a replacement or deletion leaves stale variants in durable storage and at the edge.

That is why D1 was in this design. It was not the image store. It was the inventory that made invalidation a normal indexed operation instead of an incident-time reconstruction.

Lifecycle operations also need a separate boundary from public reads. Cache purge and object deletion cannot be open the way image reads can. In this pipeline, a short-lived signed request from the origin to the Worker authorized those operations, with a timestamp window, a body digest, and a signature against the deployment secret.

None of that is decoration. Once you cache, correctness depends on being able to find, remove, and later refill the right objects.

What to look at in your own stack

A useful place to start is the questions this design had to answer. They apply to any team whose origin is spending a lot of CPU on repeated media delivery.

Is the origin serving the same image more than once? If image-path CPU is high and the underlying files are not changing, the origin is doing repeat work that belongs on a different layer.

After an edge-cache miss, where does the request go? If the next stop is always the origin, a cold location or a cache purge will recreate the original problem. A durable copy in object storage, checked before the origin, changes that.

When an image is replaced or deleted, can you find every derived variant? If you cannot, invalidation becomes an incident-time reconstruction, and stale variants remain in ways that are hard to see.

Who is allowed to purge or delete, and how is that request authenticated? A public read path is not a control path.

Do browsers with a valid copy receive a 304, and does the validator actually change when the bytes change?

Can you tell which layer served a given response: browser, edge cache, object storage, or origin? Without that, you cannot tell whether a slow or wrong response is a miss-path problem or an origin problem.

These checks come from the requirements this pipeline was built around. They are the difference between a cache that serves bytes and a cache that serves the correct representation after a replacement, a deletion, or a disagreement between layers.

The tradeoff is consistency

Caching reduced repeat work and added a consistency problem. The layers do not agree at the same time. A cache miss is not proof the origin deleted the image. A stale cache hit is not proof object storage is stale. The lifecycle path has to establish the order of deletion, purge, and repopulation, and it needs an explicit policy for disagreement.

The origin remains necessary for misses and for generation. Moving delivery to the edge does not eliminate that dependency. What it should eliminate is the habit of treating the origin as the place every request goes by default.

If you want the read path, the key format, and the invalidation flow, they are in the full write-up. The case study has the shorter version.

If you have a problem like this, I help businesses turn engineering problems into working systems through Developer Labs.

Bhavin Kamani

A small command line for a curious visitor.