Caching Sharp-Processed Images in Static Builds

A documentation site I worked on processed thousands of images on every build. Sharp converted each one to webp, measured it, and emitted several sizes. The source images almost never changed, so the work was the same every time, and CI redid all of it from scratch. The build times showed it.

The site carries about fifteen hundred images across more than five thousand pages, and CI rebuilt every one of them on every run.

The cache boundary

My first instinct was a runtime cache. That helped locally and did nothing for CI, where every run starts cold. What I actually needed was for the processed images to survive between builds, so I moved the boundary: the webp outputs and their metadata went into the repo, committed under a generated folder.

Now the build asks a simpler question. Do I already have a processed version of this image? If yes, skip Sharp and copy the cached file. If no, process it and add it.

Detecting real changes

Treating every touched file as a miss would defeat the point. I needed a hash that captured what actually changes the output.

Modified time was the first thing to throw out. Git rewrites mtime on checkout, on rebase, on anything that touches a file’s index entry, so a cache keyed on it invalidates every time someone pulls. An early version made exactly that mistake. The fix was to ignore mtime and key on path plus size instead.

Two files with the same path and size produce the same result, so they get the same hash. There is a small collision risk, two different images at the same path with the same byte length, and I accepted it: a bad hit gives you a wrong image, not a broken build, and the fix is to bump a version field and regenerate rather than debug a crash.

Hashing more would have been safer in theory, but every extra field is another way to invalidate the cache by accident. So the rule was to hash only the inputs whose change would alter the output. Source contents through size, the processing parameters, and the Sharp major version. Nothing else.

const cacheKey = (src: string, stat: Stats, params: ProcessParams) =>
  createHash('sha256')
    .update(
      JSON.stringify({
        path: relative(root, src),
        size: stat.size,
        params,
        sharpMajor: sharpVersion.split('.')[0],
        revision: 1,
      }),
    )
    .digest('hex')
    .slice(0, 16);

The CI side

CI sets an environment variable. When the build sees it, the image step short-circuits and trusts the committed cache. New or changed images get processed in their own pull request, where the cache update lands in the diff and reviewers see the new webp next to the source.

Image work went from minutes to near zero on most runs. Sharp only wakes up when an image actually changed, which is exactly when you want it to.