Cutting Backend CI by 76%

A Django backend I work on ran CI in a median of 18.5 minutes, p90 22.3. It now runs in 4.5, p90 8.0, measured over 90 runs on the main branch.

Almost none of that came from making a test faster. That is worth saying first, because the instinct when CI is slow is to go hunting for slow tests. There were slow tests. They were not the problem.

The time went on work the pipeline repeated because nobody told it not to, work it ran in sequence because that was the default, and one resource limit nobody had written down.

Where the time went

Two costs dominated, and neither was a test.

Every job replayed all 843 migrations in sequence across four database aliases before running anything, about 4:20 each time. That was paid on every job of every run and had nothing to do with the change being tested. The schema those migrations produce depends only on the migration files, and those change a few times a week.

The second was a legacy suite of 2035 tests taking about 15:30 on a single two-vCPU runner. There was no speeding that up in place, because the limit was a core count rather than a function. You cannot profile your way out of not having enough cores.

Repeated work that need not repeat, and sequential work that need not be sequential. Different problems with different fixes, and neither one shows up in a profiler pointed at the tests.

Stop running what you do not need

The cheapest minutes came from work that should never have started.

The repository holds a frontend tree the backend tests never read, so the backend jobs now check out without it. Change detection runs per job, so a pull request touching one subsystem does not pay for the others, and the change-detection gate skips checkout entirely, because working out what changed does not need the files. A run that touches nothing relevant finishes in about 1.6 minutes, most of which is scheduling.

I also moved performance profiling and the N+1 query report off the pull request path and onto a nightly schedule. Both are useful. Neither answers a question you need answered before merging. Every check on that path is a tax on every engineer on every push, and the bar for staying there should be that somebody would act on it now.

The last fixed cost was tool bootstrap. Our runner image was rebuilt to include the language toolchain and build headers each job had been installing for itself. Dropping those steps saved 15 to 25 seconds a job. On its own that is nothing. Times six shards plus the nightly job, on every run, it stops being nothing. Fixed costs are worth attacking exactly when you have fanned out, because fanning out multiplies them.

Cache the schema, shard the suite

The migration replay became a cache. Dump the fully migrated schemas, key the cache on the contents of the migration files, restore before the run. Restore takes about 2.4 seconds against the 4:20 it replaces, and a cache miss rebuilds by migrating the four aliases at once instead of one after another, so even the bad path beats the old good path.

Keying on file contents rather than a version string matters. The cache invalidates exactly when the schema could have changed and never otherwise. I put that hash in one module so the test fixtures and the cache scripts compute it the same way. Two places deciding separately whether the schema changed is how you get a cache that is quietly wrong.

The legacy suite became a matrix. The detail I would keep in any version of this is that the sharding runner does nothing unless an environment variable is set, so running tests locally is exactly what it was before. A speedup that changes how everyone works locally is not free, whatever the CI graph says.

My first attempt split the shards round-robin over discovery order, which assumes every test class costs the same. They do not, and a matrix finishes when its slowest shard finishes, so an unbalanced split wastes most of what sharding bought. I replaced it with longest-processing-time bin packing over recorded per-class timings. Then I found the timings left out setUpClass and setUpTestData, which for database-heavy classes is most of the cost, and folded those in. Balancing on a measurement I had not checked was its own small lesson.

The limit nobody had budgeted

The last piece was a flake rather than a speedup. The suite would die every so often on “sorry, too many clients already”.

The cause was arithmetic. Automatic worker detection fanned out to roughly 55 workers, each holding a connection to all four aliases, against a Postgres container capped at 100. It passed whenever the timing happened to stay under the cap, which is the worst kind of green: not a broken test, not a reliable failure, just a coin flip that got worse as the suite got faster.

The fix took four passes, and the order mattered more than where it ended up. Cap the workers to stop the bleeding. Measure actual peak connections under load instead of reasoning about what they should be. Raise the server limit now that the real number is known. Then raise the workers again against the new limit, and settle one step below the fastest setting, because the fastest one traded a few seconds of median for a tail-latency problem.

Workers times aliases times connections per alias has to fit under the limit. That is a capacity plan, and it belongs in a comment next to the number it justifies, because the next person to see a green pipeline and a spare core will raise it back.

What I learned

Look for repeated work and unbudgeted resources before looking for slow code. The wins here were a cache, some deletions, a bin-packing problem and a connection budget, and not one of them touched a test.

The habit underneath is asking what a pipeline pays for on every run that only changes weekly, and what it pays for that nobody reads. Migration replay was the first. Profiling on every pull request was the second. Both had been there long enough to look like the cost of doing business.