Why your pipeline takes 40 minutes, and the four fixes that actually work

A 40-minute pipeline costs more than build minutes. It costs the context switch: the developer opens something else, comes back an hour later, finds a failure, and the change that should have merged before lunch merges tomorrow. Ten engineers merging three times a day each turn a 30-minute saving into roughly a full engineer of recovered throughput.

The fix is almost never the CI vendor. It is these four things, in this order.

1. Measure the stages before you optimise anything

Every CI system reports per-stage duration. Pull the last 200 runs of your main pipeline and get the median of each stage. The distribution is nearly always one of:

• One dominant stage (usually tests, sometimes the image build). Go to section 2 or 3.
• Flat across many stages — this is usually setup cost repeated per job: installing dependencies, pulling images, cloning the whole repo history. Go to section 4.
• Large gap between queued and started. Not a build problem at all; you are short of executors. Fix capacity or concurrency first, because none of the other work will show up in the numbers while jobs sit in a queue.

Also get p95, not just the median. A p95 at three times p50 means flaky tests and retries, and retries are pure waste — a suite with a 2 percent flake rate across 40 jobs fails a third of pipelines for no reason.

2. Parallelise the test suite properly

Splitting tests across N jobs is the biggest single win and the one most often done badly. Splitting by file count gives you one job at 12 minutes and seven at 90 seconds, because test durations are wildly uneven.

Split by recorded duration:

# store timings from the previous run, then partition by them
pytest --splits 8 --group "$CI_NODE_INDEX" --durations-path .test_durations

Most ecosystems have an equivalent (knapsack for Ruby, --shard with a timing file for Jest, Gradle's test distribution). The goal is every shard within 20 percent of the mean.

Then stop the shards from re-doing setup: the container image, the dependency install and the database schema should be built once in a prior job and reused, not repeated eight times.

3. Cache the build, do not just cache the dependencies

Dependency caching is table stakes. The bigger win is caching compilation.

For container images, use BuildKit with a registry cache so cache layers survive ephemeral runners:

docker buildx build \
  --cache-from type=registry,ref=ghcr.io/acme/orders:buildcache \
  --cache-to   type=registry,ref=ghcr.io/acme/orders:buildcache,mode=max \
  --tag ghcr.io/acme/orders:${GIT_SHA} --push .

mode=max caches intermediate layers, not just the final one, which is the difference between a 30 second and a 6 minute rebuild after a source change.

Order the Dockerfile so the layer that changes most often is last. Copying the whole source tree before npm ci invalidates the dependency layer on every commit — this single mistake is responsible for an enormous share of slow container builds.

For compiled languages, a shared remote cache (Gradle build cache, sccache, Bazel remote cache) turns a full rebuild into a download. This is the step where ephemeral agents pay for themselves rather than costing you.

4. Do less work per commit

Most pipelines run everything on every commit because that was the simplest thing to write.

• Path filters. In a monorepo, a change to docs/ should not run the integration suite. Both GitLab (rules: changes:) and Actions (paths:) support this in a couple of lines.
• Shallow clone. fetch-depth: 1 instead of a full history clone saves 30 to 90 seconds per job on an old repository, multiplied by every parallel job.
• Split the fast gate from the slow one. Lint, unit tests and build on every push — under 10 minutes, blocking. Integration, end-to-end, security scans and performance on merge to main or on a schedule, with failures routed to the owner. A container scan that adds four minutes to every push and blocks on nothing actionable is pure tax.

What good looks like

Push to green on the blocking pipeline in under 10 minutes, p95 under 15. Full verification within an hour of merging. Flake rate under 0.5 percent, with a quarantine list that someone actually works through.

If you are at 40 minutes now, the measurement in section 1 usually takes an afternoon and tells you which of the other three sections is worth a week. It is rarely more than one of them.