A 41-minute pipeline with a 3% cache hit rate and 1.9 GB of artifacts

Why 3%. key: $CI_COMMIT_SHA is unique per commit, so each commit writes a brand-new cache entry and restores from a key that has never been populated. The only hits are retries of the same commit - hence a low single-digit rate. The key design guarantees a miss.

Second problem: the wrong thing is cached. npm ci deletes node_modules before installing, so a restored node_modules is discarded anyway. Caching it costs upload and download time for no benefit, and cached native modules can be wrong for the runner architecture. Cache the package manager's download cache instead.

.node_base:
  variables:
    npm_config_cache: $CI_PROJECT_DIR/.npm
  cache:
    key:
      files:
        - package-lock.json
    fallback_keys:
      - deps-$CI_DEFAULT_BRANCH
    paths:
      - .npm/
    policy: pull-push

key:files hashes package-lock.json, so the cache is reused across every commit until dependencies actually change, and fallback_keys gives a partial warm start on the first pipeline after a lockfile bump. Give read-only consumers (lint, unit, e2e) policy: pull so they never re-upload the same archive.

Distributed cache. With six autoscaling runners, a locally stored cache is per instance and disappears when the instance is replaced. Confirm the [runners.cache] S3 block is present and identical on all six, otherwise hits will look random. Treat the cache as best-effort: a miss must only cost time, never correctness - never put build output that must be reproducible in the cache.

Current cost: 1.9 GB per pipeline, retained 30 days. On a busy repository with, say, 40 pipelines a day that is roughly 1.9 GB × 40 × 30 ≈ 2.3 TB of standing storage, plus every job that downloads it.

Separate two different needs. Reproducibility of a released build and convenience for the next job are not the same requirement. Split them:

1. Default retention short. expire_in: 1 week on the build job; the vast majority of pipelines are never inspected after a week.
2. Long retention only for releases. Add a second upload - or a job rules-gated on tags - that keeps the artifact longer (or better, publish the release bundle to the package/container registry, where it is versioned and independently retained).
3. Shrink the payload. artifacts: exclude: the source maps (.map) into a separate job that only runs when someone needs to debug production; that alone typically removes the majority of the volume. Check whether coverage/ and test reports are being uploaded by every job.
4. Stop pulling it everywhere. Jobs download artifacts from all previous stages by default. e2e and deploy-review should use needs: with explicit job names (or dependencies:) so each pulls only build's artifact - and lint/unit should pull nothing at all.
5. Do not download when not needed. A job that only needs the artifact on failure does not need it on success.

Time arithmetic. Today the critical path is install 9 + build 6 + e2e 14 = 29 minutes of serial work, plus repeated npm ci and the 1.9 GB transfer. With needs:, lint and unit start as soon as install finishes rather than waiting for build, removing about 6 minutes from their path; a trimmed artifact cuts the e2e download substantially. Keep when: on_success semantics in mind - needs does not change whether a job runs, only when.

340 tests at 14 minutes is a clean sharding target.

e2e:
  needs: [build]
  parallel: 8
  script:
    - npx playwright test --shard=$CI_NODE_INDEX/$CI_NODE_TOTAL

parallel: 8 creates eight jobs named e2e 1/8 … e2e 8/8, each with CI_NODE_INDEX (1-based) and CI_NODE_TOTAL set. Playwright's --shard flag consumes exactly this shape, so no custom splitting logic is needed. Ideal wall clock drops from 14 minutes to roughly 2 minutes, but only if the shards are balanced - Playwright shards by test file, so a single 4-minute spec file caps the gain.

Runner constraint. Six runners at concurrent = 4 gives at most 24 slots, but lint, unit and build are also competing. Eight e2e shards plus the rest of the graph is comfortable; do not set parallel: 24 and assume it is free - every shard that cannot be scheduled is a queued job, and queueing shows up as wall clock. Also account for the artifact download: eight shards × the build artifact means eight simultaneous pulls, which is another reason to shrink dist/ first.

Costs to watch.

• Use parallel:matrix if you later need browsers crossed with shards; the job count multiplies.
• Flaky tests become eight times more visible; quarantine them before sharding, or one bad spec fails a different shard on every run.
• Do not add resource_group here - it serialises jobs and would defeat the point.
• Keep a single non-sharded nightly run of the full suite so a sharding bug cannot hide a test that never executes.