Tag mismatch and executor drift on a 14-runner fleet

Arithmetic. A job runs only on a runner that has all of its tags. The 180 jobs require docker + linux + prod, so the eligible set is 3 runners, not 14. Effective parallelism per runner is min(concurrent, limit) = min(4, 6) = 4. Two of the three are paused, leaving 1 × 4 = 4 slots. At roughly 3 minutes per job, 180 jobs need about 180 ÷ 4 × 3 ≈ 135 minutes of wall clock, so a standing backlog of 60 pending jobs is exactly what the numbers predict. The 11 idle runners are excluded by tag, not by lack of capacity - the pending message names the tag set, not saturation.

Root cause. Tags are being used as a queue partition (prod) and as a capability descriptor at the same time. The nightly matrix needs neither production access nor isolation; it inherited prod by copy-paste.

Plan.

1. Split runner pools by purpose: 11 runners tagged docker, linux for build/test work; the 3 tagged docker, linux, prod reserved for deployment jobs only. Deploy jobs keep prod, so an untagged runner can never deploy.
2. Remove prod from the 180 matrix jobs and re-run the timing to confirm the backlog clears.
3. Unpause the two runners only after confirming disk and CPU headroom for 4 concurrent Docker executor jobs each.
4. Consider raising concurrent above 4 only after checking instance CPU; oversubscribing makes each job slower and lengthens the batch.
5. Make prod runners protected so only protected branches and tags can use them.

The job is identical everywhere, so the difference is in runner configuration, not in the job. Docker-in-Docker requires the inner daemon to start inside the job container, which needs privileged = true in [runners.docker] in the affected runner's config.toml (and a kernel//dev setup that permits it). The seven working runners were provisioned with that flag; the other seven were not. A secondary cause to rule out is a mismatched DOCKER_TLS_CERTDIR / DOCKER_HOST between client and service - but that would fail consistently across all runners.

Options.

• Add privileged = true to the seven and give DinD builds a dedicated tag (e.g. dind). Simple, but a privileged container can escape to the host; these runners must be treated as untrusted: isolated ASG, no shared volumes, no credentials beyond CI_JOB_TOKEN, egress-restricted.
• Avoid privileged entirely: rootless BuildKit or Kaniko for image builds, or docker buildx against a remote builder. Slower to adopt, but removes the escape risk and lets the whole fleet stay unprivileged.

Judgement. Because the fleet also carries prod, do not enable privileged on the shared pool. Introduce a separate dind pool that carries neither prod nor deployment credentials, move the image-build jobs there by tag, and plan a migration to rootless builds.

Whichever route is chosen, make the required runner configuration part of the runner's provisioning code so the fleet cannot drift again.

The shell executor runs job commands directly on the host, as the gitlab-runner user, in a build directory on the host filesystem. That means:

• No isolation between projects. Jobs from unrelated projects share a directory tree and a home directory, so one job can read files another job left behind, including .npmrc, .aws, ~/.docker/config.json or an .env that a previous job wrote to disk.
• Cross-job contamination. Artefacts and caches land in a shared path; a dirty workspace produces failures that cannot be reproduced on a clean runner.
• Persistence. Anything it wrote, including credentials exported to files, survives the job.
• Root-owned files. With docker volumes or a root-run step, later non-root jobs cannot clean up and fail on checkout.

Remediation order.

1. Pause the runner immediately (/admin/runners or the API) so it stops accepting jobs while you investigate.
2. Identify which jobs it picked up - query the runner's jobs history - and treat any project variables those jobs used as potentially exposed. Rotate anything long-lived (deploy tokens, cloud keys); CI_JOB_TOKEN is short-lived and scoped, so it is lower priority.
3. Unregister the runner and tighten the registration token so a shell executor cannot be registered to the group again.
4. Clean the host build directory and home directory of leftovers.
5. Replace it with a Docker executor runner using a pinned image, and set GIT_STRATEGY: fetch plus a workspace clean hook so stale files never survive between jobs.

Do not simply relabel the shell runner - tags do not fix isolation.