A job stuck pending, two deploys racing, and a Docker daemon that is not there

Pending means no runner has picked the job up; it is a scheduling problem, not a script problem. Read the failure text first, because GitLab tells you which case you are in: "no runners that match all of the job's tags" is a tag mismatch; "no active runners online" means runners exist but are paused, offline, or have no capacity left.

Checklist, in order:

1. Tags. A job with no tags: can only run on a runner with run_untagged = true. docker-a has run_untagged = false and k8s-small is tagged, so the only eligible runner is the group runner - and if that runner is paused, at its limit, or belongs to a group the project is not in, the job waits forever.
2. Capacity and limits. Runner limit, instance concurrent, and queued jobs on shell-legacy (concurrent = 1) can serialise everything behind one long job.
3. Protected status. A protected runner only accepts jobs on protected branches; an unprotected feature branch job waits indefinitely with no obvious message.
4. Rules. Confirm the job exists in the pipeline at all - a rules mismatch removes it from the pipeline instead of queueing it.

Fix: be explicit. Add tags: [docker] to unit so it lands on a known runner, and keep run_untagged = false on project-specific runners so future untagged jobs cannot silently monopolise them. Tagging is the durable fix; flipping run_untagged on every runner is how tag hygiene collapses.

Use resource_group on the job. Jobs that share a resource group never run concurrently: the second one waits until the first finishes, then proceeds.

deploy:prod:
  stage: deploy
  resource_group: production
  environment:
    name: production
  script:
    - ./migrate.sh
    - ./deploy.sh

It is keyed by name and scoped to the project, so the same name in two projects does not serialise across them - pick names per real-world resource (production-db, production). Note it serialises, it does not cancel: a stale pipeline will still deploy after the newer one, in arrival order.

What it does not solve, and what to layer on:

• Stale pipelines deploying at all. Add interruptible: true and enable auto-cancel of redundant pipelines, or require a manual when: manual gate on the deploy job.
• Unauthorised deploys. Use protected environments with allowed users/approvals so the job cannot start without a human.
• Migration safety itself. Run migrations as a separate serialised job with an advisory lock in the database; resource_group only guarantees GitLab does not start two of them at once.
• Runner capacity. shell-legacy at concurrent = 1 already serialises, which is why the deadlock was surprising - the two jobs ran on different runners, or two runner processes were started from the same config.

docker build is talking to a Docker daemon that does not exist on that socket. In a docker-executor job, docker is only the service hostname if a dind service is declared, and current docker:dind images require TLS by default on port 2376 - with TLS enabled, the unencrypted 2375 listener is off, so the client's connect attempt is refused.

build:image:
  tags: [docker]
  image: docker:24-cli
  services:
    - name: docker:24-dind
  variables:
    DOCKER_HOST: tcp://docker:2376
    DOCKER_TLS_CERTDIR: "/certs"
    DOCKER_TLS_VERIFY: 1
    DOCKER_CERT_PATH: "$DOCKER_TLS_CERTDIR/client"
  before_script:
    - docker info

Two prerequisites people miss: the runner must have privileged = true under [runners.docker] in config.toml (registration flag --docker-privileged), and DOCKER_TLS_CERTDIR must be identical for the client and the service so the shared cert volume resolves to the same path.

If privileged mode is not available - typical on Kubernetes executors and shared SaaS runners - do not try to force dind. Use a daemonless image builder: Kaniko, Buildah, or docker buildx against a remote BuildKit daemon. These work unprivileged (Kaniko and rootless Buildah with appropriate settings) and avoid handing every job root on the host.