Monorepo merge semantics: shared template, silent overrides, duplicate pipelines
Acme Pay runs a monorepo with 38 services. The root .gitlab-ci.yml includes templates/node.yml and then every services/<name>/.gitlab-ci.yml through include: local - 39 include entries, none of them with rules. templates/node.yml defines .node_base with image: node:20-alpine, a cache block keyed on package-lock.json, and script: [npm ci, npm run build]. Each service file defines jobs named build, test and deploy that use extends: .node_base and declare their own script:. Since four legacy repositories were merged into this one, three things broke: service builds now die with sh: vite: not found; the ledger pipeline runs the API service's test command, and its deploy job targets the API; and every push produces two pipelines that each execute all 120 jobs. The team has one week and no appetite for splitting the repository apart again.
1. Explain why service jobs fail with `sh: vite: not found` even though `.node_base` contains `npm ci`, and give two correct YAML fixes with their trade-offs.
Read the worked answer
extends performs a reverse deep merge of hashes, but arrays are replaced wholesale - never concatenated. .node_base declares script: [npm ci, npm run build]; api:build declares script: [npm run build:docker]. The merged job keeps only the service's one-element array, so npm ci never runs, node_modules is absent, and the service's own build step cannot find vite. The confusing part is that image, cache and before_script from the base still applied, so the job looks correctly inherited.
Fix A - move shared setup out of the array the service overrides. before_script is not touched by these services, so it survives the merge:
.node_base:
image: node:20-alpine
cache:
key:
files: [package-lock.json]
paths: [node_modules/]
before_script:
- npm ci --prefer-offline
The service then only declares its own script. Trade-off: any service that later defines its own before_script replaces this one entirely - keep it minimal.
Fix B - !reference when ordering inside script genuinely matters:
api:build:
extends: .node_base
script:
- !reference [.node_setup, script]
- npm run build:docker
Put the referenced commands in a plain hidden job (.node_setup) that does not itself rely on extends, because !reference copies the raw key before that job's own inheritance is resolved.
Recommend A for 38 services, B only where two commands must stay in one script list.
2. The `ledger` pipeline runs the API's test command and its deploy targets the API. Diagnose the mechanism and give two fixes, including why gating the includes on a `$SERVICE` variable will not work.
Read the worked answer
GitLab merges all included files into one flat configuration with a single namespace of job names. All 38 service files define build, test and deploy, so 38 definitions collide; when two files define the same top-level key, the later definition wins, and the winner is determined by the order of the include: entries. The ledger pipeline is not running ledger's jobs at all - it is running whichever file happens to be included last.
Fix 1 - unique job names. Prefix per service (ledger:test, api:test) or use spec:inputs to parameterise one template so each inclusion produces distinct names. This is the durable fix: job names are the only identity mechanism, and collisions are silent.
Fix 2 - gate the includes. include: rules is evaluated at pipeline-creation time, before any job, variables: block or extends exists. A condition such as if: $SERVICE == "ledger" therefore never matches, because $SERVICE defined inside the configuration is not available yet. Only predefined variables ($CI_COMMIT_BRANCH, $CI_PIPELINE_SOURCE, $CI_MERGE_REQUEST_*), project/group/instance variables and exists: can be used there. So either move the gating into per-job rules: changes: (which also gives path-based skipping inside one large pipeline), or drive it from a pipeline variable passed by the trigger/API.
Do not rely on include order - it is an implicit contract that the next contributor will break.
3. Each push creates two pipelines that run the same 120 jobs. Write the `workflow:rules` block, state exactly which pipelines survive, and state what the change does not fix.
Read the worked answer
The duplicate comes from the branch pipeline and the merge request pipeline both being created for the same commit; without a workflow: block nothing suppresses either one.
workflow:
rules:
- if: $CI_PIPELINE_SOURCE == "merge_request_event"
- if: $CI_COMMIT_BRANCH && $CI_OPEN_MERGE_REQUESTS
when: never
- if: $CI_COMMIT_BRANCH
Evaluation is top-down, first match wins. Rule 1 keeps every merge request pipeline. Rule 2 drops branch pipelines for branches that already have an open merge request - that is the duplicate. Rule 3 keeps branch pipelines where no MR exists, which preserves pipelines for the default branch and for pushes to branches without an MR.
Two things this does not fix. First, job-level rules that only allow $CI_PIPELINE_SOURCE == "push" will now silently drop those jobs from MR pipelines, because workflow:rules decides which pipeline exists and job rules decide which jobs enter it - both must be updated together. Second, it does not reduce the 120-job graph; with no path filtering, a one-line change still builds all 38 services. Combine this with per-service rules: changes: so the surviving pipeline is also smaller.
Verify after merging by checking that a push to an MR branch produces exactly one pipeline and that the default branch still produces one on merge.