A security gate that never blocks, a manual deploy from a feature branch, and exit 137

Each layer of the gate fails independently, and all three must fail before a vulnerable change ships.

1. Job rules gate on the wrong pipeline type. Scanning runs only when the branch name equals the default branch - that is, after the change has already merged. Merge request pipelines never contain the jobs, so there is nothing to fail.
2. allow_failure: true neutralises the result. Even when the job runs and fails, the pipeline status stays green, so the job is advisory rather than blocking.
3. Nothing enforces the pipeline result. Unless the project or a merge request approval policy requires a successful pipeline, a green pipeline with a red advisory job merges freely.

Redesign.

sast:
  rules:
    - if: $CI_PIPELINE_SOURCE == "merge_request_event"
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
  allow_failure: false

allow_failure: false is what makes the job's failure a pipeline failure; combined with "Pipelines must succeed" or a merge request approval policy that requires the security job, merge is blocked until the finding is fixed or explicitly approved.

Hotfix path. Add a narrowly scoped exception - a hotfix/* branch pattern that still runs scanning, plus a security-officer approval rule that can be satisfied within minutes. Keep scanning mandatory; make the approval fast, not the scan optional. Record every override in the approval audit trail so it is reviewable, and require the fix as a follow-up.

Also confirm the compliance framework's injected jobs are actually in the pipeline - if the project config overrides them by name, the policy is decorative.

A protected environment does not restrict which pipelines may run a job - it restricts which users and roles may deploy to it, and only when the environment protection rules are actually evaluated. Three things combine here:

1. when: manual only means "do not auto-start". The job sits in any pipeline that includes it, including an MR pipeline, and stays playable.
2. rules do not restrict the source branch. Without a rule pinning the job to the default branch or a tag, an MR pipeline from a feature branch contains a playable production deploy.
3. Role-based environment protection passed. A Maintainer is in the allowed-to-deploy list, so the protection check succeeded - it was never designed to distinguish pipelines by ref.

Fix, in layers.

deploy-prod:
  stage: deploy
  environment:
    name: production
    deployment_tier: production
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
    - if: $CI_COMMIT_TAG
      when: manual

No MR pipeline now contains the job at all - the strongest control, because an absent job cannot be played.

Then tighten the environment itself: restrict allowed-to-deploy to a small group, add required deployment approvals, and enable "prevent unintended deployments" style branch/tag restrictions on the environment. Finally, protect the variables the deploy uses so they are only exposed on protected refs; that way even a misconfigured job has no credentials to use.

Test the negative case: open an MR from a feature branch and confirm deploy-prod is absent, not merely unplayable.

137 = 128 + 9, i.e. the process was terminated by SIGKILL. In a containerised CI job this is almost always the kernel or container runtime killing a process for exceeding available memory - an OOM kill. The consistent timing (about nine minutes, not immediately and not at a timeout boundary) fits a scanner whose memory grows with the size of the input: 2.1 GB of repository is the trigger, not the fault.

What retries fix: nothing here. retry: on runner_system_failure or stuck_or_timeout_failure will not help; an OOM kill usually surfaces as a script failure, and retrying simply burns another nine minutes and another runner before failing the same way. Retries also hide the trend from the dashboards.

Diagnose. Confirm from the runner's system logs or the hosted runner metrics that memory hit the ceiling at the moment of the kill. Check whether the job is cloning the full history (GIT_DEPTH unset means full clone in many configurations) and whether the scanner is walking vendored dependencies, generated bundles or minified assets.

Remediate, cheapest first.

1. Shrink the input: shallower clone (GIT_DEPTH: 20), and exclude vendored/generated paths from scanning.
2. Run the job on a larger runner by tag or resource_class; this is the reliable fix when the input genuinely is that large.
3. Split the scan if the tooling supports path scoping, or move the full scan to a scheduled nightly on the large runner and keep a fast incremental scan on MRs.
4. Raise the container memory limit only if the runner host has headroom - otherwise you move the OOM kill to the host and destabilise co-tenanted jobs.

Then set an alert on job duration and peak memory so the next growth spurt is caught before it kills a pipeline.