A protected password, a base64 leak, and a variable that will not mask

Two independent filters gate this variable. The protected flag means the value is only injected into pipelines running for a protected branch or a protected tag. release/2.4 is not protected, so the variable is never defined there - the job sees an empty string and the driver reports an authentication failure rather than a missing-variable error, which is what makes it confusing. The environment scope (production) is a second filter: the job must also declare a matching environment: to receive the value, and environment scope does not bypass protection.

Other consequences of the protected flag worth knowing: merge request pipelines from forks never receive protected variables, and jobs with an empty variable usually fail late and obscurely rather than at config time.

Fix options, best first:

1. Protect the release branches (Settings > Repository > Protected branches, pattern release/*) so hotfix pipelines on those branches can read the secret. Choose who may merge/push deliberately.
2. Move the credential out of CI variables entirely: use OIDC id_tokens to obtain short-lived cloud credentials, or secrets: with an external vault keyed by environment.
3. Only as a last resort, unprotect the variable - that widens exposure to every branch and every fork-driven pipeline, so pair it with masked and scoped access if you do it.

Masking is log scrubbing, not a security control. A value can be masked only if it is a single line, at least 8 characters, contains no whitespace, and uses characters from the Base64 alphabet plus a small set of permitted specials. ! is not in that set, which is exactly why the glpat- token in incident (c) cannot be saved as masked: the checkbox is rejected, not silently ignored.

Incident (b) is the more important lesson: GitLab masks only the exact stored string. Base64, URL-encoding, JSON-escaping, a checksum, or printing half of it defeats the scrubber - and job logs are visible to everyone with report access, are archived, and are searchable.

Safer patterns:

• Never echo the value; write it to a file in $CI_PROJECT_DIR (never to artifacts: or a dotenv report) and pass the path: kubectl create secret generic db --from-file=password=./.secret.
• set +x around the command, and avoid tools that echo their arguments (kubectl with --from-literal shows the value in a failed command's error).
• Prefer federated/short-lived credentials: id_tokens: { VAULT_ID_TOKEN: { aud: https://vault.example.com } } plus secrets:vault, so no long-lived password exists in project settings at all.
• Rotate anything that has already been printed; assume the log is compromised.

Variables defined in the GitLab UI/API (project, group, instance) outrank variables defined in .gitlab-ci.yml, including a job's own variables: block. The order, highest first, is roughly: pipeline variables (trigger API, scheduled pipeline, manual run), project variables, group variables, instance variables, variables from dotenv report artifacts, job-level YAML variables, global YAML variables, deployment variables, predefined variables.

So the project-level DEPLOY_ENV=production beats the job's staging value. inherit: variables: false will not help either - it only stops the job inheriting globally defined YAML variables, not UI-defined ones.

Options that actually work:

• Pass the override as a pipeline variable when starting the pipeline (POST /projects/:id/pipeline with variables[][key]=DEPLOY_ENV), which sits at the top of the precedence order.
• Remove DEPLOY_ENV from project settings and derive it in YAML, or define environment-scoped variables for staging and production and select them by environment: name.
• Use rules to gate the job on the intended environment so a stray value cannot reach production: - if: $CI_COMMIT_BRANCH == "main" plus a protected environment with required approvals.

For a production-affecting variable, the value should carry a guard rather than relying on whoever edited the job's YAML last.