Masked is not secret: two leaks and a four-month-old access key

Masking is a log scrubber, not an access control. GitLab rewrites literal occurrences of the variable value in the job trace to [MASKED]. It does not stop the job from reading the value, piping it to a command, writing it to a file, or sending it anywhere. echo ... | base64 produces different bytes, so there is no literal match and nothing to scrub. The same is true of URL-encoding, JSON-escaping, splitting the string, printing a substring, or cat-ing a file the secret was written into. Masking also refuses values that are multi-line, shorter than the minimum length, or contain characters outside its supported set - those are stored unmasked, often without anyone noticing.

So masking is defence-in-depth against accidental env dumps, not a control against a careless or malicious script. CI_DEBUG_TRACE=true is the same class of hazard: it expands the variable list into the trace and turns a routine job into an exfiltration surface.

Remove the stored key. Use OIDC:

deploy:prod:
  id_tokens:
    AWS_ID_TOKEN:
      aud: https://gitlab.example.com
  script:
    - export $(aws sts assume-role-with-web-identity --role-arn $ROLE_ARN \
        --role-session-name gitlab-$CI_JOB_ID \
        --web-identity-token $AWS_ID_TOKEN \
        --query Credentials --output text | ...)

The job receives short-lived credentials bound to the project, branch and ref, with a CloudTrail trail per assumption. There is no long-lived secret to mask, rotate or leak. Where a static secret is unavoidable, pull it at runtime from a secrets manager via secrets: / Vault rather than storing it in project settings.

These are three independent attributes and teams routinely assume one implies another.

• Masked - log scrubbing only. It changes how the value appears in the trace; it does not change who receives the value.
• Protected - a delivery restriction. The variable is only exposed to pipelines running on protected branches or protected tags. A hotfix/refund branch is not protected, so a protected variable is simply absent from that job's environment.
• Environment scope - a matching rule on the job's environment:. A variable scoped to production is only injected into jobs that declare that environment, which keeps staging credentials out of unrelated jobs.

STAGING_DB_PASSWORD failed on protection, not on masking. The branch was unprotected, so the value was delivered to the job and env | sort printed it. Masking would have turned the plain text into [MASKED], which hides the symptom while the job still runs with the credential on an arbitrary branch - anyone with Developer access can add a job that transforms it.

Correct configuration for a staging credential: Protected and Masked, scoped to the staging environment, with deploy:staging also gated on a protected branch and a protected environment. Then an unprotected branch gets nothing to leak, an accidental env dump is scrubbed, and an unrelated job cannot see the value at all.

Note the operational consequence: protected variables are unavailable to pipelines on unprotected branches or on scheduled pipelines that run against them. That is the intended behaviour - design the pipeline around it rather than unprotecting the variable to make a broken job green.

Containment first, forensics second. Revoke or deactivate the access key in IAM immediately - do not wait to finish the investigation. Check what the key was permitted to do and whether the same role is reachable any other way.

Reconstruct the timeline.

1. Search cloud audit logs by access key ID for the first anomalous use and everything since; compare against the log of the hotfix/refund pipeline.
2. Use GitLab audit events to confirm when CI_DEBUG_TRACE was toggled and by whom, and whether variables were edited around that window.
3. Enumerate every pipeline that ran on unprotected branches in the affected period and which variables it received.
4. Assume the secret is public: rotate the AWS key and STAGING_DB_PASSWORD, and check whether either appears anywhere else (other projects, group-level variables, CI/CD variables inherited through include).

Prevent recurrence.

• Replace the static key with OIDC / id_tokens and a role that trusts the project, restricted to protected refs. Short-lived credentials make this class of leak self-limiting.
• Mark STAGING_DB_PASSWORD Protected + Masked and scope it to the staging environment; audit all variables in the group for the same mistake.
• Enforce Secret Detection and secret push protection so a committed credential is blocked before it reaches a branch.
• Remove CI_DEBUG_TRACE from templates and forbid project-level overrides; add a check or policy that fails the pipeline configuration if it is enabled.
• Add a job-level lint step that flags env, printenv, set -x and echo $SECRET patterns in .gitlab-ci.yml before merge.
• Grant Maintainer on the project sparingly; variable editing is a privileged operation.