Pipeline That Shipped a Secret and Cannot Prove What It Deployed
A 40-service platform builds Docker images in Jenkins, tags them latest and ${BUILD_NUMBER}, and pushes to ECR. Kubernetes pulls latest with imagePullPolicy: Always. A recent scan found an AWS access key in a layer of the reports-api image: the key was copied in during npm install from a .npmrc and deleted in a later layer. The image also runs as root. There is no SBOM, no image signing, and production deploys use a long-lived Jenkins AWS key with AdministratorAccess. One approver can push to production; the approval is a checkbox in Jenkins. Base images are node:latest and are rebuilt weekly, so the same tag resolves to different digests. The scan gate was set to warn-only after it blocked a release. An auditor asks for evidence that the artifact tested is the artifact deployed, plus a path to rotate the leaked key without breaking the pipeline.
1. State the containment and remediation order after the leaked key is found. Explain why deleting the layer or rebuilding the image is not sufficient.
Read the worked answer
Contain first: disable the leaked access key in IAM, review CloudTrail for use since the image was built, and rotate anything the key could reach. Deleting the layer or rebuilding the image does not un-leak the key: anyone who pulled the image still has it, and registry history may retain it.
Then rebuild from a clean context: add .npmrc to .dockerignore, use a multi-stage build so build-time credentials never land in the final stage, and inject the token as a BuildKit secret mount rather than COPY. Purge the affected tags and, if the registry supports it, enable immutable tags and scan-on-push. Replace the long-lived Jenkins AdministratorAccess key with OIDC federation to a short-lived, environment-scoped role. Break the pipeline into: secret scan (gitleaks) that fails the build, dependency/SBOM generation, image scan with a severity threshold, sign, then deploy by digest. Fix the warn-only gate; a gate that cannot fail is not a gate. Add a pre-commit hook and a repository history scan, because the same .npmrc pattern may exist in other services.
2. Design the artifact-integrity controls that answer the auditor's question. Explain how tags, digests, SBOMs, signing, and approvals fit together.
Read the worked answer
Evidence requires an immutable link from commit to artifact to running workload. Build once, push by digest, and deploy that digest. latest plus imagePullPolicy: Always means the tested tag and the running image can differ, and weekly base rebuilds silently change the digest.
Enable ECR tag immutability, or use unique tags and always reference repo@sha256:... in Kubernetes. Generate an SBOM (Syft/Trivy) per image and attach it; sign the image with cosign and verify the signature at admission. Record provenance showing the Jenkins job, commit SHA, and builder. Bind the approval to the digest: approve reports-api@sha256:abc, not 'build 412'. Deploy the same digest to staging and production; promote, do not rebuild. Add an admission policy that rejects unsigned images, root users, and the latest tag. Store scan results and SBOMs as retained pipeline artifacts. The auditor's question then has a mechanical answer: the digest in production equals the digest signed and scanned in CI, traceable to a commit.
3. Replace the long-lived Jenkins AWS credential and tighten deploy-time authorization. What controls prevent a repeat of the leaked-key path?
Read the worked answer
Use Jenkins OIDC federation to assume an AWS role per environment; no long-lived AWS keys in Jenkins. The role should allow only ECR push to specific repositories, S3 for artifacts, and Kubernetes deploy via a scoped service account, not AdministratorAccess. Separate build and deploy roles, and require a second approver for production. Short-lived credentials remove the class of leak found in the image.
For Kubernetes, replace cluster-admin with namespace-scoped RBAC and use workload identity for pods. Add admission control (OPA/Kyverno) that blocks root, latest, missing resource limits, and unsigned images. Rotate the leaked key and the Jenkins credential store; assume any secret in the credential store is suspect. Add secret scanning in CI and pre-commit, plus a periodic scan of ECR for secrets. Log every assume-role event with the commit and digest. Finally, rehearse rotation: a key rotation that requires a pipeline outage will be avoided, so make it a one-command, tested procedure.