Green Pipeline, Production Serves the Old Code
A four-person team ships a Node.js service through a CI/CD pipeline. The Dockerfile is FROM node:18, then COPY . ., then RUN npm ci, then EXPOSE 3000 and CMD ["node","server.js"]; the container runs as root. The build job runs with set -x and passes --build-arg NPM_TOKEN=..., which appears in the job log. Every build takes about 15 minutes because dependencies are reinstalled each time. Deployment builds and pushes registry.example.com/portal:latest, then runs docker compose pull && docker compose up -d on the production host. Last week a hotfix was merged, the pipeline reported success, and production kept serving the old behaviour for two days. Staging had passed. The team wants faster builds and fewer 'works in staging, wrong in production' surprises without buying new tooling.
1. Restructure the Dockerfile for layer caching and explain precisely why the current order defeats it.
Read the worked answer
Docker caches each layer keyed on the parent layer plus the instruction text. COPY . . changes whenever any tracked file changes - a README edit, a test file, a lockfile - so the cache is invalidated every build and the following RUN npm ci must run again from scratch. That is the entire 15 minutes.
Reorder so the expensive step depends only on the dependency manifest:
FROM node:18-bookworm-slim@sha256:<digest>
WORKDIR /app
COPY package.json package-lock.json ./
RUN npm ci --omit=dev
COPY . .
USER node
EXPOSE 3000
CMD ["node","server.js"]
Now source changes reuse the dependency layer and only the final COPY is rebuilt. Add a .dockerignore containing node_modules, .git, and test output so local artefacts never bloat the context or churn the cache. Use npm ci rather than npm install so the lockfile is authoritative and results are reproducible. Pin the base image by digest so the runtime cannot silently change underneath a passing build. Run as a non-root user, and drop EXPOSE-level assumptions in favour of explicit port configuration.
2. Explain why the hotfix never reached production, and what you would change so a deploy is verifiable.
Read the worked answer
From the pipeline output you cannot determine what production is running, and that is itself the finding: a mutable latest tag identifies no artefact. The pipeline pushed a tag that may or may not have moved, the host pulled a reference that may or may not have changed, and docker compose up -d only compares container configuration, so it can report success while the running container still holds older code. Nothing in the pipeline asserts which image is live, which is why the gap went unnoticed for two days.
Changes:
- Tag every image with the immutable commit SHA, for example
portal:${GIT_SHA}, and deploy that exact tag. Keeplatestonly as a convenience pointer, never as the deploy reference. - Record the resolved image digest in the build, pass it to the deploy job, and have the deploy fail if the digest running on the host differs from the digest built.
- Force the update explicitly:
docker compose pull --policy alwaysfollowed bydocker compose up -d --force-recreate. - Add a post-deploy gate: hit a
/versionendpoint and compare it with the commit SHA. A deploy that cannot be verified is not a deploy.
Staging passing is not evidence here; the artefact that reached production was never identified.
3. Fix the secret handling and describe how developers could debug failing builds without production credentials.
Read the worked answer
Build args are visible in docker history and image metadata, and set -x echoes the expansion straight into the job log, so NPM_TOKEN is compromised. Rotate it now, purge the log artefacts, and add a pre-commit/repository secret scan so it cannot recur.
Replace the build arg with a BuildKit secret mount: RUN --mount=type=secret,id=npmrc,target=/root/.npmrc npm ci, supplied via --secret id=npmrc,src=$HOME/.npmrc, which never persists in a layer. Never bake runtime secrets into images; inject them at deploy time from the platform's secret store or compose secrets with restrictive file modes. Remove set -x or move it to a narrowly scoped debug step, and use CI log masking.
For debugging without production access: publish test reports and build artefacts from failing jobs, keep logs centrally searchable, and give developers read-only access to production logs through a proxy rather than shared credentials. Ephemeral preview environments built from the same image let them reproduce issues safely, and docker build --progress=plain locally with the same pinned base image closes most staging-versus-production gaps.