A 1.9 GB Image and a Probe That Kills Itself
A team containerises a Node.js 20 API. The Dockerfile is FROM node:20, WORKDIR /app, COPY . ., RUN npm install, ENV DB_PASSWORD=s3cr3t-prod, CMD node src/server.js. The image is 1.9 GB, a cold CI build takes 11 minutes, and a one-line source change rebuilds dependencies every time. node_modules and a developer's .env sit inside the image because there is no .dockerignore; the process runs as root and the scanner reports 340 CVEs, mostly in devDependencies and the base image. In Kubernetes the original exec: curl -f http://localhost:3000/health liveness probe failed because curl is not installed, so the team switched to httpGet /health. That endpoint returns 503 whenever the database is unreachable, and during a 40-second migration at deploy time all six pods entered CrashLoopBackOff and stayed there. Leadership wants smaller images, faster builds, and no more crash loops.
1. Restructure the Dockerfile and build context. For each change, state which symptom it fixes and why.
Read the worked answer
Start from what each symptom actually costs you.
Layer ordering and caching. COPY . . before RUN npm install means any source change invalidates the dependency layer. Copy package.json and package-lock.json first, run npm ci --omit=dev, then copy source. That alone takes the 11-minute rebuild down to about a minute on a cache hit, because the expensive layer is only invalidated when the lockfile changes.
Multi-stage. Stage one installs all dependencies and runs the build/TypeScript compile; stage two copies only node_modules (production), build output and package.json. DevDependencies - test runners, compilers, linters - never reach the runtime image. Combined with a slim base (node:20-slim, or distroless), 1.9 GB falls to roughly 200–300 MB. That also removes most of the 340 CVEs, which live in dev tooling and the full base image.
.dockerignore. Exclude node_modules, .git, .env, test, *.md. This shrinks the build context (faster upload), prevents host node_modules overwriting the container's, and stops a developer's .env from being baked in.
Non-root. USER node before CMD, with COPY --chown=node:node. Root in a container is a privilege-escalation foothold and is rejected by most admission policies.
Signal handling. Use CMD node src/server.js rather than npm start so the app is PID 1 and receives SIGTERM directly for graceful shutdown.
2. Redesign the health checking so that a slow database migration cannot put the deployment into CrashLoopBackOff.
Read the worked answer
The root cause is a category error: liveness answered "can I reach the database?" when it should answer "is this process broken beyond self-repair?". A dependency outage is not self-repair-able, so restarting the pod cannot help - and restarting makes it worse, because CrashLoopBackOff backs off exponentially up to five minutes, so the deployment stays down long after the migration finished.
Split into three probes with different jobs. A startupProbe on a trivial endpoint (for example /healthz, returning 200 as soon as the HTTP server listens) with failureThreshold: 30, periodSeconds: 2, giving a 60-second budget so slow boots are not killed by liveness. A readinessProbe on /ready, which checks the pool can acquire a connection and the expected migration version is applied; periodSeconds: 5, failureThreshold: 3. When it fails, the pod is removed from Service endpoints and simply stops receiving traffic - no restart, and the deployment waits. A livenessProbe on /healthz that checks only process health, with a generous periodSeconds and failureThreshold, and which must not fail on downstream errors.
Then remove the coupling entirely: run migrations as a pre-deploy Job (or an initContainer) so pods start only after schema is current. Add a PodDisruptionBudget so rolling updates cannot drop below a serving capacity, and set the rollout to wait for readiness rather than assuming success.
3. The hard-coded `ENV DB_PASSWORD` is in the Dockerfile. What is the exposure, and what does a correct secret flow look like?
Read the worked answer
ENV writes the value into the image configuration, which is part of every layer's metadata. Anyone who can pull the image can read it with docker history or by inspecting the image config, and a later ENV DB_PASSWORD= or RUN unset does not remove it - history is append-only. The image is in a registry, so the credential should be treated as already compromised: rotate it now, before designing the replacement. A secret scanner over image history and the Git history is the way to find out whether other credentials leaked the same way.
Correct flow: build the image with no secrets at all, and inject at runtime. In Kubernetes, reference a Secret via valueFrom.secretKeyRef, ideally backed by an external store (Vault, AWS Secrets Manager, or External Secrets Operator) so rotation does not require a rebuild. Mount as files rather than environment variables where the library allows it, because environment variables leak into crash dumps, child processes and kubectl describe. For build-time needs such as a private npm token, use BuildKit secret mounts (RUN --mount=type=secret,id=npmrc), which are never committed to a layer.
Add guardrails so this cannot recur: a .dockerignore entry for .env, a pre-commit and CI secret scan, and a policy check that fails the pipeline if ENV names match /PASSWORD|SECRET|TOKEN|KEY/.