Docker Certified Associate: A production password in a public repo, an image layer, and an env var

Rotate unconditionally - two exposures have already crossed the trust boundary

1. Git history. Fourteen contributors means fourteen clones and any forks. git rm does not remove it from history or from those clones; the credential is public to that population.
2. An image layer in the registry. docker history / docker save on that tag reveals the .env contents, and the layer is in every node's local image store and any cache that pulled it. This is the one people forget - deleting the Dockerfile line does not delete the layer.

Environment-only exposure (the compose file) is also worth rotating, but the image layer makes rotation mandatory rather than prudent. Sequence: create the new secret, roll the service, rotate the DB password, then docker secret rm the old one - never the reverse, or the running tasks cannot reconnect after a restart.

What is visible where

Data	docker service inspect	Inside the container
environment: values	plaintext in the spec - anyone with API access sees it	visible in env / /proc/<pid>/environ
Swarm secret	not shown - only name/ID, target, uid/gid, mode	/run/secrets/<target>, backed by tmpfs
Image reference	whichever tag/digest you deployed	docker ps, /.dockerenv

The honest caveat: secrets are not encryption. Any process in the container that can read /run/secrets/... can exfiltrate it, and the value is materialised on the node running the task. The win is that it never appears in image layers, the compose file in git, docker inspect, or docker service inspect - a large, real reduction in blast radius, not immunity.

Create the secret out-of-band, never in the repo or in CI logs

read -rs PGPW && printf '%s' "$PGPW" | docker secret create pg_password -

compose file

secrets:
  pg_password:
    external: true          # created above; or file: ./secrets/pg_password.txt (gitignored)

services:
  db:
    image: postgres:16.4@sha256:<digest>
    environment:
      POSTGRES_PASSWORD_FILE: /run/secrets/pg_password
    secrets:
      - source: pg_password
        target: pg_password
        uid: "999"
        gid: "999"
        mode: 0440
  api:
    secrets:
      - pg_password
    # reads PGPASSFILE or cat at connect time, see below

Application side, smallest possible change. The official Postgres image already supports _FILE variables. For the app, prefer reading at connect time over re-exporting at boot:

# entrypoint.sh
export PGPASSFILE=/run/secrets/pg_password     # driver reads it only when connecting

Re-exporting into PGPASSWORD at start-up puts the value back into docker exec env - a smaller leak than before, but the same class of leak. If the driver truly requires PGPASSWORD, read it as late as possible and accept that compromise knowingly.

Constraints that will bite

• Swarm only. docker compose up (non-swarm) does not create secrets, so local dev needs file: secrets or an env fallback behind a dev profile.
• ~500 KB limit per secret - fine here, fatal for a keystore.
• Default landing spot is /run/secrets/<name> with mode 0444; set uid/gid/mode in the long syntax so the non-root process can read it (0440 above).
• A compromised node can read the secret for any task it runs. Secrets protect against accidental disclosure, not against host compromise.

Verify

docker service inspect db --format '{{json .Spec.TaskTemplate.ContainerSpec.Secrets}}'
docker exec <db-task> ls -l /run/secrets
docker service inspect api | grep -i -E 'password|postgres://'   # must be empty

Ship these together - they are all one-line changes

  api:
    image: registry.internal:5000/api@sha256:<digest>
    read_only: true
    tmpfs:
      - /tmp
    user: "10001:10001"
    cap_drop: [ALL]
    security_opt:
      - no-new-privileges:true
    # no `ports:` for db at all - overlay-only; if host access is required, 127.0.0.1:5432:5432
    deploy:
      resources:
        limits: { cpus: "2", memory: 2g }

Reasoning and verification

Control	Why	Check
user: 10001	removes the trivial root→host escalation path	docker exec <c> id
cap_drop: ALL	a web app needs none; add back NET_BIND_SERVICE only if binding <1024	grep Cap /proc/self/status in the container
read_only: true + tmpfs	an uploaded webshell cannot be written into the app dir	docker exec <c> touch /x must fail; touch /tmp/x must succeed
no-new-privileges	blocks setuid escalation inside the container	docker inspect --format '{{.HostConfig.SecurityOpt}}' <c>
digest pin	latest cannot silently become a different, unreviewed build	docker service inspect --format '{{.Spec.TaskTemplate.ContainerSpec.Image}}' api
resource limits	a runaway task cannot starve the node and take the DB with it	docker stats under load

Two things that will bite immediately:

1. read_only: true breaks images that write into their own directory at start-up (pid files, /var/cache). Run once with --read-only outside prod, read docker logs, then add the specific tmpfs or named volume - do not disable the control because the first attempt fails.
2. Postgres needs a writable data dir; give it a named volume, never a bind mount into the repo, and never tmpfs.

Judgement call: the highest-value single change here is not the secret - it is dropping root plus pinning by digest, because those two close whole classes of future incidents, while the secret fix closes exactly one already-open one.