Unsigned images and vulnerable base images in a DTR/UCP pipeline
A regulated team runs Docker EE 3.0 with UCP and DTR 2.7. DTR has image scanning enabled and a policy that blocks deployment of images with critical vulnerabilities. The CI pipeline builds registry.example.com/finance/ledger:latest and pushes it with docker push using a service account. The pipeline has no Notary signing step. A production deploy via docker service create succeeds, but UCP audit logs show the image was not signed. After a base-image CVE is announced, DTR scans the image and marks it with three critical CVEs. The team wants to enforce signed images and prevent vulnerable images from running, but also needs an emergency path for a zero-day fix. They use UCP client bundles and a single DTR replica.
1. Why did the deploy succeed despite the DTR policy, and what must change?
Read the worked answer
The deploy succeeded because DTR scanning and content trust are separate controls, and neither was enforced at the point of deployment. DTR can scan and mark an image, but UCP must be configured to block pulls or runs based on scan results; otherwise a node with a cached copy, or a pull completed before the scan, can start the service. Content trust is also opt-in: without DOCKER_CONTENT_TRUST=1 in CI and a signed tag, docker push and docker service create do not require Notary signatures. The mutable latest tag makes this worse. Confirm with docker trust inspect registry.example.com/finance/ledger:latest and the DTR scan report. The fix is to enforce both controls: configure UCP to require signed images and block vulnerable images, and set DOCKER_CONTENT_TRUST=1 in the pipeline.
2. Design the CI and emergency-path changes that enforce signing without blocking urgent fixes.
Read the worked answer
CI should build an immutable tag such as the git SHA, rely on DTR scanning or run docker scan, sign with docker trust sign registry.example.com/finance/ledger:<sha>, and push with content trust enabled. Configure a Notary delegation for the CI service account so it can sign without the root key, and store the delegation key in a secret manager. Deploy by digest, not latest: docker service update --image registry.example.com/finance/ledger@sha256:<digest>. For emergencies, define a break-glass delegation with a separate key held by the security lead, require a second approver, and allow a time-bound DTR policy exception only for the specific CVE and image digest. The hotfix must still be built from a patched base and signed; the exception should not permit unsigned images. Record the exception and revoke it after the fix.
3. What do you do about containers already running the vulnerable image?
Read the worked answer
First identify running tasks by image digest using docker service ls, docker service ps --no-trunc, and docker inspect on tasks. Build and sign a patched image, then perform a rolling update with docker service update --image ... --update-parallelism 1 --update-delay 30s; this avoids killing all vulnerable containers at once. If DTR blocks the pull because of scan policy, the break-glass exception must be scoped to the new digest, not the old one. After the update, confirm no task references the old digest with docker service ps --no-trunc and docker image ls --digests. Keep the old image for rollback only if policy allows it; otherwise remove it from the registry after the incident. Finally, add a CI gate that fails builds with critical CVEs so the same image cannot be rebuilt and redeployed.