Disk pressure on a UCP worker with pending service tasks
A Docker Enterprise 3.0 cluster runs UCP 3.0.12 and DTR 2.7.8 on 12 nodes: three managers that also run workloads and nine dedicated workers. DTR has two replicas and stores images on a local volume on each replica. Over a weekend the CI pipeline pushes 180 new tagged images. On Monday, worker-07 reports no space left on device; its Docker data root holds 92 GB of images and 41 GB of stopped containers on a 250 GB filesystem. The affected service is a stateless API with 24 replicas and a restart policy of any. UCP reports managers healthy, but docker service ps shows six tasks pending. The team does not want to delete images still referenced by running services and wants a repeatable cleanup policy. DTR garbage collection runs weekly and retains the last five tags per repository.
1. What is the most likely cause of the six pending tasks, and what would you check before changing anything?
Read the worked answer
Disk exhaustion is the likely trigger, but pending tasks are a scheduler state, not a disk state. First confirm which tasks are pending and why: docker service ps api --no-trunc shows the task error, such as No suitable node or a failed image pull with no space left on device. Then inspect the node with docker node inspect worker-07 --pretty, df -h /var/lib/docker, and docker system df -v. Test whether the daemon can still pull: a docker pull on that node should fail if the filesystem is full. Run docker events --filter node=worker-07 to see whether the daemon is emitting disk-pressure errors. Do not prune yet. Capture the six error strings first, because a scheduling constraint and a pull failure require different fixes. Also check whether the 24-replica service has reservation headroom elsewhere, since pending tasks may simply have nowhere to go.
2. Design a remediation that frees space without breaking running services, and state the order of operations.
Read the worked answer
Free space in this order. First remove stopped containers with docker container prune; they consume 41 GB and are not needed by running tasks. Second remove unused images only: docker image prune -a --filter until=168h deletes images not referenced by any container and older than seven days; running containers keep their images. Third, if space is still tight, drain worker-07 with docker node update --availability drain worker-07, let its replicas reschedule, run the prunes, verify with df -h, then uncordon. Draining is disruptive but avoids killing tasks mid-request. Before draining, confirm the other workers can absorb the replicas by checking docker node ls and service reservations. Do not delete DTR volumes on the worker; DTR data lives on its replica nodes, and touching registry storage is not the fix for a worker image cache.
3. Which policy and operational changes would prevent this from recurring?
Read the worked answer
Make cleanup routine rather than reactive. On each worker, schedule docker container prune -f --filter until=24h and docker image prune -a -f --filter until=168h through cron or a systemd timer; these preserve images used by running containers. Configure DTR garbage collection more frequently, for example nightly, and set tag retention to match the release policy. DTR GC only affects registry storage, not worker image caches, so both controls are needed. Add disk alerts at 70 percent and 85 percent, and put /var/lib/docker on its own volume or enlarge it. Use node labels and reservations so large images land on workers with capacity, and avoid placing stateful DTR replicas on workers with high-churn CI workloads. Document the drain, prune, and uncordon sequence in the runbook and test it in staging before relying on it during an incident.