Docker Certified Associate: Four replicas, four different upload directories

• --mount type=volume uses the local driver, which is node-scoped. The name uploads is unique only per node; it is not a cluster-wide identifier.
• With no docker volume create, dockerd auto-creates uploads on whichever node first runs a task. Four replicas on four nodes → four independent uploads volumes, each with its own directory in /var/lib/docker/volumes/uploads/_data on its own node.
• The scheduler places tasks freely and re-places them after any restart or node reboot. The upload is written on node-1; the next request is served by node-3's task, whose volume has never seen that file → 404. A refresh that lands back on node-1 succeeds, which is exactly why it looks like a caching or CDN bug.
• The docker run -v uploads:/app/uploads alpine ls on node-2 attached node-2's copy. It is empty because node-2's tasks happened to receive no uploads. Nothing was deleted anywhere.

Confirm (do this before proposing a fix)

docker volume ls
docker volume inspect uploads              # "Scope": "local", no device/driver_opts
docker service ps web --no-trunc           # which node runs each task
docker exec <task-on-node1> ls -l /app/uploads | head
docker exec <task-on-node3> ls -l /app/uploads | head   # different contents

The proof is two tasks of the same service showing different directory contents at the same instant.

Judgement note: docker run -v against a service's volume name is a diagnostic on that node only - it is not evidence about the service's data, and it must never be used to "restore" or clean up. Also note docker system prune --volumes on any node would delete that node's copy silently.

Option A - shared NFS volume (keeps 4 replicas, modest effort)

Declare it once in the stack file so every node materialises it with identical options:

volumes:
  uploads:
    driver: local
    driver_opts:
      type: nfs
      o: "addr=10.20.30.40,nfsvers=4.1,hard,timeo=600,retrans=2"
      device: ":/exports/uploads"

docker stack deploy runs the equivalent of the following on each node as tasks land:

docker volume create --driver local \
  --opt type=nfs --opt o=addr=10.20.30.40,nfsvers=4.1,hard,timeo=600,retrans=2 \
  --opt device=:/exports/uploads uploads

All four tasks now see one dataset, and the definition lives in version control instead of in someone's shell history.

Trade-offs: the NFS server is a SPOF and a hard dependency - with hard mounts, tasks block in D-state if it dies; with soft, writes fail with EIO. UID/GID mapping and no_root_squash are the usual second-day bug, and throughput is only adequate for profile photos.

Option B - pin the workload to one node

docker node update --label-add uploads=true node-1
docker service update --constraint 'node.labels.uploads==true' --replicas 1 web

No new infrastructure and no shared-storage dependency, but --replicas 1 means no HA and a hard scale ceiling; a node maintenance window becomes an outage. Acceptable for an internal admin tool, not for a user-facing app.

Option C - the architecturally correct answer: put objects in S3-compatible storage and keep only metadata in the database. The service becomes stateless, replicas scale horizontally, and volume topology stops being an application concern. Highest value, highest effort; usually the right target to schedule rather than to reject.

Do not try to reconcile the four local volumes with a cron rsync. That is eventual consistency with extra steps, and it corrupts under concurrent writes.

1. Check headroom first. With 4 replicas across 4 nodes, draining one leaves 3 nodes carrying 4 tasks:

docker node ps node-3
docker node inspect node-3 --format '{{json .Description.Resources}}'

If the service is pinned by label or by a node-local volume, draining will not move the tasks - stop and fix placement first.

2. Drain, so swarm reschedules before you touch the disk

docker node update --availability drain node-3

3. Stop the daemon cleanly (a hard kill mid-write risks a corrupt metadata db)

systemctl stop docker docker.socket containerd

4. Copy preserving hardlinks, xattrs and numeric ownership

rsync -aHAX --numeric-ids /var/lib/docker/ /data/docker/

5. Edit /etc/docker/daemon.json

{
  "data-root": "/data/docker",
  "live-restore": true,
  "log-driver": "json-file",
  "log-opts": { "max-size": "50m", "max-file": "3" }
}

6. Start and verify before deleting anything

systemctl start docker
docker info | grep -i 'root dir'          # must show /data/docker
docker images | wc -l ; docker volume ls ; docker run --rm hello-world

7. Bring it back

docker node update --availability active node-3
docker service ps web --no-trunc          # watch tasks land and go Running

Keep /var/lib/docker until images, volumes and a test container all check out; only then remove it.

On the log growth

• log-opts apply only to containers created after the daemon restart. The 240 GB already on disk belongs to existing containers and stays until those containers are recreated - docker service update --force web will do it, or a rolling restart does it naturally.
• The local log driver compresses and rotates by default, still supports docker logs, and is the better default than raw json-file; check any log-shipping pipeline before switching.
• For immediate relief without restarts: find /var/lib/docker/containers -name '*-json.log' -size +1G to find offenders, then truncate with : > <file> - space is reclaimed and the container keeps running.
• Put max-size/max-file in the node image or cloud-init so node-4 does not repeat this in three months.