Rolling Update Stalls Under Node Memory Pressure
A three-node swarm runs a stateless API service with a single replica per node, each container limited to 512 MiB and reserving 256 MiB. A rolling update to a new image tag was issued with docker service update --image api:2.4 --update-parallelism 1 --update-delay 30s. After the first task restarts, it stays in pending for six minutes, then rejected with no suitable node. The other two old tasks keep serving. Node A reports 94% memory allocated, Node B 71%, Node C 58%. The updated image's entrypoint now spawns two worker processes instead of one, and its healthcheck takes about 90 seconds to pass under load. The team wants zero-downtime deployment and cannot add nodes this week.
1. Diagnose why the updated task cannot be scheduled and explain the interaction between reservations, limits, and the scheduler's view of the node.
Read the worked answer
The scheduler places tasks based on reservations, not limits. Each existing task reserves 256 MiB. On Node A, 94% of allocatable memory is already reserved, so a new 256 MiB reservation will not fit; the scheduler marks the node unsuitable and the task is rejected rather than merely slow.
The 512 MiB limit is a runtime ceiling and does not affect placement. A common misdiagnosis is to raise the limit; that changes nothing for scheduling. The relevant lever is the reservation (or the number of tasks per node).
Because --update-parallelism 1 stops the rollout after the first task, the service is left with one pending task and two healthy old tasks. The pending task is retried, but until capacity is freed it will keep failing on the same constraint. The double-worker entrypoint may also push real usage toward the 512 MiB limit, risking OOM kills after placement, so memory headroom must be verified, not just reservation math.
2. Propose a concrete remediation sequence that restores zero-downtime rollout without adding nodes, and justify each step's ordering.
Read the worked answer
- Stop the rollout so it does not thrash:
docker service update --update-parallelism 0 api(or roll back). A stalled update with a pending task should be paused before changing constraints. - Reclaim reservation headroom. Either lower the reservation to 192 MiB if measured steady-state usage supports it, or reduce per-node task density by letting the service spread across B and C. Verify with
docker node inspectanddocker service psthat the pending task clears. - Fix the healthcheck timing. A 90-second startup under a 30-second update delay means the orchestrator may consider the new task unhealthy before it stabilizes. Add a
start_period(or lengthen--update-delay) so the delay exceeds worst-case startup. Otherwise the rollout can pause on a false failure. - Resume with
--update-parallelism 1 --update-delay 120s, watchingdocker service psbetween steps. - Validate memory after the double-worker change with
docker stats; if usage approaches 512 MiB, raise the limit and the reservation together, since the reservation is what the scheduler honors.
Ordering matters: pausing prevents repeated rejections, capacity changes unblock placement, and healthcheck timing prevents a second stall during resume.