I85PREP
⌘ K
DCA study materials
DCA study resources

Docker Certified Associate: The rollout that parked itself at 50%

DCA · Put your knowledge to work in context.

All case studies

The rollout that parked itself at 50%

A swarm has three managers and two workers. Service checkout runs 8 replicas of checkout:1, published on port 443. Image checkout:2 returns 503 on /healthz for about 25 seconds after process start, then 200. The service was created with --health-cmd "curl -fsS http://localhost:8080/healthz" --health-interval 5s --health-retries 3 and otherwise default update settings. Running docker service update --image checkout:2 stalls: docker service ps shows new tasks cycling starting → running → Rejected, and the service sits at 4 tasks on v1 and 4 on v2. docker service rollback checkout also pauses, this time with v1 tasks rejected.

1. Diagnose the situation: why did the forward update pause, and why did the rollback pause too? Name the specific parameters and defaults responsible.

Read the worked answer

What swarm does during an update

  • docker service update replaces tasks under the service's UpdateConfig. Defaults: parallelism: 1, delay: 0s, order: stop-first, monitor: 5s, failure_action: pause.
  • A replacement task must reach Running and pass its health check. A task that fails --health-retries (3) at --health-interval (5s) is killed and retried.

Why v2 was rejected

  • --health-start-period defaults to 0s, so the probe counts from the first second of container life.
  • checkout:2 returns 503 for ~25s. The task gets 3 × 5s = 15s of grace, is declared unhealthy, restarted, and never converges. docker service ps --no-trunc shows Rejected / task: non-zero exit entries.
  • Swarm hits the failure action: pause. That is exactly the 4/4 split observed.

Why the rollback paused as well

  • A rollback replays the previous spec through the same machinery - and the spec still carries the aggressive health parameters. Cold-starting v1 tasks under simultaneous restart also miss the 15s window.
  • Nothing is wrong with v1 or the registry. The health check is the defect, and it applies in both directions. This is the key judgement: don't debug the image, debug the probe.

Confirm before changing anything

docker service ps --no-trunc checkout | head -20
docker service inspect --format '{{json .UpdateStatus}}' checkout
docker inspect --format '{{json .State.Health}}' <container-id>
docker service logs checkout --tail 50

UpdateStatus.Message will state the update was paused due to failure or early termination of a task.

2. Give the command sequence to converge on v2 without a second outage, and justify the ordering of the changes.

Read the worked answer

One update, not three. Every docker service update starts a new rollout, so bundle the image, the health fix and the update policy into a single invocation:

docker service update \
  --image checkout:2 \
  --health-cmd "curl -fsS --max-time 3 http://localhost:8080/healthz" \
  --health-interval 10s \
  --health-retries 3 \
  --health-start-period 45s \
  --update-order start-first \
  --update-parallelism 2 \
  --update-delay 20s \
  --update-monitor 60s \
  --update-failure-action rollback \
  checkout

Why each piece

  • --health-start-period 45s - failures during warm-up no longer count against retries; 45s > 25s with margin. This alone would have prevented the original hang.
  • --update-order start-first - the new task is scheduled, started and healthy before the old one is removed, so capacity never dips below the current level. Requires a free slot; fine with ingress-published ports.
  • --update-parallelism 2 + --update-delay 20s - bounds blast radius and lets the routing mesh converge between waves.
  • --update-monitor 60s - how long swarm watches a task after it is healthy before advancing, so slow failures surface inside the same slot.
  • --update-failure-action rollback - if it still fails, swarm reverts automatically instead of parking half-deployed at 3 a.m.

Verify

docker service ps checkout
docker service inspect --format '{{.UpdateStatus.State}}' checkout   # completed
docker service rollback checkout                                     # manual escape hatch

If start-first cannot schedule (no headroom), fall back to stop-first with parallelism 1 - a slower rollout is better than leaving the service split.

3. What should go into the service definition or CI pipeline so the *next* bad release fails fast and visibly instead of hanging halfway?

Read the worked answer

Treat the health check as part of the artifact, not an afterthought.

  1. Model the start-up reality. start_period must exceed real warm-up (DB migrations, cache fill) with margin; interval × retries should be a genuine failure budget, not 15 seconds. Separate liveness (/livez, is the process wedged?) from readiness (/readyz, can it serve?) and gate the update on readiness.
  2. Test the probe in CI. Start the built image, sleep past warm-up, and curl --fail the health endpoint. A probe that fails on a good image is a release blocker, not a runtime surprise.
  3. Make the policy explicit in the compose/stack file, not left to defaults:
    healthcheck:
      test: ["CMD", "curl", "-fsS", "--max-time", "3", "http://localhost:8080/healthz"]
      interval: 10s
      timeout: 3s
      retries: 3
      start_period: 45s
    deploy:
      update_config:
        order: start-first
        parallelism: 2
        delay: 20s
        monitor: 60s
        failure_action: rollback
  1. Add an early-warning layer: a canary service with a separate DNS name or port at 1 replica, promoted only after it stays healthy; or alert on UpdateStatus.State != completed and on rolling restart counts, so a paused update pages someone within minutes.
  2. Make failure loud: --restart-condition on-failure with a --restart-delay, plus a log/metric check on 5xx from /healthz, so a slow-burn regression is caught before the next deploy.