Rolling update rollback caused by a missing health-check start period
A 20-replica service payments-api runs on a six-node Swarm in Docker EE 19.03. The service uses update_config: parallelism: 4, delay: 10s, order: start-first, failure_action: rollback, monitor: 30s. A new image is deployed at 14:00. The image starts a JVM that takes about 45 seconds to open its HTTP port, but the health check is curl -f http://localhost:8080/health with interval: 5s, timeout: 2s, retries: 3 and no start_period. At 14:02 the deployment is marked rolled back, and docker service ps shows many tasks with task: non-zero exit (137) and some with health starting. The old tasks were removed cleanly, and the service is now at 16 running replicas, with four tasks in Rejected state. The team wants zero-downtime deployment and wants rollback to happen only for genuine application failures.
1. Explain why the rollback occurred and why four tasks are in Rejected state.
Read the worked answer
The rollback is a health-check false negative. The JVM needs about 45 seconds, but the health check starts immediately; with interval: 5s, retries: 3, and no start_period, Docker can mark the container unhealthy after roughly 15 seconds. Swarm then kills it, and exit code 137 is SIGKILL. Because order: start-first and parallelism: 4, Swarm starts four new tasks while old tasks still run, so the monitor window sees failures and triggers failure_action: rollback. The four Rejected tasks likely reflect the scheduler refusing placements during rollback, often because start-first needs spare capacity for both old and new tasks and the six nodes are already reserved. Confirm with docker service ps payments-api --no-trunc and docker service inspect --pretty; look for no suitable node or reservation errors. This is not evidence of an application defect.
2. What exact service changes would make deployments reliable without giving up zero downtime?
Read the worked answer
Set a health-check start period longer than the JVM boot time and make the monitor window longer than the start period. Example: docker service update --health-cmd 'curl -fsS http://localhost:8080/health || exit 1' --health-interval 10s --health-timeout 3s --health-retries 3 --health-start-period 90s --update-monitor 3m --update-parallelism 2 payments-api. Keep start-first for zero downtime, but reduce parallelism to 2 and ensure each node has reservation headroom for one extra task. Add --reserve-memory and --limit-memory so the scheduler accounts for the doubled footprint during start-first. If capacity is tight, use stop-first with extra replicas instead; zero downtime then depends on the service absorbing the temporary loss. Do not rely on failure_action: rollback alone: a correct start period prevents rollback on slow boot, while a longer monitor catches real failures.
3. How would you validate the new update settings in staging before using them in production?
Read the worked answer
Validate in a staging swarm that mirrors node count and reservations but uses two replicas. Deploy the candidate image with --update-parallelism 1 and --update-monitor 3m, then watch docker service ps --no-trunc, docker events --filter service=staging-api, and docker inspect health status. Simulate a slow start by adding a 60-second sleep before the server binds, and confirm tasks stay starting rather than unhealthy. Then simulate a genuine failure by pointing the health check at a closed port; the update should pause or roll back only after the start period and monitor window. Test docker service rollback and record timings. Finally, rehearse in production during a low-traffic window with a canary or a single replica behind the routing mesh, and keep the previous image digest available for immediate rollback.