Swarm quorum loss after manager decommissioning
A Docker EE Swarm has five manager nodes and ten workers. During a maintenance window, an operator runs docker swarm leave --force on two managers that were being decommissioned, then reboots a third manager for kernel updates. The remaining two managers cannot form a quorum, and docker node ls on them returns Error response from daemon: rpc error: code = DeadlineExceeded desc = context deadline exceeded. The UCP controller is unavailable, but application services on workers continue running. The team has a UCP backup from 36 hours ago and a DTR backup from 7 days ago. They want to restore management with minimal application downtime and avoid a split brain. No manager has been reimaged.
1. Explain the cluster state and the safest recovery sequence.
Read the worked answer
The cluster has lost raft quorum. A five-manager swarm tolerates two failures, but two managers were force-left and a third was rebooted; if the rebooted manager has not rejoined, only two managers are active, below the quorum of three. docker node ls times out because the API requires quorum. Application containers on workers keep running because Swarm keeps local tasks, but no management operations are possible. Safest sequence: bring the rebooted manager back and confirm it has its raft data; if it rejoins, quorum may return without a restore. If it does not, stop all manager daemons, choose the manager with the most recent raft log, start only that one, and run docker swarm init --force-new-cluster. Then rejoin the other managers one at a time and restore UCP from backup. Never run --force-new-cluster on more than one node: that creates split brain.
2. What should be done with the two force-left managers, and in what order?
Read the worked answer
The two force-left managers are no longer members; their local state is stale and must not be treated as authoritative. After quorum is restored, rejoin them as managers using a fresh join token from docker swarm join-token manager, but only after wiping or validating their Docker state. Do this one at a time and wait for docker node ls to show the manager as Ready and for raft health before adding the next. Do not rejoin them while the surviving managers are still partitioned, and do not restore an old UCP backup onto a manager that has newer swarm state without reconciling. If the force-new-cluster path was used, the old managers must join the new cluster rather than restoring their previous identity. Verify services, networks, and secrets before returning the nodes to service.
3. What runbook and backup changes prevent this failure mode?
Read the worked answer
Prevent recurrence with a manager-change runbook. Never force-leave more than one manager at a time; decommission one, wait for raft health and docker node ls stability, then proceed. Keep an odd number of managers, ideally five for production, and never take two below quorum during maintenance. Back up UCP and DTR on a schedule, test restores quarterly, and record the backup timestamp in the incident channel. Add monitoring for manager count, raft leader changes, and docker node ls failures. Use docker swarm leave without --force on managers and demote to worker first where possible. Document the exact force-new-cluster procedure, including stopping other managers and selecting the latest raft log, and require a second operator to confirm before execution.