Safe Ansible rollout with apt lock contention and load balancer draining
An Ansible control node manages 200 Ubuntu 22.04 web hosts. The playbook deploy.yml uses serial: 20, a template task for /etc/nginx/conf.d/app.conf, a systemd task that restarts nginx, and a handler. During the last run, 34 hosts failed at the apt task with 'Could not get lock /var/lib/dpkg/lock-frontend'; 12 hosts restarted nginx while the load balancer still had them in rotation, causing 502s; and 3 hosts stayed drained after a later task failed. The load balancer health check calls /health and expects HTTP 200 within 5 seconds. The change window is 30 minutes, and no more than 5% of hosts may be out of rotation at once. The playbook has max_fail_percentage: 10 and no any_errors_fatal. The team wants the playbook to be safe to rerun after a partial failure.
1. Why did the apt lock failures occur, and what playbook changes make the apt step retryable and idempotent?
Read the worked answer
The lock failures are contention with another dpkg user, usually unattended-upgrades or a previous Ansible run, not a broken repository. max_fail_percentage does not wait for a lock and does not make the apt task idempotent; it only controls when a batch aborts. Check systemctl status unattended-upgrades and ps aux | grep -E 'apt|dpkg' on a failed host, and look for a leftover /var/lib/dpkg/lock-frontend holder rather than deleting locks blindly.
Use the apt module with a bounded wait, for example lock_timeout: 180, and keep update_cache explicit rather than running it on every task. If the environment permits, schedule the rollout outside unattended-upgrade windows or disable unattended-upgrades during the window with a documented, reversible task. Add a pre-check that fails fast with a clear message if a lock is held longer than expected. Reruns should be safe because apt package installation is idempotent; the retry is for transient lock contention, not for masking a corrupted dpkg state.
2. Design the serial and load balancer flow so no more than 5% of hosts are out of rotation and nginx is never restarted while still receiving traffic.
Read the worked answer
Use serial: 10, because 5% of 200 is 10. A canary ramp such as serial: [1, 9, 10] is acceptable only if the total drained at any time stays at or below 10. Within each batch, drain the host through the load balancer API, wait until active connections reach zero or a bounded timeout, deploy, run meta: flush_handlers so nginx restarts while the host is still drained, then run the /health check and re-add the host. Keep max_fail_percentage low enough to stop the rollout on a bad batch; do not rely on it to protect the load balancer.
The 502s happened because restart was not ordered around drain. Put drain and re-add in the same per-host workflow, not in a separate later play. Use throttle on load balancer API calls to avoid rate limits, and add a pre-task that refuses to proceed if the drain API is unavailable. With 30 minutes for 200 hosts, measure the per-batch deploy and health-check time first; if 20 batches cannot fit, reduce scope rather than exceed the 5% drain limit.
3. How do you guarantee that a host is re-added after a failed deploy, and what would you log for the next partial failure?
Read the worked answer
Wrap the deployment in block, rescue, and always; set a fact when the host is successfully drained, and in always re-add only if that fact is true. Delegate the load balancer calls to localhost, make them idempotent, and do not let a failed health check leave the host permanently disabled. If the play aborts unexpectedly, a separate cleanup play with the same serial limit should scan for drained hosts and re-add them after confirming nginx is healthy; otherwise an operator must have an explicit, documented recovery command.
Log the host, batch number, drain start and end, active connections at drain completion, deploy result, health-check status code and latency, and re-add result. Preserve the Ansible retry file or use a callback that records failed hosts, and check the load balancer API state after the run rather than trusting only Ansible's summary. For the next run, test the rescue path on one canary host by forcing a health-check failure, then confirm the host returns to rotation.