Rolling restart orchestration for a 180-node Java web tier
You manage a 180-node Java web tier behind an AWS ALB. Each node must be drained from the target group, allowed 120 seconds for in-flight requests, restarted, and returned to service only after /health returns 200 for five consecutive checks. The current deploy script runs knife ssh 'role:web AND chef_environment:production' 'sudo chef-client' -C 20. During the last release, three nodes failed the Chef run because of a bad JVM heap attribute; the script did not stop, and the remaining nodes were restarted while the ALB still had them in service. The result was an 18-minute partial outage. The application team wants to keep using Chef for configuration, but the platform team wants a repeatable orchestration that can be audited. You have no external workflow tool budget, but you do have Chef Infra Server, a CI runner, and AWS APIs.
1. What are the specific design flaws in the current `knife ssh` approach for this rolling restart, and what invariants must any replacement enforce?
Read the worked answer
The script conflates configuration convergence with service orchestration. knife ssh with concurrency 20 gives no ordering, no target-group drain, no health gate, and no stop-on-failure. It also has no durable record of which node was in which phase, so an operator cannot tell whether a failed node is drained, restarted, or still serving. The three failed Chef runs should have halted the rollout before any further restarts; instead, the remaining nodes were restarted while still registered.
Any replacement must enforce several invariants: at most a bounded number of nodes out of service at once; drain before restart and wait for connection draining; restart only after a successful Chef run; return to service only after the health check passes; abort the whole rollout on a failure threshold; and record per-node state transitions for audit. The orchestration must also be idempotent, so rerunning it after a partial failure does not double-drain or skip a node. Chef remains responsible for configuration; the orchestrator is responsible for sequencing and cloud state.
2. Design an orchestration using Chef and AWS primitives. Where should the drain and health logic live-inside a recipe, a custom resource, or an external script? Justify.
Read the worked answer
Keep drain, wait, and health-check logic outside the recipe. A recipe should converge the node to the desired configuration; it should not own the cloud target group state, because that state is external, time-dependent, and must survive a Chef run failure. A custom resource can wrap a restart with guards, but the orchestration still needs a state machine that knows about the ALB, batches, and abort conditions.
Implement a small CI-run script or service that: selects a batch from a reviewed node list; calls the ALB API to deregister each target; waits for the drain timeout and for the target to be unused; runs chef-client on the node with a bounded timeout; verifies the local health endpoint; re-registers the target; waits for healthy; and records the result. Use short-lived AWS credentials from the CI runner and Chef client keys already on the nodes. Run batches of one or a small fixed number, depending on capacity. The script should write a per-node state record to a log or artifact so the audit can reconstruct the rollout. Do not put the ALB API calls in a recipe that runs on every node, because that gives every node the ability to change fleet membership.
3. How would you test and canary this orchestration before the next production release, and what metrics would you use to decide whether to continue or abort?
Read the worked answer
Test the failure path first, not the happy path. In staging, use the same ALB mechanics and inject a bad JVM heap attribute on one node; verify that the orchestrator drains that node, detects the Chef failure, stops the rollout, and leaves the node out of service rather than re-registering it. Then test a health-check failure after a successful Chef run and verify the abort and rollback behavior. Add a test for a rerun after partial failure to prove idempotency.
For production, canary one node or a small batch outside peak traffic. Watch application 5xx rate, p95 latency, unhealthy host count, Chef run failure rate, and time per node. Abort if the 5xx rate exceeds the agreed threshold over a five-minute window, if two consecutive nodes fail Chef, or if the healthy host count drops below capacity. Continue only if the canary completes within the expected drain-plus-restart time and the error rate returns to baseline. Record the canary result in the change ticket before widening to the next batch.