Chef search as load balancer membership during autoscaling

Chef search is an index over node objects saved by Chef Infra Client, not a real-time health or cloud inventory service. A newly bootstrapped node is not in search until it completes a run and saves its node object, so 15 nodes that were still bootstrapping, waiting on a package mirror, or failing their first converge will be absent for exactly the window observed. A terminated instance does not delete its node object; search continues to return it until someone deletes the node and client. ohai_time only changes when the node checks in, so a terminated node can look recent for a while and then look stale, but search has no TTL by itself.

Collect evidence before changing architecture: compare knife node list with the cloud API instance list; inspect ohai_time for the 22 stale nodes; check bootstrap and first-run logs for the 15 missing nodes; and confirm whether the runbook uses the search index immediately after bootstrap. The key judgement is that search is acceptable for coarse configuration discovery, but it is the wrong authority for load balancer membership because it cannot prove current health. Membership should come from a successful converge plus an application health check, and deregistration should come from the cloud lifecycle event.

Move membership decisions to the node and the cloud lifecycle, not to a periodic search. On boot, the instance should receive its Chef client identity through a short-lived, scoped bootstrap mechanism rather than a shared validator key. One workable pattern is an Auto Scaling lifecycle hook that calls a small bootstrap service or CI job with temporary credentials; that job creates a client-scoped credential, runs the first converge, and records the result. The node registers itself with the target group only after Chef exits zero and its /health endpoint returns 200. A node that fails its first converge stays out of the target group and is terminated or quarantined by the lifecycle hook after a bounded retry.

On scale-in, the lifecycle hook deregisters the target, waits for connection draining, then runs any final cleanup and deletes the Chef node and client. Do not depend on a human deleting nodes later. The runbook that rebuilds the target group should read from the cloud target group or a health API, not from Chef search. Chef remains the configuration authority; the cloud API remains the membership authority. This also gives the security team the per-instance credentials it wants instead of one shared validator.

First classify, do not bulk-delete. For each node, compare ohai_time with the cloud instance state, the instance launch time, and the last known target group health. A node whose cloud instance is terminated can be deleted with low risk. A node whose instance is running but has not checked in is a different case: it may be partitioned, have a broken chef-client, or be in a maintenance state, and deleting its client can break its next run. Use a grace period longer than the normal chef-client interval and require cloud state terminated or stopped before deletion.

For cleanup, script against the Chef Infra Server API or knife node bulk delete after generating a reviewed list; delete both node and client objects so a recreated instance with the same name does not inherit stale state. Run the job on a schedule and log what was deleted. Prevent recurrence with lifecycle-hook deregistration and a short TTL report for nodes with no cloud instance. Keep a manual override for known long-running maintenance nodes. The risk is not deletion itself; it is deleting a node that is merely unreachable and then having it re-register with a new key or fail its next run because its client identity vanished.