Rolling out an nginx cookbook bump across a heterogeneous production fleet
Your team manages 420 nodes with Chef Infra Server 14. Two environments exist: staging (30 nodes) and production (390). A role named web_base sets node.default['nginx']['worker_processes'] from a computed value based on node['cpu']['total']. After a cookbook version bump to nginx 9.2.0, 37 production nodes fail the chef-client run with undefined method '[]' for nil:NilClass in the nginx.conf.erb template. The other 353 nodes converge. The failing 37 are Ubuntu 18.04 hosts that also have an environment-level pin to apt 7.1.0; staging has no such pin and passes. The previous on-call engineer left a stale /var/chef/cache/chef-client-running.pid on 5 of the failing nodes. A change freeze starts in 36 hours, and the business wants the nginx tuning rolled out to production before then. You have one maintenance window of 90 minutes and no ability to reboot nodes.
1. What is the most likely root cause of the 37 failures, and what sequence of commands or Chef constructs would you use to confirm it without changing production state?
Read the worked answer
The template error points to a nil lookup inside ERB, not to a general cookbook syntax failure. Start by comparing one passing node and one failing node at the attribute level: run knife node show <node> -l for both, then use knife ssh 'name:<node>' 'ohai cpu' to see whether node['cpu'] is populated. Because the template references node['cpu']['total'], an empty cpu Mash explains the exact undefined method '[]' for nil:NilClass message.
Next, confirm that the run compiles far enough to reach the template. Run chef-client --why-run -l debug on one failing node; why-run performs the compile and resource evaluation without converging. Check /etc/chef/ohai/plugins and the client config for a legacy plugin disablement that exists only on those Ubuntu 18.04 hosts. The apt pin is probably a red herring: it explains a production-only cookbook resolution difference, but the nil is in the cpu attribute, not in apt. Also check ps before removing the stale pid file; delete the pid only if no chef-client process is running. Do not delete node objects or force a run until the attribute source is understood, because a partial converge could leave nginx in a worse state.
2. Given the 36-hour freeze and a 90-minute window, how would you roll out the fix to the remaining production nodes? Specify ordering, concurrency, pinning, and abort criteria.
Read the worked answer
Fix should be a patch-level cookbook change, not an ad hoc node edit. Make the template resilient with node.dig('cpu','total') || 2, and separately correct or remove the Ohai plugin disablement. Upload the cookbook, then pin the production environment to the new patch version so every node resolves the same artifact. Do not rely on a role-level floating version.
Use a canary-first rollout. Select five failing nodes and five currently passing nodes, clean the stale pid only on those five after checking ps, and run chef-client --once on them with a bounded knife ssh concurrency such as -C 5. Verify nginx config syntax, service state, and a local HTTP 200 before widening. Then roll in batches of 10 percent of the remaining production nodes. Abort if any batch has more than one non-zero chef-client exit, if nginx fails to reload, or if the canary health check drops below the agreed threshold. Keep the old cookbook version available for rollback. The staging environment should get the same pin first if the freeze permits a non-production change; otherwise use the canary as the only pre-production signal and record why staging was skipped.
3. Should you migrate this cookbook to Policyfiles before the next change, or keep roles and environments? Argue both sides and recommend a path given six months and an audit requirement to show exactly what ran on each node.
Read the worked answer
Roles and environments are familiar and already carry production overrides, but they make it hard to prove an exact cookbook set: a node can resolve through role run_lists, environment pins, and cookbook dependency solving, and the lock is implicit. Policyfiles give an explicit lock, a single policy revision, and a clearer audit trail of what was intended for a node. That matters for the audit requirement.
The cost is migration risk: roles with computed attributes, environment-level pins such as the apt pin, and search-driven recipes do not map one-for-one. A six-month window is enough for a pilot, not a big-bang rewrite. Recommend a pilot on the web_base role and this nginx cookbook first. Export the current run_list and attributes, build a policyfile lock, run it in staging and on a small production canary, and compare node attributes and cookbook versions. Keep roles for nodes not yet migrated, but require any new production role to use a policyfile. Record the policy name and revision in the change ticket so the auditor can see exactly what ran.