Automatic Mode Produces Failures, Reboots, and a Reporting Requirement
Automation was switched from Recommend to Automatic in a 420-VM vSphere cluster after change advisory board approval. In the first 48 hours, 340 resize actions were recommended: 61 executed, 190 were rejected, and 89 never left pending. The audit log shows three error families - 'vCenter task failed: Invalid configuration for device 0', 'Action not executed because the entity is in a maintenance window' although no window is defined on those VMs, and a large batch of 'Action discarded'. Two VMs rebooted unexpectedly during a memory hot-add because their guest operating systems do not support hot-plug. Security now requires a failure alert within five minutes of any failed action, and a daily report proving no action ran outside approved maintenance windows, delivered to a Teams channel.
1. Triage the three error families and explain which of them actually represent failures.
Read the worked answer
These are three different problems and should not be reported as one metric.
- 'Invalid configuration for device 0' is a vCenter-side rejection of the reconfigure. Typical causes: a mounted CD-ROM or ISO, a passthrough or unsupported device, a snapshot present, or a target memory size beyond what the VM hardware version or guest supports. These are per-VM facts, so filter the failing actions by VM and inspect hardware version and devices before suspecting the platform.
- 'Maintenance window' with nothing defined on the VM almost always means an inherited scheduling policy on a group the VM belongs to, or a scope applied by another administrator. Read the effective policy on one affected VM rather than the global policy page.
- 'Action discarded' is usually not a failure at all. The market re-evaluated and the action stopped being valid because utilization dropped, the VM moved, or a constraint changed. Excluding that state from failure counts is essential or your automation health looks far worse than reality.
The 89 pending items are a separate signal: check whether they sit in Manual mode somewhere in their effective policy, or are blocked by a constraint. The action detail states the reason. Reconcile the arithmetic (61 + 190 + 89 = 340) before drawing conclusions; blending discarded and failed states is the most common reporting error in this exact situation.
2. Explain the unexpected reboots and how you would prevent a recurrence without abandoning automation.
Read the worked answer
Memory hot-add requires three things at once: a guest OS that supports it, the VM setting that enables memory hot plug, and sufficient reserved memory. If any is missing, the reconfigure either fails or only takes effect after a reboot - so an action the platform classifies as a resize can power-cycle a VM. That classification is about the intent of the action, not about your guest's capabilities.
Prevention:
- Identify the population that cannot hot-add (older guest OS, hardware version below the threshold, hot-plug disabled) and take it out of Automatic. Put those VMs in a group with automation set to Recommend so an administrator schedules the resize into a maintenance window.
- Run a canary before widening scope: enable Automatic on a small non-critical group for a week and review the action outcomes, including any actions that required a power cycle.
- Where the business tolerates no downtime, exclude powered-on VMs that would require a restart from the automated scope rather than assuming resize is always online.
- Record the approval, the canary evidence and the exclusion list with the change record, because that is precisely what an auditor will request after an unexpected reboot.
The lesson to carry forward is that Automatic is a per-action-class and per-entity decision, not a single switch for the cluster.
3. Design the five-minute failure alerting and the daily maintenance-window attestation report.
Read the worked answer
Alerting. Use the platform's notification capability configured for action events, with a webhook destination pointed at a small relay (or directly at a Teams incoming webhook if the payload can be templated). Configure failure events and, separately, approval-pending events, or the approvers will not know the queue is filling. Forward syslog to the SIEM so the audit trail survives independently of the platform's own retention. Keep an API poll of failed actions as a heartbeat: if the relay dies, the poll still catches the failure. Prove the five minutes by forcing a known-bad action in a lab scope - a VM with a mounted ISO - and timing the alert end to end. A configured alert is not a tested alert.
Reporting. Schedule an actions-executed report scoped to the estate over a 24-hour window, delivered to the Teams channel through the same relay. Two validations before publishing it: confirm audit retention covers the reporting window (90-day retention with a 1-day report is fine; the reverse is not), and confirm the maintenance windows exist as platform policies so the report can be evidenced against a documented control. If the windows live only in a spreadsheet, the report attests to nothing an auditor will accept.