I85PREP
⌘ K
VCP-VVF Admin (2V0-16.25) study materials
VCP-VVF Admin (2V0-16.25) study resources

SD-Card Boot Hosts Losing Logs, Cores, and Time

VCP-VVF Admin (2V0-16.25) · Put your knowledge to work in context.

All case studies

SD-Card Boot Hosts Losing Logs, Cores, and Time

A customer runs six ESXi 8 hosts, each with 512 GB RAM, booting from a mirrored pair of 32 GB SD cards. The hosts have been in production for 14 months. After a purple diagnostic screen on esx-04 last week, the team could not find a vmcore file anywhere, and /scratch showed as a 285 MB ramdisk that is now 100% full. Syslog is still pointed at the default local path. During the incident, vCenter task timestamps were roughly four minutes behind the storage array logs, and each host's NTP server list is empty. The storage team will publish an NFS export for host logs but will not carve out iSCSI LUNs for scratch. Management has decided no future build will boot from SD or USB. The hosts currently have no local spinning disk or SSD beyond the SD pair.

1. Explain why no vmcore exists and why /scratch is a full ramdisk. State what you would change on esx-04 today to make scratch and coredumps survive a reboot, given the constraints.

Read the worked answer

Why this happened

  • Since ESXi 7.0, SD/USB boot media are not used to host the scratch partition. With no other suitable local disk, the host falls back to a ramdisk (about 285 MB) for /scratch - it fills quickly and is wiped on reboot.
  • SD/USB media are also not supported as a coredump target, so the PSOD had nowhere to write. That is why no vmcore exists.

What to set on esx-04 now

  1. Scratch - point it at persistent storage. With no local disk and no iSCSI LUN, use a VMFS datastore presented over FC/NFS if one exists; set ScratchConfig.ConfiguredScratchLocation to a directory on that datastore (e.g. /vmfs/volumes/<ds>/scratch-esx04), accepting that a reboot is required. Confirm after reboot that /scratch resolves to the new path, not the ramdisk.
  2. Logs - set Syslog.global.logDir to [nfs-datastore] /esx-logs/esx04 and enable Syslog.global.logDirUnique so each host gets its own subdirectory. This also satisfies the storage team's NFS offer.
  3. Coredumps - with no supported local device, configure the ESXi Dump Collector and point the host at it with esxcli system coredump network set plus esxcli system coredump network set --enable true. Verify with esxcli system coredump network get and confirm the partition list shows the network dump active.

Reasoning - the RAM-based scratch is a symptom of the boot-device choice, not a misconfiguration. Fix the targets first so the next incident is diagnosable, then fix the boot device permanently.

2. Nobody has configured time on these hosts. What is the risk, how do you fix it, and what must also be checked on the vCenter side?

Read the worked answer

Risk - a four-minute skew is already close to the ~5-minute Kerberos tolerance used by AD authentication. Symptoms to expect: intermittent SSO login failures for AD users, failed AD-over-SSL joins, misleading log correlation across vCenter, hosts, and array, and certificate validation oddities.

Fix on the hosts

  • Configure NTP with esxcli system ntp set --server=<ntp1>,<ntp2> --enabled=true and start ntpd.
  • Open the NTP client firewall rule (esxcli network firewall ruleset set --ruleset-id ntpClient --enabled true).
  • Point every host at the same stratum-2 sources vCenter uses - a host syncing to a different reference reintroduces skew.

Also check

  • vCenter's own time source; if vCenter is the skewed one, every task timestamp and every certificate check is suspect.
  • Don't rely on VMware Tools time sync as the primary mechanism; it is a coarse correction and does not survive the host being powered off.

Reasoning - treat time as an identity dependency, not a monitoring nicety. In an SSO/AD environment a skew fix is a security fix.

3. Management wants the next build to avoid SD/USB entirely. Recommend a boot layout and a low-risk migration path for the existing six hosts.

Read the worked answer

Target layout

  • Boot from an enterprise-grade M.2 or SSD device (RAID-1 mirror where the platform supports it), or boot from SAN if the array is on the HCL for boot. Since ESXi 7.0, a single device can hold boot, scratch, and a VMFS datastore - that removes the ramdisk problem permanently and gives a supported coredump target.
  • Size the boot device to hold the ESXi image plus log/scratch headroom; do not size it exactly to the current image.

Migration path

  1. Confirm the new boot device and firmware/driver stack are on the HCL for the exact ESXi build.
  2. Add the boot device to one host at a time; put the host in maintenance mode with the appropriate evacuation option (full evacuation if DRS/HA can absorb the load, otherwise ensure vMotion-compatible placement).
  3. Reinstall with the same ESXi build, then attach the host back under a vLCM image or baseline so it converges to the reference image rather than drifting.
  4. Re-apply the scratch/syslog/coredump settings from the image or a host profile so the fix is enforced, not manual.
  5. Validate NTP, lockdown/AD, and certificate state before moving to the next host.

Reasoning - one host at a time plus vLCM enforcement converts a hardware refresh into a repeatable, reversible change instead of six bespoke rebuilds.