Systems Management and Automation
The Systems Management and Automation domain tests out-of-band management, declarative automation, monitoring that reflects service health, and the change discipline around firmware and patching. Items often present a script or an API call and ask what happens on the second run.
Core rules
- Out-of-band: the BMC (iDRAC, iLO, XCC) gives console access, power control, and hardware inventory independent of the OS. Redfish is the modern REST/JSON interface; IPMI is the older, less secure fallback and should be segmented or disabled where possible.
- Idempotency is the dividing line between automation and scripting: a declarative task that sets NTP servers converges to the same state on the tenth run; an imperative script that appends a line drifts with every execution.
- Keep everything in version control, including the automation that touches production. Secrets belong in a vault, never in the repository.
- Monitoring layers: metrics for trends and thresholds, logs for causality, traces for request paths. Alert on symptoms users feel - latency and error rate - rather than only on component states.
- Firmware and driver updates follow the vendor compatibility matrix and are staged: pilot hosts, then a wave, with a documented rollback.
Worked example
You must boot a host from PXE exactly once. Read GET /redfish/v1/Systems/1 for PowerState and Boot, then PATCH {"Boot": {"BootSourceOverrideTarget": "Pxe", "BootSourceOverrideEnabled": "Once"}}, then POST /redfish/v1/Systems/1/Actions/ComputerSystem.Reset with ResetType On or ForceRestart. The reasoning: Once prevents a boot loop when the PXE server is unreachable, and checking PowerState first means the reset type matches reality instead of failing on an already-powered host.
Traps
- Presenting a non-idempotent script as automation, or one with no rollback path.
- Committing credentials and API tokens to the repository.
- Updating firmware without checking the driver and firmware matrix, then blaming hardware.
- Alerting on every threshold and calling it monitoring; alert fatigue hides the real outage.
- Keeping event and audit logs only on local disk.