System Monitoring, Fault Diagnosis and Troubleshooting
Domain 6 asks which metric proves a bottleneck, which troubleshooting strategy fits a symptom, and what sequence of diagnostic steps is defensible. Items often include a short timeline and expect you to spot the correlation.
What to monitor
Availability and service ports, CPU utilisation and run queue, memory and swap, disk capacity, inode usage, disk latency and utilisation, network throughput and errors, application response time, and log error rates. Set thresholds from a baseline captured during normal operation, route alerts to an owner, and tune noisy rules - alert fatigue is itself a failure mode that hides real events.
Diagnosis method
Define the symptom precisely (who, when, what changed), gather evidence before acting, form a hypothesis, test one variable at a time, verify the fix with the user, then document. Strategies: bottom-up from hardware, top-down from the application, divide-and-conquer from the middle, follow-the-path along the traffic route, and compare against a known-good peer. Always check what changed recently - a deployment, a configuration edit, a certificate expiry, a scheduled job.
Worked example
Users report the reporting system is slow every day at about 02:00 and fine by 03:00. Steps: confirm the pattern in monitoring history; check iostat and see %util near 100% with high await; check the scheduler and find a full backup job starting at 02:00 on the same volume. Hypothesis: disk saturation from backup I/O, not an application defect. Test: throttle the backup or move it to a snapshot-based method and observe the next night. Verify and document, then raise a change to stagger the schedule permanently. Reasoning: time correlation plus a resource metric pointing at the same interval beats speculation about code.
Traps
- Rebooting first: it destroys evidence and hides the cause until it recurs.
- Changing several settings at once so the effective fix cannot be identified.
- Reading load average as a CPU percentage.
- Ignoring inode exhaustion because df shows free space.
- Closing the ticket when the symptom disappears without recording the cause.