OLE: System Monitoring, Logging, and Performance Troubleshooting

This domain tests reading the right tool for the symptom, understanding journald and rsyslog, and knowing which metrics identify CPU, memory, disk, or I/O bottlenecks.

Live and historical metrics

CPU and load: uptime, top, mpstat -P ALL 1, pidstat 1. Linux load average counts runnable plus uninterruptible (D-state) tasks, so it can be high with idle CPUs.
Memory: free -m, vmstat 1 (watch si/so for swapping, wa for I/O wait).
Disk and I/O: df -h, du -sh /var/*, iostat -xz 1 (%util, await, r_await), iotop.
History: sar -u 1 5, sar -d, stored in /var/log/sa/ by the sysstat service.

Logging

journald: journalctl -u sshd -b, journalctl -p err --since today, journalctl -f, journalctl --disk-usage, journalctl --vacuum-size=500M.
Persistent journals require Storage=persistent in /etc/systemd/journald.conf and the /var/log/journal directory.
rsyslog writes /var/log/messages and /var/log/secure; rules live in /etc/rsyslog.conf and /etc/rsyslog.d/. logger generates test messages.
Rotation is configured in /etc/logrotate.d/; test with logrotate -d /etc/logrotate.conf.

Tuning

tuned-adm list, tuned-adm active, tuned-adm profile throughput-performance; nice/renice for priority, ulimit -a for limits, /etc/sysctl.d/ for kernel parameters applied with sysctl --system.

Worked example

df -h reports /var at 100% but du -sh /var/* totals far less. Run lsof +L1 to list deleted-but-still-open files. Reasoning: space held by a deleted file whose file descriptor is open is not counted by du; restarting (or truncating safely via the process) the holding service releases it.

Traps

Reading high load average as CPU saturation without checking vmstat wa and iostat.
Journals lost at reboot because storage was left volatile.
sar produces no history when the sysstat timer/service is not enabled.
du cannot see deleted open files; df can.