LFCE: Troubleshooting and Automation

What this domain covers

The final domain tests method plus repeatability: a layered diagnostic approach (kernel, service, network, permissions/SELinux, resources) and automation that produces the same result every run - shell scripts, Ansible, systemd timers, and log rotation.

Core rules

• Work the stack in order. Kernel and hardware: dmesg -T, journalctl -k. Service: systemctl status, journalctl -u. Listener: ss -ltnp. Path: tcpdump -i any port 443. Process: strace -f -p PID, lsof -p PID. Resources: vmstat, iostat, sar, perf top. Boot time: systemd-analyze blame and critical-chain.
• Distinguish refused (nothing listening or firewall reject) from timeout (filtered, routing, or forwarding problem) before touching configuration.
• Scripts: set -euo pipefail, quote variables, log to stderr, exit non-zero on failure so cron and CI notice.
• Ansible: prefer modules (copy, template, service, firewalld) over command/shell so tasks are idempotent; use --check and --diff to preview; handlers restart services only when a task reports change; facts drive conditionals.
• systemd timers: OnCalendar=, Persistent=true to catch missed runs, and systemctl list-timers to confirm. Timers see a minimal environment, unlike an interactive shell.
• logrotate: test with logrotate -d /etc/logrotate.conf; choose copytruncate for daemons that cannot reopen their log file.

Worked example

Users report a web app unreachable. ss -ltnp shows nginx listening on 443, so the process is fine. curl -v https://localhost succeeds locally, so TLS is fine. From a remote host, curl times out rather than being refused, which points at filtering or routing, not the app. firewall-cmd --list-all shows no https service, so firewall-cmd --permanent --add-service=https && firewall-cmd --reload fixes it - no nginx change was justified.

Traps that cost marks

Restarting services before reading logs; assuming a network fault when an AVC denial is the cause; using shell tasks where a module exists, breaking idempotency; cron or timer jobs failing because PATH differs from your shell; timers without Persistent missing downtime windows.