CTLA: Troubleshooting and Operational Procedures

The final domain tests method rather than trivia: given a symptom, do you gather evidence before changing anything, isolate the failing layer, change one variable, and leave a record? CTLA items here are often "what should you do first" or "which tool confirms this hypothesis", so the choice of next action carries the marks.

A defensible sequence

1. Define the symptom precisely - who, what, since when, what changed.
2. Check recent changes and the change log before touching the system.
3. Reproduce if possible.
4. Isolate by layer: hardware → kernel → network → service → application.
5. Form one hypothesis, test it with a read-only command.
6. Apply a single change, verify, then document and close.

Matching tool to question

• Load and CPU: uptime, top, vmstat 1, mpstat.
• Memory: free -h (check available, not free), swap in/out in vmstat.
• Disk I/O: iostat -x 1 (look at %util and await), iotop.
• Kernel and boot: dmesg -T, journalctl -b -p err.
• Network: ss -s, ss -lntp, ip route, tcpdump -i any port 443.
• Process-level: strace -p PID, lsof -p PID.
• Filesystem: df -h, df -i, mount.

Reading load correctly

Linux load average counts runnable and uninterruptible (I/O-blocked) tasks, so a load of 12 on a 4-vCPU host may indicate a saturated disk rather than a CPU problem. Confirm with vmstat wa and iostat before sizing or resizing anything.

Operational discipline

Work inside an approved change window, take a backup or snapshot first, write the rollback step before the change, and communicate status. Preserve evidence - logs, timestamps, command output - before restarting or clearing state. Escalate with the evidence attached, not with "it doesn't work".

Worked example

Users report an application server is "slow". uptime shows load 12.0 on 4 vCPUs. top shows low user CPU. vmstat 1 shows high wa and steady si/so; iostat -x 1 shows one device at 100% %util with high await. Reasoning: the queue is I/O-bound. High si/so suggests swapping, which is itself caused by memory pressure, not by disk failure. The next read-only steps are free -h and identifying the top I/O consumer with iotop or pidstat -d. Only then choose a remedy: throttle or reschedule the job, add memory to stop swapping, or apply a cgroup IOWeight. Rebooting would clear the evidence and probably reproduce the condition.

Traps

• Rebooting first - it destroys the state you needed.
• Changing several settings at once so the fix is unattributable.
• Reading load average as pure CPU.
• Comparing log timestamps across time zones without normalising.
• Closing a ticket without documenting the root cause or the verification step.