Monitoring, Troubleshooting, and Documentation
This domain covers how you find out what is broken, how you know what normal looks like, and how you record it so the next person does not start from zero.
Troubleshooting methodology
Define the problem and gather information, form a hypothesis, test it, act, verify, and document. Top-down starts at the application and works toward the wire - good when one application fails. Bottom-up starts at the physical layer - good when nothing works. Divide-and-conquer starts in the middle, for example by pinging the gateway, then moves toward the faulty side. Always ask what changed.
Tools
- ping (ICMP echo), traceroute/tracert (increasing TTL), pathping.
- ipconfig/ifconfig/ip, nslookup/dig, arp -a, netstat -an, route print.
- Cable tester, toner probe, loopback plug; OTDR and light meter for fiber.
- Protocol analysers such as Wireshark and tcpdump; SNMP for statistics.
Monitoring and logging
An SNMP manager polls agents on UDP 161 and receives traps on UDP 162; the MIB defines the OIDs you graph. Syslog (UDP 514) carries severity 0 emergency through 7 debug. Log verbose detail at 6 and 7, but alert on 4 and below. Baselines let you spot anomalies instead of guessing at thresholds.
Documentation
Keep physical and logical topology diagrams, an IP address management sheet, an asset inventory with serial numbers and warranty dates, switch port maps, cable labels at both ends, rack elevations, and change records. Documentation is the first thing an auditor or the next engineer asks for.
Worked example
A user cannot load an intranet site. Ping the default gateway - it replies. Ping 8.8.8.8 - it replies, so routing and internet access are fine. Ping the hostname - it fails. nslookup against the configured DNS server returns the wrong address. The fault is name resolution, not connectivity, so correct the DNS record or the client's DNS setting.
Traps
Replacing hardware before testing it. Ignoring a recent change. Reading stars in traceroute as proof of a dead link when ICMP may simply be filtered.