Monitoring, Logging, and Troubleshooting
What to know
Understand the three pillars: metrics (numeric time series), logs (event records), and traces (request paths across services). Know the difference between monitoring (known failure modes, dashboards, alerts) and observability (ability to ask new questions from telemetry). Cover alerting design - thresholds vs anomaly detection, avoiding alert fatigue, and routing to on-call. Familiarize yourself with log levels, structured logging, retention and aggregation, and centralized log collection. Distributed tracing concepts (span, trace ID) matter for microservices. Finally, a structured troubleshooting method: reproduce, isolate by layer, compare against a known-good baseline, then remediate and add a monitor to prevent recurrence.
Traps
Questions often confuse metrics with logs - "count of errors per minute" is a metric; "the stack trace" is a log. Another trap is alerting on a symptom that is noisy (CPU spikes) instead of a user-facing signal (error rate, latency). Beware answers that say "increase logging" without retention or cost consideration. Also watch for correlation vs causation in root-cause questions; the correct answer usually includes verifying with a trace or baseline rather than assuming the first anomaly is the cause.
Worked example
Users report slow checkout but CPU looks normal. Metrics show p99 latency rose at 14:02; traces show the payment call dominates; logs show retries. The root cause is downstream retry amplification, not the front end. Rule of thumb: start from the user-facing symptom, follow the trace to the slowest span, then confirm in logs.