Observability, Monitoring, and Incident Response
What to know
Monitoring asks known questions; observability lets you ask new ones. Know the three pillars: metrics (aggregated numbers, cheap, good for alerting), logs (discrete events, rich detail, costly at volume), and traces (request flow across services, span timing). Understand SLIs (measured indicators), SLOs (targets), SLAs (contracts with consequences), and error budgets (allowed unreliability that funds change velocity). Know alerting on symptoms vs. causes, alert fatigue, dashboards, and distributed tracing with correlation IDs. For incidents, know severity levels, roles (incident commander, communications lead, subject-matter experts), runbooks, escalation paths, and postmortem follow-through.
Traps
SLO and SLA are frequently swapped: an SLO is an internal target; an SLA is an external commitment. Another trap is alerting on every CPU spike-symptom-based alerts (latency, error rate, saturation) reduce noise. Confusing mean with percentile latency hides tail pain; use p95/p99. Also, an incident commander coordinates and decides-they do not debug, which is a common distractor.
Rule of thumb
Alert on user-visible symptoms, page only for urgent actionable issues, and let error budgets govern release pace. If you exhaust the budget, prioritize reliability work over new features.
Worked example
An SLO is 99.9% availability over 30 days, roughly 43 minutes of allowed downtime. A bad deploy consumes 20 minutes. The team should slow risky releases until the budget recovers and record the cause in a blameless postmortem with owned action items.