Operations, Monitoring, and Troubleshooting
Measure what users feel
Service level indicators (SLIs) are measurements such as request success rate or p95 latency. Service level objectives (SLOs) are the target for an SLI over a window. Service level agreements (SLAs) are contractual promises, usually looser than internal SLOs and carrying consequences. The gap between 100 percent and your SLO is the error budget: spend it on releases, and alert on burn rate rather than on a single spike.
Signals and tooling
Start with the golden signals: latency, traffic, errors, saturation. Metrics are cheap and aggregate; logs are detailed and expensive; traces answer where time went across services. Alert on user-visible symptoms with actionable runbooks, and use dashboards for diagnosis. Correlation across all three is what shortens mean time to recovery.
Traps candidates fall for
- Paging on CPU at 80 percent when users are fine; high utilization is a cause, not a symptom.
- Quoting averages that hide tail latency; report p95 or p99 for user-facing paths.
- Confusing an SLI (measurement) with an SLO (target) or an SLA (contract).
- Alerts without owners or runbooks, which train responders to ignore them.
- Ignoring quotas and connection limits, which surface as outages at the worst moment.
- Retaining all logs forever, then cutting observability to control cost.
Rule of thumb
If an alert fires and there is nothing to do, delete the alert. If an alert fires and nobody can say what users are experiencing, add an SLI first.
Worked example
An SLO of 99.9 percent availability over 30 days leaves an error budget of about 43 minutes (43,200 minutes x 0.001). A burn-rate alert fires when the budget is consumed roughly 14 times faster than the window allows, giving hours of warning instead of a midnight page. After the incident, a blameless postmortem identifies contributing factors and one concrete change.