Monitoring and Service Level Indicators
What to monitor
Monitoring exists to answer: is the service healthy from the user's perspective, and if not, why? Instrument four golden signals - latency (especially tail latency, not just the mean), traffic, errors and saturation. Method-specific frameworks help: RED (rate, errors, duration) for request-driven services and USE (utilisation, saturation, errors) for resources such as hosts and queues.
Distinguish white-box monitoring (internal metrics, logs, traces, exported state) from black-box monitoring (probing behaviour as a user would). Black-box testing catches the failures internal dashboards miss because the dashboards share the failure domain.
Alerting philosophy
Alert on symptoms users feel, not on causes they do not. ‘CPU is 85%’ is a cause and often harmless; ‘p99 latency exceeds the SLO for ten minutes’ is a symptom. High-cardinality cause data belongs on dashboards for debugging, not on pagers. Page only for conditions that are urgent, actionable and require human intervention; everything else is a ticket or a dashboard.
Traps
- Alerting on resource thresholds and generating fatigue; the on-call learns to ignore the pager.
- Using averages for latency. A 200 ms mean can hide a p99 of four seconds.
- Confusing monitoring (known-unknowns, predefined metrics) with observability (arbitrary questions answered from rich telemetry). They are complementary.
- Measuring availability at the server level instead of the request level.
Rule of thumb
For each user-facing service, define SLIs on the request path, alert on the budget burn rate for those SLIs, and keep cause-based signals as diagnostic dashboards. If an alert has no runbook action and no user impact, it should not page anyone.