Observability Culture and Principles
Observability culture and principles
This domain tests whether you can separate monitoring from observability, state what each of the three pillars is actually for, and describe the cultural practices - SLOs, blameless postmortems, developer-owned instrumentation - that turn telemetry into answers.
Monitoring is not observability
Monitoring watches a known set of signals against known failure modes: dashboards, thresholds, alerts. Observability, borrowed from control theory (Kálmán), is the property of a system whose internal state can be inferred from its external outputs. Monitoring answers "is it broken?"; observability answers "why, for whom, and where?" for a failure nobody predicted. A stack can be fully monitored and still not observable.
The three pillars and their jobs
- Metrics - cheap numeric time series, ideal for alerting and trends. Cost is driven by cardinality, not volume.
- Logs - discrete timestamped events with full detail. Only useful if structured (JSON/key-value) and correlated with a trace ID.
- Traces - causal per-request view across services. The only pillar that shows where time went inside one request.
Many teams add events and continuous profiling as a fourth signal.
Culture is the load-bearing part
Observability fails as a procurement exercise. It works when instrumentation is part of the definition of done, teams own the telemetry of the services they run, SLOs and error budgets (not raw CPU thresholds) drive paging, postmortems are blameless and produce tracked actions, and telemetry spend is budgeted like any other resource.
Worked example
Checkout p99 rises from 400 ms to 900 ms. Error rate and CPU are flat, so no alert fires. Metrics only confirm the latency shift. Logs contain no errors. A duration-sorted trace sample shows the extra ~500 ms inside a client span calling a third-party tax API, and only for EU carts. The fix is a provider-side cache plus a region-scoped latency SLI. The reasoning: metrics localise the symptom, traces localise the cause, and the SLI prevents the next silent regression.
Traps that cost marks
- Equating telemetry volume with observability; unlabelled data is noise.
- Alerting on causes (CPU, memory) rather than symptoms users feel.
- Postmortems that identify a person instead of a systemic gap.
- Buying a backend before defining SLIs.