Observability: logs, metrics, traces, APM, and SLOs
What this domain covers
Observability tests whether you can move between logs, metrics, and traces in Kibana, read APM data correctly, and convert a service-level objective into alerting that on-call engineers trust instead of mute.
Signals and where they live
- Logs arrive through Elastic Agent integrations managed by Fleet and land in data streams such as
logs-nginx.access-default. Retention comes from ILM or data stream lifecycle;@timestampand thedata_stream.*fields are what Discover and the Logs view depend on. - Metrics live in time series data streams in
time_seriesindex mode: metric fields carrytime_series_metric, dimension fields are the grouping keys, and downsampling reduces storage for older data. - Traces come from Elastic APM agents or OpenTelemetry through APM Server into
traces-apm-*,metrics-apm.*, andlogs-apm.*. A trace is the set of transactions and spans sharingtrace.id; each document also carriestransaction.idorspan.id,service.name, andservice.environment. W3Ctraceparentpropagates context between services. - The service map is derived from span destination data, so a missing edge usually means missing instrumentation rather than a broken call.
- Tail-based sampling decides after a trace completes, which lets you keep every error and slow trace while sampling out routine traffic.
- SLOs define an SLI - APM availability or latency, a custom KQL good/total ratio, or a timeslice metric - over a rolling window, producing an error budget and burn-rate alerting.
- Alert rule types include APM latency and anomaly, log threshold, metric threshold, and ES query rules; they act through connectors.
Worked example
Checkout must serve 99.5% of requests under 500 ms over 30 days. Create an SLO with an APM latency SLI scoped to service.name: checkout, threshold 500 ms, 30-day rolling window, target 99.5%; the error budget is the remaining 0.5%. Alert on burn rate, for example 14.4x over one hour, rather than on individual slow requests. Add tail-based sampling that keeps traces over one second plus all failed traces. The reasoning: the SLI defines what counts as good, the budget converts it into a consumable rate, and burn-rate alerting filters out isolated spikes that a static latency rule would page on.
Traps
Reading transaction.duration.us as milliseconds. Comparing latency across documents with different service.environment values. Assuming logs contain trace.id without log correlation configured in the agent or ECS logging. Expecting tail-based sampling to compensate for missing instrumentation. Building an SLO whose good-events query matches nothing - it then reports no data rather than zero, which looks healthy.