LF Observability case studies

LF Observability · Put your knowledge to work in context.

The Linux Foundation Certified Observability Specialist (LFCOS) is positioned as a Linux Foundation certification aimed at engineers who build and operate observable systems. Before you plan study time or budget, note an important limitation: the only permitted source for this page, the Linux Foundation training site's LFCOS certification URL, returned a "Page Not Found" error when retrieved. The LF Observability case studies are 5 cases and about 5,165 words.

Checkout API: the SLI that says everything is fine while users fail

Your team owns a checkout API serving about 40,000 requests per minute across three regions. The availability SLO is 99.95% over a rolling 28 days, measured from Envoy sidecar metrics as the ratio of non-5xx responses to total responses.

900 words, about 5 min 2

Prometheus at 4.1M series: ranking a cardinality cleanup

Your platform runs a single Prometheus 2.53 scraping 240 targets every 15s, with remote write to Thanos. Active series grew from 800k to 4.1M in six weeks.

1,042 words, about 5 min 3

Tail sampling with round-robin routing is producing broken traces

A 60-service platform emits about 2.4M spans per minute at peak. Each node runs an OpenTelemetry Collector agent (OTLP receiver, batch processor, exporter).

1,057 words, about 5 min 4

Kafka workers producing orphan traces and unknown_service

A Java Spring Boot order service publishes to Kafka and a Go worker consumes. The Java service is instrumented only with the OpenTelemetry Java agent; the Go worker uses the OTel Go SDK with otelhttp for its HTTP calls and hand-written spans around the Kafka consumer loop.

1,035 words, about 5 min 5

A $180k observability bill and 2,300 mostly-useless alerts

A central observability platform serves 22 product teams. Monthly spend is 180k: 95k on logs (14 TB/day into a hot Elasticsearch cluster with 30-day retention and three replicas), 45k on metrics (4.2M active series in a managed Prometheus), and 40k on traces (100% head sampling, about 900M spans/day).

1,131 words, about 6 min