High-Cardinality Metrics, Noisy 5xx Alerts, and 100% Tracing
An e-commerce platform runs 30 microservices on Kubernetes. Prometheus scrapes 1.2 million active series; the largest offenders are HTTP metrics labeled with user_id and request_id. The 5xx alert fires whenever any service exceeds a 1% error rate for 5 minutes, producing 40 pages per week, most of which resolve without action. Tracing is configured with 100% sampling, and the tracing backend drops spans under load. Logs are shipped unstructured at 4 TB/day, and engineers still search them with grep across pods. The SLO for checkout is 99.9% availability over 30 days, but nobody can state the current error budget. The team wants fewer pages and faster diagnosis during incidents.
1. Explain the impact of the `user_id` and `request_id` labels and propose a cardinality-reduction plan that preserves useful debugging.
Read the worked answer
Every unique label value creates a separate time series. user_id and request_id are unbounded, so each request or user multiplies series count, memory, scrape size, and query cost. Prometheus stores one series per label combination, and high cardinality causes slow queries, ingestion lag, and out-of-memory risk. The 1.2 million active series is likely dominated by these labels.
Remove user_id and request_id from Prometheus labels. Keep bounded dimensions such as service, route template, method, status code class, and region. Route templates should be normalized, for example /orders/{id} rather than /orders/12345; otherwise path parameters recreate the same problem. If per-request debugging is needed, use logs or traces, not metrics. Add a relabeling rule in Prometheus to drop unknown or high-cardinality labels at ingestion, and enforce a metric naming and label review in code review.
For aggregation, keep histograms with a bounded set of buckets and record exemplars if supported, which link a metric observation to a trace ID without making the trace ID a label. Use recording rules to precompute service-level error ratios and latency quantiles. After the change, measure active series per job and set a budget or alert on cardinality growth so the problem does not return.
2. Replace the per-service 5xx alert with SLO-based alerting for checkout, and explain how you would calculate and use the error budget.
Read the worked answer
The current alert pages on a 1% error rate for any service for 5 minutes. That is symptom-based and not tied to user impact, so many pages are for services whose errors do not affect checkout or are already masked by retries. SLO-based alerting pages when the error budget is being consumed fast enough that the 99.9% monthly target is at risk.
Define the checkout SLO precisely: for example, 99.9% of checkout requests succeed over a rolling 30-day window, where success means HTTP status below 500 and latency under a threshold, measured at the edge or at the checkout service. The error budget is 0.1% of valid requests. Calculate it from a Prometheus recording rule: sum(rate(checkout_requests_total{status=~"5.."}[5m])) / sum(rate(checkout_requests_total[5m])) as the error ratio, and compute multi-window burn rates against the budget.
Use burn-rate alerts with multiple windows: a fast burn such as 14.4x over 1 hour and 5 minutes pages immediately, while a slower burn such as 6x over 6 hours and 30 minutes creates a ticket. This reduces noise and catches both sudden outages and slow degradation. Display remaining budget on a dashboard. When the budget is exhausted, freeze nonessential changes until reliability is restored. The team can then state the current budget and justify pages by user impact.
3. Tracing at 100% sampling overloads the backend, and logs are unstructured and expensive. Propose a sampling and logging strategy that improves incident diagnosis without increasing total cost.
Read the worked answer
At 100% sampling, every request creates spans, and the tracing backend drops data under load, which is the worst outcome: high cost and incomplete traces. Use head-based sampling with a reasonable baseline, such as 5-10% of requests, and always sample errors and high-latency requests. Tail-based sampling is better if the collector can buffer: keep 100% of traces with errors, slow traces above a latency threshold, and a small random sample of successful traffic. Propagate the sampling decision through context so a trace is complete across services. Keep trace IDs in logs and in response headers so a log line can be tied to a trace.
For logs, replace free-text grep with structured JSON logs containing timestamp, level, service, trace ID, request ID, route, status, and duration. Ship logs through a collector that parses and enriches them, and route them to a searchable backend with retention tiers: short hot retention for debugging and cheaper object storage for audit. Reduce volume at the source by lowering debug logging in production, sampling repetitive logs, and not logging full payloads or secrets. Set daily ingestion budgets and alerts.
During an incident, the workflow becomes: SLO burn alert, dashboard showing the failing service and route, trace search filtered by error or latency, then correlated structured logs by trace ID. This replaces broad grep with targeted queries and makes the dropped-span problem visible rather than silent.