Observability Spend Above the Cost of the Cluster
After moving to a managed observability backend, the bill reached USD 38,000 per month. Prometheus scrapes 300 services every 15 seconds and holds 4.2 million active series; roughly 60% carry user_id, request_id, or full pod-name labels. Logs ingest 4 TB per day with 90-day retention, and sampling shows 85% of that volume is never queried. Tracing is head-sampled at 0.1%. There are 1,400 alert rules, and on-call estimates 70% of pages need no action; the rest arrive without a runbook link or an owner. In the last Sev-2 incident, the on-call engineer spent 35 minutes scrolling a 60-panel dashboard and never found the faulty dependency; MTTR was 90 minutes. Budget pressure requires a 40% spend cut, but the team is afraid of losing the data that would explain the next outage.
1. Where is the spend actually coming from, and how would you cut 40% without losing diagnostic capability? Give the analysis steps and the expected savings.
Read the worked answer
Find the cost driver before deleting anything. Series count times scrape frequency sets metric ingest: 4.2M series at 15 s is about 280,000 samples per second, which is almost always the dominant metric cost. Run topk(20, count by (__name__)({__name__=~".+"})) and topk(20, count by (__name__, label_name)) style queries to find the heaviest metrics and the labels inflating them. Expect a small number of metrics with unbounded label values to dominate.
Metrics. Drop user_id and request_id labels at scrape time via relabeling or at the collector; keep user_id in traces and logs, where it belongs and is queried by identity. Replace full pod name with workload plus a short hash where cardinality is not needed. Drop histogram buckets nobody queries, and precompute dashboards with recording rules so queries are cheap even if raw series stay. Target: remove 50-60% of active series, roughly 2.4M, which is most of the 40% target on its own.
Logs. 4 TB per day times 90 days is 360 TB, and 85% is never queried. Filter at the collector: drop debug-level in production, drop health-check and readiness noise, sample high-volume info logs at 10%, keep all warn and error. Reduce retention to 30 days hot with 1 year in cheap object storage for compliance, not in the query tier. Expected log savings: 60-70% of ingest.
Traces. Move from 0.1% head sampling to tail sampling that keeps 100% of errors and all spans over 1 s while sampling 1% of successful traffic. This reduces retention volume and increases usefulness at the same time.
Combined, metric series cuts and log filtering typically land between 45% and 60% of total spend. Do it in stages with a two-week observation window so you can detect the metric someone actually needed.
2. Redesign alerting so that pages are actionable, and describe how you would migrate 1,400 rules without missing a real outage.
Read the worked answer
Page on symptoms, ticket on causes. A page should mean a user-visible SLO is at risk. Define SLOs per service (availability and latency) and alert on multi-window burn rates: a fast burn (for example 2% of the 30-day error budget consumed in 1 hour, checked every 5 minutes) pages, a slow burn (5% in 6 hours) opens a ticket. This automatically suppresses pages for a single pod restarting or a CPU spike that has not hurt anyone.
Triage the existing 1,400 rules into four buckets:
- Symptom rules with a clear action: keep, add owner, runbook URL, and a dashboard deep link.
- Cause rules (CPU high, pod restarts, queue depth): route to a ticket or to a dashboard annotation, not a page.
- Duplicates and dead rules (targets that no longer exist): delete. Expect a large fraction to be duplicates of two or three underlying conditions.
- Missing coverage: add rules for the SLOs you just defined and for dependency health you know is unmonitored.
Migrate safely. Run the new SLO rules in shadow mode alongside the old ones for two to four weeks, routing them to a silent channel, and compare: anything the old rules caught that the new ones miss gets fixed before cutover. Then disable old pages in batches of about 100, one service group at a time, with a one-week soak. Keep the old rules defined but non-paging for 30 days.
Make every page self-serving. Each page carries the service owner, a runbook link, the SLO breached, the current burn rate, and a link to the relevant dashboard and recent deploys. Track pages per shift and the fraction of pages acknowledged as actionable; aim for under two pages per shift and above 90% actionability. Alert quality is measurable and should be reviewed monthly like any other SLO.
3. The last incident took 90 minutes to diagnose. What would you change in dashboards, tracing, and correlation so the next one is shorter, and how would you prove it worked?
Read the worked answer
Reduce the search space, do not add panels. A 60-panel dashboard forces serial scanning. Replace it with a single service overview generated per service showing request rate, error rate, and duration, plus saturation and dependency health, ordered top to bottom by likelihood of being the cause. Every panel links to the underlying traces and logs for the same time window.
Make correlation structural. Emit trace_id in every structured log line and attach exemplars to metric histograms, so a latency spike on a chart links directly to a slow trace, and the trace links to its logs. Without this, the on-call engineer jumps between three tools matching timestamps by eye, which is exactly the 35 minutes observed.
Fix trace sampling for diagnosis. Head sampling at 0.1% means error traces are almost never retained. Tail sampling that keeps all error traces and all traces above the latency threshold, at 1-5% of successes, costs little more and captures the incidents you care about. Add a service dependency graph derived from traces so the faulty dependency is visible without reading every service's dashboard.
Standardize on one incident view. A single page that takes a service name and time window and shows deploys, SLO burn, error budget, top errors by count, slowest dependency, and recent configuration changes. The most common root cause in a deploy-driven system is a recent change, so deploy markers on every chart are non-negotiable.
Prove it. Run a game day that injects a fault into one dependency without telling the on-call engineer, and measure time-to-detect (alert fired), time-to-localize (correct service identified), and time-to-mitigate. Do this before and after the change with the same scenario. Also track MTTR and the fraction of incidents where the root cause was identified from the first dashboard opened. Publish both monthly; if time-to-localize does not fall, the dashboard design is still the problem.