Latency SLO hides a dependency failure

Start at the service boundary with the four golden signals: request rate, error ratio, latency distribution, and saturation. For this SLO, the decision metric is successful requests divided by total requests over a rolling window, plus burn rate against the 30-day 99.9% objective. Aggregate by service and, where it changes action, customer_tier; do not average p95 across pods. A p95 is a distribution property, so averaging per-pod p95 values can hide a small set of slow pods; instead compute the aggregate histogram quantile or use request-weighted aggregation. Pod-level CPU is a diagnostic signal, not the customer-impact signal. The 8% 5xx ratio already indicates customer impact, so the on-call should treat the SLO burn alert as primary. Use logs for the legacy card-auth service and compare its error ratio and latency with its dependency calls. The dashboard should show one service-level row first, then allow drill-down to pod or node only after the service-level signal is abnormal.

request_id is unbounded: every request can create a new series, so the active series count grows with traffic and never stabilizes. At 2.4 million series, Prometheus spends memory on the index and head chunks, and every dashboard query must evaluate a huge label space, which explains the 30-second timeouts. The safest fix is to stop storing request_id as a Prometheus label. Remove it at ingestion with metric_relabel_configs if the application still emits it, or fix the instrumentation to put request_id in logs, traces, or exemplars. Exemplars attach a trace ID to a histogram bucket without creating a new series, which preserves debugging. If request_id is needed for ad-hoc analysis, use logs or a tracing backend, not metrics. Keep service, pod, status_code, and customer_tier only if they are bounded; customer_tier is fine and pod is moderate. After the change, verify the series count drops and rebuild recording rules aggregated by service and status.

First confirm customer impact with the service-level error ratio and latency histograms; the SLO burn alert is the trigger, not proof of the cause. Second, split errors by status_code and route, then compare the affected service's dependency RED metrics (rate, errors, duration) to its baseline. Third, check saturation signals for the dependency: CPU throttling, memory, connection pool, and queue depth, not just host CPU. Fourth, use pod-level and node-level metrics only to localize the blast radius after a service or dependency anomaly appears. Because card-auth has no traces, inspect its logs and the error ratio of its downstream calls. If the failure is isolated to one customer_tier, check routing or tenant-specific limits. Record the hypothesis and the metric that would falsify it; this avoids dashboard tourism. The output of the first 10 minutes should be a mitigation decision such as rollback, scale, fail over, or load shedding, plus a named owner, not a complete root-cause analysis.