Alert Fatigue, a Metric Cardinality Explosion, and a Cascading Timeout

Start from the SLO and work backwards; the current alert set is organised around causes, which is why 91% of it is noise.

Delete or demote.

• CPU > 80% on 40 pods: this is a cause, and a cause that frequently coexists with perfectly healthy service. Move it to a dashboard panel and a weekly capacity review. Keep only a saturation alert when the resource is provably tied to a user-visible symptom - for example connection pool utilisation above 90% for 5 minutes, because that specific resource produces failed checkouts, not slow ones.
• Single-pod restarts, single-instance 5xx spikes, and per-endpoint error counters below the SLO threshold: these generate the bulk of the 420 and should be anomaly digests, not pages.

Keep or add (symptom-based).

• Multi-window, multi-burn-rate alerts on the SLO error budget: page at 14.4x burn over 1 hour (about 2% of the monthly budget consumed in an hour) and 6x over 6 hours; open a ticket at 1x over 3 days. Two windows per severity keeps the alert both fast and resistant to blips.
• A synthetic checkout probe every 30 seconds from three regions, running a real authorised test transaction to a canary merchant. This is the specific control that fixes the 19-minute detection gap: synthetic monitoring does not wait for a customer to complain.
• Saturation alerts on the two resources that produce immediate user-visible failure: database connection pool and queue consumer lag.
• A payment-processor error-budget alert tied to the processor's own SLO, separate from yours, so a dependency problem is visible without being smuggled into your availability number.

Ownership and hygiene. Every alert must name an owner, a runbook, and the user-visible symptom it represents; anything that cannot answer the symptom question goes to a dashboard. Review pages weekly and track the ratio of actioned to ignored pages as a metric with a target above 70%. A page that is routinely ignored is worse than no alert, because it trains the rotation to stop reading.

Mechanism. Fraud scoring degraded from tens of milliseconds to 8 seconds. Checkout's connection pool has 50 slots; with 8-second calls, throughput through those slots collapses to about 6 calls per second, so checkout threads queue behind the pool. Callers time out and retry, and because retries have no backoff they arrive as a burst, roughly quadrupling offered load. Retries also consume pool slots, so the pool never drains, and the failing behaviour propagates upstream to anything calling checkout. Classic retry-amplified metastable failure: the system cannot recover while the retries continue, even after the dependency recovers.

Timeout arithmetic. Set a latency budget from the SLO and divide it, never the other way round. If the checkout SLO target is p99 under 1.5 s end to end, and checkout makes two downstream calls plus its own database work: own work 300 ms, database 400 ms, fraud scoring 300 ms, leaving 500 ms of headroom for scheduling and tail effects. The 8-second default is a violation of that budget by an order of magnitude. Enforce a hierarchy: every hop's timeout must be shorter than its caller's remaining budget, so checkout moves on at 300 ms instead of dying at 8 s.

Retries. Cap total retries as a fraction of requests - a retry budget around 10% of successful traffic - rather than per-call limits. Exponential backoff with full jitter, and retry only idempotent operations or POSTs carrying an idempotency key. Never retry on a timeout of a non-idempotent write.

Isolation.

• Circuit breaker on fraud scoring: open at a 50% failure rate over a 10-second window with a minimum request volume, half-open with a small probe rate. When open, apply the documented degraded path - a deterministic rules-based decision or queue the transaction for review - rather than failing checkout outright. Decide the fallback with the risk team in advance; failing open on fraud is a business decision, not an engineering one.
• Bulkhead: give fraud calls their own small pool or semaphore (for example 12 of 50 slots) so a slow dependency cannot consume all capacity. Checkout keeps serving.
• Bound the queue: reject fast when the pool wait exceeds the remaining budget rather than queueing past the timeout. A fast 503 with a retry hint is better for everyone than a 30-second hang.

Verification. Inject 8 seconds of latency into fraud scoring in staging, then confirm checkout p99 stays under 1.5 s, pool wait stays under 20 ms, and the breaker opens within two windows. Test the recovery path too, since metastable failures often persist after the trigger clears.

Cardinality. The user_id label on a histogram is unbounded - with hundreds of thousands of users and roughly 12 buckets plus _sum and _count, a single label turns a few hundred series into millions, and series churn continues because each new user creates new series until the retention window clears. The OOM kills are the direct consequence.

• Remove user_id from all metric labels. Debugging one user's latency is a tracing problem: attach user_id (hashed or partially redacted) as a span attribute and use exemplars on the histogram to jump from a slow bucket to a trace.
• Aggregate the histogram at route, method, status_class, and region. That is a bounded product and stays under a few thousand series.
• Establish a cardinality guardrail in CI: a job that scans recording rules and instrumentation configs for known-high-cardinality label names (user_id, order_id, session_id, request_id, raw paths) and fails the build. Track total series count as a capacity metric with an alert at 70% of the provisioned limit.
• Consider a selective-retention strategy for any genuinely per-entity metric: keep it in a separate short-retention store rather than the main HA pair, so an experiment cannot take down the monitoring that incident response depends on.

SLO carve-out. Excluding 5xx from one dependency is not automatically wrong - it is wrong when it is unbounded, undocumented, and applied unilaterally. Do three things. First, define the user-visible outcome: if the dependency failure means the customer cannot pay, it counts against the SLO even if the root cause is external. Second, where a genuine degraded-but-successful path exists (for example payment processed via a fallback processor within the latency target), that counts as success and needs no carve-out. Third, if you keep any exclusion, tie it to a time-boxed, written dependency SLO with an owner and a review date, publish it alongside the SLO, and report the excluded error rate as a separate series so the number is visible rather than hidden.

Rank the SLO definition by customer experience, not by which service the error came from. A carve-out that exists because it makes the number achievable is the definition of gaming the metric, and it will be discovered by someone less sympathetic than the team - typically during a renewal conversation.