Ninety alarms a day and the incident nobody saw

The current design is cause-based, per-resource, flat-severity and unconditional. That combination produces exactly the observed outcome: 90% noise, and an engineer who has learned to mute the channel. Four principles to apply.

1. Alarm on symptoms customers feel, page on those only. The payment API's error rate and latency are the symptoms. CPU on instance 17 of 24 is a cause and belongs on a dashboard, not a pager.

2. Aggregate per-resource rules into group rules. Replace "any of 24 ECS above 80% CPU" with a rule on the AS group: alarm only when more than 30% of instances exceed the threshold for 10 minutes. One instance at 80% is normal in a load-balanced tier; a majority is a trend.

3. Separate paging from ticketing. Three severities with different channels: critical (customer-visible, pages on-call), warning (ticket, next business day), info (dashboard only). Disk at 86% on a volume that grows 1% a month is a ticket, never a page.

4. Make suppression deliberate and time-boxed. Replace the informal mute with proper alarm masking tied to a change or maintenance window, with a defined end time. A permanent mute is how the 502s were missed.

Concrete replacements:

• ECS CPU per-instance, 5 min → AS-group rule, >30% of instances >85% for 10 min, warning.
• Disk >85% → forecasting rule: predicted full within 7 days, ticket; within 24 hours, page.
• ELB 5xx >1% over 5 min → error-budget burn rate on the API SLO, evaluated over both a short and a long window, paging. A 1% absolute threshold over 5 minutes is either too sensitive at low traffic or too blunt at high traffic.
• Add the two missing signals: Kafka consumer-group lag growth rate, and RDS connection utilisation above 80% of max_connections.

Three separate failures, and only one of them is about the threshold.

The window. A 5-minute evaluation window with a 1% threshold means the rule needs the error rate to stay above 1% long enough for the evaluation to complete and the alarm to transition to firing. A 6-minute event sits right at the boundary; depending on when the evaluation samples land, the rule can fire late, fire and clear before notification, or miss entirely. Short, sharp outages are exactly what long averaging windows hide.

The threshold in absolute terms. A 1% absolute 5xx rate is a very different signal at 03:00 than at 20:00 peak. During peak, a partial failure of one availability zone might produce 0.8% and never cross the line while still affecting thousands of payments. During off-peak, 1% might be three requests. The threshold is not measuring customer harm.

The delivery path. This is the decisive one. The engineer had muted the channel. No threshold change fixes a muted channel - the alarm could have fired correctly and still been invisible.

Would lowering the threshold help? No. It would have increased the daily alarm count above 90, increased muting behaviour, and made the next miss more likely. The fix is structural: an SLO-based burn-rate rule that evaluates a short window against a long window so a 6-minute event is detected without firing on every blip, routed to a channel that is never muted, plus a masking mechanism that expires automatically so that muting is a bounded action rather than an indefinite state. The lag and connection alarms are the corroborating signals that would have told the on-call what was actually happening.

The three data points - Kafka lag 240,000, RDS connections pinned at 1,000 for 11 minutes, ELB 502s for 6 minutes - fit one chain, and the ordering matters.

Most likely chain. A downstream slowdown begins the sequence, not the 502s. Either the RDS instances hit an IO or lock bottleneck, or a consumer group stalled on one bad partition. Consumers stop committing offsets, so lag climbs to 240,000. Each consumer keeps its connection open and, as the backlog grows, the application opens more connections or holds them longer trying to drain it. Active connections reach the 1,000 cap. New connection attempts queue or fail, request-handling threads in the web tier block waiting on a connection, the ELB's backend health checks or request timeouts trip, and the ELB returns 502 to clients. The 502s are the last symptom, and they lasted 6 minutes while the connection saturation lasted 11 - consistent with recovery lag after the underlying block cleared.

What I would verify rather than assume. Whether the lag was a cause or an effect: if the consumers were healthy and the producers spiked, the chain runs the other way. Check offset commit history per partition and whether one partition was stalled, which points to a poison message or a hot key rather than load.

First change. Cap and alarm on connection utilisation before anything else. Set max_connections with explicit headroom, size the application pool so 24 web instances cannot collectively exceed it, alarm at 80% of the cap, and add a bounded wait with a fast failure rather than an unbounded queue. Connection exhaustion is the point where a slow dependency becomes a total outage, and it is the cheapest link in this chain to make non-catastrophic. Only after that would I tune the consumer side and add consumer-lag alarms.