CloudMonitor Alarm Fatigue on a Database Tier
Your RDS instance and 8 ECS app servers generate CloudMonitor alarms. Over two weeks the on-call rotation received 340 alerts: 210 were CPU >80% on app servers lasting under 2 minutes and self-resolving, 90 were memory >85% on the same servers, and 40 were genuine RDS connection-count alarms. On-call is ignoring pages, and one real RDS incident went unnoticed for 40 minutes.
1. Diagnose why the alerting is failing and propose a restructured alarm design.
Read the worked answer
The system is alerting on symptoms with no duration or aggregation, so transient noise drowns the signal. 300 of 340 alerts are short CPU/memory blips that resolve on their own.
Restructure:
- Add sustained-duration conditions: alarm only if CPU >80% for 5+ consecutive minutes (use the multi-period/
countsetting). Short spikes disappear. - Aggregate per tier, not per instance: alarm on the fleet (e.g., >30% of app servers over threshold) rather than 8 independent alarms.
- Separate severity: RDS connection count is a page (P1); sustained CPU is a ticket (P3).
- Route by severity to different channels so pages are rare and meaningful.
Target: page volume should drop by an order of magnitude while the RDS signal stays immediate.
2. How would you validate the new design before trusting it, and what metric proves it worked?
Read the worked answer
Validate in two ways:
- Replay history: apply the new duration/aggregation rules to the last two weeks of metric data. Confirm the 300 noisy alerts would have been suppressed and the 40 RDS alarms would still fire. Most monitoring tools let you test threshold logic against historical data or a staging alarm.
- Synthetic test: deliberately drive CPU above threshold for 6 minutes on one server and confirm exactly one alarm fires; then drive a 1-minute spike and confirm silence.
Success metric: page volume per week (target a large reduction) and mean time to acknowledge real incidents (should fall, since on-call trusts pages again). Track both - low volume alone could mean alarms are too quiet. Keep the RDS page as the canary that the pipeline still works.