Alert Fatigue from Interface Flap Monitoring
Your CloudVision / telemetry stack alerts on any interface that goes down for more than 30 seconds. Over a week you get ~400 alerts, mostly from access ports that flap when users unplug laptops, plus a handful of genuine uplink failures. The NOC is ignoring alerts. You have 200 switches, ~8,000 access ports, and ~400 uplinks. You want to cut noise without missing real failures.
1. How would you restructure the alerting to reduce noise while preserving detection of real uplink failures?
Read the worked answer
- Segment by role: create separate alert rules for uplinks vs access ports. Uplinks: alert on down >30s (or even 10s) - these are high-severity. Access ports: alert only on flap rate (e.g., >5 transitions in 10 minutes) or on down >10 minutes, not on every unplug.
- Use interface descriptions/tags to classify ports automatically rather than maintaining lists.
- Suppress known maintenance windows and scheduled changes.
- Deduplicate and group alerts by device/rack so one event doesn't generate 20 pages.
- Route by severity: uplink failures page; access-port flaps go to a daily digest.
- Baseline: measure current flap rates to set thresholds that catch anomalies without catching normal laptop churn.
2. How would you validate that the new rules still catch the genuine failures, given the historical data?
Read the worked answer
- Replay historical data: take the past week's telemetry and run it through the new rule set; confirm the handful of genuine uplink failures still fire and the ~400 access-port alerts collapse to a small digest.
- Compare counts: target a large reduction in access-port alerts (e.g., 400 → <30) while keeping 100% of uplink failures.
- Check for false negatives: ensure no genuine failure is now suppressed by the new thresholds - especially short but real uplink blips that the old 30s rule caught.
- Pilot on a subset of switches or a shadow channel for a few days before switching the NOC over, and keep the old rules running in parallel (to a silent channel) during the pilot.