1,400 Alerts a Week, 3% Actionable, and a Customer-Reported Routing Loop

Interface utilisation >70%. Delete as a page. Utilisation is a cause-space metric, not a symptom of customer harm; a 70% link delivering traffic fine is not an incident, and the threshold was almost certainly set because it was available rather than because it was meaningful. Convert to a capacity-planning signal on a weekly report, with a page only for sustained saturation that is degrading a service-level indicator.

BGP flaps under 30 seconds. Convert and aggregate. Transient flaps are normal; the alert should fire on flap rate exceeding a threshold within a window, tied to a circuit or peer, and be de-duplicated per neighbour. Short-lived state changes are noise by construction.

CPU on two known devices. Suppress with an explicit, expiring known-issue marker and an owner, not silently. A permanent silent suppression is how a genuinely new problem gets missed later; an expiry forces a review.

The principle: alert on symptoms users or services can feel, from a small set of indicators, and push everything else to dashboards and reports. Precision is the metric - 3% actionable means 97% of pages taught the team to triage by pattern rather than by reading.

Why MTTD survived. The monitoring watched components (utilisation, session state) and never modelled the path a customer's traffic takes. A routing loop can be perfectly healthy at the interface and session layer - every link under 70%, every session up - while packets circulate. With no probe of end-to-end reachability and no SLO on the customer's actual experience, nothing in the alert set could fire. Polling at 5-minute intervals also sets a floor: worst case is 5 minutes plus evaluation, so detection speed is bounded by what is polled and how often, neither of which was the binding problem here.

Choose SLIs from the customer's point of view. The routing-loop incident names the right one for you: the customer cares whether their prefix is reachable and their traffic takes a sane path, not whether a link is at 68%.

Candidate SLIs, all measurable with existing instrumentation plus a small probe layer:

• Reachability: proportion of 1-minute intervals in which synthetic probes from at least two vantage points reach each customer prefix. This is the SLI that would have caught the loop.
• Path sanity: proportion of intervals in which the observed AS path to a prefix has not changed more than N times without a corresponding change record.
• Change success: proportion of network changes completed without a rollback or a customer-visible incident - the SLI most directly under the team's control.
• Latency/loss: proportion of probe intervals under a loss and RTT threshold per service tier.

Starter SLO: 99.9% reachability per customer prefix over a rolling 28 days, measured at 1-minute resolution - that is roughly 40 minutes of budget per 28 days, which sits sensibly against a 47-minute detection and 190-minute repair. If detection and repair together exceed the budget for routine incidents, either the SLO is unrealistic or the response process needs work; the budget makes that argument quantitative.

How the budget changes behaviour:

• While budget remains, the team ships changes and accepts risk; the on-call can deploy without escalation.
• As burn accelerates (a multi-window burn-rate alert on the budget, not a raw threshold), change velocity is throttled automatically and the on-call escalates.
• When exhausted, the error budget policy freezes non-essential network change until reliability work restores margin. This converts an argument about caution into a pre-agreed rule, which is the point.
• Burn-rate alerts also replace many threshold alerts: they are inherently customer-centric and naturally low-noise.

"5,400 alerts handled" measures the monitoring system's own workload, and it improves when the monitoring is made worse - add rules, get a bigger number. It is precisely inverted.

Measures that answer the question:

• Actionable precision: share of pages that produced an action. Currently 3%; target 50%+ over two quarters. This tracks the removal of noise.
• Detected-by-us vs. reported-by-customer: percentage of customer-impacting incidents first surfaced by monitoring rather than a phone call. The routing loop was a phone call, so the baseline is poor and the target is unambiguous.
• MTTD and MTTR trends per incident class, not as a single average. Averages hide the fact that detection is fast for down interfaces and absent for path-level faults.
• Toil: on-call hours spent on non-actionable alerts, converted to a cost figure. Twelve people sharing 1,400 weekly alerts is a staffing argument that executives understand.
• SLO attainment and burn, once defined, as the customer-facing summary.

Where change records fit. Correlating change records with alerts is the cheapest diagnostic available and is currently impossible because the records are in an unlinked spreadsheet. Joining them answers two questions at once: what fraction of incidents follow a change (which turns change management from a gate into a risk signal), and which alerts fire without any change (which are the ones most likely to be genuine environmental faults). It also lets the team suppress the predictable post-change burst of alerts, which is a large slice of the 1,400.

Recommendation: report precision, detection source, and SLO attainment monthly, and retire the alert count from executive reporting entirely - with a short explanation of why a higher count is bad news.