Alert Noise at Harborline: Supervised Classifier or Unsupervised Correlation?
Harborline is an online retailer running 1,800 hosts across two regions. Its monitoring stack emits about 42,000 alerts per day from six tools: a metrics platform, a log analytics service, an APM product, a cloud provider's native alarms, a synthetic-check service, and a ticketing webhook. Nine SREs share on-call. Last quarter, 61% of alerts were closed as 'no action required', mean time to acknowledge was 27 minutes, and only three to five alerts per week were linked to a confirmed incident outcome. A data scientist proposes training a supervised classifier to label each alert 'critical' or 'not critical', using the historical ticket closes as training labels. A vendor offers an unsupervised clustering product that groups alerts by temporal co-occurrence and claims 94% precision. The SRE manager has budget for one approach and twelve weeks before the peak trading season.
1. Which of the two options should the manager fund for the twelve-week window, and how should the vendor's 94% precision claim be treated inside that decision?
Read the worked answer
The binding constraint is label supply, not algorithm choice. Three to five confirmed outcomes per week is roughly 150-250 events a year, far too thin to train and validate an alert-level criticality classifier over 42,000 daily alerts, and the labels are contaminated: 'no action required' records a tired human's triage behaviour, not whether the alert actually mattered. A supervised model built on that would be unmeasurable within twelve weeks.
The clustering offer solves a different problem - grouping and suppression of related alerts - and needs no labels, so it is fundable now and produces an auditable output: alert volume before and after.
Treat 94% precision as uninterpretable until the positive class, the population, and the base rate are stated. Precision on a selected subset tells you nothing at 61% noise prevalence, and no vendor number survives contact with a new topology. Fund clustering, and in parallel start capturing outcome labels from incident records so a supervised approach becomes possible in two or three quarters.
2. Define the evaluation metric and baseline you would hold the chosen approach to.
Read the worked answer
Pick a metric tied to the decision the SREs actually make. Raw accuracy is useless when 61% of alerts are noise and incidents are rare.
Proposed primary metric: precision at k, where k is the number of alerts nine SREs can genuinely action in a day - roughly 60. Of the top 60 surfaced alerts per day, what fraction link to a confirmed incident outcome?
Safety metric: for every Sev1 and Sev2 in the evaluation window, was the actionable alert present in the top 20? A suppression system that hides one Sev1 has failed regardless of volume reduction.
Operational outcome: mean time to acknowledge against the current 27-minute baseline, and the no-action rate against 61%.
Baseline is the current unfiltered feed scored on the same window. Without a frozen, replayed baseline, any improvement claim is a comparison against memory.
3. Harborline's peak season doubles traffic for ten days. What is the risk to a co-occurrence clustering model, and how do you mitigate it before deployment?
Read the worked answer
Co-occurrence clusters are learned from normal weeks. At peak, entirely new alert combinations appear (checkout queue saturation paired with payment gateway latency), while seasonal scaling makes previously rare co-occurrences common. The model will either fragment known clusters into noise or merge unrelated ones, and suppression rules learned in steady state may hide the very alerts that matter during the highest-revenue period.
Mitigations: evaluate with a temporal split that holds out a previous peak period, not a random split; condition or retrain per time band and per traffic regime rather than using one global model; cap suppression so that any alert belonging to a Sev1 service route is never suppressed, only grouped; monitor cluster assignment stability and alert-to-incident ratio daily during peak; and set a pre-agreed rollback to the unfiltered feed if the actionability SLO burns at twice the normal rate.