Alert flood after monitoring consolidation
A logistics company has consolidated four monitoring tools into one platform. Since go-live, the platform generates 4,200 alerts per day and auto-creates 1,800 incident tickets. About 65% are auto-closed; on-call engineers estimate only 3% require action. The most common alerts are CPU spikes, disk 80% warnings, and duplicate network latency alerts from three regions. An AIOps correlation tool has been proposed, but the CMDB is 30% incomplete and many servers are not mapped to business services. Incident managers cannot see which customers are affected. Service owners complain about alert noise, while the monitoring team says 'we cannot remove alerts because we might miss a real outage'. A recent P1 outage was detected by a customer before any alert fired. The CIO wants a dashboard showing incident volumes and MTTR to prove the new platform is working. The service desk manager wants to turn off auto-ticketing entirely.
1. Explain why the consolidated monitoring platform has made incident management worse. Which information and data-quality problems should be fixed before investing in AIOps?
Read the worked answer
The platform has increased alert volume without increasing actionable signal. Monitoring tools detect events; incident management needs to know which events affect a service and a customer. Here, CPU spikes and disk 80% warnings are being treated as incidents even when there is no service impact. Duplicate network latency alerts from three regions amplify the same condition. Auto-ticketing then converts noise into work, and the 65% auto-closure rate shows that many tickets should never have been created.
The deeper problem is missing service context. The CMDB is 30% incomplete, and many servers are not mapped to business services, so incident managers cannot assess customer impact or route work correctly. The recent P1 detected by a customer shows that the monitoring is also missing the signals that matter, possibly because thresholds and service-level health checks are poorly designed.
AIOps depends on reliable topology, ownership, and historical alert-to-incident outcomes. Buying correlation before fixing the CMDB would automate and amplify bad data. I would first map the critical business services and their CIs, define service health indicators, and establish ownership. Then remove duplicate and low-value alerts, tune thresholds, and separate event records from incident records. AIOps should be a later step, piloted against a clean, measured baseline.
2. Recommend a prioritised set of changes to tooling, alert design, and process. Include how you would measure success and respond to the CIO's dashboard request.
Read the worked answer
Priority one is service mapping for the top 20 business services and their critical CIs, because without it neither alert correlation nor customer impact assessment is possible. Priority two is alert hygiene: deduplicate the three regional latency alerts, suppress known maintenance and transient CPU spikes, replace raw disk 80% warnings with capacity-forecast thresholds, and define which alerts are actionable. Priority three is routing: only high-confidence, service-impacting alerts should auto-create incidents; the rest should go to an event queue or a review queue. Auto-ticketing should be narrowed, not switched off entirely, because some known patterns are reliable.
Priority four is process: a daily alert-quality review, clear ownership for alert rules, and an incident classification guide that distinguishes events from incidents. Priority five is a pilot of AIOps after the data quality improves, measured against the same baseline.
The CIO's dashboard should not show incident volume and MTTR alone, because those can look better simply by suppressing tickets. It should include alert-to-incident ratio, percentage of actionable alerts, customer-reported incidents, MTTD, MTTR for service-impacting incidents, reopened incidents, and coverage of critical services in the CMDB. That gives the CIO evidence of signal quality, not just activity.