Alert Noise and a Blame-Based 5 Whys
A utility company's operations center receives about 4,200 alerts per week from 14 monitoring tools. Roughly 60% are duplicates or flapping. Engineers have stopped trusting the alert console. Problem management has a backlog of 30 problems. For the last major outage, the review used a 5 Whys session that ended at 'engineer entered the wrong configuration'. The action was 'retrain the engineer'. The same misconfiguration happened 7 weeks later on a different shift. The CMDB has 71% coverage and stale ownership for 18% of servers. The problem manager wants to buy an AIOps correlation tool. The CIO asks whether the tool will fix problem management.
1. Assess whether the AIOps tool is the right first investment. What method and data issues must be addressed?
Read the worked answer
Do not buy the tool as the first move. The symptoms are alert noise, stale ownership, and repeated misconfiguration; a correlation tool may reduce noise but will not fix data quality or RCA method. First, measure alert quality by source and rule, retire or tune the worst rules, and establish ownership data for the 18% stale servers. Improve CMDB coverage for the services in the 30-problem backlog. Then define the RCA method: use causal trees or fault trees for complex incidents, Kepner-Tregoe for differential diagnosis, and 5 Whys only for simple, linear cases. An AIOps tool can then be evaluated against a specific use case, such as grouping related alerts. Otherwise it automates bad data and creates false confidence.
2. Critique the previous root cause analysis and propose a better approach for the recurring misconfiguration.
Read the worked answer
The previous review stopped at the individual, which is a classic error. 'Wrong configuration' describes an action, not a systemic cause. A better approach asks why the configuration could be entered and pass checks: was there a peer review, automated validation, a change template, shift handover, or tooling constraint? Build a causal tree with branches for human, process, technology, and organization. Test the branches against the second occurrence on a different shift; if the same gap appears, training is not the fix. Actions should change the system: validation at entry, automated comparison with the baseline, a second check for high-risk changes, and learning from near misses. Verify the fix by reproducing the error path or monitoring for recurrence.
3. Define three measures to show whether the new approach is reducing problems, not just alert volume.
Read the worked answer
Use outcome measures, not alert volume alone. Examples: repeat incident rate for the same known error, percentage of problems with a verified root cause and accepted action, known-error reuse rate at the service desk, mean time from problem acceptance to verified cause, change failure rate for changes linked to problems, and age of the problem backlog. Add a quality measure: percentage of root-cause actions that changed a system control rather than only retraining people. Track alert-to-incident precision as a supporting measure, not the goal. Review monthly with the CIO, and stop reporting any measure that teams can improve without reducing harm.