Alert Storm After Monitoring Automation
TransFleet operates dispatch and tracking for 900 vehicles. It recently deployed auto-remediation: restart a service when CPU exceeds 85% for five minutes, and delete temp files when disk exceeds 90%. The database was not excluded from the automation scope. Within two weeks, the monitoring team receives 1,200 alerts per day, about 40% duplicates. Auto-remediation restarted the primary database 18 times, causing replication lag and lost tracking events. The CMDB is outdated: 22% of CI relationships are missing, so impact mapping shows the wrong services. Change management wants to roll back all automation. Operations wants to tune thresholds. The service owner wants to keep the benefits and asks for a structured decision.
1. Diagnose the failure across tooling, automation rules, and data. Which actions would you take first?
Read the worked answer
There are three distinct failures. First, the automation rule is unsafe: restarting a stateful primary database is not equivalent to restarting a stateless application service. The CPU threshold is also global, so it applies to components with different baselines. Second, the tooling lacks event correlation and deduplication, producing 1,200 alerts per day with 40% duplicates; this overloads operators and hides real incidents. Third, the CMDB data is incomplete, so impact mapping cannot distinguish a database restart from a minor service restart.
Take the highest-risk action first: suspend auto-remediation for the database and any stateful or clustered component. Keep the temp-file cleanup only if it has guardrails, such as excluding database and backup volumes. Then add latency, replication, and service-level alerts rather than relying on CPU alone. Deduplicate alerts by CI and event type, and suppress child alerts when a parent service is down.
Do not tune thresholds before excluding unsafe targets. Tuning a dangerous automation rule only makes the danger less frequent.
2. Would you roll back, tune, or use a phased approach? Justify with risk and governance.
Read the worked answer
Use a phased approach, not a blanket rollback. A blanket rollback loses the benefits and may leave the estate less controlled, but continuing unsafe automation is unacceptable. Roll back the highest-risk rules immediately: database restarts and any action without a tested rollback or blast-radius limit. Keep low-risk, reversible actions such as temp-file cleanup under guardrails.
For the retained automation, introduce approval gates and scope restrictions: named CI classes, maintenance windows, rate limits, and a canary group. Require a runbook, a rollback test, and a post-action verification step. Route every automation action to a log with before/after state. Put the changes through change management as a standard change only when the risk is understood and pre-authorised; otherwise use normal change with CAB review.
Governance should include an automation risk assessment covering statefulness, blast radius, reversibility, and dependency. The service owner should accept residual risk explicitly. This is safer than either extreme.
3. Which metrics would show whether the automation and CMDB are improving?
Read the worked answer
Measure both operational noise and automation safety. For alerting: total alerts per day, duplicate rate, alert-to-incident ratio, percentage of alerts actioned, and mean time to acknowledge. For automation: number of automated actions, success rate, actions causing incidents, actions requiring manual reversal, and percentage of automation with rollback tests. For service impact: replication lag, lost tracking events, and MTTR for dispatch incidents.
For the CMDB: completeness of CI relationships, accuracy against discovery data, percentage of services with mapped dependencies, and impact-mapping errors found during incidents. Track the proportion of automation decisions made with complete dependency data.
Set targets and review them at the service review. A useful target is not 'zero alerts' but a material reduction in duplicates and automation-caused incidents while maintaining or improving MTTR. If the CMDB accuracy does not improve, impact mapping will remain unreliable and automation should stay constrained.