CPU Alarm Storm in an Autoscaling Web Fleet

A Monitoring alarm evaluates the metric query against matching metric streams. If CpuUtilization is queried at compartment scope without a dimension filter, the batch instance is one of the streams being evaluated; a breach on that stream can put the alarm into FIRING and publish to the destination. The 5-minute repeat notification then republishes while the condition persists, so one hot host looks like a fleet incident.

Inspect first:

• The alarm's metric query and dimension filters: is it scoped to the web pool, a compartment, or every instance?
• The repeat notification interval and severity: CRITICAL plus 5-minute repeat drives incident creation.
• The notification message format: it omits the instance OCID/name and pool, so responders cannot see that only one host is affected.
• ONS subscriptions: PagerDuty may auto-create an incident for every message; email recipients may simply thread them.

The scale-in flapping is a separate issue: terminated instances stop producing the stream, so the alarm can change state because data disappears, not because CPU recovered. Do not rely on a disappearing stream as a health signal.

Separate the signals instead of tuning one alarm until it is useless.

1. Fleet saturation: if the metric supports aggregation, use an MQL expression that filters out the batch host or restricts to the web pool and aggregates across instances. If the platform metric cannot aggregate across instance streams, publish a custom metric such as WebPoolCpuSaturated from a small Functions job or autoscaling lifecycle hook that computes the percentage of web instances above 80%. Alarm on that single stream.
2. Batch host: exclude it by dimension if it has a stable dimension, or place batch and web instances in separate compartments/pools and scope the alarm accordingly. Do not exclude by compartment alone if both pools share it.
3. Canary: keep a per-instance alarm on the canary with a shorter window and a distinct ONS topic. A canary breach should page, but it should not page the whole fleet distribution list.
4. Notification routing: CRITICAL fleet saturation goes to prod-pager with a 15-30 minute repeat; canary goes to a small on-call topic; batch warnings go to email only. Put instance name, pool, metric value, and runbook URL in the message.
5. Missing data: add a heartbeat custom metric from the autoscaling group so a real loss of monitoring is detected independently of CPU streams disappearing during scale-in.

Validate in a non-production compartment first. Create a test instance that deliberately burns CPU and a test canary that fails; confirm the fleet alarm does not fire for the single batch-like host, the canary alarm does fire, and each ONS topic reaches only the intended subscription. Use a maintenance window for any production test and avoid lowering production thresholds as a test method.

After rollout, measure:

• PagerDuty incidents per week tagged to web-cpu-high, compared with the batch job schedule.
• Alarm state transitions per day; a healthy fleet should show few OK-to-FIRING transitions.
• Notification volume per topic and per subscription; a sudden rise usually means repeat is too aggressive or a dimension filter is too broad.
• Whether real saturation still pages: inject a controlled load test on enough web instances to cross the fleet threshold.

If incidents still correlate with the batch window, the exclusion or aggregation is wrong. If canary pages are missed, the canary alarm is too dependent on the same missing-data condition as the fleet alarm.