Critical Alarm Reaches Only Part of the On-Call List

ONS subscriptions are not active until they are confirmed. Check each subscription state in prod-db-critical: email subscriptions may be PENDING or never confirmed, and the new DBA's subscription may not have been confirmed after being added. The HTTPS subscription has its own confirmation and endpoint health requirements; an endpoint that returns errors or does not complete confirmation will not receive messages.

Also check:

• The alarm destination is the correct topic OCID and compartment, not a topic with a similar name.
• The alarm has actually published to ONS; look at alarm history and ONS metrics for published messages.
• IAM policy allows the alarm resource to publish to the topic. A missing policy can leave the alarm FIRING while ONS receives nothing.
• The topic's subscriptions are enabled and not filtered by an unsupported severity condition. ONS does not automatically route by alarm severity unless you create separate topics.
• Email delivery filters: the three recipients may have received the first message, while others are unconfirmed or have mail rules.

The pattern of three email recipients plus zero HTTPS strongly suggests unconfirmed subscriptions rather than a metric problem.

Use separate topics per routing behavior rather than one topic with every subscription.

• prod-db-critical: subscriptions for the incident tool over HTTPS and for DBA phone/SMS where supported. This topic receives only CRITICAL alarms.
• prod-db-warning: email subscriptions for the DBA roster. This topic receives WARNING alarms.
• Alarm destinations: atp-cpu-critical goes to prod-db-critical; warning alarms go to prod-db-warning. Do not rely on ONS to inspect the alarm severity field.
• Repeat: set the CRITICAL alarm repeat to 15-30 minutes, or disable repeat if the incident tool owns escalation. A 5-minute repeat creates duplicate incidents unless the tool deduplicates by alarm name and resource.
• Message format: include the database OCID, compartment, metric value, threshold, and runbook URL. Use a stable deduplication key such as alarm OCID plus resource OCID.
• Acknowledgment: ONS does not know that a human acknowledged an incident. Suppress repeats in the incident tool or set the alarm repeat longer; do not use email replies as acknowledgment.

Test with a non-production alarm and verify each subscription receives only its intended severity.

Improve the message with a custom format that includes the alarm name, severity, metric display name, threshold, current value, database OCID, compartment, and runbook URL. Add the resource's display name if it is stable; do not rely on the default JSON, which often lacks the resource identifier responders need.

Test in layers:

• Test ONS subscriptions directly with a test message or a low-severity test alarm on a non-production topic. Confirm each email and the HTTPS incident endpoint receive it and that the deduplication key behaves as expected.
• Test the alarm evaluation with a non-production database and a deliberately low threshold. Do not lower the production threshold as a test; that can page the whole roster.
• Test severity routing by sending one CRITICAL test and one WARNING test to the separate topics and checking that only the intended subscriptions receive each.
• Verify IAM by checking alarm history and ONS published-message metrics during the test.

After the test, remove the temporary alarm or restore the threshold and confirm the production alarm is back in OK state.