Litware: An Alert That Never Fired and an Ingestion Bill That Will Not Stop

The rule never saw a qualifying data point.

A log alert queries the Perf table on its own schedule. Two independent windows are in play: the query time range / bin size (5 minutes, averaged) and the evaluation frequency (every 5 minutes). A 4-minute excursion at 94% is diluted when averaged into 5-minute bins that also contain idle time, so the bin average can stay below 90. If evaluations are offset relative to the spike, the spike may not be inside a queried window at all.

Re-author it:

• Use a metric alert on the platform metric Percentage CPU instead of a log alert on a collected counter. Platform metrics are emitted at 1-minute granularity, are retained for 93 days, and are evaluated by the platform rather than by a scheduled query.
• Set the alert window to something matched to the event you care about - e.g. 3 minutes - and decide explicitly whether to use static or dynamic thresholds. Static 90 with a 3-minute window would have caught 94% sustained for 4 minutes.
• If the condition genuinely needs log data (multi-signal, correlated), keep a log alert but set the bin to 1 minute and the evaluation frequency to 1 minute, and test with a deliberate load.

A 10-second Perf sample interval does not help: the alert aggregates, so finer sampling only buys precision that the 5-minute bin throws away.

Cut at the collection layer first - it is the only lever that reduces every downstream cost.

1. Replace legacy agents with Azure Monitor Agent and Data Collection Rules. Per-DCR control means the 300 VMs that do not need six counters stop sending six counters. Target: keep CPU, memory and disk at 1-minute for critical tiers, 5-minute for the rest, and drop the 10-second interval - it is the primary driver of the 4.2 TB.
2. Filter Syslog and Event Log at the source. Collect Warning and above for Syslog; exclude the high-volume, low-value Windows event IDs. This is usually the largest single reduction in a mixed Linux/Windows estate.
3. Table plans and per-table retention. Keep 90 days interactive only on the tables you actually query during incidents; set other tables to a shorter interactive period with total retention (archive) covering the rest. Archived data stays queryable through search jobs and restore, so the 90-day investigative promise is kept at a fraction of the interactive cost.
4. Daily cap as a circuit breaker, not a budget tool - set it above normal peaks so it never truncates an incident day.
5. Deduplicate workspaces. Three subscriptions into one workspace is already right; check no VM is dual-homed through both a legacy agent and AMA during migration, which silently doubles rows.

Measure per-table GB/day before and after each change, so the 30% is attributable rather than assumed.

Separate retention from queryability, and treat the two audiences differently.

• Security events: set the SecurityEvent table (or the AMA equivalent) to a short interactive retention with total retention of 365 days. The data is preserved and can be queried through a search job or restored to a table for investigation; it simply is not sitting in the hot, always-indexed tier.
• Alternative or complement: export to an immutable storage account via a data export rule and keep it under a legal-hold or immutability policy. This is often what auditors actually want - an unalterable copy outside the SIEM, which also survives workspace deletion.
• Do not raise workspace-wide retention to 365 days to solve one table's requirement; that bills every table at the long-retention rate.

Then close the operational gap: an alert must still fire on the security events in real time - the retention change affects storage and query, not detection. Verify the analytics rules run against the live table.

Finally, state the assumption explicitly in the design doc: 'retained' must be defined as retrievable within an agreed time. A search job over archived data is not instantaneous, and if the requirement is 'produce within 15 minutes', the answer changes to interactive retention plus export.