Log Analytics Cost, Table Retention, and Diagnostic Gaps
Litware has 220 VMs across three subscriptions, all sending performance counters and event logs to a single Log Analytics workspace (Pay-as-you-go, 90-day retention) via one data collection rule. The workspace ingests 4.8 TB per month, of which 3.1 TB is Perf and 1.2 TB is the SecurityEvent table. The SOC wants 400 days of SecurityEvent retention, and the platform team wants to reduce spend by 35% without losing the ability to investigate an incident up to 30 days old. Current alerting is 45 metric alerts plus 12 log search alerts that run every 5 minutes over 30 days of data; the log alerts take 40+ seconds to fire and three of them have been disabled because they never fired. Diagnostic settings exist for 60 of the 220 VMs; the rest send only guest OS data. A new compliance requirement says all administrative operations at subscription scope must be retained for two years.
1. Design the retention configuration that satisfies the 400-day SecurityEvent requirement and the 30-day investigation requirement while cutting cost. Show the arithmetic and the settings involved.
Read the worked answer
Separate interactive and total retention. Log Analytics bills two different things: interactive retention, where data is queryable with normal KQL and usable by alert rules, and total retention, where data is retained at a much lower rate and must be restored or searched with a search job. Requirement analysis: investigations need 30 days interactive; the SOC wants 400 days of availability, not 400 days of hot querying. So set the workspace or table default interactive retention to 30 days and set the SecurityEvent table's total retention to 400 days. That satisfies the requirement at a fraction of the cost of 400 days interactive.
Arithmetic. SecurityEvent at 1.2 TB/month over 400 days is roughly 16 TB. At 400 days interactive this dominates the bill. With 30 days interactive (about 1.2 TB) and 370 days at the long-term rate, the expensive tier shrinks by more than 90% for that table, which alone can be most of the 35% target.
Perf is the bigger lever. 3.1 TB per month of performance counters is 65% of ingestion and is mostly debugging data, not evidence. Cut it at the source rather than at retention: edit the data collection rule to collect a named list of the counters actually used (typically processor, available memory, logical disk free space, and a couple of network counters) instead of the full set, drop the per-core duplicates, and lengthen the sampling interval from the default to 60 seconds where the alert thresholds tolerate it. Platform metrics are retained by Azure Monitor for 93 days at no ingestion cost, so anything that only needs a graph should not be in the workspace at all.
Keep alert-relevant data interactive. Alert rules can only query interactive data, so do not reduce retention below the window your log alerts use; a 30-day interactive tier is fine here.
Result to present. Perf ingestion reduced by filtering, SecurityEvent hot tier reduced to 30 days with 400-day total retention, remaining interactive retention at 30 days.
2. The log search alerts take 40+ seconds to fire and three never fire at all. Diagnose and redesign the alerting.
Read the worked answer
Why they are slow. A scheduled query rule evaluates its query over the time range set in the rule, not over the evaluation frequency. Scanning 30 days of a multi-terabyte table every 5 minutes is expensive and slow, and the 40+ seconds is query execution plus the evaluation cycle. The rule is also paying for history it does not need: an alert on 'errors in the last 5 minutes' should query 5-15 minutes of data, not 30 days.
Why three never fire. The usual causes, in order of likelihood: the query references a table or column that is not ingested (for example a Perf counter the DCR does not collect, or a column renamed upstream), so the result set is always empty; the threshold or aggregation is wrong for the actual data shape; or the rule's scope points at the wrong workspace or resource. Also check the rule is enabled and its action group is intact, and inspect the rule's own execution history and any failed-query errors rather than guessing.
Redesign. (1) Convert anything based on infrastructure signals to metric alert rules - CPU, memory via VM insights, disk, availability - which evaluate in near real time and cost less. (2) For the remaining log alerts, set the query time range to 5-15 minutes, keep frequency at 1-5 minutes, and use 'number of results greater than 0' with a small window where possible. (3) Use dimensions and splitting so one rule covers many resources instead of 12 hand-written queries. (4) Set the rule's severity and action group deliberately, and add an auto-mitigation or 'resolved' notification so the on-call is not paged twice. (5) Delete or fix the three dead rules - a disabled alert is worse than no alert because it implies coverage.
Verify. Test-fire each rule, confirm the expected rows exist in the workspace for the chosen window, and record the query's own measured runtime in the rule description so the next person can see the budget.
3. Only 60 of 220 VMs have diagnostic settings, and the compliance requirement covers all subscription-level administrative operations for two years. What do you configure, and where should the Activity Log data live?
Read the worked answer
Two different data sources, two different fixes. VM diagnostic settings cover guest and platform data for a resource. Subscription-level administrative operations come from the Activity Log, which is a separate pipeline with its own configuration: a subscription diagnostic setting. Deploying VM diagnostic settings will never capture role assignments, policy changes, or resource deletions.
Activity Log for two years. Create a diagnostic setting at each of the three subscriptions, sending the Administrative, Security, Policy, and Alert categories to a destination with a two-year retention guarantee. Options: a Log Analytics workspace with table-level total retention of 730 days on the relevant table, or a storage account with an immutability policy of two years. For pure audit evidence, storage with immutable blobs plus a resource lock is cheaper and harder to tamper with; for queryability during investigations, Log Analytics is better. A defensible combination is Log Analytics for 90-180 days interactive plus archive to storage with the immutability policy for the balance.
Do not rely on the default. Activity Log retention in the portal is 90 days; anything older is gone unless exported. Export must be configured before the audit period starts; there is no backfill.
Closing the VM gap. The 160 VMs sending only guest OS data are missing platform logs and metrics. Rather than configuring 160 VMs by hand, use Azure Monitor Agent with data collection rule associations, and enforce it with an Azure Policy using a DeployIfNotExists effect that creates the DCR association for any VM without one, plus a remediation task for the existing 160. Also confirm whether any of those VMs still run the legacy Log Analytics agent, which is retired, and whether the guest data is arriving through a different path than you assume.
Prove coverage. Query Heartbeat or AzureActivity and count distinct resources against the expected 220, and create an alert when a VM has not reported for 24 hours - that catches both broken agents and configuration drift.