Infrastructure Monitoring
What this domain tests
The infrastructure domain asks how hosts, Kubernetes clusters and cloud services get instrumented, which event types hold the data, and how to alert on the right aggregation without flooding on-call.
Core rules
- The infrastructure agent runs on the host and emits sample events such as SystemSample (CPU, memory, disk, network), ProcessSample, StorageSample and NetworkSample. Kubernetes adds K8sNodeSample, K8sPodSample and K8sContainerSample.
- On-host integrations (for example nginx or MySQL) and cloud integrations for AWS, Azure and GCP bring in service metrics. Polling integrations fetch data on a fixed interval of several minutes; Metric Streams pushes near-real-time data, so the two disagree on short spikes.
- Tags and labels become entity tags, which drive filtering, dashboards, workloads and alert scopes.
- High-cardinality attributes - container IDs, request paths, per-user IDs - inflate ingest volume and make aggregation useless. Filter or drop them at the agent.
- The host-not-reporting condition is a built-in infrastructure alert type that fires when the agent heartbeat stops; it catches a dead agent or host that CPU alerts cannot.
- Infrastructure metrics explain resource pressure; they do not tell you which transaction was slow. Correlate through entity relationships into APM.
Worked example
A 12-node cluster averages 55% memory, so a fleet-level alert at 90% never fires. One node is actually at 98% and is evicting pods. Faceting fixes it: SELECT max(host.memoryUsedPercent) FROM SystemSample FACET hostname shows the outlier, and the alert should be scoped per node or use outlier detection against peers. The same logic applies to CPU: a 1-minute spike to 100% disappears inside a 5-minute average, so query max or a high percentile rather than average.
Traps
- Alerting on fleet averages instead of per-entity values.
- Forgetting that cloud integration polling lags real time.
- Letting high-cardinality attributes into metrics.
- Assuming a silent host is healthy rather than not reporting.