New Relic Reliability Engineer cheat sheet

APM 360 bundles APM plus four capability groups: Error Tracking (formerly Errors Inbox), Service Levels, Application Security, and CodeStream integrations; the APM agent itself is required for all of them.

Golden signals in New Relic APM: throughput (requests per minute, rpm), response time (average, percentile, and Apdex), error rate (percentage of transactions failing), and saturation via host/infra metrics.

Apdex formula: (satisfied + tolerating/2) / total, where satisfied = response time <= T and tolerating = T < response time <= 4T; T is set per app in the agent config, default 0.5 s for web transactions.

Transaction traces capture slow transactions above a threshold; distributed traces (with trace.id) stitch spans across services and are the basis for the distributed tracing UI and service maps.

Error Tracking groups errors by fingerprint; the error group is the unit for triage, and error profiles add stack-trace-level aggregation. Marking an error group ignored or resolved changes its status, not the underlying event data.

Service Levels are defined from NRQL queries or from entity/APM metrics with an SLI (good events / valid events) and an SLO target over a rolling window (for example 99.9% over 28 days); error budget = 1 - SLO target.

Service maps derive dependencies from observed traffic between entities; uninstrumented or external calls appear as inferred external services and can be hidden or grouped in the map settings.

APM agent sampling: the agent reports a sample of transactions (default ~10% of non-error transactions, configurable via transaction_tracer settings), so metrics are extrapolated while traces are sampled.

Browser monitoring has two injection modes: auto-instrumentation via the APM agent (copy/paste or agent-injected) and the standalone browser agent (a JS snippet); the standalone agent can run without APM.

Core Web Vitals tracked: LCP (Largest Contentful Paint, good <= 2.5 s), INP (Interaction to Next Paint, good <= 200 ms, replaced FID in 2024), and CLS (Cumulative Layout Shift, good <= 0.1).

Browser PageView events carry timing attributes such as duration, timeToFirstByte, firstContentfulPaint, largestContentfulPaint, and interactionToNextPaint; query with NRQL on the PageView event type.

AJAX and JavaScript error events are separate event types (AjaxRequest, JavaScriptError); JS errors are grouped by error message and stack, and source maps are required to map minified stacks to original code.

Session replay and session traces: session traces capture a sampled waterfall of a page load; session replay records DOM interactions and is privacy-gated by masking rules configured per app.

Synthetic monitoring: simple (ping), browser (scripted Selenium-like), and API (multi-step HTTP) monitors; scripted monitors run on public or private (containerized) minion locations.

Synthetics results surface as SyntheticCheck and SyntheticRequest events; failures can open issues and feed alert policies like any other NRQL condition.

Mobile monitoring (Android, iOS, React Native, etc.) reports crash reports, handled exceptions, network requests, and MobileRequest/MobileHandledException event types, with crash symbolication via dSYM/ProGuard mapping uploads.

Real User Monitoring vs synthetic: RUM reflects actual user traffic (and is subject to ad blockers and sampling), while synthetic is deterministic, location-controlled, and runs on a schedule.

New Relic AI (the AI assistant) answers natural-language questions by generating and running NRQL against the account; it is scoped by the user's permissions and data access.

Applied Intelligence detects anomalies and correlates them into incident-like issues; anomaly detection uses baselines (predicted vs actual) rather than static thresholds.

Alert condition types relevant to automation: NRQL, baseline/anomaly, outlier, and static; baseline conditions learn per-entity behavior and fire on deviation, which reduces hand-tuned thresholds.

AI-driven root cause: related-entity analysis and incident correlation surface probable causes by comparing signals across entities in the same service topology.

Change tracking events (Deployment, Change) annotate dashboards and can be correlated with error/latency shifts; the deployment marker API records them via the GraphQL/NerdGraph mutation.

Error profiles and error analysis use clustering/ML to group similar stack traces, so triage works on groups rather than individual occurrences.

Automation surfaces: workflows (alert-driven actions such as notifications, webhooks, and incident creation) and the NerdGraph API for programmatic alert and dashboard management.

AI outputs must be validated: generated NRQL should be reviewed for correct event types, time ranges, and facets before being promoted to dashboards or alert conditions.

Guardrails: do not send secrets or PII into prompts; AI features operate within the account's data boundary and respect the same role-based access controls as the UI.

The infrastructure agent is a host-based agent (Linux/Windows) that collects system metrics (CPU, memory, disk, network) and forwards them as SystemSample, ProcessSample, StorageSample, and NetworkSample events.

On-host integrations (for example Apache, MySQL, NGINX, Redis) run as part of the infrastructure agent and report into the same event namespace; each integration has its own event type.

Cloud integrations (AWS, Azure, GCP) are polled by New Relic rather than pushed by an agent; the integration polls cloud APIs and requires read-only credentials or a linked account.

Kubernetes: the New Relic Kubernetes integration uses a DaemonSet (and a kube-state-metrics deployment) to report K8sNodeSample, K8sPodSample, K8sDeploymentSample, and related events, plus cluster explorer views.

Entity model: infrastructure entities are hosts, containers, and cloud services; they appear in the entity explorer and can be tagged for filtering and alert scoping.

Metric vs event data: infrastructure dimensional metrics (Metric event type) support NRQL aggregation with facet and are the recommended path for high-cardinality infrastructure data.

Host not reporting: the infrastructure agent sends a heartbeat; a host with no data for a period is flagged, and alert conditions can target the Infrastructure agent's host-not-reporting signal.

Tagging strategy: use consistent tags (environment, team, service) applied via agent config or cloud resource tags; tags drive entity grouping, dashboards, and alert scope.

Logs are ingested via the infrastructure agent (fluent-bit based log forwarding), the Log API (HTTP endpoint), or direct integrations; each log line becomes a Log event with message, level, and timestamp attributes.

NRQL log queries use FROM Log with attributes such as level, message, and entity.guid; common patterns are SELECT count(*) FROM Log FACET level and filtering with WHERE level = 'ERROR'.

Log parsing: grok patterns and built-in parsing rules extract structured attributes from unstructured messages; parsed attributes become queryable fields.

Log patterns cluster similar messages into templates so high-volume logs can be triaged by pattern rather than by individual line.

Data retention and partitioning: logs are stored in the Log event type with account-level retention; high-volume accounts should use drop filters and sampling to control ingest cost.

Correlation: logs can be linked to APM entities via entity.guid or trace.id, enabling navigation from a slow transaction or error to related log lines.

Drop filters and data partitions: drop filters discard matching log events at ingest; data partitions split data into named partitions with independent retention.

Alerting on logs: NRQL alert conditions can target Log (for example error count over a window), with thresholds evaluated on the aggregated query result.

Security-relevant logs: audit and security events should be routed to a separate data partition and retained longer, since default retention may be shorter than compliance requires.

Alerting model: policies contain conditions; conditions evaluate NRQL or metric queries; violations open incidents; issues group related incidents per entity and condition.

Alert condition evaluation: a sliding window (for example 5 minutes) plus a threshold and a duration (for example 5 minutes) determines when a violation opens; the aggregation (average, sum, count) must match the signal.

Alert severity levels: Critical, Warning, and Info; notification destinations (email, Slack, PagerDuty, webhooks) are attached to workflows or policies.

Dashboards are built from widgets backed by NRQL; template variables parameterize queries, and dashboard pages group widgets; dashboards can be shared via permalinks and public read-only links.

NerdGraph (GraphQL) is the programmatic API for creating dashboards, alert policies, and entities; the same API backs the CLI and Terraform provider.

OpenTelemetry: New Relic accepts OTLP over HTTP/gRPC at the OTLP endpoint with a license key; OTel spans map to Span events and metrics to dimensional Metric data.

OTel mapping: service.name maps to the entity name, and resource attributes become entity tags; the OTLP exporter is configured with the New Relic endpoint and api-key header.

Fleet Control (Fleet Management): a centralized way to manage agents and integrations across hosts, with fleet policies, version pinning, and staged rollouts.

Fleet Control uses a control plane (the New Relic fleet manager) and per-host agents that report health and version, so upgrades can be applied by policy rather than host by host.

Application Security (formerly Vulnerability Management) ingests scanner findings (SAST, DAST, SCA, container) and correlates them with the APM entities that actually receive traffic.

Risk prioritization is based on exposure: a vulnerability in a library used by an internet-facing, high-traffic service ranks above the same vulnerability in an unused dependency.

Security scores combine severity (CVSS), exploitability, and runtime exposure; the score drives the ordering in the security UI and can feed alert conditions.

Vulnerability events are queryable via NRQL; findings carry attributes such as severity, package, and the linked entity, enabling dashboards of open criticals by service.

Security alerting: NRQL conditions over vulnerability data can open issues when a critical finding appears on a monitored entity, routing to the owning team.

SBOM and dependency tracking: software composition analysis identifies third-party packages and versions, and version drift is the signal for upgrade prioritization.

Runtime vs static: runtime evidence (which service actually calls the vulnerable path) is the differentiator; static findings without runtime exposure are deprioritized.

Access control: security data respects account roles; restricting who can view or acknowledge findings is part of the risk workflow, and audit logs record changes.

Compliance mapping: findings can be mapped to frameworks (for example OWASP Top 10 or CWE) to support reporting, but the mapping is a reporting layer over the raw finding.

APM 360 bundles APM plus Error Tracking, Service Levels, Application Security, and CodeStream integrations; the APM agent is required for all of them.

Golden signals: throughput (rpm), response time (average, percentile, Apdex), error rate, and saturation via host/infra metrics.

Apdex = (satisfied + tolerating/2) / total, with satisfied <= T and tolerating <= 4T; T defaults to 0.5 s for web transactions.

Transaction traces capture slow transactions above a threshold; distributed traces with trace.id stitch spans across services and feed service maps.

Error Tracking groups errors by fingerprint; the error group is the triage unit, and marking resolved or ignored changes status, not the underlying event data.

Service Levels define an SLI (good/valid events) and an SLO target over a rolling window; error budget = 1 - SLO target.

Service maps derive dependencies from observed traffic; uninstrumented calls appear as inferred external services.

APM agents sample transactions (default ~10% of non-error transactions), so metrics are extrapolated while traces are sampled.

Browser monitoring has two injection modes: auto-instrumentation via the APM agent and the standalone browser agent JS snippet, which can run without APM.

Core Web Vitals thresholds: LCP good <= 2.5 s, INP good <= 200 ms (replaced FID in 2024), CLS good <= 0.1.

Browser PageView events carry duration, timeToFirstByte, firstContentfulPaint, largestContentfulPaint, and interactionToNextPaint.

AJAX and JavaScript errors are separate event types (AjaxRequest, JavaScriptError); source maps map minified stacks to original code.

Session traces capture a sampled page-load waterfall; session replay records DOM interactions and is gated by per-app masking rules.

Synthetic monitors are simple (ping), browser (scripted), or API (multi-step HTTP), running on public or private minion locations.

Synthetics results surface as SyntheticCheck and SyntheticRequest events and can feed alert policies like any NRQL condition.

Mobile monitoring reports crashes, handled exceptions, and network requests; crash symbolication requires dSYM or ProGuard mapping uploads.

RUM reflects actual traffic and is subject to ad blockers and sampling; synthetic is deterministic, location-controlled, and scheduled.

New Relic AI answers natural-language questions by generating and running NRQL scoped to the user's permissions and data access.

Applied Intelligence detects anomalies and correlates them into issues using baselines (predicted vs actual) rather than static thresholds.

Alert condition types: NRQL, baseline/anomaly, outlier, and static; baseline conditions learn per-entity behavior and fire on deviation.

Related-entity analysis and incident correlation surface probable causes by comparing signals across entities in the same topology.

Deployment and Change events annotate dashboards and can be correlated with error or latency shifts; markers are recorded via NerdGraph.

Error profiles cluster similar stack traces so triage works on groups rather than individual occurrences.

Automation surfaces: workflows (alert-driven notifications, webhooks, incident creation) and the NerdGraph API for programmatic management.

Generated NRQL must be reviewed for correct event types, time ranges, and facets before promotion to dashboards or alert conditions.

Never send secrets or PII into prompts; AI features respect the same role-based access controls as the UI.

The infrastructure agent (Linux/Windows) collects system metrics and forwards SystemSample, ProcessSample, StorageSample, and NetworkSample events.

On-host integrations (Apache, MySQL, NGINX, Redis) run within the infrastructure agent and report their own event types.

Cloud integrations (AWS, Azure, GCP) are polled by New Relic using read-only credentials or a linked account, not pushed by an agent.

Kubernetes integration uses a DaemonSet plus kube-state-metrics to report K8sNodeSample, K8sPodSample, and K8sDeploymentSample events.

Infrastructure entities are hosts, containers, and cloud services; they appear in the entity explorer and can be tagged for filtering and alert scoping.

Infrastructure dimensional metrics (Metric event type) support NRQL aggregation with facet and are recommended for high-cardinality data.

The agent sends a heartbeat; a host with no data for a period is flagged, and alert conditions can target host-not-reporting.

Use consistent tags (environment, team, service) via agent config or cloud resource tags to drive entity grouping, dashboards, and alert scope.

Logs are ingested via the infrastructure agent (fluent-bit based forwarding), the Log API HTTP endpoint, or direct integrations; each line becomes a Log event.

NRQL log queries use FROM Log with attributes such as level, message, and entity.guid; typical pattern: SELECT count(*) FROM Log FACET level.

Grok patterns and built-in parsing rules extract structured attributes from unstructured messages, making them queryable fields.

Log patterns cluster similar messages into templates so high-volume logs can be triaged by pattern rather than individual line.

Logs are stored in the Log event type with account-level retention; drop filters and sampling control ingest cost.

Logs link to APM entities via entity.guid or trace.id, enabling navigation from a slow transaction to related log lines.

Drop filters discard matching events at ingest; data partitions split data into named partitions with independent retention.

NRQL alert conditions can target Log, with thresholds evaluated on the aggregated query result over a sliding window.

Route audit and security logs to a separate data partition with longer retention when default retention is shorter than compliance requires.

Alerting model: policies contain conditions; conditions evaluate NRQL or metric queries; violations open incidents; issues group related incidents per entity and condition.

A condition combines a sliding window, a threshold, and a duration; the aggregation (average, sum, count) must match the signal being measured.

Severity levels are Critical, Warning, and Info; notification destinations (email, Slack, PagerDuty, webhooks) attach to workflows or policies.

Dashboards are built from NRQL-backed widgets; template variables parameterize queries, pages group widgets, and public read-only links can be shared.

NerdGraph (GraphQL) is the programmatic API for dashboards, alert policies, and entities, and backs the CLI and Terraform provider.

OpenTelemetry: New Relic accepts OTLP over HTTP/gRPC with a license key; OTel spans map to Span events and metrics to dimensional Metric data.

OTel mapping: service.name becomes the entity name and resource attributes become entity tags; configure the exporter with the New Relic endpoint and api-key header.

Fleet Control centrally manages agents and integrations across hosts with fleet policies, version pinning, and staged rollouts.

Fleet Control uses a control plane plus per-host agents that report health and version, so upgrades apply by policy rather than host by host.

Application Security ingests scanner findings (SAST, DAST, SCA, container) and correlates them with APM entities that actually receive traffic.

Risk prioritization is exposure-based: a vulnerability in a library used by an internet-facing, high-traffic service outranks the same issue in an unused dependency.

Security scores combine severity (CVSS), exploitability, and runtime exposure, driving ordering in the security UI and feeding alert conditions.

Vulnerability findings are queryable via NRQL with attributes such as severity, package, and linked entity, enabling dashboards of open criticals by service.

NRQL conditions over vulnerability data can open issues when a critical finding appears on a monitored entity and route to the owning team.

Software composition analysis identifies third-party packages and versions; version drift is the signal for upgrade prioritization.

Runtime evidence (which service actually calls the vulnerable path) differentiates findings; static findings without runtime exposure are deprioritized.

Security data respects account roles; restricting who can view or acknowledge findings is part of the risk workflow, and audit logs record changes.

Findings can be mapped to frameworks such as OWASP Top 10 or CWE for reporting, but the mapping is a reporting layer over the raw finding.