Data Onboarding and Ingestion: Elastic Agent, Integrations, and ECS
The onboarding pipeline
Modern onboarding flows through Fleet: you install a Fleet Server, enroll Elastic Agents with an enrollment token, create an agent policy, and attach integrations to that policy. Each integration supplies the ingest pipeline, index template, and data stream definition for a source; a saved policy change is rolled out to agents on their next check-in. Integration data lands in data streams named logs-<dataset>-<namespace> (for example logs-windows.sysmon_operational-default), with data_stream.dataset and event.dataset usable as filters. The namespace field on the policy controls that suffix and is the usual reason an index pattern doesn't match what a colleague sees.
Elastic Common Schema
ECS is the contract that makes prebuilt rules and dashboards work. Fields such as event.category, event.action, event.outcome, host.name, user.name, source.ip, destination.ip, process.name, process.parent.name, and process.command_line must be populated by the integration's ingest pipeline. When a source lacks a native integration, the Custom Logs integration (or Logstash plus ingest pipelines) lets you ship raw text, but you then own the field mapping and ECS normalization.
Traps worth rehearsing
Policy edits do nothing until agents check in - a "missing integration" is often an undeployed policy change. Assigning a host to two policies is not how it works; one agent, one policy, many integrations. Deleting an integration does not delete previously indexed data. Retention is governed by the data stream's ILM policy, not by the Security app. And an agent can report healthy while a single integration's data stream is empty, so verify document counts rather than agent status.
Rule of thumb: prefer a supported integration over Custom Logs; if you must go custom, prove ECS mapping by checking event.category on a sample document before writing detection rules.