Elastic Observability fundamentals and data ingestion
Domain 1: Elastic Observability fundamentals and data ingestion
This domain tests whether you can move data from a source into Elasticsearch in a shape that later searches, dashboards, and lifecycle policies can use. Expect questions on Elastic Common Schema (ECS), data streams, ingest pipelines, index and component templates, Fleet-managed Elastic Agent policies, and ILM/data tiers.
ECS, documents, and data streams
Elastic Observability assumes ECS field names. Logs usually carry @timestamp, message, log.level, host.name, service.name, and event.dataset; metrics add metricset.name and numeric fields. A data stream is the write target for append-only event data and follows ds-<type>-<dataset>-<namespace>, for example logs-nginx.access-production. Writes go to the stream name, not a backing index; rollover creates hidden .ds-* backing indices. Data streams require @timestamp and the data_stream.* fields, which integrations add automatically.
Ingestion paths and parsing
Fleet-managed Elastic Agent runs integrations that ship with an ingest pipeline. Logstash and standalone Beats are alternatives; all can send to Elasticsearch ingest nodes. Ingest pipelines are ordered processors: grok or dissect parses text, json expands structured logs, date sets @timestamp, drop removes noise, rename/set align fields to ECS, and enrich adds lookup data. A data stream's index template can set index.default_pipeline and index.final_pipeline.
Templates, mappings, and lifecycle
Index templates are composed of component templates and apply by index pattern or data stream name. Explicit mappings prevent dynamic mapping from turning a status code into text with a .keyword subfield. ILM moves indices through hot, warm, cold, frozen, and delete phases; rollover is triggered by size, age, or document count. Time series data streams (TSDS) add time_series_dimension and metric_type settings and support downsampling.
Worked example
A custom JSON log has {"ts":"2025-01-04T10:15:00Z","level":"warn","msg":"queue full","svc":"billing"} and must land in logs-billing.app-production with ECS names. Reasoning: the JSON processor parses the message; a date processor reads ts into @timestamp with the correct format; rename maps msg to message, level to log.level, and svc to service.name; a drop processor with if: ctx.log.level == 'debug' removes debug noise. The component template maps log.level and service.name as keyword so aggregations work, and the index template sets the pipeline as default.
Traps
Do not assume the event's own time becomes @timestamp; without a date processor Elasticsearch uses ingest time. Do not send to a backing index. Dynamic mapping on unbounded fields causes mapping explosions. Fleet policy changes need the agent to check in; an offline agent keeps the old policy. ILM only applies when the index matches a template with a lifecycle setting.