Elastic Certified Engineer: Data Ingestion

What to know

Documents enter through the single-document APIs, _bulk, _reindex, or external shippers such as Logstash and Beats. _bulk is NDJSON: an action line (index, create, update, delete), an optional source line, Content-Type: application/x-ndjson, and a trailing newline. Ingest pipelines (PUT _ingest/pipeline/<id>) chain processors - set, rename, remove, grok, date, convert, split, script, foreach, pipeline - and can be attached per request (?pipeline=), as index.default_pipeline, or as final_pipeline. Composable index templates (_index_template) use index_patterns, priority, composed_of and template to control mappings and settings for newly created indices.

Traps

• A _bulk request returns HTTP 200 even when individual documents fail. You must check the top-level "errors": true flag and the status of each item; a 409 usually means create hit an existing document.
• create fails if the ID exists; index overwrites.
• Grok expressions must match the whole field value unless you deliberately anchor them. Test with _ingest/pipeline/_simulate before shipping.
• Pipelines run at index time only. Existing documents must be reprocessed with _update_by_query using the same pipeline.
• _reindex aborts on version conflicts by default; add "conflicts": "proceed", and consider slices for parallelism.

Rule of thumb

Never trust a _bulk response without parsing the body: check errors, then inspect every item with status >= 300 before acknowledging success.