Firewall dataset rejects four percent of events after a parser change

The datasets write to separate data streams, so each backing index builds its own mapping. Dynamic mapping is decided per field by the first document indexed into that backing index. If source.port was first seen as a JSON number, it is mapped as long; the later value - cannot be coerced to a number, so the whole document is rejected with a document_parsing_exception while the rest of the bulk request succeeds. nginx never sends that shape, so it is unaffected. The count gap matches a per-document rejection, not an agent or queue failure.

Confirm before editing:

• GET logs-pfsense.log-default/_mapping/field/source.port and GET logs-pfsense.log-default/_field_caps?fields=source.port,destination.port to see the current type.
• GET _index_template/logs-pfsense.log-default and the referenced component templates to check whether mappings were declared or left dynamic.
• Reproduce with POST _ingest/pipeline/_simulate, submitting one numeric-port document and one - document through the actual pipeline; the failure message names the field and target type.
• Check the agent logs and Elasticsearch logs for mapper_parsing_exception or document_parsing_exception with the pfsense index name.
• Compare source appliance counts with _count on event.dataset: pfsense.log to size the loss.

Do not try to change a mapping in place; mappings are immutable per backing index. Create a new version of the component template that explicitly maps source.port and destination.port as keyword. Keyword accepts 443 and - without rejection, and port values are usually used for term filters and aggregations rather than range math. Map user.id as keyword as well, which accepts both UUID strings and numeric legacy IDs. Keep dynamic: false for the firewall dataset so unseen fields are stored in _source but not indexed, and add a flattened field for vendor-specific extras.

Roll the data stream with POST logs-pfsense.log-default/_rollover so a new backing index picks up the template. Existing backing indices keep the old mapping; query them with a runtime field such as to_long(source.port) if numeric comparison is needed, or reindex only the pfsense data stream with op_type: create into the data stream during a low-traffic window. Because rejected documents never entered Elasticsearch, recover them from the original appliance or agent buffer if either still holds them; otherwise accept the gap.

Add an on_failure block that writes error.message and routes documents to a quarantine data stream rather than silently dropping them. Test both document shapes through _simulate before rollout.

Alert on volume deviation per dataset rather than on cluster health. A simple rule compares the current hourly count for each event.dataset against a 7-day same-hour baseline and fires at a 10 percent drop; this catches rejection, agent policy mistakes and pipeline drops. Pair it with a data-quality check that counts documents containing error.message in the quarantine stream, and alert on any nonzero rate.

Make mapping changes reviewable: store the ingest pipeline and component templates in version control, and run _simulate in CI with a fixture set that includes known bad shapes such as -, empty strings, numeric strings and nested objects. Use dynamic: strict for application datasets whose schema the team controls, so an unexpected field fails during testing instead of silently creating a mapping that blocks later data. For vendor datasets, dynamic: false plus a flattened catch-all prevents future type wars.

Finally, give each integration its own data stream and dataset rather than sharing one stream across teams, and record the owning team in the template metadata. That keeps one team's parser change from altering another team's mapping and makes the volume alert route to the right on-call.