Syslog JSON Parsing, Time Skew, and Duplicate MES Events
Metro Manufacturing sends JSON events from a custom MES application to a Cortex XSIAM Broker VM over syslog. The payload is valid JSON, but the timestamp field is a string of epoch milliseconds in local time, and some stack traces span five lines. The application writes each event twice - once to a primary collector and once to a secondary collector for redundancy - and both streams reach the same dataset. The MES server clock is 12 minutes behind the domain controllers. After onboarding, analysts see 30% of events with no parsed fields, _time values that are several hours off, and duplicate alerts in the incident queue. The SOC wants queries that reliably find the unparsed events and duplicates, and a parser design that preserves the original event for audit.
1. Diagnose the parsing and timestamp failures and outline the parser changes.
Read the worked answer
Treat the three symptoms separately. First, the 30% unparsed events are likely the multiline stack traces and any event where the syslog RFC header precedes the JSON, so the parser sees a non-JSON prefix. Strip the syslog header, extract the JSON body, and keep the full original payload in _raw_log for audit. Configure multiline handling so lines beginning with whitespace or stack-trace markers are joined to the preceding event; otherwise each line becomes a separate malformed event.
Second, _time is wrong because the source field is an epoch-milliseconds string in local time while the platform expects epoch milliseconds in UTC. Parse the field as an integer, apply the known timezone offset to produce UTC, and assign that to _time. Preserve the original local-time value in a separate field such as event_time_local. Do not simply use ingest time; that hides the offset and breaks timelines.
Third, the 12-minute clock skew is a source problem. Fix NTP on the MES server and domain controllers. A parser can apply a temporary correction for a known offset, but a permanent parser workaround will break when the clock is fixed.
Map parsed fields to XDM, define the dataset explicitly, and test with representative samples including a five-line stack trace. Monitor parse success rate after deployment. For duplicates, add a stable source event ID and dedupe at the source or in queries; do not assume the platform will drop duplicate deliveries.
2. Write XQL to find (a) events missing a critical parsed field and (b) duplicate event IDs in the last 24 hours. Explain how you would validate the results.
Read the worked answer
Replace metro_mes_raw with the actual dataset name. For missing parsed fields:
dataset = metro_mes_raw
| filter _time > current_time() - 86400000
| filter event_id = null or event_id = ""
| fields _time, _raw_log, event_id, event_type
| limit 100
For duplicate event IDs:
dataset = metro_mes_raw
| filter _time > current_time() - 86400000
| comp count() as occurrences by event_id
| filter occurrences > 1
| sort desc occurrences
Validation: run the queries as an admin first to separate data problems from permission problems. Pick a known duplicate event ID from the raw collector and confirm it appears with the expected count. Check that _time matches the corrected UTC event time, not ingest time. Run a narrow one-hour window against a source count to confirm the query is not missing events because of retention or dataset routing. Then run as a tier-1 analyst to confirm RBAC does not change the result. Finally, compare _raw_log for a duplicate pair to confirm the two copies are truly the same event, not two legitimate events that share a business key.
3. What controls would you add to prevent recurrence and keep auditability?
Read the worked answer
Prevent recurrence at the source and at the parser boundary:
- Enforce NTP and monitor clock offset; alert when a log source drifts beyond a threshold.
- Require a unique event ID from the MES application and configure the secondary collector as standby rather than active-active duplicate delivery, or dedupe on the event ID before upload.
- Version the custom parser and keep sample inputs as regression tests; run the samples before every parser change.
- Preserve
_raw_logand the original local timestamp in dedicated fields so audit can reconstruct what was received. - Monitor parse success rate and duplicate rate per dataset; create an alert when the unparsed percentage exceeds an agreed threshold.
- Document the field mappings and timezone assumption in the data source runbook.
Separate raw and normalized access where possible: analysts who need to hunt raw payloads should not automatically have broad query access to unrelated datasets. If the duplicate alerts are generated by correlation rules, add a dedupe key based on event_id and a time window rather than suppressing by rule name alone. After the fix, replay a day of historical data and compare incident counts to confirm duplicates actually fell.