The Midnight Backfill That Doubled a Segment
Your team ingests order events from a Kafka source into a dataset whose XDM schema uses commerce.purchases as an array field. A nightly Airflow job replays the previous 24 hours of events to catch late arrivals, keying each record on orderId. For three weeks the 'Recent High-Value Buyers' audience, which qualifies on total purchase value over 30 days, has been growing roughly 8% per week faster than the finance team's order data supports. The source team confirms no duplicate orders exist upstream. Profile fragment counts for the dataset are also climbing faster than distinct orderId values. The nightly job currently writes with default ingestion settings and no primary identity on the record beyond orderId, which is not mapped to an identity namespace.
1. Diagnose the most likely cause of the inflated audience and fragment counts, and explain why the current replay design produces it.
Read the worked answer
The symptom pair - audience growth outstripping source truth and fragment counts outpacing distinct orderId - points to duplicate profile fragments from non-idempotent replay, not bad source data.
Reasoning chain:
orderIdis not mapped to an identity namespace, so the platform has no key on which to recognize a replayed record as the same entity. Each nightly write creates a new fragment rather than updating an existing one.- The
commerce.purchasesarray is appended on merge rather than replaced, so a replayed order adds its value again. Total purchase value over 30 days therefore inflates monotonically with each replay. - Because the replay window is 24h but late arrivals can exceed that, some events are written more than once across runs, compounding the effect.
The audience is not 'wrong' in its evaluation logic - it is correctly summing an inflated profile. Fixing the segment definition would mask the root cause.
2. Propose a corrected ingestion design and state the specific schema and dataset settings you would change.
Read the worked answer
Schema/identity changes
- Map
orderId(or a composite oforderId+ customer key) to an identity namespace, or better, ensure the record carries a stable person-level identity (e.g., hashed email or CRM ID) so fragments merge onto the right profile. - For the purchases array, use an array upsert with
orderIdas the array key so a replayed order overwrites its own element instead of appending a second copy.
Dataset changes
- Set the dataset's profile ingestion to upsert semantics, not append.
- Enable deduplication on the dataset with an appropriate dedup key and a dedup window that comfortably exceeds your late-arrival tail (e.g., 7 days if late arrivals can lag that long).
Pipeline changes
- Make the replay window configurable and derive it from observed late-arrival latency, not a fixed 24h.
- Emit a run-level idempotency key so a re-run of the same window does not double-write.
Validation
- After the fix, compare distinct
orderIdcount against profile fragment count for the dataset; they should converge. Re-check the audience size against finance's order totals before declaring success - historical inflation will not self-correct for already-appended data, so a one-time data correction or profile deletion may be needed.