The Truncated Nightly Feed
Your team runs a nightly workflow that ingests a 2.1 GB CSV of point-of-sale transactions from a cloud object store into Designer Cloud, then joins it to a 40 MB store-master file and outputs a cleaned table. The workflow has run cleanly for weeks, but for the last three nights the output row count is roughly 30% lower than the source row count, and the missing rows are concentrated in the final third of the file. No errors appear in the run history. The source file's byte size has been growing steadily. A colleague suggests the input tool is silently dropping rows and proposes adding a filter to 'remove bad records.'
1. What is the most likely cause of the shortfall, and how would you confirm it before changing any logic?
Read the worked answer
The pattern - missing rows clustered at the tail of the file, no error, and a file that has been growing - points to an incomplete or partially written source file or a read that terminated early, not to bad records. Real bad records are usually scattered; a contiguous trailing block is a truncation signature.
Confirmation steps:
- Compare the source object's byte size and row count (count lines in the object store) against what the input step reports. If the object itself is short, the problem is upstream of Designer Cloud.
- Check whether the object is still being written when the workflow starts - look at the object's last-modified time versus the workflow start time. A race with the producer produces exactly this tail-truncation pattern.
- Re-run the input alone on a known-complete historical file and verify row counts match.
Only after this would you touch logic. Adding a filter to 'remove bad records' would mask the symptom and silently bake a data-loss step into the pipeline.
2. Assuming the cause is confirmed as a producer/consumer race, what change would you make and why?
Read the worked answer
Break the race rather than the data. Options, in order of preference:
- Land-then-read: have the producer write to a temp/staging prefix and atomically rename or move to the final prefix only when complete; the workflow reads only the final prefix. This is the cleanest fix because completeness is guaranteed by the producer.
- Readiness gate: if you cannot change the producer, add an explicit check (e.g., a manifest file, a sentinel object, or a last-modified-age threshold) that the workflow evaluates before ingesting, and fail fast if not ready.
- File-size/row-count reconciliation: add a validation step that compares ingested row count to an expected count and routes failures to an error path instead of producing a plausible-but-wrong output.
Do not simply retry on a schedule - retries shrink the window but do not close it, and they hide the defect. The exam-tested judgement is: fix the contract with the source, and add a reconciliation check so a future recurrence fails loudly rather than silently.