DataWorks scheduling dependency causes silent data gaps
A media company runs a DataWorks workflow: task A extracts from an upstream API, task B cleans and writes to MaxCompute, task C aggregates into a reporting table. A depends on a daily trigger at 02:00. B depends on A; C depends on B. Last month the upstream API was down for 40 minutes on two separate days, so A failed. The team had configured B and C to 'skip' when upstream fails to avoid alert noise. Nobody noticed the reporting table was missing those two days until finance reconciled month-end. The team wants to keep noise down but never ship silent gaps again.
1. Explain how 'skip on failure' produced silent gaps and why it is the wrong default here.
Read the worked answer
'Skip' converts a failure into a successful-looking non-run. Downstream dependencies see the task as satisfied (or skipped), so B and C don't run, but no failure propagates to alerting. The result: the reporting table simply lacks those two days, and nothing signals it.
The core problem is that the pipeline's correctness contract is 'every day has data.' Skip-on-failure violates that contract silently. The team optimized for alert noise (a real concern) but chose a mechanism that trades observability for quiet. The right approach is to reduce noise without hiding gaps: e.g., suppress duplicate alerts, route to a queue, or auto-retry, but always surface that a day is missing. Finance found the gap by reconciliation - the system should have found it first.
Also note: skip semantics can differ from 'fail' in dependency evaluation, so downstream may treat skipped as satisfied. That is exactly how the gap stays invisible.
2. Propose a scheduling and alerting design that keeps noise low but guarantees no silent gaps.
Read the worked answer
Design around explicit data-completeness signals:
- Retry before failing: give A bounded automatic retries (e.g., 3 attempts with backoff) to absorb short API outages like the 40-minute ones. This removes most noise at the source.
- Fail loudly, not silently: replace skip with fail-and-alert. Downstream B and C should not run on missing input; instead they should fail with a clear 'upstream missing' reason so the gap is visible.
- Completeness check as a gate: add a validation task after B that asserts today's partition row count is within an expected range and that the expected date partition exists. If not, fail the workflow and alert. This catches gaps even if A 'succeeds' with zero rows.
- Alert routing and dedup: one alert per incident, routed to the on-call owner, with deduplication so a 40-minute outage doesn't page repeatedly. Separate 'retry in progress' from 'needs human.'
- Reconciliation job: a daily job comparing reporting table dates to expected calendar dates; missing dates raise a ticket automatically.
- Backfill runbook: when a day is missed, a documented backfill re-runs A→C for that partition.
Noise is controlled by retries and dedup, not by hiding failures.
3. How would you verify the new design actually prevents silent gaps before trusting it?
Read the worked answer
Test the failure modes deliberately:
- Simulate API outage: make A fail for a run and confirm: retries fire, then a single alert is raised, B and C do not silently succeed, and the completeness check flags the missing partition.
- Simulate 'success with zero rows': have A return empty data; confirm the completeness gate fails the workflow rather than writing an empty partition that looks valid.
- Simulate partial data: have A return fewer rows than expected; confirm the range check catches it.
- Verify dedup: confirm multiple retries produce one incident, not a page storm.
- Verify backfill: run the backfill for a missed date and confirm the reporting table is restored and reconciliation passes.
- Check dependency semantics: confirm downstream tasks treat a failed upstream as blocked, not satisfied.
Only after these pass should the team trust the pipeline. Document the test cases so future schedule changes are re-validated.