I85PREP
⌘ K
CWIDP study materials
CWIDP study resources

CWIDP: Data Infrastructure and Automation: idempotent, observable pipelines

CWIDP · Work through your exam, one topic at a time.

All guide sections

Data Infrastructure and Automation: idempotent, observable pipelines

What this domain tests

Data Infrastructure and Automation tests whether pipelines are reliable, repeatable, and observable. Expect orchestration, idempotency, delivery semantics, schema evolution, backfills, CI/CD, and data observability rather than tool-specific commands.

Core rules

  • Model pipelines as DAGs with explicit dependencies, retries, timeouts, and owners.
  • Make every task idempotent: rerunning it must produce the same result. Use merge/upsert on a natural key or overwrite a partition, not blind append.
  • Know your delivery semantics: at-most-once, at-least-once, and exactly-once. Exactly-once usually requires idempotent writes plus deduplication, not a guarantee from the queue alone.
  • Use watermarks and checkpoints so a failed run resumes without reprocessing everything.
  • Manage schema evolution with a registry or data contract: add nullable fields compatibly, never silently rename or retype.
  • Separate backfills from daily runs; isolate compute and throttle to protect downstream SLAs.
  • Monitor the four pillars: freshness, volume, schema, and distribution. Alert on anomalies, not just job failure.
  • Keep infrastructure as code and promote changes through CI/CD with tests.

Worked example

A nightly orders pipeline appends raw events, then aggregates revenue. It fails at 02:00 and is rerun at 04:00. Reasoning: a blind append duplicates the morning's events. The fix is to write raw events with a deterministic event_id and merge on it, or overwrite the affected event_date partition from a staging table. The aggregate recomputes from the deduplicated raw layer. A freshness check compares max(event_time) to now; a volume check compares row counts to a 7-day median; a schema check rejects an unexpected null in order_id. The backfill runs with a separate queue and a date range, then reconciliation compares source and warehouse counts before the SLA report is published.

Traps that cost marks

  • Non-idempotent appends that double-count on retry.
  • Alerting only on job failure while data silently arrives stale.
  • Hardcoded credentials or connection strings in pipeline code.
  • Breaking downstream models with an incompatible schema change.
  • No backfill plan, so recovery means manual edits.