I85PREP
⌘ K
GCP Data Practitioner study materials
GCP Data Practitioner study resources

GCP Data Practitioner: Orchestrate Data Pipelines

GCP Data Practitioner · Work through your exam, one topic at a time.

All guide sections

Orchestrate Data Pipelines

What this domain tests

Orchestrate data pipelines tests scheduling, dependencies, retries, and monitoring for repeated data movement and transformation. Expect Cloud Composer (managed Apache Airflow), Cloud Scheduler, Workflows, Dataflow templates, and idempotent design.

Core rules

  • Cloud Composer runs Airflow DAGs. Define tasks with operators such as BigQueryInsertJobOperator, GCSToBigQueryOperator, and DataflowStartFlexTemplateOperator; set dependencies with >>.
  • Use sensors to wait for files or partitions, but set timeout and mode carefully so a task does not hang a worker.
  • Cloud Scheduler is a cron trigger for a single HTTP endpoint, Pub/Sub message, or App Engine route. It does not provide multi-step dependency management by itself.
  • Workflows is serverless orchestration for API calls defined in YAML or JSON; it is suited to coordinating services, not to heavy data processing.
  • Dataflow templates make pipelines reusable; Flex templates package custom code. Schedule them from Composer or Cloud Scheduler when the pipeline is a single step.
  • Design for idempotency: write to a date-partition overwrite, use MERGE on a business key, or name output files deterministically. Retries must not create duplicate rows.
  • Set retries, execution timeouts, and alerting on task failure. Airflow parse errors come from top-level code; keep DAG files free of heavy work and dynamic datetime calls at parse time.

Worked example

A daily pipeline must wait for a CSV in Cloud Storage, load it to BigQuery, run a transformation query, and refresh a dashboard table. Reasoning: use a Composer DAG. A GCS sensor waits for the file; a BigQuery load task writes to a staging table partitioned by run date; a transformation task runs a MERGE into the reporting table; dependencies chain wait >> load >> transform. Set retries=2 and an SLA alert. Make the load idempotent by overwriting the same date partition, so a retry does not duplicate data. Cloud Scheduler alone would be insufficient because it cannot express the file-wait dependency or the multi-step chain.

Traps

  • Using Cloud Scheduler for a multi-step pipeline with dependencies.
  • Appending on every retry and creating duplicates.
  • Forgetting time zone differences in cron schedules.
  • Running heavy code at DAG parse time, which slows the scheduler.
  • Using a sensor with no timeout and blocking a worker slot.
  • Assuming Workflows is a data processing engine; it orchestrates calls.