Maintain and Automate Data Workloads
Orchestrate deliberately
Cloud Composer (managed Airflow) is the default for multi-step DAGs across services: scheduling, dependencies, retries, SLAs, and backfills. Set catchup=False deliberately rather than by accident, use retries with exponential backoff, and prefer deferrable or reschedule-mode sensors so idle waiting does not pin worker slots. Cloud Scheduler fires a single cron-triggered request (a DAG, a Cloud Run job, a BigQuery query). Workflows chains HTTP/gRPC calls with retries and error branches. Eventarc turns Cloud Storage object creation or Pub/Sub messages into triggers when a full DAG is overkill.
Operate and control cost
Monitor with Cloud Monitoring dashboards and alerting policies on Dataflow job metrics (system lag, backlog, worker count), Composer health, and BigQuery slot utilization. Use log-based metrics and log sinks for audit, and INFORMATION_SCHEMA.JOBS for who spent what. Control cost with custom quotas, maximum bytes billed per query, BigQuery editions and reservations instead of unbounded on-demand, and Dataflow maxWorkers ceilings.
Automate the lifecycle
Treat pipelines as code: Terraform or Config Controller for infrastructure, Cloud Build for CI/CD of Flex templates and DAG bundles, and versioned SQL models. Plan for DR with multi-region or dual-region buckets, cross-region BigQuery dataset copies, and tested restore procedures. Apply CMEK and VPC Service Controls consistently.
Traps
Cancelling a Dataflow job kills in-flight data while draining lets windows finish - the exam expects draining for a safe stop. A DAG task that writes without a deterministic output path is not idempotent, so backfills duplicate rows. Alerting only on job failure misses the silent failure of a job that runs but produces zero rows - alert on data freshness too.
Rule of thumb: every task should be safely re-runnable, and every alert should tie to a user-visible SLO.