Ingest and Process the Data
Batch, CDC, and streaming paths
For online bulk copies use Storage Transfer Service (GCS, S3, HTTP sources; scheduled, bandwidth-controlled). For petabytes over a constrained link use Transfer Appliance - offline shipping beats any tuned rsync. For database replication use Datastream, which reads change streams from MySQL, PostgreSQL, Oracle, and SQL Server and lands Avro/JSON in GCS or rows in BigQuery; use Database Migration Service when the goal is migrating the database itself. SaaS sources (Ads, Campaign Manager, Teradata) come through BigQuery Data Transfer Service connectors.
Streaming ingestion is usually Pub/Sub feeding Dataflow. Know ordering keys, exactly-once delivery subscriptions, dead-letter topics (which need a forwarding subscription and IAM), and seek/replay. For Kafka estates, the Pub/Sub connector or Kafka Connect preserves the producer contract.
The processing layer
Prefer columnar, splittable formats: Avro for records, Parquet/ORC for columnar scans. Compression matters - gzip is not splittable, so one large .gz file is read by a single worker, a classic slow-pipeline trap. Aim for files in the hundreds of MB, never millions of tiny files. Use Dataflow templates (classic or Flex) for repeatable, parameterized jobs, and run Sensitive Data Protection (DLP) de-identification inside the pipeline when PII must be redacted before landing.
Traps
Dead-letter queues capture bad records only if explicitly configured; retries without backoff amplify load; a pipeline that dies on one malformed record needs a dead-letter branch, not more workers. Datastream is replication, not general-purpose ETL.