Ingesting and Processing the Data
Pick the ingestion service by source type
Object stores and HTTP endpoints - including S3, Azure Blob and on-prem NAS via a transfer agent - use Storage Transfer Service, which supports schedules and incremental transfers. Very large one-time moves from a bandwidth-constrained data center use Transfer Appliance, shipped physically. SaaS sources (Google Ads, Campaign Manager, YouTube, Amazon S3, Teradata, MySQL) use BigQuery Data Transfer Service. Ongoing row-level change capture from Oracle, MySQL, PostgreSQL or AlloyDB uses Datastream, which writes to Cloud Storage, BigQuery or Cloud SQL. Application events land in Pub/Sub.
Land, then transform
A reliable pattern is to land raw data immutably in Cloud Storage (partitioned by date, one prefix per source), then transform into BigQuery. Load jobs from GCS or local files are free of ingestion charge; the Storage Write API and legacy streaming inserts are billed per byte and should be reserved for low-latency needs. External tables and BigLake let you query GCS objects without loading, which is right for open formats queried occasionally but slower and costlier per byte scanned than native tables.
Traps
- Datastream is for continuous CDC, not a one-time dump; a single migration uses a dump and load.
- Storage Transfer Service moves objects, not database rows.
- BigQuery load jobs fail whole files on a single bad record unless you allow max bad records.
- Compress and shard files (100 MB–1 GB) - thousands of tiny files slow every downstream job.
Rule of thumb
If moving the data over the wire would take more than about a week, use Transfer Appliance; otherwise use Storage Transfer Service with a schedule.