GCP Data Engineer cheat sheet

Batch pipelines operate on bounded data with a known end; streaming pipelines operate on unbounded data that never ends - Apache Beam expresses both with the same PTransform API.

Beam windowing types: fixed (tumbling), sliding, and session windows; each requires a trigger and allowed lateness to define when late data stops being accepted for a pane.

A watermark is the system's estimate that all event-time data before that point has arrived; a trigger such as AfterWatermark fires results once the watermark passes the window end.

Accumulation modes control how repeated firings for one window are emitted: discarding emits only new elements per pane, accumulating re-emits the full pane each time.

Pub/Sub delivers at least once; exactly-once correctness in Dataflow streaming is achieved by deduplicating on a stable message attribute or by making writes idempotent.

Choose Cloud SQL for regional OLTP under a few TB; Spanner for horizontally scalable, globally consistent relational workloads needing 99.999% availability; Bigtable for high-throughput low-latency point reads/writes above roughly 1 TB.

Choose BigQuery for serverless analytical SQL over large datasets, Dataproc for lift-and-shift Spark/Hadoop with existing code, and Dataflow for unified batch/stream pipelines with autoscaling.

Bigtable schemas are designed around a single row key sorted lexicographically: sequential keys concentrate traffic on one node, so hash or reverse prefixes to spread writes.

Ephemeral Dataproc clusters created per job and deleted afterward remove the need for long-lived cluster sizing and are the default pattern for bursty Spark workloads.

Storage Transfer Service performs scheduled, managed copies from on-premises, other clouds, or other buckets into Cloud Storage; Transfer Appliance is used for offline petabyte transfers when bandwidth is the constraint.

Datastream is a serverless CDC service that streams inserts, updates, and deletes from MySQL, PostgreSQL, Oracle, and SQL Server into BigQuery, Cloud Storage, or Cloud SQL.

Pub/Sub message retention is per subscription: the default is 7 days and the maximum is 31 days, after which unacknowledged messages are discarded.

Pub/Sub subscription types include pull, push, BigQuery, Cloud Storage, and BigLake, which write messages directly to the target without custom subscriber code.

BigQuery Data Transfer Service handles recurring, code-free loads from SaaS sources such as Google Ads, YouTube, and Amazon S3.

Dataflow Flex templates package pipeline code and custom dependencies into a container image, allowing runtime parameterization and non-standard libraries.

Dataproc autoscaling adds and removes workers based on pending YARN memory and cores, keeping ephemeral clusters aligned with job demand.

Cloud Scheduler combined with Cloud Functions or Cloud Run is the lightweight ingestion pattern for small, periodic API pulls where a full Dataflow pipeline is unnecessary.

Cloud Storage minimum storage durations: Standard none, Nearline 30 days, Coldline 90 days, Archive 365 days; deleting earlier bills the remaining duration.

Autoclass moves objects between Standard and colder classes based on observed access patterns without early-deletion or retrieval transition charges.

Cloud Storage lifecycle rules act on conditions such as object age, creation date, version count, and storage class, and can delete, transition, or apply Autoclass.

Bucket retention policies and bucket lock enforce immutability for a fixed period, supporting compliance and WORM requirements.

BigQuery partitioned tables support at most 4,000 partitions; exceeding this raises errors, so choose day, month, or year granularity to match data volume.

BigQuery clustering supports up to four columns, sorted in declared order; pruning works best when filter columns appear in the same order as declared.

Bigtable garbage collection is configured per column family and can delete cells older than a maximum age or retain only the most recent N versions.

BigLake tables expose Cloud Storage data in open formats such as Parquet, ORC, and Avro to BigQuery and other engines with row- and column-level access control.

CMEK encrypts data with customer-managed Cloud KMS keys and can be applied per bucket, per BigQuery dataset, or per Bigtable cluster and table.

BigQuery is designed for ELT: load raw data, then transform in SQL with CREATE TABLE AS SELECT or scheduled queries rather than transforming before load.

Policy tags in Data Catalog applied to BigQuery columns enforce column-level access control, while authorized views grant dataset access without exposing base tables.

Row-level security in BigQuery restricts returned rows through row access policies; both row and column controls are enforced at query time.

BigQuery ML trains and serves models with SQL: CREATE MODEL specifies the algorithm and feature columns, then ML.PREDICT or ML.EVALUATE is called in queries.

BI Engine caches hot data in memory to serve sub-second dashboard queries; capacity is purchased as a reservation measured in gigabytes and attached to a project.

Dataplex unifies discovery, lineage, data quality tasks, and zone-level governance across Cloud Storage lakes and BigQuery datasets.

BigQuery compute is allocated as slots, either on-demand per query or through reservations under Standard, Enterprise, or Enterprise Plus editions.

Use UNNEST to flatten ARRAY columns and STRUCT field access with dot notation when modeling nested JSON-like payloads in BigQuery.

Cloud Composer is managed Apache Airflow on a GKE-backed environment; DAGs are Python files with operators, dependencies, retries, and SLA settings.

Draining a Dataflow streaming pipeline processes in-flight data and saves state for restart; cancelling stops immediately and discards the final state.

A Pub/Sub dead-letter topic receives messages that a subscription fails to acknowledge after the maximum delivery attempt count, enabling later inspection and replay.

Log-based metrics in Cloud Monitoring are built from Logging query filters and can drive alerting policies; sink exports route logs to BigQuery, Pub/Sub, or Cloud Storage.

Cloud Scheduler runs cron-style jobs that trigger HTTP endpoints, Pub/Sub topics, or App Engine handlers and is paired with Cloud Functions or Workflows for retry logic.

BigQuery INFORMATION_SCHEMA views such as JOBS_BY_PROJECT, TABLE_STORAGE, and PARTITIONS expose usage and cost data for auditing and optimization queries.

Dataproc workflow templates atomically orchestrate multi-step jobs on ephemeral clusters and can be triggered by Cloud Composer or Cloud Scheduler.

Cost and safety controls include Cloud Billing budgets and alerts, project quotas, VPC Service Controls perimeters, and Committed Use Discounts or slot commitments for predictable capacity.