GCP Data Engineer Pro flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

FlexRS

Dataflow's Flexible Resource Scheduling runs batch jobs on a cheaper mix of preemptible and standard workers with delayed scheduling, reducing batch cost substantially while giving up a guaranteed start time.

Side input in Beam

A supplementary PCollection broadcast to every worker and read inside a ParDo, used for enrichment or filtering with small reference datasets instead of a full shuffle-based join.

Datastream versus Pub/Sub

Datastream performs change data capture from relational databases into BigQuery or Cloud Storage; Pub/Sub transports independently produced event messages. Choose Datastream when the source is a database change log.

Batch versus streaming rule

A bounded source with acceptable latency of minutes to hours calls for batch processing; an unbounded source or a seconds-level latency requirement calls for a streaming pipeline such as Pub/Sub into Dataflow.

Spanner versus Cloud SQL

Cloud SQL is a regional managed relational database with a single primary. Spanner shards horizontally for global strong consistency and a 99.999% multi-region SLA at higher cost and design complexity.

Idempotent pipeline writes

Because delivery is often at-least-once, writes should use deterministic keys with MERGE or deduplication so that reprocessing the same record leaves the final state unchanged.

Watermark in Dataflow

A watermark estimates that all data up to a given timestamp has arrived; windows fire when the watermark passes the window end, and later arrivals are handled by allowed lateness and triggers.

Allowed lateness

A Dataflow windowing setting that keeps window state alive after the watermark passes, so elements arriving late are still added to the correct window instead of being dropped.

Pub/Sub message retention

A subscription retains unacknowledged messages for a configurable duration from ten minutes up to seven days, with seven days as the default; seek and snapshots allow replay.

Storage Write API

BigQuery's preferred streaming ingestion interface, offering exactly-once semantics with streams, higher throughput, and lower cost than the legacy streaming insert API.

Beam CoGroupByKey

A transform that groups multiple PCollections sharing a key type into one collection of key-to-multiple-iterables, forming the basis for inner, outer, and full joins.

Dataflow templates

Prepackaged and Flex templates let you launch the same pipeline with runtime parameters from the console, gcloud, or an orchestrator, which is required for scheduled and event-driven runs.

CDC ingestion pattern

Capture row-level changes from a database log, publish them in commit order, then apply inserts, updates, and deletes to the target; Datastream implements this natively for BigQuery and Cloud Storage.

BigQuery partitioning

Splitting a table by ingestion time, a date or timestamp column, or an integer range so queries scan fewer bytes; pruning applies when filters cover the partition column.

BigQuery clustering

Sorting table data by up to four columns so BigQuery can skip blocks inside each partition; most useful when query filters do not target the partition column.

BigQuery long-term storage

Partitions left unmodified for ninety consecutive days are billed at roughly half the active storage rate, automatically, with no configuration or query change required.

Bigtable row key design

Keys are sorted lexicographically and stored in contiguous ranges, so avoid strictly increasing keys like raw timestamps; use hashed prefixes or field-reversed keys to spread writes evenly.

Cloud Storage Nearline

A storage class intended for data accessed less than once a month, with a thirty-day minimum storage duration; deleting earlier still charges for the remaining days.

BigQuery external table

A table that reads data in place from Cloud Storage, BigLake, or Drive instead of loading it; queries are slower and billed by bytes read, but data stays in the lake.

Spanner interleaved tables

Child tables declared INTERLEAVE IN PARENT are physically co-located with their parent rows, so parent-child joins avoid network hops; use for hierarchical data with matching key prefixes.

QUALIFY clause

A BigQuery clause that filters the result of window functions without a subquery, applied after window computation and before ORDER BY, similar to HAVING for analytic functions.

BI Engine

An in-memory analysis service that accelerates BigQuery SQL for dashboards, providing sub-second latency and high concurrency for queries whose data fits within the reservation.

Materialized view refresh

BigQuery materialized views refresh automatically after base table changes and support smart tuning, where a query against the base table is rewritten to read the view instead.

BigQuery ML CREATE MODEL

Trains a model inside BigQuery using standard SQL by naming a model type such as LINEAR_REG or BOOSTED_TREE_CLASSIFIER; predictions use ML.PREDICT and evaluation uses ML.EVALUATE.

Dry run cost estimate

Validating a query without executing it returns the bytes that would be processed, letting you estimate on-demand cost; maximum bytes billed rejects queries that would exceed the limit.

Policy tags

Data Catalog taxonomy tags attached to BigQuery columns that enforce column-level access control through fine-grained catalog permissions, restricting who can read specific fields in shared tables.

Dataflow drain versus cancel

Drain stops a streaming pipeline after it processes the backlog and is used before a compatible update; cancel stops immediately, discarding buffered data and leaving the job cancelled.

Composer BigQuery operator

Airflow's BigQueryInsertJobOperator submits SQL jobs as tasks, letting pipelines schedule transformations and retry failures with dependency ordering across datasets.

System lag metric

The maximum time a Dataflow streaming element has been buffered without processing; a rising value signals the pipeline is falling behind its input rate and needs more workers.

Pub/Sub dead-letter topic

A subscription setting that forwards messages failing delivery after a maximum delivery attempt count to a separate topic, so poison messages can be inspected without blocking the pipeline.

Scheduled query

BigQuery can run SQL on a recurring schedule, optionally writing results to a destination table and publishing a Pub/Sub notification on completion or failure.

Dataproc idle TTL

A cluster setting that deletes the cluster after a period of inactivity, preventing cost from forgotten ephemeral clusters created for scheduled Spark jobs.