GCP Data Engineer flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Bounded vs. unbounded data

Bounded data has a finite, known end and is processed in batch. Unbounded data arrives continuously and never ends, requiring streaming; Apache Beam models both with one API.

Beam watermark

Beam's estimate of when all event-time data up to a point has arrived. It decides when window results can fire and how long late data is still accepted.

Dataflow Streaming Engine

Moves pipeline state and shuffle processing to a Google-managed backend instead of worker VMs, reducing CPU and memory needs and allowing smaller worker machine types.

FlexRS

Flexible Resource Scheduling runs Dataflow batch jobs on a mix of preemptible and standard VMs with a six-hour completion SLA in exchange for lower cost.

Bigtable row key design

Rows are sorted lexicographically by key. Sequential or timestamp-prefixed keys concentrate writes on one node, so hash or reverse prefixes to distribute load.

Spanner vs. Cloud SQL

Spanner scales horizontally with strong consistency and 99.999% multi-region availability. Cloud SQL is a simpler regional relational service suited to smaller, single-region workloads.

Pub/Sub at-least-once delivery

Subscribers can receive duplicate messages. Correctness requires idempotent writes or deduplication, such as Dataflow's use of a stable message ID attribute.

Storage Transfer Service

Managed service that copies data on a schedule from on-premises systems, other clouds, or other buckets into Cloud Storage, with bandwidth and retry controls.

Transfer Appliance

Rackable hardware shipped to your site to capture data offline, then shipped back and uploaded to Cloud Storage. Used when network bandwidth cannot meet transfer timelines.

Datastream

Serverless change data capture service that streams inserts, updates, and deletes from MySQL, PostgreSQL, Oracle, or SQL Server into BigQuery, Cloud Storage, or Cloud SQL.

Pub/Sub retention limits

Each subscription retains unacknowledged messages for its message retention duration, defaulting to seven days with a maximum of thirty-one days.

BigQuery Data Transfer Service

Automates recurring, code-free data loads into BigQuery from SaaS and external sources such as Google Ads, YouTube, and Amazon S3 on a defined schedule.

Dataflow Flex templates

Packages pipeline code and dependencies as a container image, enabling custom libraries and runtime parameterization when a job is launched.

Dataproc autoscaling

Dynamically adds or removes worker nodes based on pending YARN memory and cores, keeping ephemeral clusters sized to actual job demand.

GCS minimum storage durations

Nearline requires 30 days, Coldline 90 days, and Archive 365 days. Deleting or moving an object earlier bills the remaining minimum storage duration.

Cloud Storage Autoclass

Automatically transitions objects between Standard and colder storage classes based on access patterns, without early deletion fees for the automatic transitions.

BigQuery partition limit

A partitioned BigQuery table supports up to 4,000 partitions. Exceeding that causes query errors, so partition granularity must match expected data volume.

BigQuery clustering limit

Up to four columns can be clustering keys. BigQuery uses them to prune blocks inside partitions, and pruning works best when filters follow declared key order.

Bigtable garbage collection policy

A per-column-family rule that removes data after a maximum age or keeps only the N most recent cell versions, controlling storage growth and read cost.

BigLake tables

BigQuery table objects over Cloud Storage data in open formats such as Parquet and ORC, providing fine-grained access control to BigQuery and other query engines.

BigQuery policy tags

Taxonomy terms defined in Data Catalog and attached to BigQuery columns to enforce column-level access control, restricting who can read sensitive fields.

Authorized views

A view granted access to a source dataset so users can query aggregated or filtered results without holding direct read permission on the base tables.

BigQuery BI Engine

In-memory analysis service that accelerates small interactive dashboard queries. Capacity is purchased as a reservation measured in gigabytes and attached to a project.

BigQuery ML

Trains and evaluates models using SQL statements such as CREATE MODEL, ML.EVALUATE, and ML.PREDICT, supporting linear and logistic regression, k-means, and boosted trees.

Dataplex

A metadata and governance layer providing discovery, lineage, data quality tasks, and zone-level policy enforcement across Cloud Storage lakes and BigQuery datasets.

BigQuery slot reservations

Compute capacity purchased in slots and assigned to projects, folders, or organizations. Editions are Standard, Enterprise, and Enterprise Plus with different scaling behavior.

Cloud Composer

Managed Apache Airflow running on a GKE-backed environment, used to schedule and monitor workflows defined as Python DAGs with operators, retries, and dependencies.

Dataflow draining vs. cancelling

Draining stops a streaming pipeline after processing in-flight data and preserves state for restart. Cancelling halts immediately and discards the final pipeline state.

Pub/Sub dead-letter topic

A topic that receives messages a subscription cannot acknowledge after the configured maximum delivery attempts, allowing later inspection, debugging, and replay.

Log-based metrics

Cloud Monitoring metrics generated from Cloud Logging query filters, used to count or extract values from log entries and drive alerting policies.

Cloud Scheduler

Fully managed cron service that invokes HTTP endpoints, publishes to Pub/Sub topics, or calls App Engine handlers at defined intervals with retry configuration.

BigQuery INFORMATION_SCHEMA

Built-in views such as JOBS_BY_PROJECT, TABLE_STORAGE, and PARTITIONS that expose job history, table metadata, and partition details for cost and usage auditing.