GCP Data Engineer Pro key terms

An open-source programming model and SDK for defining batch and streaming pipelines once, then running them on runners such as Google Cloud Dataflow, Apache Spark, or a local direct runner.

A fully managed Google Cloud service that executes Apache Beam pipelines, automatically scaling worker capacity for batch or streaming jobs and providing features such as windowing, watermarks, and exactly-once streaming delivery.

Beam's abstraction for a distributed dataset that a pipeline transforms; it may be bounded for batch input or unbounded for streaming input, and it is immutable.

Dividing an unbounded stream into finite groups by event time or session boundaries, such as fixed, sliding, or session windows, so aggregations can be computed over continuous data.

A Beam estimate of when all event-time data up to a point has arrived; records arriving later are considered late and can be handled with allowed lateness and triggers.

A managed visual data-integration service, built on CDAP, where users design pipelines with a drag-and-drop interface and run them on a Dataproc cluster.

A managed Google Cloud service for running Apache Hadoop and Apache Spark clusters or serverless jobs, useful for lift-and-shift big-data workloads and short-lived batch processing.

A named message stream in Pub/Sub where publishers send data; messages are fanned out to every subscription attached to the topic, decoupling producers from consumers.

A resource attached to a Pub/Sub topic that buffers messages and delivers each one to a single subscriber, with settings for acknowledgement deadline, retention, and retry behavior.

A separate Pub/Sub topic that receives messages after a subscription's delivery attempts fail, allowing failed messages to be inspected or reprocessed without blocking the subscription.

A serverless change data capture service that reads database logs from MySQL, PostgreSQL, SQL Server, or Oracle and streams row changes to BigQuery, Cloud Storage, or Pub/Sub.

A managed service that copies large datasets into Cloud Storage from other cloud providers, HTTP(S) sources, or another bucket, with scheduling, retries, and integrity checks.

A managed scheduler that loads data into BigQuery from SaaS sources such as Google Ads and YouTube, or from Cloud Storage, on a recurring basis without custom code.

A per-object setting such as Standard, Nearline, Coldline, or Archive that balances storage cost against retrieval cost and latency; lifecycle rules can change classes automatically.

A bucket-level condition based on object age, creation date, or storage class that automatically deletes or reclassifies objects, helping control retention and cost without custom code.

A BigQuery table divided into segments by a date, timestamp, or integer column, so queries that filter on the partitioning column read fewer bytes and cost less.

A table whose rows are physically sorted by up to four columns; when a query filters on those columns, BigQuery skips blocks of data, improving performance and reducing bytes scanned.

A precomputed BigQuery view whose results are stored and refreshed automatically; the query optimizer can rewrite compatible queries to use it, reducing cost and latency.

A BigQuery table that references data stored outside BigQuery, such as Parquet, ORC, Avro, CSV, or JSON files in Cloud Storage, allowing queries without loading the data first.

A storage engine that lets BigQuery, Spark, and other engines query data in Cloud Storage while enforcing fine-grained access control and providing table metadata, combining data-lake and warehouse access.

A petabyte-scale, wide-column NoSQL database designed for low-latency reads and writes by row key, commonly used for time-series, IoT, and serving large analytical datasets.

A globally distributed relational database that offers SQL, strong consistency, and horizontal scaling, chosen when transactional semantics and multi-region availability are both required.

A BigQuery view that is granted access to a source dataset without giving users direct access to its tables, enabling controlled sharing across datasets or projects.

A label in a BigQuery or Dataplex taxonomy that classifies columns, so IAM or fine-grained access policies can restrict who reads sensitive fields such as personally identifiable information.

A BigQuery optimization that avoids reading partitions whose values cannot match the query filter on the partitioning column, reducing bytes scanned and lowering query cost.

A BigQuery feature for creating, training, and serving machine-learning models with SQL, keeping data in place and supporting models such as linear regression, boosted trees, and imported TensorFlow models.

A BigQuery service for publishing and subscribing to shared datasets as listings within exchanges, letting organizations share data across boundaries without copying it.

A Google Cloud service that discovers, classifies, and de-identifies sensitive data in storage and streams using masking, tokenization, or redaction, helping protect information such as credit card numbers.

A managed Apache Airflow service for authoring, scheduling, and monitoring workflows as Python DAGs, integrating with Google Cloud services and providing retries, logs, and alerting.

A directed acyclic graph in Airflow that defines tasks and their dependencies; edges set execution order, and the schedule determines when the workflow runs.

A fully managed cron service that invokes HTTP endpoints, publishes to Pub/Sub topics, or triggers App Engine jobs on a schedule, often used to start pipelines or queries.

A saved SQL statement configured to run on a recurring schedule, optionally writing results to a destination table, commonly used for recurring transformations and reports.

A rule that evaluates a metric or log condition and sends notifications or triggers actions when the threshold is crossed, used to detect pipeline failures and latency problems.

Immutable records of administrative activity, data access, and system events on Google Cloud resources, used for compliance, security investigation, and troubleshooting.

A non-human identity used by applications and services to authenticate to Google Cloud APIs; IAM roles grant it permissions instead of attaching access to a person.

A set of BigQuery system views that expose metadata about datasets, tables, partitions, jobs, and access history, queried with SQL to monitor usage and cost.

A YAML configuration attached to a Dataproc cluster that adds or removes worker nodes based on YARN memory and CPU pressure, reducing cost for variable workloads.

A capacity commitment that assigns a fixed number of BigQuery slots to a project, folder, or organization, giving predictable query performance and cost compared with on-demand pricing.