GCP Data Engineer key terms

Combining related tables into fewer, wider tables so reads avoid joins. It reduces query latency and cost in analytical systems at the expense of duplicated storage and more careful update logic.

A dimensional model with one central fact table referencing surrounding dimension tables. It keeps queries simple and fast for BI tools and is a common pattern in data warehouse designs.

Splitting data across multiple databases or nodes by a key so each holds a subset. It enables horizontal scaling and usually requires a routing layer in the application.

A performance problem where too many requests or writes target one node or partition, overloading it while others sit idle. Even key distribution and key salting help avoid it.

A managed asynchronous messaging service with topics and subscriptions that decouples publishers from subscribers. It buffers messages, supports push and pull delivery, and scales with traffic.

A guarantee that every message is delivered one or more times, so duplicates are possible and consumers must be idempotent. Pub/Sub subscriptions deliver at least once by default.

A fully managed runner for Apache Beam pipelines that handles batch and streaming with autoscaling and exactly-once processing. The same pipeline code runs in either mode.

Dividing an unbounded stream into finite chunks, such as fixed, sliding, or session windows, so aggregations can be computed. The window choice determines which late events still count.

A stream processing signal that estimates how far event time has advanced, telling the system when a window is complete. Events arriving after the watermark are treated as late data.

A managed service for running Apache Hadoop, Spark, Hive, and Presto clusters. Clusters can be created per job, scaled, and deleted, and integrate with Cloud Storage and BigQuery.

A managed, code-free data integration service built on CDAP. It offers a visual pipeline designer, many source connectors, and lineage tracking for building ETL and ELT flows.

A serverless change data capture service that streams changes from databases such as MySQL, PostgreSQL, and Oracle into BigQuery, Cloud Storage, or Pub/Sub with minimal impact on the source.

A managed service for moving or copying large datasets into Cloud Storage from other clouds, HTTP sources, or on-premises file systems, with scheduling and integrity checks.

A high-throughput BigQuery ingestion interface that streams records with exactly-once semantics and supports batch commits, often at lower cost than legacy streaming inserts.

A serverless compute service for running small event-driven functions in response to Pub/Sub messages, Cloud Storage changes, or HTTP calls, without provisioning or managing servers.

An approach that loads raw data into the target warehouse first and transforms it there using the warehouse's own compute. It contrasts with transforming data before loading.

Google Cloud's object storage service for unstructured data such as files, images, and data lake layers. Objects live in buckets, are addressed by key, and are replicated across locations.

A rule set that automatically moves objects to cheaper storage classes or deletes them after a set age or condition, controlling cost for long-lived data in Cloud Storage.

A fully managed service for MySQL, PostgreSQL, and SQL Server. It handles backups, patching, and replication, and suits transactional workloads that fit on a single primary instance.

A globally distributed, horizontally scalable relational database that offers strong consistency and external consistency using TrueTime, suitable for mission-critical transactional workloads across regions.

A wide-column NoSQL database built for very high throughput and low-latency reads at massive scale. It suits time series, IoT, and analytics data accessed by a single row key.

How you choose the unique key that determines how Bigtable stores and sorts rows. Keys should be evenly distributed and queried by prefix to avoid hotspots and slow scans.

A file layout such as Parquet or ORC that stores values column by column, so analytical queries read only needed columns and compress efficiently. Common for data lake tables in Cloud Storage.

A BigQuery table whose data stays in an external source such as Cloud Storage files or Cloud SQL. Queries read the source directly, so data is not duplicated but performance can be lower.

A storage engine that lets BigQuery and other engines query data in Cloud Storage while enforcing fine-grained access control, row-level security, and consistent governance across file formats.

Google Cloud's serverless, columnar data warehouse. It separates storage from compute, supports standard SQL, and scales query execution automatically, so teams analyze huge datasets without managing servers.

Splitting a large table into smaller segments along a column value such as date or ingestion time, so queries filtering on that column scan fewer rows and cost less.

Sorting a table's data by up to four columns so BigQuery can skip blocks that cannot match a query's filters. Clustering works alongside partitioning to improve scan efficiency.

A precomputed, automatically refreshed result set stored by BigQuery. Queries matching the view's SQL can be answered from the cached result, reducing cost and latency.

A dimension table technique for tracking attribute changes over time. Type 2 keeps history using version rows and validity dates, while type 1 overwrites old values in place.

A managed Apache Airflow service for authoring, scheduling, and monitoring workflows. It orchestrates pipelines across many Google Cloud services and supports retries, dependencies, and backfills.

A directed acyclic graph describing tasks and their dependencies in an orchestration system. It defines the order work runs in and which tasks can execute in parallel.

The property that running an operation multiple times produces the same result as running it once. It is essential for retry-safe pipeline tasks and duplicate-tolerant stream consumers.

A managed cron service that triggers jobs on a schedule through HTTP, Pub/Sub, or App Engine targets. It is often used to start pipelines, invoke functions, or refresh data on a timer.

A non-human identity used by workloads and applications to authenticate to Google Cloud APIs. It holds IAM roles and can be attached to virtual machines, functions, and Dataflow jobs.

Customer-managed encryption keys, where you create and control keys in Cloud KMS that Google Cloud services use to encrypt data at rest, enabling key rotation and revocation.

A security perimeter that restricts access to Google Cloud services so data cannot be moved outside defined projects and networks, helping prevent data exfiltration.

A governance service that catalogs data assets across BigQuery, Cloud Storage, and data lakes, offering search, tagging, lineage, and policy enforcement for discovery and control.

Slots are units of computational capacity used to run BigQuery queries. Reservations assign committed slots to projects or folders for predictable performance and cost control.

A separate Pub/Sub topic where messages that repeatedly fail processing are forwarded, so bad data can be inspected and retried without blocking the main pipeline.