I85PREP
⌘ K
GCP Data Engineer study materials
GCP Data Engineer study resources

GCP Data Engineer practice questions

One question at a time. Build your GCP Data Engineer understanding.

First 50 of 200 questions. Each one lists the choices, the correct answer, and an explanation. The checker at the top of this page saves progress in this browser when it is available.

Continue in the practice tool above to load more questions, 50 at a time.

Question 1Design data processing systems

A financial application publishes trade events to Pub/Sub and subscribers must process events for the same account strictly in the order they were published. What should the publisher and subscriber configure?

  1. Create a separate topic per account and have each subscriber attach to every topic.
  2. Set an ordering key on each message when publishing and enable message ordering on the subscription. Correct answer.
  3. Increase the subscription ack deadline to the maximum so messages are redelivered in sequence.
  4. Publish all related messages from one client instance with batching disabled and a single outstanding request.

Explanation

Pub/Sub preserves order only within an ordering key, and the subscriber must opt into ordered delivery on the subscription. Messages with the same ordering key are delivered in publish order to a single subscriber client. Without an ordering key, Pub/Sub gives no ordering guarantee regardless of batching or ack settings.

Question 2Design data processing systems

A 5 TB BigQuery table holds five years of events and most queries filter on the event_date column for a short date range. Which change most directly reduces bytes scanned and query cost?

  1. Partition the table on event_date so range-filtered queries read less data. Correct answer.
  2. Create a logical view over the table so queries scan only the filtered rows.
  3. Set a table expiration so older rows are removed before queries run.
  4. Cluster the table on event_date so BigQuery prunes micro-partitions.

Explanation

Partitioning physically splits a table by a column value such as a date, and BigQuery prunes partitions that cannot match the filter, so only the relevant date ranges are scanned. Clustering helps within the scanned data but does not provide partition-level pruning. Views and expiration do not reduce the bytes read for the retained rows.

Question 3Design data processing systems

A team lands raw ingestion files in a Cloud Storage bucket. Files older than 30 days are almost never read but must stay retrievable for seven years at the lowest storage cost. Which design meets the requirement?

  1. Move the objects to a bucket in a cheaper region and disable object versioning.
  2. Keep all objects in Standard storage and query them through BigQuery external tables.
  3. Add Object Lifecycle Management rules that transition objects to Coldline or Archive storage after 30 days. Correct answer.
  4. Enable Object Versioning and set a bucket retention policy of seven years.

Explanation

Object Lifecycle Management can automatically change an object's storage class based on age, which is the supported way to move rarely accessed data into cheaper classes while keeping it retrievable. Location choice and versioning affect durability and price per GB but not the storage-class transition. Retention policies block deletion without lowering storage cost.

Question 4Design data processing systems

A pipeline has twelve dependent steps spanning BigQuery, Dataproc, and Cloud Storage, and operators need scheduling, per-step retries, and historical backfills. Which service is designed for this?

  1. A single long-running Workflows execution that runs all twelve steps.
  2. Cloud Scheduler jobs that invoke Cloud Functions for each step.
  3. A Dataflow pipeline with a timer transform that chains the steps.
  4. Cloud Composer, the managed Apache Airflow service. Correct answer.

Explanation

Cloud Composer provides a managed Airflow environment with DAG definitions, per-task retries, scheduling, and backfill of past execution dates across heterogeneous services. Its operators exist for BigQuery, Dataproc, and Cloud Storage, which matches a multi-service twelve-step dependency graph.

Question 5Design data processing systems

An operations team must relaunch the same Apache Beam pipeline nightly with different input and output locations, driven by an automated trigger, without rebuilding or recompiling code. Which approach fits?

  1. Copy the pipeline source into a Cloud Function and invoke it on a schedule.
  2. Package the pipeline as a JAR and submit it to Dataproc each night.
  3. Create a BigQuery scheduled query that calls the Dataflow API.
  4. Build a Dataflow Flex Template and launch it with runtime parameters. Correct answer.

Explanation

Flex Templates package the pipeline and its dependencies into an image that can be launched through the Dataflow API or console with runtime parameters, so the graph is not recompiled per run. This is the supported mechanism for parameterized, repeatable launches from automation.

Question 6Design data processing systems

A Dataflow job reads from a Pub/Sub subscription and writes to BigQuery. Which identity configuration limits blast radius if the pipeline is compromised?

  1. Create a dedicated service account with Pub/Sub Subscriber and BigQuery Data Editor roles, and run the job as that account. Correct answer.
  2. Run the job with a developer's personal Google account so every action is tied to a named person.
  3. Run the job as the default Compute Engine service account, which has the Editor role.
  4. Grant the Owner role to the Dataflow service agent at the project level.

Explanation

A pipeline should run under a dedicated service account whose roles are limited to exactly the resources the pipeline touches, here subscribing to Pub/Sub and writing BigQuery data. This keeps the credential independent of any individual and bounds what a compromise can reach.

Question 7Design data processing systems

A team of analysts with no programming background must build repeatable pipelines that join data from Cloud Storage and BigQuery using a drag-and-drop interface. Which product fits best?

  1. Cloud Data Fusion. Correct answer.
  2. Dataproc with PySpark notebooks.
  3. Dataflow with custom DoFn transforms.
  4. BigQuery scheduled queries alone.

Explanation

Cloud Data Fusion is a managed, code-free data integration service with a visual pipeline designer and prebuilt connectors and transforms. It targets exactly the case of non-programmers assembling repeatable pipelines across sources.

Question 8Design data processing systems

A company must move 400 TB from an on-premises S3-compatible object store into Cloud Storage and then re-sync only new or changed objects on a recurring schedule. Which service is purpose-built for this?

  1. BigQuery Data Transfer Service.
  2. Dataproc DistCp.
  3. gsutil rsync run from a single Compute Engine VM.
  4. Storage Transfer Service. Correct answer.

Explanation

Storage Transfer Service is designed for large-scale transfers into Cloud Storage from other object stores, HTTP endpoints, or on-premises agents, and it supports scheduled recurring transfers with change detection. A single VM or DistCp lacks the managed parallelism and scheduling.

Question 9Design data processing systems

A Bigtable table stores sensor readings with the row key set to the reading timestamp. Write throughput is confined to a small number of nodes because load concentrates on the newest rows. Which row key change addresses the root cause?

  1. Use the timestamp alone as the row key but format it as a fixed-width string.
  2. Add more column families so writes are spread across the table.
  3. Store one row per device and put every reading in a new column qualifier.
  4. Prefix the key with a hash of the device ID, then the timestamp. Correct answer.

Explanation

Bigtable sorts rows lexicographically and splits them into tablets, so a monotonically increasing key such as a timestamp sends all new writes to the last tablet. Prefixing with a high-cardinality value such as a hashed device ID spreads writes across the key space while keeping per-device history ordered.

Question 10Design data processing systems

A subscriber occasionally receives messages it can never process successfully. The team wants these messages set aside automatically for later analysis without blocking delivery of other messages. Which configuration achieves this?

  1. Raise the subscription ack deadline to the maximum value.
  2. Ack each message before processing it so failures never accumulate.
  3. Configure a dead-letter topic on the subscription with a maximum delivery attempts policy. Correct answer.
  4. Schedule a seek operation on the subscription every hour.

Explanation

A dead-letter topic with a maximum delivery attempts policy makes Pub/Sub forward a message after the configured number of failed deliveries, so it stops blocking the subscription and is preserved for inspection. The subscriber keeps receiving the remaining messages normally.

Question 11Design data processing systems

Analysts need to query Parquet files in Cloud Storage from BigQuery with row-level access control, and the same files must also be readable by Dataproc jobs. Which design satisfies both?

  1. Load the Parquet files into a native BigQuery table every night.
  2. Create a BigQuery view over a Cloud SQL federated query.
  3. Create a BigLake table over the Cloud Storage path using a Cloud Resource connection. Correct answer.
  4. Create a BigQuery external table directly over the Cloud Storage path.

Explanation

BigLake tables expose Cloud Storage files to BigQuery while supporting fine-grained security such as row-level and column-level policies, because BigQuery accesses the data through a Cloud Resource connection. The files stay in place, so Dataproc can read the same objects.

Question 12Design data processing systems

A streaming pipeline must aggregate per-user activity where bursts of events are separated by arbitrary periods of inactivity. Which windowing strategy models this directly?

  1. Sliding windows of thirty minutes that advance every five minutes.
  2. Session windows with a gap duration. Correct answer.
  3. A global window with a trigger that fires every 1,000 elements.
  4. Fixed-time windows of five minutes with one hour of allowed lateness.

Explanation

Session windows group elements into activity bursts and start a new window whenever the gap between consecutive elements exceeds the configured duration, which is exactly the inactivity-defined boundary described. The other options impose boundaries that are independent of user behaviour.

Question 13Design data processing systems

A streaming pipeline must enrich each event with a small reference table stored in BigQuery that changes a few times per day. Per-element BigQuery calls are too slow and expensive. Which approach is appropriate?

  1. Apply a GroupByKey over the reference table for every element.
  2. Buffer all events and join them with the reference table in an hourly batch job.
  3. Call the BigQuery API inside the DoFn for each element.
  4. Load the reference table as a side input and refresh it on a timer. Correct answer.

Explanation

A side input broadcasts a small, slowly changing dataset to all workers so each element can be enriched in memory without a remote call. Refreshing it on a timer keeps the enrichment current without per-element queries.

Question 14Design data processing systems

A pipeline streams rows into BigQuery and must avoid duplicates when the writer retries, without depending on best-effort insertId deduplication. Which approach fits?

  1. Write to Bigtable and query it through a BigQuery external table.
  2. Run a BigQuery load job every ten seconds with micro-batches.
  3. Use tabledata.insertAll with an insertId generated per row.
  4. Use the Storage Write API with exactly-once delivery and record offsets per stream. Correct answer.

Explanation

The Storage Write API supports exactly-once semantics when the client tracks the stream offset and reconnects at that offset, so retried writes are not applied twice. It also avoids the per-row charges and short-lived deduplication window of the legacy streaming API.

Question 15Design data processing systems

A Spark job runs twice a day over 5 TB of data in Cloud Storage. The team wants the lowest cost with no HDFS durability to manage. Which cluster design is appropriate?

  1. A Dataproc cluster backed by large Persistent Disks for HDFS storage.
  2. A Dataflow batch pipeline using the Dataflow Shuffle service with a custom file sink.
  3. A long-running Dataproc cluster that persists all data in HDFS.
  4. An ephemeral Dataproc cluster that reads and writes through the Cloud Storage connector. Correct answer.

Explanation

Because the data already lives in Cloud Storage, the Cloud Storage connector lets workers read and write objects without copying into HDFS, and an ephemeral cluster can be deleted after each run so the team pays only for job time. Durability is provided by Cloud Storage itself.

Question 16Design data processing systems

Order events contain a variable number of line items and are queried to total revenue and to list individual items. Which BigQuery schema avoids joins and keeps the data queryable with standard SQL?

  1. Separate orders and line_items tables joined on order_id at query time.
  2. One row per line item, with the order-level fields duplicated in every row.
  3. A flat table with one row per order and a JSON string column holding the line items.
  4. A RECORD field with mode REPEATED holding the line items inside the order row. Correct answer.

Explanation

A repeated RECORD column stores nested line items inside the parent order row, which BigQuery handles natively with ARRAY and STRUCT expressions. This keeps order-level and item-level analytics possible without a join and without parsing strings.

Question 17Design data processing systems

A team needs near-real-time replication of changes from an on-premises MySQL database into BigQuery while minimizing load on the source. Which service should be the ingestion layer?

  1. BigQuery Data Transfer Service pointed at the MySQL host.
  2. BigQuery federated queries issued directly against the MySQL instance.
  3. Cloud Data Fusion running a nightly full extract from MySQL.
  4. Datastream reading the MySQL change stream into Cloud Storage, with Dataflow loading into BigQuery. Correct answer.

Explanation

Datastream performs log-based change data capture from MySQL, which reads the binary log rather than polling tables and therefore adds little load. It writes change records to Cloud Storage or BigQuery, and Dataflow can transform and land them in BigQuery.

Question 18Design data processing systems

A 3 TB event table is filtered almost exclusively by event_date, and analysts sometimes further filter by customer_id, which has 500,000 distinct values. Which physical design is most cost effective?

  1. Partition by customer_id using integer range partitioning into 500,000 ranges.
  2. Partition by event_date and cluster by customer_id. Correct answer.
  3. Cluster by event_date only and rely on the query optimizer to prune.
  4. Partition by event_date and keep a separate table for each customer.

Explanation

Partitioning by the column used in almost every filter gives partition-level pruning, and clustering on the secondary high-cardinality column lets BigQuery skip blocks within the scanned partitions. This combination matches both filter patterns without creating an unmanageable number of objects.

Question 19Design data processing systems

In an event-time streaming pipeline, a small fraction of events arrives several hours late, and results that include those events must still be emitted. Which configuration is required?

  1. Increase the number of workers so late elements are processed faster.
  2. Set allowed lateness on the windows and configure a trigger that fires when late elements arrive. Correct answer.
  3. Switch the windows to processing time so arrival order determines the window.
  4. Set allowed lateness to zero on the windows to finalize results immediately.

Explanation

Allowed lateness keeps a window's state alive after the watermark passes so late elements are still assigned to the correct window, and a trigger such as accumulating or repeating firing causes updated results to be emitted when they arrive. Without both settings, late events are dropped.

Question 20Design data processing systems

A nightly Dataflow batch job reads about 200,000 CSV files of roughly 50 KB each and spends most of its time opening files. Which design change addresses the root cause?

  1. Compact the files into a few large objects or convert them to a splittable format such as Avro or Parquet before the job. Correct answer.
  2. Increase the boot disk size of each worker.
  3. Run the job with the DirectRunner on a large VM.
  4. Increase the maximum number of workers from 20 to 200.

Explanation

Many small files create per-file open, seek, and metadata overhead that dominates runtime, and CSV is not splittable in a way that lets workers seek into the middle of a file. Compacting or converting to a splittable columnar or block format lets the runner split work into large ranges and reads far more efficiently.

Question 21Design data processing systems

A streaming pipeline groups events by country, a low-cardinality key, and one country generates most of the traffic, leaving a single worker far behind. Which change mitigates the skew?

  1. Replace GroupByKey with CoGroupByKey.
  2. Increase the ack deadline on the source subscription.
  3. Add a random shard suffix to the hot key before grouping and combine the partial results afterwards. Correct answer.
  4. Configure processing-time timers for the aggregation.

Explanation

Key skew concentrates all elements for one key on one worker, so splitting the hot key into randomized shards spreads the work across many workers and a second, cheap aggregation merges the partial results. This is the standard pre-aggregation pattern for hot keys in Beam.

Question 22Design data processing systems

Which statement best describes the benefit of the Dataflow Shuffle service for batch pipelines?

  1. It moves shuffle data to a managed service so workers do not need local disk capacity and can scale down while the shuffle is served. Correct answer.
  2. It guarantees exactly-once processing for streaming pipelines that use GroupByKey.
  3. It removes the need for windowing in batch pipelines.
  4. It stores intermediate shuffle data on worker Persistent Disks so no network transfer occurs.

Explanation

Dataflow Shuffle offloads the shuffle to a managed backend, which decouples shuffle capacity from worker disks and allows workers to be released while shuffle data is served. That reduces disk usage and can lower cost for batch jobs with large shuffles.

Question 23Design data processing systems

A global application needs relational, ACID transactions across regions with strong consistency and 99.999% availability. Which storage design meets these requirements?

  1. Cloud Spanner with a multi-region instance configuration. Correct answer.
  2. BigQuery with streaming inserts and table decorators.
  3. Bigtable with multi-cluster routing and cell timestamps.
  4. Cloud SQL with a cross-region read replica and manual failover.

Explanation

Cloud Spanner is a horizontally scalable relational database that provides externally consistent transactions, and a multi-region instance configuration replicates across regions with a 99.999% availability SLA. This combination of relational semantics, cross-region strong consistency, and that SLA is unique to Spanner among these options.

Question 24Design data processing systems

Files arrive in a Cloud Storage bucket at irregular intervals, and a streaming pipeline must process each file as soon as it lands without processing any file twice. Which design achieves this?

  1. A batch job that lists the bucket every five minutes and processes any file it has not seen.
  2. A Cloud Function triggered on object finalize that launches one Dataflow job per file.
  3. Cloud Storage notifications published to Pub/Sub, consumed by a streaming pipeline that tracks the object generation to deduplicate. Correct answer.
  4. A streaming pipeline that polls the bucket with a GenerateSequence transform every second.

Explanation

Object notifications give event-driven, low-latency delivery of new object events to Pub/Sub, and the object generation number identifies an exact object version so retries or duplicate notifications can be filtered. A streaming pipeline over that subscription processes each arrival promptly and idempotently.

Question 25Design data processing systems

Analysts repeatedly run the same expensive join between a large fact table and a small dimension table that changes hourly, and results may be at most 30 minutes stale. Which design reduces cost while meeting the freshness requirement?

  1. Query results cached in a destination table with the default expiration.
  2. A logical view that re-executes the join each time it is queried.
  3. A scheduled query that truncates and reloads a results table every hour.
  4. A BigQuery materialized view with max_staleness set to 30 minutes. Correct answer.

Explanation

A materialized view stores the precomputed join and is refreshed incrementally, and max_staleness lets BigQuery serve the stored result while it is within the staleness bound, falling back to the base tables only when necessary. This cuts scanned bytes for repeated queries while bounding staleness.

Question 26Design data processing systems

A Cloud Storage data lake holds raw event logs that are queried heavily for the first 30 days, occasionally for audit up to a year, and must be retained for seven years for compliance. Which design gives the required access at the lowest storage cost?

  1. Lifecycle rules that move objects to Nearline, Coldline, and Archive as they age Correct answer.
  2. Object Versioning enabled on the bucket with a default storage class of Standard
  3. A Standard-class bucket protected by a bucket retention policy that blocks deletion for seven years
  4. A second Standard-class bucket that older objects are copied into each month

Explanation

Object Lifecycle Management transitions objects between storage classes based on age, and Archive is the cheapest class for data retained for years but rarely read. Combining transitions with the retention requirement satisfies both access patterns and compliance.

Question 27Design data processing systems

An enterprise has hundreds of existing Apache Hive and Spark jobs on-premises that read from HDFS and must move to Google Cloud with as few code changes as possible. Which service should the design use?

  1. Dataflow pipelines with the Spark code rewritten as Apache Beam transforms
  2. Cloud Data Fusion with a visually built pipeline for each existing job
  3. Dataproc clusters that use Cloud Storage as the default filesystem Correct answer.
  4. BigQuery with scheduled queries that replace the Hive transformations

Explanation

Dataproc provides managed Hadoop and Spark, and the Cloud Storage connector lets existing jobs address gs:// paths in place of HDFS with minimal edits. Beam and Data Fusion would require reimplementing the job logic rather than rehosting it.

Question 28Design data processing systems

A publisher emits account updates to Pub/Sub. Downstream processing must apply updates for the same account in publication order, while updates for different accounts should still be processed concurrently. Which feature should the design use?

  1. A dead-letter topic attached to the subscription with a retry policy
  2. A single subscription with a message retention duration of seven days
  3. An ordering key set to the account ID on each published message Correct answer.
  4. Exactly-once delivery enabled on the subscription

Explanation

Messages published with the same ordering key are delivered to a single subscriber client in publish order, while messages with different keys can be delivered in parallel. Retention, exactly-once delivery, and dead-lettering govern durability and duplicate handling, not ordering.

Question 29Design data processing systems

A BigQuery table is partitioned by day, and only the most recent 90 days of data are needed. Old data should be removed automatically without running extra jobs. What should be configured on the table?

  1. A clustered table on the date column with a row access policy
  2. A partition expiration of 90 days on the partitioned table Correct answer.
  3. A logical view that filters on the date column for the last 90 days
  4. A scheduled query that runs DELETE statements every night

Explanation

Partition expiration makes BigQuery delete partitions whose partition date is older than the configured interval, with no maintenance jobs required. Filtering queries or deleting rows leaves the underlying partition data stored and billable.

Question 30Design data processing systems

A Dataflow batch pipeline reads a handful of very large gzip-compressed CSV files from Cloud Storage. The job scales to many workers, but each file is processed by only one worker and the job runs slowly. Which change improves read parallelism?

  1. Add a repartition transform before the read to spread files across workers
  2. Raise the maximum number of workers so each worker receives its own file
  3. Re-encode the input as Avro container files or as uncompressed text files Correct answer.
  4. Enable the Dataflow Shuffle service so the compressed files can be split

Explanation

A gzip stream cannot be split, so one worker must read each file from start to finish regardless of cluster size. Avro container files and uncompressed text files expose seek points, allowing Beam to distribute chunks of a single file across workers.

Question 31Design data processing systems

A streaming Dataflow pipeline keeps large amounts of per-key state and many timers. The team wants to reduce CPU and memory pressure on the workers without changing pipeline logic. What should they do?

  1. Store the state in BigQuery and query it for each element inside a DoFn
  2. Switch the pipeline to the batch runner and use windowed reads of the source
  3. Enable Streaming Engine so state and watermark tracking move off the workers Correct answer.
  4. Run the pipeline with one worker harness thread per worker and larger boot disks

Explanation

Streaming Engine moves pipeline state, watermark tracking, and timer handling to the Dataflow backend, so workers no longer hold that state in memory. It is switched on with a pipeline option and requires no code change.

Question 32Design data processing systems

Each night a staging table in BigQuery contains inserts, updates, and deletes that must be applied to a large target table, and the full history is too large to reprocess. Which approach should the design use?

  1. Append all staging rows to the target and deduplicate at read time with QUALIFY
  2. Run a MERGE that matches on the business key and applies each change Correct answer.
  3. Create one table per day and union the tables together in a view
  4. Truncate the target table and reload it from the full source each night

Explanation

MERGE performs set-based upserts and deletes in one statement by matching source rows to target rows on a business key. It reads only the staging rows and the rows they match, so the full history is never rescanned.

Question 33Design data processing systems

A pipeline spans Cloud Storage, Dataproc, and BigQuery. Operators need dependency management, automatic retries, backfills, and a UI to monitor every run across all three systems. Which service should be central to the design?

  1. Cloud Composer with DAGs that define task dependencies Correct answer.
  2. Eventarc triggers that fire on Cloud Storage object finalize events
  3. Cloud Scheduler jobs that call each step at fixed times
  4. Cloud Workflows that run the steps as a linear sequence of HTTP calls

Explanation

Cloud Composer is managed Airflow, which provides DAG-based dependency management, retries, backfill runs, and a monitoring UI, plus operators for Dataproc, BigQuery, and Cloud Storage. Timetable triggers and event triggers can start work but do not track cross-system dependencies.

Question 34Design data processing systems

Records containing names, email addresses, and credit card numbers arrive in a pipeline that will land them in Cloud Storage and then BigQuery. Security requires the PII to be masked before the data reaches the data lake. Which design meets this requirement?

  1. Encrypt the bucket with customer-managed keys and restrict IAM to the pipeline account
  2. De-identify records with Cloud DLP templates before writing them to Cloud Storage Correct answer.
  3. Store raw records in a bucket with a retention policy and enable Data Access audit logs
  4. Load raw records into BigQuery and apply policy tags and column-level access controls

Explanation

Cloud DLP de-identification templates can be applied in-flight so sensitive values are masked, tokenized, or redacted before any object is written. Controls applied after landing leave the raw PII at rest in the lake.

Question 35Design data processing systems

Producers add optional fields to event records over several releases, and streaming consumers must keep running while both old and new records are in flight. Which serialization approach best supports this?

  1. CSV with a fixed column order agreed by producers and consumers
  2. JSON records where every field is required and validated in a DoFn
  3. Avro with a schema registry and a backward-compatible evolution policy Correct answer.
  4. Protobuf messages whose field numbers are reassigned at each release

Explanation

Avro schemas are carried with the data and a schema registry enforces compatibility rules, so readers tolerate fields added by newer writers. Positional or fixed-schema formats and reassigned field numbers break consumers of older records.

Question 36Design data processing systems

Nightly BigQuery batch jobs need several thousand slots at peak for about four hours, and the system is idle the rest of the day. Cost must stay predictable and below on-demand scanning of the same data. Which capacity option fits best?

  1. Materialized views that precompute the nightly outputs ahead of time
  2. On-demand pricing combined with a project-level custom quota on bytes billed
  3. A flat-rate annual commitment sized for the nightly peak slot count
  4. BigQuery editions with an autoscaling reservation sized to the nightly peak Correct answer.

Explanation

An autoscaling reservation adds slots only while demand rises and releases them afterwards, so the team pays for the four-hour peak rather than for peak capacity around the clock. That gives a predictable capacity cost while allowing the job to finish quickly.

Question 37Design data processing systems

A Dataflow pipeline reads from Pub/Sub and writes to BigQuery, and the security team requires that the worker VMs have no external IP addresses. Which configuration satisfies this requirement?

  1. Set --usePublicIps=false with Private Google Access or Cloud NAT Correct answer.
  2. Run the workers with public IPs and restrict ingress with firewall rules
  3. Attach a Cloud VPN gateway directly to the Dataflow worker pool
  4. Use the default Compute Engine service account for the worker VMs

Explanation

Workers without external IPs still need a route to Google APIs, which Private Google Access on the subnet (for Google-owned APIs) or Cloud NAT (for general egress) provides. The flag itself removes the external addresses from the worker VMs.

Question 38Design data processing systems

A Bigtable table stores the latest reading per device, with one row per device and a single column family. Storage has grown far beyond the expected row count because old cell versions are retained. Which configuration bounds storage while keeping the newest value readable?

  1. A renamed column family so that versions written earlier fall out of scope
  2. A GC policy that keeps one version per column family, with an optional TTL Correct answer.
  3. A row key prefix that spreads writes evenly across all tablet servers
  4. A second replication cluster in another region with single-cluster routing

Explanation

Bigtable garbage collection policies are configured per column family and can cap the number of retained cell versions and expire cells after a TTL. Keeping only the newest version removes accumulated history and bounds the bytes the table stores.

Question 39Design data processing systems

A streaming pipeline sends events to an external service that is efficient only for batched calls. The team groups events with GroupIntoBatches on a single global key, and throughput is now capped and latency spikes. Which change should they make?

  1. Use GroupIntoBatches with sharded keys so batches spread across workers Correct answer.
  2. Apply a Reshuffle before grouping while keeping the single global key
  3. Increase the number of workers and raise the maximum bundle size per worker
  4. Add a session window with a longer gap duration before grouping

Explanation

Grouping on one key funnels every element through a single keyed state and a single downstream writer, which serializes the calls. Using withShardedKey spreads the grouping over many keys so batches are produced and written in parallel while the per-batch size limit is still respected.

Question 40Design data processing systems

An IoT team wants to partition an event table by hour to make hourly queries fast, but five years of retention would create roughly 44,000 partitions. What design should they adopt?

  1. Create one table per day and query the tables with wildcard table suffixes
  2. Partition by ingestion time and filter queries on the partition decorator
  3. Keep hourly partitions and rely on partition expiration to prune metadata
  4. Partition by day or month and cluster on the timestamp column instead Correct answer.

Explanation

BigQuery performs best with a few thousand partitions per table, and hourly partitions over years far exceed that, inflating metadata and query planning time. Day or month partitions keep pruning effective while clustering on the timestamp preserves fine-grained filtering.

Question 41Ingest and process the data

An application publishes events to a Pub/Sub topic. A second, independent consumer team needs to receive every event without affecting the first team's acknowledgment behavior. What should you create?

  1. A second subscription on the same topic Correct answer.
  2. A snapshot of the existing subscription
  3. A new topic and republish the events to it
  4. A dead-letter topic attached to the existing subscription

Explanation

Each Pub/Sub subscription receives its own copy of every message published to the topic. Acknowledging a message in one subscription therefore does not remove it from another subscription, so a second subscription gives the new consumer an independent stream.

Question 42Ingest and process the data

You have a 200 GB set of CSV files in Cloud Storage and need to load them into BigQuery once per day. Which approach is the most direct managed ingestion method?

  1. Run a MapReduce job on a Dataproc cluster
  2. Publish each file name to Pub/Sub for BigQuery to consume
  3. Stream rows with the BigQuery Storage Write API
  4. Run bq load or create a BigQuery load job with Cloud Storage URIs Correct answer.

Explanation

BigQuery load jobs read directly from Cloud Storage and are the standard managed batch ingestion path for large file sets. The bq load command and the jobs.insert API both create these load jobs.

Question 43Ingest and process the data

In an Apache Beam pipeline, which transform keeps only the events whose amount field is greater than zero?

  1. Window
  2. Filter Correct answer.
  3. Combine
  4. GroupByKey

Explanation

Filter applies a predicate to each element and emits only the elements for which the predicate returns true. It is a per-element transform, so no grouping or window assignment is required.

Question 44Ingest and process the data

Which Google Cloud service is a managed environment for running Apache Spark and Apache Hadoop clusters?

  1. Dataflow
  2. Dataproc Correct answer.
  3. Datastream
  4. Cloud Data Fusion

Explanation

Dataproc is Google Cloud's managed service for Hadoop and Spark clusters, including related components such as Hive and Pig. It handles cluster provisioning and can run clusters that exist only for the duration of a job.

Question 45Ingest and process the data

You need to copy several terabytes of objects from an Amazon S3 bucket into a Cloud Storage bucket on a recurring schedule. Which service is designed for this?

  1. gcloud storage cp run from a Compute Engine VM
  2. BigQuery Data Transfer Service
  3. Google Cloud Transfer Appliance
  4. Storage Transfer Service Correct answer.

Explanation

Storage Transfer Service supports scheduled transfers from Amazon S3, Azure Blob Storage, HTTP/HTTPS sources, and other Cloud Storage buckets into Cloud Storage. It is managed and designed for large or recurring object transfers.

Question 46Ingest and process the data

A Pub/Sub subscription repeatedly fails to process a subset of messages. You want those messages isolated in a separate topic after a maximum number of delivery attempts. What should you configure?

  1. An ordering key on the topic
  2. A subscription expiration policy
  3. A message retention duration of seven days
  4. A dead-letter topic on the subscription Correct answer.

Explanation

A dead-letter topic on a subscription receives messages that fail processing after the configured maximum delivery attempts. This isolates poison messages so normal traffic can continue.

Question 47Ingest and process the data

Which API is Google's recommended high-throughput method for streaming data into BigQuery?

  1. The BigQuery Data Transfer Service
  2. The legacy tabledata.insertAll REST method
  3. The bq extract command
  4. The BigQuery Storage Write API Correct answer.

Explanation

The BigQuery Storage Write API is the recommended high-throughput ingestion API and supports both streaming and batch-style commits. It also provides stronger semantics than the legacy streaming insert method.

Question 48Ingest and process the data

You want operations staff to launch a Dataflow pipeline from the console with runtime parameters, without rebuilding the pipeline code. What should you use?

  1. A BigQuery scheduled query
  2. A Dataproc workflow template
  3. A Dataflow template Correct answer.
  4. A Cloud Composer DAG

Explanation

Dataflow templates package a pipeline and let users launch it with runtime parameters from the console, gcloud, or the REST API without compiling code. They are intended for reusable, parameterized pipeline deployment.

Question 49Ingest and process the data

Publishers send updates for the same account ID and need subscribers to receive those updates in publish order. What must be set?

  1. An ordering key on the messages plus message ordering enabled on the subscription Correct answer.
  2. A single subscriber client with flow control disabled
  3. The maximum message retention duration on the topic
  4. A longer acknowledgment deadline on the subscription

Explanation

Pub/Sub preserves order only when messages share an ordering key and the subscription has message ordering enabled. Messages with the same ordering key are then delivered in publish order to a single subscriber client.

Question 50Ingest and process the data

A pipeline analyzes user click bursts and groups events separated by gaps of less than five minutes into the same window. Which windowing strategy matches?

  1. Session windows with a five-minute gap duration Correct answer.
  2. Sliding windows of five minutes every one minute
  3. Fixed-time windows of five minutes
  4. A global window with a trigger every five minutes

Explanation

Session windows group elements that arrive close together and close a window when no element arrives for the configured gap duration. A five-minute gap therefore matches bursts separated by more than five minutes.