I85PREP
⌘ K
GCP Data Engineer Pro study materials
GCP Data Engineer Pro study resources

GCP Data Engineer Pro practice questions

One question at a time. Build your GCP Data Engineer Pro understanding.

First 50 of 200 questions. Each one lists the choices, the correct answer, and an explanation. The checker at the top of this page saves progress in this browser when it is available.

Continue in the practice tool above to load more questions, 50 at a time.

Question 1Design data processing systems

Which Cloud Storage storage class is designed for data that is accessed less than once a year and carries a 365-day minimum storage duration?

  1. Coldline
  2. Standard
  3. Archive Correct answer.
  4. Nearline

Explanation

Archive is the lowest-cost Cloud Storage class and targets data accessed less than once a year; it enforces a 365-day minimum storage duration and charges an early-deletion fee if objects are removed sooner.

Question 2Design data processing systems

Which Pub/Sub capability ensures that messages sharing the same key are delivered to subscribers in the order in which they were published?

  1. Dead-letter topics
  2. Attribute-based message filtering
  3. Exactly-once delivery
  4. Ordering keys Correct answer.

Explanation

When a publisher sets an ordering key, Pub/Sub delivers messages for that key in publish order to subscribers that process them in order; ordering is guaranteed per key, not across different keys.

Question 3Design data processing systems

Which window type should a streaming pipeline use to compute a metric over fixed, non-overlapping intervals such as every 5 minutes?

  1. Session windows
  2. The global window
  3. Fixed windows Correct answer.
  4. Sliding windows

Explanation

Fixed (tumbling) windows divide a stream into contiguous, equal-length, non-overlapping intervals, so each element belongs to exactly one window and one result is produced per interval.

Question 4Design data processing systems

A 5 TB BigQuery table is queried mostly with filters on an event_date column. Which table design reduces the bytes scanned by those queries most directly?

  1. Replicate the table into several datasets
  2. Add a search index on event_date
  3. Partition the table by event_date Correct answer.
  4. Create a logical view over the table that filters on event_date

Explanation

Partitioning by the column used in the filter enables partition pruning, so BigQuery reads only the partitions that satisfy the predicate and the scanned bytes drop accordingly.

Question 5Design data processing systems

Which fully managed Google Cloud service runs Apache Airflow environments for authoring and scheduling workflow DAGs?

  1. Dataproc
  2. Workflows
  3. Cloud Composer Correct answer.
  4. Cloud Data Fusion

Explanation

Cloud Composer is the managed Apache Airflow offering, so existing Airflow DAGs and operators run with minimal modification while the environment itself is operated by Google.

Question 6Design data processing systems

An organization already has Apache Spark jobs and wants to run them with minimal code changes on managed clusters that can be created per job, and deleted afterwards. Which service fits best?

  1. Dataproc Correct answer.
  2. Dataflow
  3. BigQuery
  4. Pub/Sub

Explanation

Dataproc is a managed Spark and Hadoop service, so existing jobs run largely unmodified, and its ephemeral-cluster model lets you pay for compute only while the job runs.

Question 7Design data processing systems

Which storage format is columnar, supports column pruning and predicate pushdown, and is commonly used for analytical data in Cloud Storage?

  1. Avro
  2. JSON
  3. CSV
  4. Parquet Correct answer.

Explanation

Parquet stores values column by column with per-block metadata, so engines read only the columns a query needs and can skip blocks whose statistics fail the predicate.

Question 8Design data processing systems

Which service continuously replicates changes from operational databases such as MySQL and PostgreSQL into BigQuery or Cloud Storage with minimal load on the source?

  1. Datastream Correct answer.
  2. Storage Transfer Service
  3. BigQuery Data Transfer Service
  4. Cloud Scheduler

Explanation

Datastream performs log-based change data capture from sources such as MySQL, PostgreSQL, SQL Server and Oracle, and writes the change stream to destinations including BigQuery and Cloud Storage.

Question 9Design data processing systems

A streaming Dataflow pipeline uses fixed windows, the default trigger, and no configured allowed lateness. What happens to an element that arrives after the watermark has passed the end of its window?

  1. It triggers reprocessing of the entire window from the source
  2. It is sent to a dead-letter queue automatically
  3. It is reassigned to the next window
  4. It is dropped because allowed lateness defaults to zero Correct answer.

Explanation

Allowed lateness defaults to zero, so once the watermark passes the end of a window the window's state is discarded; elements arriving later cannot be added to that window and are dropped unless allowed lateness and a late-firing trigger are configured.

Question 10Design data processing systems

An organization with BigQuery flat-rate commitments needs to divide slot capacity between departments so that one department's burst cannot starve another. Which BigQuery resource should they use?

  1. A separate materialized view for every department's dashboards
  2. Partitioned tables with short partition expiration
  3. Placing each department's datasets in its own region
  4. Reservations, with child reservations to isolate workloads Correct answer.

Explanation

Committed slots are allocated through reservations, and assignments attach projects to them; child reservations split capacity within a parent reservation so each workload is bounded by its own share.

Question 11Design data processing systems

A time-series table in Cloud Bigtable stores many measurements per device. Queries fetch the most recent measurements for a given device, and writes must not concentrate on a single tablet. Which row key design meets both goals?

  1. A monotonically increasing timestamp as the key prefix
  2. A random UUID per row
  3. The device ID followed by a reversed timestamp Correct answer.
  4. A hash of the entire row as the key

Explanation

Leading with the device ID keeps each device's measurements in a contiguous range for efficient scans, and a reversed timestamp sorts the newest rows first so recent data is read with a small row limit while spread across many device prefixes.

Question 12Design data processing systems

A streaming Dataflow job accumulates large amounts of window state. The team wants the runner to move state and shuffle storage off worker disks into a managed backend. Which option provides this?

  1. Horizontal autoscaling
  2. Flexible Resource Scheduling (FlexRS)
  3. Larger worker machine types with more local SSD
  4. Streaming Engine Correct answer.

Explanation

Streaming Engine moves streaming shuffle data and window state out of worker VMs into a Dataflow-managed backend, decoupling state size from worker persistent disk and speeding up snapshotting and autoscaling.

Question 13Design data processing systems

A team streams high volumes into BigQuery and needs no duplicate rows even when writers retry after transient failures. Which approach provides this?

  1. The Storage Write API with application-created streams in exactly-once mode Correct answer.
  2. Legacy streaming inserts that set an insertId value on every row to deduplicate
  3. Repeated batch load jobs that append the same Cloud Storage files each hour
  4. BigQuery Data Transfer Service running on a one-minute schedule

Explanation

Application-created streams in the Storage Write API track an offset per stream, so a retried append is deduplicated and committed exactly once, and the API is also cheaper than legacy streaming inserts.

Question 14Design data processing systems

Each streamed event must be enriched with a lookup table that is rebuilt every few minutes and fits comfortably in memory. Which approach keeps the lookup fresh without restarting the pipeline?

  1. Package the table into the pipeline artifact at build time
  2. Use a side input built from a periodically refreshed read of the table Correct answer.
  3. Broadcast the table through Pub/Sub to every worker with each event
  4. Call a remote API once per element inside the DoFn

Explanation

A side input is the Beam construct that makes a bounded or slowly changing dataset available to every element in a DoFn, and re-reading the source on a schedule refreshes it without redeploying the pipeline.

Question 15Design data processing systems

Some messages on a Pub/Sub subscription consistently crash the subscriber. The team wants those messages set aside after several attempts so processing of other messages continues. What should they configure?

  1. A dead-letter topic with a maximum delivery attempt count Correct answer.
  2. Enable message ordering on the subscription
  3. Increase the ack deadline to the maximum of 600 seconds
  4. Increase the max outstanding messages setting on the client

Explanation

A dead-letter policy forwards a message to a dead-letter topic once it has been delivered a configured number of times, so poison messages stop being redelivered and the rest of the stream keeps flowing.

Question 16Design data processing systems

Analysts must query Parquet files in Cloud Storage from BigQuery with column-level access control and materialized views, without loading the data into BigQuery native storage. Which table type fits?

  1. A BigLake table over the Cloud Storage bucket Correct answer.
  2. A Cloud SQL federated query
  3. A BigQuery native table populated by a nightly load job
  4. A BigQuery external table over the Cloud Storage bucket

Explanation

BigLake tables read data in Cloud Storage while exposing BigQuery governance features such as row- and column-level access control, and they support materialized views over the external data.

Question 17Design data processing systems

A nightly Spark job runs for about 40 minutes, and the team wants the lowest operational burden and cost. Which approach should they choose?

  1. Rewriting the job as a set of Cloud Functions invocations
  2. Dataproc Serverless, or an ephemeral cluster deleted after each job Correct answer.
  3. A single-node cluster with local SSDs kept running continuously
  4. A long-running cluster with autoscaling enabled that stays up between jobs

Explanation

Paying for compute only while the job runs minimizes cost; Dataproc Serverless removes cluster management entirely, and an ephemeral cluster achieves the same by being destroyed when the job finishes.

Question 18Design data processing systems

Which Beam transform computes a per-key sum in a streaming pipeline while minimizing the amount of data shuffled between workers?

  1. Combine.perKey with a sum CombineFn Correct answer.
  2. A Reshuffle after an identity ParDo
  3. GroupByKey followed by a ParDo that sums the values
  4. A ParDo that reads all values from a side input

Explanation

Combine.perKey performs partial combining on each worker before the shuffle, so far fewer values cross the network than with GroupByKey, which shuffles every value for every key.

Question 19Design data processing systems

A dashboard runs the same expensive aggregation over a large BigQuery table many times a day, and results may be a few minutes stale. Which feature reduces cost and latency with the least application change?

  1. Exporting the aggregate to Bigtable and querying it from the dashboard
  2. A scheduled query that appends results to a new table every hour
  3. Adding more clustering columns to the base table
  4. A materialized view that BigQuery refreshes automatically Correct answer.

Explanation

A materialized view stores the precomputed aggregate, is refreshed automatically, and the query optimizer can rewrite matching queries to read the view, so the dashboard query text stays unchanged.

Question 20Design data processing systems

A workload needs very high-throughput messaging confined to a single zone, and the team is willing to provision capacity explicitly in exchange for lower cost. Which service fits?

  1. Pub/Sub Lite Correct answer.
  2. Eventarc
  3. Cloud Tasks
  4. Pub/Sub with ordering keys

Explanation

Pub/Sub Lite is a zonal messaging service with explicitly provisioned throughput capacity, which makes it cheaper than Pub/Sub when traffic is predictable and cross-zone durability is not required.

Question 21Design data processing systems

A streaming pipeline emits early results for a window using a speculative trigger. Consumers need each emitted pane to contain the complete count for the window so far, not just the elements added since the previous firing. Which setting achieves this?

  1. A single global window instead of fixed windows
  2. Discarding panes for the trigger
  3. A shorter allowed lateness value
  4. Accumulating panes for the trigger Correct answer.

Explanation

Accumulation mode determines pane contents: accumulating panes repeat the full aggregation for the window on every firing, whereas discarding panes emit only the elements received since the previous firing.

Question 22Design data processing systems

A BigQuery table is partitioned on the TIMESTAMP column event_ts and clustered on customer_id. Which predicate pattern lets the engine prune both partitions and clustered blocks?

  1. WHERE customer_id = 'C-100'
  2. WHERE event_ts BETWEEN '2024-05-01' AND '2024-05-02'
  3. WHERE FORMAT_TIMESTAMP('%Y-%m-%d', event_ts) = '2024-05-01' AND customer_id = 'C-100'
  4. WHERE event_ts BETWEEN '2024-05-01' AND '2024-05-02' AND customer_id = 'C-100' Correct answer.

Explanation

Partition pruning requires a comparison directly on the partitioning column with constant values, which the BETWEEN range supplies; the additional equality on the clustering column then lets the engine skip blocks inside the surviving partitions.

Question 23Design data processing systems

A streaming pipeline reads from two unbounded sources and joins the results. One source goes idle for a long period while the other keeps producing. What happens to the watermark?

  1. The watermark advances with the fastest source, so each window closes as soon as that source passes it
  2. Each source keeps its own independent watermark and windows close separately for each source
  3. The watermark is based only on processing time, so an idle source has no effect on window closure
  4. The watermark is the minimum across inputs, so the idle source holds it back Correct answer.

Explanation

Beam tracks a watermark per input and the downstream watermark is the minimum of the upstream watermarks; a source that stops producing without being marked idle therefore prevents windows from closing.

Question 24Design data processing systems

A Dataflow batch pipeline reads a single 2 TB gzip-compressed object from Cloud Storage. Even with 50 workers, only one worker stays busy. What is the most effective fix?

  1. Increase the maximum number of workers to 200 so the file is read in parallel
  2. Attach larger persistent disks and more local SSD to each worker
  3. Enable Streaming Engine so state is offloaded from the workers
  4. Rewrite the source as many evenly sized Avro or Parquet shards Correct answer.

Explanation

gzip is not splittable, so a single object can be read by only one worker regardless of cluster size; writing the data as many shards, ideally in a splittable columnar format, lets Cloud Storage reads be distributed across workers.

Question 25Design data processing systems

A streaming pipeline computes per-user aggregates by grouping on user_id. A handful of very active users generate most of the traffic and those keys become a processing bottleneck. Which technique best mitigates the skew?

  1. Enable discarding panes on the window trigger to reduce output size
  2. Increase the maximum number of workers so the hot keys are spread out
  3. Switch from fixed windows to the global window to reduce per-window state
  4. Salt the key, combine per salted key, then re-combine the partial results Correct answer.

Explanation

Adding a random suffix to the hot key distributes one logical key across many workers, and a second combine merges the partial results, which is the standard hot-key mitigation pattern in Beam.

Question 26Design data processing systems

A data pipeline writes each event to Cloud Storage as a single complete record, and downstream consumers read entire records. New optional fields are added to the record schema frequently, and readers must handle both old and new files without rewriting them. Which file format should you choose?

  1. Avro, which stores the schema with the data and supports evolution Correct answer.
  2. Parquet, a columnar format optimized for scanning a few columns of wide tables
  3. ORC, a columnar format used by Hive-style engines with predicate pushdown
  4. CSV with a header row that consumers parse themselves

Explanation

Avro is row-oriented and embeds the writer schema in each file, so a reader can resolve older files against a newer schema. Adding optional fields with defaults is a compatible change, which suits record-at-a-time writes of evolving events.

Question 27Design data processing systems

An orders feed has one order header and a variable number of line items. Almost every query reads a full order together with all of its line items, and analysts must avoid joins and must keep exactly one row per order so header-level aggregations need no DISTINCT. Which BigQuery table design should you choose?

  1. One row per line item with the order header columns duplicated on every row
  2. One table per line item, joined to an orders table at query time
  3. A single table with the line items in a repeated RECORD column Correct answer.
  4. A single table with the line items serialized into a JSON STRING column

Explanation

BigQuery stores nested records efficiently, and a repeated RECORD keeps an order and its line items in one row, so header-level aggregations need no DISTINCT and no join. This denormalized shape is the recommended pattern for one-to-many relationships.

Question 28Design data processing systems

Multiple publishers send commands about the same account to one Pub/Sub topic, and a subscriber must process commands for a given account in the order they were published. What should you configure?

  1. Set the ordering key to the account ID and enable ordered delivery on the subscription Correct answer.
  2. Increase the acknowledgement deadline on the subscription so that slow consumers do not lose messages
  3. Set the message retention duration to the maximum of seven days
  4. Create one topic per account, each with its own subscription

Explanation

Pub/Sub preserves order only for messages that share an ordering key, and only when message ordering is enabled on the subscription; messages with the same key are delivered in publish order. Topic structure and deadlines do not affect ordering.

Question 29Design data processing systems

A team wants to reuse one Dataflow pipeline from Cloud Scheduler, the gcloud CLI, and the console, passing runtime parameters and using third-party Python libraries that are not in the default worker image. Which packaging approach should they use?

  1. A subscription filter that launches the job when a message arrives
  2. A classic template built from a pipeline that has no extra dependencies and no runtime parameters
  3. Running the pipeline locally with the DirectRunner on a Compute Engine VM
  4. A Flex Template: a container image plus a spec file declaring runtime parameters Correct answer.

Explanation

A Flex Template packages the pipeline and its dependencies into a container image and ships a metadata file describing the runtime parameters, so the same job can be launched from the console, gcloud, or the REST API. Third-party libraries are baked into the image.

Question 30Design data processing systems

A Bigtable table stores sensor readings with the row key built as the reading timestamp followed by the sensor ID. Write throughput stays low because only one tablet serves all new writes. Which row key change spreads writes across tablets while keeping a sensor's readings contiguous?

  1. Prefix the row key with the reading time truncated to the hour
  2. Append a random suffix to the end of each timestamp-based row key
  3. Prefix the row key with a hash of the sensor ID, then the timestamp Correct answer.
  4. Keep one row per sensor and store each reading in a new column, using the timestamp as the column qualifier

Explanation

Bigtable sorts rows lexicographically and splits them into tablets by key range, so keys that begin with a steadily increasing timestamp send all new writes to the last tablet. Hashing the sensor ID into the prefix spreads writes over the key space while keeping each sensor's rows contiguous.

Question 31Design data processing systems

A nightly batch job reprocesses the last three days of order events and writes them to a BigQuery table. Rerunning the job must not create duplicate rows, and orders whose status changed must be updated in place. Which approach should you use?

  1. Truncate the table and reload only the three days that were reprocessed
  2. Append the reprocessed rows to the table with a load job using WRITE_APPEND
  3. Append every reprocessed row and let analysts remove duplicates with SELECT DISTINCT in their queries
  4. MERGE the reprocessed rows on the order ID, updating matches and inserting new rows Correct answer.

Explanation

MERGE matches rows on a business key and applies updates and inserts in a single statement, so rerunning the same window converges to identical table contents. That idempotency is what makes reprocessing safe.

Question 32Design data processing systems

A daily pipeline must run a Dataproc job, then several BigQuery queries, then a data quality check, with retries, alerting, and the ability to backfill a range of past dates. The team wants Python DAGs with one operator per service. Which service should orchestrate it?

  1. A Dataflow pipeline whose steps are implemented as a chain of DoFns
  2. Pub/Sub topics chained through push subscriptions to trigger each service
  3. Cloud Scheduler jobs that call each service directly on cron expressions
  4. Cloud Composer with an Airflow DAG that has one task per step Correct answer.

Explanation

Cloud Composer runs Airflow DAGs, which model task dependencies, retries, alerting, and date-parameterized backfills, and it provides operators for Dataproc and BigQuery. That combination matches a multi-step pipeline spanning several services.

Question 33Design data processing systems

Files arrive unpredictably in a Cloud Storage bucket, dozens per hour, and each must be picked up for transformation within a few seconds of landing. The team wants a managed, event-driven trigger without polling. Which design should they use?

  1. Pub/Sub notifications on the bucket for OBJECT_FINALIZE, consumed by a streaming pipeline Correct answer.
  2. A cron job that lists the bucket every minute and starts one batch job per new file
  3. A long-running Dataproc cluster that polls the bucket continuously with a Hadoop file system watcher
  4. A BigQuery scheduled query over an external table that points at the bucket

Explanation

Cloud Storage can publish OBJECT_FINALIZE notifications to Pub/Sub, giving an event-driven signal for each object as soon as it is written. A streaming pipeline reading that subscription processes each file within seconds without polling.

Question 34Design data processing systems

A large batch pipeline runs once a night, and the business can accept results up to six hours after the job is submitted. Cost matters far more than latency. Which Dataflow configuration fits?

  1. The DirectRunner on one large Compute Engine VM
  2. Flexible Resource Scheduling (FlexRS) for the batch job Correct answer.
  3. Streaming Engine enabled on the batch pipeline
  4. A fixed worker pool sized for peak load with autoscaling disabled

Explanation

FlexRS schedules batch jobs on a mix of preemptible and standard workers and may delay execution, which lowers cost for workloads that tolerate hours of latency. Streaming-oriented features and fixed pools do not deliver that saving.

Question 35Design data processing systems

A streaming pipeline writes micro-batch output to Cloud Storage as thousands of small Avro files per hour. Downstream Spark and BigQuery jobs over these files are slow and metadata-heavy. Which design change should you make?

  1. Add a compaction step that periodically rewrites many small files into fewer large files Correct answer.
  2. Convert the output to CSV so that readers can parse the files line by line
  3. Write each micro-batch as one gzip-compressed file instead of many files
  4. Increase the number of files written per micro-batch so that each individual file is smaller and faster to create

Explanation

Many tiny objects force every reader to list and open each one, which dominates runtime for Spark and BigQuery and adds listing costs. Periodic compaction into a handful of large files restores sequential scanning and keeps the object count manageable.

Question 36Design data processing systems

A data lake in Cloud Storage must let the team rebuild any derived table after a logic bug is found, without asking source systems to resend data. Which layout best supports this?

  1. Raw data left only in source system backups, read back during rebuilds
  2. A single bucket where the raw data is overwritten in place by cleansed data after every successful run
  3. Only the cleansed and aggregated outputs, to keep storage costs low
  4. An immutable raw zone holding the original data, plus cleansed and curated zones derived from it Correct answer.

Explanation

Keeping the original data immutable in a raw zone means any derived dataset can be recomputed with corrected logic, and each zone gains a clear owner and retention rule. The downstream zones then become disposable outputs rather than sources of truth.

Question 37Design data processing systems

Three independent consumers must each process every event published to a topic: a real-time dashboard, a data lake loader, and an alerting service. Each runs at a different speed. What should you design?

  1. One subscription with a filter that routes events to the right consumer
  2. One subscription shared by all three consumers so each event is processed once
  3. One topic with three separate subscriptions, one per consumer Correct answer.
  4. Three topics, with publishers writing every event to each of them

Explanation

A subscription is an independent stream of messages, so each consumer needs its own subscription to receive every event at its own pace, while a single topic keeps publishing simple. A shared subscription would instead divide the messages among consumers.

Question 38Design data processing systems

Each incoming event must be enriched with a product record. The catalog has tens of millions of rows, changes throughout the day, and does not fit in worker memory, while enrichment must add only a few milliseconds. What design should you use?

  1. Load the catalog as a side input and refresh it every five minutes
  2. Serve lookups from Bigtable, kept current by a separate pipeline, with a client-side cache Correct answer.
  3. Use a global window and GroupByKey on product ID so that events and catalog rows meet in memory
  4. Run a BigQuery query from inside the DoFn for every event to fetch the matching product record

Explanation

When the dimension is too large for worker memory and changes continuously, lookups should be served from a system built for low-latency key access, with a cache absorbing repeated keys and a separate pipeline applying catalog updates. This keeps per-event latency low and memory bounded.

Question 39Design data processing systems

A Dataflow streaming pipeline writes aggregates to an external system whose API offers neither deduplication nor transactions. The runner may retry a bundle after a worker failure. How should you design the writes so retries do not double count?

  1. Rely on the runner's exactly-once processing guarantee to suppress the duplicate call to the external system
  2. Derive a deterministic key per window and key, and have the sink apply set or conditional-update semantics Correct answer.
  3. Buffer writes in memory and flush them only when the pipeline is drained
  4. Increase the number of workers so that retried bundles are handled in parallel

Explanation

Beam's exactly-once guarantees cover pipeline state and supported sinks inside the pipeline, but a retried bundle can still emit an external side effect twice. Idempotent writes keyed deterministically make a retry produce the same result as the original attempt.

Question 40Design data processing systems

A pipeline runs on Dataflow workers in europe-west1, reads raw files from a bucket in the same region, and writes to a BigQuery dataset created in the US multi-region. Analysts then join that dataset with tables stored in the EU. What design problem will this cause?

  1. The dataset must be converted from multi-region to regional before any load can succeed
  2. Queries cannot join tables in different locations, and cross-region movement adds latency and egress cost Correct answer.
  3. Nothing; BigQuery automatically copies data between locations when a query needs it
  4. The Dataflow job will fail to start because the input bucket and the output dataset are in different locations

Explanation

BigQuery requires every table referenced by a query to be in the same location, so a US multi-region dataset cannot be joined with EU tables. Writing pipeline output across regions also adds latency and network egress charges, so the dataset, bucket, workers, and analysts should share a location.

Question 41Ingest and process the data

An application using a Pub/Sub pull subscription needs more time to process a message than the subscription's ack deadline allows. What should the application do to avoid redelivery while processing?

  1. Extend the message ack deadline. Correct answer.
  2. Convert the subscription to push delivery.
  3. Create a second topic and republish the message.
  4. Increase the topic message retention duration.

Explanation

A pull subscriber can call modifyAckDeadline to extend the deadline for a specific message while it is being processed. The ack deadline is the retry timer, so extending it prevents redelivery during long processing.

Question 42Ingest and process the data

Which Google Cloud service is designed to continuously replicate changes from operational databases such as MySQL and PostgreSQL into BigQuery with minimal code?

  1. Datastream Correct answer.
  2. Dataproc
  3. Cloud Composer
  4. Cloud Data Fusion

Explanation

Datastream is a serverless change data capture service that continuously streams changes from operational databases into BigQuery or Cloud Storage. It handles schema and data replication with minimal code.

Question 43Ingest and process the data

Which service should be used to author and schedule a directed acyclic graph that coordinates multiple batch ingestion jobs?

  1. BigQuery with scheduled queries
  2. Pub/Sub with a push subscription
  3. Dataflow with a streaming job
  4. Cloud Composer Correct answer.

Explanation

Cloud Composer is managed Apache Airflow and is used to author, schedule, and monitor DAGs that coordinate jobs across services. A DAG defines task dependencies and retry behavior for the ingestion workflow.

Question 44Ingest and process the data

Which service can trigger a Cloud Run service when a new object is created in a Cloud Storage bucket without writing polling code?

  1. Cloud Tasks
  2. Data Transfer Service
  3. Cloud Scheduler
  4. Eventarc Correct answer.

Explanation

Eventarc routes events from Google Cloud sources to Cloud Run, Cloud Functions, and GKE, so an object-finalize event can invoke a service. This avoids polling Cloud Storage.

Question 45Ingest and process the data

A batch of CSV files is already in Cloud Storage and must be loaded once into BigQuery. Which approach is the most direct managed batch load?

  1. Use Cloud Composer to issue one API call per row.
  2. Create a Dataproc cluster and run a Hadoop job.
  3. Run a BigQuery load job with a wildcard URI. Correct answer.
  4. Stream each file through Pub/Sub into BigQuery.

Explanation

BigQuery load jobs can load multiple Cloud Storage objects in one job using a wildcard URI and are the managed batch-load path. Streaming, Dataproc, or per-row API calls add unnecessary complexity.

Question 46Ingest and process the data

Which Dataproc option runs Spark workloads without requiring you to create or manage a cluster?

  1. Dataproc personal cluster authentication
  2. Dataproc Metastore
  3. Dataproc Serverless Correct answer.
  4. Dataproc on Compute Engine with preemptible workers

Explanation

Dataproc Serverless runs Spark workloads on managed resources without requiring you to provision or manage a cluster. Dataproc on Compute Engine still requires cluster configuration and lifecycle management.

Question 47Ingest and process the data

In Pub/Sub, which setting controls how long unacknowledged messages are retained after they are published, independent of the ack deadline?

  1. Subscription ack deadline configuration
  2. Message ordering key configuration
  3. Message retention duration Correct answer.
  4. Dead-letter topic setting

Explanation

The subscription's message retention duration controls how long Pub/Sub retains unacknowledged messages even after acknowledgement deadlines expire. It is separate from the ack deadline, which controls when an individual delivery is retried.

Question 48Ingest and process the data

Which Google Cloud compute option is best for running a small event-driven Python function that responds to Pub/Sub messages without managing servers?

  1. Cloud Composer
  2. Dataproc
  3. Cloud Functions Correct answer.
  4. Google Kubernetes Engine

Explanation

Cloud Functions provides serverless execution for small event-driven functions and can be triggered directly by Pub/Sub messages. It removes the need to manage servers or clusters for that handler.

Question 49Ingest and process the data

A subscription has a dead-letter topic configured after five delivery attempts. A message repeatedly fails because the subscriber throws an exception before acknowledging. What happens after the fifth failed delivery?

  1. Forward it to the dead-letter topic. Correct answer.
  2. Retry forever with the same delay.
  3. Delete the message permanently.
  4. Republish it to the original topic.

Explanation

When a subscription has a dead-letter topic and a message exceeds the maximum delivery attempts, Pub/Sub forwards that message to the dead-letter topic. This isolates failing messages for inspection.

Question 50Ingest and process the data

An advertisement click stream is unbounded, and the business needs total clicks per ad in five-minute, non-overlapping intervals. Which Dataflow windowing strategy should be used?

  1. Session windows with a 30-minute gap
  2. Fixed-time windows of five minutes Correct answer.
  3. A single global window with no trigger
  4. Sliding windows of five minutes every one minute

Explanation

Fixed-time windows divide an unbounded stream into non-overlapping intervals, so five-minute fixed windows produce one aggregate per ad per interval. Session and sliding windows create different groupings.