GCP Data Engineer Pro common mistakes
GCP Data Engineer Pro · Understand the traps and how to avoid them.
Selecting the processing engine by product familiarity ("big data means Dataproc") rather than by the workload's actual shape - e.g., proposing a Dataproc cluster for a small unbounded streaming pipeline, or proposing Dataflow for a lift-and-shift migration of existing Hadoop/Spark jobs.
Scenario stems are long and candidates latch onto a keyword (streaming, Spark, petabytes) instead of extracting the four decision axes: bounded vs unbounded, latency requirement, existing code/skills, and operational ownership. They also assume a managed cluster is always cheaper than a serverless runner.
What to do instead
Before answering, write the axes down and map each candidate service to them. Unbounded/low-latency/autoscaling/no-ops → Dataflow (Apache Beam). Existing Spark/Hive code or fine-grained cluster control → Dataproc (ephemeral, with autoscaling and scheduled deletion). Pure SQL/BI aggregation → BigQuery. Single-step event-driven glue → Cloud Run functions or Workflows. Eliminate any option that fails an axis before comparing cost.
Designing a pipeline with no deduplication or replay story, assuming the chosen services give end-to-end exactly-once as a matter of course.
Marketing-level "exactly-once" claims for individual services are misread as an end-to-end guarantee. Candidates forget that a sink retry, a worker restart, or a backfill can re-deliver the same record downstream.
What to do instead
In every design answer, state the idempotency mechanism explicitly: a deterministic business key plus MERGE/upsert into the sink, a dedupe window keyed on message ID in the streaming job, and re-run-safe jobs that overwrite a partition rather than blind-append. Also state how state is checkpointed and how a backfill is bounded by date range.
Treating Pub/Sub delivery as exactly-once and writing subscribers that assume one copy of each message; ignoring ordering keys and never setting a dead-letter topic.
Pub/Sub is at-least-once by default, and exactly-once delivery is opt-in, regional, and only applies within the subscription/pull path - it does not protect the write to the sink. Ordering is also opt-in and per-key, which candidates overlook when a scenario mentions sequence-dependent events.
What to do instead
State at-least-once and deduplicate in the consumer using the message ID plus a sink-side MERGE. Enable ordering keys only when the scenario requires per-entity sequence, and remember ordered delivery constrains the subscriber's ack pattern. Always pair a subscription with a dead-letter topic, a retry policy, and monitoring on undelivered/oldest-unacked messages.
Picking the wrong ingestion mechanism for the source type - e.g., using Storage Transfer Service or a Transfer Appliance for ongoing replication out of an operational relational database, or using Datastream for a one-off bulk file move.
All the transfer-named services blur together under exam pressure. Candidates match on the word "transfer" rather than on whether the source is a database with a change log, a file/object store, or offline media.
What to do instead
Memorise the source-to-service mapping: relational DB change capture → Datastream (CDC, minimal source impact, lands in GCS/BigQuery); object/file stores and HTTP endpoints, scheduled or one-off → Storage Transfer Service; very large offline datasets with poor bandwidth → Transfer Appliance; continuous third-party SaaS/ads/logs → the vendor-native transfer service. Then check the scenario's latency and whether deletes/updates must propagate.
Choosing a BigQuery partition column or granularity that does not match the dominant filter, or clustering on the wrong columns in the wrong order, then assuming the query will still be cheap.
Candidates memorise "partition and cluster" without connecting it to the actual WHERE clause in the scenario. They partition on a high-cardinality ID (hitting the partition limit and gaining nothing), or put the lowest-selectivity column first in the clustering list, or filter with a function-wrapped date that defeats pruning.
What to do instead
Read the scenario's filter pattern first: partition on the DATE/TIMESTAMP/DATETIME or ingestion-time column that most queries constrain, keep partitions coarse (day/month), and cluster on the columns that appear next in WHERE/JOIN, ordered by descending selectivity. Verify the fix by asserting that filters use a literal range on the raw partition column, not a cast or function of it.
Recommending a colder Cloud Storage class to save money on data that is read frequently, or lifecycle-deleting objects without accounting for the minimum storage duration and retrieval charges.
The per-GB storage price is the only number candidates recall; the retrieval fee and the 30/90/365-day minimum-duration early-deletion charge are what actually dominate cost, and the scenario's access frequency is usually stated in one easily-missed sentence.
What to do instead
For each answer, pair the class with the observed access pattern and state the trade-off out loud: frequently read → Standard; roughly monthly → Nearline (30-day minimum); quarterly → Coldline (90-day); archival only → Archive (365-day). Show that lifecycle rules are based on measured age or access, and mention that Autoclass exists when the access pattern is genuinely unpredictable.
Assuming an external table, a logical view, or a standard view behaves like a native BigQuery table for performance and cost, or assuming a view pre-computes its results.
Candidates conflate "the data is queryable in BigQuery" with "the data is stored in BigQuery's columnar format". They then pick an external table over a native table in a high-concurrency dashboard scenario, or propose a view where the requirement is repeated expensive aggregation.
What to do instead
Map the requirement to the object explicitly: repeated heavy aggregation → materialized view (with its own refresh and staleness semantics); federated/lake data needing only occasional scans → external table or BigLake table; a security or abstraction boundary only → logical view (which costs nothing to store but re-executes every time). Prefer native tables when the workload is frequent and latency-sensitive.
Enforcing fine-grained access with dataset-level IAM alone when the scenario needs per-row or per-column restrictions, or granting primitive roles (Editor/Owner) to make a permissions problem go away.
Candidates know BigQuery has IAM but not the layered model underneath it: dataset-level roles, table-level and column-level ACLs/policy tags, row-level security filters, and authorized views that let a view read a source the caller cannot query directly. Under time pressure the broadest role looks like the safe answer.
What to do instead
Learn the escalation ladder and pick the narrowest rung: dataset-level predefined roles for whole-dataset access → column-level policy tags via Data Catalog/Dataplex for PII masking → row-level access policies for tenant filtering → authorized views when consumers must never see the base table. Never satisfy a least-privilege requirement with a primitive role; state the role you grant and to which resource.
Defaulting every scheduling or orchestration requirement to Cloud Composer, or conversely using Cloud Scheduler for a multi-step dependency chain, and never specifying retries, alerting, or idempotent task design.
Cloud Composer is the most familiar orchestrator name, so it becomes the reflex answer even for a single daily job or a simple two-step event chain. Candidates also treat orchestration as scheduling only and skip the failure-handling half of the requirement.
What to do instead
Size the orchestrator to the DAG: one cron-style trigger → Cloud Scheduler; a short retryable sequence of API/service calls → Workflows; cross-system DAGs with backfills, dependencies, and SLA monitoring → Cloud Composer; recurring Dataflow runs → Flex Templates with Scheduler. Whatever you pick, state retry/backoff, failure notification, and how a rerun avoids duplicating output.
Diagnosing slow or expensive pipelines by adding workers or slots, instead of identifying data skew, shuffle volume, or the pricing model mismatch (on-demand vs capacity-based) that is actually causing the problem.
Scaling out is the intuitive fix and the button is one flag away. Candidates miss the clues in the scenario - a few hot keys, an unbounded group-by, repeated full-table scans, or steady high monthly spend that would favour committed slots over per-query pricing - because they never look at the metric named in the stem.
What to do instead
Name the metric before the remedy: hot keys or stragglers → pre-aggregate or re-key (combine before shuffle), and check Dataflow's shuffle/system-lag metrics rather than worker count; high and predictable BigQuery spend → capacity commitments/reservations with job-level slot controls; intermittent heavy scans → fix partition pruning and clustering first. Attach a verification step (monitor the specific metric, add an alert) so the answer closes the loop.