GCP Data Engineer common mistakes
GCP Data Engineer · Understand the traps and how to avoid them.
Selecting Dataflow (or a custom Python script on a VM) as the answer for every ingestion scenario, including one-off bulk transfers from an on-premises NAS or a scheduled pull from an SaaS API.
Candidates learn Dataflow early and treat it as the universal ingestion tool, so they stop evaluating the scenario's actual constraints (volume, source type, one-time vs recurring, network bandwidth).
What to do instead
Build a decision table before the exam: Storage Transfer Service for object-to-object and on-prem-to-GCS bulk moves, Transfer Appliance for multi-hundred-TB or low-bandwidth links, Datastream for CDC from relational databases, BigQuery Data Transfer Service for SaaS connectors, Cloud Scheduler + Cloud Functions/Workflows for lightweight scheduled pulls, and Dataflow for transformation-heavy or streaming pipelines.
Designing a Pub/Sub-to-BigQuery streaming pipeline that assumes each message is processed once and writes directly to the sink with no deduplication key.
Candidates conflate 'the pipeline did not crash' with 'exactly-once', and forget that Pub/Sub messaging is at-least-once, so redeliveries after ack deadline expiry or worker restarts create duplicate rows.
What to do instead
State the delivery guarantee explicitly in the design: Pub/Sub is at-least-once, so choose either an idempotent sink (BigQuery streaming inserts with insertId deduplication, or a MERGE on a natural/business key) or a stateful dedup step keyed on message ID with an appropriate state/TTL, and say which one you picked and why.
Proposing a fixed-window streaming pipeline with a default trigger and no allowed lateness when the scenario explicitly mentions mobile clients that deliver events minutes or hours late.
Candidates memorise window types as vocabulary (fixed, sliding, session) but never connect 'late data' in the prompt to watermark behaviour, so they pick a window that silently drops the late records.
What to do instead
When the prompt mentions out-of-order or delayed events, choose the window type from the business question, then explicitly set allowed lateness and a trigger (for example, emit early results with speculative triggers and a final pane after lateness expires) or use session windows when the grouping is activity-based rather than time-based.
Creating a BigQuery table with no partitioning, or partitioning by ingestion time when every dashboard query filters on a business timestamp column, and treating clustering as interchangeable with partitioning.
Candidates know partitioning 'improves performance' but not that pruning only happens when the query filters on the partition column, and they mix up the distinct roles of partitioning (data lifecycle and coarse pruning) and clustering (within-partition sort order for filtered and aggregated queries).
What to do instead
Choose the partition column by matching the WHERE clause the workload actually uses, prefer column-based time partitioning over ingestion-time when the event time is available, and add clustering on the high-cardinality columns that appear in filters and joins; remember clustering also improves the effectiveness of partition pruning.
Describing a Dataproc design with a long-running cluster that persists between jobs and stores intermediate data on attached persistent disks (HDFS).
Candidates transfer on-premises Hadoop habits into GCP, where the cost-efficient and resilient pattern is a job-scoped cluster that reads and writes through the Cloud Storage connector.
What to do instead
Default to ephemeral, job-scoped clusters (or Dataproc Serverless for Spark) with data in Cloud Storage via the GCS connector, keep local disks only for shuffle/temporary use, and reserve long-lived clusters for interactive or shared-workload cases where you can justify the cost.
Assuming a logical view in BigQuery precomputes and caches its results, so several teams can query it repeatedly at no cost, and using a view where the scenario asks for a repeatedly refreshed aggregate.
Candidates treat BigQuery views like indexed relational views; in fact a logical view is free to store but re-executes its query on every access, and every query is billed for the bytes scanned.
What to do instead
Distinguish the three options: logical view for access control and query simplification with no storage cost, materialized view for repeated aggregate queries with automatic refresh (subject to its SQL restrictions), and a scheduled query writing to a partitioned table when the transformation is complex or the refresh cadence must be controlled.
Answering a cost-isolation or performance-guarantee question with 'on-demand pricing is fine' or 'just optimise the queries', without mentioning slots or reservations.
Candidates learn BigQuery slot mechanics superficially and cannot map a scenario like 'finance queries must never be starved by the ETL load' onto the reservation model.
What to do instead
Learn the two billing modes as a design choice: on-demand bills per byte scanned and shares a project-level pool, while flat-rate/autoscale reservations allocate slots that can be split into reservations and assignments per project or folder for workload isolation; combine this with partition pruning and custom query quotas as the cost guardrails.
Recommending Cloud Composer to run a single nightly SQL statement or a single API call, because it is the only orchestrator the candidate can name.
Candidates equate 'automate a data workload' with 'use Airflow', which produces an over-engineered design and often the wrong answer when the scenario stresses operational simplicity or cost.
What to do instead
Match orchestration weight to dependency complexity: Cloud Scheduler for a single cron trigger, Cloud Scheduler plus Workflows or Cloud Functions for a short linear chain, scheduled queries for in-warehouse transforms, and Composer only when you need multi-step DAGs, backfills, cross-system dependencies, retries, and observability across many tasks.
Leaving monitoring out of the design, or answering 'the job shows as failed in the console' when asked how the team will detect a stuck streaming pipeline or a growing subscription backlog.
Candidates study building pipelines far more than operating them, so they cannot name the metrics that reveal silent degradation rather than hard failure.
What to do instead
Name the signals and the alerting path: Cloud Monitoring alerting policies on Dataflow system lag and job/dropped-due-to-lateness counters, Pub/Sub oldest unacked message age and undelivered message count, Dataproc job and YARN metrics, and BigQuery slot utilisation; route them through alerting policies to a notification channel and pair them with Cloud Logging-based log metrics for pipeline-specific errors.
Designing the pipeline with a personal user account running gcloud commands and a downloaded service account key, and omitting CMEK or VPC Service Controls, even when the scenario says the data is regulated or must stay in a region.
Candidates focus on the data-flow boxes and treat identity, encryption, and network perimeter as implementation details that do not affect the design answer.
What to do instead
Treat the security and residency constraints in the prompt as hard requirements: dedicated user-managed service accounts with least-privilege predefined or custom roles per component, Workload Identity or attached service accounts instead of exported keys, CMEK for BigQuery, Cloud Storage, and Pub/Sub where mandated, and VPC Service Controls plus private IP and regional resource placement when data exfiltration or residency is called out.