GCP Data Engineer Pro case studies

GCP Data Engineer Pro · Put your knowledge to work in context.

The Professional Data Engineer certification is Google Cloud's credential for practitioners who build and operate data platforms: collecting, transforming, storing, and delivering data for analytics, applications, and downstream teams. It is aimed at data engineers and data architects who already work with pipelines, warehouses, and streaming systems and want independent validation of that skill rather than an entry-level badge. The GCP Data Engineer Pro case studies are 5 cases and about 4,937 words.

Sessionizing a late-arriving clickstream without double-counting

A retail platform publishes clickstream to Pub/Sub at 12,000 messages/s average and 40,000/s at peak (about 1.5 KB each, keyed by session id). Product wants sessionized funnels in BigQuery with at most 5 minutes of latency.

1,031 words, about 5 min 2

Replacing a nine-hour nightly dump with change data capture

An on-premises PostgreSQL 13 database holds 4 TB across 900 tables. The analytics team needs only 60 of them, but those include a 1.2-billion-row orders table where rows are updated in place and hard-deleted on GDPR requests (about 200 deletions/day).

925 words, about 5 min 3

An 8 TB event table that scans 1.1 TB for a single-customer query

A BigQuery table events holds 8 TB over 24 months (~11 GB/day, ~400 million rows/month). It is partitioned on ingestion time and has no clustering.

1,001 words, about 5 min 4

Forty staggered scheduled queries with no real dependency graph

A BI team maintains 40 BigQuery scheduled queries that build the mart. Each is a CREATE OR REPLACE TABLE fired 15 minutes after the previous one, from 01:00 to 11:00 UTC; the chain burns 6 hours of wall clock for roughly 40 minutes of compute.

1,000 words, about 5 min 5

Autoscaling stalls at 40 workers while the backlog grows

A Dataflow streaming pipeline (Streaming Engine on, --maxNumWorkers=200) reads Pub/Sub at a steady 25,000 msg/s, enriches each element with a BigQuery lookup, aggregates per account id, and writes to BigQuery with the Storage Write API. Since a new customer onboarded, system lag has climbed from 5 seconds to 90 minutes and Pub/Sub oldest unacked message age keeps rising, yet average worker CPU is 22% and the job is pinned at 40 workers.

980 words, about 5 min