GCP Data Engineer case studies
GCP Data Engineer · Put your knowledge to work in context.
The Professional Data Engineer certification is a Google Cloud credential for practitioners who collect, transform, store, and deliver data for diverse applications. According to Google Cloud, a Professional Data Engineer designs and builds robust data infrastructure, optimizes for performance and security, evaluates and selects solutions to meet business and regulatory needs, administers data platforms, and applies current technologies for data processing, cleaning, enrichment, and query generation and translation. The GCP Data Engineer case studies are 5 cases and about 4,334 words.
1Clickstream Funnels With Skewed Device Clocks
A retail platform emits 45,000 clickstream events per second at peak and about 12,000 on average, published to Pub/Sub with the device clock timestamp in the payload. Roughly 3% of devices have clocks that run up to 40 minutes slow, and a handful run fast.
776 words, about 4 min2Nightly Oracle and File Ingestion Under a Shrinking Window
An insurer runs 60 on-premises jobs that drop CSV extracts onto an SFTP server plus 14 Oracle tables replicated to BigQuery. Nightly volume is 2.5 TB and everything must be queryable before the 07:00 claims stand-up.
863 words, about 4 min3A 4 PB BigQuery Table and a Five-Fold Bill
A media company keeps 6 PB in BigQuery across 40 datasets. One table, events , holds 4 PB and is partitioned by ingestion time on PARTITIONTIME at DAY granularity.
867 words, about 4 min4A 90-Second Dashboard and Silently Dropped Revenue
A logistics company has a 900 GB BigQuery shipments fact table and 12 dimension tables. A Looker dashboard used by 300 operations staff joins shipments to 6 dimensions and takes 90 seconds per load; the average user refreshes it 40 times an hour.
897 words, about 4 min5220 Airflow DAGs, Duplicate Rows, and a 02:00 Out-of-Memory
A subscription company runs 220 Airflow DAGs on a Cloud Composer 2 environment with 3 workers. Nightly, each DAG triggers either a Dataproc Serverless batch or a Dataflow job that appends results to BigQuery.
931 words, about 5 min