Designing Data Processing Systems
Start from latency and source shape
Two questions decide most designs: how fresh must the data be, and is the input bounded (a file, a table export) or unbounded (clickstream events, change data capture)? Bounded input with hourly-or-slower output means batch: Cloud Storage or BigQuery as source, Dataproc Serverless or Dataflow batch for transforms, Composer or scheduled queries for orchestration. Unbounded input with sub-minute freshness means streaming: Pub/Sub as the ingestion buffer, Dataflow for windowed stateful processing, and BigQuery or Bigtable for serving.
Match compute to the work
Dataflow is the default for Apache Beam pipelines: it autoscales, supports event-time windowing, watermarks and triggers, and provides exactly-once semantics for stateful transforms inside the pipeline. Dataproc is for Spark/Hadoop workloads you already own or when you need a specific ecosystem library; Dataproc Serverless removes cluster management but still bills per workload. Cloud Data Fusion offers a codeless UI for ETL-oriented teams. Cloud Composer orchestrates all of the above but is a control plane, not a data plane.
Traps
- BigQuery is an analytics warehouse, not an OLTP store; single-row lookups belong in Bigtable, transactional writes in Cloud SQL or Spanner.
- Pub/Sub delivery is at-least-once, so a pipeline is not end-to-end exactly-once unless the sink is idempotent or you deduplicate on message ID.
- Serverless does not mean free: a streaming Dataflow job bills for minimum workers even when idle.
- Denormalize for BigQuery; repeated joins across huge tables are the usual cost culprit.
Rule of thumb
If the answer can take five minutes to appear, batch it. If a human or system must react to an individual event, stream it.