Design Data Processing Systems
Frame the decision
Every design question reduces to a few axes: batch or streaming, latency SLA, whether the transformation is mostly SQL or mostly code, and whether you are migrating an existing Hadoop/Spark estate. Dataflow (Apache Beam) is the answer for stateful, event-time streaming and unified batch/streaming pipelines. Dataproc or Dataproc Serverless fits when lifting Spark/Hive/Hadoop jobs, using Spark ML libraries, or needing fine cluster control. Cloud Data Fusion is the GUI-driven, connector-rich choice for low-code pipelines with lineage built in. BigQuery itself handles scheduled SQL, ELT, and simple loads.
Streaming correctness
Know the vocabulary: event time versus processing time, watermarks, windowing (fixed, sliding, session), triggers, accumulation modes, and allowed lateness. Late data needs allowed lateness plus a deliberate trigger strategy. Design for idempotency: exactly-once holds inside the pipeline, but sinks need deterministic keys or MERGE-based upserts to tolerate replays. Use Streaming Engine and autoscaling rather than hand-sizing workers.
Traps
Pub/Sub is a queue with a bounded replay window, not a database - questions that hinge on replaying months of history are answered by GCS or BigQuery. Choosing a high-cardinality partition key is always a wrong answer. Dataflow does not deliver exactly-once into a non-idempotent external system, and a bigger worker does not fix a poison record.
Rule of thumb: if the transform is expressible in SQL and the source already lives in BigQuery or GCS, do it in BigQuery; if you need per-key state and event-time semantics, use Beam on Dataflow.