I85PREP
⌘ K
Cloudera CDP Data Analyst study materials
Cloudera CDP Data Analyst study resources

Cloudera CDP Data Analyst: Apache Spark and Cloudera Data Engineering for Analysts

Cloudera CDP Data Analyst · Work through your exam, one topic at a time.

All guide sections

Apache Spark and Cloudera Data Engineering for Analysts

This domain tests Spark's execution model and how Cloudera Data Engineering (CDE) packages Spark work for production: lazy evaluation, shuffles, join strategies, file sizing, and the job/session lifecycle.

Execution model points that appear on exams

  • Transformations (select, filter, join) are lazy; nothing runs until an action (count, write, show, collect).
  • Narrow transformations avoid data movement; wide ones (groupBy, join, distribute by, repartition) force a shuffle governed by spark.sql.shuffle.partitions (default 200).
  • repartition(n) does a full shuffle and can increase parallelism; coalesce(n) only merges partitions and cannot increase it.
  • Adaptive Query Execution (spark.sql.adaptive.enabled) coalesces small shuffle partitions and can convert sort-merge joins to broadcast joins at runtime.
  • Broadcast joins apply under spark.sql.autoBroadcastJoinThreshold (default 10 MB); force with /*+ BROADCAST(t) */ when statistics are missing.
  • Spark SQL reads Hive Metastore tables, so the same COMPUTE STATS/ANALYZE TABLE work improves its join choices; cached DataFrames go stale after underlying writes.

CDE specifics

  • Virtual clusters provide isolated Spark compute; jobs are submitted and can be scheduled, while sessions are interactive.
  • Jobs are created and run through the CDE CLI/API (cde job create, cde job run) or orchestrated externally, e.g. Airflow operators.
  • Resources (driver/executor cores and memory) are declared per job or per resource definition, so tuning is a job configuration change, not a cluster restart.

Worked example

A daily job writes 20,000 files of 2 MB each into a Hive partition. Impala queries then take minutes opening files. Reasoning: the default shuffle partition count was far too high for the output volume. Fix: before writing, df.repartition(col("dt")) (or reduce shuffle partitions) and cap maxRecordsPerFile so each file lands near 128–256 MB, then refresh Impala metadata. The same job's 3-hour runtime from collect() on a billion rows is fixed by aggregating in Spark and writing results.

Traps

  • Calling collect() or toPandas() on a large DataFrame - driver OOM.
  • Using a Python UDF where a built-in function exists; UDFs break Catalyst optimization and codegen.
  • Caching a DataFrame and expecting it to reflect later writes.
  • Assuming AQE fixes every skew problem - extreme key skew may still need salting.
  • Raising executor memory first when the real problem is too many shuffle partitions or a missing broadcast.