I85PREP
⌘ K
Cloudera CDP-3002 study materials
Cloudera CDP-3002 study resources

Cloudera CDP-3002: Cloudera Data Platform Data Engineering

Cloudera CDP-3002 · Work through your exam, one topic at a time.

All guide sections

Cloudera Data Platform Data Engineering

The Data Engineering domain tests how you design Hive and Impala tables, use Spark on YARN, and orchestrate pipelines on CDP. Expect questions on partitioning, ACID tables and compaction, statistics, file formats, and Oozie dependencies.

Table design and formats

  • Hive managed tables own their data; dropping one deletes the data. External tables drop only metadata. In Hive 3, managed tables are transactional by default, so plan for compaction.
  • Full ACID tables use ORC and bucketing. Partitioning prunes data only when queries filter on partition columns; over-partitioning creates small files and NameNode pressure.
  • Parquet and ORC are columnar formats suited to analytics. Text formats are for landing zones, not repeated large scans.

ACID and compaction

  • ACID tables accumulate delta files. Minor compaction merges delta files; major compaction merges the base and delta files. The initiator runs automatically, but you can inspect with SHOW COMPACTIONS and request ALTER TABLE ... COMPACT 'MAJOR'.
  • Compaction is not optional housekeeping: too many delta files slow reads and can exhaust resources.

Impala, Spark, and Oozie

  • Impala uses the catalog service. After Hive DDL, run INVALIDATE METADATA or REFRESH on the affected table. Run COMPUTE STATS for join ordering and distribution; use incremental stats on partitioned tables.
  • Spark on YARN needs executor memory plus overhead. Dynamic allocation, spark.sql.shuffle.partitions, and avoiding collect() to the driver are common tuning points.
  • Oozie workflows define actions in workflow.xml; coordinators add time-based triggers; bundles group coordinators. Nominal time is the scheduled time, not the actual start time.

Worked example

A query on sales filters region='EU', but the table is partitioned by sale_date, so the query scans every partition.

Reasoning: partition pruning applies only to partition columns. Add a sale_date predicate, or repartition by region if that is the dominant filter. Check EXPLAIN to confirm partition pruning, then run COMPUTE STATS if join cardinality estimates are stale.

Traps

  • Assuming DROP TABLE on an external table deletes files; it does not.
  • Assuming Impala immediately sees Hive metadata changes without refresh or invalidation.
  • Running full COMPUTE STATS on a huge table during peak hours.
  • Treating Oozie nominal time as actual execution time when diagnosing delays.