Data Access, Ingestion, and Preparation
What the domain tests
This domain covers getting data from CDP storage and external systems into a form models can use. Questions target HDFS, Ozone, S3, ADLS, Hive and Impala tables, the Hive Metastore, Iceberg, JDBC sources, file formats, schemas, partitioning, and Spark-based preparation. The exam favors governed access through Ranger and lineage through Atlas or Data Catalog over ad hoc copies.
Core rules
- Prefer tables over raw files when possible. Hive, Impala, and Iceberg tables provide schema, partition pruning, and SQL access; raw CSV in object storage does not.
- Use Spark for large-scale ingestion and transformation. Read with explicit schemas, filter early, select only needed columns, and write columnar formats such as Parquet or ORC. Use Iceberg when you need ACID writes, snapshots, and schema evolution.
- Avoid collecting large datasets to the driver. Keep transformations distributed; collect only aggregated or sampled results.
- Partition and bucket for access patterns. Over-partitioning creates small files; under-partitioning forces full scans.
- Check Ranger policies before assuming access. A notebook that works for one user may fail for a service account used by a job.
- Preserve lineage: register outputs in the Metastore or catalog, and document source-to-target mappings.
Worked example
A daily CSV lands in S3 with inconsistent date formats and duplicate order IDs. The engineer reads it with Spark using an explicit schema, parses dates with a fixed format, drops duplicates by order_id and ingestion timestamp, joins to a customer Hive table, and writes an Iceberg table partitioned by order_date. Reasoning: explicit schema avoids inference errors; deduplication before the join prevents row explosion; Iceberg gives snapshot isolation for downstream jobs. Writing Parquet inside Iceberg keeps scans columnar and partition-pruned.
Traps
- Letting Spark infer schemas on production CSV, causing type drift.
- Calling toPandas() on a multi-gigabyte DataFrame.
- Joining before deduplicating, which multiplies rows.
- Writing thousands of tiny partition files.
- Using a personal account for a scheduled ingestion job.
- Ignoring time zones and daylight-saving shifts in date parsing.