I85PREP
⌘ K
CDESS study materials
CDESS study resources

CDESS: Open Data Lakehouse and Apache Iceberg: Format, Evolution, and Maintenance

CDESS · Work through your exam, one topic at a time.

All guide sections

Open Data Lakehouse and Apache Iceberg: Format, Evolution, and Maintenance

Open Data Lakehouse and Apache Iceberg

This domain tests the table format that lets Spark, Impala, Flink, and Trino share one copy of data on object storage with ACID behavior, schema evolution, and time travel.

Anatomy and commits

An Iceberg table is a metadata file pointing to a snapshot. Each snapshot references a manifest list, manifests, and data files. A commit writes new metadata and swaps the current metadata pointer atomically, which provides isolation. Writers use optimistic concurrency, and a conflicting commit is retried.

Features the exam probes

  • Hidden partitioning: partitions derive from a transform such as days(ts) or bucket(16, id); queries filter on the source column and pruning happens automatically.
  • Partition evolution: a new partition spec applies to new writes; old files keep their spec and no rewrite is required.
  • Schema evolution: columns are tracked by field ID, so add, drop, rename, and reorder are metadata-only. Type widening is supported; arbitrary type changes are not.
  • Row-level deletes: copy-on-write rewrites affected files; merge-on-read writes delete files applied at read time, which is cheaper to write and slower to read.
  • Maintenance: rewrite data files for compaction, rewrite manifests, and expire snapshots.

Worked example

A streaming job commits every minute, producing thousands of small files, and queries slow down. The correct response is to run data-file compaction and manifest rewriting, then set a snapshot expiration policy. Reasoning: small files inflate manifest and file-open overhead; compaction merges them, and manifest rewriting keeps query planning fast. Expiring snapshots trims metadata, but it also removes the ability to time-travel to those snapshots, so retention must match governance requirements.

Traps

  • Believing partition evolution rewrites existing data; readers must handle multiple partition specs.
  • Assuming Iceberg requires the Hive Metastore; REST, Hadoop, and Nessie catalogs are also valid.
  • Forgetting that merge-on-read tables apply delete files at read time, which affects engine support and performance.
  • Expiring snapshots without checking time-travel or audit requirements.