CDESS key terms

An open specification for large analytic tables that tracks files, snapshots, and schema in metadata rather than in a directory listing, so multiple engines can read and write the same table safely and consistently.

A complete, immutable view of an Iceberg table at one point in time, listing the set of data files that made up the table then. Queries read a single snapshot, so concurrent writes never disturb a running read.

An Iceberg metadata file that lists data files together with per-file statistics such as record counts and column-level min/max values. Query planners use these statistics to prune files without opening them.

An Iceberg metadata file that names the manifest files belonging to one snapshot, along with partition summaries that let a reader skip whole manifests. It is the entry point from a snapshot to the actual data files.

The JSON document at the root of an Iceberg table's metadata tree. It records the table schema, partition spec, sort order, properties, and the current snapshot pointer that readers consult first.

The service or component that maps a table name to the location of its current metadata file. It provides the atomic pointer swap that makes a new Iceberg snapshot visible to readers.

The Iceberg write protocol step in which a writer conditionally replaces the table's metadata pointer only if no other commit happened first. Conflicts are detected and retried rather than silently overwritten.

A concurrency model in which each query sees a stable, consistent snapshot of the table taken when it started, while writers create new snapshots independently. Readers and writers therefore never block each other.

Querying an Iceberg table as of an earlier snapshot ID or timestamp. It supports reproducing past reports, auditing changes, and recovering from a bad write without restoring from backup.

An Iceberg feature where partition values are derived from a source column by a transform, so queries filter on the source column and the engine prunes partitions automatically without the user naming partition columns.

A rule that maps a source column value to a partition value, such as identity, bucket, truncate, year, month, day, or hour. Transforms are stored in table metadata and applied consistently by every engine.

Changing a table's partition scheme without rewriting existing data files. Old files keep their original layout, and new writes follow the new spec, so the table can adapt as data volume or query patterns change.

Skipping data files whose partition values cannot match a query's filter. Combined with file-level statistics, it reduces I/O dramatically on large tables.

An Iceberg write mode where an update or delete rewrites entire affected data files into new files. Reads stay cheap because no delete files must be merged, but write amplification is higher.

An Iceberg write mode where updates and deletes are recorded in separate delete files and applied when the data is read. Writes are fast, but readers pay extra cost until compaction removes the deletes.

An Iceberg file that records rows to be treated as removed, either by position or by equality predicate, without rewriting the underlying data files. It is merged at read time in merge-on-read tables.

A maintenance job that rewrites many small or delete-heavy files into fewer larger files and applies pending deletes, restoring read performance and reducing metadata overhead on an Iceberg table.

A maintenance operation that removes old snapshots and the data files no longer referenced by any retained snapshot. It reclaims storage while preserving the snapshots still needed for time travel or rollback.

The performance degradation that occurs when a table contains a very large number of tiny files, inflating metadata, listing, and open costs. Compaction and better write batching are the usual remedies.

A relational-backed service that stores table and partition definitions for the Hadoop ecosystem. In CDP it commonly acts as the catalog that Iceberg and other table formats use to locate current metadata.

A SQL-on-Hadoop engine and metadata layer widely used in CDP for batch queries and table management. It can read Iceberg tables through the metastore catalog and is often the migration source for lakehouse moves.

Cloudera's low-latency massively parallel SQL engine for data stored in HDFS and object storage. It runs coordinator and executor daemons and relies on catalog services to find tables.

A distributed in-memory processing engine used in CDP for batch and streaming jobs. Spark SQL and DataFrames are common ways to write Iceberg tables and to run structured streaming pipelines.

A distributed, partitioned, replicated commit log for publishing and subscribing to event streams. Producers append records to topics, and consumers read them at their own pace using offsets.

An ordered, append-only shard of a Kafka topic. Partitions are the unit of parallelism and replication, and ordering is guaranteed only within a single partition, not across a topic.

A monotonically increasing identifier assigned to each record within a Kafka partition. Consumers commit offsets to record how far they have processed, enabling resume and replay.

A set of Kafka consumers that cooperatively divide the partitions of subscribed topics so each partition is processed by exactly one member at a time. Membership changes trigger rebalancing.

The number of copies of each Kafka partition kept on different brokers. It determines how many brokers may fail without data loss, provided in-sync replicas remain available.

A processing guarantee where each input record affects the output state once, even after failures. It typically combines idempotent writes or transactional commits with checkpointed, replayable source offsets.

The Spark API for incremental processing of continuous data as an unbounded table. It tracks progress with checkpoints, supports event-time windows and watermarks, and can write results to sinks such as Iceberg.

Durable state a streaming engine writes to reliable storage so it can recover after failure. It records source offsets, aggregation state, and metadata needed to resume processing without reprocessing everything.

A threshold in event-time stream processing that tells the engine how long to wait for late data. Events older than the watermark are typically dropped or handled separately, bounding state size.

The timestamp when an event actually occurred, embedded in the record, as opposed to the time the system processed it. Correct windowing and late-data handling depend on it.

A flow-based data movement tool for routing, transforming, and buffering data between systems. Flows are built from processors connected by queues, with provenance and back-pressure built in.

A service that stores and versions schemas for streaming records so producers and consumers agree on structure. It enforces compatibility rules and lets consumers deserialize data without out-of-band contracts.

The central administration console for CDP clusters. It deploys and configures services, monitors health, manages roles and hosts, and drives rolling restarts and upgrades across the cluster.

A binary distribution format used by Cloudera Manager to install and upgrade service software on cluster hosts. Parcels are activated and distributed in a controlled order to keep versions consistent.

A named set of configuration values applied to all instances of a service role, such as all DataNodes. It lets administrators manage configuration at scale instead of per host.

A centralized authorization framework that enforces fine-grained access policies on Hadoop and lakehouse components. Policies are defined per resource and audited centrally, replacing scattered file permissions.

A network authentication protocol based on tickets issued by a trusted key distribution center. CDP clusters use it so users and services prove identity without sending passwords over the network.

A deployment pattern with redundant service instances and automatic failover, so the loss of one host does not stop the service. It usually requires shared state such as a database or quorum.