CDESS flashcards
CDESS · Build recall with a focused review session.
32 cards. Read the term and the definition here. The deck at the top of this page is optional.
- Three Vs
- Volume, velocity, and variety describe data scale, arrival rate, and schema heterogeneity. Each drives a different ingestion pattern and storage decision in pipeline architecture.
- Idempotent write
- A write that produces the same final state when replayed. Achieved with deterministic business keys, MERGE/upsert logic, or deduplication on a processing timestamp.
- SCD Type 2
- A dimension modeling pattern that preserves history by inserting a new row per change, with surrogate key, effective dates, and a current-record flag.
- Star schema
- A model with a central fact table joined to denormalized dimension tables. Fewer joins and simpler queries than snowflake, at the cost of some redundancy.
- Partition pruning
- A query optimization where filters on partition columns let the engine skip irrelevant files or partitions, reducing scanned data and runtime.
- Data quality assertion
- A testable rule such as null-rate threshold, uniqueness, referential integrity, or freshness bound that gates promotion of data between pipeline layers.
- Surrogate key
- A system-generated identifier assigned to a dimension row. It is independent of the natural business key and enables history tracking across changes.
- Late-arriving data
- Records that reach the pipeline after their event-time window closed. Handled by reprocessing, allowed lateness, or routing to a side output.
- Kafka partition
- The unit of parallelism and ordering in a topic. Each partition is consumed by at most one consumer per group, and order holds only within a partition.
- Consumer lag
- The gap between the log-end offset and a consumer group's committed offset. Rising lag signals consumer throughput below producer throughput.
- min.insync.replicas
- A topic setting that sets the minimum replicas that must acknowledge a write. Combined with acks=all, it controls durability under broker failure.
- Log compaction
- A retention mode that keeps only the latest record per key rather than deleting by age. Suited to changelog and state topics needing full key recovery.
- Watermark
- A stream-processing marker estimating event-time progress. It determines when windows fire and when late records are dropped or emitted separately.
- Tumbling window
- A fixed-size, non-overlapping time window. Every event belongs to exactly one window, making it the simplest aggregation boundary.
- Exactly-once semantics
- End-to-end guarantee that each record affects results once. Requires idempotent producers, transactional commits, and read_committed consumers.
- Consumer group rebalance
- The process of reassigning partitions among consumers when membership changes. During rebalance, consumption pauses until assignment completes.
- Checkpointed state store
- Processor state persisted to durable storage so a failed task can rebuild state from the changelog and resume without data loss.
- Iceberg snapshot
- An immutable table state recorded in metadata. Each commit creates a new snapshot with a parent pointer, enabling time travel and rollback.
- Manifest list
- An Iceberg metadata file listing manifest files for a snapshot. Query planning reads it to prune manifests before opening data files.
- Hidden partitioning
- Iceberg derives partitions from source columns via transforms and does not expose them as separate columns, so queries prune automatically.
- Partition evolution
- Changing an Iceberg table's partition spec without rewriting data. Existing files retain their original spec and are planned accordingly.
- Copy-on-write
- An Iceberg update strategy that rewrites affected data files on change. It favors read-heavy workloads because readers see no delete files.
- Merge-on-read
- An Iceberg update strategy that writes delete files applied during reads. It favors write-heavy workloads but adds read-time overhead.
- Snapshot expiration
- Removing old Iceberg snapshots from metadata. It does not delete data files still referenced by retained snapshots; orphan removal is separate.
- Iceberg catalog
- The component that tracks the current metadata pointer and coordinates atomic commits using compare-and-swap, preventing silent concurrent write conflicts.
- Schema evolution by ID
- Iceberg tracks columns by unique ID, so adding, dropping, renaming, or reordering columns does not break reads of historical data files.
- SDX
- Cloudera's Shared Data Experience provides centralized metastore, Ranger policies, Atlas lineage, and credential management across clusters and workloads.
- Apache Ranger
- A centralized authorization framework that evaluates per-resource allow and deny policies for services such as HDFS, Hive, HBase, Kafka, and YARN.
- Ranger deny precedence
- In Ranger policy evaluation, an explicit deny overrides any allow, and resource-specific policies take precedence over wildcard policies.
- Kerberos authentication
- A ticket-based authentication protocol used for service-to-service and user authentication in Cloudera Private Cloud Base clusters, separate from IdP login.
- HDFS encryption zone
- A directory whose data is transparently encrypted at rest using keys managed by the KMS. Access requires both HDFS and KMS permissions.
- Atlas lineage
- Cloudera's metadata and lineage service. It records dataset provenance and supplies classifications used by tag-based Ranger masking and filtering policies.