CDESS cheat sheet

Three Vs define pipeline design pressure: volume (scale of stored records), velocity (arrival rate), and variety (schema heterogeneity); each maps to a distinct ingestion and storage pattern.

Batch vs streaming decision rule: if downstream consumers tolerate latency equal to the scheduled interval, use batch; if latency must be below the smallest business decision interval, use streaming.

Idempotency rule: every write path must be replayable without duplicating rows - key on a deterministic business key plus processing timestamp, or use MERGE/upsert semantics.

Slowly changing dimensions: Type 1 overwrites history; Type 2 adds a new row with surrogate key, effective_start, effective_end, and current_flag; Type 3 stores prior value in a column.

Star schema vs snowflake: star keeps denormalized dimensions for fewer joins; snowflake normalizes dimension hierarchies to reduce redundancy at the cost of join depth.

Partitioning rule: partition on low-cardinality, frequently filtered columns (date, region); avoid partitioning on high-cardinality IDs because it creates small files and metadata pressure.

Exactly-once end to end requires idempotent sinks plus checkpointed sources; at-least-once delivery plus deduplication is the practical fallback when sinks cannot guarantee atomicity.

Data quality gates should be expressed as assertions (null rate, uniqueness, referential integrity, freshness) that block promotion between bronze/silver/gold layers.

Kafka topic partition count sets maximum consumer parallelism within a group; one partition is consumed by at most one consumer in that group at a time.

Ordering is guaranteed only within a partition; to preserve per-key order, set the message key so all related events hash to the same partition.

Delivery semantics: at-most-once (commit before processing), at-least-once (commit after processing), exactly-once via transactions with idempotent producer and read_committed consumers.

Consumer lag = log-end offset minus committed consumer offset; sustained growth indicates consumer throughput below producer throughput, not necessarily a broker fault.

Replication factor and min.insync.replicas together determine durability: with RF=3 and min.insync.replicas=2, writes survive one broker loss while acks=all is enforced.

Compaction retains the latest record per key and is appropriate for changelog/state topics; delete retention is appropriate for event streams with time-bounded replay needs.

Windowing choices: tumbling (fixed, non-overlapping), sliding (fixed size, overlapping by step), session (gap-bounded); watermarks define when late data is dropped or side-output.

State stores in stream processors must be checkpointed to durable storage; recovery replays the changelog to rebuild state before processing resumes.

Iceberg table layout: metadata file → manifest list → manifest files → data files; snapshots are immutable and each commit creates a new snapshot with a parent pointer.

ACID in Iceberg comes from atomic metadata swaps; readers resolve the current snapshot from the catalog, so writers never mutate data files in place.

Hidden partitioning: partitions are derived from a source column via transforms (identity, bucket[N], truncate[W], year/month/day/hour) and are not exposed as table columns in queries.

Partition evolution allows changing the partition spec without rewriting existing data; old files keep their original spec, and planning prunes per spec.

Copy-on-write rewrites whole data files on update/delete; merge-on-read writes delete files applied at read time - COW favors read-heavy, MOR favors write-heavy workloads.

Snapshot expiration and orphan file removal are separate maintenance operations; expiring snapshots does not delete files still referenced by retained snapshots.

Time travel queries a snapshot by ID or timestamp; rollback sets the table's current snapshot pointer back to a prior snapshot without rewriting data.

Catalogs coordinate commits (Hive Metastore, REST, JDBC, Nessie); the catalog enforces the compare-and-swap that prevents concurrent writers from conflicting silently.

Schema evolution supports safe add, drop, rename, reorder, and widen (int→long, float→double); Iceberg tracks columns by ID, so renames do not break historical files.

CDP deployment models: Public Cloud (control plane + environment-specific data plane) and Private Cloud Base (on-premises, managed via Cloudera Manager); both share the SDX governance layer.

Authentication is delegated to an external identity provider via Knox or the control plane; Kerberos is used for service-to-service authentication in Private Cloud Base clusters.

Authorization chain: coarse-grained access at the compute cluster/service level, then Ranger policies for HDFS, Hive, HBase, Kafka, and YARN; Ranger plugins evaluate per-resource allow/deny with audit.

Apache Ranger policy evaluation order: explicit deny wins over allow; resource-specific policies take precedence over wildcard policies; tag-based policies apply through Atlas classifications.

SDX (Shared Data Experience) provides centralized metastore, Ranger policies, Atlas lineage, and credential management so workloads across clusters inherit consistent governance.

HDFS transparent encryption uses encryption zones plus key management (KMS); data at rest in object storage relies on server-side encryption configured at the storage layer.

Resource management: YARN queues with capacity/fair scheduling and cgroup-based isolation limit runaway workloads; per-queue ACLs control who can submit.

Audit and lineage: Ranger audit records policy decisions, Atlas captures dataset lineage; both are prerequisites for data-masking and row-filter policy validation.

Backup and disaster recovery: Cloudera Manager manages service-level backups; replication (HDFS/Snapshot, Hive Replication, or object storage versioning) defines the RPO for cross-cluster recovery.