I85PREP
⌘ K
CDESS study materials
CDESS study resources

CDESS: Data in Motion / Streaming: Kafka, Schema Registry, and NiFi

CDESS · Work through your exam, one topic at a time.

All guide sections

Data in Motion / Streaming: Kafka, Schema Registry, and NiFi

Data in Motion / Streaming

This domain covers ingestion and event transport in CDP: Apache Kafka for durable logs, Streams Messaging Manager (SMM) and Schema Registry for operations and contracts, Streams Replication Manager (SRM) for cross-cluster replication, and Apache NiFi (Cloudera Flow Management) for flow-based ingestion.

Kafka rules that drive exam answers

  • Ordering is guaranteed only within a partition. The producer key determines the partition, so choose a key matching the ordering unit, such as account ID.
  • Durability requires replication.factor of at least 3, min.insync.replicas of at least 2, and acks=all on the producer; acks=1 can lose records when a leader fails.
  • Idempotent producers plus transactions give exactly-once semantics for Kafka-to-Kafka pipelines; consumers still need deduplication when writing to non-transactional sinks.
  • Retention deletes by time or size; compaction retains the latest record per key. Compaction is not a substitute for retention when history must be replayed.
  • A consumer group assigns each partition to at most one consumer; consumers beyond the partition count stay idle.

NiFi and schema

NiFi flows are directed graphs of processors. Connections carry backpressure thresholds for object count and data size plus prioritizers. Record-oriented processors rely on controller services such as readers and writers, and provenance records lineage.

Worked example

A team must publish both the latest customer profile and all profile changes. A single topic with seven-day retention fails because the latest record for an inactive customer is deleted. Correct design: a compacted topic keyed by customer ID for current state, plus a separate topic with time retention for the change history. Reasoning: compaction keeps the last value per key indefinitely, while retention is time-based and cannot guarantee a surviving latest record.

Traps

  • Adding partitions to a keyed topic and assuming per-key order is preserved; the key-to-partition mapping changes.
  • Using more consumers than partitions and expecting higher throughput.
  • Setting acks=all while leaving min.insync.replicas=1, which still permits data loss.
  • Treating Schema Registry compatibility settings as enforcement that prevents all bad writes; compatibility applies at registration time.