I85PREP
⌘ K
Cloudera Data Operator study materials
Cloudera Data Operator study resources

Cloudera Data Operator: Data in Motion with CDF and CSP

Cloudera Data Operator · Work through your exam, one topic at a time.

All guide sections

Data in Motion with CDF and CSP

The Data in Motion domain tests streaming ingestion, flow orchestration, and stateful stream processing with Cloudera DataFlow (CDF) and Cloudera Stream Processing (CSP). Expect questions on NiFi versus Kafka versus Flink roles, edge-to-cloud patterns, delivery semantics, and schema handling.

Core concepts

  • CDF is built on Apache NiFi for flow-based data movement. It supports Flow Management (centralized NiFi clusters), NiFi Registry for versioned flows, and MiNiFi agents at the edge. Use CDF for ingestion, routing, transformation, and protocol mediation.
  • Kafka is the durable, distributed log for publish-subscribe. It decouples producers and consumers. Schema Registry manages Avro, JSON, or Protobuf schemas and compatibility rules.
  • CSP includes Apache Flink for stateful stream processing and SQL Stream Builder (SSB) for SQL over streams. Flink provides event time, watermarks, windows, checkpointing, and exactly-once state.
  • Delivery semantics: NiFi is typically at-least-once; exactly-once requires transactional guarantees and careful design. Kafka can provide exactly-once with idempotent producers and transactions.
  • Edge pattern: MiNiFi collects and filters at the edge; CDF routes to Kafka or cloud storage; CSP enriches and aggregates; the lakehouse stores refined results.

Worked example

Factory sensors emit 10,000 events per second; you need sub-second alerts and long-term storage. Use MiNiFi at the edge to filter noise; CDF/NiFi to route to Kafka with Schema Registry; Flink (CSP) to compute rolling averages with event-time windows and emit alerts; sink refined events to Iceberg in the lakehouse. Reasoning: NiFi handles edge ingestion; Kafka buffers and decouples; Flink provides stateful, low-latency processing; Iceberg gives ACID storage for downstream BI.

Traps

  • Using Kafka as a long-term archive; it is a log, not a lakehouse.
  • Assuming NiFi guarantees exactly-once by default; it is typically at-least-once.
  • Forgetting Schema Registry compatibility rules; changing schemas can break consumers.
  • Ignoring event time and watermarks; processing-time windows give different results for late data.
  • Treating SQL Stream Builder as batch SQL; it runs continuous queries over unbounded streams.
  • Overlooking backpressure and checkpoint storage in Flink.