Cloudera Data Operator cheat sheet

Cloudera Runtime component roles are assigned to hosts by Cloudera Manager; a role is a single instance of a service component (for example, one DataNode or one NameNode), and a host can run many roles.

Cloudera Manager's key ports: 7180 for the admin web UI, 7182 for the Agent-to-Server heartbeat/command channel (TLS), and 7183 for the embedded PostgreSQL database used by the management server.

Cloudera Manager agents poll the server over a heartbeat; the default agent heartbeat interval is 15 seconds, and a host is marked 'bad' or 'stale' when heartbeats stop arriving for the configured threshold.

A parcel is a binary distribution of Cloudera Runtime that is downloaded, distributed, and activated on hosts; activation is required before the parcel's services can start, and multiple parcels can coexist on disk.

Configuration precedence in Cloudera Manager: service-level safety-valve / advanced config overrides role-group config, which overrides role-level config; host-level overrides apply last for host-specific values.

Kerberos-enabling a cluster requires an MIT KDC or Active Directory, a keytab per service, and a Cloudera Manager principal; after enabling, HDFS, YARN, Hive, and Impala must be restarted so they pick up their keytabs.

HDFS replication factor is per-path (default 3); NameNode HA uses an active/standby pair with JournalNodes (odd number, minimum 3) providing the shared edit log, and automatic failover relies on ZooKeeper and the ZKFC process.

YARN capacity-scheduler queues are hierarchical and configured with capacity, maximum-capacity, and user-limit-factor; YARN memory is allocated in units of yarn.scheduler.minimum-allocation-mb (default 1024 MB).

Cloudera Manager's 'Host Inspector' and 'Cluster Validation' checks report version mismatches, missing Kerberos principals, and network/DNS problems before a service start is attempted.

Cloudera's enterprise AI stack layers model serving and orchestration on top of CDP: CML (Cloudera Machine Learning) provides workspaces, sessions, jobs, and model deployment endpoints inside the cluster.

A CML project is backed by a Git repository; a 'session' is an interactive compute pod, a 'job' is a scheduled or on-demand run, and an 'application' exposes a long-running web endpoint.

CML model deployments expose REST endpoints and support 'model governance' metadata; an experiment is a tracked run with parameters, metrics, and artifacts logged for comparison.

GPU-backed sessions require GPU-enabled node pools; the runtime image (engine) determines available Python/R libraries, and custom engines are built as container images pushed to the cluster's registry.

Model risk controls in regulated deployments: lineage from training data to model version, approval gates before promotion to production, and drift monitoring on live inference traffic.

Inference patterns: batch scoring runs as a scheduled job against a table; real-time serving runs behind a REST endpoint; streaming inference consumes Kafka topics with a stateful engine such as Flink.

Data for AI must be governed: row-level and column-level masking, tag-based policies, and audit logs apply to training data just as they do to BI queries.

RAG (retrieval-augmented generation) pipelines combine an embedding model, a vector index, and a generative LLM; the vector store can be a purpose-built index or a table with approximate nearest-neighbor search.

Fine-tuning and prompt engineering are complementary: fine-tuning changes weights with labeled examples, while prompt engineering and RAG change inputs at inference time without retraining.

Apache Kafka topics are partitioned and replicated; each partition is an ordered, immutable log, and ordering is guaranteed only within a partition, not across a topic.

Consumer group semantics: each partition is assigned to at most one consumer in a group, so maximum parallelism equals the partition count; adding consumers beyond that leaves them idle.

Kafka retention is time-based (retention.ms) or size-based (retention.bytes) per topic; log compaction (cleanup.policy=compact) retains the latest value per key instead of time-based deletion.

Producer durability is governed by acks: acks=0 (fire and forget), acks=1 (leader only), acks=all (all in-sync replicas); with acks=all, min.insync.replicas controls how many replicas must acknowledge.

Exactly-once semantics in Kafka Streams require processing.guarantee=exactly_once_v2 plus a transactional producer and a consumer configured with isolation.level=read_committed.

Kafka Connect runs source and sink connectors as workers; a connector's tasks are the unit of parallelism, and converters (Avro, JSON Schema, Protobuf) determine the serialization format written to topics.

Schema Registry enforces compatibility modes: BACKWARD (new schema can read old data), FORWARD, FULL, and their transitive variants; the subject is the scope of the compatibility check.

Kafka MirrorMaker 2 replicates topics between clusters using the source cluster's consumer group and the target cluster's producer; it preserves offsets in checkpoints but not consumer group offsets by default.

Flink and Kafka Streams are the common stream processors on CDP: Flink provides event-time windows and exactly-once checkpoints, Kafka Streams provides a library embedded in a JVM application.

Apache Iceberg tables store data as Parquet files plus a metadata tree: metadata.json, manifest lists, and manifest files; snapshots are immutable and time travel reads a prior snapshot by ID or timestamp.

Iceberg hidden partitioning means queries filter on source columns without naming partition columns; partition transforms include identity, bucket[N], truncate[W], year, month, day, and hour.

Iceberg supports schema evolution (add, drop, rename, reorder columns) and partition evolution without rewriting existing data files; each snapshot records its own partition spec.

Apache Hive ACID tables require the table to be transactional (ORC format, bucketed) and use delta files plus compaction; insert-only vs. full ACID determines whether updates and deletes are allowed.

Delta Lake uses a transaction log (_delta_log) of JSON commits plus Parquet checkpoints; OPTIMIZE compacts small files and Z-ORDER co-locates related data for data-skipping.

Apache Kudu provides a columnar store with primary-key upserts and scans that combine with Impala for update-heavy analytical workloads; it complements HDFS/object storage rather than replacing it.

Small-file problems hurt lakehouse query performance: compaction (Iceberg rewrite_data_files, Delta OPTIMIZE, Hive compaction) reduces file count and metadata overhead.

Table formats differ on concurrency: Iceberg uses optimistic concurrency with a metadata pointer swap; Hive ACID uses locks; Delta uses optimistic concurrency with a log-based conflict check.

Object storage (S3, Ozone, ADLS) is the default lakehouse substrate; the catalog (Hive Metastore, Iceberg REST catalog, Nessie) determines how clients discover table metadata.

A data fabric virtualizes access across sources: a query engine (Impala, Hive, Trino-style federation) reads external tables registered in the metastore without copying data.

Apache Atlas provides metadata and lineage; classifications (tags) propagate through lineage, and tag-based policies in Ranger enforce masking and row filtering at query time.

Apache Ranger policies are evaluated per resource (database, table, column, URL) and per user/group; deny policies take precedence over allow policies.

External tables point at a location and drop only metadata, while managed tables drop both metadata and data; the distinction drives data lifecycle and backup design.

Data lineage in Atlas is captured from Hive/Impala query hooks and Spark jobs; lineage edges connect input datasets to output datasets, enabling impact analysis before schema changes.

Federation trade-offs: virtualization avoids duplication and staleness but pushes predicates to remote sources; materialization improves latency at the cost of refresh jobs and storage.

A data mesh organizes ownership by domain with data products as the contract; a fabric supplies the shared governance, catalog, and access layer that makes those products discoverable and usable.

Row-level filtering and column masking in Ranger are applied by the query engine at read time, so the underlying files remain unchanged and access is consistent across tools.