Cloudera CDP-3002 cheat sheet

CDP has two deployment models: Public Cloud (CDP Control Plane hosted by Cloudera, environments in AWS/Azure/GCP) and Private Cloud Base (Cloudera Manager manages the cluster, Data Hub, and Machine Learning workspaces).

In Private Cloud Base, the management hierarchy is Cloudera Manager -> Clusters (services/roles) -> Data Hub clusters (created from Cluster Templates) -> Machine Learning workspaces.

A Data Hub cluster template fixes the services, instance types, and node counts; you cannot change service composition after provisioning, so a new template and cluster are required for service changes.

CDP Runtime versions run from 7.1.x upward; the base cluster and Data Hub clusters can run different runtime versions, and Cloudera Manager upgrades the base cluster before Data Hubs.

Service role placement: NameNode, ResourceManager, and JobHistory Server are typically on master nodes; DataNodes and NodeManagers on worker nodes; edge nodes host gateway roles only.

Cloudera Manager parcel lifecycle: download -> distribute -> activate -> (optionally) deactivate old parcel -> delete; services must be restarted after activating a new parcel.

The Cloudera Manager API is reachable at http(s)://:7183/api//... and supports cluster, service, role, and command operations used for automation.

Resource management uses YARN queues configured via capacity scheduler or fair scheduler; per-queue capacities, maximum capacities, and ACLs are defined in the scheduler configuration and require a YARN refresh.

Hive metastore (HMS) is the shared catalog; Impala, Hive, and Spark all read table definitions from it, so DDL changes made in one engine are visible to the others after cache refresh.

Iceberg tables in CDP are created with CREATE TABLE ... STORED BY ICEBERG, support ACID, time travel (AS OF / VERSION AS OF), and hidden partitioning; Impala and Spark can both query them.

Impala requires REFRESH

or INVALIDATE METADATA after external DDL; the catalogd service propagates metadata to impalad automatically when catalogd is enabled.

Impala join strategies: broadcast joins for small build sides, partitioned joins for evenly distributed large-large joins, and shuffle joins when statistics are missing; COMPUTE STATS is required for good plans.

Spark on YARN runs in client or cluster deploy mode; in cluster mode the ApplicationMaster runs the driver, so the client can disconnect after submission.

Hive ACID requires transactional tables (ORC format, bucketed, transactional=true); INSERT/UPDATE/DELETE/MERGE are supported, but only Hive and certain engines can read them.

Partitioning in Hive uses PARTITIONED BY columns stored as directory names; MSCK REPAIR TABLE or ALTER TABLE ADD PARTITION registers existing HDFS directories in the metastore.

Oozie workflows consist of an XML application definition plus job.properties; coordinator jobs schedule workflows on a time and data dependency basis.

Cloudera Manager auto-TLS provisions an internal CA; Kerberos (via MIT KDC or AD) is required for a secure cluster, and enabling Kerberos is a prerequisite for HDFS transparent encryption and Sentry/Ranger authorization.

Ranger policies are evaluated as allow or deny with a deny-override or allow-override evaluation order configured at the service level; a resource not matched by any policy is denied by default.

In CDP Private Cloud Base, Ranger replaces Sentry for Hive/Impala authorization; the Hive service definition in Ranger governs both engines via the shared metastore.

HDFS transparent data encryption uses Key Management Server (KMS) keys and encryption zones; files inside a zone cannot be moved out and must be copied.

Atlas provides metadata lineage and tagging; Ranger tag-based policies reference Atlas classifications, so tag synchronization between Atlas and Ranger must be enabled.

Hue authorization uses Hadoop SQL policies for Hive/Impala and Ranger policies for HDFS; the logged-in user's group memberships determine the effective policy set.

TLS for the CDP Control Plane uses certificates managed by the control plane; Data Lake and Data Hub clusters in public cloud inherit the environment's TLS configuration.

Audit logs: Navigator (legacy) or Ranger audit stores events in HDFS, Solr, or a database; Cloudera Manager's audit log captures administrative actions on services and roles.

Cloudera Manager health tests run periodically per role; a red health icon indicates a failed health test, while a yellow icon indicates a warning or a running command.

Common Impala failure: 'out of memory' on impalad -> raise mem_limit or reduce query concurrency; 'metadata not found' -> INVALIDATE METADATA; 'table not found after refresh' -> check HMS connectivity.

YARN container allocation failures: 'Container is running beyond physical memory limits' means the container exceeded yarn.nodemanager.resource.memory-mb for the process; adjust the container size or the NodeManager memory.

HDFS safe mode blocks writes; use hdfs dfsadmin -safemode get/leave and verify block reports and NameNode heap before leaving safe mode.

The Cloudera Manager diagnostic bundle (Support -> Diagnostic Bundle) collects logs, configs, and metrics for support cases; it is the standard artifact for escalation.

Cloudera Manager charts show host and service metrics; the 'Activities' page lists running and completed commands, and a failed command's details show the stderr of the role.

Kerberos ticket issues appear as 'GSS initiate failed' or 'No valid credentials'; check the keytab, the ticket lifetime, and clock skew between hosts (allow <= 5 minutes).

Backup and disaster recovery: HDFS snapshots for directory-level recovery, distcp for cross-cluster copy, and the metastore database must be backed up separately from HDFS data.