Cloudera CDP Administrator - Public Cloud cheat sheet
Cloudera CDP Administrator - Public Cloud · Keep the key ideas within reach.
Platform Provisioning and Cluster Management
CDP Public Cloud control plane runs in the Cloudera cloud account; the workload (data) plane runs inside the customer's VPC/VNet via a cross-account IAM role (AWS), service principal/role assignment (Azure), or service account (GCP).
Environment types: a Data Hub is a workload cluster with a chosen template (e.g., Data Mart, Data Engineering, Streams, Operational Database); a Data Lake is the singleton SDX cluster hosting HMS, Ranger, Atlas, and Knox.
Data Lake creation requires: environment registered, IDBroker/credential mappings, storage location (S3/ADLS/ABFS/GCS), and a database catalog; Data Hub templates reference the Data Lake by CRN.
Cluster definition vs. cluster: definitions are reusable templates; clusters are running instances. Scaling a cluster changes node count for a specific instance group; autoscaling uses load-based policies per group.
Recipes are per-instance-group scripts (pre/post service start phases) executed on hosts; they are not the same as parcel or CSD installation.
CDP CLI (cdp) authenticates with an API access key (access key ID + private key) generated per user in the management console; the CDP control plane API is region-scoped.
Deletion order matters: delete Data Hub clusters before the Data Lake, and delete the environment last; the Data Lake cannot be deleted while registered clusters remain.
Environment registration validates network access (egress to control plane, cross-account role trust) and stores cloud storage credentials used by the Data Lake IDBroker.
Security, Authentication, Authorization, and Governance (SDX)
SDX separates authentication (Knox IDBroker, IDP) from authorization (Ranger) and metadata/audit (Atlas); the Data Lake hosts the SDX services for all attached Data Hubs.
IDBroker maps cloud identities to Hadoop users: it assumes the IDBroker role and issues short-lived STS/credential tokens so workloads never store long-lived cloud keys.
Ranger policies are stored in the Data Lake's relational DB (embedded PostgreSQL for light use, external DB for production) and are evaluated by plugins on each cluster; the plugin polls the Data Lake for policy updates.
Ranger resource hierarchy for HDFS/Hive: database → table → column, and URL/path-based for HDFS; deny policies override allow policies.
Authentication to CDP services uses either local user management or an external IDP via SAML/OIDC; Knox Gateway provides the single ingress (CDP Proxy) for service UIs and REST APIs.
Atlas lineage is captured from Hive/Spark hooks; HDFS and Hive audit events flow to the Data Lake's audit store (Ranger audit → Solr or external).
TLS: the Data Lake and Data Hubs use auto-generated or custom certificates; Knox terminates TLS at the CDP Proxy and requires the cluster's public hostname/FQDN.
Data Lake HA: the Data Lake can run with HA (multiple master nodes) for HMS, Ranger, and Knox; a non-HA Data Lake is a single point of failure for SDX.
Data Services, Storage, and Workload Operations
Storage is decoupled: Hive/Impala tables live on cloud object storage (S3/ADLS/GCS) accessed via the Data Lake's RAZ-governed paths; local HDFS is used for scratch/temp unless configured.
Ranger Authorization (RAZ) enforces path-based access on cloud storage for Hive/Impala; the storage credential is brokered by IDBroker, so the cluster's IAM role must trust IDBroker.
Hive Metastore (HMS) is centralized on the Data Lake; Data Hubs connect to it via the HMS endpoint - multiple clusters share one catalog.
Impala requires the coordinator/executor split; queries read from object storage and the catalog; Impala-specific config (e.g., scratch dirs, admission control) is per-cluster.
Workloads: YARN queues for batch, Livy for Spark job submission, Oozie/Airflow for orchestration; Data Hub templates determine which services are deployed.
Table formats: Hive ACID tables need transactional tables on HDFS/object storage; Iceberg tables use a catalog (HMS or REST) and are queried by Impala/Spark.
Scaling: add worker nodes to an instance group to increase capacity; compute clusters can be stopped/started to save cost while the Data Lake remains running.
Data Hub upgrade: the Data Lake must be upgraded before attached Data Hubs; the runtime version of the Data Lake must be >= the Data Hub runtime.
Monitoring, Observability, Troubleshooting, and Maintenance
Cloudera Observability (formerly Workload Manager/Telemetry) collects metrics and logs from clusters; the Cloudera Manager (on Data Hubs) exposes per-service health and per-role logs.
Diagnostic bundles: Cloudera Manager generates support bundles per cluster; CDP control plane has environment/cluster diagnostics for provisioning failures.
Common provisioning failure: cross-account role trust or network egress blocked → environment registration fails; verify the trust policy and security group/NACL egress.
Service start failure on a Data Hub: check the recipe output and role logs on the failing host; a failed recipe blocks service start.
Data Lake issues (HMS/Ranger/Knox) affect all Data Hubs: if Ranger policies stop applying, check the Ranger plugin's connection to the Data Lake and the audit DB.
IDBroker failures cause cloud storage access errors (403) in Hive/Impala; check the IDBroker role logs and the trust relationship between the cluster role and IDBroker role.
Upgrade/maintenance: use the CDP control plane to upgrade runtime versions; validate with a pre-upgrade check; rollback is limited - upgrade Data Lake first, then Data Hubs.
Cost/health: stop idle compute clusters, monitor instance group utilization, and review autoscaling events; the Data Lake should remain running for SDX availability.