Cloudera CDP Generalist cheat sheet

CDP is delivered in two form factors: CDP Public Cloud (control plane managed by Cloudera on AWS, Azure, or GCP) and CDP Private Cloud (Base and Data Services running on customer-managed Kubernetes/OpenShift).

Core CDP runtime is built on Apache Hadoop 3.x with HDFS, YARN, Hive 3 (Tez execution), and Ranger for centralized authorization; Ozone is available as an HDFS-compatible object store.

CDP Public Cloud experience types: Data Hub (Data Engineering, Data Warehouse, Data Flow, Stream Processing, Custom) and Data Mart (Data Warehouse, Operational Database, Machine Learning). The environment hosts clusters, and the control plane is Cloudera-managed.

Shared Data Experience (SDX) provides a single security and governance layer across clusters: Ranger policies, Atlas metadata/lineage, and Kerberos/LDAP identity, so data access rules follow data across services.

Data Lake (formerly Data Lake cluster) is the shared storage and governance foundation; Data Hub clusters attach to it, inheriting SDX policies and metadata rather than duplicating them.

Cloudera Manager is the operational UI/API for CDP Private Cloud Base: it deploys parcels, manages roles, monitors health, and drives rolling restarts; parcels are versioned binary distributions, not RPM-installed packages.

CDP Private Cloud Data Services runs on Kubernetes (OpenShift or embedded k8s) with compute clusters isolated per workload; the Cloudera Data Platform control plane manages namespaces, environments, and Data Services experiences.

Replication and disaster recovery: Cloudera Replication Manager (part of CDP) replicates HDFS, Hive, and Ozone data between clusters, using Hive Replication for external tables and snapshot-based HDFS replication.

CDP upgrade path from CDH 5/6 and HDP 2/3 uses Cloudera Manager upgrade orchestration plus in-place or export/import migration; HDP-to-CDP migrations typically use distro upgrade tooling and require re-validation of Hive and Ranger policies.

Ranger authorization model: policies attach to resources (database/table/column/URL/HDFS path) and are evaluated by service-specific plugins; row-filter and column-masking policies apply at query time for Hive and Impala.

Atlas is the metadata and lineage service in SDX; it captures entity types (hive_table, hdfs_path, kafka_topic), tracks lineage across Hive/Spark/NiFi, and can propagate classification tags to Ranger for tag-based access control.

Tag-based policies in Ranger use Atlas classifications (for example PII, SENSITIVE); the tag must be assigned to the asset in Atlas before the Ranger tag policy takes effect.

Kerberos provides authentication in CDP; LDAP/Active Directory integration is done via Cloudera Manager or the CDP control plane, and Hadoop services use SPNEGO for web UI authentication.

TLS is configured per service role; Cloudera Manager manages certificates and keystores, and the CDP control plane enforces TLS on public endpoints by default.

Knox Gateway provides a reverse proxy and perimeter security for Hadoop REST APIs and UIs; in CDP Public Cloud, Knox is used for the Data Lake and Data Hub endpoints, with identity federation via JWT.

Auditing: Ranger audits policy evaluations to Solr (default) or HDFS; Cloudera Manager and CDP control plane record administrative actions; audit retention is configured per deployment.

Data encryption: HDFS Transparent Data Encryption (TDE) uses KMS-backed encryption zones; Ozone supports its own encryption keys; at-rest encryption for object storage uses the cloud provider's KMS.

Cloudera Data Warehouse (CDW) provides virtual warehouses backed by Impala or Hive; each Virtual Warehouse is an isolated compute cluster that can be resized and auto-suspended, sharing the same Data Lake storage and SDX policies.

Cloudera Data Engineering (CDE) runs Spark workloads on Kubernetes with virtual clusters; jobs are submitted via the CDE CLI/API, dependencies managed via Airflow or the built-in scheduler, and resources are isolated per virtual cluster.

Cloudera Machine Learning (CML) provides workspaces for data science with Jupyter, RStudio, and ML runtimes; models are deployed as REST endpoints, and projects are backed by Git and governed by SDX.

Cloudera DataFlow (CDF) includes NiFi for flow-based ingestion and Stream Processing (Flink) for stateful stream processing; NiFi flows can be deployed to DataFlow deployments in CDP Public Cloud.

Cloudera Operational Database (COD) is an operational NoSQL service based on Apache HBase and Phoenix, exposed via the CDP control plane with autoscaling and backup/restore.

Cloudera Data Visualization (CDV) connects to CDW, CDE, and CML data sources to build dashboards; it is a separate CDP experience, not part of the core Data Hub runtime.

Workload placement: Data Hub clusters and CDW virtual warehouses are ephemeral compute attached to a persistent Data Lake; storage and governance outlive the compute, which is the key cost and design principle in CDP Public Cloud.

Data Flow deployment sizing: NiFi cluster nodes are sized by flow throughput and back-pressure; the CDP control plane scales deployment nodes, and auto-scaling is available for NiFi and Stream Processing deployments.

Cloudera Replication Manager in CDP Public Cloud replicates between Data Lakes and on-prem clusters; it supports HDFS, Hive, and Ozone policies with scheduled or on-demand runs.

CDP Private Cloud Base uses parcels stored in a local or remote parcel repository; Cloudera Manager distributes and activates parcels across hosts, and rolling restarts minimize downtime during activation.

Host requirements: Cloudera Manager Server, a database (PostgreSQL, MySQL, or Oracle) for CM and for Hive Metastore/Ranger/Oozie, plus a JDK; CDP 7.x requires JDK 8 for most services and supports JDK 11 in later releases.

CDP Private Cloud Data Services requires a Kubernetes cluster (OpenShift 4.x or embedded k8s) with persistent storage (for example Longhorn or cloud block storage) and a load balancer; the control plane namespace hosts the CDP control plane services.

Networking: CDP Public Cloud environments require a VPC/VNet with subnets for the Data Lake and Data Hubs, plus endpoints for the control plane (CDP control plane API) and for object storage (S3, ADLS, GCS).

Identity in CDP Public Cloud: a Cloudera account with an identity provider (Cloudera SSO, or federated SAML/OIDC); machine users and workload users are managed via the control plane and mapped to Ranger/Knox identities.

Credential management: CDP Public Cloud uses cross-account IAM roles (AWS), service principals (Azure), or service accounts (GCP) to access object storage; these are created during environment registration.

Backup: Cloudera Manager's backup/restore covers CM configuration and the CM database; Hive Metastore and Ranger databases require separate database-level backups; Data Lake data lives in object storage or HDFS and is backed up via replication.

Troubleshooting: Cloudera Manager diagnostics bundles collect logs and configs from hosts; in CDP Public Cloud, support bundles are collected per cluster, and control plane logs are accessed via the CDP console.

Capacity planning for CDP Private Cloud Base: size by YARN memory/vcores per host, HDFS raw capacity with replication factor (default 3), and separate disks for DataNode and NodeManager when possible.

Impala is the MPP SQL engine in CDP; it reads Hive Metastore tables and uses its own daemons (impalad, statestored, catalogd). REFRESH/INVALIDATE METADATA propagate DDL changes; COMPUTE STATS gathers table and column statistics for the optimizer.

Hive on Tez executes queries as DAGs; Tez container sizing and the number of reducers are tuned via hive.tez.* and tez.* properties; LLAP provides a persistent daemon for low-latency Hive queries.

YARN scheduling: Capacity Scheduler is the default in CDP; queues have capacity, maximum capacity, and user limits; the Fair Scheduler is available but not the default. Preemption is configured per queue.

HDFS architecture: NameNode (with HA via JournalNodes and ZKFC), DataNodes, and a standby NameNode; block size default is 128 MB, replication default 3; erasure coding is available as an alternative to replication.

Ozone is a scalable object store in CDP with buckets and keys, managed by the Ozone Manager and Storage Container Manager; it provides S3-compatible API and can replace HDFS for some workloads.

Spark on YARN runs in cluster or client mode; dynamic allocation is controlled by spark.dynamicAllocation.*, and the external shuffle service is required for dynamic allocation on YARN.

HBase on CDP uses HDFS or Ozone for storage, RegionServers for serving, and the HBase Master for assignment; Phoenix provides SQL over HBase with secondary indexes.

Kafka in CDP is managed by Cloudera Streams Messaging; topics are partitioned and replicated, with MirrorMaker for cross-cluster replication and Schema Registry for Avro schemas.

Workload sizing rule: one YARN container per vcore by default (yarn.nodemanager.resource.cpu-vcores); memory per container is set by yarn.scheduler.minimum-allocation-mb and maximum-allocation-mb.

Data Lake storage in CDP Public Cloud is object storage (S3, ADLS Gen2, GCS); the Data Lake also hosts the Hive Metastore, Ranger, and Atlas services that Data Hubs and CDW use.

External tables in CDP Public Cloud point to object storage paths; managed tables default to the warehouse directory in the Data Lake. Table location and storage format (Parquet, ORC, Avro) are set at creation.

Iceberg tables are supported in CDP (Impala, Spark, Hive) for ACID, time travel, and schema evolution; the Iceberg catalog can be the Hive Metastore or a REST catalog.

Small-file problem: many small files increase NameNode/object-store overhead; compaction via Hive/Spark jobs, or Iceberg compaction, reduces file count and improves query performance.

Partitioning strategy: partition on low-cardinality, frequently filtered columns; avoid over-partitioning (thousands of partitions) and prefer dynamic partitioning with hive.exec.dynamic.partition.mode=nonstrict.

File formats: Parquet and ORC are columnar and preferred for analytics; Avro is row-based and used for streaming/write-heavy pipelines; text is used only for ingestion or interchange.

Statistics: Impala COMPUTE STATS and Hive ANALYZE TABLE ... COMPUTE STATISTICS feed the cost-based optimizers; stale stats cause bad join orders and broadcast decisions.

Data lifecycle: CDP supports storage policies and Ranger masking/row filters rather than moving data; archival to cheaper object storage tiers is done via cloud lifecycle rules on the bucket.

CDP Public Cloud environments are registered with a cloud provider account and a region; the environment contains a Data Lake and one or more Data Hubs, and all resources are tagged for cost tracking.

Cluster definitions and templates: Data Hub clusters are created from cluster definitions (blueprints); custom definitions can be added, and the template controls instance types, node counts, and volume sizes.

Autoscaling in CDP Public Cloud: Data Hubs support schedule-based and load-based autoscaling for certain services; CDW virtual warehouses support auto-suspend and auto-scale.

Stop/start semantics: stopping a Data Hub releases compute but preserves the cluster definition and data in the Data Lake; deleting a cluster removes compute and any ephemeral HDFS data, not the Data Lake.

Recipes in Cloudera Manager apply scripts or parcel/config changes at cluster creation or on role restart; in CDP Public Cloud, bootstrap scripts and parcel repositories are configured via the cluster definition.

Monitoring: Cloudera Manager charts and the CDP control plane dashboards show cluster health; metrics can be exported to Prometheus/Grafana, and alerts are configured in Cloudera Manager or the control plane.

Cost control levers: auto-suspend for CDW, scheduled stop for Data Hubs, right-sizing instance types, and object storage lifecycle policies; compute is billed per running node/hour, storage per GB-month.

Upgrades in CDP Public Cloud: the control plane is upgraded by Cloudera; Data Hub and Data Lake runtime upgrades are performed via the console with rolling restart, and the Data Lake must be upgraded before attached Data Hubs.

Security baseline: enable TLS, Kerberos/LDAP, Ranger policies, and audit; restrict public endpoints, use private subnets, and grant least-privilege IAM roles to the CDP cross-account role.

CDP supports multiple engines on shared data: Impala, Hive, Spark, and NiFi/Flink read the same Hive Metastore tables and object storage, so schema and governance changes propagate to all consumers.

Hive Metastore is the shared catalog; it stores database, table, column, and partition metadata and points to storage locations. Multiple compute clusters can share one HMS in CDP Public Cloud via the Data Lake.

Ranger policy evaluation order: explicit deny overrides allow; tag-based policies are evaluated alongside resource policies; the most specific matching policy wins, and default is deny when no policy matches.

Atlas lineage is captured from Hive/Spark/Impala query hooks; lineage edges connect input and output entities, and tags applied to upstream entities can be used for downstream tag policies.

Data masking in Ranger: column masking policies replace values at query time with null, redact, or custom expressions; masking does not change stored data, so it is safe for shared datasets.

Row filtering in Ranger: row-filter policies append a filter expression to queries for matching users/groups; Impala and Hive honor these filters, and the filter is not visible in the user's SQL.

Workload isolation: CDW virtual warehouses and CDE virtual clusters isolate compute, while SDX keeps policies centralized; this avoids the classic shared-cluster noisy-neighbor problem.

Schema evolution: Iceberg supports add/drop/rename columns and partition evolution without rewriting data; Hive ACID tables support limited schema evolution; Impala requires REFRESH after DDL.

Interoperability check: a table created by Spark with a Hive-compatible format is visible to Impala after INVALIDATE METADATA; Iceberg tables created by Spark are visible to Impala via the Iceberg catalog.

Cloudera Manager role types: NameNode, DataNode, ResourceManager, NodeManager, HiveServer2, Hive Metastore, impalad, Ranger Admin, Atlas Server, and the Cloudera Manager Server/Agent; each role has a config and log directory.

Health tests: Cloudera Manager runs role health tests (for example NameNode HA, DataNode disk, Hive Metastore connectivity) and surfaces bad health with a red status; the tests are configurable per role.

Log locations: service logs default to /var/log/ on each host; Cloudera Manager Agent logs are in /var/log/cloudera-scm-agent; the Server log is /var/log/cloudera-scm-server.

Parcel management: parcels are stored in /opt/cloudera/parcels; the parcel repository URL is configured in Cloudera Manager, and activation requires distribution to all hosts and a restart of affected roles.

Database backups: the Cloudera Manager database, Hive Metastore database, Ranger database, and Oozie database are separate; back them up with the native database tools (pg_dump, mysqldump) before upgrades.

Rolling restart: Cloudera Manager restarts roles in a controlled order (for example DataNodes before NameNode) to avoid data loss; the order is defined by the service descriptor and can be tuned.

Kerberos troubleshooting: clock skew (default 5 minutes) is the most common cause of authentication failures; verify krb5.conf, keytabs, and time sync (NTP/chrony) on all hosts.

HA: HDFS NameNode HA uses JournalNodes and ZKFC; YARN ResourceManager HA uses ZooKeeper; Hive Metastore and HiveServer2 HA use a load balancer or multiple instances behind a proxy.

Quota and permissions: HDFS quotas (space and namespace) are set per directory; Ranger HDFS policies control path access; Hive warehouse permissions are enforced by Ranger, not by POSIX alone.

CDP Public Cloud control plane API: environments, clusters, Data Hubs, and Data Lake are managed via the CDP CLI and REST API; authentication uses an API access key (machine user) or SSO session.

CDP CLI commands: cdp environments list, cdp environments create-aws-environment, cdp datahub create-aws-cluster, cdp datahub describe-cluster, and cdp dw create-cluster are the core verbs for provisioning.

Infrastructure as a Service: CDP supports environment registration via the console, CLI, or Terraform provider; the Terraform provider manages environments, Data Hubs, and Data Lakes declaratively.

Data Lake upgrade: the Data Lake must be upgraded before Data Hubs; the console offers a rolling upgrade, and the Data Lake services (Ranger, Atlas, HMS) are upgraded in place while the compute clusters remain attached.

Cluster resize: Data Hub node groups can be resized up or down; scale-down drains NodeManager/DataNode roles first, and HDFS decommissioning is required for DataNodes in CDP Private Cloud Base.

Data Hub templates: the template specifies instance type, volume size, and node count per group; changing the template requires a cluster restart or recreation depending on the change type.

CDW virtual warehouse lifecycle: create, start, stop, resize, and delete; auto-suspend stops the warehouse after idle time, and the next query restarts it (cold start latency).

CDE job runs: jobs are defined with a Spark application, dependencies, and schedule; runs are tracked with logs and status, and the CDE CLI (cde job create, cde job run) is the automation path.

CML model deployment: models are built in a project session, registered, and deployed as a model endpoint with a replica count; the endpoint is governed by SDX and accessible via REST.

Migration from CDH/HDP to CDP: assess compatibility (Hive, Spark, MapReduce, Oozie, HBase, Kafka versions), then use Cloudera Manager upgrade or export/import via Replication Manager; HDP migrations often use the Cloudera distro upgrade tooling.

Hive ACID in CDP: transactional tables require ORC format and bucketing; ACID is enabled by default for managed tables in Hive 3, and Impala can read ACID tables but has limited write support.

Impala limitations: no UPDATE/DELETE on non-Kudu tables (except through Iceberg/Hive ACID support in recent versions), and complex types are supported but with performance caveats; Kudu is not part of CDP Public Cloud Data Hub by default.

Spark version in CDP 7.x is Spark 3.x with the Catalyst optimizer and AQE (adaptive query execution); AQE coalesces shuffle partitions and handles skew joins when enabled.

NiFi in CDP DataFlow: flows are built from processors, connections, and process groups; back-pressure is set per connection, and provenance records data lineage at the flow level.

Flink in CDP Stream Processing: jobs use checkpointing and savepoints for state; exactly-once is achieved with checkpointing plus idempotent sinks or transactional sinks.

Kafka in CDP: replication factor default 3, min.insync.replicas 2 for durability; consumer group offsets are stored in __consumer_offsets; MirrorMaker 2 replicates topics between clusters.

Oozie is deprecated in CDP; workflows should be migrated to Airflow, CDE schedules, or NiFi; Oozie is still present in CDP Private Cloud Base for legacy workflows.

Hue provides a SQL and notebook UI in CDP; it connects to Hive, Impala, and Spark, and honors Ranger policies for the logged-in user.

CDP Public Cloud regions and availability zones: an environment is bound to one region; the Data Lake and Data Hubs can span availability zones for HA, and object storage is regional.

Private link/private connectivity: CDP Public Cloud supports private endpoints (AWS PrivateLink, Azure Private Link) so traffic to the control plane and object storage does not traverse the public internet.

Data Lake HA: the Data Lake services (HMS, Ranger, Atlas) run with multiple replicas behind a load balancer in CDP Public Cloud; the control plane manages failover.

Disaster recovery: use Replication Manager to replicate Hive Metastore and data to a secondary environment; the DR environment can be a separate CDP environment in another region, and failover is a DNS/endpoint switch.

Backup of the Data Lake: object storage versioning and cross-region replication protect data; the HMS/Ranger/Atlas databases are backed up by the control plane and can be restored via support.

Security in transit: TLS between CDP services, between clients and Knox, and between the control plane and the environment; certificate management is automated in CDP Public Cloud.

Compliance: CDP supports audit log export to customer SIEM; Ranger audit records include user, resource, action, and result, and can be forwarded to HDFS or Solr for retention.

Cost allocation: tag environments and clusters with cost-center tags; CDP propagates tags to cloud resources, and cloud cost explorer reports by tag.

Quota limits: cloud provider quotas (vCPUs, IP addresses, storage) must be raised before large Data Hub deployments; CDP environment creation fails if quotas are insufficient.

Cloudera Manager API: the /api/vX/ endpoints expose clusters, services, roles, and commands; the API is used for automation and is the same API the UI uses; authentication is basic auth or Kerberos.

Cloudera Manager commands: role commands (start, stop, restart), service commands (start, stop, restart, deploy client configs), and cluster commands (deploy client configs, restart) are asynchronous and return a command ID.

Parcel API: parcels can be downloaded, distributed, and activated via the API; the parcel repository URL is per-cluster and can be a local HTTP server or a remote repository.

Configuration overrides: role config groups allow per-role overrides; the safety valve mechanism lets you add raw XML/properties to a service config without editing the descriptor.

Service monitoring: Cloudera Manager collects metrics via the Agent and stores them in the Service Monitor and Host Monitor; charts are built from these time-series, and alerts are triggered by thresholds.

Events and alerts: Cloudera Manager generates events for health changes, config changes, and commands; alerts can be sent via email, SNMP, or a webhook.

User management: Cloudera Manager users and roles (Full Administrator, Operator, Configurator, etc.) control access to the UI/API; external authentication via LDAP/SAML is configured in the CM server.

Audit in Cloudera Manager: the audit log records user actions in the UI/API; it is separate from Ranger audit and is stored in the CM database.

High availability for Cloudera Manager: the CM server can run in HA with a load balancer and a shared database; the Agent points to the VIP.

CDP Public Cloud Data Hub types: Data Engineering (Spark, Hive, YARN), Data Warehouse (Impala, Hive), Data Flow (NiFi), Stream Processing (Flink, Kafka), and Custom (user-defined). The type determines the services deployed.

Data Hub scaling: node groups are scaled independently; the master group hosts YARN ResourceManager and HDFS NameNode (if HDFS is used), and the worker group hosts NodeManager/DataNode.

Storage choice: Data Hubs can use object storage (S3/ADLS/GCS) via the Data Lake or local HDFS; HDFS on Data Hubs is ephemeral and lost when the cluster is deleted, so persistent data belongs in object storage.

CDW virtual warehouse types: Impala and Hive; the warehouse size (XSmall to XXLarge) determines the number of executors and memory; auto-scale adds executors under load.

CDE virtual clusters: a virtual cluster is a Kubernetes namespace with a Spark runtime; jobs run in the virtual cluster, and resources (CPU/memory) are configured per virtual cluster.

CML workspaces: a workspace is a Kubernetes namespace with a project; sessions are runtimes (Jupyter, RStudio, etc.), and jobs run batch workloads; models are deployed from the project.

CDF deployments: NiFi deployments are sized by node count and instance type; Stream Processing deployments run Flink jobs with a job manager and task managers.

COD: operational database clusters are HBase/Phoenix clusters with a gateway; they support autoscaling and backup to object storage, and are accessed via the Phoenix Query Server or HBase REST.

CDV: data visualization connects to CDW/CDE/CML data sources; dashboards are shared within the organization, and access is governed by the underlying data policies.

Ranger policy components: resource (database, table, column, URL, path), users/groups, permissions (select, insert, update, delete, create, drop, alter), and conditions; the policy is evaluated per request.

Ranger audit: each policy evaluation writes an audit record with access type, result (allowed/denied), user, and resource; audit can be stored in Solr (with HDFS as fallback) or a database.

Atlas entity model: entities have types, attributes, classifications, and relationships; lineage is a relationship between entities, and the Atlas UI shows a lineage graph.

Atlas hooks: Hive, Spark, and Impala publish metadata to Atlas via hooks; the hook is configured in the service config, and lineage is captured for CREATE/INSERT/SELECT operations.

Tag propagation: Atlas can propagate classifications along lineage (for example, a PII tag on a source table propagates to derived tables) when the propagation option is enabled.

Ranger tag policies: a tag policy references an Atlas classification; the policy applies to all resources with that tag, so tagging a table in Atlas automatically subjects it to the tag policy.

Kerberos delegation: HiveServer2 and Impala use delegation tokens for long-running queries; token renewal is handled by the client or by the service, and expired tokens cause query failures.

LDAP group mapping: Hadoop group mapping (hadoop.security.group.mapping) resolves LDAP groups for authorization; Ranger policies can reference these groups, and the mapping must be consistent across services.

TLS keystore: each service role has a keystore/truststore; Cloudera Manager can generate self-signed certs or use a CA, and the truststore must include the CA for mutual TLS.

CDP Private Cloud Base installation: use the Cloudera Manager installer or the parcel-based manual install; hosts need Python, a JDK, and the Cloudera Manager Agent; the CM server database must be created first.

Parcel vs package: parcels are the supported deployment method in CDP; packages are deprecated; parcels allow multiple versions side by side and atomic activation.

Cloudera Manager database sizing: the CM database grows with events and metrics; the Service Monitor and Host Monitor use separate databases in large deployments (or the same with retention tuning).

Networking ports: Cloudera Manager Server 7180 (HTTP) and 7183 (HTTPS); Agent 9000; NameNode 8020; ResourceManager 8032; HiveServer2 10000; Impala 21050; Ranger Admin 6080; Atlas 21000.

Host inspection: Cloudera Manager runs host inspector checks (time sync, DNS, entropy, transparent huge pages) before cluster installation; failures must be fixed before proceeding.

Service dependencies: Hive requires HMS and a database; Impala requires HMS and HDFS; Ranger requires a database and is a dependency for Hive/Impala/HDFS authorization when enabled.

Cluster sizing: separate master and worker hosts; master hosts run NameNode, ResourceManager, HMS, Ranger, and Atlas; worker hosts run DataNode and NodeManager; use dedicated disks for DataNode.

Upgrade order in CDP Private Cloud Base: Cloudera Manager first, then the CDP parcels, then the services; the order is defined by the service dependency graph, and the upgrade wizard enforces it.

Rollback: parcels can be deactivated to roll back to the previous version, but database schema changes (HMS, Ranger) may not be reversible; back up databases before upgrading.

CDP Public Cloud Data Lake services: HMS, Ranger, Atlas, and the Data Lake storage; the Data Lake is created once per environment and shared by all Data Hubs.

Data Lake creation prerequisites: an environment with a VPC, subnets, security groups, an IAM role/service principal, and an object storage bucket for the Data Lake warehouse.

Data Hub creation prerequisites: a running Data Lake in the same environment, a cluster definition, and a subnet/security group; the Data Hub inherits the Data Lake's SDX config.

CDP control plane IAM: the cross-account role must allow the control plane to create EC2, S3, KMS, and IAM resources; the role trust policy references the Cloudera control plane account.

Azure: the environment uses a service principal with Contributor on the subscription or resource group; ADLS Gen2 storage is the Data Lake store, and the service principal needs Storage Blob Data Contributor.

GCP: the environment uses a service account with roles for Compute, Storage, and IAM; the Data Lake bucket is a GCS bucket, and the service account needs object admin on the bucket.

Environment registration failure causes: insufficient IAM permissions, quota limits, overlapping CIDR ranges, or an unsupported region; the control plane reports the failing step in the environment status.

Data Hub creation failure causes: Data Lake not running, cluster definition referencing an unavailable parcel, or subnet with no free IPs; the cluster status shows the failing step.

Deletion order: delete Data Hubs before the Data Lake, and delete the environment before the cloud resources; deleting the Data Lake with attached Data Hubs is blocked.

Query performance: Impala uses runtime filters and broadcast joins for small dimension tables; the optimizer chooses broadcast when the build side is below the broadcast threshold.

Hive on Tez: the number of reducers is estimated from input size (hive.exec.reducers.bytes.per.reducer, default 256 MB); Tez groups small tasks and avoids unnecessary reducers.

Partition pruning: queries must filter on partition columns for pruning; using functions on partition columns disables pruning, and dynamic partition pruning is available in Impala and Spark.

File size target: aim for 128 MB-1 GB per file for Parquet/ORC; too many small files cause metadata overhead and slow scans, and too few large files reduce parallelism.

Compression: Parquet/ORC with Snappy or Zstd is the default for analytics; compression reduces storage and I/O but adds CPU; Zstd offers a better ratio than Snappy at higher CPU cost.

Bucketing: Hive bucketing (CLUSTERED BY) co-locates rows for joins and sampling; it must match the join key and bucket count to be effective, and Iceberg hidden partitioning is preferred in new designs.

Join strategy: map-side joins for small tables, bucketed map joins for bucketed tables, and sort-merge joins for large-large joins; the optimizer picks based on statistics.

Statistics: table and column stats (NDV, row count, size) drive join order and broadcast decisions; missing stats cause the optimizer to guess, often badly.

Resource tuning: YARN memory per NodeManager (yarn.nodemanager.resource.memory-mb) and vcores determine container capacity; Impala memory limits per query are set by MEM_LIMIT and admission control.

CDP Public Cloud SLA and support: the control plane is operated by Cloudera; the customer operates the Data Hubs and Data Lake services; support cases are opened in the Cloudera support portal.

Diagnostic bundles: CDP Public Cloud clusters can generate a support bundle from the console; it includes service logs, configs, and system info, and is uploaded to Cloudera support.

Cloudera Manager diagnostics: the Diagnostics page collects logs and metrics from selected roles and hosts into a bundle; it can be run on demand or scheduled.

Log aggregation: CDP services write logs to local disk and optionally to a centralized location; in CDP Public Cloud, logs can be exported to object storage for retention.

Metrics export: Cloudera Manager can export metrics to Prometheus via the Prometheus exporter; Grafana dashboards are commonly used for visualization.

Alerting: Cloudera Manager alerts on health, thresholds, and events; in CDP Public Cloud, the control plane shows environment and cluster status, and alerts can be integrated with the customer's monitoring.

Common failure: HDFS safe mode is entered when the NameNode starts without enough block reports; check DataNode health and exit safe mode manually only after verifying block availability.

Common failure: YARN queue capacity exceeded causes applications to wait; check the Capacity Scheduler queue state and the user limit factor.

Common failure: Hive Metastore connection pool exhaustion causes query failures; increase the HMS connection pool or the database max connections.

CDP Public Cloud Data Hub autoscaling: load-based autoscaling adds/removes worker nodes based on YARN metrics; schedule-based autoscaling changes node counts on a time schedule; both require the cluster definition to support it.

CDW auto-scale: the virtual warehouse adds executors when queries queue and removes them when idle; auto-suspend stops the warehouse after the configured idle timeout.

CDE resource allocation: the virtual cluster has a CPU/memory quota; jobs that exceed the quota queue or fail; the quota is set per virtual cluster in the CDE UI/CLI.

CML session sizing: sessions have a resource profile (CPU/memory/GPU); the profile determines the pod resources, and the project's quota limits concurrent sessions.

NiFi back-pressure: connection thresholds (object count and data size) stop upstream processors when exceeded; this prevents memory exhaustion and provides flow control.

Kafka consumer lag: lag is the difference between the log-end offset and the consumer offset; lag indicates slow consumers, and scaling consumers (up to partition count) improves throughput.

Flink checkpointing: checkpoint interval and state backend (RocksDB for large state) determine recovery time; savepoints are used for upgrades and planned restarts.

Spark AQE: spark.sql.adaptive.enabled=true coalesces shuffle partitions and handles skew; it requires the shuffle exchange to be materialized, and it is on by default in Spark 3.x.

Iceberg maintenance: compaction, manifest rewriting, and snapshot expiration keep table metadata and file sizes healthy; without maintenance, query planning slows and storage grows.

CDP Public Cloud identity: Cloudera SSO or a federated IdP (SAML/OIDC) authenticates users; the control plane maps users to roles (Admin, User, etc.) per environment or resource.

Ranger in CDP Public Cloud: Ranger policies are managed in the Data Lake; Data Hubs and CDW inherit the policies, and the Ranger UI is accessed via the Data Lake endpoint.

Knox in CDP Public Cloud: Knox provides the gateway to the Data Lake and Data Hubs; it authenticates via the IdP and authorizes via Ranger, and it proxies the service UIs.

Workload identity: in CDP Public Cloud, users are mapped to cloud identities for object storage access via Ranger and the environment's IAM role; the user does not need direct cloud credentials.

Audit export: Ranger audit records can be exported to a customer S3 bucket or SIEM; the export is configured in Ranger and runs on a schedule.

Data masking in CDP Public Cloud: Ranger masking policies apply to Impala/Hive queries; the masked value is computed at query time and does not affect stored data.

Row-level filtering: Ranger row-filter policies apply to Hive/Impala; the filter is injected into the query plan, and the user sees only permitted rows.

Tag-based access: Atlas classifications (PII, SENSITIVE) drive Ranger tag policies; the classification must be applied in Atlas, and the tag policy must exist in Ranger for enforcement.

Compliance reporting: Atlas lineage plus Ranger audit provides a data-access trail; the lineage graph shows data flow, and the audit log shows who accessed what.

CDP Public Cloud environment types: the environment is the top-level container; it holds the Data Lake, Data Hubs, and Data Services experiences, and it is bound to a cloud provider, region, and network.

Data Lake sizing: the Data Lake services are small (HMS, Ranger, Atlas) and run on a few nodes; the storage is object storage, so the Data Lake does not need large local disks.

Data Hub sizing: worker nodes carry the compute; size by vcores/memory for YARN, and by storage for HDFS if used; instance types are chosen per node group.

CDW sizing: warehouse size maps to executor count and memory; start small and use auto-scale; the warehouse shares the Data Lake storage, so no data copy is needed.

CDE sizing: virtual cluster CPU/memory quota determines concurrent job capacity; Spark driver/executor sizes are set per job, and the quota must accommodate them.

CML sizing: session profiles and replica counts determine resource use; GPU profiles are available for deep learning, and the project quota limits total resources.

CDF sizing: NiFi node count and instance type determine throughput; Stream Processing job manager/task manager sizes determine Flink parallelism and state capacity.

COD sizing: region servers and storage determine throughput and capacity; HBase on object storage (via Ozone or HDFS) changes the IO profile, and autoscaling adjusts region servers.

Cost model: compute is billed while running; storage is billed per GB-month; data transfer between regions or to the internet is billed separately; auto-suspend and scheduled stop are the main cost controls.

CDP Public Cloud CLI authentication: cdp configure sets the CDP profile with an API access key and private key; the CLI signs requests and the control plane authorizes the machine user.

CDP CLI output: --output json or table; commands return resource CRNs (Cloudera Resource Names), which are used in IAM policies and API calls.

CDP IAM: machine users and workload users have roles (Environment Admin, Data Hub Admin, etc.); roles are assigned at the environment or resource level, and CRNs scope the permission.

Terraform: the Cloudera Terraform provider manages environments, Data Lakes, Data Hubs, and CDW; state files must be secured, and the provider uses the CDP API credentials.

CDP API versioning: the API is versioned (for example /api/v1/environments); the CLI wraps the API, and the API is the stable automation interface.

CDP control plane endpoints: the API endpoint is region-specific (for example api.us-west-1.cdp.cloudera.com); the CLI and Terraform must target the correct endpoint.

CDP environment registration via CLI: cdp environments create-aws-environment requires the VPC, subnets, security groups, and IAM role ARN; the command returns a CRN and the environment status.

CDP Data Hub creation via CLI: cdp datahub create-aws-cluster requires the environment CRN, cluster definition name, and node group configuration; the command returns a cluster CRN.

CDP CDW creation via CLI: cdp dw create-cluster requires the environment CRN, warehouse type (Impala/Hive), and size; the warehouse is then started with cdp dw start-cluster.

CDP Public Cloud Data Lake upgrade: the Data Lake runtime is upgraded via the console; the control plane orchestrates the upgrade of HMS, Ranger, and Atlas, and the Data Hubs remain attached.

Data Hub upgrade: the Data Hub runtime is upgraded after the Data Lake; the console offers a rolling upgrade, and the cluster definition must reference the target parcel version.

Parcel availability: the target parcel must be available in the parcel repository; CDP Public Cloud uses a Cloudera-managed repository, and Private Cloud uses a local or remote repository.

Upgrade prerequisites: back up the HMS, Ranger, and Atlas databases; verify disk space for parcels; check compatibility of custom configs and bootstrap scripts.

Rollback: in CDP Public Cloud, the Data Lake upgrade is not reversible; the Data Hub upgrade can be rolled back by recreating the cluster with the previous definition, but data in object storage is unaffected.

Service restart order: dependent services restart after their dependencies; for example, Hive restarts after HMS, and Impala restarts after HMS and HDFS.

Client configuration: after an upgrade, client configs (hive-site.xml, core-site.xml, etc.) must be redeployed; Cloudera Manager's Deploy Client Configuration pushes them to hosts.

Compatibility: custom UDFs, Spark jars, and NiFi processors must be compatible with the new runtime; test in a non-production environment before upgrading production.

Versioning: CDP Private Cloud Base 7.x runtimes are versioned (7.1.x); CDP Public Cloud runtimes are updated by Cloudera, and the runtime version is shown in the cluster details.

CDP security layers: authentication (Kerberos/LDAP/IdP), authorization (Ranger), auditing (Ranger audit), encryption (TLS, TDE, KMS), and governance (Atlas). Each layer is configured independently.

Kerberos principal naming: service principals follow the pattern /@REALM; the keytab must be readable by the service user, and the principal must match the host's FQDN.

Ranger plugin: each service (HDFS, Hive, Impala, HBase, Kafka, Knox) has a Ranger plugin that downloads policies from Ranger Admin; the plugin caches policies and refreshes on a configurable interval.

Ranger policy conditions: IP range, time of day, and custom conditions can restrict a policy; conditions are evaluated with the policy and can be used for break-glass access.

Atlas authentication: Atlas uses Kerberos or simple auth; in CDP Public Cloud, Atlas is behind Knox and uses the IdP; the Atlas UI is accessed via the Knox gateway.

TLS for Knox: Knox terminates TLS and proxies to services; the Knox master secret stores credentials, and the topology files define the proxied services.

Data encryption keys: HDFS TDE uses a KMS (Key Trustee or a cloud KMS) to store encryption zone keys; losing the KMS key makes the encrypted data unreadable.

Audit retention: Ranger audit records are stored in Solr with a retention policy; long-term retention requires export to HDFS or an external SIEM.

Least privilege: grant Ranger policies to groups, not users; use tag policies for broad classifications, and review audit logs for denied access to tune policies.

CDP Public Cloud Data Hub templates: the template defines instance types, volume sizes, and node counts; the cluster definition (blueprint) defines the services and roles; both are referenced at cluster creation.

Custom cluster definitions: a blueprint is a JSON/XML definition of services and role configs; custom definitions can be uploaded and used for Data Hubs, and they must be compatible with the CDP runtime.

Bootstrap scripts: a bootstrap script runs on every node at cluster creation; it is used for custom packages, config changes, or agent installation; it must be idempotent and fail-safe.

Parcel repositories: a custom parcel repository URL can be set per cluster; the repository must host the parcels and the manifest, and the cluster downloads and activates them.

Cluster resize limits: scale-up is limited by cloud quotas and the cluster definition; scale-down is limited by the minimum node count per group and HDFS replication if HDFS is used.

Decommissioning: in CDP Private Cloud Base, DataNode decommissioning requires the exclude file and a refresh; in CDP Public Cloud, scale-down handles decommissioning automatically for HDFS.

Cluster deletion: deleting a Data Hub deletes the compute and ephemeral HDFS; data in object storage remains; the Data Lake is unaffected.

Cluster stop: stopping a Data Hub stops the compute nodes but keeps the cluster definition and metadata; starting it recreates the nodes from the definition.

Cluster repair: if a node fails, the control plane can repair the cluster by replacing the node; the repair uses the cluster definition and the node group configuration.

CDP Public Cloud Data Lake storage: the warehouse directory is in object storage (for example s3a://bucket/warehouse/tablespace/managed/hive); managed tables are stored there, and external tables point to other paths.

Object storage semantics: S3/ADLS/GCS are eventually consistent for some operations; Hive/Impala use the HMS as the source of truth, and directory listings can be stale, so INVALIDATE METADATA is sometimes needed.

S3A configuration: fs.s3a.access.key and secret key, or IAM role; the S3A committer (magic or directory) affects performance and correctness; the magic committer is preferred for Spark on S3.

ADLS Gen2: abfs:// paths; authentication via OAuth (service principal) or managed identity; the filesystem is hierarchical and supports atomic rename, unlike S3.

GCS: gs:// paths; authentication via service account JSON or workload identity; the GCS connector supports hierarchical namespace and atomic operations in some modes.

HDFS on Data Hubs: ephemeral and lost on cluster deletion; use it for shuffle and temporary data, not for persistent tables; replication factor 3 by default.

Ozone: ofs:// or s3a:// paths; Ozone buckets and keys; the Ozone Manager and Storage Container Manager manage metadata and containers; Ozone is available in CDP Private Cloud Base.

Table location: managed tables use the warehouse directory; external tables can point anywhere; changing the location of a managed table requires ALTER TABLE SET LOCATION and data movement.

Storage format migration: convert text/CSV to Parquet/ORC with Spark or Hive INSERT OVERWRITE; the conversion rewrites data and should be done in partitions to control resource use.

CDP Public Cloud Data Services experiences: CDW, CDE, CML, CDF, COD, and CDV; each is activated per environment and uses the Data Lake for storage and SDX for governance.

CDW: virtual warehouses are created per environment; the warehouse uses the Data Lake HMS and storage; queries run on Impala or Hive, and the warehouse can be resized and auto-suspended.

CDE: virtual clusters run Spark jobs; jobs are defined with a Spark application and dependencies; the CDE CLI and API automate job submission, and Airflow can orchestrate jobs.

CML: projects are backed by Git; sessions run Jupyter/RStudio; models are deployed as endpoints; the CML API automates project and model operations.

CDF: NiFi flows are deployed as deployments; Stream Processing jobs are Flink jobs; the CDF CLI and API manage deployments and jobs.

COD: HBase/Phoenix clusters are created per environment; the cluster has a gateway for client access; backup and restore use object storage, and autoscaling adjusts region servers.

CDV: dashboards connect to CDW/CDE/CML; the visualization is a separate experience, and data access is governed by the underlying policies.

Cross-experience: a CDE job can write to a table that CDW queries; the HMS and SDX policies are shared, so no data copy or policy duplication is needed.

Experience activation: an experience must be enabled for the environment before use; enabling an experience may require additional cloud resources or quotas.

CDP Private Cloud Base services: HDFS, YARN, Hive, Impala, HBase, Kafka, Solr, Oozie, ZooKeeper, Ranger, Atlas, and Cloudera Manager; each service has roles distributed across hosts.

Cloudera Manager role assignment: roles are assigned to hosts via role config groups; the assignment must respect dependencies (for example, HMS on a host with a database client).

Parcel activation: a parcel must be downloaded, distributed, and activated; activation changes the symlink in /opt/cloudera/parcels and requires a restart of the roles using it.

Service start order: ZooKeeper, HDFS, YARN, then dependent services (Hive, Impala, HBase); Cloudera Manager enforces the order via the service dependency graph.

Configuration changes: some configs require a restart, others are dynamic; Cloudera Manager marks the restart requirement, and the stale configuration indicator shows which roles need a restart.

Safety valves: advanced configs can be added via safety valves (for example, HDFS Service Advanced Configuration Snippet); they override the generated config and are not validated.

Host decommissioning: decommissioning a host moves roles off it; DataNode decommissioning replicates blocks, and NodeManager decommissioning waits for running containers to finish.

Cloudera Manager HA: the CM server can run in HA with a shared database and a load balancer; the Agent and clients point to the VIP, and the database must be highly available.

Backup and restore: the CM database backup includes configs and events; the HMS, Ranger, and Atlas databases are separate; restore requires stopping services, restoring the database, and restarting.

CDP Public Cloud Data Hub types and services: Data Engineering (YARN, Hive, Spark, HDFS optional), Data Warehouse (Impala, Hive), Data Flow (NiFi), Stream Processing (Flink, Kafka), Custom (any service set).

Data Engineering: Spark jobs run on YARN; Hive on Tez is available; HDFS is optional and ephemeral; the cluster is sized by worker nodes.

Data Warehouse: Impala and Hive; the cluster is sized by worker nodes; queries use the Data Lake HMS and storage; the cluster is often ephemeral and recreated per workload.

Data Flow: NiFi runs on the cluster; flows are authored in NiFi and can be parameterized; the cluster is sized by node count and instance type.

Stream Processing: Flink and Kafka; Flink jobs run on the cluster; Kafka topics are created and managed; the cluster is sized by node count and instance type.

Custom: the user selects the services and roles; the cluster definition must be valid, and the services must be compatible with the CDP runtime.

Service availability: not all services are available in all Data Hub types; for example, HBase is available in Custom but not in Data Warehouse; check the cluster definition.

Data Hub lifecycle: create, start, stop, resize, upgrade, repair, delete; each operation is asynchronous and reported in the cluster status.

Data Hub access: services are accessed via Knox (for example, Hive JDBC via Knox), and the client configs are downloaded from the cluster details.

CDP Public Cloud Data Lake services: HMS (Hive Metastore), Ranger Admin, Atlas Server, and the Data Lake storage; the Data Lake also hosts the Knox gateway for the environment.

HMS in the Data Lake: the HMS is shared by all Data Hubs and CDW; the HMS database is managed by the control plane, and the HMS is highly available behind a load balancer.

Ranger in the Data Lake: Ranger Admin manages policies for all services; the Ranger plugins in Data Hubs and CDW download policies; the Ranger database is managed by the control plane.

Atlas in the Data Lake: Atlas stores metadata and lineage; the Atlas hooks in Data Hubs publish metadata; the Atlas database is managed by the control plane.

Knox in the Data Lake: Knox provides the gateway for the Data Lake and Data Hubs; it authenticates via the IdP and authorizes via Ranger; the Knox topology is managed by the control plane.

Data Lake storage: the warehouse directory is in object storage; the storage location is set at Data Lake creation; the Data Lake can use multiple buckets for different purposes.

Data Lake HA: the Data Lake services run with multiple replicas; the control plane manages failover; the Data Lake is designed to be highly available within a region.

Data Lake upgrade: the Data Lake runtime is upgraded via the console; the upgrade is rolling and the Data Hubs remain attached; the Data Lake must be upgraded before the Data Hubs.

Data Lake deletion: the Data Lake can be deleted only after all Data Hubs are deleted; deleting the Data Lake does not delete the object storage data unless the bucket is deleted.

CDP Public Cloud Data Services: CDW, CDE, CML, CDF, COD, CDV; each is a separate experience with its own CLI/API and UI, sharing the Data Lake and SDX.

CDW CLI: cdp dw create-cluster, cdp dw start-cluster, cdp dw stop-cluster, cdp dw describe-cluster; the warehouse is identified by a CRN.

CDE CLI: cde job create, cde job run, cde job list, cde run list; the CLI uses a virtual cluster and a job definition (Spark application, args, dependencies).

CML CLI: cml project create, cml session create, cml model create, cml model deploy; the CLI uses the CML API and the project ID.

CDF CLI: cdf deployment create, cdf deployment start, cdf deployment stop; the CLI manages NiFi and Stream Processing deployments.

COD CLI: cdp opdb create-cluster, cdp opdb start-cluster, cdp opdb stop-cluster; the CLI manages operational database clusters.

CDV CLI: cdv dashboard create, cdv dashboard list; the CLI manages dashboards and data sources.

API authentication: the CDP CLI uses the API access key; the Data Services CLIs use the same credentials and the CDP API endpoint; the CLIs are wrappers over the REST APIs.

Automation: use the CLI or Terraform for repeatable provisioning; use Airflow or the CDE scheduler for job orchestration; use the CDP API for custom integrations.

CDP Public Cloud Data Hub autoscaling: load-based autoscaling uses YARN metrics (pending containers, available memory) to add/remove worker nodes; schedule-based autoscaling changes node counts on a time schedule.

Autoscaling prerequisites: the cluster definition must have autoscaling enabled, and the node group must have a minimum and maximum node count; the cluster must be running.

Scale-down behavior: the control plane drains the node before removal; running YARN containers are allowed to finish, and HDFS blocks are replicated if HDFS is used.

CDW auto-scale: the warehouse adds executors when queries queue; the scale-up is bounded by the warehouse size and the environment quota; auto-suspend stops the warehouse after the idle timeout.

CDE resource quota: the virtual cluster has a CPU/memory quota; jobs that exceed the quota queue; the quota is set per virtual cluster and can be changed via the CLI/API.

CML quota: the project has a quota for sessions and replicas; the quota limits concurrent resource use; the quota is set per project and can be changed via the API.

CDF autoscaling: NiFi deployments support autoscaling based on metrics; Stream Processing jobs scale by parallelism, not by autoscaling the cluster.

COD autoscaling: region servers are added/removed based on load; the autoscaling policy is configured per cluster; the cluster must have a minimum and maximum region server count.

Cost monitoring: use the CDP control plane's cost dashboards and cloud cost explorer; tag resources for cost allocation; set budgets and alerts in the cloud provider.

CDP Public Cloud Data Hub troubleshooting: check the cluster status in the console; the status shows the failing step (provisioning, bootstrap, cluster install); the support bundle includes the logs.

Bootstrap failure: the bootstrap script failed or the parcel download failed; check the script logs and the parcel repository URL; the cluster creation fails and the nodes are terminated.

Cluster install failure: a service role failed to start; check the role logs in the support bundle; common causes are database connectivity, Kerberos, or port conflicts.

Knox access failure: the Knox topology is misconfigured or the IdP is unreachable; check the Knox logs and the IdP configuration; the Knox gateway is in the Data Lake.

Ranger policy not applied: the plugin has not refreshed, or the policy is not in the correct service; check the Ranger plugin logs and the policy audit; restart the plugin or wait for the refresh interval.

HMS connection failure: the HMS is down or the database is unreachable; check the HMS logs and the database connectivity; the HMS is in the Data Lake.

Query failure: the table is not visible because the HMS metadata is stale; run INVALIDATE METADATA in Impala or REFRESH; check the HMS logs for the DDL.

Object storage access failure: the IAM role or service principal lacks permissions; check the S3/ADLS/GCS access logs and the environment's IAM role; the error is often AccessDenied.

Performance issue: check the query profile, the table statistics, and the file sizes; small files and missing stats are the most common causes of slow queries.

CDP Public Cloud Data Lake and Data Hub networking: the environment uses a VPC/VNet with subnets; the Data Lake and Data Hubs are deployed in the subnets; security groups control traffic.

Private connectivity: AWS PrivateLink, Azure Private Link, and GCP Private Service Connect provide private access to the control plane and object storage; the endpoints are created in the customer VPC.

DNS: the environment uses the VPC DNS; the Data Lake and Data Hubs have private DNS names; the Knox gateway has a public or private endpoint depending on the configuration.

Load balancer: the Data Lake services are behind a load balancer; the load balancer is created by the control plane and has a DNS name; the clients use the load balancer DNS.

Firewall rules: allow traffic between the Data Lake and Data Hubs, and between the Data Hubs and object storage; allow outbound to the control plane endpoints; allow inbound to Knox from the client network.

CIDR planning: the VPC CIDR must not overlap with other networks; the subnets must have enough IPs for the Data Lake and Data Hubs; the control plane requires specific endpoints.

Cross-region: Data Hubs must be in the same region as the Data Lake; cross-region access to object storage is possible but adds latency and data transfer cost.

On-premises connectivity: use VPN or Direct Connect/ExpressRoute to connect the on-premises network to the VPC; the Data Lake and Data Hubs are reachable via the private network.

Security groups: the security groups must allow the CDP control plane to manage the instances; the control plane uses specific ports and IP ranges.

CDP Public Cloud Data Hub storage: HDFS is optional; if enabled, it is ephemeral and lost on cluster deletion; object storage is persistent and shared via the Data Lake.

HDFS on Data Hubs: the NameNode runs on the master node group; DataNodes run on worker nodes; the replication factor is 3 by default; the block size is 128 MB.

Object storage on Data Hubs: the Data Hub accesses the Data Lake's object storage via the environment's IAM role; the paths are s3a://, abfs://, or gs://.

Data Lake warehouse: the warehouse directory is the default location for managed tables; it is created at Data Lake creation and is in object storage.

External tables: the location can be any object storage path; the table metadata is in the HMS; the data is not moved.

Table formats: Parquet, ORC, Avro, JSON, CSV, and Iceberg; Iceberg tables are supported in Impala, Hive, and Spark; the Iceberg catalog is the HMS or a REST catalog.

ACID: Hive ACID tables are supported in CDP Private Cloud Base; in CDP Public Cloud, Iceberg provides ACID semantics; Impala supports Iceberg reads and writes in recent versions.

Compaction: Iceberg compaction rewrites small files; Hive ACID compaction merges delta files; both are scheduled jobs and should be monitored.

Storage cost: object storage is billed per GB-month; lifecycle policies move older data to cheaper tiers; HDFS on Data Hubs is billed as compute, not storage.

CDP Public Cloud Data Services: CDW, CDE, CML, CDF, COD, CDV; each is activated per environment and has its own UI, CLI, and API.

CDW: virtual warehouses; Impala or Hive; auto-suspend and auto-scale; shared Data Lake storage and SDX policies.

CDE: virtual clusters; Spark jobs; job dependencies and schedules; Airflow integration; the CDE CLI is the automation path.

CML: projects; sessions (Jupyter, RStudio); models and endpoints; Git integration; the CML API automates project and model operations.

CDF: NiFi deployments and Stream Processing (Flink) jobs; the CDF CLI manages deployments and jobs; flows are authored in NiFi.

COD: HBase/Phoenix clusters; a gateway for client access; backup/restore to object storage; autoscaling of region servers.

CDV: dashboards and data sources; connects to CDW, CDE, and CML; access is governed by the underlying data policies.

Shared services: all experiences use the Data Lake HMS, Ranger, and Atlas; policies and metadata are centralized; no data copy is needed between experiences.

Cost: each experience bills for its compute; auto-suspend and scheduled stop reduce cost; storage is billed separately in object storage.

CDP Public Cloud Data Hub upgrade: the cluster definition must reference the target parcel version; the console offers a rolling upgrade; the Data Lake must be upgraded first.

Upgrade prerequisites: back up the HMS and Ranger databases; verify the parcel is available; check custom configs and bootstrap scripts for compatibility.

Rolling upgrade: the control plane restarts roles in a controlled order; the cluster remains available during the upgrade; the upgrade can be paused or rolled back in some cases.

Rollback: the Data Hub upgrade can be rolled back by recreating the cluster with the previous definition; data in object storage is unaffected; ephemeral HDFS is lost.

Client configs: after the upgrade, download the new client configs from the cluster details; the old configs may not work with the new runtime.

Compatibility: custom UDFs, Spark jars, and NiFi processors must be compatible with the new runtime; test in a non-production environment first.

Data Lake upgrade: the Data Lake runtime upgrade is rolling; the HMS, Ranger, and Atlas are upgraded; the Data Hubs remain attached; the Data Lake must be upgraded before the Data Hubs.

Version visibility: the runtime version is shown in the cluster details; the parcel version is shown in the parcel list; the two must match for a successful upgrade.

Support: Cloudera support provides upgrade guidance and can assist with failures; the support bundle includes the upgrade logs.

CDP Public Cloud Data Lake and Data Hub security: authentication via the IdP (SSO/SAML/OIDC); authorization via Ranger; auditing via Ranger audit; encryption via TLS and object storage encryption.

IdP integration: the IdP is configured at the CDP account level; users and groups are synchronized; the control plane maps groups to roles; the IdP must support SAML 2.0 or OIDC.

Ranger policies: managed in the Data Lake Ranger; the policies apply to all Data Hubs and CDW; the policies can be resource-based or tag-based.

Knox: the gateway authenticates via the IdP and authorizes via Ranger; the Knox topology defines the proxied services; the Knox master secret stores credentials.

Audit: Ranger audit records are stored in Solr in the Data Lake; the audit can be exported to object storage or a SIEM; the audit retention is configurable.

Encryption: TLS between services and clients; object storage encryption at rest (SSE-S3, SSE-KMS, ADLS encryption, GCS encryption); HDFS TDE for HDFS on Data Hubs.

Key management: the cloud KMS keys are managed by the customer; the environment's IAM role must have permission to use the keys; losing the key makes the data unreadable.

Least privilege: grant Ranger policies to groups; use tag policies for broad classifications; review audit logs for denied access to tune policies.

Compliance: Atlas lineage plus Ranger audit provides a data-access trail; the lineage graph shows data flow, and the audit log shows who accessed what.

CDP Public Cloud Data Hub and Data Lake monitoring: the control plane shows environment and cluster status; Cloudera Manager shows service and role health; the two are linked.

Cloudera Manager charts: CPU, memory, disk, network, and service-specific metrics; the charts are built from the Agent metrics; the retention is configurable.

Alerts: Cloudera Manager alerts on health, thresholds, and events; the alerts can be sent via email, SNMP, or a webhook; the alert rules are configurable.

Metrics export: the Prometheus exporter exposes Cloudera Manager metrics; Grafana dashboards visualize them; the exporter is configured in Cloudera Manager.

Logs: service logs are on the hosts; the support bundle collects them; in CDP Public Cloud, the logs can be exported to object storage for retention.

Diagnostics: the Cloudera Manager Diagnostics page collects logs and metrics; the support bundle is uploaded to Cloudera support; the bundle includes the cluster definition and configs.

Health tests: Cloudera Manager runs role health tests; the tests are configurable; a failed test turns the role status red and triggers an alert.

Events: Cloudera Manager records events for health changes, config changes, and commands; the events are searchable and can be used for auditing.

Capacity: the Cloudera Manager database grows with events and metrics; the retention and purge policies must be tuned for large clusters.

CDP Public Cloud Data Services: CDW, CDE, CML, CDF, COD, CDV; each is a separate experience with its own UI, CLI, and API, sharing the Data Lake and SDX.

CDW: virtual warehouses; Impala or Hive; auto-suspend and auto-scale; shared Data Lake storage and SDX policies; the warehouse is identified by a CRN.

CDE: virtual clusters; Spark jobs; job dependencies and schedules; Airflow integration; the CDE CLI is the automation path; the virtual cluster has a CPU/memory quota.

CML: projects; sessions (Jupyter, RStudio); models and endpoints; Git integration; the CML API automates project and model operations; the project has a quota.

CDF: NiFi deployments and Stream Processing (Flink) jobs; the CDF CLI manages deployments and jobs; flows are authored in NiFi; back-pressure controls flow.

COD: HBase/Phoenix clusters; a gateway for client access; backup/restore to object storage; autoscaling of region servers; the cluster is identified by a CRN.

CDV: dashboards and data sources; connects to CDW, CDE, and CML; access is governed by the underlying data policies.

Shared services: all experiences use the Data Lake HMS, Ranger, and Atlas; policies and metadata are centralized; no data copy is needed between experiences.

Cost: each experience bills for its compute; auto-suspend and scheduled stop reduce cost; storage is billed separately in object storage.

CDP Public Cloud Data Hub types: Data Engineering, Data Warehouse, Data Flow, Stream Processing, Custom; the type determines the services and roles deployed.

Data Engineering: YARN, Hive, Spark, HDFS optional; the cluster is sized by worker nodes; Spark jobs run on YARN.

Data Warehouse: Impala and Hive; the cluster is sized by worker nodes; queries use the Data Lake HMS and storage; the cluster is often ephemeral.

Data Flow: NiFi; the cluster is sized by node count and instance type; flows are authored in NiFi and can be parameterized.

Stream Processing: Flink and Kafka; the cluster is sized by node count and instance type; Flink jobs run on the cluster.

Custom: the user selects the services and roles; the cluster definition must be valid; the services must be compatible with the CDP runtime.

Service availability: not all services are available in all Data Hub types; check the cluster definition before creating the cluster.

Data Hub lifecycle: create, start, stop, resize, upgrade, repair, delete; each operation is asynchronous and reported in the cluster status.

Data Hub access: services are accessed via Knox; the client configs are downloaded from the cluster details; the Knox gateway is in the Data Lake.

CDP Public Cloud Data Lake services: HMS, Ranger Admin, Atlas Server, and the Data Lake storage; the Data Lake also hosts the Knox gateway.

HMS: shared by all Data Hubs and CDW; the HMS database is managed by the control plane; the HMS is highly available behind a load balancer.

Ranger: manages policies for all services; the plugins in Data Hubs and CDW download policies; the Ranger database is managed by the control plane.

Atlas: stores metadata and lineage; the hooks in Data Hubs publish metadata; the Atlas database is managed by the control plane.

Knox: provides the gateway for the Data Lake and Data Hubs; authenticates via the IdP and authorizes via Ranger; the topology is managed by the control plane.

Data Lake storage: the warehouse directory is in object storage; the storage location is set at Data Lake creation; the Data Lake can use multiple buckets.

Data Lake HA: the services run with multiple replicas; the control plane manages failover; the Data Lake is designed to be highly available within a region.

Data Lake upgrade: the runtime is upgraded via the console; the upgrade is rolling and the Data Hubs remain attached; the Data Lake must be upgraded before the Data Hubs.

Data Lake deletion: the Data Lake can be deleted only after all Data Hubs are deleted; deleting the Data Lake does not delete the object storage data unless the bucket is deleted.

CDP Public Cloud Data Hub and Data Lake CLI: cdp environments, cdp datahub, cdp dw, cdp opdb; the CLI uses the CDP API and the API access key.

Environment commands: cdp environments list, cdp environments describe-environment, cdp environments create-aws-environment, cdp environments delete-environment.

Data Lake commands: cdp datalake create-aws-datalake, cdp datalake describe-datalake, cdp datalake upgrade-datalake, cdp datalake delete-datalake.

Data Hub commands: cdp datahub create-aws-cluster, cdp datahub describe-cluster, cdp datahub list-clusters, cdp datahub delete-cluster, cdp datahub repair-cluster.

CDW commands: cdp dw create-cluster, cdp dw start-cluster, cdp dw stop-cluster, cdp dw describe-cluster, cdp dw delete-cluster.

COD commands: cdp opdb create-cluster, cdp opdb start-cluster, cdp opdb stop-cluster, cdp opdb describe-cluster, cdp opdb delete-cluster.

CRNs: each resource has a CRN; the CRN is used in IAM policies and API calls; the CLI returns the CRN on creation.

Output: --output json or table; the JSON output is used for automation; the table output is for humans.

Authentication: cdp configure sets the profile; the API access key and private key are stored in the profile; the CLI signs requests.

CDP Public Cloud Data Hub and Data Lake IAM: the environment's IAM role/service principal is used by the Data Lake and Data Hubs to access cloud resources.

AWS: the cross-account role trust policy references the Cloudera control plane account; the role has permissions for EC2, S3, KMS, IAM, and CloudWatch; the role is created during environment registration.

Azure: the service principal has Contributor on the subscription or resource group; the service principal has Storage Blob Data Contributor on the ADLS Gen2 account.

GCP: the service account has roles for Compute, Storage, and IAM; the service account has object admin on the GCS bucket.

Least privilege: the IAM role should have only the permissions required by CDP; the permissions are documented by Cloudera; extra permissions increase risk.

Key management: the IAM role must have permission to use the KMS keys for object storage encryption; the keys are managed by the customer.

Networking: the IAM role must have permission to create and manage the network resources (security groups, load balancers) used by CDP.

Troubleshooting: AccessDenied errors in the Data Lake or Data Hubs usually indicate missing IAM permissions; check the cloud provider's access logs and the role policy.

Rotation: the IAM role credentials are rotated by the control plane; the customer does not manage the credentials; the role trust policy must remain valid.

CDP Public Cloud Data Hub and Data Lake cost: compute is billed per running node/hour; storage is billed per GB-month; data transfer is billed separately.

Cost controls: auto-suspend for CDW, scheduled stop for Data Hubs, right-sizing instance types, and object storage lifecycle policies.

Tagging: tag environments and clusters with cost-center tags; CDP propagates tags to cloud resources; the cloud cost explorer reports by tag.

Budgets: set budgets and alerts in the cloud provider; the CDP control plane shows cost dashboards; the two should be reconciled.

Reserved instances: cloud provider reserved instances or savings plans reduce compute cost; CDP uses on-demand instances by default; the customer can use reserved capacity where supported.

Storage tiers: object storage lifecycle policies move older data to cheaper tiers (S3 Glacier, ADLS Archive, GCS Coldline); the data remains accessible but with retrieval cost.

Data transfer: cross-region and internet egress are billed; keep the Data Lake and Data Hubs in the same region; use private connectivity to avoid internet egress.

Idle resources: stopped Data Hubs do not incur compute cost; deleted Data Hubs release all compute; the Data Lake services incur a small ongoing cost.

Chargeback: use tags and the CDP cost dashboards to allocate cost to teams; the CRN identifies the resource for billing.

CDP Public Cloud Data Hub and Data Lake support: the control plane is operated by Cloudera; the customer operates the Data Hubs and Data Lake services; support cases are opened in the Cloudera support portal.

Support bundle: the cluster details page has a Generate Support Bundle action; the bundle includes logs, configs, and system info; it is uploaded to Cloudera support.

Diagnostics: the Cloudera Manager Diagnostics page collects logs and metrics; the bundle can be run on demand or scheduled; the bundle is stored on the host and can be downloaded.

Logs: the service logs are on the hosts; the support bundle collects them; in CDP Public Cloud, the logs can be exported to object storage for retention.

Metrics: the control plane shows cluster and environment metrics; Cloudera Manager shows service and role metrics; the Prometheus exporter exposes metrics for external monitoring.

Alerts: the control plane shows environment and cluster status; Cloudera Manager alerts on health, thresholds, and events; the alerts can be integrated with the customer's monitoring.

Escalation: open a support case with the support bundle and the CRN; Cloudera support can access the control plane logs for the environment.

SLA: the control plane SLA is defined by Cloudera; the Data Hubs and Data Lake services are operated by the customer; the SLA for those services depends on the customer's configuration.

Documentation: the Cloudera documentation site has the CDP Public Cloud and Private Cloud guides; the API reference and CLI reference are available online.

CDP Public Cloud Data Hub and Data Lake upgrade: the Data Lake must be upgraded before the Data Hubs; the console offers a rolling upgrade; the upgrade can be paused or rolled back in some cases.

Upgrade prerequisites: back up the HMS and Ranger databases; verify the parcel is available; check custom configs and bootstrap scripts for compatibility.

Rolling upgrade: the control plane restarts roles in a controlled order; the cluster remains available during the upgrade; the upgrade can be paused or rolled back in some cases.

Rollback: the Data Hub upgrade can be rolled back by recreating the cluster with the previous definition; data in object storage is unaffected; ephemeral HDFS is lost.

Client configs: after the upgrade, download the new client configs from the cluster details; the old configs may not work with the new runtime.

Compatibility: custom UDFs, Spark jars, and NiFi processors must be compatible with the new runtime; test in a non-production environment first.

Version visibility: the runtime version is shown in the cluster details; the parcel version is shown in the parcel list; the two must match for a successful upgrade.

Support: Cloudera support provides upgrade guidance and can assist with failures; the support bundle includes the upgrade logs.

Documentation: the Cloudera documentation site has the upgrade guide for CDP Public Cloud and Private Cloud; the guide lists the supported upgrade paths.

CDP Public Cloud Data Hub and Data Lake security: authentication via the IdP (SSO/SAML/OIDC); authorization via Ranger; auditing via Ranger audit; encryption via TLS and object storage encryption.

IdP integration: the IdP is configured at the CDP account level; users and groups are synchronized; the control plane maps groups to roles; the IdP must support SAML 2.0 or OIDC.

Ranger policies: managed in the Data Lake Ranger; the policies apply to all Data Hubs and CDW; the policies can be resource-based or tag-based.

Knox: the gateway authenticates via the IdP and authorizes via Ranger; the Knox topology defines the proxied services; the Knox master secret stores credentials.

Audit: Ranger audit records are stored in Solr in the Data Lake; the audit can be exported to object storage or a SIEM; the audit retention is configurable.

Encryption: TLS between services and clients; object storage encryption at rest (SSE-S3, SSE-KMS, ADLS encryption, GCS encryption); HDFS TDE for HDFS on Data Hubs.

Key management: the cloud KMS keys are managed by the customer; the environment's IAM role must have permission to use the keys; losing the key makes the data unreadable.

Least privilege: grant Ranger policies to groups; use tag policies for broad classifications; review audit logs for denied access to tune policies.

Compliance: Atlas lineage plus Ranger audit provides a data-access trail; the lineage graph shows data flow, and the audit log shows who accessed what.