Cloudera CDP Data Analyst flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Hive metastore role

Shared catalog storing databases, tables, columns, partitions, and storage locations. Hive, Impala, and Spark all read it, so table definitions are visible across engines after metadata refresh.

Managed vs external table

Dropping a managed table deletes its data; dropping an external table deletes only metadata. External tables are safer when data is shared or written by other tools.

Parquet vs Avro

Parquet is columnar with compression and predicate pushdown, ideal for analytical scans. Avro is row-based and suited to write-heavy ingestion and schema evolution. Use Parquet/ORC for BI queries.

Bucketing purpose

CLUSTERED BY hashes rows into a fixed number of files. It speeds joins on the bucket key and enables efficient sampling, unlike partitioning which splits by directory value.

Impala INVALIDATE METADATA

Tells Impala to reload table and partition metadata from the metastore after DDL or new tables. Use REFRESH table for new files/partitions on an existing table.

ROW_NUMBER window function

Assigns a unique sequential number per partition ordered by a sort key. Common for deduplication: filter WHERE rn = 1 after ordering by updated_at DESC.

Hive LATERAL VIEW explode

Expands an ARRAY or MAP column into multiple rows. Syntax: LATERAL VIEW explode(arr) t AS item. Impala supports explode in the FROM clause instead.

COUNT(*) vs COUNT(col)

COUNT(*) counts all rows including those with NULLs. COUNT(col) counts only non-null values of that column, so results differ when NULLs are present.

COMPUTE STATS command

Impala command that gathers table and column statistics used by the optimizer for join order and distribution. Run after bulk loads or significant data changes.

Deduplication with ROW_NUMBER

Partition by business key, order by updated_at DESC, keep row number 1. Produces one current row per key and is portable across Hive, Impala, and Spark.

Small-file problem

Many tiny files increase scan overhead and metadata load in Hive and Impala. Fix by rewriting to larger files, Hive compaction, or Iceberg rewrite procedures.

SCD Type 2

Preserves history by adding a new row per change with effective and expiry dates plus a current flag. Queries filter on the current flag for latest state.

Kudu primary key

Every Kudu table requires a primary key that defines row sort order. Key values cannot be updated in place; choose keys matching common range-scan patterns.

Kudu hash and range partitioning

Combine RANGE partitioning by date for pruning with HASH partitioning on a high-cardinality column to spread writes across tablets and avoid hotspots.

HBase rowkey hotspotting

Sequential rowkeys concentrate writes on one RegionServer. Salt, hash-prefix, or reverse the key to distribute load evenly across regions.

Phoenix on HBase

SQL layer over HBase with secondary indexes and JDBC access. Preferred over raw scans for analyst-style queries; direct Get by rowkey remains fastest for point lookups.

Spark lazy evaluation

Transformations build a DAG; only actions like count, show, or write trigger execution. Cache reused DataFrames to avoid recomputing the same lineage.

Broadcast join in Spark

Replicates a small DataFrame to every executor to avoid a shuffle. Trigger with broadcast(df) or the broadcast threshold; effective when one side is small.

Cloudera Data Engineering jobs

CDE runs Spark jobs and sessions as containerized workloads on Kubernetes virtual clusters, with scheduling, dependencies, and run monitoring in the UI or CLI.

Iceberg MERGE INTO

Spark SQL statement that upserts rows into an Iceberg table by matching on a key, supporting updates and deletes without rewriting the whole table.

JDBC AuthMech for Impala

Driver parameter selecting authentication: 0 none, 1 Kerberos, 3 username, 4 username plus password. Match it to the cluster security mode to avoid connection failures.

Live query vs extract

Live BI queries hit Impala or Hive directly for fresh data but load the cluster. Extracts are cached in the BI engine, faster but stale. Choose by freshness need.

Knox gateway for BI

Single authenticated endpoint proxying to Impala and Hive on secured clusters. BI tools connect to the Knox URL rather than individual host and port pairs.

Pre-aggregated summary tables

Tables storing rolled-up metrics refreshed on a schedule. Dashboards query them instead of raw data, cutting scan volume and latency for repeated reporting.

Ranger deny precedence

Deny policies override allow policies in Apache Ranger. A user matching both an allow and a deny rule is denied access to the resource.

Ranger column masking

Policy that transforms column values at query time, such as showing only the last four digits. Applied consistently across Hive, Impala, and connected BI tools.

Atlas lineage

Apache Atlas records entities and processes to trace data flow from source tables to reports. Useful for impact analysis when schemas or metrics change.

Tag-based policies in SDX

Atlas tags such as PII drive Ranger policies, so classification propagates to access control without editing each table policy individually.

Stale statistics symptom

Missing or outdated table and column stats cause poor join order, wrong broadcast choices, and slow queries. Refresh with COMPUTE STATS or ANALYZE TABLE.

Partition pruning blockers

Functions or casts applied to the partition column, such as substr(dt,1,7), prevent pruning. Filter with constants or stable expressions on the raw partition column.

Impala EXPLAIN and SUMMARY

EXPLAIN shows the planned fragments, join strategies, and scans. SUMMARY reports runtime counters per operator, revealing skew and where time is spent.

Admission control resource pools

Impala queues and limits queries by resource pool. Separate pools or virtual warehouses for ETL and BI prevent heavy jobs from starving dashboards.