Cloudera CDP Data Analyst key terms

A massively parallel SQL engine that runs directly against data files in HDFS, object storage, or Kudu, without moving data into a separate store. It favors low-latency interactive queries and uses its own daemons rather than MapReduce.

A SQL-on-Hadoop layer that translates SQL into distributed jobs, historically MapReduce and now Tez or Spark. It stores table definitions in a metastore and is well suited to large batch transformations rather than sub-second interactivity.

The shared catalog service holding database, table, column, partition, and location metadata. Both Hive and Impala read from it, so a table created in one engine is generally visible to the other after a metadata refresh.

An Impala statement that discards cached table and partition metadata so the coordinator re-reads it from the metastore. Analysts run it after external tools add partitions or change schemas, because Impala otherwise serves stale definitions.

An Impala statement that reloads the file and block locations for one specific table, picking up newly written data files. It is lighter than a full metadata invalidation and is the usual fix after another engine loads partitions.

An Impala command that scans a table to gather column-level and partition-level statistics used by the cost-based optimizer. Missing or stale statistics commonly cause bad join orders and slow queries, so it is a routine maintenance step.

The query planner component that estimates row counts and selects join order and join strategy from table and column statistics. When statistics are absent it falls back to guesses, which often produce inefficient broadcast or shuffle decisions.

A join strategy that sends a copy of the small table to every worker node, avoiding a shuffle of the large table. It is efficient when one side is genuinely small and can be forced with a query hint.

A join strategy in which both sides are repartitioned by the join key across the cluster before matching rows. It scales to large-large joins but costs network and disk, so it is slower than broadcasting a small dimension table.

A filter Impala builds during execution from the values actually present on one side of a join, then pushes to the scanning side to skip irrelevant rows and blocks early. It improves join performance without changing query results.

Skipping entire partitions whose key values cannot satisfy a query's WHERE clause. It only works when predicates reference the partition columns directly, so filtering on a derived expression or wrapping the column in a function defeats it.

Pushing filter conditions as close to the data as possible, such as into the file scan or storage layer, so fewer rows are read and transferred. It reduces I/O and is a primary reason to keep filters simple and sargable.

A Hive or Impala load pattern where partition values are derived from the incoming data rather than named explicitly in the INSERT statement. It simplifies loading many partitions but can create huge numbers of small files if misused.

A condition where a table holds thousands of tiny files, forcing the engine to open each one and adding per-file overhead that dominates query time. Compaction or rewriting into fewer, larger files usually fixes it.

A background or manual process that merges many small data files, or merge-on-read delta files, into fewer larger ones. On Kudu it consolidates update deltas; on HDFS tables it reduces file count and improves scan throughput.

A file layout that stores values of each column together rather than row by row. It enables reading only referenced columns and achieves high compression, which is why formats like Parquet and ORC suit analytical scans.

An open-source columnar file format with per-column encoding, compression, and embedded statistics such as min and max values. These statistics let query engines skip row groups that cannot match a filter.

A columnar file format developed in the Hive ecosystem, offering stripe-level indexes, compression, and built-in support for ACID transactions in Hive tables. It is a common alternative to Parquet for Hive-centric workloads.

Guarantees that reads and writes behave atomically, consistently, in isolation, and durably. In Hive these require transactional tables backed by a suitable format, and they let concurrent readers see a consistent snapshot while writers append or update.

A distributed storage engine combining fast random reads and writes with columnar analytical scans. It suits tables that are both updated frequently and queried with SQL, filling the gap between slow HDFS files and key-value stores.

In Kudu, one or more columns that uniquely identify a row and define its physical ordering. Key design drives scan efficiency, so leading key columns should match common range filters and avoid heavy hashing.

Concentration of write load on a small number of servers because keys arrive in monotonic order, such as timestamps or sequential IDs. Hash or randomize the leading key component to spread writes evenly.

A distributed, sparse, sorted key-value store modeled on Google Bigtable, accessed by row key rather than SQL. It serves very large tables with fast single-row lookups but offers limited analytical scan performance.

The practice of choosing HBase row keys so that reads and writes distribute evenly and support the access patterns in use. Keys are sorted lexicographically, so prefix choices determine locality and scan range.

A general-purpose distributed processing engine that keeps intermediate data in memory and exposes APIs in Scala, Python, SQL, and Java. On CDP it handles ETL, machine learning, and large-scale SQL beyond what interactive engines do well.

The Spark module that runs SQL and DataFrame operations through the Catalyst optimizer and Tungsten execution engine. It lets analysts reuse Hive tables and metadata while gaining Spark's scale and language flexibility.

Spark SQL's rule- and cost-based planner that rewrites logical plans, pushes filters and projections down, and chooses join strategies. Reading its explain output is the standard way to diagnose a slow Spark query.

The redistribution of data across partitions by key, required by wide transformations such as groupBy and join. It writes to disk and crosses the network, making it the most common bottleneck in Spark jobs.

Uneven distribution of rows across partitions, leaving a few tasks far larger than the rest and dominating job runtime. Salting keys, repartitioning, or broadcasting the small side are typical mitigations.

A service on CDP for building, scheduling, and operating Spark jobs as containerized workloads with versioned job definitions, dependency management, and monitoring. It replaces hand-managed spark-submit scripts with managed pipelines.

A workflow scheduler that expresses pipelines as directed acyclic graphs of tasks with dependencies, retries, and scheduling. It is often paired with CDE to orchestrate multi-step data pipelines around Spark jobs.

A centralized authorization and audit framework that enforces fine-grained policies on Hive, HDFS, HBase, and other services. Policies can restrict rows and mask or filter columns, and access decisions are logged for audit.

A Ranger policy capability that rewrites or restricts which rows a user can see, such as limiting a regional analyst to their own region. The filter applies regardless of how the query is written.

A Ranger policy that replaces sensitive column values with masked output, such as showing only the last four digits of an identifier, for users lacking full access. The underlying data is unchanged; only query results are altered.

A metadata and governance service that catalogs datasets, tracks lineage between inputs and outputs, and applies classification tags that can drive access policies. It answers where data came from and what it contains.

The recorded dependency chain showing which datasets and processes produced a given table or column. Analysts use it to trace the source of a suspect number and to assess the blast radius before changing an upstream job.

Cloudera's shared data experience: a set of security and governance services, including Ranger and Atlas, managed once at the environment level and inherited by workloads. It avoids reconfiguring policies separately for each cluster or engine.

A web-based notebook that runs SQL and Spark snippets with inline charts and shared paragraphs. Analysts use it for exploratory work and to publish dashboards without leaving the CDP environment.

An open-source business intelligence tool for building charts and dashboards from SQL-speaking databases. On CDP it connects to Hive or Impala through a driver and supports saved queries, alerts, and row-level security.