HCIA-Big Data key terms

Data whose volume, velocity, or variety exceeds what conventional relational systems handle cheaply, requiring distributed storage and parallel processing to extract value.

A common framing of big data characteristics: volume (scale), velocity (speed of arrival), variety (formats and schemas), and value (usefulness after analysis). Some lists add veracity.

A file system that spreads file blocks across many machines and replicates them, so storage capacity and read throughput grow with cluster size while tolerating node failure.

Hadoop Distributed File System: a write-once, append-oriented file system with a single namespace, large blocks, and rack-aware replication, designed for streaming reads over huge files.

The HDFS master service that holds the file system namespace and block-to-location mapping in memory, and coordinates client reads, writes, and block placement decisions.

An HDFS worker process that stores actual data blocks on local disk and serves read and write requests, sending periodic heartbeats and block reports to the NameNode.

The fixed-size unit into which HDFS splits files, typically 128 MB by default, so each block can be stored and processed independently on a different machine.

The number of copies kept for each data block, commonly three, which determines fault tolerance and read parallelism at the cost of usable storage capacity.

A placement policy that considers physical rack location when choosing replica targets, so copies land on different racks and survive a rack-level power or switch failure.

A helper that periodically merges the NameNode edit log with the fsimage checkpoint to keep restart time bounded; it is not a standby master and does not serve reads.

Yet Another Resource Negotiator: Hadoop's cluster resource manager, which allocates containers of CPU and memory to applications through a global scheduler and per-application masters.

The YARN master daemon that tracks cluster capacity, runs the pluggable scheduler, and grants container allocations to application masters across the cluster.

The YARN agent on each worker node that launches and monitors containers, reports resource usage to the ResourceManager, and enforces memory and CPU limits.

A YARN allocation unit granting an application a bounded amount of memory and virtual cores on one node, within which a task process such as a map runs.

A batch programming model that splits input into independent map tasks, then groups intermediate keys and processes them in reduce tasks, hiding distribution and failure handling.

The first user-supplied stage of a MapReduce job, applied record by record to input splits, emitting intermediate key-value pairs that the framework later partitions and sorts.

The user-supplied stage that receives all intermediate values sharing one key and aggregates them, typically emitting a smaller result set that is written to the output directory.

The framework phase between map and reduce that partitions map output by reducer, sorts it by key, and transfers it across the network so each reducer sees grouped keys.

An optional map-side mini-reducer that pre-aggregates intermediate values before network transfer, reducing shuffle traffic when the aggregation is associative and commutative.

A pluggable function that decides which reducer receives a given intermediate key, usually by hashing the key modulo the number of reduce tasks.

A logical chunk of input assigned to one map task, usually aligned with an HDFS block, so the task can read its data mostly from the local node.

Scheduling tasks on nodes that already hold the needed data, avoiding network transfer; it is why Hadoop moves computation to stored data rather than the reverse.

A data warehouse layer over Hadoop that stores schemas in a metastore and translates SQL-like HiveQL queries into distributed jobs, suited to batch analytics rather than transactions.

A relational database service holding Hive table and partition metadata such as columns, locations, and formats, shared by query engines that read the same warehouse tables.

Hive's SQL dialect, supporting familiar select, join, and group-by syntax plus extensions for partitioned and complex-typed data, executed as MapReduce or Tez jobs.

A physical subdivision of a Hive table, typically mapped to a directory named by column value, so queries filtering on that column skip irrelevant data.

A fixed number of hash-based files inside a Hive partition, used to spread data evenly and enable efficient map-side joins on the bucketed column.

A pattern where raw files are stored without validation and structure is applied only when queried, giving flexibility but pushing errors to read time.

A pattern where data is validated and conformed to a defined structure before storage, as in a traditional warehouse, giving predictable queries at the cost of upfront modeling.

A curated analytical store organized around subject areas and historical snapshots, loaded through ETL and optimized for aggregated reporting rather than per-row transactions.

A dimensional model with one central fact table holding measures and foreign keys, surrounded by denormalized dimension tables that describe each measure's context.

Extract, transform, load: the pipeline that pulls data from sources, cleans and reshapes it, then writes it into a warehouse or lake in query-ready form.

A central repository holding raw files in native formats at any scale, with structure applied at query time, often serving as landing zone and archive for many engines.

A distributed, column-family NoSQL store modeled on Google Bigtable, giving random real-time read and write access to billions of rows keyed by a sorted row key.

The unique, byte-ordered identifier of an HBase row; because rows are sorted by it, its design determines read patterns, scan efficiency, and hotspot avoidance.

A named group of HBase columns defined at table creation and stored together in separate files, so related columns should share a family for efficient reads.

A horizontal shard of an HBase table holding a contiguous row-key range; regions split as data grows and are assigned to RegionServers for serving requests.

The HBase worker process that hosts regions, serves reads and writes, and flushes in-memory updates to store files, coordinating with HDFS for block storage.

The HBase control process that assigns regions to RegionServers, balances load, and handles table schema changes such as create, alter, and split.

A principle stating that a distributed data system facing a network partition must choose between consistency and availability for that period, guiding NoSQL design trade-offs.

A broad class of non-relational databases, including key-value, column-family, document, and graph stores, that trade strict schemas and joins for scale and flexible models.

A commercial big data platform that packages Hadoop ecosystem components with management, security, and monitoring tooling for enterprise cluster deployment and operations.

The ability to host several independent tenants or projects on one cluster while isolating their resources, data, and permissions through queues, quotas, and access controls.

A ticket-based network authentication protocol used to prove the identity of users and services in a Hadoop cluster, preventing unauthenticated access to data and APIs.

Processing large bounded datasets as scheduled jobs, accepting higher latency in exchange for throughput and simplicity, as with MapReduce or Hive over stored files.

Continuous computation over unbounded event flows with low latency, updating results as records arrive rather than waiting for a complete stored dataset.