HCIP-Big Data Developer key terms

The HDFS master daemon that stores the namespace metadata (file-to-block mapping, permissions, replication settings) in memory and on disk, and coordinates block placement with DataNodes. It does not carry file payload data.

An HDFS worker daemon that stores actual block replicas on local disks, serves read and write requests to clients, and sends periodic heartbeats and block reports to the NameNode so placement and health can be tracked.

The fixed-size unit (default 128 MB in FusionInsight) into which HDFS splits files. Blocks are stored independently across DataNodes, enabling parallel reads, simple replication, and support for files far larger than any single disk.

The number of copies of each HDFS block kept in the cluster, set per file or directory. Higher values improve read parallelism and fault tolerance but consume more storage and network bandwidth during writes and re-replication.

A placement policy in which the NameNode distributes block replicas across different racks using a network-topology script, so a single rack or switch failure cannot lose all copies and cross-rack traffic stays controlled.

The append-only journal in which the NameNode records every namespace change, such as file creation, rename, or block allocation. On restart the namespace image is replayed with the EditLog to rebuild in-memory metadata.

A checkpoint snapshot of the HDFS namespace held on disk by the NameNode. Combined with replaying the EditLog, it allows fast restart without rescanning every DataNode block report.

A helper daemon that periodically merges the FsImage with the EditLog to produce a fresh checkpoint and truncates the journal. It is not a standby NameNode and provides no automatic failover.

The inefficiency where many files much smaller than one block each consume a NameNode metadata entry while wasting block and disk capacity, raising memory pressure on the NameNode and slowing scans. Archiving or concatenation mitigates it.

A Hadoop Archive file that packs many small HDFS files into a single larger container while preserving their paths, reducing NameNode memory use. Reading through a HAR adds overhead, so it suits cold or archival data.

The YARN master service that tracks cluster capacity, arbitrates resource requests from applications, and launches ApplicationMasters on NodeManagers. It enforces scheduling policy and queue configuration across tenants.

The YARN agent on each worker node that launches and monitors containers, reports resource usage to the ResourceManager, and enforces per-container memory and CPU limits. It also serves shuffle data for MapReduce jobs.

A per-application YARN process that negotiates containers from the ResourceManager, places tasks on NodeManagers, monitors their progress, and handles retries and failure reporting for one submitted job or application instance.

The unit of resource allocation in YARN: a bounded amount of memory and virtual CPU granted on one NodeManager, inside which a task or ApplicationMaster runs. Container size and count drive job concurrency.

A YARN scheduler that divides cluster resources into hierarchical queues with guaranteed capacities and configurable maximum limits, supporting elasticity and multi-tenant sharing. It is the default scheduler in FusionInsight clusters.

The MapReduce phase between map output and reduce input where records are partitioned by key, sorted, spilled to disk, and fetched across the network. Poorly tuned shuffle is often the dominant cost of a job.

An optional map-side mini-reducer that pre-aggregates output before it is written and shuffled, cutting network traffic and disk I/O. It is only safe when the reduce function is associative and commutative.

A fault-tolerance mechanism that starts duplicate attempts of slow tasks on other nodes and accepts the first to finish, mitigating stragglers caused by hardware degradation or uneven load. It costs extra resources.

An uneven distribution of records across partitions or keys, so a few tasks process far more data than others and dominate job runtime. Salting keys, pre-aggregation, or custom partitioners reduce it.

The service and relational database that stores Hive table definitions, columns, partitions, and locations. Engines such as Spark SQL and Flink can share it, making it the central catalog of a data warehouse.

A directory-based subdivision of a Hive table by column values, letting queries prune irrelevant data. Partition columns must be chosen with care, since too many small partitions overload the Metastore and NameNode.

A Hive technique that hashes rows into a fixed number of files by a chosen column, enabling efficient map-side joins and sampling when tables are bucketed identically. It complements rather than replaces partitioning.

A Hive table whose data files live outside its managed directory. Dropping the table removes only metadata and leaves the underlying HDFS data intact, which suits shared or imported datasets.

A join strategy in which the smaller table is broadcast into memory so each map task joins locally without a shuffle. It requires the small table to fit in memory or to be loaded via distributed cache.

A horizontal range of an HBase table's row-key space, served by one RegionServer. Regions split as they grow and move for load balancing, so row-key design directly shapes read and write distribution.

The HBase worker process that hosts regions, serves reads and writes through the block cache and memstore, and flushes data to HFiles. Its failure triggers region reassignment by the HMaster.

An in-memory write buffer inside each HBase store. Writes land in the memstore and write-ahead log first, then flush to immutable HFiles when a threshold is reached, which is why flush tuning affects write latency.

The immutable on-disk storage format of HBase, containing sorted key-value blocks with indexes and bloom filters. Compaction merges multiple HFiles to bound read amplification and reclaim deleted data.

The practice of choosing HBase row keys to avoid hotspots by distributing writes across regions, typically via salting, hashing, or reversed timestamps, while keeping scan patterns efficient for common queries.

Spark's foundational distributed dataset abstraction: an immutable, partitioned collection with a lineage graph that enables recomputation after failures. Transformations are lazy and execute only when an action runs.

Narrow dependencies let each parent partition feed one child partition, enabling pipelining within a stage. Wide dependencies such as groupByKey require a shuffle and form stage boundaries in the Spark DAG.

The directed acyclic graph of transformations Spark builds from a job. The scheduler splits it at shuffle boundaries into stages and tasks, which determines parallelism and where data must be exchanged.

A Spark SQL abstraction representing a distributed table with a schema. Catalyst optimizes its logical plan and Tungsten generates code, usually making it faster than hand-written RDD transformations on structured data.

Spark SQL's rule- and cost-based query optimizer, which rewrites logical plans through predicate pushdown, constant folding, and join reordering before physical execution planning.

Persisting an operator's state to durable storage so a streaming job can recover after failure. In Flink it is coordinated by barriers and configured through interval and state-backend settings.

A Flink mechanism that advances event time by signaling no events older than a timestamp should still arrive, allowing window operators to fire results despite out-of-order data. Late events can be dropped or side-output.

A bounded grouping of stream elements for aggregation, defined by assigner type (tumbling, sliding, session) and trigger. Window semantics determine whether results are emitted per window or continuously.

Event time is when an event actually occurred, recorded in the data; processing time is when it is handled. Event-time processing yields deterministic results on replay but must tolerate disorder and latency.

A delivery guarantee that each record affects the result exactly once even across failures, achieved in Flink via checkpointed state plus transactional or idempotent sinks, and in Kafka via transactions.

An ordered, append-only log segment of a Kafka topic that is the unit of parallelism and replication. Records with the same key land in the same partition, preserving per-key ordering for consumers.

A set of consumers that cooperatively consume a topic, with each partition assigned to at most one member. Adding members up to the partition count increases throughput; rebalancing redistributes assignments.

A monotonically increasing identifier for a record's position within a partition. Consumers commit offsets to track progress, and resetting them determines where consumption restarts after failure or replay.

The buffer between a Flume source and sink, holding events until they are written downstream. Memory channels are fast but volatile; file channels survive agent restarts at the cost of latency.

An independent Flume process composed of a source, channel, and sink that moves streaming events from an origin such as logs or a spool directory to a destination such as HDFS or Kafka.

A batch tool that transfers data between relational databases and Hadoop storage using generated MapReduce or YARN jobs, supporting incremental import via append or last-modified modes and direct fast paths.

The ticket-based authentication protocol used by FusionInsight to prove the identity of users and services. Clients obtain a ticket-granting ticket and service tickets before accessing HDFS, Hive, HBase, or YARN.

A file holding encrypted long-term keys for a Kerberos principal, letting services and scheduled jobs authenticate without an interactive password. It must be protected and rotated like any credential.

A centralized authorization rule in Apache Ranger granting or denying access to specific resources such as HDFS paths, Hive columns, or HBase tables for named users, groups, or roles, with audit logging.

A fine-grained permission list attached to a file or directory that grants named users or groups specific rights beyond the classic owner-group-other mode bits, evaluated alongside Kerberos identity.

Sharing one cluster among independent teams or projects with isolated resources, queues, storage quotas, and permissions, so one tenant's load or data cannot disrupt or expose another's.

An optimization that moves filter conditions as close to the data source as possible, so storage formats and external systems skip irrelevant rows and less data crosses the network.

Scheduling tasks on nodes that already hold the needed HDFS blocks, avoiding network transfer. Locality levels (node, rack, any) degrade gracefully when resources are scarce, trading speed for progress.