HCIA-Big Data flashcards

32 cards. Read the term and the definition here. The deck at the top of this page is optional.

Volume, velocity, variety

The core characteristics of big data: volume is the size of data, velocity is the speed of generation and processing, and variety is the mix of structured, semi-structured and unstructured formats.

Batch versus stream processing

Batch processing handles large bounded datasets with high latency, while stream processing handles continuous unbounded data with low latency for near real-time results.

Sqoop versus Flume

Sqoop transfers structured data between relational databases and HDFS or Hive in bulk. Flume collects and aggregates streaming log and event files into HDFS.

CAP theorem

A distributed system can guarantee at most two of consistency, availability and partition tolerance. Because partitions are unavoidable, systems trade consistency against availability.

Move computation to data

A distributed computing principle: run processing where the data is stored instead of transferring data to the compute node, since network movement dominates cost.

Hadoop ecosystem roles

HDFS stores data, YARN allocates resources, MapReduce computes, Hive provides SQL over HDFS, HBase offers random access, ZooKeeper coordinates, and Kafka buffers streaming data.

Semi-structured data

Data that carries self-describing tags but no rigid table schema, such as JSON, XML and log lines, sitting between structured relational data and unstructured media.

HDFS NameNode and DataNode

The NameNode holds metadata such as the namespace and block mapping. DataNodes store the actual data blocks and report block status to the NameNode.

HDFS block size default

The default block size is 128 MB in Hadoop 2.x and later, and 64 MB in Hadoop 1.x. Files smaller than one block do not fill a block on disk.

Secondary NameNode role

The Secondary NameNode is not a hot standby. It periodically merges the edit log with the fsimage to produce a checkpoint, reducing NameNode restart time.

Rack-aware replica placement

With replication factor three, HDFS places two replicas on the local rack and one on a remote rack, so the cluster survives the loss of a single rack.

MapReduce shuffle phase

Between map and reduce, map output is buffered, spilled to disk at the threshold, partitioned by reducer, and merged; reducers fetch their partitions over the network.

Combiner function

A map-side pre-aggregation step that reduces shuffle traffic. It is valid only for commutative and associative operations such as sum or max, not average.

YARN core components

The ResourceManager allocates cluster resources, the NodeManager manages containers on one node, and the ApplicationMaster negotiates resources for a single application.

Number of map tasks

The number of map tasks equals the number of input splits, which is derived from file size and split size and is not always equal to the block count.

Hive internal versus external table

Dropping an internal managed table deletes metadata and data. Dropping an external table deletes only metadata, leaving the data in HDFS untouched.

Hive partitioning

Partitioning stores rows in separate directories by column value, so queries that filter on the partition column scan only the relevant directories instead of the whole table.

Hive bucketing

Bucketing hashes a column into a fixed number of files using CLUSTERED BY, enabling efficient sampling and map-side joins on the bucketed column.

WHERE versus HAVING

WHERE filters individual rows before grouping and aggregation. HAVING filters aggregated groups after GROUP BY, so it can reference aggregate functions.

ORDER BY versus SORT BY

ORDER BY produces one global sorted result using a single reducer. SORT BY sorts data within each reducer, so output order is only guaranteed per reducer.

ORC and Parquet formats

Columnar file formats with compression and predicate pushdown. ORC is optimized for Hive, while Parquet is widely used across Hive, Spark and other engines.

Data warehouse layering

ODS holds raw source data, DWD holds cleaned detail data, DWS holds summarized aggregates, and ADS serves reports and applications to end users.

HBase data model

HBase is a sparse, sorted, multidimensional map: Table, RowKey, ColumnFamily, Column Qualifier, Timestamp and Value. Cells are versioned by timestamp.

HBase RowKey design

RowKeys are sorted lexicographically, so short, well-distributed keys enable fast gets and range scans. Sequential keys concentrate writes on one region and cause hotspots.

HBase Region and RegionServer

A table is split horizontally into Regions, each served by one RegionServer. The HMaster assigns regions, balances load and handles schema changes but is not on the read/write path.

HBase WAL and MemStore

Writes are first appended to the Write-Ahead Log and buffered in MemStore. When MemStore reaches its flush threshold, it is flushed to HFiles on HDFS.

HBase major compaction

Major compaction merges all HFiles of a region into one file and removes deleted and expired versions, reclaiming space and improving read performance.

HBase ZooKeeper role

ZooKeeper stores the hbase:meta location and cluster state and elects the active HMaster, helping clients and servers find the live cluster components.

NoSQL database categories

Main types are key-value such as Redis, column family such as HBase and Cassandra, document such as MongoDB, and graph such as Neo4j, each suited to different access patterns.

FusionInsight Manager functions

The management console installs and configures services, monitors cluster health and alarms, manages users and roles, and supports scaling, patching and tenant resource isolation.

HDFS safe mode

In safe mode the NameNode is read-only while it waits for DataNodes to report blocks. It is entered at startup or when blocks are under-replicated.

Scaling and decommissioning nodes

Adding a DataNode triggers HDFS rebalancing and adding a RegionServer triggers region reassignment. Decommissioning moves a node's data away before removal to prevent loss.