HCIA-Big Data cheat sheet

The Vs of big data: volume, velocity, variety, veracity, value; volume is data size, velocity is the rate of generation and processing, variety covers structured, semi-structured and unstructured data.

Data volume units scale by 1024: 1 KB = 1024 B, 1 MB = 1024 KB, 1 GB = 1024 MB, 1 TB = 1024 GB, 1 PB = 1024 TB, 1 EB = 1024 PB.

Processing modes: batch processing handles large bounded datasets with high latency; stream processing handles continuous unbounded data with low latency; interactive/OLAP query serves ad hoc analysis.

Hadoop ecosystem roles: HDFS stores data, YARN allocates resources, MapReduce computes, Hive provides SQL over HDFS, HBase provides random read/write on HDFS, ZooKeeper coordinates, Flume and Sqoop move data, Kafka buffers streams.

Data movement decision rule: relational database to HDFS/Hive uses Sqoop; log files to HDFS uses Flume; distributed message queue for real-time pipelines uses Kafka.

Structured data fits a predefined schema (relational tables); semi-structured data carries self-describing tags (JSON, XML, log lines); unstructured data has no fixed schema (images, video, plain text).

Distributed computing principle: move computation to data rather than data to computation, because data movement across the network dominates cost.

CAP theorem: a distributed system can simultaneously guarantee at most two of consistency, availability, and partition tolerance; network partitions are unavoidable, so real systems trade consistency against availability.

HDFS architecture: one active NameNode holds metadata (namespace, block mapping); multiple DataNodes store blocks; a Standby NameNode plus JournalNodes provide HA; a Secondary NameNode performs checkpointing, not hot standby.

Default block size in Hadoop 2.x and later is 128 MB; in Hadoop 1.x it is 64 MB. A file smaller than one block does not occupy a full block on disk.

Replication factor default is 3. Rack-aware placement puts one replica on a remote rack and two on the local rack, so the cluster tolerates loss of one rack.

HDFS is write-once, append-oriented; it favors sequential reads over random writes and is not suitable for many small files because NameNode memory is consumed per file, block, and directory entry.

MapReduce phases: input split, map, shuffle and sort, reduce, output. The number of map tasks equals the number of input splits, not the number of blocks necessarily.

Shuffle details: map output is buffered in a circular memory buffer (default 100 MB), spilled to disk at the spill threshold (default 0.8), partitioned by reducer, then merged; the reducer fetches its partitions.

Combiner runs on the map side after map and before shuffle to pre-aggregate and reduce network traffic; it is valid only when the operation is commutative and associative, such as sum or max, not average.

YARN components: ResourceManager allocates cluster resources, NodeManager manages a node's containers, ApplicationMaster negotiates resources for one application, and the container is the unit of resource (memory and vCPU).

Hadoop 1.x uses JobTracker and TaskTracker; Hadoop 2.x replaces them with YARN's ResourceManager, NodeManager and ApplicationMaster, removing the single JobTracker bottleneck.

Hive stores metadata in a metastore (typically a relational database such as MySQL) and data in HDFS; it translates HQL into MapReduce, Tez, or Spark jobs.

Internal (managed) table: dropping the table deletes both metadata and data. External table: dropping the table deletes only metadata, data remains in HDFS.

Partitioning splits data into directories by column value and prunes scanned data; static partition requires specifying the value on load, dynamic partition derives it from the data and needs hive.exec.dynamic.partition=true.

Bucketing hashes a column into a fixed number of files (CLUSTERED BY ... INTO n BUCKETS) and supports efficient sampling and map-side joins.

Order of execution: FROM and JOIN, WHERE, GROUP BY, HAVING, SELECT, ORDER BY, LIMIT. WHERE filters rows before aggregation; HAVING filters after aggregation.

Sort types: ORDER BY produces a single global sort with one reducer; SORT BY sorts within each reducer; DISTRIBUTE BY controls which reducer receives which rows; CLUSTER BY equals DISTRIBUTE BY plus SORT BY.

File formats: TextFile is readable but uncompressed by default; ORC and Parquet are columnar with compression and predicate pushdown; ORC performs best with Hive, Parquet is common across engines.

Data warehouse layering: ODS holds raw source data, DWD holds cleaned detail data, DWS holds summarized/aggregated data, ADS or APP serves application and report output.

Join types: inner join returns matching rows; left outer join keeps all left rows; full outer join keeps all rows from both sides; map-side join works when one table is small enough to fit in memory.

HBase is a column-family-oriented, distributed, sparse, sorted map on HDFS, modeled as Table > RowKey > ColumnFamily > Column Qualifier > Timestamp > Value (Cell).

RowKey design rules: keep it short, distribute it to avoid hotspots, and exploit its lexicographic sort for range scans; monotonically increasing RowKeys concentrate writes on one RegionServer.

A table is split horizontally into Regions; each Region is served by one RegionServer. HMaster assigns regions, balances load, and handles schema changes; it is not on the read/write data path.

ZooKeeper in HBase stores the hbase:meta location and cluster state and elects the HMaster; -ROOT- was removed in HBase 0.96 and later, leaving hbase:meta as the single meta table.

Writes go to the Write-Ahead Log (WAL) and MemStore; when MemStore reaches the flush threshold it flushes to HFile. Reads merge MemStore and HFiles; a Bloom filter and block cache reduce disk reads.

Compaction: minor compaction merges adjacent small HFiles; major compaction merges all HFiles of a region into one and removes deleted and expired versions.

Versioning: each cell can hold multiple timestamped versions; the default max versions is 1, and TTL (time to live) expires cells by age.

NoSQL categories: key-value (Redis), column family (HBase, Cassandra), document (MongoDB), graph (Neo4j). HBase suits sparse, large-scale random read/write and range scans, not multi-table joins or transactions across rows.

RowKey access patterns: get by RowKey is fastest; range scan uses the sorted order; a full table scan is slow. Salt, hash, or reverse the RowKey to spread writes when keys are sequential.

FusionInsight Manager is the cluster management console: it installs and configures services, monitors health and alarms, manages users and roles, and performs scaling and patching.

FusionInsight components include HDFS, YARN, MapReduce, Hive, HBase, Spark, Kafka, Flink, ZooKeeper, Flume, Loader (bulk data movement), and Elasticsearch; roles are assigned per host.

Scaling: adding a DataNode triggers HDFS data rebalancing; adding a RegionServer triggers HBase region reassignment. Decommissioning a node moves its data before removal to avoid data loss.

Alarm handling order: read the alarm name and severity, check the affected service and host, inspect logs and metrics (CPU, memory, disk, network, JVM heap), then clear the root cause before clearing the alarm.

Capacity rules: keep HDFS usage below roughly 80-85% to leave room for temporary files and compaction; monitor DataNode disk balance and NameNode heap as the metadata grows.

Backup and recovery: NameNode metadata is protected by checkpointing and HA journal; HBase uses snapshots and WAL replay; Hive data backups depend on HDFS replication and distcp.

Security: Kerberos authentication plus role-based authorization; tenant resources are isolated with YARN queues and HDFS directory quotas and permissions.

Troubleshooting signals: a DataNode in safe mode means blocks are under-replicated; HBase RegionServer down shifts regions to other servers; a full NameNode heap causes slow metadata operations and needs more heap or fewer small files.