Hadoop and MapReduce Fundamentals
This domain tests HDFS architecture, YARN resource management, and the MapReduce execution pipeline. You should know which daemon does what, how blocks and splits differ, and where shuffle, sort, combiner, and partitioner sit.
HDFS and YARN
- HDFS: the NameNode holds metadata; DataNodes hold blocks. Default block size is 128 MB in Hadoop 2.x/3.x, with replication 3 and rack awareness. Writes use a pipeline.
- The SecondaryNameNode checkpoints edit logs; it is not a hot standby. HA uses a standby NameNode with JournalNodes.
- YARN: ResourceManager allocates, NodeManager runs containers, and each application has an ApplicationMaster. A container is the resource unit.
MapReduce pipeline
Input splits → map → spill/partition/sort → shuffle → reduce. The combiner is an optional local aggregation and may not run. The partitioner maps each key to a reducer, usually by hash.
Worked example
A 1 GB file with 128 MB blocks yields 8 blocks and, by default, 8 splits and 8 map tasks. With 3 reducers, the partitioner assigns each map output key to reducer 0, 1, or 2 by hash; shuffle sorts by key before reduce. A combiner that sums per-mapper counts cuts network traffic, but the final sum still occurs in reduce. A map-only job sets reducers to 0.
Traps
- Block size is physical storage; split size is logical input. They are often equal but not guaranteed.
- The combiner is an optimization, not a guaranteed stage; never put final business logic only there.
- The NameNode does not stream file data; clients read blocks from DataNodes.