MapReduce and YARN Application Development
YARN roles
ResourceManager schedules cluster resources; NodeManager launches and monitors containers on each node; the ApplicationMaster runs inside a container and negotiates resources for a job. A MapReduce job's AM requests map and reduce containers, monitors tasks, and retries failures. Capacity Scheduler divides queues with guaranteed capacities and limits; Fair Scheduler shares resources more dynamically.
MapReduce flow
Input splits create map tasks. Map output is buffered, spilled, partitioned by reducer, and sorted. The shuffle fetches map output to reducers; reduce runs the user function. A combiner is an optional mini-reducer that cuts shuffle data but may run zero, one, or many times. The number of reducers sets the number of output part files.
Worked example
A 1.28 GB input with 128 MB blocks yields 10 map tasks. Set 3 reducers: each map partitions records into 3 buckets by hash of key, and each reducer fetches its bucket. For word count, a combiner can aggregate counts before shuffle, reducing network traffic, but the final sum must still occur in the reducer.
Traps that cost marks
Do not assume a combiner replaces the reducer. Do not set reducers to zero when global aggregation is needed unless map output is already complete. Speculative execution duplicates a slow task, not input data. Small jobs may use uber mode, running map and reduce in the AM container.