Heterogeneous Worker Nodes and a Balancer Run at Peak

The symptoms separate into a local disk failure and a cluster-wide load failure, so the order should reduce load before changing data placement.

1. Stop the running balancer. It is the only actor generating sustained cross-rack traffic and extra NameNode block-report work during peak. Killing it removes the largest avoidable contributor to 900 ms RPC latency without touching data.
2. Exclude the eight new nodes from YARN scheduling using an exclude file and a refresh. Executors then stop landing on volumes that are about to fill, so the SLA damage is contained. Do not decommission HDFS yet: decommissioning would trigger replication traffic and make the NameNode problem worse.
3. Rolling-restart NodeManagers with all four disks in yarn.nodemanager.local-dirs, one node at a time. This fixes the root cause of DiskChecker$DiskErrorException without an outage.
4. Only after latency is stable, change reserved space and resume balancing off-peak with a bandwidth cap.

Restarting all nodes first or decommissioning the new hosts would deepen the outage. Stopping the balancer and draining YARN are reversible and buy time.

dfs.datanode.du.reserved is enforced per volume, not per node. On an 8 TB disk, 10 GB is about 0.125% of capacity. A burst of 45 MB/s fills that margin in roughly four minutes, after which the volume is treated as unavailable and its blocks become effectively offline-exactly the failure pattern seen on the new nodes. Reserved space must absorb ingest burst plus compaction and replication. A practical floor is 2–5% of volume size, so about 200 GB on 8 TB disks and 100 GB on 4 TB disks. HDFS takes an absolute value, so set a conservative absolute value that is safe for the smallest volume class unless you manage per-node configuration groups.

Also:

• Put all four mount points in yarn.nodemanager.local-dirs; otherwise shuffle and spill concentrate on two disks regardless of reserved space.
• Set yarn.nodemanager.disk-health-checker.max-disk-utilization-per-disk-percentage to about 90 so a filling disk marks the node unhealthy before it fails hard.
• Run the balancer off-peak with -bandwidth 50 (MB/s) and -threshold 5, and cap dfs.balancer.max-size-to-move so one run cannot monopolize the network.

The reason to treat 8 TB and 4 TB differently is that the same absolute reserve gives very different time-to-full, and time-to-full is what determines whether a node survives a peak.

Validate in layers, because each change addresses a different failure mode.

First, check configuration propagation: confirm all eight NodeManagers report four local dirs and the new reserved value through JMX or the ResourceManager UI, and confirm the exclude file no longer lists them after re-admission. A single stale NodeManager would reproduce the disk error.

Second, test disk headroom under load: run a controlled Tez canary that writes shuffle data and spills to disk on one new node at a time while watching df on all four volumes. The pass criterion is that no volume crosses the health-checker threshold and no DiskErrorException appears.

Third, test NameNode stability: run the balancer off-peak with the new bandwidth cap and monitor RPC p99, block-report queue length, and dfsadmin -report for missing or under-replicated blocks. Compare against the 40 ms baseline, not against the degraded 900 ms.

Finally, verify the reporting SLA with a representative query set during a pre-peak window, and keep the balancer in a paused state until the first full peak has passed. Evidence should include before/after RPC percentiles and per-volume free-space curves; if either still trends toward exhaustion, re-exclude the nodes and repeat.