Rack Failure, Safe Mode, and Replication Drift
At 02:10 a top-of-rack switch fails in rack B of a 24-node CDP Private Cloud Base HDFS cluster. Eight DataNodes become unreachable. The cluster stores 14 PB at 71% HDFS utilization with replication factor 3; rack awareness is enabled. The NameNode remains active, but hdfs fsck / reports 1.9 million under-replicated blocks and 1,200 blocks with no live replicas from legacy tables created before rack awareness. Hive and Impala queries on recent partitions fail with BlockMissingException. At 04:00 the switch is replaced and DataNodes restart. To reduce pressure during the outage, an operator had set dfs.replication=2 in hdfs-site.xml on the NameNodes and restarted them; the NameNode is now in safe mode with 99.1% of blocks reported. The business wants queries restored by 08:00. You must decide the recovery order and what not to do.
1. What is the safest recovery sequence from 04:00 to 08:00? Include safe mode checks, fsck, replication, and actions to avoid.
Read the worked answer
First confirm that all 24 DataNodes have re-registered and are sending block reports: use hdfs dfsadmin -report and check live/dead counts, not just process status. Check hdfs dfsadmin -safemode get. Capture the damage before changing anything: hdfs fsck / -list-corruptfileblocks and a scoped hdfs fsck /path -files -blocks -locations for the failing recent partitions. Save the list of 1,200 missing blocks; distinguish legacy tables with no live replica from blocks that are merely under-replicated.
Do not force the NameNode out of safe mode immediately. With 99.1% reported and a 99.9% threshold, it will not exit on its own, and -safemode forceExit before all live DataNodes report can start deletion or replication decisions on incomplete information. Wait for the safemode extension period and confirm block reports are still increasing. If the business requires read restoration and the missing blocks are isolated to known legacy tables, a controlled forceExit may be justified after snapshotting the corrupt-block list. If not, keep safe mode and restore from backup.
Restore dfs.replication to 3 before creating new files. Do not run the balancer while 1.9 million blocks are under-replicated; balancer traffic competes with replication and does not fix missing replicas. Do not delete files to clear fsck errors. Do not repeatedly restart the NameNodes; each restart re-enters safe mode and extends recovery. Do not set replication to 1 to make the percentage look healthy. Once out of safe mode, repair missing blocks from backup or accept documented loss, then run hdfs dfs -setrep -w 3 -R on scoped directories, starting with the most critical, while monitoring NameNode heap and DataNode network.
2. Why does changing `dfs.replication` back to 3 not restore existing files? How would you repair replication without saturating the cluster?
Read the worked answer
dfs.replication is a default applied when a file is created. The replication factor is stored per inode. Files created while the operator had dfs.replication=2 keep two replicas, and files that already existed keep whatever factor they had. Changing the configuration back to 3 affects new files only, so the cluster can appear correctly configured while millions of existing blocks remain under-replicated.
Repair with hdfs dfs -setrep -w 3 -R /path, but do not run it across all 14 PB at once. Scope it by data domain or date, start with critical tables, and omit -w if waiting on a huge tree would block operations; instead submit the setrep and monitor. Setting replication on an already healthy file is a no-op; on an under-replicated file it schedules replication work. Watch for NameNode RPC latency, DataNode network saturation, and the replication queue. Throttle by lowering dfs.namenode.replication.max-streams or adjusting DataNode transfer threads if the cluster is also serving queries. Verify with hdfs fsck /path -files -blocks and track under-replicated blocks to zero. Do not use the balancer as a substitute; balancing moves existing replicas, while setrep creates missing ones.
3. What controls would prevent this incident from recurring? Consider topology, replication, change management, and monitoring.
Read the worked answer
Verify rack awareness rather than assuming it. Run hdfs dfsadmin -printTopology and confirm the topology script maps 24 nodes across the intended racks. The 1,200 missing blocks came from legacy tables created before rack awareness, so identify remaining pre-rack-aware data and rewrite or re-replicate it so no block has all replicas in one rack. Ensure the topology script is available on every NameNode and DataNode restart path; a missing script can silently fall back to a single default rack.
Standardize replication at 3 for critical datasets and monitor per-directory exceptions. Do not allow an operator to change the global default during an incident without a documented exception and a follow-up setrep plan. Put the replication setting under configuration management so a NameNode restart does not silently change the default for new files. For legacy tables that must stay at 2, document the risk and keep backups.
Add monitoring for under-replicated blocks, missing blocks, safe mode state, live DataNode count by rack, NameNode heap, and RPC latency. Alert before HDFS reaches the capacity or heap headroom that makes recovery slow. Test a rack failure in a non-production cluster, including NameNode restart behavior and setrep throughput. Finally, define a runbook that says: confirm DataNode registration, capture fsck, do not force safe mode exit prematurely, do not run the balancer during recovery, restore replication with setrep, and verify block counts before closing the incident.