Triaging a Multi-Tenant FusionInsight Cluster After a Failed ETL

The most urgent risk is NameNode heap at 92%, because NameNode failure affects the entire cluster, including the 'bi' tenant. The likely root cause is the 1.2 billion small files: each file and block consumes NameNode heap (roughly 150 bytes per object), so the namespace itself is the problem. HDFS capacity at 86% is a symptom and a secondary risk; it is high but not yet critical. DN-17 at 98% is a local symptom of uneven block placement or a failing disk, not a cluster-wide root cause. The 1,800 pending containers in the 'etl' queue are a symptom of insufficient queue capacity and of the ETL slowing down under the NameNode and storage pressure.

Restarting the NameNode is dangerous at 92% heap because a restart must load the fsimage and replay edits; it may fail or take a long time. Adding 10 DataNodes does not fix NameNode heap. Treat small-file accumulation as the root cause, capacity and DN-17 as concurrent risks, and queue backlog as the visible failure.

Recover in a controlled order, using FusionInsight Manager to observe each step.

1. Protect the NameNode: check heap and GC trends in Manager. Avoid full fsck or namespace-heavy operations. If heap is still climbing, temporarily reduce nonessential ingestion and stop low-priority jobs.
2. Free HDFS capacity: delete expired temporary directories, old Hive scratch data, and obsolete partitions after confirming they are not needed. Empty the trash only after verification.
3. Address DN-17: check disk health and block distribution. If the disk is failing, decommission or isolate it; if it is only full, run the HDFS balancer with a bandwidth limit so it does not saturate the network during ETL recovery.
4. Restore the ETL queue: temporarily raise the 'etl' queue capacity or preempt lower-priority work, while preserving the 'bi' queue's 30% guarantee. Retry the ETL with reduced parallelism to lower NameNode and DataNode pressure.
5. Monitor: watch pending containers, NameNode heap, and HDFS capacity. Do not restart the NameNode unless Manager shows an imminent failure; if a restart is unavoidable, do it after freeing namespace pressure and with a rollback plan.

Small files: the 1.2 billion files must be reduced at the source. Change the streaming ingestion to batch or buffer writes into larger files (target 128–256 MB), and run a compaction job over existing small files using Hive or MapReduce. Set a retention policy and HDFS quotas to prevent recurrence. This directly lowers NameNode heap.

Namespace scalability: if file count remains high, evaluate HDFS Federation or NameNode HA with more heap, but treat federation as a structural change, not a quick fix. Confirm that management nodes have sufficient memory and that the NameNode heap is sized for the projected object count.

Tenancy: review YARN queue capacities in FusionInsight Manager. The 'etl' queue needs enough guaranteed capacity for the nightly load, while the 'bi' queue keeps its 30% guarantee. Use queue ACLs and scheduling policies so one tenant cannot starve another.

Capacity and balance: add DataNodes only after the small-file fix, or the new nodes will fill with small blocks. Run the balancer regularly, monitor disk skew, and set HDFS capacity alarms at 80% and 90%. Archive cold partitions to free space and align the ORC partition layout with the ETL write pattern.