NameNode Heap Pressure from 3.2 Million Small Files per Day

The NameNode keeps an in-memory object for every file inode and every block, plus replication bookkeeping, leases and snapshot references. A widely used sizing rule is about 150 bytes per object.

Here the live object count is roughly 90M files + 95M blocks = 185M objects, i.e. about 28 GB of live heap before replication maps, RPC buffers, the lease manager and GC headroom. On a 64 GB heap that is a ~45% live set. With G1, a live set that large means increasingly long mixed/full GCs; while the JVM is paused the NameNode cannot drain its call queue, so RPC queue length climbs, ls latency grows, fsimage load takes tens of minutes and file descriptors pile up. The trajectory is the real problem: 3.2M new files/day x ~150 bytes is ~480 MB/day of permanent heap growth, so even without traffic growth the node crosses a safe heap within months.

dfs.blocksize controls only how a single file is chunked. An 8 KB file consumes exactly one block object regardless of whether the block size is 128 MB or 256 MB, and at replication 3 it also creates three replica entries. So changing block size alters neither the inode count nor the block count for this workload. Only reducing the number of files (or adding NameNodes via federation) changes the object count. Block size tuning matters for large files and for map task counts, not for small-file metadata pressure.

Order the work as: stop creating objects, then remove objects, then tune, then scale.

1. Fix at the source (highest impact, days of effort). Do not let 8 KB events land as individual HDFS files. Buffer them: Kafka topic -> Flink or Spark Structured Streaming job with a 128-256 MB rolling file writer producing Parquet, or Flume with hdfs.rollSize plus an aggregation interceptor. 3.2M files/day at 8 KB collapses to roughly 100-250 files/day at 128 MB, removing essentially all future heap growth.

2. Compact the existing backlog (weeks, but batch). For each closed directory, run a Spark or Hive INSERT OVERWRITE that reads the small files and rewrites them at 128 MB target size, verify row counts, then delete the originals with a trash retention window. This directly removes inodes and blocks.

3. Make the query layer tolerate what remains. Use CombineHiveInputFormat or mapreduce.input.fileinputformat.split.maxsize=268435456 so small files do not become one task each; avoid full partition listings in Hive and avoid MSCK REPAIR on huge trees.

4. NameNode operations. Raise ulimit -n to 65535, set dfs.namenode.handler.count near 20*log(120) ~ 100, use G1 with a pause target, and monitor heap, object count and GC time in FusionInsight Manager.

5. Scale out. Move cold, immutable data to OBS via DistCp, and plan HDFS federation or Router-based federation so object count is partitioned across NameNodes.

HDFS archives (HAR) are a legitimate step-2 option for cold data, but see the caveat in the next task.

A HAR archive is a read-only snapshot of the directory at creation time. Files written into the source directory afterwards are outside the archive, so a still-active pipeline keeps appending to a path that queries no longer read through the har:// view. The result is either silent data loss or two divergent copies, depending on whether the original files are deleted. Neither outcome is acceptable in a compliance-bound platform.

Second, HAR reduces NameNode objects, not DataNode storage: the archived files are packed into large part files, so block and replica bytes remain. Reading through har:// is also slower and requires every downstream engine to handle the archive filesystem explicitly; MapReduce needs HarFileSystem, and Spark/Flink jobs usually need path changes. Third, archiving a directory with millions of files is a NameNode-heavy metadata operation that can itself cause an RPC spike if run during business hours.

The governing constraint is therefore: archive only immutable, closed data (for example dt older than 90 days) whose writers have stopped, run it off-peak in bounded batches, record the archived path in the metastore or catalog, and keep the original data until a read-back verification query succeeds. For active directories, fix the writer instead of archiving.