Hadoop Distributed File System (HDFS) Design and Development
Storage model
HDFS splits a file into blocks, 128 MB by default in FusionInsight HD, and stores each block as replicas, default 3. The NameNode holds the namespace and block-to-DataNode mapping in memory and persists namespace edits; DataNodes hold the actual blocks. A write creates a client-to-DataNode pipeline; packets flow forward and acknowledgements flow back. hflush pushes data to the pipeline but does not guarantee disk durability; hsync does.
HA and federation
HA uses two NameNodes, active and standby, a JournalNode quorum, and ZooKeeper Failover Controllers. Three JournalNodes tolerate one failure; five tolerate two. Federation separates namespaces so multiple NameNodes scale metadata. The Secondary NameNode is not an HA standby; it performs checkpointing in non-HA deployments.
Worked example
A 1 GB file with 128 MB blocks uses 8 blocks. With replication 3, the cluster stores 24 block replicas, about 3 GB raw. If one DataNode fails, the NameNode re-replicates blocks that fall below the target, but only after the failed node is declared dead following heartbeat expiry.
Traps that cost marks
Do not say the NameNode stores file data. Do not confuse block size with file size for small files: a 1 MB file still consumes one block metadata slot. Do not treat hflush as durable. Do not assume a Secondary NameNode enables automatic failover.