I85PREP
⌘ K
HCIP-Big Data Developer study materials
HCIP-Big Data Developer study resources

HCIP-Big Data Developer practice questions

One question at a time. Build your HCIP-Big Data Developer understanding.

First 50 of 200 questions. Each one lists the choices, the correct answer, and an explanation. The checker at the top of this page saves progress in this browser when it is available.

Continue in the practice tool above to load more questions, 50 at a time.

Question 1Big Data Architecture and Huawei FusionInsight Overview

In the FusionInsight big data platform, which component is primarily responsible for distributed offline batch processing using a MapReduce-style programming model?

  1. Spark Correct answer.
  2. Hive
  3. HDFS
  4. YARN

Explanation

Spark provides a distributed in-memory computing engine that supports batch processing with an RDD/DataFrame programming model and is the main offline batch engine promoted on FusionInsight. HDFS is storage, YARN is resource scheduling, and Hive is a SQL-on-Hadoop data warehouse layer rather than the execution engine itself.

Question 2Big Data Architecture and Huawei FusionInsight Overview

Which FusionInsight component acts as the centralized resource management and scheduling framework that allocates CPU and memory containers to distributed applications?

  1. Kafka
  2. ZooKeeper
  3. Flume
  4. YARN Correct answer.

Explanation

YARN separates resource management from computation: the ResourceManager tracks cluster resources and NodeManagers launch containers for applications such as Spark or MapReduce. The other listed components handle messaging, coordination, and data collection respectively.

Question 3Big Data Architecture and Huawei FusionInsight Overview

A FusionInsight cluster stores very large files reliably across many commodity servers. Which HDFS characteristic directly provides fault tolerance when a DataNode fails?

  1. Clients cache the entire file locally
  2. The NameNode stores all file contents in memory
  3. Data is compressed before storage
  4. Multiple block replicas are placed on different nodes Correct answer.

Explanation

HDFS splits files into blocks and stores multiple replicas on different DataNodes, so a lost replica can be served from another node and re-replicated automatically. Compression reduces size but does not provide redundancy, and the NameNode holds metadata rather than file data.

Question 4Big Data Architecture and Huawei FusionInsight Overview

In the FusionInsight architecture, which component provides a distributed, highly available coordination service used for configuration management and leader election?

  1. Sqoop
  2. Oozie
  3. ZooKeeper Correct answer.
  4. HBase

Explanation

ZooKeeper is the coordination service that maintains a hierarchical namespace of znodes and supports leader election, configuration synchronization, and distributed locks. HBase is a columnar NoSQL database, Sqoop transfers relational data, and Oozie orchestrates workflows.

Question 5Big Data Architecture and Huawei FusionInsight Overview

Which FusionInsight component is a distributed column-oriented NoSQL database designed for random real-time read and write access to very large tables?

  1. Flume
  2. HBase Correct answer.
  3. Spark SQL
  4. Hive

Explanation

HBase models data as a sparse, distributed, multidimensional sorted map and serves random point and range reads at low latency on top of HDFS. Hive and Spark SQL are analytical query layers with higher latency, and Flume is a collection agent.

Question 6Big Data Architecture and Huawei FusionInsight Overview

A team needs to ingest continuously generated application logs into HDFS with minimal custom code. Which FusionInsight component is purpose-built for reliable streaming collection and aggregation of log data?

  1. YARN
  2. Flume Correct answer.
  3. Hive
  4. Hue

Explanation

Flume uses sources, channels, and sinks to collect, buffer, and deliver streaming event data such as logs into HDFS or Kafka. Hive queries stored data, YARN schedules resources, and Hue is a web UI.

Question 7Big Data Architecture and Huawei FusionInsight Overview

An engineer wants to copy a large table from a relational database into HDFS in parallel using generated MapReduce jobs. Which FusionInsight tool is designed for this task?

  1. Sqoop Correct answer.
  2. HBase
  3. ZooKeeper
  4. Oozie

Explanation

Sqoop imports relational data into HDFS or Hive and exports it back, splitting the transfer into parallel map tasks based on a split key. Oozie schedules jobs, ZooKeeper coordinates, and HBase stores data rather than importing it from RDBMS.

Question 8Big Data Architecture and Huawei FusionInsight Overview

In YARN, what is the role of the ApplicationMaster for a running application?

  1. It replaces the NameNode during failover
  2. It compiles user code into bytecode before submission
  3. It stores HDFS block metadata for the application
  4. It negotiates resources with the ResourceManager and manages task containers Correct answer.

Explanation

Each application gets an ApplicationMaster that requests containers from the ResourceManager, launches tasks on NodeManagers, monitors progress, and handles failures. It does not manage storage metadata, perform NameNode failover, or compile code.

Question 9Big Data Architecture and Huawei FusionInsight Overview

A FusionInsight HDFS cluster has a block size of 128 MB. A file of 300 MB is written with replication factor 3. How many block replicas in total are stored across the cluster?

  1. 12
  2. 6
  3. 9 Correct answer.
  4. 3

Explanation

The 300 MB file spans three blocks (128 + 128 + 44 MB). With a replication factor of 3, each block has three replicas, giving 3 blocks x 3 replicas = 9 replicas. Replication multiplies the block count rather than the file count.

Question 10Big Data Architecture and Huawei FusionInsight Overview

Which statement best describes how HBase organizes data physically for scalable writes?

  1. Tables are split into regions distributed across RegionServers, with data persisted in HFiles on HDFS Correct answer.
  2. All rows are held in a single in-memory table on the Master node
  3. Data is stored only in ZooKeeper znodes for durability
  4. Each row is stored as a separate file on the NameNode

Explanation

HBase tables are horizontally partitioned into regions assigned to RegionServers, and flushed data is written as HFiles into HDFS. The HBase Master handles metadata and region assignment but does not store table data, and ZooKeeper only coordinates.

Question 11Big Data Architecture and Huawei FusionInsight Overview

A FusionInsight cluster runs Spark jobs on YARN. Which deployment mode runs the Spark driver inside a YARN container on the cluster rather than on the submitting client?

  1. Cluster mode Correct answer.
  2. Standalone mode
  3. Client mode
  4. Local mode

Explanation

In cluster mode the driver runs in an ApplicationMaster container on the cluster, so the client can disconnect after submission. Client mode keeps the driver on the submitting machine, while local and standalone modes do not use YARN for the driver.

Question 12Big Data Architecture and Huawei FusionInsight Overview

An analyst needs SQL-like querying over HDFS data with a metastore that tracks table schemas, and the underlying execution should be handled by a distributed engine. Which FusionInsight component fills this role?

  1. ZooKeeper
  2. Hive Correct answer.
  3. Flume
  4. Sqoop

Explanation

Hive provides a SQL interface over HDFS-backed tables and stores schema definitions in a metastore, delegating execution to an engine such as Spark or MapReduce. The other components handle ingestion, coordination, or transfer.

Question 13Big Data Architecture and Huawei FusionInsight Overview

FusionInsight supports multi-tenant cluster usage. Which mechanism is used to isolate resources such as CPU and memory among different departments sharing one cluster?

  1. HBase column family compression
  2. YARN queue-based scheduling with capacity or fair policies Correct answer.
  3. ZooKeeper znode versioning
  4. HDFS block replication

Explanation

YARN queues configured with capacity or fair scheduling policies partition cluster resources among tenants, and FusionInsight's tenant feature maps users and queues to enforce isolation. Replication, znode versioning, and compression address durability, coordination, and storage efficiency, not resource isolation.

Question 14Big Data Architecture and Huawei FusionInsight Overview

In an HDFS cluster with NameNode high availability enabled, which component ensures that only one NameNode is active at a time?

  1. The YARN ResourceManager
  2. ZooKeeper-based failover controller with a shared edit log Correct answer.
  3. The DataNode heartbeat counter
  4. The Hive metastore

Explanation

HA pairs an active and a standby NameNode that share an edit log on JournalNodes, while ZooKeeper-based failover controllers elect the active node and prevent split brain. DataNodes report to both, but heartbeats do not elect the active node.

Question 15Big Data Architecture and Huawei FusionInsight Overview

A FusionInsight deployment needs a message bus that buffers high-throughput event streams and decouples producers from multiple downstream consumers. Which component should be selected?

  1. HBase
  2. Hive
  3. Oozie
  4. Kafka Correct answer.

Explanation

Kafka is a distributed, partitioned, replicated commit log that lets producers publish streams and multiple consumer groups read them independently with retention-based buffering. Hive, Oozie, and HBase serve analytics, orchestration, and random storage roles instead.

Question 16Big Data Architecture and Huawei FusionInsight Overview

Which FusionInsight component coordinates a directed acyclic graph of dependent jobs, such as running a Hive load only after a Sqoop import succeeds?

  1. Oozie Correct answer.
  2. Spark SQL
  3. Flume
  4. ZooKeeper

Explanation

Oozie defines workflows and coordinators as DAGs of actions with dependencies, so downstream actions start only after upstream ones complete successfully. Flume collects data, ZooKeeper coordinates services, and Spark SQL executes queries.

Question 17Big Data Architecture and Huawei FusionInsight Overview

A Spark application on YARN runs with dynamic resource allocation enabled. Which behavior should the engineer expect as the workload changes?

  1. The driver is migrated to a different cluster automatically
  2. The HDFS replication factor is adjusted to match the load
  3. YARN queues are deleted when no tasks remain
  4. Executors are added or removed based on pending tasks and idle time Correct answer.

Explanation

Dynamic allocation lets the ApplicationMaster request additional executors when tasks are backlogged and release idle executors after a timeout, improving cluster utilization. It does not move the driver, change HDFS replication, or delete queues.

Question 18Big Data Architecture and Huawei FusionInsight Overview

Why does HBase rely on a write-ahead log (WAL) in addition to in-memory MemStores?

  1. To enforce column family versioning limits
  2. To compress HFiles before compaction
  3. To elect the HBase Master
  4. To recover unflushed edits if a RegionServer crashes Correct answer.

Explanation

Writes first append to the WAL and then to the MemStore; if a RegionServer fails before flushing, the WAL replays the edits so no acknowledged write is lost. Compaction, master election, and versioning are handled by other mechanisms.

Question 19Big Data Architecture and Huawei FusionInsight Overview

A tenant complains that Spark jobs are queued behind long-running batch jobs in the same YARN queue. Which change best isolates the interactive workload without adding hardware?

  1. Increase the HDFS replication factor for the tenant
  2. Disable the WAL on HBase tables used by the tenant
  3. Create a separate YARN queue with a guaranteed capacity share and submit interactive jobs there Correct answer.
  4. Increase the Hive metastore heap size

Explanation

Capacity or fair scheduling with separate queues guarantees each workload a share of cluster resources, so short interactive jobs are not starved by long batch jobs. Replication, WAL settings, and metastore heap do not affect YARN scheduling.

Question 20Big Data Architecture and Huawei FusionInsight Overview

A FusionInsight cluster must guarantee that a Kafka topic survives the loss of one broker while still serving reads. Which configuration achieves this?

  1. Replication factor 2 with min.insync.replicas=1
  2. Replication factor 1 with acks=0
  3. Replication factor 3 with unclean.leader.election disabled Correct answer.
  4. Replication factor 1 with log retention disabled

Explanation

Three replicas tolerate one broker failure, and disabling unclean leader election prevents an out-of-sync replica from becoming leader and losing committed data. A single replica has no redundancy, and acks=0 gives no delivery guarantee.

Question 21Hadoop Distributed File System (HDFS) Design and Development

A developer configures a Hadoop cluster and needs to choose a default block size for a new large dataset. Which statement correctly describes the role of the HDFS block size?

  1. The block size determines the maximum file size that HDFS can store.
  2. The block size must equal the underlying disk sector size for correctness.
  3. The block size sets the number of replicas stored for each block.
  4. The block size is the unit of storage allocation and replication on DataNodes. Correct answer.

Explanation

HDFS splits files into blocks, and each block is stored as an independent unit on DataNodes and replicated separately. The block size therefore controls the granularity of allocation, replication, and parallel read/write. It does not cap total file size, does not need to match disk sectors, and is unrelated to the replication factor.

Question 22Hadoop Distributed File System (HDFS) Design and Development

A developer writes a file to HDFS using the default configuration. Where is the file's metadata such as block-to-DataNode mapping stored?

  1. In the client's local staging directory until the file is closed
  2. In the NameNode's memory and edit log, with a checkpoint image Correct answer.
  3. Inside each block on the DataNodes as a sidecar file
  4. In the DataNode's block scanner report cache

Explanation

The NameNode holds the namespace and block mapping in memory and persists changes through the edit log, periodically compacted into an fsimage checkpoint. DataNodes store only the actual block data and report block lists to the NameNode. Clients and block scanners do not serve as the authoritative metadata store.

Question 23Hadoop Distributed File System (HDFS) Design and Development

A developer must read a large HDFS file from a MapReduce task running on a DataNode that hosts one of the file's replicas. Which read strategy does HDFS's short-circuit local read optimize?

  1. Reading the block via the NameNode's data transfer protocol
  2. Reading the block from the nearest remote rack to balance load
  3. Reading the block through the local DataNode over a loopback TCP socket
  4. Reading the block directly from the local disk file bypassing the DataNode Correct answer.

Explanation

Short-circuit local read lets a client on the same node open the block file directly from local disk, avoiding DataNode data-transfer overhead. It still requires the DataNode's cooperation to obtain the file descriptor and validate access. It does not route through the NameNode or a loopback socket for the data path.

Question 24Hadoop Distributed File System (HDFS) Design and Development

A developer wants to append data to an existing HDFS file that was created with default settings. Which condition must hold for append to succeed?

  1. The file must have replication factor 1.
  2. The file must be smaller than one block.
  3. The client must hold the file's lease and the file must not be closed by another writer. Correct answer.
  4. The file's owner must be the superuser.

Explanation

Append requires the client to acquire or extend the lease on the file; only one writer may hold the lease at a time. Replication factor, file size relative to a block, and superuser ownership are not prerequisites. If another client holds the lease, the append is rejected.

Question 25Hadoop Distributed File System (HDFS) Design and Development

A developer observes that a NameNode restart takes a long time because it must replay a very large edit log. Which mechanism reduces restart time by compacting the edit log into a new namespace image?

  1. SecondaryNameNode or Standby NameNode checkpointing Correct answer.
  2. Balancer
  3. DataNode block scanner
  4. DiskChecker

Explanation

Checkpointing merges the current fsimage with the edit log to produce a new fsimage, so restart replays only the edits since the last checkpoint. In HA deployments the Standby NameNode performs this role. Block scanning, balancing, and disk checking address data health and placement, not metadata compaction.

Question 26Hadoop Distributed File System (HDFS) Design and Development

A developer sets dfs.replication to 3 for a file, but one DataNode holding a replica fails permanently. What does HDFS do to restore the replication factor?

  1. It immediately fails all reads of the affected file.
  2. The NameNode detects under-replication and schedules a new replica on another DataNode. Correct answer.
  3. It reduces the file's replication factor to 2 permanently.
  4. It marks the block corrupt and requires the client to rewrite it.

Explanation

The NameNode tracks block locations and replication levels; when a replica disappears it marks the block under-replicated and queues replication work to another live DataNode. Reads continue from surviving replicas in the meantime. The replication factor setting is not changed by replica loss.

Question 27Hadoop Distributed File System (HDFS) Design and Development

A developer configures rack awareness in a cluster with replication factor 3. Which placement policy does HDFS aim for by default?

  1. One replica per rack across three distinct racks always.
  2. Two replicas on one rack and one on a different rack. Correct answer.
  3. All three replicas on the same DataNode for locality.
  4. All three replicas on the same rack to minimize cross-rack traffic.

Explanation

The default rack-aware placement puts two replicas on one rack and the third on a different rack, balancing write cost against rack-failure tolerance. It does not require three distinct racks, which is not guaranteed in small clusters. Placing all replicas together defeats rack-failure protection.

Question 28Hadoop Distributed File System (HDFS) Design and Development

A developer uses the HDFS Java API and calls FileSystem.create() on a path that already exists as a file. What happens with default options?

  1. A new file with a numeric suffix is created.
  2. The call appends to the existing file automatically.
  3. An IOException is thrown because overwrite defaults to false. Correct answer.
  4. The existing file is truncated and overwritten.

Explanation

FileSystem.create() has an overwrite parameter that defaults to false, so creating an existing path fails with an IOException. To replace the file the caller must pass overwrite=true. HDFS never silently appends or renames on create.

Question 29Hadoop Distributed File System (HDFS) Design and Development

A developer needs to copy a file from the local filesystem into HDFS from the command line. Which command performs this correctly?

  1. hdfs dfs -put /local/data.txt /user/dev/data.txt Correct answer.
  2. hdfs dfs -get /local/data.txt /user/dev/data.txt
  3. hdfs dfs -cat /local/data.txt > /user/dev/data.txt
  4. hdfs dfs -mv /local/data.txt /user/dev/data.txt

Explanation

The -put command copies local files into HDFS, and -copyFromLocal is an equivalent. The -get command moves data in the opposite direction, from HDFS to local disk. Shell redirection and -mv do not implement local-to-HDFS upload.

Question 30Hadoop Distributed File System (HDFS) Design and Development

A developer must ensure that a write to HDFS is durable on disk before the close() call returns. Which mechanism does HDFS use to achieve this?

  1. The client reads the block back from a random replica to verify it.
  2. The client waits for the NameNode to fsync the edit log only.
  3. The client receives acknowledgements from the DataNode pipeline after each packet is written and the final close flushes and confirms. Correct answer.
  4. The client relies on the OS page cache flushing on the DataNode.

Explanation

HDFS writes data in packets through a pipeline of DataNodes, and each packet is acknowledged back to the client. On close, the pipeline flushes and the NameNode is told the file is complete, making the data durable according to the configured checksum and sync policy. Reading back is not part of the write protocol.

Question 31Hadoop Distributed File System (HDFS) Design and Development

A developer is designing a directory layout for a multi-tenant analytics platform. Which HDFS feature allows setting storage quotas and namespace quotas per directory?

  1. Space quota and namespace quota via hdfs dfsadmin -setQuota and -setSpaceQuota Correct answer.
  2. dfs.blocksize per directory
  3. dfs.replication per file
  4. dfs.datanode.du.reserved

Explanation

HDFS supports namespace quotas (number of names) and space quotas (bytes) on directories through dfsadmin commands. These let administrators bound tenant usage. Replication and block size settings govern storage behavior but do not cap usage per directory.

Question 32Hadoop Distributed File System (HDFS) Design and Development

A developer runs a MapReduce job and wants input splits to align with HDFS block boundaries to maximize locality. Which factor primarily determines the split size?

  1. The HDFS block size and the mapreduce.input.fileinputformat.split.maxsize setting. Correct answer.
  2. The number of reducers configured for the job.
  3. The replication factor of the input files.
  4. The DataNode's disk sector size.

Explanation

FileInputFormat computes splits from the file's block boundaries, bounded by the minimum and maximum split size settings. Aligning splits with blocks lets each map task read a local replica. Reducer count, replication factor, and sector size do not determine split boundaries.

Question 33Hadoop Distributed File System (HDFS) Design and Development

A developer wants to read a file in HDFS using the Java API and process it line by line. Which class is the standard choice for reading a text file as lines?

  1. PathFilter
  2. FileStatus
  3. FSDataInputStream wrapped by a BufferedReader via InputStreamReader Correct answer.
  4. FSDataOutputStream

Explanation

FSDataInputStream provides the byte stream, and wrapping it in an InputStreamReader and BufferedReader yields line-oriented reading. FSDataOutputStream is for writing, FileStatus describes metadata, and PathFilter filters paths during listing. None of those read file content.

Question 34Hadoop Distributed File System (HDFS) Design and Development

A developer enables HDFS HA with two NameNodes. Which component ensures that only one NameNode is active at a time and prevents split-brain?

  1. The SecondaryNameNode's checkpoint schedule
  2. The Balancer's bandwidth limit
  3. The DataNode block report interval
  4. The JournalNode quorum combined with a fencing mechanism such as ZooKeeper or an SSH fence Correct answer.

Explanation

In HA, the JournalNode quorum shares the edit log, and a fencing mechanism ensures the previously active NameNode cannot continue serving as active. Together they prevent two active NameNodes, avoiding split-brain. Block reports, balancing, and checkpoint schedules do not arbitrate active state.

Question 35Hadoop Distributed File System (HDFS) Design and Development

A developer notices that a DataNode is repeatedly marked dead by the NameNode. Which configuration primarily controls how long the NameNode waits without a heartbeat before declaring the DataNode dead?

  1. dfs.replication
  2. dfs.namenode.heartbeat.recheck-interval combined with dfs.heartbeat.interval Correct answer.
  3. dfs.blocksize
  4. dfs.datanode.balance.bandwidthPerSec

Explanation

The dead-node timeout is derived from the heartbeat interval and the recheck interval, typically as twice the recheck interval plus ten heartbeat intervals. Tuning these values changes how quickly the NameNode declares a DataNode dead. Block size, replication, and balance bandwidth do not govern heartbeat expiry.

Question 36Hadoop Distributed File System (HDFS) Design and Development

A developer uses the HDFS Java API to list files in a directory recursively. Which method call achieves this?

  1. FileSystem.listFiles(path, true) Correct answer.
  2. FileSystem.getFileStatus(path)
  3. FileSystem.listStatus(path)
  4. FileSystem.globStatus(path)

Explanation

listFiles(path, recursive) returns a RemoteIterator of LocatedFileStatus entries and descends into subdirectories when recursive is true. listStatus returns only one level, getFileStatus returns a single path's metadata, and globStatus expands wildcard patterns but does not recurse.

Question 37Hadoop Distributed File System (HDFS) Design and Development

A developer wants to compress MapReduce intermediate output to reduce shuffle traffic. Which setting controls this, and what is the default?

  1. io.file.buffer.size, default 4096
  2. mapreduce.map.output.compress, default false Correct answer.
  3. mapreduce.output.fileoutputformat.compress, default true
  4. dfs.replication, default 3

Explanation

mapreduce.map.output.compress enables compression of map output before the shuffle, and it defaults to false. The file output format compression setting applies to final job output, not intermediate data. Replication and buffer size settings do not affect shuffle compression.

Question 38Hadoop Distributed File System (HDFS) Design and Development

A developer is choosing a compression codec for HDFS files that must support splitting for parallel MapReduce processing. Which codec supports splittable compression?

  1. Snappy
  2. Gzip
  3. LZO without an index
  4. Bzip2 Correct answer.

Explanation

Bzip2 blocks carry synchronization markers that allow input splits to begin at block boundaries, so it is splittable. Gzip and Snappy produce non-splittable streams, and LZO is only splittable when an index is built. For large files processed in parallel, splittability matters for locality and task count.

Question 39Hadoop Distributed File System (HDFS) Design and Development

A developer needs to move a large file within HDFS from /staging to /final without copying data through the client. Which command is most efficient?

  1. hdfs dfs -put /staging/file /final/file
  2. hdfs dfs -get /staging/file /final/file
  3. hdfs dfs -mv /staging/file /final/file Correct answer.
  4. hdfs dfs -cp /staging/file /final/file

Explanation

Renaming within the same filesystem is a metadata operation performed by the NameNode, so no block data moves. The -cp command creates a full copy, and -get and -put transfer data between local disk and HDFS. For same-cluster moves, rename is the efficient choice.

Question 40Hadoop Distributed File System (HDFS) Design and Development

A developer observes that a small file workload creates millions of tiny files in HDFS. Which consequence is most accurate?

  1. DataNode disk usage grows disproportionately because each tiny file consumes a full block on disk.
  2. NameNode memory pressure grows because each file, block, and directory consumes namespace objects. Correct answer.
  3. The replication factor automatically increases to protect small files.
  4. The block size automatically shrinks to fit each small file.

Explanation

Each file, block, and directory is an object in the NameNode's heap, so many small files exhaust NameNode memory long before disk fills. HDFS does not auto-adjust replication or block size for small files, and DataNodes store only the bytes actually written. This is why small-file problems are a namespace scaling concern.

Question 41Hadoop Distributed File System (HDFS) Design and Development

A developer wants to safely decommission a DataNode so that its blocks are replicated elsewhere before it is shut down. Which action should be taken first?

  1. Add the DataNode's host to the exclude file and refresh the NameNode. Correct answer.
  2. Run the Balancer with maximum bandwidth.
  3. Stop the DataNode process immediately to test failover.
  4. Delete the DataNode's blocks manually.

Explanation

Listing the host in the exclude file and refreshing the NameNode triggers decommissioning, during which HDFS replicates the node's blocks to other nodes before marking it decommissioned. Stopping the process abruptly or deleting blocks risks data loss and unnecessary re-replication. The Balancer only evens out disk usage.

Question 42Hadoop Distributed File System (HDFS) Design and Development

A developer configures a file with a replication factor larger than the number of available DataNodes. What happens?

  1. The file is written with as many replicas as available and remains under-replicated until more DataNodes appear. Correct answer.
  2. The NameNode refuses to create the file.
  3. The write fails immediately with an error.
  4. The replication factor is automatically reduced to match the cluster size.

Explanation

HDFS honors the requested replication factor as a target; if fewer DataNodes are available, the file is written with fewer replicas and the NameNode keeps trying to reach the target. The configured factor is not silently rewritten. Writes do not fail solely because the target exceeds current node count.

Question 43Hadoop Distributed File System (HDFS) Design and Development

A developer uses the HDFS Java API and wants to obtain a FileSystem instance for a remote cluster identified by a URI. Which approach is correct?

  1. new DistributedFileSystem(path)
  2. FileSystem.get(uri, conf) Correct answer.
  3. new FileSystem(uri, conf)
  4. FileSystem.getLocal(conf)

Explanation

FileSystem.get(uri, conf) resolves the scheme in the URI, such as hdfs://, and returns a cached FileSystem instance configured by the given Configuration. The FileSystem constructor is not public, getLocal returns the local filesystem, and DistributedFileSystem is not constructed with a Path. This is the standard entry point for HDFS access.

Question 44Hadoop Distributed File System (HDFS) Design and Development

A developer wants to guarantee that a file's data is visible to other readers immediately after the writer closes it, and also wants the NameNode to persist the namespace change before returning. Which client action provides this guarantee?

  1. Calling sync() on the local disk of the client
  2. Relying on the DataNode's periodic block report
  3. Calling close() on the output stream after writing all data Correct answer.
  4. Calling hflush() after every write without closing the file

Explanation

close() completes the write pipeline, flushes remaining packets, and notifies the NameNode that the file is complete, making the data visible and durable per the configured policy. hflush() only pushes data to the pipeline without finalizing the file. Local disk sync and block reports do not finalize an HDFS file.

Question 45MapReduce and YARN Application Development

In a MapReduce job, which component is responsible for sorting and grouping the map output records by key before they are passed to the reducer?

  1. The InputFormat, which splits input data before the map task starts
  2. The JobHistoryServer, which records completed task attempts
  3. The shuffle and sort phase, executed by the framework between map and reduce Correct answer.
  4. The OutputCommitter, which finalizes task output directories

Explanation

Between map and reduce, the framework runs the shuffle and sort stage: map outputs are partitioned by reducer, transferred, merged, and sorted by key so each reduce call receives one key with all its values grouped. InputFormat and OutputCommitter operate at input splitting and output commit boundaries, not on inter-task record ordering.

Question 46MapReduce and YARN Application Development

A developer sets the number of reduce tasks to zero for a job. What is the resulting behavior?

  1. The framework automatically assigns one reducer to preserve grouping semantics
  2. Map tasks buffer all output in memory until a reducer is configured later
  3. The map output is written directly as final job output with no shuffle or reduce stage Correct answer.
  4. The job fails at submission because at least one reducer is mandatory

Explanation

When mapreduce.job.reduces is 0, the framework skips the shuffle and reduce phases entirely and each map task writes its records directly to the job output path. This is a common pattern for map-only ETL jobs where no aggregation across keys is required.

Question 47MapReduce and YARN Application Development

Which YARN component is responsible for negotiating resources from the ResourceManager and launching containers on a NodeManager?

  1. The ApplicationMaster of the running application Correct answer.
  2. The ResourceManager scheduler plug-in
  3. The YARN client that submitted the application
  4. The NodeManager container executor

Explanation

Each application has an ApplicationMaster that requests containers from the ResourceManager scheduler, receives allocated containers, and asks NodeManagers to launch tasks in them. It also monitors task progress and handles failures for its own application.

Question 48MapReduce and YARN Application Development

Which class must an application implement to define the map function in the Hadoop MapReduce API?

  1. org.apache.hadoop.mapreduce.Partitioner
  2. org.apache.hadoop.mapreduce.RecordReader
  3. org.apache.hadoop.mapreduce.OutputFormat
  4. org.apache.hadoop.mapreduce.Mapper Correct answer.

Explanation

The Mapper class provides the map method that receives a key-value pair and an output Context, and applications subclass it to emit intermediate records. Partitioner, RecordReader, and OutputFormat define partitioning, input parsing, and output writing respectively.

Question 49MapReduce and YARN Application Development

A developer needs to join a small lookup table with a large dataset in a MapReduce job without a reduce-side join. Which approach fits best?

  1. Load the small table into memory in setup and perform the join inside the map function Correct answer.
  2. Disable the combiner so intermediate records are not merged before the join
  3. Increase the number of reducers so the join completes faster in the reduce phase
  4. Set the input format to KeyValueTextInputFormat for both datasets

Explanation

A map-side join works when one side is small enough to be cached in each map task's memory via setup, letting the map function enrich each large-dataset record locally. This avoids the shuffle cost of a reduce-side join but assumes the small table fits in memory.

Question 50MapReduce and YARN Application Development

In a MapReduce job, what does the Partitioner control?

  1. How input splits are sized across DataNodes
  2. How many times the combiner runs per spill
  3. Which compression codec is used for map output
  4. Which reduce task receives each intermediate key Correct answer.

Explanation

The partitioner computes a partition number from the intermediate key and value, and that number determines which reducer processes the key. The default HashPartitioner hashes the key modulo the number of reduce tasks.