HCIP-Big Data Developer common mistakes
HCIP-Big Data Developer · Understand the traps and how to avoid them.
Treating FusionInsight Manager components (Manager, HDFS, YARN, Hive, HBase, Spark, Flink, Kafka, Loader, Flume, Sqoop) as independent products and answering questions about which service owns a function without mapping it to the FusionInsight layering (management plane, big data platform services, and multi-tenant resource model).
Candidates often study Hadoop components in isolation from open-source documentation and never internalize how FusionInsight packages them under a unified management and multi-tenancy layer, so questions about component roles, dependencies, and Manager-managed configuration are answered from generic Hadoop knowledge.
What to do instead
Build a one-page component map: list each FusionInsight service, its upstream open-source project, the port/role it exposes, and whether it is configured via Manager or client-side XML. For each service, note what Manager centralizes (installation, config distribution, monitoring, alarms, tenant resource assignment). Practice explaining how a tenant's resource pool flows into YARN queues and HDFS directories.
Answering HDFS design and development questions by choosing replication factor 1 for reliability, ignoring rack awareness, or assuming small files are harmless because NameNode metadata is 'just pointers'.
Candidates memorize the default block size and replication factor of 3 but do not reason about NameNode memory per block/file object, the cost of per-file objects, or how rack awareness affects write pipelines and read locality. They also confuse HDFS block size with the physical disk block size.
What to do instead
Learn the numbers: default block size (128 MB in modern releases, 64 MB in older ones), default replication 3, NameNode heap cost per object, and why small files stress NameNode. Practice scenarios: choosing replication and block size for a given workload, explaining rack-aware placement, and describing how to mitigate small files (HAR, SequenceFile, CombineFileInputFormat, compaction).
Writing MapReduce/YARN answers that treat the ApplicationMaster as optional, place the ResourceManager on every node, or claim the JobTracker still schedules tasks in YARN-based clusters.
Many candidates learned pre-YARN Hadoop (JobTracker/TaskTracker) or only used Hive/Spark and never wrote a raw MapReduce job, so the YARN roles (ResourceManager, NodeManager, ApplicationMaster, Container) and the job submission lifecycle are fuzzy.
What to do instead
Draw the YARN job lifecycle end to end: client submits to ResourceManager, RM allocates a container for the ApplicationMaster, AM requests containers from RM via the scheduler, NodeManagers launch containers, AM monitors and retries tasks, then unregisters. Memorize the three schedulers (FIFO, Capacity, Fair) and their queue behavior. Practice questions on speculative execution, task retry counts, and container memory/vcore configuration.
Answering Hive questions by assuming Hive stores data in a proprietary format, that partitioning and bucketing are interchangeable, or that a partitioned table automatically prunes partitions regardless of the WHERE clause.
Candidates use Hive mostly through simple SELECTs and do not distinguish managed vs external tables, partition columns vs data columns, or static vs dynamic partitioning. They also conflate partitioning (directory-based) with bucketing (hash-based file splitting).
What to do instead
Study the Hive data model explicitly: managed vs external tables, partition columns stored as directories, bucketing into a fixed number of files by hash, and file formats (TextFile, ORC, Parquet, Avro). Practice writing queries where partition pruning applies vs where it is defeated by functions on the partition column. Learn dynamic partition properties (hive.exec.dynamic.partition, mode=nonstrict) and when to use them.
Designing HBase schemas with row keys that are sequential (timestamps, auto-increment IDs) and then answering performance questions as if the hotspotting were unavoidable or solvable only by adding RegionServers.
Candidates learn the HBase data model (row key, column family, cell versioning) but do not connect row-key design to Region splitting and write hotspots. They also forget that column families are defined at table creation and that too many families hurt performance.
What to do instead
Practice row-key design patterns: salting, hashing, reversing, and composite keys that distribute writes while preserving scan order. Know the read/write path (WAL, MemStore, HFile, BlockCache), when a Region splits, and how to pre-split a table. Study column-family best practices (keep families few, similar access patterns) and the difference between Get, Scan, and filter usage.
Treating Spark RDD transformations and actions as interchangeable, or claiming that a groupByKey is always preferable to reduceByKey because it is simpler, without discussing shuffle and memory cost.
Candidates often write Spark code that works on small data and never reason about lazy evaluation, lineage, narrow vs wide dependencies, or shuffle. They also confuse DataFrame/Dataset APIs with RDD APIs and assume Spark SQL always optimizes the same way.
What to do instead
Learn the narrow vs wide transformation distinction and which operations trigger a shuffle. Practice choosing reduceByKey/aggregateByKey over groupByKey, using broadcast joins for small tables, and reading a physical plan with explain(). Study Spark SQL Catalyst optimization, partitioning and coalesce, and the difference between cache() and persist() storage levels.
Answering Flink questions as if it were micro-batch Spark, ignoring event time vs processing time, watermark semantics, and checkpoint/savepoint configuration for exactly-once state.
Candidates with Spark Streaming background assume Flink windowing behaves the same and do not study Flink's time semantics, state backends, or checkpoint barriers. They also confuse at-least-once and exactly-once guarantees.
What to do instead
Study Flink's core model: event time, ingestion time, processing time; watermark generation and late-data handling; window types (tumbling, sliding, session); state backends (MemoryStateBackend, FsStateBackend, RocksDBStateBackend); checkpointing and savepoints; and the two-phase-commit sink pattern for exactly-once. Practice tracing a job from source to sink with a failure and recovery.
Confusing the roles of Kafka, Flume, Sqoop, and Loader: using Sqoop for streaming ingestion, Flume for relational batch import, or Loader for Kafka topic consumption, and ignoring Kafka partition/consumer-group semantics.
The ingestion domain bundles several tools with overlapping-sounding purposes, and candidates often learn only one in depth. They also miss Kafka's ordering guarantee (per partition), consumer-group rebalancing, and offset management.
What to do instead
Build a comparison table: Kafka (distributed log, pub/sub, streaming), Flume (agent-based log/event collection with sources, channels, sinks), Sqoop (batch RDBMS to HDFS/Hive import/export), Loader (FusionInsight batch data loading between relational databases and Hadoop). For Kafka, memorize partition count vs consumer parallelism, consumer-group offset commit, and replication/ISR behavior. Match each tool to a scenario in practice.
Answering security and multi-tenancy questions by describing only Kerberos authentication and forgetting authorization (Ranger policies), tenant resource isolation, and data-level permissions across HDFS, Hive, HBase, and Kafka.
Candidates treat 'security' as a single login step and do not study FusionInsight's multi-tenant model, where tenants map to resource pools, HDFS directories, and YARN queues, with role-based access controlled centrally. They also miss the distinction between authentication, authorization, and auditing.
What to do instead
Learn the three pillars separately: authentication (Kerberos, keytabs, principal/realm), authorization (Ranger policies per service, HDFS ACLs, Hive roles, HBase ACLs), and auditing. Study how a FusionInsight tenant is created and what it provisions (HDFS dir, YARN queue, user/group). Practice scenarios where a user can authenticate but is denied access, and identify which layer to fix.
Giving generic tuning advice ('add more memory', 'increase parallelism') instead of diagnosing from symptoms such as data skew, spill, GC pressure, small files, or queue contention, and skipping the troubleshooting workflow.
Performance and troubleshooting questions require mapping a symptom to a metric to a configuration knob, but candidates often memorize a list of parameters without the diagnostic chain. They also ignore FusionInsight Manager alarms and logs as the first troubleshooting step.
What to do instead
Practice symptom-to-cause mapping: task stragglers to data skew (salting, repartition), long GC to heap/executor sizing, excessive spill to memory fraction and partition count, small files to compaction, queue wait to scheduler configuration. Learn where to look first: Manager alarms, YARN application logs, Spark/Flink UI stages, HDFS fsck, and HBase RegionServer metrics. Write a short runbook for each common failure.