HCIP-Big Data Developer case studies

HCIP-Big Data Developer · Put your knowledge to work in context.

The HCIP-Big Data Developer credential is Huawei's professional-level certification for engineers who build and operate applications on the FusionInsight big data platform. It targets developers, data engineers, and platform administrators who already know distributed computing basics and now need to write production code against HDFS, YARN, Hive, HBase, Spark, Flink, and Kafka, plus the ingestion and security layers around them. The HCIP-Big Data Developer case studies are 5 cases and about 5,030 words.

NameNode Heap Pressure from 3.2 Million Small Files per Day

A media-analytics platform runs FusionInsight HD 6.5.1 on 120 DataNodes (12 x 4 TB disks each) storing 1.1 PB at replication 3. About 4,000 edge collectors ship 8 KB JSON events through Flume at a rate that creates 3.2 million files per day; HDFS currently holds roughly 90 million files and 95 million blocks.

918 words, about 5 min 2

A Single Reducer Owns 55% of a 2 TB Join

An e-commerce batch platform runs a nightly MapReduce join on a 200-node YARN cluster (20 vcores, 128 GB RAM per node). Queue etl uses the Capacity Scheduler with 30% capacity and 50% maximum; the other two queues are about 40% idle at night.

982 words, about 5 min 3

HBase Hot Region from a Timestamp-First Rowkey

A monitoring platform stores 60 TB of host metrics in HBase 2.2 on 12 RegionServers (32 GB heap, 12 x 2 TB disks). The rowkey is (Long.MAX VALUE - timestamp) followed by hostId, chosen so that scanning 'newest rows first' is a simple forward scan.

1,004 words, about 5 min 4

Flink Checkpoints Stuck Behind a Backpressured Sink

A ride-hailing company runs Flink 1.12 on YARN in exactly-once mode. The job consumes a 60-partition Kafka topic at 480,000 events/s, enriches events with asynchronous HBase dimension lookups, keeps 2.4 TB of keyed RocksDB state for 24-hour session windows, and writes results to Kafka and HBase.

1,081 words, about 5 min 5

Spark SQL Join Misestimated and Skewed on NULL store_id

A retail BI team runs Spark 3.1 SQL on a 60-node cluster (8 cores, 32 GB RAM each) against a Hive ORC table store sales of 4 TB, partitioned by dt into 1,095 daily partitions. A nightly query joins store sales to store dim (2 MB, 1,500 rows) and customer dim (400 MB, 3 million rows), filters one month of dt , and aggregates revenue by region.

1,045 words, about 5 min