HCIP-Big Data Developer cheat sheet

FusionInsight Manager is the cluster management plane; it deploys, monitors, and configures services (HDFS, YARN, Hive, HBase, Spark, Flink, Kafka, Flume) through a web UI and REST API.

Core component roles: HDFS (storage), YARN (resource scheduling), ZooKeeper (coordination and leader election), and the SQL/NoSQL/streaming layer (Hive, HBase, Spark, Flink, Kafka) on top.

FusionInsight supports three deployment modes: full-stack cluster, hybrid cloud (data on OBS), and multi-cluster/federation for geo-redundancy.

A tenant in FusionInsight is a logical resource partition: it binds a set of services, queues, HDFS directories, and users, isolating CPU/memory/storage quotas.

Client access uses a downloaded client configuration package containing core-site.xml, hdfs-site.xml, krb5.conf, and user.keytab for Kerberos-authenticated access.

HA design: two NameNodes (active/standby) via ZooKeeper/ZKFC, ResourceManager HA, and HBase Master HA; a single point of failure in ZooKeeper quorum breaks automatic failover.

FusionInsight Manager health check aggregates alarms by severity (Critical, Major, Minor, Warning); service-level alarms propagate from role instances.

Scale-out adds DataNode/NodeManager roles; HDFS rebalancing (balancer) must be run after adding DataNodes to redistribute blocks.

Huawei's big data stack aligns with Apache Hadoop ecosystem APIs, so standard Hadoop/Hive/Spark clients work with only configuration and Kerberos authentication changes.

Default block size is 128 MB (Hadoop 2.x/3.x); a file smaller than one block does not occupy a full block on disk but consumes one block slot in the NameNode namespace.

Replication factor default is 3; block placement policy is one replica on the local/writer node, one on a remote rack, and one on a different node in the remote rack.

NameNode stores metadata in memory (fsimage + edit log); SecondaryNameNode/CheckpointNode merges edits but is NOT a standby NameNode and provides no HA.

Safe mode: NameNode blocks writes until the block-report threshold is met; hdfs dfsadmin -safemode leave forces exit (use only when the namespace is consistent).

HDFS is optimized for large sequential reads and append-only writes; it does not support random writes or modifying an existing block in place.

Small-file problem: each file/block/directory consumes ~150 bytes of NameNode heap; use HAR archives, sequence files, or CombineFileInputFormat to reduce object count.

hdfs fsck /path -files -blocks -locations reports missing/under-replicated blocks; hdfs dfsadmin -report gives per-DataNode capacity and usage.

Short-circuit local reads bypass the DataNode over a Unix domain socket when the client and block are co-located; enable via dfs.client.read.shortcircuit.

Erasure coding (Hadoop 3.x) replaces replication for cold data: RS-6-3 uses 6 data + 3 parity units, reducing storage overhead from 200% to 50% at higher reconstruction cost.

MapReduce job phases: split -> map -> combine (optional) -> partition -> shuffle/sort -> reduce -> output; the shuffle is the dominant network and disk cost.

Combiner runs on the map side and must be commutative and associative; it is not guaranteed to run, so reducer logic must not depend on it.

Number of mappers = number of input splits (roughly file size / split size); number of reducers is set explicitly via mapreduce.job.reduces.

Partitioner determines which reducer key goes to; default HashPartitioner uses key.hashCode() % numReduceTasks, so changing reducer count changes key distribution.

YARN components: ResourceManager (scheduling), NodeManager (per-node agent), ApplicationMaster (per-application lifecycle), and containers (resource units of vcores + memory).

Capacity Scheduler supports queues, hierarchical queue trees, capacity/maximum-capacity percentages, and user-limit factors; Fair Scheduler shares resources equally over time.

Memory settings: mapreduce.map.memory.mb is the container request; mapreduce.map.java.opts -Xmx should be ~0.75-0.8 of the container size to avoid kill-by-OOM.

Speculative execution launches duplicate tasks for stragglers; disable it for non-idempotent tasks that write to external systems.

Data locality levels in order: NODE_LOCAL, RACK_LOCAL, OFF_SWITCH; YARN delays scheduling to wait for a better locality level (yarn.scheduler.capacity.node-locality-delay).

Managed (internal) tables drop data with the table; external tables drop only metadata, leaving HDFS data intact. Use external tables for shared or raw data.

Partitioning creates HDFS subdirectories (e.g., dt=2024-01-01); static partitions are specified in the INSERT, dynamic partitions require hive.exec.dynamic.partition=true and a non-strict mode.

Bucketing uses CLUSTERED BY ... INTO n BUCKETS with hash(key) % n; it enables efficient sampling (TABLESAMPLE) and bucket-map joins when bucket counts match.

File formats: ORC and Parquet are columnar with predicate pushdown and compression; TextFile is row-based and slow; ORC with ZLIB/SNAPPY is common for Hive warehouse tables.

Hive execution engines: MapReduce (legacy), Tez (DAG, default in many distros), and Spark; Hive-on-Spark requires a Spark session and is tuned via hive.execution.engine.

Join strategies: map-side join (small table broadcast, hive.auto.convert.join), bucket-map join, and sort-merge join (reduce-side, default for large-large joins).

The metastore is a relational database (MySQL/PostgreSQL) holding schema and partition metadata; HMS (Hive Metastore service) exposes it via Thrift to Hive, Spark, and Flink.

ACID tables (transactional=true) support UPDATE/DELETE/MERGE with ORC and compaction; they require the metastore and a valid transaction manager.

hive.exec.max.dynamic.partitions and max.dynamic.partitions.pernode cap dynamic partition creation to prevent NameNode overload from too many small directories.

HBase data model: table -> column families -> columns (qualifiers) -> cells with versions; the row key is the only index and rows are sorted lexicographically by row key bytes.

Row key design rules: avoid monotonically increasing keys (hotspotting), use salting/hashing or reverse keys, keep keys short, and order fields by query pattern.

Region split: a region grows to hbase.hregion.max.filesize (default 10 GB) then splits into two; pre-splitting with split keys avoids write hotspots on a new table.

Column families should be few (1-3); each family has its own store files and memstore, so many families multiply flush and compaction overhead.

Write path: client -> RegionServer -> WAL (HLog) append -> MemStore; data is queryable from MemStore before flush; flush to HFile happens when memstore size thresholds are hit.

Read path: client asks ZooKeeper/hbase:meta for the region location, then reads BlockCache and HFiles; bloom filters (ROW/ROWCOL) skip HFiles that cannot contain the key.

Compaction: minor compaction merges small HFiles; major compaction merges all store files in a region and purges deleted/expired versions. Major compaction is I/O heavy.

TTL (per family) and version count (VERSIONS) control cell expiry; deletes are tombstones applied only at compaction, so deleted data still consumes space until then.

HBase depends on ZooKeeper for master election and region assignment, and on HDFS for storage; a RegionServer failure triggers WAL splitting and region reassignment.

RDD transformations are lazy; actions (count, collect, saveAsTextFile) trigger a job. Narrow transformations (map, filter) avoid shuffle; wide ones (groupByKey, join) require it.

Use reduceByKey over groupByKey: reduceByKey combines on the map side before the shuffle, reducing network traffic.

Persist levels: MEMORY_ONLY, MEMORY_AND_DISK, DISK_ONLY, and _SER variants; MEMORY_AND_DISK_SER is a common default for iterative jobs. Always unpersist when done.

Spark SQL Catalyst optimizer applies rule-based rewrites and cost-based join reordering; check plans with explain(true) and look for BroadcastHashJoin vs SortMergeJoin.

AQE (spark.sql.adaptive.enabled) coalesces shuffle partitions, splits skewed partitions, and converts sort-merge joins to broadcast joins at runtime.

Partition tuning: target ~128 MB per partition; spark.sql.shuffle.partitions defaults to 200 and is often wrong for large or small jobs.

Data skew fix: salting the hot key, broadcast join for the small side, or AQE skew join handling (spark.sql.adaptive.skewJoin.enabled).

Broadcast join threshold is spark.sql.autoBroadcastJoinThreshold (default 10 MB); a broadcast variable must fit in executor memory.

Spark on YARN modes: client (driver on the submitting host) vs cluster (driver inside the ApplicationMaster, required for production and Kerberos delegation).

Flink runtime has JobManager (scheduling, checkpoints) and TaskManagers (slots executing tasks); a task slot is a fixed slice of TaskManager resources.

Watermarks mark event-time progress; a window fires when the watermark passes window end + allowed lateness. Use bounded-out-of-orderness or periodic watermark strategies.

Windowing types: tumbling (fixed, non-overlapping), sliding (fixed, overlapping), session (gap-based), and global (custom trigger).

Checkpointing uses a barrier-based snapshot (Chandy-Lamport); state backends are HashMapStateBackend (heap) and EmbeddedRocksDBStateBackend (large state, disk).

Exactly-once end-to-end requires source replayability (Kafka offsets) + checkpointing + a transactional/idempotent sink (e.g., Kafka producer with two-phase commit or an upsert sink).

Savepoints are manually triggered, versioned snapshots used for upgrades and rescaling; checkpoints are automatic and optimized for recovery speed.

Keyed state is partitioned by key and only usable after keyBy; operator state is per-parallel-subtask (e.g., Kafka source offsets).

Backpressure appears as high outPoolUsage/latency in the Flink UI; remedies are increasing parallelism, adding TaskManagers, or reducing per-record cost.

Flink SQL supports temporal joins, window TVFs, and changelog streams; the Table API and SQL share the same planner and can run in batch or streaming mode.

Kafka topic = ordered, partitioned log; each partition is append-only with an offset; ordering is guaranteed only within a partition.

Replication factor and min.insync.replicas control durability; acks=all with min.insync.replicas=2 tolerates one broker failure without data loss.

Consumer groups: each partition is assigned to at most one consumer in a group; adding consumers beyond partition count leaves consumers idle.

Offset commit: enable.auto.commit=true risks duplicates/loss on rebalance; manual commit after processing gives at-least-once semantics.

Kafka retention is time- or size-based (log.retention.hours/bytes); log compaction keeps the last value per key for changelog topics.

Flume architecture: Source -> Channel -> Sink. Channel types are Memory (fast, non-durable) and File (durable); use File channel for reliability.

Flume sink processors: default (one sink), failover (priority-ordered), and load-balancing (round-robin); interceptors modify or filter events in-flight.

Sqoop imports RDBMS tables to HDFS/Hive via map-only jobs; --split-by and --num-mappers control parallelism, and --incremental append/lastmodified supports delta loads.

Loader (FusionInsight) moves data between relational databases, HDFS, HBase, and Hive using connector jobs with configurable concurrency and field mapping.

Kerberos provides authentication: the client gets a TGT from the KDC, then service tickets; keytabs store principals and keys for non-interactive jobs.

Authentication proves identity; authorization decides access. Hadoop uses HDFS ACLs, Hive Ranger/Sentry policies, and HBase ACLs for authorization.

Ranger centralizes policy management across HDFS, Hive, HBase, YARN, and Kafka, with audit logging of allow/deny decisions.

HDFS permissions follow POSIX (owner/group/other rwx) plus ACLs; enable dfs.namenode.acls.enabled and use setfacl for fine-grained grants.

Hive authorization: SQL-standard mode (GRANT/REVOKE on objects) vs Ranger/Sentry policy plugins; column masking and row filtering are policy features.

Multi-tenancy in YARN: queue hierarchies with capacity/max-capacity, user-limit-factor, and ACLs on queues isolate tenants' CPU and memory.

HDFS tenant isolation uses directory ownership, quotas (dfsadmin -setQuota/-setSpaceQuota), and Ranger path policies; quotas prevent one tenant from filling the cluster.

Data encryption: HDFS transparent encryption (KMS + encryption zones) at rest, TLS for in-transit RPC/HTTP, and masked columns for sensitive fields.

Audit and compliance: Ranger audit logs, HDFS audit log (log4j), and Kerberos KDC logs together reconstruct who accessed what and when.

HDFS tuning levers: block size, replication, balancer runs, avoiding small files, and dfs.datanode.balance.bandwidthPerSec for rebalancing throttling.

MapReduce tuning: container memory vs -Xmx ratio, io.sort.mb/mapreduce.task.io.sort.mb, spill count, and reducer count matched to data volume.

Hive tuning: partition pruning, ORC + predicate pushdown, vectorized execution (hive.vectorized.execution.enabled), CBO (hive.cbo.enable), and Tez container sizing.

Spark tuning: partition sizing (~128 MB), broadcast joins for small tables, AQE, serialization (Kryo), and avoiding collect() on large datasets.

HBase tuning: pre-split regions, block cache sizing (hfile.block.cache.size), memstore size, and avoiding hot row keys; watch compaction queue length.

Flink tuning: parallelism per operator, RocksDB state backend for large state, checkpoint interval vs duration, and network buffer sizing.

Common failure triage: check YARN application logs and RM UI for container kills (memory exceeded), HDFS for missing blocks, and ZooKeeper for session timeouts.

Data skew symptoms: a few tasks run far longer than others and one reduce/partition dominates shuffle read; fix with salting, broadcast, or AQE skew handling.

Capacity planning: YARN available memory = sum(NodeManager memory) - reserved; HDFS usable = raw capacity x replication factor (or EC overhead).