HCIP-Big Data Developer flashcards

38 cards. Read the term and the definition here. The deck at the top of this page is optional.

FusionInsight Manager

The web-based management plane of a FusionInsight cluster. It installs, configures, monitors, and alarms on services such as HDFS, YARN, Hive, HBase, Spark, Flink, and Kafka.

FusionInsight tenant

A logical resource partition binding services, YARN queues, HDFS directories, and users together. It enforces CPU, memory, and storage quotas so one business unit cannot starve another.

NameNode HA with ZKFC

Two NameNodes run active/standby; ZooKeeper and the ZKFailoverController elect the active node and trigger failover. A broken ZooKeeper quorum prevents automatic failover.

HDFS default block size

128 MB in Hadoop 2.x and 3.x. A file smaller than one block still consumes one namespace entry, and the block size governs split count for MapReduce and Spark reads.

HDFS replica placement

With replication 3: one replica on the writer node, one on a different node in a remote rack, and one on another node of that remote rack. This balances durability and write bandwidth.

HDFS safe mode

A NameNode state where writes are blocked until enough block reports arrive. It protects against premature deletion of under-replicated blocks during startup or recovery.

HDFS small file problem

Each file, block, and directory consumes roughly 150 bytes of NameNode heap. Millions of small files exhaust NameNode memory and slow metadata operations, so archive or combine them.

MapReduce combiner

An optional map-side aggregation step that reduces shuffle volume. It must be commutative and associative, and it may run zero or many times, so reducer logic cannot depend on it.

YARN container

The unit of resource allocation in YARN, granting a fixed amount of memory and virtual cores on one NodeManager. Tasks run inside containers managed by the ApplicationMaster.

Capacity Scheduler queues

Hierarchical queues with capacity and maximum-capacity percentages, plus user-limit factors and ACLs. They guarantee minimum shares while allowing borrowing of idle capacity.

Speculative execution

YARN launches duplicate attempts of slow tasks and keeps the first to finish. It helps with stragglers but must be disabled for non-idempotent tasks writing to external systems.

Hive managed vs external table

Dropping a managed table deletes its HDFS data; dropping an external table deletes only metadata. External tables suit shared, raw, or externally written data.

Hive dynamic partitioning

Partition values are derived from the last columns of the SELECT at load time. It requires enabling dynamic partitions and capping counts to avoid flooding the NameNode with directories.

Hive bucketing

CLUSTERED BY ... INTO n BUCKETS hashes a column into n files per partition. It enables efficient sampling and bucket-map joins when both tables use matching bucket counts.

ORC and Parquet formats

Columnar file formats with compression, predicate pushdown, and column pruning. They dramatically cut I/O versus text files for analytic queries in Hive, Spark, and Flink.

Hive metastore

A relational database plus Thrift service storing table schemas, partitions, and locations. Hive, Spark, and Flink share it so tables are visible across engines.

HBase row key design

Row keys are sorted lexicographically and are the only index. Avoid monotonic keys that hotspot one region; salt, hash, or reverse them and keep them short.

HBase write path

Writes append to the write-ahead log, then to the MemStore, and become readable immediately. MemStore flushes to HFiles when size thresholds are reached.

HBase major compaction

Merges all store files of a region into one and physically removes deleted and expired cells. It is I/O intensive and is usually scheduled off-peak.

HBase region pre-splitting

Creating a table with predefined split keys distributes regions across RegionServers from the start. It prevents write hotspots while a new table grows from a single region.

reduceByKey versus groupByKey

reduceByKey combines values on the map side before the shuffle, cutting network traffic. groupByKey ships all values, which can cause out-of-memory errors on skewed keys.

Spark AQE

Adaptive Query Execution re-optimizes a plan at runtime: it coalesces small shuffle partitions, splits skewed partitions, and switches to broadcast joins when sizes allow.

Spark broadcast join

A join where the small side is shipped to every executor, eliminating the shuffle. It applies when the table is under spark.sql.autoBroadcastJoinThreshold, default 10 MB.

Spark persist levels

Storage levels such as MEMORY_ONLY, MEMORY_AND_DISK, and their serialized variants control how cached RDDs or DataFrames are retained. Unpersist when the dataset is no longer reused.

Flink watermark

A marker flowing through the stream that signals event-time progress. Windows fire when the watermark passes the window end plus allowed lateness.

Flink checkpointing

Periodic barrier-based snapshots of operator state and source offsets. On failure the job restores from the latest checkpoint, giving exactly-once state consistency with replayable sources.

Flink state backends

HashMapStateBackend keeps state on the JVM heap for speed; EmbeddedRocksDBStateBackend stores state on local disk, supporting state far larger than memory at some latency cost.

Kafka consumer group

Consumers sharing a group.id split partition assignments so each partition is read by exactly one consumer. Consumers beyond the partition count stay idle.

Kafka acks=all

The producer waits for all in-sync replicas to acknowledge a write. Combined with min.insync.replicas=2 it survives one broker failure without losing acknowledged records.

Flume channel types

Memory channels are fast but lose events on agent failure; File channels persist events to disk and provide durability at higher latency. Choose File for reliable pipelines.

Kerberos keytab

A file holding a principal and its encrypted keys, letting services and scheduled jobs authenticate without a password. Protect it, since possession grants the principal's identity.

Apache Ranger policies

Centralized authorization rules for HDFS, Hive, HBase, YARN, and Kafka, with column masking, row filters, and audit logs of every allow or deny decision.

YARN queue ACLs

Per-queue submit and administer ACLs restrict which users and groups may run applications in a queue, forming the resource-isolation layer of multi-tenancy.

HDFS space quota

Set with hdfs dfsadmin -setSpaceQuota, it caps the bytes a directory tree may consume. It stops one tenant or project from filling the cluster.

Container memory exceeded kill

YARN kills a container whose physical memory passes the limit, often because -Xmx is set too close to the container size. Keep heap near 75-80 percent of the container.

Data skew symptom

A few tasks run far longer than peers and one reduce or shuffle partition dominates read size. Fix with salting, broadcast joins, or AQE skew join handling.

HDFS balancer

A tool that moves blocks between DataNodes to equalize usage after adding nodes. Throttle it with dfs.datanode.balance.bandwidthPerSec to protect foreground workloads.

Spark shuffle partition sizing

Aim for roughly 128 MB of data per shuffle partition. The default spark.sql.shuffle.partitions of 200 is often too high for small jobs and too low for large ones.