Big Data Overview and the Ecosystem
This domain tests whether you can map a data problem onto the right components and explain the 4V/5V model, batch versus stream processing, and the collection–storage–compute–analysis–management layers of a Hadoop-centered stack. Expect terminology and architecture-matching questions rather than deep tuning.
Core rules
- The Vs are Volume, Velocity, Variety, Veracity, and Value. Veracity is data trustworthiness; Value is the insight yield. Exams often swap these two.
- Processing modes: batch (high throughput, minutes or more), stream (continuous, sub-second to seconds), and interactive SQL. Lambda combines batch and speed layers; Kappa is stream-first.
- Ecosystem by role: ingest (Flume, Kafka, Sqoop, FusionInsight Loader), store (HDFS, HBase, object storage), compute (MapReduce, Spark, Flink), SQL (Hive, Spark SQL), coordinate (ZooKeeper), schedule (Oozie), manage (FusionInsight Manager, Ambari).
Worked example
A retailer receives 50,000 clickstream events per second and wants real-time dashboards plus next-day behavior analysis. Reasoning: Kafka buffers and decouples producers; Flink reads Kafka for live dashboards; the same stream lands in HDFS as partitioned Parquet or ORC; Hive or Spark SQL serves next-day queries. HBase is added only if dashboards need millisecond lookup by user ID. ZooKeeper coordinates the cluster but does not store business events.
Traps
- Kafka is an ingestion buffer and log, not the long-term analytical store.
- Hive is a SQL warehouse over HDFS, not an OLTP database; row-level updates are not its strength.
- ZooKeeper provides coordination and leader election, not scheduling or data storage.