Kafka and Data Ingestion / Integration (Flume, Loader, Sqoop)
Kafka fundamentals
A topic is split into partitions; each partition has a leader and followers. Replication factor copies partitions across brokers. ISR is the set of replicas caught up with the leader. Producers choose acks: 0 for no wait, 1 for leader only, all for all ISR. Consumers in a group divide partitions; offsets record progress. Retention and log compaction are different cleanup policies.
Ingestion tools
Flume moves streaming logs with source-channel-sink; a file channel is durable, a memory channel is fast but lossy on failure. Loader handles batch data integration in FusionInsight. Sqoop imports relational data to HDFS or Hive and exports back.
Worked example
Set replication factor 3, min.insync.replicas=2, and acks=all. A producer write succeeds only if the leader and at least one follower acknowledge. One broker can fail without losing acknowledged data; if the ISR falls below 2, writes fail rather than risk loss. With acks=1, a leader failure can lose data that followers never received.
Traps that cost marks
More partitions increase parallelism but also metadata and rebalance cost. A consumer group rebalance pauses consumption. Flume memory channels do not survive agent crash. Sqoop null handling and delimiter choices can corrupt downstream Hive tables if not matched to the table definition.