Performance Tuning, Troubleshooting, and Operations for Big Data Applications
Tuning method
Start from a symptom and a metric, not a parameter. FusionInsight Manager shows service health, alarms, and role logs. For HDFS, watch capacity, missing blocks, and small files; run the balancer after adding DataNodes. For YARN, check queue capacity, pending containers, and container memory. For Hive, check partition pruning, file format, statistics, and skew. For HBase, check region hotspots, compaction queues, and block cache. For Spark, check shuffle spill and task skew. For Flink, check checkpoint duration and backpressure. For Kafka, check under-replicated partitions and consumer lag.
Worked example
A Hive query slows from minutes to hours. EXPLAIN shows a full table scan although the query filters dt. The predicate is substr(dt,1,10)='2024-05-01', which disables partition pruning. Rewrite as dt='2024-05-01', run ANALYZE TABLE, and confirm ORC files are not thousands of small files. Only then adjust Tez container size.
Traps that cost marks
Adding executor memory without fixing skew wastes resources. Running major compaction on HBase during peak load causes latency spikes. Restarting a service before reading its role log destroys evidence. Raising a queue's maximum capacity can starve sibling queues. Treating every alarm as a root cause leads to unnecessary restarts.