CCAAK: Monitor Apache Kafka Clusters

Domain 3, Monitor Apache Kafka clusters, tests whether you can pick the metrics that actually predict incidents, interpret them correctly, and distinguish broker problems from client or data-distribution problems. The exam favors JMX metric names and commands over generic dashboards.

Broker health metrics

• UnderReplicatedPartitions above zero for a sustained period indicates a follower is not keeping up or a broker is down.
• OfflinePartitionsCount above zero means a partition has no leader and clients cannot produce or consume; treat it as an immediate page.
• ActiveControllerCount should be exactly 1 across the cluster. Zero means no controller; two means a split-brain symptom during failover.
• IsrShrinksPerSec and IsrExpandsPerSec show flapping replicas; frequent shrinks often precede under-replication.
• UncleanLeaderElectionsPerSec above zero means availability was chosen over durability and data may have been lost.

Request and resource metrics

• Request latency decomposes into RequestQueueTimeMs, LocalTimeMs, RemoteTimeMs, ResponseQueueTimeMs and ResponseSendTimeMs. High queue time points to saturation or slow disk; high remote time points to replication or follower issues.
• RequestHandlerAvgIdlePercent below roughly 20–30% signals thread-pool saturation.
• Track BytesInPerSec, BytesOutPerSec, disk utilization and log-flush latency together; network and disk are the usual bottlenecks, not CPU.

Consumer and client metrics

• Consumer lag from kafka-consumer-groups.sh --describe matters as a trend, not an absolute number. A lag that grows steadily means consumption capacity is below production.
• Watch consumer group state and rebalance rate: frequent rebalances usually indicate session timeout or processing-time problems, not broker faults.
• Producer-side error rates, retry counts and record send latency reveal broker pressure seen from the client side.

Tooling

JMX is the source of truth; export it with a JMX exporter to Prometheus and alert on the metrics above. Cruise Control adds rebalance and anomaly detection; Burrow or Kafka Exporter focus on consumer lag.

Worked example

One partition of a topic shows rising lag while sibling partitions are stable, and broker metrics are normal. This is key skew or a hot partition, not a broker outage: check the message key distribution and partition count before adding brokers. If all partitions of one topic lag while others are fine, suspect the consumer group instead.

Traps

• Alerting on absolute lag rather than its rate of change.
• Ignoring ActiveControllerCount when it reads 0 or 2.
• Treating high request queue time as CPU exhaustion when the underlying cause is disk saturation.