CCAAK: Support Apache Kafka Clusters

Domain 4, Support Apache Kafka clusters, tests operational triage and lifecycle work: diagnosing incidents in the right order, reassigning and decommissioning, applying quotas and security, and recovering safely without silently losing data.

Triage order

1. Cluster level: offline partitions, active controller count, under-replication, disk full.
2. Broker level: GC pauses, request queue saturation, network errors, log flush latency.
3. Client level: producer/consumer errors, timeouts, rebalance storms, authorization failures.

Server logs, controller.log, state-change.log and GC logs are the primary evidence. kafka-dump-log.sh inspects segment contents and index files when corruption or truncation is suspected.

Common support operations

• Reassign partitions with kafka-reassign-partitions.sh, then run a preferred-leader election with kafka-leader-election.sh --election-type PREFERRED.
• Decommission a broker by reassigning all its partitions, waiting for under-replication to clear, then stopping it.
• Apply client quotas with kafka-configs.sh --entity-type clients --entity-name <id> --alter --add-config 'producer_byte_rate=...,consumer_byte_rate=...'.
• Manage access with kafka-acls.sh and SASL/SCRAM or TLS; verify keystore and truststore paths before restarting brokers.
• For large messages, broker message.max.bytes, topic max.message.bytes, replica.fetch.max.bytes, producer max.request.size and consumer fetch.max.bytes must all accommodate the size, or replication or consumption fails.

Worked example

Producers time out; broker metrics show high RequestQueueTimeMs and a log volume at 100% disk. The queue backs up because flushing and fetch reads compete for the same saturated device. The fix is to free space (reduce retention, delete old segments, add capacity) and then address the growth trend, not to increase heap or producer timeouts, which only hide the bottleneck.

Traps

• Deleting a topic and expecting instant space reclamation: segments are removed only after the retention check interval and, for compacted topics, delete.retention.ms.
• Enabling unclean leader election to restore availability without accounting for the data loss.
• Setting replica.fetch.max.bytes below the message size, which breaks replication while producers appear to succeed.
• Increasing heap to fix long GC pauses instead of tuning garbage collection and reducing cache pressure.
• Restarting several brokers at once during an incident, turning under-replication into offline partitions.