CCAAK: Configure Apache Kafka Clusters

Domain 1, Configure Apache Kafka clusters, tests whether you can translate workload requirements into broker, topic and client settings, and whether you know which settings are static, which are dynamic, and where each one applies.

Static, dynamic and scoped configuration

• server.properties is read at broker startup. Settings such as broker.id/node.id, log.dirs and listeners are static and need a restart.
• Many properties can be changed at runtime with kafka-configs.sh: --entity-type brokers --entity-name 0, --entity-type topics, or --entity-type clients for quotas. Dynamic entries live in cluster metadata and survive restarts.
• Precedence: a topic override beats the broker default, and a client quota beats the broker default quota.

Replication and durability

The durability triangle is replication factor, min.insync.replicas and producer acks.

• With RF=3 and min.insync.replicas=2, one broker may be lost and acks=all writes still succeed.
• Setting min.insync.replicas equal to RF means any single broker outage blocks writes with NotEnoughReplicas.
• unclean.leader.election.enable=false protects durability; enabling it restores availability at the cost of silent data loss.
• replica.lag.time.max.ms controls ISR eviction; a lagging follower that is evicted shrinks the effective ISR and can make acks=all fail.

Log, retention and partitioning

• log.dirs accepts multiple directories; Kafka spreads partitions across them, so capacity planning must allow for imbalance rather than assuming perfect striping.
• Retention (log.retention.ms, log.retention.bytes) is enforced as segments are rolled and checked, so log.segment.bytes and log.roll.ms affect how promptly disk space is reclaimed.
• num.partitions and default.replication.factor apply mainly to auto-created topics. Partitions can be increased but never decreased.
• compression.type can be set at producer, broker or topic level; broker-side recompression costs CPU, while producer-side compression saves network.

Worked example

A team runs topic orders with RF=3, min.insync.replicas=2, acks=all and 7-day retention, and wants to survive one broker failure without data loss. The setting combination already achieves this: two of three replicas must acknowledge. If they instead chose RF=2 with min.insync.replicas=2, a single broker failure would stop producers; the correct fix is RF=3, not lowering acks.

Traps

• Forgetting that auto.create.topics.enable=true creates topics with one partition and the broker default replication factor.
• Lowering acks to cure producer timeouts, converting a durability requirement into an outage risk.
• Raising min.insync.replicas above the currently in-sync replica count during a rolling restart, which blocks writes.
• Assuming a topic-level retention override follows the broker default after the override is deleted.