CCAAK key terms

A single Kafka server process that stores partitions, serves producer and consumer requests, and participates in cluster coordination. Each broker is identified by a unique integer id set in its configuration.

The broker elected to manage cluster metadata: assigning partition leaders and replicas, handling broker failures, and propagating changes. In KRaft mode the controller role is separated from the broker role.

Kafka's built-in consensus protocol that replaces ZooKeeper for metadata management. Controller quorum members replicate a metadata log, and brokers fetch metadata from the quorum leader.

The external coordination service used by older Kafka releases to store cluster metadata, controller election state, and configuration. Removed in Kafka 4.0 in favor of KRaft.

An ordered, immutable log shard of a topic. Partitions provide parallelism and are the unit of replication and leader election; ordering is guaranteed only within a single partition.

A copy of a partition stored on a broker. One replica is the leader serving reads and writes; others are followers that fetch from the leader to stay in sync.

The replica that currently handles all produce and fetch requests for a partition. Clients are routed to the leader, and followers replicate from it.

A replica that does not serve clients but continuously fetches from the partition leader to remain caught up. A follower becomes leader if the current leader fails.

A replica that has fetched up to the leader's high watermark within the configured lag threshold. Only in-sync replicas are eligible for leader election when acks=all is required.

The offset up to which all in-sync replicas have replicated a partition's data. Consumers can only read messages below the high watermark, which bounds visible data.

The offset of the next message that would be appended to a partition log on a given replica. Comparing log end offsets across replicas reveals replication lag.

A monotonically increasing integer that uniquely identifies a record's position within a partition. Consumers commit offsets to track their progress.

A named logical stream of records, divided into partitions and replicated across brokers. Producers write to topics and consumers subscribe to them.

The number of copies of each partition maintained across the cluster. It must not exceed the broker count, and a value of three is typical for production durability.

A topic or broker setting specifying the minimum number of in-sync replicas that must acknowledge a write when the producer uses acks=all. Otherwise the produce fails with NotEnoughReplicas.

A producer setting controlling how many acknowledgments are required before a write succeeds: 0 (none), 1 (leader only), or all (all in-sync replicas). Higher values increase durability but add latency.

A producer mode that attaches sequence numbers and a producer id so the broker deduplicates retries, preventing duplicate records from network retries. Enabled with enable.idempotence=true.

A producer that writes atomically across multiple partitions using a transaction id, coordinated by the transaction coordinator. Consumers with read_committed isolation see only committed records.

A broker component that manages transactional producer state, including transaction markers and partition assignments for a given transactional.id, so writes commit or abort atomically.

A set of consumers that jointly consume a topic, with each partition assigned to exactly one member at a time. Group membership and offsets are tracked by the group coordinator.

The broker responsible for a consumer group's membership, heartbeats, and offset commits. It is determined by hashing the group id against the __consumer_offsets partition count.

The process of redistributing partition assignments among consumer group members when membership changes or subscriptions are altered. During a rebalance, consumption pauses briefly.

A consumer configuration using group.instance.id that lets a restarted consumer rejoin without triggering a rebalance within the session timeout, reducing churn in rolling deployments.

The broker address list a client contacts first to discover cluster metadata. Only one reachable address is needed; the client then learns all brokers and partition leaders.

A named network endpoint a broker binds to, combining protocol (PLAINTEXT, SSL, SASL_PLAINTEXT, SASL_SSL), host, and port. Clients connect using the listener name in their security protocol.

The listener addresses a broker publishes to clients in metadata responses. Misconfigured values cause clients to receive unreachable addresses, a common cause of connection timeouts.

The broker configuration listing directories where partition log segments are stored. Multiple directories on separate disks improve throughput and allow log data to span volumes.

A file group holding a contiguous range of a partition's records, rolled when size or time thresholds are reached. Segments are the unit of retention deletion and compaction.

Policy controlling how long or how much log data is kept, set by retention.ms and retention.bytes at topic or broker level. Expired segments are deleted in the background.

A cleanup policy that retains at least the latest record for each key within a partition, discarding older values. Used for changelog and state-restoration topics.

The administrative operation of moving partition replicas to different brokers, typically to rebalance load after adding brokers. It is executed via kafka-reassign-partitions and monitored until completion.

The replica listed first in a partition's assignment, which should ideally be leader. The preferred leader election moves leadership back to it to balance load.

Allowing an out-of-sync replica to become leader when no in-sync replica is available, risking data loss. Controlled by unclean.leader.election.enable, normally disabled.

Broker or client limits on replication and client byte rates, expressed as bytes per second. Used during reassignment and catch-up to avoid saturating network or disk.

The difference between a leader's log end offset and a follower's, indicating how far behind replication is. Sustained lag shrinks the in-sync replica set and threatens durability.

A measurement exposed by Kafka through Java Management Extensions, such as BytesInPerSec or UnderReplicatedPartitions. Monitoring tools scrape these MBeans to detect broker health issues.

A broker-level JMX gauge counting partitions whose in-sync replica count is below the assigned replica count. A sustained non-zero value signals broker or network trouble.

The internal compacted topic storing committed offsets per consumer group. Its partition count determines group coordinator placement and should not be changed casually.

An access control entry granting or denying a principal operations on a resource such as a topic or group. ACLs are enforced when an authorizer is configured on the broker.

The authentication framework Kafka supports for client and inter-broker connections, with mechanisms including PLAIN, SCRAM, and GSSAPI. It is typically layered under TLS.

The identity asserted by an authenticated client, used when evaluating ACLs. For SASL/PLAIN it is the configured username; for mTLS it derives from the certificate subject.