Confluent Kafka Developer key terms

A single Kafka server that stores partition replicas and serves produce and fetch requests. Brokers register with a controller that tracks cluster metadata and leader assignment.

A named, durable, append-only log that producers write to and consumers read from. A topic is divided into partitions and is the main unit of data organization in Kafka.

An ordered, append-only subsequence of a topic's records stored on disk. Partitions let a topic scale across brokers, and ordering is guaranteed only within one partition.

A monotonically increasing integer identifying a record's position inside a partition. Consumers record their progress by committing the offset of the last processed record.

The number of brokers that keep a copy of each partition. With a factor of three, the cluster can lose two brokers and still retain all data.

The replicas that are fully caught up with the current partition leader. Only in-sync replicas can be elected leader, and acks=all waits for them.

A topic or broker setting giving the minimum number of in-sync replicas that must acknowledge a write for it to succeed when producers use acks=all.

A client that publishes records to a topic, choosing the target partition by key hash, explicit partition number, or a sticky round-robin strategy.

A set of consumers sharing a group id that cooperatively read a topic. Each partition is assigned to one member at a time, giving parallel processing without duplicate reads.

The group protocol that reassigns partitions among consumer group members when members join, leave, or change subscriptions, pausing consumption while the new generation is formed.

The gap between the newest offset in a partition and the last offset a consumer group has committed, showing how far behind processing currently is.

A consumer property deciding where to begin when the group has no committed offset for a partition: earliest, latest, or none, which raises an exception.

A consumer setting that, when true, periodically commits offsets in the background. Turning it off lets the application commit explicitly and control duplicate or lost processing.

The producer durability setting: zero ignores the broker response, one waits for the leader, and all waits for every in-sync replica.

A producer mode that stamps records with a producer id and sequence number so brokers discard duplicates caused by internal retries, avoiding repeated records from network errors.

The contract for how many times a record is processed: at most once with possible loss, at least once with possible duplicates, or exactly once using idempotence and transactions.

A cleanup policy that keeps at least the most recent record for each key and eventually removes older ones, turning a topic into a changelog of current values.

An optional record field hashed to pick a partition. Records sharing a key always go to the same partition, and keys enable compaction and stateful operations.

A client component that converts typed objects into byte arrays before sending. The consumer must use a matching deserializer, often driven by a schema id.

A framework that runs reusable connectors to move data between Kafka and external systems, handling scaling, offset storage, and fault tolerance without custom client code.

A connector that reads data from an external system such as a database table or a file and produces that data into Kafka topics.

A connector that consumes records from Kafka topics and writes them into an external system such as a database, a search index, or an object store.

A Connect plugin that converts Kafka record bytes to and from Connect's internal data representation using formats such as Avro, JSON Schema, or Protobuf.

A lightweight transformation applied per record inside a connector, such as filtering, dropping fields, or masking values, without a separate stream processing application.

A Kafka topic where a connector or stream processor sends records it could not process because of errors, so they can be inspected, fixed, and replayed.

A Connect runtime mode in which several workers share connectors and tasks, storing configuration and offsets in Kafka topics so work survives worker failure.

A service that stores versioned schemas per subject and assigns numeric ids, letting producers and consumers validate and evolve record formats independently of application code.

The name under which schema versions are registered in Schema Registry, usually the topic name with a -key or -value suffix.

A per-subject rule deciding which new schema versions may be registered, for example BACKWARD, FORWARD, FULL, or NONE, controlling how producers and consumers roll out changes.

The integer Schema Registry assigns to a registered schema version. Producers embed it in each record so consumers can fetch the exact schema used to write the bytes.

A compact binary serialization format defined by a JSON schema. It supports schema evolution through default values and is the most common format used with Schema Registry.

A Kafka Streams abstraction for an append-only sequence of records where each record is an independent event; nothing is overwritten by later records with the same key.

A Kafka Streams abstraction for a changelog of key-value updates, where a new record for a key replaces the previous value, representing current state rather than full history.

A table replicated in full to every instance of a Kafka Streams application, enabling lookups and joins without co-partitioning the other side of the join.

Local storage, typically RocksDB backed and changelog backed, that Kafka Streams uses for joins, aggregations, and windows, and restores after failures.

A fixed-size, non-overlapping time window used in stream aggregations, where each record falls into exactly one window based on its event timestamp.

A Kafka Streams capability that exposes local state stores through an API so an application can read its own state and, with remote calls, state held on other instances.

An access control entry that allows or denies a principal a specific operation, such as read or write, on a resource like a topic, consumer group, or cluster.