Confluent Kafka Developer cheat sheet

A topic is split into partitions; ordering is guaranteed only within one partition, never across partitions of the same topic.

Each record has a partition-specific offset that is unique and monotonically increasing within that partition.

Replication factor N stores N copies; acks=all plus min.insync.replicas=R requires at least R replicas acknowledged before a produce succeeds.

A consumer group assigns each partition to at most one consumer, so maximum useful consumer parallelism equals the partition count.

Log compaction retains at least the latest record per key instead of deleting by time or size; it is configured with cleanup.policy=compact.

The idempotent producer prevents duplicate writes caused by retries; it is enabled by default in modern clients and requires acks=all.

Exactly-once delivery uses transactions, a transactional.id, and consumers configured with isolation.level=read_committed.

KRaft replaces ZooKeeper for metadata management; controllers store cluster metadata in an internal Raft log.

Record key determines partition assignment; identical keys land on the same partition, preserving per-key order for consumers.

max.poll.interval.ms bounds processing time between poll calls; exceeding it triggers a rebalance and partition revocation.

Manual commits let an application control at-least-once behavior; commit only after downstream processing completes.

Exactly-once stream processing requires a transactional producer, read_committed consumers, and transactional offset commits.

ErrorHandlingDeserializer wraps a delegate deserializer and routes poison records to an error handler instead of failing the poll loop.

Co-partitioning requires joined topics to have the same partition count and matching key partitioning strategy.

An idempotent consumer deduplicates by a business key or offset store so repeated deliveries do not create duplicate side effects.

Schema compatibility must be verified before deployment; a breaking change requires a new subject or a versioned topic.

Kafka Connect runs connectors inside worker processes; connectors split work into tasks that execute in parallel.

Source connectors ingest external systems into Kafka topics; sink connectors export Kafka topics to external systems.

Converters serialize and deserialize Connect data, commonly Avro, JSON Schema, or Protobuf, and integrate with Schema Registry.

Single Message Transforms modify individual records in flight, such as masking fields, routing, or flattening, without a separate stream processor.

Connect stores configuration, offsets, and status in internal topics named config.storage.topic, offset.storage.topic, and status.storage.topic.

A dead letter queue captures records that repeatedly fail so the connector can continue processing other records.

Change data capture streams row-level database changes into Kafka, typically with one topic per source table.

Sink connectors that support exactly-once use transactional writes or idempotent upserts keyed by the record key.

Schema Registry stores versioned schemas under subjects; a subject commonly maps to a topic-value or topic-key schema.

BACKWARD compatibility allows new schemas to read old data; FORWARD allows old schemas to read new data; FULL requires both.

Kafka Streams distinguishes KStream, an append-only record stream, from KTable, a changelog-backed table of latest values per key.

A GlobalKTable is fully replicated to every stream task and supports joins without co-partitioning.

ksqlDB persistent queries run continuously and write results to a new topic; transient queries return a current result and stop.

RBAC grants roles to principals at organization, environment, cluster, topic, group, or transactional-id scope.

A service account is a non-human identity used by applications and Connect clusters to authenticate with API keys.

Confluent Cloud API keys are scoped to a resource and paired with a secret; they authenticate clients and administrative tools.