Kafka Application Design
Producer and consumer design decisions
Partitioning: the record key is hashed (murmur2) to pick a partition, so the key controls ordering and distribution. A null key uses the sticky partitioner - good for throughput, useless for ordering. Choose keys with high cardinality to avoid hot partitions; if you must change the key of a stream, you must repartition before joining.
Delivery semantics: at-most-once (commit before processing), at-least-once (process then commit; default risk of duplicates), exactly-once (idempotent producer plus transactions, or Kafka Streams processing.guarantee=exactly_once_v2). The idempotent producer is on by default in recent versions and requires acks=all, retries>0, and max.in.flight.requests.per.connection<=5. Transactional consumers must set isolation.level=read_committed to skip aborted records.
Consumer loops: enable.auto.commit=true commits offsets of records returned by poll, not records successfully processed - the classic duplicate/loss trap. Manual commitSync/commitAsync after processing gives at-least-once. Exceeding max.poll.interval.ms triggers eviction and a rebalance mid-batch; tune max.poll.records or use pause/resume instead of long blocking work.
Error handling: retry topics with backoff, a dead-letter topic for poison messages, and idempotent sinks so retries are safe.
Worked example
A consumer takes 6 minutes to process a 500-record batch (max.poll.records=500) while max.poll.interval.ms=300000. The group rebalances every batch and partitions are reprocessed. Fix: lower max.poll.records so each batch finishes well inside the interval, or raise the interval - do not raise session.timeout.ms hoping to fix it; that is a heartbeat concern, not a poll-loop concern.