Kafka Fundamentals
The mental model
A Kafka topic is a named, partitioned, replicated append-only log. Each partition is an ordered sequence with monotonically increasing offsets; ordering is guaranteed only within a partition, never across a topic. One broker is the partition leader (all produce and, in modern clients, consume traffic); followers replicate. The ISR (in-sync replica set) is the subset of replicas that are caught up within replica.lag.time.max.ms.
Durability knobs: acks (0, 1, all), topic replication.factor, and min.insync.replicas interact. acks=all means all in-sync replicas, not all configured replicas.
Retention vs compaction: cleanup.policy=delete drops whole log segments whose newest record is older than retention.ms (or when retention.bytes is exceeded) - a record can outlive the configured time until its segment rolls. cleanup.policy=compact retains the last value per key forever, so it is not a size- or time-bounded window; a null value is a tombstone that removes the key after delete.retention.ms.
Trap cluster: auto.offset.reset only fires when no committed offset exists (not on lag); a consumer group can never have more active consumers than partitions; unclean.leader.election is enabled by trading durability for availability; __consumer_offsets is a compacted internal topic.
Rule of thumb
RF=3, min.insync.replicas=2, acks=all tolerates one broker failure for writes; with two brokers down producers get NotEnoughReplicas. Note that min.insync.replicas is only enforced when acks=all - pairing it with acks=1 gives you nothing.