Confluent Kafka Developer common mistakes
Confluent Kafka Developer · Understand the traps and how to avoid them.
Treating auto.offset.reset=earliest as a way to replay a topic, and being surprised when a consumer with an existing committed offset resumes at the committed position instead of the beginning.
The setting is frequently described as 'where the consumer starts reading', so candidates memorize it as a replay switch rather than as a fallback used only when no valid committed offset exists (or the offset is out of range).
What to do instead
Memorize the precise trigger: the reset policy applies only when the group has no committed offset for the partition, or the committed offset falls outside the retained range. To actually replay, use a new group.id, seek()/seekToBeginning(), or offset reset tooling.
Adding partitions to an existing topic to increase throughput while assuming per-key ordering is preserved, or assuming the number of partitions can be reduced.
Candidates internalize 'ordering is guaranteed per key' and 'more partitions = more parallelism' without connecting the two: the partition for a key is computed by hashing the key modulo the partition count, so changing the count remaps keys.
What to do instead
State the rule explicitly: ordering is guaranteed only within a partition, and partition count can be increased but never decreased. If keys must stay pinned, create a new topic with the target partition count and migrate, or accept that keys may move partitions.
Writing a consumer loop that commits offsets after processing but never implements ConsumerRebalanceListener, so records from partitions revoked mid-batch are processed and committed after another consumer already owns them.
The happy-path tutorial consumer works fine with one consumer; rebalances only appear with multiple consumers or restarts, so the failure is invisible during local testing.
What to do instead
Implement ConsumerRebalanceListener, commit synchronously in onPartitionsRevoked before partitions are reassigned, and prefer the cooperative sticky assignor so only affected partitions are revoked. Track in-flight offsets and discard results for revoked partitions.
Enabling exactly-once by only setting a transactional.id on the producer (or only setting isolation.level on the consumer) and then claiming the pipeline is exactly-once end to end.
Exactly-once spans three settings in different components, so candidates who learn one knob assume it is the switch for the whole guarantee.
What to do instead
For plain clients: producer transactional.id + initTransactions/beginTransaction/sendOffsetsToTransaction/commitTransaction, and consumer isolation.level=read_committed. For Kafka Streams: processing.guarantee=exactly_once_v2. Sinks must also be idempotent or transactional.
Producing records with a null key and expecting global ordering, or lowering acks to 0/1 and assuming no data loss is possible.
The default partitioner behavior for null keys (round-robin/sticky) and the durability trade-offs of acks are easy to skim past because producers 'just work' with defaults.
What to do instead
Recall that a null key means records spread across partitions with no ordering guarantee - supply a meaningful key when order matters. For durability use acks=all with enable.idempotence=true and min.insync.replicas >= 2 on the topic.
Assuming two topics can be joined in Kafka Streams or a stream can be joined to a table without checking partition counts and keying, leading to records silently never matching.
Joins look like ordinary in-memory operations in the DSL, and the co-partitioning requirement lives in the details rather than the method signature.
What to do instead
Before a join, verify both sides use the same key and the same partition count; otherwise selectKey/groupByKey with a repartition topic (or through()) so both sides are co-partitioned. Remember stream-stream joins also require a window.
Reaching for Single Message Transforms in Kafka Connect to do filtering, routing, joins, or aggregation across records, or expecting SMTs to see deserialized business objects with full type fidelity.
SMTs and Streams both appear under 'transform data', so candidates conflate per-record stateless transformation with stateful stream processing.
What to do instead
Use SMTs only for stateless, single-record changes (field masking, topic renaming, dropping fields, ExtractField/InsertField). Any stateful work - joins, windows, aggregations, cross-record dedup - belongs in Kafka Streams or a stream processing engine.
Deploying a connector with a value.converter that does not match the actual data format - for example pointing a connector at Avro data with the JSON converter or the plain JSON converter without schemas.enable - and then debugging the resulting serialization exceptions or mangled payloads.
Converters and serializers are often conflated; candidates configure 'the JSON one' without realizing the converter must match the exact wire format written by the upstream producer.
What to do instead
Match converter to wire format: AvroConverter with schema.registry.url for Avro, ProtobufConverter for Protobuf, JsonConverter with schemas.enable matching the producer, and ByteArrayConverter/StringConverter for raw payloads. Set converters on both the worker and the connector where overrides are needed.
Assuming Schema Registry subjects are always <topic>-value and that registering a differently shaped value under an existing topic is fine, without reasoning about the subject naming strategy or the configured compatibility level.
TopicNameStrategy is the default and matches most tutorials, so the strategy is treated as fixed rather than as a configurable decision that changes compatibility semantics.
What to do instead
Know the three strategies: TopicNameStrategy (-key/-value), RecordNameStrategy (record full name), TopicRecordNameStrategy (topic + record name). Change the strategy when multiple event types share a topic, and check the subject's compatibility level (BACKWARD default; FULL/TRANSITIVE for stricter rules) before evolving schemas.
Granting a service account only Kafka topic-level ACLs or RBAC roles and then expecting Connect, Schema Registry, or ksqlDB to work, or assuming an ACL on a topic implicitly covers its consumer group or internal topics.
Governance is studied as a separate chapter, so the cross-component permission surface - group ACLs, transaction IDs, internal Connect topics, Schema Registry subject-level RBAC - is easy to forget.
What to do instead
Enumerate the required permissions per component: READ/WRITE on data topics plus READ on the consumer group and transactional IDs, Connect needs its config/offset/status internal topics, Schema Registry needs subject-level resource permissions, and RBAC roles must be assigned at the correct scope (cluster, environment, or resource).