Kafka Rebalance Storm and Duplicate Clickstream Records
A retail clickstream platform runs Apache Kafka 3.6 on five brokers. The topic has 12 partitions and peaks at 45,000 messages/s. Six NiFi nodes consume with a Kafka consumer group, enrich each record through a synchronous HTTP lookup averaging 600 ms, and write to HDFS. Consumer lag has reached 2.1 million records, rebalances occur roughly every eight minutes, and 0.4% of records arrive duplicated downstream. Current settings include max.poll.records=500, max.poll.interval.ms=300000, session.timeout.ms=10000, and heartbeat.interval.ms=3000. Each NiFi node runs one consumer. The business requires per-session ordering for clickstream reconstruction and reliable delivery into the lakehouse. Adding brokers is possible but takes two weeks; adding NiFi nodes is immediate.
1. Diagnose the rebalance and duplicate pattern from the numbers, not from general Kafka advice.
Read the worked answer
The rebalance interval is explained by the poll budget. With 500 records per poll and 600 ms of synchronous enrichment per record, one poll cycle takes about 300 seconds. That equals max.poll.interval.ms=300000. The consumer finishes processing at the boundary, then may fail to call poll again before the coordinator expires it, so the group rebalances every time a batch is processed. The eight-minute observation is consistent with 300 seconds of processing plus rebalance and catch-up time.
The 10-second session timeout is not the primary cause, because heartbeats run on a background thread during processing. However, 10 seconds is tight for GC pauses and network jitter, so it can add spurious rebalances once the group is already unstable.
Duplicates follow from at-least-once consumption. Offsets are committed after the HDFS write. When a rebalance occurs mid-batch, records already written but not committed are re-delivered to the new owner. A 0.4% duplicate rate is plausible for 2.1 million lag and frequent ownership changes.
The synchronous 600 ms HTTP call is the real amplifier: it makes each batch exceed the poll contract and caps throughput at roughly 1.7 records/s per consumer, far below what six consumers need at 45,000 messages/s.
2. Specify the configuration and topology changes you would make now, without waiting for new brokers, and explain the ordering implications.
Read the worked answer
First, restore a valid poll contract. Reduce max.poll.records to 50, which gives about 30 seconds of processing at 600 ms per record, or decouple enrichment so the HTTP call is asynchronous and bounded. Raise max.poll.interval.ms to 600000 as a safety margin, and set session.timeout.ms to 30000 with heartbeat.interval.ms to 10000. These changes are restart-only and can be rolled out one NiFi node at a time.
Second, add NiFi nodes. With 12 partitions, the group supports at most 12 active consumers; going from six to twelve doubles parallelism and halves per-consumer lag. This is immediate and does not alter key-to-partition mapping.
Third, address ordering before expanding partitions. Per-session ordering depends on all events for a session key landing on the same partition. Increasing partitions from 12 to 60 changes hash(key) % partitions, so in-flight sessions can be split across old and new partitions during the transition. Either pause producers briefly during the change, dual-publish to an expanded topic, or use a custom partitioner with a stable mapping. Adding consumers is safe; adding partitions is not ordering-neutral.
Finally, stop treating the sink as exactly-once by configuration alone. Use an idempotent write keyed on topic, partition, and offset, or a MERGE on a deterministic event id, so replays after rebalance do not duplicate rows.
3. The business insists on per-session ordering and exactly-once delivery into the lakehouse. What can you guarantee, and how would you prove it?
Read the worked answer
Kafka can guarantee order within a partition, and the consumer group preserves partition ownership, so per-session ordering is achievable if the producer keys by session id and no partition change occurs mid-flight. Exactly-once end-to-end is a stronger claim: NiFi consumption is at-least-once unless the entire read-process-write path is transactional or idempotent. The honest design is ordered at-least-once consumption plus idempotent sink writes, which yields effectively-once results.
Implementation: key by session id, keep the partition count stable during a traffic window, and write to HDFS or Iceberg with a deterministic event id derived from topic-partition-offset or a producer-supplied UUID. Use MERGE or deduplication on that id. Commit the consumer offset only after the sink write is durable, accepting replays rather than data loss.
Proof should be empirical:
- Inject a controlled rebalance during a known batch and verify no duplicate event ids survive the sink.
- Replay an offset range and confirm row counts and per-session sequence remain correct.
- Run a reconciliation query comparing distinct event ids and expected counts per session.
- Monitor consumer lag, rebalance rate, and duplicate-rejection counts as ongoing evidence.
If the business truly requires transactional exactly-once, the write path must be a transactional producer or a sink with atomic commit; otherwise state the guarantee precisely rather than overpromising.