Rebalance Loop Caused by a Synchronous Downstream Call
ledger-writer is a plain Kafka consumer (no Streams) in group ledger reading payments (24 partitions) with 8 instances. For each record it makes a synchronous HTTPS call to a ledger service: p50 700 ms, p99 2.8 s. max.poll.records is left at the default 500; max.poll.interval.ms, session.timeout.ms and heartbeat.interval.ms are all defaults. Logs show a rebalance every 4-6 minutes, CommitFailedException: Offset commit cannot be completed since the consumer is no longer a member of the group, and the finance team reports duplicate ledger entries for the same payment_id. Someone has already tried raising session.timeout.ms to 120000 with no effect.
1. Explain the mechanism producing the rebalances and the duplicates, and why raising `session.timeout.ms` did nothing.
Read the worked answer
The consumer processes a whole poll batch before polling again. With 500 records at ~700 ms mean, a batch takes ~350 s, which exceeds max.poll.interval.ms (default 300000 ms). Heartbeats continue on the background thread, so the session stays alive - that is exactly why session.timeout.ms is irrelevant here. But when the poll loop exceeds max.poll.interval.ms, the client proactively leaves the group, triggering a rebalance and partition reassignment.
The outgoing consumer may finish processing records whose offsets it can no longer commit (CommitFailedException), so those offsets are committed by neither old nor new owner, and the new owner re-reads them. Combined with at-least-once delivery, the ledger sees the same payment_id twice. The 120000 s session timeout only widens the heartbeat window; it does not move the poll deadline.
2. Rank the remediation options and state the constraint each one imposes.
Read the worked answer
1. Shrink the batch. Set max.poll.records to something like 50 so a batch's worst case (~140 s at p99) sits well under 300 s. Simplest, no code change, but lowers throughput per consumer.
2. Decouple polling from processing. Keep one thread owning poll(), hand records to a bounded worker pool, and call pause()/resume() on partitions when the queue is full. You must keep calling poll() while paused (otherwise the same timeout bites), and you must not commit offsets past an unfinished record - commit the highest contiguous completed offset.
3. Raise max.poll.interval.ms (e.g. 900000). Treat as a stopgap: it hides the tail latency and delays detection of genuinely stuck consumers.
4. Reduce the call cost: batch the ledger API (many payment ids per request) or add a circuit breaker so p99 spikes don't stretch the batch.
Do not simply add instances - 8 consumers already fit within 24 partitions, and more instances means more rebalances on every deploy.
3. Even after the config fix, duplicates remain. What is the correct final line of defence?
Read the worked answer
Configuration only changes the frequency of redelivery; the consumer contract is at-least-once, so the sink must be idempotent.
- Dedupe key: use
payment_id(a business key) or the tuple(topic, partition, offset)as the ledger's upsert key, with a uniqueness constraint so a second write is a no-op rather than a second entry. - Persist the offset with the effect: write the ledger row and its offset in the same database transaction, then commit the Kafka offset after the DB commit. On restart, skip records whose offset is already recorded.
- If the ledger is itself a Kafka topic, the real fix is a transactional read-process-write:
isolation.level=read_committeddownstream andsendOffsetsToTransactionso offsets and outputs commit atomically. That option is off the table while the ledger is an external HTTPS service - you cannot enlist it in a Kafka transaction, which is why the business-key upsert is the correct answer here.