CCAAK: Lag Rises While the Consumer Group Rebalances Every Few Minutes

Lag Rises While the Consumer Group Rebalances Every Few Minutes

A 16-partition payments-events topic is consumed by the ledger-writer group with eight consumers. For 45 minutes, total consumer lag rises by about 40,000 messages per minute. The group state oscillates between Stable and PreparingRebalance every three to five minutes. Broker metrics show no under-replicated partitions and normal request latency. Consumer metrics show records-consumed-rate near zero during each rebalance, poll-latency-avg around 280 ms, and an average batch processing time of 9 seconds. Each consumer is configured with max.poll.records=500 and the default max.poll.interval.ms. The ledger API called during processing has intermittent 45-second timeouts, and the consumer retries the same record until the API responds. Operations proposes increasing the topic to 32 partitions and doubling consumers to 16.

1. Which metrics or log evidence would you use to confirm the root cause, and what mechanism most likely produces the rebalance loop?

Read the worked answer

The most likely mechanism is exceeding max.poll.interval.ms (default 300000 ms). The consumer fetches up to 500 records, then calls the ledger API for each record. A 45-second timeout plus unlimited retry on the same record can keep the consumer out of poll() for over five minutes. The heartbeat thread keeps sending heartbeats, but the group coordinator removes a consumer that has not polled within max.poll.interval.ms, triggering a rebalance.

Evidence to collect:

Consumer JMX poll-time-max exceeding 300000 ms, correlated with the rebalance times.
rebalance-rate-per-hour and last-rebalance-seconds-ago showing repeated rebalances.
Consumer logs containing Consumer poll timeout has expired or heartbeat failures during rebalance.
API client metrics showing 45-second timeouts and retry attempts on the same offset.
records-lag-max and committed offsets that stall or move backward during rebalances.

poll-latency-avg around 280 ms is not the problem; that measures fetch wait time, not processing time. Normal broker metrics rule out a broker-side outage. The rebalance loop is caused by slow, unbounded processing, not by insufficient partitions.

2. Why will adding partitions and consumers not fix the lag, and what changes would you make to stabilize the group?

Read the worked answer

The group already has eight consumers for 16 partitions, so each consumer owns two partitions. Moving to 32 partitions with 16 consumers keeps the same two partitions per consumer. More consumers do not reduce the time a single consumer spends blocked on a 45-second API timeout, and more concurrent calls may increase pressure on the failing ledger API. The lag is caused by processing stalls and group instability, not by insufficient partition parallelism.

Stabilization changes:

Reduce max.poll.records substantially (for example to 50–100) so a batch cannot occupy the consumer for many minutes.
Put a bounded timeout and retry limit around the ledger call. After the limit, send the record to a dead-letter topic or stop processing that partition with an alert rather than retrying forever.
Raise max.poll.interval.ms only as a safety net (for example 600000) after reducing batch size; a larger interval alone hides the problem.
Use static group membership with a unique group.instance.id per consumer to avoid full rebalances on brief restarts.
Consider CooperativeStickyAssignor to reduce stop-the-world rebalances.
Keep session.timeout.ms and heartbeat.interval.ms separate; heartbeats are not the failing signal here.

Adding partitions may be justified later for throughput, but only after processing is bounded and the downstream API is healthy.

3. What monitoring and alerting would you add to catch this earlier?

Read the worked answer

Add consumer-side alerts on poll-time-max relative to max.poll.interval.ms, for example warn at 50% and page at 70–80%. Alert on rebalance-rate-per-hour above a baseline and on last-rebalance-seconds-ago being repeatedly short for a stable group. Track records-lag-max and per-partition lag, not just total lag, so a single stalled partition is visible.

On the application side, instrument the ledger API call with timeout counts, retry counts, and per-record processing latency. Alert when retries for the same offset exceed a threshold. Correlate rebalances with API latency spikes so the on-call engineer can distinguish a consumer bug from a broker problem.

Also monitor commit-latency-avg and failed commits. Add a dashboard panel showing group state transitions alongside lag and downstream latency. A simple alert on repeated PreparingRebalance transitions within 15 minutes would have flagged this before 45 minutes of lag accumulated.