Ordering, Rebalances and Disk Pressure on an Event Streams Rating Pipeline

The rebalances are almost certainly poll-interval driven. A consumer is removed from the group if it does not call poll within max.poll.interval.ms (default five minutes). A 500-record batch taking seven minutes when the rating engine is slow exceeds that, so the pod is evicted, its partitions are reassigned, and the same records are delivered to another pod because offsets were never committed. That is the duplicate rating mechanism, and it also explains the rebalance frequency: each eviction triggers another round.

Changes, in order of value:

1. Reduce records per poll. 500 records at 45 seconds to seven minutes is a huge unit of work. Cap max.poll.records so a batch completes well inside the interval, and let progress be committed more often. This alone typically eliminates the storm.
2. Raise max.poll.interval.ms deliberately, not blindly. It must exceed worst-case processing time with margin. Setting it very high hides a stuck consumer for longer, so pair it with liveness checks outside the poll loop.
3. Use static group membership where supported, so a pod that restarts briefly or is rescheduled does not trigger a full rebalance of the group.
4. Use a cooperative assignor so a rebalance does not stop every consumer while only a few partitions move.
5. Make rating idempotent. Rebalances, restarts and rescaling will always produce some redelivery. Deduplicate on a natural key (subscriber plus usage event identifier) at the sink, so at-least-once delivery does not become double billing. This is the durable fix; the tuning above only reduces frequency.

Also confirm the six pods map sensibly to six partitions: six consumers on six partitions means any membership change leaves a pod idle, and there is no headroom for a rolling restart without a rebalance.

Cause: no key. With a null key the producer distributes records across all six partitions, and Kafka guarantees order only within a partition. Two events for the same subscriber can land on different partitions and be consumed in parallel, so the order the consumer sees is not the order the producer sent. Any downstream logic that assumes latest-state-wins is therefore wrong.

Fix: key by subscriber. Keying on the subscriber identifier sends all events for that subscriber to one partition, which restores per-subscriber ordering while keeping parallelism across subscribers. Accept the consequence honestly: distribution becomes as uneven as the subscriber base is uneven. A few large subscribers can create hot partitions, so monitor per-partition byte rates and produce rates, not just totals.

Interaction with 24 partitions - this is the part the proposal misses:

• Increasing partition count changes the mapping from key to partition for most keys. During and after the change, events for a subscriber can be written to a different partition than before, while unsent or in-flight events for that subscriber are still heading to the old one. The ordering guarantee you just built is broken exactly across the change window. Plan it as a migration: drain and pause production for the topic, or dual-write and reconcile, rather than resizing live and hoping.
• More partitions also means more open files, more replication traffic and more consumer slots per broker. Moving from 6 to 24 partitions with three brokers changes broker-side resource use, which is the opposite of what you want while disks are at 71 percent and climbing.
• Partitions are the unit of parallelism, not a throughput dial. If the consumer is the bottleneck, more partitions with the same six pods changes nothing about capacity.

Do the keying fix, tune the consumer, and only then consider partition count as a capacity decision made with the migration plan written down.

Durability consequences are real and asymmetric. Dropping from replication factor 3 to 2 means any single broker loss removes the only other copy of some partitions; and because min.insync.replicas is 2, a producer with acks=all needs both replicas available. With replication factor 2 and min.insync.replicas 2, taking one broker down for maintenance or a restart makes those partitions unavailable for writes - the pipeline stops. You have traded disk for availability, which is the wrong direction for a rating pipeline that must not lose usage records. You also lose the ability to survive an unclean event without data loss on the affected partitions. Do not do this to save disk.

Alternatives, cheapest and least risky first:

1. Shorten retention where the business allows it. Seven days at 4 percent per day growth means you are storing roughly a week of data that the rating pipeline consumes within minutes. Set retention by time per topic according to actual consumer lag and replay requirements, not by a default applied to everything.
2. Increase broker disk or add brokers. The direct fix. Adding a broker also redistributes existing partitions and buys throughput headroom.
3. Reduce bytes on the wire. Check compression type and average record size; usage records with verbose JSON payloads compress well. A change here reduces disk, network and replication traffic simultaneously.
4. Find the real growth source. Four percent per day on a fixed input rate suggests either a volume increase or a retention policy that is not being enforced as expected. Verify actual segment sizes against configured retention before assuming you need more capacity.

Present the trade-off explicitly: replication factor 3 is a durability decision that was made for a reason. Relieving disk pressure by weakening it usually costs more during the next incident than the storage would have.