CCAAK: Compacted Topic Reaches 92% Disk While Consumers See Duplicate Keys

Compacted Topic Reaches 92% Disk While Consumers See Duplicate Keys

A 9-broker Kafka 3.6 cluster hosts a user-profiles topic with 24 partitions, replication factor 3, and cleanup.policy=compact. Producers emit about 4,000 keyed updates per second for roughly 180 million user IDs, averaging 1 KB per record. Each broker has a 2 TB data volume, log.segment.bytes is 1 GB, and log.retention.ms is 7 days. Six weeks after launch, three brokers report 88–92% disk used, and the cluster is approaching a hard stop. Consumers also report seeing the same key more than once within a single partition fetch. The team proposes setting log.retention.ms=86400000 to free space and asks whether that is safe. No broker restarts or topic configuration changes have been made since launch. Broker log.cleaner.threads remains at 1, max.compaction.lag.ms is not set, and the topic contains no tombstones.

1. Diagnose why compaction is not reclaiming disk and why consumers still see duplicate keys. Which settings and cleaner mechanics explain both symptoms, and what evidence would you collect?

Read the worked answer

Compaction is a background rewrite of inactive segments, not a read-time deduplication guarantee. Two mechanics matter here:

The active segment is never compacted. A key can therefore appear multiple times in a fetch that spans the active segment and segments that have not yet been cleaned.
A segment is eligible only after it rolls and its dirty ratio reaches min.cleanable.dirty.ratio (default 0.5). With max.compaction.lag.ms unset, there is no upper bound forcing the cleaner to process a segment that stays below the threshold or waits behind other work.

With 24 partitions × RF 3 across 9 brokers, each broker owns about 8 replica partitions. One cleaner thread (log.cleaner.threads=1) handling 1 GB segments at roughly 4,000 records/s can fall behind, so old versions remain. log.retention.ms is irrelevant for a compact-only topic; it does not delete segments.

Evidence to collect: kafka-log-dirs.sh --describe for per-partition size, kafka-configs.sh --describe --topic user-profiles, and cleaner JMX such as max-dirty-percent, time-since-last-run-ms, and cleaner-records-cleaned-rate. Look for large uncleanable or persistent dirty bytes. Duplicate keys are expected until cleaning catches up; consumers must deduplicate by key and offset rather than assuming exactly one value per key per fetch.

2. Evaluate the proposed `log.retention.ms=86400000`. What would happen if applied as-is, and what safer combination of topic, broker, and capacity changes would you recommend? Include validation and rollback.

Read the worked answer

Applied as-is, log.retention.ms=86400000 has no effect because cleanup.policy=compact does not use time-based retention. Only a policy containing delete uses retention.ms/retention.bytes. Changing the policy to compact,delete would delete whole segments older than 24 hours, including the latest value for any key not updated recently. That breaks compacted-table semantics and can cause silent data loss; it is not a safe space-reclamation step.

Safer changes:

Keep cleanup.policy=compact. Set max.compaction.lag.ms (for example 86400000) so segments are cleaned even when below the dirty-ratio threshold.
Increase log.cleaner.threads to 2–4 on brokers with high cleaner backlog; this is a static broker setting and needs a rolling restart.
Reduce segment.bytes to 256–512 MB so the cleaner works in smaller units and rolls more frequently. Consider min.cleanable.dirty.ratio=0.3 if dirty ratios stay low.
Address capacity separately: 180 million keys × ~1 KB is about 180 GB per replica before overhead. Add broker/disk capacity or reassign partitions if growth outpaces cleaning.

Validate with kafka-log-dirs.sh before and after, disk-free trends, and cleaner metrics. Roll back topic-level settings with kafka-configs.sh --alter; roll back broker settings by restoring server.properties and restarting one broker at a time.