Confluent Kafka Developer: Flash-Sale Hot Key vs. Repartitioning a Joined Stream

Flash-Sale Hot Key vs. Repartitioning a Joined Stream

The orders topic has 6 partitions, RF=3, and producers key records by merchant_id using the default partitioner. Peak traffic is 12k records/s. During a flash sale one merchant, M-1042, accounts for roughly 45% of all events. Consumer group order-loader runs 6 instances; partition 3 holds 900k records of lag while the other five sit near zero. A Kafka Streams job enriches orders with a KTable built from the compacted merchants topic (also 6 partitions), joining on merchant_id. Two changes are queued for the same window: raise orders from 6 to 12 partitions and add 6 loader instances, and re-key the producer as merchant_id + "-" + random(0,9) to spread the hot merchant. Both are scheduled together, with a 30-minute rollback budget.

1. Evaluate the proposal to raise `orders` from 6 to 12 partitions. Does it relieve the hot partition, and what else does it affect?

Read the worked answer

It does not relieve the hot key. Partitioner assignment is hash(key) % numPartitions, and all records for M-1042 still share one key, so they land on exactly one of the 12 partitions. You spread load only if the number of distinct hot keys exceeds the partition count.

What it does break:

Key-to-partition remapping. Roughly 5/6 of existing keys move to a different partition, so any consumer holding partition-local state (Streams stores, local caches) sees reshuffled input, and per-key ordering is only preserved for records produced after the cutover.
Co-partitioning. The Streams join requires orders and merchants to have the same partition count. At task assignment the assignor fails co-partitioning validation (Topics not co-partitioned) and the job will not run until merchants is also rebuilt with 12 partitions, which in turn requires repartitioning the compacted table topic.

Adding loader instances is also neutral: extra consumers sit idle because there are no free partitions.

2. Evaluate the re-key proposal (`merchant_id + "-" + random(0,9)`). What does it buy and what does it silently destroy?

Read the worked answer

It buys spreading: each merchant's traffic is split across up to 10 keys, so M-1042's 45% is distributed over 10 partitions rather than pinned to one. Throughput improves and the loader scales with instances.

It destroys per-merchant ordering, and ordering is the only reason to key by merchant. Kafka guarantees order within a partition, not within a key spread over partitions. The loader will happily write merchant 1042's refund before its authorization, and any downstream balance or state machine breaks.

If ordering per merchant is a hard requirement, the standard pattern is two-stage:

Produce with merchant_id|bucket so ingest parallelises.
In a second stage, re-key back to merchant_id and aggregate into a single-partition-per-merchant topic (a repartition topic), where the aggregation is single-writer per key.

This trades a second hop and a repartition topic for ordering. Also: random suffixes must be deterministic per record (e.g. derived from order id) if you ever need to locate a record again.

3. You have one change window and a 30-minute rollback budget. What do you actually do, in order?

Read the worked answer

Buy time cheaply first. Raise loader instances to 6 (already at 6, so no gain) - instead reduce per-record cost: batch the downstream writes and check whether the loader, not the broker, is the bottleneck. Confirm the hot partition is genuinely CPU-bound and not blocked on the sink.
Do not repartition orders in this window. A partition increase forces a matching rebuild of the merchants table topic and a Streams restart, which will not fit in 30 minutes with a safe rollback. Split it into its own change.
Relieve the hot key without breaking ordering by moving M-1042 traffic to a dedicated topic (orders.hot) with its own partition count and consumer group, using a producer-side routing rule keyed off the merchant allowlist. Ordering per merchant is preserved because one merchant still has one writer path. This is reversible by flipping the routing list.
Plan the real repartition offline: create orders.v2 with 12 partitions, dual-write, rebuild merchants at 12, migrate consumers, then cut over. Never increase partitions on a topic whose consumers rely on key affinity.