AZ Outage Takes 14 Topics Offline After Rack Configuration Is Left Empty
An operations team deploys a new Kafka 3.7 cluster with six brokers, two in each of three availability zones, and sets default.replication.factor=3, min.insync.replicas=2, and producer acks=all. The deployment template leaves broker.rack unset because a variable resolves to an empty string. Applications auto-create 60 topics with three replicas. Three weeks later, AZ-B loses power. Four brokers remain healthy, but 14 topics reject writes with NotEnoughReplicasException, and monitoring shows 180 under-replicated partitions. Reads continue for most partitions. The team expected that losing one AZ would leave two replicas per partition and that acks=all would still succeed. They want to know why the remaining four brokers cannot absorb the load and whether adding two more brokers in AZ-A and AZ-C will fix it.
1. Explain why 14 topics became write-unavailable despite four brokers remaining. What does `min.insync.replicas` actually evaluate against?
Read the worked answer
min.insync.replicas is evaluated per partition against the current ISR, not against the number of live brokers in the cluster. With broker.rack empty, auto-created topics were assigned replicas by broker ID rather than by zone. A partition can therefore have two of its three replicas on brokers in AZ-B.
When AZ-B fails, those partitions drop to an ISR of 1. The producer uses acks=all, and the partition requires at least 2 in-sync replicas, so the broker rejects the write with NotEnoughReplicasException. Four healthy brokers do not help because the missing replicas are specific to those partitions. Reads can continue because at least one replica remains (six brokers, two per AZ, so at most two replicas of a partition can be in one AZ) and a new leader can be elected from a surviving replica.
The 180 under-replicated partitions include partitions that lost replicas but still have ISR ≥ 2 and can accept writes, plus partitions that are fully write-blocked. The key diagnostic is to inspect the replica list per partition and map each broker ID to its rack.
2. How would you verify replica placement and correct it? Give the order of operations, including client-visible risks.
Read the worked answer
Verification:
- Run
kafka-topics.sh --describeand capture the replica broker IDs for each partition. - Map every broker ID to its rack from each broker's
server.properties(or from AdminClientdescribeCluster().nodes(), which exposes the rack). - Count distinct racks per partition. Any partition with fewer than three distinct racks is vulnerable.
- Use
kafka-topics.sh --under-replicated-partitionsand--unavailable-partitionsto scope impact.
Correction order:
- Fix
broker.rack=az-a,az-b, oraz-con every broker. Roll restart one broker at a time withcontrolled.shutdown.enable=true; wait for URP=0 and full ISR before moving to the next broker. Existing assignments do not change automatically. - Generate a rack-aware reassignment. Use Cruise Control's rack-aware strategy or compute explicit assignments that place one replica in each AZ. Apply in batches with
kafka-reassign-partitions.sh --execute --throttle; throttling prevents network saturation. - Re-verify distinct racks per partition and RF=3.
Client-visible risk: reassignment moves data and can temporarily increase replication traffic and ISR churn. Run during low traffic, monitor URP, and avoid combining reassignment with broker restarts. Do not lower min.insync.replicas as a permanent fix.
3. Will adding two brokers in AZ-A and AZ-C fix the problem? Justify your answer.
Read the worked answer
No. Adding brokers increases capacity but does not change existing replica placement. Kafka does not automatically move existing partitions to new brokers; new brokers receive partitions only through reassignment or new topic creation. The 14 write-blocked topics would still have two replicas in AZ-B and fail the same way on the next AZ-B outage.
After adding brokers, the cluster might have three brokers in AZ-A, three in AZ-C, and two in AZ-B. That is still asymmetric, and an AZ-B failure would leave six brokers. Capacity may improve, but availability still depends on each partition having replicas in distinct racks. The required step is a rack-aware reassignment after broker.rack is correctly set. Adding brokers without reassignment can even create new topics with skewed placement if auto-create remains enabled. The durable fix is correct rack metadata plus controlled reassignment, verified partition by partition.