Read-Only Log Directory Leaves 36 Partitions Offline Without a Broker Crash

Kafka writes to each log directory independently. When broker 7 received I/O errors on /data/kafka/3, the log directory failure handler marked that directory offline. Partitions whose logs lived only on that directory were taken offline on broker 7. For RF=3 partitions, the controller elected new leaders from replicas on other brokers, and the partitions remained available if ISR stayed at or above min.insync.replicas=2. The broker still heartbeats because the process is alive, so this is not a broker failure from the controller's perspective.

Restarting first is risky for three reasons:

• It does not fix the read-only SAN mount. The broker will fail the same directory again, possibly after a costly restart.
• It can trigger unnecessary leader elections and re-registration while the cluster is already handling 240 under-replicated partitions.
• If the broker cannot start cleanly, all partitions on broker 7, including those on healthy log directories, may be unavailable until it returns.

For RF=1 topics, there is no other replica. The controller cannot elect a leader, so those partitions are offline and producers fail. A restart may bring them back only if the volume is healthy and the log data is intact; it cannot create a missing replica.

Recovery sequence:

1. Treat storage first. Check mount, dmesg, and SAN path status; confirm whether /data/kafka/3 is still read-only. Do not delete Kafka log or checkpoint files.
2. Inspect scope with kafka-log-dirs.sh --describe --bootstrap-server ... and kafka-topics.sh --describe --unavailable-partitions; also capture --under-replicated-partitions.
3. If the SAN team confirms the path is unstable, keep broker 7 running but expect its failed directory to remain offline. Fix or replace the LUN before restarting.
4. Once the volume is writable, restart broker 7 during a controlled window. With controlled.shutdown.enable=true, leaders on healthy directories move first. After restart, watch for the directory to come online and for URP to fall.
5. Run kafka-log-dirs.sh again to confirm the directory is online, then wait for URP=0 and full ISR before closing the incident.

RF=1 topics: if the volume recovers with logs intact, broker 7 may bring the partitions online on restart. If the data is lost, recreate the topics with RF=3 and replay from the upstream source. Do not enable unclean.leader.election.enable to force them online; with RF=1 there is no alternative replica, and with RF=3 it risks data loss.

Prevent RF=1 topics from being created. Set default.replication.factor=3, set min.insync.replicas=2 on critical topics, and disable auto.create.topics.enable so applications cannot silently create weak topics. Add a policy check that scans kafka-topics.sh --describe for RF < 3 and alerts topic owners.

Monitor at the partition and log-directory level: OfflinePartitionsCount, UnderReplicatedPartitions, and broker logs for KafkaStorageException and log-directory failure messages. Alert on any offline partition and on a log directory transitioning offline. Do not rely only on broker liveness, because this broker stayed up.

Storage design: ensure each log directory is on an independent path and that a SAN failover cannot remount all LUNs read-only at once. Confirm mount options and path health monitoring. For large clusters, keep metadata.log.dir separate where applicable so metadata writes are not blocked by a data-volume failure. Finally, test a single log-directory failure in a non-production cluster to confirm alerts and failover behavior.