I85PREP
⌘ K
CCAAK study materials
CCAAK study resources

Rolling Upgrade Stalls When the First Broker Is Set to the New Protocol Too Early

CCAAK · Put your knowledge to work in context.

All case studies

Rolling Upgrade Stalls When the First Broker Is Set to the New Protocol Too Early

An 8-broker ZooKeeper-based Kafka 3.4 cluster is being upgraded to 3.7. The runbook says to replace binaries broker by broker and then perform a second rolling restart to finalize the protocol version. During the first broker restart, the operator also sets inter.broker.protocol.version=3.7 on broker 1 because the new binary is installed. Broker 1 starts, but the controller logs UnsupportedVersionException when broker 1 tries to join, and broker 1 does not appear in the broker list. The remaining seven brokers continue serving traffic, with some under-replicated partitions. The operator proposes continuing with broker 2 and setting the same value so the cluster converges faster. Client applications use the Java client 3.4 and a Go client that negotiates API versions. No client upgrade is planned.

1. Why did broker 1 fail to join, and why is continuing to broker 2 with the same setting unsafe?

Read the worked answer

inter.broker.protocol.version controls the protocol version brokers use for inter-broker requests. It must be supported by all brokers in the cluster. Broker 1 was started with 3.7 while the other seven brokers were still running 3.4 and expecting the 3.4 protocol. The controller and older brokers could not process broker 1's newer inter-broker requests, producing UnsupportedVersionException, so broker 1 never joined the active broker list.

Continuing with broker 2 and the same setting is unsafe because it repeats the failure and reduces capacity further. With two brokers removed, more partitions fall below full ISR. If ISR drops below min.insync.replicas, producers using acks=all will fail. Repeated join failures also create avoidable leader elections and URP churn. The correct move is to revert broker 1 to the old protocol setting, bring it back cleanly, and then follow the two-phase upgrade rather than trying to make the cluster converge by removing more brokers.

2. State the correct two-phase rolling-upgrade sequence from 3.4 to 3.7 for a ZooKeeper cluster, including which configs change in each phase and how to verify readiness.

Read the worked answer

Phase 1 - upgrade binaries while keeping the old inter-broker protocol:

  1. Ensure every broker has inter.broker.protocol.version=3.4 in server.properties before any binary change. Do not set it to 3.7 yet.
  2. For one broker at a time: confirm controlled.shutdown.enable=true, stop the broker, install the 3.7 binaries, start it, and verify it registers with the controller.
  3. Before moving to the next broker, verify kafka-broker-api-versions.sh shows the broker, kafka-topics.sh --under-replicated-partitions is empty, and there are no offline partitions.
  4. Repeat until all eight brokers run 3.7 binaries.

Phase 2 - finalize the protocol:

  1. Perform a second rolling restart, setting inter.broker.protocol.version=3.7 on each broker one at a time.
  2. After each restart, wait for full ISR and URP=0 before proceeding.
  3. If the release still uses log.message.format.version, leave it at the old value through phase 1 and update it in a final pass; for 3.4-to-3.7 this is normally unnecessary because old message formats are already unsupported.

Verify readiness with broker logs, controller logs, kafka-broker-api-versions.sh, --under-replicated-partitions, and consumer-group describe output. Revert the incorrect setting on broker 1 first, then restart it and confirm it rejoins before continuing.

3. What client-side risks and verification steps apply to this upgrade?

Read the worked answer

inter.broker.protocol.version affects broker-to-broker communication, not the protocol version clients must use. Kafka clients negotiate API versions, so the Java 3.4 client and the Go client should remain compatible with 3.7 brokers, but this must be tested rather than assumed.

Verification steps:

  • Run the Java 3.4 and Go clients against a staging cluster upgraded with the same two-phase process. Exercise produce, consume, commit, and error paths.
  • Watch broker logs for unsupported request versions or repeated disconnects from those clients.
  • Monitor client-side produce/fetch error rates and consumer rebalances during the upgrade window.
  • Keep clients unchanged during the broker upgrade so any regression can be attributed to the brokers.

Do not upgrade clients and brokers simultaneously. If a client uses a very old message format or an unsupported API, validate it before phase 2. Also avoid using the upgrade window to change topic settings, reassign partitions, or enable new features; isolate variables and keep rollback to the previous binaries straightforward.