Kafka without ZooKeeper: KRaft explained, and how to move to Kafka 4

Kafka no longer uses ZooKeeper. Since Kafka 4.0 it runs only in KRaft mode, where a few Kafka servers act as controllers and keep the cluster's metadata themselves, in a replicated log inside Kafka. If your cluster still runs on ZooKeeper, you cannot upgrade it to Kafka 4 directly: you migrate it to KRaft on Kafka 3.9, the bridge release, and upgrade after. A cluster already on KRaft, on 3.3 or newer, upgrades to the current release, 4.4, with an ordinary rolling upgrade.

What KRaft changes

Kafka always needed somewhere to keep its metadata: which brokers are alive, which topics exist, which broker leads each partition, the configs and access rules. For most of its life that place was ZooKeeper, a separate system with its own servers to run. Any broker could become the controller, and it kept the metadata in ZooKeeper.

KRaft moves that metadata into Kafka. You choose a few servers to be controllers. One is active and the others are hot standbys, and together they keep the metadata in a log replicated among them. Brokers read the metadata from the controllers, and there is no second system to run beside Kafka.

Each Kafka server gets a role through process.roles:

Apache Kafka says combined mode is not recommended in critical deployments: you cannot roll or scale the controllers apart from the brokers.

How many controllers to run

Run 3 or 5. A majority of the controllers must be up for the cluster to keep working, so the count sets how many failures you survive. Choose the route your cluster needs, and then the controller count.

Try itYour route to Kafka 4.4

ZooKeeperKRaft

Kafka version

·Older than 3.3·3.3 to 3.8·3.9·4.0 or newer

35

Migrate on 3.9 first

1. 1Upgrade the ZooKeeper cluster to 3.9.1 and set inter.broker.protocol.version=3.9. 3.9 is the bridge release: the only one that can migrate.
2. 2Start new KRaft controllers formatted with the cluster ID from ZooKeeper, with zookeeper.metadata.migration.enable=true.
3. 3Restart the brokers one by one in migration mode. The metadata copies over when the last one restarts.
4. 4Restart each broker as a KRaft broker: node.id instead of broker.id, process.roles=broker, ZooKeeper settings removed.
5. 5Finalize: remove zookeeper.metadata.migration.enable from each controller and restart them one at a time. After this there is no way back to ZooKeeper, and ZooKeeper can be shut down.
6. 6Rolling upgrade to 4.4.

Controller quorum

C1C2C3

1 down, 2 still up: still a majority, so the cluster keeps its metadata.

3 controllers survive 1 failure and 5 survive 2. An even number buys nothing: 4 controllers still need 3 up, so they survive only 1 failure, the same as 3. For sizing the brokers themselves, partitions and disk, our Kafka sizing calculator does the sums, and how many Kafka partitions you need explains the rule behind them.

Setting up a new KRaft cluster

A KRaft cluster does not format its own storage. You run kafka-storage.sh format on every server yourself. Apache Kafka explains why: formatting by itself can hide an error, and if a majority of controllers started with empty logs, a leader could be elected that is missing committed data.

# 1. Make a cluster ID once
bin/kafka-storage.sh random-uuid

# 2. Format the first controller as the only voter (Kafka's recommended way)
bin/kafka-storage.sh format --cluster-id <cluster-id> --standalone \
  --config config/controller.properties

# 3. Check the quorum
bin/kafka-metadata-quorum.sh --bootstrap-server localhost:9092 describe --status

Then add the other controllers to the running quorum. A quorum formatted like this is dynamic: servers find it through controller.quorum.bootstrap.servers, and you can add and remove controllers later. The older static style lists every controller in controller.quorum.voters, and the 3.9 docs say a static quorum cannot yet be converted to a dynamic one. To see which kind you have, run kafka-features.sh describe: a kraft.version of 0 or none means static, 1 or more means dynamic. Before you shut down a controller for good, remove it from the quorum first with remove-controller.

Migrating from ZooKeeper to KRaft

The migration runs on Kafka 3.9, the bridge release; Kafka 4 has no ZooKeeper mode left to migrate from. The 3.9 guide names 5 phases: initial, metadata load, hybrid (a KRaft controller with some brokers still in ZooKeeper mode), dual-write (all brokers on KRaft, the controller still writing metadata to ZooKeeper) and finalized.

1. Upgrade to 3.9.1 and set inter.broker.protocol.version=3.9 on every broker. Don't change the metadata version again until the migration is done; Kafka warns this can break the cluster.
2. Start the KRaft controllers with the same cluster ID as the ZooKeeper cluster, which zookeeper-shell.sh localhost:2181 get /cluster/id prints. Their config adds zookeeper.metadata.migration.enable=true and zookeeper.connect. Their node.id values must not clash with any existing broker.id, because brokers and controllers now share one ID space.
3. Restart the brokers in migration mode, one at a time, with the same migration flag and the controllers' address. When the last one restarts, the copy starts, and the active controller logs "Completed migration of metadata from Zookeeper to KRaft".
4. Restart each broker as a KRaft broker: broker.id becomes node.id with the same number, add process.roles=broker, and remove the ZooKeeper settings. If you use AclAuthorizer, switch to StandardAuthorizer now.
5. Finalize. Remove zookeeper.metadata.migration.enable from each controller and restart them one at a time. Then you can shut ZooKeeper down.

Until step 5 you can go back to ZooKeeper; the 3.9 docs have a table for undoing each step. After step 5 you cannot. Rehearse on a copy of the cluster before you touch production.

Upgrading to Kafka 4.4

Once the cluster is on KRaft at 3.3 or newer, the move to 4.4 is a rolling upgrade: one broker at a time, stop it, update the code, start it. A KRaft cluster older than 3.3 should go to 3.9 first, because KRaft before 3.3 was not considered ready for production.

Kafka 4 also raised the floor for everything around the brokers. Check these before the upgrade, not after:

• Java. Brokers, Kafka Connect and the tools need Java 17. Clients and Kafka Streams need at least Java 11.
• Logging. Kafka moved from Log4j to Log4j2, so custom logging configs need rewriting.
• Old clients. 4.0 removed old versions of the protocol's APIs, so every client, Connect and Streams included, must be on 2.1 or newer before the brokers move to 4.0. Clients from outside Apache Kafka are covered in KIP-896.

If you are deciding whether to run Kafka at all for a lakehouse feed, Databricks Zerobus vs Kafka compares the two for writing straight into Delta tables. For an oversized record problem on the way, see Kafka's maximum message size.