In previous Kafka architectures, a separate ZooKeeper ensemble was responsible for a critical task: managing the cluster's metadata. This included tracking which brokers were active, what topics existed, and which broker was the leader for each partition. While functional, ZooKeeper often became a bottleneck for scalability and added operational complexity. Kafka now manages this internally using a built-in Raft consensus protocol, known as KRaft.

Instead of ZooKeeper, a select group of brokers are designated as controllers. These controllers form a quorum that collectively manages the cluster's state. They use the Raft protocol to ensure that they all agree on every change to the cluster metadata. One broker in this quorum is elected as the active leader. This leader is the single source of truth for metadata changes, while the others remain on standby as followers, ready to take over if the leader fails. This self-contained model simplifies the architecture and removes the external dependency on ZooKeeper.

So, where does this quorum of controllers store the metadata? It uses a mechanism Kafka is already excellent at: a replicated log. All cluster metadata is written to a special, internal topic called __cluster_metadata. Every change, from creating a new topic to a broker going offline, is recorded as a message in this log.

The controllers treat this topic just like any other replicated data topic. The active controller writes new metadata events to the log, and the follower controllers replicate these events. This ensures every controller has an identical, up-to-date copy of the cluster's history. If the active controller fails, a follower with the complete log can be elected as the new leader, guaranteeing a consistent and durable state without data loss.

This approach is far more efficient than the previous method of reading and writing small pieces of data (znodes) in ZooKeeper. Using a log allows for high-throughput, sequential writes, which is a core strength of Kafka's design.

In a KRaft-based cluster, it's important to distinguish between two different types of leader election.

1. Controller Leader Election: This election happens within the controller quorum. The controller nodes use the Raft protocol to elect a single active controller. This process is about deciding which node gets to manage the cluster's metadata. It's a democratic election among a small group of specialized nodes.

2. Partition Leader Election: This election is managed by the active controller for all the data partitions in the cluster. For every partition of every topic, one replica is designated the leader. This leader handles all read and write requests for that partition. When a broker hosting a partition leader fails, the active controller is responsible for appointing a new leader from the remaining replicas.

The key to a safe partition leader election is the In-Sync Replica (ISR) list. This is a list, maintained by the controller, of all the replicas for a given partition that are fully caught up with the leader's log. When the leader fails, the controller will only promote a replica from the ISR list to be the new leader. This guarantees that no committed messages are lost during the failover process. Replicas that fall too far behind are removed from the ISR list and must catch up before they are eligible to become a leader.

Removing ZooKeeper wasn't just about simplification; it was about unlocking a new level of scale and performance. ZooKeeper's consensus protocol was not optimized for the kind of metadata workload a massive Kafka cluster generates.

By moving metadata management into Kafka itself, the system can leverage its own highly optimized, log-based architecture. This allows KRaft-based clusters to support millions of partitions, a scale that was previously impractical. This is a huge advantage for multi-tenant environments and use cases with a very high number of topics. The startup and recovery times are also significantly faster, as the controller can simply replay its metadata log from the __cluster_metadata topic rather than loading state from ZooKeeper.

Ultimately, KRaft makes Kafka more resilient, scalable, and easier to operate, consolidating it into a single, self-sufficient distributed system.

Time to check your understanding of Kafka's modern architecture.

With this architectural foundation, you can build resilient and highly scalable Kafka clusters without external dependencies.