Kafka rebalancing topics between brokers

Engineering Jun 18, 2018

To discard an old set of kafka brokers and migrate them to newer set,

  1. Get the existing list of topics.
  2. Make a file topics-to-move.json with contents in the format:
    {"topics": [{"topic": "sampleTopic_A"}],"version":1}
  3. In any of the kafka brokers, run the following command
./$PATH_TO_KAFKA/bin/kafka-reassign-partitions.sh --generate --zookeeper "zk_node_1:2181,zk_node_2:2181,zk_node_3:2181/kafka" --topics-to-move-json-file /$PATH_TO_ABOVE_FILE/topics-to-move.json --broker-list "<ids of new brokers like 6,7,8>"

It should give a json output with a proposed configuration.
Save it to a file, I'm naming it to immigration.json
It looks like:

  1. Preferably when traffic is low, run the following command:
./$PATH_TO_KAFKA/bin/kafka-reassign-partitions.sh --zookeeper "zk_node_1:2181,zk_node_2:2181,zk_node_3:2181/kafka" --execute --reassignment-json-file $PATH_TO_immigration.json

This step involves migrating topics between kafka brokers and hence, increases pressure on brokers.

5.Run this to verify the progress (based on amount of data, this can go longer):

./$PATH_TO_KAFKA/bin/kafka-reassign-partitions.sh --verify --zookeeper "zk_node_1:2181,zk_node_2:2181,zk_node_3:2181/kafka" --reassignment-json-file $PATH_TO_immigration.json
  1. Once the verification is complete, run the following to re-elect leaders for partitions:
./$PATH_TO_KAFKA/bin/kafka-preferred-replica-election.sh --path-to-json-file $PATH_TO_immigration.json --zookeeper "zk_node_1:2181,zk_node_2:2181,zk_node_3:2181/kafka"
  1. Verify with describe:
./$PATH_TO_KAFKA/bin/kafka-topics.sh --describe --zookeeper "zk_node_1:2181,zk_node_2:2181,zk_node_3:2181/kafka"


You should go to about section on this site.

Great! You've successfully subscribed.
Great! Next, complete checkout for full access.
Welcome back! You've successfully signed in.
Success! Your account is fully activated, you now have access to all content.