Kafka rebalancing topics between brokers

To discard an old set of kafka brokers and migrate them to newer set,

  1. Get the existing list of topics.
  2. Make a file topics-to-move.json with contents in the format:
    {"topics": [{"topic": "sampleTopic_A"}],"version":1}
  3. In any of the kafka brokers, run the following command
./$PATH_TO_KAFKA/bin/kafka-reassign-partitions.sh --generate --zookeeper "zk_node_1:2181,zk_node_2:2181,zk_node_3:2181/kafka" --topics-to-move-json-file /$PATH_TO_ABOVE_FILE/topics-to-move.json --broker-list "<ids of new brokers like 6,7,8>"

It should give a json output with a proposed configuration.
Save it to a file, I'm naming it to immigration.json
It looks like:

{"version":1,"partitions":[{"topic":"sampleTopic_A","partition":3,"replicas":[11,12]},{"topic":"sampleTopic_B","partition":0,"replicas":[8,9]}]}
  1. Preferably when traffic is low, run the following command:
./$PATH_TO_KAFKA/bin/kafka-reassign-partitions.sh --zookeeper "zk_node_1:2181,zk_node_2:2181,zk_node_3:2181/kafka" --execute --reassignment-json-file $PATH_TO_immigration.json

This step involves migrating topics between kafka brokers and hence, increases pressure on brokers.

5.Run this to verify the progress (based on amount of data, this can go longer):

./$PATH_TO_KAFKA/bin/kafka-reassign-partitions.sh --verify --zookeeper "zk_node_1:2181,zk_node_2:2181,zk_node_3:2181/kafka" --reassignment-json-file $PATH_TO_immigration.json
  1. Once the verification is complete, run the following to re-elect leaders for partitions:
./$PATH_TO_KAFKA/bin/kafka-preferred-replica-election.sh --path-to-json-file $PATH_TO_immigration.json --zookeeper "zk_node_1:2181,zk_node_2:2181,zk_node_3:2181/kafka"
  1. Verify with describe:
./$PATH_TO_KAFKA/bin/kafka-topics.sh --describe --zookeeper "zk_node_1:2181,zk_node_2:2181,zk_node_3:2181/kafka"