A few strategies for migrating Kafka to new cluster from classic to VPC:

Strategy 1:

Launch a new cluster (Zookeeper+Kafka brokers),
point producers to new cluster and drain the older cluster once consumers are also pointed to new cluster.

pros:

  • Existing cluster isn't impacted, and hence helps in easy revert.
  • Smooth transition on kafka cluster end. No config changes or zookeeper changes.

cons:

  • In case of revert, potential loss of messages.
  • Consumers may have to take some downtime.
  • Few consumers consume and produce to the cluster. They've to engineer to adapt both kafka clusters in their application.

Strategy 2:

Launch a new cluster (Zookeeper + Kafka brokers),
point producers to both the clusters. Let data in both clusters be eventually consistent. Once both the clusters are identical, drain the older cluster.

pros:

  • Minimal engineering required is on producers (also, those who produce by consumption).
  • Dummy consumers can be put up, and testing can be done somewhat smoothly.

cons:

  • Potential risk of duplicate consumption of messages.
  • Application need changes.

Strategy 3:

Add new brokers one by one, drain one by one.
Once the brokers are migrated, move zookeeper similarly.

pros:

  • applications are agnostic to these changes. No engineering changes required.

cons:

  • Draining a broker would turn out expensive. Cluster would take significant bandwidth and throttle upon network, causing distress to applications.
    There's a way to put throttling limits on bandwidth and let migration happen in controlled manner.
  • Operationally, lengthy procedure. Overall transition time would be in weeks.