Skip to content

Instantly share code, notes, and snippets.

@anosulchik

anosulchik/kafka-reassign-partitions.sh

Last active October 9, 2023 18:41
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save anosulchik/388effd73ecabf5148b7ab56f611059a to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save anosulchik/388effd73ecabf5148b7ab56f611059a to your computer and use it in GitHub Desktop.
Download ZIP

Revisions

  1. anosulchik revised this gist Oct 9, 2023. 1 changed file with 1 addition and 1 deletion.
    2 changes: 1 addition & 1 deletion kafka-reassign-partitions.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -1,7 +1,7 @@
    # https://catalog.workshops.aws/msk-labs/en-US/addingbrokers/reassignpartitions

    $ cat <<EOF > topics-to-move.json
    { "topics": [ { "topic" : "my-topic"}], "version":1}
    { "topics": [ { "topic" : "my-topic"}, { "topic" : "__consumer_offsets"} ], "version":1}
    EOF

    $ cat <<EOF > client.properties
  2. anosulchik created this gist Jun 23, 2023.
    21 changes: 21 additions & 0 deletions kafka-reassign-partitions.sh
    View file
    Original file line number Diff line number Diff line change
    @@ -0,0 +1,21 @@
    # https://catalog.workshops.aws/msk-labs/en-US/addingbrokers/reassignpartitions

    $ cat <<EOF > topics-to-move.json
    { "topics": [ { "topic" : "my-topic"}], "version":1}
    EOF

    $ cat <<EOF > client.properties
    security.protocol=SSL
    # ...
    EOF

    $ export KAFKA_BROKERS="..."

    # Example of repalancing partitions amoung 6 brokers:
    $ bin/kafka-reassign-partitions.sh --bootstrap-server "${KAFKA_BROKERS}" --command-config client.properties --topics-to-move-json-file topics-to-move.json --broker-list "1,2,3,4,5,6" --generate | awk -F: '/Proposed partition reassignment configuration/ { getline; print $0 }' | jq . > expand-cluster-reassignment.json

    $ cat expand-cluster-reassignment.json

    $ bin/kafka-reassign-partitions.sh --bootstrap-server "${KAFKA_BROKERS}" --command-config client.properties --reassignment-json-file expand-cluster-reassignment.json --execute

    $ bin/kafka-reassign-partitions.sh --bootstrap-server "${KAFKA_BROKERS}" --command-config client.properties --reassignment-json-file expand-cluster-reassignment.json --verify