Skip to content

[Breaking Change] Multi-Cluster Kafka Support
Compare
Choose a tag to compare
@sciclon2 sciclon2 released this 06 Sep 10:59
· 75 commits to main since this release
v2.0.0
3a50903
This commit was created on GitHub.com and signed with GitHub’s verified signature.
GPG key ID: B5690EEEBB952194
Learn about vigilant mode.

This release introduces support for multiple Kafka clusters, which requires a change in the configuration file format. Previously, the configuration supported only a single Kafka cluster, but now it accepts multiple clusters in the form of an array. As a result, you will need to update your configuration file to reflect the new structure.

What Changed

•	The kafka section of the configuration now supports multiple clusters as an array under the kafka_clusters key.
•	The previous configuration format, which assumed a single Kafka cluster, is no longer supported.

Old Configuration Format (v1.x.x)

kafka:
  brokers:
    - "broker1:9092"
  client_request_timeout: "30s"
  # other settings...

New Configuration Format (v2.x.x)

kafka_clusters:
  - name: "cluster-1"
    brokers:
      - "broker1:9092"
    client_request_timeout: "30s"
    # other settings...
  - name: "cluster-2"
    brokers:
      - "broker2:9092"
    client_request_timeout: "30s"
    # other settings...

How to Migrate

To migrate to the new version:

1.	Update the kafka section of your configuration to use the kafka_clusters key.
2.	Define each Kafka cluster within the kafka_clusters array.
3.	Ensure that each cluster has a name field, as this is now required.

You can refer to the updated example configuration file configs/config.yaml

Assets 2
Loading