New Prometheus Remote Write Feature & E2E Test Enhancements (Breaking Changes)

authentication methods (Basic Auth and Bearer Token). We’ve also refactored the prometheus.go file into separate files for local and remote functionality, improving code clarity.

Additionally, the E2E test framework has been enhanced with self-discovery of test folders, ensuring better test isolation and coverage. Full documentation and examples have been updated accordingly.

Breaking Changes: Please note that there are breaking configuration changes from the previous 2.x version. Review the changelog for more details.

Redis Authentication & Authorization (AuthN & AuthZ) + TLS Support

•	Added support for Redis authentication (ACLs), authorization, and TLS encryption.
•	Now you can secure your Redis connections by providing username and password authentication via the auth configuration.
•	TLS support for encrypted communication is also available for enhanced security, especially when working with Redis 6.0+.
•	This update ensures that connections to Redis clusters are secure, enabling safe use in production environments with sensitive data.
•	Older versions of Redis without ACLs or TLS are still supported for backward compatibility (unencrypted connections).

Minor improvements

•	Logging Enhancements: Improved both debug and info log levels for better visibility, including additional context such as clusterName in the logs.
•	Configuration Example: Added a config.yaml example file to guide configuration setup.
•	Race Condition Fix: Fixed a race condition in TestPersistLatestProducedOffsetsMultipleClusters, ensuring predictable order in results.
•	Mermaid Diagram Fix: Corrected the broken Mermaid diagram in the architecture documentation to accurately reflect the application’s flow.
•	E2E Test Fix: Updated the end-to-end tests to properly use their own configuration file (config.yaml), ensuring isolated and correct test runs.

[Breaking Change] Multi-Cluster Kafka Support

This release introduces support for multiple Kafka clusters, which requires a change in the configuration file format. Previously, the configuration supported only a single Kafka cluster, but now it accepts multiple clusters in the form of an array. As a result, you will need to update your configuration file to reflect the new structure.

What Changed

•	The kafka section of the configuration now supports multiple clusters as an array under the kafka_clusters key.
•	The previous configuration format, which assumed a single Kafka cluster, is no longer supported.

Old Configuration Format (v1.x.x)

kafka:
  brokers:
    - "broker1:9092"
  client_request_timeout: "30s"
  # other settings...

New Configuration Format (v2.x.x)

kafka_clusters:
  - name: "cluster-1"
    brokers:
      - "broker1:9092"
    client_request_timeout: "30s"
    # other settings...
  - name: "cluster-2"
    brokers:
      - "broker2:9092"
    client_request_timeout: "30s"
    # other settings...

How to Migrate

To migrate to the new version:

1.	Update the kafka section of your configuration to use the kafka_clusters key.
2.	Define each Kafka cluster within the kafka_clusters array.
3.	Ensure that each cluster has a name field, as this is now required.

You can refer to the updated example configuration file configs/config.yaml

Small improvements

• Adding missing unit tests
• Adding template of e2e tests
• Improving main.go moving away some functions
• Allowing to speedup iteration wait time for testing porpoises

First release

Please see Readme and Architecture section