Merge pull request #13 from sciclon2/feat-add-prometheus-remote-write

 Feature: Prometheus Remote Write Support

.github/workflows/ci-cd.yml

@@ -26,7 +26,7 @@ jobs:
       run: |
         if [ ! -f go.mod ]; then
           go mod init github.com/sciclon2/kafka-lag-go
-          go mod edit -go=1.20  # Set the Go version here
+          go mod edit -go=1.21  # Set the Go version here
         fi
         go mod tidy
 
@@ -35,31 +35,56 @@ jobs:
         docker run --rm \
           -v "${{ github.workspace }}:/app" \
           -w /app \
-          golang:1.20 \
+          golang:1.21 \
           bash -c "go mod tidy && go test ./pkg/... ./cmd/kafka-lag-go/... -v -cover"
 
+  discover-e2e-tests:
+    name: Discover E2E Test Folders
+    needs: [test]
+    runs-on: ubuntu-latest
+    if: github.event_name == 'pull_request' || (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'release'
+    outputs:
+      e2e-folders: ${{ steps.set-outputs.outputs.e2e-folders }}
+    steps:
+
+    - name: Checkout code
+      uses: actions/checkout@v3
+
+    - name: Discover test folders
+      id: set-outputs
+      run: |
+        folders=$(find test/e2e/tests/* -maxdepth 0 -type d)
+        echo "e2e-folders=$(echo "$folders" | jq -R -s -c 'split("\n")[:-1]')" >> $GITHUB_OUTPUT
+
   e2e-test:
-    name: E2E test Stage
+    name: E2E Test Stage
+    needs: [discover-e2e-tests]
     runs-on: ubuntu-latest
-    needs: test
-    if: github.event_name == 'push' && github.ref == 'refs/heads/main' || github.event_name == 'release'
+    if: github.event_name == 'pull_request' || (github.event_name == 'push' && github.ref == 'refs/heads/main') || github.event_name == 'release'
+    strategy:
+      matrix:
+        folder: ${{ fromJson(needs.discover-e2e-tests.outputs.e2e-folders) }}
 
     steps:
     - name: Checkout code
       uses: actions/checkout@v3
 
-    - name: Install Docker Compose
-      run: sudo apt-get update && sudo apt-get install -y docker-compose
+    - name: Install Docker Compose v2
+      run: |
+        mkdir -p ~/.docker/cli-plugins/
+        curl -SL https://github.com/docker/compose/releases/download/v2.15.1/docker-compose-linux-x86_64 -o ~/.docker/cli-plugins/docker-compose
+        chmod +x ~/.docker/cli-plugins/docker-compose
 
-    - name: Run E2E tests in Go container
-      run: ./test/e2e/run.sh
+    - name: Run E2E tests
+      run: |
+        echo "Running E2E tests for folder: ${{ matrix.folder }}"
+        ./test/e2e/run.sh ${{ matrix.folder }}
 
   build-and-push-dev:
     name: Build and Push DEV Stage
     runs-on: ubuntu-latest
     needs: e2e-test
-    if: github.event_name == 'push' && github.ref == 'refs/heads/main'
-
+    if: github.event_name == 'push' && github.ref == 'refs/heads/main' 
     steps:
     - name: Checkout code
       uses: actions/checkout@v3

.gitignore

@@ -43,3 +43,7 @@ go.sum
 
 # binaries
 bin/
+
+# Ignore Nginx TLS certificates
+**/nginx.key
+**/nginx.crt

CHANGELOG.md

@@ -1,5 +1,28 @@
 # Changelog
 
+## [v3.0.0] - Prometheus Remote Write Support and E2E Test Enhancements (12th Sept 2024)
+
+### Breaking Changes
+- **Configuration File Format**: 
+  - Introduced a new structure in the configuration file to separate Prometheus local scraping and Prometheus Remote Write.
+  - Users now need to update their `config.yaml` file to accommodate the new structure, including new sections for `prometheus_local` and `prometheus_remote_write`.
+
+### Feat
+- **Prometheus Remote Write**: 
+  - Added support for Prometheus Remote Write, allowing metrics to be sent directly to a remote Prometheus instance.
+  - The remote write feature supports both **Basic Auth** and **Bearer Token** authentication, as well as optional **TLS configuration** for secure communication.
+- **Prometheus Local and Remote Split**: 
+  - Split the original `prometheus.go` file into two separate files: one for **Prometheus Local** metrics and one for **Prometheus Remote Write**, improving clarity and maintainability.
+- **E2E Test Enhancements**: 
+  - Created a new structure for **self-discovering end-to-end tests** that run in isolated environments. Each E2E test is automatically detected and executed in a separate job in GitHub Actions for better isolation and parallelism.
+  - Added a specific **E2E test for Prometheus Remote Write** to ensure the metrics are correctly sent and received by the remote Prometheus server.
+
+### Documentation
+- **Prometheus Documentation**: 
+  - Updated the documentation with details on how to configure **Prometheus Remote Write**, including **Basic Auth**, **Bearer Token**, and **TLS** settings.
+  - Provided examples for both local and remote Prometheus configurations, helping users understand how to transition to the new structure.
+
+---
 
 ## [v2.0.2] - Redis Authentication, Authorization, and TLS Support (10th Sept 2024)
 

README.md

@@ -21,7 +21,9 @@
 9. [Health Check Feature](#health-check-feature)
 10. [Prometheus Metrics](#prometheus-metrics-overview)
 11. [Next Steps](#next-steps)
-12. [License](#license)
+12. [How to Contribute](#how-to-contribute)
+13. [License](#license)
+
 
 # Kafka Lag Go
 
@@ -104,12 +106,29 @@ docker build -t kafka-lag-go:latest .
 Kafka Lag Monitor requires a YAML configuration file. Create a file named `config.yaml` and customize it based on your environment. Below is an example configuration:
 
 ```yaml
-prometheus:
+prometheus_local:
   metrics_port: 9090
   labels:
     env: production
     service: kafka-lag-go
 
+prometheus_remote_write:
+  enabled: true
+  url: "https://remote-prometheus-server.com/"
+  headers:
+    X-Custom-Header: "myCustomHeaderValue" 
+  timeout: "30s"
+  basic_auth:
+    username: "myUsername"
+    password: "myPassword"
+  bearer_token: "myBearerToken" 
+  tls_config:
+    enabled: true
+    cert_file: "/path/to/cert.pem"
+    key_file: "/path/to/key.pem"
+    ca_cert_file: "/path/to/ca_cert.pem"
+    insecure_skip_verify: true
+
 kafka_clusters:
   - name: "kafka-cluster-1"
     brokers:
@@ -203,14 +222,28 @@ You can download the Docker image from Docker Hub using the following command:
 docker pull sciclon2/kafka-lag-go
 ```
 
-
 ## Configuration
 
 The Kafka Lag Monitor requires a YAML configuration file to customize its behavior. Below is a description of the available configuration options:
 
 - **Prometheus Metrics**:
-  - `prometheus.metrics_port`: The port on which Prometheus metrics will be exposed.
-  - `prometheus.labels`: Additional labels to be attached to the exported metrics.
+  - `prometheus_local.metrics_port`: The port on which Prometheus metrics will be exposed locally.
+  - `prometheus_local.labels`: Additional labels to be attached to the exported metrics.
+  - `prometheus_remote_write`: Configuration for Prometheus Remote Write to export metrics remotely.
+    - `url`: The URL of the Prometheus remote write endpoint.
+    - `headers`: Optional HTTP headers (e.g., custom headers) for the remote write request.
+    - `timeout`: Timeout duration for sending the metrics.
+    - **Authentication**:
+      - `basic_auth`: For basic authentication, specify:
+        - `username`: The username for basic auth.
+        - `password`: The password for basic auth.
+      - `bearer_token`: If using bearer token authentication, specify the token here.
+    - **TLS Settings**:
+      - `tls_config.enabled`: Whether TLS encryption is enabled for Prometheus remote write.
+      - `tls_config.cert_file`: Path to the client certificate file.
+      - `tls_config.key_file`: Path to the client key file.
+      - `tls_config.ca_cert_file`: Path to the CA certificate file for verifying the server's certificate.
+      - `tls_config.insecure_skip_verify`: Whether to skip TLS certificate verification.
 
 - **Kafka Clusters**:
   - `kafka_clusters`: An array of Kafka cluster configurations, each with the following options:
@@ -254,6 +287,7 @@ The Kafka Lag Monitor requires a YAML configuration file to customize its behavi
   - `app.health_check_port`: The port on which the health check endpoint will be exposed.
   - `app.health_check_path`: The path of the health check endpoint.
 
+
 Please refer to the `config.go` file for more details on each configuration option.
 
 ## Running Unit and End-to-End (E2E) Tests
@@ -272,20 +306,9 @@ go test ./... -v -cover -count=1
 
 ### End-to-End (E2E) Tests
 
-For E2E tests, we simulate the full system, including Kafka, storage (Redis in this case), and compile the `kafka-lag-go` binary. You can run the E2E tests using the provided script:
-
-```bash
-test/e2e/run.sh
-```
-
-This script will:
-1. Create the necessary infrastructure, including Kafka and Redis.
-2. Compile the `kafka-lag-go` binary.
-3. Execute the end-to-end tests to verify the system's behavior under real conditions.
+The E2E tests ensure that the full system operates as expected by validating the successful export of key metrics, confirming that these metrics reach their final destinations in either Prometheus Local or Prometheus Remote Write.
 
-The E2E test script is designed to handle cleanup automatically. Whether the tests pass or fail, the script will catch the signal and ensure that the Docker Compose environment is properly cleaned up after the tests have finished.
-
-Make sure you have Docker installed and running, as it is used to set up the test environment.
+For more detailed information on the E2E test setup and execution, please see the [E2E Test Documentation](docs/e2e-tests.md).
 
 ## Health Check Feature
 
@@ -331,7 +354,10 @@ http://<docker-host-ip>:<metrics_port>/metrics
 
 ## Next Steps
 Please check issues section.
-For more details on usage and advanced configuration, refer to the full documentation (coming soon).
+For more details on usage and advanced configuration, refer to the full documentation located in docs folder.
+
+## How to Contribute
+Contributions are welcome! Please refer to the [How to Contribute](docs/how-to-contribute.md) guide for more information on how to get started, guidelines for submitting pull requests, and code review standards.
 
 ## License
 

cmd/kafka-lag-go/main.go

@@ -80,8 +80,9 @@ func main() {
 	store.StartNodeMonitoring(monitorInterval)
 
 	// Initialize Prometheus metrics
-	prometheusMetrics := metrics.NewPrometheusMetrics(cfg.Prometheus.Labels)
-	prometheusMetrics.RegisterMetrics()
+	prometheusMetrics := metrics.NewLocalPrometheusMetrics(cfg.PrometheusLocal.Labels)
+	prometheusMetrics.RegisterLocalMetrics()
+	prometheusMetrics.StartRemoteWriteExporter(cfg)
 
 	// Main processing loop that runs continuously.
 	for {
@@ -118,7 +119,7 @@ func main() {
 		metricsToExportChan := lp.GenerateMetrics(groupsComplete, cfg.App.NumWorkers)
 
 		// Start processing metrics concurrently
-		prometheusMetrics.ProcessMetrics(metricsToExportChan, cfg.App.NumWorkers, startTime)
+		prometheusMetrics.ProcessLocalMetrics(metricsToExportChan, cfg.App.NumWorkers, startTime)
 
 		// Wait for the next iteration or handle a signal
 		SleepToMaintainInterval(startTime, iterationInterval, sigChan)

docs/ACLs.md

@@ -62,7 +62,6 @@ kafka-acls --bootstrap-server kafka-broker:9092 \
   --operation DESCRIBE --cluster
 ```
 
-
 ### Topic Level ACLs
 To allow the application to describe topics, fetch offsets, and perform operations on partitions, apply the following ACLs for topics:
 

docs/e2e-tests.md

@@ -0,0 +1,55 @@
+## End-to-End (E2E) Tests
+
+### Overview
+
+End-to-End (E2E) tests are essential to validate the overall system behavior by simulating real-world scenarios. In Kafka-Lag-Go, these tests cover the entire pipeline, ensuring that data flows correctly from Kafka to Redis and, finally, to Prometheus for metric collection and visualization.
+
+### What Does Kafka-Lag-Go Do?
+
+Kafka-Lag-Go collects data from Kafka, stores relevant information in Redis, and generates metrics that can be either scraped locally or pushed to a central Prometheus instance using Prometheus Remote Write.
+
+#### Differences Between Local Prometheus Scrape and Prometheus Remote Write
+
+- **Local Prometheus Scrape (`/metrics` endpoint)**: This method captures a snapshot of the current state of the system's metrics at any given time when Prometheus performs its scheduled scrapes. Metrics are queried directly from the local endpoint, which gives the latest values for the tracked metrics.
+
+- **Prometheus Remote Write**: In contrast, this approach pushes metrics to a central Prometheus instance at regular intervals. Remote Write allows for scalable and centralized monitoring, where multiple services or distributed instances can send their metrics to a single, central Prometheus server for aggregation.
+
+### E2E Test Structure
+
+The E2E tests for Kafka-Lag-Go are housed in the `test/e2e/` directory. This folder contains all necessary components for each E2E test, including configuration files for Docker Compose and Go-based test files. The main script `run.sh` is used to automate the execution of each test, taking the directory of the specific E2E test as an argument.
+
+Example structure:
+
+- `test/e2e/run.sh`: Main script to run E2E tests.
+- `test/e2e/tests/`: Directory holding individual E2E tests.
+- `test/e2e/tests/[test_name]/`: Each test has its own folder with Docker Compose setup and test configurations.
+
+The tests are run using the `run.sh` script, which ensures that the infrastructure (Kafka, Redis, etc.) is correctly set up before executing the tests. For example:
+```bash
+$ test/e2e/run.sh test/e2e/tests/prometheus_local -d 
+```
+
+### Automating E2E Tests in GitHub Actions
+
+In the GitHub Actions workflow, E2E tests are automatically detected and executed in isolated jobs. Each test folder is run in a separate job to ensure proper isolation since different tests may start their own infrastructure.
+
+When a new folder is added under `test/e2e/tests/`, it will be automatically picked up by the GitHub workflow and executed as part of the CI pipeline. Each test's infrastructure is provisioned via Docker Compose, and the tests are executed using the Go test files located within each folder.
+
+### Adding a New E2E Test
+
+To add a new E2E test:
+
+1. Copy an existing folder inside `test/e2e/tests/` and modify it according to your test's needs.
+2. Update the Docker Compose configuration with any necessary components (e.g., Kafka, Redis, Nginx).
+3. Write a new E2E test in Go and place it inside the `go_test/` folder.
+
+You can test the new E2E test locally using the `run.sh` script by pointing it to the test folder:
+```bash
+$ cp -a test/e2e/tests/prometheus_remote_write_basic_auth test/e2e/tests/my-new-cool-test
+$ # make sure the docker compose file has all the needed services
+$ # edit your go_test/e2e_test.go
+$ # finally run the test locally
+$ test/e2e/run.sh test/e2e/tests/my-new-cool-test -d # -d is for debuging
+```
+
+E2E tests are automatically triggered when you merge to the main branch or create a new release.

docs/how-to-contribute.md

@@ -0,0 +1,73 @@
+# How to Contribute
+
+We welcome contributions to the Kafka Lag Go project, whether it’s reporting an issue, requesting a feature, or contributing code. Here are some guidelines to get you started:
+
+## 1. Check the Documentation
+
+Before contributing, take a look at the documentation in the `docs` folder. It includes valuable information such as:
+
+- [Architecture Design](docs/Architecture.md): An overview of the system's design, which is helpful for understanding the codebase.
+- [E2E Tests](docs/e2e-tests.md): Details on how our end-to-end testing framework works.
+
+Understanding these documents will help you navigate the project and align your contribution with its goals.
+
+## 2. Reporting an Issue
+
+If you encounter a bug or have an idea for a new feature, please create an issue in the GitHub repository. When submitting an issue:
+
+- **For Bugs**: Include as much information as possible, such as:
+  - Steps to reproduce the issue
+  - Expected behavior vs. actual behavior
+  - Relevant logs or screenshots
+  - Version of the application and environment details
+  - **Enable Debug Mode**: To gather more information, you can enable the debug mode in your `config.yaml` file by setting the `log_level` under `app` to `debug`. This will provide additional diagnostic logs that can help in troubleshooting.
+
+
+- **For Feature Requests**: Provide a detailed description of the feature you would like to see added. Include any specific use cases or examples to clarify the request.
+
+## 3. Contributing Code
+
+If you'd like to contribute directly by fixing a bug or adding a feature, we encourage you to fork the repository and submit a pull request (PR).
+
+### Steps for Contributing Code:
+
+1. **Fork the Repository**: 
+   - Fork the repo on GitHub and clone your fork to your local machine.
+
+2. **Create a Branch**: 
+   - Create a descriptive branch for your changes, for example:
+     ```
+     git checkout -b feature/new-awesome-feature
+     ```
+
+3. **Make Your Changes**: 
+   - Add the necessary code changes to implement the feature or fix the bug.
+   - Ensure you write the corresponding unit tests for your changes.
+   - For larger features, add end-to-end (E2E) tests if applicable.
+
+4. **Run Tests**: 
+   - Before submitting your PR, make sure all unit tests and E2E tests pass by running them locally.
+   - For unit tests:
+     ```bash
+     go test ./... -v -cover -count=1
+     ```
+   - For E2E tests, follow the instructions in the [E2E Test Documentation](docs/e2e-tests.md).
+
+5. **Submit a Pull Request**: 
+   - Push your changes to your forked repository and submit a pull request to the main repo.
+   - Provide a clear description of your changes, linking to the relevant issue if it exists.
+
+## 4. Writing Unit and E2E Tests
+
+Every contribution that includes new functionality or changes should also include corresponding tests to ensure the stability of the project.
+
+### Unit Tests
+- **Unit tests** should cover the core functionality of the new feature or fix. They should be located in the corresponding package where the change is made.
+
+### End-to-End (E2E) Tests
+- **E2E tests** may be necessary if your contribution affects how the system interacts with external services such as Kafka, Redis, or Prometheus. These tests ensure the full integration works as expected.
+- Refer to the `test/e2e/` directory for examples of existing E2E tests.
+
+---
+
+Thank you for your contributions! We look forward to your participation in the Kafka Lag Go project.