Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Grafana Live View not working when victorialogs is behind nginx #197

Open
dani opened this issue Dec 31, 2024 · 1 comment
Open

Grafana Live View not working when victorialogs is behind nginx #197

dani opened this issue Dec 31, 2024 · 1 comment
Labels
bug Something isn't working

Comments

@dani
Copy link

dani commented Dec 31, 2024

Describe the bug

I run victorialogs behind an nginx proxy (which handles adding extra_filters query arg depending on some authentication factors). When doing so, Live View from Grafana (using victorialogs datasource) is not working : logs are not streamd, I can only see logs up until the time I pressed the Live View button.
Everything else is working (standard query from Grafana, query from VMUI, query or tailing using curl). Only Live View from Grafana is broken.
As soon as I remove the nginx proxy (so Grafana talks to victoriametrics "directly"), everything is working, including Live View

To Reproduce

Configure Grafana victorialogs datasource to talk to victorialogs through an nginx reverse proxy
Try to use Live View

My nginx conf
worker_processes        auto;
error_log               /dev/stderr warn;
pid                     /tmp/nginx.pid;

events {
  worker_connections    1024;
}

http {
  proxy_temp_path       /tmp/proxy_temp;
  client_body_temp_path /tmp/client_temp;
  fastcgi_temp_path     /tmp/fastcgi_temp;
  uwsgi_temp_path       /tmp/uwsgi_temp;
  scgi_temp_path        /tmp/scgi_temp;
  include               /etc/nginx/mime.types;
  default_type          application/octet-stream;
  log_format            main  '$remote_addr - $remote_user [$time_local] "$request" '
                              '$status $body_bytes_sent "$http_referer" '
                              '"$http_user_agent" "$http_x_forwarded_for"';
  access_log            /dev/stdout main;
  sendfile              on;
  keepalive_timeout     65;

  set_real_ip_from 127.0.0.1;
  real_ip_header X-Forwarded-For;
  real_ip_recursive on;
  # Detect if header X-Forwarded-Tls-Client-Cert-Info is present
  map $http_x_forwarded_tls_client_cert_info $has_x_forwarded_tls_client_cert_info {
    default 0;
    ~^(Subject%3D%22CN%3D)?dani(%40ct.acme.org%22|@ct.acme.org)? 1;
  }
  # Detect if header X-Grafana-User is present
  map $http_x_grafana_user $has_x_grafana_user {
    default 0;
    ~^(Subject%3D%22CN%3D)?dani(%40ct.acme.org%22|@ct.acme.org)? 1;
  }
  # map permissions to a header dependant (X-Forwarded-Tls-Client-Cert-Info) variable
  map $http_x_forwarded_tls_client_cert_info $extra_filters_x_forwarded_tls_client_cert_info {
    default "NOMAD_NAMESPACE%3A%3Dfake";
    ~^(Subject%3D%22CN%3D)?dani(%40ct.acme.org%22|@ct.acme.org)? "";
  }
  # map permissions to a header dependant (X-Grafana-User) variable
  map $http_x_grafana_user $extra_filters_x_grafana_user {
    default "NOMAD_NAMESPACE%3A%3Dfake";
    ~^(Subject%3D%22CN%3D)?dani(%40ct.acme.org%22|@ct.acme.org)? "";
  }
  server {
    listen 127.0.0.1:9427 default_server;
    server_tokens off;
    proxy_buffering off;
    client_max_body_size 0;
    proxy_http_version 1.1;
    proxy_request_buffering off;

    # Clear any extra_filters query arg
    if ($args ~ (.*)extra_filters=[^&]*(.*)){
      set $args $1$2;
    }
    # Set default extra_filters
    set $extra_filters "NOMAD_NAMESPACE%3A%3Dfake";

    # Set permissions for header X-Forwarded-Tls-Client-Cert-Info
    if ($has_x_forwarded_tls_client_cert_info != 0){
      set $extra_filters "$extra_filters_x_forwarded_tls_client_cert_info";
    }
    # Set permissions for header X-Grafana-User
    if ($has_x_grafana_user != 0){
      set $extra_filters "$extra_filters_x_grafana_user";
    }
    set $args "$args&extra_filters=$extra_filters";

    location ~ /(insert/|health) {
      access_log off;
      proxy_pass http://127.0.0.1:9428;
    }

    location /select/ {
      proxy_pass http://127.0.0.1:9428$uri$is_args$args;
    }

    location / {
      proxy_pass http://127.0.0.1:9428;
    }
  }
}

Version

victoria-logs-20241222-133402-tags-v1.4.0-victorialogs-0-g58f09fe3f8

Logs

No response

Screenshots

No response

Used command-line flags

-envflag.enable="true"
-envflag.prefix="VLOGS_"
-http.disableResponseCompression="true"
-httpListenAddr="127.0.0.1:9428"
-loggerOutput="stdout"
-loggerTimezone="Europe/Paris"
-retention.maxDiskSpaceUsageBytes="28252570KiB"
-retentionPeriod="90d"
-search.maxConcurrentRequests="16"
-search.maxQueryDuration="30s"
-search.maxQueueDuration="20s"
-storage.minFreeDiskSpaceBytes="500MB"
-storageDataPath="/data"

### Additional information

_No response_
@dani dani added the bug Something isn't working label Dec 31, 2024
@zekker6 zekker6 transferred this issue from VictoriaMetrics/VictoriaMetrics Jan 2, 2025
@dani
Copy link
Author

dani commented Jan 2, 2025

After some more debugging, the issue comes from the fact that the X-Grafana-User header is not sent by Grafana when using Live View mode (so it ended with a default extra_filters which filters everything out). Not sure if it's in the datasource plugin or in Grafana itself

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
bug Something isn't working
Projects
None yet
Development

No branches or pull requests

1 participant