Inserting Logs to qryn

Input Formats

qryn supports any Loki Agent using HTTP/JSON or Protobuf inserts through its PUSH API.

The following clients are natively supported:

• PaStash
• Grafana Agent
• Promtail
• Docker Driver
• Fluentd
• Fluent Bit
• Logstash
• Lambda Promtail
• AWS
• Datadog Vector

---

paStash

The paStash Agent can be used to easily ship logs to qryn from a variety of inputs

Setup

Install paStash next-gen and the Loki output plugin globally using npm:

# npm install -g @pastash/pastash @pastash/output_loki

Recipes

Log Files

In this super simple recipe we'll use the file input and loki output, with an optional wss socket for live logs:

input {
  file {
    path => "/var/log/*.log"
  }
}

filter {
  compute_field {
    field => type
    value => "my_logs"
  }
}

output {
  loki {
    host => localhost
    port => 3100
    path => "/loki/api/v1/push"
  }
}

Example: Clickhouse Logs

• Clickhouse Logs

---

Real-Time Logs via Secure WebSockets for streaming clients

input {
  ws {
    host => 0.0.0.0
    port => 5654
    ssl => true
    ssl_key => /etc/letsencrypt/live/host.domain.ext/privkey.pem
    ssl_cert => /etc/letsencrypt/live/host.domain.ext/fullchain.pem
  }
}

filter {
  compute_field {
    field => type
    value => "webrtc"
  }
}

output {
  loki {
    host => localhost
    port => 3100
    path => "/loki/api/v1/push"
  }
}

---

• Multiple input methods can be used in parallel.

Once ready, save the pastash recipe to your preferred location (ie: /etc/pastash_loki.json)

Usage

./bin/pastash --config_file=/etc/pastash_loki.json

Docker

For those using Docker there's a ready to go container for Loki output

pastash:
    image: qxip/pastash-loki
    container_name: pastash
    volumes:
      - ./conf/pastash_loki.json:/pastash.conf
      - /var/log:/var/log:ro

Sending docker logs to qryn is as simple as configuring a logging driver:

version: "3.4"

services:
  my-nginx-service:
    image: nginx
    container_name: my-nginx-service
    ports:
      - 8000:80
    environment:
      - FOO=bar
    logging:
      driver: loki
      options:
        loki-url: http://localhost:3100/loki/api/v1/push
        loki-external-labels: job=dockerlogs,owner=ruan,environment=development

Documentation

Please consult the paStash Wiki for additional documentation and examples

---

Grafana Agent

Install [Grafana Agent] using the official documentation.

The agent configuration is stored in /etc/grafana-agent.yaml

This example will scrape and send info from all logs in /var/log that end in log. They are labeled with varlogs as the job and job_name

loki:
  configs:
  - name: default
    positions:
      filename: /tmp/positions.yaml
    scrape_configs:
      - job_name: varlogs
        static_configs:
          - targets: [localhost]
            labels:
              job: varlogs
              __path__: /var/log/*log
    clients:
      - url: http://qryn.host:3100/loki/api/v1/push
        basic_auth:
          username: <User>
          password: <Password>

Here is another example, scraping logs for a minecraft server with logs stored in a subdirectory of the /home directory of a special minecraft user.

      - job_name: minecraftlog
        static_configs:
          - targets: [localhost]
            labels:
              job: minecraft
              __path__: /home/MCuser/minecraft/logs/latest.log

Anytime you change the agent configuration, you must restart the agent for the new configuration to take effect.

sudo systemctl restart grafana-agent.service