acquisition/victorialogs: add new datasource

[zekker6]

Related issue: #3307

Data source supports:

• cat mode with automatic adjustment of poll interval (same as one at Loki datasource)
• tail mode by using tailing API

Most of the logic is exactly the same as Loki datasource.
The only difference is related to adding support of live dailing and different response formats.
VictoriaLogs uses newline delimeted JSON for both reqular query and tail modes.

Tested locally with the following config:

source: victorialogs
log_level: debug
url: https://vlogs-endpoint
mode: tail
limit: 1000
query: |
  app_name:nginx
labels:
  type: nginx
---
source: victorialogs
log_level: debug
url: https://vlogs-endpoint
mode: cat
since: 1m
limit: 1000
query: |
  app_name:nginx
labels:
  type: nginx

[github-actions]

@zekker6: There are no 'kind' label on this PR. You need a 'kind' label to generate the release automatically.

• /kind feature
• /kind enhancement
• /kind refactoring
• /kind fix
• /kind chore
• /kind dependencies

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[github-actions]

@zekker6: There are no area labels on this PR. You can add as many areas as you see fit.

• /area agent
• /area local-api
• /area cscli
• /area appsec
• /area security
• /area configuration

Details

I am a bot created to help the crowdsecurity developers manage community feedback and contributions. You can check out my manifest file to understand my behavior and what I can do. If you want to use this for your project, you can check out the BirthdayResearch/oss-governance-bot repository.

[codecov]

Codecov Report

Attention: Patch coverage is 60.81633% with 192 lines in your changes missing coverage. Please review.

Project coverage is 59.14%. Comparing base (9ef5f58) to head (896ee99).

Files with missing lines	Patch %	Lines
...odules/victorialogs/internal/vlclient/vl_client.go	52.45%	102 Missing and 14 partials ⚠️
...g/acquisition/modules/victorialogs/victorialogs.go	69.26%	60 Missing and 15 partials ⚠️
pkg/acquisition/victorialogs.go	50.00%	0 Missing and 1 partial ⚠️

Additional details and impacted files

@@            Coverage Diff             @@
##           master    #3310      +/-   ##
==========================================
+ Coverage   59.11%   59.14%   +0.03%     
==========================================
  Files         359      350       -9     
  Lines       38730    38351     -379     
==========================================
- Hits        22894    22682     -212     
+ Misses      13908    13772     -136     
+ Partials     1928     1897      -31     

Flag	Coverage Δ
bats	41.72% <0.20%> (-0.50%)	⬇️
unit-linux	33.16% <60.81%> (+0.37%)	⬆️
unit-windows	?

Flags with carried forward coverage won't be shown. Click here to find out more.

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

[mmetc]

this should be listed in the Makefile too -- the only reason to "register" datasources is to optionally exclude from the builds with a tag, and still list them in "cscsli version"

[zekker6]

Added this to Makefile in f4603e1, thanks for noticing

[denisgolius]

AFAIK it requires to install curl every time when mostly Linux Distributions have installed wget out-from-box. Do you think curl will be better decision?

[blotus]

AFAIK it requires to install curl every time when mostly Linux Distributions have installed wget out-from-box. Do you think curl will be better decision?

This change only impacts the CI, and AFAIK curl is installed by default on the images provided by github.

I made the change just to test/debug why the victoria logs container seems to fail to start (or at least, github/docker thinks it has failed, I cannot reproduce the issue locally).

[zekker6]

Hello @blotus, I've updated healthcheck so it should work now. This can be tested locally with:

docker run --rm -it --name=victorialogs1 \
                            --health-cmd "wget -q -O - http://0.0.0.0:9428" \
                            --health-interval 30s \
                            --health-timeout 10s \
                            --health-retries 5 \
                            --health-start-period 30s \
                  victoriametrics/victoria-logs:v1.5.0-victorialogs

Previously, it was using localhost which does not work in alpine based images.

[zekker6]

@blotus Hopefully I've fixed the last reason for the tests to fail at 2b42a43, could you please re-run tests?