Skip to content

Added all missing YAML examples an standardized Filters docs. Fixes #1882. #1883

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 10 commits into from
Jul 8, 2025
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
37 changes: 35 additions & 2 deletions pipeline/filters/checklist.md
Original file line number Diff line number Diff line change
Expand Up @@ -17,7 +17,37 @@ The plugin supports the following configuration parameters

## Example configuration

```python
{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:
inputs:
- name: tail
tag: test1
path: test1.log
read_from_head: true
parser: json

filters:
- name: checklist
match: test1
file: ip_list.txt
lookup_key: $remote_addr
record:
- ioc abc
- badurl null
log_level: debug

outputs:
- name: stdout
match: test1
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[INPUT]
name tail
tag test1
Expand All @@ -39,6 +69,9 @@ The plugin supports the following configuration parameters
match test1
```

{% endtab %}
{% endtabs %}

The following configuration reads a file `test1.log` that includes the following values:

```text
Expand All @@ -64,4 +97,4 @@ The configuration uses `$remote_addr` as the lookup key, and `7.7.7.7` is malici

```text
{"remote_addr": "7.7.7.7", "ioc":"abc", "url":"https://badurl.com/payload.htm","badurl":"null"}
```
```
179 changes: 99 additions & 80 deletions pipeline/filters/grep.md
Original file line number Diff line number Diff line change
Expand Up @@ -42,53 +42,54 @@ When using the command line, pay close attention to quote the regular expression
The following command loads the [tail](../../pipeline/inputs/tail) plugin and reads the content of `lines.txt`. Then the `grep` filter applies a regular expression rule over the `log` field created by the `tail` plugin and only passes records with a field value starting with `aa`:

```shell
bin/fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdout
$ ./fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdout
```

### Configuration file

{% tabs %}
{% tab title="fluent-bit.conf" %}

```python
[SERVICE]
parsers_file /path/to/parsers.conf

[INPUT]
name tail
path lines.txt
parser json

[FILTER]
name grep
match *
regex log aa

[OUTPUT]
name stdout
match *
```

{% endtab %}

{% tab title="fluent-bit.yaml" %}

```yaml
service:
parsers_file: /path/to/parsers.conf

pipeline:
inputs:
- name: tail
path: lines.txt
parser: json

filters:
- name: grep
match: '*'
regex: log aa

outputs:
- name: stdout
match: '*'
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[SERVICE]
parsers_file /path/to/parsers.conf

[INPUT]
name tail
path lines.txt
parser json

[FILTER]
name grep
match *
regex log aa

[OUTPUT]
name stdout
match *
```

{% endtab %}
Expand All @@ -102,7 +103,7 @@ To match or exclude records based on nested values, you can use [Record Accessor

Consider the following record example:

```javascript
```text
{
"log": "something",
"kubernetes": {
Expand All @@ -122,25 +123,27 @@ Consider the following record example:
For example, to exclude records that match the nested field `kubernetes.labels.app`, use the following rule:

{% tabs %}
{% tab title="fluent-bit.conf" %}

```python
[FILTER]
Name grep
Match *
Exclude $kubernetes['labels']['app'] myapp
```

{% endtab %}
{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:

filters:
- name: grep
match: '*'
exclude: $kubernetes['labels']['app'] myapp
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[FILTER]
Name grep
Match *
Exclude $kubernetes['labels']['app'] myapp
```

{% endtab %}
{% endtabs %}

Expand All @@ -154,9 +157,25 @@ key fails this check.
The following example checks for a specific valid value for the key:

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:

filters:
# Use Grep to verify the contents of the iot_timestamp value.
# If the iot_timestamp key does not exist, this will fail
# and exclude the row.
- name: grep
alias: filter-iots-grep
match: iots_thread.*
regex: iot_timestamp ^\d{4}-\d{2}-\d{2}
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```python
```text
# Use Grep to verify the contents of the iot_timestamp value.
# If the iot_timestamp key does not exist, this will fail
# and exclude the row.
Expand All @@ -167,17 +186,6 @@ The following example checks for a specific valid value for the key:
Regex iot_timestamp ^\d{4}-\d{2}-\d{2}
```

{% endtab %}
{% tab title="fluent-bit.yaml" %}

```yaml
filters:
- name: grep
alias: filter-iots-grep
match: iots_thread.*
regex: iot_timestamp ^\d{4}-\d{2}-\d{2}
```

{% endtab %}
{% endtabs %}

Expand All @@ -196,27 +204,6 @@ If you want to set multiple `Regex` or `Exclude`, you can use `Logical_Op` prope
If `Logical_Op` is set, setting both `Regex` and `Exclude` results in an error.

{% tabs %}
{% tab title="fluent-bit.conf" %}

```python
[INPUT]
Name dummy
Dummy {"endpoint":"localhost", "value":"something"}
Tag dummy

[FILTER]
Name grep
Match *
Logical_Op or
Regex value something
Regex value error

[OUTPUT]
Name stdout
```

{% endtab %}

{% tab title="fluent-bit.yaml" %}

```yaml
Expand All @@ -225,15 +212,39 @@ pipeline:
- name: dummy
dummy: '{"endpoint":"localhost", "value":"something"}'
tag: dummy

filters:
- name: grep
match: '*'
logical_op: or
regex:
- value something
- value error

outputs:
- name: stdout
match: '*'
```

{% endtab %}
{% tab title="fluent-bit.conf" %}

```text
[INPUT]
Name dummy
Dummy {"endpoint":"localhost", "value":"something"}
Tag dummy

[FILTER]
Name grep
Match *
Logical_Op or
Regex value something
Regex value error

[OUTPUT]
Name stdout
Match *
```

{% endtab %}
Expand All @@ -242,20 +253,28 @@ pipeline:
The output looks similar to:

```text
Fluent Bit v2.0.9
* Copyright (C) 2015-2022 The Fluent Bit Authors
Fluent Bit v4.0.3
* Copyright (C) 2015-2025 The Fluent Bit Authors
* Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
* https://fluentbit.io

[2023/01/22 09:46:49] [ info] [fluent bit] version=2.0.9, commit=16eae10786, pid=33268
[2023/01/22 09:46:49] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2023/01/22 09:46:49] [ info] [cmetrics] version=0.5.8
[2023/01/22 09:46:49] [ info] [ctraces ] version=0.2.7
[2023/01/22 09:46:49] [ info] [input:dummy:dummy.0] initializing
[2023/01/22 09:46:49] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2023/01/22 09:46:49] [ info] [filter:grep:grep.0] OR mode
[2023/01/22 09:46:49] [ info] [sp] stream processor started
[2023/01/22 09:46:49] [ info] [output:stdout:stdout.0] worker #0 started
______ _ _ ______ _ _ ___ _____
| ___| | | | | ___ (_) | / || _ |
| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' |
| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| |
| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ /
\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/


[2025/07/03 16:15:34] [ info] [fluent bit] version=4.0.3, commit=3a91b155d6, pid=23196
[2025/07/03 16:15:34] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128
[2025/07/03 16:15:34] [ info] [simd ] disabled
[2025/07/03 16:15:34] [ info] [cmetrics] version=1.0.3
[2025/07/03 16:15:34] [ info] [ctraces ] version=0.6.6
[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] initializing
[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only)
[2025/07/03 16:15:34] [ info] [output:stdout:stdout.0] worker #0 started
[2025/07/03 16:15:34] [ info] [sp] stream processor started
[0] dummy: [1674348410.558341857, {"endpoint"=>"localhost", "value"=>"something"}]
[0] dummy: [1674348411.546425499, {"endpoint"=>"localhost", "value"=>"something"}]
```
```
Loading