From f678302aea192868483b3eb256418e62970c5f90 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 8 Jul 2025 11:56:34 +0200 Subject: [PATCH 01/10] Adding YAML examples to Checklist filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell --- pipeline/filters/checklist.md | 37 +++++++++++++++++++++++++++++++++-- 1 file changed, 35 insertions(+), 2 deletions(-) diff --git a/pipeline/filters/checklist.md b/pipeline/filters/checklist.md index c71f7892d..36eeb5f55 100644 --- a/pipeline/filters/checklist.md +++ b/pipeline/filters/checklist.md @@ -17,7 +17,37 @@ The plugin supports the following configuration parameters ## Example configuration -```python +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: tail + tag: test1 + path: test1.log + read_from_head: true + parser: json + + filters: + - name: checklist + match: test1 + file: ip_list.txt + lookup_key: $remote_addr + record: + - ioc abc + - badurl null + log_level: debug + + outputs: + - name: stdout + match: test1 +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text [INPUT] name tail tag test1 @@ -39,6 +69,9 @@ The plugin supports the following configuration parameters match test1 ``` +{% endtab %} +{% endtabs %} + The following configuration reads a file `test1.log` that includes the following values: ```text @@ -64,4 +97,4 @@ The configuration uses `$remote_addr` as the lookup key, and `7.7.7.7` is malici ```text {"remote_addr": "7.7.7.7", "ioc":"abc", "url":"https://badurl.com/payload.htm","badurl":"null"} -``` +``` \ No newline at end of file From 88e933b7c56e0f3e2aae0fb97a1e0893e39b6e54 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 8 Jul 2025 12:04:18 +0200 Subject: [PATCH 02/10] Standardizing on YAML and shell usage for grep filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell --- pipeline/filters/grep.md | 179 ++++++++++++++++++++++----------------- 1 file changed, 99 insertions(+), 80 deletions(-) diff --git a/pipeline/filters/grep.md b/pipeline/filters/grep.md index 7ffbb697c..395c08705 100644 --- a/pipeline/filters/grep.md +++ b/pipeline/filters/grep.md @@ -42,53 +42,54 @@ When using the command line, pay close attention to quote the regular expression The following command loads the [tail](../../pipeline/inputs/tail) plugin and reads the content of `lines.txt`. Then the `grep` filter applies a regular expression rule over the `log` field created by the `tail` plugin and only passes records with a field value starting with `aa`: ```shell -bin/fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdout +$ ./fluent-bit -i tail -p 'path=lines.txt' -F grep -p 'regex=log aa' -m '*' -o stdout ``` ### Configuration file {% tabs %} -{% tab title="fluent-bit.conf" %} - -```python -[SERVICE] - parsers_file /path/to/parsers.conf - -[INPUT] - name tail - path lines.txt - parser json - -[FILTER] - name grep - match * - regex log aa - -[OUTPUT] - name stdout - match * -``` - -{% endtab %} - {% tab title="fluent-bit.yaml" %} ```yaml service: parsers_file: /path/to/parsers.conf + pipeline: inputs: - name: tail path: lines.txt parser: json + filters: - name: grep match: '*' regex: log aa + outputs: - name: stdout match: '*' +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[SERVICE] + parsers_file /path/to/parsers.conf + +[INPUT] + name tail + path lines.txt + parser json +[FILTER] + name grep + match * + regex log aa + +[OUTPUT] + name stdout + match * ``` {% endtab %} @@ -102,7 +103,7 @@ To match or exclude records based on nested values, you can use [Record Accessor Consider the following record example: -```javascript +```text { "log": "something", "kubernetes": { @@ -122,25 +123,27 @@ Consider the following record example: For example, to exclude records that match the nested field `kubernetes.labels.app`, use the following rule: {% tabs %} -{% tab title="fluent-bit.conf" %} - -```python -[FILTER] - Name grep - Match * - Exclude $kubernetes['labels']['app'] myapp -``` - -{% endtab %} {% tab title="fluent-bit.yaml" %} ```yaml +pipeline: + filters: - name: grep match: '*' exclude: $kubernetes['labels']['app'] myapp ``` +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[FILTER] + Name grep + Match * + Exclude $kubernetes['labels']['app'] myapp +``` + {% endtab %} {% endtabs %} @@ -154,9 +157,25 @@ key fails this check. The following example checks for a specific valid value for the key: {% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + + filters: + # Use Grep to verify the contents of the iot_timestamp value. + # If the iot_timestamp key does not exist, this will fail + # and exclude the row. + - name: grep + alias: filter-iots-grep + match: iots_thread.* + regex: iot_timestamp ^\d{4}-\d{2}-\d{2} +``` + +{% endtab %} {% tab title="fluent-bit.conf" %} -```python +```text # Use Grep to verify the contents of the iot_timestamp value. # If the iot_timestamp key does not exist, this will fail # and exclude the row. @@ -167,17 +186,6 @@ The following example checks for a specific valid value for the key: Regex iot_timestamp ^\d{4}-\d{2}-\d{2} ``` -{% endtab %} -{% tab title="fluent-bit.yaml" %} - -```yaml - filters: - - name: grep - alias: filter-iots-grep - match: iots_thread.* - regex: iot_timestamp ^\d{4}-\d{2}-\d{2} -``` - {% endtab %} {% endtabs %} @@ -196,27 +204,6 @@ If you want to set multiple `Regex` or `Exclude`, you can use `Logical_Op` prope If `Logical_Op` is set, setting both `Regex` and `Exclude` results in an error. {% tabs %} -{% tab title="fluent-bit.conf" %} - -```python -[INPUT] - Name dummy - Dummy {"endpoint":"localhost", "value":"something"} - Tag dummy - -[FILTER] - Name grep - Match * - Logical_Op or - Regex value something - Regex value error - -[OUTPUT] - Name stdout -``` - -{% endtab %} - {% tab title="fluent-bit.yaml" %} ```yaml @@ -225,6 +212,7 @@ pipeline: - name: dummy dummy: '{"endpoint":"localhost", "value":"something"}' tag: dummy + filters: - name: grep match: '*' @@ -232,8 +220,31 @@ pipeline: regex: - value something - value error + outputs: - name: stdout + match: '*' +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name dummy + Dummy {"endpoint":"localhost", "value":"something"} + Tag dummy + +[FILTER] + Name grep + Match * + Logical_Op or + Regex value something + Regex value error + +[OUTPUT] + Name stdout + Match * ``` {% endtab %} @@ -242,20 +253,28 @@ pipeline: The output looks similar to: ```text -Fluent Bit v2.0.9 -* Copyright (C) 2015-2022 The Fluent Bit Authors +Fluent Bit v4.0.3 +* Copyright (C) 2015-2025 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io -[2023/01/22 09:46:49] [ info] [fluent bit] version=2.0.9, commit=16eae10786, pid=33268 -[2023/01/22 09:46:49] [ info] [storage] ver=1.2.0, type=memory, sync=normal, checksum=off, max_chunks_up=128 -[2023/01/22 09:46:49] [ info] [cmetrics] version=0.5.8 -[2023/01/22 09:46:49] [ info] [ctraces ] version=0.2.7 -[2023/01/22 09:46:49] [ info] [input:dummy:dummy.0] initializing -[2023/01/22 09:46:49] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only) -[2023/01/22 09:46:49] [ info] [filter:grep:grep.0] OR mode -[2023/01/22 09:46:49] [ info] [sp] stream processor started -[2023/01/22 09:46:49] [ info] [output:stdout:stdout.0] worker #0 started +______ _ _ ______ _ _ ___ _____ +| ___| | | | | ___ (_) | / || _ | +| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | +| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| | +| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / +\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ + + +[2025/07/03 16:15:34] [ info] [fluent bit] version=4.0.3, commit=3a91b155d6, pid=23196 +[2025/07/03 16:15:34] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/03 16:15:34] [ info] [simd ] disabled +[2025/07/03 16:15:34] [ info] [cmetrics] version=1.0.3 +[2025/07/03 16:15:34] [ info] [ctraces ] version=0.6.6 +[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] initializing +[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only) +[2025/07/03 16:15:34] [ info] [output:stdout:stdout.0] worker #0 started +[2025/07/03 16:15:34] [ info] [sp] stream processor started [0] dummy: [1674348410.558341857, {"endpoint"=>"localhost", "value"=>"something"}] [0] dummy: [1674348411.546425499, {"endpoint"=>"localhost", "value"=>"something"}] -``` +``` \ No newline at end of file From fbd7edca8ad029cfb874ffff30dfd1c7d1a9d825 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 8 Jul 2025 13:10:35 +0200 Subject: [PATCH 03/10] Adding YAML examples and standard shell usage for log to metric filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell --- pipeline/filters/log_to_metrics.md | 362 +++++++++++++++++++++++++++-- 1 file changed, 338 insertions(+), 24 deletions(-) diff --git a/pipeline/filters/log_to_metrics.md b/pipeline/filters/log_to_metrics.md index 1f3b5110c..c0ec2c2ff 100644 --- a/pipeline/filters/log_to_metrics.md +++ b/pipeline/filters/log_to_metrics.md @@ -11,7 +11,9 @@ The _log to metrics_ filter lets you generate log-derived metrics. It supports m This filter does not actually act as a record filter and therefore does not change or drop records. All records will pass through this filter untouched, and any generated metrics will be emitted into a separate metric pipeline. {% hint style="warning" %} + This filter is an experimental feature and is not recommended for production use. Configuration parameters and other capabilities are subject to change without notice. + {% endhint %} ## Configuration parameters @@ -36,11 +38,53 @@ The plugin supports the following configuration parameters: ## Examples +{% hint style="info" %} + +All examples below assume Prometheus is running on the local machine as shown in the Fluent Bit configurations. + +{% endhint %} + ### Counter -The following example takes records from two `dummy` inputs and counts all messages that pass through the `log_to_metrics` filter. It then generate metric records, which are provided to the `prometheus_exporter` output: +The following example takes records from two `dummy` inputs and counts all messages that pass through the `log_to_metrics` filter. It then generates metric records, which are provided to the `prometheus_exporter` output: + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +service: + flush: 1 + log_level: info + +pipeline: + inputs: + - name: dummy + dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' + tag: dummy.log + + - name: dummy + dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' + tag: dummy.log2 + + filters: + - name: log_to_metrics + match: 'dummy.log*' + tag: test_metric + metric_mode: counter + metric_name: count_all_dummy_messages + metric_description: 'This metric counts dummy messages' + + outputs: + - name: prometheus_exporter + match: '*' + host: 0.0.0.0 + port: 9999 +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} -```python +```text [SERVICE] flush 1 log_level info @@ -67,13 +111,34 @@ The following example takes records from two `dummy` inputs and counts all messa name prometheus_exporter match * host 0.0.0.0 - port 2021 + port 9999 ``` -You can then use a tool like curl to retrieve the generated metric: +{% endtab %} +{% tab title="prometheus.yml" %} + +Run this configuration file with Prometheus to collect the metrics from the Fluent Bit configurations. +```yaml +# config +global: + scrape_interval: 5s + +scrape_configs: + + # Scraping Fluent Bit example. + - job_name: "fluentbit" + static_configs: + - targets: ["localhost:9999"] ``` -> curl -s http://127.0.0.1:2021/metrics + +{% endtab %} +{% endtabs %} + +You can then use a tool like curl to retrieve the generated metric: + +```shell +$ ./curl -s http://127.0.0.1:9999/metrics # HELP log_metric_counter_count_all_dummy_messages This metric counts dummy messages @@ -85,7 +150,64 @@ log_metric_counter_count_all_dummy_messages 49 The `gauge` mode needs a `value_field` to specify where to generate the metric values from. This example also applies a `regex` filter and enables the `kubernetes_mode` option: -```python +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +service: + flush: 1 + log_level: info + +pipeline: + inputs: + - name: dummy + dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' + tag: dummy.log + + - name: dummy + dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' + tag: dummy.log2 + + filters: + - name: log_to_metrics + match: 'dummy.log*' + tag: test_metric + metric_mode: gauge + metric_name: current_duration + metric_description: 'This metric shows the current duration' + value_field: duration + kubernetes_mode: on + regex: 'message .*el.*' + add_label: app $kubernetes['labels']['app'] + label_field: + - color + - shape + + outputs: + - name: prometheus_exporter + match: '*' + host: 0.0.0.0 + port: 9999 +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[SERVICE] + flush 1 + log_level info + +[INPUT] + Name dummy + Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} + Tag dummy.log + +[INPUT] + Name dummy + Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} + Tag dummy.log2 + [FILTER] name log_to_metrics match dummy.log* @@ -99,12 +221,39 @@ The `gauge` mode needs a `value_field` to specify where to generate the metric v add_label app $kubernetes['labels']['app'] label_field color label_field shape + +[OUTPUT] + name prometheus_exporter + match * + host 0.0.0.0 + port 9999 ``` -You can then use a tool like curl to retrieve the generated metric: +{% endtab %} +{% tab title="prometheus.yml" %} + +Run this configuration file with Prometheus to collect the metrics from the Fluent Bit configurations. + +```yaml +# config +global: + scrape_interval: 5s +scrape_configs: + + # Scraping Fluent Bit example. + - job_name: "fluentbit" + static_configs: + - targets: ["localhost:9999"] ``` -> curl -s http://127.0.0.1:2021/metrics + +{% endtab %} +{% endtabs %} + +You can then use a tool like curl to retrieve the generated metric: + +```shell +$ ./curl -s http://127.0.0.1:9999/metrics # HELP log_metric_gauge_current_duration This metric shows the current duration @@ -120,26 +269,70 @@ If you execute the example curl command multiple times, the example metric value #### Metric `label_values` -The label sets defined by `add_label` and `label_field` are added to the metric. The lines in the metric represent every combination of labels. Only combinations that are actualy used are displayed here. To see this, you can add a `dummy` input to your configuration. +The label sets defined by `add_label` and `label_field` are added to the metric. The lines in the metric represent every combination of labels. Only combinations that are actually used are displayed here. -The metric output would then look like: +### Histogram +Similar to the `gauge` mode, the `histogram` mode needs a `value_field` to specify where to generate the metric values from. This example also applies a `regex` filter and enables the `kubernetes_mode` option: + +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +service: + flush: 1 + log_level: info + +pipeline: + inputs: + - name: dummy + dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' + tag: dummy.log + + - name: dummy + dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' + tag: dummy.log2 + + filters: + - name: log_to_metrics + match: 'dummy.log*' + tag: test_metric + metric_mode: histogram + metric_name: current_duration + metric_description: 'This metric shows the request duration' + value_field: duration + kubernetes_mode: on + regex: 'message .*el.*' + add_label: app $kubernetes['labels']['app'] + label_field: + - color + - shape + + outputs: + - name: prometheus_exporter + match: '*' + host: 0.0.0.0 + port: 9999 ``` -> curl -s http://127.0.0.1:2021/metrics -# HELP log_metric_gauge_current_duration This metric shows the current duration -# TYPE log_metric_gauge_current_duration gauge -log_metric_gauge_current_duration{namespace_name="default",pod_name="pod1",container_name="mycontainer",docker_id="abc123",pod_id="def456",app="app1",color="blue",shape="square"} 60 -log_metric_gauge_current_duration{namespace_name="default",pod_name="pod1",container_name="mycontainer",docker_id="abc123",pod_id="def456",app="app1",color="red",shape="circle"} 20 +{% endtab %} +{% tab title="fluent-bit.conf" %} -``` -You can also see that all Kubernetes labels have been attached to the metric accordingly. +```text +[SERVICE] + flush 1 + log_level info -### Histogram +[INPUT] + Name dummy + Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} + Tag dummy.log -Similar to the `gauge` mode, the `histogram` mode needs a `value_field` to specify where to generate the metric values from. This example also applies a `regex` filter and enables the `kubernetes_mode` option: +[INPUT] + Name dummy + Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} + Tag dummy.log2 -```python [FILTER] name log_to_metrics match dummy.log* @@ -153,12 +346,39 @@ Similar to the `gauge` mode, the `histogram` mode needs a `value_field` to speci add_label app $kubernetes['labels']['app'] label_field color label_field shape + +[OUTPUT] + name prometheus_exporter + match * + host 0.0.0.0 + port 9999 ``` -You can then use a tool like curl to retrieve the generated metric: +{% endtab %} +{% tab title="prometheus.yml" %} + +Run this configuration file with Prometheus to collect the metrics from the Fluent Bit configurations. +```yaml +# config +global: + scrape_interval: 5s + +scrape_configs: + + # Scraping Fluent Bit example. + - job_name: "fluentbit" + static_configs: + - targets: ["localhost:9999"] ``` -> curl -s http://127.0.0.1:2021/metrics + +{% endtab %} +{% endtabs %} + +You can then use a tool like curl to retrieve the generated metric: + +```shell +$ ./curl -s http://127.0.0.1:2021/metrics # HELP log_metric_histogram_current_duration This metric shows the request duration @@ -195,7 +415,72 @@ log_metric_histogram_current_duration_count{namespace_name="default",pod_name="p In the resulting output, there are several buckets by default: `0.005, 0.01, 0.025, 0.05, 0.1, 0.25, 0.5, 1.0, 2.5, 5.0, 10.0` and `+Inf`. Values are sorted into these buckets. A sum and a counter are also part of this metric. You can specify own buckets in the configuration, like in the following example: -```python +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +service: + flush: 1 + log_level: info + +pipeline: + inputs: + - name: dummy + dummy: '{"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"}' + tag: dummy.log + + - name: dummy + dummy: '{"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"}' + tag: dummy.log2 + + filters: + - name: log_to_metrics + match: 'dummy.log*' + tag: test_metric + metric_mode: histogram + metric_name: current_duration + metric_description: 'This metric shows the HTTP request duration as histogram in milliseconds' + value_field: duration + kubernetes_mode: on + bucket: + - 1 + - 5 + - 10 + - 50 + - 1000 + - 250 + - 500 + - 1000 + regex: 'message .*el.*' + label_field: + - color + - shape + + outputs: + - name: prometheus_exporter + match: '*' + host: 0.0.0.0 + port: 9999 +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[SERVICE] + flush 1 + log_level info + +[INPUT] + Name dummy + Dummy {"message":"dummy", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 20, "color": "red", "shape": "circle"} + Tag dummy.log + +[INPUT] + Name dummy + Dummy {"message":"hello", "kubernetes":{"namespace_name": "default", "docker_id": "abc123", "pod_name": "pod1", "container_name": "mycontainer", "pod_id": "def456", "labels":{"app": "app1"}}, "duration": 60, "color": "blue", "shape": "square"} + Tag dummy.log2 + [FILTER] name log_to_metrics match dummy.log* @@ -216,10 +501,39 @@ In the resulting output, there are several buckets by default: `0.005, 0.01, 0.0 regex message .*el.* label_field color label_field shape + +[OUTPUT] + name prometheus_exporter + match * + host 0.0.0.0 + port 9999 ``` +{% endtab %} +{% tab title="prometheus.yml" %} + +Run this configuration file with Prometheus to collect the metrics from the Fluent Bit configurations. + +```yaml +# config +global: + scrape_interval: 5s + +scrape_configs: + + # Scraping Fluent Bit example. + - job_name: "fluentbit" + static_configs: + - targets: ["localhost:9999"] +``` + +{% endtab %} +{% endtabs %} + {% hint style="info" %} + The `+Inf` bucket will always be included regardless of the buckets you specify. The buckets in a histogram are cumulative, so a value added to one bucket will be added to all larger buckets, too. + {% endhint %} -This filter also attaches Kubernetes labels to each metric, identical to the behavior of `label_field`. This results in two sets for the histogram. +This filter also attaches Kubernetes labels to each metric, identical to the behavior of `label_field`. This results in two sets for the histogram. \ No newline at end of file From 62335b595e1df498ada968d2b9c4dbd0deffd9bf Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 8 Jul 2025 13:14:19 +0200 Subject: [PATCH 04/10] Updating Fluent Bit output and small layout fixes for parser filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell --- pipeline/filters/parser.md | 72 +++++++++++++++++++------------------- 1 file changed, 36 insertions(+), 36 deletions(-) diff --git a/pipeline/filters/parser.md b/pipeline/filters/parser.md index 061acfc4d..52f2649e3 100644 --- a/pipeline/filters/parser.md +++ b/pipeline/filters/parser.md @@ -32,7 +32,6 @@ parsers: ``` {% endtab %} - {% tab title="fluent-bit.conf" %} ```text @@ -72,7 +71,6 @@ pipeline: ``` {% endtab %} - {% tab title="fluent-bit.conf" %} ```text @@ -107,10 +105,11 @@ $ ./fluent-bit --config fluent-bit.yaml # For classic configuration. $ ./fluent-bit --config fluent-bit.conf -Fluent Bit v4.0.0 +Fluent Bit v4.0.3 * Copyright (C) 2015-2025 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io + ______ _ _ ______ _ _ ___ _____ | ___| | | | | ___ (_) | / || _ | | |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | @@ -118,15 +117,16 @@ ______ _ _ ______ _ _ ___ _____ | | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / \_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ -[2025/06/19 10:58:47] [ info] [fluent bit] version=4.0.0, commit=3a91b155d6, pid=76206 -[2025/06/19 10:58:47] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128 -[2025/06/19 10:58:47] [ info] [simd ] disabled -[2025/06/19 10:58:47] [ info] [cmetrics] version=0.9.9 -[2025/06/19 10:58:47] [ info] [ctraces ] version=0.6.2 -[2025/06/19 10:58:47] [ info] [input:dummy:dummy.0] initializing -[2025/06/19 10:58:47] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only) -[2025/06/19 10:58:47] [ info] [output:stdout:stdout.0] worker #0 started -[2025/06/19 10:58:47] [ info] [sp] stream processor started + +[2025/07/03 16:15:34] [ info] [fluent bit] version=4.0.3, commit=3a91b155d6, pid=23196 +[2025/07/03 16:15:34] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/03 16:15:34] [ info] [simd ] disabled +[2025/07/03 16:15:34] [ info] [cmetrics] version=1.0.3 +[2025/07/03 16:15:34] [ info] [ctraces ] version=0.6.6 +[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] initializing +[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only) +[2025/07/03 16:15:34] [ info] [output:stdout:stdout.0] worker #0 started +[2025/07/03 16:15:34] [ info] [sp] stream processor started [0] dummy.data: [[1750323528.603308000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}] [0] dummy.data: [[1750323529.603788000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}] [0] dummy.data: [[1750323530.604204000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example"}] @@ -153,7 +153,6 @@ parsers: ``` {% endtab %} - {% tab title="parsers.conf" %} ```text @@ -194,7 +193,6 @@ pipeline: ``` {% endtab %} - {% tab title="fluent-bit.conf" %} ```text @@ -230,10 +228,11 @@ $ ./fluent-bit --config fluent-bit.yaml # For classic configuration. $ ./fluent-bit --config fluent-bit.conf -Fluent Bit v4.0.0 +Fluent Bit v4.0.3 * Copyright (C) 2015-2025 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io + ______ _ _ ______ _ _ ___ _____ | ___| | | | | ___ (_) | / || _ | | |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | @@ -241,15 +240,16 @@ ______ _ _ ______ _ _ ___ _____ | | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / \_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ -[2025/06/19 10:58:47] [ info] [fluent bit] version=4.0.0, commit=3a91b155d6, pid=76206 -[2025/06/19 10:58:47] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128 -[2025/06/19 10:58:47] [ info] [simd ] disabled -[2025/06/19 10:58:47] [ info] [cmetrics] version=0.9.9 -[2025/06/19 10:58:47] [ info] [ctraces ] version=0.6.2 -[2025/06/19 10:58:47] [ info] [input:dummy:dummy.0] initializing -[2025/06/19 10:58:47] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only) -[2025/06/19 10:58:47] [ info] [output:stdout:stdout.0] worker #0 started -[2025/06/19 10:58:47] [ info] [sp] stream processor started + +[2025/07/03 16:15:34] [ info] [fluent bit] version=4.0.3, commit=3a91b155d6, pid=23196 +[2025/07/03 16:15:34] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/03 16:15:34] [ info] [simd ] disabled +[2025/07/03 16:15:34] [ info] [cmetrics] version=1.0.3 +[2025/07/03 16:15:34] [ info] [ctraces ] version=0.6.6 +[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] initializing +[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only) +[2025/07/03 16:15:34] [ info] [output:stdout:stdout.0] worker #0 started +[2025/07/03 16:15:34] [ info] [sp] stream processor started [0] dummy.data: [[1750325238.681398000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "key1"=>"value1", "key2"=>"value2"}] [0] dummy.data: [[1750325239.682090000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "key1"=>"value1", "key2"=>"value2"}] [0] dummy.data: [[1750325240.682903000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "key1"=>"value1", "key2"=>"value2"}] @@ -269,7 +269,6 @@ parsers: ``` {% endtab %} - {% tab title="parsers.conf" %} ```text @@ -311,7 +310,6 @@ pipeline: ``` {% endtab %} - {% tab title="fluent-bit.conf" %} ```text @@ -348,10 +346,11 @@ $ ./fluent-bit --config fluent-bit.yaml # For classic configuration. $ ./fluent-bit --config fluent-bit.conf -Fluent Bit v4.0.0 +Fluent Bit v4.0.3 * Copyright (C) 2015-2025 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io + ______ _ _ ______ _ _ ___ _____ | ___| | | | | ___ (_) | / || _ | | |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | @@ -359,15 +358,16 @@ ______ _ _ ______ _ _ ___ _____ | | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / \_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ -[2025/06/19 10:58:47] [ info] [fluent bit] version=4.0.0, commit=3a91b155d6, pid=76206 -[2025/06/19 10:58:47] [ info] [storage] ver=1.5.2, type=memory, sync=normal, checksum=off, max_chunks_up=128 -[2025/06/19 10:58:47] [ info] [simd ] disabled -[2025/06/19 10:58:47] [ info] [cmetrics] version=0.9.9 -[2025/06/19 10:58:47] [ info] [ctraces ] version=0.6.2 -[2025/06/19 10:58:47] [ info] [input:dummy:dummy.0] initializing -[2025/06/19 10:58:47] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only) -[2025/06/19 10:58:47] [ info] [output:stdout:stdout.0] worker #0 started -[2025/06/19 10:58:47] [ info] [sp] stream processor started + +[2025/07/03 16:15:34] [ info] [fluent bit] version=4.0.3, commit=3a91b155d6, pid=23196 +[2025/07/03 16:15:34] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/03 16:15:34] [ info] [simd ] disabled +[2025/07/03 16:15:34] [ info] [cmetrics] version=1.0.3 +[2025/07/03 16:15:34] [ info] [ctraces ] version=0.6.6 +[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] initializing +[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only) +[2025/07/03 16:15:34] [ info] [output:stdout:stdout.0] worker #0 started +[2025/07/03 16:15:34] [ info] [sp] stream processor started [0] dummy.data: [[1750325678.572817000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "data"=>"100 0.5 true This is example", "key1"=>"value1", "key2"=>"value2"}] [0] dummy.data: [[1750325679.574538000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "data"=>"100 0.5 true This is example", "key1"=>"value1", "key2"=>"value2"}] [0] dummy.data: [[1750325680.569750000, {}], {"INT"=>"100", "FLOAT"=>"0.5", "BOOL"=>"true", "STRING"=>"This is example", "data"=>"100 0.5 true This is example", "key1"=>"value1", "key2"=>"value2"}] From cf5483946450fe0c7d2a052e599c7b71c3ee10e8 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 8 Jul 2025 13:27:03 +0200 Subject: [PATCH 05/10] Updated for YAML examples first for modify filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell --- pipeline/filters/modify.md | 217 ++++++++++++++++++------------------- 1 file changed, 107 insertions(+), 110 deletions(-) diff --git a/pipeline/filters/modify.md b/pipeline/filters/modify.md index 5751d9cfe..8258abe63 100644 --- a/pipeline/filters/modify.md +++ b/pipeline/filters/modify.md @@ -94,49 +94,24 @@ which outputs data similar to the following: Using the command line mode requires quotes parse the wildcard properly. The use of a configuration file is recommended. -```text -bin/fluent-bit -i mem \ - -p 'tag=mem.local' \ - -F modify \ - -p 'Add=Service1 SOMEVALUE' \ - -p 'Add=Service2 SOMEVALUE3' \ - -p 'Add=Mem.total2 TOTALMEM2' \ - -p 'Rename=Mem.free MEMFREE' \ - -p 'Rename=Mem.used MEMUSED' \ - -p 'Rename=Swap.total SWAPTOTAL' \ - -p 'Add=Mem.total TOTALMEM' \ - -m '*' \ - -o stdout +```shell +$ ./fluent-bit -i mem \ + -p 'tag=mem.local' \ + -F modify \ + -p 'Add=Service1 SOMEVALUE' \ + -p 'Add=Service2 SOMEVALUE3' \ + -p 'Add=Mem.total2 TOTALMEM2' \ + -p 'Rename=Mem.free MEMFREE' \ + -p 'Rename=Mem.used MEMUSED' \ + -p 'Rename=Swap.total SWAPTOTAL' \ + -p 'Add=Mem.total TOTALMEM' \ + -m '*' \ + -o stdout ``` ### Configuration file {% tabs %} -{% tab title="fluent-bit.conf" %} - -```python -[INPUT] - Name mem - Tag mem.local - -[OUTPUT] - Name stdout - Match * - -[FILTER] - Name modify - Match * - Add Service1 SOMEVALUE - Add Service3 SOMEVALUE3 - Add Mem.total2 TOTALMEM2 - Rename Mem.free MEMFREE - Rename Mem.used MEMUSED - Rename Swap.total SWAPTOTAL - Add Mem.total TOTALMEM -``` - -{% endtab %} - {% tab title="fluent-bit.yaml" %} ```yaml @@ -144,6 +119,7 @@ pipeline: inputs: - name: mem tag: mem.local + filters: - name: modify match: '*' @@ -156,11 +132,36 @@ pipeline: - Mem.free MEMFREE - Mem.used MEMUSED - Swap.total SWAPTOTAL + outputs: - name: stdout match: '*' ``` +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name mem + Tag mem.local + +[FILTER] + Name modify + Match * + Add Service1 SOMEVALUE + Add Service3 SOMEVALUE3 + Add Mem.total2 TOTALMEM2 + Rename Mem.free MEMFREE + Rename Mem.used MEMUSED + Rename Swap.total SWAPTOTAL + Add Mem.total TOTALMEM + +[OUTPUT] + Name stdout + Match * +``` + {% endtab %} {% endtabs %} @@ -181,9 +182,45 @@ The output of both the command line and configuration invocations should be iden ### Use a configuration file {% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: mem + tag: mem.local + interval_sec: 1 + + filters: + - name: modify + match: mem.* + Condition: + - Key_Does_Not_Exist cpustats + - Key_Exists Mem.used + Set: cpustats UNKNOWN + + - name: modify + match: mem.* + Condition: Key_Value_Does_Not_Equal cpustats KNOWN + Add: sourcetype memstats + + - name: modify + match: mem.* + Condition: Key_Value_Equals cpustats UNKNOWN + Remove_wildcard: + - Mem + - Swap + Add: cpustats_more STILL_UNKNOWN + + outputs: + - name: stdout + match: '*' +``` + +{% endtab %} {% tab title="fluent-bit.conf" %} -```python +```text [INPUT] Name mem Tag mem.local @@ -192,26 +229,20 @@ The output of both the command line and configuration invocations should be iden [FILTER] Name modify Match mem.* - Condition Key_Does_Not_Exist cpustats Condition Key_Exists Mem.used - Set cpustats UNKNOWN [FILTER] Name modify Match mem.* - Condition Key_Value_Does_Not_Equal cpustats KNOWN - Add sourcetype memstats [FILTER] Name modify Match mem.* - Condition Key_Value_Equals cpustats UNKNOWN - Remove_wildcard Mem Remove_wildcard Swap Add cpustats_more STILL_UNKNOWN @@ -221,39 +252,6 @@ The output of both the command line and configuration invocations should be iden Match * ``` -{% endtab %} - -{% tab title="fluent-bit.yaml" %} - -```yaml -pipeline: - inputs: - - name: mem - tag: mem.local - interval_sec: 1 - filters: - - name: modify - match: mem.* - Condition: - - Key_Does_Not_Exist cpustats - - Key_Exists Mem.used - Set: cpustats UNKNOWN - - name: modify - match: mem.* - Condition: Key_Value_Does_Not_Equal cpustats KNOWN - Add: sourcetype memstats - - name: modify - match: mem.* - Condition: Key_Value_Equals cpustats UNKNOWN - Remove_wildcard: - - Mem - - Swap - Add: cpustats_more STILL_UNKNOWN - outputs: - - name: stdout - match: '*' -``` - {% endtab %} {% endtabs %} @@ -272,33 +270,6 @@ pipeline: ### Emoji configuration File {% tabs %} -{% tab title="fluent-bit.conf" %} - -```python -[INPUT] - Name mem - Tag mem.local - -[OUTPUT] - Name stdout - Match * - -[FILTER] - Name modify - Match * - - Remove_Wildcard Mem - Remove_Wildcard Swap - Set This_plugin_is_on 🔥 - Set 🔥 is_hot - Copy 🔥 💦 - Rename 💦 ❄️ - Set ❄️ is_cold - Set 💦 is_wet -``` - -{% endtab %} - {% tab title="fluent-bit.yaml" %} ```yaml @@ -307,25 +278,51 @@ pipeline: - name: mem tag: mem.local interval_sec: 1 + filters: - name: modify match: mem.* Remove_wildcard: - Mem - Swap - Set: + set: - This_plugin_is_on 🔥 - 🔥 is_hot - Copy: 🔥 💦 - Rename: 💦 ❄️ - Set: - ❄️ is_cold - 💦 is_wet + copy: 🔥 💦 + rename: 💦 ❄️ + outputs: - name: stdout match: '*' ``` +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name mem + Tag mem.local + +[FILTER] + Name modify + Match * + Remove_Wildcard Mem + Remove_Wildcard Swap + Set This_plugin_is_on 🔥 + Set 🔥 is_hot + Copy 🔥 💦 + Rename 💦 ❄️ + Set ❄️ is_cold + Set 💦 is_wet + +[OUTPUT] + Name stdout + Match * +``` + {% endtab %} {% endtabs %} @@ -338,4 +335,4 @@ pipeline: [2] mem.local: [1528926374.000181042, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}] [3] mem.local: [1528926375.000090841, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}] [0] mem.local: [1528926376.000610974, {"This_plugin_is_on"=>"🔥", "🔥"=>"is_hot", "❄️"=>"is_cold", "💦"=>"is_wet"}] -``` +``` \ No newline at end of file From 9229ab3e6629a2be209356341825c777afa2800d Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 8 Jul 2025 13:34:09 +0200 Subject: [PATCH 06/10] Updated for YAML examples first for nest filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell --- pipeline/filters/nest.md | 290 ++++++++++++++++++++------------------- 1 file changed, 152 insertions(+), 138 deletions(-) diff --git a/pipeline/filters/nest.md b/pipeline/filters/nest.md index a162f985b..f682712f0 100644 --- a/pipeline/filters/nest.md +++ b/pipeline/filters/nest.md @@ -90,42 +90,21 @@ The following command loads the _mem_ plugin. Then the _nest_ filter matches the wildcard rule to the keys and nests the keys matching `Mem.*` under the new key `NEST`. -```shell copy -bin/fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout +```shell +$ ./fluent-bit -i mem -p 'tag=mem.local' -F nest -p 'Operation=nest' -p 'Wildcard=Mem.*' -p 'Nest_under=Memstats' -p 'Remove_prefix=Mem.' -m '*' -o stdout ``` ### Nest configuration file {% tabs %} -{% tab title="fluent-bit.conf" %} - -```python copy -[INPUT] - Name mem - Tag mem.local - -[OUTPUT] - Name stdout - Match * - -[FILTER] - Name nest - Match * - Operation nest - Wildcard Mem.* - Nest_under Memstats - Remove_prefix Mem. -``` - -{% endtab %} - {% tab title="fluent-bit.yaml" %} -```yaml copy +```yaml pipeline: inputs: - name: mem tag: mem.local + filters: - name: nest match: '*' @@ -133,11 +112,33 @@ pipeline: wildcard: Mem.* nest_under: Memstats remove_prefix: Mem. + outputs: - name: stdout match: '*' ``` +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name mem + Tag mem.local + +[FILTER] + Name nest + Match * + Operation nest + Wildcard Mem.* + Nest_under Memstats + Remove_prefix Mem. + + [OUTPUT] + Name stdout + Match * +``` + {% endtab %} {% endtabs %} @@ -157,42 +158,14 @@ This example nests all `Mem.*` and `Swap.*` items under the `Stats` key and then ### `nest` and `lift` undo configuration file {% tabs %} -{% tab title="fluent-bit.conf" %} - -```python copy -[INPUT] - Name mem - Tag mem.local - -[OUTPUT] - Name stdout - Match * - -[FILTER] - Name nest - Match * - Operation nest - Wildcard Mem.* - Wildcard Swap.* - Nest_under Stats - Add_prefix NESTED - -[FILTER] - Name nest - Match * - Operation lift - Nested_under Stats - Remove_prefix NESTED -``` - -{% endtab %} {% tab title="fluent-bit.yaml" %} -```yaml copy +```yaml pipeline: inputs: - name: mem tag: mem.local + filters: - name: nest match: '*' @@ -202,16 +175,47 @@ pipeline: - Swap.* Nest_under: Stats Add_prefix: NESTED + - name: nest match: '*' Operation: lift Nested_under: Stats Remove_prefix: NESTED + outputs: - name: stdout match: '*' ``` +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name mem + Tag mem.local + +[FILTER] + Name nest + Match * + Operation nest + Wildcard Mem.* + Wildcard Swap.* + Nest_under Stats + Add_prefix NESTED + +[FILTER] + Name nest + Match * + Operation lift + Nested_under Stats + Remove_prefix NESTED + +[OUTPUT] + Name stdout + Match * +``` + {% endtab %} {% endtabs %} @@ -230,17 +234,46 @@ which is then nested under `LAYER2`, which is nested under `LAYER3`. ### Deep `nest` configuration file {% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: mem + tag: mem.local + + filters: + - name: nest + match: '*' + Operation: nest + Wildcard: Mem.* + Nest_under: LAYER1 + + - name: nest + match: '*' + Operation: nest + Wildcard: LAYER1* + Nest_under: LAYER2 + + - name: nest + match: '*' + Operation: nest + Wildcard: LAYER2* + Nest_under: LAYER3 + + outputs: + - name: stdout + match: '*' +``` + +{% endtab %} {% tab title="fluent-bit.conf" %} -```python copy +```text [INPUT] Name mem Tag mem.local -[OUTPUT] - Name stdout - Match * - [FILTER] Name nest Match * @@ -261,35 +294,10 @@ which is then nested under `LAYER2`, which is nested under `LAYER3`. Operation nest Wildcard LAYER2* Nest_under LAYER3 -``` -{% endtab %} -{% tab title="fluent-bit.yaml" %} - -```yaml copy -pipeline: - inputs: - - name: mem - tag: mem.local - filters: - - name: nest - match: '*' - Operation: nest - Wildcard: Mem.* - Nest_under: LAYER1 - - name: nest - match: '*' - Operation: nest - Wildcard: LAYER1* - Nest_under: LAYER2 - - name: nest - match: '*' - Operation: nest - Wildcard: LAYER2* - Nest_under: LAYER3 - outputs: - - name: stdout - match: '*' +[OUTPUT] + Name stdout + Match * ``` {% endtab %} @@ -324,17 +332,64 @@ This example uses the 3-level deep nesting of Example 2 and applies the `lift` f ### `nest` and `lift` prefix configuration file {% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: mem + tag: mem.local + + filters: + - name: nest + match: '*' + Operation: nest + Wildcard: Mem.* + Nest_under: LAYER1 + + - name: nest + match: '*' + Operation: nest + Wildcard: LAYER1* + Nest_under: LAYER2 + + - name: nest + match: '*' + Operation: nest + Wildcard: LAYER2* + Nest_under: LAYER3 + + - name: nest + match: '*' + Operation: lift + Nested_under: LAYER3 + Add_prefix: Lifted3_ + + - name: nest + match: '*' + Operation: lift + Nested_under: Lifted3_LAYER2 + Add_prefix: Lifted3_Lifted2_ + + - name: nest + match: '*' + Operation: lift + Nested_under: Lifted3_Lifted2_LAYER1 + Add_prefix: Lifted3_Lifted2_Lifted1_ + + outputs: + - name: stdout + match: '*' +``` + +{% endtab %} {% tab title="fluent-bit.conf" %} -```python copy +```text [INPUT] Name mem Tag mem.local -[OUTPUT] - Name stdout - Match * - [FILTER] Name nest Match * @@ -376,51 +431,10 @@ This example uses the 3-level deep nesting of Example 2 and applies the `lift` f Operation lift Nested_under Lifted3_Lifted2_LAYER1 Add_prefix Lifted3_Lifted2_Lifted1_ -``` -{% endtab %} - -{% tab title="fluent-bit.yaml" %} - -```yaml copy -pipeline: - inputs: - - name: mem - tag: mem.local - filters: - - name: nest - match: '*' - Operation: nest - Wildcard: Mem.* - Nest_under: LAYER1 - - name: nest - match: '*' - Operation: nest - Wildcard: LAYER1* - Nest_under: LAYER2 - - name: nest - match: '*' - Operation: nest - Wildcard: LAYER2* - Nest_under: LAYER3 - - name: nest - match: '*' - Operation: lift - Nested_under: LAYER3 - Add_prefix: Lifted3_ - - name: nest - match: '*' - Operation: lift - Nested_under: Lifted3_LAYER2 - Add_prefix: Lifted3_Lifted2_ - - name: nest - match: '*' - Operation: lift - Nested_under: Lifted3_Lifted2_LAYER1 - Add_prefix: Lifted3_Lifted2_Lifted1_ - outputs: - - name: stdout - match: '*' +[OUTPUT] + Name stdout + Match * ``` {% endtab %} @@ -439,4 +453,4 @@ pipeline: "Lifted3_Lifted2_Lifted1_Mem.used"=>1253912, "Lifted3_Lifted2_Lifted1_Mem.free"=>2796996 } -``` +``` \ No newline at end of file From 29f7dce3801ed509a64408ccfc7dd25b647a15c7 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 8 Jul 2025 13:37:11 +0200 Subject: [PATCH 07/10] Updated for YAML examples first for rewrite tag filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell --- pipeline/filters/rewrite-tag.md | 88 ++++++++++++++++++++------------- 1 file changed, 53 insertions(+), 35 deletions(-) diff --git a/pipeline/filters/rewrite-tag.md b/pipeline/filters/rewrite-tag.md index d5b0b2d7a..df3e7fe76 100644 --- a/pipeline/filters/rewrite-tag.md +++ b/pipeline/filters/rewrite-tag.md @@ -42,7 +42,7 @@ $KEY REGEX NEW_TAG KEEP The key represents the name of the _record key_ that holds the `value` to use to match the regular expression. A key name is specified and prefixed with a `$`. Consider the following structured record (formatted for readability): -```javascript +```text { "name": "abc-123", "ss": { @@ -117,9 +117,34 @@ You can use `true` or `false` to decide the expected behavior. This field is man The following configuration example will emit a dummy record. The filter will rewrite the tag, discard the old record, and print the new record to the standard output interface: {% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +service: + flush: 1 + log_level: info + +pipeline: + inputs: + - name: dummy + tag: test_tag + dummy: '{"tool": "fluent", "sub": {"s1": {"s2": "bit"}}}' + + filters: + - name: rewrite_tag + match: test_tag + rule: $tool ^(fluent)$ from.$TAG.new.$tool.$sub['s1']['s2'].out false + emitter_name: re_emitted + + outputs: + - name: stdout + match: from.* +``` + +{% endtab %} {% tab title="fluent-bit.conf" %} -```python +```text [SERVICE] Flush 1 Log_Level info @@ -140,43 +165,36 @@ The following configuration example will emit a dummy record. The filter will re Match from.* ``` -{% endtab %} - -{% tab title="fluent-bit.yaml" %} - -```yaml -service: - flush: 1 - log_level: info -pipeline: - inputs: - - name: dummy - tag: test_tag - dummy: '{"tool": "fluent", "sub": {"s1": {"s2": "bit"}}}' - filters: - - name: rewrite_tag - match: test_tag - rule: $tool ^(fluent)$ from.$TAG.new.$tool.$sub['s1']['s2'].out false - emitter_name: re_emitted - outputs: - - name: stdout - match: from.* -``` - {% endtab %} {% endtabs %} The original tag `test_tag` will be rewritten as `from.test_tag.new.fluent.bit.out`: -```bash -$ bin/fluent-bit -c example.conf -Fluent Bit v1.x.x -* Copyright (C) 2019-2020 The Fluent Bit Authors -* Copyright (C) 2015-2018 Treasure Data +```shell +$ ./fluent-bit -c example.conf + +Fluent Bit v4.0.3 +* Copyright (C) 2015-2025 The Fluent Bit Authors * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd * https://fluentbit.io -... +______ _ _ ______ _ _ ___ _____ +| ___| | | | | ___ (_) | / || _ | +| |_ | |_ _ ___ _ __ | |_ | |_/ /_| |_ __ __/ /| || |/' | +| _| | | | | |/ _ \ '_ \| __| | ___ \ | __| \ \ / / /_| || /| | +| | | | |_| | __/ | | | |_ | |_/ / | |_ \ V /\___ |\ |_/ / +\_| |_|\__,_|\___|_| |_|\__| \____/|_|\__| \_/ |_(_)___/ + + +[2025/07/03 16:15:34] [ info] [fluent bit] version=4.0.3, commit=3a91b155d6, pid=23196 +[2025/07/03 16:15:34] [ info] [storage] ver=1.5.3, type=memory, sync=normal, checksum=off, max_chunks_up=128 +[2025/07/03 16:15:34] [ info] [simd ] disabled +[2025/07/03 16:15:34] [ info] [cmetrics] version=1.0.3 +[2025/07/03 16:15:34] [ info] [ctraces ] version=0.6.6 +[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] initializing +[2025/07/03 16:15:34] [ info] [input:dummy:dummy.0] storage_strategy='memory' (memory only) +[2025/07/03 16:15:34] [ info] [output:stdout:stdout.0] worker #0 started +[2025/07/03 16:15:34] [ info] [sp] stream processor started [0] from.test_tag.new.fluent.bit.out: [1580436933.000050569, {"tool"=>"fluent", "sub"=>{"s1"=>{"s2"=>"bit"}}}] ``` @@ -190,13 +208,13 @@ The `rewrite_tag` filter emits new records that go through the beginning of the Using the previously provided configuration, when you query the metrics exposed in the HTTP interface: -```bash -curl http://127.0.0.1:2020/api/v1/metrics/ | jq +```shell +$ ./curl http://127.0.0.1:2020/api/v1/metrics/ | jq ``` You will see metrics output similar to the following: -```javascript +```text { "input": { "dummy.0": { @@ -235,4 +253,4 @@ The records generated are handled by the internal emitter, so the new records ar The _Emitter_ is an internal Fluent Bit plugin that allows other components of the pipeline to emit custom records. On this case `rewrite_tag` creates an emitter instance to use it exclusively to emit records, allowing for granular control of who is emitting what. -Change the Emitter name in the metrics by adding the `Emitter_Name` configuration property described previously. +Change the Emitter name in the metrics by adding the `Emitter_Name` configuration property described previously. \ No newline at end of file From c6e8838ec395901e26de41ddf3af3bdc99c1d925 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 8 Jul 2025 13:38:08 +0200 Subject: [PATCH 08/10] Updated for YAML examples first for sysinfo filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell --- pipeline/filters/sysinfo.md | 45 +++++++++++++++++++------------------ 1 file changed, 23 insertions(+), 22 deletions(-) diff --git a/pipeline/filters/sysinfo.md b/pipeline/filters/sysinfo.md index da0e30233..b5aab964f 100644 --- a/pipeline/filters/sysinfo.md +++ b/pipeline/filters/sysinfo.md @@ -23,26 +23,6 @@ To start filtering records, you can run the filter from the command line or thro The following configuration file is to append the Fluent Bit version and operating system name. {% tabs %} -{% tab title="fluent-bit.conf" %} - -```python -[INPUT] - Name dummy - Tag test - -[FILTER] - Name sysinfo - Match * - Fluentbit_version_key flb_ver - Os_name_key os_name - -[OUTPUT] - name stdout - match * -``` - -{% endtab %} - {% tab title="fluent-bit.yaml" %} ```yaml @@ -50,27 +30,48 @@ pipeline: inputs: - name: dummy tag: test + filters: - name: sysinfo match: '*' Fluentbit_version_key: flb_ver Os_name_key: os_name + outputs: - name: stdout match: '*' ``` +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name dummy + Tag test + +[FILTER] + Name sysinfo + Match * + Fluentbit_version_key flb_ver + Os_name_key os_name + +[OUTPUT] + name stdout + match * +``` + {% endtab %} {% endtabs %} You can also run the filter from command line. ```shell -fluent-bit -i dummy -o stdout -F sysinfo -m '*' -p fluentbit_version_key=flb_ver -p os_name_key=os_name +$ ./fluent-bit -i dummy -o stdout -F sysinfo -m '*' -p fluentbit_version_key=flb_ver -p os_name_key=os_name ``` The output will be something like the following: ```text [0] dummy.0: [[1699172858.989654355, {}], {"message"=>"dummy", "flb_ver"=>"2.2.0", "os_name"=>"linux"}] -``` +``` \ No newline at end of file From a7ccd61481a55de50cbef88510aa3a0fe6510277 Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 8 Jul 2025 13:39:29 +0200 Subject: [PATCH 09/10] Updated for YAML examples first for type converter filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell --- pipeline/filters/type-converter.md | 47 +++++++++++++++--------------- 1 file changed, 24 insertions(+), 23 deletions(-) diff --git a/pipeline/filters/type-converter.md b/pipeline/filters/type-converter.md index 51a9611f7..af0fe2649 100644 --- a/pipeline/filters/type-converter.md +++ b/pipeline/filters/type-converter.md @@ -38,32 +38,13 @@ The plugin outputs `uint` values and `filter_type_converter` converts them into ### Convert `uint` to string {% tabs %} -{% tab title="fluent-bit.conf" %} - -```python -[INPUT] - Name mem - -[FILTER] - Name type_converter - Match * - uint_key Mem.total Mem.total_str string - uint_key Mem.used Mem.used_str string - uint_key Mem.free Mem.free_str string - -[OUTPUT] - Name stdout - Match * -``` - -{% endtab %} - {% tab title="fluent-bit.yaml" %} ```yaml pipeline: inputs: - name: mem + filters: - name: type_converter match: '*' @@ -71,22 +52,42 @@ pipeline: - Mem.total Mem.total_str string - Mem.used Mem.used_str string - Mem.free Mem.free_str string + outputs: - name: stdout match: '*' ``` +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text +[INPUT] + Name mem + +[FILTER] + Name type_converter + Match * + uint_key Mem.total Mem.total_str string + uint_key Mem.used Mem.used_str string + uint_key Mem.free Mem.free_str string + +[OUTPUT] + Name stdout + Match * +``` + {% endtab %} {% endtabs %} You can also run the filter from command line. ```shell -fluent-bit -i mem -o stdout -F type_converter -p 'uint_key=Mem.total Mem.total_str string' -p 'uint_key=Mem.used Mem.used_str string' -p 'uint_key=Mem.free Mem.free_str string' -m '*' +$ ./fluent-bit -i mem -o stdout -F type_converter -p 'uint_key=Mem.total Mem.total_str string' -p 'uint_key=Mem.used Mem.used_str string' -p 'uint_key=Mem.free Mem.free_str string' -m '*' ``` The output will be -```python +```text [0] mem.0: [1639915154.160159749, {"Mem.total"=>8146052, "Mem.used"=>4513564, "Mem.free"=>3632488, "Swap.total"=>1918356, "Swap.used"=>0, "Swap.free"=>1918356, "Mem.total_str"=>"8146052", "Mem.used_str"=>"4513564", "Mem.free_str"=>"3632488"}] -``` +``` \ No newline at end of file From 9636936df7d9556d9cb16f64cd5fff454edb92ef Mon Sep 17 00:00:00 2001 From: "Eric D. Schabell" Date: Tue, 8 Jul 2025 13:44:59 +0200 Subject: [PATCH 10/10] Added YAML examples for WASAM filter doc, part of issue #1882. Signed-off-by: Eric D. Schabell --- pipeline/filters/wasm.md | 31 ++++++++++++++++++++++++++++++- 1 file changed, 30 insertions(+), 1 deletion(-) diff --git a/pipeline/filters/wasm.md b/pipeline/filters/wasm.md index 13d83ff2c..89d2367b0 100644 --- a/pipeline/filters/wasm.md +++ b/pipeline/filters/wasm.md @@ -29,7 +29,33 @@ The plugin supports the following configuration parameters: Here is a configuration example. -```python +{% tabs %} +{% tab title="fluent-bit.yaml" %} + +```yaml +pipeline: + inputs: + - name: dummy + tag: dummy.local + + filters: + - name: wasm + match: 'dummy.*' + event_format: json # or msgpack + wasm_path: /path/to/wasm_program.wasm + function_name: filter_function_name + # Note: run Fluent Bit from the 'wasm_path' location. + accessible_paths: /path/to/accessible + + outputs: + - name: stdout + match: '*' +``` + +{% endtab %} +{% tab title="fluent-bit.conf" %} + +```text [INPUT] Name dummy Tag dummy.local @@ -46,3 +72,6 @@ Here is a configuration example. Name stdout Match * ``` + +{% endtab %} +{% endtabs %} \ No newline at end of file