Skip to content

Added YAML examples to buffering and storage docs. Fixes #1740 #1741

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
114 changes: 108 additions & 6 deletions administration/buffering-and-storage.md
Original file line number Diff line number Diff line change
Expand Up @@ -84,9 +84,27 @@ register new records. This can happen with any input source plugin. The goal of

For a full data safety guarantee, use filesystem buffering.

Here is an example input definition:
Choose your preferred format for an example input definition:

{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
pipeline:
inputs:
- name: tcp
listen: 0.0.0.0
port: 5170
format: none
tag: tcp-logs
mem_buf_limit: 50MB
```

{% endtab %}

{% tab title="fluent-bit.conf" %}

```python
```text
[INPUT]
Name tcp
Listen 0.0.0.0
Expand All @@ -96,6 +114,9 @@ Here is an example input definition:
Mem_Buf_Limit 50MB
```

{% endtab %}
{% endtabs %}

If this input uses more than 50 MB memory to buffer logs, you will get a warning like
this in the Fluent Bit logs:

Expand Down Expand Up @@ -207,7 +228,25 @@ The Service section refers to the section defined in the main

A Service section will look like this:

```python
{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
service:
flush: 1
log_level: info
storage.path: /var/log/flb-storage/
storage.sync: normal
storage.checksum: off
storage.backlog.mem_limit: 5M
storage.backlog.flush_on_shutdown: off
```

{% endtab %}

{% tab title="fluent-bit.conf" %}

```text
[SERVICE]
flush 1
log_Level info
Expand All @@ -218,6 +257,9 @@ A Service section will look like this:
storage.backlog.flush_on_shutdown off
```

{% endtab %}
{% endtabs %}

This configuration sets an optional buffering mechanism where the route to the data
is `/var/log/flb-storage/`. It uses `normal` synchronization mode, without
running a checksum and up to a maximum of 5 MB of memory when processing backlog data.
Expand All @@ -236,7 +278,33 @@ The following example configures a service offering filesystem buffering
capabilities and two input plugins being the first based in filesystem and the second
with memory only.

```python
{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
service:
flush: 1
log_level: info
storage.path: /var/log/flb-storage/
storage.sync: normal
storage.checksum: off
storage.max_chunks_up: 128
storage.backlog.mem_limit: 5M

pipeline:
inputs:
- name: cpu
storage.type: filesystem

- name: mem
storage.type: memory
```

{% endtab %}

{% tab title="fluent-bit.conf" %}

```text
[SERVICE]
flush 1
log_Level info
Expand All @@ -255,6 +323,9 @@ with memory only.
storage.type memory
```

{% endtab %}
{% endtabs %}

### Output Section Configuration

If certain chunks are filesystem `storage.type` based, it's possible to control the
Expand All @@ -269,7 +340,35 @@ The following example creates records with CPU usage samples in the filesystem w
are delivered to Google Stackdriver service while limiting the logical queue
(buffering) to `5M`:

```python
{% tabs %}
{% tab title="fluent-bit.yaml" %}

```yaml
service:
flush: 1
log_level: info
storage.path: /var/log/flb-storage/
storage.sync: normal
storage.checksum: off
storage.max_chunks_up: 128
storage.backlog.mem_limit: 5M

pipeline:
inputs:
- name: cpu
storage.type: filesystem

outputs:
- name: stackdriver
match: '*'
storage.total_limit_size: 5M
```

{% endtab %}

{% tab title="fluent-bit.conf" %}

```text
[SERVICE]
flush 1
log_Level info
Expand All @@ -289,5 +388,8 @@ are delivered to Google Stackdriver service while limiting the logical queue
storage.total_limit_size 5M
```

{% endtab %}
{% endtabs %}

If Fluent Bit is offline because of a network issue, it will continue buffering CPU
samples, keeping a maximum of 5 MB of the newest data.
samples, keeping a maximum of 5 MB of the newest data.