Skip to content

Commit bc48314

Browse files
authored
Added YAML examples to buffering and storage docs. (#1741)
Signed-off-by: Eric D. Schabell <[email protected]>
1 parent 1cfde96 commit bc48314

File tree

1 file changed

+108
-6
lines changed

1 file changed

+108
-6
lines changed

administration/buffering-and-storage.md

Lines changed: 108 additions & 6 deletions
Original file line numberDiff line numberDiff line change
@@ -84,9 +84,27 @@ register new records. This can happen with any input source plugin. The goal of
8484

8585
For a full data safety guarantee, use filesystem buffering.
8686

87-
Here is an example input definition:
87+
Choose your preferred format for an example input definition:
88+
89+
{% tabs %}
90+
{% tab title="fluent-bit.yaml" %}
91+
92+
```yaml
93+
pipeline:
94+
inputs:
95+
- name: tcp
96+
listen: 0.0.0.0
97+
port: 5170
98+
format: none
99+
tag: tcp-logs
100+
mem_buf_limit: 50MB
101+
```
102+
103+
{% endtab %}
104+
105+
{% tab title="fluent-bit.conf" %}
88106
89-
```python
107+
```text
90108
[INPUT]
91109
Name tcp
92110
Listen 0.0.0.0
@@ -96,6 +114,9 @@ Here is an example input definition:
96114
Mem_Buf_Limit 50MB
97115
```
98116

117+
{% endtab %}
118+
{% endtabs %}
119+
99120
If this input uses more than 50&nbsp;MB memory to buffer logs, you will get a warning like
100121
this in the Fluent Bit logs:
101122

@@ -207,7 +228,25 @@ The Service section refers to the section defined in the main
207228

208229
A Service section will look like this:
209230

210-
```python
231+
{% tabs %}
232+
{% tab title="fluent-bit.yaml" %}
233+
234+
```yaml
235+
service:
236+
flush: 1
237+
log_level: info
238+
storage.path: /var/log/flb-storage/
239+
storage.sync: normal
240+
storage.checksum: off
241+
storage.backlog.mem_limit: 5M
242+
storage.backlog.flush_on_shutdown: off
243+
```
244+
245+
{% endtab %}
246+
247+
{% tab title="fluent-bit.conf" %}
248+
249+
```text
211250
[SERVICE]
212251
flush 1
213252
log_Level info
@@ -218,6 +257,9 @@ A Service section will look like this:
218257
storage.backlog.flush_on_shutdown off
219258
```
220259

260+
{% endtab %}
261+
{% endtabs %}
262+
221263
This configuration sets an optional buffering mechanism where the route to the data
222264
is `/var/log/flb-storage/`. It uses `normal` synchronization mode, without
223265
running a checksum and up to a maximum of 5&nbsp;MB of memory when processing backlog data.
@@ -236,7 +278,33 @@ The following example configures a service offering filesystem buffering
236278
capabilities and two input plugins being the first based in filesystem and the second
237279
with memory only.
238280

239-
```python
281+
{% tabs %}
282+
{% tab title="fluent-bit.yaml" %}
283+
284+
```yaml
285+
service:
286+
flush: 1
287+
log_level: info
288+
storage.path: /var/log/flb-storage/
289+
storage.sync: normal
290+
storage.checksum: off
291+
storage.max_chunks_up: 128
292+
storage.backlog.mem_limit: 5M
293+
294+
pipeline:
295+
inputs:
296+
- name: cpu
297+
storage.type: filesystem
298+
299+
- name: mem
300+
storage.type: memory
301+
```
302+
303+
{% endtab %}
304+
305+
{% tab title="fluent-bit.conf" %}
306+
307+
```text
240308
[SERVICE]
241309
flush 1
242310
log_Level info
@@ -255,6 +323,9 @@ with memory only.
255323
storage.type memory
256324
```
257325

326+
{% endtab %}
327+
{% endtabs %}
328+
258329
### Output Section Configuration
259330

260331
If certain chunks are filesystem `storage.type` based, it's possible to control the
@@ -269,7 +340,35 @@ The following example creates records with CPU usage samples in the filesystem w
269340
are delivered to Google Stackdriver service while limiting the logical queue
270341
(buffering) to `5M`:
271342

272-
```python
343+
{% tabs %}
344+
{% tab title="fluent-bit.yaml" %}
345+
346+
```yaml
347+
service:
348+
flush: 1
349+
log_level: info
350+
storage.path: /var/log/flb-storage/
351+
storage.sync: normal
352+
storage.checksum: off
353+
storage.max_chunks_up: 128
354+
storage.backlog.mem_limit: 5M
355+
356+
pipeline:
357+
inputs:
358+
- name: cpu
359+
storage.type: filesystem
360+
361+
outputs:
362+
- name: stackdriver
363+
match: '*'
364+
storage.total_limit_size: 5M
365+
```
366+
367+
{% endtab %}
368+
369+
{% tab title="fluent-bit.conf" %}
370+
371+
```text
273372
[SERVICE]
274373
flush 1
275374
log_Level info
@@ -289,5 +388,8 @@ are delivered to Google Stackdriver service while limiting the logical queue
289388
storage.total_limit_size 5M
290389
```
291390

391+
{% endtab %}
392+
{% endtabs %}
393+
292394
If Fluent Bit is offline because of a network issue, it will continue buffering CPU
293-
samples, keeping a maximum of 5&nbsp;MB of the newest data.
395+
samples, keeping a maximum of 5&nbsp;MB of the newest data.

0 commit comments

Comments
 (0)