Skip to content

pipeline: parsers: decoders: general cleanup #1878

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
39 changes: 15 additions & 24 deletions pipeline/parsers/decoders.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,6 @@
# Decoders

There are cases where the log messages being parsed contain encoded data. A typical
use case can be found in containerized environments with Docker. Docker logs its
data in JSON format, which uses escaped strings.
There are cases where the log messages you want to parse contain encoded data. A typical use case can be found in containerized environments with Docker. Docker logs its data in JSON format, which uses escaped strings.

Consider the following message generated by the application:

Expand All @@ -16,20 +14,18 @@ The Docker log message encapsulates something like this:
{"log":"{\"status\": \"up and running\"}\r\n","stream":"stdout","time":"2018-03-09T01:01:44.851160855Z"}
```

The original message is handled as an escaped string. Fluent Bit wants to use the
original structured message and not a string.
The original message is handled as an escaped string. Fluent Bit will use the original structured message, and not a string.

## Getting Started
## Get started

Decoders are a built-in feature available through the Parsers file. Each parser
definition can optionally set one or more decoders. There are two types of decoders:
Decoders are a built-in feature of parsers in Fluent Bit. Each parser definition can optionally set one or more decoders. Select from one of these decoder types:

- `Decode_Field`: If the content can be decoded in a structured message, append
the structured message (keys and values) to the original log message.
- `Decode_Field_As`: Any decoded content (unstructured or structured) will be
replaced in the same key/value, and no extra keys are added.

Our pre-defined Docker parser has the following definition:
For example, the predefined Docker parser has the following definition:

{% tabs %}
{% tab title="parsers.yaml" %}
Expand Down Expand Up @@ -64,34 +60,29 @@ parsers:
{% endtab %}
{% endtabs %}

Each line in the parser with a key `Decode_Field` instructs the parser to apply
a specific decoder on a given field. Optionally, it offers the option to take an
extra action if the decoder doesn't succeed.
Each line in the parser with a key `Decode_Field` instructs the parser to apply a specific decoder on a given field. Optionally, it offers the option to take an extra action if the decoder doesn't succeed.

### Decoder options

| Name | Description |
| -------------- | ----------- |
| `json` | Handle the field content as a JSON map. If it finds a JSON map, it replaces the content with a structured map. |
| `json` | Handle the field content as a JSON map. If the decoder finds a JSON map, it replaces the content with a structured map. |
| `escaped` | Decode an escaped string. |
| `escaped_utf8` | Decode a UTF8 escaped string. |

### Optional Actions
### Optional actions

If a decoder fails to decode the field or, you want to try another decoder, you can
define an optional action. Available actions are:
If a decoder fails to decode the field, or if you want to try another decoder, you can define an optional action. Available actions are:

| Name | Description |
| -----| ----------- |
| `try_next` | if the decoder failed, apply the next decoder in the list for the same field. |
| `do_next` | if the decoder succeeded or failed, apply the next decoder in the list for the same field. |
| `try_next` | If the decoder failed, apply the next decoder in the list for the same field. |
| `do_next` | If the decoder succeeded or failed, apply the next decoder in the list for the same field. |

Actions are affected by some restrictions:

- `Decode_Field_As`: If successful, another decoder of the same type and the same
field can be applied only if the data continues being an unstructured message (raw text).
- `Decode_Field`: If successful, can only be applied once for the same field.
`Decode`_Field` is intended to decode a structured message.
- `Decode_Field_As`: If successful, another decoder of the same type and the same field can be applied only if the data continues being an unstructured message (raw text).
- `Decode_Field`: If successful, can be applied only once for the same field. `Decode_Field` is intended to decode a structured message.

### Examples

Expand Down Expand Up @@ -124,7 +115,7 @@ Decoder example Fluent Bit configuration files:
```yaml
service:
parsers_file: parsers.yaml

pipeline:
inputs:
- name: tail
Expand Down Expand Up @@ -183,4 +174,4 @@ parsers:
```

{% endtab %}
{% endtabs %}
{% endtabs %}