You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
The Kafka input plugin allows subscribing to one or more Kafka topics to collect messages from an [Apache Kafka](https://kafka.apache.org/) service.
4
-
This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka)\(built-in dependency\).
3
+
The _Kafka_ input plugin subscribes to one or more Kafka topics to collect messages from an [Apache Kafka](https://kafka.apache.org/) service.
5
4
6
-
## Configuration Parameters
5
+
This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka) as a built-in dependency.
6
+
7
+
## Configuration parameters
7
8
8
9
| Key | Description | default |
9
10
| :--- | :--- | :--- |
10
-
| brokers | Single or multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092. ||
11
-
| topics | Single entry or list of topics separated by comma \(,\) that Fluent Bit will subscribe to. ||
12
-
| format | Serialization format of the messages. If set to "json", the payload will be parsed as json. |none |
13
-
|client\_id| Client id passed to librdkafka. ||
14
-
|group\_id| Group id passed to librdkafka. | fluent-bit |
15
-
|poll\_ms| Kafka brokers polling interval in milliseconds. | 500 |
16
-
|Buffer\_Max\_Size| Specify the maximum size of buffer per cycle to poll kafka messages from subscribed topics. To increase throughput, specify larger size. |4M|
17
-
| rdkafka.{property} |`{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)||
18
-
| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). |`false`|
11
+
|`brokers`| Single or multiple list of Kafka Brokers. For example: `192.168.1.3:9092`, `192.168.1.4:9092`. |_none_|
12
+
|`topics`| Single entry or list of comma-separated topics (`,`) that Fluent Bit will subscribe to. |_none_|
13
+
|`format`| Serialization format of the messages. If set to `json`, the payload will be parsed as JSON. |_none_|
14
+
|`client_id`| Client id passed to librdkafka. |_none_|
15
+
|`group_id`| Group id passed to librdkafka. |`fluent-bit`|
16
+
|`poll_ms`| Kafka brokers polling interval in milliseconds. |`500`|
17
+
|`Buffer_Max_Size`| Specify the maximum size of buffer per cycle to poll Kafka messages from subscribed topics. To increase throughput, specify larger size. |`4M`|
18
+
|`rdkafka.{property}`|`{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md)|_none_|
19
+
|`threaded`| Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). |`false`|
19
20
20
-
## Getting Started
21
+
## Get started
21
22
22
-
In order to subscribe/collect messages from Apache Kafka, you can run the plugin from the command line or through the configuration file:
23
+
To subscribe to or collect messages from Apache Kafka, run the plugin from the command line or through the configuration file:
23
24
24
-
### Command Line
25
+
### Command line
25
26
26
-
The **kafka** plugin can read parameters through the **-p** argument \(property\), e.g:
27
+
The Kafka plugin can read parameters through the `-p` argument (property):
In your main configuration file append the following _Input_ & _Output_ sections:
35
+
In your main configuration file append the following `Input` and `Output` sections:
35
36
36
37
```text
37
38
[INPUT]
@@ -44,10 +45,9 @@ In your main configuration file append the following _Input_ & _Output_ sections
44
45
Name stdout
45
46
```
46
47
47
-
#### Example of using kafka input/output plugins
48
+
#### Example of using Kafka input and output plugins
48
49
49
-
The Fluent Bit source repository contains a full example of using Fluent Bit to
50
-
process Kafka records:
50
+
The Fluent Bit source repository contains a full example of using Fluent Bit to process Kafka records:
51
51
52
52
```text
53
53
[INPUT]
@@ -69,10 +69,10 @@ process Kafka records:
69
69
topics fb-sink
70
70
```
71
71
72
-
The above will connect to the broker listening on `kafka-broker:9092` and subscribe to the `fb-source` topic, polling for new messages every 100 milliseconds.
72
+
The previous example will connect to the broker listening on `kafka-broker:9092` and subscribe to the `fb-source` topic, polling for new messages every 100 milliseconds.
73
73
74
-
Since the payload will be in json format, we ask the plugin to automatically parse the payload with `format json`.
74
+
Since the payload will be in JSON format, the plugin is configured to parse the payload with `format json`.
75
75
76
76
Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker.
77
77
78
-
The example can be executed locally with `make start` in the `examples/kafka_filter` directory (docker/compose is used).
78
+
The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).
0 commit comments