Skip to content

Commit 8ffc14e

Browse files
authored
Merge pull request #1730 from fluent/lynettemiles/sc-136182/update-fluent-bit-docs-pipeline-inputs-kafka
2 parents 656fde1 + 9f171be commit 8ffc14e

File tree

2 files changed

+27
-26
lines changed

2 files changed

+27
-26
lines changed

pipeline/inputs/kafka.md

Lines changed: 26 additions & 26 deletions
Original file line numberDiff line numberDiff line change
@@ -1,37 +1,38 @@
11
# Kafka
22

3-
The Kafka input plugin allows subscribing to one or more Kafka topics to collect messages from an [Apache Kafka](https://kafka.apache.org/) service.
4-
This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka) \(built-in dependency\).
3+
The _Kafka_ input plugin subscribes to one or more Kafka topics to collect messages from an [Apache Kafka](https://kafka.apache.org/) service.
54

6-
## Configuration Parameters
5+
This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka) as a built-in dependency.
6+
7+
## Configuration parameters
78

89
| Key | Description | default |
910
| :--- | :--- | :--- |
10-
| brokers | Single or multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092. | |
11-
| topics | Single entry or list of topics separated by comma \(,\) that Fluent Bit will subscribe to. | |
12-
| format | Serialization format of the messages. If set to "json", the payload will be parsed as json. | none |
13-
| client\_id | Client id passed to librdkafka. | |
14-
| group\_id | Group id passed to librdkafka. | fluent-bit |
15-
| poll\_ms | Kafka brokers polling interval in milliseconds. | 500 |
16-
| Buffer\_Max\_Size | Specify the maximum size of buffer per cycle to poll kafka messages from subscribed topics. To increase throughput, specify larger size. | 4M |
17-
| rdkafka.{property} | `{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) | |
18-
| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
11+
| `brokers` | Single or multiple list of Kafka Brokers. For example: `192.168.1.3:9092`, `192.168.1.4:9092`. | _none_ |
12+
| `topics` | Single entry or list of comma-separated topics (`,`) that Fluent Bit will subscribe to. | _none_ |
13+
| `format` | Serialization format of the messages. If set to `json`, the payload will be parsed as JSON. | _none_ |
14+
| `client_id` | Client id passed to librdkafka. | _none_ |
15+
| `group_id` | Group id passed to librdkafka. | `fluent-bit` |
16+
| `poll_ms` | Kafka brokers polling interval in milliseconds. | `500` |
17+
| `Buffer_Max_Size` | Specify the maximum size of buffer per cycle to poll Kafka messages from subscribed topics. To increase throughput, specify larger size. | `4M` |
18+
| `rdkafka.{property}` | `{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) | _none_ |
19+
| `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` |
1920

20-
## Getting Started
21+
## Get started
2122

22-
In order to subscribe/collect messages from Apache Kafka, you can run the plugin from the command line or through the configuration file:
23+
To subscribe to or collect messages from Apache Kafka, run the plugin from the command line or through the configuration file:
2324

24-
### Command Line
25+
### Command line
2526

26-
The **kafka** plugin can read parameters through the **-p** argument \(property\), e.g:
27+
The Kafka plugin can read parameters through the `-p` argument (property):
2728

28-
```text
29-
$ fluent-bit -i kafka -o stdout -p brokers=192.168.1.3:9092 -p topics=some-topic
29+
```shell
30+
fluent-bit -i kafka -o stdout -p brokers=192.168.1.3:9092 -p topics=some-topic
3031
```
3132

32-
### Configuration File
33+
### Configuration file
3334

34-
In your main configuration file append the following _Input_ & _Output_ sections:
35+
In your main configuration file append the following `Input` and `Output` sections:
3536

3637
```text
3738
[INPUT]
@@ -44,10 +45,9 @@ In your main configuration file append the following _Input_ & _Output_ sections
4445
Name stdout
4546
```
4647

47-
#### Example of using kafka input/output plugins
48+
#### Example of using Kafka input and output plugins
4849

49-
The Fluent Bit source repository contains a full example of using Fluent Bit to
50-
process Kafka records:
50+
The Fluent Bit source repository contains a full example of using Fluent Bit to process Kafka records:
5151

5252
```text
5353
[INPUT]
@@ -69,10 +69,10 @@ process Kafka records:
6969
topics fb-sink
7070
```
7171

72-
The above will connect to the broker listening on `kafka-broker:9092` and subscribe to the `fb-source` topic, polling for new messages every 100 milliseconds.
72+
The previous example will connect to the broker listening on `kafka-broker:9092` and subscribe to the `fb-source` topic, polling for new messages every 100 milliseconds.
7373

74-
Since the payload will be in json format, we ask the plugin to automatically parse the payload with `format json`.
74+
Since the payload will be in JSON format, the plugin is configured to parse the payload with `format json`.
7575

7676
Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker.
7777

78-
The example can be executed locally with `make start` in the `examples/kafka_filter` directory (docker/compose is used).
78+
The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used).

vale-styles/FluentBit/Spelling-exceptions.txt

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -97,6 +97,7 @@ Kube
9797
Kubernetes
9898
Kusto
9999
labelset
100+
librdkafka
100101
loadgenerator
101102
Logstash
102103
Lua

0 commit comments

Comments
 (0)