diff --git a/pipeline/inputs/kafka.md b/pipeline/inputs/kafka.md index a0c83b97d..68fb7d860 100644 --- a/pipeline/inputs/kafka.md +++ b/pipeline/inputs/kafka.md @@ -1,37 +1,38 @@ # Kafka -The Kafka input plugin allows subscribing to one or more Kafka topics to collect messages from an [Apache Kafka](https://kafka.apache.org/) service. -This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka) \(built-in dependency\). +The _Kafka_ input plugin subscribes to one or more Kafka topics to collect messages from an [Apache Kafka](https://kafka.apache.org/) service. -## Configuration Parameters +This plugin uses the official [librdkafka C library](https://github.com/edenhill/librdkafka) as a built-in dependency. + +## Configuration parameters | Key | Description | default | | :--- | :--- | :--- | -| brokers | Single or multiple list of Kafka Brokers, e.g: 192.168.1.3:9092, 192.168.1.4:9092. | | -| topics | Single entry or list of topics separated by comma \(,\) that Fluent Bit will subscribe to. | | -| format | Serialization format of the messages. If set to "json", the payload will be parsed as json. | none | -| client\_id | Client id passed to librdkafka. | | -| group\_id | Group id passed to librdkafka. | fluent-bit | -| poll\_ms | Kafka brokers polling interval in milliseconds. | 500 | -| Buffer\_Max\_Size | Specify the maximum size of buffer per cycle to poll kafka messages from subscribed topics. To increase throughput, specify larger size. | 4M | -| rdkafka.{property} | `{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) | | -| threaded | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | +| `brokers` | Single or multiple list of Kafka Brokers. For example: `192.168.1.3:9092`, `192.168.1.4:9092`. | _none_ | +| `topics` | Single entry or list of comma-separated topics (`,`) that Fluent Bit will subscribe to. | _none_ | +| `format` | Serialization format of the messages. If set to `json`, the payload will be parsed as JSON. | _none_ | +| `client_id` | Client id passed to librdkafka. | _none_ | +| `group_id` | Group id passed to librdkafka. | `fluent-bit` | +| `poll_ms` | Kafka brokers polling interval in milliseconds. | `500` | +| `Buffer_Max_Size` | Specify the maximum size of buffer per cycle to poll Kafka messages from subscribed topics. To increase throughput, specify larger size. | `4M` | +| `rdkafka.{property}` | `{property}` can be any [librdkafka properties](https://github.com/edenhill/librdkafka/blob/master/CONFIGURATION.md) | _none_ | +| `threaded` | Indicates whether to run this input in its own [thread](../../administration/multithreading.md#inputs). | `false` | -## Getting Started +## Get started -In order to subscribe/collect messages from Apache Kafka, you can run the plugin from the command line or through the configuration file: +To subscribe to or collect messages from Apache Kafka, run the plugin from the command line or through the configuration file: -### Command Line +### Command line -The **kafka** plugin can read parameters through the **-p** argument \(property\), e.g: +The Kafka plugin can read parameters through the `-p` argument (property): -```text -$ fluent-bit -i kafka -o stdout -p brokers=192.168.1.3:9092 -p topics=some-topic +```shell +fluent-bit -i kafka -o stdout -p brokers=192.168.1.3:9092 -p topics=some-topic ``` -### Configuration File +### Configuration file -In your main configuration file append the following _Input_ & _Output_ sections: +In your main configuration file append the following `Input` and `Output` sections: ```text [INPUT] @@ -44,10 +45,9 @@ In your main configuration file append the following _Input_ & _Output_ sections Name stdout ``` -#### Example of using kafka input/output plugins +#### Example of using Kafka input and output plugins -The Fluent Bit source repository contains a full example of using Fluent Bit to -process Kafka records: +The Fluent Bit source repository contains a full example of using Fluent Bit to process Kafka records: ```text [INPUT] @@ -69,10 +69,10 @@ process Kafka records: topics fb-sink ``` -The above will connect to the broker listening on `kafka-broker:9092` and subscribe to the `fb-source` topic, polling for new messages every 100 milliseconds. +The previous example will connect to the broker listening on `kafka-broker:9092` and subscribe to the `fb-source` topic, polling for new messages every 100 milliseconds. -Since the payload will be in json format, we ask the plugin to automatically parse the payload with `format json`. +Since the payload will be in JSON format, the plugin is configured to parse the payload with `format json`. Every message received is then processed with `kafka.lua` and sent back to the `fb-sink` topic of the same broker. -The example can be executed locally with `make start` in the `examples/kafka_filter` directory (docker/compose is used). +The example can be executed locally with `make start` in the `examples/kafka_filter` directory (`docker/compose` is used). diff --git a/vale-styles/FluentBit/Spelling-exceptions.txt b/vale-styles/FluentBit/Spelling-exceptions.txt index 00d01f7ab..435ca9fa7 100644 --- a/vale-styles/FluentBit/Spelling-exceptions.txt +++ b/vale-styles/FluentBit/Spelling-exceptions.txt @@ -97,6 +97,7 @@ Kube Kubernetes Kusto labelset +librdkafka loadgenerator Logstash Lua