Start consumer with kafka file

Fixes jlandersen#96 Signed-off-by: azerr <[email protected]>
jnsjunior · Feb 25, 2021 · 0178376 · 0178376
1 parent 189b58f
commit 0178376
Show file tree

Hide file tree

Showing 37 changed files with 984 additions and 5,060 deletions.
diff --git a/CHANGELOG.md b/CHANGELOG.md
@@ -11,6 +11,8 @@ All notable changes to Kafka extension will be documented in this file.
 - Added the option to enable basic SSL support for clusters without authentication. See [#84](https://github.com/jlandersen/vscode-kafka/issues/84).
 - The consumer view now provides a `Clear Consumer View` command. See [#84](https://github.com/jlandersen/vscode-kafka/issues/40).
 - Added support for consumer group deletion. See [#26](https://github.com/jlandersen/vscode-kafka/issues/26).
+- .kafka files can define a `CONSUMER` block, to start a consumer group with a given offset, partitions. See [#96](https://github.com/jlandersen/vscode-kafka/issues/96).
+- .kafka files show the selected cluster as a codelens on the first line. See [#102](https://github.com/jlandersen/vscode-kafka/issues/102).
 
 ### Changed
 - Improved the "New cluster" and "New topic" wizards: now include validation and a back button. See [#21](https://github.com/jlandersen/vscode-kafka/issues/21).

diff --git a/README.md b/README.md
@@ -35,11 +35,11 @@ The Kafka explorer shows configured clusters with their topics, brokers, consume
 
 ![Screenshot-3](docs/assets/screen-3.png)
 
-See [Kafka explorer](https://github.com/jlandersen/vscode-kafka/blob/master/docs/Explorer.md) section for more informations.
+See [Kafka explorer](https://github.com/jlandersen/vscode-kafka/blob/master/docs/Explorer.md) section for more information.
 
 ## Producing messages
 
-Producing can be done by creating a `.kafka` file. Write simple producers using the following format:
+Define simple producers in a `[.kafka](https://github.com/jlandersen/vscode-kafka/blob/master/docs/KafkaFile.md#kafkafile)` file, using the following format:
 
 ```json
 PRODUCER keyed-message
@@ -56,15 +56,24 @@ topic: json-events
 }
 ```
 
-![Screenshot-4](docs/assets/screen-4.png)
+![Producers](docs/assets/kafka-file-producers.png)
 
-See [Producing messages](https://github.com/jlandersen/vscode-kafka/blob/master/docs/Producing.md) section for more informations.
+See [Producing messages](https://github.com/jlandersen/vscode-kafka/blob/master/docs/Producing.md) section for more information.
 
 ## Consuming messages
 
-Consuming topics can be done by right clicking a topic in the explorer or from the command palette.  
+Consuming topics can be done by right-clicking on a topic in the Kafka explorer, from the command palette, or from a `[.kafka](https://github.com/jlandersen/vscode-kafka/blob/master/docs/KafkaFile.md#kafkafile)` file:
 
-See [Consuming messages](https://github.com/jlandersen/vscode-kafka/blob/master/docs/Consuming.md) section for more informations.
+```
+CONSUMER consumer-group-id
+topic: json-events
+partitions: 0
+from: 1
+```
+
+![Start Consumer with Kafka file](docs/assets/start-consumer-from-kafkafile.png)
+
+See [Consuming messages](https://github.com/jlandersen/vscode-kafka/blob/master/docs/Consuming.md) section for more information.
 
 ## CI Builds
 

diff --git a/docs/Consuming.md b/docs/Consuming.md
@@ -3,6 +3,7 @@
 Consuming topics can be done:
 
 * from the [Kafka Explorer](#kafka-explorer), by right-clicking on a topic and selecting "Start Consumer".
+* from a [.kafka](KafkaFile.md#kafkafile) file by clicking on the `Start consumer` codelens displayed above a `CONSUMER` block.
 * from the [Start Consumer](#start-consumer-command), from the command palette.
 
 ## Consume with ...
@@ -24,6 +25,38 @@ Known limitations:
 * UTF-8 encoded keys and values only. If data is encoded differently, it will not be pretty.
 * One consumer group is created per topic (may change in the future to just have one for the extension).
 
+### Kafka file
+
+Define simple consumers in a `.kafka` file, using the following format:
+
+```
+CONSUMER consumer-group-id
+topic: json-events
+partitions: 0
+from: 1
+```
+
+Click on the `Start consumer` link above the `CONSUMER` line to start the consumer group:
+
+![Start Consumer with Kafka file](assets/start-consumer-from-kafkafile.png)
+
+The `CONSUMER` block defines:
+
+ * `consumer group id` which is declared after CONSUMER *[required]*.
+ * `topic`: the topic id *[required]*.
+ * `from`: the offset from which the consumer group will start consuming messages from. Possible values are: `earliest`, `latest`, or an integer value. *[optional]*.
+ * `partitions`: the partition number(s), or a partitions range, or a combinaison of partitions ranges *[optional]*. eg:
+    * 0
+    * 0,1,2
+    * 0-2
+    * 0,2-3
+
+A codelens is displayed above each `CONSUMER` line, showing the `status` of the consumer group (started / stopped), and provides `Start consumer` / `Stop consumer` commands according to that status.
+
+Completion snippets can help you quickly bootstrap new `CONSUMER` blocks:
+
+![Consumer snippets](assets/kafka-file-consumer-snippet.png)
+
 ### Start Consumer command
 
 ![Start Consumer from command palette](assets/start-consumer-from-command.png)

diff --git a/docs/Explorer.md b/docs/Explorer.md
@@ -8,6 +8,16 @@ The Kafka explorer shows configured clusters with their topics, brokers, consume
 
 ![Screenshot-3](assets/screen-3.png)
 
+## Cluster Item
+
+In the Kafka explorer, right-click on a cluster to access several options.
+
+### Select Cluster
+
+Before using a [.kafka file](KafkaFile.md#kafkafile), a cluster must be selected, you can use the `Select cluster` menu item:
+
+![Select cluster](assets/kafka-explorer-select-cluster.png)
+
 ## Actions
 
 ### Copy Label

diff --git a/docs/KafkaFile.md b/docs/KafkaFile.md
@@ -0,0 +1,32 @@
+# Kafka file
+
+A Kafka file is a file with  the `.kafka` file extension. It provides, for the currently selected cluster, the ability to declare:
+
+ * [PRODUCER](#PRODUCER) to produce records.
+ * [CONSUMER](#CONSUMER) to start /stop a consumer group.
+
+## Select cluster
+
+Before performing actions from a `.kafka` file, a cluster must be selected:
+
+![Select cluster](assets/kafka-explorer-select-cluster.png)
+
+Once a cluster has been selected, the following codelens will appear in the `.kafka` file:
+
+![Selected cluster](assets/kafka-file-cluster.png )
+
+## PRODUCER
+
+Declare `PRODUCER` blocks to easily produce records:
+
+![Kafka file / PRODUCER](assets/kafka-file-producer.png)
+
+See [Producing messages](Producing.md#producing) for more information.
+
+## CONSUMER
+
+Declare `CONSUMER` blocks  to start/stop consumer groups on a topic, with optional group id, offset and partitions attributes:
+
+![Kafka file / CONSUMER](assets/kafka-file-consumer.png)
+
+See [Consuming messages](Consuming.md#kafka-file) for more information.
diff --git a/docs/Producing.md b/docs/Producing.md
@@ -1,6 +1,6 @@
 # Producing messages
 
-Producing can be done by creating a `.kafka` file. Write simple producers using the following format:
+Define simple producers in a [.kafka](KafkaFile.md#kafkafile) file, using the following format:
 
 ```
 PRODUCER keyed-message
@@ -17,7 +17,10 @@ topic: json-events
 }
 ```
 
-For actually producing a single record, click on the `Produce record` link above the `PRODUCER` line; for producing 10 records, click on `Produce record x 10`.
+To produce a single record, click on the `Produce record` link above the `PRODUCER` line; to produce 10 records, click on `Produce record x 10`.
+
+![Producers](assets/kafka-file-producers.png)
+
 The log about produced messages is printed in the `Kafka Producer Log` Output view.
 
 ## Randomized content

diff --git a/docs/README.md b/docs/README.md
@@ -3,5 +3,6 @@
 Welcome to the [vscode-kafka](https://github.com/jlandersen/vscode-kafka) documentation.
 
 * [Kafka Explorer](Explorer.md#explorer)
+* [Kafka file](KafkaFile.md#kafkafile)
 * [Producing messages](Producing.md#producing-messages)
 * [Consuming messages](Consuming.md#consuming-messages)
diff --git a/docs/assets/kafka-explorer-select-cluster.png b/docs/assets/kafka-explorer-select-cluster.png
diff --git a/docs/assets/kafka-file-cluster.png b/docs/assets/kafka-file-cluster.png
diff --git a/docs/assets/kafka-file-consumer-snippet.png b/docs/assets/kafka-file-consumer-snippet.png
diff --git a/docs/assets/kafka-file-consumer.png b/docs/assets/kafka-file-consumer.png
diff --git a/docs/assets/kafka-file-producer.png b/docs/assets/kafka-file-producer.png
diff --git a/docs/assets/kafka-file-producers.png b/docs/assets/kafka-file-producers.png
diff --git a/docs/assets/start-consumer-from-kafkafile.png b/docs/assets/start-consumer-from-kafkafile.png
diff --git a/language-configuration.json b/language-configuration.json
@@ -23,7 +23,7 @@
     ],
     "folding": {
         "markers": {
-            "start": "^(PRODUCER).*$",
+            "start": "^(PRODUCER|CONSUMER).*$",
             "end": "^(#{3,})$"
         }
     }