Spring kafka stream receives duplicate messages on restart

[rajagopalmani]

• My Spring boot application doesn't have a sink connect but instead a simple foreach where I do processing of message ( posting it to a Soap WebService in real application ).

• It is a stateless stream.

• enabled PROCESSING_GUARANTEE_CONFIG, "exactly_once_v2".

• To keep simplicity, when there is an error, I quit application.

• To keep simplicity I shall send numbers as String messages

• To simulate my issue I have created an if statement in foreach loop to throw error when message number % 15 == 0

• Lets say I send number 1 to 10 as String messages at first from producer. The consumer consumes it all fine.

• After a while I send 11 to 20 as String messages from producer. The consumer will fail while processing number 15 due to if case, like below:

2023-09-10T06:56:06.491+03:00  INFO 20260 --- [-StreamThread-1] c.e.k.consumer.StringStreamProcessor     : Processing key:11, value:11
2023-09-10T06:56:06.491+03:00  INFO 20260 --- [-StreamThread-1] c.e.k.consumer.StringStreamProcessor     : Processing key:12, value:12
2023-09-10T06:56:06.492+03:00  INFO 20260 --- [-StreamThread-1] c.e.k.consumer.StringStreamProcessor     : Processing key:13, value:13
2023-09-10T06:56:06.492+03:00  INFO 20260 --- [-StreamThread-1] c.e.k.consumer.StringStreamProcessor     : Processing key:14, value:14
2023-09-10T06:56:06.492+03:00  INFO 20260 --- [-StreamThread-1] c.e.k.consumer.StringStreamProcessor     : Processing key:15, value:15
2023-09-10T06:56:06.493+03:00 ERROR 20260 --- [-StreamThread-1] o.a.k.s.p.internals.TaskExecutor         : stream-thread [example-app-d17c4596-23d7-4bed-80b8-72cb90801109-StreamThread-1] Failed to process stream task 0_0 due to the following error:

org.apache.kafka.streams.errors.StreamsException: Exception caught in process. taskId=0_0, processor=KSTREAM-SOURCE-0000000000, topic=string-topic, partition=0, offset=57, stacktrace=java.lang.RuntimeException: Mocking Exception on [num % 5] while processing

• Now when I restart my consumer, it will try to reprocess from where it left last time.
• My expectation is that the stream will start from message 15 where it failed.
• However my kafka stream consumer starts from 11th message again, that means I will process messages from 11 to 14 again which I do not want.

2023-09-10T07:28:20.188+03:00  INFO 884 --- [-StreamThread-1] c.e.k.consumer.StringStreamProcessor     : Processing key:11, value:11
2023-09-10T07:28:20.188+03:00  INFO 884 --- [-StreamThread-1] c.e.k.consumer.StringStreamProcessor     : Processing key:12, value:12
2023-09-10T07:28:20.188+03:00  INFO 884 --- [-StreamThread-1] c.e.k.consumer.StringStreamProcessor     : Processing key:13, value:13
2023-09-10T07:28:20.188+03:00  INFO 884 --- [-StreamThread-1] c.e.k.consumer.StringStreamProcessor     : Processing key:14, value:14
2023-09-10T07:28:20.188+03:00  INFO 884 --- [-StreamThread-1] c.e.k.consumer.StringStreamProcessor     : Processing key:15, value:15
2023-09-10T07:28:20.189+03:00 ERROR 884 --- [-StreamThread-1] o.a.k.s.p.internals.TaskExecutor         : stream-thread [example-app-47aed0ae-a22d-4420-8001-02b015517108-StreamThread-1] Failed to process stream task 0_0 due to the following error:

• Any idea what I do wrong ?
• I have created git project for people to clone and simulate it easily
  https://github.com/rajagopalmani/kafka-stream-project.git
  Change the producer main class for loop i value to modify the number stream produced
  The kafka docker setup I used is also attached in docker-compose.yaml of project

[garyrussell]

"Exactly Once" in Kafka means a complete consume->process->produce will be performed exactly once.

However, a common misconception is that the consume->process part is always at least once.

In this case, it looks like the offsets are not being committed, for some reason; we are not Kafka Streams experts here.

This has nothing to do with Spring; you should ask questions about Kafka itself to the wider Kafka community, either on Stack Overflow or one of their mailing lists.