You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
I'm investigating DLT and retry flows with spring-kafka however, I stumbled on the difficulties of using it in tandem with a batch listener.
I can see historically things have changed around this area which has left me a little confused.
Originally I had just assumed that underneath the listener, when a batch processing exception is thrown that it would either produce the whole batch to the -retry topic or the -dlt or it would do something like binary search on the batch to find bad messages and pausing the consumer until complete.
I then discovered BatchListenerFailedException where you need to find the bad messages yourself during processing and throw this exception but I'm still not clear on what will happen here without delving into the code, is the bad message published to the dlt? do we just commit the offset up to that message and continually retry? how can this be integrated with @DltHandler?
Ultimately, if the answers to the questions I have is, "it's not supported" then that's fine although a little disappointing, but it would be good to update the documentation found here.
I also think the documentation could be better around the handling of deserialization exceptions vs processing exceptions e.g. if the error happens during processing you need to have a serializer set up in a KafkaTemplate that's able to serialize the message OR with batch I currently don't understand what the ErrorHandlingDeserializer will do with my batch mode consumer.
As for batch consuming as a concept, it's far more efficient in my scenarios where I'm consuming from a topic then performing database operations etc.
This issue describes my pain somewhat but there's not been much activity on it.
The text was updated successfully, but these errors were encountered:
Expected Behavior
there is docs
Current Behavior
there is no docs (on this page at least)
Context
I'm investigating DLT and retry flows with spring-kafka however, I stumbled on the difficulties of using it in tandem with a
batch
listener.I can see historically things have changed around this area which has left me a little confused.
Originally I had just assumed that underneath the listener, when a batch processing exception is thrown that it would either produce the whole batch to the
-retry
topic or the-dlt
or it would do something like binary search on the batch to find bad messages and pausing the consumer until complete.I then discovered
BatchListenerFailedException
where you need to find the bad messages yourself during processing and throw this exception but I'm still not clear on what will happen here without delving into the code, is the bad message published to the dlt? do we just commit the offset up to that message and continually retry? how can this be integrated with@DltHandler
?Ultimately, if the answers to the questions I have is, "it's not supported" then that's fine although a little disappointing, but it would be good to update the documentation found here.
I also think the documentation could be better around the handling of deserialization exceptions vs processing exceptions e.g. if the error happens during processing you need to have a serializer set up in a KafkaTemplate that's able to serialize the message OR with batch I currently don't understand what the
ErrorHandlingDeserializer
will do with my batch mode consumer.As for batch consuming as a concept, it's far more efficient in my scenarios where I'm consuming from a topic then performing database operations etc.
This issue describes my pain somewhat but there's not been much activity on it.
The text was updated successfully, but these errors were encountered: