apache kafka - duplicate events by consumer

Question

we observed that one of the consumer try to pick the events multiple times from kafka topic. we have the below seetings on consumer application side. spring.kafka.consumer.enable-auto-commit=false & spring.kafka.consumer.auto-offset-reset=earliest. how to avoid the duplicate by the consumer application. Do we need to fine tune the above configuration settings to avoid the consumer to pick the events multiple times from the kafka topic.

question from:https://stackoverflow.com/questions/65640985/duplicate-events-by-consumer

Answer 1

Since you've disabled auto commits, you do need to fine tune when you actually commit a record, otherwise you could have at least once processing.

You could also read the examples of the exactly once processing capabilities using transactions and idempotent producers

The auto.offset.reset only applies if your consumer group is removed, or never exists at all (you're not committing anything). In that case, you're always going to read from the beginning of the topic