Can I have multiple consumers process messages from a single queue in Apache Kafka

Question

What I want to achieve is this:

• Subscribe multiple consumers to a single topic
• Each message should be processed by only one consumer
• No consumer should be idle as long as the topic has unprocessed messages

As far as I understand I can get the first two points by defining multiple partitions for that topic, at least one partition per consumer. But that doesn't satisfy my 3rd requirement. Assume I created a topic with 3 partitions and subscribe 3 consumers (same group id). Then a producer pushes a bulk of 300 messages which are equally distributed to all three partitions. So each partition contains 100 messages and consumers start to process. For whatever reasons one consumer takes longer and at some point when 2 consumers have already processed all messages of their partitions, the 3rd consumer still has several messages left to process. In that scenario the 2 fast consumers would fall idle while the 3rd one is still processing messages.

What I have in mind is something like a topic with only one partition and all consumers subscribed share the same offset index. Then, whenever a consumer is idle it will fetch the next message from the topic that hasn't been processed by any of the consumers yet. I know that Kafka cannot have multiple consumers of the same group on one partition. It's just to explain my intentions.

Is there a way to configure my topology to meet my requirements?

question from:https://stackoverflow.com/questions/65941331/can-i-have-multiple-consumers-process-messages-from-a-single-queue-in-apache-kaf

Answer 1

Waitting for answers