Resume kafka stream when consumer is within a group

Question

I have a circuit breaker in my Spring Cloud Stream application. It is pausing/resuming stream very well when circuit is changing state and when my stream consumer is anonymous (not in a group).

When the consumer is belong to a group, pausing the stream works well, but resuming is being "ignored", which eventually ending with timeout and leaving the group. Any explanation why this inconsistent behavior occurs?

Spring cloud stream version is 3.0.8.RELEASE.

This is my circuit breaker state transition handler:

@Component
public class CircuitBreakerKafkaStream {

  private final Logger log = LoggerFactory.getLogger(CircuitBreakerKafkaStream.class);

  private List<InputBindingLifecycle> inputBindingLifecycles;

  public CircuitBreakerKafkaStream(List<InputBindingLifecycle> inputBindingLifecycles) {
    this.inputBindingLifecycles = inputBindingLifecycles;
  }

  @Override
  // Pause or resume all of input bindings by the state of circuit breaker.
  public void transitionHandler(CircuitBreaker.State toState) {
    log.info("Circuit breaker is transitioning to {} state", toState.toString());

    if (toState == CircuitBreaker.State.OPEN) {
      log.info("Pausing kafka binder...");
      gatherInputBindings().stream().forEach(binding -> binding.pause());
    } else {
      log.info("Resuming kafka binder...");
      gatherInputBindings().stream().forEach(binding -> binding.resume());
    }
  }

  private List<Binding<?>> gatherInputBindings() {
    List<Binding<?>> inputBindings = new ArrayList<>();
    for (InputBindingLifecycle inputBindingLifecycle : this.inputBindingLifecycles) {
      Collection<Binding<?>> lifecycleInputBindings =
          (Collection<Binding<?>>)
              new DirectFieldAccessor(inputBindingLifecycle).getPropertyValue("inputBindings");
      inputBindings.addAll(lifecycleInputBindings);
    }
    return inputBindings;
  }
}

Update: I think that the problem is more specific. The circuit breaker have open, half_open and close states. Once the circuit is closed and stream is resumed, it is consuming some amount of messages, and then for some reason it is stopping, until a poll timeout occurs.

question from:https://stackoverflow.com/questions/65855011/resume-kafka-stream-when-consumer-is-within-a-group

Answer 1

You should increase max.poll.timeout.ms, by default it is set to 5 minutes. If stream does not send poll request within this limit, kafka broker considers that streams as dead.

We looked for a solution for delay processing in kafka streams, we wanted to delay message delivery in couple of minutes and increasing max.poll.timeout.ms avoided to kick off stream.