ARTICLE AD BOX
I have Spring Boot with Kafka batching and a topic with 3 partitions. For example, I want to have 1 batch with 100 records continuously if I have a lot of messages in the topic, but if I have a low rate, then I want my app to send me a batch with 100 records or to wait 10 seconds and send me as big batch as possible.
Right now, I have the following properties in my test app (concurrency is 1):
And the problem is that if I have 1 partition, it works fine, but if I create a topic with 3 partitions, I start getting 3 small batches with a 10-second interval each. I assume that the listener listens for partitions separately and does not combine messages from them.
Is it possible to achieve it without the creation of a custom implementation of a listener to cache the responses, probably, and then divide them by batch size? Or are there other solutions?
This property does not help since I use it for now, but it creates just a delay between polls, and in case of a big queue, I process very slowly.
idle-between-polls