Spring boot Kafka batching for all partitions

15 hours ago 2
ARTICLE AD BOX

I have Spring Boot with Kafka batching and a topic with 3 partitions. For example, I want to have 1 batch with 100 records continuously if I have a lot of messages in the topic, but if I have a low rate, then I want my app to send me a batch with 100 records or to wait 10 seconds and send me as big batch as possible.
Right now, I have the following properties in my test app (concurrency is 1):

fetch-min-size: 60_000 fetch-max-wait: 10_000 max-poll-records: 100

And the problem is that if I have 1 partition, it works fine, but if I create a topic with 3 partitions, I start getting 3 small batches with a 10-second interval each. I assume that the listener listens for partitions separately and does not combine messages from them.
Is it possible to achieve it without the creation of a custom implementation of a listener to cache the responses, probably, and then divide them by batch size? Or are there other solutions?

This property does not help since I use it for now, but it creates just a delay between polls, and in case of a big queue, I process very slowly.

idle-between-polls
Read Entire Article