We at Intuit are working on a project and looking to use Redis Streams. I found that Spring Data Redis provides good support for Redis Streams in it's new upcoming version of 2.2.0.
So I worked on a POC to check out 2.2.0.M1 and saw that it has a good feature to have a onMessage functionality for streams. Here is a link to POC. https://github.com/akaskj/redispoc
However I noticed that a new thread is created and put to waiting stage so that it can poll the stream for every subscription we make to a stream (Attached screenshots). This will not be scalable with our use case as we may be listening to thousands of streams at a given time. So we cannot have that many treads at a given time. Here is the link to StreamPollTask
My proposal is to have a better threading design to poll streams where we can have limited number(configurable) of threads for polling and we distribute the task of polling among them.
Could you please let me know if this is something I can make changes and contribute back to Spring-data-redis?
Or was there is a use case that you had in mind for the design that is done currently?
Or am I doing anything wrong?
Anyway please let me know and I can send you more details on the design.