Uploaded image for project: 'Spring Data Redis'
  1. Spring Data Redis
  2. DATAREDIS-954

Improve thread management for Redis streams

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Major
    • Resolution: Won't Fix
    • Affects Version/s: 2.2 M2 (Moore)
    • Fix Version/s: None
    • Component/s: Core

      Description

      Hi,

      We at Intuit are working on a project and looking to use Redis Streams. I found that Spring Data Redis provides good support for Redis Streams in it's new upcoming version of 2.2.0. 

      So I worked on a POC to check out 2.2.0.M1 and saw that it has a good feature to have a onMessage functionality for streams. Here is a link to POC. https://github.com/akaskj/redispoc

      However I noticed that a new thread is created and put to waiting stage so that it can poll the stream for every subscription we make to a stream (Attached screenshots). This will not be scalable with our use case as we may be listening to thousands of streams at a given time. So we cannot have that many treads at a given time. Here is the link to StreamPollTask

      My proposal is to have a better threading design to poll streams where we can have limited number(configurable) of threads for polling and we distribute the task of polling among them.

      Could you please let me know if this is something I can make changes and contribute back to Spring-data-redis?

      Or was there is a use case that you had in mind for the design that is done currently?

      Or am I doing anything wrong?

      Anyway please let me know and I can send you more details on the design.

       

      Thank you,

      Akash

       

        Attachments

          Activity

            People

            • Assignee:
              mp911de Mark Paluch
              Reporter:
              ajayachandra Akash Kengunte Jayachandra
              Last updater:
              Mark Paluch
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: