Uploaded image for project: 'Spring Integration'
  1. Spring Integration
  2. INT-4065

Provide Redis metadata store that writes key/values instead of single hash

    XMLWordPrintable

    Details

    • Type: New Feature
    • Status: Open
    • Priority: Minor
    • Resolution: Unresolved
    • Affects Version/s: None
    • Fix Version/s: 5.2 Backlog
    • Component/s: Redis Support
    • Labels:
      None

      Description

      Took a look at the RedisMetadataStore for use with the IdempotentReceiverInterceptor for deduplication for our high throughput data streams. We noticed that it maintains a single hash object for the metadata. For this use case, having a single hash object for all metadata doesn't really work as we would blow out the memory of any server pretty quickly. Since we really don't care about data after a deterministic amount of time because we're really only trying to avoid reprocessing data we've already seen due to a replay scenario, ideally we would want the metadata to expire.

      Being that hash keys cannot have an expiration on them (i believe) and because the current implementation doesn't allow for metadata expiration, a version that doesn't use a hash but rather individual key/value pairs for each metadata record would suffice as we could turn expiration from the server. But more ideal would be to set a TTL for records individually for differing needs.

      I could see this in a few different forms.

      1) an alternative implementation of the MetadataStore (RedisKeyValueMetadataStore) that uses key/value instead of hash and that also allows for TTL
      or
      2) an implementation of the existing RedisMetadataStore that's takes in some sort of strategy for writing data to redis where one strategy would be to use a single hash and another would use key/value pairs prefixed with a key and accounted for TTL

        Attachments

          Activity

            People

            Assignee:
            grussell Gary Russell
            Reporter:
            loesak Aaron Loes
            Votes:
            2 Vote for this issue
            Watchers:
            4 Start watching this issue

              Dates

              Created:
              Updated: