Uploaded image for project: 'Spring Data Redis'
  1. Spring Data Redis
  2. DATAREDIS-834

Redis connection using LettuceConnectionFactory locks for a long time when the redis server is down.

    Details

    • Type: Improvement
    • Status: Closed
    • Priority: Minor
    • Resolution: Works as Designed
    • Affects Version/s: 1.8.13 (Ingalls SR13)
    • Fix Version/s: None
    • Component/s: Lettuce Driver
    • Labels:
      None

      Description

      I am trying to setup spring CacheManager using Redistemplate and LettuceConnectionFactory connected to a Redis sentinel

      Below is the code I used (slightly reduced for simplicity)

      @Bean
      public CacheManager cacheManager(RedisTemplate redisTemplate){
          RedisCacheManager redisCacheManager = new RedisCacheManager(redisTemplate);
          return redisCacheManager;
      }
      
      @Bean
      public RedisTemplate redisTemplate(LettuceConnectionFactory lcf){
          RedisTemplate redisTemplate = new RedisTemplate();
          redisTemplate.setConnectionFactory(ltf);
          return redisTemplate;
      }
      
      @Bean
      public LettuceConnectionFactory lcf(LettucePool lettucePool){
          LettuceConnectionFactory lcf = new LettuceConnectionFactory(lettucePool);
          return lcf;
      }
      
      @Bean
      public LettucePool lettucePool() {
        DefaultLettucePool lettucePool = new DefaultLettucePool(new      RedisSentinelConfiguration().master("mymaster") .sentinel(new RedisNode(<name>,26379))
      lettucePool.afterPropertiesSet()
      }
          

      It is not expected that the redis server is always up and working 100% of time in a production environment. I did therefore try to simulate redis server going down by restarting the redis sentinel. The problem is that when the redis sentinel was down, the application stopped responding to the requests or took almost 30-50 seconds to respond. It seems like the LettuceConnectionFactory is locking when trying to reconnect, and builds up a large queue with pending requests which takes very long time when it tries to retry every request in the queue (not sure if this is the scenario, but it is was I understood from reading the documentation).

      After lot of debugging I was able to fix this problem by copying and modifying the afterPropertiesSet method in DefaultLettucePool:

      public void afterPropertiesSet() {
          if (clientResources != null){
             this.client = Redisclient.create(clientResources, getRedisUri)
          } else {
             this.client = RedisClient.create(getRedisUri())
          }
      
          /** Custom Code **/
          this.client.setOptions(ClientOptions.builder()
             .autoReconnect(true)
             .cancelCommandsonReconnectFailure(true)
             .disconnectedBehaviour(REJECT_COMMANDS)
             .socketOptions(SocketOptions.builder.connectTimeout(200, MILLISECONDS).build()).build();
         /** Custom code end **/
      
              client.setDefaultTimeout(timeout, MILLISECONDS)
              this.internalPool = new GenericObjectPool(new CustomLettucePool.LettuceFactory(client, dbindex), poolConfig);
      
      }

      I was therefore wondering if you could add an option to set the ClientOptions of RedisClient when configuring the DefaultLettucePool or find another solution to the locking problem by adding more configuration options. It is very important for the application to keep running as usual without cache when the Redis Server is down.

      I tried to set the reconnectDelay and connectionTimeout but nothing helped. This was the only solution that actually worked. I also tried to setup without using the LettucePool.

      I tried to use JedisConnectionFactory, but the problem with JedisConnectionFactory is that it won't reconnect to the redis sentinel after redis server restart.

        Attachments

          Activity

            People

            • Assignee:
              mp911de Mark Paluch
              Reporter:
              cenar93 Ugur Alpay Cenar
              Last updater:
              Mark Paluch
            • Votes:
              0 Vote for this issue
              Watchers:
              2 Start watching this issue

              Dates

              • Created:
                Updated:
                Resolved: