-
Type:
Improvement
-
Status: Closed
-
Priority:
Minor
-
Resolution: Works as Designed
-
Affects Version/s: 1.8.13 (Ingalls SR13)
-
Fix Version/s: None
-
Component/s: Lettuce Driver
-
Labels:None
I am trying to setup spring CacheManager using Redistemplate and LettuceConnectionFactory connected to a Redis sentinel
Below is the code I used (slightly reduced for simplicity)
@Bean public CacheManager cacheManager(RedisTemplate redisTemplate){ RedisCacheManager redisCacheManager = new RedisCacheManager(redisTemplate); return redisCacheManager; } @Bean public RedisTemplate redisTemplate(LettuceConnectionFactory lcf){ RedisTemplate redisTemplate = new RedisTemplate(); redisTemplate.setConnectionFactory(ltf); return redisTemplate; } @Bean public LettuceConnectionFactory lcf(LettucePool lettucePool){ LettuceConnectionFactory lcf = new LettuceConnectionFactory(lettucePool); return lcf; } @Bean public LettucePool lettucePool() { DefaultLettucePool lettucePool = new DefaultLettucePool(new RedisSentinelConfiguration().master("mymaster") .sentinel(new RedisNode(<name>,26379)) lettucePool.afterPropertiesSet() }
It is not expected that the redis server is always up and working 100% of time in a production environment. I did therefore try to simulate redis server going down by restarting the redis sentinel. The problem is that when the redis sentinel was down, the application stopped responding to the requests or took almost 30-50 seconds to respond. It seems like the LettuceConnectionFactory is locking when trying to reconnect, and builds up a large queue with pending requests which takes very long time when it tries to retry every request in the queue (not sure if this is the scenario, but it is was I understood from reading the documentation).
After lot of debugging I was able to fix this problem by copying and modifying the afterPropertiesSet method in DefaultLettucePool:
public void afterPropertiesSet() { if (clientResources != null){ this.client = Redisclient.create(clientResources, getRedisUri) } else { this.client = RedisClient.create(getRedisUri()) } /** Custom Code **/ this.client.setOptions(ClientOptions.builder() .autoReconnect(true) .cancelCommandsonReconnectFailure(true) .disconnectedBehaviour(REJECT_COMMANDS) .socketOptions(SocketOptions.builder.connectTimeout(200, MILLISECONDS).build()).build(); /** Custom code end **/ client.setDefaultTimeout(timeout, MILLISECONDS) this.internalPool = new GenericObjectPool(new CustomLettucePool.LettuceFactory(client, dbindex), poolConfig); }
I was therefore wondering if you could add an option to set the ClientOptions of RedisClient when configuring the DefaultLettucePool or find another solution to the locking problem by adding more configuration options. It is very important for the application to keep running as usual without cache when the Redis Server is down.
I tried to set the reconnectDelay and connectionTimeout but nothing helped. This was the only solution that actually worked. I also tried to setup without using the LettucePool.
I tried to use JedisConnectionFactory, but the problem with JedisConnectionFactory is that it won't reconnect to the redis sentinel after redis server restart.