间歇性 Redis 超时答案

【问题标题】：Intermittent Redis timeouts间歇性 Redis 超时
【发布时间】：2019-09-05 19:27:07
【问题描述】：

迁移到 GCP 和 Cloud Memorystore 后，我们遇到了间歇性的 Redis 超时。

我们从 GKE 中运行的 Java Spring 应用程序进行连接。在迁移到 GCP 之前，我们没有遇到这些问题。

与旧环境的一些主要区别：

我们将 4 个 Redis 实例合并为 2 个
我们运行的是我们自己的 Redis 服务器，而不是托管 (Memorystore)

关键数据：

一个实例平均每秒获得约 5,000 次调用
另一个实例每秒获得约 1,000 次调用

我们正在使用：

        <dependency>
            <groupId>org.springframework.data</groupId>
            <artifactId>spring-data-redis</artifactId>
            <version>2.1.10.RELEASE</version>
        </dependency>
        <dependency>
            <groupId>redis.clients</groupId>
            <artifactId>jedis</artifactId>
            <version>2.9.3</version>
        </dependency>

我们正在使用以下代码来实例化连接：

    public JedisConnectionFactory getJedisConnectionFactoryOne()
    {
        JedisPoolConfig jedisPoolConfig = new JedisPoolConfig();
        jedisPoolConfig.setMaxTotal(MAX_TOTAL);
        jedisPoolConfig.setMaxIdle(MAX_IDLE);
        jedisPoolConfig.setMinIdle(MIN_IDLE);
        JedisConnectionFactory jedisConnectionFactory = new JedisConnectionFactory(jedisPoolConfig);
        jedisConnectionFactory.setHostName(REDIS_HOST_1);
        jedisConnectionFactory.setPort(REDIS_PORT);
        return jedisConnectionFactory;
    }

我们已尝试调整池大小，但没有帮助。

我们会在一天中间歇性地遇到这些异常：

05-Sep-2019 08:18:16.597 SEVERE [http-nio-8080-exec-331] org.apache.catalina.core.StandardWrapperValve.invoke Servlet.service() for servlet [appServlet] in context with path [] threw exception [Request processing failed; 
nested exception is org.springframework.data.redis.RedisConnectionFailureException: java.net.SocketTimeoutException: Read timed out; 
nested exception is redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: Read timed out] with root cause

    java.net.SocketTimeoutException: Read timed out
        at java.net.SocketInputStream.socketRead0(Native Method)
        at java.net.SocketInputStream.socketRead(SocketInputStream.java:116)
        at java.net.SocketInputStream.read(SocketInputStream.java:171)
        at java.net.SocketInputStream.read(SocketInputStream.java:141)
        at java.net.SocketInputStream.read(SocketInputStream.java:127)
        at redis.clients.util.RedisInputStream.ensureFill(RedisInputStream.java:196)
        at redis.clients.util.RedisInputStream.readByte(RedisInputStream.java:40)
        at redis.clients.jedis.Protocol.process(Protocol.java:153)
        at redis.clients.jedis.Protocol.read(Protocol.java:218)
        at redis.clients.jedis.Connection.readProtocolWithCheckingBroken(Connection.java:341)
        at redis.clients.jedis.Connection.getBinaryBulkReply(Connection.java:260)
        at redis.clients.jedis.BinaryJedis.get(BinaryJedis.java:246)
        at org.springframework.data.redis.connection.jedis.JedisStringCommands.get(JedisStringCommands.java:66)
        at org.springframework.data.redis.connection.DefaultedRedisConnection.get(DefaultedRedisConnection.java:253)
        at org.springframework.data.redis.core.DefaultValueOperations$1.inRedis(DefaultValueOperations.java:57)
        at org.springframework.data.redis.core.AbstractOperations$ValueDeserializingRedisCallback.doInRedis(AbstractOperations.java:59)
        at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:224)
        at org.springframework.data.redis.core.RedisTemplate.execute(RedisTemplate.java:184)
        at org.springframework.data.redis.core.AbstractOperations.execute(AbstractOperations.java:95)
        at org.springframework.data.redis.core.DefaultValueOperations.get(DefaultValueOperations.java:53)
        at com.ranker.app.dao.impl.redis.tag.RedisTagDAO.getTag(RedisTagDAO.java:26)
        at com.ranker.app.service.redis.tag.RedisTagService.getTag(RedisTagService.java:20)
        at com.ranker.app.service.tag.TagService.getTag(TagService.java:48)
        at com.ranker.api.service.tag.TagApiService.getTag(TagApiService.java:113)
        at com.ranker.api.service.collection.CollectionListApiService.mapListToCollectionList(CollectionListApiService.java:171)
        at com.ranker.api.service.collection.CollectionListApiService.getCollectionListsFromMappings(CollectionListApiService.java:151)
        at com.ranker.api.service.collection.CollectionListApiService.getListCollectionInfo(CollectionListApiService.java:124)
        at com.ranker.api.service.list.ListPageApiService.getListCollectionInfo(ListPageApiService.java:355)
        at com.ranker.api.service.list.ListPageApiService.getListPageModel(ListPageApiService.java:264)
        at com.ranker.api.controller.ListPageController.getPageModel(ListPageController.java:58)
        at sun.reflect.GeneratedMethodAccessor1976.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:498)
        at org.springframework.web.method.support.InvocableHandlerMethod.doInvoke(InvocableHandlerMethod.java:189)
        at org.springframework.web.method.support.InvocableHandlerMethod.invokeForRequest(InvocableHandlerMethod.java:138)
        at org.springframework.web.servlet.mvc.method.annotation.ServletInvocableHandlerMethod.invokeAndHandle(ServletInvocableHandlerMethod.java:102)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.invokeHandlerMethod(RequestMappingHandlerAdapter.java:895)
        at org.springframework.web.servlet.mvc.method.annotation.RequestMappingHandlerAdapter.handleInternal(RequestMappingHandlerAdapter.java:800)
        at org.springframework.web.servlet.mvc.method.AbstractHandlerMethodAdapter.handle(AbstractHandlerMethodAdapter.java:87)
        at org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:1038)
        at org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:942)
        at org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:1005)
        at org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:908)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:661)
        at org.springframework.web.servlet.FrameworkServlet.service(FrameworkServlet.java:882)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:742)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:231)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:316)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.invoke(FilterSecurityInterceptor.java:126)
        at org.springframework.security.web.access.intercept.FilterSecurityInterceptor.doFilter(FilterSecurityInterceptor.java:90)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:114)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:111)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.servletapi.SecurityContextHolderAwareRequestFilter.doFilter(SecurityContextHolderAwareRequestFilter.java:169)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at com.ranker.api.security.AuthenticationTokenProcessingFilter.doFilter(AuthenticationTokenProcessingFilter.java:149)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.header.HeaderWriterFilter.doFilterInternal(HeaderWriterFilter.java:64)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.context.request.async.WebAsyncManagerIntegrationFilter.doFilterInternal(WebAsyncManagerIntegrationFilter.java:53)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:91)
        at org.springframework.security.web.FilterChainProxy$VirtualFilterChain.doFilter(FilterChainProxy.java:330)
        at org.springframework.security.web.FilterChainProxy.doFilterInternal(FilterChainProxy.java:213)
        at org.springframework.security.web.FilterChainProxy.doFilter(FilterChainProxy.java:176)
        at org.springframework.web.filter.DelegatingFilterProxy.invokeDelegate(DelegatingFilterProxy.java:357)
        at org.springframework.web.filter.DelegatingFilterProxy.doFilter(DelegatingFilterProxy.java:270)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at com.ranker.api.web.core.filter.CorsFilter.doFilterInternal(CorsFilter.java:32)
        at org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:107)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:193)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:166)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:199)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:96)
        at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:493)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:137)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:81)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:87)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:343)
        at org.apache.coyote.http11.Http11Processor.service(Http11Processor.java:798)
        at org.apache.coyote.AbstractProcessorLight.process(AbstractProcessorLight.java:66)
        at org.apache.coyote.AbstractProtocol$ConnectionHandler.process(AbstractProtocol.java:808)
        at org.apache.tomcat.util.net.NioEndpoint$SocketProcessor.doRun(NioEndpoint.java:1498)
        at org.apache.tomcat.util.net.SocketProcessorBase.run(SocketProcessorBase.java:49)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
        at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
        at java.lang.Thread.run(Thread.java:748)```

【问题讨论】：

你有没有机会监控redis服务器的cpu和内存利用率（已用内存）。因为您现在只有 2 个 redis 实例，而不是旧设置中的 4 个。另外请检查应用服务器和redis之间的网络带宽和网络利用率。将此与旧配置进行比较。
CPU 在使用率较高的实例上永远不会超过 ~20%，在使用率较低的实例上不会超过 ~10%。两者的内存利用率均为 60%-70%。我无法与旧配置进行比较，因为它们已经被终止，我们的监控工具也不见了。当前网络利用率在一个实例上为 ~5MB/s 传出和 ~1.5MB/s 传入，在另一个实例上为 ~10MB/s 传出和 ~1MB/s 传入。
Redis 客户端的默认超时时间好像是 2 秒，应该绰绰有余，但是有没有人认为我们需要在客户端进行调整？
你能看看 maxmemory 策略here 我在想，根据内存的负载，它有时可能会因为尖峰而超时。我认为默认值是“volatile-lru”。看看，让我知道它是否适合你，不过我认为为了更好地检查你的确切设置，你应该打开一个谷歌云平台支持的案例。
@eespinola 谢谢，我们正在调查此事。我们确实已经在 GCP 支持下打开了一个案例，只是希望获得进一步的见解。

标签： spring-mvc redis jedis spring-data-redis google-cloud-memorystore

【解决方案1】：

将客户端中的超时从默认（2 秒）更改为 5 秒似乎已经完全停止了错误。

jedisConnectionFactory.setTimeout(5000);

这很奇怪，因为我们的 APM 监控显示 p95 redis 调用

我们将升级到 spring-data-redis 2.2.x (jedis 3.x) 一旦它脱离快照并发布。

【讨论】：

【解决方案2】：

在使用 CloudRun 进行部署 9 天后，我实际上发生了这种情况。每 20 分钟就会发生几个这样的错误，我对 redis 实例的请求速度约为 10-30 次。

我为解决它（99% 已解决，现在很少断开连接）所做的只是重新部署我的云运行容器。

以下是一些可视化日志，可以查看发生了什么：

PS：另外，我的默认超时设置为 60 秒，所以其他答案对我没有帮助。

【讨论】：