Redis（AWS Elasticache）在一段时间后停止工作答案

【问题标题】：Redis (AWS Elasticache) stops working after sometimeRedis（AWS Elasticache）在一段时间后停止工作
【发布时间】：2017-04-06 13:56:30
【问题描述】：

我们之前使用将用户会话存储在我们的数据库表 (Postgres RDS) 中
我们决定将用户会话从 DB 迁移到 Redis，并在我们的应用程序中进行了更改
对于 Redis，我们决定使用具有 1 个分片、2 个节点（主 + 副本）并启用多可用区的 Elastic Cache 服务
在 Live 环境中，一切都很顺利，直到会话数超过 50 万（下午 3 点左右）
此时，Redis Node 突然停止响应，导致我们的生产环境完全崩溃（等待连接的线程过多）
我们必须重启实例才能恢复服务
晚上 9 点左右再次发生这种情况

Java端(spring)产生的异常

2016/11/22 09:19:31.749 <a href="http-nio-8080-exec-780">http-nio-8080-exec-780</a> <a href="https://forums.aws.amazon.com/">ERROR</a> org.apache.catalina.core.ContainerBase.<a href="https://forums.aws.amazon.com/">Tomcat</a>.<a href="https://forums.aws.amazon.com/">localhost</a>.[/].<a href="https://forums.aws.amazon.com/">dispatcherServlet</a> - Servlet.service() for servlet <a href="https://forums.aws.amazon.com/">dispatcherServlet] in context with path [</a> threw exception

org.springframework.data.redis.RedisConnectionFailureException: Cannot get Jedis connection; nested exception is redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:140) ~<strike>spring-data-redis-1.4.2.RELEASE.jar!/:1.4.2.RELEASE</strike>

at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.getConnection(JedisConnectionFactory.java:229) ~<strike>spring-data-redis-1.4.2.RELEASE.jar!/:1.4.2.RELEASE</strike>

....

at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) <strike>na:1.7.0_72</strike>

at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61) <strike>tomcat-embed-core-8.0.20.jar!/:8.0.20</strike>

at java.lang.Thread.run(Thread.java:745) <strike>na:1.7.0_72</strike>

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: Could not get a resource from the pool

at redis.clients.util.Pool.getResource(Pool.java:42) ~<strike>jedis-2.5.2.jar!/:na</strike>

at redis.clients.jedis.JedisPool.getResource(JedisPool.java:84) ~<strike>jedis-2.5.2.jar!/:na</strike>

at redis.clients.jedis.JedisPool.getResource(JedisPool.java:10) ~<strike>jedis-2.5.2.jar!/:na</strike>

at org.springframework.data.redis.connection.jedis.JedisConnectionFactory.fetchJedisConnector(JedisConnectionFactory.java:133) ~<strike>spring-data-redis-1.4.2.RELEASE.jar!/:1.4.2.RELEASE</strike>

... 55 common frames omitted

Caused by: redis.clients.jedis.exceptions.JedisConnectionException: java.net.SocketTimeoutException: connect timed out

at redis.clients.jedis.Connection.connect(Connection.java:150) ~<strike>jedis-2.5.2.jar!/:na</strike>

at redis.clients.jedis.BinaryClient.connect(BinaryClient.java:71) ~<strike>jedis-2.5.2.jar!/:na</strike>

at redis.clients.jedis.BinaryJedis.connect(BinaryJedis.java:1783) ~<strike>jedis-2.5.2.jar!/:na</strike>

at redis.clients.jedis.JedisFactory.makeObject(JedisFactory.java:65) ~<strike>jedis-2.5.2.jar!/:na</strike>

at org.apache.commons.pool2.impl.GenericObjectPool.create(GenericObjectPool.java:836) ~<strike>commons-pool2-2.2.jar!/:2.2</strike>

at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:434) ~<strike>commons-pool2-2.2.jar!/:2.2</strike>

at org.apache.commons.pool2.impl.GenericObjectPool.borrowObject(GenericObjectPool.java:361) ~<strike>commons-pool2-2.2.jar!/:2.2</strike>

at redis.clients.util.Pool.getResource(Pool.java:40) ~<strike>jedis-2.5.2.jar!/:na</strike>

... 58 common frames omitted

Caused by: java.net.SocketTimeoutException: connect timed out

at java.net.PlainSocketImpl.socketConnect(Native Method) ~<strike>na:1.7.0_72</strike>

at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:339) ~<strike>na:1.7.0_72</strike>

at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:200) ~<strike>na:1.7.0_72</strike>

at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:182) ~<strike>na:1.7.0_72</strike>

at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392) ~<strike>na:1.7.0_72</strike>

at java.net.Socket.connect(Socket.java:579) ~<strike>na:1.7.0_72</strike>

at redis.clients.jedis.Connection.connect(Connection.java:144) ~<strike>jedis-2.5.2.jar!/:na</strike>

... 65 common frames omitted

我们仍然不知道造成这种情况的根本原因？

有人能指出正确的方向并帮助我们找出问题的根本原因和解决方案吗？

【问题讨论】：

标签： java spring amazon-web-services redis amazon-elasticache

【解决方案1】：

在某些版本的 Spring 框架中，Spring 不会在事务完成后关闭 redis 连接，因此连接最终会耗尽。如果您使用 setEnableTransactionSupport(true) 初始化 Jedis 模板，这可能会触发错误。将其设置为 false 应该可以解决它。

如果您需要交易，还有其他解决方法。参见本文“Spring Data Redis 中的一个事务陷阱”一节； http://www.javaworld.com/article/3062899/big-data/lightning-fast-nosql-with-spring-data-redis.html

【讨论】：