【发布时间】：2016-04-04 15:07:38
【问题描述】：
过去几天，Debian（Google Compute Engine）服务器每天都面临套接字不可用的问题，需要重新启动 tomcat。
Dec 30, 2015 1:16:53 PM org.apache.tomcat.util.net.JIoEndpoint$Acceptor run
SEVERE: Socket accept failed
java.net.SocketException: Too many open files
at java.net.PlainSocketImpl.socketAccept(Native Method)
at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
at java.net.ServerSocket.implAccept(ServerSocket.java:530)
at java.net.ServerSocket.accept(ServerSocket.java:498)
at org.apache.tomcat.util.net.DefaultServerSocketFactory.acceptSocket(DefaultServerSocketFactory.java:60)
at org.apache.tomcat.util.net.JIoEndpoint$Acceptor.run(JIoEndpoint.java:220)
at java.lang.Thread.run(Thread.java:745)
在使用netstat -ap检查时发现GOOGLE :(正在淹没服务器。
tcp 1 0 backend-tomcat:53976 wk-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:42251 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45929 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:42159 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:55348 wo-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:44973 wa-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:42148 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45729 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45721 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:42146 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:36557 wq-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45723 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45915 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45295 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:53819 wk-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 0 0 backend-tomcat:45968 wn-in-f95.1e100.n:https ESTABLISHED 5081/java
tcp 1 0 backend-tomcat:41737 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:55132 wm-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:53969 wk-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:40883 wj-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:45734 wn-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:40889 wj-in-f95.1e100.n:https CLOSE_WAIT 5081/java
tcp 1 0 backend-tomcat:41834 wl-in-f95.1e100.n:https CLOSE_WAIT 5081/java
用lsof -p 5081检查了进程，下面有很多行
java 5081 tomcat7 1181u IPv4 1959257 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48570->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1182u IPv4 1959420 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48575->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1183u IPv4 1959526 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48577->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1184u IPv4 1959676 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48583->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1185u IPv4 1960099 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48591->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1186u IPv4 1959857 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48589->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1187u IPv4 1960209 0t0 TCP backend-tomcat.c.stutzenappointments.internal:48596->wb-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1188u IPv4 1960352 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49524->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1189u IPv4 1960555 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49526->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1190u IPv4 1960721 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49531->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1191u IPv4 1960962 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49537->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1192u IPv4 1961151 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49539->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1193u IPv4 1961318 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49544->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1194u IPv4 1961557 0t0 TCP backend-tomcat.c.stutzenappointments.internal:49550->wa-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1195u IPv4 1961828 0t0 TCP backend-tomcat.c.stutzenappointments.internal:46380->wl-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1196u IPv4 1962155 0t0 TCP backend-tomcat.c.stutzenappointments.internal:46386->wl-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1197u IPv4 1962356 0t0 TCP backend-tomcat.c.stutzenappointments.internal:46392->wl-in-f95.1e100.net:https (CLOSE_WAIT)
java 5081 tomcat7 1198u IPv4 1962461 0t0 TCP backend-tomcat.c.stutzenappointments.internal:46393->wl-in-f95.1e100.net:https (CLOSE_WAIT)
从 ulimit 发现打开文件的限制设置为最大值
open files (-n) 65536
是因为 GCE 的 Cloud Debugger 功能吗？有没有办法要求谷歌不要让服务器每天流血？
【问题讨论】：
-
netstat 输出似乎是到 Google 的传出连接。您的应用程序是否使用任何 Google API？考虑安装/使用 lsof 来查看您的应用程序使用的其他文件。 tomcat 或您的应用程序是否有可能泄漏文件描述符？如果不是泄漏，而是正常运行，请考虑增加 tomcat 可用的文件描述符（ulimit）。
-
如果它是传出连接，那么它会很糟糕，因为使用谷歌的计算引擎，他们是否跟踪每一个活动？有什么办法可以阻止它？直到现在还没有使用 Google API。
-
您在帖子中提到了云调试器；你启用了吗？ Cloud Debugger 确实需要对 Google 进行 API 调用才能运行。如果您已启用 Cloud Debugger，您可能需要尝试禁用它并查看问题是否消失。
-
是的，现在已被
sudo emacs /etc/default/tomcat7删除#STUTZEN below line is commented to check if Cloud Debugger is causing the socket leak issue #JAVA_OPTS="${JAVA_OPTS} $( /opt/cdbg/format_env_gce.sh --app_class_path=/var/lib/tomcat7/webapps/ROOT --app_class=/usr/share/tomcat7-examples/examples --agent_logs_dir=/var/log/tomcat7 )"让我们等一天检查一下，到目前为止没有与该域的连接 -
3天过去了，tomcat去掉Cloud Debugger后稳定了(:|
标签： tomcat debian google-compute-engine google-cloud-platform google-cloud-debugger