hBase - 服务器有太多的连接

Question

我使用 hbase 存储来自爬虫的数据。这些数据从 hbase 索引到 Solr。但我对 hBase 有疑问。因为我现在有更大的数据库，所以每次我尝试从 hbase 索引数据时，都会发生下面显示的错误。这是日志文件hadoop.log的一部分

有类似的原因：这可能表明服务器有太多连接（默认值为 30）。 我在 SO 上发现这可能是 maxClientConnections 属性的原因所以我将此值设置为 0，但此解决方案没有帮助。

2013-11-11 15:30:22,638 ERROR zookeeper.ClientCnxn - Error while calling watcher 
java.lang.NullPointerException
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.resetZooKeeperTrackers(HConnectionManager.java:315)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.abort(HConnectionManager.java:1340)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.connectionEvent(ZooKeeperWatcher.java:343)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.process(ZooKeeperWatcher.java:261)
at org.apache.zookeeper.ClientCnxn$EventThread.run(ClientCnxn.java:488)
2013-11-11 15:30:27,925 WARN  mapred.FileOutputCommitter - Output path is null in cleanup
2013-11-11 15:30:27,925 WARN  mapred.LocalJobRunner - job_local1497313370_0001
org.apache.gora.util.GoraException: java.lang.RuntimeException: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:167)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:118)
at org.apache.gora.mapreduce.GoraOutputFormat.getRecordWriter(GoraOutputFormat.java:88)
at org.apache.hadoop.mapred.ReduceTask$NewTrackingRecordWriter.<init>(ReduceTask.java:568)
at org.apache.hadoop.mapred.ReduceTask.runNewReducer(ReduceTask.java:637)
at org.apache.hadoop.mapred.ReduceTask.run(ReduceTask.java:418)
at org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:398)
Caused by: java.lang.RuntimeException: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:127)
at org.apache.gora.store.DataStoreFactory.initializeDataStore(DataStoreFactory.java:102)
at org.apache.gora.store.DataStoreFactory.createDataStore(DataStoreFactory.java:161)
... 6 more
Caused by: org.apache.hadoop.hbase.ZooKeeperConnectionException: HBase is able to connect to ZooKeeper but the connection closes immediately. This could be a sign that the server has too many connections (30 is the default). Consider inspecting your ZK server logs for that error and then make sure you are reusing HBaseConfiguration as often as you can. See HTable's javadoc for more information.
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:155)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.getZooKeeperWatcher(HConnectionManager.java:1002)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.setupZookeeperTrackers(HConnectionManager.java:304)
at org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation.<init>(HConnectionManager.java:295)
at org.apache.hadoop.hbase.client.HConnectionManager.getConnection(HConnectionManager.java:157)
at org.apache.hadoop.hbase.client.HBaseAdmin.<init>(HBaseAdmin.java:90)
at org.apache.gora.hbase.store.HBaseStore.initialize(HBaseStore.java:109)
... 8 more
Caused by: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase
at org.apache.zookeeper.KeeperException.create(KeeperException.java:90)
at org.apache.zookeeper.KeeperException.create(KeeperException.java:42)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:809)
at org.apache.zookeeper.ZooKeeper.exists(ZooKeeper.java:837)
at org.apache.hadoop.hbase.zookeeper.ZKUtil.createAndFailSilent(ZKUtil.java:903)
at org.apache.hadoop.hbase.zookeeper.ZooKeeperWatcher.<init>(ZooKeeperWatcher.java:133)
... 14 more

请问您知道问题出在哪里吗？或者您知道如何确保我没有同时打开多个 HTable 实例吗？

Answer 1

如果有人对我如何解决这个问题感兴趣，我在 hBase 邮件列表中创建了一个线程并在那里解决了它http://apache-hbase.679495.n3.nabble.com/hBase-the-server-has-too-many-connections-maxClientConn-property-set-to-0-does-not-help-td4052728.html#a4052873

我不知道究竟是什么解决了这个问题，但我尝试将这些东西结合起来：

1) 属性 maxClientConnections 设置为：

<property>
   <name>hbase.zookeeper.property.maxClientCnxns</name>
   <value>0</value>
</property>

2) 超时时间延长

<property>
  <name>zookeeper.session.timeout</name>
  <value>1200000</value>
</property>

<property>
  <name>hbase.zookeeper.property.tickTime</name>
  <value>6000</value>
</property>

3)https://wiki.apache.org/nutch/ErrorMessagesInNutch2#Nutch_2.1_.2B-_HBase_0.90.4_cluster_settings_-WARN_zookeeper.ClientCnxn_-_Session_0x0_for_server_node1.xxxxxx.com.2Fxxx.xxx.xxx.xxx:2181.2C_unexpected_error.2C_closing_socket_connection_and_attempting_reconnect_java.io.IOException:_Connection_reset_by_peer

然后我杀死了所有 java 进程，启动和停止 hBase，然后运行 ​​rolling-restart.sh，现在一切正常。

Answer 2

HTable 不打开新连接，HTable 驻留在 Connection 对象上。如此多的 HTable 共享相同的连接。但是您必须确保每个节点不要打开太多连接。

ZK 有一个设置来设置一个 IP 地址的最大连接数，以防止连接轰炸。默认值为 30。您可以增加但不能增加太多。通常从同一主机到 ZK 的并发连接不应该太多。否则使用netstat检查哪个进程打开了太多的ZK连接，该进程可能有连接泄漏。

参考我的其他相关回答：Error HBASE-ZOOKEEPER : Too many connections