关于hbase，zookeeper.MetaTableLocator：hbase:meta验证失败，.NotServingRegionException答案

【问题标题】：About hbase, zookeeper.MetaTableLocator: Failed verification of hbase:meta, .NotServingRegionException关于hbase，zookeeper.MetaTableLocator：hbase:meta验证失败，.NotServingRegionException
【发布时间】：2018-08-15 19:28:56
【问题描述】：

我使用 hadoop2.7.3，Hbase 1.2.3，zookeeper 3.4.9。

每次我停止 Hbase 并重新启动它时，它都会抛出异常：

  [hadoop01:16000.activeMasterManager] zookeeper.MetaTableLocator: Failed        verification of hbase:meta,,1 at address=hadoop05,16020,1478663588885, exception=org.apache.hadoop.hbase.NotServingRegionException: Region hbase:meta,,1 is not online on hadoop05,16020,1478664215143
at org.apache.hadoop.hbase.regionserver.HRegionServer.getRegionByEncodedName(HRegionServer.java:2922)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegion(RSRpcServices.java:1059)
at org.apache.hadoop.hbase.regionserver.RSRpcServices.getRegionInfo(RSRpcServices.java:1339)
at org.apache.hadoop.hbase.protobuf.generated.AdminProtos$AdminService$2.callBlockingMethod(AdminProtos.java:22233)
at org.apache.hadoop.hbase.ipc.RpcServer.call(RpcServer.java:2180)
at org.apache.hadoop.hbase.ipc.CallRunner.run(CallRunner.java:112)
at org.apache.hadoop.hbase.ipc.RpcExecutor.consumerLoop(RpcExecutor.java:133)
at org.apache.hadoop.hbase.ipc.RpcExecutor$1.run(RpcExecutor.java:108)
at java.lang.Thread.run(Thread.java:745)

我已经搜索了异常，但是原因（即hdfs saftemode或服务器的大约日期）都不满足我的情况。以及zookeeper的日志：

2016-11-09 03:34:49,900 [myid:2] - INFO  [ProcessThread(sid:2    cport:-1)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x358483686110002 type:create cxid:0x1 zxid:0x100000040 txntype:-1 reqpath:n/a Error Path:/hbase/replication Error:KeeperErrorCode = NoNode for /hbase/replication
2016-11-09 03:34:50,810 [myid:2] - INFO  [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x15848364a850001 type:create cxid:0x16 zxid:0x100000048 txntype:-1 reqpath:n/a Error Path:/hbase/replication/peers Error:KeeperErrorCode = NodeExists for /hbase/replication/peers
2016-11-09 03:34:55,378 [myid:2] - INFO  [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x15848364a850000 type:delete cxid:0x5b zxid:0x100000049 txntype:-1 reqpath:n/a Error Path:/hbase/meta-region-server Error:KeeperErrorCode = NoNode for /hbase/meta-region-server
2016-11-09 03:34:56,677 [myid:2] - INFO  [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x15848364a850001 type:setData cxid:0x2d zxid:0x10000004e txntype:-1 reqpath:n/a Error Path:/hbase/meta-region-server Error:KeeperErrorCode = NoNode for /hbase/meta-region-server
2016-11-09 03:34:57,556 [myid:2] - INFO  [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x15848364a850000 type:create cxid:0x30c zxid:0x100000052 txntype:-1 reqpath:n/a Error Path:/hbase/table-lock/hbase:namespace Error:KeeperErrorCode = NoNode for /hbase/table-lock/hbase:namespace
2016-11-09 03:35:00,443 [myid:2] - INFO  [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x15848364a850000 type:create cxid:0x330 zxid:0x100000062 txntype:-1 reqpath:n/a Error Path:/hbase/namespace/default Error:KeeperErrorCode = NodeExists for /hbase/namespace/default
2016-11-09 03:35:00,510 [myid:2] - INFO  [ProcessThread(sid:2 cport:-1)::PrepRequestProcessor@649] - Got user-level KeeperException when processing sessionid:0x15848364a850000 type:create cxid:0x333 zxid:0x100000064 txntype:-1 reqpath:n/a Error Path:/hbase/namespace/hbase Error:KeeperErrorCode = NodeExists for /hbase/namespace/hbase

hbase-sie.xml

动物园.cfg

【问题讨论】：

/var/log/zookeeper/上是否有错误日志
zookeeper的leader log就像我在这里发的一样，有帮助吗？谢谢@NirmalRam
你能分享你的 hbase-site.xml
当然，我会发布的
检查 zookeeper 是否正常运行 -> $ZOOKEEPER_HOME/bin/zkServer.sh 状态

标签： hadoop hbase apache-zookeeper

【解决方案1】：

我也遇到过这个问题。首先你可以尝试修复数据

bin/hbase 清理

用法：hbase clean (--cleanZk|--cleanHdfs|--cleanAll) 选项：
--cleanZk   cleans hbase related data from zookeeper.
--cleanHdfs cleans hbase related data from hdfs.
--cleanAll  cleans hbase related data from both zookeeper and hdfs.

另外，您可以删除整个动物园管理员的数据，这可能是由 hbase-site.xml 或 zoo.cfg 配置的，甚至是默认目录，你可以检查你的配置文件

【讨论】：