【问题标题】:Phoenix connection throws error message "Failed to find any Kerberos tgt" after running for some timePhoenix 连接运行一段时间后抛出错误消息“Failed to find any Kerberos tgt”
【发布时间】:2025-12-06 21:55:02
【问题描述】:

我正在使用 Phoenix 连接到安全的 HBase。客户端启动时运行良好。我可以连接到 HBase 以查询来自 HBase 的数据。 Phoenix JDBC 连接在连接池中并且不会关闭。但几个小时后,我无法使用相同的连接从 HBase 查询任何内容。我必须重新启动客户端才能使我的应用程序正常工作。错误消息是:

Caused by: org.apache.hadoop.hbase.client.RetriesExhaustedException: Failed after attempts=1, exceptions:
Fri Nov 11 06:24:01 CST 2016, RpcRetryingCaller{globalStartTime=1478867041301, pause=100, retries=1}, java.io.IOException: Could not set up IO Streams to <regionserver>/<ip_address>:60020
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:147) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:95) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callMethod(CoprocessorRpcChannel.java:56) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.phoenix.coprocessor.generated.MetaDataProtos$MetaDataService$Stub.getTable(MetaDataProtos.java:11769) ~[phoenix-core-4.5.1-HBase-1.0.jar:4.5.1-HBase-1.0]
    at org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1301) ~[phoenix-core-4.5.1-HBase-1.0.jar:4.5.1-HBase-1.0]
    at org.apache.phoenix.query.ConnectionQueryServicesImpl$7.call(ConnectionQueryServicesImpl.java:1288) ~[phoenix-core-4.5.1-HBase-1.0.jar:4.5.1-HBase-1.0]
    at org.apache.hadoop.hbase.client.HTable$16.call(HTable.java:1737) ~[hbase-client-1.0.1.jar:1.0.1]
    at java.util.concurrent.FutureTask.run(FutureTask.java:262) ~[na:1.7.0_79]
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145) [na:1.7.0_79]
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615) [na:1.7.0_79]
    ... 1 common frames omitted
Caused by: java.io.IOException: Could not set up IO Streams to <regionserver>/<ip_address>:60020
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:772) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.writeRequest(RpcClientImpl.java:880) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.tracedWriteRequest(RpcClientImpl.java:849) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1173) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:216) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.AbstractRpcClient$BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:300) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.protobuf.generated.ClientProtos$ClientService$BlockingStub.execService(ClientProtos.java:31913) ~[hbase-protocol-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.protobuf.ProtobufUtil.execService(ProtobufUtil.java:1605) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:92) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel$1.call(RegionCoprocessorRpcChannel.java:89) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCaller.java:126) ~[hbase-client-1.0.1.jar:1.0.1]
    ... 10 common frames omitted
Caused by: java.lang.RuntimeException: SASL authentication failed. The most likely cause is missing or invalid credentials. Consider 'kinit'.
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$1.run(RpcClientImpl.java:672) ~[hbase-client-1.0.1.jar:1.0.1]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_79]
    at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_79]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) ~[hadoop-common-2.6.0.jar:na]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.handleSaslConnectionFailure(RpcClientImpl.java:630) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:738) ~[hbase-client-1.0.1.jar:1.0.1]
    ... 20 common frames omitted
    Caused by: javax.security.sasl.SaslException: GSS initiate failed
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:212) ~[na:1.7.0_79]
    at org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:179) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupSaslConnection(RpcClientImpl.java:604) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.access$600(RpcClientImpl.java:153) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:730) ~[hbase-client-1.0.1.jar:1.0.1]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection$2.run(RpcClientImpl.java:727) ~[hbase-client-1.0.1.jar:1.0.1]
    at java.security.AccessController.doPrivileged(Native Method) ~[na:1.7.0_79]
    at javax.security.auth.Subject.doAs(Subject.java:415) ~[na:1.7.0_79]
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628) ~[hadoop-common-2.6.0.jar:na]
    at org.apache.hadoop.hbase.ipc.RpcClientImpl$Connection.setupIOstreams(RpcClientImpl.java:727) ~[hbase-client-1.0.1.jar:1.0.1]
    ... 20 common frames omitted
Caused by: org.ietf.jgss.GSSException: No valid credentials provided (Mechanism level: Failed to find any Kerberos tgt)
    at sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147) ~[na:1.7.0_79]
    at sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:121) ~[na:1.7.0_79]
    at sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187) ~[na:1.7.0_79]
    at sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:223) ~[na:1.7.0_79]
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:212) ~[na:1.7.0_79]
    at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179) ~[na:1.7.0_79]
    at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:193) ~[na:1.7.0_79]
    ... 29 common frames omitted

【问题讨论】:

    标签: java hadoop hbase kerberos phoenix


    【解决方案1】:

    凭据可能已过期。在以下情况下在客户端和区域服务器上执行klist 以查看有效票证:

    • 你可以连接
    • 您无法连接

    【讨论】:

    • 感谢您的评论。有没有办法刷新凭据?我在启动应用程序时没有执行 kinit,我相信凭据是由 Phoenix 客户端本身初始化的。
    • Phoenix 似乎无法自动处理。您必须为其编写应用程序或在他们的 JIRA 跟踪器上提交功能请求。
    【解决方案2】:

    我们的环境:HDP 2.5

    我们应该按照这个在即将到期时自动续订。

    Kerberos 票证在一天后过期。所以在重新启动时它将工作并持续接下来的 24 小时。 我们可以通过这种方法自动更新 HBase Kerberos connection renewal strategy

    【讨论】: