Elasticsearch TLS 错误：空的客户端证书链答案

【问题标题】：Elasticsearch TLS error: Empty client certificate chainElasticsearch TLS 错误：空的客户端证书链
【发布时间】：2021-12-12 10:55:12
【问题描述】：

AKS Kubernetes 版本 1.20.7 ES 版本：7.12.0

我们正在获取“空客户端证书链”（请参阅下面的日志跟踪）
我们启用了 TLS（设置如下）
我们使用的是自签名证书，而不是使用 elasticsearch-certutil 工具
输出证书具有“扩展密钥用法”，可显式启用 clientAuth 和 serverAuth

非常感谢任何帮助

XPack 设置

xpack.security.enabled: true
xpack.security.transport.ssl.enabled: true
xpack.security.transport.ssl.verification_mode: certificate
xpack.security.transport.ssl.keystore.path: /usr/share/Elasticsearch/config/certs/elasticsearch.pfx
xpack.security.transport.ssl.truststore.path: /usr/share/Elasticsearch/config/certs/elasticsearch.pfx
xpack.security.http.ssl.enabled: true
xpack.security.http.ssl.client_authentication: optional
xpack.security.http.ssl.verification_mode: certificate
xpack.security.http.ssl.keystore.path: /usr/share/Elasticsearch/config/certs/elasticsearch.pfx
xpack.security.http.ssl.truststore.path: /usr/share/Elasticsearch/config/certs/elasticsearch.pfx

日志跟踪

ompletionEvent(javax.net.ssl.SSLHandshakeException: Empty client certificate chain)" }
{"type": "server", "timestamp": "2021-10-27T05:32:39,760Z", "level": "TRACE", "component": "o.e.t.n.ESLoggingHandler", "cluster.name": "test-rp-search", "node.name": "test-rp-search-master-0", "message": "[id: 0x7272c902, L:/10.155.12.55:9300 ! R:/10.155.12.34:42246] EXCEPTION: io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
"stack trace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at java.lang.Thread.run(Thread.java:832) [?:?]",
"Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
"at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]",
"at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?]",
"at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1194) ~[?:?]",
"at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[?:?]",
"at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]",
"at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]",
"at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]",
"at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"... 16 more"] }
{"type": "server", "timestamp": "2021-10-27T05:32:39,762Z", "level": "WARN", "component": "o.e.t.TcpTransport", "cluster.name": "test-rp-search", "node.name": "test-rp-search-master-0", "message": "exception caught on transport layer [Netty4TcpChannel{localAddress=/10.155.12.55:9300, remoteAddress=/10.155.12.34:42246, profile=default}], closing connection",
"stacktrace": ["io.netty.handler.codec.DecoderException: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:471) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.channelRead(ByteToMessageDecoder.java:276) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.fireChannelRead(AbstractChannelHandlerContext.java:357) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline$HeadContext.channelRead(DefaultChannelPipeline.java:1410) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:379) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.AbstractChannelHandlerContext.invokeChannelRead(AbstractChannelHandlerContext.java:365) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.DefaultChannelPipeline.fireChannelRead(DefaultChannelPipeline.java:919) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.AbstractNioByteChannel$NioByteUnsafe.read(AbstractNioByteChannel.java:163) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKey(NioEventLoop.java:714) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeysPlain(NioEventLoop.java:615) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.processSelectedKeys(NioEventLoop.java:578) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.channel.nio.NioEventLoop.run(NioEventLoop.java:493) [netty-transport-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:989) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74) [netty-common-4.1.49.Final.jar:4.1.49.Final]",
"at java.lang.Thread.run(Thread.java:832) [?:?]",
"Caused by: javax.net.ssl.SSLHandshakeException: Empty client certificate chain",
"at sun.security.ssl.Alert.createSSLException(Alert.java:131) ~[?:?]",
"at sun.security.ssl.Alert.createSSLException(Alert.java:117) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:356) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:312) ~[?:?]",
"at sun.security.ssl.TransportContext.fatal(TransportContext.java:303) ~[?:?]",
"at sun.security.ssl.CertificateMessage$T13CertificateConsumer.onConsumeCertificate(CertificateMessage.java:1194) ~[?:?]",
"at sun.security.ssl.CertificateMessage$T13CertificateConsumer.consume(CertificateMessage.java:1181) ~[?:?]",
"at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:396) ~[?:?]",
"at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:480) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1267) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask$DelegatedAction.run(SSLEngineImpl.java:1254) ~[?:?]",
"at java.security.AccessController.doPrivileged(AccessController.java:691) ~[?:?]",
"at sun.security.ssl.SSLEngineImpl$DelegatedTask.run(SSLEngineImpl.java:1199) ~[?:?]",
"at io.netty.handler.ssl.SslHandler.runAllDelegatedTasks(SslHandler.java:1542) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.runDelegatedTasks(SslHandler.java:1556) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.unwrap(SslHandler.java:1440) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decodeJdkCompatible(SslHandler.java:1267) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.ssl.SslHandler.decode(SslHandler.java:1314) ~[netty-handler-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.decodeRemovalReentryProtection(ByteToMessageDecoder.java:501) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"at io.netty.handler.codec.ByteToMessageDecoder.callDecode(ByteToMessageDecoder.java:440) ~[netty-codec-4.1.49.Final.jar:4.1.49.Final]",
"... 16 more"] }

【问题讨论】：

标签： elasticsearch tls1.2 azure-aks azure-keyvault elasticsearch-x-pack

【解决方案1】：

我们解决了这个问题，但结果很糟糕。下面是为什么...

这是 Azure KeyVault 的实际问题

准备自签名 PFX 文件（带有完整链）。
PFX 必须受密码保护（尽管 Elasticsearch 示例没有明确说明）才能获得完整的链
通过运行以下命令验证完整链 keytool -list -keystone yourFullChainCert.pfx -storetype pkcs12 -v
使用门户将受密码保护的 PFX（带有完整链）上传到 Azure KeyVault
下载 PFX 文件并再次运行 keytool。您将无法验证链

解决方案，一个讨厌的解决方法（讨厌，因为（需要构建我自己的图像）

形象建设

使用 aes256 alg 加密传输/SSL PFX 私钥
创建私有 Elasticsearch 映像
在容器镜像中，将所有 PEM 文件放在 /usr/share/elasticsearch/config/certs 目录下
- 您的传输 PFX root ca cert PEM，例如 rootCA.cert.pem
- 您的传输 PFX 可选颁发 ca cert PEM，例如颁发 CA.cert.pem
- 您使用 aes256 PEM 加密的传输 PFX 私钥，例如 tls.aes.key.pem
- 您的传输 PFX 证书 PEM，例如 tls.cert.pem
- 您的 ssl PFX 私钥使用 aes256 PEM 加密，ssl.aes.key.pem
- 您的 ssl PFX 证书 PEM，例如 ssl.cert.pem
将 AES256 加密密钥放在 Azure KeyVault 中
使用 CSI 驱动同步密钥，这里是代码 sn -p certysnc.yaml 文件
将 certysnc.yaml 文件放在 Elasticseach helm 图表模板目录中

apiVersion: secrets-store.csi.x-k8s.io/v1alpha1
kind: SecretProviderClass
metadata:
  name: {{ .Values.secretProviderClassName }}
  namespace: {{ .Release. Namespace }} 
spec:
  provider: azure
  secretObjects:
  - secretName: elasticsearch-tls-ssl-cert-key 
    type: Opaque
    data:
    - objectName: es-tls-ssl-cert-key
      key: xpack.security.transport.ssl.secure_key_passphrase
    - objectName: es-tls-ssl-cert-key
      key: xpack.security.http.ssl.secure_key_passphrase
  - secretName: elastic-credentials
    type: Opaque
    data:
    - objectName: elastic-username
      key: username
    - objectName: elastic-user-password
      key: password
  parameters:
    usePodIdentity: "true"
    userAssignedIdentityID: {{ .Values.userAssignedIdentityID }}
    keyvaultName: {{ .Values.keyvaultName }}
    cloudName: "AzurePublicCloud"
    objects:  |
      array:
        - |
          objectName: elastic-user-password
          objectType: secret
          objectVersion: ""
        - |
          objectName: elastic-username
          objectType: secret
          objectVersion: ""
        - |
          objectName: es-tls-ssl-cert-key
          objectType: secret
          objectVersion: ""
    tenantId: {{ .Values.tenantId }}
    resourceGroup: {{ .Values.resourceGroup }}
    subscriptionId: {{ .Values.subscriptionId }}

【讨论】：