用于结构化流的 Elasticsearch spark 连接

Question

我正在尝试从我的 spark 程序连接到 elasticsearch。 我的弹性搜索主机是 https 并且没有找到它的连接属性。 我们使用的是 spark structred streaming Java API，连接细节如下，

        SparkSession spark = SparkSession.builder()
                .config(ConfigurationOptions.ES_NET_HTTP_AUTH_USER, "username")
                .config(ConfigurationOptions.ES_NET_HTTP_AUTH_PASS, "password")
                 .config(ConfigurationOptions.ES_NODES, "my_host_url")
                 .config(ConfigurationOptions.ES_PORT, "9200")
.config(ConfigurationOptions.ES_NET_SSL_TRUST_STORE_LOCATION,"C:\\certs\\elastic\\truststore.jks")
.config(ConfigurationOptions.ES_NET_SSL_TRUST_STORE_PASS,"my_password") .config(ConfigurationOptions.ES_NET_SSL_KEYSTORE_TYPE,"jks")
                .master("local[2]")
                .appName("spark_elastic").getOrCreate();
        spark.conf().set("spark.sql.shuffle.partitions",2);
        spark.conf().set("spark.default.parallelism",2);

我收到以下错误

19/07/01 12:26:00 INFO HttpMethodDirector: I/O exception (org.apache.commons.httpclient.NoHttpResponseException) caught when processing request: The server 10.xx.xxx.xxx failed to respond
19/07/01 12:26:00 INFO HttpMethodDirector: Retrying request
19/07/01 12:26:00 ERROR NetworkClient: Node [10.xx.xxx.xxx:9200] failed (The server 10.xx.xxx.xxx failed to respond); no other nodes left - aborting...
19/07/01 12:26:00 ERROR StpMain: Error
org.elasticsearch.hadoop.EsHadoopIllegalArgumentException: Cannot detect ES version - typically this happens if the network/Elasticsearch cluster is not accessible or when targeting a WAN/Cloud instance without the proper setting 'es.nodes.wan.only'
    at org.elasticsearch.hadoop.rest.InitializationUtils.discoverClusterInfo(InitializationUtils.java:344)

可能是因为它尝试通过 http 协议启动连接，但在我的情况下，我需要 https 连接，但不知道如何配置它

Answer 1

由于 spark 无法找到信任库文件而发生错误。看来我们需要为要接受的路径添加“file:\\”。