如何使用 Programmatic API 向 Livy（在 Knox 下）提交工作？答案

【问题标题】：How to submit a Job to Livy (under Knox) with Programmatic API?如何使用 Programmatic API 向 Livy（在 Knox 下）提交工作？
【发布时间】：2021-02-27 09:55:47
【问题描述】：

Livy Docs 的这段代码运行良好：

LivyClient client = new LivyClientBuilder()
    .setURI(new URI("http://sample.ru:8999"))
    .build();
String piJar = "C:\\Users\\e.makrushin\\IdeaProjects\\untitled-maven\\out\\artifacts\\untitled_maven_jar\\untitled-maven.jar";
try {
    String date = "2020.01.07";
    System.err.printf("Job jar file %s uploading to the Spark context...\n", piJar);
    client.uploadJar(new File(piJar)).get();
    System.err.println("Job running at date " + date + " ...");
    double cheques = client.submit(new ChequesJob(date)).get();
    System.out.println("Job finished. " + cheques + " cheques found.");
} finally {
    client.stop(true);
}

DevOps 通过 Knox 路由 Livy 网关：

<service role="LIVYSERVER" name="livy" version="0.1.0">
  <routes>
    <route path="/livy/v1/sessions">
        <rewrite apply="LIVYSERVER/livy/addusername/inbound" to="request.body"/>
    </route>
    <route path="/livy/v1/**?**"/>
    <route path="/livy/v1"/>
    <route path="/livy/v1/"/>
  </routes>
  <dispatch classname="org.apache.knox.gateway.livy.LivyDispatch"/>
</service>

rewrite.xml:
<rules>
  <rule name="LIVYSERVER/livy/user-name">
    <rewrite template="{$username}"/>
  </rule>
  <rule dir="IN" name="LIVYSERVER/livy/root/inbound" pattern="*://*:*/**/livy/v1">
    <rewrite template="{$serviceUrl[LIVYSERVER]}"/>
  </rule>
  <rule dir="IN" name="LIVYSERVER/livy/path/inbound" pattern="*://*:*/**/livy/v1/{path=**}?{**}">
    <rewrite template="{$serviceUrl[LIVYSERVER]}/{path=**}?{**}"/>
  </rule>
  <filter name="LIVYSERVER/livy/addusername/inbound">
    <content type="*/json">
      <apply path="$.proxyUser" rule="LIVYSERVER/livy/user-name"/>
    </content>
  </filter>
</rules>rewrite template="{$serviceUrl[LIVYSERVER]}/"/>
    </rule>
    <rule dir="IN" name="LIVYSERVER/livy/inbound/path" pattern="*://*:*/**/livy/{**}">
        <rewrite template="{$serviceUrl[LIVYSERVER]}/{**}"/>
  </rule>
</rules>

Livy 会话列表现已在 https://knox-gateway:8443/gateway/default/livy/v1/session 上提供

之后，正如预期的那样，此代码停止工作：

Exception in thread "main" java.lang.RuntimeException: javax.net.ssl.SSLHandshakeException: PKIX path building failed: sun.security.provider.certpath.SunCertPathBuilderException: unable to find valid certification path to requested target
    at org.apache.livy.client.http.HttpClient.propagate(HttpClient.java:182)
    at org.apache.livy.client.http.HttpClient.<init>(HttpClient.java:82)
    at org.apache.livy.client.http.HttpClientFactory.createClient(HttpClientFactory.java:37)
    at org.apache.livy.LivyClientBuilder.build(LivyClientBuilder.java:130)
    at Main.main(Main.java:14)

我在 Livy 文档中找不到任何关于使用 Programmatic API 设置授权的内容。我应该在调用代码或配置文件中添加什么才能使此代码再次工作？

【问题讨论】：

标签： java apache-spark livy apache-knox

【解决方案1】：

这似乎是 TLS/SSL 信任存储问题。我相信 Livy 正在尝试调用服务器，并且正在提供它不信任的 SSL 证书。您应该查找 TLS 配置文档，而不是设置授权。也许以下内容会有所帮助。 https://community.cloudera.com/t5/Community-Articles/Enable-SSL-for-Livy-Server/tac-p/246070

【讨论】：