在 spark worker 类路径中自定义 JAR 的最佳方法

Question

我正在使用 Spark 中的 ETL 管道，我发现推送发布是时间/带宽密集型的。我的发布脚本（伪代码）：

sbt assembly
openstack object create spark target/scala-2.11/etl-$VERSION-super.jar
spark-submit \
    --class comapplications.WindowsETLElastic \
    --master spark://spark-submit.cloud \
    --deploy-mode cluster \
    --verbose \
    --conf "spark.executor.memory=16g" \
    "$JAR_URL"

它可以工作，但可能需要 4 多分钟来组装和一分钟来推动。我的 build.sbt：

name := "secmon_etl"

version := "1.2"

scalaVersion := "2.11.8"

exportJars := true

assemblyJarName in assembly := s"${name.value}-${version.value}-super.jar"

libraryDependencies ++= Seq (
  "org.apache.spark" %% "spark-core" % "2.1.0" % "provided",
  "org.apache.spark" %% "spark-streaming" % "2.1.0" % "provided",
  "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.1.0",
  "io.spray" %%  "spray-json" % "1.3.3",
//  "commons-net" % "commons-net" % "3.5",
//  "org.apache.httpcomponents" % "httpclient" % "4.5.2",
  "org.elasticsearch" % "elasticsearch-spark-20_2.11" % "5.3.1"
)

assemblyMergeStrategy in assembly <<= (assemblyMergeStrategy in assembly) {
  (old) => {
    case PathList("META-INF", xs @ _*) => MergeStrategy.discard
    case x => MergeStrategy.first
  }
}

问题似乎在于 elasticsearch-spark-20_2.11 的大小。它为我的 uberjar 增加了大约 90MB。我很乐意将其转换为对 spark 主机的 provided 依赖项，从而无需打包。问题是，最好的方法是什么？我应该手动复制 jars 还是有一种万无一失的方法来指定依赖项并让工具解析所有传递依赖项？

Answer 1

我的 spark 作业正在运行，而且现在运行得更快了。我跑了

sbt assemblyPackageDependency

它生成了一个巨大的 jar（110MB！），很容易放在 spark 工作目录 'jars' 文件夹中，所以现在我的 spark 集群的 Dockerfile 如下所示：

FROM openjdk:8-jre

ENV SPARK_VERSION 2.1.0
ENV HADOOP_VERSION hadoop2.7
ENV SPARK_MASTER_OPTS="-Djava.net.preferIPv4Stack=true"

RUN apt-get update && apt-get install -y python

RUN curl -sSLO http://mirrors.ocf.berkeley.edu/apache/spark/spark-$SPARK_VERSION/spark-$SPARK_VERSION-bin-$HADOOP_VERSION.tgz && tar xzfC /spark-$SPARK_VERSION-bin-$HADOOP_VERSION.tgz /usr/share && rm /spark-$SPARK_VERSION-bin-$HADOOP_VERSION.tgz

# master or worker's webui port, 
EXPOSE 8080
# master's rest api port
EXPOSE 7077

ADD deps.jar /usr/share/spark-$SPARK_VERSION-bin-$HADOOP_VERSION/jars/

WORKDIR /usr/share/spark-$SPARK_VERSION-bin-$HADOOP_VERSION

部署该配置后，我更改了 build.sbt，因此 kafka-streaming/elasticsearch-spark jar 和依赖项标记为 provided：

name := "secmon_etl"

version := "1.2"

scalaVersion := "2.11.8"

exportJars := true

assemblyJarName in assembly := s"${name.value}-${version.value}-super.jar"

libraryDependencies ++= Seq (
  "org.apache.spark" %% "spark-core" % "2.1.0" % "provided",
  "org.apache.spark" %% "spark-streaming" % "2.1.0" % "provided",

  "org.apache.spark" %% "spark-streaming-kafka-0-10" % "2.1.0" % "provided",
  "io.spray" %%  "spray-json" % "1.3.3" % "provided",
  "org.elasticsearch" % "elasticsearch-spark-20_2.11" % "5.3.1" % "provided"
)

assemblyMergeStrategy in assembly <<= (assemblyMergeStrategy in assembly) {
  (old) => {
    case PathList("META-INF", xs @ _*) => MergeStrategy.discard
    case x => MergeStrategy.first
  }
}

现在我的部署在 20 秒内完成！