Spark Submit on Kubernetes 退出代码答案

【问题标题】：Spark Submit on Kuberentes exit codeSpark Submit on Kubernetes 退出代码
【发布时间】：2022-10-09 03:53:22
【问题描述】：

如何在运行 spark-submit 时以编程方式检查 spark 作业是成功还是失败。通常使用 unix 退出代码。

 phase: Failed
 container status:
     container name: spark-kubernetes-driver
     container image: <regstry>/spark-py:spark3.2.1
     container state: terminated
     container started at: 2022-03-25T19:10:51Z
     container finished at: 2022-03-25T19:10:57Z
     exit code: 1
     termination reason: Error

2022-03-25 15:10:58,457 INFO submit.LoggingPodStatusWatcherImpl: Application Postgres-Minio-Kubernetes.py with submission ID spark:postgres-minio-kubernetes-py-b70d3f7fc27829ec-driver finished
2022-03-25 15:10:58,465 INFO util.ShutdownHookManager: Shutdown hook called
2022-03-25 15:10:58,466 INFO util.ShutdownHookManager: Deleting directory /tmp/spark-3321e67c-73d5-422d-a26d-642a0235cf23

进程失败，当我通过 echo $ 在 unix 中获取退出代码时？它返回一个零错误代码！

$ echo $?
0

生成的 pod 也是随机方式。除了使用 sparkonk8operator 之外，处理 spark-submit 的方式是什么？

【问题讨论】：

标签： kubernetes spark-submit

【解决方案1】：

如果您使用的是 bash，一种在输出上 grep 的方法。您可能必须在 stderr or stdout 上使用 grep，具体取决于日志输出的发送位置。

像这样的东西：

OUTPUT=`spark-submit ...`
if echo "$OUTPUT" | grep -q "exit code: 1"; then
    exit 1
fi

【讨论】：

【解决方案2】：

除了@Rico 提到的事情，我还考虑了cluster 和client 的部署模式，将$SPARK_HOME/bin 目录中的spark-submit shell 文件更改如下。

#!/usr/bin/env bash

#
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
#
#    http://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
#

if [ -z "${SPARK_HOME}" ]; then
  source "$(dirname "$0")"/find-spark-home
fi

# disable randomized hash for string in Python 3.3+
export PYTHONHASHSEED=0


# check deployment mode.
if echo "$@" | grep -q "--deploy-mode cluster";
then
    echo "cluster mode..";
    # temp log file for spark job.
    export TMP_LOG="/tmp/spark-job-log-$(date '+%Y-%m-%d-%H-%M-%S').log";
    exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@" |& tee ${TMP_LOG};
    # when exit code 1 and exception are contained in spark log, then return exit 1.
    if cat ${TMP_LOG} | grep -q "exit code: 1";
    then
      echo "exit code: 1";
      rm -rf ${TMP_LOG};
      exit 1;
    else
      echo "job succeeded.";
      rm -rf ${TMP_LOG};
      exit 0;
    fi
else
    echo "client mode..";
    exec "${SPARK_HOME}"/bin/spark-class org.apache.spark.deploy.SparkSubmit "$@"
fi

然后，我构建并推送了我的 spark docker 映像。

【讨论】：