如何在 google dataproc 上运行 spark 3.2.0？

Answer 1

可以通过以下方式实现：

1. 使用包含 pyspark 3.2 作为包的环境 (your_sample_env) 创建一个 dataproc 集群
2. 通过添加修改/usr/lib/spark/conf/spark-env.sh

   SPARK_HOME="/opt/conda/miniconda3/envs/your_sample_env/lib/python/site-packages/pyspark"
   SPARK_CONF="/usr/lib/spark/conf"
   

   最后

   3. 修改/usr/lib/spark/conf/spark-defaults.conf，注释掉以下配置

   spark.yarn.jars=local:/usr/lib/spark/jars/*
   spark.yarn.unmanagedAM.enabled=true
   
   

   现在，您的 spark 作业将使用 pyspark 3.2

Answer 2

Dataproc Serverless for Spark 刚刚发布，支持 Spark 3.2.0：https://cloud.google.com/dataproc-serverless

Answer 3

@milominderbinder 的答案在笔记本中对我不起作用。我使用了谷歌给出的pip install script，并在 main.js 中添加了以下代码。

function main() {
  install_pip
  pip install pyspark==3.2.0
  sed -i '4d;27d' /usr/lib/spark/conf/spark-defaults.conf
  cat << EOF | tee -a /etc/profile.d/custom_env.sh /etc/*bashrc >/dev/null
export SPARK_HOME=/opt/conda/miniconda3/lib/python3.8/site-packages/pyspark/
export SPARK_CONF=/usr/lib/spark/conf
EOF
  sed -i 's/\/usr\/lib\/spark/\/opt\/conda\/miniconda3\/lib\/python3.8\/site-packages\/pyspark\//g' /opt/conda/miniconda3/share/jupyter/kernels/python3/kernel.json

  if [[ -z "${PACKAGES}" ]]; then
    echo "WARNING: requirements empty"
    exit 0
  fi
  run_with_retry pip install --upgrade ${PACKAGES}

}

这使得它可以在带有 Python3 内核的 jupyterlab 中工作。

Answer 4

快速而肮脏的脚本，在 Dataproc 映像 2.0 的初始化操作中完成：

#!/usr/bin/env bash

spark_version="3.3.0"

cd /opt

if [[ ! -L /opt/spark ]]; then
    archive_filename="spark-${spark_version}-bin-without-hadoop.tgz"
    rm -rf spark*
    wget "https://dlcdn.apache.org/spark/spark-${spark_version}/${archive_filename}"
    tar xvfz "${archive_filename}"
    rm -f spark*.tgz*
    ln -s spark-* spark
fi

# This will cause spark to fallback to defaults. There's probably a better way.
sed -i '/^spark\.yarn\.jars/d' /usr/lib/spark/conf/spark-defaults.conf

# By default, Dataproc uses Hive. For unknown reasons, this doesn't work, so we replace it with 'in-memory'.
sed -i '/^spark\.sql\.catalogImplementation/d' /usr/lib/spark/conf/spark-defaults.conf
echo "spark.sql.catalogImplementation=in-memory" >>/usr/lib/spark/conf/spark-defaults.conf

{
    # shellcheck disable=SC2016
    echo 'export PATH=/opt/spark/bin:$PATH'
    echo "export SPARK_CONF_DIR=/usr/lib/spark/conf"
    echo "export SPARK_HOME=/opt/spark"
    # shellcheck disable=SC2016
    echo 'export PYTHONPATH=$(ZIPS=("$SPARK_HOME"/python/lib/*.zip); IFS=:; echo "${ZIPS[*]}"):$PYTHONPATH'
    # shellcheck disable=SC2016
    echo 'export SPARK_DIST_CLASSPATH=$(hadoop classpath)'
} >/etc/profile.d/zzzzzzzzzzzzz-custom-spark.sh
chmod +x /etc/profile.d/zzzzzzzzzzzzz-custom-spark.sh