使用 spark-on-k8s 时如何向驱动程序 pod 注入环境变量？答案

【问题标题】：How to inject evnironment variables to driver pod when using spark-on-k8s?使用 spark-on-k8s 时如何向驱动程序 pod 注入环境变量？
【发布时间】：2020-07-16 21:30:09
【问题描述】：

我正在 k8s 上使用 GCP spark 编写 Kubernetes Spark 应用程序。

目前，我无法将环境变量注入我的容器。

我正在关注文档here

清单：

apiVersion: "sparkoperator.k8s.io/v1beta2"
kind: SparkApplication
metadata:
  name: spark-search-indexer
  namespace: spark-operator
spec:
  type: Scala
  mode: cluster
  image: "gcr.io/spark-operator/spark:v2.4.5"
  imagePullPolicy: Always
  mainClass: com.quid.indexer.news.jobs.ESIndexingJob
  mainApplicationFile: "https://lala.com/baba-0.0.43.jar"
  arguments:
    - "--esSink"
    - "http://something:9200/mo-sn-{yyyy-MM}-v0.0.43/searchable-article"
    - "-streaming"
    - "--kafkaTopics"
    - "annotated_blogs,annotated_ln_news,annotated_news"
    - "--kafkaBrokers"
    - "10.1.1.1:9092"
  sparkVersion: "2.4.5"
  restartPolicy:
    type: Never
  volumes:
    - name: "test-volume"
      hostPath:
        path: "/tmp"
        type: Directory
  driver:
    cores: 1
    coreLimit: "1200m"
    memory: "512m"
    env:
      - name: "DEMOGRAPHICS_ES_URI"
        value: "somevalue"
    labels:
      version: 2.4.5
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"
  executor:
    cores: 1
    instances: 1
    memory: "512m"
    env:
      - name: "DEMOGRAPHICS_ES_URI"
        value: "somevalue"
    labels:
      version: 2.4.5
    volumeMounts:
      - name: "test-volume"
        mountPath: "/tmp"

在 pod 中设置的环境变量：

Environment:
      SPARK_DRIVER_BIND_ADDRESS:   (v1:status.podIP)
      SPARK_LOCAL_DIRS:           /var/data/spark-1ed8539d-b157-4fab-9aa6-daff5789bfb5
      SPARK_CONF_DIR:             /opt/spark/conf

【问题讨论】：

标签： apache-spark kubernetes spark-operator

【解决方案1】：

原来使用这个必须启用webhooks（如何在quick-start guide here中设置）

另一种方法是使用envVars

例子：

   spec:
       executor:
           envVars:
               DEMOGRAPHICS_ES_URI: "somevalue"

参考：https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/issues/978

【讨论】：