如何使用 GitLab CI-CD 变量参数化 pod_template_file（yaml 文件）？答案

【问题标题】：How to parameterized pod_template_file (yaml file) using GitLab CI-CD variables?如何使用 GitLab CI-CD 变量参数化 pod_template_file（yaml 文件）？
【发布时间】：2021-09-11 17:15:16
【问题描述】：

需要通过 CI-CD 变量更改 AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER & AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY 的值。以上值存在于airflow_template.yaml 文件中。我尝试替换 CI-CD 变量，但它不起作用。如果有更好的参数化方法。请告诉我。

#My project folder structure looks like below:
dataops
    -- docker
        -- base
            -- airflow.cfg
            -- **airflow_template.yaml** 
            -- Dockerfile
        -- dag-image
             --Dockerfile
    -- helm
        --Chart.yaml
        --values.yaml
        --templates
            --deployment.yaml
            --svc.yaml

**airflow_template.yaml**
apiVersion: v1
kind: Pod
metadata:
  labels: {}
spec:
  containers:
  - args: []
    command: []
    env:
    - name: AIRFLOW__KUBERNETES__WORKER_CONTAINER_REPOSITORY
      value: $DEV_AIRFLOW_CONTAINER_REPO
    - name: AIRFLOW__LOGGING__REMOTE_BASE_LOG_FOLDER
      value: $DEV_AIRFLOW_LOG_FOLDER
    envFrom: []
    imagePullPolicy: Always
    name: base
    ports: []
    volumeMounts:
    - mountPath: /usr/local/airflow/logs
      name: airflow-logs
  hostNetwork: false
  imagePullSecrets: []
  initContainers: []
  nodeSelector: {}
  restartPolicy: Never
  securityContext:
    runAsUser: 1000
  serviceAccountName: default
  volumes:
  - emptyDir: {}
    name: airflow-logs

  


gitlab-ci.yml
stages:
  - build_and_upload
  - deploy_to_dev
  - tag_prod
  - deploy_to_prod

build_and_upload:
  stage: build_and_upload
  image: docker:latest
  variables:
DOCKER_DRIVER: overlay2
DOCKER_TLS_CERTDIR: "/certs"
  services:
- docker:19.03.14-dind
  script:
- echo $DEV_CREDENTIALS > service_account.json && cat service_account.json | docker login -u _json_key --password-stdin https://gcr.io
- echo "as- $DEV_AIRFLOW_LOG_FOLDER"
- export DEV_AIRFLOW_LOG_FOLDER="${DEV_AIRFLOW_LOG_FOLDER}"
- mkdir -p edfi/operation
- cp -r airflow_dags/ dataops/docker/dag-image/airflow_dags/
- cd dataops/docker/dag-image/
- docker build -t "$DEV_DAGS_IMAGE:$CI_COMMIT_SHORT_SHA" --build-arg COMMIT_HASH=$CI_COMMIT_SHORT_SHA .
- docker tag $DEV_DAGS_IMAGE:$CI_COMMIT_SHORT_SHA $DEV_DAGS_IMAGE:latest
- docker push $DEV_DAGS_IMAGE:$CI_COMMIT_SHORT_SHA
- docker push $DEV_DAGS_IMAGE:latest
  only:
refs:
  - develop
#    variables:
#      - $CI_COMMIT_MESSAGE =~ /penguin/

deploy_to_dev:
  stage: deploy_to_dev
  image: $CI_REGISTRY_IMAGE:kube-image
  script:
- echo $DEV_CREDENTIALS > service_account.json && cat service_account.json | docker login -u _json_key --password-stdin https://gcr.io
- echo "as- $DEV_AIRFLOW_LOG_FOLDER"
- export DEV_AIRFLOW_CONTAINER_REPO="${DEV_AIRFLOW_CONTAINER_REPO}"
- export DEV_AIRFLOW_LOG_FOLDER="${DEV_AIRFLOW_LOG_FOLDER}"
- gcloud auth activate-service-account $DEV_SERVICE_ACCOUNT --key-file=./service_account.json --project=$DEV_PROJECT_NAME
- gcloud container clusters get-credentials $DEV_GKE_CLUSTER --region $REGION
- echo $DEV_DB_CONN > dataops/helm/airflow-loadbalancer/files/secrets/airflow/AIRFLOW__CORE__SQL_ALCHEMY_CONN
- cd dataops/helm/
- helm upgrade airflow-dev airflow-loadbalancer/ --install --atomic --set dags_image.tag=$CI_COMMIT_SHORT_SHA
  only:
refs:
  - develop

【问题讨论】：

这个$DEV_AIRFLOW_CONTAINER_REPO 来自哪里？还有你使用的是什么 CI-CI，Gitlab？这会对实现产生很大影响。
@JorrickSleijster，DEV_AIRFLOW_CONTAINER_REPO 来自 git lab CI-CD 变量。我们正在使用 git lab。

标签： kubernetes airflow devops kubernetes-helm kubernetes-pod

【解决方案1】：

您可以将其设为 jinja2 模板并使用一个小型 Python 程序将值插入到模板中。然后，您还可以灵活地使用环境变量或其他东西。

【讨论】：