如何为作为 kubernetes pod 运行的气流设置持久日志记录和 dag

Question

我使用安装了 unbuntu wsl 的本地 Windows 机器。

$cat pv.yml:
apiVersion: v1
kind: PersistentVolume
metadata:
  name: airflow-volume
  labels:
    type: local
spec:
  #we use local node storage here!
  #kubectl get storageclass
  storageClassName: standard
  capacity:
    storage: 1Gi
  accessModes:
    - ReadWriteMany
  hostPath:
    path: "/mnt/c/dumps"


$ cat pvc.yml
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: kube-claim
spec:
  storageClassName: standard
  accessModes:
    - ReadWriteMany
  resources:
    requests:
      storage: 50Mi

掌舵命令：

helm upgrade $RELEASE_NAME apache-airflow/airflow --namespace $NAMESPACE     --set images.airflow.repository=airflow-dags     --set images.airflow.tag=0.2.3 --set executor=KubernetesExecutor --set logs.persistence.enabled=true --set logs.persistence.existingClaim=kube-claim

来自网络服务器的日志屏幕错误：

*** Trying to get logs (last 100 lines) from worker pod batchtestpodruntest.1f82d2f60f9f4766b81ed54abb28e75a ***

*** Unable to fetch logs from worker pod batchtestpodruntest.1f82d2f60f9f4766b81ed54abb28e75a ***
(403)
Reason: Forbidden
HTTP response headers: HTTPHeaderDict({'Content-Type': 'application/json', 'X-Content-Type-Options': 'nosniff', 'Date': 'Mon,

2021 年 6 月 14 日 15:21:19 GMT'，'Content-Length': '425'}) HTTP 响应正文：b'{"kind":"Status","apiVersion":"v1","metadata":{},"status":"Failure","message":"pods \"batchtestpodruntest.1f82d2f60f9f4766b81ed54abb28e75a\" 是 禁止：用户 \"system:serviceaccount:airflow:latest-airflow-webserver\"不能 在命名空间中的 API 组 \"\" 中获取资源 \"pods/log\" \"airflow\"","re​​ason":"Forbidden","details":{"name":"batchtestpodruntest.1f82d2f60f9f4766b81ed54abb28e75a","kind":"pods"},"code":403}\n'

Answer 1

您可以配置气流以将日志发送到 s3 或弹性搜索。