气流 bashoperator CommandNotFoundError答案

【问题标题】：Airflow bashoperator CommandNotFoundError气流 bashoperator CommandNotFoundError
【发布时间】：2022-08-17 05:05:36
【问题描述】：

我想在气流 2 中运行这个 dag 文件。但它在下面的错误中不起作用。我可以在终端中运行相同的命令。但它不是气流的工作环境。

错误

AIRFLOW_CTX_DAG_RUN_ID=scheduled__2022-08-16T02:29:00+00:00
[2022-08-16, 20:30:01 KST] {subprocess.py:62} INFO - Tmp dir root location: 
 /tmp
[2022-08-16, 20:30:01 KST] {subprocess.py:74} INFO - Running command: [\'bash\', \'-c\', \'conda activate ukjo \']
[2022-08-16, 20:30:01 KST] {subprocess.py:85} INFO - Output:
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - CommandNotFoundError: Your shell has not been properly configured to use \'conda activate\'.
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - To initialize your shell, run
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -     $ conda init <SHELL_NAME>
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - Currently supported shells are:
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -   - bash
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -   - fish
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -   - tcsh
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -   - xonsh
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -   - zsh
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO -   - powershell
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - See \'conda init --help\' for more information and options.
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - IMPORTANT: You may need to close and restart your shell after running \'conda init\'.
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:92} INFO - 
[2022-08-16, 20:30:01 KST] {subprocess.py:96} INFO - Command exited with return code 1
[2022-08-16, 20:30:01 KST] {taskinstance.py:1909} ERROR - Task failed with exception
Traceback (most recent call last):
  File \"/home/ai/miniconda3/envs/ukjo/lib/python3.8/site-packages/airflow/operators/bash.py\", line 194, in execute
    raise AirflowException(
airflow.exceptions.AirflowException: Bash command failed. The command returned a non-zero exit code 1.
[2022-08-16, 20:30:01 KST] {taskinstance.py:1415} INFO - Marking task as FAILED. dag_id=dag_bash, task_id=t2, execution_date=20220816T022900, start_date=20220816T023001, end_date=20220816T023001
[2022-08-16, 20:30:01 KST] {standard_task_runner.py:92} ERROR - Failed to execute job 162361 for task t2 (Bash command failed. The command returned a non-zero exit code 1.; 4041098)
[2022-08-16, 20:30:02 KST] {local_task_job.py:156} INFO - Task exited with return code 1
[2022-08-16, 20:30:02 KST] {local_task_job.py:273} INFO - 0 downstream tasks scheduled from follow-on schedule check

DAG python 文件

from airflow.decorators import dag
from airflow.operators.bash import BashOperator
from datetime import datetime


@dag(start_date=datetime(2022, 8, 10), schedule_interval=\'* * * * *\', catchup=False)
def dag_bash():

    t1 = BashOperator(task_id=\'t1\', bash_command=\'source ~/miniconda3/etc/profile.d/conda.sh \')
    t2 = BashOperator(task_id=\'t2\', bash_command=\'conda activate ukjo \')
    t3 = BashOperator(task_id=\'t3\', bash_command=\'echo $CONDA_DEFAULT_ENV \')

    t1 >> t2 >> t3

dag = dag_bash()

bash -c \'conda activate\' 甚至尝试都没有任何意义。它的目的是激活一个 conda 环境在当前外壳内，但是当 bash -c 完成时，当前的 shell 退出。激活的效果完全被 shell 的终止所抵消，那么为什么首先要麻烦呢？
谢谢。我希望它运行这个连续的命令来使用本地 conda 环境并在我的本地服务器上运行我的可执行 python 模块。

标签： airflow airflow-2.x

【解决方案1】：

要在 conda 环境中运行 python 脚本，您只需提供 conda python 解释器的完整路径：

BashOperator(task_id='execute_python_script', bash_command='~/miniconda3/envs/ukjo/bin/python /path/to/python/script.py').

【讨论】：