【发布时间】：2021-08-10 15:57:25
【问题描述】：
在 Airflow 中，我使用 SSHOperator 来调用用于某些自动化工作的 API。工作成功运行并生成了报告，但由于 Socket 异常，Airflow 返回任务失败。
这个错误有时会发生，我想知道导致它的原因。
收到的错误信息：
[2021-07-20 08:00:07,345] {ssh.py:109} INFO - Running command: curl -u <user:pw> <URL>
[2021-07-20 08:00:07,414] {ssh.py:145} WARNING - % Total % Received % Xferd Average Speed Time Time Time Current
Dload Upload Total Spent Left Speed
0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
[2021-07-20 08:00:08,420] {ssh.py:145} WARNING -
0 0 0 0 0 0 0 0 --:--:-- 0:00:01 --:--:-- 0
[2021-07-20 08:00:09,421] {ssh.py:145} WARNING -
0 0 0 0 0 0 0 0 --:--:-- 0:00:02 --:--:-- 0
[2021-07-20 08:00:10,423] {ssh.py:145} WARNING -
0 0 0 0 0 0 0 0 --:--:-- 0:00:03 --:--:-- 0
[2021-07-20 08:00:10,615] {ssh.py:141} INFO - Report Sent Successfully.
[2021-07-20 08:00:10,616] {transport.py:1819} ERROR - Socket exception: Bad file descriptor (9)
[2021-07-20 08:00:10,633] {taskinstance.py:1481} ERROR - Task failed with exception
Traceback (most recent call last):
File "/u01/airflow-venv/lib/python3.8/site-packages/airflow/providers/ssh/operators/ssh.py", line 152, in execute
stdout.channel.close()
File "/u01/airflow-venv/lib/python3.8/site-packages/paramiko/channel.py", line 671, in close
self.transport._send_user_message(m)
File "/u01/airflow-venv/lib/python3.8/site-packages/paramiko/transport.py", line 1863, in _send_user_message
self._send_message(data)
File "/u01/airflow-venv/lib/python3.8/site-packages/paramiko/transport.py", line 1839, in _send_message
self.packetizer.send_message(data)
File "/u01/airflow-venv/lib/python3.8/site-packages/paramiko/packet.py", line 431, in send_message
self.write_all(out)
File "/u01/airflow-venv/lib/python3.8/site-packages/paramiko/packet.py", line 367, in write_all
raise EOFError()
EOFError
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/u01/airflow-venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1137, in _run_raw_task
self._prepare_and_execute_task_with_callbacks(context, task)
File "/u01/airflow-venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1311, in _prepare_and_execute_task_with_callbacks
result = self._execute_task(context, task_copy)
File "/u01/airflow-venv/lib/python3.8/site-packages/airflow/models/taskinstance.py", line 1336, in _execute_task
result = task_copy.execute(context=context)
File "/u01/airflow-venv/lib/python3.8/site-packages/airflow/providers/ssh/operators/ssh.py", line 171, in execute
raise AirflowException(f"SSH operator error: {str(e)}")
airflow.exceptions.AirflowException: SSH operator error:
--- 编辑---
generate_report = SSHOperator(
task_id = 'generate_report',
ssh_conn_id = 'ssh_123',
command = curl -u user:password "http://localhost:1234/path/to/trigger/report_creation_API?async=false",
)
【问题讨论】：
-
错误文件描述符错误通常发生在写入已关闭的套接字、从只写套接字读取或写入只读套接字时。发布minimal viable example 以帮助我们重现您的错误。
-
不幸的是，此错误仅在 Airflow 的 SSHOpertaor 成功运行并完成其工作时才会发生。因此，我没有可以定期重现错误的代码，SSHOperator 的代码已更新以获取更多信息。
-
这听起来像环境，那么。发布有关代码运行位置的更多详细信息 - 例如，它是否在容器中？您正在通过 SSH 连接的机器是否会抢先关闭连接？
-
SSHOperator需要ssh_conn_id，你能跟踪连接 ID 发生了什么吗？很可能会过早关闭。 -
您也可以尝试移动到您自己的SSHHook，看看是否会导致同样的问题。你可以试试
keepalive和timeout看看是否能解决问题。
标签： python ssh airflow paramiko