集群上的 dask-yarn：无法连接到应用程序答案

【问题标题】：dask-yarn on cluster: Unable to connect to application集群上的 dask-yarn：无法连接到应用程序
【发布时间】：2019-06-12 13:59:19
【问题描述】：

我正在尝试使用 dask-yarn 在集群上分发 Python 作业。

我正在使用以下代码来创建集群：

from dask_yarn import YarnCluster

cluster = YarnCluster(environment='.conda/envs/myconda', worker_vcores=2, worker_memory='4GB', n_workers=4)
client = Client(cluster)
cluster.shutdown()

我会假设您需要更多信息以确保它已连接到纱线。

错误信息如下图：

---------------------------------------------------------------------------
ConnectionError                           Traceback (most recent call last)
<ipython-input-16-9de74c663703> in <module>()
----> 1 cluster = YarnCluster('myconda.tar.gz')

~/.conda/envs/myconda/lib/python3.6/site-packages/dask_yarn/core.py in __init__(self, environment, n_workers, worker_vcores, worker_memory, worker_restarts, worker_env, scheduler_vcores, scheduler_memory, deploy_mode, name, queue, tags, user, skein_client)
    293                                    user=user)
    294 
--> 295         self._start_cluster(spec, skein_client)
    296 
    297     @cached_property

~/.conda/envs/myconda/lib/python3.6/site-packages/dask_yarn/core.py in _start_cluster(self, spec, skein_client)
    373             app = skein_client.submit_and_connect(spec)
    374             try:
--> 375                 scheduler_address = app.kv.wait('dask.scheduler').decode()
    376                 dashboard_address = app.kv.get('dask.dashboard')
    377                 if dashboard_address is not None:

~/.conda/envs/myconda/lib/python3.6/site-packages/skein/kv.py in wait(self, key, return_owner)
    653                 return res
    654 
--> 655             event = event_queue.get()
    656 
    657         return event.result if return_owner else event.result.value

~/.conda/envs/myconda/lib/python3.6/site-packages/skein/kv.py in get(self, block, timeout)
    279         if isinstance(out, Exception):
    280             self._exception = out
--> 281             raise out
    282         return out
    283 

ConnectionError: Unable to connect to application
```python

【问题讨论】：

只是补充一下，这是在 Jupyter 笔记本中。

标签： python python-3.x hadoop-yarn dask dask-distributed

【解决方案1】：

这可能是因为您的应用程序无法启动（因此无法连接）。我建议查看日志：

$ yarn logs -applicationId <YOUR APPLICATION ID>

通过查看上面的代码，一件奇怪的事情是指定的 conda 环境。看起来您传递的是目录路径而不是存档环境 - 您可能需要在此处阅读文档：http://yarn.dask.org/en/latest/environments.html。

【讨论】：