一次运行两个使用相同conda环境的作业时出现snakemake错误

Question

我在执行 Snakemake (6.0.0) 工作流时遇到错误，即在同一节点上同时启动两个作业，这两个作业都使用相同的 conda 环境。最小的例子如下。

一些观察：

• 在我的机构集群的节点上运行工作流时出现问题，但在我的本地计算机上没有。 （我正在使用 >1 cpu 的交互式 slurm 作业中运行 snakemake；我正在使用 Miniconda，由我的集群系统管理员以 module 的形式提供）
• 当任务被强制串行运行时，工作流完成得很好 (snakemake --use-conda -j1)。仅当-j2 或更高（不超过 slurm 分配中可用的核心数）时才会出现此问题。第一份工作似乎运行良好，但总是第二份工作很糟糕。
• 我可以在 snakemake 创建后很好地激活有问题的 conda 环境（例如，在运行工作流后，conda activate /long_path_to_cluster_project_folder/testing/conda_test/.snakemake/conda/c4751dca 工作，我可以从内部运行 R，等等）
• 如果我运行snakemake --use-conda -j2，我得到的唯一错误是shell 命令运行下的(one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)。如果我添加--verbose，则会以蓝色和红色打印一个冗长的回溯，我已将其包含在下面。相关位似乎是：

    File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/deployment/conda.py", line 505, in prefix_path
      return self.info["conda_prefix"]
  AttributeError: 'Conda' object has no attribute 'info'
  

• 怀疑某种竞争条件，我还尝试添加 --max-jobs-per-second=0.5 来限制作业，这样它们就不会同时启动，但这似乎没有效果（作业同时开始，与以前相同的错误. 我没有用--cluster 或--profile 或任何东西运行snakemake；没有创建额外的slurm 作业，只是在同一个计算节点上产生了进程）
• 如果我创建两个完全不同的 Snakemake 规则并最终同时执行，则会出现同样的问题，只要这两个规则使用相同的 conda 环境。

我对 snakemake 和 HPC 都很陌生，但这似乎介于系统/配置特定的问题（因为它只发生在集群上）和一个小的 snakemake 错误（因为 snakemake 似乎是将问题归因于我的 shell 脚本，而不是与 conda 有关）。我对如何进一步排除故障或解决问题的建议感兴趣。

谢谢！

小例子：

├── input.txt
├── results
└── workflow
    ├── Snakefile
    └── envs
        └── env1.yaml

• workflow/Snakefile:

  rule all:
      input:
          'results/output1.txt',
          'results/output2.txt',
          'results/output3.txt',
          'results/output4.txt'
  
  rule rule1:
      input: 'input.txt'
      output:
          'results/output{n}.txt'
      conda: 'envs/env1.yaml'
      shell:"""
      sleep 5s
      touch {output}
      """
  

• workflow/envs/env1.yaml:

  channels:
  - conda-forge
  - bioconda
  - defaults
  dependencies:
  - r-ggplot2
  

$ snakemake --use-conda -j2 -p --verbose
Building DAG of jobs...
Using shell: /usr/bin/bash
Provided cores: 2
Rules claiming more threads will be scaled down.
Job counts:
    count   jobs
    1   all
    4   rule1
    5

<< snip >>


[Fri Mar  5 21:01:33 2021]
Error in rule rule1:
    jobid: 2
    output: results/output2.txt
    conda-env: /long_path_to_cluster_project_folder/testing/conda_test/.snakemake/conda/c4751dca
    shell:
        
    sleep 5s
    touch results/output2.txt
    
        (one of the commands exited with non-zero exit code; note that snakemake uses bash strict mode!)

Full Traceback (most recent call last):
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
    run(
  File "/long_path_to_cluster_project_folder/testing/conda_test/workflow/Snakefile", line 33, in __rule_rule1
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/shell.py", line 141, in __new__
    cmd = Conda(container_img).shellcmd(conda_env, cmd)
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/deployment/conda.py", line 512, in shellcmd
    activate = os.path.join(self.bin_path(), "activate")
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/deployment/conda.py", line 508, in bin_path
    return os.path.join(self.prefix_path(), "bin")
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/deployment/conda.py", line 505, in prefix_path
    return self.info["conda_prefix"]
AttributeError: 'Conda' object has no attribute 'info'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 568, in _callback
    raise ex
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
    run_func(*args)
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper
    raise RuleException(
snakemake.exceptions.RuleException: AttributeError in line 13 of /long_path_to_cluster_project_folder/testing/conda_test/workflow/Snakefile:
'Conda' object has no attribute 'info'
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/long_path_to_cluster_project_folder/testing/conda_test/workflow/Snakefile", line 13, in __rule_rule1
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/deployment/conda.py", line 512, in shellcmd
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/deployment/conda.py", line 508, in bin_path
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/deployment/conda.py", line 505, in prefix_path

RuleException:
AttributeError in line 13 of /long_path_to_cluster_project_folder/testing/conda_test/workflow/Snakefile:
'Conda' object has no attribute 'info'
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 2326, in run_wrapper
  File "/long_path_to_cluster_project_folder/testing/conda_test/workflow/Snakefile", line 13, in __rule_rule1
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/deployment/conda.py", line 512, in shellcmd
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/deployment/conda.py", line 508, in bin_path
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/deployment/conda.py", line 505, in prefix_path
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 568, in _callback
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/concurrent/futures/thread.py", line 52, in run
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 554, in cached_or_run
  File "/long_path_to_cluster_project_folder/conda_envs/snakemake/lib/python3.9/site-packages/snakemake/executors/__init__.py", line 2357, in run_wrapper

Answer 1

尝试至少更新到 Snakemake v6.0.2。 问题似乎是 v6.0.0 版本的错误，并已使用 v6.0.2 版本 (Release Notes) 进行了修补。你是对的，因为这是一个竞争条件问题 (see commit)。