slurm 集群中的 gpucompute* 已关闭*答案

【问题标题】：gpucompute* is down* in slurm clusterslurm 集群中的 gpucompute* 已关闭*
【发布时间】：2021-01-11 23:57:33
【问题描述】：

我的 gpucompute 节点处于关闭状态，无法在 GPU 节点上发送作业。在遵循网络上的所有解决方案后，我无法返回我的“关闭 GPU”节点。在这个问题之前，我在运行 'NVIDIA-Linux-x86_64-410.79.run --no 解决了该错误后，我的 Nvidia 驱动程序配置出现了错误，我无法通过 'nvidia-smi' 检测到 GPU -drm' 我遇到了这个错误，这是因为节点处于关闭状态。非常感谢任何帮助！

command: sbatch md1.s
sbatch: error: Batch job submission failed: Requested node configuration is not available

command:  sinfo
PARTITION   AVAIL  TIMELIMIT  NODES  STATE NODELIST
gpucompute*    up   infinite      1  down* fwb-lab-tesla1

command:  sinfo -R
REASON               USER      TIMESTAMP           NODELIST
Not responding       slurm     2020-09-25T13:13:19 fwb-lab-tesla1

 command: sinfo -Nl
Fri Sep 25 16:35:25 2020
NODELIST        NODES   PARTITION       STATE CPUS    S:C:T MEMORY TMP_DISK WEIGHT AVAIL_FE REASON              
fwb-lab-tesla1      1 gpucompute*       down*   32   32:1:1  64000        0      1   (null)Not responding 


command: vim /etc/slurm/slurm.conf
# slurm.conf file generated by configurator easy.html.
# Put this file on all nodes of your cluster.
# See the slurm.conf man page for more information.
#
ControlMachine=FWB-Lab-Tesla
#ControlAddr=137.72.38.102
#
MailProg=/bin/mail
MpiDefault=none
#MpiParams=ports=#-#
ProctrackType=proctrack/cgroup
ReturnToService=1
SlurmctldPidFile=/var/run/slurmctld.pid
#SlurmctldPort=6817
SlurmdPidFile=/var/run/slurmd.pid
#SlurmdPort=6818
SlurmdSpoolDir=/var/spool/slurmd
#SlurmUser=slurm
SlurmdUser=root
StateSaveLocation=/var/spool/slurm/StateSave
SwitchType=switch/none
TaskPlugin=task/cgroup
#
#
# TIMERS
#KillWait=30

command: ls /etc/init.d
functions  livesys  livesys-late  netconsole  network  README

command: nvidia-smi
Fri Sep 25 16:35:01 2020    

+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.79       Driver Version: 410.79       CUDA Version: 10.0     |
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  TITAN V             Off  | 00000000:02:00.0 Off |                  N/A |
| 24%   32C    P8    N/A /  N/A |      0MiB / 12036MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   1  TITAN V             Off  | 00000000:03:00.0 Off |                  N/A |
| 23%   35C    P8    N/A /  N/A |      0MiB / 12036MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   2  TITAN V             Off  | 00000000:83:00.0 Off |                  N/A |
| 30%   44C    P8    N/A /  N/A |      0MiB / 12036MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+
|   3  TITAN V             Off  | 00000000:84:00.0 Off |                  N/A |
| 31%   42C    P8    N/A /  N/A |      0MiB / 12036MiB |      6%      Default |
---------------------------------------------------------------------------+
                                                                               
----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID   Type   Process name                             Usage      |
|=============================================================================|
|  No running processes found                                                 |

【问题讨论】：

标签： gpu nvidia slurm sbatch sysinfo

【解决方案1】：

您提到的问题可能阻止了gpucompute 上的slurmd 守护进程启动。您应该能够通过运行 systemctl status slurmd 或您的 Linux 发行版的等效命令来确认这一点。

slurmd 日志可能包含类似于

的行

slurmd[1234]: fatal: can't stat gres.conf file /dev/nvidia0: No such file or directory

尝试重新启动它

systemctl start slurmd

一旦您确定nvidia-smi 正确响应。

【讨论】：

感谢您的回复。我试过这个命令。 nvidia-smi 正确响应。虽然有些东西发生了变化，但仍然无法在节点上发送作业； sinfogpucompute* up infinite 1 unk* fwb-lab-tesla1sinfo -Nl fwb-lab-tesla1 1 gpucompute* unknown* 32 32:1:1 64000 0 1 (null) nonesbatch md1.ssbatch: error: Batch job submission failed: Requested node configuration is not available
过了一会儿，又转为down*。 gpucompute* down*
它的状态似乎在不断变化。它处于耗尽和低实际内存状态：sinfogpucompute* up infinite 1 drain fwb-lab-tesla1sinfo -Nlfwb-lab-tesla1 1 gpucompute* drained 32 32:1:1 64000 0 1 (null) Low RealMemory sinfo -RLow RealMemory root 2020-09-27T10:52:38 fwb-lab-tesla1
最终状态再次down*。
Systemd 配置为查找 .pid 文件 (/var/run/slurmd.pid) 的位置与 Slurm 配置为写入它的位置之间似乎存在不匹配。查看slurm.conf中的SlurmdPidFile参数

【解决方案2】：

我的问题已通过以下说明解决。请记住，无论何时重新启动系统，都需要在重新启动后输入命令。感谢 Joan Bryan 解决了这个问题！


slurmd -Dcvvv
reboot
ps -ef | grep slurm
kill xxxx (this is Process id number in the output of previous ps ef command)
nvidia-smi
systemctl start slurmctld
systemctl start slurmd
scontrol update nodename=fwb-lab-tesla1 state=idle

now you can run the jobs on the GPU nodes!
Cheers

【讨论】：