使 PBS 作业进入调试队列

Question

在我的新工作中，我管理一个集群，该集群使用 Torque 作为资源管理器，使用 maui 作为调度器。

目前，我正面临这个重复的问题，即特定用户的作业总是被发送到调试队列。以下是系统上的活动队列列表：

Queue            Memory CPU Time Walltime Node  Run Que Lm  State
---------------- ------ -------- -------- ----  --- --- --  -----
debug              --      --    00:20:00   --    0   0 12   E R
intel              --      --       --      --    0   0 --   E R
medium             --      --    72:00:00   --    0   0 12   E R
bighuge            --      --       --      --    0   0 --   E R
long               --      --       --      --    0   0 12   E R
                                               ----- -----
                                                   0     0

用户提交的作业的 Wall-time 以小时为单位，所以我很困惑为什么它会被发送到调试队列。

此外，这是跟踪作业的输出：

04/08/2016 15:46:48  S    enqueuing into intel, state 1 hop 1
04/08/2016 15:46:48  S    dequeuing from intel, state QUEUED
04/08/2016 15:46:48  S    enqueuing into debug, state 1 hop 1
04/08/2016 15:46:48  S    Job Queued at request of dawn@cm01, owner = dawn@cm01, job name = run01_submit.script, queue =
                          debug
04/08/2016 15:46:49  S    Job Run at request of root@cm01
04/08/2016 15:46:49  S    child reported success for job after 0 seconds (dest=n20), rc=0
04/08/2016 15:46:49  S    preparing to send 'b' mail for job 15631.cm01 to dawn@cm01 (---)
04/08/2016 15:46:49  S    Not sending email: User does not want mail of this type.
04/08/2016 15:46:49  S    obit received - updating final job usage info
04/08/2016 15:46:49  S    job exit status 1 handled
04/08/2016 15:46:49  S    preparing to send 'e' mail for job 15631.cm01 to dawn@cm01 (Exit_status=1
04/08/2016 15:46:49  S    Not sending email: User does not want mail of this type.
04/08/2016 15:46:49  S    Exit_status=1 resources_used.cput=00:00:00 resources_used.mem=0kb resources_used.vmem=0kb
                          resources_used.walltime=00:00:00
04/08/2016 15:46:49  S    on_job_exit task assigned to job
04/08/2016 15:46:49  S    req_jobobit completed
04/08/2016 15:46:49  S    JOB_SUBSTATE_EXITING
04/08/2016 15:46:49  S    JOB_SUBSTATE_STAGEOUT
04/08/2016 15:46:49  S    about to copy stdout/stderr/stageout files
04/08/2016 15:46:49  S    JOB_SUBSTATE_STAGEOUT
04/08/2016 15:46:49  S    JOB_SUBSTATE_STAGEDEL
04/08/2016 15:46:49  S    JOB_SUBSTATE_EXITED
04/08/2016 15:46:49  S    JOB_SUBSTATE_COMPLETE
04/08/2016 15:50:54  S    Request invalid for state of job COMPLETE
04/08/2016 15:51:00  S    Request invalid for state of job COMPLETE
04/08/2016 15:51:49  S    dequeuing from debug, state COMPLETE

现在的解决方法是使用qalter 命令手动更改作业的分配队列。

有什么想法吗？

Answer 1

因为作业会立即从 intel 队列跳转到调试，所以我怀疑您在 qmgr 或 Maui 中配置了自动路由。如果 intel 队列被配置为路由队列，那就可以解释了。

运行qmgr -c "print queue intel" 进行检查。

如果不是路由队列，您可能可以提高日志级别以更好地查看 pbs_server 日志中发生的情况。

当我以这种方式创建路由队列时，我在提交作业时会得到相同类型的跟踪作业输出：

05/20/2016 20:04:05.439 S enqueuing into route, state 1 hop 1 05/20/2016 20:04:05.440 S dequeuing from route, state QUEUED 05/20/2016 20:04:05.440 S enqueuing into test, state 1 hop 1 05/20/2016 20:04:05.737 S Job Run at request of root@testserver

否则，请检查 Maui 配置和日志以获取线索。