并行运行子shell并收集退出代码

Question

通过使用shell脚本，我将一个长数据文件分成8个文件，并在8个实例中并行运行。

function_child()
{
while read -r record
do
 ###process to get the data by arsdoc get##
    exit 12  ## if get fails##
 ### fp2pdf ###
    EXIT 12  ## IF AFP2PDF FAILS ###
 ### logic ###
    exit 12  ## if logic fails####
done < $1
}

## main ##
for file in /$MY_WORK/CCN_split_files/*; do
   proceed_task "$file" &
   PID="$!"
   echo "$PID:$file" | tee $tmp_file
   PID_LIST+="$PID "
done

如何在发生故障时监控子进程的退出代码和pid。 我在下面尝试过，一旦所有进程都发送到后台，我正在使用“等待”功能等待来自我们的 PID_LIST 的 PID 退出，然后捕获并打印相应的退出状态。

for process in "${PID_LIST[@]}";do
   wait "$process"
   exit_status=$?
   file_name=`egrep $process $tmp_file | awk -F ":" '{print $2}' | rev | awk -F "/" '{print $2}' | rev`
   echo "$file_name exit status: $exit_status"
done

但它给出了一个错误

 line 49: wait: `23043 23049 ': not a pid or valid job spec
grep: 23049: No such file or directory

谁能帮帮我，谢谢。

Answer 1

您在 cmets 中提到您不想使用 GNU Parallel，因此此答案适用于没有该限制的人。

doit()  {
  record="$1"
  ###process to get the data by arsdoc get##
     exit 12  ## if get fails##
  ### fp2pdf ###
     EXIT 12  ## IF AFP2PDF FAILS ###
  ### logic ###
     exit 12  ## if logic fails####
}
export -f doit

cat /$MY_WORK/CCN_split_files/* |
  parallel --joblog my.log doit
# Field 7 of my.log is the exit value

# If you have an unsplit version of the input you can have GNU Parallel process it:
# cat /$MY_WORK/CNN_big_file |
#   parallel --joblog my.log doit

Answer 2

在正确的 pid 编号上使用 wait。

function_child() {
    while read -r record; do
        # let's return a random number!
        exit ${RANDOM}
    done <<<'a'
}

mkdir -p my-home/dir
touch my-home/dir/{1..9}

for file in my-home/dir/*; do
    function_child "$file" &
    pid=$!
    echo "Backgrounded: $file (pid=$pid)"
    pids[$pid]=$file
done

for i in "${!pids[@]}"; do
    wait "$i"
    ret=$?
    echo ${pids[$i]} returned with $ret
done

outputs on repl:

Backgrounded: my-home/dir/1 (pid=84)
Backgrounded: my-home/dir/2 (pid=85)
Backgrounded: my-home/dir/3 (pid=86)
Backgrounded: my-home/dir/4 (pid=87)
Backgrounded: my-home/dir/5 (pid=88)
Backgrounded: my-home/dir/6 (pid=89)
Backgrounded: my-home/dir/7 (pid=90)
Backgrounded: my-home/dir/8 (pid=91)
Backgrounded: my-home/dir/9 (pid=92)
my-home/dir/1 returned with 241
my-home/dir/2 returned with 59
my-home/dir/3 returned with 235
my-home/dir/4 returned with 11
my-home/dir/5 returned with 6
my-home/dir/6 returned with 222
my-home/dir/7 returned with 230
my-home/dir/8 returned with 189
my-home/dir/9 returned with 195

但我认为只需使用 xargs 或其他旨在并行运行此类作业的工具。

 printf "%s\n" my-home/dir/* | xargs -n$'\n' -P8 sh -c 'echo "$1"; ###process to get the data by arsdoc get' --

@KamilCuk，appologies，更新了代码。

PID_LIST+="$PID " 创建一个带空格的长变量。 "${PID_LIST[@]}" 是用于数组的扩展。它是这样工作的，${PID_LIST[@]} 只是扩展为变量PID_LIST 的值，就像"$PID_LIST"，所以它扩展为"23043 23049 "。因为它在引号中，所以它迭代一个元素，因此它运行wait "23043 23049 "，因此您会看到错误消息。

不推荐：你可以依赖shell空间分割

for process in $PID_LIST; do
     wait "$process"

但是只使用一个数组：

    PID_LIST+=("$PID")
done

for process in "${PID_LIST[@]}"; do
    wait "$process"

如果您对 pids[$pid]=$file 关联数组感到不安全，请改用两个数组：

     onlypids+=("$pid")
     files+=("$files")
done

for i in "${!onlypids[@]}"; do
     pid="${onlypids[$i]}"
     file="${files[$i]}"
     wait "$pid"

请注意，按照惯例，大写的变量名称用于导出的变量。

Answer 3

wait 参数周围的双引号创建一个字符串。删除引号以使 shell 将字符串分解为单独的 PID。

Answer 4

请参阅：help jobs 和 help wait

在代码末尾收集返回状态

for pid in $(jobs -rp); do
  printf "Job %d handling file %q is still running\n" "$pid" "${pids[pid]}"
done

for pid in ${jobs -sp); do
  printf "Job %s handling file %q has returned with status %d\n" "$pid" "${pids[pid]}" "$(wait "$pid")"
done