等待子shell进程完成

Question

processUsageFile()
{
    #sdate=`pin_virtual_time  | awk -F" " '{print $3}'`;

    #Get all new files to be loaded to brm staging data.
    count=`ls ${PRE_STAGING}/TWN* 2>/dev/null|grep -v reprocess|wc -l`
    if [ $count -ne 0 ];then
        # Fork subshell
        (./efx_omc_brm_rpt_process.sh -t TWN & )&
        exitOnError
    fi

    #Process Rapid Report files
    count=`ls $PRE_STAGING/RR* 2>/dev/null|grep -v  reprocess|wc -l`
    if [ $count -ne 0 ];then
        (./efx_omc_brm_rpt_process.sh -t RR &)&
        exitOnError
    fi
...
...
}
#Reprocessing. Process the reprocessed files.
#This method updates the records in the BRM staging table.
reprocessingUsageFile()
{
    #Process TWN fulfillment reprocess files
    count=`ls $PRE_STAGING/TWN*reprocess* 2>/dev/null|wc -l`
    if [ $count -ne 0 ];then
        # Fork subshell
        (./efx_omc_brm_rpt_reprocess.sh -t TWN & ) &
    fi

    #Process Rapid Report files
    count=`ls $PRE_STAGING/RR*reprocess* 2>/dev/null|wc -l`
    if [ $count -ne 0 ];then
        (./efx_omc_brm_rpt_reprocess.sh -t RR &) &
    fi
...
...
}

#Pre processing
PreProcessing

# Start processing usage files.
processUsageFile

processErrFile 

上述代码的思想是做并行处理。所有方法都调用多个子 shell 并从 tty 中分离。我想知道是否有办法先等待前两个方法完成执行，然后再运行最后一个方法。

等待 PID 在某种程度上是不准确的。还在努力……

waitPids() {
echo "Testing $pids -- ${#pids[@]}"
    while [ ${#pids[@]} -ne 0 ]; do
            local range=$(eval echo {0..$((${#pids[@]}-1))})
            local i
            for i in $range; do
                if ! kill -0 ${pids[$i]} 2> /dev/null; then
                    echo "Done -- ${pids[$i]}"
                     unset pids[$i]
                fi
            done
            pids=("${pids[@]}") 
            sleep 1
        done
    }

Answer 1

使用内置等待

$ help wait
wait: wait [-n] [id ...]
    Wait for job completion and return exit status.

    Waits for each process identified by an ID, which may be a process ID or a
    job specification, and reports its termination status.  If ID is not
    given, waits for all currently active child processes, and the return
    status is zero.  If ID is a a job specification, waits for all processes
    in that job's pipeline.

    If the -n option is supplied, waits for the next job to terminate and
    returns its exit status.

    Exit Status:
    Returns the status of the last ID; fails if ID is invalid or an invalid
    option is given.

极简主义示例

$ wait -n; (sleep 3; false); echo $?
1

您的代码作为示例

后台任务立即返回。您的诀窍是将您的函数包装在一个子shell 中，这样您就可以等待子shell（而不是后台作业）完成。例如：

$ wait -n; (processUsageFile); echo $?

如果您想变得比这更复杂，您将不得不在变量中捕获您正在生成的后台任务的 PID，以便您可以使用类似 wait $pidof_process_1 $pidof_process_2 的结构等待特定进程。

将函数封装在子shell 中更简单。但是，您的具体需求可能会有所不同。

Answer 2

似乎主要问题是，您使用的是分离的子外壳。

也许最简单的解决方案是使用不同的机制来分离子外壳，因此您可以使用wait。

例如通过nohup

 nohup ./process1 &
 nohup ./process2 &
 wait

Answer 3

可能是处理和重新处理之间的“等待”命令。

来自： http://www.tldp.org/LDP/abs/html/subshells.html

示例 21-3。在子shell中运行并行进程

(cat list1 list2 list3 | sort | uniq > list123) &
(cat list4 list5 list6 | sort | uniq > list456) &
# Merges and sorts both sets of lists simultaneously.
# Running in background ensures parallel execution.
#
# Same effect as
#   cat list1 list2 list3 | sort | uniq > list123 &
#   cat list4 list5 list6 | sort | uniq > list456 &

wait   # Don't execute the next command until subshells finish.

diff list123 list456

Answer 4

我发现并行化和等待的最佳方法是导出一个函数以在 subshel​​l 中使用，并使用带有 -P 的 xargs 以获得最大数量的并行线程，同时使用 -n 或 -L 为工作函数提供特定数量的参数。

来自：https://man7.org/linux/man-pages/man1/xargs.1.html

       -P max-procs, --max-procs=max-procs
              Run up to max-procs processes at a time; the default is 1.
              If max-procs is 0, xargs will run as many processes as
              possible at a time.  Use the -n option or the -L option
              with -P;

示例代码：

# define some work function and export it
function unit_action() {
  echo action $*
  sleep 5
  echo action $* done
}
export -f unit_action

# list all arguments to feed into function
# with 2 parameters at a time in a maximum of 3 parallel threads
echo {1..9} | xargs -t -n 2 -P 3 bash -c 'unit_action $@' --
echo all done

xargs 将隐式等待，直到所有输入都被消耗掉，因此不需要显式等待命令。