如何在bash脚本中等待从该脚本派生的几个子进程完成,然后在任何子进程以code !=0结束时返回退出代码!=0?
简单的脚本:
#!/bin/bash
for i in `seq 0 9`; do
doCalculations $i &
done
wait
上面的脚本将等待所有10个子进程,但它总是给出退出状态0(参见help wait)。我如何修改这个脚本,以便它将发现衍生子进程的退出状态,并在任何子进程以code !=0结束时返回退出代码1 ?
有没有比收集子进程的pid、按顺序等待它们并求和退出状态更好的解决方案呢?
这里有一个使用等待的简单例子。
运行一些进程:
$ sleep 10 &
$ sleep 10 &
$ sleep 20 &
$ sleep 20 &
然后用wait命令等待他们:
$ wait < <(jobs -p)
或者只是等待(没有争论)。
这将等待后台的所有作业完成。
如果提供了-n选项,则等待下一个作业终止并返回其退出状态。
参见:帮助等待和帮助作业语法。
然而,缺点是这将只返回最后一个ID的状态,因此您需要检查每个子流程的状态并将其存储在变量中。
或者让你的计算函数在失败时创建一些文件(空的或有失败日志的),然后检查该文件是否存在。
$ sleep 20 && true || tee fail &
$ sleep 20 && false || tee fail &
$ wait < <(jobs -p)
$ test -f fail && echo Calculation failed.
这是有效的,应该是一个很好的,如果不是更好的@HoverHell的答案!
#!/usr/bin/env bash
set -m # allow for job control
EXIT_CODE=0; # exit code of overall script
function foo() {
echo "CHLD exit code is $1"
echo "CHLD pid is $2"
echo $(jobs -l)
for job in `jobs -p`; do
echo "PID => ${job}"
wait ${job} || echo "At least one test failed with exit code => $?" ; EXIT_CODE=1
done
}
trap 'foo $? $$' CHLD
DIRN=$(dirname "$0");
commands=(
"{ echo "foo" && exit 4; }"
"{ echo "bar" && exit 3; }"
"{ echo "baz" && exit 5; }"
)
clen=`expr "${#commands[@]}" - 1` # get length of commands - 1
for i in `seq 0 "$clen"`; do
(echo "${commands[$i]}" | bash) & # run the command via bash in subshell
echo "$i ith command has been issued as a background job"
done
# wait for all to finish
wait;
echo "EXIT_CODE => $EXIT_CODE"
exit "$EXIT_CODE"
# end
当然,我已经在一个NPM项目中保存了这个脚本,它允许你并行运行bash命令,对测试很有用:
https://github.com/ORESoftware/generic-subshell
正是为了这个目的,我写了一个bash函数:for。
注意::for不仅保留并返回失败函数的退出码,而且终止所有并行运行的实例。在这种情况下可能不需要。
#!/usr/bin/env bash
# Wait for pids to terminate. If one pid exits with
# a non zero exit code, send the TERM signal to all
# processes and retain that exit code
#
# usage:
# :wait 123 32
function :wait(){
local pids=("$@")
[ ${#pids} -eq 0 ] && return $?
trap 'kill -INT "${pids[@]}" &>/dev/null || true; trap - INT' INT
trap 'kill -TERM "${pids[@]}" &>/dev/null || true; trap - RETURN TERM' RETURN TERM
for pid in "${pids[@]}"; do
wait "${pid}" || return $?
done
trap - INT RETURN TERM
}
# Run a function in parallel for each argument.
# Stop all instances if one exits with a non zero
# exit code
#
# usage:
# :for func 1 2 3
#
# env:
# FOR_PARALLEL: Max functions running in parallel
function :for(){
local f="${1}" && shift
local i=0
local pids=()
for arg in "$@"; do
( ${f} "${arg}" ) &
pids+=("$!")
if [ ! -z ${FOR_PARALLEL+x} ]; then
(( i=(i+1)%${FOR_PARALLEL} ))
if (( i==0 )) ;then
:wait "${pids[@]}" || return $?
pids=()
fi
fi
done && [ ${#pids} -eq 0 ] || :wait "${pids[@]}" || return $?
}
使用
for.sh:
#!/usr/bin/env bash
set -e
# import :for from gist: https://gist.github.com/Enteee/c8c11d46a95568be4d331ba58a702b62#file-for
# if you don't like curl imports, source the actual file here.
source <(curl -Ls https://gist.githubusercontent.com/Enteee/c8c11d46a95568be4d331ba58a702b62/raw/)
msg="You should see this three times"
:(){
i="${1}" && shift
echo "${msg}"
sleep 1
if [ "$i" == "1" ]; then sleep 1
elif [ "$i" == "2" ]; then false
elif [ "$i" == "3" ]; then
sleep 3
echo "You should never see this"
fi
} && :for : 1 2 3 || exit $?
echo "You should never see this"
$ ./for.sh; echo $?
You should see this three times
You should see this three times
You should see this three times
1
参考文献
[1]:博客
[2]:要点
为了将此并行化…
for i in $(whatever_list) ; do
do_something $i
done
翻译成这样…
for i in $(whatever_list) ; do echo $i ; done | ## execute in parallel...
(
export -f do_something ## export functions (if needed)
export PATH ## export any variables that are required
xargs -I{} --max-procs 0 bash -c ' ## process in batches...
{
echo "processing {}" ## optional
do_something {}
}'
)
If an error occurs in one process, it won't interrupt the other processes, but it will result in a non-zero exit code from the sequence as a whole.
Exporting functions and variables may or may not be necessary, in any particular case.
You can set --max-procs based on how much parallelism you want (0 means "all at once").
GNU Parallel offers some additional features when used in place of xargs -- but it isn't always installed by default.
The for loop isn't strictly necessary in this example since echo $i is basically just regenerating the output of $(whatever_list). I just think the use of the for keyword makes it a little easier to see what is going on.
Bash string handling can be confusing -- I have found that using single quotes works best for wrapping non-trivial scripts.
You can easily interrupt the entire operation (using ^C or similar), unlike the the more direct approach to Bash parallelism.
下面是一个简化的工作示例……
for i in {0..5} ; do echo $i ; done |xargs -I{} --max-procs 2 bash -c '
{
echo sleep {}
sleep 2s
}'