我正在尝试编写一个.sh文件,同时运行许多程序

我试过了

prog1 
prog2

但是它会运行prog1,然后等待prog1结束,然后启动prog2……

那么如何并行运行呢?


当前回答

由于某些原因,我不能使用等待,我想出了这个解决方案:

# create a hashmap of the tasks name -> its command
declare -A tasks=(
  ["Sleep 3 seconds"]="sleep 3"
  ["Check network"]="ping imdb.com"
  ["List dir"]="ls -la"
)

# execute each task in the background, redirecting their output to a custom file descriptor
fd=10
for task in "${!tasks[@]}"; do
    script="${tasks[${task}]}"
    eval "exec $fd< <(${script} 2>&1 || (echo $task failed with exit code \${?}! && touch tasks_failed))"
    ((fd+=1))
done

# print the outputs of the tasks and wait for them to finish
fd=10
for task in "${!tasks[@]}"; do
    cat <&$fd
    ((fd+=1))
done

# determine the exit status
#   by checking whether the file "tasks_failed" has been created
if [ -e tasks_failed ]; then
    echo "Task(s) failed!"
    exit 1
else
    echo "All tasks finished without an error!"
    exit 0
fi

其他回答

如果你想用ctrl-c轻松地运行和杀死多个进程,这是我最喜欢的方法:在一个(…)subshell中生成多个后台进程,并捕获SIGINT来执行kill 0,这将杀死subshell组中生成的所有内容:

(trap 'kill 0' SIGINT; prog1 & prog2 & prog3)

你可以有复杂的进程执行结构,所有的事情都可以用一个ctrl-c来结束(只要确保最后一个进程是在前台运行的,也就是说,在prog1.3之后不要包含&):

(trap 'kill 0' SIGINT; prog1.1 && prog1.2 & (prog2.1 | prog2.2 || prog2.3) & prog1.3)

如果最后一个命令可能会提前退出,而您希望保持其他所有命令的运行,则添加wait作为最后一个命令。在下面的例子中,sleep 2会先退出,在sleep 4结束之前杀死它;添加wait可以让两者都运行到完成:

(trap 'kill 0' SIGINT; sleep 4 & sleep 2 & wait)

进程生成管理器

Sure, technically these are processes, and this program should really be called a process spawning manager, but this is only due to the way that BASH works when it forks using the ampersand, it uses the fork() or perhaps clone() system call which clones into a separate memory space, rather than something like pthread_create() which would share memory. If BASH supported the latter, each "sequence of execution" would operate just the same and could be termed to be traditional threads whilst gaining a more efficient memory footprint. Functionally however it works the same, though a bit more difficult since GLOBAL variables are not available in each worker clone hence the use of the inter-process communication file and the rudimentary flock semaphore to manage critical sections. Forking from BASH of course is the basic answer here but I feel as if people know that but are really looking to manage what is spawned rather than just fork it and forget it. This demonstrates a way to manage up to 200 instances of forked processes all accessing a single resource. Clearly this is overkill but I enjoyed writing it so I kept on. Increase the size of your terminal accordingly. I hope you find this useful.

ME=$(basename $0)
IPC="/tmp/$ME.ipc"      #interprocess communication file (global thread accounting stats)
DBG=/tmp/$ME.log
echo 0 > $IPC           #initalize counter
F1=thread
SPAWNED=0
COMPLETE=0
SPAWN=1000              #number of jobs to process
SPEEDFACTOR=1           #dynamically compensates for execution time
THREADLIMIT=50          #maximum concurrent threads
TPS=1                   #threads per second delay
THREADCOUNT=0           #number of running threads
SCALE="scale=5"         #controls bc's precision
START=$(date +%s)       #whence we began
MAXTHREADDUR=6         #maximum thread life span - demo mode

LOWER=$[$THREADLIMIT*100*90/10000]   #90% worker utilization threshold
UPPER=$[$THREADLIMIT*100*95/10000]   #95% worker utilization threshold
DELTA=10                             #initial percent speed change

threadspeed()        #dynamically adjust spawn rate based on worker utilization
{
   #vaguely assumes thread execution average will be consistent
   THREADCOUNT=$(threadcount)
   if [ $THREADCOUNT -ge $LOWER ] && [ $THREADCOUNT -le $UPPER ] ;then
      echo SPEED HOLD >> $DBG
      return
   elif [ $THREADCOUNT -lt $LOWER ] ;then
      #if maxthread is free speed up
      SPEEDFACTOR=$(echo "$SCALE;$SPEEDFACTOR*(1-($DELTA/100))"|bc)
      echo SPEED UP $DELTA%>> $DBG
   elif [ $THREADCOUNT -gt $UPPER ];then
      #if maxthread is active then slow down
      SPEEDFACTOR=$(echo "$SCALE;$SPEEDFACTOR*(1+($DELTA/100))"|bc)
      DELTA=1                            #begin fine grain control
      echo SLOW DOWN $DELTA%>> $DBG
   fi

   echo SPEEDFACTOR $SPEEDFACTOR >> $DBG

   #average thread duration   (total elapsed time / number of threads completed)
   #if threads completed is zero (less than 100), default to maxdelay/2  maxthreads

   COMPLETE=$(cat $IPC)

   if [ -z $COMPLETE ];then
      echo BAD IPC READ ============================================== >> $DBG
      return
   fi

   #echo Threads COMPLETE $COMPLETE >> $DBG
   if [ $COMPLETE -lt 100 ];then
      AVGTHREAD=$(echo "$SCALE;$MAXTHREADDUR/2"|bc)
   else
      ELAPSED=$[$(date +%s)-$START]
      #echo Elapsed Time $ELAPSED >> $DBG
      AVGTHREAD=$(echo "$SCALE;$ELAPSED/$COMPLETE*$THREADLIMIT"|bc)
   fi
   echo AVGTHREAD Duration is $AVGTHREAD >> $DBG

   #calculate timing to achieve spawning each workers fast enough
   # to utilize threadlimit - average time it takes to complete one thread / max number of threads
   TPS=$(echo "$SCALE;($AVGTHREAD/$THREADLIMIT)*$SPEEDFACTOR"|bc)
   #TPS=$(echo "$SCALE;$AVGTHREAD/$THREADLIMIT"|bc)  # maintains pretty good
   #echo TPS $TPS >> $DBG

}
function plot()
{
   echo -en \\033[${2}\;${1}H

   if [ -n "$3" ];then
         if [[ $4 = "good" ]];then
            echo -en "\\033[1;32m"
         elif [[ $4 = "warn" ]];then
            echo -en "\\033[1;33m"
         elif [[ $4 = "fail" ]];then
            echo -en "\\033[1;31m"
         elif [[ $4 = "crit" ]];then
            echo -en "\\033[1;31;4m"
         fi
   fi
      echo -n "$3"
      echo -en "\\033[0;39m"
}

trackthread()   #displays thread status
{
   WORKERID=$1
   THREADID=$2
   ACTION=$3    #setactive | setfree | update
   AGE=$4

   TS=$(date +%s)

   COL=$[(($WORKERID-1)/50)*40]
   ROW=$[(($WORKERID-1)%50)+1]

   case $ACTION in
      "setactive" )
         touch /tmp/$ME.$F1$WORKERID  #redundant - see main loop
         #echo created file $ME.$F1$WORKERID >> $DBG
         plot $COL $ROW "Worker$WORKERID: ACTIVE-TID:$THREADID INIT    " good
         ;;
      "update" )
         plot $COL $ROW "Worker$WORKERID: ACTIVE-TID:$THREADID AGE:$AGE" warn
         ;;
      "setfree" )
         plot $COL $ROW "Worker$WORKERID: FREE                         " fail
         rm /tmp/$ME.$F1$WORKERID
         ;;
      * )

      ;;
   esac
}

getfreeworkerid()
{
   for i in $(seq 1 $[$THREADLIMIT+1])
   do
      if [ ! -e /tmp/$ME.$F1$i ];then
         #echo "getfreeworkerid returned $i" >> $DBG
         break
      fi
   done
   if [ $i -eq $[$THREADLIMIT+1] ];then
      #echo "no free threads" >> $DBG
      echo 0
      #exit
   else
      echo $i
   fi
}

updateIPC()
{
   COMPLETE=$(cat $IPC)        #read IPC
   COMPLETE=$[$COMPLETE+1]     #increment IPC
   echo $COMPLETE > $IPC       #write back to IPC
}


worker()
{
   WORKERID=$1
   THREADID=$2
   #echo "new worker WORKERID:$WORKERID THREADID:$THREADID" >> $DBG

   #accessing common terminal requires critical blocking section
   (flock -x -w 10 201
      trackthread $WORKERID $THREADID setactive
   )201>/tmp/$ME.lock

   let "RND = $RANDOM % $MAXTHREADDUR +1"

   for s in $(seq 1 $RND)               #simulate random lifespan
   do
      sleep 1;
      (flock -x -w 10 201
         trackthread $WORKERID $THREADID update $s
      )201>/tmp/$ME.lock
   done

   (flock -x -w 10 201
      trackthread $WORKERID $THREADID setfree
   )201>/tmp/$ME.lock

   (flock -x -w 10 201
      updateIPC
   )201>/tmp/$ME.lock
}

threadcount()
{
   TC=$(ls /tmp/$ME.$F1* 2> /dev/null | wc -l)
   #echo threadcount is $TC >> $DBG
   THREADCOUNT=$TC
   echo $TC
}

status()
{
   #summary status line
   COMPLETE=$(cat $IPC)
   plot 1 $[$THREADLIMIT+2] "WORKERS $(threadcount)/$THREADLIMIT  SPAWNED $SPAWNED/$SPAWN  COMPLETE $COMPLETE/$SPAWN SF=$SPEEDFACTOR TIMING=$TPS"
   echo -en '\033[K'                   #clear to end of line
}

function main()
{
   while [ $SPAWNED -lt $SPAWN ]
   do
      while [ $(threadcount) -lt $THREADLIMIT ] && [ $SPAWNED -lt $SPAWN ]
      do
         WID=$(getfreeworkerid)
         worker $WID $SPAWNED &
         touch /tmp/$ME.$F1$WID    #if this loops faster than file creation in the worker thread it steps on itself, thread tracking is best in main loop
         SPAWNED=$[$SPAWNED+1]
         (flock -x -w 10 201
            status
         )201>/tmp/$ME.lock
         sleep $TPS
        if ((! $[$SPAWNED%100]));then
           #rethink thread timing every 100 threads
           threadspeed
        fi
      done
      sleep $TPS
   done

   while [ "$(threadcount)" -gt 0 ]
   do
      (flock -x -w 10 201
         status
      )201>/tmp/$ME.lock
      sleep 1;
   done

   status
}

clear
threadspeed
main
wait
status
echo

使用GNU Parallel http://www.gnu.org/software/parallel/,它就像:

(echo prog1; echo prog2) | parallel

或者如果你喜欢:

parallel ::: prog1 prog2

了解更多:

观看介绍视频快速介绍: https://www.youtube.com/playlist?list=PL284C9FF2488BC6D1 浏览教程(man parallel_tutorial)。您的命令行 会因此爱你的。 阅读:Ole Tange, GNU Parallel 2018 (Ole Tange, 2018)。

你可以试试ppss(废弃)。PPSS非常强大——你甚至可以创建一个迷你集群。 如果您有一批令人尴尬的并行处理要做,xargs -P也很有用。

由于某些原因,我不能使用等待,我想出了这个解决方案:

# create a hashmap of the tasks name -> its command
declare -A tasks=(
  ["Sleep 3 seconds"]="sleep 3"
  ["Check network"]="ping imdb.com"
  ["List dir"]="ls -la"
)

# execute each task in the background, redirecting their output to a custom file descriptor
fd=10
for task in "${!tasks[@]}"; do
    script="${tasks[${task}]}"
    eval "exec $fd< <(${script} 2>&1 || (echo $task failed with exit code \${?}! && touch tasks_failed))"
    ((fd+=1))
done

# print the outputs of the tasks and wait for them to finish
fd=10
for task in "${!tasks[@]}"; do
    cat <&$fd
    ((fd+=1))
done

# determine the exit status
#   by checking whether the file "tasks_failed" has been created
if [ -e tasks_failed ]; then
    echo "Task(s) failed!"
    exit 1
else
    echo "All tasks finished without an error!"
    exit 0
fi