并行执行 bash for 循环答案

【问题标题】：parallel execution bash for loop并行执行 bash for 循环
【发布时间】：2015-08-02 03:55:52
【问题描述】：

我整理了一个简单的小 bash 脚本，它遍历 100-200+ 2GB 数据包捕获（来自 daemonlogger），提示用户在 tcpdump 中匹配过滤器...然后编译所有数据包从每个单独的捕获到 1 个合并的上限。我遇到的一件事是我希望通过一次搜索多个数据包捕获来更快地执行......但不仅仅是简单地使用&作为背景。（试过了，它基本上使系统崩溃，试图将大量 2GB pcaps 加载到内存中。lol oops。）谁能告诉我如何在 for 循环中说“我想在一个每次通过的时间。”等等？

#!/bin/bash
echo '[+] example tcp dump filters:'
echo '[+] host 1.1.1.1'
echo '[+] host 1.1.1.1 dst port 80'
echo '[+] host 1.1.1.1 and host 2.2.2.2 and dst port 80'
echo 'tcpdump filter:'
read FILTER
cd /var/packet_recorder/
DATESTAMP=$(date +"%m-%d-%Y-%H:%M")
# make a specific folder to drop the filtered pcaps in
mkdir /var/packet_recorder/temp/$DATESTAMP
# iterate over all pcaps and check for an instance of your filter
for file in $(ls *.pcap); do
        tcpdump -nn -A -w temp/$DATESTAMP/$file -r $file $FILTER
        # remove empty pcaps that dont match
        if [ "`ls -l temp/$DATESTAMP/$file | awk '{print $5}'`" = "24" ]; then
                rm -f "temp/$DATESTAMP/$file"
        fi
done
echo '[+] Merging pcaps'
# cd to your pcap directory 
cd /var/packet_recorder/temp/${DATESTAMP}
# merge all of the pcaps into one file and remove the seperated files
mergecap *.pcap -w merged.pcap
rm -f InternetBorder.*
echo "\[\+\] Done. your files are in $(pwd)"

【问题讨论】：

标签： bash for-loop tcpdump

【解决方案1】：

最近我从this question 那里了解到在GNU Findutils 中使用GNU Parallel 或xargs -P 来解决这样的问题。

使用xargs -P（假设路径中没有空格）

# iterate over all pcaps and check for an instance of your filter
# process up to 3 files at a time
ls *.pcap | xargs -n1 -P3 -I{} tcpdump -nn -A -w temp/$DATESTAMP/{} -r {} $FILTER

# remove empty pcaps that dont match (remove files whose size is 24)
wc -c temp/$DATESTAMP/*.pcap | head -n -1 |
while read size path; do
    if [[ "$size" = 24 ]]; then
        rm -f "$path"
    fi
done

【讨论】：