在解析时计算已处理数据的行数

Question

我正在尝试找到一种方法来计算在解析来自数百万个小文件的传入数据时已处理的行数。

以样本数据为例，tab 是分隔符：

CLIENT1.test.com    /var    DIR 21213412    user1   root    default 2000-03-04 18:30:59.000000 PROC_MGMT
CLIENT1.test.com    /usr    DIR 212112  user1   root    default 2006-02-11 08:30:00.000000 PROC_MGMT
CLIENT2.test.com    /var/tmp/test.txt   ACTIVE  FILE    4000    sysuser sysuser NA  2001-04-11 03:00:09.000000 DEFAULT
CLIENT3.test.com    /test.out   PASSIVE FILE    4000    atuser  atgroup group   2012-05-04 02:30:59.000000 AUTOMAT
CLIENT4.test.com    /opt    DIR 542016  dbuser  dbgroup Default 2000-03-04 18:30:59.000000 SYSTEM

我的代码目前看起来像这样：

PATTERN="mssg1|mssg2|mssg3|...|mssgN"
SERVER=my_server_name

find <path> -type f -name "*.txt" -print0 | \
xargs -0 awk -v PAT="$PATTERN" '$0!~PAT' | \
awk '{gsub(/\t/",") {print}}' | \
awk -v SRV="$SERVER" 'BEGIN {FS=OFS=","} {$1=SRV OFS $1;} {if ($4 !~ /DIR/) $4=","$4;} {print}' | \
awk 'BEGIN {FS=OFS=","} {if ($9 == "") $9="01/01/1970 00:00:00 AM"; else {gsub("[:-]"," ",$9); $9=strftime("%m/%d%/Y %r", maketime($9))};} {print}' > /tmp/outputFile.log

我可以通过运行for 循环和wc -l（我猜这会很慢）来计算所有传入文件的总行数，并将其作为yyyy 行数。 我正在寻找的是计算我已经处理的行数。这样我就可以显示类似

的内容

echo "Processed xxxx lines out of yyyy lines"

其中 xxxx 可以被 1000 整除。例如：

Processed 1000 lines out of 1000000 lines.
Processed 2000 lines out of 1000000 lines.
Processed 3000 lines out of 1000000 lines.
.........
Processed 1000000 lines out of 1000000 lines.
Done.

我可以在我正在使用的awk 语句中添加一个计数器吗？

我的代码是基于RHEL 6.7 运行的bash。

Answer 1

以下awk 程序统一了您的整个管道。

可以计算您的记录数，但无法打印总行数，除非您事先知道有多少行。您确实知道有多少文件，因此您可以将其用作计数器。

PATTERN="mssg1|mssg2|mssg3|...|mssgN"
SERVER=my_server_name
find <path> -type f -name "*.txt" -print0 | \
xargs -0 awk -v PAT="$PATTERN" -v SRV="$SERVER" -v OUT=/tmp/outputFile.log '
      BEGIN {FS=OFS=","}
      (FNR==1){f++} 
      # print progress
      (NR%1000==0){ print "Processed "NR" lines and "f-1" files out of "ARGC-2 }
      # skip line matching pattern
      ($0~PAT){next}
      # substitute all tabs, prepend SRV and redefine fields
      # after this point, we inserted a new field before everything
      { gsub(/\t/,","); $0=SRV OFS $0 }
      # redefine $6 which automatically redefines fields
      # after this line, $4 will be an empty field and $5 will be the old $4
      ($4 !~ /DIR/){ $4 = OFS $4 }
      # process field 9
      { if ($9 == "") $9="01/01/1970 00:00:00 AM"
        else { gsub("[-:]"," ",$9); $9=strftime("%m/%d%/Y %r", maketime($9))} }
      # print to output file
      { print $0 > OUT }
      END{ print "Total lines processed: "NR
           print "Total files processed: "f  }'

关于日期的一般建议：避免任何不可排序的内容，当 ascii 排序时，您的格式“mm/dd/yyyy”不按日期排序，而“yyyy-mm-dd”是。此外，时间的 AM 和 PM 没有多大意义。

https://xkcd.com/1179/

如果你想在状态文件中输出状态，你可以这样做

xargs ... | awk ' ... 
    END{ print "Total lines processed: "NR > "status.txt"
         print "Total files processed: "f  > "status.txt" }'