AWK 参数列表太长（有充分的理由）答案

【问题标题】：AWK-Argument list too long (For good reason)AWK 参数列表太长（有充分的理由）
【发布时间】：2020-02-05 17:03:53
【问题描述】：

我必须处理一个 CSV 文件，并且我需要从具有特定索引号的列中替换一些值。示例

10/06-14:04:21.082467 ,1917,33219,239.255.255.250,1900,,,,
10/06-14:04:22.082715 ,1917,33219,239.255.255.250,1900,,,,
10/06-14:04:23.082940 ,1917,33219,239.255.255.250,1900,,,,
10/06-14:04:24.083256 ,1917,33219,239.255.255.250,1900,,,,
10/06-14:04:27.421793 ,1418,64878,192.168.0.13,161,0xC498BF38,0x0,,
10/06-14:04:27.522099 ,1418,,64879,192.168.0.13,161,0xC499BF39,0x0,,
10/06-14:04:33.445012 ,1421,64878,192.168.0.13,705,0xC498BF38,0x0,,
10/06-14:04:33.545144 ,1421,192.168.0.130xC498BF38,0x0,,

感兴趣的列是第 2 列。我将其提取到一个单独的 tmp 文件中，我看起来像这样：

我的预期输出是：

到目前为止，我使用以下脚本：

csvtool col 2  $file>tmp && echo " TEMPORARY Column DONE "&& echo "Start the Magic"

## Perform an AWK comparation between the integers intervals and the terget number.
##In the end write everything out in the new column .
awk '{ 
if( ($1>=363 && $1<=499) || ($1>=4645 && $1<=4646)){ print 0}  
else if( ($1>=2174 && $1<=2193)) { print 1}  
else if( ($1==500) || ($1>=12308 && $1<=12356)){ print 2} 
else if( ($1>=103 && $1<=220) || ($1>=252 && $1<=299) || ($1>=1980 && $1<=1986) || ($1>=2921 && $1<=2922)){ print 3} 
else if( ($1>=221 && $1<=251) || ($1>=8085 && $1<=8091) || ($1==8350) || ($1>=12809 && $1<=12945) || ($1>=16834 && $1<=17033)){ print 4} 
else if( ($1>=300 && $1<=362) || ($1==522) || ($1>=2923 && $1<=2925) || ($1>=3441 && $1<=3442) || ($1==4644)|| ($1>=5677 && $1<=5695) || ($1>=8082 && $1<=8083)|| ($1>=8093 && $1<=8349) || ($1>=12946 && $1<=12947) || ($1>=12986 && $1<=13215) || ($1>=13309 && $1<=13311)){ print 5}
else if( ($1>=501 && $1<=504) || ($1>=566 && $1<=600) || ($1>=613 && $1<=637) ||  ($1>=2015 && $1<=2040) ||  ($1>=2103 && $1<=2126) || ($1>=2373 && $1<=2374) || ($1>=3828 && $1<=4125) || ($1>=4237 && $1<=4636) || ($1>=4647 && $1<=4889) || ($1>=4991 && $1<=5676) || ($1>=5696 && $1<=5705) || ($1>=6502 && $1<=6595) || ($1>=8429 && $1<=8460) || ($1>=8552 && $1<=8699) || ($1>=10487 && $1<=10977) || ($1>=11326 && $1<=11617) || ($1>=11688 && $1<=11815) || ($1>=11844 && $1<=11938) || ($1>=12490 && $1<=12597) || ($1>=12973 && $1<=12982) || ($1>=13367 && $1<=13414)){ print 6}
else if( ($1>=523 && $1<=548) || ($1>=555 && $1<=565) || ($1>=2005 && $1<=2014) || ($1>=2041 && $1<=2063) || ($1>=2091 && $1<=2102) ||  ($1==2394) || ($1>=2407 && $1<=2411) || ($1>=2926 && $1<=3008) || ($1>=3443 && $1<=3473) || ($1>=3486 && $1<=3813) || ($1>=4132 && $1<=4144) || ($1>=4637 && $1<=4643) || ($1>=4916 && $1<=4981) || ($1>=5711 && $1<=5741) || ($1>=6403 && $1<=6405) || ($1>=6415 && $1<=6466) || ($1>=6701 && $1<=7002) || ($1>=7035 && $1<=7048) || ($1>=8426 && $1<=8428) || ($1>=8496 && $1<=8541) || ($1>=8857 && $1<=9323) || ($1>=9429 && $1<=9618) || ($1>=9674 && $1<=9789) || ($1>=9802 && $1<=9811) || ($1>=9850 && $1<=10009) || ($1>=10131 && $1<=10136) || ($1>=10396 && $1<=10402) || ($1>=11000 && $1<=11175) || ($1==11618) || ($1>=12100 && $1<=12111) || ($1>=12212 && $1<=12219) || ($1==12489) || ($1>=12807 && $1<=12808) || ($1==12983) || ($1>=14616 && $1<=14627) || ($1>=15723 && $1<=15897)){ print 7}
else if( ($1==521) || ($1==554) || ($1>=601 && $1<=612) || ($1>=651 && $1<=708) || ($1>=1905 && $1<=1942) || ($1>=1949 && $1<=1979) || ($1>=1987 && $1<=1993) || ($1>=2259 && $1<=2278) || ($1>=2352 && $1<=2362) || ($1>=2395 && $1<=2406) || ($1>=2412 && $1<=2449) || ($1>=2673 && $1<=2919) || ($1>=3009 && $1<=3016) || ($1>=3814 && $1<=3827) || ($1>=4126 && $1<=4131) || ($1>=4982 && $1<=4990) || ($1>=5706 && $1<=5710) || ($1>=6012 && $1<=6181) || ($1>=6285 && $1<=6339) || ($1>=6409 && $1<=6411) || ($1>=6596 && $1<=6700) || ($1>=7191 && $1<=7424) || ($1==8081) || ($1>=8550 && $1<=8551) || ($1>=8700 && $1<=8716) || ($1>=9324 && $1<=9326) || ($1>=9619 && $1<=9624) || ($1==9729) || ($1>=10018 && $1<=10064) || ($1>=10115 && $1<=10126) || ($1>=10198 && $1<=10386) || ($1==10486) || ($1>=12112 && $1<=12115) || ($1>=12209 && $1<=12211)){ print 8}
else if( ($1>=489 && $1<=498) || ($1>=505 && $1<=520) || ($1>=549 && $1<=553) || ($1>=638 && $1<=650) || ($1>=709 && $1<=1904) || ($1>=1943 && $1<=1948) || ($1>=1994 && $1<=2004) || ($1>=2064 && $1<=2090) || ($1>=2127 && $1<=2173) || ($1>=2194 && $1<=2258) || ($1>=2279 && $1<=2351) || ($1>=2363 && $1<=2372) || ($1==2393) || ($1>=2450 && $1<=2672) || ($1>=3474 && $1<=3485) || ($1>=4145 && $1<=4236) || ($1>=4890 && $1<=4915) || ($1>=5742 && $1<=6011) || ($1>=7003 && $1<=7034) || ($1>=7049 && $1<=7295) || ($1>=7425 && $1<=8080) || ($1==8084) || ($1>=8352 && $1<=8425) || ($1>=8461 && $1<=8495) || ($1>=8542 && $1<=8549) || ($1>=8717 && $1<=8856) || ($1>=9327 && $1<=9428) || ($1>=9625 && $1<=9673) || ($1>=9790 && $1<=9791) || ($1>=9793 && $1<=9801) || ($1>=9812 && $1<=9849) || ($1>=10010 && $1<=10017) || ($1>=10065 && $1<=10114) || ($1>=10128 && $1<=10130) || ($1>=10137 && $1<=10197) || ($1>=10387 && $1<=10395) || ($1>=10403 && $1<=10485) || ($1>=10978 && $1<=10999) || ($1>=11176 && $1<=11325) || ($1>=11620 && $1<=11687) || ($1>=11816 && $1<=11843) || ($1>=11939 && $1<=12099) || ($1>=12116 && $1<=12208) || ($1>=12220 && $1<=12307) || ($1>=12357 && $1<=12488) || ($1>=12598 && $1<=12806) || ($1>=12948 && $1<=12972) || ($1>=13216 && $1<=13306) || ($1>=13312 && $1<=13366) || ($1>=13415 && $1<=14615) || ($1>=14628 && $1<=15722) || ($1>=15989 && $1<=16833) || ($1>=17402 && $1<=17431)){ print 9}
}' tmp

在更新击中我之前，这项工作是一种魅力。如果到现在为止，我的最大整数值为 17431。现在我得到了 50421，这意味着我必须在我的解决方案中声明更多间隔，但是在将所有新间隔插入到它们的位置之后，脚本停止工作并出现错误：

AWK argument list too long

你知道如何在这么多的区间上操作吗？

我正在考虑休耕：鉴于我可以创建一个地图文件，例如：

Target,Index
103,1
104,1
105,2
106,5
107,8
108,9
109,6
110,9
111,6
112,9
113,9
114,9
115,9
116,9
117,9
118,9
119,9
120,9

Target 是我要查找的号码，Index 是替换我的 Target 号码的值。如何将第一列导入 Array1 并将第二列导入 Array2 并为tmp 文件中的每一行检查 Array1 的目标值位置并打印相同的 Array2 药水

示例：如果 1917 在药水 Array1[1853] 然后打印 Array1[1853]（来自同一药水的值），因为这 2 个数组在元素数量方面是相等的。

主要的静止是：有什么方法可以修复我的脚本以接受所有新的 6000 间隔？如果 AWK 不能以这种方式支持它，您有什么建议？

【问题讨论】：

你如何定义你的间隔？这背后一定有一定的逻辑。
我同意@kvantour，这些不能只是您使用的任意数字，因此某种编程方法可能是可能的，而不是硬编码所有这些条件，但如果不是 - 保存脚本在文件中并将其作为awk -f scriptfile tmp 执行，而不是将脚本作为字符串传递给awk。见stackoverflow.com/q/11475221/1745001。
除了 Ed Morton 的建议，请解释输入 csv 如何转换为预期的单列输出。
不确定这是否与问题有关，但一般来说，@kvantour 是正确的：即使条件阈值是非系统的，将它们保持在数组（可能从文件中读取！可维护性的改进！）并循环遍历它们，而不是在脚本中硬编码。
等等 - 你有地图文件吗？您是否在脚本中对从其他文件复制的值进行硬编码？

标签： arrays bash awk

【解决方案1】：

即使您确实必须对您的范围进行硬编码，也可以这样做，以便您可以验证您的范围中没有任何漏洞，或者相同值可能存在于多个范围中并且可以处理的任何情况输入只需哈希查找，而不必遍历每一行输入的范围：

$ cat tst.awk
BEGIN {
    # r[] = ranges[], v = value
    v = 0
    r[363,499]          = v
    r[4645,4646]        = v

    v = 1
    r[2174,2193]        = v
    r[500]              = v
    r[12308,12356]      = v

    populate(r,map)
}

$1 in map { print map[$1] }

function populate(ranges,map,    cnt,range,begend,beg,end,val,n,i) {
    for (range in ranges) {
        n = split(range,begend,SUBSEP)
        beg = begend[1]
        end = begend[n]
        val = ranges[range]
        for (i=beg; i<=end; i++) {
            map[i] = val
            cnt[i]++
        }
        min = ((min == "") || (min > beg) ? beg : min)
        max = ((max == "") || (max < end) ? end : max)
    }

    for (i=min; i<=max; i++) {
        if ( cnt[i] != 1 ) {
            if ( cnt[i] == 0 ) {
                printf "Hole: %d\n", i | "cat>&2"
            }
            else {
                printf "Overlap: %d\n", i | "cat>&2"
            }
        }
    }
}

$ echo 2180 | awk -f  tst.awk 2>/dev/null
1

$ echo 370 | awk -f  tst.awk 2>/dev/null
0

我将 stderr 重定向到上面的 /dev/null，因为我没有完全填充 r[]，因此将报告数百个漏洞。

显然，如果您愿意，您可以从文件中轻松填充 r[]，而不是硬编码脚本中的值，但由于 BEGIN 部分的第一部分是您指定范围的唯一位置，而且它们是写起来很简单，不太可能搞砸，在这种情况下，将数据保存在脚本本身并不是那么糟糕。

【讨论】：

【解决方案2】：

首先创建一个映射文件，如：第一列是最小值，第二列是最大值，第三列是输出。让我们将文件命名为mapfile.txt：

 363  499 0
4645 4646 0
2174 2193 1
...

然后运行 awk，例如（未经测试，预计有错别字）：

awk 'FNR == NR { ++i; min[i]=$1; max[i]=$2; result[i]=$3; }
     FNR != NR { 
          for (j = 1; j <= i; ++j) { 
              if (min[j] <= $1 && $1 <= max[j]) {
                  print result[j];
                  break
              } 
          }
     }
' mapfile.txt second_column_values.txt

首先我们将地图文件读入内存和三个数组。然后我们检查最小值/最大值的值，如果找到则打印结果。然后，如果找到结果 - 我们将退出循环。

或者，如果您处理过大文件，您可以这样做：

首先创建另一个映射文件，第一列是值，第二列是结果。它可以从上面的 mapfile.txt 生成，类似while read a b c; do seq -f "%.0f $c" $a $b; done < mapfile.txt

请记住对该文件进行排序。我们就叫它mapfile2.txt
然后从第 2 列获得数字，在每行添加一个行号，在第二列对其进行排序，然后将 join 与 mapfile2.txt 组合在一起，对行号重新排序并删除行号。

nl -w1 second_column_values.txt | sort -s -k2 |
join -12 -21 - <(<mapfile2.txt sort -s -k1) |
sort -s -k1 | cut -f2-

如果行的顺序无关紧要，也可以不给行编号，也可以这样做。我认为sort+join-ing 文件可能比在非常极端的情况下进行范围比较的普通数组查找更快。

【讨论】：

Dudi Boy 给我一个暂时可行的解决方案，因为我已经完成了间隔，但我会尝试实施您的解决方案作为进一步的开发，因为我的地图文件将定期更新并且是很容易将自己引用到单个文件然后执行不间断检查地图文件是否已更新然后生成新范围，将它们放入新脚本并运行脚本.....您的解决方案对于未来。谢谢。
好的。所以我们必须将 ++i 移到分配前面？并从j = 1迭代？

【解决方案3】：

这是一个可行的小改动。创建一个awk 脚本文件然后运行它。

script.awk

## Perform an AWK comparation between the integers intervals and the terget number.
##In the end write everything out in the new column .
{ 
if( ($1>=363 && $1<=499) || ($1>=4645 && $1<=4646)){ print 0}  
else if( ($1>=2174 && $1<=2193)) { print 1}  
else if( ($1==500) || ($1>=12308 && $1<=12356)){ print 2} 
else if( ($1>=103 && $1<=220) || ($1>=252 && $1<=299) || ($1>=1980 && $1<=1986) || ($1>=2921 && $1<=2922)){ print 3} 
else if( ($1>=221 && $1<=251) || ($1>=8085 && $1<=8091) || ($1==8350) || ($1>=12809 && $1<=12945) || ($1>=16834 && $1<=17033)){ print 4} 
else if( ($1>=300 && $1<=362) || ($1==522) || ($1>=2923 && $1<=2925) || ($1>=3441 && $1<=3442) || ($1==4644)|| ($1>=5677 && $1<=5695) || ($1>=8082 && $1<=8083)|| ($1>=8093 && $1<=8349) || ($1>=12946 && $1<=12947) || ($1>=12986 && $1<=13215) || ($1>=13309 && $1<=13311)){ print 5}
else if( ($1>=501 && $1<=504) || ($1>=566 && $1<=600) || ($1>=613 && $1<=637) ||  ($1>=2015 && $1<=2040) ||  ($1>=2103 && $1<=2126) || ($1>=2373 && $1<=2374) || ($1>=3828 && $1<=4125) || ($1>=4237 && $1<=4636) || ($1>=4647 && $1<=4889) || ($1>=4991 && $1<=5676) || ($1>=5696 && $1<=5705) || ($1>=6502 && $1<=6595) || ($1>=8429 && $1<=8460) || ($1>=8552 && $1<=8699) || ($1>=10487 && $1<=10977) || ($1>=11326 && $1<=11617) || ($1>=11688 && $1<=11815) || ($1>=11844 && $1<=11938) || ($1>=12490 && $1<=12597) || ($1>=12973 && $1<=12982) || ($1>=13367 && $1<=13414)){ print 6}
else if( ($1>=523 && $1<=548) || ($1>=555 && $1<=565) || ($1>=2005 && $1<=2014) || ($1>=2041 && $1<=2063) || ($1>=2091 && $1<=2102) ||  ($1==2394) || ($1>=2407 && $1<=2411) || ($1>=2926 && $1<=3008) || ($1>=3443 && $1<=3473) || ($1>=3486 && $1<=3813) || ($1>=4132 && $1<=4144) || ($1>=4637 && $1<=4643) || ($1>=4916 && $1<=4981) || ($1>=5711 && $1<=5741) || ($1>=6403 && $1<=6405) || ($1>=6415 && $1<=6466) || ($1>=6701 && $1<=7002) || ($1>=7035 && $1<=7048) || ($1>=8426 && $1<=8428) || ($1>=8496 && $1<=8541) || ($1>=8857 && $1<=9323) || ($1>=9429 && $1<=9618) || ($1>=9674 && $1<=9789) || ($1>=9802 && $1<=9811) || ($1>=9850 && $1<=10009) || ($1>=10131 && $1<=10136) || ($1>=10396 && $1<=10402) || ($1>=11000 && $1<=11175) || ($1==11618) || ($1>=12100 && $1<=12111) || ($1>=12212 && $1<=12219) || ($1==12489) || ($1>=12807 && $1<=12808) || ($1==12983) || ($1>=14616 && $1<=14627) || ($1>=15723 && $1<=15897)){ print 7}
else if( ($1==521) || ($1==554) || ($1>=601 && $1<=612) || ($1>=651 && $1<=708) || ($1>=1905 && $1<=1942) || ($1>=1949 && $1<=1979) || ($1>=1987 && $1<=1993) || ($1>=2259 && $1<=2278) || ($1>=2352 && $1<=2362) || ($1>=2395 && $1<=2406) || ($1>=2412 && $1<=2449) || ($1>=2673 && $1<=2919) || ($1>=3009 && $1<=3016) || ($1>=3814 && $1<=3827) || ($1>=4126 && $1<=4131) || ($1>=4982 && $1<=4990) || ($1>=5706 && $1<=5710) || ($1>=6012 && $1<=6181) || ($1>=6285 && $1<=6339) || ($1>=6409 && $1<=6411) || ($1>=6596 && $1<=6700) || ($1>=7191 && $1<=7424) || ($1==8081) || ($1>=8550 && $1<=8551) || ($1>=8700 && $1<=8716) || ($1>=9324 && $1<=9326) || ($1>=9619 && $1<=9624) || ($1==9729) || ($1>=10018 && $1<=10064) || ($1>=10115 && $1<=10126) || ($1>=10198 && $1<=10386) || ($1==10486) || ($1>=12112 && $1<=12115) || ($1>=12209 && $1<=12211)){ print 8}
else if( ($1>=489 && $1<=498) || ($1>=505 && $1<=520) || ($1>=549 && $1<=553) || ($1>=638 && $1<=650) || ($1>=709 && $1<=1904) || ($1>=1943 && $1<=1948) || ($1>=1994 && $1<=2004) || ($1>=2064 && $1<=2090) || ($1>=2127 && $1<=2173) || ($1>=2194 && $1<=2258) || ($1>=2279 && $1<=2351) || ($1>=2363 && $1<=2372) || ($1==2393) || ($1>=2450 && $1<=2672) || ($1>=3474 && $1<=3485) || ($1>=4145 && $1<=4236) || ($1>=4890 && $1<=4915) || ($1>=5742 && $1<=6011) || ($1>=7003 && $1<=7034) || ($1>=7049 && $1<=7295) || ($1>=7425 && $1<=8080) || ($1==8084) || ($1>=8352 && $1<=8425) || ($1>=8461 && $1<=8495) || ($1>=8542 && $1<=8549) || ($1>=8717 && $1<=8856) || ($1>=9327 && $1<=9428) || ($1>=9625 && $1<=9673) || ($1>=9790 && $1<=9791) || ($1>=9793 && $1<=9801) || ($1>=9812 && $1<=9849) || ($1>=10010 && $1<=10017) || ($1>=10065 && $1<=10114) || ($1>=10128 && $1<=10130) || ($1>=10137 && $1<=10197) || ($1>=10387 && $1<=10395) || ($1>=10403 && $1<=10485) || ($1>=10978 && $1<=10999) || ($1>=11176 && $1<=11325) || ($1>=11620 && $1<=11687) || ($1>=11816 && $1<=11843) || ($1>=11939 && $1<=12099) || ($1>=12116 && $1<=12208) || ($1>=12220 && $1<=12307) || ($1>=12357 && $1<=12488) || ($1>=12598 && $1<=12806) || ($1>=12948 && $1<=12972) || ($1>=13216 && $1<=13306) || ($1>=13312 && $1<=13366) || ($1>=13415 && $1<=14615) || ($1>=14628 && $1<=15722) || ($1>=15989 && $1<=16833) || ($1>=17402 && $1<=17431)){ print 9}
}

运行脚本：

awk -f script.awk input.csv

【讨论】：

【解决方案4】：

我建议这个问题的另一种解决方案。更容易维护和使用查找表，避免重复比较。

这是前 6 个过滤器的初始设置。

建议用户添加过滤器 6、7、8、9，并将循环计数器从 5 增加到 9。

script.awk

# initialize filter range pairs
# each filter is a pair of lower/upper ranges
BEGIN {
     filterArr[0] = "363,499,4645,4646";
     filterArr[1] = "2174,2193";
     filterArr[2] = "500,500,12308,12356";
     filterArr[3] = "103,220,252,299,1980,1986,2921,2921";
     filterArr[4] = "221,251,8085,8091,8350,8350,12809,12945,16834,17033";
     filterArr[5] = "300,362,522,522,2923,2925,3441,3442,4644,4644,5677,5695,8082,8083,8093,8349,12946,12947,12986,13215,13309,13311";
}

# for each input line from input file, loop through all the filters (call scanFilter utility funtion).
{
     for (filter = 0; filter <= 1; filter++) scanFilter(filter);
}

# utility function to scan input field $1 to be in ranges
function scanFilter(filterIdx) {
     # if input field alread found, return its value
     if ($1 in foundAlready) {
          print foundAlready[$1];
          return;
     }
     # need to scan the input field through all filters
     rangesCount = split(filterArr[filterIdx], rangesArr, ",");
     # rangesCount holds the range pair values (lower, upper)
     # rangesArr hold rangePairs: rangesArr[1]=1st range lower,rangesArr[2]=1st range upper,rangesArr[3]=2nd range lower,rangesArr[4]=2nd range upper, etc

     for (rangeCounter = 1; rangeCounter <= rangesCount; rangeCounter += 2) {
          # test $1 for each upper range and lower range
          if ($1 >= rangesArr[rangeCounter] && $1 <= rangesArr[rangeCounter + 1]) {
               print filterIdx;
               foundAlready[$1] = filterIdx;
               return;
          }
     }
}

【讨论】：