使用 AWK 搜索 .CSV 文件 - 仅使用第一行

Question

我一直在尝试搜索 .csv 文件的特定列以查找包含特定单词的单元格。但是，它仅适用于我的 .csv 文件中的第一行（即标题）。

该文件是一系列超过 10,000 条论坛帖子，第 1 列作为帖子关键字，第 2 列作为帖子文本。下面的标题是'key'，'annotated sentence'。

key,annotated sentence
"(212, 2)","Got evidence to back that up??

I'm not sure how a stoner's worse than an alcoholic really.

-Wez"
"(537, 5)","Forgive me for laughing; no, not really ha, ha, ha ha ha
Could it be that people here as well as Canada and the rest of the world has figured out your infantile ""grading system of States"" is a complete sham and your very reason for existing is but an anti-constitutional farce and has lost any claims you have or will make? You stand alone now brady, with simply a few still clinging to the false hope of having others pay for your failures and unConstitutional stance so you can sit on your hands while you keep harping on overturning the 2A."
"(595, 0)",So you're actually claiming that it is a lie to say that the UK has a lower gun crime rate than the US? Even if the police were miscounting crimes it's still a huge and unjustified leap in logic to conclude from that that the UK does not have a lower gun crime rate.
"(736, 3)","The anti-abortionists claim a load of **** on many issues. I don't listen to them. To put the ""life"" of an unfertilized egg above that of a person is grotesquely sick IMO. I support any such stem cell research wholeheartedly."

CSV 分隔符是逗号，文本分隔符是“。”

如果我尝试：

awk -F, '$1 ~ /key/ {print}' posts_file.csv > output_file.csv

它会输出标题行没问题。但是，我已经尝试过：

awk -F, '$1 ~ /212/ {print}' posts_file.csv > output_file.csv
awk -F, '$2 ~ /Canada/ {print}' posts_file.csv > output_file.csv

这些都不起作用 - 尽管应该有，但没有找到匹配项。我想不通为什么？有任何想法吗？提前致谢。

Answer 1

awk 来救援！

通常复杂的 csv 不起作用，但在您的情况下，由于键和注释句子具有非常不同的值类型，您可以将模式搜索扩展到整个记录而不是键和值，诀窍是定义记录，这又是根据您的格式也可以完成。例如

$ awk -v RS='\n"' '/Canada/{print RT $0}' csv    

"(537, 5)","Forgive me for laughing; no, not really ha, ha, ha ha ha
Could it be that people here as well as Canada and the rest of the world has figured out your infantile ""grading syst
em of States"" is a complete sham and your very reason for existing is but an anti-constitutional farce and has lost a
ny claims you have or will make? You stand alone now brady, with simply a few still clinging to the false hope of havi
ng others pay for your failures and unConstitutional stance so you can sit on your hands while you keep harping on ove
rturning the 2A."

还有这个

$ awk -v RS='\n"' '/(212, 2)/{print RT $0}' csv       

"(212, 2)","Got evidence to back that up??

I'm not sure how a stoner's worse than an alcoholic really.

-Wez"

Answer 2

Python 的 CSV 解析支持开箱即用的格式。

下面是一个简单的脚本，您可以按如下方式调用：

# csvfilter <1-basedFieldNdx> <regexStr> < <infile> > <outfile>
csvfilter 1 'key' < posts_file.csv > output_file.csv
csvfilter 1 '212' < posts_file.csv > output_file.csv
csvfilter 2 'Canada' < posts_file.csv > output_file.csv

示例脚本csvfilter:

#!/usr/bin/env python
# coding=utf-8

import csv, sys, re

# Assign arguments to variables.
fieldNdx = int(sys.argv[1]) - 1  # Index of field to filter; Python arrays are 0-based!
reStr = sys.argv[2] if (len(sys.argv) > 2) else '' # Filter regex

# Read from stdin...
reader = csv.reader(sys.stdin)
# ... write to stdout.
writer = csv.writer(sys.stdout, reader.dialect)

# Read each line...
for row in reader:
  # Match the target field against the filter regex and
  # print the row only if it matches.  
  if (re.search(reStr, row[fieldNdx])):
    writer.writerow(row)

Answer 3

OpenRefine 可以帮助搜索。

Answer 4

将 awk 安全地用于复杂 CSV 的一种方法是使用“csv2tsv”实用程序将 CSV 文件转换为 awk 可以正确处理的格式。 通常 TSV（“制表符分隔值”）格式正好适合这项工作。

（如果最终输出必须是 CSV，则可以使用互补的“tsv2csv”实用程序，或者 awk 本身可以完成这项工作——尽管可能需要小心才能完全正确。）

所以管道可能如下所示：

csv2tsv < input.csv | awk -F\\t 'BEGIN{OFS=FS} ....' | tsv2csv

对于 csv 到 tsv 的转换，有多种替代方法，从您自己的脚本到 Excel，但我建议您花时间检查您选择的任何工具或工具集是否满足“极端情况”要求您感兴趣的内容。