如何从文件中删除重复的行？ [复制]

Question

我有 one.txt 包含数据的文件：

 822.25 111.48 883.59 256.68
 822.25 111.48 883.59 256.68
 8.6 123.68 467.27 276.69
 0.0 186.77 165.62 375.0
 0.0 186.77 165.62 375.0
 724.76 177.83 923.52 316.78
 724.76 177.83 923.52 316.78
 724.76 177.83 923.52 316.78
 724.76 177.83 923.52 316.78
 724.76 177.83 923.52 316.78
 438.03 148.5 540.88 198.54
 511.99 170.97 571.74 215.81
 511.99 170.97 571.74 215.81

对于重复的行，我只想为它们写一行。例如：

724.76 177.83 923.52 316.78

重复5次，我只想写一次，对其他行也做同样的事情，然后将新数据写入文件。

我的代码：

with open('one.txt', 'r') as infile:
    with open('output.txt', 'w') as outfile:
        for line in infile:
            #how to do this?
            if line are repeated remove and replace them with only one line
               outfile.write(line)

Answer 1

这可以通过 linux 实用程序 uniq 完成，只需在终端输入 uniq <infile.txt >outfile.txt。这里符号> 和< 告诉shell 使用提供的文件而不是标准输入和输出。

要在 python 中重新发明这个实用程序，可以这样写：

with open('one.txt', 'r') as infile:
    with open('output.txt', 'w') as outfile:
        prev_line = infile.readline()  # read first line
        outfile.write(prev_line)
        for line in infile:
            if line != prev_line:  # if the line is a different one, print it
                prev_line = line
                outfile.write(line)

Answer 2

您可能想要itertools.groupby，如果没有比较函数，它只会为每个唯一行返回一个“组”，因此您可以完全跳过该组，只需从每个分组中写入一行。

with open('one.txt', 'r') as infile:
    with open('output.txt', 'w') as outfile:
        for line, _ in itertools.groupby(infile):
            outfile.write(line)

这只会替换出现在同一区域的组，如果重复的行可能出现在文件中的多个位置（例如a a b a 会写成a b a），那么您可以保留一组您已经看过的行

seen_lines = set()
with open('one.txt', 'r') as infile:
    with open('output.txt', 'w') as outfile:
        for line in infile:
            if line in seen_lines:
                continue
            outfile.write(line)
            seen_lines.add(line)

Answer 3

您可以利用itertools.groupby() 如下：

from itertools import groupby

data = """
822.25 111.48 883.59 256.68
 822.25 111.48 883.59 256.68
 8.6 123.68 467.27 276.69
 0.0 186.77 165.62 375.0
 0.0 186.77 165.62 375.0
 724.76 177.83 923.52 316.78
 724.76 177.83 923.52 316.78
 724.76 177.83 923.52 316.78
 724.76 177.83 923.52 316.78
 724.76 177.83 923.52 316.78
 438.03 148.5 540.88 198.54
 511.99 170.97 571.74 215.81
 511.99 170.97 571.74 215.81
"""

ones = [key 
        for key, _ in groupby(
            (line.strip() for line in data.split("\n") if line)
        )]
print(ones)

这会产生

[
 '822.25 111.48 883.59 256.68',
 '8.6 123.68 467.27 276.69', 
 '0.0 186.77 165.62 375.0', 
 '724.76 177.83 923.52 316.78', 
 '438.03 148.5 540.88 198.54', 
 '511.99 170.97 571.74 215.81'
]

Answer 4

也许可以尝试做一个 If 语句，如果它被打印一次就可以了，那么也许可以让 Elif 删除任何相似或相同的字符串。