python：删除重复的文本行组

Question

我知道如何从文本中删除重复的行和重复的字符，但我正在尝试在 python3 中完成一些更复杂的事情。我的文本文件可能包含也可能不包含在每个文本文件中重复的行组。我想编写一个 python 实用程序来查找这些重复的行块并删除除找到的第一个之外的所有行。

例如，假设file1 包含以下数据：

Now is the time
for all good men
to come to the aid of their party.

This is some other stuff.

And this is even different stuff.

Now is the time
for all good men
to come to the aid of their party.

Now is the time
for all good men
to come to the aid of their party.

That's all, folks.

我希望以下是这种转换的结果：

Now is the time
for all good men
to come to the aid of their party.

This is some other stuff.

And this is even different stuff.




That's all, folks.

当发现重复的行组不是从文件开头的位置开始时，我也希望它能够工作。假设file2 看起来像这样：

This is some text.

This is some other text,
as is this.

All around
the mulberry bush
the monkey chased the weasel.

Here is some more random stuff.
All around
the mulberry bush
the monkey chased the weasel.
... and this is another phrase.

All around
the mulberry bush
the monkey chased the weasel.

End

对于file2，这应该是转换的结果：

This is some text.

This is some other text,
as is this.

All around
the mulberry bush
the monkey chased the weasel.

Here is some more random stuff.
... and this is another phrase.


End

需要明确的是，在运行此所需实用程序之前，可能不知道可能重复的行组。该算法必须自己识别这些重复的行组。

我确信，只要有足够的工作和足够的时间，我最终就能想出我正在寻找的算法。但我希望有人可能已经解决了这个问题并将结果发布在某个地方。我一直在寻找并没有找到任何东西，但也许我忽略了一些东西。

附录：我需要更清楚地说明。行组必须是最大的组，并且每个组必须至少包含 2 行。

例如，假设file3 看起来像这样：

line1 line1 line1
line2 line2 line2
line3 line3 line3

other stuff

line1 line1 line1
line3 line3 line3
line2 line2 line2

在这种情况下，所需的算法不会删除任何行。

还有一个例子，在file4:

abc def ghi
jkl mno pqr

line1 line1 line1
line2 line2 line2
line3 line3 line3
abc def ghi
line1 line1 line1
line2 line2 line2
line3 line3 line3
line4 line4 line4
qwerty
line1 line1 line1
line2 line2 line2
line3 line3 line3
line4 line4 line4
asdfghj
line1 line1 line1
line2 line2 line2
line3 line3 line3
lkjhgfd
line2 line2 line2
line3 line3 line3
line4 line4 line4
wxyz

我正在寻找的结果是这样的：

abc def ghi
jkl mno pqr

line1 line1 line1
line2 line2 line2
line3 line3 line3
abc def ghi
line1 line1 line1
line2 line2 line2
line3 line3 line3
line4 line4 line4
qwerty
asdfghj
line1 line1 line1
line2 line2 line2
line3 line3 line3
lkjhgfd
line2 line2 line2
line3 line3 line3
line4 line4 line4
wxyz

换句话说，由于 4 行组（带有“line1 ... line2 ... line3 ... line4 ...”）是最大的重复组，因此是唯一被删除的组.

如果我想同时删除较小的重复组，我可以一直重复该过程直到文件未更改。

Answer 1

我想出了以下解决方案。它可能仍然有一些无法解释的边缘情况，而且它可能不是最有效的方法，但至少在我的初步测试之后，它似乎有效。

这个转发已经修复了我最初提交的版本中的一些错误。

欢迎提出任何改进建议。

# Remove all but the first occurrence of the longest                                                                            
# duplicated group of lines from a block of text.
# In this utility, a "group" of lines is considered
# to be two or more consecutive lines.                                                                             
#                                                                                                                               
# Much of this code has been shamelessly stolen from                                                                            
# https://programmingpraxis.com/2010/12/14/longest-duplicated-substring/                                                        

import sys

from itertools import starmap, takewhile, tee
from operator import eq, truth

# imap and izip no longer exist in python3 itertools.                                                                           
# These are simply equivalent to map and zip in python3.                                                                        
try:
    # python2 ...
    from itertools import imap
except ImportError:
    # python3 ...
    imap = map
try:
    # python2 ...
    from itertools import izip
except ImportError:
    # python3 ...
    izip = zip

def remove_longest_dup_line_group(text):
    if not text:
        return ''
    # Unlike in the original code, here we're dealing                                                                           
    # with groups of whole lines instead of strings                                                                              
    # (groups of characters). So we split the incoming                                                                          
    # data into a list of lines, and we then apply the                                                                          
    # algorithm to these lines, treating a line in the
    # same way that the original algorithm treats an
    # individual character.                                                                                                       
    lines = text.split('\n')
    ld = longest_duplicate(lines)
    if not ld:
        return text
    tokens = text.split(ld)
    if len(tokens) < 1:
        # Defensive programming: this shouldn't ever happen,                                                                    
        # but just in case ...                                                                                                  
        return text
    return '{}{}{}'.format(tokens[0], ld, ''.join(tokens[1:]))

def pairwise(iterable):
    a, b = tee(iterable)
    next(b, None)
    return izip(a,b)

def prefix(a, b):
    count = sum(takewhile(truth, imap(eq, a, b)))
    if count < 2:
        # Blocks must consist of more than one line.
        return ''
    else:
        return '{}\n'.format('\n'.join(a[:count]))

def longest_duplicate(s):
    suffixes = (s[n:] for n in range(len(s)))
    return max(starmap(prefix, pairwise(sorted(suffixes))), key=len)

if __name__ == '__main__':
    text = sys.stdin.read()
    if text:
        # Use sys.stdout.write instead of print to
        # avoid adding an extra newline at the end.
        sys.stdout.write(remove_longest_dup_line_group(text))
    sys.exit(0)

Answer 2

快速而肮脏，未针对极端情况进行测试：

#!/usr/bin/env python3

from pathlib import Path

TEXT = '''Now is the time
for all good men
to come to the aid of their party.

This is some other stuff.

And this is even different stuff.

Now is the time
for all good men
to come to the aid of their party.

Now is the time
for all good men
to come to the aid of their party.

That's all, folks.'''

def remove_duplicate_blocks(lines):
    num_lines = len(lines)

    for idx_start in range(num_lines):
        idx_end = num_lines

        for idx in range(idx_end, -1, -1):
            if idx_start < idx:
                dup_candidate_block = lines[idx_start + 1: idx]
                len_dup_block = len(dup_candidate_block)
                if len_dup_block and len_dup_block < int(num_lines / 2):
                    for scan_idx in range(idx):
                        if ((idx_start + 1) > scan_idx
                                and dup_candidate_block == lines[scan_idx: scan_idx + len_dup_block]):
                            lines[idx_start + 1: idx] = []
                            return remove_duplicate_blocks(lines)
    return lines


if __name__ == '__main__':
    clean_lines = remove_duplicate_blocks(TEXT.split('\n'))
    print('\n'.join(clean_lines))

输出：

Now is the time
for all good men
to come to the aid of their party.

This is some other stuff.
And this is even different stuff.
That's all, folks.