使用 Python 从语义上检测文本块

Question

我有这个示例日志文本块：

20190122 09:00,000 ###PERFORMANCE string1 string2 string3
20190122 09:10,500 number1 string1 string2 string3
20190122 09:24,670 number2 string1 string2 string3
20190122 10:05,000 number3 string1 string2 string3
20190122 10:33,960 number4 string1 string2 string3
20190122 11:00,321 number5 string1 string2 string3
20190122 11:40,256 ###PERFORMANCE string1 string2 string3
20190123 10:24,670 number1 string1 string2 string3 string4 date1 number2
20190123 10:32,130 number1 string1 string2 string3 string4 date1 number2
20190123 08:00,000 ###PERFORMANCE string1 string2 string3
20190123 08:10,500 number1 string1 string2 string3
20190123 08:24,670 number2 string1 string2 string3
20190123 09:05,000 number3 string1 string2 string3
20190123 10:33,960 number4 string1 string2 string3
20190123 10:00,321 number5 string1 string2 string3
20190123 13:40,256 ###PERFORMANCE string1 string2 string3
20190124 10:00,000 ###PERFORMANCE string1 string2 string3
20190124 10:10,500 number1 string1 string2 string3
20190124 10:24,670 number2 string1 string2 string3
20190124 11:05,000 number3 string1 string2 string3
20190124 12:33,960 number4 string1 string2 string3
20190124 13:00,321 number5 string1 string2 string3
20190124 13:40,256 ###PERFORMANCE string1 string2 string3

我想用 Python 做的是检测每个 ###PERFORMANCE 文本块，如下例所示：

如您所见，有 3 个感兴趣的块，每个块由字符串中的文本 ###PERFORMANCE 分隔。 第一个从第 1 行开始，到第 7 行结束。第 7 行和第 10 行之间的内容不得被视为感兴趣的块。 每个块的字符串行数也可能不同（因此按行数不是一个好主意）。

到目前为止，我所做的只是逐行读取文本文件：

logFile = "testLog.txt"

with open(logFile) as f:
    content = f.readlines()
# you may also want to remove whitespace characters like `\n` at the end of each line
content = [x.strip() for x in content]

for line in content:
    print(line)

我可以通过哪种方式来完成这项任务？使用 NLTK 是个好主意吗？它甚至可以完成这项任务吗？ 有什么一般性的建议吗？

Answer 1

由于您只是在 PERFORMANCE 分隔符上进行匹配，因此使用 NLTK 似乎有点矫枉过正。一个简单的方法是使用一个简单的匹配（是行上的预期字符串），然后根据它切换您的捕获模式。例如：

in_block = False
IDENTIFIER = 'PERFORMANCE'
with open(logfile) as f:
    for line in f.readlines():
        if IDENTIFIER in line:
            # Toggle the boolean
            in_block = not in_block
        if in_block:
            print(line)

Answer 2

我认为您可以通过简单的检查来完成您需要的操作。让我解释一下我是否理解正确。你可以有一个标志（真/假值）来检测你是否在有趣的块中。每当您找到“###PERFORMANCE”时，您都可以更改此标志。然后您可以将这两个块保存在两个列表或您喜欢的任何结构中。

在代码的 sn-p 下方

logFile = "logfile.txt"

with open(logFile) as f:
    content = f.readlines()
# you may also want to remove whitespace characters like `\n` at the end of each line
content = [x.strip() for x in content]

# flag
are_we_in_the_interesting_block = False;

# two lists to save the liens
interesting_block = [];
non_interesting_block = [];

for line in content:
    # check if there is the text ###PERFORMANCE
    is_there_performance = line.find('###PERFORMANCE');

    # if it's not there, it returns -1
    if is_there_performance > 0:
        are_we_in_the_interesting_block = not are_we_in_the_interesting_block;
    else:    
        if are_we_in_the_interesting_block:
            # here I append to a list, but you can do your processing
            interesting_block.append(line);
        else:
            # here processing of the non interesting parts
            non_interesting_block.append(line);

print('Interesting blocks')
print(interesting_block)

print('\n')
print('Non interesting blocks')
print(non_interesting_block)

产生的输出将是

Interesting blocks
['20190122 09:10,500 number1 string1 string2 string3', '20190122 09:24,670 number2 string1 string2 string3', '20190122 10:05,000 number3 string1 string2 string3', '20190122 10:33,960 number4 string1 string2 string3', '20190122 11:00,321 number5 string1 string2 string3', '20190123 08:10,500 number1 string1 string2 string3', '20190123 08:24,670 number2 string1 string2 string3', '20190123 09:05,000 number3 string1 string2 string3', '20190123 10:33,960 number4 string1 string2 string3', '20190123 10:00,321 number5 string1 string2 string3', '20190124 10:10,500 number1 string1 string2 string3', '20190124 10:24,670 number2 string1 string2 string3', '20190124 11:05,000 number3 string1 string2 string3', '20190124 12:33,960 number4 string1 string2 string3', '20190124 13:00,321 number5 string1 string2 string3']


Non interesting blocks
['20190123 10:24,670 number1 string1 string2 string3 string4 date1 number2', '20190123 10:32,130 number1 string1 string2 string3 string4 date1 number2']

如果需要，您可以访问interesting_block[n] 以获取第 n 行..