句子中引用文本的检测答案

【问题标题】：Detection of quoted text in sentences句子中引用文本的检测
【发布时间】：2021-09-08 09:12:46
【问题描述】：

我有句子在其中引用文本，例如：

Why did the author use three sentences in a row that start with the words, "it spun"?
Why did the queen most likely say  “I would have tea instead.”
Why did the fdsfdsf repeat the phrase "he waited" so many times?
Why were "the lights of his town growing smaller below them"?
What is a fdsfdsf for the word "adjust"?
Reread this: "If anybody had asked trial of answered at once, 'My nose.'" What is the correct definition of the word "trial" as it is used here?
Reread these sentences: "This was his courtship, and it lasted all through the summer." What does the word "courtship" mean?

我试图用 REGEX 掩盖引用的部分，但它不准确。比如最后一句：

txt = 'Reread these sentences: "This was his courtship, and it lasted all through the summer." What does the word "courtship" mean?'
print(re.sub(r"(?<=\").{20,}(?=\")", "<quote>", txt))

输出是：

Reread these sentences: "<quote>" mean?

相反，它应该是：

Reread these sentences: "<quote>" What does the word "courtship" mean?

由于我有超过 10k 个实例，因此很难找到适用于所有情况的通用 REGEX 模式。

我的问题是，是否有任何库（可能基于神经网络实现？）或方法来解决这个问题？

【问题讨论】：

您需要将此类问题视为“匹配一个引号，然后匹配任意数量的非引号，然后匹配一个引号”。如果你认为它是“匹配一个引用，然后是任何东西，然后是另一个引用”，你会因为正则表达式的贪婪而失败。
@anubhava，它对这个不起作用：“重读这个：“如果有人要求试用，马上回答，'我的鼻子。'”“试用”这个词的正确定义是什么“就像这里使用的那样？”
@TimRoberts 抱歉，不清楚。你能澄清你的答案吗？
@anubhava，当它是用双引号括起来的文本时。

标签： python regex machine-learning quotes

【解决方案1】：

另一种方法可能是使用与正则表达式不同的技术，shlex

shlex 类可以很容易地为类似于 Unix shell 的简单语法编写词法分析器。这对于编写小型语言（例如，在 Python 应用程序的运行控制文件中）或解析带引号的字符串通常很有用。

shlex.split 在拆分成单词时考虑引号，可选的posix 参数将引号保留在结果中。使用它的输出，您可以创建一个与您描述的字符串类似的字符串。

import shlex

lines = [
'Why did the author use three sentences in a row that start with the words, "it spun"?',
'Why did the queen most likely say  “I would have tea instead.”',
'Why did the fdsfdsf repeat the phrase "he waited" so many times?',
'Why were "the lights of his town growing smaller below them"?',
'What is a fdsfdsf for the word "adjust"?', 'Reread this: "If anybody had asked trial of answered at once, \'My nose.\'" What is the correct definition of the word "trial" as it is used here?',
'Reread these sentences: "This was his courtship, and it lasted all through the summer." What does the word "courtship" mean?',
]
for line in lines:
    print(
        " ".join(
            word
            if word[0] != '"' and word[-1] != '"' else '"<quote>"'
            for word in shlex.split(line, posix=False)
        )
    )

输出：

Why did the author use three sentences in a row that start with the words, "<quote>" ?
Why did the queen most likely say “I would have tea instead.”
Why did the fdsfdsf repeat the phrase "<quote>" so many times?
Why were "<quote>" ?
What is a fdsfdsf for the word "<quote>" ?
Reread this: "<quote>" What is the correct definition of the word "<quote>" as it is used here?
Reread these sentences: "<quote>" What does the word "<quote>" mean?

注意 1：shlex 不会将花引号解释为引号（例如第 2 行），因此如果您有它们，则应在将每一行输入之前.replace() 它们。
注意 2：这是替换所有引用的事件，但如果您只想要第一个并保留其余的，您可以这样做（很确定这可以写得更好，但将其作为概念证明）：

for line in lines:
    new_line = []
    quote_count = 0
    for word in shlex.split(line, posix=False):
        if word[0] == '"' and word[-1] == '"':
            if quote_count < 1:
                quote_count += 1
                new_line.append('"<quote>"')
            else:
                new_line.append(word)
        else:
            new_line.append(word)
    print(' '.join(new_line))

输出：

Why did the author use three sentences in a row that start with the words, "<quote>" ?
Why did the queen most likely say “I would have tea instead.”
Why did the fdsfdsf repeat the phrase "<quote>" so many times?
Why were "<quote>" ?
What is a fdsfdsf for the word "<quote>" ?
Reread this: "<quote>" What is the correct definition of the word "trial" as it is used here?
Reread these sentences: "<quote>" What does the word "courtship" mean?

【讨论】：

【解决方案2】：

对于这些示例，请使用

import re
txt = """Why did the author use three sentences in a row that start with the words, "it spun"?
Why did the queen most likely say  “I would have tea instead.”
Why did the fdsfdsf repeat the phrase "he waited" so many times?
Why were "the lights of his town growing smaller below them"?
What is a fdsfdsf for the word "adjust"?
Reread these sentences: "This was his courtship, and it lasted all through the summer." What does the word "courtship" mean?"""
txt = re.sub(r'''"([^"]*)"''', lambda m: '<quote>' if len(m.group(1))>19 else m.group(), txt)
txt = re.sub(r'“[^“”]{20,}”', '<quote>', txt)
print(txt)

见Python proof。对于各种类型的引号，使用单独的命令，这样更容易控制。

结果：

Why did the author use three sentences in a row that start with the words, "it spun"?
Why did the queen most likely say  <quote>
Why did the fdsfdsf repeat the phrase "he waited" so many times?
Why were <quote>?
What is a fdsfdsf for the word "adjust"?
Reread these sentences: <quote> What does the word "courtship" mean?

【讨论】：