从 pandas 列中用 spaCy 提取段落周围的短语答案

【问题标题】：Extract paragraph surrounding phrase with spaCy from pandas column从 pandas 列中用 spaCy 提取段落周围的短语
【发布时间】：2022-12-11 18:14:19
【问题描述】：

我有一个在一列中包含文本数据的数据框。在本专栏中，我想使用 spaCy 检索匹配词周围的句子。

考虑这个玩具数据框：

import pandas as pd
df_test: pd.DataFrame = pd.DataFrame(
    {
        "col1": ["2022-01-01", "2022-10-10", "2022-12-12"],
        "text": [
            "Sentence without the matching word. Another sentence without the matching word.",
            "Sentence with lowercase matchword_one. And a sentence without the matching word. And a sentence with matchword_two.",
            "Sentence with uppercase Matchword_ONE. And another sentence with the uppercase Matchword_one.",
        ],
    }
)

这个短语匹配器包含两个模式 matchw1 和 matchw2：

import spacy

nlp = spacy.load("en_core_web_sm")
phrase_matcher = spacy.matcher.PhraseMatcher(nlp.vocab, attr="LOWER")
patterns1 = [nlp(text) for text in ["matchword_one"]]
phrase_matcher.add("matchw1", None, *patterns1)
patterns2 = [nlp(text) for text in ["matchword_two"]]
phrase_matcher.add("matchw2", None, *patterns2)

我现在处理文本以在 text_spacy 列中包含一个 spacy 文档

df_test['text_spacy'] = [doc for doc in nlp.pipe(df_test['text'].tolist())]  # convert to spacy object
type(df_test.at[0, 'text_spacy']) # check that cell contains a spaCy Doc object

并应用匹配器：

df_test['matches_phrases'] = df_test['text_spacy'].apply(phrase_matcher)  # match patterns

到目前为止，一切都很好。现在要检索包含 sincgle 对象的匹配词的句子，我将使用：

doc = nlp(
    "Sentence with lowercase matchword_one. And a sentence without the matching word. And a sentence with matchword_two."
)
for sent in doc.sents:
    for match_id, start, end in phrase_matcher(nlp(sent.text)):
        if nlp.vocab.strings[match_id] in ["matchw1"]:
            print("matchw1", sent.text)
            print("")

        if nlp.vocab.strings[match_id] in ["matchw2"]:
            print("matchw2", sent.text)
            print("")
## Out: matchw1 Sentence with lowercase matchword_one.
## Out: matchw2 And a sentence with matchword_two.

我如何在专栏和将短语保存在具有模式名称的列中？

预期的输出是这样的：

## expected output:
#                    
#          col1  ...                                    matches_phrases  phrase_matchw1                                                                                phrase_matchw2
# 0  2022-01-01  ...                                                 []                               
# 1  2022-10-10  ...  [(15306160315042522568, 3, 4), (14646110443092...  Sentence with lowercase matchword_one.                                                        And a sentence with matchword_two.
# 2  2022-12-12  ...  [(15306160315042522568, 3, 4), (15306160315042...  Sentence with uppercase Matchword_ONE. And another sentence with the uppercase Matchword_one.

我的直觉是它会类似于 df_test['matches_phrases'].apply(lambda x: return x.text if match_id, start, end in x)（这不起作用，但我希望它能说明逻辑。

非常感谢您的提示和指点！

【问题讨论】：

标签： python pandas spacy match-phrase

【解决方案1】：

这是一种方法：

for pat in ["matchw1", "matchw2"]:
    df_test[f"phrase_{pat}"] = df_test.apply(
        lambda x: " ".join(
            [
                x["text"].split(". ")[i]
                for i, item in enumerate(x["matches_phrases"])
                if nlp.vocab.strings[item[0]] in [pat]
            ]
        ),
        axis=1,
    )

然后：

print(df_test)
# Output
         col1        
0  2022-01-01   ...   
1  2022-10-10   ...
2  2022-12-12   ...

                                                  matches_phrases  
0                                                              []   
1  [(15306160315042522568, 3, 4), (14646110443092162848, 17, 18)]   
2  [(15306160315042522568, 3, 4), (15306160315042522568, 11, 12)]   

                                                                                 phrase_matchw1  
0
1                                                         Sentence with lowercase matchword_one   
2  Sentence with uppercase Matchword_ONE And another sentence with the uppercase Matchword_one.   

                             phrase_matchw2  
0
1  And a sentence without the matching word
2

【讨论】：