如何使用 SpaCy Matcher 更快地找到匹配项？

Question

只要句子中有被动语态，我就会尝试使用SpaCy Matcher package 来检测匹配项。我写了下面的模式，它正确地找到了被动动词和句子。虽然我现在的问题是速度。我有大约 100 万条记录，每条记录大约有 10 个句子。我想知道我是否可以做任何事情来提高搜索效率？喜欢不返回结束和开始标记？

匹配器：

matcher = Matcher(nlp.vocab)
passive_rule1 = [{'DEP':'nsubjpass', 'OP':'*'}, {'DEP':'xcomp', 'OP':'*'}, {'DEP':'aux','OP':'*'},{'DEP':'auxpass'}, {'DEP':'nsubj', 'OP':'*'}, {'TAG':'VBN'}]
passive_rule2 =  [{'DEP': 'attr'}, {'DEP':'det', 'OP':'*'}, {'Tag':'NOUN', 'OP': '?'}, {'TAG':'VBN'}]

matcher.add('passive_rule1',None, passive_rule1)
matcher.add('passive_rule2 ', None, passive_rule2)

寻找匹配：

df.loc[:, 'PassiveVoice'] = df.Sentence.apply(lambda x:1 if len(matcher(nlp(x)))>0 else 0)

或者，如果有人有任何其他想法，我会很高兴听到！

Answer 1

投入 100 万。向熊猫的dataframe 发送文本，然后循环调用nlp 100 万次是个坏主意。相反，通过df["Sentence"].tolist() 将您的文档放在一个列表中，并通过nlp.pipe 有效地处理它们：

import spacy
from spacy.matcher import Matcher

nlp = spacy.load("en_core_web_md", disable=["ner"])

matcher = Matcher(nlp.vocab)
passive_rule1 = [
    {"DEP": "nsubjpass", "OP": "*"},
    {"DEP": "xcomp", "OP": "*"},
    {"DEP": "aux", "OP": "*"},
    {"DEP": "auxpass"},
    {"DEP": "nsubj", "OP": "*"},
    {"TAG": "VBN"},
]
passive_rule2 = [
    {"DEP": "attr"},
    {"DEP": "det", "OP": "*"},
    {"Tag": "NOUN", "OP": "?"},
    {"TAG": "VBN"},
]

matcher.add("passive_rule1", None, passive_rule1)
matcher.add("passive_rule2", None, passive_rule2)

texts = ["this is my first sentence. about something", "this is another"]
# texts = df["Sentence"].tolist()
docs = nlp.pipe(texts, n_process = 2, batch_size=50)

for doc in docs:
    if matcher(doc):
        #do something

另外注意，使用nlp.pipe()，您可以使用n_process=2（选择您的）打开多处理，并使用batch_size=50（选择您的）批量处理您的文本。