SPACY 自定义 NER 不返回任何实体答案

【问题标题】：SPACY custom NER is not returning any entitySPACY 自定义 NER 不返回任何实体
【发布时间】：2020-01-31 17:45:39
【问题描述】：

我正在尝试训练一个 Spacy 模型来识别一些自定义 NER，训练数据如下所示，主要与识别一些服务器模型、FY 格式的日期和 HDD 类型有关：

TRAIN_DATA = [('Send me the number of units shipped in FY21 for A566TY server', {'entities': [(39, 42, 'DateParse'),(48,53,'server')]}),
            ('Send me the number of units shipped in FY-21 for A5890Y server', {'entities': [(39, 43, 'DateParse'),(49,53,'server')]}),              
          ('How many systems sold with 3.5 inch drives in FY20-Q2 for F567?', {'entities': [(46, 52, 'DateParse'),(58,61,'server'),(27,29,'HDD')]}),              
          ('Total revenue in FY20Q2 for 3.5 HDD', {'entities': [(17, 22, 'DateParse'),(28,30,'HDD')]}),
          ('How many systems sold with 3.5 inch drives in FY20-Q2 for F567?', {'entities': [(46, 52, 'DateParse'),(58,61,'server'),(27,29,'HDD')]}),

          ('Total units shipped in FY2017-FY2021', {'entities': [(23, 28, 'DateParse'),(30,35,'DateParse')]}),
          ('Total units shipped in FY 18', {'entities': [(23, 27, 'DateParse')]}),
          ('Total units shipped between FY16 and FY2021', {'entities': [(28, 31, 'DateParse'),(37,42,'DateParse')]})
         ]
def train_spacy(data,iterations):
TRAIN_DATA = data
nlp = spacy.blank('en')  # create blank Language class
# create the built-in pipeline components and add them to the pipeline
# nlp.create_pipe works for built-ins that are registered with spaCy
if 'ner' not in nlp.pipe_names:
    ner = nlp.create_pipe('ner')
    nlp.add_pipe(ner, last=True)


# add labels
for _, annotations in TRAIN_DATA:
     for ent in annotations.get('entities'):
        ner.add_label(ent[2])

# get names of other pipes to disable them during training
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != 'ner']
with nlp.disable_pipes(*other_pipes):  # only train NER
    optimizer = nlp.begin_training()
    for itn in range(iterations):
        print("Statring iteration " + str(itn))
        random.shuffle(TRAIN_DATA)
        losses = {}
         # batch up the examples using spaCy's minibatch
        batches = minibatch(TRAIN_DATA, size=compounding(4.0, 32.0, 1.001))
        for batch in batches:
            texts, annotations = zip(*batch)
            nlp.update(
                texts,  # batch of texts
                annotations,  # batch of annotations
                drop=0.2,  # dropout - make it harder to memorise data
                losses=losses,
            )
        print("Losses", losses)
return nlp

但即使在训练数据上运行代码时，也不会返回任何实体。

prdnlp = train_spacy(TRAIN_DATA, 100)
for text, _ in TRAIN_DATA:
    doc = prdnlp(text)
    print("Entities", [(ent.text, ent.label_) for ent in doc.ents])
    print("Tokens", [(t.text, t.ent_type_, t.ent_iob) for t in doc])

输出如下：

【问题讨论】：

标签： python-3.x nlp spacy named-entity-recognition

【解决方案1】：

Spacy 目前只能从与标记边界对齐的实体注释中进行训练。主要问题是您的跨度结束字符太短了一个字符。字符开始/结束值应该就像文本的字符串切片：

text = "Send me the number of units shipped in FY21 for A566TY server"
# (39, 42, 'DateParse')
assert text[39:42] == "FY2"

你应该改用(39, 43, 'DateParse')。

第二个问题是，您可能还需要针对 FY2017-FY2021 这样的情况调整分词器，因为默认的英文分词器将其视为一个分词，因此在训练期间将忽略注释 [(23, 28, 'DateParse'),(30,35,'DateParse')]。

在这里查看更详细的解释：https://github.com/explosion/spaCy/issues/4946#issuecomment-580663925

【讨论】：