为 ner train 构建数据集

【问题标题】：construct dataset for ner train为 ner train 构建数据集
【发布时间】：2022-11-26 14:37:11
【问题描述】：

我有输入：

text = "Apple est une entreprise, James Alfred travaille ici"
spans = [
    {
"start":0,
"end":5,
"label":"ORG"
},
{
"start":26,
"end":38,
"label":"PER"
}
]

correspondance_dict = {"PER":2, "ORG": 4 , "O" : 0}

我想标记文本并根据跨度列表构建标签，即：

我想要输出：

tokenized_text = ["Apple", "est", "une", "entreprise", "," , "James","Alfred", "travaille", "ici"]
labels = [4,0,0,0,0,2,2,0,0]  #this list constructed with correspondance_dict and spans (4 because Apple is ORG and  the "2,2" because "James,Alfred" is person

【问题讨论】：

标签： python python-3.x nlp spacy

【解决方案1】：

如果您尝试在程序的其他部分使用 huggingface 的管道，则可以使用适当的策略轻松聚合输出文本块。

提供详尽解释的文档here！

from transformers import pipeline

# Initialize the NER pipeline
ner = pipeline("ner", aggregation_strategy="simple")

# Phrase
phrase = "David helped Peter enter the building, where his house is located."

# NER task
ner_result = ner(phrase)

# Print result
print(ner_result)

输出：

[{'entity_group': 'PER', 'score': 0.99642086, 'word': 'David', 'start': 0, 'end': 5}, {'entity_group': 'PER', 'score': 0.99559766, 'word': 'Peter', 'start': 13, 'end': 18}]

【讨论】：