【发布时间】：2019-09-07 18:28:25
【问题描述】：
我是 NLP 新手。从过去的 2/3 天开始工作。为此使用spacy。我正在尝试使用以下代码“训练其他实体类型”...
"""Example of training an additional entity type
This script shows how to add a new entity type to an existing pre-trained NER
model. To keep the example short and simple, only four sentences are provided
as examples. In practice, you'll need many more — a few hundred would be a
good start. You will also likely need to mix in examples of other entity
types, which might be obtained by running the entity recognizer over unlabelled
sentences, and adding their annotations to the training set.
The actual training is performed by looping over the examples, and calling
`nlp.entity.update()`. The `update()` method steps through the words of the
input. At each word, it makes a prediction. It then consults the annotations
provided on the GoldParse instance, to see whether it was right. If it was
wrong, it adjusts its weights so that the correct action will score higher
next time.
After training your model, you can save it to a directory. We recommend
wrapping models as Python packages, for ease of deployment.
For more details, see the documentation:
* Training: https://spacy.io/usage/training
* NER: https://spacy.io/usage/linguistic-features#named-entities
Compatible with: spaCy v2.1.0+
Last tested with: v2.1.0
"""
from __future__ import unicode_literals, print_function
import plac
import random
from pathlib import Path
import spacy
from spacy.util import minibatch, compounding
# new entity label
LABEL = "CATID:1000012"
# training data
# Note: If you're using an existing model, make sure to mix in examples of
# other entity types that spaCy correctly recognized before. Otherwise, your
# model might learn the new type, but "forget" what it previously knew.
# https://explosion.ai/blog/pseudo-rehearsal-catastrophic-forgetting
TRAIN_DATA = [
(
"The mobile phone can be used to communicate over long distances without wires.",
{"entities": [(11, 16, LABEL)]},
),
(
"A smartphone is a mobile phone that can do more than other phones.",
{"entities": [(2, 12, LABEL)]},
),
(
"Feature phones run on proprietary firmware with third-party software support through platforms such as Java ME or BREW.",
{"entities": [(8, 14, LABEL)]},
),
(
"As mobile phones became more popular, they began to cost less money, and more people could afford them.",
{"entities": [(10, 16, LABEL)]},
),
(
"The majority of smartphones run on Apple iOS or Google Android but others use Windows Phone or BlackBerry OS.",
{"entities": [(16, 27, LABEL)]},
),
(
"Feature phones are often more durable, less complex, and more affordable.",
{"entities": [(8, 14, LABEL)]},
),
]
@plac.annotations(
model=("Model name. Defaults to blank 'en' model.", "option", "m", str),
new_model_name=("New model name for model meta.", "option", "nm", str),
output_dir=("Optional output directory", "option", "o", Path),
n_iter=("Number of training iterations", "option", "n", int),
)
def main(model="en_core_web_sm", new_model_name="CATID", output_dir="Model", n_iter=30):
"""Set up the pipeline and entity recognizer, and train the new entity."""
random.seed(0)
if model is not None:
nlp = spacy.load(model) # load existing spaCy model
print("Loaded model '%s'" % model)
else:
nlp = spacy.blank("en") # create blank Language class
print("Created blank 'en' model")
# Add entity recognizer to model if it's not in the pipeline
# nlp.create_pipe works for built-ins that are registered with spaCy
if "ner" not in nlp.pipe_names:
ner = nlp.create_pipe("ner")
nlp.add_pipe(ner)
# otherwise, get it, so we can add labels to it
else:
ner = nlp.get_pipe("ner")
ner.add_label(LABEL) # add new entity label to entity recognizer
# Adding extraneous labels shouldn't mess anything up
ner.add_label("VEGETABLE")
if model is None:
optimizer = nlp.begin_training()
else:
optimizer = nlp.resume_training()
move_names = list(ner.move_names)
# get names of other pipes to disable them during training
other_pipes = [pipe for pipe in nlp.pipe_names if pipe != "ner"]
with nlp.disable_pipes(*other_pipes): # only train NER
sizes = compounding(1.0, 4.0, 1.001)
# batch up the examples using spaCy's minibatch
for itn in range(n_iter):
random.shuffle(TRAIN_DATA)
batches = minibatch(TRAIN_DATA, size=sizes)
losses = {}
for batch in batches:
texts, annotations = zip(*batch)
nlp.update(texts, annotations, sgd=optimizer,
drop=0.35, losses=losses)
print("Losses", losses)
# test the trained model
test_text = "Mobile phones under 10k."
doc = nlp(test_text)
print("Entities in '%s'" % test_text)
for ent in doc.ents:
print(ent.label_, ent.text)
# save model to output directory
if output_dir is not None:
output_dir = Path(output_dir)
if not output_dir.exists():
output_dir.mkdir()
nlp.meta["name"] = new_model_name # rename model
nlp.to_disk(output_dir)
print("Saved model to", output_dir)
# test the saved model
print("Loading from", output_dir)
nlp2 = spacy.load(output_dir)
# Check the classes have loaded back consistently
assert nlp2.get_pipe("ner").move_names == move_names
doc2 = nlp2(test_text)
for ent in doc2.ents:
print(ent.label_, ent.text)
if __name__ == "__main__":
plac.call(main)
我在这里所做的是，创建了一个名为“CATID:1000012”的新实体，并尝试通过提供一些训练数据来识别新实体（CATID:1000012）来教授现有模型（en_core_web_sm）。
但是，在训练之后，当我使用语句 (Mobile phones under 10k.) 进行测试以识别新实体时，我得到的词 under 被标记为我的新实体。我无法理解为什么会这样。
这里是训练结果的输出，供参考。
nlp python train_new_entity_type.py
Loaded model 'en_core_web_sm'
Losses {'ner': 97.52074194946908}
Losses {'ner': 93.69056796826771}
Losses {'ner': 104.9913784135133}
Losses {'ner': 106.799345289357}
Losses {'ner': 95.95352823211579}
Losses {'ner': 95.5921512588784}
Losses {'ner': 104.20201236551293}
Losses {'ner': 91.30133426242173}
Losses {'ner': 83.3817401985325}
Losses {'ner': 108.10902537551738}
Losses {'ner': 90.79526191594738}
Losses {'ner': 92.66721615749748}
Losses {'ner': 89.48430367704572}
Losses {'ner': 79.65045529220826}
Losses {'ner': 81.69409873239893}
Losses {'ner': 78.08388914307191}
Losses {'ner': 75.96670668302312}
Losses {'ner': 85.84131752077208}
Losses {'ner': 83.16802654485699}
Losses {'ner': 74.70389228454836}
Losses {'ner': 82.74640468226158}
Losses {'ner': 86.27583874967632}
Losses {'ner': 91.80043086154723}
Losses {'ner': 71.57743340098828}
Losses {'ner': 89.68161530740633}
Losses {'ner': 68.54411317529383}
Losses {'ner': 79.08097473334223}
Losses {'ner': 80.63091049017571}
Losses {'ner': 87.19688005072365}
Losses {'ner': 87.32719076574251}
Entities in 'Mobile phones under 10k.'
CATID:1000012 phones
CATID:1000012 under
CATID:1000012 .
Saved model to Model
Loading from Model
CATID:1000012 phones
CATID:1000012 under
VEGETABLE .
【问题讨论】：