为什么培训师在教程中培训时不报告评估指标？答案

【问题标题】：Why doesn't trainer report evaluation metrics while training in the tutorial?为什么培训师在教程中培训时不报告评估指标？
【发布时间】：2021-08-10 00:01:28
【问题描述】：

我正在学习本教程以了解培训师 API。 https://huggingface.co/transformers/training.html

我复制了如下代码：

from datasets import load_dataset

import numpy as np
from datasets import load_metric

metric = load_metric("accuracy")

def compute_metrics(eval_pred):
    logits, labels = eval_pred
    predictions = np.argmax(logits, axis=-1)
    return metric.compute(predictions=predictions, references=labels)

print('Download dataset ...')
raw_datasets = load_dataset("imdb")
from transformers import AutoTokenizer

print('Tokenize text ...')
tokenizer = AutoTokenizer.from_pretrained("bert-base-cased")
def tokenize_function(examples):
    return tokenizer(examples["text"], padding="max_length", truncation=True)
tokenized_datasets = raw_datasets.map(tokenize_function, batched=True)

print('Prepare data ...')
small_train_dataset = tokenized_datasets["train"].shuffle(seed=42).select(range(500))
small_eval_dataset = tokenized_datasets["test"].shuffle(seed=42).select(range(500))
full_train_dataset = tokenized_datasets["train"]
full_eval_dataset = tokenized_datasets["test"]

print('Define model ...')
from transformers import AutoModelForSequenceClassification
model = AutoModelForSequenceClassification.from_pretrained("bert-base-cased", num_labels=2)

print('Define trainer ...')
from transformers import TrainingArguments, Trainer
training_args = TrainingArguments("test_trainer", evaluation_strategy="epoch")
trainer = Trainer(
    model=model,
    args=training_args,
    train_dataset=small_train_dataset,
    eval_dataset=small_eval_dataset,
    compute_metrics=compute_metrics,
)

print('Fine-tune train ...')
trainer.evaluate()

但是，它没有报告任何关于训练指标的信息，而是以下消息：

Download dataset ...
Reusing dataset imdb (/Users/congminmin/.cache/huggingface/datasets/imdb/plain_text/1.0.0/4ea52f2e58a08dbc12c2bd52d0d92b30b88c00230b4522801b3636782f625c5b)
Tokenize text ...
100%|██████████| 25/25 [00:06<00:00,  4.01ba/s]
100%|██████████| 25/25 [00:06<00:00,  3.99ba/s]
100%|██████████| 50/50 [00:13<00:00,  3.73ba/s]
Prepare data ...
Define model ...
Some weights of the model checkpoint at bert-base-cased were not used when initializing BertForSequenceClassification: ['cls.seq_relationship.weight', 'cls.predictions.transform.LayerNorm.weight', 'cls.seq_relationship.bias', 'cls.predictions.transform.dense.bias', 'cls.predictions.bias', 'cls.predictions.decoder.weight', 'cls.predictions.transform.LayerNorm.bias', 'cls.predictions.transform.dense.weight']
- This IS expected if you are initializing BertForSequenceClassification from the checkpoint of a model trained on another task or with another architecture (e.g. initializing a BertForSequenceClassification model from a BertForPreTraining model).
- This IS NOT expected if you are initializing BertForSequenceClassification from the checkpoint of a model that you expect to be exactly identical (initializing a BertForSequenceClassification model from a BertForSequenceClassification model).
Some weights of BertForSequenceClassification were not initialized from the model checkpoint at bert-base-cased and are newly initialized: ['classifier.weight', 'classifier.bias']
You should probably TRAIN this model on a down-stream task to be able to use it for predictions and inference.
Define trainer ...
Fine-tune train ...
100%|██████████| 63/63 [08:35<00:00,  8.19s/it]

Process finished with exit code 0

教程不是更新了吗？我应该进行一些配置更改以报告指标吗？

【问题讨论】：

标签： python huggingface-transformers transformer

【解决方案1】：

评估函数返回指标，它不打印它们。有没有

metrics=trainer.evaluate()
print(metrics)

工作？此外，消息说您使用的是基础 bert 模型，该模型没有针对句子分类进行预训练，而是基础语言模型。因此它没有任务的初始化权重，应该进行训练

【讨论】：

嗨，Samer，但本教程正好用于微调预训练的语言模型，应该适用于句子分类。那正确吗？我从教程中粘贴的代码包含对句子分类的微调。

【解决方案2】：

你为什么要trainer.evaluate()？这只是在验证集上运行验证。如果你想微调或训练，你需要这样做：

trainer.train()

【讨论】：

【解决方案3】：

我认为您需要告诉培训师在TrainingArguments 中使用evaluation_strategy 和eval_steps 评估绩效的频率

【讨论】：

【解决方案4】：

关键是label，bert模型需要label字段是“labels”，所以要重命名column。

# check fields
print(next(iter(small_train_dataset)).keys())

# rename field
small_train_dataset = small_train_dataset.rename_column("label", "labels")
small_eval_dataset = small_eval_dataset.rename_column("label", "labels")

来自文档

将“标签”列重命名为“标签”（因为模型希望参数被命名为标签）

【讨论】：