使用 huggingface transformers trainer 方法来拥抱人脸数据集

Question

我正在尝试使用拥抱脸训练器方法和拥抱脸数据集（即“eth_py150_open”）来训练变压器（Salesforce codet5-small）。但是，我遇到了一些问题。

以下是相关代码sn-p：

import torch
import transformers
from datasets import load_dataset_builder
from datasets import load_dataset

corpus=load_dataset("eth_py150_open", split='train')

training_args = transformers.TrainingArguments( #general training arguments
    per_device_train_batch_size = 8,
    warmup_steps = 0,
    weight_decay = 0.01,
    learning_rate = 1e-4,
    num_train_epochs = 12,
    output_dir = './runs/run2/output/',
    logging_dir = './runs/run2/logging/',
    logging_steps = 50,
    save_steps= 10000,
    remove_unused_columns=False,
)

model = transformers.T5ForConditionalGeneration.from_pretrained('Salesforce/codet5-small').cuda()

trainer = transformers.Trainer(
    model = model,
   args = training_args,
    train_dataset = corpus,
)

但是，在运行 trainer.train() 时，出现以下错误：

***** Running training *****
  Num examples = 74749
  Num Epochs = 12
  Instantaneous batch size per device = 8
  Total train batch size (w. parallel, distributed & accumulation) = 8
  Gradient Accumulation steps = 1
  Total optimization steps = 112128
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-28-3435b262f1ae> in <module>
----> 1 trainer.train()

3 frames
/usr/local/lib/python3.7/dist-packages/transformers/trainer.py in _prepare_inputs(self, inputs)
   2414         if len(inputs) == 0:
   2415             raise ValueError(
-> 2416                 "The batch received was empty, your model won't be able to train on it. Double-check that your "
   2417                 f"training dataset contains keys expected by the model: {','.join(self._signature_columns)}."
   2418             )

TypeError: can only join an iterable

我尝试将语料库转换为 Torch Dataset 对象，但似乎无法弄清楚如何做到这一点。我真的很感激任何帮助！

Answer 1

您需要先标记数据集，然后才能将其传递给模型。下面我添加了一个preprocess() 函数来标记化。您还需要data_collator 来整理标记化的序列。由于 T5 是 seq2seq 模型，我猜您正在尝试生成许可证字符串，因此我已将 Trainer 替换为 Seq2SeqTrainer。 （虽然我认为如果你认为这是一个序列分类任务会更好）。这是您更新的脚本。

import torch
import transformers
from datasets import load_dataset

corpus=load_dataset("eth_py150_open", split='train')

def preprocess(examples):
    model_inputs = tokenizer(examples['filepath'], truncation=True)
    with tokenizer.as_target_tokenizer():
        labels = tokenizer(examples['license'], truncation=True)
    model_inputs['labels'] = labels['input_ids']
    return model_inputs

tokenized_dataset = corpus.map(preprocess_function, batched=True)

training_args = transformers.Seq2SeqTrainingArguments( #general training arguments
    per_device_train_batch_size = 8,
    warmup_steps = 0,
    weight_decay = 0.01,
    learning_rate = 1e-4,
    num_train_epochs = 12,
    output_dir = './runs/run2/output/',
    logging_dir = './runs/run2/logging/',
    logging_steps = 50,
    save_steps= 10000,
    remove_unused_columns=False,
)

model = transformers.T5ForConditionalGeneration.from_pretrained('Salesforce/codet5-small').cuda()
data_collator = transformers.DataCollatorForSeq2Seq(tokenizer, model=model)

trainer = transformers.Seq2SeqTrainer(
    model = model,
    args = training_args,
    train_dataset = tokenized_dataset,
    data_collator=data_collator
)