在训练 BERT 变体时，出现 IndexError: index out of range in self答案

【问题标题】：While training BERT variant, getting IndexError: index out of range in self在训练 BERT 变体时，出现 IndexError: index out of range in self
【发布时间】：2023-01-25 02:41:17
【问题描述】：

在训练XLMRobertaForSequenceClassification时：

xlm_r_model(input_ids = X_train_batch_input_ids
            , attention_mask = X_train_batch_attention_mask
            , return_dict = False
           )

我遇到以下错误：

Traceback (most recent call last):
  File "<string>", line 3, in <module>
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 1218, in forward
    return_dict=return_dict,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 849, in forward
    past_key_values_length=past_key_values_length,
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/transformers/models/roberta/modeling_roberta.py", line 132, in forward
    inputs_embeds = self.word_embeddings(input_ids)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py", line 1102, in _call_impl
    return forward_call(*input, **kwargs)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/modules/sparse.py", line 160, in forward
    self.norm_type, self.scale_grad_by_freq, self.sparse)
  File "/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py", line 2044, in embedding
    return torch.embedding(weight, input, padding_idx, scale_grad_by_freq, sparse)
IndexError: index out of range in self

以下是详细信息：

创建模型

config = XLMRobertaConfig() 
config.output_hidden_states = False
xlm_r_model = XLMRobertaForSequenceClassification(config=config)
xlm_r_model.to(device) # device is device(type='cpu')

分词器

xlmr_tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-large')

MAX_TWEET_LEN = 402

>>> df_1000.info() # describing a data frame I have pre populated
<class 'pandas.core.frame.DataFrame'>
Int64Index: 1000 entries, 29639 to 44633
Data columns (total 2 columns):
#    Column  Non-Null Count  Dtype 
---  ------  --------------  ----- 
0    text    1000 non-null   object
1    class   1000 non-null   int64 
dtypes: int64(1), object(1)
memory usage: 55.7+ KB

X_train = xlmr_tokenizer(list(df_1000[:800].text), padding=True, max_length=MAX_TWEET_LEN+5, truncation=True) # +5: a head room for special tokens / separators
>>> list(map(len,X_train['input_ids']))  # why its 105? shouldn't it be MAX_TWEET_LEN+5 = 407?
[105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, 105, ...]

>>> type(train_index) # describing (for clarity) training fold indices I pre populated
<class 'numpy.ndarray'>

>>> train_index.size 
640

X_train_fold_input_ids = np.array(X_train['input_ids'])[train_index]
X_train_fold_attention_mask = np.array(X_train['attention_mask'])[train_index]

>>> i # batch id
0
>>> batch_size
16

X_train_batch_input_ids = X_train_fold_input_ids[i:i+batch_size]
X_train_batch_input_ids = torch.tensor(X_train_batch_input_ids,dtype=torch.long).to(device)

X_train_batch_attention_mask = X_train_fold_attention_mask[i:i+batch_size]
X_train_batch_attention_mask = torch.tensor(X_train_batch_attention_mask,dtype=torch.long).to(device)

>>> X_train_batch_input_ids.size()
torch.Size([16, 105]) # why 105? Shouldnt this be MAX_TWEET_LEN+5 = 407?

>>> X_train_batch_attention_mask.size()
torch.Size([16, 105]) # why 105? Shouldnt this be MAX_TWEET_LEN+5 = 407?

在此之后，我按照此问题开头所述拨打电话xlm_r_model(...)，并以指定的错误结束。

注意到所有这些细节，我仍然无法理解为什么我会收到指定的错误。我哪里做错了？

【问题讨论】：

标签： python pytorch huggingface-transformers bert-language-model transformer-model

【解决方案1】：

根据 github 上的这篇文章，可能有很多原因。以下是从该帖子中总结的原因列表（截至 2022 年 4 月 24 日，请注意第二和第三个原因未经过测试）：

分词器和 bert 模型的词汇量不匹配。这将导致分词器生成模型无法理解的 ID。 ref
存在于不同设备（CPU、GPU、TPU）上的模型和数据ref
长度超过 512 的序列（这是类 BERT 模型的最大值）ref
就我而言，这是第一个原因，词汇大小不匹配，我已按如下方式解决此问题：

这是我解决这个问题的方法：
```
xlmr_tokenizer = XLMRobertaTokenizer.from_pretrained('xlm-roberta-large')
config = XLMRobertaConfig() 
config.vocab_size = xlmr_tokenizer.vocab_size  # setting both to have same vocab size
```

【讨论】：

【解决方案2】：

我的问题是 3. 长度超过 512 的序列（这是 BERT 类模型的最大值）请问谁解决了这个问题？我通过 config.max_embeddings_input = 1024 在 markupLM 配置中进行了更改，但徒劳无功：/

【讨论】：