输出对带有拥抱脸/变形金刚（火炬）的 bert-base-uncased 的关注

Question

我正在关注a paper 基于 BERT 的词法替换（特别是尝试实现等式 (2) - 如果有人已经实现了整篇论文，那也很棒）。因此，我想同时获得最后的隐藏层（我唯一不确定的是输出中层的顺序：最后一个还是第一个？）以及来自基本 BERT 模型（bert-base-uncased）的注意力。

但是，我有点不确定 huggingface/transformers library 是否真的会为 bert-base-uncased 输出注意力（我使用的是 Torch，但我愿意使用 TF 代替）？

从what I had read 开始，我应该得到一个 (logits, hidden_​​states, attentions) 的元组，但在下面的示例中（例如在 Google Colab 中运行），我得到的长度为 2。

我是否以错误的方式误解了我的理解或处理方式？我做了明显的测试并使用output_attention=False 而不是output_attention=True（而output_hidden_states=True 确实似乎添加了隐藏状态，正如预期的那样）并且我得到的输出没有任何变化。这显然是我对图书馆理解的一个不好的迹象，或者表明存在问题。

import numpy as np
import torch
!pip install transformers

from transformers import (AutoModelWithLMHead, 
                          AutoTokenizer, 
                          BertConfig)

bert_tokenizer = AutoTokenizer.from_pretrained("bert-base-uncased")
config = BertConfig.from_pretrained('bert-base-uncased', output_hidden_states=True, output_attention=True) # Nothign changes, when I switch to output_attention=False
bert_model = AutoModelWithLMHead.from_config(config)

sequence = "We went to an ice cream cafe and had a chocolate ice cream."
bert_tokenized_sequence = bert_tokenizer.tokenize(sequence)

indexed_tokens = bert_tokenizer.encode(bert_tokenized_sequence, return_tensors='pt')

predictions = bert_model(indexed_tokens)

########## Now let's have a look at what the predictions look like #############
print(len(predictions)) # Length is 2, I expected 3: logits, hidden_layers, attention

print(predictions[0].shape) # torch.Size([1, 16, 30522]) - seems to be logits (shape is 1 x sequence length x vocabulary

print(len(predictions[1])) # Length is 13 - the hidden layers?! There are meant to be 12, right? Is one somehow the attention?

for k in range(len(predictions[1])):
  print(predictions[1][k].shape) # These all seem to be torch.Size([1, 16, 768]), so presumably the hidden layers?

解释最终受接受答案启发的工作原理

import numpy as np
import torch
!pip install transformers

from transformers import BertModel, BertConfig, BertTokenizer

tokenizer = BertTokenizer.from_pretrained('bert-base-uncased')
config = BertConfig.from_pretrained('bert-base-uncased', output_hidden_states=True, output_attentions=True)
model = BertModel.from_pretrained('bert-base-uncased', config=config)
sequence = "We went to an ice cream cafe and had a chocolate ice cream."
tokenized_sequence = tokenizer.tokenize(sequence)
indexed_tokens = tokenizer.encode(tokenized_sequence, return_tensors='pt'
enter code here`outputs = model(indexed_tokens)
print( len(outputs) ) # 4 
print( outputs[0].shape ) #1, 16, 768 
print( outputs[1].shape ) # 1, 768
print( len(outputs[2]) ) # 13  = input embedding (index 0) + 12 hidden layers (indices 1 to 12)
print( outputs[2][0].shape ) # for each of these 13: 1,16,768 = input sequence, index of each input id in sequence, size of hidden layer
print( len(outputs[3]) ) # 12 (=attenion for each layer)
print( outputs[3][0].shape ) # 0 index = first layer, 1,12,16,16 = , layer, index of each input id in sequence, index of each input id in sequence

Answer 1

我认为现在在这里回答为时已晚，但是随着拥抱脸的变形金刚的更新，我认为我们可以使用它

config = BertConfig.from_pretrained('bert-base-uncased', 
output_hidden_states=True, output_attentions=True)  
bert_model = BertModel.from_pretrained('bert-base-uncased', 
config=config)

with torch.no_grad():
  out = bert_model(input_ids)
  last_hidden_states = out.last_hidden_state
  pooler_output = out.pooler_output
  hidden_states = out.hidden_states
  attentions = out.attentions

Answer 2

原因是您使用的是AutoModelWithLMHead，它是实际模型的包装器。它调用 BERT 模型（即BERTModel 的实例），然后使用嵌入矩阵作为词预测的权重矩阵。在底层模型之间确实返回了注意力，但包装器并不关心，只返回 logits。

您可以通过调用 AutoModel 直接获取 BERT 模型。请注意，此模型不返回 logits，而是返回隐藏状态。

bert_model = AutoModel.from_config(config)

或者您可以通过调用从BertWithLMHead 对象获取它：

wrapped_model = bert_model.base_model