如何在pytorch中的BERT多语言之上创建RNN层答案

【问题标题】：How to create RNN layer on top of BERT multilingual in pytorch如何在pytorch中的BERT多语言之上创建RNN层
【发布时间】：2022-06-12 16:35:26
【问题描述】：

我正在研究一个分类问题。我想将 BERT 嵌入传递给 RNN 层，然后在最后的 FCN 层进行分类。但是我遇到了一些问题，有没有人解决过同样的问题。

我创建了这个类如下

class BERTClass(torch.nn.Module):
    def __init__(self):
        super(BERTClass, self).__init__()
        self.l1 = BertModel.from_pretrained('bert-base-multilingual-cased', return_dict=False)
        # for param in self.l1.parameters():
        #   param.requires_grad = False
        self.l2 = torch.nn.Dropout(0.4)
        self.l3 = torch.nn.RNN(768, 1028)
        self.activation = torch.nn.ReLU()
        self.l4 = torch.nn.Dropout(0.2)
        self.l5 = torch.nn.Linear(1028, 128)
        self.activation2 = torch.nn.ReLU()
        self.l6 = torch.nn.Linear(128, 10)
        
    
    def forward(self, ids, mask, token_type_ids):
        _, output_1= self.l1(ids, attention_mask = mask, token_type_ids = token_type_ids)
        output_2 = self.l2(output_1)
        output3 = self.l3(output_2)
        act = self.activation(output3)
        output4 = self.l4(act)
        output5 = self.l5(output4)
        act2 = self.activation2(output5)
        output6 = self.l6(act2)
        return output6

model = BERTClass()

但我遇到了一个错误

<ipython-input-23-bbe09bd88901> in forward(self, ids, mask, token_type_ids)
     22         output_2 = self.l2(output_1)
     23         output3 = self.l3(output_2)
---> 24         act = self.activation(output3)
     25         output4 = self.l4(act)
     26         output5 = self.l5(output4)

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/module.py in _call_impl(self, *input, **kwargs)
   1108         if not (self._backward_hooks or self._forward_hooks or self._forward_pre_hooks or _global_backward_hooks
   1109                 or _global_forward_hooks or _global_forward_pre_hooks):
-> 1110             return forward_call(*input, **kwargs)
   1111         # Do not call functions when jit is used
   1112         full_backward_hooks, non_full_backward_hooks = [], []

/usr/local/lib/python3.7/dist-packages/torch/nn/modules/activation.py in forward(self, input)
     96 
     97     def forward(self, input: Tensor) -> Tensor:
---> 98         return F.relu(input, inplace=self.inplace)
     99 
    100     def extra_repr(self) -> str:

/usr/local/lib/python3.7/dist-packages/torch/nn/functional.py in relu(input, inplace)
   1440         result = torch.relu_(input)
   1441     else:
-> 1442         result = torch.relu(input)
   1443     return result
   1444 

TypeError: relu(): argument 'input' (position 1) must be Tensor, not tuple

【问题讨论】：

标签： pytorch multilingual recurrent-neural-network huggingface-transformers

【解决方案1】：

您必须注意RNN 层的输出。

Outputs: output, h_n

h_n 可能是您想要用于您的网络的内容。此外，默认情况下，torch RNN 不是 batch_first。

【讨论】：

【解决方案2】：

torch.nn.RNN 的输出是一个形状为(output, h_n) 的元组（有关此层的更多信息，请访问此link）
所以activation的输入应该只是RNN输出的第一个元素（output3[0]）。

最终代码为：

def forward(self, ids, mask, token_type_ids):
    _, output_1= self.l1(ids, attention_mask = mask, token_type_ids = token_type_ids)
    output_2 = self.l2(output_1)
    output3 = self.l3(output_2)
    act = self.activation(output3[0])
    output4 = self.l4(act)
    output5 = self.l5(output4)
    act2 = self.activation2(output5)
    output6 = self.l6(act2)
    return output6

【讨论】：