如何使用长模型生成句子嵌入

Question

我正在使用 Hugging Face mrm8488/longformer-base-4096-finetuned-squadv2 预训练模型 https://huggingface.co/mrm8488/longformer-base-4096-finetuned-squadv2.

我想生成句子级嵌入。我有一个包含文本列的数据框。

我正在使用此代码：

import torch
from transformers import AutoTokenizer, AutoModelForQuestionAnswering
ckpt = "mrm8488/longformer-base-4096-finetuned-squadv2"
tokenizer = AutoTokenizer.from_pretrained(ckpt)
model = AutoModelForQuestionAnswering.from_pretrained(ckpt)

text = "Huggingface has democratized NLP. Huge thanks to Huggingface for this." # I will pas text-column here from my data-frame
#question = "What has Huggingface done ?"
encoding = tokenizer(question, text, return_tensors="pt")
# I don't want to use it for Question-Answer use-case. I just need the sentence embeddings
input_ids = encoding["input_ids"]

# default is local attention everywhere
# the forward method will automatically set global attention on question tokens
attention_mask = encoding["attention_mask"] 

如何在上面的代码中进行修改以生成句子的嵌入。 ?

我有以下例子：

                           Text
i've added notes to the claim and it's been escalated for final review
after submitting the request you'll receive an email confirming the open request.
hello my name is person and i'll be assisting you
this is sam and i'll be assisting you for date.
I'll return the amount as asap.
ill return it to you.

Answer 1

Longformer 使用local attention mechanism，您需要传递global attention mask 以让一个令牌处理您序列中的所有令牌。

import torch
from transformers import LongformerTokenizer, LongformerModel
ckpt = "mrm8488/longformer-base-4096-finetuned-squadv2"
tokenizer = LongformerTokenizer.from_pretrained(ckpt)
model = LongformerModel.from_pretrained(ckpt)

text = "Huggingface has democratized NLP. Huge thanks to Huggingface for this." # I will pas text-column here from my data-frame
#question = "What has Huggingface done ?"
encoding = tokenizer(text, return_tensors="pt")

global_attention_mask = [1].extend([0]*encoding["input_ids"].shape[-1])

encoding["global_attention_mask"] = global_attention_mask

# I don't want to use it for Question-Answer use-case. I just need the sentence embeddings
# default is local attention everywhere
# the forward method will automatically set global attention on question tokens

o = model(**encoding)

sentence_embedding = o.last_hidden_state[:,0]

您应该记住，mrm8488/longformer-base-4096-finetuned-squadv2 没有经过预训练以产生有意义的句子嵌入，并且在句子嵌入方面面对相同的 issues as the MLM pre-trained BERT's。