哪个损失函数用于训练稀疏多标签文本分类问题和类偏度/不平衡

Question

我正在使用Hugging Face 模型训练一个稀疏的multi-label text classification 问题，这是SMART REPLY System 的一部分。我正在做的任务如下：

我将Customer Utterances 分类为模型的输入，并分类它属于哪个Agent Response 集群。我有60 集群，Customer Utterances 可以映射到一个或多个集群。

模型输入

Input                             Output

My account is blocked             [0,0,0,1,1,0....0,0,0,0,0]

输出是集群标签的编码向量。在上面的示例中，客户查询映射到代理响应的cluster 4 和cluster 5。

问题：

该模型总是预测非常频繁的簇数。它不需要稀有的集群。

输出标签中一次只有几个 1，其余为 0。

代码：

#Dividing the params into those which needs to be updated and rest
param_optimizer = list(model.named_parameters())
no_decay = ['bias', 'gamma', 'beta']
optimizer_grouped_parameters = [
    {
        'params': [p for n, p in param_optimizer if not any(nd in n for nd in no_decay)],
        'weight_decay_rate': 0.01
    },
    {
        'params': [p for n, p in param_optimizer if any(nd in n for nd in no_decay)],
        'weight_decay_rate': 0.0
    }
]

optimizer = BertAdam(optimizer_grouped_parameters, lr =0.05, warmup = .1)

模型训练

#Empty the GPU memory as it might be memory and CPU intensive while training
torch.cuda.empty_cache()
#Number of times the whole dataset will run through the network and model is fine-tuned
epochs = 10
epoch_count = 1
#Iterate over number of epochs
for _ in trange(epochs, desc = "Epoch"):
    #Switch model to train phase where it will update gradients
    model.train()
    #Initaite train and validation loss, number of rows passed and number of batches passed
    tr_loss = 0
    nb_tr_examples, nb_tr_steps = 0, 0
    val_loss = 0
    nb_val_examples, nb_val_steps = 0, 0
   
    #Iterate over batches within the same epoch
    for batch in tqdm(train_dataloader):
        #Shift the batch to GPU for computation
        #pdb.set_trace()
        batch = tuple(t.to(device) for t in batch)
        #Load the input ids and masks from the batch
        b_input_ids, b_input_mask, b_labels = batch
        #Initiate gradients to 0 as they tend to add up
        optimizer.zero_grad()
        #Forward pass the input data
        logits = model(b_input_ids, token_type_ids = None, attention_mask = b_input_mask)
        #We will be using the Binary Cross entropy loss with added sigmoid function after that in BCEWithLogitsLoss
        loss_func = BCEWithLogitsLoss()
        #Calculate the loss between multilabel predicted outputs and actuals
        loss = loss_func(logits, b_labels.type_as(logits))
        
        #Backpropogate the loss and calculate the gradients
        loss.backward()
        #Update the weights with the calculated gradients
        optimizer.step()
        #Add the loss of the batch to the final loss, number of rows and batches
        tr_loss += loss.item()
        nb_tr_examples += b_input_ids.size(0)
        nb_tr_steps += 1
    #Print the current training loss 
    print("Train Loss: {}".format(tr_loss/nb_tr_examples))
    
    # Save the trained model after each epoch.
#     pickle.dump(model, open("conv_bert_model_"+str(epoch_count)+".pkl", "wb"))
    epoch_count=epoch_count+1

我目前正在使用这个损失函数：

loss_func = BCEWithLogitsLoss()
#Calculate the loss between multilabel predicted outputs and actuals
loss = loss_func(logits, b_labels.type_as(logits))

有什么方法可以通过使用不同的损失函数来提高模型输出（召回率和精度）？

在 MULTI LABLES 分类的情况下，我们如何解决 Hugging 人脸模型中的聚类不平衡问题。

Answer 1

您可以对输出的每个索引使用加权交叉熵。您必须通过训练集来计算每个集群的权重。

criterion = nn.BCEWithLogitsLoss(reduction='none')
loss = criterion(output, target)
loss = (loss * weights).mean()
loss.backward()

通过这样做，不同索引的损失不会立即合并，而是分开保存。它们首先与权重相乘，然后合并。

要计算权重，假设输出是张量：

weights = torch.sum(outputs, 0)/torch.sum(outputs)

假设 numpy 数组：

weights = np.sum(outputs, 0)/np.sum(outputs)