如何解决我在pytorch中的max_step参数问题？答案

【问题标题】：How to solve my problem of max_step parameter in pytorch?如何解决我在pytorch中的max_step参数问题？
【发布时间】：2023-01-03 05:30:20
【问题描述】：

我正在尝试训练源代码。

class mymodel(pl.LightningModule):
  def __init__(self, config , learning_rate = 1e-4, max_steps = 100000//2):
    super(mymodel, self).__init__()
    
    self.config = config
    self.save_hyperparameters()
    self.training_losses = []
    self.validation_losses = []
    self.max_steps = max_steps

  def configure_optimizers(self):
    return torch.optim.AdamW(self.parameters(), lr = self.hparams['learning_rate'])


  def forward(self, batch_dict):
    return answer_vector

  def calculate_metrics(self, prediction, labels):
      batch_size = len(prediction)
      ac_score = 0

      for (pred, gt) in zip(prediction, labels):
        ac_score+= calculate_acc_score(pred.detach().cpu(), gt.detach().cpu())
      ac_score = ac_score/batch_size
      return ac_score

  def training_step(self, batch, batch_idx):
    answer_vector = self.forward(batch)
    loss = nn.CrossEntropyLoss()(answer_vector.reshape(-1,self.config['classes']), batch['answer'].reshape(-1))
    _, preds = torch.max(answer_vector, dim = -1)

    train_acc = self.calculate_metrics(preds, batch['answer'])
    train_acc = torch.tensor(train_acc)

    return loss

  def validation_step(self, batch, batch_idx):
    logits = self.forward(batch)
    loss = nn.CrossEntropyLoss()(logits.reshape(-1,self.config['classes']), batch['answer'].reshape(-1))
    _, preds = torch.max(logits, dim = -1)

    ## Validation Accuracy
    val_acc = self.calculate_metrics(preds.cpu(), batch['answer'].cpu())
    val_acc = torch.tensor(val_acc)

    ## Logging
    self.log('val_ce_loss', loss, prog_bar = True)
    self.log('val_acc', val_acc, prog_bar = True)
    
    return {'val_loss': loss, 'val_acc': val_acc}

  def optimizer_step(self, epoch_nb, batch_nb, optimizer, optimizer_i, opt_closure = None, on_tpu=False,
    using_native_amp=False, using_lbfgs=False):

        ## Warmup for 1000 steps
        if self.trainer.global_step < 1000:
            lr_scale = min(1., float(self.trainer.global_step + 1) / 1000.)
            for pg in optimizer.param_groups:
                pg['lr'] = lr_scale * self.hparams.learning_rate

        ## Linear Decay
        else:
            for pg in optimizer.param_groups:
                pg['lr'] = polynomial(self.hparams.learning_rate, self.trainer.global_step, max_iter = self.max_steps)

        optimizer.step(opt_closure)
        optimizer.zero_grad()

在第 5 个时期（可能更少或更多），我遇到了停止训练的错误。所以我增加了 max_step 。但是当我增加 max_step(max_step==100K) 时，我遇到了损失问题，并认为 loss>100 && acc==0 。我附上这个问题的屏幕。

enter image description here

在没有这个问题的情况下，我应该在源代码中做些什么改变来继续训练模型？

【问题讨论】：

标签： machine-learning pytorch pytorch-lightning

【解决方案1】：

错误的是，您是指val_ce_loss 行吗？如果是，那不是错误。这意味着当前 epoch 的 val_ce_loss 不在历史 epoch 的 top1 之内，因此检查点不会保存到磁盘。请参考检查点回调。 https://pytorch-lightning.readthedocs.io/en/stable/api/pytorch_lightning.callbacks.ModelCheckpoint.html

【讨论】：