如何创建基于批次而不是时期更新的 Keras 学习率计划

Question

我正在使用 Keras，并尝试创建一个学习率调度程序，该调度程序根据处理的批次数而不是 epoch 数进行调度。为此，我将调度代码插入到我的“优化器”的get_updates 方法中。在大多数情况下，我尝试将常规 Python 变量用于在给定训练运行期间保持不变的值，并将计算图节点仅用于实际变化的参数。

我的 2 个问题是：

1. 如果将下面的代码放在 Keras Optimizer 的 get_updates 方法中，它是否看起来应该像学习率调度程序一样正常运行。

2. 如何将此代码嵌入到类似于 LearningRateScheduler 的类中，但根据批次数而不是 epoch 数进行调度？

---

    #Copying graph node that stores original value of learning rate
    lr = self.lr 

    # Checking whether learning rate schedule is to be used
    if self.initial_lr_decay > 0:
        # this decay mimics exponential decay from 
        # tensorflow/python/keras/optimizer_v2/exponential_decay 

        # Get value of current number of processed batches from graph node
        # and convert to numeric value for use in K.pow()
        curr_batch = float(K.get_value(self.iterations))

        # Create graph node containing lr decay factor
        # Note: self.lr_decay_steps is a number, not a node
        #       self.lr_decay is a node, not a number
        decay_factor =  K.pow(self.lr_decay, (curr_batch / self.lr_decay_steps)) 

        # Reassign lr to graph node formed by
        # product of graph node containing decay factor
        # and graph node containing original learning rate.
        lr = lr * decay_factor

        # Get product of two numbers to calculate number of batches processed
        # in warmup period
        num_warmup_batches = self.steps_per_epoch_num * self.warmup_epochs

        # Make comparisons between numbers to determine if we're in warmup period
        if (self.warmup_epochs > 0) and (curr_batch < num_warmup_batches):

            # Create node with value of learning rate by multiplying a number
            # by a node, and then dividing by a number
            lr = (self.initial_lr  *
                  K.cast(self.iterations, K.floatx()) / curr_batch)

Answer 1

比弄乱 Keras 源代码更容易（这是可能的，但它既复杂又合理），您可以使用回调。

from keras.callbacks import LambdaCallback

total_batches = 0
def what_to_do_when_batch_ends(batch, logs):
   total_batches += 1 #or use the "batch" variable,
                      #which is the batch index of the last finished batch

   #change learning rate at will
   if your_condition == True:
       keras.backend.set_value(model.optimizer.lr, newLrValueAsPythonFloat)

训练时，使用回调：

lrUpdater = LambdaCallback(on_batch_end = what_to_do_when_batch_ends)
model.fit(........, callbacks = [lrUpdater, ...other callbacks...])