如何在 GradientTape 中将“步长”传递给指数衰减答案

【问题标题】：How to pass "step" to ExponentialDecay in GradientTape如何在 GradientTape 中将“步长”传递给指数衰减
【发布时间】：2019-05-15 02:52:51
【问题描述】：

我尝试使用 optimizers.schedules.ExponentialDecay 作为 Adm 优化器的 learning_rate，但在 GradientTape 中训练模型时我不知道如何将“步骤”传递给它。

我使用 tensorflow-gpu-2.0-alpha0 和 python3.6。我阅读了文档https://tensorflow.google.cn/versions/r2.0/api_docs/python/tf/optimizers/schedules/ExponentialDecay，但不知道如何解决它。

initial_learning_rate = 0.1
lr_schedule = tf.keras.optimizers.schedules.ExponentialDecay(
    initial_learning_rate,
    decay_steps=100000,
    decay_rate=0.96)

optimizer = tf.optimizers.Adam(learning_rate = lr_schedule)

for epoch in range(self.Epoch):
    ...
    ...
    with GradientTape as tape:
        pred_label = model(images)
        loss = calc_loss(pred_label, ground_label)
    grads = tape.gradient(loss, model.trainable_variables)
    optimizer.apply_gradients(zip(grads, model.trainable_variables))

# I tried this but the result seem not right.
# I want to pass "epoch" as "step" to lr_schedule

【问题讨论】：

标签： python tensorflow2.0

【解决方案1】：

使用tf.keras.optimizers.schedules.ExponentialDecay()的实例可能不适用于GadientTape，它更适合Keras的model.fit()，我的理解是您需要在一定数量的迭代/步骤后降低或安排学习率，所以有一个解决方法，您可以使用优化器类的get_config() 和from_config() 方法手动安排学习率。


def exponential_decay(optimizer, decay_rate):
    #get the optimizer configuration dictionary
    opt_cfg = optimizer.get_config() 
    """ The opt_cfg dictionary will look like this given that if you set initial learning rate to 0.1.
        {'name': 'Adam', 'learning_rate': 0.1, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 
        'epsilon': 1e-07, 'amsgrad': False}"""

    #change the value of learning rate by multiplying decay rate with learning rate to get new learning rate
    opt_cfg['learning_rate'] = opt_cfg['learning_rate']*decay_rate
    """ the changed opt_cfg dictionary will look like this, if you have initial learning rate of 0.1 and decay rate of 0.96,
        then new opt_cfg will look like this: 
        {'name': 'Adam', 'learning_rate': 0.096, 'decay': 0.0, 'beta_1': 0.9, 'beta_2': 0.999, 
        'epsilon': 1e-07, 'amsgrad': False}"""
    #now just pass this updated optimizer configuartion dictionary to from_config() method and you are done, now
    #your optimzer will use new learning rate
    optimizer = optimizer.from_config(opt_cfg)
    return optimizer

decay_steps=100000
decay_rate=0.96
optimizer = tf.optimizers.Adam(learning_rate = lr_schedule)
#loop for epoch
for epoch in range(self.Epoch):
    ...
    ...
    #loop to iterate over batches
    for itr, images in enumerate(batch_images):
        if decay_steps%itr==0:
            optimizer = exponential_decay(optimizer, decay_rate)
        with GradientTape as tape:
            pred_label = model(images)
            loss = calc_loss(pred_label, ground_label)
        grads = tape.gradient(loss, model.trainable_variables)
        optimizer.apply_gradients(zip(grads, model.trainable_variables))

所以我们在这里所做的是，当迭代/步数达到我们定义的限制或衰减步数时，假设您想在每 100K 步后更改学习率，我们调用我们的方法 exponential_deacy() 传入实例我们的优化器和衰减率，该方法将改变我们优化器的学习率并返回优化器实例，使用更新的学习率，您可以使用optimizer.get_config() 方法进行验证。更多信息请查看get_config() & from_config()。

【讨论】：