Keras ReduceLROnPlateau - 如何检查它是否被应用答案

【问题标题】：Keras ReduceLROnPlateau - How to check if it is appliedKeras ReduceLROnPlateau - 如何检查它是否被应用
【发布时间】：2021-09-11 15:28:51
【问题描述】：

请告知如何确认是否实际应用了 ReduceLROnPlateau，以及每个 epoch 应用了哪些学习率。

ReduceLROnPlateau 的耐心设置为 2 并监控var_loss。

self._history = self.model.fit(
    self.X.shuffle(1000).batch(self.batch_size).prefetch(1),
    epochs=self.num_epochs,
    batch_size=self.batch_size,
    validation_data=self.V.shuffle(1000).batch(self.batch_size).prefetch(1),
    callbacks=[
        EaryStoppingCallback(patience=self.early_stop_patience),
        ReduceLRCallback(patience=self.reduce_lr_patience),     # <---- set to 2
        TensorBoardCallback(self.log_directory)
    ]
)

class ReduceLRCallback(tf.keras.callbacks.ReduceLROnPlateau):
    """Reduce learning rate when a metric has stopped improving.
    See https://keras.io/api/callbacks/reduce_lr_on_plateau/
    """
    def __init__(self, patience=3):
        assert patience > 0
        super().__init__(
            monitor="val_loss",
            factor=0.3,
            patience=patience,
        )

训练 val_loss 增加了两倍以上，但如果应用了 ReduceLROnPlateau，则没有看到任何信息。

_________________________________________________________________
Epoch 1/20
3990/3990 [==============================] - 860s 214ms/step - loss: 0.1705 - accuracy: 0.9386 - val_loss: 0.1626 - val_accuracy: 0.9456
Epoch 2/20
3990/3990 [==============================] - 847s 212ms/step - loss: 0.1618 - accuracy: 0.9412 - val_loss: 0.1433 - val_accuracy: 0.9456
Epoch 3/20
3990/3990 [==============================] - 846s 212ms/step - loss: 0.1593 - accuracy: 0.9425 - val_loss: 0.1478 - val_accuracy: 0.9438
Epoch 4/20
3990/3990 [==============================] - 846s 212ms/step - loss: 0.1567 - accuracy: 0.9427 - val_loss: 0.1428 - val_accuracy: 0.9468
Epoch 5/20
3990/3990 [==============================] - 846s 212ms/step - loss: 0.1558 - accuracy: 0.9425 - val_loss: 0.1502 - val_accuracy: 0.9425
Epoch 6/20
3990/3990 [==============================] - 843s 211ms/step - loss: 0.1554 - accuracy: 0.9433 - val_loss: 0.1453 - val_accuracy: 0.9456
Epoch 7/20
3990/3990 [==============================] - 843s 211ms/step - loss: 0.1482 - accuracy: 0.9454 - val_loss: 0.1362 - val_accuracy: 0.9477
Epoch 8/20
3990/3990 [==============================] - 843s 211ms/step - loss: 0.1475 - accuracy: 0.9449 - val_loss: 0.1373 - val_accuracy: 0.9471
Epoch 9/20
3990/3990 [==============================] - 845s 212ms/step - loss: 0.1468 - accuracy: 0.9460 - val_loss: 0.1362 - val_accuracy: 0.9485
Epoch 10/20
3990/3990 [==============================] - 843s 211ms/step - loss: 0.1448 - accuracy: 0.9462 - val_loss: 0.1344 - val_accuracy: 0.9489
Epoch 11/20
3990/3990 [==============================] - 846s 212ms/step - loss: 0.1447 - accuracy: 0.9458 - val_loss: 0.1346 - val_accuracy: 0.9483
Epoch 12/20
3990/3990 [==============================] - 843s 211ms/step - loss: 0.1444 - accuracy: 0.9460 - val_loss: 0.1342 - val_accuracy: 0.9483

【问题讨论】：

您是否尝试过使用 verbose=1 的标准 ReduceLROnPlateau？
@MarcoCerliani，感谢您指出。我没有。

标签： tensorflow keras huggingface-transformers

【解决方案1】：

Marco Cerliani 的回答可能会为您解决。这会提供控制台输出，因此您可以验证 LR 是否确实减少了。

如果需要在代码中查看模型的学习率，可以使用

tf.keras.backend.get_value(model.optimizer.lr)

【讨论】：