【发布时间】：2018-09-19 02:45:27
【问题描述】：
在我在 Google ML Engine 的微调工作中，一些训练配置会导致 NaN 损失，从而导致错误。我希望能够忽略这些试验，并继续使用不同的参数进行微调。
我正在使用带有 fail_on_nan_loss=False 的 NanTensorHook，它在没有执行并行试验时在 ML Engine 中成功运行（maxParallelTrials：1），但在多个并行试验中失败（maxParallelTrials：3）。
以前有人遇到过这个错误吗？关于如何解决它的任何想法？
这是我的配置文件：
trainingInput:
scaleTier: CUSTOM
masterType: standard
workerType: standard
parameterServerType: standard
workerCount: 4
parameterServerCount: 1
hyperparameters:
goal: MAXIMIZE
maxTrials: 5
maxParallelTrials: 3
enableTrialEarlyStopping: False
hyperparameterMetricTag: auc
params:
- parameterName: learning_rate
type: DOUBLE
minValue: 0.0001
maxValue: 0.01
scaleType: UNIT_LOG_SCALE
- parameterName: optimizer
type: CATEGORICAL
categoricalValues:
- Adam
- Adagrad
- Momentum
- SGD
- parameterName: batch_size
type: DISCRETE
discreteValues:
- 128
- 256
- 512
这就是我设置 NanTensorHook 的方式：
hook = tf.train.NanTensorHook(loss,fail_on_nan_loss=False)
train_op = tf.contrib.layers.optimize_loss(
loss=loss, global_step=tf.train.get_global_step(),
learning_rate=lr, optimizer=optimizer)
model_fn = tf.estimator.EstimatorSpec(mode=mode, loss=loss,
eval_metric_ops=eval_metric_ops, train_op=train_op,
training_hooks=[hook])
我得到的错误信息是：
Hyperparameter Tuning Trial #4 Failed before any other successful
trials were completed. The failed trial had parameters: optimizer=SGD,
batch_size=128, learning_rate=0.00075073617775056709, . The trial's ror
message was: The replica worker 1 exited with a non-zero status of 1.
Termination reason: Error. Traceback (most recent call last): [...]
File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 421, in
train_and_evaluate executor.run() File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 522, in run
getattr(self, task_to_run)() File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 532, in
run_worker return self._start_distributed_training() File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 715, in
_start_distributed_training saving_listeners=saving_listeners) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/estimator.py", line 352, in train
loss = self._train_model(input_fn, hooks, saving_listeners) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/estimator.py", line 891, in
_train_model _, loss = mon_sess.run([estimator_spec.train_op,
estimator_spec.loss]) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 546, in
run run_metadata=run_metadata) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1022,
in run run_metadata=run_metadata) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1113,
in run raise six.reraise(*original_exc_info) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1098,
in run return self._sess.run(*args, **kwargs) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1178,
in run run_metadata=run_metadata)) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/basic_session_run_hooks.py", line
617, in after_run raise NanLossDuringTrainingError
NanLossDuringTrainingError: NaN loss during training. The replica
worker 3 exited with a non-zero status of 1. Termination reason: Error.
Traceback (most recent call last): [...] File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 421, in
train_and_evaluate executor.run() File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 522, in run
getattr(self, task_to_run)() File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 532, in
run_worker return self._start_distributed_training() File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/training.py", line 715, in
_start_distributed_training saving_listeners=saving_listeners) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/estimator.py", line 352, in train
loss = self._train_model(input_fn, hooks, saving_listeners) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/estimator/estimator.py", line 891, in
_train_model _, loss = mon_sess.run([estimator_spec.train_op,
estimator_spec.loss]) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 546, in
run run_metadata=run_metadata) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1022,
in run run_metadata=run_metadata) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1113,
in run raise six.reraise(*original_exc_info) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1098,
in run return self._sess.run(*args, **kwargs) File
"/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/monitored_session.py", line 1178,
in run run_metadata=run_metadata)) File "/usr/local/lib/python2.7/dist-
packages/tensorflow/python/training/basic_session_run_hooks.py", line
617, in after_run raise NanLossDuringTrainingError
NanLossDuringTrainingError: NaN loss during training.
提前谢谢大家！
【问题讨论】：
-
我怀疑将 maxParallelTrials 设置为大于 1 可能会激活 Tensrflow 中的另一个钩子，这可能会导致排序问题并阻止钩子（例如 tf_debug.LocalCLIDebugHook）在 nans 出现在网络中时运行。 NanTensorHook 导致程序在这些钩子的 after_run() 方法运行之前崩溃。请提供你得到的错误，并确认错误前loss的最后一个值不是NaN。
-
您好，Shahin，感谢您的回复。损失确实得到了 NaN，这就是我想设置 NaNTensorHook 的原因。我在日志中添加了错误。
-
删除 Tuner DIR 中的 ActiveWorkers SubDIR 进行试用，然后在新试用开始后重试失败工作人员的 TF 作业可能会解决问题。其他可以解决的方法可能是更改优化器（例如到 adagrad），这可能会避免 NaN 值。
-
我个人怀疑 NanTensorHook 源代码存在内部问题，无法支持多次试验。换句话说，由于每条路径仅从已完成的试验中获得的信息中受益，而无法访问同时运行的试验结果，因此它们可能在本地面临一些 NaN 值，这些值无法通过 NaNtensorhook 解决.换句话说，NaNtensorhook 只检查已完成的试验（总损失函数）。
-
感谢您的洞察力，很遗憾 NanTensorHook 在这种情况下失败，我将尝试修复 NanHook 以监听每个模型丢失。干杯
标签： tensorflow machine-learning nan google-cloud-ml