TensorFlow：去除累积梯度中的 nans

Question

对于函数逼近问题，我试图累积梯度，但我发现有时这些梯度中的一些是 nan（即未定义），即使损失总是真实的。我认为这可能是由于数值不稳定性，我基本上是在寻找一种简单的方法来从计算的梯度中删除 nans。

从solution to this question 开始，我尝试执行以下操作：

# Optimizer definition - nothing different from any classical example
opt = tf.train.AdamOptimizer()

## Retrieve all trainable variables you defined in your graph
tvs = tf.trainable_variables()
## Creation of a list of variables with the same shape as the trainable ones
# initialized with 0s
accum_vars = [tf.Variable(tf.zeros_like(tv.initialized_value()), trainable=False) for tv in tvs]
zero_ops = [tv.assign(tf.zeros_like(tv)) for tv in accum_vars]

## Calls the compute_gradients function of the optimizer to obtain... the list of gradients
gvs_ = opt.compute_gradients(rmse, tvs)

gvs =tf.where(tf.is_nan(gvs_), tf.zeros_like(gvs_), gvs_)

## Adds to each element from the list you initialized earlier with zeros its gradient (works because accum_vars and gvs are in the same order)
accum_ops = [accum_vars[i].assign_add(gv[0]) for i, gv in enumerate(gvs)]

## Define the training step (part with variable value update)
train_step = opt.apply_gradients([(accum_vars[i], gv[1]) for i, gv in enumerate(gvs)])

所以基本上，关键思想是这一行：

gvs =tf.where(tf.is_nan(gvs_), tf.zeros_like(gvs_), gvs_)

但是当我应用这个想法时，我得到以下错误：

ValueError: 试图将 'x' 转换为张量并失败。错误： 两个形状的维度 1 必须相等，但分别是 30 和 9。形状是 [2,30] 和 [2,9]。将形状 2 与其他形状合并。为了 'IsNan/packed' (op: 'Pack') 输入形状：[2,9,30], [2,30,9], [2,30]，[2,9]。

Answer 1

compute_gradients 在您的情况下返回张量列表。你可能想做：

gvs_ = [(tf.where(tf.is_nan(grad), tf.zeros_like(grad), grad), val) for grad,val in gvs_]