【发布时间】：2020-03-05 20:54:29
【问题描述】：
我的目标是通过 SGD 训练自动编码器。通过使用 tensorflow 1.x，我为我的损失函数添加了 L1 正则化，如下所示：
........
........
beta = 10e-3
n_inputs = X_Train.shape[1]
n_outputs = n_inputs
X = tf.placeholder(tf.float32, shape=[None, n_inputs])
weights1 = tf.get_variable("weights1", shape=[n_inputs, n_hidden], dtype=tf.float32, initializer = tf.contrib.layers.variance_scaling_initializer())
weights2 = tf.get_variable("weights2", shape=[n_hidden, n_outputs], dtype=tf.float32, initializer = tf.contrib.layers.variance_scaling_initializer())
biases1 = tf.get_variable("biases1", shape=[n_hidden], initializer = tf.zeros_initializer())
biases2 = tf.get_variable("biases2", shape=[n_outputs], initializer = tf.zeros_initializer())
hidden = activation(tf.matmul(X, weights1) + biases1)
outputs = tf.matmul(hidden, weights2) + biases2
reconstruction_loss = tf.reduce_mean(tf.square(outputs - X))
reg_loss = beta * (tf.reduce_sum(tf.abs(weights1)) + tf.reduce_sum(tf.abs(weights2)))
loss = reconstruction_loss + reg_loss
training_op = tf.train.AdamOptimizer(learning_rate).optimizer.minimize(loss)
init = tf.global_variables_initializer()
.......
.......
训练后，我计算了 weights1 矩阵中零的个数。我发现所有的 weights1[i][j] ≠ 0。有什么问题？
【问题讨论】：
-
我相信你的参数会在零附近振荡，通过添加/减去 delta = 学习率 * 正则化常数。如果一个参数为正且小于 delta，它将变为负数并再次小于 delta，然后返回。