我可以使用 `tf.nn.dropout` 来实现 DropConnect 吗？

Question

我（认为）我掌握了 DropOut 和 use of the TensorFlow API in implementing it 的基础知识。但是与tf.nn.dropout 中的辍学概率相关的标准化似乎不是DropConnect 的一部分。那是对的吗？如果是这样，规范化是否有任何“伤害”，或者我可以简单地将tf.nn.dropout 应用于我的权重以实现 DropConnect？

Answer 1

回答

是的，你可以使用 tf.nn.dropout 来做 DropConnect，只需使用 tf.nn.dropout 来包装你的权重矩阵而不是你的后矩阵乘法。然后，您可以通过乘以 dropout 来撤消权重变化，就像这样

dropConnect = tf.nn.dropout( m1, keep_prob ) * keep_prob

代码示例

这是一个使用 drop connect 计算 XOR 函数的代码示例。我还注释掉了可以退出并比较输出的代码。

### imports
import tensorflow as tf

### constant data
x  = [[0.,0.],[1.,1.],[1.,0.],[0.,1.]]
y_ = [[1.,0.],[1.,0.],[0.,1.],[0.,1.]]

### induction

# Layer 0 = the x2 inputs
x0 = tf.constant( x  , dtype=tf.float32 )
y0 = tf.constant( y_ , dtype=tf.float32 )

keep_prob = tf.placeholder( dtype=tf.float32 )

# Layer 1 = the 2x12 hidden sigmoid
m1 = tf.Variable( tf.random_uniform( [2,12] , minval=0.1 , maxval=0.9 , dtype=tf.float32  ))
b1 = tf.Variable( tf.random_uniform( [12]   , minval=0.1 , maxval=0.9 , dtype=tf.float32  ))


########## DROP CONNECT
# - use this to preform "DropConnect" flavor of dropout
dropConnect = tf.nn.dropout( m1, keep_prob ) * keep_prob
h1 = tf.sigmoid( tf.matmul( x0, dropConnect ) + b1 ) 

########## DROP OUT
# - uncomment this to use "regular" dropout
#h1 = tf.nn.dropout( tf.sigmoid( tf.matmul( x0,m1 ) + b1 ) , keep_prob )


# Layer 2 = the 12x2 softmax output
m2 = tf.Variable( tf.random_uniform( [12,2] , minval=0.1 , maxval=0.9 , dtype=tf.float32  ))
b2 = tf.Variable( tf.random_uniform( [2]   , minval=0.1 , maxval=0.9 , dtype=tf.float32  ))
y_out = tf.nn.softmax( tf.matmul( h1,m2 ) + b2 )


# loss : sum of the squares of y0 - y_out
loss = tf.reduce_sum( tf.square( y0 - y_out ) )

# training step : discovered learning rate of 1e-2 through experimentation
train = tf.train.AdamOptimizer(1e-2).minimize(loss)

### training
# run 5000 times using all the X and Y
# print out the loss and any other interesting info
with tf.Session() as sess:
  sess.run( tf.initialize_all_variables() )
  print "\nloss"
  for step in range(5000) :
    sess.run(train,feed_dict={keep_prob:0.5})
    if (step + 1) % 100 == 0 :
      print sess.run(loss,feed_dict={keep_prob:1.})


  results = sess.run([m1,b1,m2,b2,y_out,loss],feed_dict={keep_prob:1.})
  labels  = "m1,b1,m2,b2,y_out,loss".split(",")
  for label,result in zip(*(labels,results)) :
    print ""
    print label
    print result

print ""

输出

两种风格都能够正确地将输入分离为正确的输出

y_out
[[  7.05891490e-01   2.94108540e-01]
 [  9.99605477e-01   3.94574134e-04]
 [  4.99370173e-02   9.50062990e-01]
 [  4.39682379e-02   9.56031740e-01]]

在这里您可以看到 dropConnect 的输出能够正确地将 Y 分类为 true、true、false、false。