分配张量时的OOM

Question

如何解决ResourceExhaustedError: OOM when allocating tensor的问题？

ResourceExhaustedError（回溯见上文）：分配时出现 OOM 形状为[10000,32,28,28]的张量

我包含了几乎所有的代码

learning_rate = 0.0001
epochs = 10
batch_size = 50

# declare the training data placeholders
# input x - for 28 x 28 pixels = 784 - this is the flattened image data that is drawn from
# mnist.train.nextbatch()
x = tf.placeholder(tf.float32, [None, 784])
# dynamically reshape the input
x_shaped = tf.reshape(x, [-1, 28, 28, 1])
# now declare the output data placeholder - 10 digits
y = tf.placeholder(tf.float32, [None, 10])
def create_new_conv_layer(input_data, num_input_channels, num_filters, filter_shape, pool_shape, name):
    # setup the filter input shape for tf.nn.conv_2d
    conv_filt_shape = [filter_shape[0], filter_shape[1], num_input_channels,
                      num_filters]

    # initialise weights and bias for the filter
    weights = tf.Variable(tf.truncated_normal(conv_filt_shape, stddev=0.03),
                                      name=name+'_W')
    bias = tf.Variable(tf.truncated_normal([num_filters]), name=name+'_b')

    # setup the convolutional layer operation
    out_layer = tf.nn.conv2d(input_data, weights, [1, 1, 1, 1], padding='SAME')

    # add the bias
    out_layer += bias

    # apply a ReLU non-linear activation
    out_layer = tf.nn.relu(out_layer)

    # now perform max pooling
    ksize = [1, 2, 2, 1]
    strides = [1, 2, 2, 1]
    out_layer = tf.nn.max_pool(out_layer, ksize=ksize, strides=strides,
                               padding='SAME')

    return out_layer
# create some convolutional layers
layer1 = create_new_conv_layer(x_shaped, 1, 32, [5, 5], [2, 2], name='layer1')
layer2 = create_new_conv_layer(layer1, 32, 64, [5, 5], [2, 2], name='layer2')

flattened = tf.reshape(layer2, [-1, 7 * 7 * 64])

# setup some weights and bias values for this layer, then activate with ReLU
wd1 = tf.Variable(tf.truncated_normal([7 * 7 * 64, 1000], stddev=0.03), name='wd1')
bd1 = tf.Variable(tf.truncated_normal([1000], stddev=0.01), name='bd1')
dense_layer1 = tf.matmul(flattened, wd1) + bd1
dense_layer1 = tf.nn.relu(dense_layer1)

# another layer with softmax activations
wd2 = tf.Variable(tf.truncated_normal([1000, 10], stddev=0.03), name='wd2')
bd2 = tf.Variable(tf.truncated_normal([10], stddev=0.01), name='bd2')
dense_layer2 = tf.matmul(dense_layer1, wd2) + bd2
y_ = tf.nn.softmax(dense_layer2)
cross_entropy = tf.reduce_mean(tf.nn.softmax_cross_entropy_with_logits(logits=dense_layer2, labels=y))


# add an optimiser
optimiser = tf.train.AdamOptimizer(learning_rate=learning_rate).minimize(cross_entropy)

# define an accuracy assessment operation
correct_prediction = tf.equal(tf.argmax(y, 1), tf.argmax(y_, 1))
accuracy = tf.reduce_mean(tf.cast(correct_prediction, tf.float32))

# setup the initialisation operator
init_op = tf.global_variables_initializer() 



 with tf.Session() as sess:
            # initialise the variables
            sess.run(init_op)
            total_batch = int(len(mnist.train.labels) / batch_size)
            for epoch in range(epochs):
                avg_cost = 0
                for i in range(total_batch):
                    batch_x, batch_y = mnist.train.next_batch(batch_size=batch_size)
                    _, c = sess.run([optimiser, cross_entropy], feed_dict={x: 
         batch_x, 
            y: batch_y})
                    avg_cost += c / total_batch
                test_acc = sess.run(accuracy,feed_dict={x: mnist.test.images, y: 
            mnist.test.labels})
                print("Epoch:", (epoch + 1), "cost =", "{:.3f}".format(avg_cost), "  
            test accuracy: {:.3f}".format(test_acc))

            print("\nTraining complete!")
            print(sess.run(accuracy, feed_dict={x: mnist.test.images, y: 
            mnist.test.labels}))

并且错误中引用的那些行是： create_new_conv_layer - function

sess.run .. 在训练循环中

下面列出了我从调试器输出中复制的更多错误（有更多行，但我认为这些是主要的，其他的都是由此引起的..）

tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM 分配张量时的形状为[10000,32,28,28] [[Node: Conv2D = Conv2D[T=DT_FLOAT, data_format="NHWC", padding="SAME", strides=[1, 1, 1, 1], use_cudnn_on_gpu=true, _device="/job:localhost/replica:0/task:0/gpu:0"](Reshape, layer1_W/read)]]

我第二次运行它时发出以下错误我有 cpu 和 GPU 可以在下面的输出中看到，我可以理解一些与 cpu 问题相关的错误可能是因为我的 tensorflow 没有编译为使用这些功能，我在 windows 10 上安装了 cuda 8 和 cudnn 6，python 3.5，tensorflow 1.3.0。

2017-10-03 03:53:58.944371: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:45] TensorFlow 库未编译为使用 AVX 指令，但 这些在您的机器上可用，并且可以加速 CPU 计算。 2017-10-03 03:53:58.945563: W C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:45] TensorFlow 库未编译为使用 AVX2 指令，但 这些在您的机器上可用，并且可以加速 CPU 计算。 2017-10-03 03:53:59.230761: 我 C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:955] 找到具有以下属性的设备 0： 名称：Quadro K620 主要：5 次要：0 memoryClockRate (GHz) 1.124 pciBusID 0000:01:00.0 总内存：2.00GiB 可用内存：1.66GiB 2017-10-03 03:53:59.231109: 我 C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:976] DMA: 0 2017-10-03 03:53:59.231229: 我 C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:986] 0: 是 2017-10-03 03:53:59.231363: 我 C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\common_runtime\gpu\gpu_device.cc:1045] 创建 TensorFlow 设备 (/gpu:0) -> (设备: 0, 名称: Quadro K620, PCI总线ID：0000：01：00.0）2017-10-03 03：54：01.511141：E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_dnn.cc:371] 无法创建 cudnn 句柄：CUDNN_STATUS_NOT_INITIALIZED 2017-10-03 03:54:01.511372: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_dnn.cc:375] 获取驱动程序版本时出错：未实现：内核报告驱动程序版本未在 Windows 上实现 2017-10-03 03:54:01.511862: E C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\stream_executor\cuda\cuda_dnn.cc:338] 无法破坏 cudnn 句柄：CUDNN_STATUS_BAD_PARAM 2017-10-03 03:54:01.512074: F C:\tf_jenkins\home\workspace\rel-win\M\windows-gpu\PY\35\tensorflow\core\kernels\conv_ops.cc:672] 检查失败：stream->parent()->GetConvolveAlgorithms( conv_parameters.ShouldIncludeWinogradNonfusedAlgo(), &algorithms)

Answer 1

进程因内存不足 (OOM) 而失败，因为您一次推送了整个测试集进行评估（请参阅 this question）。很容易看出10000 * 32 * 28 * 28 * 4 几乎是 1Gb，而您的 GPU 总共只有 1.66Gb 可用，而且大部分已经被网络本身占用了。

解决方案是为神经网络提供批次，不仅用于训练，还用于测试。结果准确度将是所有批次的平均值。此外，您不需要在每个 epoch 之后都这样做：您真的对所有中间网络的测试结果感兴趣吗？

您的第二条错误消息很可能是先前失败的结果，因为 CUDNN 驱动程序似乎不再工作了。我建议你重启你的机器。