Tensorflow Python OutofRange 错误：资源耗尽

Question

我正在关注以下 TensorFlow 对象检测教程 Tensorflow Object Detection。我已经从以下 GitHub 链接 Object Models 下载了模型。我正在尝试使用自定义数据检测花朵。 TFRecords 和标签都已创建，当我运行 train.py 时，如上面的教程所示 python train.py --logtostderr --train_dir=training\ --pipeline_config_path=training\ssd_mobilenet_v1_pets.config.

我收到以下错误：

Instructions for updating:
Please switch to tf.train.create_global_step
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:depth of additional conv before box predictor: 0
INFO:tensorflow:Summary name /clone_loss is illegal; using clone_loss instead.
2018-02-09 01:45:59.841297: I C:\tf_jenkins\home\workspace\rel-win\M\windows\PY\35\tensorflow\core\platform\cpu_feature_guard.cc:137] Your CPU supports    instructions that this TensorFlow binary was not compiled to use: AVX
INFO:tensorflow:Restoring parameters from ssd_mobilenet_v1_coco_11_06_2017/model.ckpt
INFO:tensorflow:Starting Session.
INFO:tensorflow:Saving checkpoint to path training\model.ckpt
INFO:tensorflow:Starting Queues.
INFO:tensorflow:global_step/sec: 0
INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>, OOM when allocating tensor with shape[24,1,3648,5472,3]
     [[Node: batch = QueueDequeueManyV2[component_types=[DT_STRING,  DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32, DT_INT64, DT_INT32, DT_INT64, DT_INT32, DT_INT64, DT_INT32, DT_BOOL, DT_INT32, DT_FLOAT, DT_INT32, DT_STRING, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localho
 st/replica:0/task:0/device:CPU:0"](batch/padding_fifo_queue, batch/n)]]
 INFO:tensorflow:Caught OutOfRangeError. Stopping Training.
 INFO:tensorflow:Finished training! Saving model to disk.
 INFO:tensorflow:Recording summary at step 0.
 Traceback (most recent call last):
 File "train.py", line 164, in <module>
 tf.app.run()
 File "C:\Users\ML\AppData\Local\Programs\Python\Python35\lib\site-packages  \tensorflow\python\platform\app.py", line 48, in run
_sys.exit(main(_sys.argv[:1] + flags_passthrough))
 File "train.py", line 160, in main
worker_job_name, is_chief, FLAGS.train_dir)
File "C:\Users\Ashwin\Desktop\game making\OpenCV\Tensorflow Object    Detection\models\research\object_detection\trainer.py", line 332, in train
saver=saver)
File "C:\Users\Ashwin\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\contrib\slim\python\slim\learning.py", line 775, in train
sv.stop(threads, close_summary_writer=True)
File "C:\Users\Ashwin\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\supervisor.py", line 792, in stop
stop_grace_period_secs=self._stop_grace_secs)
File "C:\Users\Ashwin\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\coordinator.py", line 389, in join
six.reraise(*self._exc_info_to_raise)
File "C:\Users\Ashwin\AppData\Local\Programs\Python\Python35\lib\site-packages\six.py", line 693, in reraise
raise value
File "C:\Users\Ashwin\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\training\queue_runner_impl.py", line 238, in _run
enqueue_callable()
File "C:\Users\Ashwin\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\client\session.py", line 1231, in _single_operation_run
target_list_as_strings, status, None)
File "C:\Users\Ashwin\AppData\Local\Programs\Python\Python35\lib\site-packages\tensorflow\python\framework\errors_impl.py", line 473, in __exit__
c_api.TF_GetCode(self.status.status))
tensorflow.python.framework.errors_impl.ResourceExhaustedError: OOM when  allocating tensor with shape[24,1,3648,5472,3]
     [[Node: batch = QueueDequeueManyV2[component_types=[DT_STRING,  DT_INT32, DT_FLOAT, DT_INT32, DT_FLOAT, DT_INT32, DT_INT64, DT_INT32, DT_INT64, DT_INT32, DT_INT64, DT_INT32, DT_BOOL, DT_INT32, DT_FLOAT, DT_INT32, DT_STRING, DT_INT32, DT_STRING, DT_INT32], timeout_ms=-1, _device="/job:localho
 st/replica:0/task:0/device:CPU:0"](batch/padding_fifo_queue, batch/n)]]

这些行似乎表明了错误的性质： INFO:tensorflow:Error reported to Coordinator: <class 'tensorflow.python.framework.errors_impl.ResourceExhaustedError'>, OOM when allocating tensor with shape[24,1,3648,5472,3] INFO:tensorflow:Caught **OutOfRangeError**. Stopping Training.

我查看了 GitHub 页面，其中删除 num_epochs 似乎可以修复 OutOfRange 错误，但我在从 repo 下载的 train.py 中找不到此类参数。这是我第一次接触 TensorFlow，我无法完全理解其中的机制。

我的数据集标签大小是：训练标签中的 124 张图像和测试标签中的 93 张图像。

Answer 1

看起来您的OOM 问题（Out Of Memory 的首字母缩写）是Github issues of Tensorflow repository 上讨论的已知问题。

似乎每个人的问题并不总是相同的。我将尝试列举最流行的解决方案。

1. 减小.config 文件的批处理大小batch_size。请注意，处理时间会更长。

2. 释放计算机上的内存 (RAM)。我不知道你在哪个操作系统上，但我猜 Linux 发行版。 Here is a StackExchange question 就可以了。

3. Making more checkpoint，这意味着在您的训练模型崩溃之前更频繁地保存它并在那里恢复它。

当然，这里没有一个解决方案是最佳的，真正的解决方案是拥有更强大的 GPU 供您使用。

如果您可以访问它，您也可以考虑使用 云资源（最流行的是 AWS 和 Azure）但是它可能很昂贵。

Answer 2

如果是自定义数据，则必须使用自定义配置文件（特别是需要更改其中的类数）。

您还会在其中找到num_epochs，以及用于训练和测试的批量大小（减少它会降低出现 OOM 错误的风险）。看起来您正在 CPU 上运行，您需要确保其上有足够的 RAM 来运行训练。否则你将不得不使用更小的网络，但 MobileNets 上的 SSD 已经非常小，所以并不容易......