【发布时间】：2016-12-02 19:53:29
【问题描述】：
我正在尝试使用 TFrecord 文件在 tensorflow 中训练网络。问题是它开始运行良好，但一段时间后，它变得非常慢。甚至 GPU 利用率也会在一段时间内降至 0%。 我已经测量了迭代之间的时间，并且明显在增加。 我在某处读到这可能是由于在训练循环中向图形添加操作，这可以通过使用 graph.finalize() 来解决。
我的代码是这样的：
self.inputMR_,self.CT_GT_ = read_and_decode_single_example("data.tfrecords")
self.inputMR, self.CT_GT = tf.train.shuffle_batch([self.inputMR_, self.CT_GT_], batch_size=self.batch_size, num_threads=2,
capacity=500*self.batch_size,min_after_dequeue=2000)
batch_size_tf = tf.shape(self.inputMR)[0] #variable batchsize so we can test here
self.train_phase = tf.placeholder(tf.bool, name='phase_train')
self.G = self.Network(self.inputMR,batch_size_tf)# create the network
self.g_loss=lp_loss(self.G, self.CT_GT, self.l_num, batch_size_tf)
print 'learning rate ',self.learning_rate
self.g_optim = tf.train.GradientDescentOptimizer(self.learning_rate).minimize(self.g_loss)
self.saver = tf.train.Saver()
然后我的训练阶段是这样的：
def train(self, config):
init=tf.initialize_all_variables()
with tf.Session() as sess:
sess.run(init)
coord = tf.train.Coordinator()
threads=tf.train.start_queue_runners(sess=sess, coord=coord)
sess.graph.finalize()# **WHERE SHOULD I PUT THIS?**
try:
while not coord.should_stop():
_,loss_eval = sess.run([self.g_optim, self.g_loss],feed_dict={self.train_phase: True})
.....
except:
e = sys.exc_info()[0]
print "Exception !!!", e
finally:
coord.request_stop()
coord.join(threads)
sess.close()
当我添加 grapgh.finalize 时，有一个异常显示：type 'exceptions.RuntimeError' 谁能向我解释一下，在训练期间使用 TFrecord 文件的正确方法是什么，以及如何在不干扰 QueueRunner 执行的情况下使用 graph.finalize()？
完整的错误是：
File "main.py", line 37, in <module>
tf.app.run()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/platform/app.py", line 30, in run
sys.exit(main(sys.argv[:1] + flags_passthrough))
File "main.py", line 35, in main
gen_model.train(FLAGS)
File "/home/dongnie/Desktop/gan/TF_record_MR_CT/model.py", line 143, in train
self.global_step.assign(it).eval() # set and update(eval) global_step with index, i
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/variables.py", line 505, in assign
return state_ops.assign(self._variable, value, use_locking=use_locking)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/ops/gen_state_ops.py", line 45, in assign
use_locking=use_locking, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/op_def_library.py", line 490, in apply_op
preferred_dtype=default_dtype)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 657, in convert_to_tensor
ret = conversion_func(value, dtype=dtype, name=name, as_ref=as_ref)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 180, in _constant_tensor_conversion_function
return constant(v, dtype=dtype, name=name)
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/constant_op.py", line 167, in constant
attrs={"value": tensor_value, "dtype": dtype_value}, name=name).outputs[0]
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2337, in create_op
self._check_not_finalized()
File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/framework/ops.py", line 2078, in _check_not_finalized
raise RuntimeError("Graph is finalized and cannot be modified.")
RuntimeError: Graph is finalized and cannot be modified.
【问题讨论】：
-
通常您构建图表，然后完成它，然后执行您的第一个 session.run 调用。查看 RuntimeError 的完整堆栈跟踪会很有用
-
谢谢雅罗斯拉夫，我这样做了，使用 QueueRunner 时出现问题，然后 graph.finalize 导致错误。如何打印完整的堆栈跟踪？
-
即，复制粘贴所有打印的内容，而不仅仅是
'exceptions.RuntimeError'部分 -
我将其添加到问题中；但是没有try except，否则只会显示exceptions.RuntimeError
-
这就解释了为什么你的训练变慢了——运行
global_step.assign(it)会在每次运行时向图表添加一个新的分配操作
标签： python tensorflow deep-learning tensorflow-serving