训练RNN时张量流损失为nan

Question

运行带有单个 GRU 单元的 RNN，我遇到了以下情况，我得到以下堆栈跟踪

Traceback (most recent call last):
  File "language_model_test.py", line 15, in <module>
    test_model()
  File "language_model_test.py", line 12, in test_model
    model.train(random_data, s)
  File "/home/language_model/language_model.py", line 120, in train
    train_pp = self._run_epoch(data, sess, inputs, rnn_ouputs, loss, trainOp, verbose)
  File "/home/language_model/language_model.py", line 92, in _run_epoch
    loss, _= sess.run([loss, trainOp], feed_dict=feed)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 767, in run
    run_metadata_ptr)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 952, in _run
    fetch_handler = _FetchHandler(self._graph, fetches, feed_dict_string)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 408, in __init__
    self._fetch_mapper = _FetchMapper.for_fetch(fetches)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 230, in for_fetch
    return _ListFetchMapper(fetch)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 337, in __init__
    self._mappers = [_FetchMapper.for_fetch(fetch) for fetch in fetches]
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 238, in for_fetch
    return _ElementFetchMapper(fetches, contraction_fn)
  File "/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.py", line 271, in __init__
    % (fetch, type(fetch), str(e)))
TypeError: Fetch argument nan has invalid type <type 'numpy.float32'>, must be a string or Tensor. (Can not convert a float32 into a Tensor or Operation.)

计算损失的步骤似乎是个问题

def train(self,data, session=tf.Session(), verbose=10):

        print "initializing model"
        self._add_placeholders()
        inputs = self._add_embedding()
        rnn_ouputs, _ = self._run_rnn(inputs)
        outputs = self._projection_layer(rnn_ouputs)
        loss = self._compute_loss(outputs)
        trainOp = self._add_train_step(loss)
        start = tf.initialize_all_variables()
        saver = tf.train.Saver()

        with session as sess:
            sess.run(start)

            for epoch in xrange(self._max_epochs):
                train_pp = self._run_epoch(data, sess, inputs, rnn_ouputs, loss, trainOp, verbose)
                print "Training preplexity for batch {} - {}".format(epoch, train_pp)

这是_run_epoch的代码

任何有损失的地方都会回来nan

def _run_epoch(self, data, session, inputs, rnn_ouputs, loss, trainOp, verbose=10):
    with session.as_default() as sess:
        total_steps = sum(1 for x in data_iterator(data, self._batch_size, self._max_steps))
        train_loss = []
        for step, (x,y, l) in enumerate(data_iterator(data, self._batch_size, self._max_steps)):
            print "step - {0}".format(step)
            feed = {
                self.input_placeholder: x,
                self.label_placeholder: y,
                self.sequence_length: l,
                self._dropout_placeholder: self._dropout,
            }
            loss, _= sess.run([loss, trainOp], feed_dict=feed)
            print "loss - {0}".format(loss)
            train_loss.append(loss)
            if verbose and step % verbose == 0:
                sys.stdout.write('\r{} / {} : pp = {}'. format(step, total_steps, np.exp(np.mean(train_loss))))
                sys.stdout.flush()
            if verbose:
                sys.stdout.write('\r')

        return np.exp(np.mean(train_loss))

当我使用以下数据测试我的代码时会出现这种情况 random_data = np.random.normal(0, 100, size=[42068, 46]) 旨在模仿使用单词 id 作为输入传递。我的其余代码可以在以下gist

中找到

编辑这是我在出现此问题时运行测试套件的方式：

def test_model():
    model = Language_model(vocab=range(0,101))
    s = tf.Session()
    #1 more than step size to acoomodate for the <eos> token at the end
    random_data = np.random.normal(0, 100, size=[42068, 46])
    # file = "./data/ptb.test.txt"
    print "Fitting started"
    model.train(random_data, s)

if __name__ == "__main__":
    test_model() 

如果我将random_data 替换为其他语言模型，它们也会输出nan 作为成本。我的理解是，通过传入 feed-dict，tensorflow 应该获取数值并检索与 id 对应的适当嵌入向量，我不明白为什么 random_data 会导致 nan 用于其他楷模。

Answer 1

上面的代码有几个问题

让我们从这一行开始

random_data = np.random.normal(0, 100, size=[42068, 46])

np.random.normal(...) 不会产生不同的值，而是产生浮点值，让我们试试上面的以下示例，但大小可管理。

>>> np.random.normal(0, 100, size=[5])
array([-53.12407229,  39.57335574, -98.25406749,  90.81471139, -41.05069646])

机器学习算法无法学习这些，因为它们是嵌入模型的输入，我们有负值和浮点值。

真正想要的是以下代码：

random_data = np.random.randint(0, 101, size=...)

检查我们得到的输出

>>> np.random.randint(0, 100, size=[5])
array([27, 47, 33, 12, 24])

接下来，下面这行实际上是在制造一个微妙的问题。

def _run_epoch(self, data, session, inputs, rnn_ouputs, loss, train, verbose=10):
    with session.as_default() as sess:
        total_steps = sum(1 for x in data_iterator(data, self._batch_size, self._max_steps))
        train_loss = []
        for step, (x,y, l) in enumerate(data_iterator(data, self._batch_size, self._max_steps)):
            print "step - {0}".format(step)
            feed = {
                self.input_placeholder: x,
                self.label_placeholder: y,
                self.sequence_length: l,
                self._dropout_placeholder: self._dropout,
            }
            loss, _= sess.run([loss, train], feed_dict=feed)
            print "loss - {0}".format(loss)
            train_loss.append(loss)
            if verbose and step % verbose == 0:
                sys.stdout.write('\r{} / {} : pp = {}'. format(step, total_steps, np.exp(np.mean(train_loss))))
                sys.stdout.flush()
            if verbose:
                sys.stdout.write('\r')

        return np.exp(np.mean(train_loss))

loss 既是参数自变量又是变量，所以第一次运行时，它不再是张量，因此我们实际上不能在会话中调用它。