在 TensorFlow CIFAR10 多 GPU 上计算损失

Question

在 TensorFlow CIFAR10 示例中

# Build the portion of the Graph calculating the losses. Note that we will
# assemble the total_loss using a custom function below.
_ = cifar10.loss(logits, labels)

# Assemble all of the losses for the current tower only.
losses = tf.get_collection('losses', scope)

# Calculate the total loss for the current tower.
total_loss = tf.add_n(losses, name='total_loss')

# Attach a scalar summary to all individual losses and the total loss; do the
# same for the averaged version of the losses.
for l in losses + [total_loss]:
    # Remove 'tower_[0-9]/' from the name in case this is a multi-GPU training
    # session. This helps the clarity of presentation on tensorboard.
    loss_name = re.sub('%s_[0-9]*/' % cifar10.TOWER_NAME, '', l.op.name)
    tf.contrib.deprecated.scalar_summary(loss_name, l)

return total_loss

为什么没有使用函数 cifar10.loss 返回的损失？相反，损失由 tf.get_collection('losses', scope) 计算。

Answer 1

代码返回一个塔（单个 GPU）中所有批次的总损失。我认为这是因为他们对存储单个批次的损失不感兴趣，而是存储所有批次的总损失。

tf.get_collection 使用塔的范围检索塔的所有损失。然后我们可以使用它来计算一个塔或两个塔中所有批次的平均损失。