使用tensorflow nce_loss训练时如何做预测

Question

https://www.tensorflow.org/api_docs/python/tf/nn/nce_loss 这里说calculate the full sigmoid loss for evaluation or inference，谁能详细解释一下如何在推理期预测标签？

据我了解模型的最后一层输出是shape (batch, num_class)，在训练过程中它直接进入nce loss并被视为二元分类问题。在推理的时候，我直接在最后一层输出上取sigmoid，得到对应的entryi来代表类i的概率是不是对的？或者我可以像使用 softmax 一样直接将最大的条目视为类标签？

不太明白这个，我也没有在网上找到任何与此相关的实际例子。任何帮助表示赞赏！非常感谢！

Answer 1

当您考虑序列输入时，NCE_loss 可能是噪声对比估计，它通过选择候选采样器来改变输入以创建输出。

参考 0：https://www.tensorflow.org/api_docs/python/tf/nn/nce_loss

参考 1：https://github.com/yl-1993/tensorflow/blob/master/tensorflow/examples/tutorials/mnist/mnist_deep.py

参考 2：https://www.programcreek.com/python/example/90447/tensorflow.nce_loss

[ 样本 ]：

import os
from os.path import exists

import tensorflow as tf
import tensorflow_io as tfio

import matplotlib.pyplot as plt
import math
import numpy as np

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
[PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
None
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
physical_devices = tf.config.experimental.list_physical_devices('GPU')
assert len(physical_devices) > 0, "Not enough GPU hardware devices available"
config = tf.config.experimental.set_memory_growth(physical_devices[0], True)
print(physical_devices)
print(config)   

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
: Variables
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
learning_rate = 0.001
global_step = 0
vocabulary_size = 5000
start = 0
limit = 128
delta = 1
embedding_size = 16
n_sample = 16

tf.compat.v1.disable_eager_execution()

# Input data.

inputs = tf.compat.v1.get_variable('X', dtype = tf.int32, initializer = tf.random.uniform(shape=[1], maxval=1, dtype=tf.int32, seed=10))
labels = tf.compat.v1.get_variable('Y', dtype = tf.int32, initializer = tf.random.uniform(shape=[1, 1], maxval=1, dtype=tf.int32, seed=10))

# Look up embeddings for inputs.
embeddings = tf.Variable(
tf.random.uniform([vocabulary_size, embedding_size], -1.0, 1.0)
)
embed = tf.nn.embedding_lookup(embeddings, inputs)

# Construct the variables for the NCE loss
nce_weights = tf.Variable(
    tf.random.uniform(shape=[vocabulary_size, embedding_size], maxval=255, dtype=tf.float32,)
)
nce_biases = tf.Variable(tf.zeros([vocabulary_size]))

# Compute the average NCE loss for the batch.
# tf.nce_loss automatically draws a new sample of the negative labels each
# time we evaluate the loss.
loss = tf.reduce_mean(
tf.nn.nce_loss(nce_weights, nce_biases,
               labels=labels,
               inputs=embed,
               num_sampled=n_sample,
               num_classes=vocabulary_size),
name='loss'
)
optimizer = tf.compat.v1.train.ProximalAdagradOptimizer(
    learning_rate,
    initial_accumulator_value=0.1,
    l1_regularization_strength=0.2,
    l2_regularization_strength=0.1,
    use_locking=False,
    name='ProximalAdagrad'
)
training_op = optimizer.minimize(loss, name='minimize') 


"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
: DataSet / Input
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
X = np.reshape([ 500 ], (1))
Y = np.reshape([ 15 ], (1, 1))
history = [ ] 

"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
: Training / Optimize
"""""""""""""""""""""""""""""""""""""""""""""""""""""""""
with tf.compat.v1.Session() as sess:
    sess.run(tf.compat.v1.global_variables_initializer())
    
    for i in range(1000):
        global_step = global_step + 1
        train_loss, temp = sess.run([loss, training_op], feed_dict={inputs:X, labels:Y})
        history.append(train_loss)
        
        print( 'steps: ' + str(i) )
        
sess.close()

print( history )
plt.plot(history)
plt.show()
plt.close()

input('...')