使用 Estimator 的 Tensorflow 嵌入最佳实践

Question

我有一个带有层的 CIFAR-10 数据集的 CNN：

[IN] -> [CONV] -> [POOL] -> [CONV] -> [POOL] -> [FC] -> [DROPOUT] -> [LOGITS] -> [OUT]
                                                                 \-> [EMBEDDINGS]

估算器代码：

config = tf.contrib.learn.RunConfig(save_checkpoints_secs=30)

# Create the Estimator
classifier = tf.estimator.Estimator(model_fn=inference, config=config, model_dir=LOG_DIR)
train_images, train_labels, train_labels_onehot = Utils.load_training_data()

hooks = [
    # logging hook
    tf.train.LoggingTensorHook(tensors=tensors_to_log, every_n_iter=50),
]

train_input_fn = tf.estimator.inputs.numpy_input_fn(x={'x': train_images}, y=train_labels)

classifier.train(input_fn=train_input_fn, steps=FLAGS.steps, hooks=hooks)

推理功能代码：

def inference(self, features, labels, mode):
    try:
        images = tf.cast(features['x'], tf.float32)
        # Input Layer
        with tf.name_scope('Data'):
            input_layer = tf.reshape(images, [-1, img_width, img_height, num_channels])

        # Convolutional Layer 1
        with tf.variable_scope('ConvLayer1'):
            conv1 = tf.layers.conv2d(inputs=input_layer, filters=32, kernel_size=[5, 5], 
                                     padding="same", activation=tf.nn.relu)
            pool1 = tf.layers.max_pooling2d(inputs=conv1, pool_size=[2, 2], strides=2)

        logging.info('Convolutional Layer 1 build successful..')

        # Convolutional Layer 1
        with tf.variable_scope('ConvLayer2'):
            conv2 = tf.layers.conv2d(inputs=pool1, filters=64, kernel_size=[5, 5], 
                                     padding="same", activation=tf.nn.relu)
            pool2 = tf.layers.max_pooling2d(inputs=conv2, pool_size=[2, 2], strides=2)

        logging.info('Convolutional Layer 2 build successful..')

        # Fully Connected Layer
        with tf.variable_scope('FullyConnectedLayer'):
            pool2_flat = tf.reshape(pool2, [-1, 8 * 8 * 64])
            dense = tf.layers.dense(inputs=pool2_flat, units=1024, activation=tf.nn.relu)
            dropout = tf.layers.dropout(inputs=dense, rate=0.4, 
                                        training=(mode == tf.estimator.ModeKeys.TRAIN))

        logging.info('Fully Connected Layer build successful..')

        tf.summary.histogram('dropout', dropout)

        # Logits Layer
        logits = tf.layers.dense(inputs=dropout, units=10)

        tf.summary.histogram('logits', logits)

        logging.info('Logits Layer build successful..')

        predictions = {
            # Generate predictions (for PREDICT and EVAL mode)
            "classes": tf.argmax(input=logits, axis=1),
            # Add `softmax_tensor` to the graph. It is used for PREDICT and by the
            # `logging_hook`.
            "probabilities": tf.nn.softmax(logits, name="softmax_tensor")
        }

        if mode == tf.estimator.ModeKeys.PREDICT:
            return tf.estimator.EstimatorSpec(mode=mode, predictions=predictions, 
                                              evaluation_hooks=[])

        # Calculate Loss (for both TRAIN and EVAL modes)
        onehot_labels = tf.one_hot(indices=tf.cast(labels, tf.int32), depth=10)
        loss = tf.losses.softmax_cross_entropy(onehot_labels=onehot_labels,
                                               logits=logits)

        tf.summary.histogram('loss', loss)

        logging.info('Losses build successful..')

        # Configure the Training Op (for TRAIN mode)
        if mode == tf.estimator.ModeKeys.TRAIN:
            learning_rate = tf.train.exponential_decay(start_learning_rate, 
                                tf.train.get_global_step(), 1000, 0.9, staircase=True)
            optimizer = tf.train.AdamOptimizer(learning_rate=learning_rate)
            train_op = optimizer.minimize(loss=loss, global_step=tf.train.get_global_step())
            return tf.estimator.EstimatorSpec(mode=mode, loss=loss, train_op=train_op,
                                              scaffold=tf.train.Scaffold(
                                                  summary_op=tf.summary.merge_all(),
                                              ))

        # Add evaluation metrics (for EVAL mode)
        accuracy = tf.metrics.accuracy(labels=labels, predictions=predictions["classes"])

        tf.summary.histogram('accuracy', accuracy)

        logging.info('Accuracy metric build successful..')

        return tf.estimator.EstimatorSpec(mode=mode, loss=loss,
                                      train_op=train_op,
                                      scaffold=tf.train.Scaffold(
                                          summary_op=tf.summary.merge_all()
                                      ))

我正在尝试在 tensorflow 中使用嵌入可视化，在这里我想将 dropout 输出可视化为嵌入。

我发现使用嵌入的代码：

sess = tf.InteractiveSession()

# Input set for Embedded TensorBoard visualization
# Performed with cpu to conserve memory and processing power
with tf.device("/cpu:0"):
    embedding = tf.Variable(self._data, trainable=False, name='embedding')

sess.run(embedding.initializer)

writer = tf.summary.FileWriter(LOG_DIR + '/projector', sess.graph)

config = projector.ProjectorConfig()
embed = config.embeddings.add()
embed.tensor_name = embedding.name

embed.metadata_path = os.path.join(LOG_DIR + '/projector/metadata.tsv')

embed.sprite.image_path = os.path.join(DATA_DIR + '/cifar_10k_sprite.png')
embed.sprite.single_image_dim.extend([img_width, img_height])

projector.visualize_embeddings(writer, config)

saver = tf.train.Saver([embedding])
saver.save(sess, os.path.join(LOG_DIR, 'projector/a_model.ckpt'))

在我的情况下它不起作用，因为我正在使用 Estimator 类并且我无法访问会话。

我尝试过的方法：

1. 将 numpy.array 变量传递给 Estimator 的 model_fn，我可以在其中设置该变量的值，然后将该变量传递给 SessionRunHook，在那里我可以访问会话并将数据保存到文件中。不起作用，因为传递给 Estimator 的所有参数都变成了张量。所以这种方式行不通，因为我已经有 dropout 层张量。

2. 创建全局变量，我可以在其中放置 dropout 层的所有值。也没有工作，因为对于这种方法，我需要访问张量值。

据我了解 Estimator 架构的主要问题是让 dropout 层输出信号从 Estimator 输出，并以某种方式将其传递给 SessionRunHook 以将它们保存为嵌入。但我认为这不是最好的方法。

在 Estimator 中使用嵌入的正确方法是什么？

Answer 1

伙计们，我就是这样做的（但这可能不是最有效的方式）：

SessinRunHook：

import tensorflow as tf

from classes.Utils import Utils


class EmbeddingSaverHook(tf.train.SessionRunHook):

    def __init__(self, values, labels, captions):
        self._saver = None

        self._classes = Utils.get_classnames()

        self._dense3 = None
        self._labels = None

        self._emb_values = values
        self._emb_labels = labels
        self._emb_captions = captions

    def begin(self):
        self._dense3 = tf.get_default_graph().get_tensor_by_name("dense3/BiasAdd:0")

        self._labels = tf.get_default_graph().get_tensor_by_name("labels:0")

    def before_run(self, run_context):
        return tf.train.SessionRunArgs([self._dense3, self._labels])

    def after_run(self, run_context, run_values):
        self._emb_values.extend(run_values[0][0])
        self._emb_labels.extend(run_values[0][1])
        self._emb_captions.extend([self._classes[x] for x in run_values[0][1]])

    def end(self, session):
        pass

完整代码可以看in my github repo