sklearn MLPRegressor 的 TensorFlow 副本产生其他结果

Question

我正在尝试在 Tensorflow 中重现深度学习回归结果。如果我使用 sklearn 中的 MLPRegressor 类训练神经网络，我会得到 98% 的非常好的验证结果。

MLPRegressor：

http://scikit-learn.org/stable/modules/generated/sklearn.neural_network.MLPRegressor.html#sklearn.neural_network.MLPRegressor

我正在尝试在 Tensorflow 中重现该模型。通过在 Tensorflow 模型中复制 MLPRegressor 类的默认值。但是我不能得到相同的结果。大多数时候我只得到 75%。

我的 TF 模型：

tf.reset_default_graph()
graph = tf.Graph()
n_input = 3  # n variables
n_hidden_1 = 100
n_hidden_2 = 1
n_output = 1
beta = 0.001

learning_rate = 0.001

with graph.as_default():
    tf_train_feat = tf.placeholder(tf.float32, shape=(None, n_input))
    tf_train_label = tf.placeholder(tf.float32, shape=(None))


    tf_test_feat = tf.constant(test_feat, tf.float32)


    """
    Weights and biases. The weights matix' columns will be the output vector.

    * ndarray([rows, columns])
    * ndarray([in, out])

    tf.placeholder(None) and tf.placeholder([None, 3]) means that the row's size is not set. In the second 
    placeholder the columns are prefixed at 3.
    """
    W = {
        "layer_1": tf.Variable(tf.truncated_normal([n_input, n_hidden_1])),
        "layer_2": tf.Variable(tf.truncated_normal([n_hidden_1, n_hidden_2])),
        "layer_3": tf.Variable(tf.truncated_normal([n_hidden_2, n_output])),
    }


    b = {
        "layer_1": tf.Variable(tf.zeros([n_hidden_1])),
        "layer_2": tf.Variable(tf.zeros([n_hidden_2])),
    }

    def computation(X):
        layer_1 = tf.nn.relu(tf.matmul(X, W["layer_1"]) + b["layer_1"])
        layer_2 = tf.nn.relu(tf.matmul(layer_1, W["layer_2"]) + b["layer_2"])
        return layer_2

    tf_prediction = computation(tf_train_feat)
    tf_test_prediction = computation(tf_test_feat)

    tf_loss = tf.reduce_mean(tf.pow(tf_train_label - tf_prediction, 2)) 
    tf_loss = tf.reduce_mean( tf_loss + beta * tf.nn.l2_loss(W["layer_2"]) )
    tf_optimizer = tf.train.AdamOptimizer(learning_rate).minimize(tf_loss)
    #tf_optimizer = tf.train.GradientDescentOptimizer(learning_rate).minimize(tf_loss)

    init = tf.global_variables_initializer()

我的 TF 会话：

def accuracy(y_pred, y):
    a = 0
    for i in range(y.shape[0]):
        a += abs(1 - y_pred[i][0] / y[i])

    return round((1 - a / y.shape[0]) * 100, 3)

def accuracy_tensor(y_pred, y):
    a = 0
    for i in range(y.shape[0]):
        a += abs(1 - y_pred[i][0] / y[i])

    return round((1 - a / y.shape[0]) * 100, 3)

# Shuffles two arrays.
def shuffle_in_unison(a, b):
    assert len(a) == len(b)
    shuffled_a = np.empty(a.shape, dtype=a.dtype)
    shuffled_b = np.empty(b.shape, dtype=b.dtype)
    permutation = np.random.permutation(len(a))
    for old_index, new_index in enumerate(permutation):
        shuffled_a[new_index] = a[old_index]
        shuffled_b[new_index] = b[old_index]
    return shuffled_a, shuffled_b

train_epoch = int(5e4)
batch = int(200)

n_batch = int(X.shape[0] // batch)

prev_acc = 0
stable_count = 0

session = tf.InteractiveSession(graph=graph)
session.run(init)
print("Initialized.\n No. of epochs: %d.\n No. of batches: %d." % (train_epoch, n_batch))

for epoch in range(train_epoch):
    offset = (epoch * n_batch) % (Y.shape[0] - n_batch)


    for i in range(n_batch):
        x = X[offset:(offset + n_batch)]
        y = Y[offset:(offset + n_batch)]

        x, y = shuffle_in_unison(x, y)

        feed_dict = {tf_train_feat: x, tf_train_label: y}
        _, l, pred, pred_label = session.run([tf_optimizer, tf_loss, tf_prediction, tf_train_label], feed_dict=feed_dict)

    if epoch % 1 == 0:
        print("Epoch: %d. Batch' loss: %f" %(epoch, l))
        test_pred = tf_test_prediction.eval(session=session)

        acc_test = accuracy(test_pred, test_label)
        acc_train = accuracy_tensor(pred, pred_label)

        print("Accuracy train set %s%%" % acc_train)
        print("Accuracy test set: %s%%" % acc_test)

我在 Tensorflow 代码中遗漏了什么吗？谢谢！

Answer 1

除非您有充分的理由不使用它们，否则回归应该具有线性输出单元。不久前我遇到了类似的问题，最终使用了线性输出和线性隐藏单元，这在我的例子中似乎反映了 mlpregressor。

Goodfellow 的 Deep Learning Book 中的 chapter 6 中有一个很棒的部分，从第 181 页开始，介绍了激活函数。

至少为你的输出层试试这个

 layer_2 = tf.matmul(layer_1, W["layer_2"]) + b["layer_2"]