【发布时间】：2019-09-01 09:04:02
【问题描述】：
我创建了自己的非常简单的 1 层神经网络，专门研究二元分类问题。输入数据点乘以权重并添加偏差。整个事情被求和（加权和）并通过激活函数（例如relu 或sigmoid）馈送。那将是预测输出。不涉及其他层（即隐藏层）。
只是为了我自己对数学方面的理解，我不想使用现有的库/包（例如 Keras、PyTorch、Scikit-learn ..等），而只是想使用普通 python 创建一个神经网络代码。该模型是在一个方法 (simple_1_layer_classification_NN) 中创建的，该方法采用必要的参数进行预测。但是，我遇到了一些问题，因此将下面的问题与我的代码一起列出。
附：我真的很抱歉包含这么大一部分代码，但我不知道在不参考相关代码的情况下如何提出问题。
问题：
1 - 当我通过一些训练数据集来训练网络时，我发现最终的平均准确率随着不同的 Epoch 数量完全不同，对于某种最佳的 Epoch 数量绝对没有明确的模式。我保持其他参数相同：learning rate = 0.5、activation = sigmoid（因为它是 1 层 - 既是输入层又是输出层。不涉及隐藏层。我读过 sigmoid 比 @987654327 更适合输出层@)、cost function = squared error。以下是不同时期的结果：
纪元 = 100,000。 平均精度：50.10541638874056
纪元 = 500,000。 平均准确度：50.08965597645948
纪元 = 1,000,000。 平均准确度：97.56879179064482
纪元 = 7,500,000。 平均准确度：49.994692515332524
750,000 纪元。 平均准确度：77.0028368954157
纪元 = 100。 平均准确度：48.96967591507596
纪元 = 500。 平均准确度：48.20721972881673
纪元 = 10,000。 平均准确度：71.58066454336122
纪元 = 50,000。 平均准确度：62.52998222597177
纪元 = 100,000。 平均准确度：49.813675726563424
纪元 = 1,000,000。 平均准确度：49.993141329926374
如您所见，似乎没有任何明确的模式。我尝试了 100 万个 epoch，得到了 97.6% 的准确率。然后我尝试了 750 万个 epoch，得到了 50% 的准确率。 50 万个 epoch 也有 50% 的准确率。 100 个 epoch 的准确率达到 49%。然后真正奇怪的是，再次尝试了 100 万个 epoch 并获得了 50%。
所以我在下面分享我的代码，因为我不相信网络正在做任何学习。似乎只是随机猜测。我应用了反向传播和偏导数的概念来优化权重和偏差。所以我不确定我的代码哪里出错了。
2- 我在simple_1_layer_classification_NN 方法的参数列表中包含的参数之一是input_dimension 参数。起初我认为需要锻炼输入层所需的权重数量。然后我意识到，只要将dataset_input_matrix（特征矩阵）参数传递给方法，我就可以访问矩阵的随机索引来访问矩阵中的随机观察向量（input_observation_vector = dataset_input_matrix[ri]）。然后循环观察以访问每个特征。观察向量的循环数（或长度）将准确告诉我需要多少权重（因为每个特征都需要一个权重（作为其系数）。所以(len(input_observation_vector)) 会告诉我输入中所需的权重数量层，因此我不需要要求用户将input_dimension 参数传递给该方法。
所以我的问题很简单，是否有任何需要/理由包含 input_dimension 参数，这可以通过评估输入矩阵中的观察向量的长度来解决？
3 - 当我尝试绘制 costs 值的数组时，什么都没有显示 - plt.plot(y_costs)。 cost 值（从每个 Epoch 生成）仅每 50 个 epoch 附加到 costs 数组。这是为了避免在 epoch 数量非常多的情况下在数组中添加这么多 cost 元素。在线：
if i % 50 == 0:
costs.append(cost)
我在调试的时候发现costs数组是空的，方法返回后。我不确定为什么会这样，什么时候应该每 50 个 epoch 附加一个 cost 值。可能我忽略了一些我看不到的非常愚蠢的东西。
在此先感谢您，并再次为冗长的代码道歉。
from __future__ import print_function
import numpy as np
import matplotlib.pyplot as plt
import sys
# import os
class NN_classification:
def __init__(self):
self.bias = float()
self.weights = []
self.chosen_activation_func = None
self.chosen_cost_func = None
self.train_average_accuracy = int()
self.test_average_accuracy = int()
# -- Activation functions --:
def sigmoid(x):
return 1/(1 + np.exp(-x))
def relu(x):
return np.maximum(0.0, x)
# -- Derivative of activation functions --:
def sigmoid_derivation(x):
return NN_classification.sigmoid(x) * (1-NN_classification.sigmoid(x))
def relu_derivation(x):
if x <= 0:
return 0
else:
return 1
# -- Squared-error cost function --:
def squared_error(pred, target):
return np.square(pred - target)
# -- Derivative of squared-error cost function --:
def squared_error_derivation(pred, target):
return 2 * (pred - target)
# --- neural network structure diagram ---
# O output prediction
# / \ w1, w2, b
# O O datapoint 1, datapoint 2
def simple_1_layer_classification_NN(self, dataset_input_matrix, output_data_labels, input_dimension, epochs, activation_func='sigmoid', learning_rate=0.2, cost_func='squared_error'):
weights = []
bias = int()
cost = float()
costs = []
dCost_dWeights = []
chosen_activation_func_derivation = None
chosen_cost_func = None
chosen_cost_func_derivation = None
correct_pred = int()
incorrect_pred = int()
# store the chosen activation function to use to it later on in the activation calculation section and in the 'predict' method
# Also the same goes for the derivation section.
if activation_func == 'sigmoid':
self.chosen_activation_func = NN_classification.sigmoid
chosen_activation_func_derivation = NN_classification.sigmoid_derivation
elif activation_func == 'relu':
self.chosen_activation_func = NN_classification.relu
chosen_activation_func_derivation = NN_classification.relu_derivation
else:
print("Exception error - no activation function utilised, in training method", file=sys.stderr)
return
# store the chosen cost function to use to it later on in the cost calculation section.
# Also the same goes for the cost derivation section.
if cost_func == 'squared_error':
chosen_cost_func = NN_classification.squared_error
chosen_cost_func_derivation = NN_classification.squared_error_derivation
else:
print("Exception error - no cost function utilised, in training method", file=sys.stderr)
return
# Set initial network parameters (weights & bias):
# Will initialise the weights to a uniform distribution and ensure the numbers are small close to 0.
# We need to loop through all the weights to set them to a random value initially.
for i in range(input_dimension):
# create random numbers for our initial weights (connections) to begin with. 'rand' method creates small random numbers.
w = np.random.rand()
weights.append(w)
# create a random number for our initial bias to begin with.
bias = np.random.rand()
# We perform the training based on the number of epochs specified
for i in range(epochs):
# create random index
ri = np.random.randint(len(dataset_input_matrix))
# Pick random observation vector: pick a random observation vector of independent variables (x) from the dataset matrix
input_observation_vector = dataset_input_matrix[ri]
# reset weighted sum value at the beginning of every epoch to avoid incrementing the previous observations weighted-sums on top.
weighted_sum = 0
# Loop through all the independent variables (x) in the observation
for i in range(len(input_observation_vector)):
# Weighted_sum: we take each independent variable in the entire observation, add weight to it then add it to the subtotal of weighted sum
weighted_sum += input_observation_vector[i] * weights[i]
# Add Bias: add bias to weighted sum
weighted_sum += bias
# Activation: process weighted_sum through activation function
activation_func_output = self.chosen_activation_func(weighted_sum)
# Prediction: Because this is a single layer neural network, so the activation output will be the same as the prediction
pred = activation_func_output
# Cost: the cost function to calculate the prediction error margin
cost = chosen_cost_func(pred, output_data_labels[ri])
# Also calculate the derivative of the cost function with respect to prediction
dCost_dPred = chosen_cost_func_derivation(pred, output_data_labels[ri])
# Derivative: bringing derivative from prediction output with respect to the activation function used for the weighted sum.
dPred_dWeightSum = chosen_activation_func_derivation(weighted_sum)
# Bias is just a number on its own added to the weighted sum, so its derivative is just 1
dWeightSum_dB = 1
# The derivative of the Weighted Sum with respect to each weight is the input data point / independant variable it's multiplied by.
# Therefore I simply assigned the input data array to another variable I called 'dWeightedSum_dWeights'
# to represent the array of the derivative of all the weights involved. I could've used the 'input_sample'
# array variable itself, but for the sake of readibility, I created a separate variable to represent the derivative of each of the weights.
dWeightedSum_dWeights = input_observation_vector
# Derivative chaining rule: chaining all the derivative functions together (chaining rule)
# Loop through all the weights to workout the derivative of the cost with respect to each weight:
for dWeightedSum_dWeight in dWeightedSum_dWeights:
dCost_dWeight = dCost_dPred * dPred_dWeightSum * dWeightedSum_dWeight
dCost_dWeights.append(dCost_dWeight)
dCost_dB = dCost_dPred * dPred_dWeightSum * dWeightSum_dB
# Backpropagation: update the weights and bias according to the derivatives calculated above.
# In other word we update the parameters of the neural network to correct parameters and therefore
# optimise the neural network prediction to be as accurate to the real output as possible
# We loop through each weight and update it with its derivative with respect to the cost error function value.
for i in range(len(weights)):
weights[i] = weights[i] - learning_rate * dCost_dWeights[i]
bias = bias - learning_rate * dCost_dB
# for each 50th loop we're going to get a summary of the
# prediction compared to the actual ouput
# to see if the prediction is as expected.
# Anything in prediction above 0.5 should match value
# 1 of the actual ouptut. Any prediction below 0.5 should
# match value of 0 for actual output
if i % 50 == 0:
costs.append(cost)
# Compare prediction to target
error_margin = np.sqrt(np.square(pred - output_data_labels[ri]))
accuracy = (1 - error_margin) * 100
self.train_average_accuracy += accuracy
# Evaluate whether guessed correctly or not based on classification binary problem 0 or 1 outcome. So if prediction is above 0.5 it guessed 1 and below 0.5 it guessed incorrectly. If it's dead on 0.5 it is incorrect for either guesses. Because it's no exactly a good guess for either 0 or 1. We need to set a good standard for the neural net model.
if (error_margin < 0.5) and (error_margin >= 0):
correct_pred += 1
elif (error_margin >= 0.5) and (error_margin <= 1):
incorrect_pred += 1
else:
print("Exception error - 'margin error' for 'predict' method is out of range. Must be between 0 and 1, in training method", file=sys.stderr)
return
# store the final optimised weights to the weights instance variable so it can be used in the predict method.
self.weights = weights
# store the final optimised bias to the weights instance variable so it can be used in the predict method.
self.bias = bias
# Calculate average accuracy from the predictions of all obervations in the training dataset
self.train_average_accuracy /= epochs
# Print out results
print('Average Accuracy: {}'.format(self.train_average_accuracy))
print('Correct predictions: {}, Incorrect Predictions: {}'.format(correct_pred, incorrect_pred))
print('costs = {}'.format(costs))
y_costs = np.array(costs)
plt.plot(y_costs)
plt.show()
from numpy import array
#define array of dataset
# each observation vector has 3 datapoints or 3 columns: length, width, and outcome label (0, 1 to represent blue flower and red flower respectively).
data = array([[3, 1.5, 1],
[2, 1, 0],
[4, 1.5, 1],
[3, 1, 0],
[3.5, 0.5, 1],
[2, 0.5, 0],
[5.5, 1, 1],
[1, 1, 0]])
# separate data: split input, output, train and test data.
X_train, y_train, X_test, y_test = data[:6, :-1], data[:6, -1], data[6:, :-1], data[6:, -1]
nn_model = NN_classification()
nn_model.simple_1_layer_classification_NN(X_train, y_train, 2, 1000000, learning_rate=0.5)
【问题讨论】：
-
这是一个相当高的学习率。
-
@Chris ye，我之前尝试过 0.2，但由于学习率，我认为我没有得到太多结果。我现在又试了0.2。真正奇怪的是我尝试了相同数量的 Epoch 3 次（100,000 epoch）。我第一次得到92%的准确率结果。然后是 50%，然后是 49%。显然这里有什么不对！
-
尝试 .01 或 .001 并使用 relu
-
@Chris，抱歉回复晚了。实际上，您让我意识到 relu 对我的 1 层神经网络毫无用处。当我尝试 relu 时，我收到了错误消息，检查以确保输出预测在 0 和 1 之间（因为这是一个概率分类问题）。这是有道理的，因为 relu 的正域可以高于 1。没有限制。只有负域输出 0。这就是为什么总是建议使用 Sigmoid 或其他逻辑非线性函数来压缩一个范围（对于输出）之间的值，因为概率必须在 0 和 1 之间
标签： python python-3.x machine-learning neural-network