SGD - 一些迭代后损失开始增加

Question

我正在尝试实现具有两个约束的随机梯度下降，因此不能使用 scikit-learn。不幸的是，在没有这两个限制的情况下，我已经在使用常规 SGD 苦苦挣扎。训练集上的损失（平方损失）在一些迭代中下降，但在一段时间后开始增加，如图所示。 这些是我使用的功能：

def loss_prime_simple(w,node,feature,data):
   x = data[3]
   y = data[2]
   x_f = x[node][feature]
   y_node = y[node]
   ret = (y_node - w[feature] * x_f) * (-x_f)
   return ret

def update_weights(w,data,predecs,children,node, learning_rate):
   len_features = len(data[3][0])
   w_new = np.zeros(len_features)
   for feature_ in range(len_features):
      w_new[feature_] = loss_prime_simple(w,node,feature_,data)
   return w - learning_rate * w_new

def loss_simple(w,data):
   y_p = data[2]
   x = data[3]
   return ((y_p - np.dot(w,np.array(x).T)) ** 2).sum()

这显示了具有两种不同学习率（0.001、0.0001）http://postimg.org/image/43nbmh8x5/的训练集上的损失

任何人都可以找到错误或有如何调试的建议吗？ 谢谢

编辑：

正如 lejlot 指出的那样，拥有数据会很好。 这是我用于 x 的数据（单个样本）：http://textuploader.com/5x0f1

y=2

这让我们失去了这个：http://postimg.org/image/o9d97kt9v/

更新后的代码：

def loss_prime_simple(w,node,feature,data):
   x = data[3]
   y = data[2]
   x_f = x[node][feature]
   y_node = y[node]
   return -(y_node - w[feature] * x_f) * x_f

def update_weights(w,data,predecs,children,node, learning_rate):
   len_features = len(data[3][0])
   w_new = np.zeros(len_features)
   for feature_ in range(len_features):
      w_new[feature_] = loss_prime_simple(w,node,feature_,data)
   return w - learning_rate * w_new

def loss_simple2(w,data):
   y_p = data[2]
   x = data[3]
   return ((y_p - np.dot(w,np.array(x).T)) ** 2).sum()

import numpy as np
X = [#put array from http://textuploader.com/5x0f1 here]
y = [2]

data = None, None, y, X

w = np.random.rand(4096)

a = [ loss_simple2(w, data) ]

for _ in range(200):
    for j in range(X.shape[0]):
        w = update_weights(w,data,None,None,j, 0.0001)
        a.append( loss_simple2(w, data) )

from matplotlib import pyplot as plt
plt.figure()
plt.plot(a)
plt.show()

Answer 1

问题是我用而不是更新了权重

所以这行得通：

def update_weights(w,x,y, learning_rate):
    inner_product = 0.0    
    for f_ in range(len(x)):
        inner_product += (w[f_] * x[f_])
    dloss = inner_product - y
    for f_ in range(len(x)):
        w[f_] += (learning_rate * (-x[f_] * dloss))
    return w

Answer 2

一个可以注意到的主要错误是你reshape而不是transpose，比较：

>>> import numpy as np
>>> X = np.array(range(10)).reshape(2,-1)
>>> X
array([[0, 1, 2, 3, 4],
       [5, 6, 7, 8, 9]])
>>> X.reshape(-1, 2)
array([[0, 1],
       [2, 3],
       [4, 5],
       [6, 7],
       [8, 9]])
>>> X.T
array([[0, 5],
       [1, 6],
       [2, 7],
       [3, 8],
       [4, 9]])
>>> X.reshape(-1, 2) == X.T
array([[ True, False],
       [False, False],
       [False, False],
       [False, False],
       [False,  True]], dtype=bool)

接下来看起来很糟糕的是调用 sum(array)，你应该调用 array.sum()

>>> import numpy as np
>>> x = np.array(range(10)).reshape(2, 5)
>>> x
array([[0, 1, 2, 3, 4],
       [5, 6, 7, 8, 9]])
>>> sum(x)
array([ 5,  7,  9, 11, 13])
>>> x.sum()
45

在这之后，它就可以正常工作了

def loss_prime_simple(w,node,feature,data):
   x = data[3]
   y = data[2]
   x_f = x[node][feature]
   y_node = y[node]
   ret = w[feature]
   return -(y_node - w[feature] * x_f) * x_f

def update_weights(w,data,predecs,children,node, learning_rate):
   len_features = len(data[3][0])
   w_new = np.zeros(len_features)
   for feature_ in range(len_features):
      w_new[feature_] = loss_prime_simple(w,node,feature_,data)
   return w - learning_rate * w_new

def loss_simple(w,data):
   y_p = data[2]
   x = data[3]
   return ((y_p - np.dot(w,np.array(x).T)) ** 2).sum()

import numpy as np

X = np.random.randn(1000, 3)
y = np.random.randn(1000)

data = None, None, y, X

w = np.array([1,3,3])

loss = [loss_simple(w, data)]

for _ in range(20):
    for j in range(X.shape[0]):
        w = update_weights(w, data, None, None, j, 0.001)
        loss.append(loss_simple(w, data))

from matplotlib import pyplot as plt
plt.figure()
plt.plot(loss)
plt.show()