梯度下降的纯Python实现

Question

我自己尝试使用 Python 实现梯度下降。我知道这方面有类似的话题，但对于我的尝试，我的猜测斜率总是可以非常接近真实斜率，但猜测截距从未匹配甚至接近真实截距。有谁知道为什么会这样？

另外，我阅读了很多梯度下降的帖子和公式，它说对于每次迭代，我需要将梯度乘以负学习率并重复直到它收敛。正如您在下面的实现中所看到的，我的梯度下降仅在我将学习率乘以梯度而不是 -1 时才有效。这是为什么？我理解梯度下降是错误的还是我的实现是错误的？ （如果我将学习率和梯度乘以-1，exam_m 和exam_b 会很快溢出）

intercept = -5
slope = -4


x = []
y = []
for i in range(0, 100):
    x.append(i/300)
    y.append((i * slope + intercept)/300)

learning_rate = 0.005
# y = mx + b 
# m is slope, b is y-intercept

exam_m = 100
exam_b = 100

#iteration
#My error function is sum all (y - guess) ^2
for _ in range(20000):
    gradient_m = 0
    gradient_b = 0
    for i in range(len(x)):
        gradient_m += (y[i] - exam_m * x[i] - exam_b) * x[i]
        gradient_b += (y[i] - exam_m * x[i] - exam_b)
        #why not gradient_m -= (y[i] - exam_m * x[i] - exam_b) * x[i] like what it said in the gradient descent formula

    exam_m += learning_rate * gradient_m
    exam_b += learning_rate * gradient_b
    print(exam_m, exam_b)

Answer 1

溢出的原因是缺少因子(2/n)。为了更清楚，我已经广泛地使用了负面符号。

import numpy as np
import matplotlib.pyplot as plt

intercept = -5
slope = -4
# y = mx + b

x = []
y = []

for i in range(0, 100):
    x.append(i/300)
    y.append((i * slope + intercept)/300)

n = len(x)
x = np.array(x)
y = np.array(y)

learning_rate = 0.05
exam_m = 0
exam_b = 0
epochs = 1000

for _ in range(epochs):
    gradient_m = 0
    gradient_b = 0
    for i in range(n):
        gradient_m -= (y[i] - exam_m * x[i] - exam_b) * x[i]
        gradient_b -= (y[i] - exam_m * x[i] - exam_b)

    exam_m = exam_m - (2/n)*learning_rate * gradient_m
    exam_b = exam_b - (2/n)*learning_rate * gradient_b

print('Slope, Intercept: ', exam_m, exam_b)

y_pred = exam_m*x + exam_b
plt.xlabel('x')
plt.ylabel('y')
plt.plot(x, y_pred, '--', color='black', label='predicted_line')
plt.plot(x, y, '--', color='blue', label='orginal_line')
plt.legend()
plt.show()

输出： Slope, Intercept: -2.421033215481844 -0.2795651072061604