MXNet - 将标量乘以数组导致零答案

【问题标题】：MXNet - multiplying scalar to an array leads to zeroMXNet - 将标量乘以数组导致零
【发布时间】：2018-03-16 17:30:18
【问题描述】：

我正在尝试实现mxnetin this page的教程，计算梯度体面：

def SGD(params, lr):
for param in params:
    param[:] = param - lr * param.grad

我注意到当 lr

lr * param.grad

变成与 param.grad 大小相同的零矩阵

我不知道为什么会这样。谁能帮助我理解这一点？

非常感谢

【问题讨论】：

如果您连接了调试器，请检查param 和lr 的dtype。它们是任何变化的整数吗？你也可以打印param.dtype和lr.dtype。
我在尝试调试时发现会发生这种情况，但我不知道为什么会发生这种情况。
type(lr) 是浮点数。 type(param) 为：mxnet.ndarray.ndarray.NDArray 和 param.dtype：Out[5]：numpy.float32
如果lr 不为零，那么param.grad 必须为零。除了梯度实际上为零之外，param.grad 为零的另一种方法是，如果您实际上是通过在autograd 范围下通过前向传递然后调用backward 来计算梯度。如果您无法弄清楚，请粘贴一个重现该问题的小代码，我很乐意提供帮助。
实际上当我运行代码时它可以工作，但是当我调试它时会发生这种情况。 lr =.5 并且 param.grad 不为零-感谢@SinaAfrooze

标签： machine-learning mxnet

【解决方案1】：

我在本地的 PyCharm 中运行了该笔记本中的代码，并附加了一个调试器来尝试重现您所看到的内容。

def SGD(params, lr):
    for param in params:
        t = lr * param.grad
        param[:] = param - t

当我在上面的代码中设置断点并检查t 时，调试器显示全零。但是当我运行代码时，一切都运行良好。那很奇怪。因此，我修改了代码以在每次迭代中打印每个像素的所有梯度的总和。

def SGD(params, lr):
    for param in params:
        t = lr * param.grad
        print(nd.sum(param.grad, axis=-1))
        param[:] = param - t

我得到了this 输出。一次迭代的输出如下所示：

[  0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -6.23335197e-08
   2.91620381e-08   2.32830644e-10   1.11467671e-08  -7.45058060e-09
   2.09547579e-09  -3.11993062e-08   3.44589353e-08  -7.45058060e-09
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   1.73723835e-09  -1.45343968e-08   1.51213424e-08  -4.47034836e-08
  -1.19209290e-07  -8.10250640e-08   8.66129994e-08   3.77185643e-08
   8.47503543e-08   8.66129994e-08   8.70786607e-08   5.35510480e-09
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   5.47231149e-09  -4.99075004e-10  -4.99075004e-10
   6.05359674e-09  -9.66247171e-09   1.83936208e-08  -7.50416707e-09
   1.12872200e-07   9.31513569e-08   2.45261617e-07  -1.86264515e-07
  -5.28991222e-07  -2.68220901e-07  -1.22934580e-07   0.00000000e+00
   2.08616257e-07   1.89989805e-07   4.33064997e-08   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00  -2.75546541e-09   2.39708999e-08  -2.04163371e-08
   6.70552254e-08   6.14672899e-08   4.40049917e-08   8.94069672e-08
   8.94069672e-08   2.98023224e-07   1.19209290e-07  -5.96046448e-08
  -2.38418579e-07  -4.76837158e-07  -4.76837158e-07  -2.38418579e-07
   1.78813934e-07   4.13507223e-07   2.83122063e-07   3.09199095e-07
   1.34285074e-07   6.17840215e-08   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00  -2.19943086e-09   1.58324838e-08   6.26314431e-08
   4.47034836e-08   5.96046448e-08   2.38418579e-07   4.76837158e-07
   2.38418579e-07   4.76837158e-07   5.66244125e-07   1.49011612e-07
  -4.17232513e-07  -2.38418579e-07  -3.57627869e-07   0.00000000e+00
   0.00000000e+00   1.22934580e-07   1.97440386e-07   2.73808837e-07
   4.55183908e-08   1.16733858e-08   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   7.33416528e-09   4.35829861e-08
   4.36230039e-08   3.72456270e-08   1.67638063e-08   5.96046448e-08
   1.49011612e-07   2.38418579e-07  -1.19209290e-07  -2.38418579e-07
  -1.78813934e-07   2.38418579e-07   6.25848770e-07   3.57627869e-07
   2.98023224e-08  -4.17232513e-07  -3.57627869e-07  -5.96046448e-08
  -1.04308128e-07   1.19209290e-07   1.08033419e-07  -6.14672899e-08
  -1.40164047e-07  -1.22236088e-07  -1.07334927e-07   0.00000000e+00
   0.00000000e+00   0.00000000e+00  -2.17623892e-08   6.13508746e-08
   7.71033228e-08   4.65661287e-09   0.00000000e+00  -5.96046448e-08
   0.00000000e+00   2.38418579e-07   1.19209290e-07   3.57627869e-07
   4.76837158e-07   1.19209290e-07  -2.38418579e-07  -1.78813934e-07
  -4.76837158e-07   0.00000000e+00   1.78813934e-07   1.19209290e-07
   5.06639481e-07   1.78813934e-07  -1.19209290e-07  -1.19209290e-07
  -1.19209290e-07   0.00000000e+00  -1.34168658e-08   0.00000000e+00
   0.00000000e+00   0.00000000e+00   4.71118256e-10   2.32830644e-09
   2.53785402e-08  -2.83122063e-07   1.19209290e-07   1.19209290e-07
   2.98023224e-07   0.00000000e+00   2.38418579e-07   3.20374966e-07
   0.00000000e+00   3.57627869e-07  -2.38418579e-07   2.38418579e-07
  -9.53674316e-07  -7.15255737e-07   1.19209290e-07   3.57627869e-07
   4.17232513e-07  -7.45058060e-08  -5.96046448e-08  -1.19209290e-07
   1.49011612e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -2.59024091e-09
  -5.21540642e-08  -7.45058060e-09  -1.78813934e-07   5.96046448e-08
   1.78813934e-07   1.19209290e-07   4.76837158e-07   2.98023224e-07
  -5.96046448e-08   2.38418579e-07   3.57627869e-07   5.96046448e-08
  -4.76837158e-07   1.78813934e-07   4.17232513e-07  -1.19209290e-07
   2.98023224e-07   5.96046448e-08   2.04890966e-08  -1.30385160e-08
  -4.83123586e-09   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   3.23634595e-08
  -5.58793545e-08  -1.34110451e-07  -1.49011612e-08  -5.96046448e-08
   1.49011612e-07   1.49011612e-08   1.78813934e-07   0.00000000e+00
   0.00000000e+00   8.94069672e-08   5.96046448e-08  -1.19209290e-07
  -3.57627869e-07   0.00000000e+00   4.76837158e-07  -2.38418579e-07
   0.00000000e+00   1.02445483e-07   1.46217644e-07  -1.32713467e-08
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -2.79396772e-08  -1.56462193e-07  -1.93715096e-07   1.82539225e-07
   3.68803740e-07   4.32133675e-07   5.96046448e-08  -1.19209290e-07
   0.00000000e+00   2.98023224e-07  -1.19209290e-07  -5.06639481e-07
   1.19209290e-07   4.76837158e-07   8.94069672e-08   0.00000000e+00
   0.00000000e+00   5.96046448e-08   8.61473382e-08   4.39467840e-08
   4.85442797e-11   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -2.79396772e-08  -1.86264515e-07  -1.93715096e-07   2.98023224e-08
   3.57627869e-07   3.57627869e-07  -2.38418579e-07  -1.19209290e-07
  -2.38418579e-07  -1.78813934e-07  -1.78813934e-07   0.00000000e+00
  -4.02331352e-07  -2.23517418e-08   2.08616257e-07  -2.98023224e-08
   2.38418579e-07   2.08616257e-07   2.37952918e-07   7.72388375e-08
   3.80787242e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -7.45058060e-08  -2.98023224e-08   1.11758709e-07   1.78813934e-07
   4.76837158e-07   3.57627869e-07  -1.19209290e-07   0.00000000e+00
  -3.57627869e-07   2.38418579e-07  -1.19209290e-07   5.96046448e-08
   1.19209290e-07   1.49011612e-07   8.94069672e-08   1.78813934e-07
   4.17232513e-07   4.47034836e-07   4.47034836e-07   8.42846930e-08
  -3.70637281e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -1.04308128e-07   0.00000000e+00   3.35276127e-07   2.68220901e-07
   7.74860382e-07   7.15255737e-07   2.38418579e-07   1.19209290e-07
  -2.38418579e-07  -3.57627869e-07  -3.57627869e-07  -8.94069672e-08
   3.87430191e-07   3.57627869e-07   3.57627869e-07   5.96046448e-08
   2.98023224e-07   4.76837158e-07   5.36441803e-07   4.84287739e-08
   4.59044713e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -9.77270531e-09
  -1.19209290e-07   7.45058060e-09   0.00000000e+00   3.57627869e-07
   2.38418579e-07   5.96046448e-07   3.57627869e-07   9.53674316e-07
   4.76837158e-07   1.19209290e-07  -2.38418579e-07   1.19209290e-07
  -2.98023224e-07  -1.19209290e-07  -5.96046448e-08  -3.57627869e-07
   1.78813934e-07   4.47034836e-07   3.87430191e-07   1.08033419e-07
   3.81596692e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -4.18361026e-08
  -5.63450158e-08  -4.84287739e-08   5.96046448e-08   2.38418579e-07
   4.76837158e-07   3.57627869e-07   5.96046448e-07   1.78813934e-07
  -1.19209290e-07  -1.19209290e-07  -2.68220901e-07   0.00000000e+00
   0.00000000e+00   2.38418579e-07  -5.96046448e-07  -3.87430191e-07
   1.19209290e-07   3.57627869e-07   3.87430191e-07   1.24797225e-07
   6.91925379e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -4.36812968e-08
  -5.60366971e-08  -8.84756446e-08   1.19209290e-07  -1.19209290e-07
   3.57627869e-07   5.96046448e-08   0.00000000e+00   0.00000000e+00
  -5.96046448e-07   1.19209290e-07  -2.98023224e-08   1.78813934e-07
  -2.38418579e-07  -2.38418579e-07   3.57627869e-07  -3.57627869e-07
   2.38418579e-07   2.38418579e-07   3.42726707e-07   1.14087015e-07
   2.49773286e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00  -1.28651578e-09
  -1.01268569e-07  -1.50499545e-07  -4.47034836e-08   0.00000000e+00
   5.96046448e-08   3.57627869e-07   1.19209290e-07  -3.57627869e-07
  -3.57627869e-07   2.98023224e-07   3.12924385e-07   5.96046448e-08
   0.00000000e+00  -3.57627869e-07  -4.76837158e-07   5.96046448e-08
   2.98023224e-07   2.98023224e-07   1.78813934e-07   7.29633030e-08
   1.60424936e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -7.72104300e-08  -3.55039447e-08   2.83529516e-07   2.63098627e-08
   3.25031579e-07   7.45058060e-09   1.19209290e-07  -5.96046448e-08
  -3.27825546e-07   6.70552254e-08   3.12924385e-07   4.17232513e-07
  -3.57627869e-07  -4.76837158e-07  -1.19209290e-07   1.78813934e-07
   2.98023224e-07   1.71363354e-07   6.27478585e-08   1.92048901e-08
  -3.83870313e-09   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
  -6.35419752e-08  -5.20981303e-08   2.64481059e-08   6.12053555e-08
   2.30735168e-07  -2.51457095e-08  -2.98023224e-07  -3.57627869e-07
   1.78813934e-07   6.55651093e-07   2.38418579e-07  -5.96046448e-08
  -3.57627869e-07   3.57627869e-07   1.19209290e-07   1.78813934e-07
   1.49011612e-07   8.19563866e-08   4.22005542e-08   2.06365929e-08
   2.06365929e-08   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   2.09386428e-08   5.31318420e-08  -5.19480636e-09  -1.55114321e-07
   7.33416528e-08  -2.98023224e-08  -2.98023224e-08  -2.98023224e-07
   4.76837158e-07   2.38418579e-07   5.96046448e-08   3.57627869e-07
   1.19209290e-07   0.00000000e+00   2.38418579e-07   1.04308128e-07
   3.91155481e-08   4.49217623e-08   2.17316245e-08   2.06365929e-08
   2.23254459e-09   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   1.64938871e-08   6.39862421e-08   5.16827185e-08   1.46355887e-07
  -1.34110451e-07  -5.96046448e-08   2.38418579e-07   7.74860382e-07
   9.53674316e-07   1.19209290e-07   0.00000000e+00   3.57627869e-07
   2.38418579e-07   0.00000000e+00   1.19209290e-07   1.44354999e-08
   2.26427801e-08   1.19798682e-09  -4.99121011e-09  -4.99121011e-09
  -2.63172950e-09   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   3.20321369e-08   5.63813956e-08   8.19563866e-08
   0.00000000e+00  -1.19209290e-07   1.19209290e-07   1.19209290e-07
   1.19209290e-07  -1.19209290e-07   0.00000000e+00   0.00000000e+00
   5.96046448e-08  -7.45058060e-09   4.09781933e-08   1.63381628e-08
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00  -7.45058060e-08  -5.96046448e-08
  -1.78813934e-07   5.96046448e-08  -2.88709998e-08  -3.35276127e-08
   6.28060661e-08   2.98023224e-08   1.19209290e-07   5.96046448e-08
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00  -1.19209290e-07  -1.19209290e-07
   0.00000000e+00  -1.49011612e-08  -2.67755240e-08  -1.49157131e-09
   7.06131686e-09   6.01576176e-08  -2.69501470e-08   1.49011612e-08
   0.00000000e+00   0.00000000e+00  -7.45058060e-09   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00
   0.00000000e+00   0.00000000e+00   0.00000000e+00   0.00000000e+00]
<NDArray 784 @cpu(0)>

请注意，前几十个像素的权重梯度始终为零。由于调试器只显示 ndarray 上的前几个值，所以你看到的只是零。

前几十个像素的权重梯度很可能为零，因为这些像素不包含对图像中的数字进行分类的任何有用信息。请注意，MNIST 图像在图像中居中。沿边界的像素只有 0。

希望对您有所帮助。

【讨论】：

谢谢@indhu Bharathi。我尝试了其他一些缩放器来进行 ndarray mulyiplication，当缩放器小于一时，结果变为零（仅在调试模式下）。你也可能会遇到这种情况。而在运行中同样的乘法很好！！！