2.7 Adam

这一篇写得特别详细：
深度学习优化算法解析(Momentum, RMSProp, Adam)
Adam（Adaptive Moment Estimation）
初始化：

vdW=0,vdb=0,SdW=0,Sdb=0$$v_{dW}=0,v_{db}=0,S_{dW}=0,S_{db}=0$$

On iteration t:
compute dW,db$dW,db$ using mini batch

Momentum:vdW=β1vdW+(1−β1)dWvdb=β1vdb+(1−β1)dbRMSprop:SdW=β2SdW+(1−β2)dW2Sdb=β2Sdb+(1−β2)db2Biascorrection:vcorrecteddW=vdW1−β21vcorrecteddb=vdb1−β21ScorrecteddW=SdW1−β22Scorrecteddb=Sdb1−β22Computation:W=W−αvdWSdW−−−−√b=b−αvdbSdb−−−√$$\begin{array}{rl}& Momentum:\\ & v_{dW}={\beta }_1{v}_{dW}+(1-{\beta }_1)dW{v}_{db}={\beta }_1{v}_{db}+(1-{\beta }_1)db\\ & RMSprop:\\ & S_{dW}={\beta }_2{S}_{dW}+(1-{\beta }_2)d{W}^2{S}_{db}={\beta }_2{S}_{db}+(1-{\beta }_2)d{b}^2\\ & \\ & Biascorrection:\\ & v_{dW}^{corrected}=\frac{v_{dW}}{1-{\beta }_1^2}{v}_{db}^{corrected}=\frac{v_{db}}{1-{\beta }_1^2}\\ & S_{dW}^{corrected}=\frac{S_{dW}}{1-{\beta }_2^2}{S}_{db}^{corrected}=\frac{S_{db}}{1-{\beta }_2^2}\\ & \\ & Computation:\\ & W=W-\alpha \frac{v_{dW}}{\sqrt{S_{dW}}}\\ & b=b-\alpha \frac{v_{db}}{\sqrt{S_{db}}}\end{array}$$

When implementing Adam, what people usually do is just use the default value. So, β1${\beta }_1$ and β2${\beta }_2$ as well as ϵ$ϵ$. I don’t think anyone ever really tunes Epsilon. And then, try a range of values of Alpha to see what works best.

So, where does the term ‘Adam’ come from?

Adam stands for Adaptive Moment Estimation. So β1${\beta }_1$ is computing the mean of the derivatives. This is called the first moment. And β2${\beta }_2$ is used to compute exponentially weighted average of the squares and that’s called the second moment. So that gives rise to the name adaptive moment estimation.