Dissecting Adam The Sign, Magnitude and Variance of Stochastic Gradients

DissectingAdam:TheSign,MagnitudeandVarianceofStochasticGradientsLukasBalles1PhilippHennig1AbstractwhichisarandomvariablewithE[g(θ)]=∇L(θ).Animportantquantityforthispaperwillbethe(element-wise)Th...