Adafactor Adaptive Learning Rates with Sublinear Memory Cost

Adafactor:AdaptiveLearningRateswithSublinearMemoryCostNoamShazeer1MitchellStern12Abstractvectorsummarizingthehistoryofsquaredgradients,usuallyobtainedthroughsummationasinAdagrad(Duchietal.,Insevera...