Metric Perspective of Stochastic Optimizers
Date:
In this talk, I explain several major stochastic optimizers from the perspective of the metric, that is the definition of the parameter space of the model. This talk covers algorithms such as
- Quasi-Newton Method Type
- Finite-Difference Method: SGD-QN, AdaDelta, VSGD
- Extended Gauss-Newton: KSD, SMD, HF
- LBFGS: Stochastic LBFGS, RES
- Natural Gradient Type: Natural Gradient, TONGA
- Root Mean Square Type: AdaGrad, RMSProp, Adam