Metric Perspective of Stochastic Optimizers
Date:
This talk gives a high-level overview for the major classes of stochastic optimizers from the perspective of the model parameter space. The talk covers following algorithms:
- Quasi-Newton Methods: Finite-Difference (SGD-QN, AdaDelta, VSGD), Extended Gauss-Newton (KSD, SMD, HF), LBFGS (LBFGS, RES)
- Natural Gradient Type: Natural Gradient, TONGA
- Root Mean Square Type: AdaGrad, RMSProp, Adam