Abstract

In this paper, we provide an overview of first-order and second-ordervariants of the gradient descent method that are commonly used in machinelearning. We propose a general framework in which 6 of these variants can beinterpreted as different instances of the same approach. They are the vanillagradient descent, the classical and generalized Gauss-Newton methods, thenatural gradient descent method, the gradient covariance matrix approach, andNewton's method. Besides interpreting these methods within a single framework,we explain their specificities and show under which conditions some of themcoincide.

 

Quick Read (beta)