Abstract

In this paper, we provide an overview of first-order and second-ordervariants of the gradient descent methods commonly used in machine learning. Wepropose a general framework in which 6 of these methods can be interpreted asdifferent instances of the same approach. These methods are the vanillagradient descent, the classical and generalized Gauss-Newton methods, thenatural gradient descent method, the gradient covariance matrix approach, andNewton's method. Besides interpreting these methods within a single framework,we explain their specificities and show under which conditions some of themcoincide.

 

Quick Read (beta)