Abstract

Gradient-based optimization methods are the most popular choice for findinglocal optima for classical minimization and saddle point problems. Here, wehighlight a systemic issue of gradient dynamics that arise for saddle pointproblems, namely the presence of undesired stable stationary points that are nolocal optima. We propose a novel optimization approach that exploits curvatureinformation in order to escape from these undesired stationary points. We provethat different optimization methods, including gradient method and adagrad,equipped with curvature exploitation can escape non-optimal stationary points.We also provide empirical results on common saddle point problems which confirmthe advantage of using curvature exploitation.

 

Quick Read (beta)