Abstract

We study the bias of generic optimization methods, including Mirror Descent,Natural Gradient Descent and Steepest Descent with respect to differentpotentials and norms, when optimizing underdetermined linear regression orseparable linear classification problems. We ask the question of whether theglobal minimum (among the many possible global minima) reached by optimizationalgorithms can be characterized in terms of the potential or norm, andindependently of hyperparameter choices such as step size and momentum.

 

Quick Read (beta)