A Stable High-order Tuner for General Convex Functions

  • 2020-11-19 17:50:53
  • José M. Moreu, Anuradha M. Annaswamy
  • 0

Abstract

Iterative gradient-based algorithms have been increasingly applied for thetraining of a broad variety of machine learning models including largeneural-nets. In particular, momentum-based methods, with accelerated learningguarantees, have received a lot of attention due to their provable guaranteesof fast learning in certain classes of problems and multiple algorithms havebeen derived. However, properties for these methods hold true only for constantregressors. When time-varying regressors occur, which is commonplace in dynamicsystems, many of these momentum-based methods cannot guarantee stability.Recently, a new High-order Tuner (HT) was developed and shown to have 1)stability and asymptotic convergence for time-varying regressors and 2)non-asymptotic accelerated learning guarantees for constant regressors. Theseresults were derived for a linear regression framework producing a quadraticloss function. In this paper, we extend and discuss the results of this same HTfor general convex loss functions. Through the exploitation of convexity andsmoothness definitions, we establish similar stability and asymptoticconvergence guarantees. Additionally we conjecture that the HT has anaccelerated convergence rate. Finally, we provide numerical simulationssupporting the satisfactory behavior of the HT algorithm as well as theconjecture of accelerated learning.

 

Quick Read (beta)

loading the full paper ...