A Unified Framework of Online Learning Algorithms for Training Recurrent Neural Networks

  • 2019-07-05 01:49:45
  • Owen Marschall, Kyunghyun Cho, Cristina Savin
  • 28

Abstract

We present a framework for compactly summarizing many recent results inefficient and/or biologically plausible online training of recurrent neuralnetworks (RNN). The framework organizes algorithms according to severalcriteria: (a) past vs. future facing, (b) tensor structure, (c) stochastic vs.deterministic, and (d) closed form vs. numerical. These axes reveal latentconceptual connections among several recent advances in online learning.Furthermore, we provide novel mathematical intuitions for their degree ofsuccess. Testing various algorithms on two synthetic tasks shows thatperformances cluster according to our criteria. Although a similar clusteringis also observed for gradient alignment, alignment with exact methods does notalone explain ultimate performance, especially for stochastic algorithms. Thissuggests the need for better comparison metrics.

 

Quick Read (beta)

loading the full paper ...