Abstract
Vanilla RNN with ReLU activation have a simple structure that is amenable tosystematic dynamical systems analysis and interpretation, but they suffer fromthe exploding vs. vanishing gradients problem. Recent attempts to retain thissimplicity while alleviating the gradient problem are based on properinitialization schemes or orthogonality/unitary constraints on the RNN'srecurrence matrix, which, however, comes with limitations to its expressivepower with regards to dynamical systems phenomena like chaos ormulti-stability. Here, we instead suggest a regularization scheme that pushespart of the RNN's latent subspace toward a line attractor configuration thatenables long short-term memory and arbitrarily slow time scales. We show thatour approach excels on a number of benchmarks like the sequential MNIST ormultiplication problems, and enables reconstruction of dynamical systems whichharbor widely different time scales.