Understanding Recurrent Neural Networks Using Nonequilibrium Response Theory

  • 2021-01-18 17:28:17
  • Soon Hoe Lim
  • 0

Abstract

Recurrent neural networks (RNNs) are brain-inspired models widely used inmachine learning for analyzing sequential data. The present work is acontribution towards a deeper understanding of how RNNs process input signalsusing the response theory from nonequilibrium statistical mechanics. For aclass of continuous-time stochastic RNNs (SRNNs) driven by an input signal, wederive a Volterra type series representation for their output. Thisrepresentation is interpretable and disentangles the input signal from the SRNNarchitecture. The kernels of the series are certain recursively definedcorrelation functions with respect to the unperturbed dynamics that completelydetermine the output. Exploiting connections of this representation and itsimplications to rough paths theory, we identify a universal feature -- theresponse feature, which turns out to be the signature of tensor product of theinput signal and a natural support basis. In particular, we show that SRNNs,with only the weights in the readout layer optimized and the weights in thehidden layer kept fixed and not optimized, can be viewed as kernel machinesoperating on a reproducing kernel Hilbert space associated with the responsefeature.

 

Quick Read (beta)

loading the full paper ...