Complex Evolution Recurrent Neural Networks (ceRNNs)

  • 2019-06-05 18:51:26
  • Izhak Shafran, Tom Bagby, R. J. Skerry-Ryan
  • 25

Abstract

Unitary Evolution Recurrent Neural Networks (uRNNs) have three attractiveproperties: (a) the unitary property, (b) the complex-valued nature, and (c)their efficient linear operators. The literature so far does not address -- howcritical is the unitary property of the model? Furthermore, uRNNs have not beenevaluated on large tasks. To study these shortcomings, we propose the complexevolution Recurrent Neural Networks (ceRNNs), which is similar to uRNNs butdrops the unitary property selectively. On a simple multivariate linearregression task, we illustrate that dropping the constraints improves thelearning trajectory. In copy memory task, ceRNNs and uRNNs perform identically,demonstrating that their superior performance over LSTMs is due tocomplex-valued nature and their linear operators. In a large scale real-worldspeech recognition, we find that pre-pending a uRNN degrades the performance ofour baseline LSTM acoustic models, while pre-pending a ceRNN improves theperformance over the baseline by 0.8% absolute WER.

 

Quick Read (beta)

loading the full paper ...