Abstract

Associative memory using fast weights is a short-term memory mechanism thatsubstantially improves the memory capacity and time scale of recurrent neuralnetworks (RNNs). As recent studies introduced fast weights only to regularRNNs, it is unknown whether fast weight memory is beneficial to gated RNNs. Inthis work, we report a significant synergy between long short-term memory(LSTM) networks and fast weight associative memories. We show that thiscombination, in learning associative retrieval tasks, results in much fastertraining and lower test error, a performance boost most prominent at highmemory task difficulties.

 

Quick Read (beta)