Abstract

Given the success of the gated recurrent unit, a natural question is whetherall the gates of the long short-term memory (LSTM) network are necessary.Previous research has shown that the forget gate is one of the most importantgates in the LSTM. Here we show that a forget-gate-only version of the LSTMwith chrono-initialized biases, not only provides computational savings butoutperforms the standard LSTM on multiple benchmark datasets and competes withsome of the best contemporary models. Our proposed network, the JANET, achievesaccuracies of 99% and 92.5% on the MNIST and pMNIST datasets, outperforming thestandard LSTM which yields accuracies of 98.5% and 91%.

 

Quick Read (beta)