Abstract
Recurrent neural networks (RNNs) have been widely used for processingsequential data. However, RNNs are commonly difficult to train due to thewell-known gradient vanishing and exploding problems and hard to learnlong-term patterns. Long short-term memory (LSTM) and gated recurrent unit(GRU) were developed to address these problems, but the use of hyperbolictangent and the sigmoid action functions results in gradient decay over layers.Consequently, construction of an efficiently trainable deep network ischallenging. In addition, all the neurons in an RNN layer are entangledtogether and their behaviour is hard to interpret. To address these problems, anew type of RNN, referred to as independently recurrent neural network(IndRNN), is proposed in this paper, where neurons in the same layer areindependent of each other and they are connected across layers. We have shownthat an IndRNN can be easily regulated to prevent the gradient exploding andvanishing problems while allowing the network to learn long-term dependencies.Moreover, an IndRNN can work with non-saturated activation functions such asrelu (rectified linear unit) and be still trained robustly. Multiple IndRNNscan be stacked to construct a network that is deeper than the existing RNNs.Experimental results have shown that the proposed IndRNN is able to processvery long sequences (over 5000 time steps), can be used to construct very deepnetworks (21 layers used in the experiment) and still be trained robustly.Better performances have been achieved on various tasks by using IndRNNscompared with the traditional RNN and LSTM.