Abstract
Breakthroughs in deep learning and memory networks have made major advancesin natural language understanding. Language is sequential and informationcarried through the sequence can be captured through memory networks. Learningthe sequence is one of the key aspects in learning the language. However,memory networks are not capable of holding infinitely long sequences in theirmemories and are limited by various constraints such as the vanishing orexploding gradient problem. Therefore, natural language understanding modelsare affected when presented with long sequential text. We introduce Long TermMemory network (LTM) to learn from infinitely long sequences. LTM givespriority to the current inputs to allow it to have a high impact. Languagemodeling is an important factor in natural language understanding. LTM wastested in language modeling, which requires long term memory. LTM is tested onPenn Tree bank dataset, Google Billion Word dataset and WikiText-2 dataset. Wecompare LTM with other language models which require long term memory.