Abstract
Neural language modeling (LM) has led to significant improvements in severalapplications, including Automatic Speech Recognition. However, they typicallyrequire large amounts of training data, which is not available for many domainsand languages. In this study, we propose a multilingual neural language modelarchitecture, trained jointly on the domain-specific data of severallow-resource languages. The proposed multilingual LM consists of languagespecific word embeddings in the encoder and decoder, and one language specificLSTM layer, plus two LSTM layers with shared parameters across the languages.This multilingual LM model facilitates transfer learning across the languages,acting as an extra regularizer in very low-resource scenarios. We integrate ourproposed multilingual approach with a state-of-the-art highly-regularizedneural LM, and evaluate on the conversational data domain for four languagesover a range of training data sizes. Compared to monolingual LMs, the resultsshow significant improvements of our proposed multilingual LM when the amountof available training data is limited, indicating the advantages ofcross-lingual parameter sharing in very low-resource language modeling.