Abstract
Character-based Neural Network Language Models (NNLM) have the advantage ofsmaller vocabulary and thus faster training times in comparison to NNLMs basedon multi-character units. However, in low-resource scenarios, both thecharacter and multi-character NNLMs suffer from data sparsity. In suchscenarios, cross-lingual transfer has improved multi-character NNLM performanceby allowing information transfer from a source to the target language. In thesame vein, we propose to use cross-lingual transfer for character NNLMs appliedto low-resource Automatic Speech Recognition (ASR). However, applyingcross-lingual transfer to character NNLMs is not as straightforward. We observethat relatedness of the source language plays an important role incross-lingual pretraining of character NNLMs. We evaluate this aspect on ASRtasks for two target languages: Finnish (with English and Estonian as source)and Swedish (with Danish, Norwegian, and English as source). Prior work hasobserved no difference between using the related or unrelated language formulti-character NNLMs. We, however, show that for character-based NNLMs, onlypretraining with a related language improves the ASR performance, and using anunrelated language may deteriorate it. We also observe that the benefits arelarger when there is much lesser target data than source data.