Effects of Language Relatedness for Cross-lingual Transfer Learning in Character-Based Language Models

  • 2020-07-22 19:52:34
  • Mittul Singh, Peter Smit, Sami Virpioja, Mikko Kurimo
  • 10

Abstract

Character-based Neural Network Language Models (NNLM) have the advantage ofsmaller vocabulary and thus faster training times in comparison to NNLMs basedon multi-character units. However, in low-resource scenarios, both thecharacter and multi-character NNLMs suffer from data sparsity. In suchscenarios, cross-lingual transfer has improved multi-character NNLM performanceby allowing information transfer from a source to the target language. In thesame vein, we propose to use cross-lingual transfer for character NNLMs appliedto low-resource Automatic Speech Recognition (ASR). However, applyingcross-lingual transfer to character NNLMs is not as straightforward. We observethat relatedness of the source language plays an important role incross-lingual pretraining of character NNLMs. We evaluate this aspect on ASRtasks for two target languages: Finnish (with English and Estonian as source)and Swedish (with Danish, Norwegian, and English as source). Prior work hasobserved no difference between using the related or unrelated language formulti-character NNLMs. We, however, show that for character-based NNLMs, onlypretraining with a related language improves the ASR performance, and using anunrelated language may deteriorate it. We also observe that the benefits arelarger when there is much lesser target data than source data.

 

Quick Read (beta)

loading the full paper ...