Transferring Monolingual Model to Low-Resource Language: The Case of Tigrinya

  • 2020-06-19 15:00:02
  • Abrhalei Tela, Abraham Woubie, Ville Hautamaki
  • 0

Abstract

In recent years, transformer models have achieved great success in naturallanguage processing (NLP) tasks. Most of the current state-of-the-art NLPresults are achieved by using monolingual transformer models, where the modelis pre-trained using a single language unlabelled text corpus. Then, the modelis fine-tuned to the specific downstream task. However, the cost ofpre-training a new transformer model is high for most languages. In this work,we propose a cost-effective transfer learning method to adopt a strong sourcelanguage model, trained from a large monolingual corpus to a low-resourcelanguage. Thus, using XLNet language model, we demonstrate competitiveperformance with mBERT and a pre-trained target language model on thecross-lingual sentiment (CLS) dataset and on a new sentiment analysis datasetfor low-resourced language Tigrinya. With only 10k examples of the givenTigrinya sentiment analysis dataset, English XLNet has achieved 78.88% F1-Scoreoutperforming BERT and mBERT by 10% and 7%, respectively. More interestingly,fine-tuning (English) XLNet model on the CLS dataset has promising resultscompared to mBERT and even outperformed mBERT for one dataset of the Japaneselanguage.

 

Quick Read (beta)

loading the full paper ...