Abstract

Pretrained language models are promising particularly for low-resourcelanguages as they only require unlabelled data. However, training existingmodels requires huge amounts of compute, while pretrained cross-lingual modelsoften underperform on low-resource languages. We propose Multi-lingual languagemodel Fine-Tuning (MultiFiT) to enable practitioners to train and fine-tunelanguage models efficiently in their own language. In addition, we propose azero-shot method using an existing pretrained cross-lingual model. We evaluateour methods on two widely used cross-lingual classification datasets where theyoutperform models pretrained on orders of magnitude more data and compute. Werelease all models and code.

 

Quick Read (beta)