Abstract
Massively multilingual language models such as multilingual BERT offerstate-of-the-art cross-lingual transfer performance on a range of NLP tasks.However, due to limited capacity and large differences in pretraining datasizes, there is a profound performance gap between resource-rich andresource-poor target languages. The ultimate challenge is dealing withunder-resourced languages not covered at all by the models and written inscripts unseen during pretraining. In this work, we propose a series of noveldata-efficient methods that enable quick and effective adaptation of pretrainedmultilingual models to such low-resource languages and unseen scripts. Relyingon matrix factorization, our methods capitalize on the existing latentknowledge about multiple languages already available in the pretrained model'sembedding matrix. Furthermore, we show that learning of the new dedicatedembedding matrix in the target language can be improved by leveraging a smallnumber of vocabulary items (i.e., the so-called lexically overlapping tokens)shared between mBERT's and target language vocabulary. Our adaptationtechniques offer substantial performance gains for languages with unseenscripts. We also demonstrate that they can yield improvements for low-resourcelanguages written in scripts covered by the pretrained model.