Abstract
Massively multilingual language models such as multilingual BERT (mBERT) andXLM-R offer state-of-the-art cross-lingual transfer performance on a range ofNLP tasks. However, due to their limited capacity and large differences inpretraining data, there is a profound performance gap between resource-rich andresource-poor target languages. The ultimate challenge is dealing withunder-resourced languages not covered at all by the models, which are alsowritten in scripts \textit{unseen} during pretraining. In this work, we proposea series of novel data-efficient methods that enable quick and effectiveadaptation of pretrained multilingual models to such low-resource languages andunseen scripts. Relying on matrix factorization, our proposed methodscapitalize on the existing latent knowledge about multiple languages alreadyavailable in the pretrained model's embedding matrix. Furthermore, we show thatlearning of the new dedicated embedding matrix in the target language can beimproved by leveraging a small number of vocabulary items (i.e., the so-called\textit{lexically overlapping} tokens) shared between mBERT's and targetlanguage vocabulary. Our adaptation techniques offer substantial performancegains for languages with unseen scripts. We also demonstrate that they can alsoyield improvements for low-resource languages written in scripts covered by thepretrained model.