Emerging Cross-lingual Structure in Pretrained Language Models

  • 2019-11-10 06:55:02
  • Shijie Wu, Alexis Conneau, Haoran Li, Luke Zettlemoyer, Veselin Stoyanov
  • 0

Abstract

We study the problem of multilingual masked language modeling, i.e. thetraining of a single model on concatenated text from multiple languages, andpresent a detailed study of several factors that influence why these models areso effective for cross-lingual transfer. We show, contrary to what waspreviously hypothesized, that transfer is possible even when there is no sharedvocabulary across the monolingual corpora and also when the text comes fromvery different domains. The only requirement is that there are some sharedparameters in the top layers of the multi-lingual encoder. To better understandthis result, we also show that representations from independently trainedmodels in different languages can be aligned post-hoc quite effectively,strongly suggesting that, much like for non-contextual word embeddings, thereare universal latent symmetries in the learned embedding spaces. Formultilingual masked language modeling, these symmetries seem to beautomatically discovered and aligned during the joint training process.

 

Quick Read (beta)

loading the full paper ...