Multi-Source Cross-Lingual Model Transfer: Learning What to Share

  • 2019-06-05 04:04:07
  • Xilun Chen, Ahmed Hassan Awadallah, Hany Hassan, Wei Wang, Claire Cardie
  • 0

Abstract

Modern NLP applications have enjoyed a great boost utilizing neural networksmodels. Such deep neural models, however, are not applicable to most humanlanguages due to the lack of annotated training data for various NLP tasks.Cross-lingual transfer learning (CLTL) is a viable method for building NLPmodels for a low-resource target language by leveraging labeled data from other(source) languages. In this work, we focus on the multilingual transfer settingwhere training data in multiple source languages is leveraged to further boosttarget language performance. Unlike most existing methods that rely only on language-invariant featuresfor CLTL, our approach coherently utilizes both language-invariant andlanguage-specific features at instance level. Our model leverages adversarialnetworks to learn language-invariant features, and mixture-of-experts models todynamically exploit the similarity between the target language and eachindividual source language. This enables our model to learn effectively what toshare between various languages in the multilingual setup. Moreover, whencoupled with unsupervised multilingual embeddings, our model can operate in azero-resource setting where neither target language training data norcross-lingual resources are available. Our model achieves significantperformance gains over prior art, as shown in an extensive set of experimentsover multiple text classification and sequence tagging tasks including alarge-scale industry dataset.

 

Quick Read (beta)

loading the full paper ...