Cross-Lingual Natural Language Generation via Pre-Training

  • 2019-11-22 09:24:46
  • Zewen Chi, Li Dong, Furu Wei, Wenhui Wang, Xian-Ling Mao, Heyan Huang
  • 0

Abstract

In this work we focus on transferring supervision signals of natural languagegeneration (NLG) tasks between multiple languages. We propose to pretrain theencoder and the decoder of a sequence-to-sequence model under both monolingualand cross-lingual settings. The pre-training objective encourages the model torepresent different languages in the shared space, so that we can conductzero-shot cross-lingual transfer. After the pre-training procedure, we usemonolingual data to fine-tune the pre-trained model on downstream NLG tasks.Then the sequence-to-sequence model trained in a single language can bedirectly evaluated beyond that language (i.e., accepting multi-lingual inputand producing multi-lingual output). Experimental results on questiongeneration and abstractive summarization show that our model outperforms themachine-translation-based pipeline methods for zero-shot cross-lingualgeneration. Moreover, cross-lingual transfer improves NLG performance oflow-resource languages by leveraging rich-resource language data. Ourimplementation and data are available at https://github.com/CZWin32768/xnlg.

 

Quick Read (beta)

loading the full paper ...