Abstract
In this work we focus on transferring supervision signals of natural languagegeneration (NLG) tasks between multiple languages. We propose to pretrain theencoder and the decoder of a sequence-to-sequence model under both monolingualand cross-lingual settings. The pre-training objective encourages the model torepresent different languages in the shared space, so that we can conductzero-shot cross-lingual transfer. After the pre-training procedure, we usemonolingual data to fine-tune the pre-trained model on downstream NLG tasks.Then the sequence-to-sequence model trained in a single language can bedirectly evaluated beyond that language (i.e., accepting multi-lingual inputand producing multi-lingual output). Experimental results on questiongeneration and abstractive summarization show that our model outperforms themachine-translation-based pipeline methods for zero-shot cross-lingualgeneration. Moreover, cross-lingual transfer improves NLG performance oflow-resource languages by leveraging rich-resource language data. Ourimplementation and data are available at https://github.com/CZWin32768/xnlg.